Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2273 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.2565. Min Act: -0.0001

Data Index: 1171492 (C4 (Web Text))

Max Activating Token Index: 100

Text #1

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.2486. Min Act: -0.0001

Data Index: 738401 (C4 (Web Text))

Max Activating Token Index: 680

Text #2

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.2307. Min Act: -0.0001

Data Index: 1029332 (C4 (Web Text))

Max Activating Token Index: 875

Text #3

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.2153. Min Act: -0.0001

Data Index: 923171 (C4 (Web Text))

Max Activating Token Index: 610

Text #4

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.2110. Min Act: -0.0001

Data Index: 313749 (C4 (Web Text))

Max Activating Token Index: 808

Text #5

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.1865. Min Act: -0.0001

Data Index: 880730 (C4 (Web Text))

Max Activating Token Index: 843

Text #6

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.2039. Min Act: -0.0001

Data Index: 24077 (C4 (Web Text))

Max Activating Token Index: 516

Text #7

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.1881. Min Act: -0.0001

Data Index: 1300531 (C4 (Web Text))

Max Activating Token Index: 717

Text #8

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.1959. Min Act: -0.0001

Data Index: 813432 (C4 (Web Text))

Max Activating Token Index: 484

Text #9

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.2099. Min Act: -0.0001

Data Index: 333672 (C4 (Web Text))

Max Activating Token Index: 528

Text #10

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.2008. Min Act: -0.0001

Data Index: 284121 (C4 (Web Text))

Max Activating Token Index: 827

Text #11

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.2056. Min Act: -0.0001

Data Index: 560659 (C4 (Web Text))

Max Activating Token Index: 504

Text #12

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.2112. Min Act: -0.0001

Data Index: 1289087 (C4 (Web Text))

Max Activating Token Index: 816

Text #13

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.1844. Min Act: -0.0001

Data Index: 217135 (C4 (Web Text))

Max Activating Token Index: 563

Text #14

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.1883. Min Act: -0.0001

Data Index: 129683 (C4 (Web Text))

Max Activating Token Index: 1009

Text #15

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.1797. Min Act: -0.0001

Data Index: 839875 (C4 (Web Text))

Max Activating Token Index: 778

Text #16

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.1698. Min Act: -0.0001

Data Index: 1197119 (C4 (Web Text))

Max Activating Token Index: 868

Text #17

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.1831. Min Act: -0.0001

Data Index: 429429 (C4 (Web Text))

Max Activating Token Index: 734

Text #18

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.1822. Min Act: -0.0001

Data Index: 1005310 (C4 (Web Text))

Max Activating Token Index: 77

Text #19

Max Range: 0.2565. Min Range: -0.2565

Max Act: 0.1862. Min Act: -0.0001

Data Index: 46729 (C4 (Web Text))

Max Activating Token Index: 846