Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3962 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.0717. Min Range: -4.0717

Max Act: 4.0717. Min Act: -0.0000

Data Index: 90084 (C4 (Web Text))

Max Activating Token Index: 34

Text #1

Max Range: 4.0717. Min Range: -4.0717

Max Act: 2.7227. Min Act: -0.0000

Data Index: 1338597 (C4 (Web Text))

Max Activating Token Index: 913

Text #2

Max Range: 4.0717. Min Range: -4.0717

Max Act: 2.8224. Min Act: -0.0000

Data Index: 1174654 (C4 (Web Text))

Max Activating Token Index: 17

Text #3

Max Range: 4.0717. Min Range: -4.0717

Max Act: 2.9698. Min Act: -0.0000

Data Index: 338029 (C4 (Web Text))

Max Activating Token Index: 701

Text #4

Max Range: 4.0717. Min Range: -4.0717

Max Act: 2.7305. Min Act: -0.0000

Data Index: 424176 (C4 (Web Text))

Max Activating Token Index: 315

Text #5

Max Range: 4.0717. Min Range: -4.0717

Max Act: 2.3172. Min Act: -0.0000

Data Index: 542692 (C4 (Web Text))

Max Activating Token Index: 793

Text #6

Max Range: 4.0717. Min Range: -4.0717

Max Act: 2.6453. Min Act: -0.0000

Data Index: 163246 (C4 (Web Text))

Max Activating Token Index: 765

Text #7

Max Range: 4.0717. Min Range: -4.0717

Max Act: 2.5031. Min Act: -0.0000

Data Index: 406925 (C4 (Web Text))

Max Activating Token Index: 264

Text #8

Max Range: 4.0717. Min Range: -4.0717

Max Act: 2.2682. Min Act: -0.0000

Data Index: 1231628 (C4 (Web Text))

Max Activating Token Index: 158

Text #9

Max Range: 4.0717. Min Range: -4.0717

Max Act: 2.3340. Min Act: -0.0000

Data Index: 487251 (C4 (Web Text))

Max Activating Token Index: 744

Text #10

Max Range: 4.0717. Min Range: -4.0717

Max Act: 1.9082. Min Act: -0.0000

Data Index: 753907 (C4 (Web Text))

Max Activating Token Index: 708

Text #11

Max Range: 4.0717. Min Range: -4.0717

Max Act: 2.1194. Min Act: -0.0000

Data Index: 986474 (C4 (Web Text))

Max Activating Token Index: 532

Text #12

Max Range: 4.0717. Min Range: -4.0717

Max Act: 2.0781. Min Act: -0.0000

Data Index: 1196786 (C4 (Web Text))

Max Activating Token Index: 396

Text #13

Max Range: 4.0717. Min Range: -4.0717

Max Act: 1.9170. Min Act: -0.0000

Data Index: 741191 (C4 (Web Text))

Max Activating Token Index: 515

Text #14

Max Range: 4.0717. Min Range: -4.0717

Max Act: 1.9049. Min Act: -0.0000

Data Index: 534557 (C4 (Web Text))

Max Activating Token Index: 544

Text #15

Max Range: 4.0717. Min Range: -4.0717

Max Act: 1.9978. Min Act: -0.0000

Data Index: 185952 (C4 (Web Text))

Max Activating Token Index: 219

Text #16

Max Range: 4.0717. Min Range: -4.0717

Max Act: 2.2905. Min Act: -0.0000

Data Index: 566444 (C4 (Web Text))

Max Activating Token Index: 261

Text #17

Max Range: 4.0717. Min Range: -4.0717

Max Act: 1.7077. Min Act: -0.0000

Data Index: 553078 (C4 (Web Text))

Max Activating Token Index: 821

Text #18

Max Range: 4.0717. Min Range: -4.0717

Max Act: 1.9789. Min Act: -0.0000

Data Index: 789790 (C4 (Web Text))

Max Activating Token Index: 129

Text #19

Max Range: 4.0717. Min Range: -4.0717

Max Act: 1.7163. Min Act: -0.0000

Data Index: 861603 (C4 (Web Text))

Max Activating Token Index: 9