Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1847 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 5.9454. Min Range: -5.9454

Max Act: 5.9454. Min Act: -0.0000

Data Index: 408315 (C4 (Web Text))

Max Activating Token Index: 33

Text #1

Max Range: 5.9454. Min Range: -5.9454

Max Act: 4.7262. Min Act: -0.0000

Data Index: 103861 (C4 (Web Text))

Max Activating Token Index: 241

Text #2

Max Range: 5.9454. Min Range: -5.9454

Max Act: 4.9429. Min Act: -0.0000

Data Index: 193199 (C4 (Web Text))

Max Activating Token Index: 513

Text #3

Max Range: 5.9454. Min Range: -5.9454

Max Act: 4.3989. Min Act: -0.0000

Data Index: 776499 (C4 (Web Text))

Max Activating Token Index: 987

Text #4

Max Range: 5.9454. Min Range: -5.9454

Max Act: 4.3068. Min Act: -0.0000

Data Index: 259957 (C4 (Web Text))

Max Activating Token Index: 948

Text #5

Max Range: 5.9454. Min Range: -5.9454

Max Act: 4.0954. Min Act: -0.0000

Data Index: 439266 (C4 (Web Text))

Max Activating Token Index: 152

Text #6

Max Range: 5.9454. Min Range: -5.9454

Max Act: 4.0367. Min Act: -0.0000

Data Index: 536873 (C4 (Web Text))

Max Activating Token Index: 700

Text #7

Max Range: 5.9454. Min Range: -5.9454

Max Act: 3.9703. Min Act: -0.0000

Data Index: 1309176 (C4 (Web Text))

Max Activating Token Index: 673

Text #8

Max Range: 5.9454. Min Range: -5.9454

Max Act: 4.1325. Min Act: -0.0000

Data Index: 616315 (C4 (Web Text))

Max Activating Token Index: 765

Text #9

Max Range: 5.9454. Min Range: -5.9454

Max Act: 4.0523. Min Act: -0.0000

Data Index: 689587 (C4 (Web Text))

Max Activating Token Index: 461

Text #10

Max Range: 5.9454. Min Range: -5.9454

Max Act: 3.6888. Min Act: -0.0000

Data Index: 31962 (C4 (Web Text))

Max Activating Token Index: 286

Text #11

Max Range: 5.9454. Min Range: -5.9454

Max Act: 3.4895. Min Act: -0.0000

Data Index: 737235 (C4 (Web Text))

Max Activating Token Index: 629

Text #12

Max Range: 5.9454. Min Range: -5.9454

Max Act: 3.3774. Min Act: -0.0000

Data Index: 359657 (C4 (Web Text))

Max Activating Token Index: 740

Text #13

Max Range: 5.9454. Min Range: -5.9454

Max Act: 3.6492. Min Act: -0.0000

Data Index: 784305 (C4 (Web Text))

Max Activating Token Index: 722

Text #14

Max Range: 5.9454. Min Range: -5.9454

Max Act: 3.4793. Min Act: -0.0000

Data Index: 222424 (C4 (Web Text))

Max Activating Token Index: 510

Text #15

Max Range: 5.9454. Min Range: -5.9454

Max Act: 3.4568. Min Act: -0.0000

Data Index: 1153039 (C4 (Web Text))

Max Activating Token Index: 973

Text #16

Max Range: 5.9454. Min Range: -5.9454

Max Act: 3.0354. Min Act: -0.0000

Data Index: 247149 (C4 (Web Text))

Max Activating Token Index: 692

Text #17

Max Range: 5.9454. Min Range: -5.9454

Max Act: 3.4424. Min Act: -0.0000

Data Index: 489976 (C4 (Web Text))

Max Activating Token Index: 944

Text #18

Max Range: 5.9454. Min Range: -5.9454

Max Act: 3.4160. Min Act: -0.0000

Data Index: 586659 (C4 (Web Text))

Max Activating Token Index: 1014

Text #19

Max Range: 5.9454. Min Range: -5.9454

Max Act: 3.2593. Min Act: -0.0000

Data Index: 348003 (C4 (Web Text))

Max Activating Token Index: 175