Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 630 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1414. Min Range: -2.1414

Max Act: 2.1414. Min Act: -0.0000

Data Index: 392612 (C4 (Web Text))

Max Activating Token Index: 144

Text #1

Max Range: 2.1414. Min Range: -2.1414

Max Act: 2.0122. Min Act: -0.0000

Data Index: 140825 (C4 (Web Text))

Max Activating Token Index: 872

Text #2

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.6125. Min Act: -0.0000

Data Index: 1060396 (C4 (Web Text))

Max Activating Token Index: 938

Text #3

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.7600. Min Act: -0.0000

Data Index: 1094209 (C4 (Web Text))

Max Activating Token Index: 693

Text #4

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.6747. Min Act: -0.0000

Data Index: 726767 (C4 (Web Text))

Max Activating Token Index: 193

Text #5

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.7142. Min Act: -0.0000

Data Index: 346261 (C4 (Web Text))

Max Activating Token Index: 752

Text #6

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.6050. Min Act: -0.0000

Data Index: 61169 (C4 (Web Text))

Max Activating Token Index: 64

Text #7

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.6802. Min Act: -0.0000

Data Index: 362110 (C4 (Web Text))

Max Activating Token Index: 504

Text #8

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.6821. Min Act: -0.0000

Data Index: 931610 (C4 (Web Text))

Max Activating Token Index: 406

Text #9

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.6801. Min Act: -0.0000

Data Index: 732964 (C4 (Web Text))

Max Activating Token Index: 583

Text #10

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.6729. Min Act: -0.0000

Data Index: 625863 (C4 (Web Text))

Max Activating Token Index: 164

Text #11

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.6697. Min Act: -0.0000

Data Index: 955733 (C4 (Web Text))

Max Activating Token Index: 828

Text #12

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.5606. Min Act: -0.0000

Data Index: 183771 (C4 (Web Text))

Max Activating Token Index: 936

Text #13

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.6485. Min Act: -0.0000

Data Index: 535748 (C4 (Web Text))

Max Activating Token Index: 44

Text #14

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.5524. Min Act: -0.0000

Data Index: 216426 (C4 (Web Text))

Max Activating Token Index: 844

Text #15

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.6419. Min Act: -0.0000

Data Index: 873254 (C4 (Web Text))

Max Activating Token Index: 300

Text #16

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.6333. Min Act: -0.0000

Data Index: 204675 (C4 (Web Text))

Max Activating Token Index: 869

Text #17

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.4603. Min Act: -0.0000

Data Index: 1254703 (C4 (Web Text))

Max Activating Token Index: 731

Text #18

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.6251. Min Act: -0.0000

Data Index: 4423 (C4 (Web Text))

Max Activating Token Index: 946

Text #19

Max Range: 2.1414. Min Range: -2.1414

Max Act: 1.5287. Min Act: -0.0000

Data Index: 569895 (C4 (Web Text))

Max Activating Token Index: 717