Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5031 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.6896. Min Range: -3.6896

Max Act: 3.6896. Min Act: -0.0000

Data Index: 725399 (C4 (Web Text))

Max Activating Token Index: 891

Text #1

Max Range: 3.6896. Min Range: -3.6896

Max Act: 3.2521. Min Act: -0.0000

Data Index: 690857 (C4 (Web Text))

Max Activating Token Index: 185

Text #2

Max Range: 3.6896. Min Range: -3.6896

Max Act: 3.4408. Min Act: -0.0000

Data Index: 285296 (C4 (Web Text))

Max Activating Token Index: 331

Text #3

Max Range: 3.6896. Min Range: -3.6896

Max Act: 2.7899. Min Act: -0.0000

Data Index: 113209 (C4 (Web Text))

Max Activating Token Index: 45

Text #4

Max Range: 3.6896. Min Range: -3.6896

Max Act: 3.1075. Min Act: -0.0000

Data Index: 702726 (C4 (Web Text))

Max Activating Token Index: 956

Text #5

Max Range: 3.6896. Min Range: -3.6896

Max Act: 3.1194. Min Act: -0.0000

Data Index: 757899 (C4 (Web Text))

Max Activating Token Index: 907

Text #6

Max Range: 3.6896. Min Range: -3.6896

Max Act: 2.6550. Min Act: -0.0000

Data Index: 447351 (C4 (Web Text))

Max Activating Token Index: 866

Text #7

Max Range: 3.6896. Min Range: -3.6896

Max Act: 3.1827. Min Act: -0.0000

Data Index: 1165039 (C4 (Web Text))

Max Activating Token Index: 569

Text #8

Max Range: 3.6896. Min Range: -3.6896

Max Act: 2.1008. Min Act: -0.0000

Data Index: 422464 (C4 (Web Text))

Max Activating Token Index: 720

Text #9

Max Range: 3.6896. Min Range: -3.6896

Max Act: 3.1818. Min Act: -0.0000

Data Index: 509463 (C4 (Web Text))

Max Activating Token Index: 145

Text #10

Max Range: 3.6896. Min Range: -3.6896

Max Act: 3.1248. Min Act: -0.0000

Data Index: 1064049 (C4 (Web Text))

Max Activating Token Index: 888

Text #11

Max Range: 3.6896. Min Range: -3.6896

Max Act: 2.7990. Min Act: -0.0000

Data Index: 1305057 (C4 (Web Text))

Max Activating Token Index: 657

Text #12

Max Range: 3.6896. Min Range: -3.6896

Max Act: 2.7345. Min Act: -0.0000

Data Index: 1097809 (C4 (Web Text))

Max Activating Token Index: 856

Text #13

Max Range: 3.6896. Min Range: -3.6896

Max Act: 2.6940. Min Act: -0.0000

Data Index: 450390 (C4 (Web Text))

Max Activating Token Index: 652

Text #14

Max Range: 3.6896. Min Range: -3.6896

Max Act: 2.6699. Min Act: -0.0000

Data Index: 969113 (C4 (Web Text))

Max Activating Token Index: 634

Text #15

Max Range: 3.6896. Min Range: -3.6896

Max Act: 2.0326. Min Act: -0.0000

Data Index: 753782 (C4 (Web Text))

Max Activating Token Index: 724

Text #16

Max Range: 3.6896. Min Range: -3.6896

Max Act: 2.7700. Min Act: -0.0000

Data Index: 501381 (C4 (Web Text))

Max Activating Token Index: 6

Text #17

Max Range: 3.6896. Min Range: -3.6896

Max Act: 2.2217. Min Act: -0.0000

Data Index: 783120 (C4 (Web Text))

Max Activating Token Index: 825

Text #18

Max Range: 3.6896. Min Range: -3.6896

Max Act: 2.6048. Min Act: -0.0000

Data Index: 367781 (C4 (Web Text))

Max Activating Token Index: 727

Text #19

Max Range: 3.6896. Min Range: -3.6896

Max Act: 2.6889. Min Act: -0.0000

Data Index: 593992 (C4 (Web Text))

Max Activating Token Index: 864