Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 411 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.2156. Min Act: -0.0000

Data Index: 493560 (C4 (Web Text))

Max Activating Token Index: 524

Text #1

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.2493. Min Act: -0.0000

Data Index: 185958 (C4 (Web Text))

Max Activating Token Index: 837

Text #2

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.3099. Min Act: -0.0000

Data Index: 188960 (C4 (Web Text))

Max Activating Token Index: 816

Text #3

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.2394. Min Act: -0.0000

Data Index: 782868 (C4 (Web Text))

Max Activating Token Index: 80

Text #4

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.1882. Min Act: -0.0000

Data Index: 513903 (C4 (Web Text))

Max Activating Token Index: 869

Text #5

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.1737. Min Act: -0.0000

Data Index: 123955 (C4 (Web Text))

Max Activating Token Index: 30

Text #6

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.1323. Min Act: -0.0000

Data Index: 1356670 (C4 (Web Text))

Max Activating Token Index: 637

Text #7

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.1359. Min Act: 0.0000

Data Index: 949924 (C4 (Web Text))

Max Activating Token Index: 290

Text #8

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.1530. Min Act: -0.0000

Data Index: 1069105 (C4 (Web Text))

Max Activating Token Index: 244

Text #9

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.0742. Min Act: -0.0000

Data Index: 682656 (C4 (Web Text))

Max Activating Token Index: 398

Text #10

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.0581. Min Act: -0.0000

Data Index: 442330 (C4 (Web Text))

Max Activating Token Index: 572

Text #11

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.0477. Min Act: -0.0000

Data Index: 560930 (C4 (Web Text))

Max Activating Token Index: 796

Text #12

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.0337. Min Act: -0.0000

Data Index: 1018114 (C4 (Web Text))

Max Activating Token Index: 201

Text #13

Max Range: 1.3099. Min Range: -1.3099

Max Act: 0.9708. Min Act: -0.0000

Data Index: 296420 (C4 (Web Text))

Max Activating Token Index: 938

Text #14

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.0222. Min Act: 0.0000

Data Index: 1278171 (C4 (Web Text))

Max Activating Token Index: 186

Text #15

Max Range: 1.3099. Min Range: -1.3099

Max Act: 0.9637. Min Act: -0.0000

Data Index: 1055951 (C4 (Web Text))

Max Activating Token Index: 271

Text #16

Max Range: 1.3099. Min Range: -1.3099

Max Act: 0.9645. Min Act: -0.0000

Data Index: 780778 (C4 (Web Text))

Max Activating Token Index: 914

Text #17

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.0170. Min Act: -0.0000

Data Index: 224855 (C4 (Web Text))

Max Activating Token Index: 464

Text #18

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.0087. Min Act: -0.0000

Data Index: 397933 (C4 (Web Text))

Max Activating Token Index: 830

Text #19

Max Range: 1.3099. Min Range: -1.3099

Max Act: 1.0115. Min Act: -0.0000

Data Index: 402703 (C4 (Web Text))

Max Activating Token Index: 492