Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3970 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0466. Min Range: -1.0466

Max Act: 1.0466. Min Act: -0.0000

Data Index: 733182 (C4 (Web Text))

Max Activating Token Index: 222

Text #1

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.9383. Min Act: -0.0000

Data Index: 1347093 (C4 (Web Text))

Max Activating Token Index: 94

Text #2

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.9714. Min Act: -0.0000

Data Index: 799073 (C4 (Web Text))

Max Activating Token Index: 49

Text #3

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.9073. Min Act: -0.0000

Data Index: 466732 (C4 (Web Text))

Max Activating Token Index: 43

Text #4

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.9231. Min Act: -0.0000

Data Index: 819023 (C4 (Web Text))

Max Activating Token Index: 770

Text #5

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.9374. Min Act: -0.0000

Data Index: 1109811 (C4 (Web Text))

Max Activating Token Index: 195

Text #6

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.9110. Min Act: -0.0000

Data Index: 659760 (C4 (Web Text))

Max Activating Token Index: 942

Text #7

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.9203. Min Act: -0.0000

Data Index: 725735 (C4 (Web Text))

Max Activating Token Index: 620

Text #8

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.8628. Min Act: -0.0000

Data Index: 861058 (C4 (Web Text))

Max Activating Token Index: 611

Text #9

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.8566. Min Act: -0.0000

Data Index: 30329 (C4 (Web Text))

Max Activating Token Index: 472

Text #10

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.8514. Min Act: -0.0000

Data Index: 1272440 (C4 (Web Text))

Max Activating Token Index: 321

Text #11

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.8784. Min Act: -0.0000

Data Index: 269333 (C4 (Web Text))

Max Activating Token Index: 635

Text #12

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.8707. Min Act: -0.0000

Data Index: 114368 (C4 (Web Text))

Max Activating Token Index: 1014

Text #13

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.8494. Min Act: -0.0000

Data Index: 119721 (C4 (Web Text))

Max Activating Token Index: 881

Text #14

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.8721. Min Act: -0.0000

Data Index: 822262 (C4 (Web Text))

Max Activating Token Index: 82

Text #15

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.8747. Min Act: -0.0000

Data Index: 1198864 (C4 (Web Text))

Max Activating Token Index: 537

Text #16

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.7995. Min Act: -0.0000

Data Index: 533827 (C4 (Web Text))

Max Activating Token Index: 28

Text #17

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.8844. Min Act: -0.0000

Data Index: 410669 (C4 (Web Text))

Max Activating Token Index: 94

Text #18

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.8224. Min Act: -0.0000

Data Index: 1202344 (C4 (Web Text))

Max Activating Token Index: 23

Text #19

Max Range: 1.0466. Min Range: -1.0466

Max Act: 0.7926. Min Act: -0.0000

Data Index: 813244 (C4 (Web Text))

Max Activating Token Index: 343