Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 380 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6926. Min Act: -0.0001

Data Index: 1224587 (C4 (Web Text))

Max Activating Token Index: 87

Text #1

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6941. Min Act: -0.0001

Data Index: 835701 (C4 (Web Text))

Max Activating Token Index: 158

Text #2

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6633. Min Act: -0.0001

Data Index: 662976 (C4 (Web Text))

Max Activating Token Index: 1023

Text #3

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6710. Min Act: -0.0001

Data Index: 947777 (C4 (Web Text))

Max Activating Token Index: 145

Text #4

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6669. Min Act: -0.0001

Data Index: 1181789 (C4 (Web Text))

Max Activating Token Index: 1023

Text #5

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6614. Min Act: -0.0001

Data Index: 791578 (C4 (Web Text))

Max Activating Token Index: 95

Text #6

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6679. Min Act: -0.0001

Data Index: 1231212 (C4 (Web Text))

Max Activating Token Index: 3

Text #7

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6497. Min Act: -0.0001

Data Index: 813869 (C4 (Web Text))

Max Activating Token Index: 3

Text #8

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6573. Min Act: -0.0001

Data Index: 877347 (C4 (Web Text))

Max Activating Token Index: 29

Text #9

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6421. Min Act: -0.0001

Data Index: 1236054 (C4 (Web Text))

Max Activating Token Index: 35

Text #10

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6499. Min Act: -0.0001

Data Index: 467567 (C4 (Web Text))

Max Activating Token Index: 1023

Text #11

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6344. Min Act: -0.0001

Data Index: 1061324 (C4 (Web Text))

Max Activating Token Index: 1023

Text #12

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6323. Min Act: -0.0001

Data Index: 136325 (C4 (Web Text))

Max Activating Token Index: 1023

Text #13

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6294. Min Act: -0.0001

Data Index: 264500 (C4 (Web Text))

Max Activating Token Index: 4

Text #14

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6386. Min Act: -0.0001

Data Index: 964121 (C4 (Web Text))

Max Activating Token Index: 4

Text #15

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6314. Min Act: -0.0001

Data Index: 209360 (C4 (Web Text))

Max Activating Token Index: 821

Text #16

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6428. Min Act: -0.0001

Data Index: 803562 (C4 (Web Text))

Max Activating Token Index: 4

Text #17

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6403. Min Act: -0.0001

Data Index: 1033187 (C4 (Web Text))

Max Activating Token Index: 1023

Text #18

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6275. Min Act: -0.0001

Data Index: 197131 (C4 (Web Text))

Max Activating Token Index: 1023

Text #19

Max Range: 0.6941. Min Range: -0.6941

Max Act: 0.6320. Min Act: -0.0001

Data Index: 130117 (C4 (Web Text))

Max Activating Token Index: 1023