Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4456 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.7209. Min Range: -4.7209

Max Act: 4.7209. Min Act: -0.0000

Data Index: 998242 (C4 (Web Text))

Max Activating Token Index: 52

Text #1

Max Range: 4.7209. Min Range: -4.7209

Max Act: 4.2692. Min Act: -0.0000

Data Index: 1071366 (C4 (Web Text))

Max Activating Token Index: 345

Text #2

Max Range: 4.7209. Min Range: -4.7209

Max Act: 3.6965. Min Act: -0.0000

Data Index: 1320029 (C4 (Web Text))

Max Activating Token Index: 978

Text #3

Max Range: 4.7209. Min Range: -4.7209

Max Act: 3.7586. Min Act: -0.0000

Data Index: 1189510 (C4 (Web Text))

Max Activating Token Index: 599

Text #4

Max Range: 4.7209. Min Range: -4.7209

Max Act: 3.4548. Min Act: -0.0000

Data Index: 710541 (C4 (Web Text))

Max Activating Token Index: 644

Text #5

Max Range: 4.7209. Min Range: -4.7209

Max Act: 3.3787. Min Act: -0.0000

Data Index: 1063521 (C4 (Web Text))

Max Activating Token Index: 552

Text #6

Max Range: 4.7209. Min Range: -4.7209

Max Act: 3.2202. Min Act: -0.0000

Data Index: 243213 (C4 (Web Text))

Max Activating Token Index: 740

Text #7

Max Range: 4.7209. Min Range: -4.7209

Max Act: 3.1655. Min Act: -0.0000

Data Index: 447231 (C4 (Web Text))

Max Activating Token Index: 934

Text #8

Max Range: 4.7209. Min Range: -4.7209

Max Act: 3.1307. Min Act: -0.0000

Data Index: 465286 (C4 (Web Text))

Max Activating Token Index: 1022

Text #9

Max Range: 4.7209. Min Range: -4.7209

Max Act: 3.0267. Min Act: -0.0000

Data Index: 584562 (C4 (Web Text))

Max Activating Token Index: 541

Text #10

Max Range: 4.7209. Min Range: -4.7209

Max Act: 2.8791. Min Act: -0.0000

Data Index: 907582 (C4 (Web Text))

Max Activating Token Index: 254

Text #11

Max Range: 4.7209. Min Range: -4.7209

Max Act: 2.5772. Min Act: -0.0000

Data Index: 998050 (C4 (Web Text))

Max Activating Token Index: 120

Text #12

Max Range: 4.7209. Min Range: -4.7209

Max Act: 2.4278. Min Act: -0.0000

Data Index: 873370 (C4 (Web Text))

Max Activating Token Index: 289

Text #13

Max Range: 4.7209. Min Range: -4.7209

Max Act: 2.6979. Min Act: -0.0000

Data Index: 678823 (C4 (Web Text))

Max Activating Token Index: 32

Text #14

Max Range: 4.7209. Min Range: -4.7209

Max Act: 2.3209. Min Act: -0.0000

Data Index: 47431 (C4 (Web Text))

Max Activating Token Index: 801

Text #15

Max Range: 4.7209. Min Range: -4.7209

Max Act: 2.4005. Min Act: -0.0000

Data Index: 1057462 (C4 (Web Text))

Max Activating Token Index: 390

Text #16

Max Range: 4.7209. Min Range: -4.7209

Max Act: 2.4952. Min Act: -0.0000

Data Index: 608 (C4 (Web Text))

Max Activating Token Index: 66

Text #17

Max Range: 4.7209. Min Range: -4.7209

Max Act: 2.5914. Min Act: -0.0000

Data Index: 601626 (C4 (Web Text))

Max Activating Token Index: 657

Text #18

Max Range: 4.7209. Min Range: -4.7209

Max Act: 2.2288. Min Act: -0.0000

Data Index: 980788 (C4 (Web Text))

Max Activating Token Index: 801

Text #19

Max Range: 4.7209. Min Range: -4.7209

Max Act: 2.2214. Min Act: -0.0000

Data Index: 789095 (C4 (Web Text))

Max Activating Token Index: 811