Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3709 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.8570. Min Act: -0.0000

Data Index: 298079 (C4 (Web Text))

Max Activating Token Index: 823

Text #1

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.9250. Min Act: -0.0000

Data Index: 1046501 (C4 (Web Text))

Max Activating Token Index: 587

Text #2

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.8514. Min Act: -0.0000

Data Index: 1350639 (C4 (Web Text))

Max Activating Token Index: 816

Text #3

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.7831. Min Act: -0.0000

Data Index: 340516 (C4 (Web Text))

Max Activating Token Index: 164

Text #4

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.7797. Min Act: -0.0000

Data Index: 85312 (C4 (Web Text))

Max Activating Token Index: 252

Text #5

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.7337. Min Act: -0.0000

Data Index: 892738 (C4 (Web Text))

Max Activating Token Index: 372

Text #6

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.7749. Min Act: -0.0000

Data Index: 70871 (C4 (Web Text))

Max Activating Token Index: 388

Text #7

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.7091. Min Act: -0.0000

Data Index: 158560 (C4 (Web Text))

Max Activating Token Index: 189

Text #8

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.6577. Min Act: -0.0000

Data Index: 546576 (C4 (Web Text))

Max Activating Token Index: 463

Text #9

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.7174. Min Act: -0.0000

Data Index: 272992 (C4 (Web Text))

Max Activating Token Index: 415

Text #10

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.7359. Min Act: -0.0000

Data Index: 402614 (C4 (Web Text))

Max Activating Token Index: 92

Text #11

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.6618. Min Act: -0.0000

Data Index: 366586 (C4 (Web Text))

Max Activating Token Index: 976

Text #12

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.6939. Min Act: -0.0000

Data Index: 12837 (C4 (Web Text))

Max Activating Token Index: 61

Text #13

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.6387. Min Act: -0.0000

Data Index: 1004127 (C4 (Web Text))

Max Activating Token Index: 109

Text #14

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.6867. Min Act: -0.0000

Data Index: 1198934 (C4 (Web Text))

Max Activating Token Index: 114

Text #15

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.6275. Min Act: -0.0000

Data Index: 92033 (C4 (Web Text))

Max Activating Token Index: 277

Text #16

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.6569. Min Act: -0.0000

Data Index: 362890 (C4 (Web Text))

Max Activating Token Index: 641

Text #17

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.6023. Min Act: -0.0000

Data Index: 788694 (C4 (Web Text))

Max Activating Token Index: 250

Text #18

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.6211. Min Act: -0.0000

Data Index: 166711 (C4 (Web Text))

Max Activating Token Index: 615

Text #19

Max Range: 0.9250. Min Range: -0.9250

Max Act: 0.6107. Min Act: -0.0000

Data Index: 1153196 (C4 (Web Text))

Max Activating Token Index: 698