Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4587 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.9808. Min Act: -0.0000

Data Index: 1174614 (C4 (Web Text))

Max Activating Token Index: 825

Text #1

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.9935. Min Act: -0.0000

Data Index: 123867 (C4 (Web Text))

Max Activating Token Index: 788

Text #2

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.8527. Min Act: -0.0000

Data Index: 415913 (C4 (Web Text))

Max Activating Token Index: 510

Text #3

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.8538. Min Act: -0.0000

Data Index: 364673 (C4 (Web Text))

Max Activating Token Index: 159

Text #4

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.8040. Min Act: -0.0000

Data Index: 686540 (C4 (Web Text))

Max Activating Token Index: 347

Text #5

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.7917. Min Act: -0.0000

Data Index: 140138 (C4 (Web Text))

Max Activating Token Index: 764

Text #6

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.7478. Min Act: -0.0000

Data Index: 153982 (C4 (Web Text))

Max Activating Token Index: 340

Text #7

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.7424. Min Act: -0.0000

Data Index: 477333 (C4 (Web Text))

Max Activating Token Index: 481

Text #8

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.7260. Min Act: -0.0000

Data Index: 11065 (C4 (Web Text))

Max Activating Token Index: 186

Text #9

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.7212. Min Act: -0.0000

Data Index: 896608 (C4 (Web Text))

Max Activating Token Index: 169

Text #10

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.6936. Min Act: -0.0000

Data Index: 856720 (C4 (Web Text))

Max Activating Token Index: 59

Text #11

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.6794. Min Act: -0.0000

Data Index: 751811 (C4 (Web Text))

Max Activating Token Index: 164

Text #12

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.6690. Min Act: -0.0000

Data Index: 878824 (C4 (Web Text))

Max Activating Token Index: 210

Text #13

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.6905. Min Act: -0.0000

Data Index: 700166 (C4 (Web Text))

Max Activating Token Index: 54

Text #14

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.6872. Min Act: -0.0000

Data Index: 1156176 (C4 (Web Text))

Max Activating Token Index: 312

Text #15

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.6600. Min Act: -0.0000

Data Index: 595290 (C4 (Web Text))

Max Activating Token Index: 219

Text #16

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.7008. Min Act: -0.0000

Data Index: 463603 (C4 (Web Text))

Max Activating Token Index: 268

Text #17

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.7217. Min Act: -0.0000

Data Index: 1342866 (C4 (Web Text))

Max Activating Token Index: 259

Text #18

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.6965. Min Act: -0.0000

Data Index: 1281682 (C4 (Web Text))

Max Activating Token Index: 55

Text #19

Max Range: 0.9935. Min Range: -0.9935

Max Act: 0.6538. Min Act: -0.0000

Data Index: 1123525 (C4 (Web Text))

Max Activating Token Index: 158