Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3118 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.9826. Min Act: -0.0000

Data Index: 277032 (C4 (Web Text))

Max Activating Token Index: 203

Text #1

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.6482. Min Act: -0.0000

Data Index: 524750 (C4 (Web Text))

Max Activating Token Index: 770

Text #2

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.8915. Min Act: -0.0001

Data Index: 676611 (C4 (Web Text))

Max Activating Token Index: 644

Text #3

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.7975. Min Act: -0.0000

Data Index: 639491 (C4 (Web Text))

Max Activating Token Index: 619

Text #4

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.1775. Min Act: -0.0000

Data Index: 1016643 (C4 (Web Text))

Max Activating Token Index: 137

Text #5

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.1569. Min Act: -0.0000

Data Index: 251531 (C4 (Web Text))

Max Activating Token Index: 331

Text #6

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.4903. Min Act: -0.0000

Data Index: 790047 (C4 (Web Text))

Max Activating Token Index: 584

Text #7

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.4352. Min Act: -0.0000

Data Index: 444237 (C4 (Web Text))

Max Activating Token Index: 908

Text #8

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.4158. Min Act: -0.0000

Data Index: 240405 (C4 (Web Text))

Max Activating Token Index: 154

Text #9

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.4270. Min Act: -0.0000

Data Index: 750419 (C4 (Web Text))

Max Activating Token Index: 590

Text #10

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.2671. Min Act: -0.0000

Data Index: 513334 (C4 (Web Text))

Max Activating Token Index: 178

Text #11

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.4220. Min Act: -0.0000

Data Index: 211044 (C4 (Web Text))

Max Activating Token Index: 295

Text #12

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.4675. Min Act: -0.0000

Data Index: 357521 (C4 (Web Text))

Max Activating Token Index: 116

Text #13

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.5025. Min Act: -0.0001

Data Index: 963013 (C4 (Web Text))

Max Activating Token Index: 143

Text #14

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.4152. Min Act: -0.0001

Data Index: 181452 (C4 (Web Text))

Max Activating Token Index: 706

Text #15

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.4490. Min Act: -0.0000

Data Index: 1082908 (C4 (Web Text))

Max Activating Token Index: 854

Text #16

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.4367. Min Act: -0.0000

Data Index: 484950 (C4 (Web Text))

Max Activating Token Index: 563

Text #17

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.7155. Min Act: -0.0000

Data Index: 865693 (C4 (Web Text))

Max Activating Token Index: 700

Text #18

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.3676. Min Act: -0.0000

Data Index: 167780 (C4 (Web Text))

Max Activating Token Index: 128

Text #19

Max Range: 1.9826. Min Range: -1.9826

Max Act: 1.3555. Min Act: -0.0000

Data Index: 566789 (C4 (Web Text))

Max Activating Token Index: 529