Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1079 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.0739. Min Range: -1.0739

Max Act: 1.0739. Min Act: -0.0000

Data Index: 1090478 (C4 (Web Text))

Max Activating Token Index: 982

Text #1

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.8914. Min Act: -0.0000

Data Index: 1043315 (C4 (Web Text))

Max Activating Token Index: 770

Text #2

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.9342. Min Act: -0.0000

Data Index: 515656 (C4 (Web Text))

Max Activating Token Index: 1008

Text #3

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.7802. Min Act: -0.0000

Data Index: 198869 (C4 (Web Text))

Max Activating Token Index: 822

Text #4

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.7974. Min Act: -0.0000

Data Index: 294452 (C4 (Web Text))

Max Activating Token Index: 211

Text #5

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.7501. Min Act: -0.0000

Data Index: 659406 (C4 (Web Text))

Max Activating Token Index: 629

Text #6

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.7625. Min Act: -0.0000

Data Index: 260691 (C4 (Web Text))

Max Activating Token Index: 495

Text #7

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.7347. Min Act: -0.0000

Data Index: 295846 (C4 (Web Text))

Max Activating Token Index: 517

Text #8

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.7633. Min Act: -0.0000

Data Index: 370147 (C4 (Web Text))

Max Activating Token Index: 705

Text #9

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.7328. Min Act: -0.0000

Data Index: 1160732 (C4 (Web Text))

Max Activating Token Index: 49

Text #10

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.7165. Min Act: -0.0000

Data Index: 1032716 (C4 (Web Text))

Max Activating Token Index: 670

Text #11

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.6729. Min Act: -0.0000

Data Index: 671212 (C4 (Web Text))

Max Activating Token Index: 345

Text #12

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.6241. Min Act: -0.0000

Data Index: 862477 (C4 (Web Text))

Max Activating Token Index: 194

Text #13

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.6297. Min Act: -0.0000

Data Index: 200461 (C4 (Web Text))

Max Activating Token Index: 256

Text #14

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.6407. Min Act: -0.0000

Data Index: 385545 (C4 (Web Text))

Max Activating Token Index: 342

Text #15

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.6346. Min Act: -0.0000

Data Index: 682346 (C4 (Web Text))

Max Activating Token Index: 561

Text #16

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.6378. Min Act: -0.0000

Data Index: 583224 (C4 (Web Text))

Max Activating Token Index: 355

Text #17

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.6540. Min Act: -0.0000

Data Index: 197251 (C4 (Web Text))

Max Activating Token Index: 923

Text #18

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.6445. Min Act: -0.0000

Data Index: 535750 (C4 (Web Text))

Max Activating Token Index: 980

Text #19

Max Range: 1.0739. Min Range: -1.0739

Max Act: 0.5845. Min Act: -0.0000

Data Index: 1013323 (C4 (Web Text))

Max Activating Token Index: 355