Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3389 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.7765. Min Act: -0.0000

Data Index: 26269 (C4 (Web Text))

Max Activating Token Index: 69

Text #1

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.7550. Min Act: -0.0000

Data Index: 487032 (C4 (Web Text))

Max Activating Token Index: 778

Text #2

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.7413. Min Act: -0.0000

Data Index: 1060034 (C4 (Web Text))

Max Activating Token Index: 140

Text #3

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6969. Min Act: -0.0000

Data Index: 564404 (C4 (Web Text))

Max Activating Token Index: 322

Text #4

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6802. Min Act: -0.0000

Data Index: 942935 (C4 (Web Text))

Max Activating Token Index: 558

Text #5

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6918. Min Act: -0.0000

Data Index: 709050 (C4 (Web Text))

Max Activating Token Index: 845

Text #6

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6461. Min Act: -0.0000

Data Index: 5831 (C4 (Web Text))

Max Activating Token Index: 717

Text #7

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6714. Min Act: -0.0000

Data Index: 262214 (C4 (Web Text))

Max Activating Token Index: 839

Text #8

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6781. Min Act: -0.0000

Data Index: 41832 (C4 (Web Text))

Max Activating Token Index: 186

Text #9

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6486. Min Act: -0.0000

Data Index: 274561 (C4 (Web Text))

Max Activating Token Index: 1019

Text #10

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6315. Min Act: -0.0000

Data Index: 1116460 (C4 (Web Text))

Max Activating Token Index: 744

Text #11

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6064. Min Act: -0.0000

Data Index: 254385 (C4 (Web Text))

Max Activating Token Index: 935

Text #12

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6353. Min Act: -0.0000

Data Index: 1290326 (C4 (Web Text))

Max Activating Token Index: 675

Text #13

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6149. Min Act: -0.0000

Data Index: 157308 (C4 (Web Text))

Max Activating Token Index: 361

Text #14

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6505. Min Act: -0.0000

Data Index: 776135 (C4 (Web Text))

Max Activating Token Index: 541

Text #15

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.5940. Min Act: -0.0000

Data Index: 967448 (C4 (Web Text))

Max Activating Token Index: 283

Text #16

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6525. Min Act: -0.0000

Data Index: 883870 (C4 (Web Text))

Max Activating Token Index: 462

Text #17

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6063. Min Act: -0.0000

Data Index: 1200087 (C4 (Web Text))

Max Activating Token Index: 513

Text #18

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6399. Min Act: -0.0000

Data Index: 576030 (C4 (Web Text))

Max Activating Token Index: 983

Text #19

Max Range: 0.7765. Min Range: -0.7765

Max Act: 0.6190. Min Act: -0.0000

Data Index: 1122491 (C4 (Web Text))

Max Activating Token Index: 406