Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2802 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 5.5341. Min Range: -5.5341

Max Act: 5.5341. Min Act: -0.0000

Data Index: 415300 (C4 (Web Text))

Max Activating Token Index: 34

Text #1

Max Range: 5.5341. Min Range: -5.5341

Max Act: 4.3557. Min Act: -0.0000

Data Index: 446209 (C4 (Web Text))

Max Activating Token Index: 162

Text #2

Max Range: 5.5341. Min Range: -5.5341

Max Act: 4.2259. Min Act: -0.0000

Data Index: 463 (C4 (Web Text))

Max Activating Token Index: 45

Text #3

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.9210. Min Act: -0.0000

Data Index: 168611 (C4 (Web Text))

Max Activating Token Index: 51

Text #4

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.9672. Min Act: -0.0000

Data Index: 850049 (C4 (Web Text))

Max Activating Token Index: 49

Text #5

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.9544. Min Act: -0.0000

Data Index: 1202398 (C4 (Web Text))

Max Activating Token Index: 10

Text #6

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.9105. Min Act: -0.0000

Data Index: 751524 (C4 (Web Text))

Max Activating Token Index: 107

Text #7

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.8369. Min Act: -0.0000

Data Index: 166948 (C4 (Web Text))

Max Activating Token Index: 282

Text #8

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.4659. Min Act: -0.0000

Data Index: 496468 (C4 (Web Text))

Max Activating Token Index: 82

Text #9

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.6036. Min Act: -0.0000

Data Index: 254305 (C4 (Web Text))

Max Activating Token Index: 395

Text #10

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.7404. Min Act: -0.0000

Data Index: 1318996 (C4 (Web Text))

Max Activating Token Index: 43

Text #11

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.6582. Min Act: -0.0000

Data Index: 478813 (C4 (Web Text))

Max Activating Token Index: 47

Text #12

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.4580. Min Act: -0.0000

Data Index: 187038 (C4 (Web Text))

Max Activating Token Index: 15

Text #13

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.4450. Min Act: -0.0000

Data Index: 248086 (C4 (Web Text))

Max Activating Token Index: 839

Text #14

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.5888. Min Act: -0.0000

Data Index: 761918 (C4 (Web Text))

Max Activating Token Index: 229

Text #15

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.7677. Min Act: -0.0000

Data Index: 1098807 (C4 (Web Text))

Max Activating Token Index: 176

Text #16

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.4330. Min Act: -0.0000

Data Index: 1135350 (C4 (Web Text))

Max Activating Token Index: 996

Text #17

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.5880. Min Act: -0.0000

Data Index: 341298 (C4 (Web Text))

Max Activating Token Index: 49

Text #18

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.5528. Min Act: -0.0000

Data Index: 97208 (C4 (Web Text))

Max Activating Token Index: 22

Text #19

Max Range: 5.5341. Min Range: -5.5341

Max Act: 3.5351. Min Act: -0.0000

Data Index: 96180 (C4 (Web Text))

Max Activating Token Index: 957