Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1521 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.2848. Min Range: -4.2848

Max Act: 4.2848. Min Act: 0.0000

Data Index: 60647 (C4 (Web Text))

Max Activating Token Index: 1004

Text #1

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.7883. Min Act: -0.0000

Data Index: 1135988 (C4 (Web Text))

Max Activating Token Index: 87

Text #2

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.7826. Min Act: -0.0000

Data Index: 1173300 (C4 (Web Text))

Max Activating Token Index: 245

Text #3

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.7209. Min Act: -0.0000

Data Index: 201724 (C4 (Web Text))

Max Activating Token Index: 277

Text #4

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.3281. Min Act: -0.0000

Data Index: 1157554 (C4 (Web Text))

Max Activating Token Index: 592

Text #5

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.4672. Min Act: -0.0000

Data Index: 103224 (C4 (Web Text))

Max Activating Token Index: 335

Text #6

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.4628. Min Act: -0.0000

Data Index: 691093 (C4 (Web Text))

Max Activating Token Index: 250

Text #7

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.8062. Min Act: -0.0000

Data Index: 935792 (C4 (Web Text))

Max Activating Token Index: 204

Text #8

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.5353. Min Act: -0.0000

Data Index: 836727 (C4 (Web Text))

Max Activating Token Index: 233

Text #9

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.6923. Min Act: -0.0000

Data Index: 216641 (C4 (Web Text))

Max Activating Token Index: 66

Text #10

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.3336. Min Act: -0.0000

Data Index: 343778 (C4 (Web Text))

Max Activating Token Index: 156

Text #11

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.3998. Min Act: -0.0000

Data Index: 375937 (C4 (Web Text))

Max Activating Token Index: 352

Text #12

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.4318. Min Act: -0.0000

Data Index: 206558 (C4 (Web Text))

Max Activating Token Index: 161

Text #13

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.2105. Min Act: -0.0000

Data Index: 1086745 (C4 (Web Text))

Max Activating Token Index: 220

Text #14

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.1873. Min Act: -0.0000

Data Index: 1201908 (C4 (Web Text))

Max Activating Token Index: 306

Text #15

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.3346. Min Act: -0.0000

Data Index: 1015216 (C4 (Web Text))

Max Activating Token Index: 139

Text #16

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.0461. Min Act: -0.0000

Data Index: 1315341 (C4 (Web Text))

Max Activating Token Index: 367

Text #17

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.3384. Min Act: -0.0000

Data Index: 143541 (C4 (Web Text))

Max Activating Token Index: 944

Text #18

Max Range: 4.2848. Min Range: -4.2848

Max Act: 3.1376. Min Act: -0.0000

Data Index: 240376 (C4 (Web Text))

Max Activating Token Index: 488

Text #19

Max Range: 4.2848. Min Range: -4.2848

Max Act: 2.9328. Min Act: -0.0000

Data Index: 1246872 (C4 (Web Text))

Max Activating Token Index: 245