Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4550 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.8414. Min Act: -0.0001

Data Index: 1104432 (C4 (Web Text))

Max Activating Token Index: 477

Text #1

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.8392. Min Act: -0.0001

Data Index: 112153 (C4 (Web Text))

Max Activating Token Index: 327

Text #2

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.8148. Min Act: -0.0001

Data Index: 258789 (C4 (Web Text))

Max Activating Token Index: 552

Text #3

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.3596. Min Act: -0.0001

Data Index: 1239695 (C4 (Web Text))

Max Activating Token Index: 92

Text #4

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.3339. Min Act: -0.0001

Data Index: 775271 (C4 (Web Text))

Max Activating Token Index: 246

Text #5

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.5746. Min Act: -0.0001

Data Index: 532308 (C4 (Web Text))

Max Activating Token Index: 194

Text #6

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.4203. Min Act: -0.0001

Data Index: 554947 (C4 (Web Text))

Max Activating Token Index: 291

Text #7

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.7198. Min Act: -0.0001

Data Index: 1076202 (C4 (Web Text))

Max Activating Token Index: 805

Text #8

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.5078. Min Act: -0.0001

Data Index: 768982 (C4 (Web Text))

Max Activating Token Index: 854

Text #9

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.5319. Min Act: -0.0001

Data Index: 417647 (C4 (Web Text))

Max Activating Token Index: 240

Text #10

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.5060. Min Act: -0.0001

Data Index: 998353 (C4 (Web Text))

Max Activating Token Index: 283

Text #11

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.8463. Min Act: -0.0001

Data Index: 462366 (C4 (Web Text))

Max Activating Token Index: 364

Text #12

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.6455. Min Act: -0.0001

Data Index: 861820 (C4 (Web Text))

Max Activating Token Index: 274

Text #13

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.4735. Min Act: -0.0001

Data Index: 565989 (C4 (Web Text))

Max Activating Token Index: 986

Text #14

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.6119. Min Act: -0.0001

Data Index: 631921 (C4 (Web Text))

Max Activating Token Index: 834

Text #15

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.4378. Min Act: -0.0001

Data Index: 702904 (C4 (Web Text))

Max Activating Token Index: 453

Text #16

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.4383. Min Act: -0.0001

Data Index: 312523 (C4 (Web Text))

Max Activating Token Index: 836

Text #17

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.4204. Min Act: -0.0001

Data Index: 1072844 (C4 (Web Text))

Max Activating Token Index: 177

Text #18

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.5811. Min Act: -0.0001

Data Index: 1330018 (C4 (Web Text))

Max Activating Token Index: 420

Text #19

Max Range: 3.8463. Min Range: -3.8463

Max Act: 3.0360. Min Act: -0.0001

Data Index: 1307967 (C4 (Web Text))

Max Activating Token Index: 76