Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3841 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 5.4587. Min Range: -5.4587

Max Act: 5.4587. Min Act: -0.0000

Data Index: 1062568 (C4 (Web Text))

Max Activating Token Index: 956

Text #1

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.7898. Min Act: -0.0000

Data Index: 872246 (C4 (Web Text))

Max Activating Token Index: 635

Text #2

Max Range: 5.4587. Min Range: -5.4587

Max Act: 4.0314. Min Act: -0.0000

Data Index: 663115 (C4 (Web Text))

Max Activating Token Index: 395

Text #3

Max Range: 5.4587. Min Range: -5.4587

Max Act: 4.1454. Min Act: -0.0000

Data Index: 818483 (C4 (Web Text))

Max Activating Token Index: 318

Text #4

Max Range: 5.4587. Min Range: -5.4587

Max Act: 4.1094. Min Act: -0.0000

Data Index: 460505 (C4 (Web Text))

Max Activating Token Index: 486

Text #5

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.6582. Min Act: -0.0000

Data Index: 255236 (C4 (Web Text))

Max Activating Token Index: 460

Text #6

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.7062. Min Act: -0.0000

Data Index: 1088881 (C4 (Web Text))

Max Activating Token Index: 798

Text #7

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.5730. Min Act: -0.0000

Data Index: 679886 (C4 (Web Text))

Max Activating Token Index: 618

Text #8

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.3514. Min Act: -0.0000

Data Index: 290556 (C4 (Web Text))

Max Activating Token Index: 685

Text #9

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.7057. Min Act: -0.0000

Data Index: 1310551 (C4 (Web Text))

Max Activating Token Index: 674

Text #10

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.5026. Min Act: -0.0000

Data Index: 868978 (C4 (Web Text))

Max Activating Token Index: 583

Text #11

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.2463. Min Act: -0.0000

Data Index: 1162838 (C4 (Web Text))

Max Activating Token Index: 521

Text #12

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.3384. Min Act: -0.0000

Data Index: 1087324 (C4 (Web Text))

Max Activating Token Index: 46

Text #13

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.2289. Min Act: -0.0000

Data Index: 1268921 (C4 (Web Text))

Max Activating Token Index: 606

Text #14

Max Range: 5.4587. Min Range: -5.4587

Max Act: 2.5912. Min Act: -0.0000

Data Index: 72674 (C4 (Web Text))

Max Activating Token Index: 917

Text #15

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.0244. Min Act: -0.0000

Data Index: 1304656 (C4 (Web Text))

Max Activating Token Index: 186

Text #16

Max Range: 5.4587. Min Range: -5.4587

Max Act: 2.6548. Min Act: -0.0000

Data Index: 541126 (C4 (Web Text))

Max Activating Token Index: 123

Text #17

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.1172. Min Act: -0.0000

Data Index: 1076383 (C4 (Web Text))

Max Activating Token Index: 416

Text #18

Max Range: 5.4587. Min Range: -5.4587

Max Act: 3.0549. Min Act: -0.0000

Data Index: 166765 (C4 (Web Text))

Max Activating Token Index: 905

Text #19

Max Range: 5.4587. Min Range: -5.4587

Max Act: 2.9488. Min Act: -0.0000

Data Index: 575067 (C4 (Web Text))

Max Activating Token Index: 191