Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3897 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.7362. Min Range: -3.7362

Max Act: 3.7362. Min Act: -0.0001

Data Index: 164549 (C4 (Web Text))

Max Activating Token Index: 234

Text #1

Max Range: 3.7362. Min Range: -3.7362

Max Act: 3.2576. Min Act: -0.0001

Data Index: 169336 (C4 (Web Text))

Max Activating Token Index: 767

Text #2

Max Range: 3.7362. Min Range: -3.7362

Max Act: 3.3280. Min Act: -0.0001

Data Index: 1081604 (C4 (Web Text))

Max Activating Token Index: 430

Text #3

Max Range: 3.7362. Min Range: -3.7362

Max Act: 3.3599. Min Act: -0.0001

Data Index: 48878 (C4 (Web Text))

Max Activating Token Index: 327

Text #4

Max Range: 3.7362. Min Range: -3.7362

Max Act: 3.1096. Min Act: -0.0001

Data Index: 1207153 (C4 (Web Text))

Max Activating Token Index: 339

Text #5

Max Range: 3.7362. Min Range: -3.7362

Max Act: 2.9253. Min Act: -0.0001

Data Index: 413633 (C4 (Web Text))

Max Activating Token Index: 603

Text #6

Max Range: 3.7362. Min Range: -3.7362

Max Act: 2.9693. Min Act: -0.0001

Data Index: 276078 (C4 (Web Text))

Max Activating Token Index: 503

Text #7

Max Range: 3.7362. Min Range: -3.7362

Max Act: 3.0587. Min Act: -0.0001

Data Index: 108568 (C4 (Web Text))

Max Activating Token Index: 90

Text #8

Max Range: 3.7362. Min Range: -3.7362

Max Act: 3.3449. Min Act: -0.0001

Data Index: 772043 (C4 (Web Text))

Max Activating Token Index: 825

Text #9

Max Range: 3.7362. Min Range: -3.7362

Max Act: 3.0763. Min Act: -0.0001

Data Index: 139232 (C4 (Web Text))

Max Activating Token Index: 750

Text #10

Max Range: 3.7362. Min Range: -3.7362

Max Act: 2.9497. Min Act: -0.0001

Data Index: 360052 (C4 (Web Text))

Max Activating Token Index: 609

Text #11

Max Range: 3.7362. Min Range: -3.7362

Max Act: 2.9429. Min Act: -0.0001

Data Index: 358775 (C4 (Web Text))

Max Activating Token Index: 697

Text #12

Max Range: 3.7362. Min Range: -3.7362

Max Act: 3.0532. Min Act: -0.0001

Data Index: 935127 (C4 (Web Text))

Max Activating Token Index: 837

Text #13

Max Range: 3.7362. Min Range: -3.7362

Max Act: 2.7061. Min Act: -0.0001

Data Index: 1127203 (C4 (Web Text))

Max Activating Token Index: 677

Text #14

Max Range: 3.7362. Min Range: -3.7362

Max Act: 2.9601. Min Act: -0.0001

Data Index: 205110 (C4 (Web Text))

Max Activating Token Index: 910

Text #15

Max Range: 3.7362. Min Range: -3.7362

Max Act: 2.6911. Min Act: -0.0001

Data Index: 13789 (C4 (Web Text))

Max Activating Token Index: 757

Text #16

Max Range: 3.7362. Min Range: -3.7362

Max Act: 2.5515. Min Act: -0.0001

Data Index: 984294 (C4 (Web Text))

Max Activating Token Index: 356

Text #17

Max Range: 3.7362. Min Range: -3.7362

Max Act: 2.5673. Min Act: -0.0001

Data Index: 1144501 (C4 (Web Text))

Max Activating Token Index: 846

Text #18

Max Range: 3.7362. Min Range: -3.7362

Max Act: 2.9364. Min Act: -0.0001

Data Index: 496102 (C4 (Web Text))

Max Activating Token Index: 361

Text #19

Max Range: 3.7362. Min Range: -3.7362

Max Act: 2.7528. Min Act: -0.0001

Data Index: 596471 (C4 (Web Text))

Max Activating Token Index: 804