Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3413 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 5.3549. Min Range: -5.3549

Max Act: 5.3549. Min Act: -0.0001

Data Index: 194690 (C4 (Web Text))

Max Activating Token Index: 314

Text #1

Max Range: 5.3549. Min Range: -5.3549

Max Act: 4.3940. Min Act: -0.0000

Data Index: 1348418 (C4 (Web Text))

Max Activating Token Index: 882

Text #2

Max Range: 5.3549. Min Range: -5.3549

Max Act: 4.2526. Min Act: -0.0001

Data Index: 1061303 (C4 (Web Text))

Max Activating Token Index: 732

Text #3

Max Range: 5.3549. Min Range: -5.3549

Max Act: 3.8560. Min Act: -0.0001

Data Index: 993237 (C4 (Web Text))

Max Activating Token Index: 550

Text #4

Max Range: 5.3549. Min Range: -5.3549

Max Act: 3.5626. Min Act: -0.0001

Data Index: 909806 (C4 (Web Text))

Max Activating Token Index: 489

Text #5

Max Range: 5.3549. Min Range: -5.3549

Max Act: 3.4519. Min Act: -0.0001

Data Index: 199154 (C4 (Web Text))

Max Activating Token Index: 887

Text #6

Max Range: 5.3549. Min Range: -5.3549

Max Act: 3.3463. Min Act: -0.0000

Data Index: 284185 (C4 (Web Text))

Max Activating Token Index: 798

Text #7

Max Range: 5.3549. Min Range: -5.3549

Max Act: 3.3106. Min Act: -0.0001

Data Index: 469649 (C4 (Web Text))

Max Activating Token Index: 802

Text #8

Max Range: 5.3549. Min Range: -5.3549

Max Act: 3.1772. Min Act: -0.0001

Data Index: 502190 (C4 (Web Text))

Max Activating Token Index: 309

Text #9

Max Range: 5.3549. Min Range: -5.3549

Max Act: 2.8576. Min Act: -0.0001

Data Index: 1098085 (C4 (Web Text))

Max Activating Token Index: 59

Text #10

Max Range: 5.3549. Min Range: -5.3549

Max Act: 3.0649. Min Act: -0.0001

Data Index: 797376 (C4 (Web Text))

Max Activating Token Index: 868

Text #11

Max Range: 5.3549. Min Range: -5.3549

Max Act: 3.0197. Min Act: -0.0001

Data Index: 1016366 (C4 (Web Text))

Max Activating Token Index: 773

Text #12

Max Range: 5.3549. Min Range: -5.3549

Max Act: 2.6626. Min Act: -0.0001

Data Index: 1213368 (C4 (Web Text))

Max Activating Token Index: 503

Text #13

Max Range: 5.3549. Min Range: -5.3549

Max Act: 2.5701. Min Act: -0.0001

Data Index: 345930 (C4 (Web Text))

Max Activating Token Index: 903

Text #14

Max Range: 5.3549. Min Range: -5.3549

Max Act: 2.5421. Min Act: -0.0001

Data Index: 1110571 (C4 (Web Text))

Max Activating Token Index: 556

Text #15

Max Range: 5.3549. Min Range: -5.3549

Max Act: 2.5339. Min Act: -0.0001

Data Index: 831794 (C4 (Web Text))

Max Activating Token Index: 902

Text #16

Max Range: 5.3549. Min Range: -5.3549

Max Act: 2.6204. Min Act: -0.0001

Data Index: 1255826 (C4 (Web Text))

Max Activating Token Index: 518

Text #17

Max Range: 5.3549. Min Range: -5.3549

Max Act: 2.3480. Min Act: -0.0001

Data Index: 525640 (C4 (Web Text))

Max Activating Token Index: 458

Text #18

Max Range: 5.3549. Min Range: -5.3549

Max Act: 2.5931. Min Act: -0.0001

Data Index: 413861 (C4 (Web Text))

Max Activating Token Index: 635

Text #19

Max Range: 5.3549. Min Range: -5.3549

Max Act: 2.2498. Min Act: -0.0001

Data Index: 745686 (C4 (Web Text))

Max Activating Token Index: 848