Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5619 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0509. Min Range: -1.0509

Max Act: 1.0509. Min Act: -0.0000

Data Index: 589332 (C4 (Web Text))

Max Activating Token Index: 422

Text #1

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.9012. Min Act: -0.0000

Data Index: 20628 (C4 (Web Text))

Max Activating Token Index: 1011

Text #2

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.8610. Min Act: -0.0000

Data Index: 372827 (C4 (Web Text))

Max Activating Token Index: 359

Text #3

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.9008. Min Act: -0.0000

Data Index: 269060 (C4 (Web Text))

Max Activating Token Index: 604

Text #4

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.8907. Min Act: -0.0000

Data Index: 868825 (C4 (Web Text))

Max Activating Token Index: 647

Text #5

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.8864. Min Act: -0.0000

Data Index: 555844 (C4 (Web Text))

Max Activating Token Index: 360

Text #6

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.8567. Min Act: -0.0000

Data Index: 162348 (C4 (Web Text))

Max Activating Token Index: 973

Text #7

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.8398. Min Act: -0.0000

Data Index: 1307217 (C4 (Web Text))

Max Activating Token Index: 843

Text #8

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.8661. Min Act: -0.0000

Data Index: 611421 (C4 (Web Text))

Max Activating Token Index: 739

Text #9

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.8096. Min Act: -0.0000

Data Index: 777703 (C4 (Web Text))

Max Activating Token Index: 817

Text #10

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.8343. Min Act: -0.0000

Data Index: 1127379 (C4 (Web Text))

Max Activating Token Index: 28

Text #11

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.8456. Min Act: -0.0000

Data Index: 90534 (C4 (Web Text))

Max Activating Token Index: 778

Text #12

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.7439. Min Act: -0.0000

Data Index: 939684 (C4 (Web Text))

Max Activating Token Index: 919

Text #13

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.7501. Min Act: -0.0000

Data Index: 396755 (C4 (Web Text))

Max Activating Token Index: 777

Text #14

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.7571. Min Act: -0.0000

Data Index: 1284652 (C4 (Web Text))

Max Activating Token Index: 282

Text #15

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.7779. Min Act: -0.0000

Data Index: 552187 (C4 (Web Text))

Max Activating Token Index: 895

Text #16

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.7841. Min Act: -0.0000

Data Index: 855690 (C4 (Web Text))

Max Activating Token Index: 627

Text #17

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.7492. Min Act: -0.0000

Data Index: 535996 (C4 (Web Text))

Max Activating Token Index: 331

Text #18

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.7524. Min Act: -0.0000

Data Index: 974329 (C4 (Web Text))

Max Activating Token Index: 338

Text #19

Max Range: 1.0509. Min Range: -1.0509

Max Act: 0.7510. Min Act: -0.0000

Data Index: 165174 (C4 (Web Text))

Max Activating Token Index: 550