Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1312 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.8753. Min Act: -0.0000

Data Index: 727452 (C4 (Web Text))

Max Activating Token Index: 110

Text #1

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.8154. Min Act: -0.0000

Data Index: 769951 (C4 (Web Text))

Max Activating Token Index: 81

Text #2

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.7998. Min Act: -0.0000

Data Index: 1040797 (C4 (Web Text))

Max Activating Token Index: 72

Text #3

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.7901. Min Act: -0.0000

Data Index: 825763 (C4 (Web Text))

Max Activating Token Index: 10

Text #4

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.7535. Min Act: -0.0000

Data Index: 742641 (C4 (Web Text))

Max Activating Token Index: 133

Text #5

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.7127. Min Act: -0.0000

Data Index: 87123 (C4 (Web Text))

Max Activating Token Index: 837

Text #6

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.7098. Min Act: -0.0000

Data Index: 986828 (C4 (Web Text))

Max Activating Token Index: 41

Text #7

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.7161. Min Act: -0.0000

Data Index: 1084186 (C4 (Web Text))

Max Activating Token Index: 902

Text #8

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.7094. Min Act: -0.0000

Data Index: 1218461 (C4 (Web Text))

Max Activating Token Index: 109

Text #9

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.6775. Min Act: -0.0000

Data Index: 558964 (C4 (Web Text))

Max Activating Token Index: 359

Text #10

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.7271. Min Act: -0.0000

Data Index: 13930 (C4 (Web Text))

Max Activating Token Index: 662

Text #11

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.7434. Min Act: -0.0000

Data Index: 640767 (C4 (Web Text))

Max Activating Token Index: 155

Text #12

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.7345. Min Act: -0.0000

Data Index: 1011543 (C4 (Web Text))

Max Activating Token Index: 528

Text #13

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.7021. Min Act: -0.0000

Data Index: 314633 (C4 (Web Text))

Max Activating Token Index: 498

Text #14

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.7244. Min Act: -0.0000

Data Index: 776248 (C4 (Web Text))

Max Activating Token Index: 579

Text #15

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.6937. Min Act: -0.0000

Data Index: 563604 (C4 (Web Text))

Max Activating Token Index: 429

Text #16

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.6893. Min Act: -0.0000

Data Index: 997804 (C4 (Web Text))

Max Activating Token Index: 376

Text #17

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.6721. Min Act: -0.0000

Data Index: 985751 (C4 (Web Text))

Max Activating Token Index: 17

Text #18

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.6687. Min Act: -0.0000

Data Index: 940459 (C4 (Web Text))

Max Activating Token Index: 446

Text #19

Max Range: 0.8753. Min Range: -0.8753

Max Act: 0.6873. Min Act: -0.0000

Data Index: 988068 (C4 (Web Text))

Max Activating Token Index: 181