Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4193 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3847. Min Range: -1.3847

Max Act: 1.3847. Min Act: -0.0000

Data Index: 948027 (C4 (Web Text))

Max Activating Token Index: 41

Text #1

Max Range: 1.3847. Min Range: -1.3847

Max Act: 1.2385. Min Act: -0.0000

Data Index: 1233043 (C4 (Web Text))

Max Activating Token Index: 40

Text #2

Max Range: 1.3847. Min Range: -1.3847

Max Act: 1.2238. Min Act: -0.0000

Data Index: 1025612 (C4 (Web Text))

Max Activating Token Index: 57

Text #3

Max Range: 1.3847. Min Range: -1.3847

Max Act: 1.2153. Min Act: -0.0000

Data Index: 902008 (C4 (Web Text))

Max Activating Token Index: 787

Text #4

Max Range: 1.3847. Min Range: -1.3847

Max Act: 1.0088. Min Act: -0.0000

Data Index: 223583 (C4 (Web Text))

Max Activating Token Index: 473

Text #5

Max Range: 1.3847. Min Range: -1.3847

Max Act: 1.0916. Min Act: -0.0000

Data Index: 1291564 (C4 (Web Text))

Max Activating Token Index: 543

Text #6

Max Range: 1.3847. Min Range: -1.3847

Max Act: 1.0453. Min Act: -0.0000

Data Index: 785417 (C4 (Web Text))

Max Activating Token Index: 54

Text #7

Max Range: 1.3847. Min Range: -1.3847

Max Act: 1.0198. Min Act: -0.0000

Data Index: 346104 (C4 (Web Text))

Max Activating Token Index: 35

Text #8

Max Range: 1.3847. Min Range: -1.3847

Max Act: 0.7335. Min Act: -0.0000

Data Index: 574302 (C4 (Web Text))

Max Activating Token Index: 796

Text #9

Max Range: 1.3847. Min Range: -1.3847

Max Act: 1.2111. Min Act: -0.0000

Data Index: 1191612 (C4 (Web Text))

Max Activating Token Index: 905

Text #10

Max Range: 1.3847. Min Range: -1.3847

Max Act: 0.9060. Min Act: -0.0000

Data Index: 1382046 (Python Code)

Max Activating Token Index: 956

Text #11

Max Range: 1.3847. Min Range: -1.3847

Max Act: 1.0201. Min Act: -0.0000

Data Index: 693465 (C4 (Web Text))

Max Activating Token Index: 64

Text #12

Max Range: 1.3847. Min Range: -1.3847

Max Act: 0.8704. Min Act: -0.0000

Data Index: 1459540 (Python Code)

Max Activating Token Index: 814

Text #13

Max Range: 1.3847. Min Range: -1.3847

Max Act: 0.9934. Min Act: -0.0000

Data Index: 907490 (C4 (Web Text))

Max Activating Token Index: 45

Text #14

Max Range: 1.3847. Min Range: -1.3847

Max Act: 0.9868. Min Act: -0.0000

Data Index: 782264 (C4 (Web Text))

Max Activating Token Index: 42

Text #15

Max Range: 1.3847. Min Range: -1.3847

Max Act: 0.9370. Min Act: -0.0000

Data Index: 1217573 (C4 (Web Text))

Max Activating Token Index: 571

Text #16

Max Range: 1.3847. Min Range: -1.3847

Max Act: 0.9771. Min Act: -0.0000

Data Index: 830290 (C4 (Web Text))

Max Activating Token Index: 773

Text #17

Max Range: 1.3847. Min Range: -1.3847

Max Act: 0.8394. Min Act: -0.0000

Data Index: 252517 (C4 (Web Text))

Max Activating Token Index: 484

Text #18

Max Range: 1.3847. Min Range: -1.3847

Max Act: 0.8501. Min Act: -0.0000

Data Index: 518999 (C4 (Web Text))

Max Activating Token Index: 35

Text #19

Max Range: 1.3847. Min Range: -1.3847

Max Act: 0.8526. Min Act: -0.0000

Data Index: 472617 (C4 (Web Text))

Max Activating Token Index: 482