Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3303 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.7912. Min Act: -0.0001

Data Index: 272628 (C4 (Web Text))

Max Activating Token Index: 515

Text #1

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.7168. Min Act: -0.0001

Data Index: 1253965 (C4 (Web Text))

Max Activating Token Index: 615

Text #2

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.6877. Min Act: -0.0002

Data Index: 1149417 (C4 (Web Text))

Max Activating Token Index: 32

Text #3

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.6216. Min Act: -0.0002

Data Index: 857004 (C4 (Web Text))

Max Activating Token Index: 663

Text #4

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.6666. Min Act: -0.0001

Data Index: 796761 (C4 (Web Text))

Max Activating Token Index: 699

Text #5

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.6698. Min Act: -0.0002

Data Index: 1311169 (C4 (Web Text))

Max Activating Token Index: 673

Text #6

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.6030. Min Act: -0.0001

Data Index: 996699 (C4 (Web Text))

Max Activating Token Index: 708

Text #7

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.6412. Min Act: -0.0002

Data Index: 126629 (C4 (Web Text))

Max Activating Token Index: 716

Text #8

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.6364. Min Act: -0.0002

Data Index: 340276 (C4 (Web Text))

Max Activating Token Index: 126

Text #9

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.6486. Min Act: -0.0002

Data Index: 517053 (C4 (Web Text))

Max Activating Token Index: 294

Text #10

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.6124. Min Act: -0.0002

Data Index: 516548 (C4 (Web Text))

Max Activating Token Index: 13

Text #11

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.6460. Min Act: -0.0001

Data Index: 529681 (C4 (Web Text))

Max Activating Token Index: 747

Text #12

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.6540. Min Act: -0.0002

Data Index: 924409 (C4 (Web Text))

Max Activating Token Index: 178

Text #13

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.5933. Min Act: -0.0001

Data Index: 1213684 (C4 (Web Text))

Max Activating Token Index: 890

Text #14

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.5769. Min Act: -0.0001

Data Index: 964072 (C4 (Web Text))

Max Activating Token Index: 95

Text #15

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.5884. Min Act: -0.0001

Data Index: 279894 (C4 (Web Text))

Max Activating Token Index: 491

Text #16

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.5269. Min Act: -0.0002

Data Index: 994158 (C4 (Web Text))

Max Activating Token Index: 302

Text #17

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.6114. Min Act: -0.0002

Data Index: 361191 (C4 (Web Text))

Max Activating Token Index: 1013

Text #18

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.5886. Min Act: -0.0002

Data Index: 693313 (C4 (Web Text))

Max Activating Token Index: 771

Text #19

Max Range: 0.7912. Min Range: -0.7912

Max Act: 0.5230. Min Act: -0.0002

Data Index: 96934 (C4 (Web Text))

Max Activating Token Index: 57