Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1380 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.6938. Min Act: -0.0000

Data Index: 994838 (C4 (Web Text))

Max Activating Token Index: 52

Text #1

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.6669. Min Act: -0.0000

Data Index: 778 (C4 (Web Text))

Max Activating Token Index: 319

Text #2

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.6532. Min Act: -0.0000

Data Index: 895629 (C4 (Web Text))

Max Activating Token Index: 351

Text #3

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.6047. Min Act: -0.0000

Data Index: 1264333 (C4 (Web Text))

Max Activating Token Index: 347

Text #4

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.6176. Min Act: -0.0000

Data Index: 1265940 (C4 (Web Text))

Max Activating Token Index: 780

Text #5

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.5854. Min Act: -0.0000

Data Index: 1224622 (C4 (Web Text))

Max Activating Token Index: 853

Text #6

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.6387. Min Act: -0.0000

Data Index: 807307 (C4 (Web Text))

Max Activating Token Index: 118

Text #7

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.5894. Min Act: -0.0000

Data Index: 722077 (C4 (Web Text))

Max Activating Token Index: 499

Text #8

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.6121. Min Act: -0.0000

Data Index: 592060 (C4 (Web Text))

Max Activating Token Index: 592

Text #9

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.5919. Min Act: -0.0000

Data Index: 891610 (C4 (Web Text))

Max Activating Token Index: 672

Text #10

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.5677. Min Act: -0.0000

Data Index: 109882 (C4 (Web Text))

Max Activating Token Index: 891

Text #11

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.5995. Min Act: -0.0000

Data Index: 554919 (C4 (Web Text))

Max Activating Token Index: 704

Text #12

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.6171. Min Act: -0.0000

Data Index: 456479 (C4 (Web Text))

Max Activating Token Index: 754

Text #13

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.5642. Min Act: -0.0000

Data Index: 1187527 (C4 (Web Text))

Max Activating Token Index: 478

Text #14

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.6294. Min Act: -0.0000

Data Index: 383060 (C4 (Web Text))

Max Activating Token Index: 872

Text #15

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.5914. Min Act: -0.0000

Data Index: 1368153 (Python Code)

Max Activating Token Index: 786

Text #16

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.5648. Min Act: -0.0000

Data Index: 132764 (C4 (Web Text))

Max Activating Token Index: 543

Text #17

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.5586. Min Act: -0.0000

Data Index: 1123417 (C4 (Web Text))

Max Activating Token Index: 294

Text #18

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.5508. Min Act: -0.0000

Data Index: 1305917 (C4 (Web Text))

Max Activating Token Index: 961

Text #19

Max Range: 0.6938. Min Range: -0.6938

Max Act: 0.5075. Min Act: -0.0000

Data Index: 385457 (C4 (Web Text))

Max Activating Token Index: 170