Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2392 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.7319. Min Act: -0.0000

Data Index: 1298266 (C4 (Web Text))

Max Activating Token Index: 600

Text #1

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.6897. Min Act: -0.0000

Data Index: 953585 (C4 (Web Text))

Max Activating Token Index: 756

Text #2

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.6054. Min Act: -0.0000

Data Index: 516542 (C4 (Web Text))

Max Activating Token Index: 218

Text #3

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.6329. Min Act: -0.0000

Data Index: 911264 (C4 (Web Text))

Max Activating Token Index: 258

Text #4

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.5743. Min Act: -0.0000

Data Index: 720233 (C4 (Web Text))

Max Activating Token Index: 209

Text #5

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.4667. Min Act: -0.0000

Data Index: 978315 (C4 (Web Text))

Max Activating Token Index: 88

Text #6

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.4666. Min Act: -0.0000

Data Index: 515722 (C4 (Web Text))

Max Activating Token Index: 875

Text #7

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.3955. Min Act: -0.0000

Data Index: 1227174 (C4 (Web Text))

Max Activating Token Index: 1000

Text #8

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.4098. Min Act: -0.0000

Data Index: 605393 (C4 (Web Text))

Max Activating Token Index: 61

Text #9

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.3716. Min Act: 0.0000

Data Index: 97429 (C4 (Web Text))

Max Activating Token Index: 185

Text #10

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.3498. Min Act: 0.0000

Data Index: 1108592 (C4 (Web Text))

Max Activating Token Index: 216

Text #11

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.3425. Min Act: -0.0000

Data Index: 278383 (C4 (Web Text))

Max Activating Token Index: 1017

Text #12

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.3111. Min Act: -0.0000

Data Index: 1314645 (C4 (Web Text))

Max Activating Token Index: 925

Text #13

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.3188. Min Act: -0.0000

Data Index: 394490 (C4 (Web Text))

Max Activating Token Index: 585

Text #14

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.3245. Min Act: 0.0000

Data Index: 549493 (C4 (Web Text))

Max Activating Token Index: 171

Text #15

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.2986. Min Act: -0.0000

Data Index: 515280 (C4 (Web Text))

Max Activating Token Index: 1019

Text #16

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.2758. Min Act: -0.0000

Data Index: 847884 (C4 (Web Text))

Max Activating Token Index: 440

Text #17

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.2813. Min Act: -0.0000

Data Index: 607846 (C4 (Web Text))

Max Activating Token Index: 862

Text #18

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.3009. Min Act: -0.0000

Data Index: 897936 (C4 (Web Text))

Max Activating Token Index: 620

Text #19

Max Range: 0.7319. Min Range: -0.7319

Max Act: 0.2844. Min Act: -0.0000

Data Index: 716031 (C4 (Web Text))

Max Activating Token Index: 102