Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1304 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.7218. Min Act: -0.0000

Data Index: 546585 (C4 (Web Text))

Max Activating Token Index: 586

Text #1

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.4347. Min Act: -0.0000

Data Index: 195298 (C4 (Web Text))

Max Activating Token Index: 146

Text #2

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.4417. Min Act: -0.0000

Data Index: 603003 (C4 (Web Text))

Max Activating Token Index: 885

Text #3

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.4290. Min Act: -0.0000

Data Index: 900478 (C4 (Web Text))

Max Activating Token Index: 904

Text #4

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.4048. Min Act: -0.0000

Data Index: 434053 (C4 (Web Text))

Max Activating Token Index: 974

Text #5

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.4426. Min Act: -0.0000

Data Index: 1182854 (C4 (Web Text))

Max Activating Token Index: 898

Text #6

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.4157. Min Act: -0.0000

Data Index: 946914 (C4 (Web Text))

Max Activating Token Index: 61

Text #7

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.4739. Min Act: -0.0000

Data Index: 180716 (C4 (Web Text))

Max Activating Token Index: 960

Text #8

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.3665. Min Act: -0.0000

Data Index: 1171454 (C4 (Web Text))

Max Activating Token Index: 647

Text #9

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.3329. Min Act: -0.0000

Data Index: 557010 (C4 (Web Text))

Max Activating Token Index: 757

Text #10

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.3219. Min Act: -0.0000

Data Index: 742091 (C4 (Web Text))

Max Activating Token Index: 216

Text #11

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.3487. Min Act: -0.0000

Data Index: 817271 (C4 (Web Text))

Max Activating Token Index: 332

Text #12

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.3327. Min Act: -0.0000

Data Index: 372285 (C4 (Web Text))

Max Activating Token Index: 313

Text #13

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.3248. Min Act: -0.0000

Data Index: 934763 (C4 (Web Text))

Max Activating Token Index: 61

Text #14

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.3250. Min Act: -0.0000

Data Index: 827817 (C4 (Web Text))

Max Activating Token Index: 754

Text #15

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.3454. Min Act: -0.0000

Data Index: 200030 (C4 (Web Text))

Max Activating Token Index: 701

Text #16

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.3310. Min Act: -0.0000

Data Index: 1135302 (C4 (Web Text))

Max Activating Token Index: 956

Text #17

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.3235. Min Act: -0.0000

Data Index: 123086 (C4 (Web Text))

Max Activating Token Index: 563

Text #18

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.3083. Min Act: -0.0000

Data Index: 980024 (C4 (Web Text))

Max Activating Token Index: 895

Text #19

Max Range: 0.7218. Min Range: -0.7218

Max Act: 0.2874. Min Act: -0.0000

Data Index: 826963 (C4 (Web Text))

Max Activating Token Index: 848