Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5988 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.4099. Min Range: -3.4099

Max Act: 3.0595. Min Act: -0.0000

Data Index: 547965 (C4 (Web Text))

Max Activating Token Index: 983

Text #1

Max Range: 3.4099. Min Range: -3.4099

Max Act: 3.4099. Min Act: -0.0000

Data Index: 1133375 (C4 (Web Text))

Max Activating Token Index: 280

Text #2

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.9229. Min Act: -0.0000

Data Index: 858109 (C4 (Web Text))

Max Activating Token Index: 728

Text #3

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.8875. Min Act: -0.0000

Data Index: 454463 (C4 (Web Text))

Max Activating Token Index: 86

Text #4

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.7963. Min Act: -0.0000

Data Index: 523176 (C4 (Web Text))

Max Activating Token Index: 444

Text #5

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.7234. Min Act: -0.0000

Data Index: 608759 (C4 (Web Text))

Max Activating Token Index: 424

Text #6

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.5510. Min Act: -0.0000

Data Index: 266268 (C4 (Web Text))

Max Activating Token Index: 82

Text #7

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.3896. Min Act: -0.0000

Data Index: 38812 (C4 (Web Text))

Max Activating Token Index: 20

Text #8

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.5729. Min Act: -0.0000

Data Index: 427858 (C4 (Web Text))

Max Activating Token Index: 594

Text #9

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.4448. Min Act: -0.0000

Data Index: 1071673 (C4 (Web Text))

Max Activating Token Index: 511

Text #10

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.3892. Min Act: -0.0000

Data Index: 298084 (C4 (Web Text))

Max Activating Token Index: 745

Text #11

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.2078. Min Act: -0.0000

Data Index: 102193 (C4 (Web Text))

Max Activating Token Index: 187

Text #12

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.2806. Min Act: -0.0000

Data Index: 885003 (C4 (Web Text))

Max Activating Token Index: 830

Text #13

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.0452. Min Act: -0.0000

Data Index: 611835 (C4 (Web Text))

Max Activating Token Index: 540

Text #14

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.3801. Min Act: -0.0000

Data Index: 1279381 (C4 (Web Text))

Max Activating Token Index: 965

Text #15

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.3449. Min Act: 0.0000

Data Index: 43188 (C4 (Web Text))

Max Activating Token Index: 763

Text #16

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.1866. Min Act: -0.0000

Data Index: 812266 (C4 (Web Text))

Max Activating Token Index: 334

Text #17

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.1556. Min Act: 0.0000

Data Index: 587024 (C4 (Web Text))

Max Activating Token Index: 718

Text #18

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.1713. Min Act: 0.0003

Data Index: 111310 (C4 (Web Text))

Max Activating Token Index: 156

Text #19

Max Range: 3.4099. Min Range: -3.4099

Max Act: 2.0466. Min Act: -0.0000

Data Index: 196654 (C4 (Web Text))

Max Activating Token Index: 379