Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1478 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.7595. Min Act: -0.0000

Data Index: 1358137 (C4 (Web Text))

Max Activating Token Index: 573

Text #1

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.7210. Min Act: -0.0000

Data Index: 425685 (C4 (Web Text))

Max Activating Token Index: 254

Text #2

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.8367. Min Act: -0.0000

Data Index: 299724 (C4 (Web Text))

Max Activating Token Index: 63

Text #3

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.6461. Min Act: -0.0000

Data Index: 794532 (C4 (Web Text))

Max Activating Token Index: 74

Text #4

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.6030. Min Act: -0.0000

Data Index: 559878 (C4 (Web Text))

Max Activating Token Index: 107

Text #5

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.3724. Min Act: -0.0000

Data Index: 523948 (C4 (Web Text))

Max Activating Token Index: 617

Text #6

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.4748. Min Act: -0.0000

Data Index: 845615 (C4 (Web Text))

Max Activating Token Index: 165

Text #7

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.3365. Min Act: -0.0000

Data Index: 11266 (C4 (Web Text))

Max Activating Token Index: 354

Text #8

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.4582. Min Act: -0.0000

Data Index: 981610 (C4 (Web Text))

Max Activating Token Index: 508

Text #9

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.3135. Min Act: -0.0000

Data Index: 161115 (C4 (Web Text))

Max Activating Token Index: 92

Text #10

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.4334. Min Act: -0.0000

Data Index: 83792 (C4 (Web Text))

Max Activating Token Index: 532

Text #11

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.3100. Min Act: -0.0000

Data Index: 272302 (C4 (Web Text))

Max Activating Token Index: 779

Text #12

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.3343. Min Act: -0.0000

Data Index: 1079450 (C4 (Web Text))

Max Activating Token Index: 124

Text #13

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.2109. Min Act: -0.0000

Data Index: 609392 (C4 (Web Text))

Max Activating Token Index: 829

Text #14

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.1235. Min Act: -0.0000

Data Index: 41772 (C4 (Web Text))

Max Activating Token Index: 220

Text #15

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.1368. Min Act: -0.0000

Data Index: 422906 (C4 (Web Text))

Max Activating Token Index: 491

Text #16

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.0989. Min Act: -0.0000

Data Index: 629592 (C4 (Web Text))

Max Activating Token Index: 141

Text #17

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.1090. Min Act: -0.0000

Data Index: 1105648 (C4 (Web Text))

Max Activating Token Index: 208

Text #18

Max Range: 2.8367. Min Range: -2.8367

Max Act: 1.9563. Min Act: -0.0000

Data Index: 620499 (C4 (Web Text))

Max Activating Token Index: 341

Text #19

Max Range: 2.8367. Min Range: -2.8367

Max Act: 2.0574. Min Act: -0.0000

Data Index: 508573 (C4 (Web Text))

Max Activating Token Index: 613