Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2599 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8155. Min Range: -1.8155

Max Act: 1.8155. Min Act: -0.0000

Data Index: 1319441 (C4 (Web Text))

Max Activating Token Index: 198

Text #1

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.8764. Min Act: -0.0000

Data Index: 684287 (C4 (Web Text))

Max Activating Token Index: 510

Text #2

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.8295. Min Act: -0.0000

Data Index: 679037 (C4 (Web Text))

Max Activating Token Index: 336

Text #3

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.8538. Min Act: -0.0000

Data Index: 989098 (C4 (Web Text))

Max Activating Token Index: 1018

Text #4

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.7676. Min Act: -0.0000

Data Index: 192874 (C4 (Web Text))

Max Activating Token Index: 678

Text #5

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.7879. Min Act: -0.0000

Data Index: 881852 (C4 (Web Text))

Max Activating Token Index: 134

Text #6

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.8274. Min Act: -0.0000

Data Index: 623074 (C4 (Web Text))

Max Activating Token Index: 881

Text #7

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.6885. Min Act: -0.0000

Data Index: 925935 (C4 (Web Text))

Max Activating Token Index: 719

Text #8

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.7353. Min Act: -0.0000

Data Index: 1135914 (C4 (Web Text))

Max Activating Token Index: 352

Text #9

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.7232. Min Act: -0.0000

Data Index: 159798 (C4 (Web Text))

Max Activating Token Index: 165

Text #10

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.6063. Min Act: -0.0000

Data Index: 1220234 (C4 (Web Text))

Max Activating Token Index: 237

Text #11

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.7087. Min Act: -0.0000

Data Index: 944439 (C4 (Web Text))

Max Activating Token Index: 323

Text #12

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.7356. Min Act: -0.0000

Data Index: 509141 (C4 (Web Text))

Max Activating Token Index: 183

Text #13

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.7472. Min Act: -0.0000

Data Index: 561132 (C4 (Web Text))

Max Activating Token Index: 188

Text #14

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.6745. Min Act: -0.0000

Data Index: 275030 (C4 (Web Text))

Max Activating Token Index: 1022

Text #15

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.6500. Min Act: -0.0000

Data Index: 1288945 (C4 (Web Text))

Max Activating Token Index: 162

Text #16

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.5828. Min Act: -0.0000

Data Index: 1225081 (C4 (Web Text))

Max Activating Token Index: 327

Text #17

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.6182. Min Act: -0.0000

Data Index: 671103 (C4 (Web Text))

Max Activating Token Index: 854

Text #18

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.6190. Min Act: -0.0000

Data Index: 1271269 (C4 (Web Text))

Max Activating Token Index: 308

Text #19

Max Range: 1.8155. Min Range: -1.8155

Max Act: 0.6341. Min Act: -0.0000

Data Index: 526133 (C4 (Web Text))

Max Activating Token Index: 543