Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3040 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.9471. Min Act: -0.0000

Data Index: 1473599 (Python Code)

Max Activating Token Index: 21

Text #1

Max Range: 3.4196. Min Range: -3.4196

Max Act: 3.1219. Min Act: -0.0000

Data Index: 1364176 (Python Code)

Max Activating Token Index: 785

Text #2

Max Range: 3.4196. Min Range: -3.4196

Max Act: 3.4196. Min Act: -0.0000

Data Index: 1364972 (Python Code)

Max Activating Token Index: 875

Text #3

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.7364. Min Act: -0.0000

Data Index: 1557273 (Python Code)

Max Activating Token Index: 115

Text #4

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.8557. Min Act: -0.0000

Data Index: 1632312 (Python Code)

Max Activating Token Index: 132

Text #5

Max Range: 3.4196. Min Range: -3.4196

Max Act: 3.1374. Min Act: -0.0000

Data Index: 1628297 (Python Code)

Max Activating Token Index: 728

Text #6

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.8133. Min Act: -0.0000

Data Index: 1530622 (Python Code)

Max Activating Token Index: 416

Text #7

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.7247. Min Act: -0.0000

Data Index: 1369264 (Python Code)

Max Activating Token Index: 995

Text #8

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.6795. Min Act: -0.0000

Data Index: 1635594 (Python Code)

Max Activating Token Index: 340

Text #9

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.4893. Min Act: -0.0000

Data Index: 1508863 (Python Code)

Max Activating Token Index: 160

Text #10

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.8767. Min Act: -0.0000

Data Index: 1386422 (Python Code)

Max Activating Token Index: 185

Text #11

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.8852. Min Act: -0.0000

Data Index: 1440910 (Python Code)

Max Activating Token Index: 875

Text #12

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.8438. Min Act: -0.0000

Data Index: 1444608 (Python Code)

Max Activating Token Index: 300

Text #13

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.5354. Min Act: -0.0000

Data Index: 1547251 (Python Code)

Max Activating Token Index: 18

Text #14

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.7695. Min Act: -0.0000

Data Index: 1453880 (Python Code)

Max Activating Token Index: 850

Text #15

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.7095. Min Act: -0.0000

Data Index: 1653612 (Python Code)

Max Activating Token Index: 800

Text #16

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.4292. Min Act: -0.0000

Data Index: 1570461 (Python Code)

Max Activating Token Index: 868

Text #17

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.5556. Min Act: -0.0000

Data Index: 1418276 (Python Code)

Max Activating Token Index: 200

Text #18

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.5005. Min Act: -0.0000

Data Index: 1574572 (Python Code)

Max Activating Token Index: 607

Text #19

Max Range: 3.4196. Min Range: -3.4196

Max Act: 2.3496. Min Act: -0.0000

Data Index: 1496132 (Python Code)

Max Activating Token Index: 32