Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4564 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.8121. Min Act: -0.0000

Data Index: 310923 (C4 (Web Text))

Max Activating Token Index: 1001

Text #1

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.7882. Min Act: -0.0000

Data Index: 1570855 (Python Code)

Max Activating Token Index: 765

Text #2

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.6801. Min Act: -0.0000

Data Index: 536609 (C4 (Web Text))

Max Activating Token Index: 664

Text #3

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.6686. Min Act: -0.0000

Data Index: 1389890 (Python Code)

Max Activating Token Index: 665

Text #4

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.4491. Min Act: -0.0000

Data Index: 1574570 (Python Code)

Max Activating Token Index: 748

Text #5

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.5201. Min Act: -0.0000

Data Index: 1186763 (C4 (Web Text))

Max Activating Token Index: 434

Text #6

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.4219. Min Act: -0.0000

Data Index: 1392578 (Python Code)

Max Activating Token Index: 411

Text #7

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.4619. Min Act: -0.0000

Data Index: 1604233 (Python Code)

Max Activating Token Index: 205

Text #8

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.4473. Min Act: -0.0000

Data Index: 853630 (C4 (Web Text))

Max Activating Token Index: 219

Text #9

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.4003. Min Act: -0.0000

Data Index: 1500877 (Python Code)

Max Activating Token Index: 657

Text #10

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.3978. Min Act: -0.0000

Data Index: 537702 (C4 (Web Text))

Max Activating Token Index: 454

Text #11

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.2938. Min Act: -0.0000

Data Index: 57370 (C4 (Web Text))

Max Activating Token Index: 599

Text #12

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.3577. Min Act: -0.0000

Data Index: 1630583 (Python Code)

Max Activating Token Index: 213

Text #13

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.5140. Min Act: -0.0000

Data Index: 1022819 (C4 (Web Text))

Max Activating Token Index: 452

Text #14

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.3184. Min Act: -0.0000

Data Index: 408802 (C4 (Web Text))

Max Activating Token Index: 457

Text #15

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.2884. Min Act: -0.0000

Data Index: 1254172 (C4 (Web Text))

Max Activating Token Index: 571

Text #16

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.2408. Min Act: -0.0000

Data Index: 660396 (C4 (Web Text))

Max Activating Token Index: 400

Text #17

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.3468. Min Act: -0.0000

Data Index: 918538 (C4 (Web Text))

Max Activating Token Index: 372

Text #18

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.2006. Min Act: -0.0000

Data Index: 1438721 (Python Code)

Max Activating Token Index: 102

Text #19

Max Range: 1.8121. Min Range: -1.8121

Max Act: 1.3247. Min Act: -0.0000

Data Index: 1531511 (Python Code)

Max Activating Token Index: 225