Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5649 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.5378. Min Range: -2.5378

Max Act: 2.5378. Min Act: -0.0000

Data Index: 266224 (C4 (Web Text))

Max Activating Token Index: 749

Text #1

Max Range: 2.5378. Min Range: -2.5378

Max Act: 2.3856. Min Act: -0.0000

Data Index: 1528725 (Python Code)

Max Activating Token Index: 295

Text #2

Max Range: 2.5378. Min Range: -2.5378

Max Act: 2.0966. Min Act: 0.0000

Data Index: 431176 (C4 (Web Text))

Max Activating Token Index: 185

Text #3

Max Range: 2.5378. Min Range: -2.5378

Max Act: 2.0230. Min Act: -0.0000

Data Index: 1518767 (Python Code)

Max Activating Token Index: 437

Text #4

Max Range: 2.5378. Min Range: -2.5378

Max Act: 2.0133. Min Act: -0.0000

Data Index: 1560310 (Python Code)

Max Activating Token Index: 666

Text #5

Max Range: 2.5378. Min Range: -2.5378

Max Act: 2.0702. Min Act: -0.0000

Data Index: 1588944 (Python Code)

Max Activating Token Index: 783

Text #6

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.9381. Min Act: -0.0000

Data Index: 21899 (C4 (Web Text))

Max Activating Token Index: 31

Text #7

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.8964. Min Act: -0.0000

Data Index: 1605024 (Python Code)

Max Activating Token Index: 119

Text #8

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.8693. Min Act: -0.0000

Data Index: 25958 (C4 (Web Text))

Max Activating Token Index: 497

Text #9

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.8862. Min Act: -0.0000

Data Index: 442667 (C4 (Web Text))

Max Activating Token Index: 842

Text #10

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.7684. Min Act: -0.0000

Data Index: 1466060 (Python Code)

Max Activating Token Index: 875

Text #11

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.8765. Min Act: -0.0000

Data Index: 1401966 (Python Code)

Max Activating Token Index: 956

Text #12

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.8726. Min Act: -0.0000

Data Index: 1437486 (Python Code)

Max Activating Token Index: 617

Text #13

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.7639. Min Act: -0.0000

Data Index: 1542424 (Python Code)

Max Activating Token Index: 123

Text #14

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.8695. Min Act: -0.0000

Data Index: 1375066 (Python Code)

Max Activating Token Index: 131

Text #15

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.7570. Min Act: -0.0000

Data Index: 1645097 (Python Code)

Max Activating Token Index: 523

Text #16

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.7554. Min Act: -0.0000

Data Index: 1577074 (Python Code)

Max Activating Token Index: 636

Text #17

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.7433. Min Act: -0.0000

Data Index: 610691 (C4 (Web Text))

Max Activating Token Index: 69

Text #18

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.7396. Min Act: -0.0000

Data Index: 1398390 (Python Code)

Max Activating Token Index: 760

Text #19

Max Range: 2.5378. Min Range: -2.5378

Max Act: 1.8000. Min Act: -0.0000

Data Index: 1542836 (Python Code)

Max Activating Token Index: 418