Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 402 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.6704. Min Act: -0.0000

Data Index: 1550590 (Python Code)

Max Activating Token Index: 236

Text #1

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.6679. Min Act: -0.0000

Data Index: 1485293 (Python Code)

Max Activating Token Index: 524

Text #2

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.4159. Min Act: -0.0000

Data Index: 1597282 (Python Code)

Max Activating Token Index: 937

Text #3

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.4876. Min Act: -0.0000

Data Index: 1581016 (Python Code)

Max Activating Token Index: 410

Text #4

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.4649. Min Act: -0.0000

Data Index: 1540419 (Python Code)

Max Activating Token Index: 640

Text #5

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.4537. Min Act: -0.0000

Data Index: 1499647 (Python Code)

Max Activating Token Index: 732

Text #6

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.3227. Min Act: -0.0000

Data Index: 1595668 (Python Code)

Max Activating Token Index: 142

Text #7

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.4169. Min Act: -0.0000

Data Index: 1360478 (Python Code)

Max Activating Token Index: 1023

Text #8

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.4125. Min Act: -0.0000

Data Index: 1556343 (Python Code)

Max Activating Token Index: 286

Text #9

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.3925. Min Act: -0.0000

Data Index: 1361061 (Python Code)

Max Activating Token Index: 716

Text #10

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.2795. Min Act: -0.0000

Data Index: 1565768 (Python Code)

Max Activating Token Index: 841

Text #11

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.4746. Min Act: -0.0000

Data Index: 1459654 (Python Code)

Max Activating Token Index: 699

Text #12

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.3450. Min Act: -0.0000

Data Index: 1506684 (Python Code)

Max Activating Token Index: 520

Text #13

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.3818. Min Act: -0.0000

Data Index: 1572362 (Python Code)

Max Activating Token Index: 489

Text #14

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.3664. Min Act: -0.0000

Data Index: 1650100 (Python Code)

Max Activating Token Index: 201

Text #15

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.3816. Min Act: -0.0000

Data Index: 1522169 (Python Code)

Max Activating Token Index: 141

Text #16

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.3363. Min Act: -0.0000

Data Index: 1484244 (Python Code)

Max Activating Token Index: 251

Text #17

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.4619. Min Act: -0.0000

Data Index: 1513114 (Python Code)

Max Activating Token Index: 731

Text #18

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.3677. Min Act: -0.0000

Data Index: 1458341 (Python Code)

Max Activating Token Index: 529

Text #19

Max Range: 1.6704. Min Range: -1.6704

Max Act: 1.3192. Min Act: -0.0000

Data Index: 1626889 (Python Code)

Max Activating Token Index: 807