Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3378 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3724. Min Range: -1.3724

Max Act: 1.3724. Min Act: -0.0000

Data Index: 1377255 (Python Code)

Max Activating Token Index: 24

Text #1

Max Range: 1.3724. Min Range: -1.3724

Max Act: 1.1527. Min Act: -0.0000

Data Index: 1491287 (Python Code)

Max Activating Token Index: 37

Text #2

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.9385. Min Act: -0.0000

Data Index: 1526835 (Python Code)

Max Activating Token Index: 35

Text #3

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.8896. Min Act: -0.0000

Data Index: 1599153 (Python Code)

Max Activating Token Index: 36

Text #4

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.8335. Min Act: -0.0000

Data Index: 1436814 (Python Code)

Max Activating Token Index: 865

Text #5

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.9293. Min Act: -0.0000

Data Index: 1367176 (Python Code)

Max Activating Token Index: 1009

Text #6

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.7671. Min Act: -0.0000

Data Index: 1605953 (Python Code)

Max Activating Token Index: 755

Text #7

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.8541. Min Act: -0.0000

Data Index: 1583068 (Python Code)

Max Activating Token Index: 567

Text #8

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.9086. Min Act: -0.0000

Data Index: 1605333 (Python Code)

Max Activating Token Index: 967

Text #9

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.7352. Min Act: -0.0000

Data Index: 1580390 (Python Code)

Max Activating Token Index: 974

Text #10

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.7367. Min Act: -0.0000

Data Index: 1558580 (Python Code)

Max Activating Token Index: 9

Text #11

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.7948. Min Act: -0.0000

Data Index: 1627894 (Python Code)

Max Activating Token Index: 35

Text #12

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.7156. Min Act: -0.0000

Data Index: 1654035 (Python Code)

Max Activating Token Index: 661

Text #13

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.7929. Min Act: -0.0000

Data Index: 1362845 (Python Code)

Max Activating Token Index: 84

Text #14

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.7411. Min Act: -0.0000

Data Index: 1647727 (Python Code)

Max Activating Token Index: 26

Text #15

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.6884. Min Act: -0.0000

Data Index: 1555683 (Python Code)

Max Activating Token Index: 738

Text #16

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.6898. Min Act: -0.0000

Data Index: 1561097 (Python Code)

Max Activating Token Index: 351

Text #17

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.6963. Min Act: -0.0000

Data Index: 1490931 (Python Code)

Max Activating Token Index: 21

Text #18

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.7406. Min Act: -0.0000

Data Index: 1641151 (Python Code)

Max Activating Token Index: 669

Text #19

Max Range: 1.3724. Min Range: -1.3724

Max Act: 0.6876. Min Act: -0.0000

Data Index: 1533972 (Python Code)

Max Activating Token Index: 904