Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5013 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.2795. Min Range: -2.2795

Max Act: 2.2704. Min Act: -0.0000

Data Index: 1597412 (Python Code)

Max Activating Token Index: 991

Text #1

Max Range: 2.2795. Min Range: -2.2795

Max Act: 2.2795. Min Act: -0.0000

Data Index: 1392941 (Python Code)

Max Activating Token Index: 694

Text #2

Max Range: 2.2795. Min Range: -2.2795

Max Act: 2.0350. Min Act: -0.0000

Data Index: 1630533 (Python Code)

Max Activating Token Index: 946

Text #3

Max Range: 2.2795. Min Range: -2.2795

Max Act: 2.0835. Min Act: -0.0000

Data Index: 1456082 (Python Code)

Max Activating Token Index: 862

Text #4

Max Range: 2.2795. Min Range: -2.2795

Max Act: 2.0466. Min Act: -0.0000

Data Index: 1362396 (Python Code)

Max Activating Token Index: 569

Text #5

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.9967. Min Act: -0.0000

Data Index: 1564728 (Python Code)

Max Activating Token Index: 1007

Text #6

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.9463. Min Act: -0.0000

Data Index: 1544459 (Python Code)

Max Activating Token Index: 92

Text #7

Max Range: 2.2795. Min Range: -2.2795

Max Act: 2.0452. Min Act: -0.0000

Data Index: 1503625 (Python Code)

Max Activating Token Index: 963

Text #8

Max Range: 2.2795. Min Range: -2.2795

Max Act: 2.0213. Min Act: -0.0000

Data Index: 1400805 (Python Code)

Max Activating Token Index: 783

Text #9

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.7906. Min Act: -0.0000

Data Index: 1415222 (Python Code)

Max Activating Token Index: 881

Text #10

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.7253. Min Act: -0.0000

Data Index: 1576257 (Python Code)

Max Activating Token Index: 872

Text #11

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.8195. Min Act: -0.0000

Data Index: 1592279 (Python Code)

Max Activating Token Index: 890

Text #12

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.7836. Min Act: -0.0000

Data Index: 1474365 (Python Code)

Max Activating Token Index: 739

Text #13

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.7894. Min Act: -0.0000

Data Index: 1584614 (Python Code)

Max Activating Token Index: 82

Text #14

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.6838. Min Act: -0.0000

Data Index: 1407982 (Python Code)

Max Activating Token Index: 919

Text #15

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.7785. Min Act: -0.0000

Data Index: 1513578 (Python Code)

Max Activating Token Index: 868

Text #16

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.6748. Min Act: -0.0000

Data Index: 1647055 (Python Code)

Max Activating Token Index: 869

Text #17

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.6549. Min Act: -0.0000

Data Index: 1587844 (Python Code)

Max Activating Token Index: 150

Text #18

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.7574. Min Act: -0.0000

Data Index: 1450986 (Python Code)

Max Activating Token Index: 68

Text #19

Max Range: 2.2795. Min Range: -2.2795

Max Act: 1.6549. Min Act: -0.0000

Data Index: 1429538 (Python Code)

Max Activating Token Index: 615