Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3815 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0457. Min Act: -0.0002

Data Index: 1552997 (Python Code)

Max Activating Token Index: 40

Text #1

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0446. Min Act: -0.0002

Data Index: 1583422 (Python Code)

Max Activating Token Index: 39

Text #2

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0432. Min Act: -0.0002

Data Index: 1398269 (Python Code)

Max Activating Token Index: 20

Text #3

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0428. Min Act: -0.0002

Data Index: 1631628 (Python Code)

Max Activating Token Index: 12

Text #4

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0411. Min Act: -0.0002

Data Index: 1557779 (Python Code)

Max Activating Token Index: 5

Text #5

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0425. Min Act: -0.0002

Data Index: 1536491 (Python Code)

Max Activating Token Index: 10

Text #6

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0423. Min Act: -0.0002

Data Index: 1593532 (Python Code)

Max Activating Token Index: 16

Text #7

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0423. Min Act: -0.0002

Data Index: 1367093 (Python Code)

Max Activating Token Index: 40

Text #8

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0423. Min Act: -0.0002

Data Index: 1536941 (Python Code)

Max Activating Token Index: 21

Text #9

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0422. Min Act: -0.0002

Data Index: 1604091 (Python Code)

Max Activating Token Index: 135

Text #10

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0421. Min Act: -0.0002

Data Index: 1591603 (Python Code)

Max Activating Token Index: 10

Text #11

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0420. Min Act: -0.0002

Data Index: 1592350 (Python Code)

Max Activating Token Index: 16

Text #12

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0411. Min Act: -0.0002

Data Index: 1534637 (Python Code)

Max Activating Token Index: 46

Text #13

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0418. Min Act: -0.0002

Data Index: 1455129 (Python Code)

Max Activating Token Index: 35

Text #14

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0416. Min Act: -0.0002

Data Index: 1446975 (Python Code)

Max Activating Token Index: 12

Text #15

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0408. Min Act: -0.0002

Data Index: 1508824 (Python Code)

Max Activating Token Index: 33

Text #16

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0416. Min Act: -0.0002

Data Index: 1643531 (Python Code)

Max Activating Token Index: 11

Text #17

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0424. Min Act: -0.0002

Data Index: 1645056 (Python Code)

Max Activating Token Index: 11

Text #18

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0415. Min Act: -0.0002

Data Index: 1387484 (Python Code)

Max Activating Token Index: 35

Text #19

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0414. Min Act: -0.0002

Data Index: 1404974 (Python Code)

Max Activating Token Index: 10