Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2137 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8389. Min Range: -1.8389

Max Act: 1.7890. Min Act: -0.0000

Data Index: 1634912 (Python Code)

Max Activating Token Index: 600

Text #1

Max Range: 1.8389. Min Range: -1.8389

Max Act: 1.8389. Min Act: -0.0000

Data Index: 1548843 (Python Code)

Max Activating Token Index: 965

Text #2

Max Range: 1.8389. Min Range: -1.8389

Max Act: 1.4802. Min Act: -0.0000

Data Index: 1384460 (Python Code)

Max Activating Token Index: 837

Text #3

Max Range: 1.8389. Min Range: -1.8389

Max Act: 1.4352. Min Act: -0.0000

Data Index: 1368675 (Python Code)

Max Activating Token Index: 63

Text #4

Max Range: 1.8389. Min Range: -1.8389

Max Act: 1.4598. Min Act: -0.0000

Data Index: 1483045 (Python Code)

Max Activating Token Index: 140

Text #5

Max Range: 1.8389. Min Range: -1.8389

Max Act: 1.1641. Min Act: -0.0000

Data Index: 1531060 (Python Code)

Max Activating Token Index: 47

Text #6

Max Range: 1.8389. Min Range: -1.8389

Max Act: 1.0704. Min Act: -0.0000

Data Index: 1526902 (Python Code)

Max Activating Token Index: 165

Text #7

Max Range: 1.8389. Min Range: -1.8389

Max Act: 1.0405. Min Act: -0.0000

Data Index: 1511999 (Python Code)

Max Activating Token Index: 158

Text #8

Max Range: 1.8389. Min Range: -1.8389

Max Act: 1.0163. Min Act: -0.0000

Data Index: 1365303 (Python Code)

Max Activating Token Index: 931

Text #9

Max Range: 1.8389. Min Range: -1.8389

Max Act: 0.9849. Min Act: -0.0000

Data Index: 1631451 (Python Code)

Max Activating Token Index: 998

Text #10

Max Range: 1.8389. Min Range: -1.8389

Max Act: 0.8388. Min Act: -0.0000

Data Index: 1410345 (Python Code)

Max Activating Token Index: 367

Text #11

Max Range: 1.8389. Min Range: -1.8389

Max Act: 0.8489. Min Act: -0.0000

Data Index: 1551655 (Python Code)

Max Activating Token Index: 520

Text #12

Max Range: 1.8389. Min Range: -1.8389

Max Act: 0.9068. Min Act: -0.0000

Data Index: 1521571 (Python Code)

Max Activating Token Index: 1001

Text #13

Max Range: 1.8389. Min Range: -1.8389

Max Act: 0.8446. Min Act: -0.0000

Data Index: 1422258 (Python Code)

Max Activating Token Index: 977

Text #14

Max Range: 1.8389. Min Range: -1.8389

Max Act: 0.8641. Min Act: -0.0000

Data Index: 1655455 (Python Code)

Max Activating Token Index: 789

Text #15

Max Range: 1.8389. Min Range: -1.8389

Max Act: 0.7160. Min Act: -0.0000

Data Index: 1478146 (Python Code)

Max Activating Token Index: 54

Text #16

Max Range: 1.8389. Min Range: -1.8389

Max Act: 0.8299. Min Act: -0.0000

Data Index: 1512689 (Python Code)

Max Activating Token Index: 151

Text #17

Max Range: 1.8389. Min Range: -1.8389

Max Act: 0.7981. Min Act: -0.0000

Data Index: 781342 (C4 (Web Text))

Max Activating Token Index: 826

Text #18

Max Range: 1.8389. Min Range: -1.8389

Max Act: 0.7496. Min Act: -0.0000

Data Index: 1646108 (Python Code)

Max Activating Token Index: 92

Text #19

Max Range: 1.8389. Min Range: -1.8389

Max Act: 0.6783. Min Act: -0.0000

Data Index: 1408776 (Python Code)

Max Activating Token Index: 786