Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4490 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.2575. Min Act: -0.0000

Data Index: 1605161 (Python Code)

Max Activating Token Index: 410

Text #1

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.2524. Min Act: -0.0000

Data Index: 1536279 (Python Code)

Max Activating Token Index: 338

Text #2

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.2332. Min Act: -0.0000

Data Index: 1488470 (Python Code)

Max Activating Token Index: 533

Text #3

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.2352. Min Act: -0.0000

Data Index: 1578637 (Python Code)

Max Activating Token Index: 449

Text #4

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.2386. Min Act: -0.0000

Data Index: 1361278 (Python Code)

Max Activating Token Index: 460

Text #5

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.2208. Min Act: -0.0000

Data Index: 1624249 (Python Code)

Max Activating Token Index: 728

Text #6

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.2223. Min Act: -0.0000

Data Index: 1438195 (Python Code)

Max Activating Token Index: 853

Text #7

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.2197. Min Act: -0.0000

Data Index: 1615600 (Python Code)

Max Activating Token Index: 710

Text #8

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.2045. Min Act: -0.0000

Data Index: 1637340 (Python Code)

Max Activating Token Index: 836

Text #9

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.1967. Min Act: -0.0000

Data Index: 1451036 (Python Code)

Max Activating Token Index: 716

Text #10

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.1912. Min Act: -0.0000

Data Index: 1611839 (Python Code)

Max Activating Token Index: 833

Text #11

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.1906. Min Act: -0.0000

Data Index: 1373987 (Python Code)

Max Activating Token Index: 431

Text #12

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.1941. Min Act: -0.0000

Data Index: 1652379 (Python Code)

Max Activating Token Index: 107

Text #13

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.1783. Min Act: -0.0000

Data Index: 1528106 (Python Code)

Max Activating Token Index: 111

Text #14

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.1686. Min Act: -0.0000

Data Index: 1549566 (Python Code)

Max Activating Token Index: 238

Text #15

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.1252. Min Act: -0.0000

Data Index: 1628974 (Python Code)

Max Activating Token Index: 36

Text #16

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.1185. Min Act: -0.0000

Data Index: 1464036 (Python Code)

Max Activating Token Index: 36

Text #17

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.1084. Min Act: -0.0000

Data Index: 917051 (C4 (Web Text))

Max Activating Token Index: 688

Text #18

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.1042. Min Act: -0.0000

Data Index: 1633691 (Python Code)

Max Activating Token Index: 22

Text #19

Max Range: 0.2575. Min Range: -0.2575

Max Act: 0.1020. Min Act: -0.0000

Data Index: 906968 (C4 (Web Text))

Max Activating Token Index: 628