Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3512 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.9345. Min Range: -3.9345

Max Act: 3.9345. Min Act: -0.0000

Data Index: 1619725 (Python Code)

Max Activating Token Index: 323

Text #1

Max Range: 3.9345. Min Range: -3.9345

Max Act: 3.1233. Min Act: -0.0000

Data Index: 1290351 (C4 (Web Text))

Max Activating Token Index: 586

Text #2

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.9349. Min Act: -0.0000

Data Index: 1385580 (Python Code)

Max Activating Token Index: 328

Text #3

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.9851. Min Act: -0.0000

Data Index: 1288018 (C4 (Web Text))

Max Activating Token Index: 228

Text #4

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.7169. Min Act: -0.0000

Data Index: 1361084 (Python Code)

Max Activating Token Index: 835

Text #5

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.6554. Min Act: -0.0000

Data Index: 1520989 (Python Code)

Max Activating Token Index: 213

Text #6

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.3406. Min Act: -0.0000

Data Index: 1441144 (Python Code)

Max Activating Token Index: 250

Text #7

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.4269. Min Act: -0.0000

Data Index: 1543973 (Python Code)

Max Activating Token Index: 37

Text #8

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.4299. Min Act: -0.0000

Data Index: 1131281 (C4 (Web Text))

Max Activating Token Index: 375

Text #9

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.3637. Min Act: -0.0000

Data Index: 1336475 (C4 (Web Text))

Max Activating Token Index: 732

Text #10

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.2449. Min Act: -0.0000

Data Index: 179506 (C4 (Web Text))

Max Activating Token Index: 163

Text #11

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.3311. Min Act: -0.0000

Data Index: 1447282 (Python Code)

Max Activating Token Index: 333

Text #12

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.0539. Min Act: -0.0000

Data Index: 1647678 (Python Code)

Max Activating Token Index: 378

Text #13

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.4086. Min Act: -0.0000

Data Index: 1218812 (C4 (Web Text))

Max Activating Token Index: 814

Text #14

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.2505. Min Act: -0.0000

Data Index: 1369852 (Python Code)

Max Activating Token Index: 884

Text #15

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.2394. Min Act: -0.0000

Data Index: 1170566 (C4 (Web Text))

Max Activating Token Index: 865

Text #16

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.1059. Min Act: -0.0000

Data Index: 666436 (C4 (Web Text))

Max Activating Token Index: 743

Text #17

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.1891. Min Act: -0.0000

Data Index: 286249 (C4 (Web Text))

Max Activating Token Index: 878

Text #18

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.1838. Min Act: -0.0000

Data Index: 1625783 (Python Code)

Max Activating Token Index: 7

Text #19

Max Range: 3.9345. Min Range: -3.9345

Max Act: 2.0709. Min Act: -0.0000

Data Index: 1580730 (Python Code)

Max Activating Token Index: 95