Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 584 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2621. Min Range: -1.2621

Max Act: 1.2621. Min Act: -0.0000

Data Index: 1540164 (Python Code)

Max Activating Token Index: 203

Text #1

Max Range: 1.2621. Min Range: -1.2621

Max Act: 1.1101. Min Act: -0.0000

Data Index: 1529421 (Python Code)

Max Activating Token Index: 93

Text #2

Max Range: 1.2621. Min Range: -1.2621

Max Act: 1.1061. Min Act: -0.0000

Data Index: 1606973 (Python Code)

Max Activating Token Index: 91

Text #3

Max Range: 1.2621. Min Range: -1.2621

Max Act: 1.0770. Min Act: -0.0000

Data Index: 1420965 (Python Code)

Max Activating Token Index: 434

Text #4

Max Range: 1.2621. Min Range: -1.2621

Max Act: 0.9343. Min Act: -0.0000

Data Index: 1508028 (Python Code)

Max Activating Token Index: 250

Text #5

Max Range: 1.2621. Min Range: -1.2621

Max Act: 1.0512. Min Act: -0.0000

Data Index: 1487541 (Python Code)

Max Activating Token Index: 70

Text #6

Max Range: 1.2621. Min Range: -1.2621

Max Act: 1.0781. Min Act: -0.0000

Data Index: 1521027 (Python Code)

Max Activating Token Index: 153

Text #7

Max Range: 1.2621. Min Range: -1.2621

Max Act: 1.1641. Min Act: -0.0000

Data Index: 1422375 (Python Code)

Max Activating Token Index: 146

Text #8

Max Range: 1.2621. Min Range: -1.2621

Max Act: 1.0189. Min Act: -0.0000

Data Index: 1425099 (Python Code)

Max Activating Token Index: 130

Text #9

Max Range: 1.2621. Min Range: -1.2621

Max Act: 0.9890. Min Act: -0.0000

Data Index: 1491375 (Python Code)

Max Activating Token Index: 220

Text #10

Max Range: 1.2621. Min Range: -1.2621

Max Act: 0.9987. Min Act: -0.0001

Data Index: 1495037 (Python Code)

Max Activating Token Index: 119

Text #11

Max Range: 1.2621. Min Range: -1.2621

Max Act: 0.9654. Min Act: -0.0000

Data Index: 1565141 (Python Code)

Max Activating Token Index: 302

Text #12

Max Range: 1.2621. Min Range: -1.2621

Max Act: 0.9386. Min Act: -0.0000

Data Index: 1457022 (Python Code)

Max Activating Token Index: 221

Text #13

Max Range: 1.2621. Min Range: -1.2621

Max Act: 0.9661. Min Act: -0.0000

Data Index: 1466942 (Python Code)

Max Activating Token Index: 407

Text #14

Max Range: 1.2621. Min Range: -1.2621

Max Act: 0.9505. Min Act: -0.0000

Data Index: 1576043 (Python Code)

Max Activating Token Index: 742

Text #15

Max Range: 1.2621. Min Range: -1.2621

Max Act: 0.9448. Min Act: -0.0000

Data Index: 1633543 (Python Code)

Max Activating Token Index: 490

Text #16

Max Range: 1.2621. Min Range: -1.2621

Max Act: 0.9478. Min Act: -0.0000

Data Index: 1467673 (Python Code)

Max Activating Token Index: 858

Text #17

Max Range: 1.2621. Min Range: -1.2621

Max Act: 0.8562. Min Act: -0.0000

Data Index: 1466788 (Python Code)

Max Activating Token Index: 216

Text #18

Max Range: 1.2621. Min Range: -1.2621

Max Act: 1.0318. Min Act: -0.0000

Data Index: 1637503 (Python Code)

Max Activating Token Index: 957

Text #19

Max Range: 1.2621. Min Range: -1.2621

Max Act: 0.9102. Min Act: -0.0000

Data Index: 1392024 (Python Code)

Max Activating Token Index: 677