Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 38 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.8435. Min Act: -0.0000

Data Index: 1523825 (Python Code)

Max Activating Token Index: 854

Text #1

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.7919. Min Act: -0.0000

Data Index: 1637647 (Python Code)

Max Activating Token Index: 1005

Text #2

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.7182. Min Act: -0.0000

Data Index: 1387678 (Python Code)

Max Activating Token Index: 1012

Text #3

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.7590. Min Act: -0.0000

Data Index: 1407345 (Python Code)

Max Activating Token Index: 939

Text #4

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6908. Min Act: -0.0000

Data Index: 1571462 (Python Code)

Max Activating Token Index: 657

Text #5

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.7110. Min Act: -0.0000

Data Index: 1560041 (Python Code)

Max Activating Token Index: 252

Text #6

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.7038. Min Act: -0.0000

Data Index: 1639107 (Python Code)

Max Activating Token Index: 818

Text #7

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.7165. Min Act: -0.0000

Data Index: 1435764 (Python Code)

Max Activating Token Index: 446

Text #8

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6567. Min Act: -0.0000

Data Index: 1614573 (Python Code)

Max Activating Token Index: 971

Text #9

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6760. Min Act: -0.0000

Data Index: 1385467 (Python Code)

Max Activating Token Index: 299

Text #10

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6502. Min Act: -0.0000

Data Index: 1630694 (Python Code)

Max Activating Token Index: 308

Text #11

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6345. Min Act: -0.0000

Data Index: 1527689 (Python Code)

Max Activating Token Index: 965

Text #12

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6281. Min Act: -0.0000

Data Index: 1505393 (Python Code)

Max Activating Token Index: 545

Text #13

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6434. Min Act: -0.0000

Data Index: 1619570 (Python Code)

Max Activating Token Index: 107

Text #14

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6373. Min Act: -0.0000

Data Index: 1480957 (Python Code)

Max Activating Token Index: 880

Text #15

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6392. Min Act: -0.0000

Data Index: 1360451 (Python Code)

Max Activating Token Index: 196

Text #16

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6352. Min Act: -0.0000

Data Index: 1589860 (Python Code)

Max Activating Token Index: 706

Text #17

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6014. Min Act: -0.0000

Data Index: 1599021 (Python Code)

Max Activating Token Index: 213

Text #18

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6228. Min Act: -0.0000

Data Index: 1361962 (Python Code)

Max Activating Token Index: 600

Text #19

Max Range: 0.8435. Min Range: -0.8435

Max Act: 0.6229. Min Act: -0.0000

Data Index: 1368457 (Python Code)

Max Activating Token Index: 227