Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 409 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.8288. Min Act: -0.0000

Data Index: 1580541 (Python Code)

Max Activating Token Index: 276

Text #1

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.5263. Min Act: -0.0000

Data Index: 1579960 (Python Code)

Max Activating Token Index: 47

Text #2

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.5029. Min Act: -0.0000

Data Index: 1511882 (Python Code)

Max Activating Token Index: 588

Text #3

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.5481. Min Act: -0.0000

Data Index: 1442880 (Python Code)

Max Activating Token Index: 421

Text #4

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.4247. Min Act: -0.0000

Data Index: 1570552 (Python Code)

Max Activating Token Index: 214

Text #5

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.5237. Min Act: -0.0000

Data Index: 1524095 (Python Code)

Max Activating Token Index: 293

Text #6

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.2615. Min Act: -0.0000

Data Index: 1544681 (Python Code)

Max Activating Token Index: 748

Text #7

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.2900. Min Act: -0.0000

Data Index: 1395183 (Python Code)

Max Activating Token Index: 64

Text #8

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.2248. Min Act: -0.0000

Data Index: 1500223 (Python Code)

Max Activating Token Index: 213

Text #9

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.3628. Min Act: -0.0000

Data Index: 1437962 (Python Code)

Max Activating Token Index: 701

Text #10

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.2306. Min Act: -0.0000

Data Index: 1382187 (Python Code)

Max Activating Token Index: 195

Text #11

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.2185. Min Act: -0.0000

Data Index: 1465323 (Python Code)

Max Activating Token Index: 933

Text #12

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.1915. Min Act: -0.0000

Data Index: 1489013 (Python Code)

Max Activating Token Index: 918

Text #13

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.1951. Min Act: -0.0000

Data Index: 1527771 (Python Code)

Max Activating Token Index: 101

Text #14

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.1637. Min Act: -0.0000

Data Index: 1490447 (Python Code)

Max Activating Token Index: 778

Text #15

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.2735. Min Act: -0.0000

Data Index: 1488479 (Python Code)

Max Activating Token Index: 739

Text #16

Max Range: 2.8288. Min Range: -2.8288

Max Act: 1.9852. Min Act: -0.0000

Data Index: 1445818 (Python Code)

Max Activating Token Index: 573

Text #17

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.0881. Min Act: -0.0000

Data Index: 1625434 (Python Code)

Max Activating Token Index: 186

Text #18

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.1779. Min Act: -0.0000

Data Index: 1566781 (Python Code)

Max Activating Token Index: 40

Text #19

Max Range: 2.8288. Min Range: -2.8288

Max Act: 2.0505. Min Act: -0.0000

Data Index: 1410325 (Python Code)

Max Activating Token Index: 330