Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 930 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0433. Min Act: -0.0002

Data Index: 1596783 (Python Code)

Max Activating Token Index: 189

Text #1

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0424. Min Act: -0.0002

Data Index: 1393286 (Python Code)

Max Activating Token Index: 159

Text #2

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0391. Min Act: -0.0002

Data Index: 1567416 (Python Code)

Max Activating Token Index: 540

Text #3

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0372. Min Act: -0.0002

Data Index: 1533882 (Python Code)

Max Activating Token Index: 389

Text #4

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0395. Min Act: -0.0002

Data Index: 1561517 (Python Code)

Max Activating Token Index: 49

Text #5

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0368. Min Act: -0.0002

Data Index: 1540131 (Python Code)

Max Activating Token Index: 39

Text #6

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0375. Min Act: -0.0002

Data Index: 1566568 (Python Code)

Max Activating Token Index: 439

Text #7

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0360. Min Act: -0.0002

Data Index: 1567074 (Python Code)

Max Activating Token Index: 17

Text #8

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0348. Min Act: -0.0002

Data Index: 1618545 (Python Code)

Max Activating Token Index: 410

Text #9

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0350. Min Act: -0.0002

Data Index: 1501676 (Python Code)

Max Activating Token Index: 945

Text #10

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0346. Min Act: -0.0002

Data Index: 1542008 (Python Code)

Max Activating Token Index: 689

Text #11

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0338. Min Act: -0.0002

Data Index: 1633057 (Python Code)

Max Activating Token Index: 231

Text #12

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0325. Min Act: -0.0002

Data Index: 1438714 (Python Code)

Max Activating Token Index: 881

Text #13

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0322. Min Act: -0.0002

Data Index: 1434021 (Python Code)

Max Activating Token Index: 680

Text #14

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0306. Min Act: -0.0002

Data Index: 1632087 (Python Code)

Max Activating Token Index: 211

Text #15

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0296. Min Act: -0.0002

Data Index: 1436925 (Python Code)

Max Activating Token Index: 181

Text #16

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0307. Min Act: -0.0002

Data Index: 1480493 (Python Code)

Max Activating Token Index: 354

Text #17

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0303. Min Act: -0.0002

Data Index: 1463826 (Python Code)

Max Activating Token Index: 87

Text #18

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0288. Min Act: -0.0002

Data Index: 1589290 (Python Code)

Max Activating Token Index: 133

Text #19

Max Range: 0.0433. Min Range: -0.0433

Max Act: 0.0291. Min Act: -0.0002

Data Index: 1542728 (Python Code)

Max Activating Token Index: 23