Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3621 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1182. Min Range: -2.1182

Max Act: 2.1182. Min Act: 0.0000

Data Index: 1599622 (Python Code)

Max Activating Token Index: 605

Text #1

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.9613. Min Act: 0.0000

Data Index: 1517028 (Python Code)

Max Activating Token Index: 59

Text #2

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.9072. Min Act: 0.0000

Data Index: 1458662 (Python Code)

Max Activating Token Index: 44

Text #3

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.8375. Min Act: 0.0000

Data Index: 1524098 (Python Code)

Max Activating Token Index: 413

Text #4

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.8389. Min Act: -0.0000

Data Index: 1406999 (Python Code)

Max Activating Token Index: 68

Text #5

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.6842. Min Act: 0.0000

Data Index: 1471037 (Python Code)

Max Activating Token Index: 76

Text #6

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.7711. Min Act: 0.0000

Data Index: 1570476 (Python Code)

Max Activating Token Index: 47

Text #7

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.6603. Min Act: 0.0000

Data Index: 1421490 (Python Code)

Max Activating Token Index: 541

Text #8

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.8366. Min Act: 0.0000

Data Index: 1470200 (Python Code)

Max Activating Token Index: 280

Text #9

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.6450. Min Act: 0.0000

Data Index: 1631765 (Python Code)

Max Activating Token Index: 571

Text #10

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.7358. Min Act: 0.0000

Data Index: 1367485 (Python Code)

Max Activating Token Index: 266

Text #11

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.6367. Min Act: 0.0000

Data Index: 1486432 (Python Code)

Max Activating Token Index: 90

Text #12

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.6279. Min Act: 0.0000

Data Index: 1586626 (Python Code)

Max Activating Token Index: 26

Text #13

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.7232. Min Act: 0.0000

Data Index: 1404921 (Python Code)

Max Activating Token Index: 921

Text #14

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.7075. Min Act: 0.0000

Data Index: 1435657 (Python Code)

Max Activating Token Index: 665

Text #15

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.6186. Min Act: -0.0000

Data Index: 1500351 (Python Code)

Max Activating Token Index: 51

Text #16

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.6963. Min Act: 0.0000

Data Index: 1639648 (Python Code)

Max Activating Token Index: 399

Text #17

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.7681. Min Act: 0.0000

Data Index: 1614058 (Python Code)

Max Activating Token Index: 113

Text #18

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.5439. Min Act: 0.0000

Data Index: 1396568 (Python Code)

Max Activating Token Index: 714

Text #19

Max Range: 2.1182. Min Range: -2.1182

Max Act: 1.6092. Min Act: 0.0000

Data Index: 1426957 (Python Code)

Max Activating Token Index: 197