Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2847 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1681. Min Act: -0.0003

Data Index: 1395829 (Python Code)

Max Activating Token Index: 368

Text #1

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1659. Min Act: -0.0003

Data Index: 1605982 (Python Code)

Max Activating Token Index: 15

Text #2

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1644. Min Act: -0.0003

Data Index: 1360906 (Python Code)

Max Activating Token Index: 132

Text #3

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1576. Min Act: -0.0003

Data Index: 1377819 (Python Code)

Max Activating Token Index: 100

Text #4

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1622. Min Act: -0.0003

Data Index: 1570989 (Python Code)

Max Activating Token Index: 533

Text #5

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1724. Min Act: -0.0003

Data Index: 1630288 (Python Code)

Max Activating Token Index: 760

Text #6

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1608. Min Act: -0.0003

Data Index: 1652761 (Python Code)

Max Activating Token Index: 147

Text #7

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1586. Min Act: -0.0003

Data Index: 1416326 (Python Code)

Max Activating Token Index: 6

Text #8

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1539. Min Act: -0.0003

Data Index: 1584860 (Python Code)

Max Activating Token Index: 159

Text #9

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1660. Min Act: -0.0003

Data Index: 1456032 (Python Code)

Max Activating Token Index: 873

Text #10

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1323. Min Act: -0.0003

Data Index: 1584454 (Python Code)

Max Activating Token Index: 854

Text #11

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1528. Min Act: -0.0003

Data Index: 1486104 (Python Code)

Max Activating Token Index: 789

Text #12

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1571. Min Act: -0.0003

Data Index: 1582950 (Python Code)

Max Activating Token Index: 42

Text #13

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1437. Min Act: -0.0003

Data Index: 1566593 (Python Code)

Max Activating Token Index: 43

Text #14

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1380. Min Act: -0.0003

Data Index: 1567134 (Python Code)

Max Activating Token Index: 6

Text #15

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1483. Min Act: -0.0003

Data Index: 1562464 (Python Code)

Max Activating Token Index: 594

Text #16

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1329. Min Act: -0.0003

Data Index: 1562797 (Python Code)

Max Activating Token Index: 137

Text #17

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1380. Min Act: -0.0003

Data Index: 1466657 (Python Code)

Max Activating Token Index: 344

Text #18

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1311. Min Act: -0.0003

Data Index: 1360853 (Python Code)

Max Activating Token Index: 343

Text #19

Max Range: 0.1724. Min Range: -0.1724

Max Act: 0.1402. Min Act: -0.0003

Data Index: 1652962 (Python Code)

Max Activating Token Index: 478