Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 22 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.2744. Min Act: -0.0002

Data Index: 1476643 (Python Code)

Max Activating Token Index: 507

Text #1

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.2513. Min Act: -0.0002

Data Index: 1486657 (Python Code)

Max Activating Token Index: 39

Text #2

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.2533. Min Act: -0.0002

Data Index: 1507963 (Python Code)

Max Activating Token Index: 723

Text #3

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.2609. Min Act: -0.0002

Data Index: 1488210 (Python Code)

Max Activating Token Index: 722

Text #4

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.2516. Min Act: -0.0002

Data Index: 1387381 (Python Code)

Max Activating Token Index: 119

Text #5

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.2493. Min Act: -0.0002

Data Index: 1362426 (Python Code)

Max Activating Token Index: 276

Text #6

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.2137. Min Act: -0.0002

Data Index: 1554881 (Python Code)

Max Activating Token Index: 24

Text #7

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.2021. Min Act: -0.0002

Data Index: 1656927 (Python Code)

Max Activating Token Index: 949

Text #8

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.2030. Min Act: -0.0002

Data Index: 1578073 (Python Code)

Max Activating Token Index: 266

Text #9

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.1957. Min Act: -0.0002

Data Index: 1483338 (Python Code)

Max Activating Token Index: 28

Text #10

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.1933. Min Act: -0.0002

Data Index: 1635155 (Python Code)

Max Activating Token Index: 52

Text #11

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.1935. Min Act: -0.0002

Data Index: 1507864 (Python Code)

Max Activating Token Index: 65

Text #12

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.1843. Min Act: -0.0002

Data Index: 1601601 (Python Code)

Max Activating Token Index: 996

Text #13

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.1861. Min Act: -0.0002

Data Index: 1423967 (Python Code)

Max Activating Token Index: 362

Text #14

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.1908. Min Act: -0.0002

Data Index: 1624686 (Python Code)

Max Activating Token Index: 688

Text #15

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.1757. Min Act: -0.0002

Data Index: 1437370 (Python Code)

Max Activating Token Index: 301

Text #16

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.1772. Min Act: -0.0002

Data Index: 1615698 (Python Code)

Max Activating Token Index: 201

Text #17

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.1710. Min Act: -0.0002

Data Index: 1424090 (Python Code)

Max Activating Token Index: 66

Text #18

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.1628. Min Act: -0.0002

Data Index: 1531317 (Python Code)

Max Activating Token Index: 4

Text #19

Max Range: 0.2744. Min Range: -0.2744

Max Act: 0.1566. Min Act: -0.0002

Data Index: 1453551 (Python Code)

Max Activating Token Index: 16