Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2791 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.7666. Min Act: -0.0000

Data Index: 1746879 (The Pile)

Max Activating Token Index: 962

Text #1

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.6234. Min Act: -0.0000

Data Index: 1672014 (The Pile)

Max Activating Token Index: 972

Text #2

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.5885. Min Act: -0.0000

Data Index: 1439756 (The Pile)

Max Activating Token Index: 906

Text #3

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.5014. Min Act: -0.0000

Data Index: 1158924 (The Pile)

Max Activating Token Index: 816

Text #4

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.4766. Min Act: -0.0000

Data Index: 898976 (The Pile)

Max Activating Token Index: 635

Text #5

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.4746. Min Act: -0.0000

Data Index: 1672343 (The Pile)

Max Activating Token Index: 701

Text #6

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.2936. Min Act: -0.0000

Data Index: 344088 (The Pile)

Max Activating Token Index: 211

Text #7

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.2938. Min Act: -0.0000

Data Index: 332772 (The Pile)

Max Activating Token Index: 863

Text #8

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.4120. Min Act: -0.0000

Data Index: 1002121 (The Pile)

Max Activating Token Index: 886

Text #9

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.4038. Min Act: -0.0000

Data Index: 458263 (The Pile)

Max Activating Token Index: 357

Text #10

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.3907. Min Act: -0.0000

Data Index: 15395 (The Pile)

Max Activating Token Index: 170

Text #11

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.2010. Min Act: -0.0000

Data Index: 464552 (The Pile)

Max Activating Token Index: 935

Text #12

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.3812. Min Act: -0.0000

Data Index: 956879 (The Pile)

Max Activating Token Index: 28

Text #13

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.2316. Min Act: -0.0000

Data Index: 1223792 (The Pile)

Max Activating Token Index: 872

Text #14

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.3613. Min Act: -0.0000

Data Index: 180419 (The Pile)

Max Activating Token Index: 228

Text #15

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.1633. Min Act: -0.0000

Data Index: 521650 (The Pile)

Max Activating Token Index: 621

Text #16

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.1736. Min Act: -0.0000

Data Index: 1012125 (The Pile)

Max Activating Token Index: 824

Text #17

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.1713. Min Act: -0.0000

Data Index: 51243 (The Pile)

Max Activating Token Index: 39

Text #18

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.4526. Min Act: -0.0000

Data Index: 1153024 (The Pile)

Max Activating Token Index: 981

Text #19

Max Range: 3.7666. Min Range: -3.7666

Max Act: 3.3005. Min Act: -0.0000

Data Index: 1412731 (The Pile)

Max Activating Token Index: 852