Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3844 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 10.2920. Min Range: -10.2920

Max Act: 10.2920. Min Act: -0.0000

Data Index: 1829555 (The Pile)

Max Activating Token Index: 33

Text #1

Max Range: 10.2920. Min Range: -10.2920

Max Act: 9.9786. Min Act: -0.0000

Data Index: 345743 (The Pile)

Max Activating Token Index: 102

Text #2

Max Range: 10.2920. Min Range: -10.2920

Max Act: 9.3443. Min Act: -0.0000

Data Index: 1911933 (The Pile)

Max Activating Token Index: 590

Text #3

Max Range: 10.2920. Min Range: -10.2920

Max Act: 8.6384. Min Act: -0.0000

Data Index: 1798647 (The Pile)

Max Activating Token Index: 350

Text #4

Max Range: 10.2920. Min Range: -10.2920

Max Act: 8.7709. Min Act: -0.0000

Data Index: 1709439 (The Pile)

Max Activating Token Index: 469

Text #5

Max Range: 10.2920. Min Range: -10.2920

Max Act: 8.7025. Min Act: -0.0000

Data Index: 130823 (The Pile)

Max Activating Token Index: 124

Text #6

Max Range: 10.2920. Min Range: -10.2920

Max Act: 8.3223. Min Act: -0.0000

Data Index: 100852 (The Pile)

Max Activating Token Index: 714

Text #7

Max Range: 10.2920. Min Range: -10.2920

Max Act: 8.6401. Min Act: -0.0000

Data Index: 86793 (The Pile)

Max Activating Token Index: 695

Text #8

Max Range: 10.2920. Min Range: -10.2920

Max Act: 8.3771. Min Act: -0.0000

Data Index: 1793313 (The Pile)

Max Activating Token Index: 979

Text #9

Max Range: 10.2920. Min Range: -10.2920

Max Act: 8.1227. Min Act: -0.0000

Data Index: 924087 (The Pile)

Max Activating Token Index: 158

Text #10

Max Range: 10.2920. Min Range: -10.2920

Max Act: 8.1095. Min Act: -0.0000

Data Index: 1688220 (The Pile)

Max Activating Token Index: 285

Text #11

Max Range: 10.2920. Min Range: -10.2920

Max Act: 8.1545. Min Act: -0.0000

Data Index: 368006 (The Pile)

Max Activating Token Index: 165

Text #12

Max Range: 10.2920. Min Range: -10.2920

Max Act: 7.8064. Min Act: -0.0000

Data Index: 10427 (The Pile)

Max Activating Token Index: 93

Text #13

Max Range: 10.2920. Min Range: -10.2920

Max Act: 8.2887. Min Act: -0.0000

Data Index: 146538 (The Pile)

Max Activating Token Index: 880

Text #14

Max Range: 10.2920. Min Range: -10.2920

Max Act: 7.9089. Min Act: -0.0000

Data Index: 450848 (The Pile)

Max Activating Token Index: 659

Text #15

Max Range: 10.2920. Min Range: -10.2920

Max Act: 7.8472. Min Act: -0.0000

Data Index: 474042 (The Pile)

Max Activating Token Index: 321

Text #16

Max Range: 10.2920. Min Range: -10.2920

Max Act: 7.5068. Min Act: -0.0000

Data Index: 1475296 (The Pile)

Max Activating Token Index: 356

Text #17

Max Range: 10.2920. Min Range: -10.2920

Max Act: 7.4719. Min Act: -0.0000

Data Index: 997224 (The Pile)

Max Activating Token Index: 661

Text #18

Max Range: 10.2920. Min Range: -10.2920

Max Act: 7.8140. Min Act: -0.0000

Data Index: 308053 (The Pile)

Max Activating Token Index: 467

Text #19

Max Range: 10.2920. Min Range: -10.2920

Max Act: 7.4640. Min Act: -0.0000

Data Index: 1144183 (The Pile)

Max Activating Token Index: 803