Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 1600 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.7607. Min Act: -0.0000

Data Index: 1305419 (The Pile)

Max Activating Token Index: 192

Text #1

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.7158. Min Act: -0.0000

Data Index: 1881178 (The Pile)

Max Activating Token Index: 970

Text #2

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.7293. Min Act: -0.0000

Data Index: 742468 (The Pile)

Max Activating Token Index: 972

Text #3

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.7236. Min Act: -0.0000

Data Index: 1053588 (The Pile)

Max Activating Token Index: 827

Text #4

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.6983. Min Act: -0.0000

Data Index: 859867 (The Pile)

Max Activating Token Index: 697

Text #5

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.6757. Min Act: -0.0000

Data Index: 886417 (The Pile)

Max Activating Token Index: 38

Text #6

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.6500. Min Act: -0.0000

Data Index: 1322995 (The Pile)

Max Activating Token Index: 34

Text #7

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.6640. Min Act: -0.0000

Data Index: 1856578 (The Pile)

Max Activating Token Index: 918

Text #8

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.6811. Min Act: -0.0000

Data Index: 805825 (The Pile)

Max Activating Token Index: 791

Text #9

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.6328. Min Act: -0.0000

Data Index: 1842388 (The Pile)

Max Activating Token Index: 818

Text #10

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.6218. Min Act: -0.0000

Data Index: 426642 (The Pile)

Max Activating Token Index: 141

Text #11

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.6194. Min Act: -0.0000

Data Index: 637728 (The Pile)

Max Activating Token Index: 160

Text #12

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.6279. Min Act: -0.0000

Data Index: 265562 (The Pile)

Max Activating Token Index: 569

Text #13

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.6078. Min Act: -0.0000

Data Index: 1630354 (The Pile)

Max Activating Token Index: 169

Text #14

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.5857. Min Act: -0.0000

Data Index: 1598564 (The Pile)

Max Activating Token Index: 947

Text #15

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.5822. Min Act: -0.0000

Data Index: 913380 (The Pile)

Max Activating Token Index: 494

Text #16

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.6189. Min Act: -0.0000

Data Index: 576984 (The Pile)

Max Activating Token Index: 897

Text #17

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.6207. Min Act: -0.0000

Data Index: 234091 (The Pile)

Max Activating Token Index: 963

Text #18

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.5786. Min Act: -0.0000

Data Index: 994612 (The Pile)

Max Activating Token Index: 249

Text #19

Max Range: 0.7607. Min Range: -0.7607

Max Act: 0.5467. Min Act: -0.0000

Data Index: 1877178 (The Pile)

Max Activating Token Index: 533