Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 742 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 4.3536. Min Range: -4.3536

Max Act: 4.3536. Min Act: -0.0000

Data Index: 676560 (The Pile)

Max Activating Token Index: 69

Text #1

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.9834. Min Act: -0.0000

Data Index: 694143 (The Pile)

Max Activating Token Index: 168

Text #2

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.6912. Min Act: -0.0000

Data Index: 817583 (The Pile)

Max Activating Token Index: 49

Text #3

Max Range: 4.3536. Min Range: -4.3536

Max Act: 4.1144. Min Act: -0.0000

Data Index: 14722 (The Pile)

Max Activating Token Index: 814

Text #4

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.8253. Min Act: -0.0000

Data Index: 1873613 (The Pile)

Max Activating Token Index: 355

Text #5

Max Range: 4.3536. Min Range: -4.3536

Max Act: 4.1194. Min Act: -0.0000

Data Index: 1076018 (The Pile)

Max Activating Token Index: 75

Text #6

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.7951. Min Act: -0.0000

Data Index: 1277339 (The Pile)

Max Activating Token Index: 93

Text #7

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.7266. Min Act: -0.0000

Data Index: 221738 (The Pile)

Max Activating Token Index: 495

Text #8

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.2868. Min Act: -0.0000

Data Index: 605973 (The Pile)

Max Activating Token Index: 605

Text #9

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.7029. Min Act: -0.0000

Data Index: 910568 (The Pile)

Max Activating Token Index: 851

Text #10

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.5891. Min Act: -0.0000

Data Index: 900991 (The Pile)

Max Activating Token Index: 298

Text #11

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.7120. Min Act: -0.0000

Data Index: 1053800 (The Pile)

Max Activating Token Index: 877

Text #12

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.4780. Min Act: -0.0000

Data Index: 1644136 (The Pile)

Max Activating Token Index: 11

Text #13

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.4285. Min Act: -0.0000

Data Index: 95128 (The Pile)

Max Activating Token Index: 907

Text #14

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.3031. Min Act: -0.0000

Data Index: 1459326 (The Pile)

Max Activating Token Index: 106

Text #15

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.7073. Min Act: -0.0000

Data Index: 384316 (The Pile)

Max Activating Token Index: 736

Text #16

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.2602. Min Act: -0.0000

Data Index: 112041 (The Pile)

Max Activating Token Index: 236

Text #17

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.2447. Min Act: -0.0000

Data Index: 864366 (The Pile)

Max Activating Token Index: 623

Text #18

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.0871. Min Act: -0.0000

Data Index: 1779927 (The Pile)

Max Activating Token Index: 109

Text #19

Max Range: 4.3536. Min Range: -4.3536

Max Act: 3.8479. Min Act: -0.0000

Data Index: 408416 (The Pile)

Max Activating Token Index: 735