Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 1453 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.7089. Min Range: -5.7089

Max Act: 5.7089. Min Act: -0.0000

Data Index: 962196 (The Pile)

Max Activating Token Index: 14

Text #1

Max Range: 5.7089. Min Range: -5.7089

Max Act: 5.5689. Min Act: -0.0000

Data Index: 415350 (The Pile)

Max Activating Token Index: 11

Text #2

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.9718. Min Act: -0.0000

Data Index: 1378821 (The Pile)

Max Activating Token Index: 10

Text #3

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.8670. Min Act: -0.0000

Data Index: 933304 (The Pile)

Max Activating Token Index: 30

Text #4

Max Range: 5.7089. Min Range: -5.7089

Max Act: 5.0251. Min Act: -0.0000

Data Index: 1693396 (The Pile)

Max Activating Token Index: 518

Text #5

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.6464. Min Act: -0.0000

Data Index: 1433513 (The Pile)

Max Activating Token Index: 30

Text #6

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.2863. Min Act: -0.0000

Data Index: 166326 (The Pile)

Max Activating Token Index: 616

Text #7

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.4974. Min Act: -0.0000

Data Index: 130564 (The Pile)

Max Activating Token Index: 87

Text #8

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.6378. Min Act: -0.0000

Data Index: 857130 (The Pile)

Max Activating Token Index: 9

Text #9

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.3886. Min Act: -0.0000

Data Index: 674023 (The Pile)

Max Activating Token Index: 21

Text #10

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.5671. Min Act: -0.0000

Data Index: 1559367 (The Pile)

Max Activating Token Index: 15

Text #11

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.1540. Min Act: -0.0000

Data Index: 1384209 (The Pile)

Max Activating Token Index: 69

Text #12

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.3111. Min Act: -0.0000

Data Index: 1965274 (The Pile)

Max Activating Token Index: 938

Text #13

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.1062. Min Act: -0.0000

Data Index: 446991 (The Pile)

Max Activating Token Index: 15

Text #14

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.3968. Min Act: -0.0000

Data Index: 1352075 (The Pile)

Max Activating Token Index: 34

Text #15

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.4059. Min Act: -0.0000

Data Index: 79757 (The Pile)

Max Activating Token Index: 264

Text #16

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.2282. Min Act: -0.0000

Data Index: 49281 (The Pile)

Max Activating Token Index: 18

Text #17

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.3530. Min Act: -0.0000

Data Index: 1658450 (The Pile)

Max Activating Token Index: 24

Text #18

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.2907. Min Act: -0.0000

Data Index: 1621908 (The Pile)

Max Activating Token Index: 898

Text #19

Max Range: 5.7089. Min Range: -5.7089

Max Act: 4.6262. Min Act: -0.0000

Data Index: 1543016 (The Pile)

Max Activating Token Index: 13