Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 489 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 7.9151. Min Range: -7.9151

Max Act: 7.9151. Min Act: -0.0000

Data Index: 1977586 (The Pile)

Max Activating Token Index: 229

Text #1

Max Range: 7.9151. Min Range: -7.9151

Max Act: 7.0227. Min Act: -0.0000

Data Index: 1640829 (The Pile)

Max Activating Token Index: 884

Text #2

Max Range: 7.9151. Min Range: -7.9151

Max Act: 5.9679. Min Act: -0.0000

Data Index: 778583 (The Pile)

Max Activating Token Index: 459

Text #3

Max Range: 7.9151. Min Range: -7.9151

Max Act: 5.0488. Min Act: -0.0000

Data Index: 1309288 (The Pile)

Max Activating Token Index: 568

Text #4

Max Range: 7.9151. Min Range: -7.9151

Max Act: 5.9364. Min Act: -0.0000

Data Index: 1974039 (The Pile)

Max Activating Token Index: 303

Text #5

Max Range: 7.9151. Min Range: -7.9151

Max Act: 5.3854. Min Act: -0.0000

Data Index: 366503 (The Pile)

Max Activating Token Index: 906

Text #6

Max Range: 7.9151. Min Range: -7.9151

Max Act: 4.5590. Min Act: -0.0000

Data Index: 1697263 (The Pile)

Max Activating Token Index: 933

Text #7

Max Range: 7.9151. Min Range: -7.9151

Max Act: 6.3019. Min Act: -0.0000

Data Index: 1323233 (The Pile)

Max Activating Token Index: 882

Text #8

Max Range: 7.9151. Min Range: -7.9151

Max Act: 6.1165. Min Act: -0.0000

Data Index: 862769 (The Pile)

Max Activating Token Index: 801

Text #9

Max Range: 7.9151. Min Range: -7.9151

Max Act: 6.1054. Min Act: -0.0000

Data Index: 1618570 (The Pile)

Max Activating Token Index: 273

Text #10

Max Range: 7.9151. Min Range: -7.9151

Max Act: 4.2798. Min Act: -0.0000

Data Index: 1946737 (The Pile)

Max Activating Token Index: 950

Text #11

Max Range: 7.9151. Min Range: -7.9151

Max Act: 5.3876. Min Act: -0.0000

Data Index: 646286 (The Pile)

Max Activating Token Index: 745

Text #12

Max Range: 7.9151. Min Range: -7.9151

Max Act: 4.5912. Min Act: -0.0000

Data Index: 584755 (The Pile)

Max Activating Token Index: 215

Text #13

Max Range: 7.9151. Min Range: -7.9151

Max Act: 5.1536. Min Act: -0.0000

Data Index: 6021 (The Pile)

Max Activating Token Index: 949

Text #14

Max Range: 7.9151. Min Range: -7.9151

Max Act: 5.0545. Min Act: -0.0000

Data Index: 138763 (The Pile)

Max Activating Token Index: 826

Text #15

Max Range: 7.9151. Min Range: -7.9151

Max Act: 4.2924. Min Act: -0.0000

Data Index: 346351 (The Pile)

Max Activating Token Index: 971

Text #16

Max Range: 7.9151. Min Range: -7.9151

Max Act: 5.8023. Min Act: -0.0000

Data Index: 1556853 (The Pile)

Max Activating Token Index: 554

Text #17

Max Range: 7.9151. Min Range: -7.9151

Max Act: 4.5023. Min Act: -0.0000

Data Index: 1020867 (The Pile)

Max Activating Token Index: 152

Text #18

Max Range: 7.9151. Min Range: -7.9151

Max Act: 3.8280. Min Act: -0.0000

Data Index: 1544372 (The Pile)

Max Activating Token Index: 136

Text #19

Max Range: 7.9151. Min Range: -7.9151

Max Act: 4.4684. Min Act: -0.0000

Data Index: 919686 (The Pile)

Max Activating Token Index: 830