Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3059 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.5794. Min Range: -5.5794

Max Act: 5.5794. Min Act: -0.0000

Data Index: 1705625 (The Pile)

Max Activating Token Index: 220

Text #1

Max Range: 5.5794. Min Range: -5.5794

Max Act: 4.2105. Min Act: -0.0000

Data Index: 1603378 (The Pile)

Max Activating Token Index: 932

Text #2

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.5874. Min Act: -0.0000

Data Index: 954488 (The Pile)

Max Activating Token Index: 667

Text #3

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.6359. Min Act: -0.0000

Data Index: 617057 (The Pile)

Max Activating Token Index: 70

Text #4

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.5867. Min Act: -0.0000

Data Index: 333387 (The Pile)

Max Activating Token Index: 302

Text #5

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.1636. Min Act: -0.0000

Data Index: 450698 (The Pile)

Max Activating Token Index: 910

Text #6

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.3074. Min Act: -0.0000

Data Index: 1361116 (The Pile)

Max Activating Token Index: 177

Text #7

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.4463. Min Act: -0.0000

Data Index: 874410 (The Pile)

Max Activating Token Index: 104

Text #8

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.4604. Min Act: -0.0000

Data Index: 791309 (The Pile)

Max Activating Token Index: 161

Text #9

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.4463. Min Act: -0.0000

Data Index: 130750 (The Pile)

Max Activating Token Index: 494

Text #10

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.2523. Min Act: -0.0000

Data Index: 732667 (The Pile)

Max Activating Token Index: 343

Text #11

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.3481. Min Act: -0.0000

Data Index: 1831510 (The Pile)

Max Activating Token Index: 618

Text #12

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.4918. Min Act: -0.0000

Data Index: 984561 (The Pile)

Max Activating Token Index: 264

Text #13

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.2849. Min Act: -0.0000

Data Index: 57865 (The Pile)

Max Activating Token Index: 925

Text #14

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.3072. Min Act: -0.0000

Data Index: 1319549 (The Pile)

Max Activating Token Index: 317

Text #15

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.4392. Min Act: -0.0000

Data Index: 407029 (The Pile)

Max Activating Token Index: 538

Text #16

Max Range: 5.5794. Min Range: -5.5794

Max Act: 2.9854. Min Act: -0.0000

Data Index: 1662252 (The Pile)

Max Activating Token Index: 285

Text #17

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.4355. Min Act: -0.0000

Data Index: 1781499 (The Pile)

Max Activating Token Index: 400

Text #18

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.1266. Min Act: -0.0000

Data Index: 494946 (The Pile)

Max Activating Token Index: 409

Text #19

Max Range: 5.5794. Min Range: -5.5794

Max Act: 3.3552. Min Act: -0.0000

Data Index: 529847 (The Pile)

Max Activating Token Index: 691