Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 407 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.2180. Min Act: -0.0000

Data Index: 1986627 (The Pile)

Max Activating Token Index: 788

Text #1

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.2762. Min Act: -0.0000

Data Index: 583893 (The Pile)

Max Activating Token Index: 30

Text #2

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.1108. Min Act: -0.0000

Data Index: 1373426 (The Pile)

Max Activating Token Index: 367

Text #3

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.1800. Min Act: -0.0000

Data Index: 1168632 (The Pile)

Max Activating Token Index: 256

Text #4

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.1833. Min Act: -0.0000

Data Index: 610716 (The Pile)

Max Activating Token Index: 225

Text #5

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.0721. Min Act: -0.0000

Data Index: 867407 (The Pile)

Max Activating Token Index: 35

Text #6

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.0311. Min Act: -0.0000

Data Index: 269217 (The Pile)

Max Activating Token Index: 24

Text #7

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9893. Min Act: -0.0000

Data Index: 906521 (The Pile)

Max Activating Token Index: 561

Text #8

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.0806. Min Act: -0.0000

Data Index: 587323 (The Pile)

Max Activating Token Index: 329

Text #9

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9520. Min Act: -0.0000

Data Index: 221195 (The Pile)

Max Activating Token Index: 625

Text #10

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9674. Min Act: -0.0000

Data Index: 1945771 (The Pile)

Max Activating Token Index: 740

Text #11

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9462. Min Act: -0.0000

Data Index: 6044 (The Pile)

Max Activating Token Index: 115

Text #12

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9422. Min Act: -0.0000

Data Index: 1013981 (The Pile)

Max Activating Token Index: 107

Text #13

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9672. Min Act: -0.0000

Data Index: 1679959 (The Pile)

Max Activating Token Index: 176

Text #14

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.0241. Min Act: -0.0000

Data Index: 1060801 (The Pile)

Max Activating Token Index: 33

Text #15

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9901. Min Act: -0.0000

Data Index: 638631 (The Pile)

Max Activating Token Index: 812

Text #16

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9044. Min Act: -0.0000

Data Index: 283842 (The Pile)

Max Activating Token Index: 333

Text #17

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.8998. Min Act: -0.0000

Data Index: 255474 (The Pile)

Max Activating Token Index: 134

Text #18

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.0108. Min Act: -0.0000

Data Index: 1675290 (The Pile)

Max Activating Token Index: 775

Text #19

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9476. Min Act: -0.0000

Data Index: 43044 (The Pile)

Max Activating Token Index: 188