Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 493 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.3706. Min Act: -0.0000

Data Index: 240497 (The Pile)

Max Activating Token Index: 902

Text #1

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.3392. Min Act: -0.0000

Data Index: 841896 (The Pile)

Max Activating Token Index: 635

Text #2

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.3319. Min Act: -0.0000

Data Index: 316155 (The Pile)

Max Activating Token Index: 422

Text #3

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.3127. Min Act: -0.0000

Data Index: 373829 (The Pile)

Max Activating Token Index: 592

Text #4

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.3143. Min Act: -0.0000

Data Index: 1484762 (The Pile)

Max Activating Token Index: 569

Text #5

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.3218. Min Act: -0.0000

Data Index: 1976674 (The Pile)

Max Activating Token Index: 1011

Text #6

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.3187. Min Act: -0.0000

Data Index: 990158 (The Pile)

Max Activating Token Index: 462

Text #7

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.3128. Min Act: -0.0000

Data Index: 492569 (The Pile)

Max Activating Token Index: 41

Text #8

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.2926. Min Act: -0.0000

Data Index: 818192 (The Pile)

Max Activating Token Index: 412

Text #9

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.2918. Min Act: -0.0000

Data Index: 1774104 (The Pile)

Max Activating Token Index: 1004

Text #10

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.3048. Min Act: -0.0000

Data Index: 1919750 (The Pile)

Max Activating Token Index: 198

Text #11

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.2868. Min Act: -0.0000

Data Index: 1762392 (The Pile)

Max Activating Token Index: 764

Text #12

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.3076. Min Act: -0.0000

Data Index: 1871265 (The Pile)

Max Activating Token Index: 481

Text #13

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.2784. Min Act: -0.0000

Data Index: 525192 (The Pile)

Max Activating Token Index: 812

Text #14

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.2692. Min Act: -0.0000

Data Index: 805588 (The Pile)

Max Activating Token Index: 536

Text #15

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.2815. Min Act: -0.0000

Data Index: 1666330 (The Pile)

Max Activating Token Index: 450

Text #16

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.2611. Min Act: -0.0000

Data Index: 1261801 (The Pile)

Max Activating Token Index: 683

Text #17

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.2722. Min Act: -0.0000

Data Index: 209401 (The Pile)

Max Activating Token Index: 222

Text #18

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.2618. Min Act: -0.0000

Data Index: 619755 (The Pile)

Max Activating Token Index: 909

Text #19

Max Range: 0.3706. Min Range: -0.3706

Max Act: 0.2676. Min Act: -0.0000

Data Index: 10275 (The Pile)

Max Activating Token Index: 23