Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 701 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 3.9862. Min Range: -3.9862

Max Act: 3.9862. Min Act: -0.0000

Data Index: 1620748 (The Pile)

Max Activating Token Index: 397

Text #1

Max Range: 3.9862. Min Range: -3.9862

Max Act: 3.2895. Min Act: -0.0000

Data Index: 1290996 (The Pile)

Max Activating Token Index: 7

Text #2

Max Range: 3.9862. Min Range: -3.9862

Max Act: 3.5196. Min Act: -0.0000

Data Index: 1920924 (The Pile)

Max Activating Token Index: 67

Text #3

Max Range: 3.9862. Min Range: -3.9862

Max Act: 3.4960. Min Act: -0.0000

Data Index: 1898454 (The Pile)

Max Activating Token Index: 473

Text #4

Max Range: 3.9862. Min Range: -3.9862

Max Act: 3.6587. Min Act: -0.0000

Data Index: 956668 (The Pile)

Max Activating Token Index: 30

Text #5

Max Range: 3.9862. Min Range: -3.9862

Max Act: 3.6389. Min Act: -0.0000

Data Index: 621492 (The Pile)

Max Activating Token Index: 79

Text #6

Max Range: 3.9862. Min Range: -3.9862

Max Act: 3.1400. Min Act: -0.0000

Data Index: 1113567 (The Pile)

Max Activating Token Index: 178

Text #7

Max Range: 3.9862. Min Range: -3.9862

Max Act: 3.2025. Min Act: -0.0000

Data Index: 1928762 (The Pile)

Max Activating Token Index: 306

Text #8

Max Range: 3.9862. Min Range: -3.9862

Max Act: 3.0342. Min Act: -0.0000

Data Index: 1367081 (The Pile)

Max Activating Token Index: 290

Text #9

Max Range: 3.9862. Min Range: -3.9862

Max Act: 3.2935. Min Act: -0.0000

Data Index: 1011749 (The Pile)

Max Activating Token Index: 9

Text #10

Max Range: 3.9862. Min Range: -3.9862

Max Act: 2.9899. Min Act: -0.0000

Data Index: 1006405 (The Pile)

Max Activating Token Index: 872

Text #11

Max Range: 3.9862. Min Range: -3.9862

Max Act: 2.8788. Min Act: -0.0000

Data Index: 200639 (The Pile)

Max Activating Token Index: 7

Text #12

Max Range: 3.9862. Min Range: -3.9862

Max Act: 2.8317. Min Act: -0.0000

Data Index: 1914957 (The Pile)

Max Activating Token Index: 8

Text #13

Max Range: 3.9862. Min Range: -3.9862

Max Act: 2.9689. Min Act: -0.0000

Data Index: 294401 (The Pile)

Max Activating Token Index: 191

Text #14

Max Range: 3.9862. Min Range: -3.9862

Max Act: 3.0709. Min Act: -0.0000

Data Index: 1974550 (The Pile)

Max Activating Token Index: 346

Text #15

Max Range: 3.9862. Min Range: -3.9862

Max Act: 2.9138. Min Act: -0.0000

Data Index: 1509069 (The Pile)

Max Activating Token Index: 226

Text #16

Max Range: 3.9862. Min Range: -3.9862

Max Act: 3.0540. Min Act: -0.0000

Data Index: 647846 (The Pile)

Max Activating Token Index: 382

Text #17

Max Range: 3.9862. Min Range: -3.9862

Max Act: 2.9075. Min Act: -0.0000

Data Index: 481868 (The Pile)

Max Activating Token Index: 6

Text #18

Max Range: 3.9862. Min Range: -3.9862

Max Act: 2.7441. Min Act: -0.0000

Data Index: 1685356 (The Pile)

Max Activating Token Index: 6

Text #19

Max Range: 3.9862. Min Range: -3.9862

Max Act: 2.8358. Min Act: -0.0000

Data Index: 265161 (The Pile)

Max Activating Token Index: 1022