Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 1008 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.3912. Min Range: -5.3912

Max Act: 5.3912. Min Act: -0.0000

Data Index: 45518 (The Pile)

Max Activating Token Index: 163

Text #1

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.7932. Min Act: -0.0000

Data Index: 416428 (The Pile)

Max Activating Token Index: 945

Text #2

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.7173. Min Act: -0.0000

Data Index: 1742209 (The Pile)

Max Activating Token Index: 134

Text #3

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.7926. Min Act: -0.0000

Data Index: 1130322 (The Pile)

Max Activating Token Index: 599

Text #4

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.3300. Min Act: -0.0000

Data Index: 823923 (The Pile)

Max Activating Token Index: 202

Text #5

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.1679. Min Act: -0.0000

Data Index: 1547658 (The Pile)

Max Activating Token Index: 548

Text #6

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.2919. Min Act: -0.0000

Data Index: 40012 (The Pile)

Max Activating Token Index: 746

Text #7

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.4831. Min Act: -0.0000

Data Index: 865696 (The Pile)

Max Activating Token Index: 332

Text #8

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.1795. Min Act: -0.0000

Data Index: 659159 (The Pile)

Max Activating Token Index: 920

Text #9

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.0726. Min Act: -0.0000

Data Index: 1865720 (The Pile)

Max Activating Token Index: 742

Text #10

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.1009. Min Act: -0.0000

Data Index: 10974 (The Pile)

Max Activating Token Index: 464

Text #11

Max Range: 5.3912. Min Range: -5.3912

Max Act: 3.9299. Min Act: -0.0000

Data Index: 460300 (The Pile)

Max Activating Token Index: 759

Text #12

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.0884. Min Act: -0.0000

Data Index: 733800 (The Pile)

Max Activating Token Index: 119

Text #13

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.0475. Min Act: -0.0000

Data Index: 1141952 (The Pile)

Max Activating Token Index: 83

Text #14

Max Range: 5.3912. Min Range: -5.3912

Max Act: 3.8987. Min Act: -0.0000

Data Index: 1639148 (The Pile)

Max Activating Token Index: 528

Text #15

Max Range: 5.3912. Min Range: -5.3912

Max Act: 3.8409. Min Act: -0.0000

Data Index: 121643 (The Pile)

Max Activating Token Index: 921

Text #16

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.0000. Min Act: -0.0000

Data Index: 223970 (The Pile)

Max Activating Token Index: 802

Text #17

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.1720. Min Act: -0.0000

Data Index: 1996220 (The Pile)

Max Activating Token Index: 302

Text #18

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.1591. Min Act: -0.0000

Data Index: 1395820 (The Pile)

Max Activating Token Index: 769

Text #19

Max Range: 5.3912. Min Range: -5.3912

Max Act: 4.1315. Min Act: -0.0000

Data Index: 1544629 (The Pile)

Max Activating Token Index: 471