Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 3030 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.3688. Min Act: -0.0000

Data Index: 1442890 (The Pile)

Max Activating Token Index: 193

Text #1

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.3889. Min Act: -0.0000

Data Index: 1013137 (The Pile)

Max Activating Token Index: 865

Text #2

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.3624. Min Act: -0.0000

Data Index: 1995172 (The Pile)

Max Activating Token Index: 746

Text #3

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.3353. Min Act: -0.0000

Data Index: 676710 (The Pile)

Max Activating Token Index: 248

Text #4

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.3512. Min Act: -0.0000

Data Index: 1340373 (The Pile)

Max Activating Token Index: 935

Text #5

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.3112. Min Act: -0.0000

Data Index: 1860028 (The Pile)

Max Activating Token Index: 329

Text #6

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2981. Min Act: -0.0000

Data Index: 160841 (The Pile)

Max Activating Token Index: 214

Text #7

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2880. Min Act: -0.0000

Data Index: 1051150 (The Pile)

Max Activating Token Index: 295

Text #8

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2812. Min Act: -0.0000

Data Index: 209712 (The Pile)

Max Activating Token Index: 123

Text #9

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2839. Min Act: -0.0000

Data Index: 645754 (The Pile)

Max Activating Token Index: 87

Text #10

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.3037. Min Act: -0.0000

Data Index: 90171 (The Pile)

Max Activating Token Index: 433

Text #11

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2878. Min Act: -0.0000

Data Index: 973115 (The Pile)

Max Activating Token Index: 621

Text #12

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2685. Min Act: -0.0000

Data Index: 1749908 (The Pile)

Max Activating Token Index: 579

Text #13

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2656. Min Act: -0.0000

Data Index: 1328906 (The Pile)

Max Activating Token Index: 442

Text #14

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2575. Min Act: -0.0000

Data Index: 575 (The Pile)

Max Activating Token Index: 557

Text #15

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2861. Min Act: -0.0000

Data Index: 1081473 (The Pile)

Max Activating Token Index: 655

Text #16

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2395. Min Act: -0.0000

Data Index: 1639204 (The Pile)

Max Activating Token Index: 601

Text #17

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2610. Min Act: -0.0000

Data Index: 1870536 (The Pile)

Max Activating Token Index: 369

Text #18

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2672. Min Act: -0.0000

Data Index: 718264 (The Pile)

Max Activating Token Index: 118

Text #19

Max Range: 0.3889. Min Range: -0.3889

Max Act: 0.2708. Min Act: -0.0000

Data Index: 1812542 (The Pile)

Max Activating Token Index: 890