Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3571 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.5923. Min Range: -5.5923

Max Act: 5.5923. Min Act: -0.0000

Data Index: 1302242 (The Pile)

Max Activating Token Index: 748

Text #1

Max Range: 5.5923. Min Range: -5.5923

Max Act: 5.3821. Min Act: -0.0000

Data Index: 1940244 (The Pile)

Max Activating Token Index: 26

Text #2

Max Range: 5.5923. Min Range: -5.5923

Max Act: 5.2660. Min Act: -0.0000

Data Index: 213049 (The Pile)

Max Activating Token Index: 118

Text #3

Max Range: 5.5923. Min Range: -5.5923

Max Act: 4.6258. Min Act: -0.0000

Data Index: 1687415 (The Pile)

Max Activating Token Index: 21

Text #4

Max Range: 5.5923. Min Range: -5.5923

Max Act: 4.5784. Min Act: -0.0000

Data Index: 1716239 (The Pile)

Max Activating Token Index: 63

Text #5

Max Range: 5.5923. Min Range: -5.5923

Max Act: 4.1628. Min Act: -0.0000

Data Index: 1455800 (The Pile)

Max Activating Token Index: 228

Text #6

Max Range: 5.5923. Min Range: -5.5923

Max Act: 4.1632. Min Act: -0.0000

Data Index: 700085 (The Pile)

Max Activating Token Index: 232

Text #7

Max Range: 5.5923. Min Range: -5.5923

Max Act: 3.8769. Min Act: -0.0000

Data Index: 355106 (The Pile)

Max Activating Token Index: 308

Text #8

Max Range: 5.5923. Min Range: -5.5923

Max Act: 3.7706. Min Act: -0.0000

Data Index: 413466 (The Pile)

Max Activating Token Index: 884

Text #9

Max Range: 5.5923. Min Range: -5.5923

Max Act: 3.6798. Min Act: -0.0000

Data Index: 1316497 (The Pile)

Max Activating Token Index: 35

Text #10

Max Range: 5.5923. Min Range: -5.5923

Max Act: 3.6665. Min Act: -0.0000

Data Index: 609703 (The Pile)

Max Activating Token Index: 42

Text #11

Max Range: 5.5923. Min Range: -5.5923

Max Act: 3.4173. Min Act: -0.0000

Data Index: 837816 (The Pile)

Max Activating Token Index: 661

Text #12

Max Range: 5.5923. Min Range: -5.5923

Max Act: 3.3412. Min Act: -0.0000

Data Index: 1473064 (The Pile)

Max Activating Token Index: 160

Text #13

Max Range: 5.5923. Min Range: -5.5923

Max Act: 3.3174. Min Act: -0.0000

Data Index: 720534 (The Pile)

Max Activating Token Index: 237

Text #14

Max Range: 5.5923. Min Range: -5.5923

Max Act: 2.9902. Min Act: -0.0000

Data Index: 867232 (The Pile)

Max Activating Token Index: 40

Text #15

Max Range: 5.5923. Min Range: -5.5923

Max Act: 3.0726. Min Act: -0.0000

Data Index: 1291269 (The Pile)

Max Activating Token Index: 209

Text #16

Max Range: 5.5923. Min Range: -5.5923

Max Act: 3.2099. Min Act: -0.0000

Data Index: 1424997 (The Pile)

Max Activating Token Index: 271

Text #17

Max Range: 5.5923. Min Range: -5.5923

Max Act: 2.7757. Min Act: -0.0000

Data Index: 1435810 (The Pile)

Max Activating Token Index: 126

Text #18

Max Range: 5.5923. Min Range: -5.5923

Max Act: 3.1851. Min Act: -0.0000

Data Index: 561733 (The Pile)

Max Activating Token Index: 495

Text #19

Max Range: 5.5923. Min Range: -5.5923

Max Act: 2.4614. Min Act: -0.0000

Data Index: 1521014 (The Pile)

Max Activating Token Index: 122