Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3291 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 4.3240. Min Range: -4.3240

Max Act: 4.3240. Min Act: -0.0000

Data Index: 1595394 (The Pile)

Max Activating Token Index: 36

Text #1

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.6616. Min Act: -0.0000

Data Index: 1289162 (The Pile)

Max Activating Token Index: 31

Text #2

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.8545. Min Act: -0.0000

Data Index: 1926641 (The Pile)

Max Activating Token Index: 294

Text #3

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.7078. Min Act: -0.0000

Data Index: 1600462 (The Pile)

Max Activating Token Index: 88

Text #4

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.7572. Min Act: -0.0000

Data Index: 810194 (The Pile)

Max Activating Token Index: 32

Text #5

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.4868. Min Act: -0.0000

Data Index: 1801532 (The Pile)

Max Activating Token Index: 125

Text #6

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.3674. Min Act: -0.0000

Data Index: 35083 (The Pile)

Max Activating Token Index: 385

Text #7

Max Range: 4.3240. Min Range: -4.3240

Max Act: 2.9593. Min Act: -0.0000

Data Index: 1338948 (The Pile)

Max Activating Token Index: 890

Text #8

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.4876. Min Act: -0.0000

Data Index: 1593097 (The Pile)

Max Activating Token Index: 593

Text #9

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.4575. Min Act: -0.0000

Data Index: 1687271 (The Pile)

Max Activating Token Index: 896

Text #10

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.2790. Min Act: -0.0000

Data Index: 1202384 (The Pile)

Max Activating Token Index: 189

Text #11

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.4339. Min Act: -0.0000

Data Index: 541975 (The Pile)

Max Activating Token Index: 665

Text #12

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.5458. Min Act: -0.0000

Data Index: 1330617 (The Pile)

Max Activating Token Index: 74

Text #13

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.1402. Min Act: -0.0000

Data Index: 1663525 (The Pile)

Max Activating Token Index: 171

Text #14

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.5459. Min Act: -0.0000

Data Index: 1072980 (The Pile)

Max Activating Token Index: 146

Text #15

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.6246. Min Act: -0.0000

Data Index: 1537497 (The Pile)

Max Activating Token Index: 590

Text #16

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.3496. Min Act: -0.0000

Data Index: 575828 (The Pile)

Max Activating Token Index: 505

Text #17

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.3582. Min Act: -0.0000

Data Index: 1620823 (The Pile)

Max Activating Token Index: 87

Text #18

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.3225. Min Act: -0.0000

Data Index: 1959203 (The Pile)

Max Activating Token Index: 75

Text #19

Max Range: 4.3240. Min Range: -4.3240

Max Act: 3.3241. Min Act: -0.0000

Data Index: 1067132 (The Pile)

Max Activating Token Index: 684