Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 5021 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.3111. Min Range: -3.3111

Max Act: 3.3111. Min Act: -0.0000

Data Index: 1405680 (The Pile)

Max Activating Token Index: 438

Text #1

Max Range: 3.3111. Min Range: -3.3111

Max Act: 3.0616. Min Act: -0.0000

Data Index: 402683 (The Pile)

Max Activating Token Index: 92

Text #2

Max Range: 3.3111. Min Range: -3.3111

Max Act: 3.2416. Min Act: -0.0000

Data Index: 614536 (The Pile)

Max Activating Token Index: 65

Text #3

Max Range: 3.3111. Min Range: -3.3111

Max Act: 3.2298. Min Act: -0.0000

Data Index: 703993 (The Pile)

Max Activating Token Index: 985

Text #4

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.8051. Min Act: -0.0000

Data Index: 828007 (The Pile)

Max Activating Token Index: 196

Text #5

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.7031. Min Act: -0.0000

Data Index: 1831027 (The Pile)

Max Activating Token Index: 26

Text #6

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.6918. Min Act: -0.0000

Data Index: 24718 (The Pile)

Max Activating Token Index: 174

Text #7

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.5710. Min Act: -0.0000

Data Index: 337228 (The Pile)

Max Activating Token Index: 384

Text #8

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.4548. Min Act: -0.0000

Data Index: 993401 (The Pile)

Max Activating Token Index: 264

Text #9

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.5714. Min Act: -0.0000

Data Index: 1835445 (The Pile)

Max Activating Token Index: 85

Text #10

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.5196. Min Act: -0.0000

Data Index: 1954890 (The Pile)

Max Activating Token Index: 941

Text #11

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.4563. Min Act: -0.0000

Data Index: 674388 (The Pile)

Max Activating Token Index: 245

Text #12

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.5176. Min Act: -0.0000

Data Index: 1991993 (The Pile)

Max Activating Token Index: 860

Text #13

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.3986. Min Act: -0.0000

Data Index: 827441 (The Pile)

Max Activating Token Index: 238

Text #14

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.3603. Min Act: -0.0000

Data Index: 1110812 (The Pile)

Max Activating Token Index: 57

Text #15

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.3140. Min Act: -0.0000

Data Index: 485989 (The Pile)

Max Activating Token Index: 535

Text #16

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.1768. Min Act: -0.0000

Data Index: 189156 (The Pile)

Max Activating Token Index: 423

Text #17

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.2525. Min Act: -0.0000

Data Index: 646012 (The Pile)

Max Activating Token Index: 548

Text #18

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.2589. Min Act: -0.0000

Data Index: 94756 (The Pile)

Max Activating Token Index: 229

Text #19

Max Range: 3.3111. Min Range: -3.3111

Max Act: 2.3479. Min Act: -0.0000

Data Index: 1632307 (The Pile)

Max Activating Token Index: 99