Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3235 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.9256. Min Act: -0.0000

Data Index: 249537 (The Pile)

Max Activating Token Index: 593

Text #1

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.7488. Min Act: -0.0000

Data Index: 1819317 (The Pile)

Max Activating Token Index: 975

Text #2

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.6654. Min Act: -0.0000

Data Index: 227203 (The Pile)

Max Activating Token Index: 636

Text #3

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.6838. Min Act: -0.0000

Data Index: 1404536 (The Pile)

Max Activating Token Index: 1014

Text #4

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.7026. Min Act: -0.0000

Data Index: 957745 (The Pile)

Max Activating Token Index: 156

Text #5

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.6353. Min Act: -0.0000

Data Index: 1405440 (The Pile)

Max Activating Token Index: 997

Text #6

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.6672. Min Act: -0.0000

Data Index: 1276988 (The Pile)

Max Activating Token Index: 792

Text #7

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.6862. Min Act: -0.0000

Data Index: 1579176 (The Pile)

Max Activating Token Index: 765

Text #8

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.5698. Min Act: -0.0000

Data Index: 506635 (The Pile)

Max Activating Token Index: 146

Text #9

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.5686. Min Act: -0.0000

Data Index: 1071437 (The Pile)

Max Activating Token Index: 31

Text #10

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.6431. Min Act: -0.0000

Data Index: 1255287 (The Pile)

Max Activating Token Index: 769

Text #11

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.5498. Min Act: -0.0000

Data Index: 942312 (The Pile)

Max Activating Token Index: 308

Text #12

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.5475. Min Act: -0.0000

Data Index: 597934 (The Pile)

Max Activating Token Index: 629

Text #13

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.5313. Min Act: -0.0000

Data Index: 1230792 (The Pile)

Max Activating Token Index: 190

Text #14

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.5312. Min Act: -0.0000

Data Index: 1864263 (The Pile)

Max Activating Token Index: 526

Text #15

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.5235. Min Act: -0.0000

Data Index: 607332 (The Pile)

Max Activating Token Index: 764

Text #16

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.5333. Min Act: -0.0000

Data Index: 1808421 (The Pile)

Max Activating Token Index: 207

Text #17

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.5197. Min Act: -0.0000

Data Index: 1196456 (The Pile)

Max Activating Token Index: 553

Text #18

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.5592. Min Act: -0.0000

Data Index: 1794271 (The Pile)

Max Activating Token Index: 135

Text #19

Max Range: 0.9256. Min Range: -0.9256

Max Act: 0.4732. Min Act: -0.0000

Data Index: 982895 (The Pile)

Max Activating Token Index: 808