Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2096 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.4117. Min Act: -0.0000

Data Index: 188527 (The Pile)

Max Activating Token Index: 413

Text #1

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.4167. Min Act: -0.0000

Data Index: 298962 (The Pile)

Max Activating Token Index: 359

Text #2

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.3841. Min Act: -0.0000

Data Index: 1798920 (The Pile)

Max Activating Token Index: 278

Text #3

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.3242. Min Act: -0.0000

Data Index: 758629 (The Pile)

Max Activating Token Index: 788

Text #4

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.3360. Min Act: -0.0000

Data Index: 788476 (The Pile)

Max Activating Token Index: 229

Text #5

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.3128. Min Act: -0.0000

Data Index: 1149185 (The Pile)

Max Activating Token Index: 367

Text #6

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.2895. Min Act: -0.0000

Data Index: 1234179 (The Pile)

Max Activating Token Index: 104

Text #7

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.2810. Min Act: -0.0000

Data Index: 397992 (The Pile)

Max Activating Token Index: 319

Text #8

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.2778. Min Act: -0.0000

Data Index: 349193 (The Pile)

Max Activating Token Index: 358

Text #9

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.3018. Min Act: -0.0000

Data Index: 1537838 (The Pile)

Max Activating Token Index: 906

Text #10

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.2979. Min Act: -0.0000

Data Index: 1484141 (The Pile)

Max Activating Token Index: 867

Text #11

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.2652. Min Act: -0.0000

Data Index: 961021 (The Pile)

Max Activating Token Index: 378

Text #12

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.3045. Min Act: -0.0000

Data Index: 291366 (The Pile)

Max Activating Token Index: 696

Text #13

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.2625. Min Act: -0.0000

Data Index: 1559184 (The Pile)

Max Activating Token Index: 25

Text #14

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.2691. Min Act: -0.0000

Data Index: 1127493 (The Pile)

Max Activating Token Index: 881

Text #15

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.2604. Min Act: -0.0000

Data Index: 88524 (The Pile)

Max Activating Token Index: 447

Text #16

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.2676. Min Act: -0.0000

Data Index: 1995177 (The Pile)

Max Activating Token Index: 714

Text #17

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.2581. Min Act: -0.0000

Data Index: 1776166 (The Pile)

Max Activating Token Index: 79

Text #18

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.2504. Min Act: -0.0000

Data Index: 728764 (The Pile)

Max Activating Token Index: 356

Text #19

Max Range: 0.4167. Min Range: -0.4167

Max Act: 0.2543. Min Act: -0.0000

Data Index: 1947368 (The Pile)

Max Activating Token Index: 282