Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 1271 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.7083. Min Act: -0.0000

Data Index: 1098874 (The Pile)

Max Activating Token Index: 625

Text #1

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.5332. Min Act: -0.0000

Data Index: 219576 (The Pile)

Max Activating Token Index: 107

Text #2

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.5387. Min Act: -0.0000

Data Index: 1168186 (The Pile)

Max Activating Token Index: 604

Text #3

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.4699. Min Act: -0.0000

Data Index: 1372575 (The Pile)

Max Activating Token Index: 283

Text #4

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.2462. Min Act: -0.0000

Data Index: 957072 (The Pile)

Max Activating Token Index: 32

Text #5

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.3589. Min Act: -0.0000

Data Index: 894358 (The Pile)

Max Activating Token Index: 276

Text #6

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.1498. Min Act: -0.0000

Data Index: 1530045 (The Pile)

Max Activating Token Index: 934

Text #7

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.3123. Min Act: -0.0000

Data Index: 1332704 (The Pile)

Max Activating Token Index: 854

Text #8

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.1913. Min Act: -0.0000

Data Index: 1714842 (The Pile)

Max Activating Token Index: 225

Text #9

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.2498. Min Act: -0.0000

Data Index: 618390 (The Pile)

Max Activating Token Index: 495

Text #10

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.2698. Min Act: -0.0000

Data Index: 1345112 (The Pile)

Max Activating Token Index: 256

Text #11

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.2225. Min Act: -0.0000

Data Index: 572506 (The Pile)

Max Activating Token Index: 457

Text #12

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.2725. Min Act: -0.0000

Data Index: 912802 (The Pile)

Max Activating Token Index: 638

Text #13

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.1959. Min Act: -0.0000

Data Index: 747449 (The Pile)

Max Activating Token Index: 478

Text #14

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.2466. Min Act: -0.0000

Data Index: 1922299 (The Pile)

Max Activating Token Index: 518

Text #15

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.1637. Min Act: -0.0000

Data Index: 584940 (The Pile)

Max Activating Token Index: 1023

Text #16

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.1896. Min Act: -0.0000

Data Index: 274231 (The Pile)

Max Activating Token Index: 53

Text #17

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.1394. Min Act: -0.0000

Data Index: 556695 (The Pile)

Max Activating Token Index: 34

Text #18

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.1671. Min Act: -0.0000

Data Index: 885046 (The Pile)

Max Activating Token Index: 447

Text #19

Max Range: 1.7083. Min Range: -1.7083

Max Act: 1.1559. Min Act: -0.0000

Data Index: 1946553 (The Pile)

Max Activating Token Index: 269