Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 78 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.0719. Min Range: -1.0719

Max Act: 1.0719. Min Act: -0.0000

Data Index: 809984 (The Pile)

Max Activating Token Index: 13

Text #1

Max Range: 1.0719. Min Range: -1.0719

Max Act: 1.0620. Min Act: -0.0000

Data Index: 263385 (The Pile)

Max Activating Token Index: 857

Text #2

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8908. Min Act: -0.0000

Data Index: 1929221 (The Pile)

Max Activating Token Index: 920

Text #3

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.9204. Min Act: -0.0000

Data Index: 149982 (The Pile)

Max Activating Token Index: 889

Text #4

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8873. Min Act: -0.0000

Data Index: 1679680 (The Pile)

Max Activating Token Index: 504

Text #5

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8791. Min Act: -0.0000

Data Index: 1564092 (The Pile)

Max Activating Token Index: 9

Text #6

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8982. Min Act: -0.0000

Data Index: 1140780 (The Pile)

Max Activating Token Index: 992

Text #7

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8895. Min Act: -0.0000

Data Index: 1303281 (The Pile)

Max Activating Token Index: 410

Text #8

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8514. Min Act: -0.0000

Data Index: 1768032 (The Pile)

Max Activating Token Index: 537

Text #9

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.7680. Min Act: -0.0000

Data Index: 1112886 (The Pile)

Max Activating Token Index: 726

Text #10

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8562. Min Act: -0.0000

Data Index: 257025 (The Pile)

Max Activating Token Index: 493

Text #11

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8207. Min Act: -0.0000

Data Index: 1842755 (The Pile)

Max Activating Token Index: 422

Text #12

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.7906. Min Act: -0.0000

Data Index: 368549 (The Pile)

Max Activating Token Index: 199

Text #13

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8188. Min Act: -0.0000

Data Index: 114260 (The Pile)

Max Activating Token Index: 446

Text #14

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8428. Min Act: -0.0000

Data Index: 1610610 (The Pile)

Max Activating Token Index: 625

Text #15

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8545. Min Act: -0.0000

Data Index: 1171815 (The Pile)

Max Activating Token Index: 376

Text #16

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.7923. Min Act: -0.0000

Data Index: 1412763 (The Pile)

Max Activating Token Index: 458

Text #17

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8953. Min Act: -0.0000

Data Index: 883615 (The Pile)

Max Activating Token Index: 647

Text #18

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8122. Min Act: -0.0000

Data Index: 38462 (The Pile)

Max Activating Token Index: 609

Text #19

Max Range: 1.0719. Min Range: -1.0719

Max Act: 0.8409. Min Act: -0.0000

Data Index: 517286 (The Pile)

Max Activating Token Index: 776