Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 516 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.8475. Min Act: -0.0000

Data Index: 274516 (The Pile)

Max Activating Token Index: 484

Text #1

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.7463. Min Act: -0.0000

Data Index: 1157834 (The Pile)

Max Activating Token Index: 144

Text #2

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.8314. Min Act: -0.0000

Data Index: 1342900 (The Pile)

Max Activating Token Index: 179

Text #3

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.8134. Min Act: -0.0000

Data Index: 594491 (The Pile)

Max Activating Token Index: 691

Text #4

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.8324. Min Act: -0.0000

Data Index: 1103619 (The Pile)

Max Activating Token Index: 612

Text #5

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.7097. Min Act: -0.0000

Data Index: 1284523 (The Pile)

Max Activating Token Index: 107

Text #6

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.8038. Min Act: -0.0000

Data Index: 1411903 (The Pile)

Max Activating Token Index: 861

Text #7

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.7681. Min Act: -0.0000

Data Index: 555443 (The Pile)

Max Activating Token Index: 910

Text #8

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.6406. Min Act: -0.0000

Data Index: 1753216 (The Pile)

Max Activating Token Index: 1019

Text #9

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.6799. Min Act: -0.0000

Data Index: 1460815 (The Pile)

Max Activating Token Index: 1013

Text #10

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.7036. Min Act: -0.0000

Data Index: 989884 (The Pile)

Max Activating Token Index: 930

Text #11

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.7879. Min Act: -0.0000

Data Index: 1727187 (The Pile)

Max Activating Token Index: 53

Text #12

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.6840. Min Act: -0.0000

Data Index: 1690390 (The Pile)

Max Activating Token Index: 667

Text #13

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.6640. Min Act: -0.0000

Data Index: 363715 (The Pile)

Max Activating Token Index: 470

Text #14

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.6744. Min Act: -0.0000

Data Index: 1488467 (The Pile)

Max Activating Token Index: 449

Text #15

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.6710. Min Act: -0.0000

Data Index: 1053004 (The Pile)

Max Activating Token Index: 1022

Text #16

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.6647. Min Act: -0.0000

Data Index: 1399684 (The Pile)

Max Activating Token Index: 255

Text #17

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.7038. Min Act: -0.0000

Data Index: 1013121 (The Pile)

Max Activating Token Index: 709

Text #18

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.6817. Min Act: -0.0000

Data Index: 1130149 (The Pile)

Max Activating Token Index: 138

Text #19

Max Range: 0.8475. Min Range: -0.8475

Max Act: 0.6391. Min Act: -0.0000

Data Index: 242682 (The Pile)

Max Activating Token Index: 740