Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 653 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.5569. Min Range: -5.5569

Max Act: 5.5569. Min Act: -0.0000

Data Index: 1970150 (The Pile)

Max Activating Token Index: 970

Text #1

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.9629. Min Act: -0.0000

Data Index: 1397849 (The Pile)

Max Activating Token Index: 249

Text #2

Max Range: 5.5569. Min Range: -5.5569

Max Act: 5.2269. Min Act: -0.0000

Data Index: 459440 (The Pile)

Max Activating Token Index: 933

Text #3

Max Range: 5.5569. Min Range: -5.5569

Max Act: 5.0081. Min Act: -0.0000

Data Index: 435661 (The Pile)

Max Activating Token Index: 540

Text #4

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.9444. Min Act: -0.0000

Data Index: 796716 (The Pile)

Max Activating Token Index: 154

Text #5

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.7237. Min Act: -0.0000

Data Index: 393461 (The Pile)

Max Activating Token Index: 896

Text #6

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.6598. Min Act: -0.0000

Data Index: 1219137 (The Pile)

Max Activating Token Index: 536

Text #7

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.8329. Min Act: -0.0000

Data Index: 308718 (The Pile)

Max Activating Token Index: 489

Text #8

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.8363. Min Act: -0.0000

Data Index: 726715 (The Pile)

Max Activating Token Index: 779

Text #9

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.7735. Min Act: -0.0000

Data Index: 1786913 (The Pile)

Max Activating Token Index: 132

Text #10

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.7066. Min Act: -0.0000

Data Index: 49467 (The Pile)

Max Activating Token Index: 595

Text #11

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.6907. Min Act: -0.0000

Data Index: 754493 (The Pile)

Max Activating Token Index: 441

Text #12

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.7763. Min Act: -0.0000

Data Index: 501176 (The Pile)

Max Activating Token Index: 809

Text #13

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.5671. Min Act: -0.0000

Data Index: 207681 (The Pile)

Max Activating Token Index: 645

Text #14

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.7171. Min Act: -0.0000

Data Index: 946692 (The Pile)

Max Activating Token Index: 310

Text #15

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.8947. Min Act: -0.0000

Data Index: 1593088 (The Pile)

Max Activating Token Index: 623

Text #16

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.3363. Min Act: -0.0000

Data Index: 1956211 (The Pile)

Max Activating Token Index: 680

Text #17

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.5127. Min Act: -0.0000

Data Index: 216693 (The Pile)

Max Activating Token Index: 420

Text #18

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.5014. Min Act: -0.0000

Data Index: 918405 (The Pile)

Max Activating Token Index: 744

Text #19

Max Range: 5.5569. Min Range: -5.5569

Max Act: 4.4950. Min Act: -0.0000

Data Index: 1317046 (The Pile)

Max Activating Token Index: 652