Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2793 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3325. Min Act: -0.0000

Data Index: 1290880 (The Pile)

Max Activating Token Index: 907

Text #1

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3836. Min Act: -0.0000

Data Index: 350349 (The Pile)

Max Activating Token Index: 958

Text #2

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3905. Min Act: -0.0000

Data Index: 1469296 (The Pile)

Max Activating Token Index: 1017

Text #3

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3684. Min Act: -0.0000

Data Index: 133734 (The Pile)

Max Activating Token Index: 684

Text #4

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3706. Min Act: -0.0000

Data Index: 453581 (The Pile)

Max Activating Token Index: 846

Text #5

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3649. Min Act: -0.0000

Data Index: 898904 (The Pile)

Max Activating Token Index: 456

Text #6

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3709. Min Act: -0.0000

Data Index: 479708 (The Pile)

Max Activating Token Index: 901

Text #7

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3077. Min Act: -0.0000

Data Index: 1800278 (The Pile)

Max Activating Token Index: 889

Text #8

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3497. Min Act: -0.0000

Data Index: 1517887 (The Pile)

Max Activating Token Index: 743

Text #9

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.2991. Min Act: -0.0000

Data Index: 731394 (The Pile)

Max Activating Token Index: 916

Text #10

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3501. Min Act: -0.0000

Data Index: 1124810 (The Pile)

Max Activating Token Index: 584

Text #11

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3241. Min Act: -0.0000

Data Index: 1981701 (The Pile)

Max Activating Token Index: 761

Text #12

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3149. Min Act: -0.0000

Data Index: 423355 (The Pile)

Max Activating Token Index: 852

Text #13

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3113. Min Act: -0.0000

Data Index: 1069244 (The Pile)

Max Activating Token Index: 754

Text #14

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3018. Min Act: -0.0000

Data Index: 166541 (The Pile)

Max Activating Token Index: 606

Text #15

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3038. Min Act: -0.0000

Data Index: 1455334 (The Pile)

Max Activating Token Index: 988

Text #16

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.3058. Min Act: -0.0000

Data Index: 1106059 (The Pile)

Max Activating Token Index: 663

Text #17

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.2906. Min Act: -0.0000

Data Index: 1009573 (The Pile)

Max Activating Token Index: 650

Text #18

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.2722. Min Act: -0.0000

Data Index: 1917481 (The Pile)

Max Activating Token Index: 682

Text #19

Max Range: 0.3905. Min Range: -0.3905

Max Act: 0.2671. Min Act: -0.0000

Data Index: 926020 (The Pile)

Max Activating Token Index: 619