Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2689 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.5751. Min Range: -3.5751

Max Act: 3.5751. Min Act: -0.0000

Data Index: 1700286 (The Pile)

Max Activating Token Index: 456

Text #1

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.9574. Min Act: -0.0000

Data Index: 556941 (The Pile)

Max Activating Token Index: 942

Text #2

Max Range: 3.5751. Min Range: -3.5751

Max Act: 3.1904. Min Act: -0.0000

Data Index: 1379541 (The Pile)

Max Activating Token Index: 670

Text #3

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.8067. Min Act: -0.0000

Data Index: 42791 (The Pile)

Max Activating Token Index: 753

Text #4

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.5984. Min Act: -0.0000

Data Index: 832614 (The Pile)

Max Activating Token Index: 455

Text #5

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.5649. Min Act: -0.0000

Data Index: 1280889 (The Pile)

Max Activating Token Index: 117

Text #6

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.5476. Min Act: -0.0000

Data Index: 924482 (The Pile)

Max Activating Token Index: 132

Text #7

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.5330. Min Act: -0.0000

Data Index: 739909 (The Pile)

Max Activating Token Index: 461

Text #8

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.3470. Min Act: -0.0000

Data Index: 844936 (The Pile)

Max Activating Token Index: 761

Text #9

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.4508. Min Act: -0.0000

Data Index: 1654916 (The Pile)

Max Activating Token Index: 977

Text #10

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.4561. Min Act: -0.0000

Data Index: 161567 (The Pile)

Max Activating Token Index: 595

Text #11

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.4347. Min Act: -0.0000

Data Index: 996673 (The Pile)

Max Activating Token Index: 461

Text #12

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.4145. Min Act: -0.0000

Data Index: 357600 (The Pile)

Max Activating Token Index: 721

Text #13

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.2960. Min Act: -0.0000

Data Index: 1935187 (The Pile)

Max Activating Token Index: 124

Text #14

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.4124. Min Act: -0.0000

Data Index: 466496 (The Pile)

Max Activating Token Index: 575

Text #15

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.3837. Min Act: -0.0000

Data Index: 509186 (The Pile)

Max Activating Token Index: 327

Text #16

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.3879. Min Act: -0.0000

Data Index: 1492986 (The Pile)

Max Activating Token Index: 28

Text #17

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.3909. Min Act: -0.0000

Data Index: 1852860 (The Pile)

Max Activating Token Index: 554

Text #18

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.3671. Min Act: -0.0000

Data Index: 1314531 (The Pile)

Max Activating Token Index: 268

Text #19

Max Range: 3.5751. Min Range: -3.5751

Max Act: 2.2416. Min Act: -0.0000

Data Index: 278663 (The Pile)

Max Activating Token Index: 691