Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4494 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.3801. Min Range: -5.3801

Max Act: 5.0698. Min Act: -0.0000

Data Index: 197858 (The Pile)

Max Activating Token Index: 179

Text #1

Max Range: 5.3801. Min Range: -5.3801

Max Act: 5.3801. Min Act: -0.0000

Data Index: 822192 (The Pile)

Max Activating Token Index: 1009

Text #2

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.9061. Min Act: -0.0000

Data Index: 31385 (The Pile)

Max Activating Token Index: 1018

Text #3

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.9386. Min Act: -0.0000

Data Index: 539436 (The Pile)

Max Activating Token Index: 891

Text #4

Max Range: 5.3801. Min Range: -5.3801

Max Act: 5.0922. Min Act: -0.0000

Data Index: 681548 (The Pile)

Max Activating Token Index: 13

Text #5

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.7685. Min Act: -0.0000

Data Index: 1193385 (The Pile)

Max Activating Token Index: 962

Text #6

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.8849. Min Act: -0.0000

Data Index: 383630 (The Pile)

Max Activating Token Index: 452

Text #7

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.6865. Min Act: -0.0000

Data Index: 1874670 (The Pile)

Max Activating Token Index: 353

Text #8

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.6701. Min Act: -0.0000

Data Index: 1535605 (The Pile)

Max Activating Token Index: 748

Text #9

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.6712. Min Act: -0.0000

Data Index: 611767 (The Pile)

Max Activating Token Index: 79

Text #10

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.6693. Min Act: -0.0000

Data Index: 1064700 (The Pile)

Max Activating Token Index: 539

Text #11

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.5109. Min Act: -0.0000

Data Index: 1788681 (The Pile)

Max Activating Token Index: 495

Text #12

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.2845. Min Act: -0.0000

Data Index: 129820 (The Pile)

Max Activating Token Index: 866

Text #13

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.4263. Min Act: -0.0000

Data Index: 745113 (The Pile)

Max Activating Token Index: 472

Text #14

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.2016. Min Act: -0.0000

Data Index: 187344 (The Pile)

Max Activating Token Index: 962

Text #15

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.3919. Min Act: -0.0000

Data Index: 1846549 (The Pile)

Max Activating Token Index: 562

Text #16

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.5540. Min Act: -0.0000

Data Index: 850522 (The Pile)

Max Activating Token Index: 28

Text #17

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.3515. Min Act: -0.0000

Data Index: 886354 (The Pile)

Max Activating Token Index: 288

Text #18

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.1664. Min Act: -0.0000

Data Index: 596811 (The Pile)

Max Activating Token Index: 659

Text #19

Max Range: 5.3801. Min Range: -5.3801

Max Act: 4.4266. Min Act: -0.0000

Data Index: 862692 (The Pile)

Max Activating Token Index: 898