Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2472 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.3123. Min Act: -0.0000

Data Index: 1593082 (The Pile)

Max Activating Token Index: 959

Text #1

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.3001. Min Act: -0.0000

Data Index: 72743 (The Pile)

Max Activating Token Index: 264

Text #2

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.3164. Min Act: -0.0000

Data Index: 1998139 (The Pile)

Max Activating Token Index: 853

Text #3

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.3173. Min Act: -0.0000

Data Index: 209863 (The Pile)

Max Activating Token Index: 736

Text #4

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.3018. Min Act: -0.0000

Data Index: 300283 (The Pile)

Max Activating Token Index: 30

Text #5

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.3007. Min Act: -0.0000

Data Index: 70908 (The Pile)

Max Activating Token Index: 101

Text #6

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2876. Min Act: -0.0000

Data Index: 1954644 (The Pile)

Max Activating Token Index: 619

Text #7

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2857. Min Act: -0.0000

Data Index: 781684 (The Pile)

Max Activating Token Index: 280

Text #8

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2685. Min Act: -0.0000

Data Index: 784092 (The Pile)

Max Activating Token Index: 246

Text #9

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2685. Min Act: -0.0000

Data Index: 1554748 (The Pile)

Max Activating Token Index: 79

Text #10

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2676. Min Act: -0.0000

Data Index: 1757644 (The Pile)

Max Activating Token Index: 316

Text #11

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2648. Min Act: -0.0000

Data Index: 1092036 (The Pile)

Max Activating Token Index: 197

Text #12

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2347. Min Act: -0.0000

Data Index: 1749698 (The Pile)

Max Activating Token Index: 997

Text #13

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2654. Min Act: -0.0000

Data Index: 1321725 (The Pile)

Max Activating Token Index: 138

Text #14

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2668. Min Act: -0.0000

Data Index: 1912516 (The Pile)

Max Activating Token Index: 435

Text #15

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2651. Min Act: -0.0000

Data Index: 994768 (The Pile)

Max Activating Token Index: 29

Text #16

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2612. Min Act: -0.0000

Data Index: 389540 (The Pile)

Max Activating Token Index: 39

Text #17

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2534. Min Act: -0.0000

Data Index: 1006709 (The Pile)

Max Activating Token Index: 108

Text #18

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2616. Min Act: -0.0000

Data Index: 912387 (The Pile)

Max Activating Token Index: 168

Text #19

Max Range: 0.3173. Min Range: -0.3173

Max Act: 0.2589. Min Act: -0.0000

Data Index: 838768 (The Pile)

Max Activating Token Index: 63