Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 520 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.6569. Min Act: -0.0000

Data Index: 1269874 (The Pile)

Max Activating Token Index: 957

Text #1

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.6458. Min Act: 0.0000

Data Index: 1748383 (The Pile)

Max Activating Token Index: 224

Text #2

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5687. Min Act: -0.0000

Data Index: 306166 (The Pile)

Max Activating Token Index: 237

Text #3

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5935. Min Act: -0.0000

Data Index: 1082663 (The Pile)

Max Activating Token Index: 559

Text #4

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5581. Min Act: -0.0000

Data Index: 393825 (The Pile)

Max Activating Token Index: 1006

Text #5

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5545. Min Act: -0.0000

Data Index: 83107 (The Pile)

Max Activating Token Index: 176

Text #6

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5682. Min Act: -0.0000

Data Index: 1626712 (The Pile)

Max Activating Token Index: 791

Text #7

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5469. Min Act: -0.0000

Data Index: 407454 (The Pile)

Max Activating Token Index: 559

Text #8

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5283. Min Act: -0.0000

Data Index: 761691 (The Pile)

Max Activating Token Index: 177

Text #9

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5585. Min Act: -0.0000

Data Index: 1620068 (The Pile)

Max Activating Token Index: 610

Text #10

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5506. Min Act: -0.0000

Data Index: 592939 (The Pile)

Max Activating Token Index: 770

Text #11

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5547. Min Act: -0.0000

Data Index: 576417 (The Pile)

Max Activating Token Index: 864

Text #12

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5176. Min Act: -0.0000

Data Index: 1684423 (The Pile)

Max Activating Token Index: 716

Text #13

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5419. Min Act: -0.0000

Data Index: 97715 (The Pile)

Max Activating Token Index: 114

Text #14

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5316. Min Act: -0.0000

Data Index: 966485 (The Pile)

Max Activating Token Index: 953

Text #15

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5183. Min Act: -0.0000

Data Index: 776676 (The Pile)

Max Activating Token Index: 121

Text #16

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.4904. Min Act: -0.0000

Data Index: 1750196 (The Pile)

Max Activating Token Index: 198

Text #17

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.5180. Min Act: -0.0000

Data Index: 972397 (The Pile)

Max Activating Token Index: 258

Text #18

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.4951. Min Act: -0.0000

Data Index: 1518051 (The Pile)

Max Activating Token Index: 726

Text #19

Max Range: 0.6569. Min Range: -0.6569

Max Act: 0.4999. Min Act: -0.0000

Data Index: 910022 (The Pile)

Max Activating Token Index: 963