Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3766 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.1711. Min Range: -1.1711

Max Act: 1.1711. Min Act: -0.0000

Data Index: 1061911 (The Pile)

Max Activating Token Index: 233

Text #1

Max Range: 1.1711. Min Range: -1.1711

Max Act: 1.1239. Min Act: -0.0000

Data Index: 1323699 (The Pile)

Max Activating Token Index: 794

Text #2

Max Range: 1.1711. Min Range: -1.1711

Max Act: 1.0907. Min Act: -0.0000

Data Index: 138290 (The Pile)

Max Activating Token Index: 492

Text #3

Max Range: 1.1711. Min Range: -1.1711

Max Act: 1.1343. Min Act: -0.0000

Data Index: 800265 (The Pile)

Max Activating Token Index: 975

Text #4

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.9508. Min Act: -0.0000

Data Index: 1442143 (The Pile)

Max Activating Token Index: 908

Text #5

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.9730. Min Act: -0.0000

Data Index: 1939099 (The Pile)

Max Activating Token Index: 836

Text #6

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.9147. Min Act: -0.0000

Data Index: 461780 (The Pile)

Max Activating Token Index: 761

Text #7

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.9959. Min Act: -0.0000

Data Index: 421926 (The Pile)

Max Activating Token Index: 697

Text #8

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.8852. Min Act: -0.0000

Data Index: 471574 (The Pile)

Max Activating Token Index: 380

Text #9

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.9368. Min Act: -0.0000

Data Index: 557384 (The Pile)

Max Activating Token Index: 958

Text #10

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.9341. Min Act: -0.0000

Data Index: 1107407 (The Pile)

Max Activating Token Index: 230

Text #11

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.8948. Min Act: -0.0000

Data Index: 1063525 (The Pile)

Max Activating Token Index: 432

Text #12

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.8828. Min Act: -0.0000

Data Index: 590896 (The Pile)

Max Activating Token Index: 708

Text #13

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.8241. Min Act: -0.0000

Data Index: 1684236 (The Pile)

Max Activating Token Index: 680

Text #14

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.8625. Min Act: -0.0000

Data Index: 600114 (The Pile)

Max Activating Token Index: 234

Text #15

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.9413. Min Act: -0.0000

Data Index: 1101332 (The Pile)

Max Activating Token Index: 995

Text #16

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.8841. Min Act: -0.0000

Data Index: 1835544 (The Pile)

Max Activating Token Index: 351

Text #17

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.8514. Min Act: -0.0000

Data Index: 1559034 (The Pile)

Max Activating Token Index: 349

Text #18

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.7493. Min Act: -0.0000

Data Index: 585398 (The Pile)

Max Activating Token Index: 870

Text #19

Max Range: 1.1711. Min Range: -1.1711

Max Act: 0.8475. Min Act: -0.0000

Data Index: 1744526 (The Pile)

Max Activating Token Index: 513