Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 680 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.7481. Min Act: -0.0000

Data Index: 908331 (The Pile)

Max Activating Token Index: 958

Text #1

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.7302. Min Act: -0.0000

Data Index: 536359 (The Pile)

Max Activating Token Index: 972

Text #2

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6940. Min Act: -0.0000

Data Index: 898143 (The Pile)

Max Activating Token Index: 546

Text #3

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.7344. Min Act: -0.0000

Data Index: 1596120 (The Pile)

Max Activating Token Index: 1011

Text #4

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.7560. Min Act: -0.0000

Data Index: 790938 (The Pile)

Max Activating Token Index: 925

Text #5

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.7101. Min Act: -0.0000

Data Index: 300417 (The Pile)

Max Activating Token Index: 272

Text #6

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.7189. Min Act: -0.0000

Data Index: 1730256 (The Pile)

Max Activating Token Index: 713

Text #7

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.7035. Min Act: -0.0000

Data Index: 1533227 (The Pile)

Max Activating Token Index: 765

Text #8

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6966. Min Act: -0.0000

Data Index: 1434295 (The Pile)

Max Activating Token Index: 380

Text #9

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6913. Min Act: -0.0000

Data Index: 1328576 (The Pile)

Max Activating Token Index: 456

Text #10

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6886. Min Act: -0.0000

Data Index: 1514003 (The Pile)

Max Activating Token Index: 431

Text #11

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6755. Min Act: -0.0000

Data Index: 742010 (The Pile)

Max Activating Token Index: 557

Text #12

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6750. Min Act: -0.0000

Data Index: 1856909 (The Pile)

Max Activating Token Index: 218

Text #13

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6914. Min Act: 0.0000

Data Index: 169579 (The Pile)

Max Activating Token Index: 255

Text #14

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6461. Min Act: -0.0000

Data Index: 749544 (The Pile)

Max Activating Token Index: 132

Text #15

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6833. Min Act: -0.0000

Data Index: 801579 (The Pile)

Max Activating Token Index: 790

Text #16

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6572. Min Act: -0.0000

Data Index: 276013 (The Pile)

Max Activating Token Index: 506

Text #17

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6515. Min Act: -0.0000

Data Index: 687170 (The Pile)

Max Activating Token Index: 816

Text #18

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6636. Min Act: -0.0000

Data Index: 1450069 (The Pile)

Max Activating Token Index: 882

Text #19

Max Range: 0.7560. Min Range: -0.7560

Max Act: 0.6703. Min Act: -0.0000

Data Index: 789642 (The Pile)

Max Activating Token Index: 869