Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 867 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.6679. Min Act: -0.0000

Data Index: 476576 (C4 (Web Text))

Max Activating Token Index: 969

Text #1

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.6106. Min Act: -0.0000

Data Index: 1339328 (C4 (Web Text))

Max Activating Token Index: 869

Text #2

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.6056. Min Act: -0.0000

Data Index: 664239 (C4 (Web Text))

Max Activating Token Index: 679

Text #3

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.5947. Min Act: -0.0000

Data Index: 994107 (C4 (Web Text))

Max Activating Token Index: 668

Text #4

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.5689. Min Act: -0.0000

Data Index: 1276892 (C4 (Web Text))

Max Activating Token Index: 214

Text #5

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.5632. Min Act: -0.0000

Data Index: 437744 (C4 (Web Text))

Max Activating Token Index: 814

Text #6

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.5494. Min Act: -0.0000

Data Index: 1116265 (C4 (Web Text))

Max Activating Token Index: 609

Text #7

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.5291. Min Act: -0.0000

Data Index: 565072 (C4 (Web Text))

Max Activating Token Index: 317

Text #8

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.5087. Min Act: -0.0000

Data Index: 1020950 (C4 (Web Text))

Max Activating Token Index: 955

Text #9

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.4779. Min Act: -0.0000

Data Index: 8557 (C4 (Web Text))

Max Activating Token Index: 373

Text #10

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.5048. Min Act: -0.0000

Data Index: 174822 (C4 (Web Text))

Max Activating Token Index: 910

Text #11

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.5060. Min Act: -0.0000

Data Index: 1204365 (C4 (Web Text))

Max Activating Token Index: 167

Text #12

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.5115. Min Act: -0.0000

Data Index: 941988 (C4 (Web Text))

Max Activating Token Index: 110

Text #13

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.4895. Min Act: -0.0000

Data Index: 1076443 (C4 (Web Text))

Max Activating Token Index: 937

Text #14

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.4927. Min Act: -0.0000

Data Index: 866504 (C4 (Web Text))

Max Activating Token Index: 979

Text #15

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.5126. Min Act: -0.0000

Data Index: 871093 (C4 (Web Text))

Max Activating Token Index: 269

Text #16

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.4667. Min Act: -0.0000

Data Index: 1221227 (C4 (Web Text))

Max Activating Token Index: 362

Text #17

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.4969. Min Act: -0.0000

Data Index: 930570 (C4 (Web Text))

Max Activating Token Index: 750

Text #18

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.4932. Min Act: -0.0000

Data Index: 748475 (C4 (Web Text))

Max Activating Token Index: 881

Text #19

Max Range: 0.6679. Min Range: -0.6679

Max Act: 0.4559. Min Act: -0.0000

Data Index: 1324273 (C4 (Web Text))

Max Activating Token Index: 897