Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 797 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.8945. Min Act: -0.0000

Data Index: 1054779 (C4 (Web Text))

Max Activating Token Index: 826

Text #1

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.8479. Min Act: -0.0000

Data Index: 1338046 (C4 (Web Text))

Max Activating Token Index: 136

Text #2

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.8483. Min Act: -0.0000

Data Index: 274497 (C4 (Web Text))

Max Activating Token Index: 431

Text #3

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.8118. Min Act: -0.0000

Data Index: 386524 (C4 (Web Text))

Max Activating Token Index: 361

Text #4

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.7749. Min Act: -0.0000

Data Index: 1137263 (C4 (Web Text))

Max Activating Token Index: 843

Text #5

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.7956. Min Act: -0.0000

Data Index: 343376 (C4 (Web Text))

Max Activating Token Index: 808

Text #6

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.7618. Min Act: -0.0000

Data Index: 560667 (C4 (Web Text))

Max Activating Token Index: 638

Text #7

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.7948. Min Act: -0.0000

Data Index: 1025870 (C4 (Web Text))

Max Activating Token Index: 307

Text #8

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.7289. Min Act: 0.0000

Data Index: 969969 (C4 (Web Text))

Max Activating Token Index: 988

Text #9

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.7427. Min Act: -0.0000

Data Index: 1133628 (C4 (Web Text))

Max Activating Token Index: 589

Text #10

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.7142. Min Act: -0.0000

Data Index: 977230 (C4 (Web Text))

Max Activating Token Index: 1018

Text #11

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.7096. Min Act: -0.0000

Data Index: 1094921 (C4 (Web Text))

Max Activating Token Index: 948

Text #12

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.6656. Min Act: -0.0000

Data Index: 1284984 (C4 (Web Text))

Max Activating Token Index: 134

Text #13

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.6809. Min Act: -0.0000

Data Index: 472690 (C4 (Web Text))

Max Activating Token Index: 701

Text #14

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.6936. Min Act: -0.0000

Data Index: 1216333 (C4 (Web Text))

Max Activating Token Index: 954

Text #15

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.6716. Min Act: -0.0000

Data Index: 507996 (C4 (Web Text))

Max Activating Token Index: 595

Text #16

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.6674. Min Act: -0.0000

Data Index: 895653 (C4 (Web Text))

Max Activating Token Index: 764

Text #17

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.7039. Min Act: -0.0000

Data Index: 519573 (C4 (Web Text))

Max Activating Token Index: 478

Text #18

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.6491. Min Act: -0.0000

Data Index: 481764 (C4 (Web Text))

Max Activating Token Index: 1017

Text #19

Max Range: 0.8945. Min Range: -0.8945

Max Act: 0.6286. Min Act: -0.0000

Data Index: 747337 (C4 (Web Text))

Max Activating Token Index: 1019