Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1055 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.9820. Min Act: -0.0000

Data Index: 70933 (C4 (Web Text))

Max Activating Token Index: 620

Text #1

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.9092. Min Act: -0.0000

Data Index: 141784 (C4 (Web Text))

Max Activating Token Index: 357

Text #2

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.7939. Min Act: -0.0000

Data Index: 1149151 (C4 (Web Text))

Max Activating Token Index: 729

Text #3

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.7968. Min Act: -0.0000

Data Index: 157305 (C4 (Web Text))

Max Activating Token Index: 415

Text #4

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.7879. Min Act: -0.0000

Data Index: 236297 (C4 (Web Text))

Max Activating Token Index: 664

Text #5

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.7150. Min Act: -0.0000

Data Index: 774908 (C4 (Web Text))

Max Activating Token Index: 407

Text #6

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.6791. Min Act: -0.0000

Data Index: 1070830 (C4 (Web Text))

Max Activating Token Index: 729

Text #7

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.7203. Min Act: -0.0000

Data Index: 370472 (C4 (Web Text))

Max Activating Token Index: 904

Text #8

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.7615. Min Act: -0.0000

Data Index: 186725 (C4 (Web Text))

Max Activating Token Index: 676

Text #9

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.6917. Min Act: -0.0000

Data Index: 41185 (C4 (Web Text))

Max Activating Token Index: 903

Text #10

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.7243. Min Act: -0.0000

Data Index: 767839 (C4 (Web Text))

Max Activating Token Index: 202

Text #11

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.7303. Min Act: -0.0000

Data Index: 1054972 (C4 (Web Text))

Max Activating Token Index: 915

Text #12

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.6636. Min Act: -0.0000

Data Index: 653653 (C4 (Web Text))

Max Activating Token Index: 983

Text #13

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.6851. Min Act: -0.0000

Data Index: 425101 (C4 (Web Text))

Max Activating Token Index: 516

Text #14

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.6597. Min Act: -0.0000

Data Index: 414911 (C4 (Web Text))

Max Activating Token Index: 110

Text #15

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.6281. Min Act: -0.0000

Data Index: 313521 (C4 (Web Text))

Max Activating Token Index: 251

Text #16

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.6434. Min Act: -0.0000

Data Index: 176601 (C4 (Web Text))

Max Activating Token Index: 937

Text #17

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.6978. Min Act: -0.0000

Data Index: 16170 (C4 (Web Text))

Max Activating Token Index: 620

Text #18

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.6266. Min Act: -0.0000

Data Index: 530499 (C4 (Web Text))

Max Activating Token Index: 739

Text #19

Max Range: 0.9820. Min Range: -0.9820

Max Act: 0.6219. Min Act: -0.0000

Data Index: 1281907 (C4 (Web Text))

Max Activating Token Index: 84