Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4110 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.9824. Min Act: -0.0001

Data Index: 749791 (C4 (Web Text))

Max Activating Token Index: 195

Text #1

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.9392. Min Act: -0.0001

Data Index: 1305780 (C4 (Web Text))

Max Activating Token Index: 600

Text #2

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.9060. Min Act: -0.0001

Data Index: 505213 (C4 (Web Text))

Max Activating Token Index: 233

Text #3

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.8519. Min Act: -0.0001

Data Index: 33862 (C4 (Web Text))

Max Activating Token Index: 952

Text #4

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.9050. Min Act: -0.0001

Data Index: 1108673 (C4 (Web Text))

Max Activating Token Index: 201

Text #5

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.8491. Min Act: -0.0001

Data Index: 51829 (C4 (Web Text))

Max Activating Token Index: 518

Text #6

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.8836. Min Act: -0.0001

Data Index: 822220 (C4 (Web Text))

Max Activating Token Index: 566

Text #7

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.8581. Min Act: -0.0001

Data Index: 816162 (C4 (Web Text))

Max Activating Token Index: 1010

Text #8

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.8339. Min Act: -0.0001

Data Index: 97883 (C4 (Web Text))

Max Activating Token Index: 554

Text #9

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.8209. Min Act: -0.0001

Data Index: 484982 (C4 (Web Text))

Max Activating Token Index: 13

Text #10

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.7464. Min Act: -0.0001

Data Index: 989838 (C4 (Web Text))

Max Activating Token Index: 475

Text #11

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.8025. Min Act: -0.0001

Data Index: 760751 (C4 (Web Text))

Max Activating Token Index: 924

Text #12

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.8255. Min Act: -0.0001

Data Index: 644289 (C4 (Web Text))

Max Activating Token Index: 690

Text #13

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.7759. Min Act: -0.0001

Data Index: 256310 (C4 (Web Text))

Max Activating Token Index: 438

Text #14

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.7943. Min Act: -0.0001

Data Index: 554867 (C4 (Web Text))

Max Activating Token Index: 316

Text #15

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.7430. Min Act: -0.0001

Data Index: 560830 (C4 (Web Text))

Max Activating Token Index: 403

Text #16

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.7370. Min Act: -0.0001

Data Index: 840715 (C4 (Web Text))

Max Activating Token Index: 851

Text #17

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.7580. Min Act: -0.0000

Data Index: 1062269 (C4 (Web Text))

Max Activating Token Index: 877

Text #18

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.7052. Min Act: -0.0001

Data Index: 247782 (C4 (Web Text))

Max Activating Token Index: 185

Text #19

Max Range: 0.9824. Min Range: -0.9824

Max Act: 0.7335. Min Act: -0.0001

Data Index: 540604 (C4 (Web Text))

Max Activating Token Index: 274