Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4787 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.2430. Min Act: -0.0001

Data Index: 924974 (C4 (Web Text))

Max Activating Token Index: 126

Text #1

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.2166. Min Act: -0.0001

Data Index: 1278608 (C4 (Web Text))

Max Activating Token Index: 789

Text #2

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.2076. Min Act: -0.0001

Data Index: 1026596 (C4 (Web Text))

Max Activating Token Index: 478

Text #3

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.2047. Min Act: -0.0001

Data Index: 30909 (C4 (Web Text))

Max Activating Token Index: 1006

Text #4

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1910. Min Act: -0.0001

Data Index: 609374 (C4 (Web Text))

Max Activating Token Index: 56

Text #5

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1945. Min Act: -0.0001

Data Index: 415089 (C4 (Web Text))

Max Activating Token Index: 224

Text #6

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1924. Min Act: -0.0001

Data Index: 78806 (C4 (Web Text))

Max Activating Token Index: 374

Text #7

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1944. Min Act: -0.0001

Data Index: 860673 (C4 (Web Text))

Max Activating Token Index: 32

Text #8

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1886. Min Act: -0.0001

Data Index: 1226744 (C4 (Web Text))

Max Activating Token Index: 92

Text #9

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1957. Min Act: -0.0001

Data Index: 1230388 (C4 (Web Text))

Max Activating Token Index: 820

Text #10

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1865. Min Act: -0.0001

Data Index: 213929 (C4 (Web Text))

Max Activating Token Index: 826

Text #11

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1875. Min Act: -0.0001

Data Index: 1153481 (C4 (Web Text))

Max Activating Token Index: 63

Text #12

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1882. Min Act: -0.0001

Data Index: 1119515 (C4 (Web Text))

Max Activating Token Index: 644

Text #13

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1850. Min Act: -0.0001

Data Index: 993399 (C4 (Web Text))

Max Activating Token Index: 57

Text #14

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1845. Min Act: -0.0001

Data Index: 273724 (C4 (Web Text))

Max Activating Token Index: 1020

Text #15

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1841. Min Act: -0.0001

Data Index: 744016 (C4 (Web Text))

Max Activating Token Index: 522

Text #16

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1823. Min Act: -0.0001

Data Index: 416951 (C4 (Web Text))

Max Activating Token Index: 200

Text #17

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1769. Min Act: -0.0001

Data Index: 390288 (C4 (Web Text))

Max Activating Token Index: 333

Text #18

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1748. Min Act: -0.0001

Data Index: 182520 (C4 (Web Text))

Max Activating Token Index: 84

Text #19

Max Range: 0.2430. Min Range: -0.2430

Max Act: 0.1809. Min Act: -0.0001

Data Index: 902483 (C4 (Web Text))

Max Activating Token Index: 92