Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1688 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3438. Min Range: -1.3438

Max Act: 1.3438. Min Act: -0.0000

Data Index: 284694 (C4 (Web Text))

Max Activating Token Index: 84

Text #1

Max Range: 1.3438. Min Range: -1.3438

Max Act: 1.1053. Min Act: -0.0000

Data Index: 1147267 (C4 (Web Text))

Max Activating Token Index: 937

Text #2

Max Range: 1.3438. Min Range: -1.3438

Max Act: 1.1093. Min Act: -0.0000

Data Index: 1098595 (C4 (Web Text))

Max Activating Token Index: 189

Text #3

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.9910. Min Act: -0.0000

Data Index: 1040490 (C4 (Web Text))

Max Activating Token Index: 146

Text #4

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.9088. Min Act: -0.0000

Data Index: 735625 (C4 (Web Text))

Max Activating Token Index: 723

Text #5

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.9292. Min Act: -0.0000

Data Index: 969725 (C4 (Web Text))

Max Activating Token Index: 568

Text #6

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.9871. Min Act: -0.0000

Data Index: 1308337 (C4 (Web Text))

Max Activating Token Index: 863

Text #7

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.9071. Min Act: -0.0000

Data Index: 535201 (C4 (Web Text))

Max Activating Token Index: 773

Text #8

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.8019. Min Act: -0.0000

Data Index: 1039391 (C4 (Web Text))

Max Activating Token Index: 403

Text #9

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.8455. Min Act: -0.0000

Data Index: 643052 (C4 (Web Text))

Max Activating Token Index: 483

Text #10

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.8405. Min Act: -0.0000

Data Index: 1124912 (C4 (Web Text))

Max Activating Token Index: 444

Text #11

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.8406. Min Act: -0.0000

Data Index: 1189013 (C4 (Web Text))

Max Activating Token Index: 195

Text #12

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.8458. Min Act: -0.0000

Data Index: 866479 (C4 (Web Text))

Max Activating Token Index: 721

Text #13

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.8333. Min Act: -0.0000

Data Index: 339985 (C4 (Web Text))

Max Activating Token Index: 693

Text #14

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.8173. Min Act: -0.0000

Data Index: 1306213 (C4 (Web Text))

Max Activating Token Index: 101

Text #15

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.7744. Min Act: -0.0000

Data Index: 732830 (C4 (Web Text))

Max Activating Token Index: 271

Text #16

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.8169. Min Act: -0.0000

Data Index: 1244697 (C4 (Web Text))

Max Activating Token Index: 451

Text #17

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.8071. Min Act: -0.0000

Data Index: 865230 (C4 (Web Text))

Max Activating Token Index: 574

Text #18

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.8052. Min Act: -0.0000

Data Index: 420427 (C4 (Web Text))

Max Activating Token Index: 123

Text #19

Max Range: 1.3438. Min Range: -1.3438

Max Act: 0.7987. Min Act: -0.0000

Data Index: 480931 (C4 (Web Text))

Max Activating Token Index: 478