Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 940 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1283. Min Act: -0.0002

Data Index: 925394 (C4 (Web Text))

Max Activating Token Index: 805

Text #1

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1215. Min Act: -0.0002

Data Index: 484584 (C4 (Web Text))

Max Activating Token Index: 541

Text #2

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1215. Min Act: -0.0002

Data Index: 21210 (C4 (Web Text))

Max Activating Token Index: 564

Text #3

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1167. Min Act: -0.0002

Data Index: 1278844 (C4 (Web Text))

Max Activating Token Index: 216

Text #4

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1082. Min Act: -0.0002

Data Index: 799546 (C4 (Web Text))

Max Activating Token Index: 952

Text #5

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1075. Min Act: -0.0002

Data Index: 132882 (C4 (Web Text))

Max Activating Token Index: 936

Text #6

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1094. Min Act: -0.0002

Data Index: 1336371 (C4 (Web Text))

Max Activating Token Index: 60

Text #7

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1143. Min Act: -0.0002

Data Index: 1212302 (C4 (Web Text))

Max Activating Token Index: 892

Text #8

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1087. Min Act: -0.0002

Data Index: 940504 (C4 (Web Text))

Max Activating Token Index: 928

Text #9

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1080. Min Act: -0.0002

Data Index: 374133 (C4 (Web Text))

Max Activating Token Index: 408

Text #10

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1011. Min Act: -0.0002

Data Index: 1179377 (C4 (Web Text))

Max Activating Token Index: 674

Text #11

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1085. Min Act: -0.0002

Data Index: 801225 (C4 (Web Text))

Max Activating Token Index: 167

Text #12

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1009. Min Act: -0.0002

Data Index: 356975 (C4 (Web Text))

Max Activating Token Index: 824

Text #13

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1089. Min Act: -0.0002

Data Index: 1004970 (C4 (Web Text))

Max Activating Token Index: 177

Text #14

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1054. Min Act: -0.0002

Data Index: 718910 (C4 (Web Text))

Max Activating Token Index: 301

Text #15

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1018. Min Act: -0.0002

Data Index: 397796 (C4 (Web Text))

Max Activating Token Index: 343

Text #16

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.0947. Min Act: -0.0002

Data Index: 335476 (C4 (Web Text))

Max Activating Token Index: 972

Text #17

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.0952. Min Act: -0.0002

Data Index: 1343038 (C4 (Web Text))

Max Activating Token Index: 1012

Text #18

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.0864. Min Act: -0.0002

Data Index: 164937 (C4 (Web Text))

Max Activating Token Index: 714

Text #19

Max Range: 0.1283. Min Range: -0.1283

Max Act: 0.1046. Min Act: -0.0002

Data Index: 22742 (C4 (Web Text))

Max Activating Token Index: 362