Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3384 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1380. Min Act: -0.0002

Data Index: 1062728 (C4 (Web Text))

Max Activating Token Index: 303

Text #1

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1350. Min Act: -0.0002

Data Index: 280613 (C4 (Web Text))

Max Activating Token Index: 376

Text #2

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1271. Min Act: -0.0002

Data Index: 851048 (C4 (Web Text))

Max Activating Token Index: 267

Text #3

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1241. Min Act: -0.0002

Data Index: 82716 (C4 (Web Text))

Max Activating Token Index: 673

Text #4

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1201. Min Act: -0.0002

Data Index: 1301225 (C4 (Web Text))

Max Activating Token Index: 551

Text #5

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1199. Min Act: -0.0002

Data Index: 279237 (C4 (Web Text))

Max Activating Token Index: 40

Text #6

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1184. Min Act: -0.0002

Data Index: 179863 (C4 (Web Text))

Max Activating Token Index: 282

Text #7

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1179. Min Act: -0.0002

Data Index: 1122210 (C4 (Web Text))

Max Activating Token Index: 269

Text #8

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1168. Min Act: -0.0002

Data Index: 1015965 (C4 (Web Text))

Max Activating Token Index: 351

Text #9

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1171. Min Act: -0.0002

Data Index: 1108473 (C4 (Web Text))

Max Activating Token Index: 693

Text #10

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1139. Min Act: -0.0002

Data Index: 258364 (C4 (Web Text))

Max Activating Token Index: 953

Text #11

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1179. Min Act: -0.0002

Data Index: 1034971 (C4 (Web Text))

Max Activating Token Index: 901

Text #12

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1142. Min Act: -0.0002

Data Index: 1237889 (C4 (Web Text))

Max Activating Token Index: 246

Text #13

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1175. Min Act: -0.0002

Data Index: 1021625 (C4 (Web Text))

Max Activating Token Index: 727

Text #14

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1099. Min Act: -0.0002

Data Index: 209731 (C4 (Web Text))

Max Activating Token Index: 236

Text #15

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1094. Min Act: -0.0002

Data Index: 227006 (C4 (Web Text))

Max Activating Token Index: 309

Text #16

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1122. Min Act: -0.0002

Data Index: 1009737 (C4 (Web Text))

Max Activating Token Index: 123

Text #17

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1084. Min Act: -0.0002

Data Index: 1029869 (C4 (Web Text))

Max Activating Token Index: 904

Text #18

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1075. Min Act: -0.0002

Data Index: 209206 (C4 (Web Text))

Max Activating Token Index: 337

Text #19

Max Range: 0.1380. Min Range: -0.1380

Max Act: 0.1032. Min Act: -0.0002

Data Index: 1346034 (C4 (Web Text))

Max Activating Token Index: 362