Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1542 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0207. Min Act: -0.0003

Data Index: 1263852 (C4 (Web Text))

Max Activating Token Index: 454

Text #1

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0201. Min Act: -0.0002

Data Index: 103254 (C4 (Web Text))

Max Activating Token Index: 107

Text #2

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0205. Min Act: -0.0003

Data Index: 690402 (C4 (Web Text))

Max Activating Token Index: 485

Text #3

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0196. Min Act: -0.0002

Data Index: 300151 (C4 (Web Text))

Max Activating Token Index: 492

Text #4

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0192. Min Act: -0.0002

Data Index: 233958 (C4 (Web Text))

Max Activating Token Index: 93

Text #5

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0181. Min Act: -0.0003

Data Index: 486484 (C4 (Web Text))

Max Activating Token Index: 466

Text #6

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0182. Min Act: -0.0003

Data Index: 795248 (C4 (Web Text))

Max Activating Token Index: 71

Text #7

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0176. Min Act: -0.0002

Data Index: 702155 (C4 (Web Text))

Max Activating Token Index: 11

Text #8

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0176. Min Act: -0.0002

Data Index: 1026652 (C4 (Web Text))

Max Activating Token Index: 23

Text #9

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0183. Min Act: -0.0003

Data Index: 1355806 (C4 (Web Text))

Max Activating Token Index: 73

Text #10

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0184. Min Act: -0.0003

Data Index: 680889 (C4 (Web Text))

Max Activating Token Index: 581

Text #11

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0169. Min Act: -0.0002

Data Index: 1253234 (C4 (Web Text))

Max Activating Token Index: 295

Text #12

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0169. Min Act: -0.0002

Data Index: 1157485 (C4 (Web Text))

Max Activating Token Index: 341

Text #13

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0167. Min Act: -0.0003

Data Index: 305035 (C4 (Web Text))

Max Activating Token Index: 141

Text #14

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0175. Min Act: -0.0002

Data Index: 391119 (C4 (Web Text))

Max Activating Token Index: 79

Text #15

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0170. Min Act: -0.0002

Data Index: 651776 (C4 (Web Text))

Max Activating Token Index: 276

Text #16

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0174. Min Act: -0.0002

Data Index: 1356560 (C4 (Web Text))

Max Activating Token Index: 378

Text #17

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0170. Min Act: -0.0002

Data Index: 384629 (C4 (Web Text))

Max Activating Token Index: 942

Text #18

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0175. Min Act: -0.0002

Data Index: 14300 (C4 (Web Text))

Max Activating Token Index: 157

Text #19

Max Range: 0.0207. Min Range: -0.0207

Max Act: 0.0170. Min Act: -0.0002

Data Index: 1123820 (C4 (Web Text))

Max Activating Token Index: 468