Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1829 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2318. Min Act: -0.0005

Data Index: 1013585 (C4 (Web Text))

Max Activating Token Index: 449

Text #1

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2251. Min Act: -0.0005

Data Index: 989807 (C4 (Web Text))

Max Activating Token Index: 291

Text #2

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2268. Min Act: -0.0005

Data Index: 891666 (C4 (Web Text))

Max Activating Token Index: 662

Text #3

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2122. Min Act: -0.0005

Data Index: 134123 (C4 (Web Text))

Max Activating Token Index: 557

Text #4

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2323. Min Act: -0.0005

Data Index: 5511 (C4 (Web Text))

Max Activating Token Index: 267

Text #5

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2176. Min Act: -0.0005

Data Index: 475289 (C4 (Web Text))

Max Activating Token Index: 958

Text #6

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2254. Min Act: -0.0005

Data Index: 722834 (C4 (Web Text))

Max Activating Token Index: 166

Text #7

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2236. Min Act: -0.0005

Data Index: 536429 (C4 (Web Text))

Max Activating Token Index: 527

Text #8

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.1880. Min Act: -0.0005

Data Index: 835295 (C4 (Web Text))

Max Activating Token Index: 478

Text #9

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2143. Min Act: -0.0005

Data Index: 329251 (C4 (Web Text))

Max Activating Token Index: 792

Text #10

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2139. Min Act: -0.0005

Data Index: 868985 (C4 (Web Text))

Max Activating Token Index: 546

Text #11

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2140. Min Act: -0.0005

Data Index: 185179 (C4 (Web Text))

Max Activating Token Index: 787

Text #12

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2058. Min Act: -0.0005

Data Index: 1118954 (C4 (Web Text))

Max Activating Token Index: 264

Text #13

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2123. Min Act: -0.0005

Data Index: 6298 (C4 (Web Text))

Max Activating Token Index: 526

Text #14

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2102. Min Act: -0.0005

Data Index: 400730 (C4 (Web Text))

Max Activating Token Index: 898

Text #15

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2012. Min Act: -0.0005

Data Index: 378092 (C4 (Web Text))

Max Activating Token Index: 938

Text #16

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2056. Min Act: -0.0005

Data Index: 1052537 (C4 (Web Text))

Max Activating Token Index: 26

Text #17

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.2046. Min Act: -0.0005

Data Index: 35335 (C4 (Web Text))

Max Activating Token Index: 601

Text #18

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.1988. Min Act: -0.0005

Data Index: 1044478 (C4 (Web Text))

Max Activating Token Index: 581

Text #19

Max Range: 0.2323. Min Range: -0.2323

Max Act: 0.1976. Min Act: -0.0005

Data Index: 30791 (C4 (Web Text))

Max Activating Token Index: 1005