Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5064 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0338. Min Act: -0.0001

Data Index: 1328145 (C4 (Web Text))

Max Activating Token Index: 945

Text #1

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0303. Min Act: -0.0001

Data Index: 1581631 (Python Code)

Max Activating Token Index: 445

Text #2

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0321. Min Act: -0.0001

Data Index: 1057454 (C4 (Web Text))

Max Activating Token Index: 13

Text #3

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0299. Min Act: -0.0001

Data Index: 1454730 (Python Code)

Max Activating Token Index: 139

Text #4

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0305. Min Act: -0.0001

Data Index: 810263 (C4 (Web Text))

Max Activating Token Index: 315

Text #5

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0319. Min Act: -0.0001

Data Index: 380823 (C4 (Web Text))

Max Activating Token Index: 801

Text #6

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0282. Min Act: -0.0001

Data Index: 271139 (C4 (Web Text))

Max Activating Token Index: 190

Text #7

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0269. Min Act: -0.0001

Data Index: 377366 (C4 (Web Text))

Max Activating Token Index: 725

Text #8

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0309. Min Act: -0.0001

Data Index: 396892 (C4 (Web Text))

Max Activating Token Index: 179

Text #9

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0277. Min Act: -0.0001

Data Index: 989430 (C4 (Web Text))

Max Activating Token Index: 1001

Text #10

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0274. Min Act: -0.0001

Data Index: 590890 (C4 (Web Text))

Max Activating Token Index: 419

Text #11

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0271. Min Act: -0.0001

Data Index: 189187 (C4 (Web Text))

Max Activating Token Index: 465

Text #12

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0271. Min Act: -0.0001

Data Index: 238024 (C4 (Web Text))

Max Activating Token Index: 521

Text #13

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0245. Min Act: -0.0001

Data Index: 119447 (C4 (Web Text))

Max Activating Token Index: 140

Text #14

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0274. Min Act: -0.0001

Data Index: 1298548 (C4 (Web Text))

Max Activating Token Index: 13

Text #15

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0259. Min Act: -0.0001

Data Index: 659684 (C4 (Web Text))

Max Activating Token Index: 1013

Text #16

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0260. Min Act: -0.0001

Data Index: 880764 (C4 (Web Text))

Max Activating Token Index: 288

Text #17

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0266. Min Act: -0.0001

Data Index: 1556688 (Python Code)

Max Activating Token Index: 313

Text #18

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0244. Min Act: -0.0001

Data Index: 1163831 (C4 (Web Text))

Max Activating Token Index: 956

Text #19

Max Range: 0.0338. Min Range: -0.0338

Max Act: 0.0251. Min Act: -0.0001

Data Index: 1467665 (Python Code)

Max Activating Token Index: 529