Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2448 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1507. Min Act: -0.0002

Data Index: 929835 (C4 (Web Text))

Max Activating Token Index: 339

Text #1

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1405. Min Act: -0.0002

Data Index: 216598 (C4 (Web Text))

Max Activating Token Index: 794

Text #2

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1401. Min Act: -0.0002

Data Index: 67833 (C4 (Web Text))

Max Activating Token Index: 1007

Text #3

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1327. Min Act: -0.0002

Data Index: 1137770 (C4 (Web Text))

Max Activating Token Index: 899

Text #4

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1252. Min Act: -0.0002

Data Index: 263812 (C4 (Web Text))

Max Activating Token Index: 607

Text #5

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1267. Min Act: -0.0002

Data Index: 402249 (C4 (Web Text))

Max Activating Token Index: 327

Text #6

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1342. Min Act: -0.0002

Data Index: 646441 (C4 (Web Text))

Max Activating Token Index: 545

Text #7

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1171. Min Act: -0.0002

Data Index: 696475 (C4 (Web Text))

Max Activating Token Index: 538

Text #8

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1205. Min Act: -0.0002

Data Index: 182411 (C4 (Web Text))

Max Activating Token Index: 228

Text #9

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1017. Min Act: -0.0002

Data Index: 545153 (C4 (Web Text))

Max Activating Token Index: 629

Text #10

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1175. Min Act: -0.0002

Data Index: 1181378 (C4 (Web Text))

Max Activating Token Index: 234

Text #11

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1134. Min Act: -0.0002

Data Index: 142806 (C4 (Web Text))

Max Activating Token Index: 48

Text #12

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1172. Min Act: -0.0002

Data Index: 1024769 (C4 (Web Text))

Max Activating Token Index: 2

Text #13

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1169. Min Act: -0.0002

Data Index: 1007437 (C4 (Web Text))

Max Activating Token Index: 61

Text #14

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1200. Min Act: -0.0002

Data Index: 448305 (C4 (Web Text))

Max Activating Token Index: 684

Text #15

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1187. Min Act: -0.0002

Data Index: 1235658 (C4 (Web Text))

Max Activating Token Index: 540

Text #16

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1151. Min Act: -0.0002

Data Index: 911148 (C4 (Web Text))

Max Activating Token Index: 561

Text #17

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1188. Min Act: -0.0002

Data Index: 72644 (C4 (Web Text))

Max Activating Token Index: 889

Text #18

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1144. Min Act: -0.0002

Data Index: 454265 (C4 (Web Text))

Max Activating Token Index: 483

Text #19

Max Range: 0.1507. Min Range: -0.1507

Max Act: 0.1122. Min Act: -0.0002

Data Index: 933069 (C4 (Web Text))

Max Activating Token Index: 736