Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1403 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.3165. Min Act: -0.0000

Data Index: 1066693 (C4 (Web Text))

Max Activating Token Index: 1002

Text #1

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2547. Min Act: -0.0000

Data Index: 183894 (C4 (Web Text))

Max Activating Token Index: 932

Text #2

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2433. Min Act: -0.0000

Data Index: 1231054 (C4 (Web Text))

Max Activating Token Index: 106

Text #3

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2405. Min Act: -0.0000

Data Index: 1299479 (C4 (Web Text))

Max Activating Token Index: 475

Text #4

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2390. Min Act: -0.0000

Data Index: 677981 (C4 (Web Text))

Max Activating Token Index: 881

Text #5

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2387. Min Act: -0.0000

Data Index: 123064 (C4 (Web Text))

Max Activating Token Index: 386

Text #6

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2357. Min Act: -0.0000

Data Index: 698638 (C4 (Web Text))

Max Activating Token Index: 689

Text #7

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2296. Min Act: -0.0000

Data Index: 1088176 (C4 (Web Text))

Max Activating Token Index: 531

Text #8

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2450. Min Act: -0.0000

Data Index: 220255 (C4 (Web Text))

Max Activating Token Index: 169

Text #9

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2275. Min Act: -0.0000

Data Index: 1228475 (C4 (Web Text))

Max Activating Token Index: 447

Text #10

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2356. Min Act: -0.0000

Data Index: 1267037 (C4 (Web Text))

Max Activating Token Index: 149

Text #11

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2335. Min Act: -0.0000

Data Index: 480436 (C4 (Web Text))

Max Activating Token Index: 851

Text #12

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2332. Min Act: -0.0000

Data Index: 880682 (C4 (Web Text))

Max Activating Token Index: 272

Text #13

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2081. Min Act: -0.0000

Data Index: 481060 (C4 (Web Text))

Max Activating Token Index: 441

Text #14

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2209. Min Act: -0.0000

Data Index: 1143815 (C4 (Web Text))

Max Activating Token Index: 774

Text #15

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2193. Min Act: -0.0000

Data Index: 835963 (C4 (Web Text))

Max Activating Token Index: 1022

Text #16

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2185. Min Act: -0.0000

Data Index: 655065 (C4 (Web Text))

Max Activating Token Index: 314

Text #17

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2165. Min Act: -0.0000

Data Index: 944582 (C4 (Web Text))

Max Activating Token Index: 221

Text #18

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2168. Min Act: -0.0000

Data Index: 1169361 (C4 (Web Text))

Max Activating Token Index: 408

Text #19

Max Range: 0.3165. Min Range: -0.3165

Max Act: 0.2166. Min Act: -0.0000

Data Index: 482398 (C4 (Web Text))

Max Activating Token Index: 219