Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4493 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.5195. Min Range: -2.5195

Max Act: 2.5195. Min Act: -0.0000

Data Index: 833222 (C4 (Web Text))

Max Activating Token Index: 730

Text #1

Max Range: 2.5195. Min Range: -2.5195

Max Act: 2.2128. Min Act: -0.0000

Data Index: 1003093 (C4 (Web Text))

Max Activating Token Index: 485

Text #2

Max Range: 2.5195. Min Range: -2.5195

Max Act: 2.2118. Min Act: -0.0000

Data Index: 1255295 (C4 (Web Text))

Max Activating Token Index: 942

Text #3

Max Range: 2.5195. Min Range: -2.5195

Max Act: 2.1126. Min Act: -0.0000

Data Index: 737252 (C4 (Web Text))

Max Activating Token Index: 219

Text #4

Max Range: 2.5195. Min Range: -2.5195

Max Act: 2.2531. Min Act: -0.0000

Data Index: 297693 (C4 (Web Text))

Max Activating Token Index: 385

Text #5

Max Range: 2.5195. Min Range: -2.5195

Max Act: 2.1330. Min Act: -0.0000

Data Index: 934896 (C4 (Web Text))

Max Activating Token Index: 985

Text #6

Max Range: 2.5195. Min Range: -2.5195

Max Act: 2.1187. Min Act: -0.0000

Data Index: 325986 (C4 (Web Text))

Max Activating Token Index: 355

Text #7

Max Range: 2.5195. Min Range: -2.5195

Max Act: 2.1095. Min Act: -0.0000

Data Index: 552510 (C4 (Web Text))

Max Activating Token Index: 661

Text #8

Max Range: 2.5195. Min Range: -2.5195

Max Act: 2.1130. Min Act: -0.0000

Data Index: 380369 (C4 (Web Text))

Max Activating Token Index: 514

Text #9

Max Range: 2.5195. Min Range: -2.5195

Max Act: 2.0838. Min Act: -0.0000

Data Index: 1119503 (C4 (Web Text))

Max Activating Token Index: 363

Text #10

Max Range: 2.5195. Min Range: -2.5195

Max Act: 2.0376. Min Act: -0.0000

Data Index: 1019499 (C4 (Web Text))

Max Activating Token Index: 976

Text #11

Max Range: 2.5195. Min Range: -2.5195

Max Act: 2.0077. Min Act: -0.0000

Data Index: 88615 (C4 (Web Text))

Max Activating Token Index: 580

Text #12

Max Range: 2.5195. Min Range: -2.5195

Max Act: 1.8618. Min Act: -0.0000

Data Index: 694590 (C4 (Web Text))

Max Activating Token Index: 131

Text #13

Max Range: 2.5195. Min Range: -2.5195

Max Act: 1.8894. Min Act: -0.0000

Data Index: 1197294 (C4 (Web Text))

Max Activating Token Index: 718

Text #14

Max Range: 2.5195. Min Range: -2.5195

Max Act: 1.9402. Min Act: -0.0000

Data Index: 533744 (C4 (Web Text))

Max Activating Token Index: 344

Text #15

Max Range: 2.5195. Min Range: -2.5195

Max Act: 1.8346. Min Act: -0.0000

Data Index: 506481 (C4 (Web Text))

Max Activating Token Index: 122

Text #16

Max Range: 2.5195. Min Range: -2.5195

Max Act: 1.9304. Min Act: -0.0000

Data Index: 295662 (C4 (Web Text))

Max Activating Token Index: 454

Text #17

Max Range: 2.5195. Min Range: -2.5195

Max Act: 1.9393. Min Act: -0.0000

Data Index: 228461 (C4 (Web Text))

Max Activating Token Index: 916

Text #18

Max Range: 2.5195. Min Range: -2.5195

Max Act: 1.9231. Min Act: -0.0000

Data Index: 326619 (C4 (Web Text))

Max Activating Token Index: 258

Text #19

Max Range: 2.5195. Min Range: -2.5195

Max Act: 1.9207. Min Act: -0.0000

Data Index: 864736 (C4 (Web Text))

Max Activating Token Index: 887