Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4325 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.6302. Min Act: -0.0000

Data Index: 1264307 (C4 (Web Text))

Max Activating Token Index: 1012

Text #1

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.3871. Min Act: -0.0000

Data Index: 369919 (C4 (Web Text))

Max Activating Token Index: 411

Text #2

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.3753. Min Act: -0.0000

Data Index: 523819 (C4 (Web Text))

Max Activating Token Index: 244

Text #3

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.3613. Min Act: -0.0000

Data Index: 1285611 (C4 (Web Text))

Max Activating Token Index: 492

Text #4

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.2828. Min Act: -0.0000

Data Index: 391628 (C4 (Web Text))

Max Activating Token Index: 1014

Text #5

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.2687. Min Act: -0.0000

Data Index: 9652 (C4 (Web Text))

Max Activating Token Index: 392

Text #6

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.1194. Min Act: -0.0000

Data Index: 441981 (C4 (Web Text))

Max Activating Token Index: 671

Text #7

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.2046. Min Act: -0.0000

Data Index: 214051 (C4 (Web Text))

Max Activating Token Index: 606

Text #8

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.1950. Min Act: -0.0000

Data Index: 780561 (C4 (Web Text))

Max Activating Token Index: 163

Text #9

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.1100. Min Act: -0.0000

Data Index: 192874 (C4 (Web Text))

Max Activating Token Index: 452

Text #10

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.0515. Min Act: -0.0000

Data Index: 867587 (C4 (Web Text))

Max Activating Token Index: 268

Text #11

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.0557. Min Act: -0.0000

Data Index: 790080 (C4 (Web Text))

Max Activating Token Index: 166

Text #12

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.0229. Min Act: -0.0000

Data Index: 878324 (C4 (Web Text))

Max Activating Token Index: 537

Text #13

Max Range: 2.6302. Min Range: -2.6302

Max Act: 2.0091. Min Act: -0.0000

Data Index: 254517 (C4 (Web Text))

Max Activating Token Index: 390

Text #14

Max Range: 2.6302. Min Range: -2.6302

Max Act: 1.9524. Min Act: -0.0000

Data Index: 13728 (C4 (Web Text))

Max Activating Token Index: 650

Text #15

Max Range: 2.6302. Min Range: -2.6302

Max Act: 1.9528. Min Act: -0.0000

Data Index: 81255 (C4 (Web Text))

Max Activating Token Index: 284

Text #16

Max Range: 2.6302. Min Range: -2.6302

Max Act: 1.9089. Min Act: -0.0000

Data Index: 491153 (C4 (Web Text))

Max Activating Token Index: 102

Text #17

Max Range: 2.6302. Min Range: -2.6302

Max Act: 1.8999. Min Act: -0.0000

Data Index: 37139 (C4 (Web Text))

Max Activating Token Index: 268

Text #18

Max Range: 2.6302. Min Range: -2.6302

Max Act: 1.8438. Min Act: -0.0000

Data Index: 557306 (C4 (Web Text))

Max Activating Token Index: 501

Text #19

Max Range: 2.6302. Min Range: -2.6302

Max Act: 1.7895. Min Act: -0.0000

Data Index: 900503 (C4 (Web Text))

Max Activating Token Index: 143