Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2302 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.7625. Min Act: -0.0000

Data Index: 1022446 (C4 (Web Text))

Max Activating Token Index: 146

Text #1

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.8136. Min Act: -0.0000

Data Index: 606639 (C4 (Web Text))

Max Activating Token Index: 257

Text #2

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.7568. Min Act: -0.0000

Data Index: 1166524 (C4 (Web Text))

Max Activating Token Index: 637

Text #3

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.7575. Min Act: -0.0000

Data Index: 302843 (C4 (Web Text))

Max Activating Token Index: 75

Text #4

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.7134. Min Act: -0.0000

Data Index: 437877 (C4 (Web Text))

Max Activating Token Index: 294

Text #5

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.6443. Min Act: -0.0000

Data Index: 624715 (C4 (Web Text))

Max Activating Token Index: 846

Text #6

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.6049. Min Act: -0.0000

Data Index: 643509 (C4 (Web Text))

Max Activating Token Index: 314

Text #7

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.5771. Min Act: -0.0000

Data Index: 167987 (C4 (Web Text))

Max Activating Token Index: 264

Text #8

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.5577. Min Act: -0.0000

Data Index: 700402 (C4 (Web Text))

Max Activating Token Index: 899

Text #9

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.5471. Min Act: -0.0000

Data Index: 1163505 (C4 (Web Text))

Max Activating Token Index: 402

Text #10

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.5682. Min Act: -0.0000

Data Index: 157517 (C4 (Web Text))

Max Activating Token Index: 651

Text #11

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.5660. Min Act: -0.0000

Data Index: 1188007 (C4 (Web Text))

Max Activating Token Index: 931

Text #12

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.5417. Min Act: -0.0000

Data Index: 183788 (C4 (Web Text))

Max Activating Token Index: 886

Text #13

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.5300. Min Act: -0.0000

Data Index: 874350 (C4 (Web Text))

Max Activating Token Index: 186

Text #14

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.5468. Min Act: -0.0000

Data Index: 675547 (C4 (Web Text))

Max Activating Token Index: 426

Text #15

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.5390. Min Act: -0.0000

Data Index: 664992 (C4 (Web Text))

Max Activating Token Index: 282

Text #16

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.5158. Min Act: -0.0000

Data Index: 840159 (C4 (Web Text))

Max Activating Token Index: 925

Text #17

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.4819. Min Act: -0.0000

Data Index: 328005 (C4 (Web Text))

Max Activating Token Index: 50

Text #18

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.4950. Min Act: -0.0000

Data Index: 202834 (C4 (Web Text))

Max Activating Token Index: 854

Text #19

Max Range: 0.8136. Min Range: -0.8136

Max Act: 0.4583. Min Act: -0.0000

Data Index: 1227195 (C4 (Web Text))

Max Activating Token Index: 58