Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 163 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.9606. Min Act: -0.0000

Data Index: 553029 (C4 (Web Text))

Max Activating Token Index: 563

Text #1

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.9142. Min Act: -0.0000

Data Index: 639319 (C4 (Web Text))

Max Activating Token Index: 94

Text #2

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.8667. Min Act: -0.0000

Data Index: 1083113 (C4 (Web Text))

Max Activating Token Index: 347

Text #3

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.8353. Min Act: -0.0000

Data Index: 434498 (C4 (Web Text))

Max Activating Token Index: 759

Text #4

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.8580. Min Act: -0.0000

Data Index: 1107010 (C4 (Web Text))

Max Activating Token Index: 825

Text #5

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.8416. Min Act: -0.0000

Data Index: 505628 (C4 (Web Text))

Max Activating Token Index: 137

Text #6

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.8016. Min Act: -0.0000

Data Index: 621445 (C4 (Web Text))

Max Activating Token Index: 583

Text #7

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.7650. Min Act: -0.0000

Data Index: 276771 (C4 (Web Text))

Max Activating Token Index: 33

Text #8

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.7534. Min Act: -0.0000

Data Index: 507402 (C4 (Web Text))

Max Activating Token Index: 788

Text #9

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.7953. Min Act: -0.0000

Data Index: 59929 (C4 (Web Text))

Max Activating Token Index: 772

Text #10

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.7690. Min Act: -0.0000

Data Index: 677965 (C4 (Web Text))

Max Activating Token Index: 455

Text #11

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.7411. Min Act: -0.0000

Data Index: 1200521 (C4 (Web Text))

Max Activating Token Index: 265

Text #12

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.7521. Min Act: -0.0000

Data Index: 1200113 (C4 (Web Text))

Max Activating Token Index: 60

Text #13

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.7391. Min Act: -0.0000

Data Index: 368925 (C4 (Web Text))

Max Activating Token Index: 188

Text #14

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.7277. Min Act: -0.0000

Data Index: 670477 (C4 (Web Text))

Max Activating Token Index: 246

Text #15

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.6889. Min Act: -0.0000

Data Index: 919003 (C4 (Web Text))

Max Activating Token Index: 709

Text #16

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.6855. Min Act: -0.0000

Data Index: 1307912 (C4 (Web Text))

Max Activating Token Index: 132

Text #17

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.7108. Min Act: -0.0000

Data Index: 1113607 (C4 (Web Text))

Max Activating Token Index: 280

Text #18

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.6705. Min Act: -0.0000

Data Index: 83054 (C4 (Web Text))

Max Activating Token Index: 104

Text #19

Max Range: 0.9606. Min Range: -0.9606

Max Act: 0.6831. Min Act: -0.0000

Data Index: 891210 (C4 (Web Text))

Max Activating Token Index: 190