Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4253 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0576. Min Range: -1.0576

Max Act: 1.0576. Min Act: -0.0000

Data Index: 853158 (C4 (Web Text))

Max Activating Token Index: 699

Text #1

Max Range: 1.0576. Min Range: -1.0576

Max Act: 1.0164. Min Act: -0.0000

Data Index: 1268860 (C4 (Web Text))

Max Activating Token Index: 477

Text #2

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9704. Min Act: -0.0000

Data Index: 1027261 (C4 (Web Text))

Max Activating Token Index: 888

Text #3

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9890. Min Act: -0.0000

Data Index: 904838 (C4 (Web Text))

Max Activating Token Index: 784

Text #4

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9639. Min Act: -0.0000

Data Index: 977082 (C4 (Web Text))

Max Activating Token Index: 267

Text #5

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9734. Min Act: -0.0000

Data Index: 417396 (C4 (Web Text))

Max Activating Token Index: 192

Text #6

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9912. Min Act: -0.0000

Data Index: 923425 (C4 (Web Text))

Max Activating Token Index: 546

Text #7

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9597. Min Act: -0.0000

Data Index: 1296250 (C4 (Web Text))

Max Activating Token Index: 627

Text #8

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9736. Min Act: -0.0000

Data Index: 798163 (C4 (Web Text))

Max Activating Token Index: 817

Text #9

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9695. Min Act: -0.0000

Data Index: 636371 (C4 (Web Text))

Max Activating Token Index: 153

Text #10

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9726. Min Act: -0.0000

Data Index: 1113619 (C4 (Web Text))

Max Activating Token Index: 426

Text #11

Max Range: 1.0576. Min Range: -1.0576

Max Act: 1.0361. Min Act: -0.0000

Data Index: 1325628 (C4 (Web Text))

Max Activating Token Index: 164

Text #12

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9689. Min Act: -0.0000

Data Index: 865942 (C4 (Web Text))

Max Activating Token Index: 119

Text #13

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9972. Min Act: -0.0000

Data Index: 1061248 (C4 (Web Text))

Max Activating Token Index: 782

Text #14

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9188. Min Act: -0.0000

Data Index: 475723 (C4 (Web Text))

Max Activating Token Index: 784

Text #15

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9239. Min Act: -0.0000

Data Index: 378814 (C4 (Web Text))

Max Activating Token Index: 409

Text #16

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9787. Min Act: -0.0000

Data Index: 511932 (C4 (Web Text))

Max Activating Token Index: 753

Text #17

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9684. Min Act: -0.0000

Data Index: 943622 (C4 (Web Text))

Max Activating Token Index: 615

Text #18

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9389. Min Act: -0.0000

Data Index: 637228 (C4 (Web Text))

Max Activating Token Index: 717

Text #19

Max Range: 1.0576. Min Range: -1.0576

Max Act: 0.9422. Min Act: -0.0000

Data Index: 462690 (C4 (Web Text))

Max Activating Token Index: 417