Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1062 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2171. Min Range: -1.2171

Max Act: 1.2033. Min Act: -0.0000

Data Index: 444910 (C4 (Web Text))

Max Activating Token Index: 662

Text #1

Max Range: 1.2171. Min Range: -1.2171

Max Act: 1.1682. Min Act: -0.0000

Data Index: 11827 (C4 (Web Text))

Max Activating Token Index: 930

Text #2

Max Range: 1.2171. Min Range: -1.2171

Max Act: 1.2171. Min Act: -0.0000

Data Index: 232435 (C4 (Web Text))

Max Activating Token Index: 35

Text #3

Max Range: 1.2171. Min Range: -1.2171

Max Act: 1.0299. Min Act: -0.0000

Data Index: 417741 (C4 (Web Text))

Max Activating Token Index: 724

Text #4

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.8919. Min Act: -0.0000

Data Index: 756687 (C4 (Web Text))

Max Activating Token Index: 862

Text #5

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.9103. Min Act: -0.0000

Data Index: 750129 (C4 (Web Text))

Max Activating Token Index: 921

Text #6

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.9698. Min Act: -0.0000

Data Index: 382081 (C4 (Web Text))

Max Activating Token Index: 721

Text #7

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.9729. Min Act: -0.0000

Data Index: 6678 (C4 (Web Text))

Max Activating Token Index: 851

Text #8

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.8558. Min Act: -0.0000

Data Index: 1289775 (C4 (Web Text))

Max Activating Token Index: 997

Text #9

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.8044. Min Act: -0.0000

Data Index: 51525 (C4 (Web Text))

Max Activating Token Index: 617

Text #10

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.9247. Min Act: -0.0000

Data Index: 222846 (C4 (Web Text))

Max Activating Token Index: 702

Text #11

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.8618. Min Act: -0.0000

Data Index: 49521 (C4 (Web Text))

Max Activating Token Index: 62

Text #12

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.8235. Min Act: -0.0000

Data Index: 846172 (C4 (Web Text))

Max Activating Token Index: 537

Text #13

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.8036. Min Act: -0.0000

Data Index: 190588 (C4 (Web Text))

Max Activating Token Index: 182

Text #14

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.8850. Min Act: -0.0000

Data Index: 983012 (C4 (Web Text))

Max Activating Token Index: 156

Text #15

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.8000. Min Act: -0.0000

Data Index: 574226 (C4 (Web Text))

Max Activating Token Index: 133

Text #16

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.7932. Min Act: -0.0000

Data Index: 64878 (C4 (Web Text))

Max Activating Token Index: 457

Text #17

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.8567. Min Act: -0.0000

Data Index: 97688 (C4 (Web Text))

Max Activating Token Index: 290

Text #18

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.8133. Min Act: -0.0000

Data Index: 257145 (C4 (Web Text))

Max Activating Token Index: 337

Text #19

Max Range: 1.2171. Min Range: -1.2171

Max Act: 0.7835. Min Act: -0.0000

Data Index: 1180552 (C4 (Web Text))

Max Activating Token Index: 154