Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2212 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0786. Min Act: -0.0003

Data Index: 1279207 (C4 (Web Text))

Max Activating Token Index: 211

Text #1

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0743. Min Act: -0.0003

Data Index: 1288238 (C4 (Web Text))

Max Activating Token Index: 185

Text #2

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0731. Min Act: -0.0003

Data Index: 926413 (C4 (Web Text))

Max Activating Token Index: 239

Text #3

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0651. Min Act: -0.0003

Data Index: 1056533 (C4 (Web Text))

Max Activating Token Index: 70

Text #4

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0677. Min Act: -0.0003

Data Index: 389792 (C4 (Web Text))

Max Activating Token Index: 784

Text #5

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0650. Min Act: -0.0003

Data Index: 30121 (C4 (Web Text))

Max Activating Token Index: 316

Text #6

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0649. Min Act: -0.0003

Data Index: 619095 (C4 (Web Text))

Max Activating Token Index: 517

Text #7

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0668. Min Act: -0.0003

Data Index: 127843 (C4 (Web Text))

Max Activating Token Index: 228

Text #8

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0645. Min Act: -0.0003

Data Index: 1096555 (C4 (Web Text))

Max Activating Token Index: 807

Text #9

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0633. Min Act: -0.0003

Data Index: 1331764 (C4 (Web Text))

Max Activating Token Index: 733

Text #10

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0641. Min Act: -0.0003

Data Index: 1354227 (C4 (Web Text))

Max Activating Token Index: 368

Text #11

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0626. Min Act: -0.0003

Data Index: 762216 (C4 (Web Text))

Max Activating Token Index: 682

Text #12

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0660. Min Act: -0.0003

Data Index: 219164 (C4 (Web Text))

Max Activating Token Index: 151

Text #13

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0620. Min Act: -0.0003

Data Index: 636248 (C4 (Web Text))

Max Activating Token Index: 310

Text #14

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0617. Min Act: -0.0003

Data Index: 212401 (C4 (Web Text))

Max Activating Token Index: 431

Text #15

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0615. Min Act: -0.0003

Data Index: 1115906 (C4 (Web Text))

Max Activating Token Index: 899

Text #16

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0641. Min Act: -0.0003

Data Index: 1221432 (C4 (Web Text))

Max Activating Token Index: 101

Text #17

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0605. Min Act: -0.0003

Data Index: 1055072 (C4 (Web Text))

Max Activating Token Index: 358

Text #18

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0646. Min Act: -0.0003

Data Index: 352256 (C4 (Web Text))

Max Activating Token Index: 731

Text #19

Max Range: 0.0786. Min Range: -0.0786

Max Act: 0.0620. Min Act: -0.0003

Data Index: 804776 (C4 (Web Text))

Max Activating Token Index: 106