Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2172 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.5203. Min Range: -2.5203

Max Act: 2.4606. Min Act: -0.0000

Data Index: 1609253 (Python Code)

Max Activating Token Index: 362

Text #1

Max Range: 2.5203. Min Range: -2.5203

Max Act: 2.5203. Min Act: -0.0000

Data Index: 1532454 (Python Code)

Max Activating Token Index: 543

Text #2

Max Range: 2.5203. Min Range: -2.5203

Max Act: 2.3813. Min Act: -0.0000

Data Index: 1601907 (Python Code)

Max Activating Token Index: 147

Text #3

Max Range: 2.5203. Min Range: -2.5203

Max Act: 2.1060. Min Act: -0.0000

Data Index: 1428027 (Python Code)

Max Activating Token Index: 336

Text #4

Max Range: 2.5203. Min Range: -2.5203

Max Act: 2.2646. Min Act: -0.0000

Data Index: 1522850 (Python Code)

Max Activating Token Index: 795

Text #5

Max Range: 2.5203. Min Range: -2.5203

Max Act: 2.0543. Min Act: -0.0000

Data Index: 1414952 (Python Code)

Max Activating Token Index: 563

Text #6

Max Range: 2.5203. Min Range: -2.5203

Max Act: 2.0037. Min Act: -0.0000

Data Index: 1639411 (Python Code)

Max Activating Token Index: 458

Text #7

Max Range: 2.5203. Min Range: -2.5203

Max Act: 2.0292. Min Act: -0.0000

Data Index: 1490274 (Python Code)

Max Activating Token Index: 32

Text #8

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.9836. Min Act: -0.0000

Data Index: 1576580 (Python Code)

Max Activating Token Index: 570

Text #9

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.9575. Min Act: -0.0000

Data Index: 1450540 (Python Code)

Max Activating Token Index: 217

Text #10

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.9151. Min Act: -0.0000

Data Index: 1605485 (Python Code)

Max Activating Token Index: 169

Text #11

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.7460. Min Act: -0.0000

Data Index: 1515046 (Python Code)

Max Activating Token Index: 247

Text #12

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.8916. Min Act: -0.0000

Data Index: 1548167 (Python Code)

Max Activating Token Index: 807

Text #13

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.8554. Min Act: -0.0000

Data Index: 1450594 (Python Code)

Max Activating Token Index: 161

Text #14

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.7580. Min Act: -0.0000

Data Index: 1433959 (Python Code)

Max Activating Token Index: 40

Text #15

Max Range: 2.5203. Min Range: -2.5203

Max Act: 2.0437. Min Act: -0.0000

Data Index: 1648524 (Python Code)

Max Activating Token Index: 679

Text #16

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.7835. Min Act: -0.0000

Data Index: 1585990 (Python Code)

Max Activating Token Index: 729

Text #17

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.7853. Min Act: -0.0000

Data Index: 1509703 (Python Code)

Max Activating Token Index: 409

Text #18

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.8530. Min Act: -0.0000

Data Index: 1572265 (Python Code)

Max Activating Token Index: 227

Text #19

Max Range: 2.5203. Min Range: -2.5203

Max Act: 1.7443. Min Act: -0.0000

Data Index: 1414158 (Python Code)

Max Activating Token Index: 569