Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4145 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0917. Min Act: -0.0002

Data Index: 1278436 (C4 (Web Text))

Max Activating Token Index: 879

Text #1

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0787. Min Act: -0.0002

Data Index: 887654 (C4 (Web Text))

Max Activating Token Index: 188

Text #2

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0836. Min Act: -0.0002

Data Index: 481602 (C4 (Web Text))

Max Activating Token Index: 450

Text #3

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0712. Min Act: -0.0002

Data Index: 223638 (C4 (Web Text))

Max Activating Token Index: 354

Text #4

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0739. Min Act: -0.0002

Data Index: 82225 (C4 (Web Text))

Max Activating Token Index: 769

Text #5

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0674. Min Act: -0.0002

Data Index: 440504 (C4 (Web Text))

Max Activating Token Index: 825

Text #6

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0697. Min Act: -0.0002

Data Index: 172928 (C4 (Web Text))

Max Activating Token Index: 562

Text #7

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0653. Min Act: -0.0002

Data Index: 366695 (C4 (Web Text))

Max Activating Token Index: 377

Text #8

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0642. Min Act: -0.0002

Data Index: 312619 (C4 (Web Text))

Max Activating Token Index: 919

Text #9

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0640. Min Act: -0.0002

Data Index: 1210714 (C4 (Web Text))

Max Activating Token Index: 325

Text #10

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0662. Min Act: -0.0002

Data Index: 224431 (C4 (Web Text))

Max Activating Token Index: 336

Text #11

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0591. Min Act: -0.0002

Data Index: 1049589 (C4 (Web Text))

Max Activating Token Index: 466

Text #12

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0604. Min Act: -0.0002

Data Index: 483727 (C4 (Web Text))

Max Activating Token Index: 616

Text #13

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0615. Min Act: -0.0002

Data Index: 779661 (C4 (Web Text))

Max Activating Token Index: 644

Text #14

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0594. Min Act: -0.0002

Data Index: 787846 (C4 (Web Text))

Max Activating Token Index: 334

Text #15

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0579. Min Act: -0.0002

Data Index: 829104 (C4 (Web Text))

Max Activating Token Index: 506

Text #16

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0571. Min Act: -0.0002

Data Index: 842523 (C4 (Web Text))

Max Activating Token Index: 704

Text #17

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0598. Min Act: -0.0002

Data Index: 1193698 (C4 (Web Text))

Max Activating Token Index: 383

Text #18

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0565. Min Act: -0.0002

Data Index: 457798 (C4 (Web Text))

Max Activating Token Index: 198

Text #19

Max Range: 0.0917. Min Range: -0.0917

Max Act: 0.0563. Min Act: -0.0002

Data Index: 1280869 (C4 (Web Text))

Max Activating Token Index: 959