Model: SoLU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 743 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-3l')

Text #0

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0765. Min Act: -0.0003

Data Index: 738312 (C4 (Web Text))

Max Activating Token Index: 1005

Text #1

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0741. Min Act: -0.0004

Data Index: 321133 (C4 (Web Text))

Max Activating Token Index: 822

Text #2

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0731. Min Act: -0.0003

Data Index: 295786 (C4 (Web Text))

Max Activating Token Index: 944

Text #3

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0725. Min Act: -0.0003

Data Index: 162398 (C4 (Web Text))

Max Activating Token Index: 855

Text #4

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0727. Min Act: -0.0003

Data Index: 488845 (C4 (Web Text))

Max Activating Token Index: 928

Text #5

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0724. Min Act: -0.0003

Data Index: 567287 (C4 (Web Text))

Max Activating Token Index: 396

Text #6

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0720. Min Act: -0.0003

Data Index: 896686 (C4 (Web Text))

Max Activating Token Index: 752

Text #7

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0742. Min Act: -0.0003

Data Index: 41673 (C4 (Web Text))

Max Activating Token Index: 643

Text #8

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0705. Min Act: -0.0003

Data Index: 576945 (C4 (Web Text))

Max Activating Token Index: 937

Text #9

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0709. Min Act: -0.0003

Data Index: 323123 (C4 (Web Text))

Max Activating Token Index: 979

Text #10

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0679. Min Act: -0.0004

Data Index: 143785 (C4 (Web Text))

Max Activating Token Index: 703

Text #11

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0717. Min Act: -0.0003

Data Index: 435209 (C4 (Web Text))

Max Activating Token Index: 853

Text #12

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0716. Min Act: -0.0003

Data Index: 50703 (C4 (Web Text))

Max Activating Token Index: 885

Text #13

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0697. Min Act: -0.0003

Data Index: 867860 (C4 (Web Text))

Max Activating Token Index: 294

Text #14

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0663. Min Act: -0.0004

Data Index: 542186 (C4 (Web Text))

Max Activating Token Index: 938

Text #15

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0656. Min Act: -0.0003

Data Index: 773891 (C4 (Web Text))

Max Activating Token Index: 947

Text #16

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0680. Min Act: -0.0003

Data Index: 351169 (C4 (Web Text))

Max Activating Token Index: 301

Text #17

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0666. Min Act: -0.0003

Data Index: 1355335 (C4 (Web Text))

Max Activating Token Index: 708

Text #18

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0657. Min Act: -0.0003

Data Index: 596382 (C4 (Web Text))

Max Activating Token Index: 710

Text #19

Max Range: 0.0765. Min Range: -0.0765

Max Act: 0.0675. Min Act: -0.0003

Data Index: 920055 (C4 (Web Text))

Max Activating Token Index: 930