Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1975 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0735. Min Act: -0.0002

Data Index: 335686 (C4 (Web Text))

Max Activating Token Index: 918

Text #1

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0752. Min Act: -0.0002

Data Index: 800671 (C4 (Web Text))

Max Activating Token Index: 913

Text #2

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0673. Min Act: -0.0002

Data Index: 104375 (C4 (Web Text))

Max Activating Token Index: 555

Text #3

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0650. Min Act: -0.0002

Data Index: 626758 (C4 (Web Text))

Max Activating Token Index: 352

Text #4

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0599. Min Act: -0.0002

Data Index: 1298831 (C4 (Web Text))

Max Activating Token Index: 798

Text #5

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0747. Min Act: -0.0002

Data Index: 1085153 (C4 (Web Text))

Max Activating Token Index: 992

Text #6

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0655. Min Act: -0.0002

Data Index: 1149172 (C4 (Web Text))

Max Activating Token Index: 193

Text #7

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0575. Min Act: -0.0002

Data Index: 900230 (C4 (Web Text))

Max Activating Token Index: 766

Text #8

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0673. Min Act: -0.0002

Data Index: 193626 (C4 (Web Text))

Max Activating Token Index: 544

Text #9

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0636. Min Act: -0.0002

Data Index: 125747 (C4 (Web Text))

Max Activating Token Index: 805

Text #10

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0678. Min Act: -0.0002

Data Index: 454562 (C4 (Web Text))

Max Activating Token Index: 556

Text #11

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0582. Min Act: -0.0002

Data Index: 1111263 (C4 (Web Text))

Max Activating Token Index: 736

Text #12

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0617. Min Act: -0.0002

Data Index: 700808 (C4 (Web Text))

Max Activating Token Index: 37

Text #13

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0642. Min Act: -0.0002

Data Index: 650765 (C4 (Web Text))

Max Activating Token Index: 405

Text #14

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0565. Min Act: -0.0002

Data Index: 142914 (C4 (Web Text))

Max Activating Token Index: 809

Text #15

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0564. Min Act: -0.0002

Data Index: 572849 (C4 (Web Text))

Max Activating Token Index: 65

Text #16

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0637. Min Act: -0.0002

Data Index: 1267076 (C4 (Web Text))

Max Activating Token Index: 830

Text #17

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0617. Min Act: -0.0002

Data Index: 580367 (C4 (Web Text))

Max Activating Token Index: 294

Text #18

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0554. Min Act: -0.0002

Data Index: 890478 (C4 (Web Text))

Max Activating Token Index: 144

Text #19

Max Range: 0.0752. Min Range: -0.0752

Max Act: 0.0580. Min Act: -0.0002

Data Index: 567869 (C4 (Web Text))

Max Activating Token Index: 749