Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 90 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0703. Min Act: -0.0002

Data Index: 272202 (C4 (Web Text))

Max Activating Token Index: 291

Text #1

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0646. Min Act: -0.0002

Data Index: 603721 (C4 (Web Text))

Max Activating Token Index: 684

Text #2

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0668. Min Act: -0.0002

Data Index: 921877 (C4 (Web Text))

Max Activating Token Index: 170

Text #3

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0638. Min Act: -0.0002

Data Index: 4380 (C4 (Web Text))

Max Activating Token Index: 919

Text #4

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0591. Min Act: -0.0002

Data Index: 982915 (C4 (Web Text))

Max Activating Token Index: 694

Text #5

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0676. Min Act: -0.0002

Data Index: 552464 (C4 (Web Text))

Max Activating Token Index: 705

Text #6

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0639. Min Act: -0.0002

Data Index: 1333631 (C4 (Web Text))

Max Activating Token Index: 15

Text #7

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0615. Min Act: -0.0002

Data Index: 701578 (C4 (Web Text))

Max Activating Token Index: 397

Text #8

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0658. Min Act: -0.0002

Data Index: 1280156 (C4 (Web Text))

Max Activating Token Index: 465

Text #9

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0597. Min Act: -0.0002

Data Index: 244560 (C4 (Web Text))

Max Activating Token Index: 434

Text #10

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0571. Min Act: -0.0002

Data Index: 180863 (C4 (Web Text))

Max Activating Token Index: 224

Text #11

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0689. Min Act: -0.0002

Data Index: 990805 (C4 (Web Text))

Max Activating Token Index: 704

Text #12

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0584. Min Act: -0.0002

Data Index: 576137 (C4 (Web Text))

Max Activating Token Index: 9

Text #13

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0632. Min Act: -0.0002

Data Index: 855456 (C4 (Web Text))

Max Activating Token Index: 907

Text #14

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0596. Min Act: -0.0002

Data Index: 806421 (C4 (Web Text))

Max Activating Token Index: 192

Text #15

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0596. Min Act: -0.0002

Data Index: 940447 (C4 (Web Text))

Max Activating Token Index: 119

Text #16

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0593. Min Act: -0.0002

Data Index: 331932 (C4 (Web Text))

Max Activating Token Index: 272

Text #17

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0547. Min Act: -0.0002

Data Index: 90341 (C4 (Web Text))

Max Activating Token Index: 887

Text #18

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0579. Min Act: -0.0002

Data Index: 486151 (C4 (Web Text))

Max Activating Token Index: 427

Text #19

Max Range: 0.0703. Min Range: -0.0703

Max Act: 0.0577. Min Act: -0.0002

Data Index: 505476 (C4 (Web Text))

Max Activating Token Index: 503