Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1654 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0766. Min Act: -0.0002

Data Index: 1086010 (C4 (Web Text))

Max Activating Token Index: 660

Text #1

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0716. Min Act: -0.0002

Data Index: 1276542 (C4 (Web Text))

Max Activating Token Index: 507

Text #2

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0707. Min Act: -0.0002

Data Index: 488403 (C4 (Web Text))

Max Activating Token Index: 169

Text #3

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0663. Min Act: -0.0002

Data Index: 248618 (C4 (Web Text))

Max Activating Token Index: 635

Text #4

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0685. Min Act: -0.0002

Data Index: 1236259 (C4 (Web Text))

Max Activating Token Index: 46

Text #5

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0691. Min Act: -0.0002

Data Index: 302326 (C4 (Web Text))

Max Activating Token Index: 920

Text #6

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0655. Min Act: -0.0002

Data Index: 923621 (C4 (Web Text))

Max Activating Token Index: 68

Text #7

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0671. Min Act: -0.0002

Data Index: 159075 (C4 (Web Text))

Max Activating Token Index: 438

Text #8

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0664. Min Act: -0.0002

Data Index: 911560 (C4 (Web Text))

Max Activating Token Index: 41

Text #9

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0633. Min Act: -0.0002

Data Index: 903104 (C4 (Web Text))

Max Activating Token Index: 801

Text #10

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0642. Min Act: -0.0002

Data Index: 986416 (C4 (Web Text))

Max Activating Token Index: 7

Text #11

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0618. Min Act: -0.0002

Data Index: 1083364 (C4 (Web Text))

Max Activating Token Index: 200

Text #12

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0609. Min Act: -0.0002

Data Index: 463948 (C4 (Web Text))

Max Activating Token Index: 622

Text #13

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0569. Min Act: -0.0002

Data Index: 105127 (C4 (Web Text))

Max Activating Token Index: 482

Text #14

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0611. Min Act: -0.0002

Data Index: 284634 (C4 (Web Text))

Max Activating Token Index: 1006

Text #15

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0587. Min Act: -0.0002

Data Index: 632251 (C4 (Web Text))

Max Activating Token Index: 241

Text #16

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0597. Min Act: -0.0002

Data Index: 418863 (C4 (Web Text))

Max Activating Token Index: 836

Text #17

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0596. Min Act: -0.0002

Data Index: 374405 (C4 (Web Text))

Max Activating Token Index: 67

Text #18

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0583. Min Act: -0.0002

Data Index: 970006 (C4 (Web Text))

Max Activating Token Index: 867

Text #19

Max Range: 0.0766. Min Range: -0.0766

Max Act: 0.0583. Min Act: -0.0002

Data Index: 1016454 (C4 (Web Text))

Max Activating Token Index: 25