Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2746 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0932. Min Act: -0.0001

Data Index: 1038060 (C4 (Web Text))

Max Activating Token Index: 298

Text #1

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0969. Min Act: -0.0001

Data Index: 1336231 (C4 (Web Text))

Max Activating Token Index: 16

Text #2

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0817. Min Act: -0.0001

Data Index: 1297095 (C4 (Web Text))

Max Activating Token Index: 592

Text #3

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0774. Min Act: -0.0001

Data Index: 684556 (C4 (Web Text))

Max Activating Token Index: 3

Text #4

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0755. Min Act: -0.0001

Data Index: 409135 (C4 (Web Text))

Max Activating Token Index: 796

Text #5

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0714. Min Act: -0.0001

Data Index: 180945 (C4 (Web Text))

Max Activating Token Index: 797

Text #6

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0747. Min Act: -0.0001

Data Index: 373508 (C4 (Web Text))

Max Activating Token Index: 25

Text #7

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0780. Min Act: -0.0001

Data Index: 339527 (C4 (Web Text))

Max Activating Token Index: 334

Text #8

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0688. Min Act: -0.0001

Data Index: 27798 (C4 (Web Text))

Max Activating Token Index: 937

Text #9

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0681. Min Act: -0.0001

Data Index: 792629 (C4 (Web Text))

Max Activating Token Index: 1004

Text #10

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0672. Min Act: -0.0001

Data Index: 864169 (C4 (Web Text))

Max Activating Token Index: 945

Text #11

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0670. Min Act: -0.0001

Data Index: 884950 (C4 (Web Text))

Max Activating Token Index: 41

Text #12

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0574. Min Act: -0.0001

Data Index: 308484 (C4 (Web Text))

Max Activating Token Index: 881

Text #13

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0783. Min Act: -0.0001

Data Index: 1031124 (C4 (Web Text))

Max Activating Token Index: 275

Text #14

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0668. Min Act: -0.0001

Data Index: 433202 (C4 (Web Text))

Max Activating Token Index: 713

Text #15

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0668. Min Act: -0.0001

Data Index: 1083012 (C4 (Web Text))

Max Activating Token Index: 609

Text #16

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0695. Min Act: -0.0001

Data Index: 567103 (C4 (Web Text))

Max Activating Token Index: 977

Text #17

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0636. Min Act: -0.0001

Data Index: 417899 (C4 (Web Text))

Max Activating Token Index: 933

Text #18

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0673. Min Act: -0.0001

Data Index: 958730 (C4 (Web Text))

Max Activating Token Index: 495

Text #19

Max Range: 0.0969. Min Range: -0.0969

Max Act: 0.0652. Min Act: -0.0001

Data Index: 110253 (C4 (Web Text))

Max Activating Token Index: 599