Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1443 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0632. Min Range: -1.0632

Max Act: 1.0632. Min Act: -0.0002

Data Index: 231980 (C4 (Web Text))

Max Activating Token Index: 858

Text #1

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.7644. Min Act: -0.0002

Data Index: 1158684 (C4 (Web Text))

Max Activating Token Index: 689

Text #2

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.8874. Min Act: -0.0002

Data Index: 615328 (C4 (Web Text))

Max Activating Token Index: 499

Text #3

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.8263. Min Act: -0.0002

Data Index: 125035 (C4 (Web Text))

Max Activating Token Index: 140

Text #4

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.8963. Min Act: -0.0002

Data Index: 369718 (C4 (Web Text))

Max Activating Token Index: 959

Text #5

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.7745. Min Act: -0.0002

Data Index: 677629 (C4 (Web Text))

Max Activating Token Index: 651

Text #6

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.7940. Min Act: -0.0002

Data Index: 1153825 (C4 (Web Text))

Max Activating Token Index: 922

Text #7

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.9042. Min Act: -0.0002

Data Index: 1268898 (C4 (Web Text))

Max Activating Token Index: 693

Text #8

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.8163. Min Act: -0.0002

Data Index: 600516 (C4 (Web Text))

Max Activating Token Index: 78

Text #9

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.8347. Min Act: -0.0002

Data Index: 723310 (C4 (Web Text))

Max Activating Token Index: 857

Text #10

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.7012. Min Act: -0.0002

Data Index: 290928 (C4 (Web Text))

Max Activating Token Index: 245

Text #11

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.7162. Min Act: -0.0002

Data Index: 142326 (C4 (Web Text))

Max Activating Token Index: 922

Text #12

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.7605. Min Act: -0.0002

Data Index: 875698 (C4 (Web Text))

Max Activating Token Index: 89

Text #13

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.7297. Min Act: -0.0002

Data Index: 254381 (C4 (Web Text))

Max Activating Token Index: 180

Text #14

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.7603. Min Act: -0.0002

Data Index: 37366 (C4 (Web Text))

Max Activating Token Index: 778

Text #15

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.7628. Min Act: -0.0002

Data Index: 1098880 (C4 (Web Text))

Max Activating Token Index: 151

Text #16

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.6990. Min Act: -0.0002

Data Index: 1236138 (C4 (Web Text))

Max Activating Token Index: 240

Text #17

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.7738. Min Act: -0.0002

Data Index: 1058598 (C4 (Web Text))

Max Activating Token Index: 26

Text #18

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.6833. Min Act: -0.0002

Data Index: 214279 (C4 (Web Text))

Max Activating Token Index: 941

Text #19

Max Range: 1.0632. Min Range: -1.0632

Max Act: 0.7390. Min Act: -0.0002

Data Index: 1073227 (C4 (Web Text))

Max Activating Token Index: 742