Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1900 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.4780. Min Range: -2.4780

Max Act: 2.4780. Min Act: -0.0000

Data Index: 451987 (C4 (Web Text))

Max Activating Token Index: 474

Text #1

Max Range: 2.4780. Min Range: -2.4780

Max Act: 2.3223. Min Act: -0.0000

Data Index: 252827 (C4 (Web Text))

Max Activating Token Index: 542

Text #2

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.9384. Min Act: -0.0000

Data Index: 269771 (C4 (Web Text))

Max Activating Token Index: 930

Text #3

Max Range: 2.4780. Min Range: -2.4780

Max Act: 2.0593. Min Act: -0.0000

Data Index: 101222 (C4 (Web Text))

Max Activating Token Index: 739

Text #4

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.8318. Min Act: -0.0000

Data Index: 1032884 (C4 (Web Text))

Max Activating Token Index: 623

Text #5

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.8793. Min Act: -0.0000

Data Index: 459482 (C4 (Web Text))

Max Activating Token Index: 251

Text #6

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.9365. Min Act: -0.0000

Data Index: 930211 (C4 (Web Text))

Max Activating Token Index: 731

Text #7

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.6433. Min Act: -0.0000

Data Index: 280517 (C4 (Web Text))

Max Activating Token Index: 878

Text #8

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.7878. Min Act: -0.0000

Data Index: 448838 (C4 (Web Text))

Max Activating Token Index: 8

Text #9

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.7912. Min Act: -0.0000

Data Index: 395092 (C4 (Web Text))

Max Activating Token Index: 489

Text #10

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.6717. Min Act: -0.0000

Data Index: 528437 (C4 (Web Text))

Max Activating Token Index: 984

Text #11

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.7391. Min Act: -0.0000

Data Index: 2289 (C4 (Web Text))

Max Activating Token Index: 364

Text #12

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.7392. Min Act: -0.0000

Data Index: 637473 (C4 (Web Text))

Max Activating Token Index: 39

Text #13

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.5747. Min Act: -0.0000

Data Index: 1268300 (C4 (Web Text))

Max Activating Token Index: 37

Text #14

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.5566. Min Act: -0.0000

Data Index: 555920 (C4 (Web Text))

Max Activating Token Index: 658

Text #15

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.5991. Min Act: -0.0000

Data Index: 956794 (C4 (Web Text))

Max Activating Token Index: 560

Text #16

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.5884. Min Act: -0.0000

Data Index: 564054 (C4 (Web Text))

Max Activating Token Index: 226

Text #17

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.5498. Min Act: -0.0000

Data Index: 566457 (C4 (Web Text))

Max Activating Token Index: 184

Text #18

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.4444. Min Act: -0.0000

Data Index: 597037 (C4 (Web Text))

Max Activating Token Index: 104

Text #19

Max Range: 2.4780. Min Range: -2.4780

Max Act: 1.5468. Min Act: -0.0000

Data Index: 622078 (C4 (Web Text))

Max Activating Token Index: 613