Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2290 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.2729. Min Act: -0.0000

Data Index: 1112242 (C4 (Web Text))

Max Activating Token Index: 224

Text #1

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.2453. Min Act: -0.0000

Data Index: 1290547 (C4 (Web Text))

Max Activating Token Index: 439

Text #2

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.2718. Min Act: -0.0000

Data Index: 563074 (C4 (Web Text))

Max Activating Token Index: 1008

Text #3

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.1248. Min Act: -0.0000

Data Index: 157834 (C4 (Web Text))

Max Activating Token Index: 582

Text #4

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.0870. Min Act: -0.0000

Data Index: 1027706 (C4 (Web Text))

Max Activating Token Index: 358

Text #5

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.0745. Min Act: -0.0000

Data Index: 1087760 (C4 (Web Text))

Max Activating Token Index: 205

Text #6

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.0847. Min Act: -0.0000

Data Index: 740501 (C4 (Web Text))

Max Activating Token Index: 200

Text #7

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.0807. Min Act: -0.0000

Data Index: 1162368 (C4 (Web Text))

Max Activating Token Index: 691

Text #8

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.0611. Min Act: -0.0000

Data Index: 1014642 (C4 (Web Text))

Max Activating Token Index: 821

Text #9

Max Range: 1.2729. Min Range: -1.2729

Max Act: 0.9999. Min Act: -0.0000

Data Index: 903604 (C4 (Web Text))

Max Activating Token Index: 973

Text #10

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.0374. Min Act: -0.0000

Data Index: 1243792 (C4 (Web Text))

Max Activating Token Index: 212

Text #11

Max Range: 1.2729. Min Range: -1.2729

Max Act: 0.9724. Min Act: -0.0000

Data Index: 622555 (C4 (Web Text))

Max Activating Token Index: 860

Text #12

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.0307. Min Act: -0.0000

Data Index: 1298969 (C4 (Web Text))

Max Activating Token Index: 81

Text #13

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.0291. Min Act: -0.0000

Data Index: 836499 (C4 (Web Text))

Max Activating Token Index: 637

Text #14

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.0823. Min Act: -0.0000

Data Index: 1116905 (C4 (Web Text))

Max Activating Token Index: 444

Text #15

Max Range: 1.2729. Min Range: -1.2729

Max Act: 0.9428. Min Act: -0.0000

Data Index: 945489 (C4 (Web Text))

Max Activating Token Index: 595

Text #16

Max Range: 1.2729. Min Range: -1.2729

Max Act: 0.9812. Min Act: -0.0000

Data Index: 325513 (C4 (Web Text))

Max Activating Token Index: 522

Text #17

Max Range: 1.2729. Min Range: -1.2729

Max Act: 1.0431. Min Act: -0.0000

Data Index: 782515 (C4 (Web Text))

Max Activating Token Index: 825

Text #18

Max Range: 1.2729. Min Range: -1.2729

Max Act: 0.9859. Min Act: -0.0000

Data Index: 192490 (C4 (Web Text))

Max Activating Token Index: 605

Text #19

Max Range: 1.2729. Min Range: -1.2729

Max Act: 0.9142. Min Act: -0.0000

Data Index: 1176703 (C4 (Web Text))

Max Activating Token Index: 613