Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4544 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.8540. Min Act: -0.0000

Data Index: 148081 (C4 (Web Text))

Max Activating Token Index: 777

Text #1

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.8053. Min Act: -0.0000

Data Index: 1516099 (Python Code)

Max Activating Token Index: 328

Text #2

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.7688. Min Act: -0.0000

Data Index: 1061486 (C4 (Web Text))

Max Activating Token Index: 302

Text #3

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.7321. Min Act: -0.0000

Data Index: 974340 (C4 (Web Text))

Max Activating Token Index: 375

Text #4

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.7333. Min Act: -0.0000

Data Index: 618431 (C4 (Web Text))

Max Activating Token Index: 120

Text #5

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.6770. Min Act: -0.0000

Data Index: 1491315 (Python Code)

Max Activating Token Index: 464

Text #6

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.6606. Min Act: -0.0000

Data Index: 612795 (C4 (Web Text))

Max Activating Token Index: 926

Text #7

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.6729. Min Act: -0.0000

Data Index: 738303 (C4 (Web Text))

Max Activating Token Index: 36

Text #8

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.6970. Min Act: 0.0000

Data Index: 260131 (C4 (Web Text))

Max Activating Token Index: 398

Text #9

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.6676. Min Act: -0.0000

Data Index: 1230355 (C4 (Web Text))

Max Activating Token Index: 167

Text #10

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.6497. Min Act: -0.0000

Data Index: 710516 (C4 (Web Text))

Max Activating Token Index: 132

Text #11

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.6710. Min Act: -0.0000

Data Index: 387829 (C4 (Web Text))

Max Activating Token Index: 487

Text #12

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.6135. Min Act: -0.0000

Data Index: 510192 (C4 (Web Text))

Max Activating Token Index: 743

Text #13

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.6324. Min Act: -0.0000

Data Index: 87338 (C4 (Web Text))

Max Activating Token Index: 663

Text #14

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.5785. Min Act: -0.0000

Data Index: 1371594 (Python Code)

Max Activating Token Index: 582

Text #15

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.5944. Min Act: -0.0000

Data Index: 1418585 (Python Code)

Max Activating Token Index: 85

Text #16

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.5692. Min Act: -0.0000

Data Index: 1364008 (Python Code)

Max Activating Token Index: 135

Text #17

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.5879. Min Act: -0.0000

Data Index: 838296 (C4 (Web Text))

Max Activating Token Index: 579

Text #18

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.5689. Min Act: -0.0000

Data Index: 559286 (C4 (Web Text))

Max Activating Token Index: 68

Text #19

Max Range: 0.8540. Min Range: -0.8540

Max Act: 0.5670. Min Act: -0.0000

Data Index: 173389 (C4 (Web Text))

Max Activating Token Index: 21