Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 787 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.8309. Min Range: -2.8309

Max Act: 2.8309. Min Act: -0.0000

Data Index: 1568747 (Python Code)

Max Activating Token Index: 216

Text #1

Max Range: 2.8309. Min Range: -2.8309

Max Act: 2.5632. Min Act: -0.0000

Data Index: 1537576 (Python Code)

Max Activating Token Index: 320

Text #2

Max Range: 2.8309. Min Range: -2.8309

Max Act: 2.2122. Min Act: -0.0000

Data Index: 1516786 (Python Code)

Max Activating Token Index: 271

Text #3

Max Range: 2.8309. Min Range: -2.8309

Max Act: 2.1409. Min Act: -0.0000

Data Index: 1608678 (Python Code)

Max Activating Token Index: 825

Text #4

Max Range: 2.8309. Min Range: -2.8309

Max Act: 2.1193. Min Act: -0.0000

Data Index: 1578361 (Python Code)

Max Activating Token Index: 177

Text #5

Max Range: 2.8309. Min Range: -2.8309

Max Act: 1.9659. Min Act: -0.0000

Data Index: 432506 (C4 (Web Text))

Max Activating Token Index: 1020

Text #6

Max Range: 2.8309. Min Range: -2.8309

Max Act: 1.9628. Min Act: -0.0000

Data Index: 1542651 (Python Code)

Max Activating Token Index: 718

Text #7

Max Range: 2.8309. Min Range: -2.8309

Max Act: 2.1540. Min Act: -0.0000

Data Index: 1644585 (Python Code)

Max Activating Token Index: 650

Text #8

Max Range: 2.8309. Min Range: -2.8309

Max Act: 2.0157. Min Act: -0.0000

Data Index: 1423261 (Python Code)

Max Activating Token Index: 236

Text #9

Max Range: 2.8309. Min Range: -2.8309

Max Act: 2.0050. Min Act: -0.0000

Data Index: 741244 (C4 (Web Text))

Max Activating Token Index: 431

Text #10

Max Range: 2.8309. Min Range: -2.8309

Max Act: 2.0981. Min Act: -0.0000

Data Index: 1543343 (Python Code)

Max Activating Token Index: 584

Text #11

Max Range: 2.8309. Min Range: -2.8309

Max Act: 2.0818. Min Act: -0.0000

Data Index: 1452931 (Python Code)

Max Activating Token Index: 95

Text #12

Max Range: 2.8309. Min Range: -2.8309

Max Act: 2.0477. Min Act: -0.0000

Data Index: 1614000 (Python Code)

Max Activating Token Index: 686

Text #13

Max Range: 2.8309. Min Range: -2.8309

Max Act: 2.0307. Min Act: -0.0000

Data Index: 1424760 (Python Code)

Max Activating Token Index: 187

Text #14

Max Range: 2.8309. Min Range: -2.8309

Max Act: 1.9090. Min Act: -0.0000

Data Index: 26493 (C4 (Web Text))

Max Activating Token Index: 209

Text #15

Max Range: 2.8309. Min Range: -2.8309

Max Act: 1.8904. Min Act: -0.0000

Data Index: 138822 (C4 (Web Text))

Max Activating Token Index: 35

Text #16

Max Range: 2.8309. Min Range: -2.8309

Max Act: 1.8816. Min Act: -0.0000

Data Index: 510269 (C4 (Web Text))

Max Activating Token Index: 1009

Text #17

Max Range: 2.8309. Min Range: -2.8309

Max Act: 1.7722. Min Act: -0.0000

Data Index: 1266593 (C4 (Web Text))

Max Activating Token Index: 831

Text #18

Max Range: 2.8309. Min Range: -2.8309

Max Act: 1.7891. Min Act: -0.0000

Data Index: 1557569 (Python Code)

Max Activating Token Index: 485

Text #19

Max Range: 2.8309. Min Range: -2.8309

Max Act: 1.7872. Min Act: -0.0000

Data Index: 1618504 (Python Code)

Max Activating Token Index: 640