Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4999 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.6744. Min Act: 0.0001

Data Index: 650159 (C4 (Web Text))

Max Activating Token Index: 541

Text #1

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.6053. Min Act: 0.0001

Data Index: 1634410 (Python Code)

Max Activating Token Index: 646

Text #2

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.5836. Min Act: 0.0001

Data Index: 1396855 (Python Code)

Max Activating Token Index: 676

Text #3

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.5254. Min Act: -0.0000

Data Index: 538908 (C4 (Web Text))

Max Activating Token Index: 574

Text #4

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.5275. Min Act: 0.0001

Data Index: 1567163 (Python Code)

Max Activating Token Index: 379

Text #5

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4726. Min Act: -0.0000

Data Index: 1489404 (Python Code)

Max Activating Token Index: 560

Text #6

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4715. Min Act: -0.0000

Data Index: 1412504 (Python Code)

Max Activating Token Index: 197

Text #7

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4774. Min Act: 0.0001

Data Index: 1507185 (Python Code)

Max Activating Token Index: 495

Text #8

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4762. Min Act: -0.0000

Data Index: 8727 (C4 (Web Text))

Max Activating Token Index: 37

Text #9

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4466. Min Act: -0.0000

Data Index: 1599725 (Python Code)

Max Activating Token Index: 998

Text #10

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4287. Min Act: -0.0000

Data Index: 379348 (C4 (Web Text))

Max Activating Token Index: 802

Text #11

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4175. Min Act: -0.0000

Data Index: 1584173 (Python Code)

Max Activating Token Index: 537

Text #12

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4187. Min Act: -0.0000

Data Index: 1588583 (Python Code)

Max Activating Token Index: 609

Text #13

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4027. Min Act: -0.0001

Data Index: 1066971 (C4 (Web Text))

Max Activating Token Index: 668

Text #14

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4262. Min Act: -0.0000

Data Index: 1546636 (Python Code)

Max Activating Token Index: 441

Text #15

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4011. Min Act: 0.0001

Data Index: 1637720 (Python Code)

Max Activating Token Index: 132

Text #16

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4107. Min Act: 0.0000

Data Index: 609992 (C4 (Web Text))

Max Activating Token Index: 141

Text #17

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4317. Min Act: -0.0000

Data Index: 1605428 (Python Code)

Max Activating Token Index: 355

Text #18

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4099. Min Act: -0.0001

Data Index: 112730 (C4 (Web Text))

Max Activating Token Index: 860

Text #19

Max Range: 0.6744. Min Range: -0.6744

Max Act: 0.4035. Min Act: -0.0000

Data Index: 295959 (C4 (Web Text))

Max Activating Token Index: 906