Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2166 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.2228. Min Act: -0.0001

Data Index: 1514221 (Python Code)

Max Activating Token Index: 382

Text #1

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.2232. Min Act: -0.0001

Data Index: 1506496 (Python Code)

Max Activating Token Index: 220

Text #2

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.2300. Min Act: -0.0001

Data Index: 1656763 (Python Code)

Max Activating Token Index: 885

Text #3

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1814. Min Act: -0.0001

Data Index: 1629426 (Python Code)

Max Activating Token Index: 296

Text #4

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1865. Min Act: -0.0001

Data Index: 1501124 (Python Code)

Max Activating Token Index: 138

Text #5

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1905. Min Act: -0.0000

Data Index: 1417479 (Python Code)

Max Activating Token Index: 700

Text #6

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1584. Min Act: -0.0001

Data Index: 1458655 (Python Code)

Max Activating Token Index: 982

Text #7

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1480. Min Act: -0.0001

Data Index: 76854 (C4 (Web Text))

Max Activating Token Index: 510

Text #8

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1540. Min Act: -0.0001

Data Index: 1490716 (Python Code)

Max Activating Token Index: 896

Text #9

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1442. Min Act: -0.0000

Data Index: 1521903 (Python Code)

Max Activating Token Index: 225

Text #10

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1547. Min Act: -0.0000

Data Index: 1590415 (Python Code)

Max Activating Token Index: 619

Text #11

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1580. Min Act: -0.0001

Data Index: 1571728 (Python Code)

Max Activating Token Index: 659

Text #12

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1487. Min Act: -0.0001

Data Index: 1503481 (Python Code)

Max Activating Token Index: 890

Text #13

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1451. Min Act: -0.0001

Data Index: 1461544 (Python Code)

Max Activating Token Index: 107

Text #14

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1500. Min Act: -0.0000

Data Index: 1362062 (Python Code)

Max Activating Token Index: 151

Text #15

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1617. Min Act: -0.0001

Data Index: 1576199 (Python Code)

Max Activating Token Index: 795

Text #16

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1485. Min Act: -0.0001

Data Index: 1462594 (Python Code)

Max Activating Token Index: 502

Text #17

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1477. Min Act: -0.0001

Data Index: 1609498 (Python Code)

Max Activating Token Index: 701

Text #18

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1458. Min Act: -0.0001

Data Index: 1445138 (Python Code)

Max Activating Token Index: 876

Text #19

Max Range: 0.2300. Min Range: -0.2300

Max Act: 0.1355. Min Act: -0.0001

Data Index: 1473162 (Python Code)

Max Activating Token Index: 555