Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1723 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 6.0834. Min Range: -6.0834

Max Act: 6.0834. Min Act: -0.0001

Data Index: 1446280 (Python Code)

Max Activating Token Index: 858

Text #1

Max Range: 6.0834. Min Range: -6.0834

Max Act: 5.4619. Min Act: -0.0000

Data Index: 1547303 (Python Code)

Max Activating Token Index: 66

Text #2

Max Range: 6.0834. Min Range: -6.0834

Max Act: 5.1614. Min Act: -0.0001

Data Index: 1468557 (Python Code)

Max Activating Token Index: 298

Text #3

Max Range: 6.0834. Min Range: -6.0834

Max Act: 5.0474. Min Act: -0.0001

Data Index: 1646066 (Python Code)

Max Activating Token Index: 359

Text #4

Max Range: 6.0834. Min Range: -6.0834

Max Act: 5.0875. Min Act: -0.0000

Data Index: 1478389 (Python Code)

Max Activating Token Index: 705

Text #5

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.9438. Min Act: -0.0000

Data Index: 654815 (C4 (Web Text))

Max Activating Token Index: 1016

Text #6

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.6218. Min Act: -0.0000

Data Index: 1483237 (Python Code)

Max Activating Token Index: 663

Text #7

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.1318. Min Act: -0.0001

Data Index: 1596585 (Python Code)

Max Activating Token Index: 865

Text #8

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.7227. Min Act: -0.0001

Data Index: 1646952 (Python Code)

Max Activating Token Index: 745

Text #9

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.3909. Min Act: -0.0001

Data Index: 566660 (C4 (Web Text))

Max Activating Token Index: 983

Text #10

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.7478. Min Act: -0.0001

Data Index: 1363185 (Python Code)

Max Activating Token Index: 807

Text #11

Max Range: 6.0834. Min Range: -6.0834

Max Act: 5.1518. Min Act: -0.0000

Data Index: 1417052 (Python Code)

Max Activating Token Index: 358

Text #12

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.9511. Min Act: -0.0000

Data Index: 1563636 (Python Code)

Max Activating Token Index: 956

Text #13

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.3695. Min Act: -0.0000

Data Index: 1610899 (Python Code)

Max Activating Token Index: 306

Text #14

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.5297. Min Act: -0.0000

Data Index: 1405126 (Python Code)

Max Activating Token Index: 191

Text #15

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.1994. Min Act: -0.0000

Data Index: 1597706 (Python Code)

Max Activating Token Index: 633

Text #16

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.4960. Min Act: -0.0000

Data Index: 695728 (C4 (Web Text))

Max Activating Token Index: 617

Text #17

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.5566. Min Act: -0.0000

Data Index: 1427991 (Python Code)

Max Activating Token Index: 539

Text #18

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.5258. Min Act: -0.0001

Data Index: 712623 (C4 (Web Text))

Max Activating Token Index: 997

Text #19

Max Range: 6.0834. Min Range: -6.0834

Max Act: 4.4016. Min Act: -0.0000

Data Index: 1402802 (Python Code)

Max Activating Token Index: 692