Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3961 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.2327. Min Act: -0.0000

Data Index: 1360411 (Python Code)

Max Activating Token Index: 421

Text #1

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1963. Min Act: -0.0001

Data Index: 754681 (C4 (Web Text))

Max Activating Token Index: 842

Text #2

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1980. Min Act: -0.0001

Data Index: 222757 (C4 (Web Text))

Max Activating Token Index: 588

Text #3

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.2040. Min Act: -0.0001

Data Index: 1323765 (C4 (Web Text))

Max Activating Token Index: 834

Text #4

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1974. Min Act: -0.0001

Data Index: 1557247 (Python Code)

Max Activating Token Index: 14

Text #5

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1960. Min Act: -0.0001

Data Index: 1249703 (C4 (Web Text))

Max Activating Token Index: 550

Text #6

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1954. Min Act: -0.0001

Data Index: 994 (C4 (Web Text))

Max Activating Token Index: 638

Text #7

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1831. Min Act: -0.0000

Data Index: 1387232 (Python Code)

Max Activating Token Index: 916

Text #8

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1881. Min Act: -0.0001

Data Index: 793692 (C4 (Web Text))

Max Activating Token Index: 107

Text #9

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1785. Min Act: -0.0000

Data Index: 717875 (C4 (Web Text))

Max Activating Token Index: 877

Text #10

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1724. Min Act: -0.0001

Data Index: 1377657 (Python Code)

Max Activating Token Index: 497

Text #11

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1907. Min Act: -0.0000

Data Index: 1038241 (C4 (Web Text))

Max Activating Token Index: 742

Text #12

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1711. Min Act: -0.0000

Data Index: 1115805 (C4 (Web Text))

Max Activating Token Index: 61

Text #13

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1811. Min Act: -0.0001

Data Index: 1573831 (Python Code)

Max Activating Token Index: 407

Text #14

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1659. Min Act: -0.0000

Data Index: 1603375 (Python Code)

Max Activating Token Index: 255

Text #15

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1699. Min Act: -0.0001

Data Index: 1655767 (Python Code)

Max Activating Token Index: 508

Text #16

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1687. Min Act: -0.0000

Data Index: 143828 (C4 (Web Text))

Max Activating Token Index: 978

Text #17

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1672. Min Act: -0.0001

Data Index: 1440625 (Python Code)

Max Activating Token Index: 636

Text #18

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1607. Min Act: -0.0000

Data Index: 1644018 (Python Code)

Max Activating Token Index: 823

Text #19

Max Range: 0.2327. Min Range: -0.2327

Max Act: 0.1667. Min Act: -0.0001

Data Index: 1467534 (Python Code)

Max Activating Token Index: 567