Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4216 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1373. Min Range: -1.1373

Max Act: 1.1373. Min Act: -0.0000

Data Index: 1485756 (Python Code)

Max Activating Token Index: 943

Text #1

Max Range: 1.1373. Min Range: -1.1373

Max Act: 1.0546. Min Act: 0.0000

Data Index: 1468700 (Python Code)

Max Activating Token Index: 186

Text #2

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.8816. Min Act: -0.0000

Data Index: 1638939 (Python Code)

Max Activating Token Index: 855

Text #3

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.7788. Min Act: -0.0000

Data Index: 1471416 (Python Code)

Max Activating Token Index: 996

Text #4

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.8192. Min Act: -0.0000

Data Index: 1405655 (Python Code)

Max Activating Token Index: 836

Text #5

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.7893. Min Act: -0.0000

Data Index: 1603121 (Python Code)

Max Activating Token Index: 528

Text #6

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.7395. Min Act: -0.0000

Data Index: 1436338 (Python Code)

Max Activating Token Index: 940

Text #7

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.7539. Min Act: -0.0000

Data Index: 1535294 (Python Code)

Max Activating Token Index: 267

Text #8

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.7694. Min Act: -0.0000

Data Index: 1543366 (Python Code)

Max Activating Token Index: 435

Text #9

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.6770. Min Act: -0.0000

Data Index: 1614516 (Python Code)

Max Activating Token Index: 604

Text #10

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.6881. Min Act: -0.0000

Data Index: 1566074 (Python Code)

Max Activating Token Index: 235

Text #11

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.7503. Min Act: 0.0000

Data Index: 1409327 (Python Code)

Max Activating Token Index: 388

Text #12

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.7613. Min Act: -0.0000

Data Index: 1539673 (Python Code)

Max Activating Token Index: 327

Text #13

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.7398. Min Act: -0.0000

Data Index: 1398516 (Python Code)

Max Activating Token Index: 283

Text #14

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.7436. Min Act: -0.0000

Data Index: 1427260 (Python Code)

Max Activating Token Index: 898

Text #15

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.6555. Min Act: -0.0000

Data Index: 974102 (C4 (Web Text))

Max Activating Token Index: 723

Text #16

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.6873. Min Act: -0.0000

Data Index: 1497224 (Python Code)

Max Activating Token Index: 381

Text #17

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.7119. Min Act: -0.0000

Data Index: 1456522 (Python Code)

Max Activating Token Index: 69

Text #18

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.6890. Min Act: 0.0000

Data Index: 1552226 (Python Code)

Max Activating Token Index: 1003

Text #19

Max Range: 1.1373. Min Range: -1.1373

Max Act: 0.6410. Min Act: -0.0000

Data Index: 1388172 (Python Code)

Max Activating Token Index: 67