Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 88 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.5605. Min Range: -2.5605

Max Act: 2.5021. Min Act: -0.0000

Data Index: 1498484 (Python Code)

Max Activating Token Index: 1000

Text #1

Max Range: 2.5605. Min Range: -2.5605

Max Act: 2.5605. Min Act: -0.0000

Data Index: 1369555 (Python Code)

Max Activating Token Index: 81

Text #2

Max Range: 2.5605. Min Range: -2.5605

Max Act: 2.1861. Min Act: -0.0000

Data Index: 1637812 (Python Code)

Max Activating Token Index: 25

Text #3

Max Range: 2.5605. Min Range: -2.5605

Max Act: 2.2486. Min Act: -0.0000

Data Index: 1557421 (Python Code)

Max Activating Token Index: 379

Text #4

Max Range: 2.5605. Min Range: -2.5605

Max Act: 2.2644. Min Act: -0.0000

Data Index: 1414993 (Python Code)

Max Activating Token Index: 398

Text #5

Max Range: 2.5605. Min Range: -2.5605

Max Act: 2.1036. Min Act: -0.0000

Data Index: 1483823 (Python Code)

Max Activating Token Index: 26

Text #6

Max Range: 2.5605. Min Range: -2.5605

Max Act: 2.0903. Min Act: -0.0000

Data Index: 1424803 (Python Code)

Max Activating Token Index: 583

Text #7

Max Range: 2.5605. Min Range: -2.5605

Max Act: 2.0330. Min Act: -0.0000

Data Index: 1376884 (Python Code)

Max Activating Token Index: 16

Text #8

Max Range: 2.5605. Min Range: -2.5605

Max Act: 2.0438. Min Act: -0.0000

Data Index: 1635359 (Python Code)

Max Activating Token Index: 23

Text #9

Max Range: 2.5605. Min Range: -2.5605

Max Act: 1.9774. Min Act: -0.0000

Data Index: 1496885 (Python Code)

Max Activating Token Index: 436

Text #10

Max Range: 2.5605. Min Range: -2.5605

Max Act: 2.0152. Min Act: -0.0000

Data Index: 1652554 (Python Code)

Max Activating Token Index: 93

Text #11

Max Range: 2.5605. Min Range: -2.5605

Max Act: 2.0130. Min Act: -0.0000

Data Index: 1656521 (Python Code)

Max Activating Token Index: 20

Text #12

Max Range: 2.5605. Min Range: -2.5605

Max Act: 1.9750. Min Act: -0.0000

Data Index: 1582789 (Python Code)

Max Activating Token Index: 514

Text #13

Max Range: 2.5605. Min Range: -2.5605

Max Act: 1.9347. Min Act: -0.0000

Data Index: 1441234 (Python Code)

Max Activating Token Index: 857

Text #14

Max Range: 2.5605. Min Range: -2.5605

Max Act: 1.9524. Min Act: -0.0000

Data Index: 1568898 (Python Code)

Max Activating Token Index: 379

Text #15

Max Range: 2.5605. Min Range: -2.5605

Max Act: 1.9801. Min Act: -0.0000

Data Index: 1453857 (Python Code)

Max Activating Token Index: 96

Text #16

Max Range: 2.5605. Min Range: -2.5605

Max Act: 1.9329. Min Act: -0.0000

Data Index: 1479167 (Python Code)

Max Activating Token Index: 628

Text #17

Max Range: 2.5605. Min Range: -2.5605

Max Act: 1.9229. Min Act: -0.0000

Data Index: 1567158 (Python Code)

Max Activating Token Index: 321

Text #18

Max Range: 2.5605. Min Range: -2.5605

Max Act: 1.9826. Min Act: -0.0000

Data Index: 1649087 (Python Code)

Max Activating Token Index: 841

Text #19

Max Range: 2.5605. Min Range: -2.5605

Max Act: 1.9559. Min Act: -0.0000

Data Index: 1520898 (Python Code)

Max Activating Token Index: 48