Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1024 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.8788. Min Act: -0.0000

Data Index: 1463304 (Python Code)

Max Activating Token Index: 119

Text #1

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.8644. Min Act: -0.0000

Data Index: 1511777 (Python Code)

Max Activating Token Index: 531

Text #2

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.7929. Min Act: -0.0000

Data Index: 1374914 (Python Code)

Max Activating Token Index: 99

Text #3

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.8009. Min Act: -0.0000

Data Index: 1603887 (Python Code)

Max Activating Token Index: 563

Text #4

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.6652. Min Act: -0.0000

Data Index: 1458622 (Python Code)

Max Activating Token Index: 903

Text #5

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.7368. Min Act: -0.0000

Data Index: 1590620 (Python Code)

Max Activating Token Index: 269

Text #6

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.6216. Min Act: -0.0000

Data Index: 1360020 (Python Code)

Max Activating Token Index: 235

Text #7

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.6278. Min Act: -0.0000

Data Index: 1498422 (Python Code)

Max Activating Token Index: 467

Text #8

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.7087. Min Act: -0.0000

Data Index: 1530879 (Python Code)

Max Activating Token Index: 500

Text #9

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.6201. Min Act: -0.0000

Data Index: 1652415 (Python Code)

Max Activating Token Index: 199

Text #10

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.7150. Min Act: -0.0000

Data Index: 1560007 (Python Code)

Max Activating Token Index: 30

Text #11

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.8017. Min Act: -0.0000

Data Index: 1627954 (Python Code)

Max Activating Token Index: 967

Text #12

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.5811. Min Act: -0.0000

Data Index: 1556387 (Python Code)

Max Activating Token Index: 809

Text #13

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.6772. Min Act: -0.0000

Data Index: 1435557 (Python Code)

Max Activating Token Index: 254

Text #14

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.7868. Min Act: -0.0000

Data Index: 1443973 (Python Code)

Max Activating Token Index: 919

Text #15

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.5691. Min Act: -0.0000

Data Index: 1463520 (Python Code)

Max Activating Token Index: 262

Text #16

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.5755. Min Act: -0.0000

Data Index: 1646156 (Python Code)

Max Activating Token Index: 159

Text #17

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.5737. Min Act: -0.0000

Data Index: 1362023 (Python Code)

Max Activating Token Index: 199

Text #18

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.6488. Min Act: -0.0000

Data Index: 1635680 (Python Code)

Max Activating Token Index: 53

Text #19

Max Range: 1.8788. Min Range: -1.8788

Max Act: 1.5623. Min Act: -0.0000

Data Index: 1400840 (Python Code)

Max Activating Token Index: 461