Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2077 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1589. Min Range: -1.1589

Max Act: 1.1589. Min Act: -0.0000

Data Index: 10273 (C4 (Web Text))

Max Activating Token Index: 845

Text #1

Max Range: 1.1589. Min Range: -1.1589

Max Act: 1.0603. Min Act: -0.0000

Data Index: 447361 (C4 (Web Text))

Max Activating Token Index: 925

Text #2

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.9283. Min Act: -0.0000

Data Index: 1117435 (C4 (Web Text))

Max Activating Token Index: 197

Text #3

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.9086. Min Act: -0.0000

Data Index: 33015 (C4 (Web Text))

Max Activating Token Index: 430

Text #4

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8912. Min Act: -0.0000

Data Index: 752968 (C4 (Web Text))

Max Activating Token Index: 177

Text #5

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8866. Min Act: -0.0000

Data Index: 741752 (C4 (Web Text))

Max Activating Token Index: 877

Text #6

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.9457. Min Act: -0.0000

Data Index: 882213 (C4 (Web Text))

Max Activating Token Index: 1013

Text #7

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8774. Min Act: -0.0000

Data Index: 981655 (C4 (Web Text))

Max Activating Token Index: 892

Text #8

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8627. Min Act: -0.0000

Data Index: 890879 (C4 (Web Text))

Max Activating Token Index: 232

Text #9

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8520. Min Act: -0.0000

Data Index: 673284 (C4 (Web Text))

Max Activating Token Index: 954

Text #10

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8039. Min Act: -0.0000

Data Index: 1358705 (C4 (Web Text))

Max Activating Token Index: 966

Text #11

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8201. Min Act: -0.0000

Data Index: 1010940 (C4 (Web Text))

Max Activating Token Index: 795

Text #12

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.7934. Min Act: -0.0000

Data Index: 1142495 (C4 (Web Text))

Max Activating Token Index: 939

Text #13

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8128. Min Act: -0.0000

Data Index: 791872 (C4 (Web Text))

Max Activating Token Index: 869

Text #14

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8883. Min Act: -0.0000

Data Index: 729973 (C4 (Web Text))

Max Activating Token Index: 972

Text #15

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.7817. Min Act: -0.0000

Data Index: 1317457 (C4 (Web Text))

Max Activating Token Index: 931

Text #16

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8308. Min Act: -0.0000

Data Index: 293448 (C4 (Web Text))

Max Activating Token Index: 756

Text #17

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8000. Min Act: -0.0000

Data Index: 500995 (C4 (Web Text))

Max Activating Token Index: 506

Text #18

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8049. Min Act: -0.0000

Data Index: 1069688 (C4 (Web Text))

Max Activating Token Index: 769

Text #19

Max Range: 1.1589. Min Range: -1.1589

Max Act: 0.8154. Min Act: -0.0000

Data Index: 1008928 (C4 (Web Text))

Max Activating Token Index: 676