Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5573 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 5.0438. Min Range: -5.0438

Max Act: 5.0438. Min Act: -0.0000

Data Index: 1323889 (C4 (Web Text))

Max Activating Token Index: 970

Text #1

Max Range: 5.0438. Min Range: -5.0438

Max Act: 4.5307. Min Act: -0.0000

Data Index: 42914 (C4 (Web Text))

Max Activating Token Index: 37

Text #2

Max Range: 5.0438. Min Range: -5.0438

Max Act: 4.4961. Min Act: -0.0000

Data Index: 1262165 (C4 (Web Text))

Max Activating Token Index: 832

Text #3

Max Range: 5.0438. Min Range: -5.0438

Max Act: 4.4968. Min Act: -0.0000

Data Index: 770687 (C4 (Web Text))

Max Activating Token Index: 211

Text #4

Max Range: 5.0438. Min Range: -5.0438

Max Act: 4.1085. Min Act: -0.0000

Data Index: 932206 (C4 (Web Text))

Max Activating Token Index: 453

Text #5

Max Range: 5.0438. Min Range: -5.0438

Max Act: 4.2063. Min Act: -0.0000

Data Index: 1102582 (C4 (Web Text))

Max Activating Token Index: 935

Text #6

Max Range: 5.0438. Min Range: -5.0438

Max Act: 4.3133. Min Act: -0.0000

Data Index: 625192 (C4 (Web Text))

Max Activating Token Index: 190

Text #7

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.8064. Min Act: -0.0000

Data Index: 177876 (C4 (Web Text))

Max Activating Token Index: 143

Text #8

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.8929. Min Act: -0.0000

Data Index: 63423 (C4 (Web Text))

Max Activating Token Index: 726

Text #9

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.8682. Min Act: -0.0000

Data Index: 123851 (C4 (Web Text))

Max Activating Token Index: 1013

Text #10

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.6621. Min Act: -0.0000

Data Index: 1125652 (C4 (Web Text))

Max Activating Token Index: 57

Text #11

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.6247. Min Act: -0.0000

Data Index: 1060964 (C4 (Web Text))

Max Activating Token Index: 6

Text #12

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.7363. Min Act: -0.0000

Data Index: 56337 (C4 (Web Text))

Max Activating Token Index: 464

Text #13

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.4619. Min Act: -0.0000

Data Index: 602696 (C4 (Web Text))

Max Activating Token Index: 717

Text #14

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.4361. Min Act: -0.0000

Data Index: 211274 (C4 (Web Text))

Max Activating Token Index: 551

Text #15

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.4179. Min Act: -0.0000

Data Index: 143795 (C4 (Web Text))

Max Activating Token Index: 774

Text #16

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.2477. Min Act: -0.0000

Data Index: 198402 (C4 (Web Text))

Max Activating Token Index: 142

Text #17

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.5600. Min Act: -0.0000

Data Index: 585093 (C4 (Web Text))

Max Activating Token Index: 740

Text #18

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.2852. Min Act: -0.0000

Data Index: 492551 (C4 (Web Text))

Max Activating Token Index: 539

Text #19

Max Range: 5.0438. Min Range: -5.0438

Max Act: 3.1243. Min Act: -0.0000

Data Index: 71450 (C4 (Web Text))

Max Activating Token Index: 72