Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1300 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.3976. Min Act: -0.0001

Data Index: 1272928 (C4 (Web Text))

Max Activating Token Index: 332

Text #1

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.8680. Min Act: -0.0001

Data Index: 801061 (C4 (Web Text))

Max Activating Token Index: 327

Text #2

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.6018. Min Act: -0.0001

Data Index: 621473 (C4 (Web Text))

Max Activating Token Index: 86

Text #3

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.7997. Min Act: -0.0001

Data Index: 1102235 (C4 (Web Text))

Max Activating Token Index: 206

Text #4

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.8486. Min Act: -0.0001

Data Index: 666622 (C4 (Web Text))

Max Activating Token Index: 327

Text #5

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.5662. Min Act: -0.0001

Data Index: 435150 (C4 (Web Text))

Max Activating Token Index: 891

Text #6

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.5909. Min Act: -0.0001

Data Index: 1048597 (C4 (Web Text))

Max Activating Token Index: 126

Text #7

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.7470. Min Act: -0.0001

Data Index: 769051 (C4 (Web Text))

Max Activating Token Index: 133

Text #8

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.9089. Min Act: -0.0001

Data Index: 1205725 (C4 (Web Text))

Max Activating Token Index: 735

Text #9

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.8834. Min Act: -0.0001

Data Index: 881088 (C4 (Web Text))

Max Activating Token Index: 173

Text #10

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.7333. Min Act: -0.0001

Data Index: 869535 (C4 (Web Text))

Max Activating Token Index: 711

Text #11

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.6344. Min Act: -0.0000

Data Index: 757395 (C4 (Web Text))

Max Activating Token Index: 758

Text #12

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.6920. Min Act: -0.0001

Data Index: 412102 (C4 (Web Text))

Max Activating Token Index: 904

Text #13

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.5955. Min Act: -0.0001

Data Index: 345792 (C4 (Web Text))

Max Activating Token Index: 599

Text #14

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.6011. Min Act: -0.0001

Data Index: 754213 (C4 (Web Text))

Max Activating Token Index: 215

Text #15

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.9325. Min Act: -0.0001

Data Index: 2052 (C4 (Web Text))

Max Activating Token Index: 839

Text #16

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.7667. Min Act: -0.0001

Data Index: 70696 (C4 (Web Text))

Max Activating Token Index: 839

Text #17

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.5806. Min Act: -0.0001

Data Index: 1101854 (C4 (Web Text))

Max Activating Token Index: 935

Text #18

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.7460. Min Act: -0.0001

Data Index: 418894 (C4 (Web Text))

Max Activating Token Index: 112

Text #19

Max Range: 3.9325. Min Range: -3.9325

Max Act: 3.4980. Min Act: -0.0001

Data Index: 1310453 (C4 (Web Text))

Max Activating Token Index: 290