Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3431 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.5077. Min Range: -2.5077

Max Act: 2.5077. Min Act: -0.0000

Data Index: 969995 (C4 (Web Text))

Max Activating Token Index: 60

Text #1

Max Range: 2.5077. Min Range: -2.5077

Max Act: 2.0717. Min Act: -0.0000

Data Index: 240775 (C4 (Web Text))

Max Activating Token Index: 940

Text #2

Max Range: 2.5077. Min Range: -2.5077

Max Act: 2.0720. Min Act: -0.0000

Data Index: 111114 (C4 (Web Text))

Max Activating Token Index: 806

Text #3

Max Range: 2.5077. Min Range: -2.5077

Max Act: 2.0426. Min Act: -0.0000

Data Index: 111629 (C4 (Web Text))

Max Activating Token Index: 848

Text #4

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.9756. Min Act: -0.0000

Data Index: 807366 (C4 (Web Text))

Max Activating Token Index: 518

Text #5

Max Range: 2.5077. Min Range: -2.5077

Max Act: 2.0958. Min Act: -0.0000

Data Index: 1285517 (C4 (Web Text))

Max Activating Token Index: 605

Text #6

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.8991. Min Act: -0.0000

Data Index: 637877 (C4 (Web Text))

Max Activating Token Index: 107

Text #7

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.9461. Min Act: -0.0000

Data Index: 862 (C4 (Web Text))

Max Activating Token Index: 776

Text #8

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.9277. Min Act: -0.0000

Data Index: 928950 (C4 (Web Text))

Max Activating Token Index: 804

Text #9

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.9540. Min Act: -0.0000

Data Index: 474430 (C4 (Web Text))

Max Activating Token Index: 421

Text #10

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.9047. Min Act: -0.0000

Data Index: 1278886 (C4 (Web Text))

Max Activating Token Index: 396

Text #11

Max Range: 2.5077. Min Range: -2.5077

Max Act: 2.0014. Min Act: -0.0000

Data Index: 59215 (C4 (Web Text))

Max Activating Token Index: 590

Text #12

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.9127. Min Act: -0.0000

Data Index: 1181941 (C4 (Web Text))

Max Activating Token Index: 514

Text #13

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.8949. Min Act: -0.0000

Data Index: 67137 (C4 (Web Text))

Max Activating Token Index: 565

Text #14

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.8630. Min Act: -0.0000

Data Index: 874401 (C4 (Web Text))

Max Activating Token Index: 560

Text #15

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.7794. Min Act: -0.0000

Data Index: 1227176 (C4 (Web Text))

Max Activating Token Index: 883

Text #16

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.7579. Min Act: -0.0000

Data Index: 667154 (C4 (Web Text))

Max Activating Token Index: 818

Text #17

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.8469. Min Act: -0.0000

Data Index: 892053 (C4 (Web Text))

Max Activating Token Index: 145

Text #18

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.6599. Min Act: -0.0000

Data Index: 907110 (C4 (Web Text))

Max Activating Token Index: 812

Text #19

Max Range: 2.5077. Min Range: -2.5077

Max Act: 1.8333. Min Act: -0.0000

Data Index: 1286048 (C4 (Web Text))

Max Activating Token Index: 252