Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2420 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1154. Min Range: -1.1154

Max Act: 1.1154. Min Act: -0.0000

Data Index: 478613 (C4 (Web Text))

Max Activating Token Index: 930

Text #1

Max Range: 1.1154. Min Range: -1.1154

Max Act: 1.0512. Min Act: -0.0000

Data Index: 912161 (C4 (Web Text))

Max Activating Token Index: 557

Text #2

Max Range: 1.1154. Min Range: -1.1154

Max Act: 1.0506. Min Act: -0.0000

Data Index: 314430 (C4 (Web Text))

Max Activating Token Index: 362

Text #3

Max Range: 1.1154. Min Range: -1.1154

Max Act: 1.0140. Min Act: 0.0000

Data Index: 223620 (C4 (Web Text))

Max Activating Token Index: 251

Text #4

Max Range: 1.1154. Min Range: -1.1154

Max Act: 1.0099. Min Act: -0.0000

Data Index: 670759 (C4 (Web Text))

Max Activating Token Index: 68

Text #5

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.9958. Min Act: -0.0000

Data Index: 761703 (C4 (Web Text))

Max Activating Token Index: 558

Text #6

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.8370. Min Act: -0.0000

Data Index: 595369 (C4 (Web Text))

Max Activating Token Index: 143

Text #7

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.9545. Min Act: -0.0000

Data Index: 1166224 (C4 (Web Text))

Max Activating Token Index: 124

Text #8

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.9192. Min Act: 0.0000

Data Index: 800077 (C4 (Web Text))

Max Activating Token Index: 710

Text #9

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.9716. Min Act: -0.0000

Data Index: 1222207 (C4 (Web Text))

Max Activating Token Index: 203

Text #10

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.9680. Min Act: -0.0000

Data Index: 931274 (C4 (Web Text))

Max Activating Token Index: 531

Text #11

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.8949. Min Act: -0.0000

Data Index: 238241 (C4 (Web Text))

Max Activating Token Index: 936

Text #12

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.9170. Min Act: -0.0000

Data Index: 36696 (C4 (Web Text))

Max Activating Token Index: 289

Text #13

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.9370. Min Act: 0.0001

Data Index: 1111973 (C4 (Web Text))

Max Activating Token Index: 43

Text #14

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.9526. Min Act: -0.0000

Data Index: 1227713 (C4 (Web Text))

Max Activating Token Index: 417

Text #15

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.8624. Min Act: 0.0000

Data Index: 1305063 (C4 (Web Text))

Max Activating Token Index: 85

Text #16

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.8550. Min Act: -0.0000

Data Index: 467479 (C4 (Web Text))

Max Activating Token Index: 68

Text #17

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.8799. Min Act: -0.0000

Data Index: 621710 (C4 (Web Text))

Max Activating Token Index: 74

Text #18

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.8788. Min Act: -0.0000

Data Index: 253264 (C4 (Web Text))

Max Activating Token Index: 415

Text #19

Max Range: 1.1154. Min Range: -1.1154

Max Act: 0.8929. Min Act: -0.0000

Data Index: 762903 (C4 (Web Text))

Max Activating Token Index: 251