Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1972 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3504. Min Range: -1.3504

Max Act: 1.3504. Min Act: 0.0000

Data Index: 1291463 (C4 (Web Text))

Max Activating Token Index: 639

Text #1

Max Range: 1.3504. Min Range: -1.3504

Max Act: 1.0852. Min Act: 0.0000

Data Index: 322997 (C4 (Web Text))

Max Activating Token Index: 35

Text #2

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.9367. Min Act: 0.0000

Data Index: 79729 (C4 (Web Text))

Max Activating Token Index: 730

Text #3

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.8885. Min Act: 0.0000

Data Index: 1156527 (C4 (Web Text))

Max Activating Token Index: 700

Text #4

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.9386. Min Act: -0.0000

Data Index: 1279982 (C4 (Web Text))

Max Activating Token Index: 833

Text #5

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.8398. Min Act: -0.0000

Data Index: 446239 (C4 (Web Text))

Max Activating Token Index: 923

Text #6

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.8640. Min Act: -0.0000

Data Index: 1028512 (C4 (Web Text))

Max Activating Token Index: 431

Text #7

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.7537. Min Act: -0.0000

Data Index: 227763 (C4 (Web Text))

Max Activating Token Index: 242

Text #8

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.7941. Min Act: 0.0000

Data Index: 993065 (C4 (Web Text))

Max Activating Token Index: 650

Text #9

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.7569. Min Act: -0.0000

Data Index: 763222 (C4 (Web Text))

Max Activating Token Index: 440

Text #10

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.8147. Min Act: -0.0000

Data Index: 808520 (C4 (Web Text))

Max Activating Token Index: 255

Text #11

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.8272. Min Act: 0.0000

Data Index: 391717 (C4 (Web Text))

Max Activating Token Index: 322

Text #12

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.7577. Min Act: 0.0000

Data Index: 434499 (C4 (Web Text))

Max Activating Token Index: 880

Text #13

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.7697. Min Act: -0.0000

Data Index: 747364 (C4 (Web Text))

Max Activating Token Index: 883

Text #14

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.7533. Min Act: 0.0000

Data Index: 242090 (C4 (Web Text))

Max Activating Token Index: 63

Text #15

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.7786. Min Act: -0.0000

Data Index: 115285 (C4 (Web Text))

Max Activating Token Index: 860

Text #16

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.7149. Min Act: 0.0000

Data Index: 616028 (C4 (Web Text))

Max Activating Token Index: 336

Text #17

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.7512. Min Act: -0.0000

Data Index: 887285 (C4 (Web Text))

Max Activating Token Index: 589

Text #18

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.6892. Min Act: -0.0000

Data Index: 1034669 (C4 (Web Text))

Max Activating Token Index: 222

Text #19

Max Range: 1.3504. Min Range: -1.3504

Max Act: 0.7022. Min Act: 0.0000

Data Index: 794921 (C4 (Web Text))

Max Activating Token Index: 26