Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2920 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.3564. Min Range: -1.3564

Max Act: 1.3564. Min Act: -0.0000

Data Index: 883243 (C4 (Web Text))

Max Activating Token Index: 239

Text #1

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.7993. Min Act: -0.0000

Data Index: 1280012 (C4 (Web Text))

Max Activating Token Index: 808

Text #2

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.9479. Min Act: -0.0000

Data Index: 375371 (C4 (Web Text))

Max Activating Token Index: 942

Text #3

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.7960. Min Act: -0.0000

Data Index: 118842 (C4 (Web Text))

Max Activating Token Index: 292

Text #4

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.7329. Min Act: -0.0000

Data Index: 934178 (C4 (Web Text))

Max Activating Token Index: 761

Text #5

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.8425. Min Act: -0.0000

Data Index: 1185530 (C4 (Web Text))

Max Activating Token Index: 519

Text #6

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.7365. Min Act: -0.0000

Data Index: 957355 (C4 (Web Text))

Max Activating Token Index: 497

Text #7

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.8092. Min Act: -0.0000

Data Index: 1163418 (C4 (Web Text))

Max Activating Token Index: 256

Text #8

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.7005. Min Act: -0.0000

Data Index: 644972 (C4 (Web Text))

Max Activating Token Index: 721

Text #9

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.6278. Min Act: -0.0000

Data Index: 173704 (C4 (Web Text))

Max Activating Token Index: 695

Text #10

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.6432. Min Act: -0.0000

Data Index: 398895 (C4 (Web Text))

Max Activating Token Index: 626

Text #11

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.6298. Min Act: -0.0000

Data Index: 35655 (C4 (Web Text))

Max Activating Token Index: 127

Text #12

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.6172. Min Act: -0.0000

Data Index: 656310 (C4 (Web Text))

Max Activating Token Index: 974

Text #13

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.5878. Min Act: -0.0000

Data Index: 1284520 (C4 (Web Text))

Max Activating Token Index: 784

Text #14

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.6376. Min Act: -0.0000

Data Index: 705099 (C4 (Web Text))

Max Activating Token Index: 452

Text #15

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.5722. Min Act: -0.0000

Data Index: 1248945 (C4 (Web Text))

Max Activating Token Index: 285

Text #16

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.6019. Min Act: -0.0000

Data Index: 353794 (C4 (Web Text))

Max Activating Token Index: 536

Text #17

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.5813. Min Act: -0.0000

Data Index: 167825 (C4 (Web Text))

Max Activating Token Index: 361

Text #18

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.5802. Min Act: -0.0000

Data Index: 370592 (C4 (Web Text))

Max Activating Token Index: 661

Text #19

Max Range: 1.3564. Min Range: -1.3564

Max Act: 0.6112. Min Act: -0.0000

Data Index: 130499 (C4 (Web Text))

Max Activating Token Index: 425