Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 691 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.8277. Min Act: -0.0000

Data Index: 635060 (C4 (Web Text))

Max Activating Token Index: 780

Text #1

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.7047. Min Act: -0.0000

Data Index: 1087970 (C4 (Web Text))

Max Activating Token Index: 876

Text #2

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.6125. Min Act: -0.0000

Data Index: 276674 (C4 (Web Text))

Max Activating Token Index: 949

Text #3

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.6251. Min Act: -0.0000

Data Index: 643432 (C4 (Web Text))

Max Activating Token Index: 992

Text #4

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.6013. Min Act: -0.0000

Data Index: 1019514 (C4 (Web Text))

Max Activating Token Index: 152

Text #5

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.5868. Min Act: -0.0000

Data Index: 209165 (C4 (Web Text))

Max Activating Token Index: 597

Text #6

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.5874. Min Act: 0.0000

Data Index: 571676 (C4 (Web Text))

Max Activating Token Index: 983

Text #7

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.5455. Min Act: -0.0000

Data Index: 350792 (C4 (Web Text))

Max Activating Token Index: 789

Text #8

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.5509. Min Act: -0.0000

Data Index: 382460 (C4 (Web Text))

Max Activating Token Index: 827

Text #9

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.5587. Min Act: 0.0000

Data Index: 174507 (C4 (Web Text))

Max Activating Token Index: 468

Text #10

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.5844. Min Act: -0.0000

Data Index: 1130397 (C4 (Web Text))

Max Activating Token Index: 832

Text #11

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.5843. Min Act: -0.0000

Data Index: 899553 (C4 (Web Text))

Max Activating Token Index: 302

Text #12

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.5454. Min Act: -0.0000

Data Index: 1074560 (C4 (Web Text))

Max Activating Token Index: 545

Text #13

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.5557. Min Act: -0.0000

Data Index: 1229062 (C4 (Web Text))

Max Activating Token Index: 524

Text #14

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.5382. Min Act: -0.0000

Data Index: 641797 (C4 (Web Text))

Max Activating Token Index: 489

Text #15

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.5401. Min Act: -0.0000

Data Index: 712727 (C4 (Web Text))

Max Activating Token Index: 973

Text #16

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.5254. Min Act: -0.0000

Data Index: 950450 (C4 (Web Text))

Max Activating Token Index: 701

Text #17

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.4992. Min Act: -0.0000

Data Index: 530730 (C4 (Web Text))

Max Activating Token Index: 514

Text #18

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.4953. Min Act: -0.0000

Data Index: 492656 (C4 (Web Text))

Max Activating Token Index: 195

Text #19

Max Range: 0.8277. Min Range: -0.8277

Max Act: 0.4903. Min Act: -0.0000

Data Index: 1023697 (C4 (Web Text))

Max Activating Token Index: 928