Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4247 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.6643. Min Act: -0.0000

Data Index: 906113 (C4 (Web Text))

Max Activating Token Index: 452

Text #1

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.6002. Min Act: -0.0000

Data Index: 232239 (C4 (Web Text))

Max Activating Token Index: 659

Text #2

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.6225. Min Act: -0.0000

Data Index: 528847 (C4 (Web Text))

Max Activating Token Index: 371

Text #3

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.6052. Min Act: -0.0000

Data Index: 527737 (C4 (Web Text))

Max Activating Token Index: 504

Text #4

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.5690. Min Act: -0.0000

Data Index: 580186 (C4 (Web Text))

Max Activating Token Index: 713

Text #5

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.4817. Min Act: -0.0000

Data Index: 676739 (C4 (Web Text))

Max Activating Token Index: 214

Text #6

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.4639. Min Act: -0.0000

Data Index: 1277810 (C4 (Web Text))

Max Activating Token Index: 17

Text #7

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.5086. Min Act: -0.0000

Data Index: 125669 (C4 (Web Text))

Max Activating Token Index: 746

Text #8

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.4430. Min Act: -0.0000

Data Index: 833654 (C4 (Web Text))

Max Activating Token Index: 930

Text #9

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.4567. Min Act: -0.0000

Data Index: 180766 (C4 (Web Text))

Max Activating Token Index: 654

Text #10

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.4316. Min Act: -0.0000

Data Index: 664244 (C4 (Web Text))

Max Activating Token Index: 534

Text #11

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.4279. Min Act: -0.0000

Data Index: 405204 (C4 (Web Text))

Max Activating Token Index: 901

Text #12

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.4043. Min Act: -0.0000

Data Index: 874142 (C4 (Web Text))

Max Activating Token Index: 83

Text #13

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.3468. Min Act: -0.0000

Data Index: 886127 (C4 (Web Text))

Max Activating Token Index: 849

Text #14

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.3667. Min Act: -0.0000

Data Index: 1277374 (C4 (Web Text))

Max Activating Token Index: 901

Text #15

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.3789. Min Act: -0.0000

Data Index: 627174 (C4 (Web Text))

Max Activating Token Index: 40

Text #16

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.3493. Min Act: -0.0000

Data Index: 1006779 (C4 (Web Text))

Max Activating Token Index: 451

Text #17

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.3376. Min Act: -0.0000

Data Index: 1291872 (C4 (Web Text))

Max Activating Token Index: 949

Text #18

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.3357. Min Act: -0.0000

Data Index: 1093470 (C4 (Web Text))

Max Activating Token Index: 976

Text #19

Max Range: 0.6643. Min Range: -0.6643

Max Act: 0.3447. Min Act: -0.0000

Data Index: 575145 (C4 (Web Text))

Max Activating Token Index: 332