Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 117 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0489. Min Act: -0.0003

Data Index: 159107 (C4 (Web Text))

Max Activating Token Index: 557

Text #1

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0573. Min Act: -0.0003

Data Index: 1343814 (C4 (Web Text))

Max Activating Token Index: 746

Text #2

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0511. Min Act: -0.0003

Data Index: 388672 (C4 (Web Text))

Max Activating Token Index: 785

Text #3

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0534. Min Act: -0.0003

Data Index: 715077 (C4 (Web Text))

Max Activating Token Index: 581

Text #4

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0515. Min Act: -0.0003

Data Index: 886939 (C4 (Web Text))

Max Activating Token Index: 237

Text #5

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0554. Min Act: -0.0003

Data Index: 103894 (C4 (Web Text))

Max Activating Token Index: 335

Text #6

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0500. Min Act: -0.0003

Data Index: 549644 (C4 (Web Text))

Max Activating Token Index: 207

Text #7

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0441. Min Act: -0.0003

Data Index: 224338 (C4 (Web Text))

Max Activating Token Index: 421

Text #8

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0508. Min Act: -0.0003

Data Index: 1092929 (C4 (Web Text))

Max Activating Token Index: 674

Text #9

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0445. Min Act: -0.0003

Data Index: 857689 (C4 (Web Text))

Max Activating Token Index: 872

Text #10

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0443. Min Act: -0.0003

Data Index: 737597 (C4 (Web Text))

Max Activating Token Index: 329

Text #11

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0490. Min Act: -0.0003

Data Index: 410077 (C4 (Web Text))

Max Activating Token Index: 901

Text #12

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0473. Min Act: -0.0003

Data Index: 613461 (C4 (Web Text))

Max Activating Token Index: 437

Text #13

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0461. Min Act: -0.0003

Data Index: 996852 (C4 (Web Text))

Max Activating Token Index: 864

Text #14

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0469. Min Act: -0.0003

Data Index: 843244 (C4 (Web Text))

Max Activating Token Index: 383

Text #15

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0461. Min Act: -0.0003

Data Index: 568400 (C4 (Web Text))

Max Activating Token Index: 794

Text #16

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0458. Min Act: -0.0003

Data Index: 978789 (C4 (Web Text))

Max Activating Token Index: 747

Text #17

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0427. Min Act: -0.0003

Data Index: 452577 (C4 (Web Text))

Max Activating Token Index: 141

Text #18

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0486. Min Act: -0.0003

Data Index: 1114921 (C4 (Web Text))

Max Activating Token Index: 429

Text #19

Max Range: 0.0573. Min Range: -0.0573

Max Act: 0.0457. Min Act: -0.0003

Data Index: 1201203 (C4 (Web Text))

Max Activating Token Index: 901