Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2065 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.6359. Min Act: -0.0000

Data Index: 690283 (C4 (Web Text))

Max Activating Token Index: 1023

Text #1

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.5385. Min Act: -0.0000

Data Index: 1219680 (C4 (Web Text))

Max Activating Token Index: 525

Text #2

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.5000. Min Act: -0.0000

Data Index: 532725 (C4 (Web Text))

Max Activating Token Index: 140

Text #3

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.4278. Min Act: -0.0000

Data Index: 526621 (C4 (Web Text))

Max Activating Token Index: 350

Text #4

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.3847. Min Act: -0.0000

Data Index: 514404 (C4 (Web Text))

Max Activating Token Index: 716

Text #5

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.4144. Min Act: -0.0000

Data Index: 1043315 (C4 (Web Text))

Max Activating Token Index: 173

Text #6

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.4061. Min Act: -0.0000

Data Index: 520545 (C4 (Web Text))

Max Activating Token Index: 41

Text #7

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.3354. Min Act: -0.0000

Data Index: 774184 (C4 (Web Text))

Max Activating Token Index: 372

Text #8

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.3865. Min Act: 0.0000

Data Index: 1030886 (C4 (Web Text))

Max Activating Token Index: 1016

Text #9

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.3551. Min Act: -0.0000

Data Index: 410376 (C4 (Web Text))

Max Activating Token Index: 940

Text #10

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.3370. Min Act: 0.0000

Data Index: 791623 (C4 (Web Text))

Max Activating Token Index: 718

Text #11

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.3305. Min Act: -0.0000

Data Index: 1242016 (C4 (Web Text))

Max Activating Token Index: 582

Text #12

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.2993. Min Act: -0.0000

Data Index: 207174 (C4 (Web Text))

Max Activating Token Index: 196

Text #13

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.2917. Min Act: -0.0000

Data Index: 546519 (C4 (Web Text))

Max Activating Token Index: 425

Text #14

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.3011. Min Act: -0.0000

Data Index: 660567 (C4 (Web Text))

Max Activating Token Index: 530

Text #15

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.2667. Min Act: -0.0000

Data Index: 843232 (C4 (Web Text))

Max Activating Token Index: 28

Text #16

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.3153. Min Act: -0.0000

Data Index: 742664 (C4 (Web Text))

Max Activating Token Index: 613

Text #17

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.2976. Min Act: -0.0000

Data Index: 700959 (C4 (Web Text))

Max Activating Token Index: 879

Text #18

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.3158. Min Act: 0.0000

Data Index: 725148 (C4 (Web Text))

Max Activating Token Index: 724

Text #19

Max Range: 0.6359. Min Range: -0.6359

Max Act: 0.2872. Min Act: -0.0000

Data Index: 209418 (C4 (Web Text))

Max Activating Token Index: 614