Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 794 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1461. Min Range: -2.1461

Max Act: 2.1461. Min Act: -0.0000

Data Index: 200813 (C4 (Web Text))

Max Activating Token Index: 526

Text #1

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.8084. Min Act: -0.0000

Data Index: 1352480 (C4 (Web Text))

Max Activating Token Index: 596

Text #2

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.9703. Min Act: -0.0000

Data Index: 1242467 (C4 (Web Text))

Max Activating Token Index: 211

Text #3

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.8858. Min Act: -0.0000

Data Index: 1304029 (C4 (Web Text))

Max Activating Token Index: 693

Text #4

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.7417. Min Act: -0.0000

Data Index: 767807 (C4 (Web Text))

Max Activating Token Index: 903

Text #5

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.6825. Min Act: -0.0000

Data Index: 689552 (C4 (Web Text))

Max Activating Token Index: 78

Text #6

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.5374. Min Act: -0.0000

Data Index: 671922 (C4 (Web Text))

Max Activating Token Index: 87

Text #7

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.5840. Min Act: -0.0000

Data Index: 760341 (C4 (Web Text))

Max Activating Token Index: 69

Text #8

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.5038. Min Act: -0.0000

Data Index: 1462850 (Python Code)

Max Activating Token Index: 1019

Text #9

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.6618. Min Act: -0.0000

Data Index: 101015 (C4 (Web Text))

Max Activating Token Index: 410

Text #10

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.5494. Min Act: -0.0000

Data Index: 444449 (C4 (Web Text))

Max Activating Token Index: 185

Text #11

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.4543. Min Act: -0.0000

Data Index: 577455 (C4 (Web Text))

Max Activating Token Index: 315

Text #12

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.4203. Min Act: -0.0000

Data Index: 270395 (C4 (Web Text))

Max Activating Token Index: 937

Text #13

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.4950. Min Act: -0.0000

Data Index: 541134 (C4 (Web Text))

Max Activating Token Index: 641

Text #14

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.4743. Min Act: -0.0000

Data Index: 959682 (C4 (Web Text))

Max Activating Token Index: 394

Text #15

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.4626. Min Act: -0.0000

Data Index: 913662 (C4 (Web Text))

Max Activating Token Index: 175

Text #16

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.4562. Min Act: -0.0000

Data Index: 850319 (C4 (Web Text))

Max Activating Token Index: 517

Text #17

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.4406. Min Act: -0.0000

Data Index: 243559 (C4 (Web Text))

Max Activating Token Index: 775

Text #18

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.4288. Min Act: -0.0000

Data Index: 1162601 (C4 (Web Text))

Max Activating Token Index: 69

Text #19

Max Range: 2.1461. Min Range: -2.1461

Max Act: 1.4097. Min Act: -0.0000

Data Index: 626342 (C4 (Web Text))

Max Activating Token Index: 345