Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4393 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.6628. Min Act: -0.0000

Data Index: 931536 (C4 (Web Text))

Max Activating Token Index: 234

Text #1

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5815. Min Act: -0.0000

Data Index: 219895 (C4 (Web Text))

Max Activating Token Index: 327

Text #2

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.6042. Min Act: -0.0000

Data Index: 33230 (C4 (Web Text))

Max Activating Token Index: 771

Text #3

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5974. Min Act: -0.0000

Data Index: 749658 (C4 (Web Text))

Max Activating Token Index: 106

Text #4

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5929. Min Act: -0.0000

Data Index: 198595 (C4 (Web Text))

Max Activating Token Index: 770

Text #5

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5677. Min Act: -0.0000

Data Index: 359186 (C4 (Web Text))

Max Activating Token Index: 600

Text #6

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5525. Min Act: -0.0000

Data Index: 794321 (C4 (Web Text))

Max Activating Token Index: 379

Text #7

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5552. Min Act: -0.0000

Data Index: 268445 (C4 (Web Text))

Max Activating Token Index: 654

Text #8

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5675. Min Act: -0.0000

Data Index: 1221173 (C4 (Web Text))

Max Activating Token Index: 904

Text #9

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5391. Min Act: 0.0001

Data Index: 390207 (C4 (Web Text))

Max Activating Token Index: 22

Text #10

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5316. Min Act: -0.0000

Data Index: 621300 (C4 (Web Text))

Max Activating Token Index: 792

Text #11

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5280. Min Act: -0.0000

Data Index: 1272660 (C4 (Web Text))

Max Activating Token Index: 292

Text #12

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5234. Min Act: -0.0000

Data Index: 994765 (C4 (Web Text))

Max Activating Token Index: 28

Text #13

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5603. Min Act: -0.0000

Data Index: 680379 (C4 (Web Text))

Max Activating Token Index: 583

Text #14

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5445. Min Act: -0.0000

Data Index: 1291295 (C4 (Web Text))

Max Activating Token Index: 109

Text #15

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5383. Min Act: -0.0000

Data Index: 663034 (C4 (Web Text))

Max Activating Token Index: 519

Text #16

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5344. Min Act: -0.0000

Data Index: 1011257 (C4 (Web Text))

Max Activating Token Index: 631

Text #17

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5468. Min Act: -0.0000

Data Index: 1011925 (C4 (Web Text))

Max Activating Token Index: 951

Text #18

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5094. Min Act: -0.0000

Data Index: 308073 (C4 (Web Text))

Max Activating Token Index: 517

Text #19

Max Range: 0.6628. Min Range: -0.6628

Max Act: 0.5066. Min Act: -0.0000

Data Index: 96081 (C4 (Web Text))

Max Activating Token Index: 146