Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4424 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.4448. Min Range: -2.4448

Max Act: 2.4448. Min Act: -0.0000

Data Index: 349912 (C4 (Web Text))

Max Activating Token Index: 943

Text #1

Max Range: 2.4448. Min Range: -2.4448

Max Act: 2.1397. Min Act: -0.0000

Data Index: 62033 (C4 (Web Text))

Max Activating Token Index: 942

Text #2

Max Range: 2.4448. Min Range: -2.4448

Max Act: 2.0092. Min Act: -0.0000

Data Index: 535951 (C4 (Web Text))

Max Activating Token Index: 574

Text #3

Max Range: 2.4448. Min Range: -2.4448

Max Act: 2.1949. Min Act: -0.0000

Data Index: 1174359 (C4 (Web Text))

Max Activating Token Index: 946

Text #4

Max Range: 2.4448. Min Range: -2.4448

Max Act: 2.0670. Min Act: -0.0000

Data Index: 135517 (C4 (Web Text))

Max Activating Token Index: 646

Text #5

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.9050. Min Act: -0.0000

Data Index: 107395 (C4 (Web Text))

Max Activating Token Index: 566

Text #6

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.7688. Min Act: -0.0000

Data Index: 494537 (C4 (Web Text))

Max Activating Token Index: 396

Text #7

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.8065. Min Act: -0.0000

Data Index: 102439 (C4 (Web Text))

Max Activating Token Index: 923

Text #8

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.5772. Min Act: -0.0000

Data Index: 899471 (C4 (Web Text))

Max Activating Token Index: 217

Text #9

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.6701. Min Act: -0.0000

Data Index: 1117023 (C4 (Web Text))

Max Activating Token Index: 298

Text #10

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.5412. Min Act: -0.0000

Data Index: 269101 (C4 (Web Text))

Max Activating Token Index: 567

Text #11

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.5968. Min Act: -0.0000

Data Index: 1338054 (C4 (Web Text))

Max Activating Token Index: 685

Text #12

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.5605. Min Act: -0.0000

Data Index: 779911 (C4 (Web Text))

Max Activating Token Index: 965

Text #13

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.5065. Min Act: -0.0000

Data Index: 979784 (C4 (Web Text))

Max Activating Token Index: 792

Text #14

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.5327. Min Act: -0.0000

Data Index: 268911 (C4 (Web Text))

Max Activating Token Index: 258

Text #15

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.3472. Min Act: -0.0000

Data Index: 465090 (C4 (Web Text))

Max Activating Token Index: 545

Text #16

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.4955. Min Act: -0.0000

Data Index: 1156791 (C4 (Web Text))

Max Activating Token Index: 943

Text #17

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.4817. Min Act: -0.0000

Data Index: 1038217 (C4 (Web Text))

Max Activating Token Index: 927

Text #18

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.3905. Min Act: -0.0000

Data Index: 215915 (C4 (Web Text))

Max Activating Token Index: 906

Text #19

Max Range: 2.4448. Min Range: -2.4448

Max Act: 1.4567. Min Act: -0.0000

Data Index: 607300 (C4 (Web Text))

Max Activating Token Index: 799