Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3984 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.7747. Min Act: -0.0000

Data Index: 354947 (C4 (Web Text))

Max Activating Token Index: 692

Text #1

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.7495. Min Act: -0.0000

Data Index: 443774 (C4 (Web Text))

Max Activating Token Index: 670

Text #2

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.7626. Min Act: -0.0000

Data Index: 228049 (C4 (Web Text))

Max Activating Token Index: 1021

Text #3

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.7306. Min Act: -0.0000

Data Index: 1248792 (C4 (Web Text))

Max Activating Token Index: 720

Text #4

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.7142. Min Act: -0.0000

Data Index: 593394 (C4 (Web Text))

Max Activating Token Index: 695

Text #5

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.7038. Min Act: -0.0000

Data Index: 892107 (C4 (Web Text))

Max Activating Token Index: 483

Text #6

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.7274. Min Act: -0.0000

Data Index: 39190 (C4 (Web Text))

Max Activating Token Index: 1022

Text #7

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.7193. Min Act: -0.0000

Data Index: 452282 (C4 (Web Text))

Max Activating Token Index: 829

Text #8

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.6930. Min Act: -0.0000

Data Index: 1010996 (C4 (Web Text))

Max Activating Token Index: 966

Text #9

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.6872. Min Act: -0.0000

Data Index: 262137 (C4 (Web Text))

Max Activating Token Index: 622

Text #10

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.6568. Min Act: -0.0000

Data Index: 984360 (C4 (Web Text))

Max Activating Token Index: 389

Text #11

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.6435. Min Act: -0.0000

Data Index: 879524 (C4 (Web Text))

Max Activating Token Index: 372

Text #12

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.6377. Min Act: -0.0000

Data Index: 901389 (C4 (Web Text))

Max Activating Token Index: 877

Text #13

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.6330. Min Act: -0.0000

Data Index: 311786 (C4 (Web Text))

Max Activating Token Index: 870

Text #14

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.6670. Min Act: -0.0000

Data Index: 42698 (C4 (Web Text))

Max Activating Token Index: 588

Text #15

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.6758. Min Act: -0.0000

Data Index: 54311 (C4 (Web Text))

Max Activating Token Index: 708

Text #16

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.6324. Min Act: -0.0000

Data Index: 1290180 (C4 (Web Text))

Max Activating Token Index: 573

Text #17

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.6287. Min Act: -0.0000

Data Index: 110828 (C4 (Web Text))

Max Activating Token Index: 251

Text #18

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.6518. Min Act: -0.0000

Data Index: 1303127 (C4 (Web Text))

Max Activating Token Index: 77

Text #19

Max Range: 0.7747. Min Range: -0.7747

Max Act: 0.6419. Min Act: -0.0000

Data Index: 1147323 (C4 (Web Text))

Max Activating Token Index: 734