Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4125 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.7102. Min Range: -3.7102

Max Act: 3.7102. Min Act: -0.0001

Data Index: 1034826 (C4 (Web Text))

Max Activating Token Index: 40

Text #1

Max Range: 3.7102. Min Range: -3.7102

Max Act: 3.2948. Min Act: -0.0001

Data Index: 883493 (C4 (Web Text))

Max Activating Token Index: 978

Text #2

Max Range: 3.7102. Min Range: -3.7102

Max Act: 3.1757. Min Act: -0.0001

Data Index: 427520 (C4 (Web Text))

Max Activating Token Index: 893

Text #3

Max Range: 3.7102. Min Range: -3.7102

Max Act: 3.4329. Min Act: -0.0001

Data Index: 1161473 (C4 (Web Text))

Max Activating Token Index: 370

Text #4

Max Range: 3.7102. Min Range: -3.7102

Max Act: 3.2056. Min Act: -0.0001

Data Index: 95051 (C4 (Web Text))

Max Activating Token Index: 376

Text #5

Max Range: 3.7102. Min Range: -3.7102

Max Act: 3.1896. Min Act: -0.0001

Data Index: 593905 (C4 (Web Text))

Max Activating Token Index: 783

Text #6

Max Range: 3.7102. Min Range: -3.7102

Max Act: 3.0587. Min Act: -0.0001

Data Index: 1096096 (C4 (Web Text))

Max Activating Token Index: 919

Text #7

Max Range: 3.7102. Min Range: -3.7102

Max Act: 2.8789. Min Act: -0.0001

Data Index: 469761 (C4 (Web Text))

Max Activating Token Index: 681

Text #8

Max Range: 3.7102. Min Range: -3.7102

Max Act: 2.9463. Min Act: -0.0001

Data Index: 1348942 (C4 (Web Text))

Max Activating Token Index: 399

Text #9

Max Range: 3.7102. Min Range: -3.7102

Max Act: 2.8157. Min Act: -0.0001

Data Index: 825564 (C4 (Web Text))

Max Activating Token Index: 186

Text #10

Max Range: 3.7102. Min Range: -3.7102

Max Act: 3.1353. Min Act: -0.0001

Data Index: 1164225 (C4 (Web Text))

Max Activating Token Index: 163

Text #11

Max Range: 3.7102. Min Range: -3.7102

Max Act: 2.7471. Min Act: -0.0001

Data Index: 189311 (C4 (Web Text))

Max Activating Token Index: 751

Text #12

Max Range: 3.7102. Min Range: -3.7102

Max Act: 2.7705. Min Act: -0.0001

Data Index: 299420 (C4 (Web Text))

Max Activating Token Index: 339

Text #13

Max Range: 3.7102. Min Range: -3.7102

Max Act: 2.6306. Min Act: -0.0001

Data Index: 54515 (C4 (Web Text))

Max Activating Token Index: 967

Text #14

Max Range: 3.7102. Min Range: -3.7102

Max Act: 2.6075. Min Act: -0.0001

Data Index: 1252283 (C4 (Web Text))

Max Activating Token Index: 755

Text #15

Max Range: 3.7102. Min Range: -3.7102

Max Act: 2.5972. Min Act: -0.0001

Data Index: 1293483 (C4 (Web Text))

Max Activating Token Index: 501

Text #16

Max Range: 3.7102. Min Range: -3.7102

Max Act: 3.0189. Min Act: -0.0001

Data Index: 1077726 (C4 (Web Text))

Max Activating Token Index: 695

Text #17

Max Range: 3.7102. Min Range: -3.7102

Max Act: 2.5919. Min Act: -0.0001

Data Index: 1321807 (C4 (Web Text))

Max Activating Token Index: 286

Text #18

Max Range: 3.7102. Min Range: -3.7102

Max Act: 2.4363. Min Act: -0.0001

Data Index: 76463 (C4 (Web Text))

Max Activating Token Index: 711

Text #19

Max Range: 3.7102. Min Range: -3.7102

Max Act: 2.8037. Min Act: -0.0001

Data Index: 65600 (C4 (Web Text))

Max Activating Token Index: 946