Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 644 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.2493. Min Range: -3.2493

Max Act: 3.2493. Min Act: -0.0001

Data Index: 19047 (C4 (Web Text))

Max Activating Token Index: 422

Text #1

Max Range: 3.2493. Min Range: -3.2493

Max Act: 2.5820. Min Act: -0.0001

Data Index: 705009 (C4 (Web Text))

Max Activating Token Index: 545

Text #2

Max Range: 3.2493. Min Range: -3.2493

Max Act: 2.3076. Min Act: -0.0001

Data Index: 465889 (C4 (Web Text))

Max Activating Token Index: 214

Text #3

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.7881. Min Act: -0.0001

Data Index: 76733 (C4 (Web Text))

Max Activating Token Index: 200

Text #4

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.8397. Min Act: -0.0001

Data Index: 464273 (C4 (Web Text))

Max Activating Token Index: 287

Text #5

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.7790. Min Act: -0.0001

Data Index: 1028748 (C4 (Web Text))

Max Activating Token Index: 636

Text #6

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.6940. Min Act: -0.0001

Data Index: 800481 (C4 (Web Text))

Max Activating Token Index: 992

Text #7

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.6406. Min Act: -0.0001

Data Index: 1039629 (C4 (Web Text))

Max Activating Token Index: 891

Text #8

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.4123. Min Act: -0.0001

Data Index: 534570 (C4 (Web Text))

Max Activating Token Index: 215

Text #9

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.5605. Min Act: -0.0001

Data Index: 26562 (C4 (Web Text))

Max Activating Token Index: 262

Text #10

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.6469. Min Act: -0.0001

Data Index: 986410 (C4 (Web Text))

Max Activating Token Index: 994

Text #11

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.4893. Min Act: -0.0001

Data Index: 811285 (C4 (Web Text))

Max Activating Token Index: 783

Text #12

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.6540. Min Act: -0.0001

Data Index: 28057 (C4 (Web Text))

Max Activating Token Index: 899

Text #13

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.4631. Min Act: -0.0001

Data Index: 389256 (C4 (Web Text))

Max Activating Token Index: 398

Text #14

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.4179. Min Act: -0.0001

Data Index: 694863 (C4 (Web Text))

Max Activating Token Index: 176

Text #15

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.4906. Min Act: -0.0001

Data Index: 430696 (C4 (Web Text))

Max Activating Token Index: 340

Text #16

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.3913. Min Act: -0.0001

Data Index: 1282639 (C4 (Web Text))

Max Activating Token Index: 295

Text #17

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.6544. Min Act: -0.0001

Data Index: 1251554 (C4 (Web Text))

Max Activating Token Index: 758

Text #18

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.5335. Min Act: -0.0001

Data Index: 907997 (C4 (Web Text))

Max Activating Token Index: 205

Text #19

Max Range: 3.2493. Min Range: -3.2493

Max Act: 1.4247. Min Act: -0.0001

Data Index: 1015280 (C4 (Web Text))

Max Activating Token Index: 969