Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3777 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.3326. Min Act: -0.0000

Data Index: 528187 (C4 (Web Text))

Max Activating Token Index: 959

Text #1

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.3585. Min Act: -0.0000

Data Index: 55281 (C4 (Web Text))

Max Activating Token Index: 952

Text #2

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.3118. Min Act: -0.0000

Data Index: 906705 (C4 (Web Text))

Max Activating Token Index: 383

Text #3

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2777. Min Act: -0.0000

Data Index: 484817 (C4 (Web Text))

Max Activating Token Index: 429

Text #4

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2734. Min Act: -0.0000

Data Index: 149281 (C4 (Web Text))

Max Activating Token Index: 666

Text #5

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2935. Min Act: -0.0000

Data Index: 1303601 (C4 (Web Text))

Max Activating Token Index: 1021

Text #6

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2668. Min Act: -0.0000

Data Index: 938759 (C4 (Web Text))

Max Activating Token Index: 815

Text #7

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2725. Min Act: -0.0000

Data Index: 321100 (C4 (Web Text))

Max Activating Token Index: 363

Text #8

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2634. Min Act: -0.0000

Data Index: 810258 (C4 (Web Text))

Max Activating Token Index: 968

Text #9

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2710. Min Act: -0.0000

Data Index: 465695 (C4 (Web Text))

Max Activating Token Index: 347

Text #10

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2670. Min Act: -0.0000

Data Index: 843578 (C4 (Web Text))

Max Activating Token Index: 877

Text #11

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2444. Min Act: -0.0000

Data Index: 1294087 (C4 (Web Text))

Max Activating Token Index: 676

Text #12

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2572. Min Act: -0.0000

Data Index: 1353827 (C4 (Web Text))

Max Activating Token Index: 346

Text #13

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2495. Min Act: -0.0000

Data Index: 720352 (C4 (Web Text))

Max Activating Token Index: 416

Text #14

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2561. Min Act: -0.0000

Data Index: 199936 (C4 (Web Text))

Max Activating Token Index: 794

Text #15

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2375. Min Act: -0.0000

Data Index: 730736 (C4 (Web Text))

Max Activating Token Index: 318

Text #16

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2527. Min Act: -0.0000

Data Index: 1284842 (C4 (Web Text))

Max Activating Token Index: 159

Text #17

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2523. Min Act: -0.0000

Data Index: 1039642 (C4 (Web Text))

Max Activating Token Index: 346

Text #18

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2449. Min Act: -0.0000

Data Index: 733090 (C4 (Web Text))

Max Activating Token Index: 339

Text #19

Max Range: 0.3585. Min Range: -0.3585

Max Act: 0.2346. Min Act: -0.0000

Data Index: 1061872 (C4 (Web Text))

Max Activating Token Index: 689