Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 117 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0949. Min Act: -0.0001

Data Index: 26603 (C4 (Web Text))

Max Activating Token Index: 362

Text #1

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0919. Min Act: -0.0001

Data Index: 1122893 (C4 (Web Text))

Max Activating Token Index: 370

Text #2

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0861. Min Act: -0.0001

Data Index: 1136849 (C4 (Web Text))

Max Activating Token Index: 651

Text #3

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0861. Min Act: -0.0001

Data Index: 823035 (C4 (Web Text))

Max Activating Token Index: 524

Text #4

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0866. Min Act: -0.0001

Data Index: 584016 (C4 (Web Text))

Max Activating Token Index: 400

Text #5

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0866. Min Act: -0.0001

Data Index: 310392 (C4 (Web Text))

Max Activating Token Index: 437

Text #6

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0832. Min Act: -0.0001

Data Index: 208400 (C4 (Web Text))

Max Activating Token Index: 116

Text #7

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0859. Min Act: -0.0001

Data Index: 634104 (C4 (Web Text))

Max Activating Token Index: 973

Text #8

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0832. Min Act: -0.0001

Data Index: 1160329 (C4 (Web Text))

Max Activating Token Index: 122

Text #9

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0799. Min Act: -0.0001

Data Index: 1279609 (C4 (Web Text))

Max Activating Token Index: 178

Text #10

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0729. Min Act: -0.0001

Data Index: 888324 (C4 (Web Text))

Max Activating Token Index: 693

Text #11

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0808. Min Act: -0.0001

Data Index: 309070 (C4 (Web Text))

Max Activating Token Index: 900

Text #12

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0809. Min Act: -0.0001

Data Index: 1274145 (C4 (Web Text))

Max Activating Token Index: 905

Text #13

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0810. Min Act: -0.0001

Data Index: 555209 (C4 (Web Text))

Max Activating Token Index: 348

Text #14

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0819. Min Act: -0.0001

Data Index: 517353 (C4 (Web Text))

Max Activating Token Index: 865

Text #15

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0796. Min Act: -0.0001

Data Index: 676547 (C4 (Web Text))

Max Activating Token Index: 14

Text #16

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0792. Min Act: -0.0001

Data Index: 288621 (C4 (Web Text))

Max Activating Token Index: 161

Text #17

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0790. Min Act: -0.0001

Data Index: 1085656 (C4 (Web Text))

Max Activating Token Index: 89

Text #18

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0767. Min Act: -0.0001

Data Index: 160416 (C4 (Web Text))

Max Activating Token Index: 870

Text #19

Max Range: 0.0949. Min Range: -0.0949

Max Act: 0.0793. Min Act: -0.0001

Data Index: 638079 (C4 (Web Text))

Max Activating Token Index: 660