Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5620 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.4884. Min Act: -0.0000

Data Index: 618154 (C4 (Web Text))

Max Activating Token Index: 717

Text #1

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.4861. Min Act: -0.0000

Data Index: 1137169 (C4 (Web Text))

Max Activating Token Index: 121

Text #2

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.4008. Min Act: -0.0000

Data Index: 394553 (C4 (Web Text))

Max Activating Token Index: 541

Text #3

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3916. Min Act: -0.0000

Data Index: 554054 (C4 (Web Text))

Max Activating Token Index: 118

Text #4

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3402. Min Act: -0.0000

Data Index: 143087 (C4 (Web Text))

Max Activating Token Index: 689

Text #5

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3334. Min Act: -0.0000

Data Index: 325870 (C4 (Web Text))

Max Activating Token Index: 298

Text #6

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3712. Min Act: -0.0000

Data Index: 4159 (C4 (Web Text))

Max Activating Token Index: 991

Text #7

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3782. Min Act: -0.0000

Data Index: 1337797 (C4 (Web Text))

Max Activating Token Index: 721

Text #8

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3551. Min Act: -0.0000

Data Index: 686489 (C4 (Web Text))

Max Activating Token Index: 564

Text #9

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3383. Min Act: -0.0000

Data Index: 62361 (C4 (Web Text))

Max Activating Token Index: 403

Text #10

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3755. Min Act: -0.0000

Data Index: 1072894 (C4 (Web Text))

Max Activating Token Index: 803

Text #11

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3606. Min Act: -0.0000

Data Index: 1117202 (C4 (Web Text))

Max Activating Token Index: 765

Text #12

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3390. Min Act: -0.0000

Data Index: 360841 (C4 (Web Text))

Max Activating Token Index: 698

Text #13

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3567. Min Act: -0.0000

Data Index: 1296439 (C4 (Web Text))

Max Activating Token Index: 898

Text #14

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3542. Min Act: -0.0000

Data Index: 237384 (C4 (Web Text))

Max Activating Token Index: 934

Text #15

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3448. Min Act: -0.0000

Data Index: 196160 (C4 (Web Text))

Max Activating Token Index: 920

Text #16

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3390. Min Act: -0.0000

Data Index: 475625 (C4 (Web Text))

Max Activating Token Index: 595

Text #17

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3121. Min Act: -0.0000

Data Index: 1047211 (C4 (Web Text))

Max Activating Token Index: 578

Text #18

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.3524. Min Act: -0.0000

Data Index: 1005581 (C4 (Web Text))

Max Activating Token Index: 92

Text #19

Max Range: 0.4884. Min Range: -0.4884

Max Act: 0.2861. Min Act: -0.0000

Data Index: 1004274 (C4 (Web Text))

Max Activating Token Index: 766