Model: SoLU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1569 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-2l')

Text #0

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.5590. Min Act: -0.0003

Data Index: 39131 (C4 (Web Text))

Max Activating Token Index: 859

Text #1

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3978. Min Act: -0.0003

Data Index: 445899 (C4 (Web Text))

Max Activating Token Index: 623

Text #2

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3876. Min Act: -0.0003

Data Index: 1114966 (C4 (Web Text))

Max Activating Token Index: 1022

Text #3

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3544. Min Act: -0.0003

Data Index: 508622 (C4 (Web Text))

Max Activating Token Index: 968

Text #4

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3434. Min Act: -0.0003

Data Index: 475514 (C4 (Web Text))

Max Activating Token Index: 169

Text #5

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3439. Min Act: -0.0003

Data Index: 393817 (C4 (Web Text))

Max Activating Token Index: 843

Text #6

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3335. Min Act: -0.0003

Data Index: 550086 (C4 (Web Text))

Max Activating Token Index: 269

Text #7

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3284. Min Act: -0.0003

Data Index: 611505 (C4 (Web Text))

Max Activating Token Index: 122

Text #8

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3149. Min Act: -0.0003

Data Index: 427839 (C4 (Web Text))

Max Activating Token Index: 412

Text #9

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3242. Min Act: -0.0003

Data Index: 884949 (C4 (Web Text))

Max Activating Token Index: 603

Text #10

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3046. Min Act: -0.0003

Data Index: 1015278 (C4 (Web Text))

Max Activating Token Index: 627

Text #11

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.2968. Min Act: -0.0003

Data Index: 734529 (C4 (Web Text))

Max Activating Token Index: 357

Text #12

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3027. Min Act: -0.0003

Data Index: 461082 (C4 (Web Text))

Max Activating Token Index: 409

Text #13

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3045. Min Act: -0.0003

Data Index: 551273 (C4 (Web Text))

Max Activating Token Index: 642

Text #14

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3016. Min Act: -0.0003

Data Index: 103178 (C4 (Web Text))

Max Activating Token Index: 172

Text #15

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.3089. Min Act: -0.0003

Data Index: 677595 (C4 (Web Text))

Max Activating Token Index: 557

Text #16

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.2997. Min Act: -0.0003

Data Index: 586699 (C4 (Web Text))

Max Activating Token Index: 620

Text #17

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.2945. Min Act: -0.0003

Data Index: 19728 (C4 (Web Text))

Max Activating Token Index: 501

Text #18

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.2935. Min Act: -0.0004

Data Index: 956555 (C4 (Web Text))

Max Activating Token Index: 351

Text #19

Max Range: 0.5590. Min Range: -0.5590

Max Act: 0.2837. Min Act: -0.0003

Data Index: 269432 (C4 (Web Text))

Max Activating Token Index: 223