Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4821 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.8998. Min Act: -0.0000

Data Index: 889998 (C4 (Web Text))

Max Activating Token Index: 234

Text #1

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.8528. Min Act: -0.0000

Data Index: 933681 (C4 (Web Text))

Max Activating Token Index: 168

Text #2

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.7863. Min Act: -0.0001

Data Index: 59718 (C4 (Web Text))

Max Activating Token Index: 351

Text #3

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.6808. Min Act: -0.0000

Data Index: 244313 (C4 (Web Text))

Max Activating Token Index: 303

Text #4

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.7111. Min Act: -0.0000

Data Index: 816655 (C4 (Web Text))

Max Activating Token Index: 167

Text #5

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.7487. Min Act: -0.0000

Data Index: 29042 (C4 (Web Text))

Max Activating Token Index: 839

Text #6

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.6858. Min Act: -0.0000

Data Index: 1142681 (C4 (Web Text))

Max Activating Token Index: 327

Text #7

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.6225. Min Act: -0.0000

Data Index: 635042 (C4 (Web Text))

Max Activating Token Index: 993

Text #8

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.6449. Min Act: -0.0000

Data Index: 580523 (C4 (Web Text))

Max Activating Token Index: 523

Text #9

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.6684. Min Act: -0.0000

Data Index: 657067 (C4 (Web Text))

Max Activating Token Index: 817

Text #10

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.7028. Min Act: -0.0000

Data Index: 1321036 (C4 (Web Text))

Max Activating Token Index: 388

Text #11

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.6025. Min Act: -0.0000

Data Index: 447759 (C4 (Web Text))

Max Activating Token Index: 838

Text #12

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.6513. Min Act: -0.0000

Data Index: 119138 (C4 (Web Text))

Max Activating Token Index: 386

Text #13

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.6748. Min Act: -0.0000

Data Index: 621364 (C4 (Web Text))

Max Activating Token Index: 152

Text #14

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.5863. Min Act: -0.0000

Data Index: 1107325 (C4 (Web Text))

Max Activating Token Index: 987

Text #15

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.5714. Min Act: -0.0000

Data Index: 657262 (C4 (Web Text))

Max Activating Token Index: 951

Text #16

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.5479. Min Act: -0.0000

Data Index: 1244100 (C4 (Web Text))

Max Activating Token Index: 568

Text #17

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.4766. Min Act: -0.0000

Data Index: 453853 (C4 (Web Text))

Max Activating Token Index: 665

Text #18

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.5010. Min Act: -0.0000

Data Index: 1247216 (C4 (Web Text))

Max Activating Token Index: 70

Text #19

Max Range: 0.8998. Min Range: -0.8998

Max Act: 0.4887. Min Act: -0.0000

Data Index: 36476 (C4 (Web Text))

Max Activating Token Index: 462