Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 895 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.0558. Min Range: -2.0558

Max Act: 2.0558. Min Act: -0.0000

Data Index: 911728 (C4 (Web Text))

Max Activating Token Index: 133

Text #1

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.7114. Min Act: -0.0000

Data Index: 293288 (C4 (Web Text))

Max Activating Token Index: 904

Text #2

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.2040. Min Act: -0.0000

Data Index: 1224382 (C4 (Web Text))

Max Activating Token Index: 259

Text #3

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.1590. Min Act: -0.0000

Data Index: 462512 (C4 (Web Text))

Max Activating Token Index: 282

Text #4

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.2262. Min Act: -0.0000

Data Index: 1043697 (C4 (Web Text))

Max Activating Token Index: 355

Text #5

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.2018. Min Act: -0.0000

Data Index: 135743 (C4 (Web Text))

Max Activating Token Index: 769

Text #6

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.1590. Min Act: -0.0000

Data Index: 378351 (C4 (Web Text))

Max Activating Token Index: 67

Text #7

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.1291. Min Act: -0.0000

Data Index: 1298017 (C4 (Web Text))

Max Activating Token Index: 131

Text #8

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.1275. Min Act: -0.0000

Data Index: 1225513 (C4 (Web Text))

Max Activating Token Index: 150

Text #9

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.1294. Min Act: -0.0000

Data Index: 540163 (C4 (Web Text))

Max Activating Token Index: 335

Text #10

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.0806. Min Act: -0.0000

Data Index: 633917 (C4 (Web Text))

Max Activating Token Index: 776

Text #11

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.1080. Min Act: -0.0000

Data Index: 454103 (C4 (Web Text))

Max Activating Token Index: 152

Text #12

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.0508. Min Act: -0.0000

Data Index: 372361 (C4 (Web Text))

Max Activating Token Index: 504

Text #13

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.0363. Min Act: -0.0000

Data Index: 833922 (C4 (Web Text))

Max Activating Token Index: 582

Text #14

Max Range: 2.0558. Min Range: -2.0558

Max Act: 0.9983. Min Act: -0.0000

Data Index: 1179148 (C4 (Web Text))

Max Activating Token Index: 196

Text #15

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.0896. Min Act: -0.0000

Data Index: 1214049 (C4 (Web Text))

Max Activating Token Index: 46

Text #16

Max Range: 2.0558. Min Range: -2.0558

Max Act: 0.9878. Min Act: -0.0000

Data Index: 178089 (C4 (Web Text))

Max Activating Token Index: 327

Text #17

Max Range: 2.0558. Min Range: -2.0558

Max Act: 0.9991. Min Act: -0.0000

Data Index: 461681 (C4 (Web Text))

Max Activating Token Index: 82

Text #18

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.0145. Min Act: -0.0000

Data Index: 1050265 (C4 (Web Text))

Max Activating Token Index: 44

Text #19

Max Range: 2.0558. Min Range: -2.0558

Max Act: 1.0219. Min Act: -0.0000

Data Index: 244892 (C4 (Web Text))

Max Activating Token Index: 715