Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1663 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1024. Min Range: -2.1024

Max Act: 2.1024. Min Act: -0.0001

Data Index: 604630 (C4 (Web Text))

Max Activating Token Index: 289

Text #1

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.9236. Min Act: -0.0001

Data Index: 86035 (C4 (Web Text))

Max Activating Token Index: 39

Text #2

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.8766. Min Act: -0.0001

Data Index: 1002063 (C4 (Web Text))

Max Activating Token Index: 26

Text #3

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.7434. Min Act: -0.0001

Data Index: 375882 (C4 (Web Text))

Max Activating Token Index: 59

Text #4

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.6162. Min Act: -0.0001

Data Index: 207789 (C4 (Web Text))

Max Activating Token Index: 202

Text #5

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.5704. Min Act: -0.0001

Data Index: 690886 (C4 (Web Text))

Max Activating Token Index: 218

Text #6

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.5347. Min Act: -0.0001

Data Index: 1086488 (C4 (Web Text))

Max Activating Token Index: 992

Text #7

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.5199. Min Act: -0.0001

Data Index: 472110 (C4 (Web Text))

Max Activating Token Index: 495

Text #8

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.5170. Min Act: -0.0001

Data Index: 953901 (C4 (Web Text))

Max Activating Token Index: 42

Text #9

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.2568. Min Act: -0.0001

Data Index: 895241 (C4 (Web Text))

Max Activating Token Index: 695

Text #10

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.5643. Min Act: -0.0001

Data Index: 194396 (C4 (Web Text))

Max Activating Token Index: 264

Text #11

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.3774. Min Act: -0.0001

Data Index: 126039 (C4 (Web Text))

Max Activating Token Index: 266

Text #12

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.4391. Min Act: -0.0001

Data Index: 682076 (C4 (Web Text))

Max Activating Token Index: 243

Text #13

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.5020. Min Act: -0.0001

Data Index: 897265 (C4 (Web Text))

Max Activating Token Index: 840

Text #14

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.3243. Min Act: -0.0001

Data Index: 300093 (C4 (Web Text))

Max Activating Token Index: 168

Text #15

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.0609. Min Act: -0.0001

Data Index: 320390 (C4 (Web Text))

Max Activating Token Index: 213

Text #16

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.3152. Min Act: -0.0001

Data Index: 1309892 (C4 (Web Text))

Max Activating Token Index: 226

Text #17

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.4469. Min Act: -0.0001

Data Index: 283663 (C4 (Web Text))

Max Activating Token Index: 17

Text #18

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.3287. Min Act: -0.0001

Data Index: 891914 (C4 (Web Text))

Max Activating Token Index: 14

Text #19

Max Range: 2.1024. Min Range: -2.1024

Max Act: 1.2918. Min Act: -0.0001

Data Index: 164854 (C4 (Web Text))

Max Activating Token Index: 930