Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1982 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.8218. Min Act: -0.0000

Data Index: 307907 (C4 (Web Text))

Max Activating Token Index: 845

Text #1

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.8211. Min Act: -0.0000

Data Index: 1122055 (C4 (Web Text))

Max Activating Token Index: 302

Text #2

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.8188. Min Act: -0.0000

Data Index: 688006 (C4 (Web Text))

Max Activating Token Index: 506

Text #3

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.4549. Min Act: -0.0000

Data Index: 153992 (C4 (Web Text))

Max Activating Token Index: 882

Text #4

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.5403. Min Act: -0.0000

Data Index: 167746 (C4 (Web Text))

Max Activating Token Index: 423

Text #5

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.3893. Min Act: -0.0000

Data Index: 1004694 (C4 (Web Text))

Max Activating Token Index: 304

Text #6

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.6527. Min Act: -0.0000

Data Index: 768076 (C4 (Web Text))

Max Activating Token Index: 66

Text #7

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.3755. Min Act: -0.0000

Data Index: 65362 (C4 (Web Text))

Max Activating Token Index: 564

Text #8

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.4909. Min Act: -0.0000

Data Index: 1136185 (C4 (Web Text))

Max Activating Token Index: 546

Text #9

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.2083. Min Act: -0.0000

Data Index: 124950 (C4 (Web Text))

Max Activating Token Index: 120

Text #10

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.2628. Min Act: -0.0000

Data Index: 1280055 (C4 (Web Text))

Max Activating Token Index: 849

Text #11

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.1498. Min Act: -0.0000

Data Index: 530626 (C4 (Web Text))

Max Activating Token Index: 621

Text #12

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.2589. Min Act: -0.0000

Data Index: 860811 (C4 (Web Text))

Max Activating Token Index: 859

Text #13

Max Range: 1.8218. Min Range: -1.8218

Max Act: 0.9026. Min Act: -0.0000

Data Index: 810221 (C4 (Web Text))

Max Activating Token Index: 799

Text #14

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.1113. Min Act: -0.0000

Data Index: 549648 (C4 (Web Text))

Max Activating Token Index: 273

Text #15

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.0964. Min Act: -0.0000

Data Index: 268914 (C4 (Web Text))

Max Activating Token Index: 274

Text #16

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.3098. Min Act: -0.0000

Data Index: 916946 (C4 (Web Text))

Max Activating Token Index: 945

Text #17

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.0241. Min Act: -0.0000

Data Index: 274517 (C4 (Web Text))

Max Activating Token Index: 44

Text #18

Max Range: 1.8218. Min Range: -1.8218

Max Act: 1.1183. Min Act: -0.0000

Data Index: 393078 (C4 (Web Text))

Max Activating Token Index: 816

Text #19

Max Range: 1.8218. Min Range: -1.8218

Max Act: 0.9786. Min Act: -0.0000

Data Index: 732541 (C4 (Web Text))

Max Activating Token Index: 664