Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4770 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.8042. Min Act: -0.0001

Data Index: 1251651 (C4 (Web Text))

Max Activating Token Index: 796

Text #1

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7497. Min Act: -0.0001

Data Index: 763947 (C4 (Web Text))

Max Activating Token Index: 201

Text #2

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.8101. Min Act: -0.0000

Data Index: 1287056 (C4 (Web Text))

Max Activating Token Index: 137

Text #3

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7922. Min Act: -0.0001

Data Index: 1250287 (C4 (Web Text))

Max Activating Token Index: 658

Text #4

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7642. Min Act: -0.0001

Data Index: 866879 (C4 (Web Text))

Max Activating Token Index: 890

Text #5

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7261. Min Act: -0.0000

Data Index: 240816 (C4 (Web Text))

Max Activating Token Index: 179

Text #6

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7641. Min Act: -0.0001

Data Index: 915572 (C4 (Web Text))

Max Activating Token Index: 846

Text #7

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7533. Min Act: -0.0000

Data Index: 247896 (C4 (Web Text))

Max Activating Token Index: 69

Text #8

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7917. Min Act: -0.0001

Data Index: 1308113 (C4 (Web Text))

Max Activating Token Index: 239

Text #9

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.6987. Min Act: -0.0001

Data Index: 532533 (C4 (Web Text))

Max Activating Token Index: 678

Text #10

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7177. Min Act: -0.0000

Data Index: 1013289 (C4 (Web Text))

Max Activating Token Index: 469

Text #11

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7200. Min Act: -0.0001

Data Index: 976335 (C4 (Web Text))

Max Activating Token Index: 411

Text #12

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7170. Min Act: -0.0001

Data Index: 611940 (C4 (Web Text))

Max Activating Token Index: 921

Text #13

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7266. Min Act: -0.0001

Data Index: 757601 (C4 (Web Text))

Max Activating Token Index: 940

Text #14

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7085. Min Act: -0.0001

Data Index: 946252 (C4 (Web Text))

Max Activating Token Index: 469

Text #15

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7112. Min Act: -0.0001

Data Index: 567730 (C4 (Web Text))

Max Activating Token Index: 277

Text #16

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7780. Min Act: -0.0001

Data Index: 617717 (C4 (Web Text))

Max Activating Token Index: 644

Text #17

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7303. Min Act: -0.0000

Data Index: 827390 (C4 (Web Text))

Max Activating Token Index: 116

Text #18

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7282. Min Act: -0.0001

Data Index: 988058 (C4 (Web Text))

Max Activating Token Index: 208

Text #19

Max Range: 0.8101. Min Range: -0.8101

Max Act: 0.7551. Min Act: -0.0000

Data Index: 762725 (C4 (Web Text))

Max Activating Token Index: 155