Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4184 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.7712. Min Act: -0.0000

Data Index: 977492 (C4 (Web Text))

Max Activating Token Index: 749

Text #1

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.7242. Min Act: -0.0000

Data Index: 1030970 (C4 (Web Text))

Max Activating Token Index: 81

Text #2

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.6593. Min Act: -0.0000

Data Index: 195025 (C4 (Web Text))

Max Activating Token Index: 62

Text #3

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.5908. Min Act: -0.0000

Data Index: 565760 (C4 (Web Text))

Max Activating Token Index: 466

Text #4

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.5775. Min Act: -0.0000

Data Index: 330346 (C4 (Web Text))

Max Activating Token Index: 66

Text #5

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.5205. Min Act: -0.0000

Data Index: 448911 (C4 (Web Text))

Max Activating Token Index: 656

Text #6

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.5443. Min Act: -0.0000

Data Index: 796014 (C4 (Web Text))

Max Activating Token Index: 335

Text #7

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.5569. Min Act: -0.0000

Data Index: 696901 (C4 (Web Text))

Max Activating Token Index: 204

Text #8

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.5360. Min Act: -0.0000

Data Index: 1172202 (C4 (Web Text))

Max Activating Token Index: 369

Text #9

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.4652. Min Act: -0.0000

Data Index: 742319 (C4 (Web Text))

Max Activating Token Index: 877

Text #10

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.5248. Min Act: -0.0000

Data Index: 729755 (C4 (Web Text))

Max Activating Token Index: 567

Text #11

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.5201. Min Act: -0.0000

Data Index: 315310 (C4 (Web Text))

Max Activating Token Index: 893

Text #12

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.5031. Min Act: -0.0000

Data Index: 1152491 (C4 (Web Text))

Max Activating Token Index: 889

Text #13

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.4908. Min Act: -0.0000

Data Index: 76481 (C4 (Web Text))

Max Activating Token Index: 539

Text #14

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.4323. Min Act: -0.0000

Data Index: 643866 (C4 (Web Text))

Max Activating Token Index: 239

Text #15

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.4289. Min Act: -0.0000

Data Index: 1125167 (C4 (Web Text))

Max Activating Token Index: 349

Text #16

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.4725. Min Act: -0.0000

Data Index: 69798 (C4 (Web Text))

Max Activating Token Index: 672

Text #17

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.4630. Min Act: -0.0000

Data Index: 695911 (C4 (Web Text))

Max Activating Token Index: 570

Text #18

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.4609. Min Act: -0.0000

Data Index: 391105 (C4 (Web Text))

Max Activating Token Index: 669

Text #19

Max Range: 0.7712. Min Range: -0.7712

Max Act: 0.4578. Min Act: -0.0000

Data Index: 1359410 (C4 (Web Text))

Max Activating Token Index: 578