Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2133 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0901. Min Act: -0.0002

Data Index: 915701 (C4 (Web Text))

Max Activating Token Index: 778

Text #1

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0834. Min Act: -0.0002

Data Index: 611429 (C4 (Web Text))

Max Activating Token Index: 12

Text #2

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0689. Min Act: -0.0002

Data Index: 1057356 (C4 (Web Text))

Max Activating Token Index: 253

Text #3

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0813. Min Act: -0.0002

Data Index: 398845 (C4 (Web Text))

Max Activating Token Index: 487

Text #4

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0765. Min Act: -0.0002

Data Index: 798733 (C4 (Web Text))

Max Activating Token Index: 634

Text #5

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0817. Min Act: -0.0002

Data Index: 211301 (C4 (Web Text))

Max Activating Token Index: 189

Text #6

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0779. Min Act: -0.0002

Data Index: 1298416 (C4 (Web Text))

Max Activating Token Index: 908

Text #7

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0748. Min Act: -0.0002

Data Index: 617017 (C4 (Web Text))

Max Activating Token Index: 303

Text #8

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0780. Min Act: -0.0002

Data Index: 600991 (C4 (Web Text))

Max Activating Token Index: 439

Text #9

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0854. Min Act: -0.0002

Data Index: 906393 (C4 (Web Text))

Max Activating Token Index: 734

Text #10

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0805. Min Act: -0.0002

Data Index: 533972 (C4 (Web Text))

Max Activating Token Index: 832

Text #11

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0760. Min Act: -0.0002

Data Index: 1302419 (C4 (Web Text))

Max Activating Token Index: 384

Text #12

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0785. Min Act: -0.0002

Data Index: 1204403 (C4 (Web Text))

Max Activating Token Index: 847

Text #13

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0778. Min Act: -0.0002

Data Index: 551526 (C4 (Web Text))

Max Activating Token Index: 511

Text #14

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0780. Min Act: -0.0002

Data Index: 674475 (C4 (Web Text))

Max Activating Token Index: 925

Text #15

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0704. Min Act: -0.0002

Data Index: 442986 (C4 (Web Text))

Max Activating Token Index: 731

Text #16

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0748. Min Act: -0.0002

Data Index: 468785 (C4 (Web Text))

Max Activating Token Index: 860

Text #17

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0714. Min Act: -0.0002

Data Index: 92557 (C4 (Web Text))

Max Activating Token Index: 614

Text #18

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0716. Min Act: -0.0002

Data Index: 396774 (C4 (Web Text))

Max Activating Token Index: 717

Text #19

Max Range: 0.0901. Min Range: -0.0901

Max Act: 0.0792. Min Act: -0.0002

Data Index: 415965 (C4 (Web Text))

Max Activating Token Index: 134