Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2564 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.9581. Min Act: -0.0000

Data Index: 994332 (C4 (Web Text))

Max Activating Token Index: 397

Text #1

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.5724. Min Act: -0.0000

Data Index: 1242494 (C4 (Web Text))

Max Activating Token Index: 659

Text #2

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.5348. Min Act: -0.0000

Data Index: 1165995 (C4 (Web Text))

Max Activating Token Index: 282

Text #3

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.4242. Min Act: -0.0000

Data Index: 479659 (C4 (Web Text))

Max Activating Token Index: 755

Text #4

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.3832. Min Act: -0.0000

Data Index: 411784 (C4 (Web Text))

Max Activating Token Index: 844

Text #5

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.3471. Min Act: -0.0000

Data Index: 177288 (C4 (Web Text))

Max Activating Token Index: 653

Text #6

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2795. Min Act: -0.0000

Data Index: 60329 (C4 (Web Text))

Max Activating Token Index: 878

Text #7

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2684. Min Act: -0.0000

Data Index: 1110965 (C4 (Web Text))

Max Activating Token Index: 531

Text #8

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2901. Min Act: -0.0000

Data Index: 158445 (C4 (Web Text))

Max Activating Token Index: 137

Text #9

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2929. Min Act: -0.0000

Data Index: 920938 (C4 (Web Text))

Max Activating Token Index: 640

Text #10

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2715. Min Act: -0.0000

Data Index: 1342410 (C4 (Web Text))

Max Activating Token Index: 28

Text #11

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2691. Min Act: -0.0000

Data Index: 887579 (C4 (Web Text))

Max Activating Token Index: 752

Text #12

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2629. Min Act: -0.0000

Data Index: 26996 (C4 (Web Text))

Max Activating Token Index: 595

Text #13

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2774. Min Act: -0.0000

Data Index: 394435 (C4 (Web Text))

Max Activating Token Index: 953

Text #14

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2785. Min Act: -0.0000

Data Index: 992627 (C4 (Web Text))

Max Activating Token Index: 955

Text #15

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2801. Min Act: -0.0000

Data Index: 442844 (C4 (Web Text))

Max Activating Token Index: 126

Text #16

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2453. Min Act: -0.0000

Data Index: 13692 (C4 (Web Text))

Max Activating Token Index: 127

Text #17

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2661. Min Act: -0.0000

Data Index: 810745 (C4 (Web Text))

Max Activating Token Index: 971

Text #18

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2655. Min Act: -0.0000

Data Index: 1129874 (C4 (Web Text))

Max Activating Token Index: 835

Text #19

Max Range: 0.9581. Min Range: -0.9581

Max Act: 0.2309. Min Act: -0.0000

Data Index: 1281158 (C4 (Web Text))

Max Activating Token Index: 849