Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1677 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.6563. Min Act: -0.0001

Data Index: 1030458 (C4 (Web Text))

Max Activating Token Index: 759

Text #1

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.2878. Min Act: -0.0001

Data Index: 974747 (C4 (Web Text))

Max Activating Token Index: 619

Text #2

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.3218. Min Act: -0.0001

Data Index: 114966 (C4 (Web Text))

Max Activating Token Index: 522

Text #3

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.2235. Min Act: -0.0001

Data Index: 536203 (C4 (Web Text))

Max Activating Token Index: 898

Text #4

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.3045. Min Act: -0.0001

Data Index: 699883 (C4 (Web Text))

Max Activating Token Index: 428

Text #5

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.2655. Min Act: -0.0001

Data Index: 611938 (C4 (Web Text))

Max Activating Token Index: 1019

Text #6

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.3015. Min Act: -0.0001

Data Index: 483243 (C4 (Web Text))

Max Activating Token Index: 74

Text #7

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.2186. Min Act: -0.0001

Data Index: 664644 (C4 (Web Text))

Max Activating Token Index: 659

Text #8

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.2811. Min Act: -0.0001

Data Index: 357944 (C4 (Web Text))

Max Activating Token Index: 1009

Text #9

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.3137. Min Act: -0.0001

Data Index: 415400 (C4 (Web Text))

Max Activating Token Index: 20

Text #10

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.2714. Min Act: -0.0001

Data Index: 135121 (C4 (Web Text))

Max Activating Token Index: 516

Text #11

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.3286. Min Act: -0.0001

Data Index: 518173 (C4 (Web Text))

Max Activating Token Index: 932

Text #12

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.1600. Min Act: -0.0001

Data Index: 85810 (C4 (Web Text))

Max Activating Token Index: 302

Text #13

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.2404. Min Act: -0.0001

Data Index: 623480 (C4 (Web Text))

Max Activating Token Index: 933

Text #14

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.2145. Min Act: -0.0001

Data Index: 40342 (C4 (Web Text))

Max Activating Token Index: 570

Text #15

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.3355. Min Act: -0.0001

Data Index: 825646 (C4 (Web Text))

Max Activating Token Index: 823

Text #16

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.2025. Min Act: -0.0001

Data Index: 500334 (C4 (Web Text))

Max Activating Token Index: 753

Text #17

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.1665. Min Act: -0.0001

Data Index: 1232024 (C4 (Web Text))

Max Activating Token Index: 317

Text #18

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.2000. Min Act: -0.0001

Data Index: 470441 (C4 (Web Text))

Max Activating Token Index: 420

Text #19

Max Range: 1.6563. Min Range: -1.6563

Max Act: 1.1937. Min Act: -0.0001

Data Index: 428837 (C4 (Web Text))

Max Activating Token Index: 323