Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3521 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.2762. Min Act: -0.0002

Data Index: 541404 (C4 (Web Text))

Max Activating Token Index: 434

Text #1

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.2619. Min Act: -0.0002

Data Index: 21095 (C4 (Web Text))

Max Activating Token Index: 971

Text #2

Max Range: 1.2762. Min Range: -1.2762

Max Act: 1.0909. Min Act: -0.0002

Data Index: 717458 (C4 (Web Text))

Max Activating Token Index: 762

Text #3

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9868. Min Act: -0.0002

Data Index: 200505 (C4 (Web Text))

Max Activating Token Index: 744

Text #4

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9649. Min Act: -0.0002

Data Index: 658834 (C4 (Web Text))

Max Activating Token Index: 794

Text #5

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9840. Min Act: -0.0002

Data Index: 345696 (C4 (Web Text))

Max Activating Token Index: 931

Text #6

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.9051. Min Act: -0.0002

Data Index: 257213 (C4 (Web Text))

Max Activating Token Index: 662

Text #7

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.8282. Min Act: -0.0002

Data Index: 523144 (C4 (Web Text))

Max Activating Token Index: 26

Text #8

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.7963. Min Act: -0.0002

Data Index: 232694 (C4 (Web Text))

Max Activating Token Index: 351

Text #9

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.8991. Min Act: -0.0002

Data Index: 417012 (C4 (Web Text))

Max Activating Token Index: 866

Text #10

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.8443. Min Act: -0.0002

Data Index: 227017 (C4 (Web Text))

Max Activating Token Index: 27

Text #11

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.8859. Min Act: -0.0002

Data Index: 371048 (C4 (Web Text))

Max Activating Token Index: 582

Text #12

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.8037. Min Act: -0.0002

Data Index: 253499 (C4 (Web Text))

Max Activating Token Index: 739

Text #13

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.8193. Min Act: -0.0002

Data Index: 687751 (C4 (Web Text))

Max Activating Token Index: 969

Text #14

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.7643. Min Act: -0.0002

Data Index: 1072213 (C4 (Web Text))

Max Activating Token Index: 189

Text #15

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.6595. Min Act: -0.0002

Data Index: 522372 (C4 (Web Text))

Max Activating Token Index: 790

Text #16

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.7217. Min Act: -0.0002

Data Index: 856515 (C4 (Web Text))

Max Activating Token Index: 787

Text #17

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.6614. Min Act: -0.0002

Data Index: 853345 (C4 (Web Text))

Max Activating Token Index: 739

Text #18

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.7070. Min Act: -0.0002

Data Index: 395520 (C4 (Web Text))

Max Activating Token Index: 110

Text #19

Max Range: 1.2762. Min Range: -1.2762

Max Act: 0.7087. Min Act: -0.0002

Data Index: 1247445 (C4 (Web Text))

Max Activating Token Index: 769