Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2280 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.8906. Min Act: -0.0003

Data Index: 49339 (C4 (Web Text))

Max Activating Token Index: 666

Text #1

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.7960. Min Act: -0.0003

Data Index: 295695 (C4 (Web Text))

Max Activating Token Index: 869

Text #2

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.7846. Min Act: -0.0003

Data Index: 341155 (C4 (Web Text))

Max Activating Token Index: 59

Text #3

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.8011. Min Act: -0.0003

Data Index: 702037 (C4 (Web Text))

Max Activating Token Index: 389

Text #4

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.7752. Min Act: -0.0003

Data Index: 1294654 (C4 (Web Text))

Max Activating Token Index: 943

Text #5

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.7568. Min Act: -0.0003

Data Index: 590792 (C4 (Web Text))

Max Activating Token Index: 890

Text #6

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.7590. Min Act: -0.0003

Data Index: 864860 (C4 (Web Text))

Max Activating Token Index: 913

Text #7

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.7162. Min Act: -0.0003

Data Index: 1146843 (C4 (Web Text))

Max Activating Token Index: 156

Text #8

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.6716. Min Act: -0.0003

Data Index: 280228 (C4 (Web Text))

Max Activating Token Index: 602

Text #9

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.7547. Min Act: -0.0003

Data Index: 162859 (C4 (Web Text))

Max Activating Token Index: 29

Text #10

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.7245. Min Act: -0.0003

Data Index: 691670 (C4 (Web Text))

Max Activating Token Index: 856

Text #11

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.7451. Min Act: -0.0003

Data Index: 185446 (C4 (Web Text))

Max Activating Token Index: 529

Text #12

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.6895. Min Act: -0.0003

Data Index: 1143459 (C4 (Web Text))

Max Activating Token Index: 667

Text #13

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.6665. Min Act: -0.0003

Data Index: 1330530 (C4 (Web Text))

Max Activating Token Index: 116

Text #14

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.7040. Min Act: -0.0003

Data Index: 710261 (C4 (Web Text))

Max Activating Token Index: 954

Text #15

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.6829. Min Act: -0.0003

Data Index: 30872 (C4 (Web Text))

Max Activating Token Index: 129

Text #16

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.7030. Min Act: -0.0003

Data Index: 844652 (C4 (Web Text))

Max Activating Token Index: 725

Text #17

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.6318. Min Act: -0.0003

Data Index: 933360 (C4 (Web Text))

Max Activating Token Index: 389

Text #18

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.6730. Min Act: -0.0003

Data Index: 793004 (C4 (Web Text))

Max Activating Token Index: 46

Text #19

Max Range: 0.8906. Min Range: -0.8906

Max Act: 0.6658. Min Act: -0.0003

Data Index: 962594 (C4 (Web Text))

Max Activating Token Index: 892