Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3651 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6993. Min Act: -0.0000

Data Index: 717617 (C4 (Web Text))

Max Activating Token Index: 753

Text #1

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.7374. Min Act: -0.0000

Data Index: 929629 (C4 (Web Text))

Max Activating Token Index: 914

Text #2

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6931. Min Act: -0.0000

Data Index: 504074 (C4 (Web Text))

Max Activating Token Index: 691

Text #3

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.7606. Min Act: -0.0000

Data Index: 207871 (C4 (Web Text))

Max Activating Token Index: 772

Text #4

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6984. Min Act: -0.0000

Data Index: 1075505 (C4 (Web Text))

Max Activating Token Index: 774

Text #5

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.7110. Min Act: -0.0000

Data Index: 495846 (C4 (Web Text))

Max Activating Token Index: 912

Text #6

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.7135. Min Act: -0.0000

Data Index: 92392 (C4 (Web Text))

Max Activating Token Index: 777

Text #7

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6928. Min Act: -0.0000

Data Index: 661203 (C4 (Web Text))

Max Activating Token Index: 874

Text #8

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6752. Min Act: -0.0000

Data Index: 1221490 (C4 (Web Text))

Max Activating Token Index: 573

Text #9

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6940. Min Act: -0.0000

Data Index: 277443 (C4 (Web Text))

Max Activating Token Index: 882

Text #10

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.7477. Min Act: -0.0000

Data Index: 369815 (C4 (Web Text))

Max Activating Token Index: 998

Text #11

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.7066. Min Act: -0.0000

Data Index: 604469 (C4 (Web Text))

Max Activating Token Index: 603

Text #12

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6487. Min Act: -0.0000

Data Index: 304003 (C4 (Web Text))

Max Activating Token Index: 851

Text #13

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6242. Min Act: -0.0000

Data Index: 1023933 (C4 (Web Text))

Max Activating Token Index: 695

Text #14

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6516. Min Act: -0.0000

Data Index: 1092008 (C4 (Web Text))

Max Activating Token Index: 767

Text #15

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6479. Min Act: -0.0000

Data Index: 1218461 (C4 (Web Text))

Max Activating Token Index: 896

Text #16

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6678. Min Act: -0.0000

Data Index: 1022828 (C4 (Web Text))

Max Activating Token Index: 454

Text #17

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6210. Min Act: -0.0000

Data Index: 1127156 (C4 (Web Text))

Max Activating Token Index: 606

Text #18

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6804. Min Act: -0.0000

Data Index: 525499 (C4 (Web Text))

Max Activating Token Index: 831

Text #19

Max Range: 0.7606. Min Range: -0.7606

Max Act: 0.6216. Min Act: -0.0000

Data Index: 1164433 (C4 (Web Text))

Max Activating Token Index: 1010