Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2833 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.7354. Min Act: -0.0000

Data Index: 1119937 (C4 (Web Text))

Max Activating Token Index: 826

Text #1

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.7396. Min Act: -0.0000

Data Index: 83856 (C4 (Web Text))

Max Activating Token Index: 744

Text #2

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.6737. Min Act: -0.0000

Data Index: 1125879 (C4 (Web Text))

Max Activating Token Index: 959

Text #3

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.6667. Min Act: -0.0000

Data Index: 567205 (C4 (Web Text))

Max Activating Token Index: 362

Text #4

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.6669. Min Act: -0.0000

Data Index: 1290870 (C4 (Web Text))

Max Activating Token Index: 1002

Text #5

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.6550. Min Act: -0.0000

Data Index: 1339261 (C4 (Web Text))

Max Activating Token Index: 788

Text #6

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.5463. Min Act: -0.0000

Data Index: 837427 (C4 (Web Text))

Max Activating Token Index: 252

Text #7

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.6222. Min Act: -0.0000

Data Index: 98961 (C4 (Web Text))

Max Activating Token Index: 310

Text #8

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.6125. Min Act: -0.0000

Data Index: 1174464 (C4 (Web Text))

Max Activating Token Index: 910

Text #9

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.6240. Min Act: -0.0000

Data Index: 1033151 (C4 (Web Text))

Max Activating Token Index: 638

Text #10

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.5970. Min Act: -0.0000

Data Index: 1166160 (C4 (Web Text))

Max Activating Token Index: 886

Text #11

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.5562. Min Act: -0.0000

Data Index: 265370 (C4 (Web Text))

Max Activating Token Index: 549

Text #12

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.5937. Min Act: -0.0000

Data Index: 770677 (C4 (Web Text))

Max Activating Token Index: 6

Text #13

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.6053. Min Act: -0.0000

Data Index: 1145842 (C4 (Web Text))

Max Activating Token Index: 759

Text #14

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.6038. Min Act: -0.0000

Data Index: 536593 (C4 (Web Text))

Max Activating Token Index: 405

Text #15

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.6060. Min Act: -0.0000

Data Index: 1173648 (C4 (Web Text))

Max Activating Token Index: 776

Text #16

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.5830. Min Act: -0.0000

Data Index: 202184 (C4 (Web Text))

Max Activating Token Index: 782

Text #17

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.5896. Min Act: -0.0000

Data Index: 1145639 (C4 (Web Text))

Max Activating Token Index: 606

Text #18

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.5839. Min Act: -0.0000

Data Index: 1180747 (C4 (Web Text))

Max Activating Token Index: 571

Text #19

Max Range: 0.7396. Min Range: -0.7396

Max Act: 0.5776. Min Act: -0.0000

Data Index: 623903 (C4 (Web Text))

Max Activating Token Index: 129