Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2321 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.7543. Min Act: -0.0003

Data Index: 1270426 (C4 (Web Text))

Max Activating Token Index: 205

Text #1

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.7262. Min Act: -0.0003

Data Index: 113668 (C4 (Web Text))

Max Activating Token Index: 734

Text #2

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6720. Min Act: -0.0003

Data Index: 792205 (C4 (Web Text))

Max Activating Token Index: 522

Text #3

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.7500. Min Act: -0.0003

Data Index: 867900 (C4 (Web Text))

Max Activating Token Index: 164

Text #4

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.7015. Min Act: -0.0003

Data Index: 180103 (C4 (Web Text))

Max Activating Token Index: 720

Text #5

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6909. Min Act: -0.0003

Data Index: 882865 (C4 (Web Text))

Max Activating Token Index: 756

Text #6

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6607. Min Act: -0.0003

Data Index: 109308 (C4 (Web Text))

Max Activating Token Index: 426

Text #7

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6533. Min Act: -0.0003

Data Index: 78687 (C4 (Web Text))

Max Activating Token Index: 844

Text #8

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6543. Min Act: -0.0003

Data Index: 1036908 (C4 (Web Text))

Max Activating Token Index: 168

Text #9

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6963. Min Act: -0.0003

Data Index: 975873 (C4 (Web Text))

Max Activating Token Index: 115

Text #10

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6871. Min Act: -0.0003

Data Index: 658032 (C4 (Web Text))

Max Activating Token Index: 164

Text #11

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6466. Min Act: -0.0003

Data Index: 1311407 (C4 (Web Text))

Max Activating Token Index: 916

Text #12

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6276. Min Act: -0.0003

Data Index: 1340319 (C4 (Web Text))

Max Activating Token Index: 192

Text #13

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.5920. Min Act: -0.0003

Data Index: 399446 (C4 (Web Text))

Max Activating Token Index: 709

Text #14

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6679. Min Act: -0.0003

Data Index: 137412 (C4 (Web Text))

Max Activating Token Index: 547

Text #15

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6678. Min Act: -0.0003

Data Index: 1119787 (C4 (Web Text))

Max Activating Token Index: 920

Text #16

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6171. Min Act: -0.0003

Data Index: 940430 (C4 (Web Text))

Max Activating Token Index: 777

Text #17

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6616. Min Act: -0.0003

Data Index: 732337 (C4 (Web Text))

Max Activating Token Index: 682

Text #18

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6396. Min Act: -0.0003

Data Index: 85227 (C4 (Web Text))

Max Activating Token Index: 773

Text #19

Max Range: 0.7543. Min Range: -0.7543

Max Act: 0.6576. Min Act: -0.0003

Data Index: 390741 (C4 (Web Text))

Max Activating Token Index: 105