Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 714 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.1333. Min Act: -0.0001

Data Index: 878185 (C4 (Web Text))

Max Activating Token Index: 709

Text #1

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.1349. Min Act: -0.0000

Data Index: 531600 (C4 (Web Text))

Max Activating Token Index: 171

Text #2

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0845. Min Act: -0.0001

Data Index: 856820 (C4 (Web Text))

Max Activating Token Index: 847

Text #3

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0757. Min Act: -0.0001

Data Index: 502317 (C4 (Web Text))

Max Activating Token Index: 193

Text #4

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0910. Min Act: -0.0001

Data Index: 173068 (C4 (Web Text))

Max Activating Token Index: 982

Text #5

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0787. Min Act: -0.0001

Data Index: 356283 (C4 (Web Text))

Max Activating Token Index: 809

Text #6

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0805. Min Act: -0.0001

Data Index: 772650 (C4 (Web Text))

Max Activating Token Index: 346

Text #7

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0828. Min Act: -0.0001

Data Index: 843924 (C4 (Web Text))

Max Activating Token Index: 442

Text #8

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0733. Min Act: -0.0001

Data Index: 933771 (C4 (Web Text))

Max Activating Token Index: 798

Text #9

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0853. Min Act: -0.0000

Data Index: 70149 (C4 (Web Text))

Max Activating Token Index: 367

Text #10

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0815. Min Act: -0.0001

Data Index: 1037857 (C4 (Web Text))

Max Activating Token Index: 752

Text #11

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0791. Min Act: -0.0001

Data Index: 458169 (C4 (Web Text))

Max Activating Token Index: 383

Text #12

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0818. Min Act: -0.0001

Data Index: 530506 (C4 (Web Text))

Max Activating Token Index: 511

Text #13

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0782. Min Act: -0.0001

Data Index: 955727 (C4 (Web Text))

Max Activating Token Index: 158

Text #14

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0787. Min Act: -0.0001

Data Index: 1004172 (C4 (Web Text))

Max Activating Token Index: 427

Text #15

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0742. Min Act: -0.0001

Data Index: 1277350 (C4 (Web Text))

Max Activating Token Index: 120

Text #16

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0825. Min Act: -0.0000

Data Index: 1169452 (C4 (Web Text))

Max Activating Token Index: 923

Text #17

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0763. Min Act: -0.0001

Data Index: 949989 (C4 (Web Text))

Max Activating Token Index: 236

Text #18

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0680. Min Act: -0.0001

Data Index: 698116 (C4 (Web Text))

Max Activating Token Index: 320

Text #19

Max Range: 0.1349. Min Range: -0.1349

Max Act: 0.0737. Min Act: -0.0001

Data Index: 1139516 (C4 (Web Text))

Max Activating Token Index: 77