Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 321 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.9709. Min Act: -0.0001

Data Index: 1046552 (C4 (Web Text))

Max Activating Token Index: 887

Text #1

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.8885. Min Act: -0.0001

Data Index: 346196 (C4 (Web Text))

Max Activating Token Index: 385

Text #2

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.9391. Min Act: -0.0001

Data Index: 90863 (C4 (Web Text))

Max Activating Token Index: 534

Text #3

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.8261. Min Act: -0.0001

Data Index: 284967 (C4 (Web Text))

Max Activating Token Index: 777

Text #4

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.9103. Min Act: -0.0001

Data Index: 655104 (C4 (Web Text))

Max Activating Token Index: 752

Text #5

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.8650. Min Act: -0.0001

Data Index: 1110481 (C4 (Web Text))

Max Activating Token Index: 64

Text #6

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.9762. Min Act: -0.0001

Data Index: 208569 (C4 (Web Text))

Max Activating Token Index: 933

Text #7

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.9194. Min Act: -0.0001

Data Index: 269449 (C4 (Web Text))

Max Activating Token Index: 917

Text #8

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.8629. Min Act: -0.0001

Data Index: 521738 (C4 (Web Text))

Max Activating Token Index: 943

Text #9

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.8610. Min Act: -0.0001

Data Index: 717801 (C4 (Web Text))

Max Activating Token Index: 965

Text #10

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.8149. Min Act: -0.0001

Data Index: 570015 (C4 (Web Text))

Max Activating Token Index: 188

Text #11

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.8196. Min Act: -0.0001

Data Index: 294108 (C4 (Web Text))

Max Activating Token Index: 210

Text #12

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.7905. Min Act: -0.0001

Data Index: 1317793 (C4 (Web Text))

Max Activating Token Index: 942

Text #13

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.8722. Min Act: -0.0001

Data Index: 737965 (C4 (Web Text))

Max Activating Token Index: 683

Text #14

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.9081. Min Act: -0.0001

Data Index: 997152 (C4 (Web Text))

Max Activating Token Index: 854

Text #15

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.8462. Min Act: -0.0001

Data Index: 364531 (C4 (Web Text))

Max Activating Token Index: 21

Text #16

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.8316. Min Act: -0.0001

Data Index: 613598 (C4 (Web Text))

Max Activating Token Index: 80

Text #17

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.7320. Min Act: -0.0001

Data Index: 509271 (C4 (Web Text))

Max Activating Token Index: 206

Text #18

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.8402. Min Act: -0.0001

Data Index: 1189273 (C4 (Web Text))

Max Activating Token Index: 29

Text #19

Max Range: 0.9762. Min Range: -0.9762

Max Act: 0.8679. Min Act: -0.0001

Data Index: 445093 (C4 (Web Text))

Max Activating Token Index: 808