Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3301 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.7790. Min Act: -0.0000

Data Index: 880361 (C4 (Web Text))

Max Activating Token Index: 820

Text #1

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.7352. Min Act: -0.0000

Data Index: 1128843 (C4 (Web Text))

Max Activating Token Index: 928

Text #2

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.7892. Min Act: -0.0000

Data Index: 406926 (C4 (Web Text))

Max Activating Token Index: 544

Text #3

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.8580. Min Act: -0.0000

Data Index: 467705 (C4 (Web Text))

Max Activating Token Index: 940

Text #4

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.8317. Min Act: 0.0000

Data Index: 1088425 (C4 (Web Text))

Max Activating Token Index: 742

Text #5

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.7726. Min Act: -0.0000

Data Index: 1134167 (C4 (Web Text))

Max Activating Token Index: 813

Text #6

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.8236. Min Act: -0.0000

Data Index: 1115539 (C4 (Web Text))

Max Activating Token Index: 692

Text #7

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.8207. Min Act: -0.0000

Data Index: 1071821 (C4 (Web Text))

Max Activating Token Index: 43

Text #8

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.6883. Min Act: -0.0000

Data Index: 751207 (C4 (Web Text))

Max Activating Token Index: 850

Text #9

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.7781. Min Act: -0.0000

Data Index: 48569 (C4 (Web Text))

Max Activating Token Index: 461

Text #10

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.7774. Min Act: -0.0000

Data Index: 578167 (C4 (Web Text))

Max Activating Token Index: 571

Text #11

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.7071. Min Act: -0.0000

Data Index: 1150173 (C4 (Web Text))

Max Activating Token Index: 740

Text #12

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.7327. Min Act: -0.0000

Data Index: 1006759 (C4 (Web Text))

Max Activating Token Index: 641

Text #13

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.7119. Min Act: -0.0000

Data Index: 1350903 (C4 (Web Text))

Max Activating Token Index: 589

Text #14

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.7323. Min Act: -0.0000

Data Index: 451319 (C4 (Web Text))

Max Activating Token Index: 478

Text #15

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.6929. Min Act: -0.0000

Data Index: 548391 (C4 (Web Text))

Max Activating Token Index: 334

Text #16

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.5925. Min Act: -0.0000

Data Index: 944134 (C4 (Web Text))

Max Activating Token Index: 286

Text #17

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.6687. Min Act: -0.0000

Data Index: 1126571 (C4 (Web Text))

Max Activating Token Index: 71

Text #18

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.5606. Min Act: -0.0000

Data Index: 558692 (C4 (Web Text))

Max Activating Token Index: 715

Text #19

Max Range: 0.8580. Min Range: -0.8580

Max Act: 0.6446. Min Act: -0.0000

Data Index: 1278719 (C4 (Web Text))

Max Activating Token Index: 952