Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 662 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.7822. Min Act: -0.0000

Data Index: 177009 (C4 (Web Text))

Max Activating Token Index: 660

Text #1

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.5593. Min Act: -0.0000

Data Index: 1346558 (C4 (Web Text))

Max Activating Token Index: 353

Text #2

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.5030. Min Act: -0.0000

Data Index: 1063586 (C4 (Web Text))

Max Activating Token Index: 903

Text #3

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.5217. Min Act: -0.0000

Data Index: 461552 (C4 (Web Text))

Max Activating Token Index: 628

Text #4

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4995. Min Act: -0.0000

Data Index: 328571 (C4 (Web Text))

Max Activating Token Index: 223

Text #5

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4871. Min Act: -0.0000

Data Index: 798374 (C4 (Web Text))

Max Activating Token Index: 654

Text #6

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4832. Min Act: -0.0000

Data Index: 883897 (C4 (Web Text))

Max Activating Token Index: 71

Text #7

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4688. Min Act: -0.0000

Data Index: 748207 (C4 (Web Text))

Max Activating Token Index: 496

Text #8

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4964. Min Act: -0.0000

Data Index: 970870 (C4 (Web Text))

Max Activating Token Index: 633

Text #9

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4639. Min Act: -0.0000

Data Index: 355292 (C4 (Web Text))

Max Activating Token Index: 209

Text #10

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4737. Min Act: -0.0000

Data Index: 164861 (C4 (Web Text))

Max Activating Token Index: 319

Text #11

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.5075. Min Act: -0.0000

Data Index: 1215857 (C4 (Web Text))

Max Activating Token Index: 70

Text #12

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4774. Min Act: -0.0000

Data Index: 899736 (C4 (Web Text))

Max Activating Token Index: 581

Text #13

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4479. Min Act: -0.0000

Data Index: 523949 (C4 (Web Text))

Max Activating Token Index: 483

Text #14

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4378. Min Act: -0.0000

Data Index: 1202058 (C4 (Web Text))

Max Activating Token Index: 951

Text #15

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4226. Min Act: -0.0000

Data Index: 838612 (C4 (Web Text))

Max Activating Token Index: 438

Text #16

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4345. Min Act: -0.0000

Data Index: 224067 (C4 (Web Text))

Max Activating Token Index: 596

Text #17

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4261. Min Act: -0.0000

Data Index: 444449 (C4 (Web Text))

Max Activating Token Index: 730

Text #18

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4386. Min Act: -0.0000

Data Index: 1187736 (C4 (Web Text))

Max Activating Token Index: 423

Text #19

Max Range: 0.7822. Min Range: -0.7822

Max Act: 0.4132. Min Act: -0.0000

Data Index: 621903 (C4 (Web Text))

Max Activating Token Index: 870