Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 262 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.7936. Min Act: -0.0000

Data Index: 864460 (C4 (Web Text))

Max Activating Token Index: 494

Text #1

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.7609. Min Act: -0.0000

Data Index: 894420 (C4 (Web Text))

Max Activating Token Index: 912

Text #2

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.7829. Min Act: -0.0000

Data Index: 1015480 (C4 (Web Text))

Max Activating Token Index: 440

Text #3

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.7478. Min Act: -0.0001

Data Index: 1543849 (Python Code)

Max Activating Token Index: 955

Text #4

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.7025. Min Act: -0.0000

Data Index: 767831 (C4 (Web Text))

Max Activating Token Index: 82

Text #5

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.7435. Min Act: -0.0000

Data Index: 503081 (C4 (Web Text))

Max Activating Token Index: 443

Text #6

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.7010. Min Act: -0.0000

Data Index: 249628 (C4 (Web Text))

Max Activating Token Index: 534

Text #7

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.6938. Min Act: -0.0000

Data Index: 1551766 (Python Code)

Max Activating Token Index: 1015

Text #8

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.6790. Min Act: -0.0000

Data Index: 383 (C4 (Web Text))

Max Activating Token Index: 750

Text #9

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.7224. Min Act: -0.0000

Data Index: 870269 (C4 (Web Text))

Max Activating Token Index: 306

Text #10

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.6891. Min Act: -0.0000

Data Index: 500488 (C4 (Web Text))

Max Activating Token Index: 55

Text #11

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.6444. Min Act: -0.0000

Data Index: 533589 (C4 (Web Text))

Max Activating Token Index: 730

Text #12

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.6279. Min Act: -0.0000

Data Index: 1197752 (C4 (Web Text))

Max Activating Token Index: 165

Text #13

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.6201. Min Act: -0.0000

Data Index: 535159 (C4 (Web Text))

Max Activating Token Index: 688

Text #14

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.5781. Min Act: -0.0000

Data Index: 1079172 (C4 (Web Text))

Max Activating Token Index: 753

Text #15

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.6344. Min Act: -0.0000

Data Index: 918431 (C4 (Web Text))

Max Activating Token Index: 456

Text #16

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.6300. Min Act: -0.0000

Data Index: 1192309 (C4 (Web Text))

Max Activating Token Index: 514

Text #17

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.5879. Min Act: -0.0000

Data Index: 1528609 (Python Code)

Max Activating Token Index: 718

Text #18

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.6308. Min Act: -0.0000

Data Index: 1205328 (C4 (Web Text))

Max Activating Token Index: 727

Text #19

Max Range: 0.7936. Min Range: -0.7936

Max Act: 0.6006. Min Act: -0.0000

Data Index: 439923 (C4 (Web Text))

Max Activating Token Index: 331