Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4855 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.7508. Min Act: -0.0000

Data Index: 57778 (C4 (Web Text))

Max Activating Token Index: 937

Text #1

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.7234. Min Act: 0.0001

Data Index: 610286 (C4 (Web Text))

Max Activating Token Index: 85

Text #2

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.6577. Min Act: -0.0000

Data Index: 1130707 (C4 (Web Text))

Max Activating Token Index: 936

Text #3

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.6404. Min Act: -0.0000

Data Index: 1320680 (C4 (Web Text))

Max Activating Token Index: 918

Text #4

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.6339. Min Act: -0.0000

Data Index: 1078064 (C4 (Web Text))

Max Activating Token Index: 541

Text #5

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.6170. Min Act: -0.0000

Data Index: 598573 (C4 (Web Text))

Max Activating Token Index: 156

Text #6

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.6076. Min Act: -0.0000

Data Index: 1224562 (C4 (Web Text))

Max Activating Token Index: 251

Text #7

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.6249. Min Act: -0.0000

Data Index: 1139384 (C4 (Web Text))

Max Activating Token Index: 693

Text #8

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.6061. Min Act: 0.0000

Data Index: 1340432 (C4 (Web Text))

Max Activating Token Index: 643

Text #9

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.6120. Min Act: -0.0000

Data Index: 692987 (C4 (Web Text))

Max Activating Token Index: 944

Text #10

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.6132. Min Act: 0.0001

Data Index: 427508 (C4 (Web Text))

Max Activating Token Index: 477

Text #11

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.6108. Min Act: -0.0000

Data Index: 193580 (C4 (Web Text))

Max Activating Token Index: 38

Text #12

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.6068. Min Act: -0.0000

Data Index: 940549 (C4 (Web Text))

Max Activating Token Index: 676

Text #13

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.5745. Min Act: -0.0000

Data Index: 1302434 (C4 (Web Text))

Max Activating Token Index: 1000

Text #14

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.5672. Min Act: -0.0000

Data Index: 600355 (C4 (Web Text))

Max Activating Token Index: 73

Text #15

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.5644. Min Act: -0.0000

Data Index: 1159618 (C4 (Web Text))

Max Activating Token Index: 32

Text #16

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.5662. Min Act: 0.0001

Data Index: 117846 (C4 (Web Text))

Max Activating Token Index: 876

Text #17

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.5608. Min Act: -0.0000

Data Index: 308271 (C4 (Web Text))

Max Activating Token Index: 66

Text #18

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.5429. Min Act: -0.0000

Data Index: 1103820 (C4 (Web Text))

Max Activating Token Index: 782

Text #19

Max Range: 0.7508. Min Range: -0.7508

Max Act: 0.5589. Min Act: -0.0000

Data Index: 933556 (C4 (Web Text))

Max Activating Token Index: 793