Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 281 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.7181. Min Act: -0.0000

Data Index: 130441 (C4 (Web Text))

Max Activating Token Index: 547

Text #1

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.6879. Min Act: -0.0000

Data Index: 339340 (C4 (Web Text))

Max Activating Token Index: 688

Text #2

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.7020. Min Act: -0.0000

Data Index: 836899 (C4 (Web Text))

Max Activating Token Index: 1002

Text #3

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.7247. Min Act: -0.0000

Data Index: 571779 (C4 (Web Text))

Max Activating Token Index: 988

Text #4

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.6532. Min Act: -0.0000

Data Index: 462505 (C4 (Web Text))

Max Activating Token Index: 894

Text #5

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.6896. Min Act: -0.0000

Data Index: 908585 (C4 (Web Text))

Max Activating Token Index: 416

Text #6

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.6619. Min Act: -0.0000

Data Index: 1191315 (C4 (Web Text))

Max Activating Token Index: 456

Text #7

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.5873. Min Act: -0.0000

Data Index: 107422 (C4 (Web Text))

Max Activating Token Index: 550

Text #8

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.6172. Min Act: -0.0000

Data Index: 460343 (C4 (Web Text))

Max Activating Token Index: 88

Text #9

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.6096. Min Act: -0.0000

Data Index: 150585 (C4 (Web Text))

Max Activating Token Index: 220

Text #10

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.6123. Min Act: -0.0000

Data Index: 182529 (C4 (Web Text))

Max Activating Token Index: 267

Text #11

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.5839. Min Act: -0.0000

Data Index: 930666 (C4 (Web Text))

Max Activating Token Index: 65

Text #12

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.5861. Min Act: -0.0000

Data Index: 1114498 (C4 (Web Text))

Max Activating Token Index: 392

Text #13

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.5785. Min Act: -0.0000

Data Index: 1267064 (C4 (Web Text))

Max Activating Token Index: 452

Text #14

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.5940. Min Act: -0.0000

Data Index: 608989 (C4 (Web Text))

Max Activating Token Index: 651

Text #15

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.5783. Min Act: -0.0000

Data Index: 1352413 (C4 (Web Text))

Max Activating Token Index: 402

Text #16

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.5532. Min Act: -0.0000

Data Index: 363474 (C4 (Web Text))

Max Activating Token Index: 575

Text #17

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.5854. Min Act: -0.0000

Data Index: 483705 (C4 (Web Text))

Max Activating Token Index: 855

Text #18

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.5994. Min Act: -0.0000

Data Index: 620275 (C4 (Web Text))

Max Activating Token Index: 938

Text #19

Max Range: 0.7247. Min Range: -0.7247

Max Act: 0.5571. Min Act: -0.0000

Data Index: 406270 (C4 (Web Text))

Max Activating Token Index: 507