Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 371 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.9476. Min Act: -0.0000

Data Index: 1340947 (C4 (Web Text))

Max Activating Token Index: 240

Text #1

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.7645. Min Act: -0.0000

Data Index: 724291 (C4 (Web Text))

Max Activating Token Index: 244

Text #2

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.7980. Min Act: -0.0000

Data Index: 1194506 (C4 (Web Text))

Max Activating Token Index: 514

Text #3

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.7603. Min Act: -0.0000

Data Index: 840820 (C4 (Web Text))

Max Activating Token Index: 549

Text #4

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.6981. Min Act: -0.0000

Data Index: 1058966 (C4 (Web Text))

Max Activating Token Index: 518

Text #5

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.6892. Min Act: -0.0000

Data Index: 1323977 (C4 (Web Text))

Max Activating Token Index: 85

Text #6

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.7248. Min Act: -0.0000

Data Index: 879860 (C4 (Web Text))

Max Activating Token Index: 147

Text #7

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.7106. Min Act: -0.0000

Data Index: 1267962 (C4 (Web Text))

Max Activating Token Index: 100

Text #8

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.7102. Min Act: -0.0000

Data Index: 909178 (C4 (Web Text))

Max Activating Token Index: 929

Text #9

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.7023. Min Act: -0.0000

Data Index: 402405 (C4 (Web Text))

Max Activating Token Index: 806

Text #10

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.6746. Min Act: -0.0000

Data Index: 1220928 (C4 (Web Text))

Max Activating Token Index: 802

Text #11

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.6705. Min Act: -0.0000

Data Index: 322006 (C4 (Web Text))

Max Activating Token Index: 708

Text #12

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.5678. Min Act: -0.0000

Data Index: 1173259 (C4 (Web Text))

Max Activating Token Index: 693

Text #13

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.6601. Min Act: -0.0000

Data Index: 742877 (C4 (Web Text))

Max Activating Token Index: 564

Text #14

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.6507. Min Act: -0.0000

Data Index: 543146 (C4 (Web Text))

Max Activating Token Index: 734

Text #15

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.6707. Min Act: -0.0000

Data Index: 526305 (C4 (Web Text))

Max Activating Token Index: 744

Text #16

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.7208. Min Act: -0.0000

Data Index: 842383 (C4 (Web Text))

Max Activating Token Index: 888

Text #17

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.6347. Min Act: -0.0000

Data Index: 528601 (C4 (Web Text))

Max Activating Token Index: 609

Text #18

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.5607. Min Act: -0.0000

Data Index: 166046 (C4 (Web Text))

Max Activating Token Index: 840

Text #19

Max Range: 0.9476. Min Range: -0.9476

Max Act: 0.6387. Min Act: -0.0000

Data Index: 147945 (C4 (Web Text))

Max Activating Token Index: 553