Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 907 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.1210. Min Act: -0.0001

Data Index: 236451 (C4 (Web Text))

Max Activating Token Index: 756

Text #1

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0773. Min Act: -0.0001

Data Index: 404220 (C4 (Web Text))

Max Activating Token Index: 245

Text #2

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0766. Min Act: -0.0001

Data Index: 1287842 (C4 (Web Text))

Max Activating Token Index: 432

Text #3

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0760. Min Act: -0.0001

Data Index: 21590 (C4 (Web Text))

Max Activating Token Index: 428

Text #4

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0692. Min Act: -0.0001

Data Index: 647504 (C4 (Web Text))

Max Activating Token Index: 530

Text #5

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0728. Min Act: -0.0001

Data Index: 532967 (C4 (Web Text))

Max Activating Token Index: 93

Text #6

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0661. Min Act: -0.0001

Data Index: 767034 (C4 (Web Text))

Max Activating Token Index: 60

Text #7

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0717. Min Act: -0.0001

Data Index: 1047355 (C4 (Web Text))

Max Activating Token Index: 622

Text #8

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0600. Min Act: -0.0001

Data Index: 1311454 (C4 (Web Text))

Max Activating Token Index: 616

Text #9

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0623. Min Act: -0.0001

Data Index: 549073 (C4 (Web Text))

Max Activating Token Index: 424

Text #10

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0643. Min Act: -0.0001

Data Index: 458710 (C4 (Web Text))

Max Activating Token Index: 480

Text #11

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0644. Min Act: -0.0001

Data Index: 168846 (C4 (Web Text))

Max Activating Token Index: 205

Text #12

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0628. Min Act: -0.0001

Data Index: 800550 (C4 (Web Text))

Max Activating Token Index: 889

Text #13

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0598. Min Act: -0.0001

Data Index: 990259 (C4 (Web Text))

Max Activating Token Index: 760

Text #14

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0617. Min Act: -0.0001

Data Index: 847536 (C4 (Web Text))

Max Activating Token Index: 701

Text #15

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0624. Min Act: -0.0001

Data Index: 791513 (C4 (Web Text))

Max Activating Token Index: 342

Text #16

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0615. Min Act: -0.0001

Data Index: 572035 (C4 (Web Text))

Max Activating Token Index: 223

Text #17

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0616. Min Act: -0.0001

Data Index: 494495 (C4 (Web Text))

Max Activating Token Index: 252

Text #18

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0545. Min Act: -0.0001

Data Index: 446880 (C4 (Web Text))

Max Activating Token Index: 863

Text #19

Max Range: 0.1210. Min Range: -0.1210

Max Act: 0.0631. Min Act: -0.0001

Data Index: 28049 (C4 (Web Text))

Max Activating Token Index: 457