Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2791 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.8417. Min Act: -0.0000

Data Index: 1197321 (C4 (Web Text))

Max Activating Token Index: 779

Text #1

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.6830. Min Act: -0.0000

Data Index: 67441 (C4 (Web Text))

Max Activating Token Index: 638

Text #2

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.6936. Min Act: -0.0000

Data Index: 782725 (C4 (Web Text))

Max Activating Token Index: 181

Text #3

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.6684. Min Act: -0.0000

Data Index: 875039 (C4 (Web Text))

Max Activating Token Index: 451

Text #4

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.6349. Min Act: -0.0000

Data Index: 1210350 (C4 (Web Text))

Max Activating Token Index: 133

Text #5

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.6638. Min Act: -0.0000

Data Index: 388855 (C4 (Web Text))

Max Activating Token Index: 934

Text #6

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.5872. Min Act: -0.0000

Data Index: 339833 (C4 (Web Text))

Max Activating Token Index: 499

Text #7

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.5747. Min Act: -0.0000

Data Index: 684691 (C4 (Web Text))

Max Activating Token Index: 464

Text #8

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.5866. Min Act: -0.0000

Data Index: 491452 (C4 (Web Text))

Max Activating Token Index: 513

Text #9

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.5642. Min Act: -0.0000

Data Index: 845898 (C4 (Web Text))

Max Activating Token Index: 544

Text #10

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.5190. Min Act: -0.0000

Data Index: 1023160 (C4 (Web Text))

Max Activating Token Index: 320

Text #11

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.5347. Min Act: -0.0000

Data Index: 245607 (C4 (Web Text))

Max Activating Token Index: 672

Text #12

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.5307. Min Act: -0.0000

Data Index: 1311269 (C4 (Web Text))

Max Activating Token Index: 460

Text #13

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.4945. Min Act: -0.0000

Data Index: 610240 (C4 (Web Text))

Max Activating Token Index: 1012

Text #14

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.4431. Min Act: -0.0000

Data Index: 670069 (C4 (Web Text))

Max Activating Token Index: 350

Text #15

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.5096. Min Act: -0.0000

Data Index: 797559 (C4 (Web Text))

Max Activating Token Index: 1009

Text #16

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.4776. Min Act: -0.0000

Data Index: 60498 (C4 (Web Text))

Max Activating Token Index: 354

Text #17

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.4674. Min Act: -0.0000

Data Index: 625332 (C4 (Web Text))

Max Activating Token Index: 623

Text #18

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.4372. Min Act: -0.0000

Data Index: 567827 (C4 (Web Text))

Max Activating Token Index: 251

Text #19

Max Range: 0.8417. Min Range: -0.8417

Max Act: 0.4810. Min Act: -0.0000

Data Index: 1146017 (C4 (Web Text))

Max Activating Token Index: 104