Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1945 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0988. Min Act: -0.0006

Data Index: 1468317 (Python Code)

Max Activating Token Index: 839

Text #1

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0860. Min Act: -0.0006

Data Index: 1609211 (Python Code)

Max Activating Token Index: 527

Text #2

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0862. Min Act: -0.0006

Data Index: 1598802 (Python Code)

Max Activating Token Index: 374

Text #3

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0791. Min Act: -0.0006

Data Index: 1639668 (Python Code)

Max Activating Token Index: 712

Text #4

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0741. Min Act: -0.0006

Data Index: 1583774 (Python Code)

Max Activating Token Index: 280

Text #5

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0704. Min Act: -0.0006

Data Index: 1418366 (Python Code)

Max Activating Token Index: 944

Text #6

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0725. Min Act: -0.0006

Data Index: 1642808 (Python Code)

Max Activating Token Index: 448

Text #7

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0778. Min Act: -0.0006

Data Index: 1509259 (Python Code)

Max Activating Token Index: 40

Text #8

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0787. Min Act: -0.0006

Data Index: 1606407 (Python Code)

Max Activating Token Index: 347

Text #9

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0745. Min Act: -0.0006

Data Index: 1454930 (Python Code)

Max Activating Token Index: 816

Text #10

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0757. Min Act: -0.0006

Data Index: 1567287 (Python Code)

Max Activating Token Index: 921

Text #11

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0742. Min Act: -0.0006

Data Index: 1496886 (Python Code)

Max Activating Token Index: 681

Text #12

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0751. Min Act: -0.0006

Data Index: 1499940 (Python Code)

Max Activating Token Index: 977

Text #13

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0763. Min Act: -0.0006

Data Index: 1588818 (Python Code)

Max Activating Token Index: 886

Text #14

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0691. Min Act: -0.0006

Data Index: 1577059 (Python Code)

Max Activating Token Index: 251

Text #15

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0746. Min Act: -0.0006

Data Index: 1365258 (Python Code)

Max Activating Token Index: 730

Text #16

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0705. Min Act: -0.0006

Data Index: 1552316 (Python Code)

Max Activating Token Index: 1019

Text #17

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0760. Min Act: -0.0006

Data Index: 1508304 (Python Code)

Max Activating Token Index: 45

Text #18

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0631. Min Act: -0.0006

Data Index: 1465381 (Python Code)

Max Activating Token Index: 697

Text #19

Max Range: 0.0988. Min Range: -0.0988

Max Act: 0.0764. Min Act: -0.0006

Data Index: 1463397 (Python Code)

Max Activating Token Index: 557