Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1549 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.8959. Min Range: -1.8959

Max Act: 1.8959. Min Act: -0.0001

Data Index: 1549655 (Python Code)

Max Activating Token Index: 349

Text #1

Max Range: 1.8959. Min Range: -1.8959

Max Act: 1.2415. Min Act: -0.0001

Data Index: 1489408 (Python Code)

Max Activating Token Index: 256

Text #2

Max Range: 1.8959. Min Range: -1.8959

Max Act: 1.0788. Min Act: -0.0001

Data Index: 1385212 (Python Code)

Max Activating Token Index: 796

Text #3

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.8867. Min Act: -0.0001

Data Index: 1589477 (Python Code)

Max Activating Token Index: 789

Text #4

Max Range: 1.8959. Min Range: -1.8959

Max Act: 1.1021. Min Act: -0.0001

Data Index: 1585822 (Python Code)

Max Activating Token Index: 395

Text #5

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.8919. Min Act: -0.0001

Data Index: 519683 (C4 (Web Text))

Max Activating Token Index: 178

Text #6

Max Range: 1.8959. Min Range: -1.8959

Max Act: 1.0186. Min Act: -0.0001

Data Index: 1429967 (Python Code)

Max Activating Token Index: 985

Text #7

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.8510. Min Act: -0.0001

Data Index: 724882 (C4 (Web Text))

Max Activating Token Index: 963

Text #8

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.8826. Min Act: -0.0001

Data Index: 1584459 (Python Code)

Max Activating Token Index: 673

Text #9

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.9717. Min Act: -0.0001

Data Index: 1457293 (Python Code)

Max Activating Token Index: 89

Text #10

Max Range: 1.8959. Min Range: -1.8959

Max Act: 1.0104. Min Act: -0.0001

Data Index: 1500004 (Python Code)

Max Activating Token Index: 369

Text #11

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.9389. Min Act: -0.0001

Data Index: 1493617 (Python Code)

Max Activating Token Index: 305

Text #12

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.8588. Min Act: -0.0001

Data Index: 1581636 (Python Code)

Max Activating Token Index: 893

Text #13

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.9552. Min Act: -0.0001

Data Index: 1465298 (Python Code)

Max Activating Token Index: 284

Text #14

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.9901. Min Act: -0.0001

Data Index: 1410411 (Python Code)

Max Activating Token Index: 708

Text #15

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.9018. Min Act: -0.0001

Data Index: 1527688 (Python Code)

Max Activating Token Index: 987

Text #16

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.9160. Min Act: -0.0001

Data Index: 1460264 (Python Code)

Max Activating Token Index: 801

Text #17

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.8745. Min Act: -0.0001

Data Index: 1479232 (Python Code)

Max Activating Token Index: 869

Text #18

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.7702. Min Act: -0.0001

Data Index: 1140526 (C4 (Web Text))

Max Activating Token Index: 602

Text #19

Max Range: 1.8959. Min Range: -1.8959

Max Act: 0.8997. Min Act: -0.0001

Data Index: 1445438 (Python Code)

Max Activating Token Index: 374