Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3677 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.0454. Min Range: -2.0454

Max Act: 2.0454. Min Act: -0.0000

Data Index: 1056724 (C4 (Web Text))

Max Activating Token Index: 1004

Text #1

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.8598. Min Act: -0.0000

Data Index: 534532 (C4 (Web Text))

Max Activating Token Index: 548

Text #2

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.7863. Min Act: -0.0000

Data Index: 741767 (C4 (Web Text))

Max Activating Token Index: 772

Text #3

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.7541. Min Act: -0.0000

Data Index: 113609 (C4 (Web Text))

Max Activating Token Index: 789

Text #4

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.7126. Min Act: -0.0000

Data Index: 770211 (C4 (Web Text))

Max Activating Token Index: 844

Text #5

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.7277. Min Act: -0.0000

Data Index: 954577 (C4 (Web Text))

Max Activating Token Index: 804

Text #6

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.7124. Min Act: -0.0000

Data Index: 1008438 (C4 (Web Text))

Max Activating Token Index: 661

Text #7

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.6959. Min Act: -0.0000

Data Index: 1356813 (C4 (Web Text))

Max Activating Token Index: 467

Text #8

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.6824. Min Act: -0.0000

Data Index: 1060869 (C4 (Web Text))

Max Activating Token Index: 736

Text #9

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.6804. Min Act: -0.0000

Data Index: 664783 (C4 (Web Text))

Max Activating Token Index: 866

Text #10

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.7606. Min Act: -0.0000

Data Index: 969209 (C4 (Web Text))

Max Activating Token Index: 829

Text #11

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.5770. Min Act: -0.0000

Data Index: 1237850 (C4 (Web Text))

Max Activating Token Index: 636

Text #12

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.6725. Min Act: -0.0000

Data Index: 588580 (C4 (Web Text))

Max Activating Token Index: 710

Text #13

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.6654. Min Act: -0.0000

Data Index: 1321083 (C4 (Web Text))

Max Activating Token Index: 715

Text #14

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.6677. Min Act: -0.0000

Data Index: 505410 (C4 (Web Text))

Max Activating Token Index: 941

Text #15

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.5516. Min Act: -0.0000

Data Index: 1277890 (C4 (Web Text))

Max Activating Token Index: 888

Text #16

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.6506. Min Act: -0.0000

Data Index: 94537 (C4 (Web Text))

Max Activating Token Index: 808

Text #17

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.7361. Min Act: -0.0000

Data Index: 621768 (C4 (Web Text))

Max Activating Token Index: 639

Text #18

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.6358. Min Act: -0.0000

Data Index: 837433 (C4 (Web Text))

Max Activating Token Index: 856

Text #19

Max Range: 2.0454. Min Range: -2.0454

Max Act: 1.7063. Min Act: -0.0000

Data Index: 565045 (C4 (Web Text))

Max Activating Token Index: 637