Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4576 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.5971. Min Act: -0.0000

Data Index: 585597 (C4 (Web Text))

Max Activating Token Index: 406

Text #1

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.5618. Min Act: -0.0000

Data Index: 309327 (C4 (Web Text))

Max Activating Token Index: 496

Text #2

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.5391. Min Act: -0.0000

Data Index: 1099312 (C4 (Web Text))

Max Activating Token Index: 738

Text #3

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.5535. Min Act: -0.0000

Data Index: 562697 (C4 (Web Text))

Max Activating Token Index: 639

Text #4

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.5338. Min Act: -0.0000

Data Index: 822095 (C4 (Web Text))

Max Activating Token Index: 743

Text #5

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.5230. Min Act: -0.0000

Data Index: 174087 (C4 (Web Text))

Max Activating Token Index: 747

Text #6

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.5087. Min Act: -0.0000

Data Index: 209545 (C4 (Web Text))

Max Activating Token Index: 241

Text #7

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.5174. Min Act: -0.0000

Data Index: 508296 (C4 (Web Text))

Max Activating Token Index: 967

Text #8

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.5067. Min Act: -0.0000

Data Index: 1107586 (C4 (Web Text))

Max Activating Token Index: 654

Text #9

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.4670. Min Act: -0.0000

Data Index: 60012 (C4 (Web Text))

Max Activating Token Index: 340

Text #10

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.4664. Min Act: -0.0000

Data Index: 1155012 (C4 (Web Text))

Max Activating Token Index: 903

Text #11

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.4497. Min Act: -0.0000

Data Index: 460806 (C4 (Web Text))

Max Activating Token Index: 733

Text #12

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.4544. Min Act: -0.0000

Data Index: 1063764 (C4 (Web Text))

Max Activating Token Index: 915

Text #13

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.4621. Min Act: -0.0000

Data Index: 1249564 (C4 (Web Text))

Max Activating Token Index: 206

Text #14

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.4272. Min Act: -0.0000

Data Index: 1291534 (C4 (Web Text))

Max Activating Token Index: 1011

Text #15

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.4773. Min Act: -0.0000

Data Index: 214395 (C4 (Web Text))

Max Activating Token Index: 355

Text #16

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.4301. Min Act: -0.0000

Data Index: 263131 (C4 (Web Text))

Max Activating Token Index: 853

Text #17

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.4599. Min Act: -0.0000

Data Index: 1048382 (C4 (Web Text))

Max Activating Token Index: 846

Text #18

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.4399. Min Act: -0.0000

Data Index: 890778 (C4 (Web Text))

Max Activating Token Index: 569

Text #19

Max Range: 0.5971. Min Range: -0.5971

Max Act: 0.4190. Min Act: -0.0000

Data Index: 74285 (C4 (Web Text))

Max Activating Token Index: 622