Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1901 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0839. Min Act: -0.0002

Data Index: 494747 (C4 (Web Text))

Max Activating Token Index: 11

Text #1

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0822. Min Act: -0.0002

Data Index: 1561106 (Python Code)

Max Activating Token Index: 375

Text #2

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0807. Min Act: -0.0002

Data Index: 1025461 (C4 (Web Text))

Max Activating Token Index: 341

Text #3

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0783. Min Act: -0.0002

Data Index: 1328370 (C4 (Web Text))

Max Activating Token Index: 29

Text #4

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0778. Min Act: -0.0002

Data Index: 97696 (C4 (Web Text))

Max Activating Token Index: 479

Text #5

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0730. Min Act: -0.0002

Data Index: 289200 (C4 (Web Text))

Max Activating Token Index: 173

Text #6

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0770. Min Act: -0.0002

Data Index: 755078 (C4 (Web Text))

Max Activating Token Index: 238

Text #7

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0751. Min Act: -0.0002

Data Index: 293909 (C4 (Web Text))

Max Activating Token Index: 769

Text #8

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0747. Min Act: -0.0002

Data Index: 1551307 (Python Code)

Max Activating Token Index: 27

Text #9

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0748. Min Act: -0.0002

Data Index: 764705 (C4 (Web Text))

Max Activating Token Index: 609

Text #10

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0764. Min Act: -0.0002

Data Index: 595144 (C4 (Web Text))

Max Activating Token Index: 161

Text #11

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0760. Min Act: -0.0002

Data Index: 941238 (C4 (Web Text))

Max Activating Token Index: 798

Text #12

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0748. Min Act: -0.0002

Data Index: 1225377 (C4 (Web Text))

Max Activating Token Index: 745

Text #13

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0685. Min Act: -0.0002

Data Index: 472174 (C4 (Web Text))

Max Activating Token Index: 923

Text #14

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0652. Min Act: -0.0002

Data Index: 1593174 (Python Code)

Max Activating Token Index: 309

Text #15

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0658. Min Act: -0.0002

Data Index: 554199 (C4 (Web Text))

Max Activating Token Index: 225

Text #16

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0719. Min Act: -0.0002

Data Index: 685458 (C4 (Web Text))

Max Activating Token Index: 499

Text #17

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0717. Min Act: -0.0002

Data Index: 317703 (C4 (Web Text))

Max Activating Token Index: 947

Text #18

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0735. Min Act: -0.0002

Data Index: 1131437 (C4 (Web Text))

Max Activating Token Index: 795

Text #19

Max Range: 0.0839. Min Range: -0.0839

Max Act: 0.0739. Min Act: -0.0002

Data Index: 1477380 (Python Code)

Max Activating Token Index: 1020