Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2741 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9974. Min Act: -0.0002

Data Index: 1599474 (Python Code)

Max Activating Token Index: 109

Text #1

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9764. Min Act: -0.0003

Data Index: 1437126 (Python Code)

Max Activating Token Index: 253

Text #2

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9873. Min Act: -0.0002

Data Index: 1534225 (Python Code)

Max Activating Token Index: 748

Text #3

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9825. Min Act: -0.0002

Data Index: 1611423 (Python Code)

Max Activating Token Index: 362

Text #4

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9973. Min Act: -0.0002

Data Index: 1638072 (Python Code)

Max Activating Token Index: 195

Text #5

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9884. Min Act: -0.0002

Data Index: 1520895 (Python Code)

Max Activating Token Index: 145

Text #6

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9989. Min Act: -0.0002

Data Index: 1643252 (Python Code)

Max Activating Token Index: 802

Text #7

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9636. Min Act: -0.0002

Data Index: 1581648 (Python Code)

Max Activating Token Index: 886

Text #8

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9651. Min Act: -0.0002

Data Index: 1380661 (Python Code)

Max Activating Token Index: 982

Text #9

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9982. Min Act: -0.0002

Data Index: 1456948 (Python Code)

Max Activating Token Index: 608

Text #10

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9459. Min Act: -0.0002

Data Index: 1484449 (Python Code)

Max Activating Token Index: 47

Text #11

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9696. Min Act: -0.0002

Data Index: 1536829 (Python Code)

Max Activating Token Index: 403

Text #12

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9555. Min Act: -0.0002

Data Index: 1498937 (Python Code)

Max Activating Token Index: 308

Text #13

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9215. Min Act: -0.0002

Data Index: 1389987 (Python Code)

Max Activating Token Index: 410

Text #14

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9116. Min Act: -0.0002

Data Index: 1471182 (Python Code)

Max Activating Token Index: 313

Text #15

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9295. Min Act: -0.0002

Data Index: 1506609 (Python Code)

Max Activating Token Index: 592

Text #16

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9539. Min Act: -0.0002

Data Index: 1456553 (Python Code)

Max Activating Token Index: 604

Text #17

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9171. Min Act: -0.0002

Data Index: 1603220 (Python Code)

Max Activating Token Index: 748

Text #18

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9467. Min Act: -0.0002

Data Index: 1627900 (Python Code)

Max Activating Token Index: 662

Text #19

Max Range: 0.9989. Min Range: -0.9989

Max Act: 0.9387. Min Act: -0.0002

Data Index: 1465639 (Python Code)

Max Activating Token Index: 143