Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 449 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.9623. Min Act: -0.0000

Data Index: 1444464 (Python Code)

Max Activating Token Index: 599

Text #1

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.9540. Min Act: -0.0000

Data Index: 1456632 (Python Code)

Max Activating Token Index: 449

Text #2

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.7682. Min Act: -0.0000

Data Index: 1449914 (Python Code)

Max Activating Token Index: 275

Text #3

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.8116. Min Act: -0.0000

Data Index: 1472914 (Python Code)

Max Activating Token Index: 846

Text #4

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.9040. Min Act: -0.0000

Data Index: 1550497 (Python Code)

Max Activating Token Index: 668

Text #5

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.7574. Min Act: -0.0000

Data Index: 1623136 (Python Code)

Max Activating Token Index: 768

Text #6

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.7749. Min Act: -0.0000

Data Index: 1620332 (Python Code)

Max Activating Token Index: 898

Text #7

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.7567. Min Act: -0.0000

Data Index: 1564628 (Python Code)

Max Activating Token Index: 109

Text #8

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.7412. Min Act: -0.0000

Data Index: 1545713 (Python Code)

Max Activating Token Index: 785

Text #9

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.6649. Min Act: -0.0000

Data Index: 1384177 (Python Code)

Max Activating Token Index: 235

Text #10

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.6733. Min Act: -0.0000

Data Index: 1613079 (Python Code)

Max Activating Token Index: 686

Text #11

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.6502. Min Act: -0.0000

Data Index: 1507281 (Python Code)

Max Activating Token Index: 480

Text #12

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.6062. Min Act: -0.0000

Data Index: 1455829 (Python Code)

Max Activating Token Index: 298

Text #13

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.5482. Min Act: -0.0000

Data Index: 1636710 (Python Code)

Max Activating Token Index: 959

Text #14

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.4839. Min Act: -0.0000

Data Index: 1508173 (Python Code)

Max Activating Token Index: 949

Text #15

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.4414. Min Act: -0.0000

Data Index: 797577 (C4 (Web Text))

Max Activating Token Index: 247

Text #16

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.4721. Min Act: -0.0000

Data Index: 1651835 (Python Code)

Max Activating Token Index: 539

Text #17

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.3558. Min Act: -0.0000

Data Index: 1327660 (C4 (Web Text))

Max Activating Token Index: 684

Text #18

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.4114. Min Act: -0.0000

Data Index: 1656060 (Python Code)

Max Activating Token Index: 195

Text #19

Max Range: 0.9623. Min Range: -0.9623

Max Act: 0.3909. Min Act: -0.0000

Data Index: 59357 (C4 (Web Text))

Max Activating Token Index: 932