Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5439 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1980. Min Act: -0.0002

Data Index: 574457 (C4 (Web Text))

Max Activating Token Index: 24

Text #1

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1880. Min Act: -0.0002

Data Index: 1012860 (C4 (Web Text))

Max Activating Token Index: 384

Text #2

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1737. Min Act: -0.0002

Data Index: 1090370 (C4 (Web Text))

Max Activating Token Index: 889

Text #3

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1693. Min Act: -0.0002

Data Index: 11306 (C4 (Web Text))

Max Activating Token Index: 291

Text #4

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1665. Min Act: -0.0002

Data Index: 913240 (C4 (Web Text))

Max Activating Token Index: 269

Text #5

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1619. Min Act: -0.0002

Data Index: 168394 (C4 (Web Text))

Max Activating Token Index: 517

Text #6

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1637. Min Act: -0.0002

Data Index: 983807 (C4 (Web Text))

Max Activating Token Index: 755

Text #7

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1646. Min Act: -0.0002

Data Index: 196093 (C4 (Web Text))

Max Activating Token Index: 32

Text #8

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1589. Min Act: -0.0002

Data Index: 726530 (C4 (Web Text))

Max Activating Token Index: 203

Text #9

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1583. Min Act: -0.0002

Data Index: 247097 (C4 (Web Text))

Max Activating Token Index: 841

Text #10

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1589. Min Act: -0.0002

Data Index: 55279 (C4 (Web Text))

Max Activating Token Index: 342

Text #11

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1554. Min Act: -0.0002

Data Index: 286783 (C4 (Web Text))

Max Activating Token Index: 809

Text #12

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1566. Min Act: -0.0002

Data Index: 1238480 (C4 (Web Text))

Max Activating Token Index: 657

Text #13

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1537. Min Act: -0.0002

Data Index: 182484 (C4 (Web Text))

Max Activating Token Index: 430

Text #14

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1518. Min Act: -0.0002

Data Index: 609517 (C4 (Web Text))

Max Activating Token Index: 277

Text #15

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1525. Min Act: -0.0002

Data Index: 131049 (C4 (Web Text))

Max Activating Token Index: 412

Text #16

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1494. Min Act: -0.0002

Data Index: 670481 (C4 (Web Text))

Max Activating Token Index: 162

Text #17

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1506. Min Act: -0.0002

Data Index: 86142 (C4 (Web Text))

Max Activating Token Index: 728

Text #18

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1529. Min Act: -0.0002

Data Index: 493130 (C4 (Web Text))

Max Activating Token Index: 449

Text #19

Max Range: 0.1980. Min Range: -0.1980

Max Act: 0.1509. Min Act: -0.0002

Data Index: 136841 (C4 (Web Text))

Max Activating Token Index: 261