Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3249 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.1696. Min Range: -3.1696

Max Act: 3.0910. Min Act: -0.0000

Data Index: 1433418 (Python Code)

Max Activating Token Index: 136

Text #1

Max Range: 3.1696. Min Range: -3.1696

Max Act: 3.1696. Min Act: -0.0000

Data Index: 1542364 (Python Code)

Max Activating Token Index: 729

Text #2

Max Range: 3.1696. Min Range: -3.1696

Max Act: 3.0861. Min Act: -0.0000

Data Index: 1388598 (Python Code)

Max Activating Token Index: 219

Text #3

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.8939. Min Act: -0.0000

Data Index: 1606452 (Python Code)

Max Activating Token Index: 975

Text #4

Max Range: 3.1696. Min Range: -3.1696

Max Act: 3.1189. Min Act: -0.0000

Data Index: 1435957 (Python Code)

Max Activating Token Index: 740

Text #5

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.7608. Min Act: -0.0000

Data Index: 1478726 (Python Code)

Max Activating Token Index: 49

Text #6

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.7627. Min Act: -0.0000

Data Index: 1511786 (Python Code)

Max Activating Token Index: 82

Text #7

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.8824. Min Act: -0.0000

Data Index: 1524106 (Python Code)

Max Activating Token Index: 382

Text #8

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.6768. Min Act: -0.0000

Data Index: 1440730 (Python Code)

Max Activating Token Index: 919

Text #9

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.7688. Min Act: -0.0000

Data Index: 1444735 (Python Code)

Max Activating Token Index: 821

Text #10

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.7417. Min Act: -0.0000

Data Index: 1631315 (Python Code)

Max Activating Token Index: 940

Text #11

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.6057. Min Act: -0.0000

Data Index: 1415724 (Python Code)

Max Activating Token Index: 414

Text #12

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.5773. Min Act: -0.0000

Data Index: 1435694 (Python Code)

Max Activating Token Index: 436

Text #13

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.8404. Min Act: -0.0000

Data Index: 1602883 (Python Code)

Max Activating Token Index: 628

Text #14

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.8123. Min Act: -0.0000

Data Index: 1527623 (Python Code)

Max Activating Token Index: 799

Text #15

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.4770. Min Act: -0.0000

Data Index: 1508457 (Python Code)

Max Activating Token Index: 575

Text #16

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.4574. Min Act: -0.0000

Data Index: 1383078 (Python Code)

Max Activating Token Index: 389

Text #17

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.7078. Min Act: -0.0000

Data Index: 1588580 (Python Code)

Max Activating Token Index: 311

Text #18

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.6833. Min Act: -0.0000

Data Index: 1497998 (Python Code)

Max Activating Token Index: 518

Text #19

Max Range: 3.1696. Min Range: -3.1696

Max Act: 2.6849. Min Act: -0.0000

Data Index: 1381398 (Python Code)

Max Activating Token Index: 944