Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3269 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1876. Min Act: -0.0002

Data Index: 1615514 (Python Code)

Max Activating Token Index: 458

Text #1

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1899. Min Act: -0.0002

Data Index: 1596193 (Python Code)

Max Activating Token Index: 482

Text #2

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1876. Min Act: -0.0002

Data Index: 1533240 (Python Code)

Max Activating Token Index: 737

Text #3

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1822. Min Act: -0.0002

Data Index: 1492737 (Python Code)

Max Activating Token Index: 740

Text #4

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1656. Min Act: -0.0002

Data Index: 1597946 (Python Code)

Max Activating Token Index: 594

Text #5

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1706. Min Act: -0.0002

Data Index: 1536554 (Python Code)

Max Activating Token Index: 831

Text #6

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1696. Min Act: -0.0002

Data Index: 1550641 (Python Code)

Max Activating Token Index: 89

Text #7

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1674. Min Act: -0.0002

Data Index: 1653937 (Python Code)

Max Activating Token Index: 762

Text #8

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1533. Min Act: -0.0002

Data Index: 1402395 (Python Code)

Max Activating Token Index: 1003

Text #9

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1446. Min Act: -0.0002

Data Index: 1547786 (Python Code)

Max Activating Token Index: 983

Text #10

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1550. Min Act: -0.0002

Data Index: 1382910 (Python Code)

Max Activating Token Index: 332

Text #11

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1455. Min Act: -0.0002

Data Index: 1441049 (Python Code)

Max Activating Token Index: 692

Text #12

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1511. Min Act: -0.0002

Data Index: 1470778 (Python Code)

Max Activating Token Index: 182

Text #13

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1588. Min Act: -0.0002

Data Index: 1403296 (Python Code)

Max Activating Token Index: 303

Text #14

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1549. Min Act: -0.0002

Data Index: 1367321 (Python Code)

Max Activating Token Index: 88

Text #15

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1547. Min Act: -0.0002

Data Index: 1565055 (Python Code)

Max Activating Token Index: 920

Text #16

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1522. Min Act: -0.0002

Data Index: 1538873 (Python Code)

Max Activating Token Index: 783

Text #17

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1553. Min Act: -0.0002

Data Index: 1510636 (Python Code)

Max Activating Token Index: 246

Text #18

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1450. Min Act: -0.0002

Data Index: 1471820 (Python Code)

Max Activating Token Index: 75

Text #19

Max Range: 0.1899. Min Range: -0.1899

Max Act: 0.1481. Min Act: -0.0002

Data Index: 1433951 (Python Code)

Max Activating Token Index: 525