Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5979 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.5559. Min Act: -0.0000

Data Index: 1632812 (Python Code)

Max Activating Token Index: 563

Text #1

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.4667. Min Act: -0.0000

Data Index: 1389266 (Python Code)

Max Activating Token Index: 745

Text #2

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.6881. Min Act: -0.0000

Data Index: 1417773 (Python Code)

Max Activating Token Index: 376

Text #3

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.3061. Min Act: -0.0000

Data Index: 1516334 (Python Code)

Max Activating Token Index: 439

Text #4

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.1281. Min Act: -0.0000

Data Index: 1437775 (Python Code)

Max Activating Token Index: 412

Text #5

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.4853. Min Act: -0.0000

Data Index: 1443499 (Python Code)

Max Activating Token Index: 354

Text #6

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.4556. Min Act: -0.0000

Data Index: 1580210 (Python Code)

Max Activating Token Index: 705

Text #7

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.2944. Min Act: -0.0000

Data Index: 1490215 (Python Code)

Max Activating Token Index: 260

Text #8

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.5021. Min Act: -0.0000

Data Index: 1409406 (Python Code)

Max Activating Token Index: 472

Text #9

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.4860. Min Act: -0.0000

Data Index: 1653568 (Python Code)

Max Activating Token Index: 274

Text #10

Max Range: 2.6881. Min Range: -2.6881

Max Act: 1.9974. Min Act: -0.0000

Data Index: 1616471 (Python Code)

Max Activating Token Index: 272

Text #11

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.1860. Min Act: -0.0000

Data Index: 1392795 (Python Code)

Max Activating Token Index: 672

Text #12

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.0755. Min Act: -0.0000

Data Index: 1647274 (Python Code)

Max Activating Token Index: 726

Text #13

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.3023. Min Act: -0.0000

Data Index: 1462872 (Python Code)

Max Activating Token Index: 759

Text #14

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.1575. Min Act: -0.0000

Data Index: 1581800 (Python Code)

Max Activating Token Index: 326

Text #15

Max Range: 2.6881. Min Range: -2.6881

Max Act: 1.9859. Min Act: -0.0000

Data Index: 1532412 (Python Code)

Max Activating Token Index: 215

Text #16

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.0709. Min Act: -0.0000

Data Index: 1607521 (Python Code)

Max Activating Token Index: 692

Text #17

Max Range: 2.6881. Min Range: -2.6881

Max Act: 1.9482. Min Act: -0.0000

Data Index: 1437069 (Python Code)

Max Activating Token Index: 660

Text #18

Max Range: 2.6881. Min Range: -2.6881

Max Act: 2.3878. Min Act: -0.0000

Data Index: 1603416 (Python Code)

Max Activating Token Index: 347

Text #19

Max Range: 2.6881. Min Range: -2.6881

Max Act: 1.9192. Min Act: -0.0000

Data Index: 1488395 (Python Code)

Max Activating Token Index: 542