Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2022 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.4219. Min Act: -0.0003

Data Index: 1369798 (Python Code)

Max Activating Token Index: 456

Text #1

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3970. Min Act: -0.0003

Data Index: 1423534 (Python Code)

Max Activating Token Index: 1021

Text #2

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3807. Min Act: -0.0003

Data Index: 1473291 (Python Code)

Max Activating Token Index: 772

Text #3

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3568. Min Act: -0.0003

Data Index: 1607768 (Python Code)

Max Activating Token Index: 570

Text #4

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3700. Min Act: -0.0003

Data Index: 1517026 (Python Code)

Max Activating Token Index: 589

Text #5

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3642. Min Act: -0.0003

Data Index: 1364674 (Python Code)

Max Activating Token Index: 58

Text #6

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3593. Min Act: -0.0003

Data Index: 1404510 (Python Code)

Max Activating Token Index: 906

Text #7

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3580. Min Act: -0.0003

Data Index: 1607145 (Python Code)

Max Activating Token Index: 94

Text #8

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3551. Min Act: -0.0003

Data Index: 1602339 (Python Code)

Max Activating Token Index: 1001

Text #9

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3504. Min Act: -0.0003

Data Index: 1437711 (Python Code)

Max Activating Token Index: 951

Text #10

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3610. Min Act: -0.0003

Data Index: 1491470 (Python Code)

Max Activating Token Index: 493

Text #11

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3522. Min Act: -0.0003

Data Index: 1373266 (Python Code)

Max Activating Token Index: 273

Text #12

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3357. Min Act: -0.0003

Data Index: 1634400 (Python Code)

Max Activating Token Index: 726

Text #13

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3341. Min Act: -0.0003

Data Index: 1360407 (Python Code)

Max Activating Token Index: 612

Text #14

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3583. Min Act: -0.0003

Data Index: 1452487 (Python Code)

Max Activating Token Index: 289

Text #15

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3416. Min Act: -0.0003

Data Index: 1398037 (Python Code)

Max Activating Token Index: 59

Text #16

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3496. Min Act: -0.0003

Data Index: 1446095 (Python Code)

Max Activating Token Index: 689

Text #17

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3626. Min Act: -0.0003

Data Index: 1420682 (Python Code)

Max Activating Token Index: 516

Text #18

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3488. Min Act: -0.0003

Data Index: 1578909 (Python Code)

Max Activating Token Index: 586

Text #19

Max Range: 0.4219. Min Range: -0.4219

Max Act: 0.3470. Min Act: -0.0003

Data Index: 1448886 (Python Code)

Max Activating Token Index: 1019