Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3065 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.7849. Min Act: -0.0000

Data Index: 1644901 (Python Code)

Max Activating Token Index: 403

Text #1

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.8881. Min Act: -0.0000

Data Index: 1455578 (Python Code)

Max Activating Token Index: 489

Text #2

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.6101. Min Act: -0.0000

Data Index: 1594285 (Python Code)

Max Activating Token Index: 600

Text #3

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.6890. Min Act: -0.0000

Data Index: 1601504 (Python Code)

Max Activating Token Index: 749

Text #4

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.7918. Min Act: -0.0000

Data Index: 1592802 (Python Code)

Max Activating Token Index: 710

Text #5

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.6041. Min Act: -0.0000

Data Index: 1542346 (Python Code)

Max Activating Token Index: 207

Text #6

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.6379. Min Act: -0.0000

Data Index: 1577428 (Python Code)

Max Activating Token Index: 281

Text #7

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.6219. Min Act: -0.0000

Data Index: 1506886 (Python Code)

Max Activating Token Index: 509

Text #8

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.6145. Min Act: -0.0000

Data Index: 1370761 (Python Code)

Max Activating Token Index: 408

Text #9

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.6048. Min Act: -0.0000

Data Index: 1536402 (Python Code)

Max Activating Token Index: 247

Text #10

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.6036. Min Act: -0.0000

Data Index: 1596761 (Python Code)

Max Activating Token Index: 389

Text #11

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.5090. Min Act: -0.0000

Data Index: 1383146 (Python Code)

Max Activating Token Index: 136

Text #12

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.5034. Min Act: -0.0000

Data Index: 1581794 (Python Code)

Max Activating Token Index: 944

Text #13

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.5871. Min Act: -0.0000

Data Index: 1650542 (Python Code)

Max Activating Token Index: 505

Text #14

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.5752. Min Act: -0.0000

Data Index: 1649481 (Python Code)

Max Activating Token Index: 688

Text #15

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.5654. Min Act: -0.0000

Data Index: 1445921 (Python Code)

Max Activating Token Index: 18

Text #16

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.5545. Min Act: -0.0000

Data Index: 1585402 (Python Code)

Max Activating Token Index: 147

Text #17

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.5902. Min Act: -0.0000

Data Index: 1390830 (Python Code)

Max Activating Token Index: 268

Text #18

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.4777. Min Act: -0.0000

Data Index: 1643559 (Python Code)

Max Activating Token Index: 26

Text #19

Max Range: 1.8881. Min Range: -1.8881

Max Act: 1.4784. Min Act: -0.0000

Data Index: 1480758 (Python Code)

Max Activating Token Index: 602