Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4062 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.5375. Min Act: -0.0000

Data Index: 1642501 (Python Code)

Max Activating Token Index: 470

Text #1

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.4751. Min Act: -0.0000

Data Index: 1645366 (Python Code)

Max Activating Token Index: 488

Text #2

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.4288. Min Act: -0.0000

Data Index: 1559923 (Python Code)

Max Activating Token Index: 577

Text #3

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.4365. Min Act: -0.0000

Data Index: 1617364 (Python Code)

Max Activating Token Index: 68

Text #4

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.4282. Min Act: -0.0000

Data Index: 1557325 (Python Code)

Max Activating Token Index: 718

Text #5

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.4087. Min Act: -0.0000

Data Index: 1612797 (Python Code)

Max Activating Token Index: 359

Text #6

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3891. Min Act: -0.0000

Data Index: 1376873 (Python Code)

Max Activating Token Index: 404

Text #7

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3817. Min Act: -0.0000

Data Index: 1414138 (Python Code)

Max Activating Token Index: 64

Text #8

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3815. Min Act: -0.0000

Data Index: 1413811 (Python Code)

Max Activating Token Index: 38

Text #9

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3932. Min Act: -0.0000

Data Index: 1376236 (Python Code)

Max Activating Token Index: 308

Text #10

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3887. Min Act: -0.0000

Data Index: 1450804 (Python Code)

Max Activating Token Index: 50

Text #11

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3845. Min Act: -0.0000

Data Index: 1571909 (Python Code)

Max Activating Token Index: 680

Text #12

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3620. Min Act: -0.0000

Data Index: 1646549 (Python Code)

Max Activating Token Index: 607

Text #13

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3516. Min Act: -0.0000

Data Index: 1650160 (Python Code)

Max Activating Token Index: 469

Text #14

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3588. Min Act: -0.0000

Data Index: 1466334 (Python Code)

Max Activating Token Index: 549

Text #15

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3414. Min Act: -0.0000

Data Index: 1416737 (Python Code)

Max Activating Token Index: 519

Text #16

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3370. Min Act: -0.0000

Data Index: 1508389 (Python Code)

Max Activating Token Index: 480

Text #17

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3575. Min Act: -0.0000

Data Index: 1439251 (Python Code)

Max Activating Token Index: 840

Text #18

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3466. Min Act: -0.0000

Data Index: 1532409 (Python Code)

Max Activating Token Index: 473

Text #19

Max Range: 0.5375. Min Range: -0.5375

Max Act: 0.3194. Min Act: -0.0000

Data Index: 1422599 (Python Code)

Max Activating Token Index: 20