Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 40 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.4308. Min Act: -0.0000

Data Index: 338601 (The Pile)

Max Activating Token Index: 296

Text #1

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.4023. Min Act: -0.0000

Data Index: 989291 (The Pile)

Max Activating Token Index: 980

Text #2

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3697. Min Act: -0.0000

Data Index: 1661984 (The Pile)

Max Activating Token Index: 772

Text #3

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3033. Min Act: -0.0000

Data Index: 52950 (The Pile)

Max Activating Token Index: 200

Text #4

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3843. Min Act: -0.0000

Data Index: 1861350 (The Pile)

Max Activating Token Index: 127

Text #5

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3749. Min Act: -0.0000

Data Index: 1093548 (The Pile)

Max Activating Token Index: 571

Text #6

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3590. Min Act: -0.0000

Data Index: 1346630 (The Pile)

Max Activating Token Index: 320

Text #7

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3717. Min Act: -0.0000

Data Index: 1259881 (The Pile)

Max Activating Token Index: 725

Text #8

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3566. Min Act: -0.0000

Data Index: 1718080 (The Pile)

Max Activating Token Index: 800

Text #9

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3570. Min Act: -0.0000

Data Index: 1991243 (The Pile)

Max Activating Token Index: 941

Text #10

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3705. Min Act: -0.0000

Data Index: 1662128 (The Pile)

Max Activating Token Index: 633

Text #11

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3305. Min Act: -0.0000

Data Index: 1456129 (The Pile)

Max Activating Token Index: 438

Text #12

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3514. Min Act: -0.0000

Data Index: 565786 (The Pile)

Max Activating Token Index: 875

Text #13

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3212. Min Act: -0.0000

Data Index: 1966213 (The Pile)

Max Activating Token Index: 520

Text #14

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3207. Min Act: -0.0000

Data Index: 1326307 (The Pile)

Max Activating Token Index: 284

Text #15

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3360. Min Act: -0.0000

Data Index: 1228676 (The Pile)

Max Activating Token Index: 780

Text #16

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3403. Min Act: -0.0000

Data Index: 1446897 (The Pile)

Max Activating Token Index: 435

Text #17

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3392. Min Act: -0.0000

Data Index: 195347 (The Pile)

Max Activating Token Index: 359

Text #18

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3388. Min Act: -0.0000

Data Index: 1689882 (The Pile)

Max Activating Token Index: 251

Text #19

Max Range: 0.4308. Min Range: -0.4308

Max Act: 0.3271. Min Act: -0.0000

Data Index: 1639006 (The Pile)

Max Activating Token Index: 418