Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 5023 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.4310. Min Act: -0.0000

Data Index: 1207971 (The Pile)

Max Activating Token Index: 539

Text #1

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.5508. Min Act: -0.0000

Data Index: 1771654 (The Pile)

Max Activating Token Index: 1001

Text #2

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.3533. Min Act: -0.0000

Data Index: 730999 (The Pile)

Max Activating Token Index: 974

Text #3

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.2007. Min Act: -0.0000

Data Index: 597590 (The Pile)

Max Activating Token Index: 143

Text #4

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.2291. Min Act: -0.0000

Data Index: 1119266 (The Pile)

Max Activating Token Index: 148

Text #5

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.1154. Min Act: -0.0000

Data Index: 174424 (The Pile)

Max Activating Token Index: 70

Text #6

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.1287. Min Act: -0.0000

Data Index: 1368426 (The Pile)

Max Activating Token Index: 631

Text #7

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.2066. Min Act: -0.0000

Data Index: 1589584 (The Pile)

Max Activating Token Index: 73

Text #8

Max Range: 1.5508. Min Range: -1.5508

Max Act: 0.9895. Min Act: -0.0000

Data Index: 189075 (The Pile)

Max Activating Token Index: 994

Text #9

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.0726. Min Act: -0.0000

Data Index: 953529 (The Pile)

Max Activating Token Index: 302

Text #10

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.0060. Min Act: -0.0000

Data Index: 1743783 (The Pile)

Max Activating Token Index: 739

Text #11

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.0341. Min Act: -0.0000

Data Index: 194405 (The Pile)

Max Activating Token Index: 410

Text #12

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.0192. Min Act: -0.0000

Data Index: 782073 (The Pile)

Max Activating Token Index: 473

Text #13

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.0480. Min Act: -0.0000

Data Index: 898359 (The Pile)

Max Activating Token Index: 153

Text #14

Max Range: 1.5508. Min Range: -1.5508

Max Act: 1.0396. Min Act: -0.0000

Data Index: 239490 (The Pile)

Max Activating Token Index: 385

Text #15

Max Range: 1.5508. Min Range: -1.5508

Max Act: 0.9517. Min Act: -0.0000

Data Index: 1545598 (The Pile)

Max Activating Token Index: 509

Text #16

Max Range: 1.5508. Min Range: -1.5508

Max Act: 0.9478. Min Act: -0.0000

Data Index: 479320 (The Pile)

Max Activating Token Index: 811

Text #17

Max Range: 1.5508. Min Range: -1.5508

Max Act: 0.9856. Min Act: -0.0000

Data Index: 1316625 (The Pile)

Max Activating Token Index: 365

Text #18

Max Range: 1.5508. Min Range: -1.5508

Max Act: 0.9541. Min Act: -0.0000

Data Index: 801596 (The Pile)

Max Activating Token Index: 688

Text #19

Max Range: 1.5508. Min Range: -1.5508

Max Act: 0.9383. Min Act: -0.0000

Data Index: 188674 (The Pile)

Max Activating Token Index: 318