Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 219 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.3332. Min Act: -0.0000

Data Index: 1039723 (The Pile)

Max Activating Token Index: 663

Text #1

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2777. Min Act: -0.0000

Data Index: 1737547 (The Pile)

Max Activating Token Index: 745

Text #2

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.3083. Min Act: -0.0000

Data Index: 1561354 (The Pile)

Max Activating Token Index: 542

Text #3

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2993. Min Act: -0.0000

Data Index: 1321061 (The Pile)

Max Activating Token Index: 894

Text #4

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.3112. Min Act: -0.0000

Data Index: 549024 (The Pile)

Max Activating Token Index: 365

Text #5

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2963. Min Act: -0.0000

Data Index: 771536 (The Pile)

Max Activating Token Index: 849

Text #6

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2968. Min Act: -0.0000

Data Index: 767321 (The Pile)

Max Activating Token Index: 955

Text #7

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2546. Min Act: -0.0000

Data Index: 562750 (The Pile)

Max Activating Token Index: 746

Text #8

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.3009. Min Act: -0.0000

Data Index: 1237781 (The Pile)

Max Activating Token Index: 983

Text #9

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2854. Min Act: -0.0000

Data Index: 591600 (The Pile)

Max Activating Token Index: 588

Text #10

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2532. Min Act: -0.0000

Data Index: 490821 (The Pile)

Max Activating Token Index: 856

Text #11

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2575. Min Act: -0.0000

Data Index: 586776 (The Pile)

Max Activating Token Index: 711

Text #12

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2994. Min Act: -0.0000

Data Index: 175188 (The Pile)

Max Activating Token Index: 749

Text #13

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2295. Min Act: -0.0000

Data Index: 1869385 (The Pile)

Max Activating Token Index: 765

Text #14

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2400. Min Act: -0.0000

Data Index: 218547 (The Pile)

Max Activating Token Index: 449

Text #15

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2508. Min Act: -0.0000

Data Index: 891461 (The Pile)

Max Activating Token Index: 774

Text #16

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2561. Min Act: -0.0000

Data Index: 1993208 (The Pile)

Max Activating Token Index: 701

Text #17

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2569. Min Act: -0.0000

Data Index: 1559370 (The Pile)

Max Activating Token Index: 715

Text #18

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2794. Min Act: -0.0000

Data Index: 1695844 (The Pile)

Max Activating Token Index: 880

Text #19

Max Range: 0.3332. Min Range: -0.3332

Max Act: 0.2321. Min Act: 0.0000

Data Index: 1844707 (The Pile)

Max Activating Token Index: 733