Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2643 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.9291. Min Act: -0.0000

Data Index: 344703 (The Pile)

Max Activating Token Index: 557

Text #1

Max Range: 8.0969. Min Range: -8.0969

Max Act: 8.0714. Min Act: -0.0000

Data Index: 562916 (The Pile)

Max Activating Token Index: 585

Text #2

Max Range: 8.0969. Min Range: -8.0969

Max Act: 8.0969. Min Act: -0.0000

Data Index: 311152 (The Pile)

Max Activating Token Index: 738

Text #3

Max Range: 8.0969. Min Range: -8.0969

Max Act: 8.0814. Min Act: -0.0000

Data Index: 1415908 (The Pile)

Max Activating Token Index: 478

Text #4

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.7935. Min Act: -0.0000

Data Index: 728697 (The Pile)

Max Activating Token Index: 832

Text #5

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.7492. Min Act: -0.0000

Data Index: 338750 (The Pile)

Max Activating Token Index: 525

Text #6

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.8010. Min Act: -0.0000

Data Index: 1890613 (The Pile)

Max Activating Token Index: 547

Text #7

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.7402. Min Act: -0.0000

Data Index: 1231544 (The Pile)

Max Activating Token Index: 693

Text #8

Max Range: 8.0969. Min Range: -8.0969

Max Act: 8.0508. Min Act: -0.0000

Data Index: 75312 (The Pile)

Max Activating Token Index: 435

Text #9

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.8723. Min Act: -0.0000

Data Index: 593922 (The Pile)

Max Activating Token Index: 555

Text #10

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.8537. Min Act: -0.0000

Data Index: 658679 (The Pile)

Max Activating Token Index: 453

Text #11

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.4663. Min Act: -0.0000

Data Index: 416753 (The Pile)

Max Activating Token Index: 688

Text #12

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.2379. Min Act: -0.0000

Data Index: 697878 (The Pile)

Max Activating Token Index: 641

Text #13

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.8966. Min Act: -0.0000

Data Index: 1041851 (The Pile)

Max Activating Token Index: 502

Text #14

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.7529. Min Act: -0.0000

Data Index: 619389 (The Pile)

Max Activating Token Index: 491

Text #15

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.5814. Min Act: -0.0000

Data Index: 1323342 (The Pile)

Max Activating Token Index: 421

Text #16

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.9062. Min Act: -0.0000

Data Index: 1601025 (The Pile)

Max Activating Token Index: 426

Text #17

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.8761. Min Act: -0.0000

Data Index: 981703 (The Pile)

Max Activating Token Index: 555

Text #18

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.5205. Min Act: -0.0000

Data Index: 1461575 (The Pile)

Max Activating Token Index: 665

Text #19

Max Range: 8.0969. Min Range: -8.0969

Max Act: 7.6132. Min Act: -0.0000

Data Index: 1674207 (The Pile)

Max Activating Token Index: 792