Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3818 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.8248. Min Act: -0.0000

Data Index: 1608115 (The Pile)

Max Activating Token Index: 886

Text #1

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.9594. Min Act: -0.0000

Data Index: 1969649 (The Pile)

Max Activating Token Index: 459

Text #2

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.8193. Min Act: -0.0000

Data Index: 1369392 (The Pile)

Max Activating Token Index: 804

Text #3

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.7668. Min Act: -0.0000

Data Index: 465429 (The Pile)

Max Activating Token Index: 409

Text #4

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.9024. Min Act: -0.0000

Data Index: 695142 (The Pile)

Max Activating Token Index: 660

Text #5

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.8094. Min Act: -0.0000

Data Index: 610669 (The Pile)

Max Activating Token Index: 394

Text #6

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.8293. Min Act: -0.0000

Data Index: 1371543 (The Pile)

Max Activating Token Index: 762

Text #7

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.8960. Min Act: -0.0000

Data Index: 1610470 (The Pile)

Max Activating Token Index: 733

Text #8

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.8108. Min Act: -0.0000

Data Index: 1255465 (The Pile)

Max Activating Token Index: 891

Text #9

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.6812. Min Act: -0.0000

Data Index: 1966386 (The Pile)

Max Activating Token Index: 642

Text #10

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.7373. Min Act: -0.0000

Data Index: 1912637 (The Pile)

Max Activating Token Index: 874

Text #11

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.7469. Min Act: -0.0000

Data Index: 711905 (The Pile)

Max Activating Token Index: 715

Text #12

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.6798. Min Act: -0.0000

Data Index: 304399 (The Pile)

Max Activating Token Index: 60

Text #13

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.8642. Min Act: -0.0000

Data Index: 1454857 (The Pile)

Max Activating Token Index: 151

Text #14

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.6739. Min Act: -0.0000

Data Index: 568405 (The Pile)

Max Activating Token Index: 713

Text #15

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.7538. Min Act: -0.0000

Data Index: 165562 (The Pile)

Max Activating Token Index: 418

Text #16

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.6881. Min Act: -0.0000

Data Index: 1985770 (The Pile)

Max Activating Token Index: 306

Text #17

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.7601. Min Act: -0.0000

Data Index: 731749 (The Pile)

Max Activating Token Index: 202

Text #18

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.6470. Min Act: -0.0000

Data Index: 408695 (The Pile)

Max Activating Token Index: 156

Text #19

Max Range: 0.9594. Min Range: -0.9594

Max Act: 0.6621. Min Act: -0.0000

Data Index: 61686 (The Pile)

Max Activating Token Index: 133