Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 1219 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 4.3967. Min Range: -4.3967

Max Act: 4.3596. Min Act: -0.0000

Data Index: 117713 (The Pile)

Max Activating Token Index: 587

Text #1

Max Range: 4.3967. Min Range: -4.3967

Max Act: 4.3967. Min Act: -0.0000

Data Index: 1065901 (The Pile)

Max Activating Token Index: 332

Text #2

Max Range: 4.3967. Min Range: -4.3967

Max Act: 4.1993. Min Act: -0.0000

Data Index: 1944445 (The Pile)

Max Activating Token Index: 599

Text #3

Max Range: 4.3967. Min Range: -4.3967

Max Act: 4.0113. Min Act: -0.0000

Data Index: 178292 (The Pile)

Max Activating Token Index: 813

Text #4

Max Range: 4.3967. Min Range: -4.3967

Max Act: 4.1311. Min Act: -0.0000

Data Index: 1389548 (The Pile)

Max Activating Token Index: 498

Text #5

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.9659. Min Act: -0.0000

Data Index: 493550 (The Pile)

Max Activating Token Index: 611

Text #6

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.6808. Min Act: -0.0000

Data Index: 1326946 (The Pile)

Max Activating Token Index: 714

Text #7

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.6324. Min Act: -0.0000

Data Index: 1986791 (The Pile)

Max Activating Token Index: 746

Text #8

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.9001. Min Act: -0.0000

Data Index: 1365991 (The Pile)

Max Activating Token Index: 619

Text #9

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.5446. Min Act: -0.0000

Data Index: 564721 (The Pile)

Max Activating Token Index: 681

Text #10

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.4730. Min Act: -0.0000

Data Index: 964256 (The Pile)

Max Activating Token Index: 583

Text #11

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.6247. Min Act: -0.0000

Data Index: 1635380 (The Pile)

Max Activating Token Index: 647

Text #12

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.8809. Min Act: -0.0000

Data Index: 1942790 (The Pile)

Max Activating Token Index: 751

Text #13

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.4238. Min Act: -0.0000

Data Index: 1416428 (The Pile)

Max Activating Token Index: 817

Text #14

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.6672. Min Act: -0.0000

Data Index: 363211 (The Pile)

Max Activating Token Index: 844

Text #15

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.2385. Min Act: -0.0000

Data Index: 289130 (The Pile)

Max Activating Token Index: 547

Text #16

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.3877. Min Act: -0.0000

Data Index: 1720081 (The Pile)

Max Activating Token Index: 266

Text #17

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.2378. Min Act: -0.0000

Data Index: 694198 (The Pile)

Max Activating Token Index: 65

Text #18

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.2591. Min Act: -0.0000

Data Index: 1178813 (The Pile)

Max Activating Token Index: 587

Text #19

Max Range: 4.3967. Min Range: -4.3967

Max Act: 3.2121. Min Act: -0.0000

Data Index: 400279 (The Pile)

Max Activating Token Index: 560