Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2002 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 4.2040. Min Range: -4.2040

Max Act: 4.2040. Min Act: -0.0000

Data Index: 1194440 (The Pile)

Max Activating Token Index: 875

Text #1

Max Range: 4.2040. Min Range: -4.2040

Max Act: 3.0882. Min Act: -0.0000

Data Index: 821807 (The Pile)

Max Activating Token Index: 1013

Text #2

Max Range: 4.2040. Min Range: -4.2040

Max Act: 3.0538. Min Act: -0.0000

Data Index: 447994 (The Pile)

Max Activating Token Index: 789

Text #3

Max Range: 4.2040. Min Range: -4.2040

Max Act: 3.0890. Min Act: -0.0000

Data Index: 628193 (The Pile)

Max Activating Token Index: 893

Text #4

Max Range: 4.2040. Min Range: -4.2040

Max Act: 3.0936. Min Act: -0.0000

Data Index: 458937 (The Pile)

Max Activating Token Index: 985

Text #5

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.9115. Min Act: -0.0000

Data Index: 689222 (The Pile)

Max Activating Token Index: 458

Text #6

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.8373. Min Act: -0.0000

Data Index: 1622992 (The Pile)

Max Activating Token Index: 101

Text #7

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.7756. Min Act: -0.0000

Data Index: 892833 (The Pile)

Max Activating Token Index: 434

Text #8

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.7711. Min Act: -0.0000

Data Index: 1363978 (The Pile)

Max Activating Token Index: 238

Text #9

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.5562. Min Act: -0.0000

Data Index: 406095 (The Pile)

Max Activating Token Index: 539

Text #10

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.7859. Min Act: -0.0000

Data Index: 653067 (The Pile)

Max Activating Token Index: 791

Text #11

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.6502. Min Act: -0.0000

Data Index: 1932981 (The Pile)

Max Activating Token Index: 232

Text #12

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.7716. Min Act: -0.0000

Data Index: 1666752 (The Pile)

Max Activating Token Index: 552

Text #13

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.6278. Min Act: -0.0000

Data Index: 1863622 (The Pile)

Max Activating Token Index: 802

Text #14

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.6206. Min Act: -0.0000

Data Index: 448264 (The Pile)

Max Activating Token Index: 446

Text #15

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.7444. Min Act: -0.0000

Data Index: 740536 (The Pile)

Max Activating Token Index: 965

Text #16

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.4860. Min Act: -0.0000

Data Index: 1089046 (The Pile)

Max Activating Token Index: 787

Text #17

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.5905. Min Act: -0.0000

Data Index: 1756015 (The Pile)

Max Activating Token Index: 254

Text #18

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.5567. Min Act: -0.0000

Data Index: 261737 (The Pile)

Max Activating Token Index: 248

Text #19

Max Range: 4.2040. Min Range: -4.2040

Max Act: 2.4491. Min Act: -0.0000

Data Index: 1202868 (The Pile)

Max Activating Token Index: 146