Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 897 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.5445. Min Act: -0.0000

Data Index: 1111585 (The Pile)

Max Activating Token Index: 51

Text #1

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.2549. Min Act: -0.0000

Data Index: 1080871 (The Pile)

Max Activating Token Index: 700

Text #2

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.3944. Min Act: -0.0000

Data Index: 1548256 (The Pile)

Max Activating Token Index: 618

Text #3

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.3835. Min Act: -0.0000

Data Index: 965643 (The Pile)

Max Activating Token Index: 667

Text #4

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.3490. Min Act: -0.0000

Data Index: 610715 (The Pile)

Max Activating Token Index: 230

Text #5

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.3144. Min Act: -0.0000

Data Index: 1526731 (The Pile)

Max Activating Token Index: 523

Text #6

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.2697. Min Act: -0.0000

Data Index: 1075022 (The Pile)

Max Activating Token Index: 13

Text #7

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.2680. Min Act: -0.0000

Data Index: 1565097 (The Pile)

Max Activating Token Index: 314

Text #8

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.3510. Min Act: -0.0000

Data Index: 1705780 (The Pile)

Max Activating Token Index: 168

Text #9

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.1926. Min Act: -0.0000

Data Index: 1303450 (The Pile)

Max Activating Token Index: 202

Text #10

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.0383. Min Act: -0.0000

Data Index: 1840155 (The Pile)

Max Activating Token Index: 55

Text #11

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.1234. Min Act: -0.0000

Data Index: 251891 (The Pile)

Max Activating Token Index: 612

Text #12

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.2646. Min Act: -0.0001

Data Index: 1706120 (The Pile)

Max Activating Token Index: 869

Text #13

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.0866. Min Act: -0.0000

Data Index: 1204421 (The Pile)

Max Activating Token Index: 382

Text #14

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.2413. Min Act: -0.0000

Data Index: 1785541 (The Pile)

Max Activating Token Index: 12

Text #15

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.0638. Min Act: -0.0000

Data Index: 1861919 (The Pile)

Max Activating Token Index: 20

Text #16

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.0914. Min Act: -0.0000

Data Index: 856837 (The Pile)

Max Activating Token Index: 839

Text #17

Max Range: 3.5445. Min Range: -3.5445

Max Act: 2.9301. Min Act: -0.0000

Data Index: 871393 (The Pile)

Max Activating Token Index: 253

Text #18

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.0591. Min Act: -0.0000

Data Index: 1036201 (The Pile)

Max Activating Token Index: 60

Text #19

Max Range: 3.5445. Min Range: -3.5445

Max Act: 3.1911. Min Act: -0.0000

Data Index: 1319324 (The Pile)

Max Activating Token Index: 659