Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4016 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.0731. Min Range: -3.0731

Max Act: 3.0731. Min Act: -0.0000

Data Index: 103773 (The Pile)

Max Activating Token Index: 187

Text #1

Max Range: 3.0731. Min Range: -3.0731

Max Act: 2.9610. Min Act: -0.0000

Data Index: 969053 (The Pile)

Max Activating Token Index: 606

Text #2

Max Range: 3.0731. Min Range: -3.0731

Max Act: 2.5817. Min Act: -0.0000

Data Index: 886749 (The Pile)

Max Activating Token Index: 118

Text #3

Max Range: 3.0731. Min Range: -3.0731

Max Act: 2.2644. Min Act: -0.0000

Data Index: 299969 (The Pile)

Max Activating Token Index: 498

Text #4

Max Range: 3.0731. Min Range: -3.0731

Max Act: 2.0589. Min Act: -0.0000

Data Index: 1750217 (The Pile)

Max Activating Token Index: 368

Text #5

Max Range: 3.0731. Min Range: -3.0731

Max Act: 2.0133. Min Act: -0.0000

Data Index: 847879 (The Pile)

Max Activating Token Index: 1007

Text #6

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.8087. Min Act: -0.0000

Data Index: 495796 (The Pile)

Max Activating Token Index: 694

Text #7

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.9424. Min Act: -0.0000

Data Index: 983381 (The Pile)

Max Activating Token Index: 42

Text #8

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.5729. Min Act: -0.0000

Data Index: 595931 (The Pile)

Max Activating Token Index: 521

Text #9

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.7246. Min Act: -0.0000

Data Index: 486075 (The Pile)

Max Activating Token Index: 693

Text #10

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.5466. Min Act: -0.0000

Data Index: 1878855 (The Pile)

Max Activating Token Index: 722

Text #11

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.4750. Min Act: -0.0000

Data Index: 1910161 (The Pile)

Max Activating Token Index: 420

Text #12

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.5377. Min Act: -0.0000

Data Index: 892590 (The Pile)

Max Activating Token Index: 292

Text #13

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.4732. Min Act: -0.0000

Data Index: 748272 (The Pile)

Max Activating Token Index: 320

Text #14

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.5625. Min Act: -0.0000

Data Index: 15828 (The Pile)

Max Activating Token Index: 380

Text #15

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.4629. Min Act: -0.0000

Data Index: 1028962 (The Pile)

Max Activating Token Index: 786

Text #16

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.4310. Min Act: -0.0000

Data Index: 383010 (The Pile)

Max Activating Token Index: 901

Text #17

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.4488. Min Act: -0.0000

Data Index: 1221291 (The Pile)

Max Activating Token Index: 851

Text #18

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.5102. Min Act: -0.0000

Data Index: 1988636 (The Pile)

Max Activating Token Index: 108

Text #19

Max Range: 3.0731. Min Range: -3.0731

Max Act: 1.2541. Min Act: -0.0000

Data Index: 1870824 (The Pile)

Max Activating Token Index: 264