Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4058 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 4.8790. Min Range: -4.8790

Max Act: 4.8790. Min Act: -0.0000

Data Index: 1700095 (The Pile)

Max Activating Token Index: 770

Text #1

Max Range: 4.8790. Min Range: -4.8790

Max Act: 4.2998. Min Act: -0.0000

Data Index: 1036950 (The Pile)

Max Activating Token Index: 863

Text #2

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.5068. Min Act: -0.0000

Data Index: 907599 (The Pile)

Max Activating Token Index: 443

Text #3

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.7364. Min Act: -0.0000

Data Index: 181320 (The Pile)

Max Activating Token Index: 504

Text #4

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.5156. Min Act: -0.0000

Data Index: 1647140 (The Pile)

Max Activating Token Index: 524

Text #5

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.4788. Min Act: -0.0000

Data Index: 1824838 (The Pile)

Max Activating Token Index: 820

Text #6

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.4612. Min Act: -0.0000

Data Index: 542645 (The Pile)

Max Activating Token Index: 273

Text #7

Max Range: 4.8790. Min Range: -4.8790

Max Act: 2.8317. Min Act: -0.0000

Data Index: 911415 (The Pile)

Max Activating Token Index: 851

Text #8

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.5284. Min Act: -0.0000

Data Index: 1216685 (The Pile)

Max Activating Token Index: 649

Text #9

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.4810. Min Act: -0.0000

Data Index: 1529802 (The Pile)

Max Activating Token Index: 420

Text #10

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.3336. Min Act: -0.0000

Data Index: 1110995 (The Pile)

Max Activating Token Index: 347

Text #11

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.1833. Min Act: -0.0000

Data Index: 474471 (The Pile)

Max Activating Token Index: 937

Text #12

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.1689. Min Act: -0.0000

Data Index: 267384 (The Pile)

Max Activating Token Index: 421

Text #13

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.1401. Min Act: -0.0000

Data Index: 710965 (The Pile)

Max Activating Token Index: 467

Text #14

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.1214. Min Act: -0.0000

Data Index: 906763 (The Pile)

Max Activating Token Index: 359

Text #15

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.2047. Min Act: -0.0000

Data Index: 628208 (The Pile)

Max Activating Token Index: 429

Text #16

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.3247. Min Act: -0.0000

Data Index: 1608419 (The Pile)

Max Activating Token Index: 391

Text #17

Max Range: 4.8790. Min Range: -4.8790

Max Act: 2.8692. Min Act: -0.0000

Data Index: 1738807 (The Pile)

Max Activating Token Index: 610

Text #18

Max Range: 4.8790. Min Range: -4.8790

Max Act: 2.7200. Min Act: -0.0000

Data Index: 1272750 (The Pile)

Max Activating Token Index: 869

Text #19

Max Range: 4.8790. Min Range: -4.8790

Max Act: 3.1351. Min Act: -0.0000

Data Index: 1897242 (The Pile)

Max Activating Token Index: 71