Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2991 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.4391. Min Range: -3.4391

Max Act: 3.2156. Min Act: -0.0000

Data Index: 1119769 (The Pile)

Max Activating Token Index: 856

Text #1

Max Range: 3.4391. Min Range: -3.4391

Max Act: 3.1798. Min Act: -0.0000

Data Index: 387963 (The Pile)

Max Activating Token Index: 946

Text #2

Max Range: 3.4391. Min Range: -3.4391

Max Act: 3.4391. Min Act: -0.0000

Data Index: 1529207 (The Pile)

Max Activating Token Index: 228

Text #3

Max Range: 3.4391. Min Range: -3.4391

Max Act: 3.1137. Min Act: -0.0000

Data Index: 1504242 (The Pile)

Max Activating Token Index: 545

Text #4

Max Range: 3.4391. Min Range: -3.4391

Max Act: 3.0779. Min Act: -0.0000

Data Index: 115493 (The Pile)

Max Activating Token Index: 426

Text #5

Max Range: 3.4391. Min Range: -3.4391

Max Act: 3.1315. Min Act: -0.0000

Data Index: 52189 (The Pile)

Max Activating Token Index: 320

Text #6

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.9591. Min Act: -0.0000

Data Index: 1960782 (The Pile)

Max Activating Token Index: 644

Text #7

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.9365. Min Act: -0.0000

Data Index: 1872769 (The Pile)

Max Activating Token Index: 424

Text #8

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.7745. Min Act: -0.0000

Data Index: 1249873 (The Pile)

Max Activating Token Index: 744

Text #9

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.8315. Min Act: -0.0000

Data Index: 1081878 (The Pile)

Max Activating Token Index: 791

Text #10

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.5543. Min Act: -0.0000

Data Index: 547035 (The Pile)

Max Activating Token Index: 599

Text #11

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.8735. Min Act: -0.0000

Data Index: 892368 (The Pile)

Max Activating Token Index: 997

Text #12

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.4078. Min Act: -0.0000

Data Index: 929366 (The Pile)

Max Activating Token Index: 988

Text #13

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.7654. Min Act: -0.0000

Data Index: 380532 (The Pile)

Max Activating Token Index: 512

Text #14

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.7400. Min Act: -0.0000

Data Index: 961493 (The Pile)

Max Activating Token Index: 493

Text #15

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.5898. Min Act: -0.0000

Data Index: 474898 (The Pile)

Max Activating Token Index: 23

Text #16

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.7132. Min Act: -0.0000

Data Index: 1642243 (The Pile)

Max Activating Token Index: 213

Text #17

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.8541. Min Act: -0.0000

Data Index: 1689868 (The Pile)

Max Activating Token Index: 138

Text #18

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.7287. Min Act: -0.0000

Data Index: 376318 (The Pile)

Max Activating Token Index: 924

Text #19

Max Range: 3.4391. Min Range: -3.4391

Max Act: 2.5787. Min Act: -0.0000

Data Index: 1798632 (The Pile)

Max Activating Token Index: 824