Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4756 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.7080. Min Range: -3.7080

Max Act: 3.7080. Min Act: -0.0000

Data Index: 1403927 (The Pile)

Max Activating Token Index: 833

Text #1

Max Range: 3.7080. Min Range: -3.7080

Max Act: 3.5709. Min Act: -0.0000

Data Index: 704975 (The Pile)

Max Activating Token Index: 107

Text #2

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.8941. Min Act: -0.0000

Data Index: 1473214 (The Pile)

Max Activating Token Index: 701

Text #3

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.8845. Min Act: -0.0000

Data Index: 1178263 (The Pile)

Max Activating Token Index: 355

Text #4

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.7859. Min Act: -0.0000

Data Index: 1410638 (The Pile)

Max Activating Token Index: 632

Text #5

Max Range: 3.7080. Min Range: -3.7080

Max Act: 1.9554. Min Act: -0.0000

Data Index: 256863 (The Pile)

Max Activating Token Index: 1018

Text #6

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.8134. Min Act: -0.0000

Data Index: 1874291 (The Pile)

Max Activating Token Index: 784

Text #7

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.7346. Min Act: -0.0000

Data Index: 37489 (The Pile)

Max Activating Token Index: 478

Text #8

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.6059. Min Act: -0.0000

Data Index: 1491448 (The Pile)

Max Activating Token Index: 245

Text #9

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.3596. Min Act: -0.0000

Data Index: 728840 (The Pile)

Max Activating Token Index: 127

Text #10

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.0996. Min Act: -0.0000

Data Index: 317374 (The Pile)

Max Activating Token Index: 821

Text #11

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.2819. Min Act: -0.0000

Data Index: 181349 (The Pile)

Max Activating Token Index: 945

Text #12

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.1581. Min Act: -0.0000

Data Index: 1187105 (The Pile)

Max Activating Token Index: 832

Text #13

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.3669. Min Act: -0.0000

Data Index: 1699350 (The Pile)

Max Activating Token Index: 456

Text #14

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.5505. Min Act: -0.0000

Data Index: 871110 (The Pile)

Max Activating Token Index: 744

Text #15

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.3643. Min Act: -0.0000

Data Index: 1497451 (The Pile)

Max Activating Token Index: 741

Text #16

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.4109. Min Act: -0.0000

Data Index: 1887813 (The Pile)

Max Activating Token Index: 343

Text #17

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.2808. Min Act: -0.0000

Data Index: 1712790 (The Pile)

Max Activating Token Index: 97

Text #18

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.2716. Min Act: -0.0000

Data Index: 876299 (The Pile)

Max Activating Token Index: 951

Text #19

Max Range: 3.7080. Min Range: -3.7080

Max Act: 2.0436. Min Act: -0.0000

Data Index: 1356923 (The Pile)

Max Activating Token Index: 469