Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4164 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.2072. Min Range: -3.2072

Max Act: 3.1687. Min Act: -0.0000

Data Index: 1539602 (The Pile)

Max Activating Token Index: 1001

Text #1

Max Range: 3.2072. Min Range: -3.2072

Max Act: 3.1546. Min Act: -0.0000

Data Index: 777664 (The Pile)

Max Activating Token Index: 905

Text #2

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.9851. Min Act: -0.0000

Data Index: 823914 (The Pile)

Max Activating Token Index: 538

Text #3

Max Range: 3.2072. Min Range: -3.2072

Max Act: 3.1133. Min Act: -0.0000

Data Index: 133696 (The Pile)

Max Activating Token Index: 234

Text #4

Max Range: 3.2072. Min Range: -3.2072

Max Act: 3.2072. Min Act: -0.0000

Data Index: 363719 (The Pile)

Max Activating Token Index: 779

Text #5

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.9338. Min Act: -0.0000

Data Index: 1396089 (The Pile)

Max Activating Token Index: 5

Text #6

Max Range: 3.2072. Min Range: -3.2072

Max Act: 3.0372. Min Act: -0.0000

Data Index: 1053972 (The Pile)

Max Activating Token Index: 287

Text #7

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.8472. Min Act: -0.0000

Data Index: 1908136 (The Pile)

Max Activating Token Index: 826

Text #8

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.8953. Min Act: -0.0000

Data Index: 636671 (The Pile)

Max Activating Token Index: 807

Text #9

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.6869. Min Act: -0.0000

Data Index: 1400774 (The Pile)

Max Activating Token Index: 844

Text #10

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.6891. Min Act: -0.0000

Data Index: 1143296 (The Pile)

Max Activating Token Index: 784

Text #11

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.4021. Min Act: -0.0000

Data Index: 993033 (The Pile)

Max Activating Token Index: 635

Text #12

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.6665. Min Act: -0.0000

Data Index: 1245830 (The Pile)

Max Activating Token Index: 250

Text #13

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.7962. Min Act: -0.0000

Data Index: 490026 (The Pile)

Max Activating Token Index: 974

Text #14

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.4986. Min Act: -0.0000

Data Index: 141133 (The Pile)

Max Activating Token Index: 328

Text #15

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.6166. Min Act: -0.0000

Data Index: 1203628 (The Pile)

Max Activating Token Index: 977

Text #16

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.6172. Min Act: -0.0000

Data Index: 1903439 (The Pile)

Max Activating Token Index: 886

Text #17

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.4764. Min Act: -0.0000

Data Index: 546016 (The Pile)

Max Activating Token Index: 977

Text #18

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.5588. Min Act: -0.0000

Data Index: 1280710 (The Pile)

Max Activating Token Index: 481

Text #19

Max Range: 3.2072. Min Range: -3.2072

Max Act: 2.7096. Min Act: -0.0000

Data Index: 336111 (The Pile)

Max Activating Token Index: 306