Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 154 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.5108. Min Range: -3.5108

Max Act: 3.3085. Min Act: -0.0000

Data Index: 732278 (The Pile)

Max Activating Token Index: 620

Text #1

Max Range: 3.5108. Min Range: -3.5108

Max Act: 3.4280. Min Act: -0.0000

Data Index: 441751 (The Pile)

Max Activating Token Index: 776

Text #2

Max Range: 3.5108. Min Range: -3.5108

Max Act: 3.4127. Min Act: -0.0000

Data Index: 1671906 (The Pile)

Max Activating Token Index: 160

Text #3

Max Range: 3.5108. Min Range: -3.5108

Max Act: 3.3844. Min Act: -0.0000

Data Index: 636129 (The Pile)

Max Activating Token Index: 612

Text #4

Max Range: 3.5108. Min Range: -3.5108

Max Act: 3.5108. Min Act: -0.0000

Data Index: 458937 (The Pile)

Max Activating Token Index: 692

Text #5

Max Range: 3.5108. Min Range: -3.5108

Max Act: 3.0242. Min Act: -0.0000

Data Index: 1935706 (The Pile)

Max Activating Token Index: 276

Text #6

Max Range: 3.5108. Min Range: -3.5108

Max Act: 3.1701. Min Act: -0.0000

Data Index: 1408594 (The Pile)

Max Activating Token Index: 518

Text #7

Max Range: 3.5108. Min Range: -3.5108

Max Act: 3.1971. Min Act: -0.0000

Data Index: 114903 (The Pile)

Max Activating Token Index: 462

Text #8

Max Range: 3.5108. Min Range: -3.5108

Max Act: 3.0497. Min Act: -0.0000

Data Index: 669236 (The Pile)

Max Activating Token Index: 382

Text #9

Max Range: 3.5108. Min Range: -3.5108

Max Act: 3.0086. Min Act: -0.0000

Data Index: 283195 (The Pile)

Max Activating Token Index: 827

Text #10

Max Range: 3.5108. Min Range: -3.5108

Max Act: 2.8592. Min Act: -0.0000

Data Index: 1265811 (The Pile)

Max Activating Token Index: 545

Text #11

Max Range: 3.5108. Min Range: -3.5108

Max Act: 2.9749. Min Act: -0.0000

Data Index: 1324776 (The Pile)

Max Activating Token Index: 60

Text #12

Max Range: 3.5108. Min Range: -3.5108

Max Act: 2.9666. Min Act: -0.0000

Data Index: 1770012 (The Pile)

Max Activating Token Index: 850

Text #13

Max Range: 3.5108. Min Range: -3.5108

Max Act: 2.9301. Min Act: -0.0000

Data Index: 1225334 (The Pile)

Max Activating Token Index: 384

Text #14

Max Range: 3.5108. Min Range: -3.5108

Max Act: 3.0208. Min Act: -0.0000

Data Index: 906955 (The Pile)

Max Activating Token Index: 115

Text #15

Max Range: 3.5108. Min Range: -3.5108

Max Act: 2.9866. Min Act: -0.0000

Data Index: 1339966 (The Pile)

Max Activating Token Index: 176

Text #16

Max Range: 3.5108. Min Range: -3.5108

Max Act: 2.8744. Min Act: -0.0000

Data Index: 478621 (The Pile)

Max Activating Token Index: 92

Text #17

Max Range: 3.5108. Min Range: -3.5108

Max Act: 2.8559. Min Act: -0.0000

Data Index: 1500881 (The Pile)

Max Activating Token Index: 63

Text #18

Max Range: 3.5108. Min Range: -3.5108

Max Act: 2.8482. Min Act: -0.0000

Data Index: 28876 (The Pile)

Max Activating Token Index: 610

Text #19

Max Range: 3.5108. Min Range: -3.5108

Max Act: 2.8512. Min Act: -0.0000

Data Index: 1967314 (The Pile)

Max Activating Token Index: 325