Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 254 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 4.1091. Min Range: -4.1091

Max Act: 4.1091. Min Act: -0.0000

Data Index: 1512882 (The Pile)

Max Activating Token Index: 69

Text #1

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.9033. Min Act: -0.0000

Data Index: 1801318 (The Pile)

Max Activating Token Index: 497

Text #2

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.7891. Min Act: -0.0000

Data Index: 1527851 (The Pile)

Max Activating Token Index: 146

Text #3

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.6339. Min Act: -0.0000

Data Index: 1446016 (The Pile)

Max Activating Token Index: 814

Text #4

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.7887. Min Act: -0.0000

Data Index: 1667555 (The Pile)

Max Activating Token Index: 26

Text #5

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.6044. Min Act: -0.0000

Data Index: 1080435 (The Pile)

Max Activating Token Index: 175

Text #6

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.5973. Min Act: -0.0000

Data Index: 1314732 (The Pile)

Max Activating Token Index: 429

Text #7

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.6873. Min Act: -0.0000

Data Index: 1051204 (The Pile)

Max Activating Token Index: 409

Text #8

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.6767. Min Act: -0.0000

Data Index: 611142 (The Pile)

Max Activating Token Index: 544

Text #9

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.1682. Min Act: -0.0000

Data Index: 1755880 (The Pile)

Max Activating Token Index: 814

Text #10

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.2940. Min Act: -0.0000

Data Index: 477822 (The Pile)

Max Activating Token Index: 161

Text #11

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.7789. Min Act: -0.0000

Data Index: 243146 (The Pile)

Max Activating Token Index: 987

Text #12

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.5965. Min Act: -0.0000

Data Index: 680355 (The Pile)

Max Activating Token Index: 615

Text #13

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.4381. Min Act: -0.0000

Data Index: 1807781 (The Pile)

Max Activating Token Index: 117

Text #14

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.4014. Min Act: -0.0000

Data Index: 720048 (The Pile)

Max Activating Token Index: 418

Text #15

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.3925. Min Act: -0.0000

Data Index: 843948 (The Pile)

Max Activating Token Index: 318

Text #16

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.2368. Min Act: -0.0000

Data Index: 176081 (The Pile)

Max Activating Token Index: 462

Text #17

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.4969. Min Act: -0.0000

Data Index: 757263 (The Pile)

Max Activating Token Index: 490

Text #18

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.4716. Min Act: -0.0000

Data Index: 1145638 (The Pile)

Max Activating Token Index: 614

Text #19

Max Range: 4.1091. Min Range: -4.1091

Max Act: 3.3102. Min Act: -0.0000

Data Index: 659991 (The Pile)

Max Activating Token Index: 188