Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3576 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.0569. Min Range: -5.0569

Max Act: 5.0569. Min Act: -0.0000

Data Index: 364257 (The Pile)

Max Activating Token Index: 552

Text #1

Max Range: 5.0569. Min Range: -5.0569

Max Act: 4.7621. Min Act: -0.0000

Data Index: 189123 (The Pile)

Max Activating Token Index: 585

Text #2

Max Range: 5.0569. Min Range: -5.0569

Max Act: 4.6612. Min Act: -0.0000

Data Index: 1276565 (The Pile)

Max Activating Token Index: 401

Text #3

Max Range: 5.0569. Min Range: -5.0569

Max Act: 4.4203. Min Act: -0.0000

Data Index: 728018 (The Pile)

Max Activating Token Index: 292

Text #4

Max Range: 5.0569. Min Range: -5.0569

Max Act: 4.0577. Min Act: -0.0000

Data Index: 1768035 (The Pile)

Max Activating Token Index: 1003

Text #5

Max Range: 5.0569. Min Range: -5.0569

Max Act: 4.0543. Min Act: -0.0000

Data Index: 267468 (The Pile)

Max Activating Token Index: 383

Text #6

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.8398. Min Act: -0.0000

Data Index: 1001390 (The Pile)

Max Activating Token Index: 259

Text #7

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.5740. Min Act: -0.0000

Data Index: 1316040 (The Pile)

Max Activating Token Index: 919

Text #8

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.5578. Min Act: -0.0000

Data Index: 1977883 (The Pile)

Max Activating Token Index: 135

Text #9

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.6299. Min Act: -0.0000

Data Index: 826358 (The Pile)

Max Activating Token Index: 111

Text #10

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.6159. Min Act: -0.0000

Data Index: 793325 (The Pile)

Max Activating Token Index: 334

Text #11

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.5620. Min Act: -0.0000

Data Index: 467761 (The Pile)

Max Activating Token Index: 268

Text #12

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.7121. Min Act: -0.0000

Data Index: 119157 (The Pile)

Max Activating Token Index: 146

Text #13

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.5320. Min Act: -0.0000

Data Index: 1966884 (The Pile)

Max Activating Token Index: 3

Text #14

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.6919. Min Act: -0.0000

Data Index: 897667 (The Pile)

Max Activating Token Index: 284

Text #15

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.6790. Min Act: -0.0000

Data Index: 1694033 (The Pile)

Max Activating Token Index: 308

Text #16

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.5264. Min Act: -0.0000

Data Index: 1272282 (The Pile)

Max Activating Token Index: 114

Text #17

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.5163. Min Act: -0.0000

Data Index: 660412 (The Pile)

Max Activating Token Index: 11

Text #18

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.7204. Min Act: -0.0000

Data Index: 1305378 (The Pile)

Max Activating Token Index: 739

Text #19

Max Range: 5.0569. Min Range: -5.0569

Max Act: 3.4235. Min Act: -0.0000

Data Index: 1568207 (The Pile)

Max Activating Token Index: 478