Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 441 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.7472. Min Act: -0.0000

Data Index: 1157573 (The Pile)

Max Activating Token Index: 447

Text #1

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.6922. Min Act: -0.0000

Data Index: 1983269 (The Pile)

Max Activating Token Index: 1007

Text #2

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.6459. Min Act: -0.0000

Data Index: 51867 (The Pile)

Max Activating Token Index: 169

Text #3

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.4313. Min Act: -0.0000

Data Index: 241415 (The Pile)

Max Activating Token Index: 605

Text #4

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.5815. Min Act: -0.0000

Data Index: 144782 (The Pile)

Max Activating Token Index: 279

Text #5

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.5206. Min Act: -0.0000

Data Index: 425246 (The Pile)

Max Activating Token Index: 393

Text #6

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.3514. Min Act: -0.0000

Data Index: 614036 (The Pile)

Max Activating Token Index: 863

Text #7

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.4764. Min Act: -0.0000

Data Index: 1517503 (The Pile)

Max Activating Token Index: 517

Text #8

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.3052. Min Act: -0.0000

Data Index: 1241186 (The Pile)

Max Activating Token Index: 935

Text #9

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.5808. Min Act: -0.0000

Data Index: 810051 (The Pile)

Max Activating Token Index: 704

Text #10

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.5257. Min Act: -0.0000

Data Index: 548788 (The Pile)

Max Activating Token Index: 728

Text #11

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.5532. Min Act: -0.0000

Data Index: 1900873 (The Pile)

Max Activating Token Index: 981

Text #12

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.3526. Min Act: -0.0000

Data Index: 184299 (The Pile)

Max Activating Token Index: 654

Text #13

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.1932. Min Act: -0.0000

Data Index: 202270 (The Pile)

Max Activating Token Index: 561

Text #14

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.1729. Min Act: -0.0000

Data Index: 1943124 (The Pile)

Max Activating Token Index: 284

Text #15

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.1485. Min Act: -0.0000

Data Index: 606475 (The Pile)

Max Activating Token Index: 870

Text #16

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.3269. Min Act: -0.0000

Data Index: 1725377 (The Pile)

Max Activating Token Index: 51

Text #17

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.3301. Min Act: -0.0000

Data Index: 1773722 (The Pile)

Max Activating Token Index: 785

Text #18

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.3117. Min Act: -0.0000

Data Index: 1750108 (The Pile)

Max Activating Token Index: 277

Text #19

Max Range: 3.7472. Min Range: -3.7472

Max Act: 3.1722. Min Act: -0.0000

Data Index: 1781590 (The Pile)

Max Activating Token Index: 716