Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 687 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.8750. Min Act: -0.0000

Data Index: 763129 (The Pile)

Max Activating Token Index: 991

Text #1

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.8243. Min Act: -0.0000

Data Index: 1484055 (The Pile)

Max Activating Token Index: 412

Text #2

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.8517. Min Act: -0.0000

Data Index: 544707 (The Pile)

Max Activating Token Index: 472

Text #3

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.7910. Min Act: -0.0000

Data Index: 591790 (The Pile)

Max Activating Token Index: 617

Text #4

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.7276. Min Act: -0.0000

Data Index: 1603566 (The Pile)

Max Activating Token Index: 682

Text #5

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.7286. Min Act: -0.0000

Data Index: 1018573 (The Pile)

Max Activating Token Index: 685

Text #6

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.6744. Min Act: -0.0000

Data Index: 1246728 (The Pile)

Max Activating Token Index: 788

Text #7

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.7011. Min Act: -0.0000

Data Index: 505302 (The Pile)

Max Activating Token Index: 827

Text #8

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.6631. Min Act: 0.0000

Data Index: 1542308 (The Pile)

Max Activating Token Index: 421

Text #9

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.7105. Min Act: -0.0000

Data Index: 428625 (The Pile)

Max Activating Token Index: 936

Text #10

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.6098. Min Act: 0.0000

Data Index: 1750611 (The Pile)

Max Activating Token Index: 1002

Text #11

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.5943. Min Act: -0.0000

Data Index: 1991517 (The Pile)

Max Activating Token Index: 689

Text #12

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.6056. Min Act: -0.0000

Data Index: 1001677 (The Pile)

Max Activating Token Index: 955

Text #13

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.5988. Min Act: 0.0000

Data Index: 759449 (The Pile)

Max Activating Token Index: 118

Text #14

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.6001. Min Act: -0.0000

Data Index: 108660 (The Pile)

Max Activating Token Index: 808

Text #15

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.5903. Min Act: -0.0000

Data Index: 1534635 (The Pile)

Max Activating Token Index: 270

Text #16

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.5268. Min Act: -0.0000

Data Index: 1889508 (The Pile)

Max Activating Token Index: 470

Text #17

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.5697. Min Act: 0.0000

Data Index: 1425224 (The Pile)

Max Activating Token Index: 450

Text #18

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.5475. Min Act: 0.0000

Data Index: 930589 (The Pile)

Max Activating Token Index: 323

Text #19

Max Range: 0.8750. Min Range: -0.8750

Max Act: 0.5194. Min Act: -0.0000

Data Index: 1258002 (The Pile)

Max Activating Token Index: 161