Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 1384 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.2285. Min Range: -5.2285

Max Act: 5.2285. Min Act: -0.0000

Data Index: 407253 (The Pile)

Max Activating Token Index: 61

Text #1

Max Range: 5.2285. Min Range: -5.2285

Max Act: 4.3992. Min Act: -0.0000

Data Index: 1107449 (The Pile)

Max Activating Token Index: 399

Text #2

Max Range: 5.2285. Min Range: -5.2285

Max Act: 4.3896. Min Act: -0.0000

Data Index: 1535956 (The Pile)

Max Activating Token Index: 31

Text #3

Max Range: 5.2285. Min Range: -5.2285

Max Act: 4.3684. Min Act: -0.0000

Data Index: 1931715 (The Pile)

Max Activating Token Index: 11

Text #4

Max Range: 5.2285. Min Range: -5.2285

Max Act: 4.3523. Min Act: -0.0000

Data Index: 711476 (The Pile)

Max Activating Token Index: 80

Text #5

Max Range: 5.2285. Min Range: -5.2285

Max Act: 4.2481. Min Act: -0.0000

Data Index: 1988601 (The Pile)

Max Activating Token Index: 32

Text #6

Max Range: 5.2285. Min Range: -5.2285

Max Act: 4.1745. Min Act: -0.0000

Data Index: 963244 (The Pile)

Max Activating Token Index: 5

Text #7

Max Range: 5.2285. Min Range: -5.2285

Max Act: 4.1531. Min Act: -0.0000

Data Index: 947004 (The Pile)

Max Activating Token Index: 879

Text #8

Max Range: 5.2285. Min Range: -5.2285

Max Act: 3.9251. Min Act: -0.0000

Data Index: 775059 (The Pile)

Max Activating Token Index: 77

Text #9

Max Range: 5.2285. Min Range: -5.2285

Max Act: 3.8836. Min Act: -0.0000

Data Index: 995125 (The Pile)

Max Activating Token Index: 25

Text #10

Max Range: 5.2285. Min Range: -5.2285

Max Act: 4.0346. Min Act: -0.0000

Data Index: 107030 (The Pile)

Max Activating Token Index: 11

Text #11

Max Range: 5.2285. Min Range: -5.2285

Max Act: 4.0455. Min Act: -0.0000

Data Index: 793095 (The Pile)

Max Activating Token Index: 49

Text #12

Max Range: 5.2285. Min Range: -5.2285

Max Act: 3.8056. Min Act: -0.0000

Data Index: 1103932 (The Pile)

Max Activating Token Index: 98

Text #13

Max Range: 5.2285. Min Range: -5.2285

Max Act: 3.7340. Min Act: -0.0000

Data Index: 1360227 (The Pile)

Max Activating Token Index: 647

Text #14

Max Range: 5.2285. Min Range: -5.2285

Max Act: 3.6593. Min Act: -0.0000

Data Index: 1667646 (The Pile)

Max Activating Token Index: 29

Text #15

Max Range: 5.2285. Min Range: -5.2285

Max Act: 3.5023. Min Act: -0.0000

Data Index: 704976 (The Pile)

Max Activating Token Index: 399

Text #16

Max Range: 5.2285. Min Range: -5.2285

Max Act: 3.4889. Min Act: -0.0000

Data Index: 785693 (The Pile)

Max Activating Token Index: 7

Text #17

Max Range: 5.2285. Min Range: -5.2285

Max Act: 3.5903. Min Act: -0.0000

Data Index: 1593371 (The Pile)

Max Activating Token Index: 996

Text #18

Max Range: 5.2285. Min Range: -5.2285

Max Act: 3.5889. Min Act: -0.0000

Data Index: 1196402 (The Pile)

Max Activating Token Index: 12

Text #19

Max Range: 5.2285. Min Range: -5.2285

Max Act: 3.7123. Min Act: -0.0000

Data Index: 1701503 (The Pile)

Max Activating Token Index: 999