Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 162 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.7948. Min Act: -0.0000

Data Index: 1581650 (The Pile)

Max Activating Token Index: 947

Text #1

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.7189. Min Act: -0.0000

Data Index: 1069088 (The Pile)

Max Activating Token Index: 736

Text #2

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.6964. Min Act: -0.0000

Data Index: 402748 (The Pile)

Max Activating Token Index: 110

Text #3

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.6445. Min Act: -0.0000

Data Index: 809642 (The Pile)

Max Activating Token Index: 390

Text #4

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.6425. Min Act: -0.0000

Data Index: 578353 (The Pile)

Max Activating Token Index: 347

Text #5

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.6459. Min Act: -0.0000

Data Index: 572822 (The Pile)

Max Activating Token Index: 469

Text #6

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.5832. Min Act: -0.0000

Data Index: 1571645 (The Pile)

Max Activating Token Index: 746

Text #7

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.5458. Min Act: -0.0000

Data Index: 1029926 (The Pile)

Max Activating Token Index: 35

Text #8

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.6038. Min Act: -0.0000

Data Index: 177782 (The Pile)

Max Activating Token Index: 58

Text #9

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.5996. Min Act: -0.0000

Data Index: 1468549 (The Pile)

Max Activating Token Index: 37

Text #10

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.5940. Min Act: -0.0000

Data Index: 819880 (The Pile)

Max Activating Token Index: 44

Text #11

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.5946. Min Act: -0.0000

Data Index: 554131 (The Pile)

Max Activating Token Index: 712

Text #12

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.5713. Min Act: -0.0000

Data Index: 1132819 (The Pile)

Max Activating Token Index: 591

Text #13

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.5893. Min Act: -0.0000

Data Index: 840198 (The Pile)

Max Activating Token Index: 505

Text #14

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.6090. Min Act: -0.0000

Data Index: 222227 (The Pile)

Max Activating Token Index: 94

Text #15

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.5466. Min Act: -0.0000

Data Index: 1416022 (The Pile)

Max Activating Token Index: 911

Text #16

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.5797. Min Act: -0.0000

Data Index: 543487 (The Pile)

Max Activating Token Index: 13

Text #17

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.5777. Min Act: -0.0000

Data Index: 1739507 (The Pile)

Max Activating Token Index: 248

Text #18

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.5885. Min Act: -0.0000

Data Index: 1456679 (The Pile)

Max Activating Token Index: 71

Text #19

Max Range: 0.7948. Min Range: -0.7948

Max Act: 0.5326. Min Act: -0.0000

Data Index: 1805177 (The Pile)

Max Activating Token Index: 23