Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4679 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.9152. Min Act: -0.0000

Data Index: 757314 (The Pile)

Max Activating Token Index: 524

Text #1

Max Range: 7.2721. Min Range: -7.2721

Max Act: 7.2721. Min Act: -0.0000

Data Index: 1650264 (The Pile)

Max Activating Token Index: 409

Text #2

Max Range: 7.2721. Min Range: -7.2721

Max Act: 7.1600. Min Act: -0.0000

Data Index: 753677 (The Pile)

Max Activating Token Index: 748

Text #3

Max Range: 7.2721. Min Range: -7.2721

Max Act: 7.1300. Min Act: -0.0000

Data Index: 729088 (The Pile)

Max Activating Token Index: 117

Text #4

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.6306. Min Act: -0.0000

Data Index: 1304485 (The Pile)

Max Activating Token Index: 310

Text #5

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.9265. Min Act: -0.0000

Data Index: 288547 (The Pile)

Max Activating Token Index: 652

Text #6

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.7425. Min Act: -0.0000

Data Index: 578555 (The Pile)

Max Activating Token Index: 191

Text #7

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.7379. Min Act: -0.0000

Data Index: 1934505 (The Pile)

Max Activating Token Index: 655

Text #8

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.8830. Min Act: -0.0000

Data Index: 1136682 (The Pile)

Max Activating Token Index: 36

Text #9

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.8539. Min Act: -0.0000

Data Index: 1434787 (The Pile)

Max Activating Token Index: 39

Text #10

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.7463. Min Act: -0.0000

Data Index: 152163 (The Pile)

Max Activating Token Index: 332

Text #11

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.9286. Min Act: -0.0000

Data Index: 575808 (The Pile)

Max Activating Token Index: 217

Text #12

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.6940. Min Act: -0.0000

Data Index: 1268957 (The Pile)

Max Activating Token Index: 258

Text #13

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.6839. Min Act: -0.0000

Data Index: 1974493 (The Pile)

Max Activating Token Index: 337

Text #14

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.8560. Min Act: -0.0000

Data Index: 1907332 (The Pile)

Max Activating Token Index: 518

Text #15

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.5896. Min Act: -0.0000

Data Index: 782765 (The Pile)

Max Activating Token Index: 800

Text #16

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.4686. Min Act: -0.0000

Data Index: 62078 (The Pile)

Max Activating Token Index: 571

Text #17

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.5991. Min Act: -0.0000

Data Index: 1355132 (The Pile)

Max Activating Token Index: 134

Text #18

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.7399. Min Act: -0.0000

Data Index: 1002534 (The Pile)

Max Activating Token Index: 662

Text #19

Max Range: 7.2721. Min Range: -7.2721

Max Act: 6.5450. Min Act: -0.0000

Data Index: 1471254 (The Pile)

Max Activating Token Index: 382