Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 837 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 7.9249. Min Range: -7.9249

Max Act: 7.8793. Min Act: -0.0000

Data Index: 1532131 (The Pile)

Max Activating Token Index: 693

Text #1

Max Range: 7.9249. Min Range: -7.9249

Max Act: 7.6911. Min Act: -0.0000

Data Index: 1968545 (The Pile)

Max Activating Token Index: 220

Text #2

Max Range: 7.9249. Min Range: -7.9249

Max Act: 7.9249. Min Act: -0.0000

Data Index: 347529 (The Pile)

Max Activating Token Index: 236

Text #3

Max Range: 7.9249. Min Range: -7.9249

Max Act: 7.3880. Min Act: -0.0000

Data Index: 431709 (The Pile)

Max Activating Token Index: 832

Text #4

Max Range: 7.9249. Min Range: -7.9249

Max Act: 7.1480. Min Act: -0.0000

Data Index: 86414 (The Pile)

Max Activating Token Index: 549

Text #5

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.5919. Min Act: -0.0000

Data Index: 379711 (The Pile)

Max Activating Token Index: 643

Text #6

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.8643. Min Act: -0.0000

Data Index: 1573043 (The Pile)

Max Activating Token Index: 307

Text #7

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.8426. Min Act: -0.0000

Data Index: 1744188 (The Pile)

Max Activating Token Index: 365

Text #8

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.4334. Min Act: -0.0000

Data Index: 1238640 (The Pile)

Max Activating Token Index: 516

Text #9

Max Range: 7.9249. Min Range: -7.9249

Max Act: 5.8854. Min Act: -0.0000

Data Index: 964153 (The Pile)

Max Activating Token Index: 802

Text #10

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.0982. Min Act: -0.0000

Data Index: 218121 (The Pile)

Max Activating Token Index: 997

Text #11

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.5820. Min Act: -0.0000

Data Index: 1044268 (The Pile)

Max Activating Token Index: 973

Text #12

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.2147. Min Act: -0.0000

Data Index: 95208 (The Pile)

Max Activating Token Index: 304

Text #13

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.1721. Min Act: -0.0000

Data Index: 324046 (The Pile)

Max Activating Token Index: 215

Text #14

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.1765. Min Act: -0.0000

Data Index: 1620481 (The Pile)

Max Activating Token Index: 746

Text #15

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.3281. Min Act: -0.0000

Data Index: 1963792 (The Pile)

Max Activating Token Index: 388

Text #16

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.3327. Min Act: -0.0000

Data Index: 1658327 (The Pile)

Max Activating Token Index: 214

Text #17

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.5006. Min Act: -0.0000

Data Index: 1866148 (The Pile)

Max Activating Token Index: 702

Text #18

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.0379. Min Act: -0.0000

Data Index: 550851 (The Pile)

Max Activating Token Index: 12

Text #19

Max Range: 7.9249. Min Range: -7.9249

Max Act: 6.6001. Min Act: -0.0000

Data Index: 637463 (The Pile)

Max Activating Token Index: 449