Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 513 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.8769. Min Act: -0.0000

Data Index: 902796 (The Pile)

Max Activating Token Index: 392

Text #1

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.8471. Min Act: -0.0000

Data Index: 1802411 (The Pile)

Max Activating Token Index: 191

Text #2

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.7375. Min Act: -0.0000

Data Index: 671958 (The Pile)

Max Activating Token Index: 403

Text #3

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.7019. Min Act: -0.0000

Data Index: 65309 (The Pile)

Max Activating Token Index: 10

Text #4

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.6633. Min Act: -0.0000

Data Index: 641724 (The Pile)

Max Activating Token Index: 39

Text #5

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.7362. Min Act: -0.0000

Data Index: 1417459 (The Pile)

Max Activating Token Index: 326

Text #6

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.7105. Min Act: -0.0000

Data Index: 320844 (The Pile)

Max Activating Token Index: 403

Text #7

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.6166. Min Act: -0.0000

Data Index: 1774090 (The Pile)

Max Activating Token Index: 970

Text #8

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.6402. Min Act: -0.0000

Data Index: 1603334 (The Pile)

Max Activating Token Index: 953

Text #9

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.7478. Min Act: -0.0000

Data Index: 331247 (The Pile)

Max Activating Token Index: 249

Text #10

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.6919. Min Act: -0.0000

Data Index: 1548698 (The Pile)

Max Activating Token Index: 210

Text #11

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.6918. Min Act: -0.0000

Data Index: 1682483 (The Pile)

Max Activating Token Index: 574

Text #12

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.6666. Min Act: -0.0000

Data Index: 1338973 (The Pile)

Max Activating Token Index: 31

Text #13

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.7230. Min Act: -0.0000

Data Index: 1373949 (The Pile)

Max Activating Token Index: 248

Text #14

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.6035. Min Act: -0.0000

Data Index: 1871621 (The Pile)

Max Activating Token Index: 203

Text #15

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.6912. Min Act: -0.0000

Data Index: 1977261 (The Pile)

Max Activating Token Index: 781

Text #16

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.6439. Min Act: -0.0000

Data Index: 1287912 (The Pile)

Max Activating Token Index: 61

Text #17

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.6736. Min Act: -0.0000

Data Index: 1768561 (The Pile)

Max Activating Token Index: 464

Text #18

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.6603. Min Act: -0.0000

Data Index: 968209 (The Pile)

Max Activating Token Index: 462

Text #19

Max Range: 0.8769. Min Range: -0.8769

Max Act: 0.6323. Min Act: -0.0000

Data Index: 1969960 (The Pile)

Max Activating Token Index: 784