Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 48 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.2385. Min Act: -0.0000

Data Index: 1194056 (The Pile)

Max Activating Token Index: 19

Text #1

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.2752. Min Act: -0.0000

Data Index: 1550934 (The Pile)

Max Activating Token Index: 682

Text #2

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.1442. Min Act: -0.0000

Data Index: 1187721 (The Pile)

Max Activating Token Index: 258

Text #3

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.2602. Min Act: -0.0000

Data Index: 713687 (The Pile)

Max Activating Token Index: 760

Text #4

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.1310. Min Act: -0.0000

Data Index: 80662 (The Pile)

Max Activating Token Index: 450

Text #5

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.1474. Min Act: -0.0000

Data Index: 1347847 (The Pile)

Max Activating Token Index: 588

Text #6

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.1355. Min Act: -0.0000

Data Index: 1208577 (The Pile)

Max Activating Token Index: 336

Text #7

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.1554. Min Act: -0.0000

Data Index: 656110 (The Pile)

Max Activating Token Index: 226

Text #8

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.2348. Min Act: -0.0000

Data Index: 37729 (The Pile)

Max Activating Token Index: 663

Text #9

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.2048. Min Act: -0.0000

Data Index: 878367 (The Pile)

Max Activating Token Index: 314

Text #10

Max Range: 1.2752. Min Range: -1.2752

Max Act: 0.9621. Min Act: -0.0000

Data Index: 1119987 (The Pile)

Max Activating Token Index: 814

Text #11

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.0974. Min Act: -0.0000

Data Index: 215719 (The Pile)

Max Activating Token Index: 346

Text #12

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.1633. Min Act: -0.0000

Data Index: 1882745 (The Pile)

Max Activating Token Index: 352

Text #13

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.1440. Min Act: -0.0000

Data Index: 1279280 (The Pile)

Max Activating Token Index: 359

Text #14

Max Range: 1.2752. Min Range: -1.2752

Max Act: 0.9545. Min Act: -0.0000

Data Index: 1855883 (The Pile)

Max Activating Token Index: 593

Text #15

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.0389. Min Act: -0.0000

Data Index: 281753 (The Pile)

Max Activating Token Index: 563

Text #16

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.0102. Min Act: -0.0000

Data Index: 1130386 (The Pile)

Max Activating Token Index: 433

Text #17

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.0375. Min Act: -0.0000

Data Index: 746468 (The Pile)

Max Activating Token Index: 824

Text #18

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.0816. Min Act: -0.0000

Data Index: 800892 (The Pile)

Max Activating Token Index: 115

Text #19

Max Range: 1.2752. Min Range: -1.2752

Max Act: 1.0194. Min Act: -0.0000

Data Index: 262653 (The Pile)

Max Activating Token Index: 926