Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 866 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.6615. Min Act: -0.0000

Data Index: 1947336 (The Pile)

Max Activating Token Index: 692

Text #1

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.3360. Min Act: -0.0000

Data Index: 1991852 (The Pile)

Max Activating Token Index: 920

Text #2

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.2668. Min Act: -0.0000

Data Index: 1997623 (The Pile)

Max Activating Token Index: 201

Text #3

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.1304. Min Act: -0.0000

Data Index: 252115 (The Pile)

Max Activating Token Index: 682

Text #4

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.1345. Min Act: -0.0000

Data Index: 66495 (The Pile)

Max Activating Token Index: 661

Text #5

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.1339. Min Act: -0.0000

Data Index: 1574962 (The Pile)

Max Activating Token Index: 422

Text #6

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.1229. Min Act: -0.0000

Data Index: 370176 (The Pile)

Max Activating Token Index: 541

Text #7

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.0971. Min Act: -0.0000

Data Index: 517222 (The Pile)

Max Activating Token Index: 252

Text #8

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.1746. Min Act: -0.0000

Data Index: 1639953 (The Pile)

Max Activating Token Index: 381

Text #9

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.0739. Min Act: -0.0000

Data Index: 1650077 (The Pile)

Max Activating Token Index: 532

Text #10

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.0576. Min Act: -0.0000

Data Index: 1826474 (The Pile)

Max Activating Token Index: 300

Text #11

Max Range: 1.6615. Min Range: -1.6615

Max Act: 0.9893. Min Act: -0.0000

Data Index: 119357 (The Pile)

Max Activating Token Index: 276

Text #12

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.0268. Min Act: -0.0000

Data Index: 1485233 (The Pile)

Max Activating Token Index: 448

Text #13

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.0383. Min Act: -0.0000

Data Index: 1421174 (The Pile)

Max Activating Token Index: 544

Text #14

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.1301. Min Act: -0.0000

Data Index: 305136 (The Pile)

Max Activating Token Index: 62

Text #15

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.0006. Min Act: -0.0000

Data Index: 1104647 (The Pile)

Max Activating Token Index: 490

Text #16

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.0912. Min Act: -0.0000

Data Index: 936833 (The Pile)

Max Activating Token Index: 266

Text #17

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.0304. Min Act: -0.0000

Data Index: 47835 (The Pile)

Max Activating Token Index: 296

Text #18

Max Range: 1.6615. Min Range: -1.6615

Max Act: 0.8489. Min Act: -0.0000

Data Index: 1597595 (The Pile)

Max Activating Token Index: 689

Text #19

Max Range: 1.6615. Min Range: -1.6615

Max Act: 1.1915. Min Act: -0.0000

Data Index: 1304613 (The Pile)

Max Activating Token Index: 750