Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 115 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 4.4443. Min Range: -4.4443

Max Act: 4.4443. Min Act: -0.0000

Data Index: 1012206 (The Pile)

Max Activating Token Index: 568

Text #1

Max Range: 4.4443. Min Range: -4.4443

Max Act: 4.0320. Min Act: -0.0000

Data Index: 1208672 (The Pile)

Max Activating Token Index: 946

Text #2

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.6774. Min Act: -0.0000

Data Index: 1334464 (The Pile)

Max Activating Token Index: 686

Text #3

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.9341. Min Act: -0.0000

Data Index: 677376 (The Pile)

Max Activating Token Index: 837

Text #4

Max Range: 4.4443. Min Range: -4.4443

Max Act: 4.0638. Min Act: -0.0000

Data Index: 1990642 (The Pile)

Max Activating Token Index: 530

Text #5

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.9407. Min Act: -0.0000

Data Index: 1525930 (The Pile)

Max Activating Token Index: 536

Text #6

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.7475. Min Act: -0.0000

Data Index: 825255 (The Pile)

Max Activating Token Index: 429

Text #7

Max Range: 4.4443. Min Range: -4.4443

Max Act: 4.0690. Min Act: -0.0000

Data Index: 1884837 (The Pile)

Max Activating Token Index: 485

Text #8

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.9913. Min Act: -0.0000

Data Index: 412631 (The Pile)

Max Activating Token Index: 574

Text #9

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.8061. Min Act: -0.0000

Data Index: 869353 (The Pile)

Max Activating Token Index: 445

Text #10

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.6383. Min Act: -0.0000

Data Index: 164995 (The Pile)

Max Activating Token Index: 635

Text #11

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.4993. Min Act: -0.0000

Data Index: 187360 (The Pile)

Max Activating Token Index: 404

Text #12

Max Range: 4.4443. Min Range: -4.4443

Max Act: 4.1274. Min Act: -0.0000

Data Index: 1495396 (The Pile)

Max Activating Token Index: 826

Text #13

Max Range: 4.4443. Min Range: -4.4443

Max Act: 4.3170. Min Act: -0.0000

Data Index: 1385761 (The Pile)

Max Activating Token Index: 673

Text #14

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.7877. Min Act: -0.0000

Data Index: 1725091 (The Pile)

Max Activating Token Index: 673

Text #15

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.3059. Min Act: -0.0000

Data Index: 510925 (The Pile)

Max Activating Token Index: 666

Text #16

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.6052. Min Act: -0.0000

Data Index: 1694674 (The Pile)

Max Activating Token Index: 598

Text #17

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.4158. Min Act: -0.0000

Data Index: 503304 (The Pile)

Max Activating Token Index: 960

Text #18

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.7616. Min Act: -0.0000

Data Index: 916343 (The Pile)

Max Activating Token Index: 831

Text #19

Max Range: 4.4443. Min Range: -4.4443

Max Act: 3.8413. Min Act: -0.0000

Data Index: 1360721 (The Pile)

Max Activating Token Index: 677