Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3520 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 4.3652. Min Range: -4.3652

Max Act: 4.3652. Min Act: -0.0000

Data Index: 657769 (The Pile)

Max Activating Token Index: 747

Text #1

Max Range: 4.3652. Min Range: -4.3652

Max Act: 4.3519. Min Act: -0.0000

Data Index: 1058520 (The Pile)

Max Activating Token Index: 351

Text #2

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.7568. Min Act: -0.0000

Data Index: 1480670 (The Pile)

Max Activating Token Index: 671

Text #3

Max Range: 4.3652. Min Range: -4.3652

Max Act: 4.2624. Min Act: -0.0000

Data Index: 1204660 (The Pile)

Max Activating Token Index: 389

Text #4

Max Range: 4.3652. Min Range: -4.3652

Max Act: 4.0076. Min Act: -0.0000

Data Index: 526635 (The Pile)

Max Activating Token Index: 92

Text #5

Max Range: 4.3652. Min Range: -4.3652

Max Act: 4.3063. Min Act: -0.0000

Data Index: 1366015 (The Pile)

Max Activating Token Index: 853

Text #6

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.6013. Min Act: -0.0000

Data Index: 1201810 (The Pile)

Max Activating Token Index: 131

Text #7

Max Range: 4.3652. Min Range: -4.3652

Max Act: 4.1110. Min Act: -0.0000

Data Index: 1101021 (The Pile)

Max Activating Token Index: 284

Text #8

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.7221. Min Act: -0.0000

Data Index: 202486 (The Pile)

Max Activating Token Index: 700

Text #9

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.7598. Min Act: -0.0000

Data Index: 1114283 (The Pile)

Max Activating Token Index: 386

Text #10

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.8306. Min Act: -0.0000

Data Index: 1195531 (The Pile)

Max Activating Token Index: 633

Text #11

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.6976. Min Act: -0.0000

Data Index: 804281 (The Pile)

Max Activating Token Index: 36

Text #12

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.4658. Min Act: -0.0000

Data Index: 1043200 (The Pile)

Max Activating Token Index: 48

Text #13

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.7776. Min Act: -0.0000

Data Index: 771424 (The Pile)

Max Activating Token Index: 90

Text #14

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.7756. Min Act: -0.0000

Data Index: 1793256 (The Pile)

Max Activating Token Index: 259

Text #15

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.8724. Min Act: -0.0000

Data Index: 2567 (The Pile)

Max Activating Token Index: 123

Text #16

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.6801. Min Act: -0.0000

Data Index: 1128216 (The Pile)

Max Activating Token Index: 567

Text #17

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.3979. Min Act: -0.0000

Data Index: 808880 (The Pile)

Max Activating Token Index: 742

Text #18

Max Range: 4.3652. Min Range: -4.3652

Max Act: 3.6781. Min Act: -0.0000

Data Index: 620642 (The Pile)

Max Activating Token Index: 355

Text #19

Max Range: 4.3652. Min Range: -4.3652

Max Act: 4.1278. Min Act: -0.0000

Data Index: 632061 (The Pile)

Max Activating Token Index: 867