Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 625 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.1413. Min Range: -3.1413

Max Act: 3.1150. Min Act: -0.0000

Data Index: 424628 (The Pile)

Max Activating Token Index: 486

Text #1

Max Range: 3.1413. Min Range: -3.1413

Max Act: 3.1413. Min Act: -0.0000

Data Index: 1079475 (The Pile)

Max Activating Token Index: 744

Text #2

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.8894. Min Act: -0.0000

Data Index: 187976 (The Pile)

Max Activating Token Index: 603

Text #3

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.6961. Min Act: -0.0000

Data Index: 1044786 (The Pile)

Max Activating Token Index: 679

Text #4

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.7021. Min Act: -0.0000

Data Index: 383131 (The Pile)

Max Activating Token Index: 900

Text #5

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.6979. Min Act: -0.0000

Data Index: 1335685 (The Pile)

Max Activating Token Index: 210

Text #6

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.7986. Min Act: -0.0000

Data Index: 144059 (The Pile)

Max Activating Token Index: 69

Text #7

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.7384. Min Act: -0.0000

Data Index: 1242921 (The Pile)

Max Activating Token Index: 725

Text #8

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.4933. Min Act: -0.0000

Data Index: 1158668 (The Pile)

Max Activating Token Index: 892

Text #9

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.2441. Min Act: -0.0000

Data Index: 1927301 (The Pile)

Max Activating Token Index: 633

Text #10

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.5046. Min Act: -0.0000

Data Index: 868271 (The Pile)

Max Activating Token Index: 796

Text #11

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.9620. Min Act: -0.0000

Data Index: 664171 (The Pile)

Max Activating Token Index: 714

Text #12

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.5017. Min Act: -0.0000

Data Index: 1271660 (The Pile)

Max Activating Token Index: 228

Text #13

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.4819. Min Act: -0.0000

Data Index: 1692934 (The Pile)

Max Activating Token Index: 483

Text #14

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.5548. Min Act: -0.0000

Data Index: 1490682 (The Pile)

Max Activating Token Index: 541

Text #15

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.6701. Min Act: -0.0000

Data Index: 1058248 (The Pile)

Max Activating Token Index: 764

Text #16

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.4138. Min Act: -0.0000

Data Index: 1797237 (The Pile)

Max Activating Token Index: 932

Text #17

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.0926. Min Act: -0.0000

Data Index: 1450691 (The Pile)

Max Activating Token Index: 228

Text #18

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.4322. Min Act: -0.0000

Data Index: 1579860 (The Pile)

Max Activating Token Index: 255

Text #19

Max Range: 3.1413. Min Range: -3.1413

Max Act: 2.0878. Min Act: -0.0000

Data Index: 353870 (The Pile)

Max Activating Token Index: 533