Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3294 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.1572. Min Range: -5.1572

Max Act: 5.1572. Min Act: -0.0000

Data Index: 1176954 (The Pile)

Max Activating Token Index: 797

Text #1

Max Range: 5.1572. Min Range: -5.1572

Max Act: 4.5726. Min Act: -0.0000

Data Index: 428602 (The Pile)

Max Activating Token Index: 264

Text #2

Max Range: 5.1572. Min Range: -5.1572

Max Act: 4.4465. Min Act: -0.0000

Data Index: 378429 (The Pile)

Max Activating Token Index: 973

Text #3

Max Range: 5.1572. Min Range: -5.1572

Max Act: 4.1496. Min Act: -0.0000

Data Index: 266238 (The Pile)

Max Activating Token Index: 540

Text #4

Max Range: 5.1572. Min Range: -5.1572

Max Act: 4.4785. Min Act: -0.0000

Data Index: 1410054 (The Pile)

Max Activating Token Index: 925

Text #5

Max Range: 5.1572. Min Range: -5.1572

Max Act: 4.5017. Min Act: -0.0000

Data Index: 379708 (The Pile)

Max Activating Token Index: 771

Text #6

Max Range: 5.1572. Min Range: -5.1572

Max Act: 4.2843. Min Act: -0.0000

Data Index: 1287756 (The Pile)

Max Activating Token Index: 860

Text #7

Max Range: 5.1572. Min Range: -5.1572

Max Act: 3.7937. Min Act: -0.0000

Data Index: 1977808 (The Pile)

Max Activating Token Index: 591

Text #8

Max Range: 5.1572. Min Range: -5.1572

Max Act: 4.2241. Min Act: -0.0000

Data Index: 1314812 (The Pile)

Max Activating Token Index: 683

Text #9

Max Range: 5.1572. Min Range: -5.1572

Max Act: 3.8763. Min Act: -0.0000

Data Index: 1632883 (The Pile)

Max Activating Token Index: 836

Text #10

Max Range: 5.1572. Min Range: -5.1572

Max Act: 3.8710. Min Act: -0.0000

Data Index: 950500 (The Pile)

Max Activating Token Index: 965

Text #11

Max Range: 5.1572. Min Range: -5.1572

Max Act: 4.0045. Min Act: -0.0000

Data Index: 1720742 (The Pile)

Max Activating Token Index: 843

Text #12

Max Range: 5.1572. Min Range: -5.1572

Max Act: 3.8190. Min Act: -0.0000

Data Index: 901341 (The Pile)

Max Activating Token Index: 919

Text #13

Max Range: 5.1572. Min Range: -5.1572

Max Act: 3.9493. Min Act: -0.0000

Data Index: 1412964 (The Pile)

Max Activating Token Index: 892

Text #14

Max Range: 5.1572. Min Range: -5.1572

Max Act: 4.1065. Min Act: -0.0000

Data Index: 90645 (The Pile)

Max Activating Token Index: 612

Text #15

Max Range: 5.1572. Min Range: -5.1572

Max Act: 3.4508. Min Act: -0.0000

Data Index: 308401 (The Pile)

Max Activating Token Index: 703

Text #16

Max Range: 5.1572. Min Range: -5.1572

Max Act: 3.8681. Min Act: -0.0000

Data Index: 230484 (The Pile)

Max Activating Token Index: 1010

Text #17

Max Range: 5.1572. Min Range: -5.1572

Max Act: 4.1282. Min Act: -0.0000

Data Index: 1745340 (The Pile)

Max Activating Token Index: 910

Text #18

Max Range: 5.1572. Min Range: -5.1572

Max Act: 3.4564. Min Act: -0.0000

Data Index: 95523 (The Pile)

Max Activating Token Index: 980

Text #19

Max Range: 5.1572. Min Range: -5.1572

Max Act: 3.4598. Min Act: -0.0000

Data Index: 1016846 (The Pile)

Max Activating Token Index: 828