Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 828 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.1456. Min Act: -0.0000

Data Index: 962738 (The Pile)

Max Activating Token Index: 679

Text #1

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.1398. Min Act: -0.0000

Data Index: 1500936 (The Pile)

Max Activating Token Index: 557

Text #2

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.1047. Min Act: -0.0000

Data Index: 687460 (The Pile)

Max Activating Token Index: 1018

Text #3

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.1076. Min Act: -0.0000

Data Index: 1422247 (The Pile)

Max Activating Token Index: 363

Text #4

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0976. Min Act: -0.0000

Data Index: 89352 (The Pile)

Max Activating Token Index: 665

Text #5

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0960. Min Act: -0.0000

Data Index: 1409393 (The Pile)

Max Activating Token Index: 453

Text #6

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0865. Min Act: -0.0000

Data Index: 1334262 (The Pile)

Max Activating Token Index: 732

Text #7

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0885. Min Act: -0.0000

Data Index: 1719817 (The Pile)

Max Activating Token Index: 808

Text #8

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.1004. Min Act: -0.0000

Data Index: 1663714 (The Pile)

Max Activating Token Index: 264

Text #9

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.1057. Min Act: -0.0000

Data Index: 1902022 (The Pile)

Max Activating Token Index: 286

Text #10

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0971. Min Act: -0.0001

Data Index: 1158926 (The Pile)

Max Activating Token Index: 715

Text #11

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.1035. Min Act: -0.0000

Data Index: 27051 (The Pile)

Max Activating Token Index: 87

Text #12

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0827. Min Act: -0.0000

Data Index: 1830285 (The Pile)

Max Activating Token Index: 539

Text #13

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0986. Min Act: -0.0000

Data Index: 1173599 (The Pile)

Max Activating Token Index: 461

Text #14

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0864. Min Act: -0.0000

Data Index: 1589197 (The Pile)

Max Activating Token Index: 974

Text #15

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0899. Min Act: -0.0000

Data Index: 910919 (The Pile)

Max Activating Token Index: 833

Text #16

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0929. Min Act: -0.0000

Data Index: 915647 (The Pile)

Max Activating Token Index: 698

Text #17

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0978. Min Act: -0.0000

Data Index: 209609 (The Pile)

Max Activating Token Index: 984

Text #18

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0798. Min Act: -0.0000

Data Index: 218825 (The Pile)

Max Activating Token Index: 651

Text #19

Max Range: 0.1456. Min Range: -0.1456

Max Act: 0.0919. Min Act: -0.0000

Data Index: 604118 (The Pile)

Max Activating Token Index: 400