Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4539 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.3154. Min Act: -0.0000

Data Index: 1809214 (The Pile)

Max Activating Token Index: 651

Text #1

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.3207. Min Act: -0.0000

Data Index: 137781 (The Pile)

Max Activating Token Index: 16

Text #2

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.3510. Min Act: -0.0000

Data Index: 600458 (The Pile)

Max Activating Token Index: 807

Text #3

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2819. Min Act: -0.0000

Data Index: 389597 (The Pile)

Max Activating Token Index: 901

Text #4

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2652. Min Act: -0.0000

Data Index: 1116499 (The Pile)

Max Activating Token Index: 567

Text #5

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2697. Min Act: -0.0000

Data Index: 975011 (The Pile)

Max Activating Token Index: 306

Text #6

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2642. Min Act: -0.0000

Data Index: 1780178 (The Pile)

Max Activating Token Index: 187

Text #7

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2649. Min Act: -0.0000

Data Index: 1816655 (The Pile)

Max Activating Token Index: 181

Text #8

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2628. Min Act: -0.0000

Data Index: 618523 (The Pile)

Max Activating Token Index: 709

Text #9

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2616. Min Act: -0.0000

Data Index: 299748 (The Pile)

Max Activating Token Index: 450

Text #10

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2624. Min Act: -0.0000

Data Index: 42264 (The Pile)

Max Activating Token Index: 656

Text #11

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2734. Min Act: -0.0000

Data Index: 920132 (The Pile)

Max Activating Token Index: 122

Text #12

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2455. Min Act: -0.0000

Data Index: 268497 (The Pile)

Max Activating Token Index: 373

Text #13

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2549. Min Act: -0.0000

Data Index: 935749 (The Pile)

Max Activating Token Index: 230

Text #14

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2436. Min Act: -0.0000

Data Index: 1437762 (The Pile)

Max Activating Token Index: 514

Text #15

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2434. Min Act: -0.0000

Data Index: 189312 (The Pile)

Max Activating Token Index: 624

Text #16

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2440. Min Act: -0.0000

Data Index: 1781066 (The Pile)

Max Activating Token Index: 488

Text #17

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2497. Min Act: -0.0000

Data Index: 267527 (The Pile)

Max Activating Token Index: 34

Text #18

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2667. Min Act: -0.0000

Data Index: 1382944 (The Pile)

Max Activating Token Index: 950

Text #19

Max Range: 0.3510. Min Range: -0.3510

Max Act: 0.2263. Min Act: -0.0000

Data Index: 1918540 (The Pile)

Max Activating Token Index: 407