Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 17 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.1806. Min Range: -3.1806

Max Act: 3.1806. Min Act: -0.0000

Data Index: 371137 (The Pile)

Max Activating Token Index: 743

Text #1

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.9834. Min Act: -0.0000

Data Index: 1682559 (The Pile)

Max Activating Token Index: 983

Text #2

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.8924. Min Act: -0.0000

Data Index: 1171209 (The Pile)

Max Activating Token Index: 721

Text #3

Max Range: 3.1806. Min Range: -3.1806

Max Act: 3.0292. Min Act: -0.0000

Data Index: 1337624 (The Pile)

Max Activating Token Index: 449

Text #4

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.7982. Min Act: -0.0000

Data Index: 1159973 (The Pile)

Max Activating Token Index: 786

Text #5

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.9862. Min Act: -0.0000

Data Index: 817921 (The Pile)

Max Activating Token Index: 1009

Text #6

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.6506. Min Act: -0.0000

Data Index: 663018 (The Pile)

Max Activating Token Index: 26

Text #7

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.5745. Min Act: -0.0000

Data Index: 255503 (The Pile)

Max Activating Token Index: 790

Text #8

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.4117. Min Act: -0.0000

Data Index: 1839135 (The Pile)

Max Activating Token Index: 1017

Text #9

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.2903. Min Act: -0.0000

Data Index: 1557073 (The Pile)

Max Activating Token Index: 763

Text #10

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.4313. Min Act: -0.0000

Data Index: 354009 (The Pile)

Max Activating Token Index: 903

Text #11

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.6200. Min Act: -0.0000

Data Index: 1069623 (The Pile)

Max Activating Token Index: 480

Text #12

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.6147. Min Act: -0.0000

Data Index: 580629 (The Pile)

Max Activating Token Index: 782

Text #13

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.4483. Min Act: -0.0000

Data Index: 133623 (The Pile)

Max Activating Token Index: 755

Text #14

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.5902. Min Act: -0.0000

Data Index: 460036 (The Pile)

Max Activating Token Index: 871

Text #15

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.5773. Min Act: -0.0000

Data Index: 97625 (The Pile)

Max Activating Token Index: 41

Text #16

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.3585. Min Act: -0.0000

Data Index: 1600090 (The Pile)

Max Activating Token Index: 877

Text #17

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.3067. Min Act: -0.0000

Data Index: 687812 (The Pile)

Max Activating Token Index: 790

Text #18

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.5443. Min Act: -0.0000

Data Index: 1803158 (The Pile)

Max Activating Token Index: 746

Text #19

Max Range: 3.1806. Min Range: -3.1806

Max Act: 2.3969. Min Act: -0.0000

Data Index: 255181 (The Pile)

Max Activating Token Index: 554