Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 812 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.4875. Min Range: -5.4875

Max Act: 5.4875. Min Act: -0.0000

Data Index: 123300 (The Pile)

Max Activating Token Index: 1014

Text #1

Max Range: 5.4875. Min Range: -5.4875

Max Act: 4.2972. Min Act: -0.0000

Data Index: 813832 (The Pile)

Max Activating Token Index: 76

Text #2

Max Range: 5.4875. Min Range: -5.4875

Max Act: 4.2338. Min Act: -0.0000

Data Index: 984559 (The Pile)

Max Activating Token Index: 201

Text #3

Max Range: 5.4875. Min Range: -5.4875

Max Act: 4.5795. Min Act: -0.0000

Data Index: 777445 (The Pile)

Max Activating Token Index: 97

Text #4

Max Range: 5.4875. Min Range: -5.4875

Max Act: 4.3550. Min Act: -0.0000

Data Index: 1918305 (The Pile)

Max Activating Token Index: 96

Text #5

Max Range: 5.4875. Min Range: -5.4875

Max Act: 4.0969. Min Act: -0.0000

Data Index: 122160 (The Pile)

Max Activating Token Index: 19

Text #6

Max Range: 5.4875. Min Range: -5.4875

Max Act: 4.0232. Min Act: -0.0000

Data Index: 74184 (The Pile)

Max Activating Token Index: 870

Text #7

Max Range: 5.4875. Min Range: -5.4875

Max Act: 4.3218. Min Act: -0.0000

Data Index: 932384 (The Pile)

Max Activating Token Index: 23

Text #8

Max Range: 5.4875. Min Range: -5.4875

Max Act: 4.0933. Min Act: -0.0000

Data Index: 300806 (The Pile)

Max Activating Token Index: 1018

Text #9

Max Range: 5.4875. Min Range: -5.4875

Max Act: 3.8153. Min Act: -0.0000

Data Index: 1345958 (The Pile)

Max Activating Token Index: 26

Text #10

Max Range: 5.4875. Min Range: -5.4875

Max Act: 3.7892. Min Act: -0.0000

Data Index: 390597 (The Pile)

Max Activating Token Index: 985

Text #11

Max Range: 5.4875. Min Range: -5.4875

Max Act: 3.9450. Min Act: -0.0000

Data Index: 1502709 (The Pile)

Max Activating Token Index: 32

Text #12

Max Range: 5.4875. Min Range: -5.4875

Max Act: 4.0651. Min Act: -0.0000

Data Index: 926966 (The Pile)

Max Activating Token Index: 57

Text #13

Max Range: 5.4875. Min Range: -5.4875

Max Act: 3.8876. Min Act: -0.0000

Data Index: 1293649 (The Pile)

Max Activating Token Index: 76

Text #14

Max Range: 5.4875. Min Range: -5.4875

Max Act: 4.0568. Min Act: -0.0000

Data Index: 1824639 (The Pile)

Max Activating Token Index: 50

Text #15

Max Range: 5.4875. Min Range: -5.4875

Max Act: 3.8748. Min Act: -0.0000

Data Index: 314679 (The Pile)

Max Activating Token Index: 53

Text #16

Max Range: 5.4875. Min Range: -5.4875

Max Act: 4.0226. Min Act: -0.0000

Data Index: 1679614 (The Pile)

Max Activating Token Index: 58

Text #17

Max Range: 5.4875. Min Range: -5.4875

Max Act: 3.4899. Min Act: -0.0000

Data Index: 197582 (The Pile)

Max Activating Token Index: 180

Text #18

Max Range: 5.4875. Min Range: -5.4875

Max Act: 3.6204. Min Act: -0.0000

Data Index: 1010967 (The Pile)

Max Activating Token Index: 1023

Text #19

Max Range: 5.4875. Min Range: -5.4875

Max Act: 3.4478. Min Act: -0.0000

Data Index: 812270 (The Pile)

Max Activating Token Index: 250