Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2603 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.3380. Min Act: -0.0000

Data Index: 1798497 (The Pile)

Max Activating Token Index: 1019

Text #1

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.3024. Min Act: -0.0000

Data Index: 631425 (The Pile)

Max Activating Token Index: 999

Text #2

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2249. Min Act: -0.0000

Data Index: 350525 (The Pile)

Max Activating Token Index: 744

Text #3

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2335. Min Act: -0.0000

Data Index: 1419345 (The Pile)

Max Activating Token Index: 558

Text #4

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2430. Min Act: -0.0000

Data Index: 807307 (The Pile)

Max Activating Token Index: 778

Text #5

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2574. Min Act: -0.0000

Data Index: 1782109 (The Pile)

Max Activating Token Index: 679

Text #6

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2323. Min Act: -0.0000

Data Index: 73167 (The Pile)

Max Activating Token Index: 629

Text #7

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2427. Min Act: -0.0000

Data Index: 1007839 (The Pile)

Max Activating Token Index: 971

Text #8

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2275. Min Act: -0.0000

Data Index: 334535 (The Pile)

Max Activating Token Index: 789

Text #9

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2130. Min Act: -0.0000

Data Index: 1973972 (The Pile)

Max Activating Token Index: 966

Text #10

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2121. Min Act: -0.0000

Data Index: 349900 (The Pile)

Max Activating Token Index: 1016

Text #11

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.1974. Min Act: -0.0000

Data Index: 1435503 (The Pile)

Max Activating Token Index: 1006

Text #12

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2053. Min Act: -0.0000

Data Index: 1203498 (The Pile)

Max Activating Token Index: 798

Text #13

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2294. Min Act: -0.0000

Data Index: 1199476 (The Pile)

Max Activating Token Index: 969

Text #14

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2325. Min Act: -0.0000

Data Index: 268680 (The Pile)

Max Activating Token Index: 799

Text #15

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.1952. Min Act: -0.0000

Data Index: 1147058 (The Pile)

Max Activating Token Index: 1015

Text #16

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2131. Min Act: -0.0000

Data Index: 1055029 (The Pile)

Max Activating Token Index: 888

Text #17

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.1866. Min Act: -0.0000

Data Index: 1696486 (The Pile)

Max Activating Token Index: 884

Text #18

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.2031. Min Act: -0.0000

Data Index: 1698160 (The Pile)

Max Activating Token Index: 698

Text #19

Max Range: 0.3380. Min Range: -0.3380

Max Act: 0.1944. Min Act: -0.0000

Data Index: 788968 (The Pile)

Max Activating Token Index: 596