< Prev | Home | Model | Random | Next >

Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 5119 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.1154. Min Act: -0.0000

Data Index: 79420 (The Pile)

Max Activating Token Index: 1023

Click toggle to see full text

Truncated

(\begin{smallmatrix}0&0&1&0\\1&1&0&1\\0&1&0&1\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}0&1&1&1\\

Full Text #0

<|endoftext|>&1&0&1\\1&1&1&1\\1&1&1&0\end{smallmatrix}\right)\Big\}\\[+4mm] 
\Big\{\left(\begin{smallmatrix}0&0&0&0\\0&0&0&0\\0&0&0&0\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}1&0&0&0\\0&0&1&1\\0&0&1&0\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}1&0&1&1\\1&0&0&0\\0&1&0&0\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}0&1&0&1\\1&1&0&0\\1&0&0&0\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}0&1&1&1\\0&0&1&0\\0&0&0&1\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}0&0&1&0\\0&1&0&1\\1&1&0&0\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}0&0&0&1\\1&1&1&0\\1&0&0&1\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}0&1&0&0\\1&0&0&1\\0&1&1&1\end{smallmatrix}\right),\\[+2mm] 
\left(\begin{smallmatrix}1&1&0&0\\1&0&1&0\\0&1&0&1\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}1&1&1&1\\0&0&0&1\\0&0&1&1\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}0&0&1&1\\1&0&1&1\\0&1&1&0\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}1&0&1&0\\0&1&1&0\\1&1&1&0\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}1&0&0&1\\1&1&0&1\\1&0&1&1\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}0&1&1&0\\0&1&1&1\\1&1&0&1\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}1&1&1&0\\0&1&0&0\\1&1&1&1\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}1&1&0&1\\1&1&1&1\\1&0&1&0\end{smallmatrix}\right)\Big\}\\[+4mm] 
\Big\{\left(\begin{smallmatrix}0&0&0&0\\0&0&0&0\\0&0&0&0\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}0&1&0&0\\0&0&1&0\\0&0&0&1\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}1&1&0&0\\0&1&0&0\\1&0&1&0\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}1&0&0&1\\1&0&1&0\\1&0&0&0\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}0&0&1&1\\0&0&0&1\\1&1&0&1\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}1&1&1&0\\0&1&0&1\\0&0&1&0\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}1&1&0&1\\1&0&0&0\\1&0&0&1\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}1&0&0&0\\0&1&1&0\\1&0&1&1\end{smallmatrix}\right),\\[+2mm] 
\left(\begin{smallmatrix}0&0&1&0\\1&1&0&1\\0&1&0&1\end{smallmatrix}\right)& 
\left(\begin{smallmatrix}0&1&1&1\\

Text #1

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.1151. Min Act: -0.0000

Data Index: 742374 (The Pile)

Max Activating Token Index: 1023

Click toggle to see full text

Truncated

Full Text #1

Text #2

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.1121. Min Act: -0.0000

Data Index: 1121584 (The Pile)

Max Activating Token Index: 25

Click toggle to see full text

Truncated

Full Text #2

Text #3

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.1090. Min Act: -0.0000

Data Index: 1412982 (The Pile)

Max Activating Token Index: 345

Click toggle to see full text

Truncated

Full Text #3

Text #4

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.1065. Min Act: -0.0000

Data Index: 1321267 (The Pile)

Max Activating Token Index: 7

Click toggle to see full text

Truncated

Full Text #4

Text #5

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.1062. Min Act: -0.0000

Data Index: 1544047 (The Pile)

Max Activating Token Index: 3

Click toggle to see full text

Truncated

Full Text #5

Text #6

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.1031. Min Act: -0.0000

Data Index: 638238 (The Pile)

Max Activating Token Index: 10

Click toggle to see full text

Truncated

Full Text #6

Text #7

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.1022. Min Act: -0.0000

Data Index: 1472403 (The Pile)

Max Activating Token Index: 6

Click toggle to see full text

Truncated

Full Text #7

Text #8

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.1000. Min Act: -0.0000

Data Index: 1317725 (The Pile)

Max Activating Token Index: 12

Click toggle to see full text

Truncated

Full Text #8

Text #9

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.0994. Min Act: -0.0000

Data Index: 1143894 (The Pile)

Max Activating Token Index: 7

Click toggle to see full text

Truncated

Full Text #9

Text #10

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.1026. Min Act: -0.0000

Data Index: 875462 (The Pile)

Max Activating Token Index: 10

Click toggle to see full text

Truncated

Full Text #10

Text #11

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.0987. Min Act: -0.0000

Data Index: 723849 (The Pile)

Max Activating Token Index: 18

Click toggle to see full text

Truncated

Full Text #11

Text #12

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.0981. Min Act: -0.0000

Data Index: 781681 (The Pile)

Max Activating Token Index: 885

Click toggle to see full text

Truncated

Full Text #12

Text #13

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.0968. Min Act: -0.0000

Data Index: 437537 (The Pile)

Max Activating Token Index: 7

Click toggle to see full text

Truncated

Full Text #13

Text #14

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.0967. Min Act: -0.0000

Data Index: 1005962 (The Pile)

Max Activating Token Index: 12

Click toggle to see full text

Truncated

Full Text #14

Text #15

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.0964. Min Act: -0.0000

Data Index: 257469 (The Pile)

Max Activating Token Index: 18

Click toggle to see full text

Truncated

Full Text #15

Text #16

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.0959. Min Act: -0.0000

Data Index: 361352 (The Pile)

Max Activating Token Index: 8

Click toggle to see full text

Truncated

Full Text #16

Text #17

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.0943. Min Act: -0.0000

Data Index: 129782 (The Pile)

Max Activating Token Index: 11

Click toggle to see full text

Truncated

Full Text #17

Text #18

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.0947. Min Act: -0.0000

Data Index: 403376 (The Pile)

Max Activating Token Index: 5

Click toggle to see full text

Truncated

Full Text #18

Text #19

Max Range: 0.1154. Min Range: -0.1154

Max Act: 0.0942. Min Act: -0.0000

Data Index: 1261257 (The Pile)

Max Activating Token Index: 26

Click toggle to see full text

Truncated

Full Text #19