\subsection{Functions of the edges}
\label{sec:edges}

%------------------------------------------------------------------

\begin{figure}[!htp]
\centering
\begin{verbatim}
          p1
          o
         /|\
        / | \
       /  |  \e31
   e12/   |   \
     /    |e01 \
    /     |     \
p2 o f012 | f031 o p3
    \     |     /
     \    |    /
   e20\   |   /e03
       \  |  /
        \ | /
         \|/
          o
          p0

\end{verbatim}
\caption{Edge face pair labeling.
\label{diagram:EdgeFaces}}
\end{figure}

Notation in this section is based on figure \ref{diagram:EdgeFaces}.
We are discussing functions defined on a neighborhood of edge $e_{01}$.

We assume that, for each edge, an arbitrary order is assigned to
its two vertices, which are then at the positions $\p_0,\p_1$ in the diagram.

An interior edge has 2 adjacent faces, $f_{012}$ and $f_{031}$.
We assume that these 2 faces are oriented consistently, with the labels
taken counterclockwise, so that the normal vectors point out of the page.
Each face is represented by an ordered triple of vertices,
but the order is only determined up to a circular permutation;
for example, $f_{012}$ may be represented by the ordered triples
$(\p_0,\p_1,\p_2)$, $(\p_2,\p_0,\p_1)$, or $(\p_1,\p_2,\p_0)$,
but not by
$(\p_0,\p_2,\p_1)$, $(\p_1,\p_0,\p_2)$, or $(\p_2,\p_1,\p_0)$.

For the given ordering $(\p_0,\p_1)$ of the edge,
$f_{120}$ is the edge's {\it left face}
and $f_{031}$ is the {\it right face}.

Note that we cannot assume any consistent ordering of the 4 neighboring edges;
for example, $e_{12}$ may be represented by either ordered pair
$(\p_1,\p_2)$ or $(\p_2,\p_1)$.

%------------------------------------------------------------------

\subsubsection{Edge length}
\label{sec:edge_length}

The edge tangent vector is $\p_1 - \p_0$.

The gradient of its squared length is:
\begin{equation}
\Gc{\p_i}{\| \p_1 - \p_0 \|^2}{\q} = 2 \left( \p_i - \p_{(i+1) \bmod 1} \right)
\end{equation}

The gradient of the edge length, $\|\p_1 - \p_0\|$ is:
\begin{equation}
\Gc{\p_i}{\| \p_1 - \p_0 \|}{\q} =
\frac{\left( \p_i - \p_{(i+1) \bmod 1} \right)}
{\|\p_1 - \p_0\|}
\end{equation}

%------------------------------------------------------------------

\subsubsection{Edge dihedral}
\label{sec:edge_dihedral}

In smooth surfaces,
the standard measures of curvature are all measures
of the rate of change of the surface normal vector.
For triangular meshes, it is therefore natural to
consider functions that depend on the change in
normal vectors between nearby faces.

In this section, we consider functions of the difference
in normals for 2 faces that share an edge,
which is equivalent to the {\it dihedral angle} of the edge.

We will consider a number of both real- and $\Reals^3$-valued functions on
$\Reals^{12} = \Reals^3 \oplus \Reals^3 \oplus \Reals^3 \oplus \Reals^3$.
We will use $\p = (\p_0, \p_1, \p_2, \p_3)$, $\q=\ldots$, $\r=\ldots$, etc.,
to refer to the arguments of these functions, with the meaning
of the indices determined by figure \ref{diagram:EdgeFaces}.
We are also interested in the two 9-dimensional subspaces
corresponding to the vertics of the two faces:
$\p_{012} = (\p_0,\p_1,\p_2), \p_{031} = (\p_0,\p_3,\p_1)$.
We abbreviate the two normal vectors:
$\n_{012} = \n(\p_0,\p_1,\p_2)$
and
$\n_{031} = \n(\p_0,\p_3,\p_1)$.

%------------------------------------------------------------------

\paragraph{Difference in face normals}
\label{sec:normal_difference}

One measure of the change in surface normal across an edge
is simply the vector difference of the two normals:

\begin{equation}
\label{eq:deltan}
{\mathbf \dn} (\p_0, \p_1, \p_2, \p_3)
=
\n (\p_{012}) - \n (\p_{031})
\end{equation}

The (total) derivative of the squared distance between adjacent face normals is:
\begin{eqnarray}
\Db{\|\dn(\p)\|^2}{\q}
& =
2 \ \dn ( \q )^\dagger &
\left( \Db{ ( \dn ) }{\q} \right)
\\
& =
2 \ \dn(\q)^\dagger &
\left( \Db{\n(\p_{012})}{\q} - \Db{\n(\p_{031})}{\q} \right)
\nonumber \\
& =
2 \dn(\q)^\dagger &
\{ \; \left[ \I_{\Reals^3} - \left( \n( \q_{012} ) \otimes \n( \q_{012} ) \right)
\right]
\ast \Db{\a ( \p_{012} ) }{\q}
\nonumber \\
\label{eq:deltan_derivative}
&
& - \left[ \I_{\Reals^3} - \left( \n( \q_{031} ) \otimes \n ( \q_{031} ) \right)
\right]
\ast \Db{\a ( \p_{031} ) }{\q}
\; \}
\nonumber
\end{eqnarray}

The partial derivatives, with respect to one of the vertices,
like $\Dd{\p_0}{\a ( \p_{012} ) }{\q}{\r_0}$,
all have a similar form:
\begin{equation}
\Dd{\p_0}{\a ( \p_{012} ) }{\q}{\r_0}  = (\q_1 - \q_3) \times \r_0
\end{equation}
Using this, equation \ref{eq:deltan_derivative}, equation \ref{eq:dot_cross},
and the facts that
$\dn(\q)  \perp  \n(\q_{012}) = - \left( \n(\q_{031})  \perp  \n(\q_{012}) \right)$
and
$\dn(\q)  \perp  \n(\q_{031}) = \n(\q_{012})  \perp  \n(\q_{031})$,
we can write the partial gradients without reference to the
derivative's argument $\r$:
\begin{eqnarray}
\label{eq:normal-difference-gradient}
\Gc{\p_0}{\|\dn\|^2}{\q}
& = &
\left[
{{ \n(\q_{031})  \perp  \n(\q_{012}) }
\over {A(\q_{012})}}
\times (\q_1 - \q_2)
\right]
\; + \;
\left[
{{ \n(\q_{012})  \perp  \n(\q_{031}) }
\over {A(\q_{031})}}
\times (\q_3 - \q_1)
\right]
\\
\Gc{\p_1}{\|\dn\|^2}{\q}
& = &
\left[
{{ \n(\q_{031})  \perp  \n(\q_{012}) }
\over {A(\q_{012})}}
\times (\q_2 - \q_0)
\right]
\; + \;
\left[
{{ \n(\q_{012})  \perp  \n(\q_{031}) }
\over {A(\q_{031})}}
\times (\q_0 - \q_3)
\right]
\nonumber
\\
\Gc{\p_2}{\|\dn\|^2}{\q}
& = &
\left[
{{ \n(\q_{031})  \perp  \n(\q_{012}) }
\over {A(\q_{012})}}
\times (\q_0 - \q_1)
\right]
\nonumber
\\
\Gc{\p_3}{\|\dn\|^2}{\q}
& = &
\left[
{{ \n(\q_{012})  \perp  \n(\q_{031}) }
\over {A(\q_{031})}}
\times (\q_1 - \q_0)
\right]
\nonumber
\end{eqnarray}

%------------------------------------------------------------------

\paragraph{Inner product between face normals}
\label{sec:normal_dot}

The inner product $\left( \n_{012} \bullet \n_{031} \right)$
is another important measure of edge curvature.
It is closely related to the squared distance between adjacent normals:
\begin{equation}
\label{eq:normal-distance-dot}
\| \n_{012} - \n_{031} \|^2
= \| \n_{012} \|^2
+ \| \n_{031} \|^2
- 2 \left( \n_{012} \bullet \n_{031} \right)
= 2 \left[ 1 - \left( \n_{012} \bullet \n_{031} \right) \right]
\end{equation}

The function $f(\p) = 1 - \left( \n_{012} \bullet \n_{031} \right)$
achieves its minimum, $0$, on flat face pairs,
and its maximum, $2$, on face pairs that are folded back on themselves.
It's a reasonable choice the total bending or curvature of a surface.
And $\Da{f} = - \Da{\left( \n_{012} \bullet \n_{031} \right)}$.

The derivative of
$\left( \n_{012} \bullet \n_{031} \right)$
can be calculated using equations \ref{eq:dot_derivative} and
\ref{eq:unit_normal_derivative}:
\begin{eqnarray}
\label{normal_dot_derivative}
\Db{\left( \n_{012} \bullet \n_{031} \right)}{\q}
& = & \n(\q_{031}) \bullet \Db{\n_{012}}{\q} + \n(\q_{012}) \bullet \Db{\n_{031}}{\q}
\\
\nonumber \\
& = &
\n(\q_{031}) \bullet
{{\I - \left(\n(\q_{012}) \otimes \n(\q_{012}) \right)} \over {\| \a(\q_{012}) \|}}
\; \Db{\a_{012}}{\q}
\nonumber \\
& + &
\n(\q_{012}) \bullet
{{\I - \left(\n(\q_{031}) \otimes \n(\q_{031}) \right)} \over {\| \a(\q_{031}) \|}}
\; \Db{\a_{031}}{\q}
\nonumber
\end{eqnarray}

As in \autoref{sec:normal_difference}, we can write the partial gradients
without reference to an argument:
\begin{eqnarray}
\label{eq:normal_dot_gradient}
\Gc{\p_0}{(\n_{012} \bullet \n_{031})}{\q}
& = \; &
{ { \n(\q_{031}) - \left[ \n(\q_{012}) \bullet \n(\q_{031}) \right] \n(\q_{012}) }
\over {\| \a (\q_{012}) \| } }
\times (\q_2 - \q_1)
\\
& \; + &
{ { \n(\q_{012}) - \left[ \n(\q_{012}) \bullet \n(\q_{031}) \right] \n(\q_{031})  }
\over {\| \a (\q_{031}) \| }}
\times (\q_1 - \q_3)
\nonumber \\
& & \nonumber \\
\Gc{\p_1}{(\n_{012} \bullet \n_{031})}{\q}
& = \; &
{ { \n(\q_{031}) - \left[ \n(\q_{012}) \bullet \n(\q_{031}) \right] \n(\q_{012})  }
\over {\| \a (\q_{012}) \| } }
\times (\q_0 - \q_2)
\nonumber \\
& \; + &
{{ \n(\q_{012}) - \left[ \n(\q_{012}) \bullet \n(\q_{031}) \right] \n(\q_{031})   }
\over {\| \a (\q_{031}) \| }}
\times (\q_3 - \q_0)
\nonumber \\
& & \nonumber \\
\Gc{\p_2}{(\n_{012} \bullet \n_{031})}{\q}
& = \; &
{{ \n(\q_{031}) - \left[ \n(\q_{012}) \bullet \n(\q_{031}) \right] \n(\q_{012})  }
\over {\| \a (\q_{012}) \| } }
\times (\q_1 - \q_0)
\nonumber \\
& & \nonumber \\
\Gc{\p_3}{(\n_{012} \bullet \n_{031})}{\q}
& = \; &
{ { \n(\q_{012}) - \left[ \n(\q_{012}) \bullet \n(\q_{031}) \right] \n(\q_{031}) }
\over {\| \a (\q_{031}) \| } }
\times (\q_0 - \q_1)
\nonumber
\end{eqnarray}

This can be simplified using the fact that
\(\n_i \perp \n_j = \n_i - \left[ \n_i \bullet \n_j \right] \n_j\), for unit vectors,
and the face area \(A(\q) = \frac{1}{2} \| \a(\q) \|\):
\begin{eqnarray}
\label{eq:simplified_normal_dot_gradient}
\Gc{\p_0}{(\n_{012} \bullet \n_{031})}{\q}
& = \;\;\; &
\frac{\left[ \n(\q_{031}) \perp \n(\q_{012}) \right]}{2A(\q_{012})}
\times (\q_2 - \q_1)
\\
& \;\;\; + &
\frac{\left[ \n(\q_{012}) \perp \n(\q_{031}) \right]}{2A(\q_{031})}
\times (\q_1 - \q_3)
\nonumber \\
& & \nonumber \\
\Gc{\p_1}{(\n_{012} \bullet \n_{031})}{\q}
& = \;\;\; &
\frac{\left[ \n(\q_{031}) \perp \n(\q_{012}) \right]}{2A(\q_{012})}
\times (\q_0 - \q_2)
\nonumber \\
& \;\;\; + &
\frac{\left[ \n(\q_{012}) \perp \n(\q_{031}) \right]}{2A(\q_{031})}
\times (\q_3 - \q_0)
\nonumber \\
& & \nonumber \\
\Gc{\p_2}{(\n_{012} \bullet \n_{031})}{\q}
& = \;\;\; &
\frac{\left[ \n(\q_{031}) \perp \n(\q_{012}) \right]}{2A(\q_{012})}
\times (\q_1 - \q_0)
\nonumber \\
& & \nonumber \\
\Gc{\p_3}{(\n_{012} \bullet \n_{031})}{\q}
& = \;\;\; &
\frac{\left[ \n(\q_{012}) \perp \n(\q_{031}) \right]}{2A(\q_{031})}
\times (\q_0 - \q_1)
\nonumber
\end{eqnarray}

%------------------------------------------------------------------

\paragraph{Squared inner product between face normals}
\label{sec:squared_normal_dot}

We can get a more even distribution of bending by giving
a higher weight to sharper edge bends.
A simple way to do that is to square some existing function,
for example: $\left(1 - \n_{012} \bullet \n_{031}\right)^2$.
The derivative is simply:
\begin{equation}
\Da{\left(1 - \n_{012} \bullet \n_{031}\right)^2}
= -2 \left( 1 - \n_{012} \bullet \n_{031} \right)
\Da{(\n_{012} \bullet \n_{031})}
\end{equation}

It follows from equation \ref{eq:simplified_normal_dot_gradient}
that the partial gradients are:
\begin{eqnarray}
\label{eq:squared_normal_dot_gradient}
\Gc{\p_0}{\left(1 - \n_{012} \bullet \n_{031}\right)^2}{\q}
& = \;\;\; &
\frac{\left( \n(\q_{012}) \bullet \n(\q_{031}) - 1\right)
}
{A(\q_{012}) }
\left[ \n(\q_{031}) \perp \n(\q_{012}) \right]
\times (\q_2 - \q_1)
\\
& \;\;\; + &
\frac{\left( \n(\q_{012}) \bullet \n(\q_{031}) - 1\right)
}{A(\q_{031})}
\left[ \n(\q_{012}) \perp \n(\q_{031}) \right]
\times (\q_1 - \q_3)
\nonumber \\
& & \nonumber \\
\Gc{\p_1}{\left(1 - \n_{012} \bullet \n_{031}\right)^2}{\q}
& = \;\;\; &
\frac{\left( \n(\q_{012}) \bullet \n(\q_{031}) - 1\right)
}{A(\q_{012})}
\left[ \n(\q_{031}) \perp \n(\q_{012}) \right]
\times (\q_0 - \q_2)
\nonumber \\
& \;\;\; + &
\frac{\left( \n(\q_{012}) \bullet \n(\q_{031}) - 1\right)
}{A(\q_{031})}
\left[ \n(\q_{012}) \perp \n(\q_{031}) \right]
\times (\q_3 - \q_0)
\nonumber \\
& & \nonumber \\
\Gc{\p_2}{\left(1 - \n_{012} \bullet \n_{031}\right)^2}{\q}
& = \;\;\; &
\frac{\left( \n(\q_{012}) \bullet \n(\q_{031}) - 1\right)
}{A(\q_{012})}
\left[ \n(\q_{031}) \perp \n(\q_{012}) \right]
\times (\q_1 - \q_0)
\nonumber \\
& & \nonumber \\
\Gc{\p_3}{\left(1 - \n_{012} \bullet \n_{031}\right)^2}{\q}
& = \;\;\; &
\frac{\left( \n(\q_{012}) \bullet \n(\q_{031}) - 1\right)
}{A(\q_{031})}
\left[ \n(\q_{012}) \perp \n(\q_{031}) \right]
\times (\q_0 - \q_1)
\nonumber
\end{eqnarray}

%-----------------------------------------------------------------

\paragraph{Dihedral angle}
\label{sec:Dihedral-angle}

The {\it dihedral angle},
$\theta_d$,
 of an edge is the amount
needed to rotate one face
through the inside of the surface
(in the sense of the orientation of the faces)
onto the other face.

With this definition, the dihedral angle is always
positive, in fact, between $0$ and $2\pi$.
An acutely concave edge has a dihedral angle
of a little less than $2\pi$;
a concave edge where the faces meet in a right angle
${3\pi} \over 2$;
a flat edge is $\pi$;
a convex right angle $\pi \over 2$;
and an acutely convex edge has a dihedral angle
slightly more than $0$.

The dihedral angle is a simple function of the
signed normal angle, $\theta_n$
(\autoref{sec:signed_normal_angle}),
and it is generally more convenient to work
with the signed normal angle directly,
so I will not consider the dihedral angle further.

%------------------------------------------------------------------

\paragraph{Unsigned normal angle}
\label{sec:unsigned_normal_angle}

The inner product $\left( \n_{012} \bullet \n_{031} \right)$
is the cosine of $\theta_u$,
the (unsigned) angle between the 2 face normals.
Thus the value of the angle is:
\begin{equation}
\theta_u(\p_0,\p_1,\p_2,\p_3)
= \cos^{-1} \left( \n_{012} \bullet \n_{031} \right)
\end{equation}

Recall the derivative of the $\cos^{-1}$ is:
\begin{equation}
\frac{d}{\mathit dx} \cos^{-1}(x) = { -1 \over \sqrt{1 - x^2} }
\end{equation}

The derivative of $\theta_u$ can be calculated using the chain rule
(equation \ref{eq:chain-rule}) and equation \ref{eq:normal_dot_gradient}.
\begin{equation}
\Db{\theta_u}{\q}
 = { -1 \over \sqrt{1 - \left( \n(\q_{012}) \bullet \n(\q_{031}) \right)^2} }
\; \Db{\left( \n_{012} \bullet \n_{031} \right)}{\q}
\end{equation}

As above, we can write the partial gradients without reference to an argument:
\begin{equation}
\Gc{\p_i}{\theta_u}{\q}
=
{ -1 \over \sqrt{1 - \left( \n(\q_{012}) \bullet \n(\q_{031}) \right)^2} }
\; \Gc{\p_i}{\left(\n_{012}\bullet\n_{031}\right)}{\q}
\end{equation}

%------------------------------------------------------------------

\paragraph{Signed normal angle}
\label{sec:signed_normal_angle}

The unsigned normal angle
($\theta_u$, \autoref{sec:unsigned_normal_angle})
doesn't distinguish concave and convex changes in the normal vector,
so it can't be a 1-1 function of the dihedral angle.
More importantly, it's inadequate if we want to construct
a penalty that treats concave and convex edges differently.

It is common to want to make a surface as flat as possible,
which means penalizing normal angles different from zero.
With such 2rd order (curvature minimizing) penalties,
it may not necessary to distinguish
convex and concave bending.

However, there's considerable evidence that
$3$rd order penalties ---minimizing the {\it variation}
in surface bending, rather than the bending itself---
gives better results in many situations.
To measure bending variation, it's necessary
to use correctly signed angles.

Another reason to use signed angles is to be able
to encourage the surface to bend in a certain
predetermined way at certain edges.
This is discussed in \autoref{sec:Bent-edge-neighborhoods}
below.

The {\it signed normal angle,} $\theta_n$,
is essentially just the
unsigned normal angle, multiplied by $-1$ if the surface is
concave at the edge.
To make this a bit more formal,
we can define the signed normal angle to be the amount
of righthanded rotation about the edge tangent ($\p_1 - \p_0$)
needed to bring the right normal ($\n_{031}$) to
the left normal ($\n_{012}$).
We cut our angle measurement at $\pi \sim -\pi$.

With this definition, the signed normal angle
is between $-\pi$ and $\pi$.
An acutely concave edge has a signed normal angle
of a little more than $-\pi$;
a concave edge where the faces meet in a right angle
$-{\pi \over 2}$;
a flat edge is $0$;
a convex right angle $\pi \over 2$;
and an acutely convex edge has a signed normal angle
slightly less than $\pi$.

The dihedral angle can be expressed simply in terms of the
signed normal angle: $\theta_d = \pi - \theta_n$.

To compute $\theta_n$,
it's easiest to use the informal definition:
\begin{equation}
\theta_n(\p_0,\p_1,\p_2,\p_3)
= \kappa(\p_0,\p_1,\p_2,\p_3) \ast \theta_u(\p_0,\p_1,\p_2,\p_3),
\end{equation}
where $\kappa(\p_0,\p_1,\p_2,\p_3)$ is
$+1$ if the edge is convex
and
$-1$ if the edge is concave.
One way to determine convexity/concavity
is to look at
$\left(\p_3 - \p_2 \right) \bullet \n_{031}$
which is positive if the edge is convex
and negative if it is concave.

The derivative of $\theta_n$ are simply $\kappa$
times the derivative of $\theta_u$:
$\Db{\theta_n}{\q} = \kappa(\q) \ast \Db{\theta_u}{\q}$.

%-----------------------------------------------------------------

\paragraph{Signed angle and normal distance}
\label{sec:Signed-angle-and-normal-distance}

We can tie the signed normal angle back
to the distance between normal vectors,
and save the time and complexity
of computing the $\cos^{-1}$ and its derivatives,
by considering a simple transform.

First, note that $\sin({\theta_n \over 2})$
is a monotone function of $\theta_n$,
ranging from $-1$, when $\theta_n = -\pi$,
to $1$ when $\theta_n = \pi$.
Also note that
\begin{equation}
\sin({\theta_n \over 2})
 =
\sign(\theta_n)
\left[ {1 - \cos(\theta_n)}
\over 2 \right]^{1 \over 2}
 =
\sign(\theta_n)
\left[ {1 - \left( \n_{012}\bullet\n_{031} \right)}
\over 2 \right]^{1 \over 2},
\end{equation}
which means that
\begin{equation}
\sin^2({\theta_n \over 2})
= {{1 - \left( \n_{012}\bullet\n_{031} \right)} \over 2},
\end{equation}
which is proportional to the distance between the adjacent normals
(see equation \ref{eq:normal-distance-dot}).

Note that the function that promotes evenly
distributed bending described in \autoref{sec:squared_normal_dot},
is proportional to $\sin^4({\theta_n \over 2})$.

%-----------------------------------------------------------------

\paragraph{Bent edge neighborhoods}
\label{sec:Bent-edge-neighborhoods}

In many problems \cite{hoppe-et-al-94,hoppe-thesis-94},
it's important to be able to allow the surface to have
sharp creases and cusps.
A simple way to achieve this is to mark
certain edges as ``sharp'',
and then not evaluate any bending penalty
on those edges.
With this approach, there are then 2 kinds of edges:
sharp edges, where any bend angle is equally acceptable,
and smooth edges,
where any deviation from zero bending is penalized.

However, it's not often the case that all angles
are equally desirable for a ``sharp'' edge,
or that no angle, other than $0$, is acceptable
for a ``smooth'' edge.
It's often true that we know that a given crease
must be, for example, convex ($\theta_n > 0$)
but with bounded acuteness ($\theta_n < \alpha < \pi$).
Or we may know that a given edge should be close to
a convex right angle
($\left( {\pi \over 2} - \delta \right)
< \theta_n <
\left( {\pi \over 2} + \delta \right)$)
for some small positive angle $\delta$.

Enforcing hard constraints on $\theta_n$
is relatively difficult,
but it is easy to convert any function
of the signed normal angle, $f(\theta_n)$,
that penalizes non-zero normal angles,
into one that penalizes angles
outside an interval, $\left[\theta_0,\theta_1\right]$,
of prefered angles.
($-\pi \leq \theta_0 \leq \theta_1 \leq \pi$.)

For example,
consider the function of \autoref{sec:squared_normal_dot},
which is proportional to $\sin^4({\theta_n \over 2})$.
Let $s = \sin({\theta_n \over 2})$,
a monotone function of $\theta_n$
that ranges over $\left[ -1, 1 \right]$.

Then,
given an interval $\left[ \theta_0, \theta_1 \right] \subseteq \left[ -\pi, \pi \right]$,
let the {\it bent} function be:
\begin{equation}
f(s)
=
\left\{
\begin{array}{cr}
\left[ {
\frac{\textstyle
s^2 - s_0^2}
{\textstyle
1 - s_0^2}
  } \right]^2
& -\pi \leq \theta_n \leq \theta_0
\\
\\
0
& \theta_0 \leq \theta_n \leq \theta_1
\\
\\
\left[ {
  {\textstyle s^2 - s_1^2}
  \over
  {\textstyle 1 - s_1^2}
  } \right]^2
& \theta_1 \leq \theta_n \leq \pi
\end{array}
\right.
\end{equation}
The reason for stretching $s^2$ rather than $s$
is that we get a relatively simple expression
in $\n_{012} \bullet \n_{031}$:
\begin{equation}
f(s)
=
\left\{
\begin{array}{cr}
\left[ {
  {\textstyle c_0 - \left(\n_{012} \bullet \n_{031}\right) }
  \over
  {\textstyle c_0 + 1}
   } \right]^2
& -\pi \leq \theta_n \leq \theta_0
\\
\\
0
& \theta_0 \leq \theta_n \leq \theta_1
\\
\\
\left[ {\textstyle {c_1 - \left(\n_{012} \bullet \n_{031}\right)}
    \over {\textstyle c_1 + 1}} \right]^2
& \theta_1 \leq \theta_n \leq \pi
\end{array}
\right.
\end{equation}
where $c_0 = \cos(\theta_0)$
and $c_1 = \cos(\theta_1)$.
Recall that $\cos(\theta_n) = \left(\n_{012} \bullet \n_{031}\right)$.

%-----------------------------------------------------------------

\paragraph{Weighting bend measures by length}
\label{sec:length_weighted_bend}

Let $f(\p) = f(\p_0,\p_1,\p_2,\p_3)$ be some measure of the
bend across the edge connecting $(\p_0,\p_1)$.
It may be useful to weight the bend measure by the edge length:
$\| \p_0 - \p_1 \| \ast f(\p_0,\p_1,\p_2,\p_3)$.
The partial gradients of the weighted measure are:
\begin{eqnarray}
\label{eq:length_weighted_bend_gradient}
\Gc{\p_0}{\left( \|\p_0 - \p_1 \| f(\p) \right)}{\q}
& = &
\frac{f(\q)}{\|\q_0 - \q_1\|} \left( \q_0 - \q_1 \right)
+ \| \q_0 - \q_1 \| \Gc{\p_0}{f}{q}
\\
& & \nonumber \\
\Gc{\p_1}{\left( \|\p_0 - \p_1 \| f(\p) \right)}{\q}
& = &
\frac{f(\q)}{\|\q_0 - \q_1\|} \left( \q_1 - \q_0 \right)
+ \| \q_0 - \q_1 \| \Gc{\p_1}{f}{q}
\nonumber \\
& & \nonumber \\
\Gc{\p_2}{\left( \|\p_0 - \p_1 \| f(\p) \right)}{\q}
& = &
\| \q_0 - \q_1 \| \Gc{\p_2}{f}{q}
\nonumber \\
& & \nonumber \\
\Gc{\p_3}{\left( \|\p_0 - \p_1 \| f(\p) \right)}{\q}
& = &
\| \q_0 - \q_1 \| \Gc{\p_3}{f}{q}
\nonumber
\end{eqnarray}

%------------------------------------------------------------------

\paragraph{Rate of bending}
\label{sec:Rate-of-bending}

It may be useful to weight edge bends should
by the length of the edge ---
a long sharp edge should cost more than a short sharp edge.
It may also be useful to weight inversely by the
distance to nearby edges --- two edges close together
means more curvature than the same two edges farther apart.
A measure has both these properties, at least approximately,
is:
\begin{equation}
{\mathrm dN2L2A (\p) }
=
\| \n(\p_{012}) - \n(\p_{031}) \|^2
\| \p_0 - \p_1 \|^2
\left[
\frac{1}{A(\p_{012})} +
\frac{1}{A(\p_{031})}
\right]
\end{equation}
where $\| \n(\p_{012}) - \n(\p_{031}) \|^2$ is the squared distance between
adjacent normals, as in equation \ref{eq:deltan}.
(Note the similarity of the weighting to the aspect ratio
measure discussed in \ref{sec:Squared-edge-lengths-over-area}.)

\begin{eqnarray}
\Gc{\p}{dN2L2A}{\q}
& = &
\Gc{\p}{\left(\| \n(\p_{012}) - \n(\p_{031}) \|^2 \right)}{\q}
\| \q_0 - \q_1 \|^2
\left[
\frac{1}{A(\q_{012})} +
\frac{1}{A(\q_{031})}
\right]
\\
& + &
\| \n(\q_{012}) - \n(\q_{031}) \|^2
\Gc{\p}{\left(\| \p_0 - \p_1 \|^2\right)}{\q}
\left[
\frac{1}{A(\q_{012})} +
\frac{1}{A(\q_{031})}
\right]
\nonumber
\\
& + &
\| \n(\q_{012}) - \n(\q_{012}) \|^2
\| \q_0 - \q_1 \|^2
\left[
\Gc{\p}{\left( \frac{1}{A(\p_{012})} \right)}{\q} +
\Gc{\p}{\left( \frac{1}{A(\p_{031})} \right)}{\q}
\right]
\nonumber
\end{eqnarray}

The partial gradients,
$\Gc{\p_i}{\left(\| \n(\p_{012}) - \n(\p_{031}) \|^2 \right)}{\q}$,
is given in equation \ref{eq:normal-difference-gradient}.

$\Gc{\p_0}{\left(\| \p_0 - \p_1 \|^2\right)}{\q} = 2 \left( \q_0 - \q_1 \right)$;
$\Gc{\p_1}{\left(\| \p_0 - \p_1 \|^2\right)}{\q} = 2 \left( \q_1 - \q_0 \right)$;
and the other 2 partial gradients are zero.

Using \ref{eq:area_partial_derivative}
and the chain rule, we have:
\begin{eqnarray}
\label{eq:inverse-area-gradient-012}
\Gc{\p_0}{\left( \frac{1}{A(\p_{012})} \right)}{\q}
& = &
{{\n(\q_{012}) \times \left( \q_1 - \q_2 \right)}
\over
{2 A(\q_{012})^2 } }
\\
\Gc{\p_1}{\left( \frac{1}{A(\p_{012})} \right)}{\q}
& = &
{{\n(\q_{012}) \times \left( \q_2 - \q_0 \right)}
\over
{2 A(\q_{012})^2 } }
\nonumber
\\
\Gc{\p_2}{\left( \frac{1}{A(\p_{012})} \right)}{\q}
& = &
{{\n(\q_{012}) \times \left( \q_0 - \q_1 \right)}
\over
{2 A(\q_{012})^2 } }
\nonumber
\end{eqnarray}
\begin{eqnarray}
\label{eq:inverse-area-gradient-031}
\Gc{\p_0}{\left( \frac{1}{A(\p_{031})} \right)}{\q}
& = &
{{\n(\q_{031}) \times \left( \q_3 - \q_1 \right)}
\over
{2 A(\q_{031})^2 } }
\\
\Gc{\p_1}{\left( \frac{1}{A(\p_{031})} \right)}{\q}
& = &
{{\n(\q_{031}) \times \left( \q_0 - \q_3 \right)}
\over
{2 A(\q_{031})^2 } }
\nonumber
\\
\Gc{\p_3}{\left( \frac{1}{A(\p_{031})} \right)}{\q}
& = &
{{\n(\q_{031}) \times \left( \q_1 - \q_0 \right)}
\over
{2 A(\q_{031})^2 } }
\nonumber
\end{eqnarray}

%------------------------------------------------------------------


\documentclass[twoside]{MATH77}
\usepackage{multicol}
\usepackage[fleqn,reqno,centertags]{amsmath}
\begin{document}
\begmath 7.3 Compute Polynomial Coefficients from Roots

\silentfootnote{$^\copyright$1997 Calif. Inst. of Technology, \thisyear \ Math \`a la Carte, Inc.}

\subsection{Purpose}

Given $n$ complex numbers, $z_i$, compute (complex) coefficients, $c_i$,
such that the monic polynomial defined by%
\begin{equation*}
p(z)=c_1z^n+...+c_nz+c_{n+1}
\end{equation*}
has the numbers $z_i$ as its roots. The coefficients will be computed from
the relation%
\begin{equation*}
\hspace{-10pt}c_1z^n+.\text{ . . }+c_nz+c_{n+1}=(z-z_1)(z-z_2)^{...}(z-z_n)
\end{equation*}
Note that $c_1$ is always set to~1.

\subsection{Usage}

\subsubsection{Program Prototype, Single Precision}

\begin{description}
\item[\bf INTEGER]  \ {\bf NDEG}
\item[\bf COMPLEX]  \ {\bf ROOTS}$(\geq $NDEG),{\bf COEFS}$(\geq $NDEG+1)
\end{description}
Assign values to NDEG and ROOTS().
$$
\fbox{{\bf CALL CCOEF (NDEG, ROOTS, COEFS)}}
$$
Computed quantities are returned in COEFS().

\subsubsection{Argument Definitions}

\begin{description}
\item[NDEG]  [in] Number of roots given in ROOTS(), and thus the degree of
the polynomial whose coefficients are to be computed.
\item[ROOTS()]  [in] Roots, given as complex numbers.
\item[COEFS()]  [out] Computed coefficients, stored as complex numbers. The
arrays COEFS() and ROOTS() must be distinct. The coefficient, $c_1$, will be
the coefficient of $z^{NDEG}$ and will be set to~1.
\end{description}

\subsubsection{Modifications for Double Precision}

For double precision usage change the subroutine name from CCOEF to ZCOEF.
Recall that the Fortran~77 standard does not support a double precision
complex data type, although many Fortran compilers do, using the
declaration, COMPLEX*16. To remain within the Fortran~77 standard, use the
declarations
\begin{description}
\item[DOUBLE PRECISION]  {\bf ROOTS}(2, $\geq $ NDEG$)$,
{\bf COEFFS}(2, $\geq $NDEG$+1)$
\end{description}

and use the convention that real and imaginary parts of complex numbers are
associated with the values 1 and~2, respectively, of the first subscript.
This usage is compatible with the Fortran~90 standard.  Alternatively, if
the COMPLEX*16 declaration is available and is compatible with this storage
convention, one may use the nonstandard declaration
\begin{description}
\item[COMPLEX*16]  {\bf ROOTS}$(\geq $ NDEG$)$,
{\bf COEFFS}$(\geq $ NDEG$+1)$
\end{description}

\subsection{Examples and Remarks}

The program, DRZCOEF, with its output, ODZCOEF, illustrates the use of ZCOEF
to compute the coefficients of a quadratic and a cubic polynomial.

If this subroutine is used to assess the accuracy of a polynomial root
finder we suggest use of the double precision version, even if it is a single precision
root finder, to reduce the introduction of errors from the process of
computing the polynomial coefficients.

\subsection{Functional Description}

\subparagraph{Method}

The degree, NDEG, and roots, $z_1$, ..., $z_{NDEG}$, are given. The
coefficients, $c_i$, are computed by the following algorithm. The quantities
$z_i$ and $c_i$ are complex. In the double precision version the complex arithmetic is
coded in-line in terms of operations on the real and imaginary parts to
conform to the Fortran~77 standard.
\begin{tabbing}
\hspace{.2in}\=$c_1 = 1.0$\\
\>if( NDEG .le. 0 ) return\\
\>$c_2 = -z_1$\\
\>do $i = 2$, NDEG\\
\>\ \ \ \ \=$c_{i+1} = -c_i * z_i$\\
\>\>do $j = i$, 2, $-$1\\
\>\>\ \ \ \ $c_j = c_j - c_{j-1} * z_i$\\
\>\>enddo\\
\>enddo\\
\>return
\end{tabbing}

\subparagraph{Accuracy tests}

The logic and the accuracy of this code were checked by use with a root
finder. The accuracy was consistent with the computer system being used.

\subsection{Error Procedures and Restrictions}

If NDEG $\leq$ 0 the subroutine returns, setting COEFS$(1) = 1$. The arrays
ROOTS() and COEFS() must occupy distinct storage locations.

\subsection{Supporting Information}

The source language for these subroutines is ANSI Fortran 77.

\begin{tabular}{@{\bf}l@{\hspace{5pt}}l}
Entry & \hspace{.35in} {\bf Required Files}\vspace{2pt}\\
CCOEF & \hspace{.35in} CCOEF\rule[-5pt]{0pt}{8pt}\\
ZCOEF & \hspace{.35in} ZCOEF
\end{tabular}

Designed by C. L. Lawson, JPL, May~1986.

Programmed by C. L. Lawson and S. Y. Chiu, JPL, May~1986, Feb.~1987.

\begcodenp
\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRZCOEF}\vspace{10pt}
\lstinputlisting{\codeloc{zcoef}}

\vspace{30pt}\centerline{\bf \large ODZCOEF}\vspace{10pt}
\lstset{language={}}
\lstinputlisting{\outputloc{zcoef}}
\end{document}


%% LyX 2.3.1-1 created this file.  For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[english]{extarticle}
\usepackage[T1]{fontenc}
\usepackage[latin9]{inputenc}
\usepackage{geometry}
\geometry{verbose,tmargin=2.54cm,bmargin=2.54cm,lmargin=2.54cm,rmargin=2.54cm}
\usepackage{bm}
\usepackage{esint}

\makeatletter
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% User specified LaTeX commands.
\setlength\parindent{0pt}

\makeatother

\usepackage{babel}
\begin{document}
{\LARGE{}Notes on the Taylor Green Vortex}{\LARGE\par}

Comparing Domain $(0,2\pi)^{3}$ to $(0,1)^{3}$

B.A. Wingate

\section{Introduction}

The Taylor Green vortex solution is used to compare/debug and test
DNS codes. The Sandia/LANL DNS code has a domain $(0,1)^{3}$ while
many DNS codes use the default domain $(0,2\pi)^{3}$ and this will
cause differences in the solution, such as the energy decay rate,
to occur. This document dicusses some of the details of this computation.
It has a companion sympy jupyter notebook titled \texttt{\textit{TaylorGreenExperiments.ipynb}}.

\section{In two-dimensions in $(0,2\pi)^{2}$ }

The equations we solve are,
\begin{eqnarray}
\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial x}+v\frac{\partial u}{\partial y} & = & -\frac{1}{\rho_{0}}\frac{\partial p}{\partial x}+\nu\left(\frac{\partial^{2}u}{\partial x^{2}}+\frac{\partial^{2}u}{\partial y^{2}}\right),\label{eq:nsu}\\
\frac{\partial v}{\partial t}+u\frac{\partial v}{\partial x}+v\frac{\partial v}{\partial y} & = & -\frac{1}{\rho_{0}}\frac{\partial p}{\partial y}+\nu\left(\frac{\partial^{2}v}{\partial x^{2}}+\frac{\partial^{2}v}{\partial y^{2}}\right),\label{eq:nsv}\\
\frac{\partial u}{\partial x}+\frac{\partial v}{\partial y} & = & 0.\label{eq:divu}
\end{eqnarray}

On $\Omega\in(0,2\pi)^{3}$ with initial condition,
\begin{eqnarray*}
u(\bm{x},0) & = & \sin(x)\cos(y),\\
v(\bm{x},0) & = & -\cos(x)\sin(y).
\end{eqnarray*}
 This has the following exact solution,
\begin{eqnarray*}
u(\bm{x},t) & = & \sin(x)\cos(y)e^{-2\nu t},\\
v(\bm{x},t) & = & -\cos(x)\sin(y)e^{-2\nu t},\\
p(\bm{x},t) & = & -\left(\cos(2x)+\cos(2y)\right)e^{-4\nu t}.
\end{eqnarray*}
It is quite interesting to me to be reminded that the pressure has
to decay like twice the others because the others are nonlinear (it
is more like u.u). This therefore has the total energy per unit area
decay like,
\[
E=\frac{1}{\Omega}\frac{1}{2}\int_{\Omega}\left(u^{2}+v^{2}\right)d\Omega\quad\Omega\in(0,2\pi)^{2}
\]
 which gives,
\begin{eqnarray*}
E(t) & = & \frac{\pi^{2}}{4\pi^{2}}e^{-4\nu t}\quad\Omega\in(0,2\pi)^{3}\\
 & = & \frac{1}{4}e^{-4\nu t}
\end{eqnarray*}
We can also compute the intitial energy, $E(0),$
\begin{eqnarray*}
E(0) & = & \frac{1}{4\pi^{2}}\frac{1}{2}\int_{0}^{2\pi}\int_{0}^{2\pi}\sin^{2}(x)\cos^{2}(y)+\cos^{2}(x)\sin^{2}(y)\,dx\,dy\\
 & = & \frac{1}{4\pi^{2}}\frac{1}{2}(2\pi^{2})=\frac{1}{4}
\end{eqnarray*}

\section{In two-dimensions in $(0,1)^{2}$ }

The equations are the same as (\ref{eq:nsu})-(\ref{eq:divu}), but
these are the following changes,

On $\Omega\in(0,1)$ with initial condition,
\begin{eqnarray*}
u(\bm{x},0) & = & \sin(2\pi x)\cos(2\pi y),\\
v(\bm{x},0) & = & -\cos(2\pi x)\sin(2\pi y).
\end{eqnarray*}
 This has the following exact solution,
\begin{eqnarray*}
u(\bm{x},t) & = & \sin(2\pi x)\cos(2\pi y)e^{-2\nu(2\pi)^{2}t},\\
v(\bm{x},t) & = & -\cos(2\pi x)\sin(2\pi y)e^{-2\nu(2\pi)^{2}t},\\
p(\bm{x},t) & = & -\left(\cos(2\left(2\pi x\right))+\cos(2\left(2\pi y\right))\right)e^{-4\nu(2\pi)^{2}t}.
\end{eqnarray*}
 This therefore has the total energy decay like,
\[
E=\frac{1}{\Omega}\frac{1}{2}\int_{\Omega}\left(u^{2}+v^{2}\right)d\Omega\quad\Omega\in(0,1)^{2}
\]
 which gives,
\[
E(t)=\frac{1}{2}e^{-4\nu(2\pi)^{2}t}.
\]
 Computing the energy at $t=0$,
\begin{eqnarray*}
E(0) & = & \frac{1}{1}\frac{1}{2}\int_{0}^{1}\int_{0}^{1}\sin^{2}(2\pi x)\cos^{2}(2\pi y)+\cos^{2}(2\pi x)\sin^{2}(2\pi y)\,dx\,dy\\
 & = & \frac{1}{2}(\frac{1}{2})=\frac{1}{4}.
\end{eqnarray*}

\section{In three-dimensions in $(0,2\pi)^{3}$ }

The equations we solve are,
\begin{eqnarray}
\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial x}+v\frac{\partial u}{\partial y}+w\frac{\partial u}{\partial z} & = & -\frac{1}{\rho_{0}}\frac{\partial p}{\partial x}+\nu\left(\frac{\partial^{2}u}{\partial x^{2}}+\frac{\partial^{2}u}{\partial y^{2}}+\frac{\partial^{2}u}{\partial z^{2}}\right),\label{eq:nsu-1}\\
\frac{\partial v}{\partial t}+u\frac{\partial v}{\partial x}+v\frac{\partial v}{\partial y}+w\frac{\partial v}{\partial z} & = & -\frac{1}{\rho_{0}}\frac{\partial p}{\partial y}+\nu\left(\frac{\partial^{2}v}{\partial x^{2}}+\frac{\partial^{2}v}{\partial y^{2}}+\frac{\partial^{2}v}{\partial z^{2}}\right),\label{eq:nsv-1}\\
\frac{\partial w}{\partial t}+u\frac{\partial w}{\partial x}+v\frac{\partial w}{\partial y}+w\frac{\partial w}{\partial z} & = & -\frac{1}{\rho_{0}}\frac{\partial p}{\partial z}+\nu\left(\frac{\partial^{2}w}{\partial x^{2}}+\frac{\partial^{2}w}{\partial y^{2}}+\frac{\partial^{2}w}{\partial z^{2}}\right),\\
\frac{\partial u}{\partial x}+\frac{\partial v}{\partial y}+\frac{\partial w}{\partial z} & = & 0.\label{eq:divu-1}
\end{eqnarray}
On $\Omega\in(0,2\pi)^{3}$ with initial condition,
\begin{eqnarray*}
u(\bm{x},0) & = & \sin(x)\cos(y)\cos(z),\\
v(\bm{x},0) & = & -\cos(x)\sin(y)\cos(z).\\
w(\bm{x},0) & = & 0.
\end{eqnarray*}

I think this has exact solution,
\begin{eqnarray*}
u(\bm{x},t) & = & \sin(x)\cos(y)\cos(z)e^{-3\nu t},\\
v(\bm{x},t) & = & -\cos(x)\sin(y)\cos(z)e^{-3\nu t},\\
w(\bm{x},t) & = & ?\\
p(\bm{x},t) & = & -\left(\cos(2x)+\cos(2y)\right)\left(\cos(z)\right)^{2}e^{-6\nu t}.
\end{eqnarray*}
 and therefor this has the following energy per unit volume,

\begin{eqnarray*}
E & = & \frac{1}{8\pi^{3}}\frac{1}{2}\int_{\Omega}\left(u^{2}+v^{2}+w^{2}\right)d\Omega\quad\Omega\in(0,2\pi)^{3}\\
 & \approx & \frac{1}{8\pi^{3}}\frac{1}{2}e^{-6\nu t}\int_{\Omega}\left(u^{2}+v^{2}+w^{2}\right)d\Omega\quad\Omega\in(0,2\pi)^{3}
\end{eqnarray*}

Computing the initial energy,
\begin{eqnarray*}
E(0) & = & \frac{1}{8\pi^{3}}\frac{1}{2}\int_{0}^{2\pi}\int_{0}^{2\pi}\int_{0}^{2\pi}\sin^{2}(x)\cos^{2}(y)\cos^{2}(z)+\cos^{2}(x)\sin^{2}(y)\cos^{2}(z)\,dx\,dy\,dz\\
 & = & \frac{1}{8\pi^{3}}\frac{1}{2}\left(2\pi^{3}\right)=\frac{1}{8}
\end{eqnarray*}

\section{In three-dimensions in $(0,1)^{3}$ }

The equations we solve are,
\begin{eqnarray}
\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial x}+v\frac{\partial u}{\partial y}+w\frac{\partial u}{\partial z} & = & -\frac{1}{\rho_{0}}\frac{\partial p}{\partial x}+\nu\left(\frac{\partial^{2}u}{\partial x^{2}}+\frac{\partial^{2}u}{\partial y^{2}}+\frac{\partial^{2}u}{\partial z^{2}}\right),\label{eq:nsu-1-1}\\
\frac{\partial v}{\partial t}+u\frac{\partial v}{\partial x}+v\frac{\partial v}{\partial y}+w\frac{\partial v}{\partial z} & = & -\frac{1}{\rho_{0}}\frac{\partial p}{\partial y}+\nu\left(\frac{\partial^{2}v}{\partial x^{2}}+\frac{\partial^{2}v}{\partial y^{2}}+\frac{\partial^{2}v}{\partial z^{2}}\right),\label{eq:nsv-1-1}\\
\frac{\partial w}{\partial t}+u\frac{\partial w}{\partial x}+v\frac{\partial w}{\partial y}+w\frac{\partial w}{\partial z} & = & -\frac{1}{\rho_{0}}\frac{\partial p}{\partial z}+\nu\left(\frac{\partial^{2}w}{\partial x^{2}}+\frac{\partial^{2}w}{\partial y^{2}}+\frac{\partial^{2}w}{\partial z^{2}}\right),\\
\frac{\partial u}{\partial x}+\frac{\partial v}{\partial y}+\frac{\partial w}{\partial z} & = & 0.\label{eq:divu-1-1}
\end{eqnarray}
On $\Omega\in(0,2\pi)^{3}$ with initial condition,
\begin{eqnarray*}
u(\bm{x},0) & = & \sin(2\pi x)\cos(2\pi y)\cos(2\pi z),\\
v(\bm{x},0) & = & -\cos(2\pi x)\sin(2\pi y)\cos(2\pi z).\\
w(\bm{x},0) & = & 0.
\end{eqnarray*}

I think this has exact solution,
\begin{eqnarray*}
u(\bm{x},t) & = & \sin(2\pi x)\cos(2\pi y)\cos(2\pi z)e^{-3\left(2\pi\right)^{2}\nu t},\\
v(\bm{x},t) & = & -\cos(2\pi x)\sin(2\pi y)\cos(2\pi z)e^{-3\left(2\pi\right)^{2}\nu t},\\
w(\bm{x},t) & = & ?\\
p(\bm{x},t) & = & -\left(\cos(2x2\pi)+\cos(2y2\pi)\right)\left(\cos(2\pi z)\right)^{2}e^{-6\left(2\pi\right)^{2}\nu t}.
\end{eqnarray*}
 and therefor this has the following energy,

\begin{eqnarray*}
E & = & \frac{1}{2}\int_{\Omega}\left(u^{2}+v^{2}+w^{2}\right)d\Omega\quad\Omega\in(0,2\pi)^{3}\\
 & \approx & \frac{1}{2}e^{-6\left(2\pi\right)^{2}\nu t}\int_{\Omega}\left(u^{2}+v^{2}+w^{2}\right)d\Omega\quad\Omega\in(0,2\pi)^{3}
\end{eqnarray*}
Computing the energy at $t=0,$
\begin{eqnarray*}
E(0) & = & \frac{1}{1}\frac{1}{2}\int_{0}^{1}\int_{0}^{1}\int_{0}^{1}\sin^{2}(2\pi x)\cos^{2}(2\pi y)\cos^{2}(2\pi z)+\cos^{2}(2\pi x)\sin^{2}(2\pi y)\cos^{2}(2\pi z)\,dx\,dy\,dz\\
 & = & \frac{1}{1}\frac{1}{2}\left(\frac{1}{4}\right)=\frac{1}{8}
\end{eqnarray*}

\end{document}


\section{General definitions and notation}

\begin{myNot}[Spatial dimension\coderm{BoSSS.Foundation.Grid.Simplex.SpatialDimension}]
Will be denoted as $D \in \natWoZero$ in below;
\end{myNot}

\begin{myNot}[standard symbols]
Within this text, the following symbols are used:
\begin{packed_itemize}
  \item the unit sphere $\mathcal{S}_D = \{ \vec{x} \in \real^D; \ |\vec{x}|_2 < 1 \}$
  \item the symmetric group $S^n = \{ \tau:\{0,\cdots,n-1\}\rightarrow\{0,\cdots,n-1\}; \tau \textrm{ is bijective} \}$, i.e. the group of all permutations of length $n$;
  \item the Sobolev space
        $H^l(\Omega) = \left\{ f \in L^2(\Omega); \ \sum_{|\alpha | \leq l} \left\| \partial^\alpha f \right\|_2^2 < \infty \right\}$,
        where $\alpha$ denotes a multindex.
\end{packed_itemize}
\end{myNot}

\begin{myNot}[inner product] The inner product in a domain $K \subset \real^D$,
for $f,g \in L^2(K)$ will be denoted and is defined (in BoSSS) by
\[
  \langle f,g \rangle_K := \int_K f(\vec{x}) \cdot g(\vec{x}) \ d\vec{x}.
\]
\end{myNot}

\begin{myDef}[Convex Hull]
For a given list of vertices,
$\left( \vec{v}_1,\ldots,\vec{v}_n \right) \in \real^{D \times n}$
the ($D$-dimensional) convex hull is defined as the set
\[
  \left\{ \sum_{i=1}^n \alpha_i \cdot \vec{v}_i \in \real^D, \ \textrm{where} \ \sum_{i=1}^n \alpha_i \leq 1 \right\}
  =:
  CH \left( \vec{v}_1,\ldots,\vec{v}_n \right) \subset \real^D
\]
\end{myDef}

\begin{myDef}[Quadrature rule]
A quadrature rule with $k$ Nodes within a bounded domain $K \subset \real^D$ is a Tuple
\[
 \left[ \left( \vec{y}_i\right)_{i=0,\ldots,k-1}, \left( \beta_i \right)_{i=0,\ldots,k-1}, p_\textrm{ord} \right]
 \in \real^{D\times k} \times \real^k \times \nat
\]
for which the equation
\[
  \int_{\vec{x} \in K} p( \vec{x} ) \ d\vec{x} = \sum_{i=0}^{k-1} p(\vec{y}_i) \cdot \beta_i
\]
holds for all polynomials $p$ with $\deg(p) \leq p_\textrm{ord}$. The rule is called to
be of order $p_\textrm{ord}$, $\vec{y}_i$ are called the nodes and $\beta_i$ are called the weights of the
quadrature rule.
\end{myDef}


\section{Linear algebra}\label{sec:linear_algebra}

Linear algebra is a branch of mathematics that is both very accessible and enormously useful. It studies \hyperref[def:vector_space]{vector spaces}, mostly over \hyperref[def:set_of_real_numbers]{real} or \hyperref[def:set_of_complex_numbers]{complex} numbers, and \hyperref[def:linear_operator]{linear maps} between them. For \hyperref[def:vector_space_dimension]{finite-dimensional} vector spaces, this reduces to studying \hyperref[def:array/matrix]{matrices}, which also leads to a very rich computational theory.


% ------------------------------------------------------------------
\chapter{Neural Network Computations}\label{s:fundamentals}
% ------------------------------------------------------------------

This chapter provides a brief introduction to the computational aspects of neural networks, and convolutional neural networks in particular, emphasizing the concepts required to understand and use \matconvnet.

% ------------------------------------------------------------------
\section{Overview}\label{s:cnn-structure}
% ------------------------------------------------------------------

A \emph{Neural Network} (NN) is a function $g$ mapping data $\bx$, for example an image, to an output vector $\by$, for example an image label. The function $g=f_L \circ \dots \circ f_1$ is the composition of a sequence of simpler functions $f_l$, which are called \emph{computational blocks} or \emph{layers}. Let $\bx_1,\bx_2,\dots,\bx_L$ be the outputs of each layer in the network, and let $\bx_0=\bx$ denote the network input. Each intermediate output $\bx_l = f_l(\bx_{l-1};\bw_l)$ is computed from the previous output $\bx_{l-1}$  by applying the function $f_l$ with parameters $\bw_l$.

In a \emph{Convolutional Neural Network} (CNN), the data has a spatial structure: each $\bx_l\in\mathbb{R}^{H_l \times W_l \times C_l}$ is a 3D array or \emph{tensor} where the first two dimensions $H_l$ (height) and $W_l$ (width) are interpreted as spatial dimensions. The third dimension $C_l$ is instead interpreted as the \emph{number of feature channels}. Hence, the tensor $\bx_l$ represents a $H_l \times W_l$ field of $C_l$-dimensional feature vectors, one for each spatial location. A fourth dimension $N_l$ in the tensor spans multiple data samples packed in a single \emph{batch} for efficiency parallel processing. The number of data samples $N_l$ in a batch is called the batch \emph{cardinality}. The network is called \emph{convolutional} because the functions $f_l$ are local and translation invariant operators (i.e.\ non-linear filters) like linear convolution.

It is also possible to conceive CNNs with more than two spatial dimensions, where the additional dimensions may represent volume or time. In fact, there are little \emph{a-priori} restrictions on the format of data in neural networks in general. Many useful NNs contain a mixture of convolutional layers together with layer that process other data types such as text strings, or perform other operations that do not strictly conform  to the CNN assumptions.

\matconvnet includes a variety of layers, contained in the !matlab/! directory, such as !vl_nnconv! (convolution), !vl_nnconvt! (convolution transpose or deconvolution), !vl_nnpool! (max and average pooling), !vl_nnrelu! (ReLU activation), !vl_nnsigmoid! (sigmoid activation), !vl_nnsoftmax! (softmax operator), !vl_nnloss! (classification log-loss), !vl_nnbnorm! (batch normalization), !vl_nnspnorm! (spatial normalization), !vl_nnnormalize! (locar response normalization -- LRN), or !vl_nnpdist! ($p$-distance).  There are enough layers to implement many interesting state-of-the-art networks out of the box, or even import them from other toolboxes such as Caffe.

NNs are often used as classifiers or regressors. In the example of \cref{f:demo}, the output $\hat \by = f(\bx)$ is a vector of probabilities, one for each of a 1,000 possible image labels (dog, cat, trilobite, ...).  If $\by$ is the true label of image $\bx$, we can measure the CNN performance by a loss function $\ell_\by(\hat \by)  \in \mathbb{R}$ which assigns a penalty to classification errors. The CNN parameters can then be tuned or \emph{learned} to minimize this loss averaged over a large dataset of labelled example images.

Learning generally uses a variant of \emph{stochastic gradient descent} (SGD). While this is an efficient method (for this type of problems), networks may contain several million parameters and need to be trained on millions of images; thus, efficiency is a paramount in \matlab design, as further discussed in \cref{s:speed}. SGD also requires to compute the CNN derivatives, as explained in the next section.

% ------------------------------------------------------------------
\section{Network structures}\label{s:cnn-topology}
% ------------------------------------------------------------------

In the simplest case, layers in a NN are arranged in a sequence; however, more complex interconnections are possible as well, and in fact very useful in many cases. This section discusses such configurations and introduces a graphical notation to visualize them.

% ------------------------------------------------------------------
\subsection{Sequences}\label{s:cnn-simple}
% ------------------------------------------------------------------

Start by considering a computational block $f$ in the network. This can be represented schematically as a box receiving data $\bx$ and parameters $\bw$ as inputs and producing data $\by$ as output:
\begin{center}
\begin{tikzpicture}[auto, node distance=2cm]
\node (x) [data] {$\bx$};
\node (f) [block,right of=x]{$f$};
\node (y) [data, right of=f] {$\by$};
\node (w) [data, below of=f] {$\bw$};
\draw [to] (x.east) -- (f.west) {};
\draw [to] (f.east) -- (y.west) {};
\draw [to] (w.north) -- (f.south) {};
\end{tikzpicture}
\end{center}
As seen above, in the simplest case blocks are chained in a sequence $f_1 \rightarrow f_2\rightarrow\dots\rightarrow f_L$ yielding the structure:
\begin{center}
\begin{tikzpicture}[auto, node distance=2cm]
\node (x0)  [data] {$\bx_0$};
\node (f1) [block,right of=x0]{$f_1$};
\node (f2) [block,right of=f1,node distance=3cm]{$f_2$};
\node (dots) [right of=f2]{...};
\node (fL) [block,right of=dots]{$f_L$};
\node (xL)  [data, right of=fL] {$\bx_L$};
\node (w1) [data, below of=f1] {$\bw_1$};
\node (w2) [data, below of=f2] {$\bw_2$};
\node (wL) [data, below of=fL] {$\bw_L$};
\draw [to] (x0.east) -- (f1.west) {};
\draw [to] (f1.east) -- node {$\bx_1$} (f2.west);
\draw [to] (f2.east) -- node {$\bx_2$} (dots.west) {};
\draw [to] (dots.east) -- node {$\bx_{L-1}$} (fL.west) {};
\draw [to] (fL.east) -- (xL.west) {};
\draw [to] (w1.north) -- (f1.south) {};
\draw [to] (w2.north) -- (f2.south) {};
\draw [to] (wL.north) -- (fL.south) {};
\end{tikzpicture}
\end{center}
Given an input $\bx_0$, evaluating the network is a simple matter of evaluating all the blocks from left to right, which defines a composite function $\bx_L = f(\bx_0;\bw_1,\dots,\bw_L)$.

% ------------------------------------------------------------------
\subsection{Directed acyclic graphs}\label{s:cnn-dag}
% ------------------------------------------------------------------

\begin{figure}[t]
\begin{center}
\begin{tikzpicture}[auto, node distance=0.4cm]
 \matrix (m) [matrix of math nodes,
    column sep=1.2cm,
    row sep=0.4cm]
{
& \node (f1) [block]{f_1};
& \node (x1) [datac]{\bx_1};
\\
\node (x0) [datac]{\bx_0};
&
&
& \node (f3) [block]{f_3};
& \node (x3) [datac]{\bx_3};
\\
& \node (f2) [block]{f_2};
& \node (x2) [datac]{\bx_2};
& &
& \node (f5) [block]{f_5};
& \node (x7) [datac]{\bx_7};
\\
&
& \node(x5) [datac]{\bx_5};
\\
\node (x4) [datac]{\bx_4};
& \node (f4) [block]{f_4};
\\
&
& \node(x6) [datac]{\bx_6};
\\
};
\draw[to] (x0) -- (f1);
\draw[to] (f1) -- (x1);
\draw[to] (x1) -- (f3);
\draw[to] (x0) -- (f2);
\draw[to] (f2) -- (x2);
\draw[to] (x2) -- (f3);
\draw[to] (f3) -- (x3);
\draw[to] (x3) -- (f5);
\draw[to] (f5) -- (x7);
\draw[to] (x4) -- (f4);
\draw[to] (f4) -- (x5);
\draw[to] (f4) -- (x6);
\draw[to] (x5) -- (f5);
\node(w1) [par,below=of f1]{$\bw_1$}; \draw[to] (w1) -- (f1);
\node(w2) [par,below=of f2]{$\bw_2$}; \draw[to] (w2) -- (f2);
%\node(w3) [par,below=of f3]{$\bw_3$}; \draw[to] (w3) -- (f3);
\node(w4) [par,below=of f4]{$\bw_4$}; \draw[to] (w4) -- (f4);
\draw[to] (w4) to [bend right] (f3);
\node(w5) [par,below=of f5]{$\bw_5$}; \draw[to] (w5) -- (f5);
\end{tikzpicture}
\end{center}
\vspace{-1em}
\caption{\textbf{Example DAG.}}\label{f:dag}
\end{figure}

One is not limited to chaining layers one after another. In fact, the only requirement for evaluating a NN is that, when a layer has to be evaluated, all its input have been evaluated prior to it. This is possible exactly when the interconnections between layers form a \emph{directed acyclic graph}, or DAG for short.

In order to visualize DAGs, it is useful to introduce additional nodes for the network variables, as in the  example of Fig.~\ref{f:dag}. Here boxes denote functions and circles denote variables (parameters are treated as a special kind of variables). In the example, $\bx_0$ and $\bx_4$ are the inputs of the CNN and $\bx_6$ and $\bx_7$ the outputs. Functions can take any number of inputs (e.g. $f_3$ and $f_5$ take two) and have any number of outputs (e.g. $f_4$ has two). There are a few noteworthy properties of this graph:

\begin{enumerate}
\item The graph is bipartite, in the sense that arrows always go from boxes to circles and from circles to boxes.
\item Functions can have any number of inputs or outputs; variables and parameters can have an arbitrary number of outputs (a parameter with more of one output is \emph{shared} between different layers); variables have at most one input and parameters none.
\item Variables with no incoming arrows and parameters are not computed by the network, but must be set prior to evaluation, i.e.\ they are \emph{inputs}. Any variable (or even parameter) may be used as output, although these are usually the variables with no outgoing arrows.
\item Since the graph is acyclic, the CNN can be evaluated by sorting the functions and computing them one after another (in the example, evaluating the functions in the order $f_1,f_2,f_3,f_4,f_5$ would work).
\end{enumerate}

% ------------------------------------------------------------------
\section{Computing derivatives with backpropagation}\label{s:back}
% ------------------------------------------------------------------

Learning a NN requires computing the derivative of the loss with respect to the network parameters. Derivatives are computed using an algorithm called \emph{backpropagation}, which is a memory-efficient implementation of the chain rule for derivatives. First, we discuss the derivatives of a single layer, and then of a whole network.

\subsection{Derivatives of tensor functions}

In a CNN, a layer is a function $\by = f(\bx)$ where both input $\bx \in \mathbb{R}^{H\times W \times C}$ and output $\by \in \mathbb{R}^{H'\times W' \times C'}$ are tensors. The derivative of the function $f$ contains the derivative of each output component $y_{i'j'k'}$ with respect to each input component $x_{ijk}$, for a total of $H'\times W'\times C'\times H\times W\times C$ elements naturally arranged in a 6D tensor. Instead of expressing derivatives as tensors, it is often useful  to switch to a matrix notation by \emph{stacking} the input and output tensors into vectors. This is done by the $\vv$ operator, which visits each element of a tensor in lexicographical order and produces a vector:
\[
  \vv \bx
  =
  \begin{bmatrix}
  x_{111} \\
  x_{211} \\
  \vdots
  \\
  x_{H11} \\
  x_{121} \\
  \vdots \\
  x_{HWC}
  \end{bmatrix}.
\]
By stacking both input and output, each layer $f$ can be seen reinterpreted as vector function $\vv f$, whose derivative is the conventional Jacobian matrix:
\[
\renewcommand*{\arraystretch}{1.5}
\frac{d \vv f}{d(\vv \bx)^\top}
=
\begin{bmatrix}
\frac{\partial y_{111}}{\partial x_{111}} &
\frac{\partial y_{111}}{\partial x_{211}} &
\dots &
\frac{\partial y_{111}}{\partial x_{H11}} &
\frac{\partial y_{111}}{\partial x_{121}} &
\dots &
\frac{\partial y_{111}}{\partial x_{HWC}} \\
\frac{\partial y_{211}}{\partial x_{111}} &
\frac{\partial y_{211}}{\partial x_{211}} &
\dots &
\frac{\partial y_{211}}{\partial x_{H11}} &
\frac{\partial y_{211}}{\partial x_{121}} &
\dots &
\frac{\partial y_{211}}{\partial x_{HWC}} \\
\vdots & \vdots & \dots & \vdots & \vdots & \dots & \vdots \\
\frac{\partial y_{H'11}}{\partial x_{111}} &
\frac{\partial y_{H'11}}{\partial x_{211}} &
\dots &
\frac{\partial y_{H'11}}{\partial x_{H11}} &
\frac{\partial y_{H'11}}{\partial x_{121}} &
\dots &
\frac{\partial y_{H'11}}{\partial x_{HWC}} \\
\frac{\partial y_{121}}{\partial x_{111}} &
\frac{\partial y_{121}}{\partial x_{211}} &
\dots &
\frac{\partial y_{121}}{\partial x_{H11}} &
\frac{\partial y_{121}}{\partial x_{121}} &
\dots &
\frac{\partial y_{121}}{\partial x_{HWC}} \\
\vdots & \vdots & \dots & \vdots & \vdots & \dots & \vdots \\
\frac{\partial y_{H'W'C'}}{\partial x_{111}} &
\frac{\partial y_{H'W'C'}}{\partial x_{211}} &
\dots &
\frac{\partial y_{H'W'C'}}{\partial x_{H11}} &
\frac{\partial y_{H'W'C'}}{\partial x_{121}} &
\dots &
\frac{\partial y_{H'W'C'}}{\partial x_{HWC}}
\end{bmatrix}.
\]
This notation for the derivatives of tensor functions is taken from~\cite{kinghorn96integrals} and is used throughout this document.

While it is easy to express the derivatives of tensor functions as matrices, these matrices are in general extremely large. Even for moderate data sizes (e.g. $H=H'=W=W'=32$ and $C=C'=128$), there are $H'W'C'HWC \approx 17 \times 10^9$ elements in the Jacobian. Storing that requires 68 GB of space in single precision. The purpose of the backpropagation algorithm is to compute the derivatives required for learning without incurring this huge memory cost.

\subsection{Derivatives of function compositions}

In order to understand backpropagation, consider first a simple CNN terminating in a loss function $f_L = \ell_\by$:
\begin{center}
\begin{tikzpicture}[auto, node distance=2cm]
\node (x0)  [data] {$\bx_0$};
\node (f1) [block,right of=x0]{$f_1$};
\node (f2) [block,right of=f1,node distance=3cm]{$f_2$};
\node (dots) [right of=f2]{...};
\node (fL) [block,right of=dots]{$f_L$};
\node (w1) [data, below of=f1] {$\bw_1$};
\node (w2) [data, below of=f2] {$\bw_2$};
\node (wL) [data, below of=fL] {$\bw_L$};
\node (xL) [data, right of=fL] {$x_l\in\real$};
\draw [to] (x0.east) -- (f1.west) {};
\draw [to] (f1.east) -- node {$\bx_1$} (f2.west);
\draw [to] (f2.east) -- node {$\bx_2$} (dots.west) {};
\draw [to] (dots.east) -- node {$\bx_{L-1}$} (fL.west) {};
\draw [to] (fL.east) -- (xL.west) {};
\draw [to] (w1.north) -- (f1.south) {};
\draw [to] (w2.north) -- (f2.south) {};
\draw [to] (wL.north) -- (fL.south) {};
\end{tikzpicture}
\end{center}
The goal is to compute the gradient of the loss value $x_L$ (output) with respect to each network parameter $\bw_l$:
\[
\frac{df}{d(\vv \bw_l)^\top} =
\frac{d}{d(\vv \bw_l)^\top}
\left[f_L(\cdot;\bw_L) \circ ... \circ
f_2(\cdot;\bw_2) \circ f_1(\bx_0;\bw_1)\right].
\]
By applying the chain rule and by using the matrix notation introduced above, the derivative can be written as
\begin{equation}\label{e:chain-rule}
\frac{df}{d(\vv \bw_l)^\top}
=
\frac{d\vv f_L(\bx_{L-1};\bw_{L})}{d(\vv\bx_{L-1})^\top}
\times
\dots
\times
\frac{d\vv f_{l+1}(\bx_{l};\bw_{l+1})}{d(\vv\bx_{l})^\top}
\times
\frac{d\vv f_l(\bx_{l-1};\bw_{l})}{d(\vv\bw_l^\top)}
\end{equation}
where the derivatives are computed at the working point determined by the input $\bx_0$ and the current value of the parameters.

Note that, since the network output $x_l$ is a \emph{scalar} quantity, the target derivative $df/d(\vv \bw_l)^\top$ has the same number of elements of the parameter vector $\bw_l$, which is moderate. However, the intermediate Jacobian factors have, as seen above, an unmanageable size. In order to avoid computing these factor explicitly, we can proceed as follows.

Start by multiplying the output of the last layer by a tensor $p_L=1$ (note that this tensor is a scalar just like the variable $x_L$):
\begin{align*}
p_L \times \frac{df}{d(\vv \bw_l)^\top}
&=
\underbrace{p_L \times \frac{d\vv f_L(\bx_{L-1};\bw_{L})}{d(\vv\bx_{L-1})^\top}}_{(\vv \bp_{L-1})^\top}
\times
\dots
\times
\frac{d\vv f_{l+1}(\bx_{l};\bw_{l+1})}{d(\vv\bx_{l})^\top}
\times
\frac{d\vv f_l(\bx_{l-1};\bw_{l})}{d(\vv\bw_l^\top)}
\\
&=
(\vv \bp_{L-1})^\top
\times
\dots
\times
\frac{d\vv f_{l+1}(\bx_{l};\bw_{l+1})}{d(\vv\bx_{l})^\top}
\times
\frac{d\vv f_l(\bx_{l-1};\bw_{l})}{d(\vv\bw_l^\top)}
\end{align*}
In the second line the last two factors to the left have been multiplied obtaining a new tensor $\bp_{L-1}$ that has the same size as the variable $\bx_{L-1}$. The factor $\bp_{L-1}$ can therefore be explicitly stored. The construction is then repeated by multiplying pairs of factors from left to right, obtaining a sequence of tensors $\bp_{L-2},\dots,\bp_{l}$ until the desired derivative is obtained. Note that, in doing so, no large tensor is ever stored in memory. This process is known as \emph{backpropagation}.

In general, tensor $\bp_{l}$ is obtained from $\bp_{l+1}$ as the product:
\[
(\vv \bp_{l})^\top = (\vv \bp_{l+1})^\top \times \frac{d\vv f_{l+1}(\bx_{l};\bw_{l+1})}{d(\vv\bx_{l})^\top}.
\]
The key to implement backpropagation is to be able to compute these products without explicitly computing and storing in memory the second factor, which is a large Jacobian matrix. Since computing the derivative is a linear operation, this product can be interpreted as the \emph{derivative of the layer projected along direction $\bp_{l+1}$}:
\begin{equation}\label{e:projected}
\bp_{l} =
\frac{d \langle \bp_{l+1}, f(\bx_l;\bw_l) \rangle}
{d \bx_{l}}.
\end{equation}
Here $\langle \cdot,\cdot \rangle$ denotes the inner product between tensors, which results in a scalar quantity. Hence the derivative \eqref{e:projected} needs not to use the $\vv$ notation, and yields a tensor $\bp_l$ that has the same size as $\bx_l$ as expected.

In order to implement backpropagation, a CNN toolbox provides implementations of each layer $f$ that provide:
\begin{itemize}
\item A \textbf{forward mode}, computing the output $\by = f(\bx;\bw)$ of the layer given its input $\bx$ and parameters $\bw$.
\item A \textbf{backward mode}, computing the projected derivatives
\[
\frac{d \langle \bp, f(\bx;\bw) \rangle}
{d \bx}
\quad\text{and}\quad
\frac{d \langle \bp, f(\bx;\bw) \rangle}
{d \bw},
\]
given, in addition to the input $\bx$ and parameters $\bw$, a tensor $\bp$ that the same size as $\by$.
\end{itemize}
This is best illustrated with an example. Consider a layer $f$ such as the convolution operator implemented by the \matconvnet\ !vl_nnconv! command. In the ``forward'' mode, one calls the function as !y = vl_nnconv(x,w,[])! to apply the filters !w! to the input !x! and obtain the output !y!. In the ``backward mode'', one calls ![dx, dw] = vl_nnconv(x,w,[],p)!.  As explained above, !dx!, !dw!, and !p! have the same size as !x!, !w!, and !y!, respectively. The computation of large Jacobian is encapsulated in the function call and never carried out explicitly.

\subsection{Backpropagation networks}\label{s:bpnets}

In this section, we provide a schematic interpretation of backpropagation and show how it can be implemented by ``reversing'' the NN computational graph.

The projected derivative of eq.~\eqref{e:projected} can be seen as the derivative of the following mini-network:
\begin{center}
\begin{tikzpicture}[auto, node distance=2cm]
\node (x) [data] {$\bx$};
\node (f) [block,right of=x ] {$f$};
\node (dot)[block,right of=f ] {$\langle \cdot, \cdot \rangle$};
\node (z) [data, right of=dot] {$z \in \mathbb{R}$};
\node (w) [data, below of=f ] {$\bw$};
\node (p) [data, below of=dot] {$\bp$};
\draw [to] (x.east) -- (f.west) {};
\draw [to] (f.east) -- node {$\by$}  (dot.west) {};
\draw [to] (w.north) -- (f.south) {};
\draw [to] (dot.east) -- (z.west) {};
\draw [to] (p.north) -- (dot.south) {};
\end{tikzpicture}
\end{center}
In the context of back-propagation, it can be useful to think of the projection $\bp$ as the ``linearization'' of the rest of the network from variable $\by$ down to the loss. The projected derivative can also be though of as a new layer $(d\bx, d\bw) = df(\bx,\bw,\bp)$ that, by computing the derivative of the mini-network, operates in the reverse direction:
\begin{center}
\begin{tikzpicture}[auto, node distance=2cm]
\node (df) [block,right of=x] {$df$};
\node (dx) [data,left of=df] {$d\bx$};
\node (dw) [data,below of=df] {$d\bw$};
\node (w) [data,above of=df,xshift=0.6em] {$\bw$};
\node (x) [data,above of=df,xshift=-0.6em] {$\bx$};
\node (p) [data,right of=df] {$\bp$};
\draw [to] (df.west) -- (dx.east)  {};
\draw [to] (df.south) -- (dw.north)  {};
\draw [to] (p.west) -- (f.east) {};
\draw [to] (w.south) -- ([xshift=0.6em]df.north) {};
\draw [to] (x.south) -- ([xshift=-0.6em]df.north) {};
\end{tikzpicture}
\end{center}
By construction (see eq.~\eqref{e:projected}), the function $df$ is \emph{linear} in the argument $\bp$.

Using this notation, the forward and backward passes through the original network can be rewritten as evaluating an extended network which contains a BP-reverse of the original one (in blue in the diagram):
\begin{center}
\begin{tikzpicture}[auto, node distance=2cm]
\node (x0) [data] {$\bx_0$};
%
\node (f1) [block,right of=x0] {$f_1$};
\node (x1) [data,right of=f1] {$\bx_{1}$};
\node (w1) [data,below of=f1] {$\bw_1$};
%
\node (f2) [block,right of=x1] {$f_2$};
\node (x2) [data,right of=f2] {$\bx_{2}$};
\node (w2) [data,below of=f2] {$\bw_2$};
%
\node (f3) [right of=x2] {$\dots$};
\node (xLm) [right of=f3] {$\bx_{L-1}$};
%
\node (fL) [block,right of=xLm] {$f_L$};
\node (xL) [data,right of=fL] {$\bx_{L}$};
\node (wL) [data,below of=fL] {$\bw_L$};
%
\draw [to] (x0.east) -- (f1.west) {};
%
\draw [to] (w1.north) -- (f1.south) {};
\draw [to] (f1.east) -- (x1.west) {};
\draw [to] (x1.east) -- (f2.west) {};
%
\draw [to] (w2.north) -- (f2.south) {};
\draw [to] (f2.east) -- (x2.west) {};
\draw [to] (x2.east) -- (f3.west) {};
%
\draw [to] (f3.east) -- (xLm.west) {};
\draw [to] (xLm.east) -- (fL.west) {};
%
\draw [to] (wL.north) -- (fL.south) {};
\draw [to] (fL.east) -- (xL.west) {};
%
\node (dfL) [block,below of=wL,bp] {$df_L$};
\node (dxL) [data,right of=dfL,bpe] {$d\bp_L$};
\node (dwL) [data,below of=dfL,bpe] {$d\bw_L$};
\node (dxLm) [data,left of=dfL,bpe] {$d\bx_{L-1}$};
%
\node (df3) [left of=dxLm,bpe] {$\dots$};
%
\node (df2) [block,below of=w2,bp] {$df_2$};
\node (dx2) [data,right of=df2,bpe] {$d\bx_{2}$};
\node (dw2) [data,below of=df2,bpe] {$d\bw_2$};
%
\node (df1) [block,below of=w1,bp] {$df_1$};
\node (dx1) [data,right of=df1,bpe] {$d\bx_{1}$};
\node (dw1) [data,below of=df1,bpe] {$d\bw_1$};
%
\node (dx0) [data,left of=df1,bpe] {$d\bx_{0}$};
%
\draw [to,bp] (wL.south) -- (dfL.north) {};
\draw [to,bp] (dfL.south) -- (dwL.north) {};
\draw [to,bp] (dxL.west) -- (dfL.east) {};
\draw [to,bp] (dfL.west) -- (dxLm.east) {};
%
\draw [to,bp] (dxLm.west) -- (df3.east) {};
\draw [to,bp] (df3.west) -- (dx2.east) {};
%
\draw [to,bp] (w2.south) -- (df2.north) {};
\draw [to,bp] (df2.south) -- (dw2.north) {};
\draw [to,bp] (dx2.west) -- (df2.east) {};
\draw [to,bp] (df2.west) -- (dx1.east) {};
%
\draw [to,bp] (w1.south) -- (df1.north) {};
\draw [to,bp] (df1.south) -- (dw1.north) {};
\draw [to,bp] (dx1.west) -- (df1.east) {};
%
\draw [to,bp] (df1.west) -- (dx0.east) {};
%
\draw [to,bp] (x0) -- (df1) {} ;
\draw [to,bp] (x1) -- (df2) {} ;
\draw [to,bp] (xLm) -- (dfL) {} ;
\end{tikzpicture}
\end{center}

% ------------------------------------------------------------------
\subsection{Backpropagation in DAGs}\label{s:dag}
% ------------------------------------------------------------------

Assume that the DAG has a single output variable $\bx_L$ and assume, without loss of generality, that all variables are sorted in order of computation $(\bx_0,\bx_1,\dots,\bx_{L-1},\bx_L)$ according to the DAG structure. Furthermore, in order to simplify the notation, assume that this list contains both data and parameter variables, as the distinction is moot for the discussion in this section.

We can cut the DAG at any point in the sequence by fixing $\bx_0, \dots, \bx_{l-1}$ to some arbitrary value and dropping all the DAG layers that feed into them, effectively transforming the first $l$ variables into inputs. Then, the rest of the DAG defines a function $h_l$ that maps these input variables to the output $\bx_L$:
\[
 \bx_L = h_l(\bx_0,\bx_1,\dots,\bx_{l-1}).
\]
Next, we show that backpropagation in a DAG iteratively computes the projected derivatives of all functions $h_1,\dots,h_L$ with respect to all their parameters.

Backpropagation starts by initializing variables $(d\bx_{0},\dots,d\bx_{l-1})$ to null tensors of the same size as $(\bx_0,\dots,\bx_{l-1})$. Next, it computes the projected derivatives of
\[
 \bx_L = h_L(\bx_0,\bx_1,\dots,\bx_{L-1}) =
 f_{\pi_L}(\bx_0,\bx_1,\dots,\bx_{L-1}).
\]
Here $\pi_l$ denotes the index of the layer $f_{\pi_l}$ that computes the value of the variable $\bx_l$. There is at most one such layer, or none if $\bx_l$ is an input or parameter of the original NN. In the first case, the layer may depend on any of the variables prior to $\bx_l$ in the sequence, so that general one has:
\[
 \bx_{l} = f_{\pi_l}(\bx_0,\dots,\bx_{l-1}).
\]
	At the beginning of backpropagation, since there are no intermediate variables between $\bx_{L-1}$ and $\bx_L$, the function $h_L$ is the same as the last layer $f_{\pi_L}$. Thus the projected derivatives of $h_L$ are the same as the projected derivatives of $f_{\pi_L}$, resulting in the equation
\[
\forall t=0,\dots,L-1:\qquad
d\bx_{t} \leftarrow d\bx_{t}
+ \frac{d\langle \bp_L, f_{\pi_L}(\bx_0,\dots,\bx_{t-1})\rangle}{d\bx_t}.
\]
Here, for uniformity with the other iterations, we use the fact that $d\bx_l$ are initialized to zero an\emph{accumulate} the values instead of storing them. In practice, the update operation needs to be carried out only for the variables $\bx_l$ that are actual inputs to $f_{\pi_L}$, which is often a tiny fraction of all the variables in the DAG.

After the update, each $d\bx_t$ contains the projected derivative of function $h_L$ with respect to the corresponding variable:
\[
\forall t=0,\dots,L-1:\qquad
d\bx_t = \frac{d\langle \bp_L, h_L(\bx_0,\dots,\bx_{l-1})\rangle}{d\bx_t}.
\]
Given this information, the next iteration of backpropagation updates the variables to contain the projected derivatives of $h_{L-1}$ instead. In general, given the derivatives of $h_{l+1}$, backpropagation computes the derivatives of $h_{l}$ by using the relation
\[
 \bx_L
 =
 h_{l}(\bx_0,\bx_1,\dots,\bx_{l-1})
 =
 h_{l+1}(\bx_0,\bx_1,\dots,\bx_{l-1},f_{\pi_L}(\bx_0,\dots,\bx_{l-1}))
\]
Applying the chain rule to this expression, for all $0\leq t \leq l-1$:
\[
\frac{d\langle \bp, h_l \rangle}{d(\vv \bx_t)^\top}
=
\frac{d\langle \bp, h_{l+1}\rangle}{d(\vv \bx_t)^\top}
+
\underbrace{\frac{d\langle \bp_L, h_{l+1}\rangle}{d(\vv \bx_l)^\top}}_{\vv d\bx_l}
\frac{d \vv f_{\pi_l}}{d(\vv \bx_t)^\top}.
\]
This yields the update equation
\begin{equation}\label{e:bp-update}
\forall t=0,\dots,l-1:\qquad
d\bx_t \leftarrow d\bx_t + \frac{d\langle \bp_l, f_{\pi_l}(\bx_0,\dots,\bx_{l-1})\rangle}{d\bx_t},
\quad
\text{where\ }
\bp_l = d\bx_l.
\end{equation}
Once more, the update needs to be explicitly carried out only for the variables $\bx_t$ that are actual inputs of $f_{\pi_l}$. In particular, if $\bx_l$ is a data input or a parameter of the original neural network, then $\bx_l$ does not depend on any other variables or parameters and $f_{\pi_l}$ is a nullary function (i.e.\ a function with no arguments). In this case, the update does not do anything.
After iteration $L-l+1$ completes, backpropagation remains with:
\begin{align*}
\forall t=0,\dots,l-1:&\qquad
d\bx_t
=
\frac{d\langle \bp_L, h_l(\bx_0,\dots,\bx_{l-1})\rangle}{d\bx_t}.
\end{align*}
Note that the derivatives for variables $\bx_t, l \leq t \leq L-1$ are not updated since $h_l$ does not depend on any of those. Thus, after all $L$ iterations are complete, backpropagation terminates with
\[
\forall l=1,\dots,L:\qquad
d\bx_{l-1}
=
\frac{d\langle \bp_L, h_{l}(\bx_0,\dots,\bx_{l-1})\rangle}{d\bx_{l-1}}.
\]
As seen above, functions $h_{l}$ are obtained from the original network $f$ by transforming variables $\bx_0,\dots,\bx_{l-1}$ into to inputs. If $\bx_{l-1}$ was already an input (data or parameter) of $f$, then the derivative $d\bx_{l-1}$ is applicable to $f$ as well.

Backpropagation can be summarized as follows:
\begin{center}
\fbox{\begin{minipage}{0.95\textwidth}
Given: a DAG neural network $f$ with a single output $\bx_L$, the values of all input variables (including the parameters), and the value of the projection $\bp_L$ (usually $\bx_L$ is a scalar and $\bp_L = p_L = 1$):
\begin{enumerate}
    \item Sort all variables by computation order $(\bx_0,\bx_1,\dots,\bx_L)$ according to the DAG.
    \item Perform a forward pass through the network to compute all the intermediate variable values.
    \item Initialize $(d\bx_0, \dots, d\bx_{L-1})$ to null tensors with the same size as the corresponding variables.
    \item For $l=L,L-1,\dots,2,1$:
  \begin{enumerate}
  \item Find the index $\pi_l$ of the layer $\bx_{l} = f_{\pi_l}(\bx_0,\dots,\bx_{l-1})$ that evaluates variable $\bx_l$. If there is no such layer (because $\bx_{l}$ is an input or parameter of the network), go to the next iteration.
  \item Update the variables using the formula:
   \[
   \forall t=0,\dots,l-1:\qquad
d\bx_t \leftarrow d\bx_t + \frac{d\langle d\bx_l, f_{\pi_l}(\bx_0,\dots,\bx_{l-1})\rangle}{d\bx_t}.
   \]
   To do so efficiently, use the ``backward mode'' of the layer $f_{\pi_l}$ to compute its derivative projected onto $d\bx_l$ as needed.
  \end{enumerate}
  \end{enumerate}
\end{minipage}}
\end{center}

% TODO: what to do with multiple outputs

\begin{figure}[t]
\begin{center}
\begin{tikzpicture}[auto, node distance=0.3cm]
 \matrix (m) [matrix of math nodes,
    column sep=1.2cm,
    row sep=0.3cm]
{
& \node (f1) [block]{f_1};
& \node (x1) [datac]{\bx_1};
\\
\node (x0) [datac]{\bx_0};
&
&
& \node (f3) [block]{f_3};
& \node (x3) [datac]{\bx_3};
\\
& \node (f2) [block]{f_2};
& \node (x2) [datac]{\bx_2};
& &
& \node (f5) [block]{f_5};
& \node (x7) [datac]{\bx_7};
\\
&
& \node(x5) [datac]{\bx_5};
\\
\node (x4) [datac]{\bx_4};
& \node (f4) [block]{f_4};
\\
&
& \node(x6) [datac]{\bx_6};
\\
% BP
& \node (df1) [block,bp]{df_1};
& \node (dx1) [datac,bp]{d\bx_1};
\\
\node (dx0) [datac,bp]{d\bx_0};
&
&
& \node (df3) [block,bp]{df_3};
& \node (dx3) [datac,bp]{d\bx_3};
\\
& \node (df2) [block,bp]{df_2};
& \node (dx2) [datac,bp]{d\bx_2};
& &
& \node (df5) [block,bp]{df_5};
& \node (dx7) [datac,bp]{\bp_7};
\\
&
& \node (dx5) [datac,bp]{d\bx_5};
\\
\node (dx4) [datac,bp]{d\bx_4};
& \node (df4) [block,bp]{df_4};
\\
&
& \node(dx6) [datac,bp]{\bp_6};
\\
};
\draw[to] (x0) -- (f1);
\draw[to] (f1) -- (x1);
\draw[to] (x1) -- (f3);
\draw[to] (x0) -- (f2);
\draw[to] (f2) -- (x2);
\draw[to] (x2) -- (f3);
\draw[to] (f3) -- (x3);
\draw[to] (x3) -- (f5);
\draw[to] (f5) -- (x7);
\draw[to] (x4) -- (f4);
\draw[to] (f4) -- (x5);
\draw[to] (f4) -- (x6);
\draw[to] (x5) -- (f5);
\node(w1) [par,below=of f1]{$\bw_1$}; \draw[to] (w1) -- (f1);
\node(w2) [par,below=of f2]{$\bw_2$}; \draw[to] (w2) -- (f2);
\node(w4) [par,below=of f4]{$\bw_4$}; \draw[to] (w4) -- (f4);
\draw[to] (w4) to [bend right] (f3);
\node(w5) [par,below=of f5]{$\bw_5$}; \draw[to] (w5) -- (f5);
\node (dx0s) [right of=dx0,xshift=20pt,draw,rectangle,bp]{$\Sigma$};
\draw[from,bp] (dx0) -- (dx0s);
\draw[from,bp] (dx0s) -- (df1);
\draw[from,bp] (df1) -- (dx1);
\draw[from,bp] (dx1) -- (df3);
\draw[from,bp] (dx0s) -- (df2);
\draw[from,bp] (df2) -- (dx2);
\draw[from,bp] (dx2) -- (df3);
\draw[from,bp] (df3) -- (dx3);
\draw[from,bp] (dx3) -- (df5);
\draw[from,bp] (df5) -- (dx7);
\draw[from,bp] (dx4) -- (df4);
\draw[from,bp] (df4) -- (dx5);
\draw[from,bp] (df4) -- (dx6);
\draw[from,bp] (dx5) -- (df5);
\node(dw1) [par,below=of df1,bp]{$d\bw_1$}; \draw[from,bp] (dw1) -- (df1);
\node(dw2) [par,below=of df2,bp]{$d\bw_2$}; \draw[from,bp] (dw2) -- (df2);
\node(dw4s) [below of=df4,draw,rectangle,bp,yshift=-25pt]{$\Sigma$}; \draw[from,bp] (dw4s) -- (df4);
\node(dw4) [par,below=of dw4s,bp]{$d\bw_4$}; \draw[from,bp] (dw4) -- (dw4s);
\draw[from,bp] (dw4s) to [bend right,bp] (df3);
\node(dw5) [par,below=of df5,bp]{$d\bw_5$}; \draw[from,bp] (dw5) -- (df5);
%
\draw[to,bpl] (x0) -| ([xshift=-0.3cm]x0.west) |- (df1);
\draw[to,bpl] (x0) -| ([xshift=-0.6cm]x0.west) |- (df2);
\draw[to,bpl] (x1) -| ([xshift=4cm]x1.west) |- ([yshift=10pt]df3.east);
\draw[to,bpl] (x2) -| (df3);
\draw[to,bpl] (x3) -| ([xshift=+5cm]x3.east) |- ([yshift=15pt]df5.east);
\draw[to,bpl] (x4) to [bend right=75] ([yshift=15pt]df4.west);
\draw[to,bpl] (x5) to [bend left] (df5);
\end{tikzpicture}
\end{center}
\vspace{-1em}
\caption{\textbf{Backpropagation network for a DAG.}}\label{f:dagbp}
\end{figure}

% ------------------------------------------------------------------
\subsection{DAG backpropagation networks}\label{s:bpnets-dag}
% ------------------------------------------------------------------

Just like for sequences, backpropagation in DAGs can be implemented as a corresponding BP-reversed DAG. To construct the reversed DAG:
\begin{enumerate}
\item For each layer $f_l$, and variable/parameter $\bx_t$ and $\bw_l$, create a corresponding layer $df_l$ and variable/parameter $d\bx_t$ and $d\bw_l$.
\item If a variable $\bx_t$ (or parameter $\bw_l$) is an input of $f_l$, then it is an input of $df_l$ as well.
\item If a variable $\bx_t$ (or parameter $\bw_l$) is an input of $f_l$, then the variable $d\bx_t$ (or the parameter $d\bw_l$) is an output $df_l$.
\item In the previous step, if a variable $\bx_t$ (or parameter $\bw_l$) is input to two or more layers in $f$, then $d\bx_t$ would be the output of two or more layers in the reversed network, which creates a conflict. Resolve these conflicts by inserting a summation layer that adds these contributions (this corresponds to the summation in the BP update equation \eqref{e:bp-update}).
\end{enumerate}
The BP network corresponding to the DAG of Fig.~\ref{f:dag} is given in Fig.~\ref{f:dagbp}.


% Created 2021-11-26 Fri 08:58
% Intended LaTeX compiler: pdflatex
\documentclass[presentation,aspectratio=1610]{beamer}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{grffile}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{textcomp}
\usepackage{amssymb}
\usepackage{capt-of}
\usepackage{hyperref}
\usepackage{khpreamble, euscript}
\DeclareMathOperator{\atantwo}{atan2}
\newcommand*{\ctrb}{\EuScript{C}}
\newcommand*{\obsv}{\EuScript{O}}
\usetheme{default}
\author{Kjartan Halvorsen}
\date{\today}
\title{Discrete-time Ouput feedback (state feedback with observer)}
\hypersetup{
 pdfauthor={Kjartan Halvorsen},
 pdftitle={Discrete-time Ouput feedback (state feedback with observer)},
 pdfkeywords={},
 pdfsubject={},
 pdfcreator={Emacs 26.3 (Org mode 9.4.6)},
 pdflang={English}}
\begin{document}

\maketitle

\section{Discret-time state space model}
\label{sec:orgdf25772}
\begin{frame}[label={sec:orge5f3d67}]{The discrete-time state-space model}
\end{frame}

\begin{frame}[label={sec:orgc315c14}]{The discrete-time state-space model}
\begin{center}
  \begin{tikzpicture}[node distance=42mm, block/.style={inner sep=6pt, rectangle, draw, minimum width=15mm}, sumnode/.style={circle, draw, inner sep=2pt}]
    \node[coordinate] (input) {};
    \node[block, right of=input, align=center] (plant)  {$x(k+1) = \Phi x(k) + \Gamma u(k)$\\$y(k) = C x(k)$};
    \node[coordinate, right of=plant] (output) {};

    \draw[->] (input) -- node[above, pos=0.3] {$u(k)$} (plant);
    \draw[->] (plant) -- node[above, near end] {$y(k)$} (output);
  \end{tikzpicture}
\end{center}
\end{frame}

\begin{frame}[label={sec:org871eb8d}]{Stability}
\end{frame}
\begin{frame}[label={sec:org5143890}]{Eigenvalues and eigenvectors}
\alert{Definition} The eigenvalues \(\lambda_i  \in \mathbb{R}\) and eigenvectors \(v_i \in \mathbb{R}^n\) of a matrix \(\Phi \in \mathbb{R}^{n\times{}n}\) are the \(n\) pairs \((\lambda_i, v_i \neq 0 ), \; i=1,2,\ldots,n\) that satisfy
\[ \Phi v_i = \lambda_i v_i \]
\end{frame}

\begin{frame}[label={sec:orga4ef6e4}]{Stability}
The system
\begin{equation*}
x(k+1)=\Phi x(k), \ \ x(0)=x_0
\end{equation*}
is \alert{stable} if  \(\underset{t\to\infty}{\lim}x(kh)=0, \quad \forall\;  x_0\in\Bbb{R}^n\).

A necessary and sufficient requirement for stability is that \alert{all the eigenvalues of \(\Phi\) are inside the unit circle.}

The \alert{eigenvalues} of \(\Phi\) are the  \alert{poles} of the system.
\end{frame}

\section{State feedback}
\label{sec:org6c54675}
\begin{frame}[label={sec:orgaa62416}]{State feedback control}
\end{frame}
\begin{frame}[label={sec:org6a0dafe}]{State feedback control}
Given
 \begin{equation}
 \begin{split}
  x(k+1) &= \Phi x(k) + \Gamma u(k)\\
  y(k) &= C x(k)
 \end{split}
 \label{eq:ssmodel}
\end{equation}
and measurements (or an estimate) of the state vector \(x(k)\).

\alert{Linear state feedback} is the control law
\begin{equation*}
\begin{split}
 u(k) &= f\big((x(k), u_c(k)\big) = -\textcolor{morange}{l_1}x_1(k) - \textcolor{morange}{l_2}x_2(k) - \cdots - \textcolor{morange}{l_n} x_n(k) + \textcolor{mbluegreen}{l_0}u_c(k)\\
      &= -\textcolor{morange}{L}x(k) + \textcolor{mbluegreen}{l_0}u_c(k),
\end{split}
\end{equation*}
where \[ \textcolor{morange}{L} = \bbm \textcolor{morange}{l_1} & \textcolor{morange}{l_2} & \cdots & \textcolor{morange}{l_n} \ebm. \]
Substituting this in the state-space model \eqref{eq:ssmodel} gives
 \begin{equation}
 \begin{split}
  x(k+1) &= \left(\Phi -\Gamma \textcolor{morange}{L} \right) x(k) + \textcolor{mbluegreen}{l_0}\Gamma u_c(k)\\
  y(k) &= C x(k)
 \end{split}
 \label{eq:closedloop}
\end{equation}
\end{frame}

\begin{frame}[label={sec:org4bdb9dc}]{Pole placement by state feedback}
Given (or choosing) a desired placement of the closed-loop poles \(p_1, p_2, \ldots, p_n\), being roots of the desired characteristic polynomial
\begin{equation}
a_c(z) = (z-p_1)(z-p_2)\cdots(z-p_n) = z^n + \alpha_1 z^{n-1} + \cdots \alpha_n.
\label{eq:desiredpoles}
\end{equation}

\pause

Linear state feedback gives the system
 \begin{equation}
 \begin{split}
  x(k+1) &= \left(\Phi -\Gamma \textcolor{morange}{L} \right) x(k) + \textcolor{mbluegreen}{l_0}\Gamma u_c(k)
 \end{split}
 \label{eq:closedloop}
\end{equation}
with characteristic polynomial
\begin{equation}
\det\left(zI - (\Phi - \Gamma \textcolor{morange}{L})\right) = z^n + \beta_1(\textcolor{morange}{l_1},\ldots,\textcolor{morange}{l_n}) z^{n-1} + \cdots \beta_n(\textcolor{morange}{l_1}, \ldots, \textcolor{morange}{l_n}).
\label{eq:poles}
\end{equation}

\pause

Set the coefficients of the desired characteristic polynomial \eqref{eq:desiredpoles} equal to the coefficients of \eqref{eq:poles} to obtain the system of equations
\begin{equation*}
\begin{split}
\beta_1(\textcolor{morange}{l_1}, \ldots, \textcolor{morange}{l_n}) &= \alpha_1\\
\beta_2(\textcolor{morange}{l_1}, \ldots, \textcolor{morange}{l_n}) &= \alpha_2\\
&\vdots\\
\beta_n(\textcolor{morange}{l_1}, \ldots, \textcolor{morange}{l_n}) &= \alpha_n
\end{split}
\label{eq:coeffs}
\end{equation*}
\end{frame}

\begin{frame}[label={sec:org9823dd3}]{Pole placement by state feedback}
The system of equations
\begin{equation*}
\begin{split}
\beta_1(\textcolor{morange}{l_1}, \ldots, \textcolor{morange}{l_n}) &= \alpha_1\\
\beta_2(\textcolor{morange}{l_1}, \ldots, \textcolor{morange}{l_n}) &= \alpha_2\\
&\vdots\\
\beta_n(\textcolor{morange}{l_1}, \ldots, \textcolor{morange}{l_n}) &= \alpha_n
\end{split}
\label{eq:coeffs}
\end{equation*}

is always linear in the parameters of the controller, hence
\begin{equation*}
M \textcolor{morange}{L}\transp = \alpha,
\end{equation*}
where \(\alpha\transp = \bbm \alpha_1 & \alpha_2 & \cdots & \alpha_n \ebm.\)
\end{frame}

\begin{frame}[label={sec:orgdec3dc3},fragile]{Pole placement by state feedback}
 Given a desired placement of the closed-loop poles \(p_1, p_2, \ldots, p_n\), being roots of the desired characteristic polynomial
\begin{equation*}
a_c(z) = (z-p_1)(z-p_2)\cdots(z-p_n) = z^n + \alpha_1 z^{n-1} + \cdots \alpha_n.
\label{eq:desiredpoles}
\end{equation*}
and closed-loop system
 \begin{equation*}
 \begin{split}
  x(k+1) &= \left(\Phi -\Gamma \textcolor{morange}{L} \right) x(k) + \textcolor{mbluegreen}{l_0}\Gamma u_c(k)\\
  y(k) &= C x(k)
 \end{split}
 \label{eq:closedloop}
\end{equation*}

The Matlab (\emph{control systems toolbox}) has methods for computing the gain vector \(L\)

\begin{enumerate}
\item \alert{Ackerman's method}
\begin{verbatim}
L = acker(Phi, Gamma, pd)
\end{verbatim}
\item \alert{Numerically more stable method}
\begin{verbatim}
L = place(Phi, Gamma, pd)
\end{verbatim}
\end{enumerate}
\end{frame}

\begin{frame}[label={sec:orga3cd068}]{The reference input gain \(l_0\)}
The closed-loop state space system
\begin{equation*}
\begin{split}
 x(k+1) &= \underbrace{\left(\Phi -\Gamma \textcolor{morange}{L} \right)}_{\Phi_c} x(k) + \textcolor{mbluegreen}{l_0}\Gamma u_c(k)\\
 y(k) &= C x(k)
\end{split}
\end{equation*}
with constant reference signal \(u_c(k) = u_{c,f}\) has the steady-state solution (\(x(k+1)=x(k)\))
\pause
\[ x_f =  \textcolor{mbluegreen}{l_0} (I - \Phi_c)^{-1}\Gamma u_{c,f}\]
\[ y_f = Cx_f = \textcolor{mbluegreen}{l_0} C(I - \Phi_c)^{-1}\Gamma u_{c,f}.\]
We want \(y_f =  u_{c,f}\),
\[ \Rightarrow \qquad \textcolor{mbluegreen}{l_0} = \frac{1}{C(I-\Phi_c)^{-1}\Gamma}\]
\end{frame}

\section{State feedback with observer}
\label{sec:org2e8ae4c}
\begin{frame}[label={sec:org7e1e842}]{State feedback with reconstructed states}
\end{frame}

\begin{frame}[label={sec:org2a1058e}]{Observer design}
Given model
 \begin{equation*}
 \begin{split}
  x(k+1) &= \Phi x(k) + \Gamma u(k)\\
  y(k) &= C x(k)
 \end{split}
 \label{eq:ssmodel}
\end{equation*}
and measurements of the output signal \(y(k)\).

The observer is given by
\begin{equation*}
\begin{split}
\hat{x}(k+1) &= \underbrace{\Phi \hat{x}(k) + \Gamma u(k)}_{\text{simulation}} + \underbrace{\textcolor{mred}{K}\big(y(k) - C\hat{x}(k)\big)}_{\text{correction}} = \left(\Phi - \textcolor{mred}{K}C\right)\hat{x}(k) +  \Gamma u(k) + \textcolor{mred}{K}y(k)
\end{split}
\end{equation*}
with poles given by the eigenvalues of the matrix \(\Phi_o = \Phi - \textcolor{mred}{K}C\)

\alert{Rule-of-thumb} (From continous-time state-space theory) Choose the poles of the observer (eigenvalues of \(\Phi-\textcolor{mred}{K}C\)) at least twice as fast as the poles (eigenvalues) of \(\Phi-\Gamma L\). In discrete-time place the observer-poles closer to the origin.
\end{frame}

\begin{frame}[label={sec:orgd560d3e}]{Control by feedback from reconstructed states}
The design problem can be separates into two problems
\begin{enumerate}
\item Determine the gain vector \(\textcolor{orange!80!black}{L}\) and the gain \(l_0\) of the control law
\[ u(k) = -\textcolor{orange!80!black}{L} \hat{x}(k) + l_0 u_c(k)\]
so that the closed-loop system has good reference tracking.
\item Determine the gain vector \(\textcolor{mred}{K}\) of the observer
\begin{equation*}
\begin{split}
\hat{x}(k+1) &= \Phi \hat{x}(k) + \Gamma u(k) + \textcolor{mred}{K} \big(y(k) - C\hat{x}(k)\big)
\end{split}
\end{equation*}
to get a good balance between disturbance rejection and noise attenuation.
\end{enumerate}
\end{frame}

\begin{frame}[label={sec:orgfae4068}]{Computing the observer gain}
A matrix \(M\) and its transpose \(M\transp\) have the same eigenvalues. Hence, the problem of determining the gain \(K\) to obtain desired eigenvalues of
\[\Phi- \textcolor{mred}{K}C\] is equivalent to determining the gain \(K\) in
\[(\Phi-KC)\transp = \Phi\transp - C\transp \textcolor{mred}{K}\transp.\]
The last problem has the exact same form as the problem of determining \(L\) to obtain desired eigenvalues of
\[\Phi - \Gamma L\]

So, the same matlab function can be used for both problems.
\end{frame}

\begin{frame}[label={sec:org988a07f},fragile]{Computing the observer gain}
 \begin{enumerate}
\item \alert{Ackerman's method}
\begin{verbatim}
K = acker(Phi', C', po)'
\end{verbatim}
\item \alert{More numerically stable method}
\begin{verbatim}
K = place(Phi', C', pd)'
\end{verbatim}
\end{enumerate}
\end{frame}

\section{Where to choose the poles}
\label{sec:org9fdaa4e}
\begin{frame}[label={sec:org5019d3d}]{Where to place the closed-loop poles?}
\end{frame}
\begin{frame}[label={sec:org4d3661a}]{Where to place the closed-loop poles?}
If the system is controllable we can place the closed-loop poles freely.

\pause

But not all placements are good choices

\pause

Take into account
\begin{itemize}
\item Desired speed
\item Desired damping
\item \alert{Poles and zeros of the plant}
\end{itemize}
\end{frame}

\begin{frame}[label={sec:orgcc4a069}]{Placing for desired speed and damping}
\begin{columns}
\begin{column}{0.4\columnwidth}
\alert{s-plane}

\begin{center}
 \includegraphics[height=.6\textheight]{../../figures/sgrid-crop}
\end{center}
\end{column}
\begin{column}{0.6\columnwidth}
\alert{z-plane}
\begin{center}
 \includegraphics[height=.59\textheight]{../../figures/zgrid-crop}
\end{center}
\end{column}
\end{columns}
\end{frame}

\begin{frame}[label={sec:orgdcea2e4}]{The effect of plant zeros and poles}
\begin{columns}
\begin{column}{0.4\columnwidth}
\begin{center}
 \includegraphics[width=1.0\linewidth]{../../figures/AM-portal.png}
\end{center}

\pause
\end{column}

\begin{column}{0.6\columnwidth}
\begin{center}
 \includegraphics[width=1.0\linewidth]{../../figures/AM-ch12.4.png}
\end{center}
\end{column}
\end{columns}
\end{frame}

\begin{frame}[label={sec:org7612825}]{Placing w.r.t plant zeros and poles}
\begin{itemize}
\item Cancel slow plant zeros
\item Place fast closed-loop poles near fast plant poles
\end{itemize}
\end{frame}
\end{document}


\documentclass[12pt]{cdblatex}
\usepackage{exercises}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% --------------------------------------------------------------------------------------------
\section*{Exercise 6.1 Evaluate -- without {\tt rhsonly = True}}

\begin{cadabra}
   {\theta, \varphi}::Coordinate.
   {a,b,c,d,e,f,g,h#}::Indices(values={\theta, \varphi}, position=independent).

   \partial{#}::PartialDerivative.

   V  := { V_{\theta} = \varphi, V_{\varphi} = \sin(\theta) }.   # cdb(ex-0601.100,V)
   dV := dV_{a b} -> \partial_{b}{V_{a}} - \partial_{a}{V_{b}}.  # cdb(ex-0601.101,dV)

   evaluate (dV, V)                                              # cdb(ex-0601.102,dV)
\end{cadabra}

Notice how {\tt evaluate} has been applied to both the left and right hand sides of the rule.

\begin{align*}
   V_{a} &= \Cdb{ex-0601.100}\\[10pt]
   & \Cdb{ex-0601.101}\\[10pt]
   & \Cdb{ex-0601.102}
\end{align*}

\clearpage

% --------------------------------------------------------------------------------------------
\section*{Exercise 6.1 Evaluate -- with {\tt rhsonly = True}}

\begin{cadabra}
   {\theta, \varphi}::Coordinate.
   {a,b,c,d,e,f,g,h#}::Indices(values={\theta, \varphi}, position=independent).

   \partial{#}::PartialDerivative.

   V  := { V_{\theta} = \varphi, V_{\varphi} = \sin(\theta) }.   # cdb(ex-0601.200,V)
   dV := dV_{a b} -> \partial_{b}{V_{a}} - \partial_{a}{V_{b}}.  # cdb(ex-0601.201,dV)

   evaluate (dV, V, rhsonly=True)                                # cdb(ex-0601.202,dV)
\end{cadabra}

This is an improvement, only the right hnd side has been expanded into components.

\begin{align*}
   V_{a} &= \Cdb{ex-0601.200}\\[10pt]
   & \Cdb{ex-0601.201}\\[10pt]
   & \Cdb{ex-0601.202}
\end{align*}

\end{document}


\documentclass[]{article}
\usepackage{caption,subcaption,graphicx,float,url,amsmath,amssymb,tocloft,wasysym,amsthm,thmtools,textcomp,listings,amsfonts,cancel}
\usepackage[hidelinks]{hyperref}
\usepackage[toc,acronym,nonumberlist]{glossaries}
\usepackage[]{algorithm2e}
\setacronymstyle{long-short}
\usepackage{glossaries-extra}
\graphicspath{{figs/}}
\setlength{\cftsubsecindent}{0em}
\setlength{\cftsecnumwidth}{3em}
\setlength{\cftsubsecnumwidth}{3em}
\newcommand\numberthis{\addtocounter{equation}{1}\tag{\theequation}}
\newtheorem{thm}{Theorem}
\newtheorem{cor}[thm]{Corollary}
\setcounter{tocdepth}{1}

%opening
\title{Computation in Complex Systems\\
	Week 5
	}

\begin{document}

\maketitle

\section{Recursive Functions}

In Table \ref{table:recursive}, the first column contains either a reference to the exercise in the Exam, or a '-' for auxiliary functions:
\begin{itemize}
	\item \emph{bool} takes us into to the world of Boolean functions\footnote{\emph{mul} can serve as a replacement for \emph{and}; $bool(add)$ replaces \emph{or}};
	\item \emph{not} is our first Boolean function;
	\item \emph{gt} is used, alongside \emph{not}, to define \emph{max} \& \emph{min}.
\end{itemize}

\begin{table}[H]
	\caption{Recursive Functions}\label{table:recursive}
	\begin{tabular}{|c|c|l|l|}\hline
		&&Base case&General\\ \hline
		a&exp&$exp(x,0)=1$&$exp(x,y+1)=mult(exp(x,y),x)$\\ \hline
		b&pred&$pred(0)=0$&$pred(x+1) =x$\\ \hline
		c&sub&$sub(x, 0)=x $&$sub(x, y+1)=pred(sub(x,y)) $\\ \hline
		-&bool&$bool(0)=0$&$bool(x+1)=1$ \\ \hline
		-&not&$not(0)=1$&$not(x+1)=0$ \\ \hline
		-&gt&&$gt(x,y)=bool(sub(x,y))$\\ \hline
		d&min&-&min(x,y)=add(mult(gt(x,y),y),mult(not(gt(x,y)),x))\\ \hline
		e&max&-&max(x,y)=add(mult(gt(x,y),x),mult(not(gt(x,y)),y))\\ \hline
	\end{tabular}
\end{table}

\section{Turing Machines}

Table \ref{table:successor} shows the rules for a Turing machine for the successor function, and Tables  \ref{table:carry}--\ref{table:halt} show the rules through the three component Turing Machines from the exam. I have assumed that all Turing machines start in a special state \includegraphics[width=9pt]{gosign} and halt in state \includegraphics[width=9pt]{stop}: there are no rules that more the Turing machine into \includegraphics[width=9pt]{gosign} or out of \includegraphics[width=9pt]{stop}. Each table comprises two columns for the character under the head and the state, followed by three columns--the character to be written, the new state, and the movement of the head.
\begin{table}[H]
	\begin{center}
		\caption{Successor. There are two states, apart from the special states: in the \emph{Carrying} state we move to the left, setting 1s to 0s as we go, until we find a 0; in the \emph{Returning} state we }\label{table:successor}
		\begin{tabular}{|c|c||c|c|c|} \hline
			$\bullet$ &\includegraphics[width=9pt,height=7pt]{gosign}&$\bullet$ &Carrying&$\leftarrow$ \\ \hline
			1&Carrying&0&Carrying&$\leftarrow$ \\ \hline
			0&Carrying&1&Returning&$\rightarrow$ \\ \hline
			0&Returning&1&Returning&$\rightarrow$ \\ \hline
			1&Returning&1&Returning&$\rightarrow$ \\ \hline
			$\bullet$ &Returning&	$\bullet$&\includegraphics[width=9pt,height=7pt]{stop}& \\ \hline
		\end{tabular}
	\end{center}
\end{table}

The machine described in Table \ref{table:successor} is equivalent to the combination of the following 3 machines: $Halt(Return(Carry(x)))$.

\begin{table}[H]
	\begin{center}
		\caption{Carry: move left, replacing 1 by 0, until we find a  0.}\label{table:carry}
		\begin{tabular}{|c|c||c|c|c|} \hline
			$\bullet$ &\includegraphics[width=9pt,height=7pt]{gosign}&$\bullet$ &Carrying&$\leftarrow$ \\ \hline
			1&Carrying&0&Carrying&$\leftarrow$ \\ \hline
			0&Carrying&1&\includegraphics[width=9pt,height=7pt]{stop}&$\rightarrow$ \\ \hline
		\end{tabular}
	\end{center}
\end{table}

\begin{table}[H]
	\begin{center}
		\caption{Return: move the head back to the final position}\label{table:return}
		\begin{tabular}{|c|c||c|c|c|} \hline
			0&\includegraphics[width=9pt,height=7pt]{gosign}&1&Returning&$\rightarrow$ \\ \hline
			0&Returning&0&Returning&$\rightarrow$ \\ \hline
			1&Returning&1&Returning&$\rightarrow$ \\ \hline
			$\bullet$ &Returning&	$\bullet$&\includegraphics[width=9pt,height=7pt]{stop}& \\ \hline
		\end{tabular}
	\end{center}
\end{table}

\begin{table}[H]
	\begin{center}
		\caption{Halt}\label{table:halt}
		\begin{tabular}{|c|c||c|c|c|} \hline
			$\bullet$ &\includegraphics[width=9pt,height=7pt]{gosign}&$\bullet$ &\includegraphics[width=9pt,height=7pt]{stop}& \\ \hline
		\end{tabular}
	\end{center}
\end{table}

% bibliography go here

\bibliographystyle{unsrt}
\raggedright
\addcontentsline{toc}{section}{Bibliography}
\bibliography{computations}

\end{document}


\subsubsection{Introduction to Induction}

\begin{frame}<beamer>{Structure}
  \tableofcontents[currentsection,currentsubsection]
\end{frame}

\begin{frame}{Induction}
  \textbf{Basics:}
  \begin{itemize}
    \item<2- | handout:1>
      You want to show that assumption $A(n)$ is valid $\forall n \in \mathbb{N}$
    \item<3- | handout:1>
      We show induction in two steps:
      \begin{enumerate}
        \item<4- | handout:1>
          \textbf{Induction basis:} we show that our assumption is valid
          for one value (for example: $n = 1, \, A(1)$).
        \item<5- | handout:1>
          \textbf{Induction step:} we show that the assumption is valid for
          all $n$ (normally one step forward: $n = n + 1, A(1), \ldots, A(n)$).
      \end{enumerate}
    \item<6- | handout:1>
      If both has been proven, then {\color{MainA}$A(n)$} holds for all natural
      numbers {\color{MainA}$n$} by \textbf{induction}
  \end{itemize}
\end{frame}

%-------------------------------------------------------------------------------

\begin{frame}{Induction - Example 1}
  \begin{block}{Claim:}
    A \textbf{complete} binary tree of depth $d$ has $v(d) = 2^d - 1$ nodes
  \end{block}
  \begin{itemize}
    \item<2- |handout:1>
      \textbf{Induction basis:} assumption holds for {\color{MainA}$d = 1$}
      \begin{columns}
        \begin{column}{0.5\linewidth}
          \begin{figure}[!h]
            \begin{adjustbox}{width=4em}
              \input{Images/Induction/BinaryTree_Root.tikz}
            \end{adjustbox}
            \caption{Tree of depth {\color{MainA}$1$} has
              {\color{MainA}$1$} node}
          \end{figure}
        \end{column}
        \begin{column}{0.5\linewidth}
          {\color{MainA}$v(1) = 2^1 - 1 = 1$}\\
          \onslide<3- |handout:1>{$\Rightarrow$ correct $\checkmark$}
        \end{column}
      \end{columns}
  \end{itemize}
\end{frame}

%-------------------------------------------------------------------------------

\begin{frame}{Induction - Example 1}
  Number of nodes $v(d)$ in a binary tree with depth $d$:
  \begin{itemize}
    \item
      \textbf{Induction assumption:}
      {\color{MainA}$v(d) = 2^d-1$}
    \item<2- |handout:1>
      \textbf{Induction basis:}
      $v(1) = 2^d - 1 = 2^1 - 1 = 1 ~{\color{Mittel-Blau}\checkmark}$
    \item<3- |handout:1>
      \textbf{Induction step:}
      to show for $d := d + 1$
      \begin{columns}
        \begin{column}<3- |handout:1>{0.6\textwidth}
          \begin{figure}%
            \begin{adjustbox}{height=7.0em}
              \input{Images/Induction/BinaryTree_SubTrees.tikz}%
            \end{adjustbox}%
            \vspace{-0.5em}
            \caption{binary tree with subtrees}%
            \label{fig:binary_tree_subtrees}%
          \end{figure}
        \end{column}
        \begin{column}{0.4\linewidth}
          \vspace{1.5em}
          \begin{align*}
            \onslide<4- |handout:1>{
              v(d + 1) = &~ 2 \cdot v(d) + 1
            }\\
            \onslide<5- |handout:1>{
              {} = &~ 2 \cdot \textcolor{MainA}{\left(2^d - 1\right)} + 1
            }\\
            \onslide<6- |handout:1>{
              {} = &~ 2^{d + 1} - 2 + 1
            }\\
            \onslide<7- |handout:1>{
              {} = &~ 2^{d + 1} - 1 ~{\color{MainA}\checkmark}
            }
          \end{align*}
          \onslide<8- |handout:1>{
            \llap{$\Rightarrow$ \textbf{By induction:}}
            $v(d) = 2^d - 1 ~\forall d \in \mathbb{N} ~\Square$
          }
        \end{column}
      \end{columns}
  \end{itemize}
\end{frame}