\subsection{Remainders}

Division is defined between natural numbers. However there are many cases where this division does not map to a natural number. For example:

\(\dfrac{7}{3}\)

We can divide \(6\) of the \(7\) by \(3\), giving \(2\) with \(1\) remaining.

Alternatively we can divide \(3\) of the \(7\) by \(3\), giving \(1\) with \(4\) remaining

Or we could divide \(0\) of the \(7\) by \(3\) giving \(0\) with \(7\) remaining.

The remainder refers to the lowest possible number - in this case \(1\).


\section{Divisibility}

\frame{
{Part 1: Divisibility}

\tableofcontents[currentsection,hideallsubsections, firstsection=1, sections={1-5}]
}

\subsection{Definitions}

\begin{frame}
  \frametitle{Talking about Division}

  How should we define the \emph{divide} operation on integers?\bigskip

  Consider $a/b$, for $a,b \in \mathbb{N}$ and $b > 0$. We have:
  \bigskip

  \begin{itemize}
  \item q = quotient(a,b)
  \item r = remainder(a,b)
  \end{itemize}

  \bigskip

  The {\bf Division Theorem} says that $\exists$ {\bf unique} $q$ and $r$ in $\mathbb{N}$ such as
  \begin{equation*}
    a = bq + r, 0 \leq r < b
  \end{equation*}\bigskip

  {\bf Example}: $16/3: a = 16, b = 3, q = 5, r = 1$;\hspace{1cm} $16 = 3\times 5 + 1$

\end{frame}

\begin{frame}
  \frametitle{Divisibility}

  We say that $c$ {\bf divides} $a (c|a)$ {\bf iff}
    \begin{equation*}
      \exists k \in \mathbb{N}, a = k\times c.
    \end{equation*}
    \begin{center}
      $c$ divides $a$ if there is a number $k$ where $a$ equals $c$ times $k$.
    \end{center}

  \bigskip

  \begin{itemize}
  \item $5 | 15$ because $15 = 3 \times 5$
  \item $n | 0$ because $0 = 0 \times n$ \hspace{1cm} every number divides 0
  \item $1 | n$ because $n = n \times 1$ \hspace{1cm} 1 divides every number
  \end{itemize}
\end{frame}

%%%%%%%%%%%%%%%%%%%

\begin{frame}{Implications of Divisibility (1/2)}

  \begin{block}{Lemma 1: $c|a \implies c|(sa)$}
    If $c$ divides $a$, then $c$ divides all multiples of $a$. {\bf Miniproof}: Multiply both sides by $s$:
    \begin{itemize}
      \item $a = kc$\hfill Definition of divisibility
      \item $sa = skc$\hfill Multiplies both sides by $s$
      \item $(sa) = (sk)\times c \implies c|sa$\qed
    \end{itemize}
  \end{block}

  \begin{block}{Lemma 2: $c|a \land c|b \implies c|(a+b)$}
    If $c$ divides $a$ and $b$, then $c$ divides $a+b$. {\bf Miniproof}: Distributive property of multiplication:
    \begin{itemize}
      \item $a = k_1c, b = k_2c$\hfill Definition of divisibility
      \item $a+b = k_1c + k_2c$\hfill Adding both equations together
      \item $(a+b) = (k_1+k_2)c \implies c|(a+b)$\qed
    \end{itemize}
  \end{block}
\end{frame}

\begin{frame}[t]{Implication of Divisibility (2/2)}

  \begin{block}{$c|a \land c|b \implies c | (sa+tb)$}
    If $c$ divides $a$ and $b$, then $c$ divides all \alert{linear combinations} of $a$ and $b$.\\
    {\bf This one is pretty important.}
  \end{block}
  Try to solve it by yourself first:\bigskip

  \begin{onlyenv}<2>
  \begin{itemize}
    \item Lemma 1: $c|a \implies c|ka$ for any $k$.\medskip
    \item Collorary: $c|a \land c|b \implies c|sa \land c|tb$\medskip
    \item Lemma 2: $c|a \land c|b \implies c|(a+b)$.\medskip
    \item Collorary: $c|sa \land c|tb \implies c|(sa+tb)$\qed
  \end{itemize}
  \end{onlyenv}
\end{frame}

\subsection{Common Divisors}

\begin{frame}
  \frametitle{Common Divisors}

    If $c|a$ and $c|b$, we say that c is a \structure{common divisor} of $a$ and $b$.\bigskip

    As we saw in the last slide, a common divisor of $a$ and $b$ will also divide the linear combinations of $a$ and $b$:
    \begin{equation*}
      c|a \land c|b \implies \forall s,t\in\mathbb{N}, c|(sa+tb)
    \end{equation*}\bigskip

    In the next section, let's talk in more details about $c$, $s$ and $t$.
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\subsection{Model fitting}
\label{subsec:fitting}
We employed a Markov Chain Monte Carlo (MCMC), using \texttt{emcee} \parencite{2013PASP..125..306F}, to fit the coefficients of a polynomial model for $r(\mathcal{M}_c)$. The MCMC uses the smoothing prior as defined in \S\ref{subsec:likelihood}. We performed a least squares fit first in order to obtain the initial guess for the coefficients of the model.

The model we used is a polynomial of degree 9. Because of the smoothing function, using higher order does not change the shape of the fit in a significant way. The MCMC best fit can be seen in Figure \ref{fig:line_MCMC}. The MCMC walkers and triangle plot are in Figures \ref{fig:MCMC_time} and \ref{fig:MCMC_triangle}.

The best fit model is
%
\begin{equation}
  r(\mathcal{M}_c) = \sum_{k = 0}^{9}\alpha_k(\log\mathcal{M}_c)^k,
%
  \label{MCMC_best_fit}
\end{equation}
%
where the coefficients, $\alpha_k$, are listed in Table \ref{table:coefficients}.

\begin{table}[ht]
\centering
\caption{Coefficients from the MCMC fit.}
\label{table:coefficients}

\begin{tabular}{l|r}
  k & $\alpha_k$ \\
  \hline
  \hline
  0 & \num{-1.903e-06} \\
  1 & \num{+1.842e-05} \\
  2 & \num{+9.999e-01} \\
  3 & \num{+1.748e-04} \\
  4 & \num{-2.441e-04} \\
  5 & \num{+2.122e-04} \\
  6 & \num{-1.130e-04} \\
  7 & \num{+3.470e-05} \\
  8 & \num{-5.416e-06} \\
  9 & \num{+3.470e-07} \\
\end{tabular}
\end{table}

\begin{figure*}[ht]
  \includegraphics[width=\textwidth]{img/line-MCMC.pdf}
  \caption{The cyan line is an example of a fit from MCMC.}
  \label{fig:line_MCMC}
\end{figure*}

\begin{figure*}[ht]
  \includegraphics[width=\textwidth]{img/line-mcmc_err.pdf}
  \caption{2000 randomly chosen fits from the MCMC.}
  \label{fig:line_MCMC_err}
\end{figure*}

\begin{figure*}[ht]
  \includegraphics[width=\textwidth]{img/line-time-1.pdf}
  \caption{Visualization of MCMC worker locations over the large number of iterations performed for coefficients $\lambda_0 $ to $\lambda_4$. }
  \label{fig:MCMC_time}
\end{figure*}

\begin{figure*}[ht]
  \includegraphics[width=\textwidth]{img/line-time-2.pdf}
  \caption{Visualization of MCMC worker locations over the large number of iterations performed for coefficients $\lambda_5$ to $\lambda_9$.}
  \label{fig:MCMC_time}
\end{figure*}

\begin{figure*}[ht]
  \includegraphics[width=\textwidth]{img/line-triangle.pdf}
  \caption{Corner plot of MCMC fit coefficients.}
  \label{fig:MCMC_triangle}
\end{figure*}


\section{Deep Residual Networks - ResNet}
Deep convolutional neural networks is today used to produce the best results on the ImageNet dataset, and this reveals that the depth of the networks is of high importance for the performance of the networks\citep{RESNET}. Some of the leading results on the ImageNet dataset have models with a depth of 16 to 30 layers. The real question is, if learning better networks is as easy as having more layers in the network?

\myFigure{plain_network}{Results from two networks run on the CIFAR-10 dataset with 20 and 56 layers \citep{RESNET}}{fig:plain}{1}

On figure \ref{fig:plain} it is shown that a convolutional neural network with 20 layers achieves a better performance than a network with 56 layers. A reason why the 56-layer network is bad could be the vanishing / exploding gradients problem, which hamper convergence from the beginning of the training. This problem can be solved by normalized initialization and intermediate normalization layers, which will make the network start converging. After solving this problem the deeper networks will be able to converge, but here a degradation problem might occur, meaning that with the network depth increasing the accuracy gets saturated. the degradation problem is also shown on figure \ref{fig:plain}. To solve this problem a deep residual network can be used.

\myFigure{res_block}{A residual building block. \citep{RESNET}}{fig:resblock}{0.5}

Figure \ref{fig:resblock} shows a residual learning building block. In a residual neural network the underlying layers are fit a residual mapping. The underlying mapping is referred to as \emph{H(x)}, where the nonlinear layers fit a mapping of \emph{F(x) = H(x) - x}. The original mapping is represented as \emph{F(x) + x}, where this mapping can be realized by feed forward neural networks with shortcut connections. Shortcut connections are connections which takes the input \emph{x} and skips it forward to the output of the stacked layers, as seen in figure \ref{fig:resblock}, this is also called identity mapping. The shortcut connections does not add extra complexity to the network. Each residual building block can be defined as:

\begin{equation} \label{eq:res}
y = F(x, {Wi}) + x
\end{equation}

In equation \ref{eq:res}, x and y are the input and output vectors of the layers in the building block. The $F(x,{Wi})$ function represents the residual mapping to be learned throughout the layers. By using figure \ref{fig:resblock} as a building block, the following function will be used for the two layers:

\begin{equation} \label{eq:func_res}
F = W2\sigma(W1x)
\end{equation}

In equation \ref{eq:func_res}, $\sigma$ is the ReLU activation function, while W2 and W1 is the weights for each layer. The shortcut connections added in equation \ref{eq:res} does not introduce any extra parameters or computation complexity. This is good for the comparison between plain and residual networks, as two networks can be compared easily as they have the same amount of parameters, depth, width and computational cost.
\newline

On figure \ref{fig:plainvsres} the first layers of a plain and a residual network is shown. The plain network consist of convolutional layers with a 3x3 filter. When the output feature maps are the same size, the layers have the same amount of filters. If the size of the feature map is divided by two, the filters will be multiplied with two, and this will preserve the time complexity per layer. Downscaling is performed by convolutional layers that have a stride 2.

\myFigure{plain_vs_res}{The left shows the first layers of a plain 34-layer network. The right shows the same layers from a 34-layer residual network. \citep{RESNET}}{fig:plainvsres}{0.5}
\FloatBarrier

The difference from the plain network to the residual network architecture is the shortcut connections. When the input and the output of the layers are of the same dimensions the identity shortcuts can be used directly, this is shown on figure \ref{fig:plainvsres} by the solid lines going from input to output. When the dimensions between input and output increase the dotted line is used in figure \ref{fig:plainvsres}. Two solutions exist to the shortcut connections between two different input and output dimensions. The first solution is to pad extra zeros to the input image, to achieve the same dimension as the output image. This solution does not add any extra paramters to equation \ref{eq:res}. Another solution is to use a projection shortcut which is used to match the dimensions between input and ouput done by 1x1 convolutions. Adding the projection shortcut gives the equation shown in equation \ref{eq:func_proj}, where Ws is added to the equation.

\begin{equation} \label{eq:func_proj}
y = F(x, {Wi}) + Wsx.
\end{equation}


\section{Regularized Dual Averaging}
In this section, we generalize the dual averaging method (\ref{generalized_dual_averaging}) to deal
with composite optimization problems, i.e. problems of the form
\begin{equation}\label{composite_optimization}
\arg\min_x \left[f(x) = F(x) + G(x)\right]
\end{equation}
where $F$ is a convex, Lipschitz function and $G$ is a convex function for which we can efficiently solve the proximal,
or backward step
\begin{equation}\label{prox_regularlized_dual_averaging}
 P_{\lambda G}(x) = \arg\min_{y} \frac{1}{2}\|y - x\|_2^2 + \lambda G(y)
\end{equation}
Two very common choices for $G$ are the indicator function of a convex set $A$
\begin{equation}
 G(x) = i_A(x) =
 \begin{cases}
                    0 &~ \text{if $x\in A$} \\
		    +\infty &~ \text{if $x\notin A$}
 \end{cases}
\end{equation}
and the l$1$-norm $G(x) = \|x\|_1$. In the former case, the proximal map is just a projection onto the set $A$ and
we obtain methods for solving constrained optimization problems. In the latter case, the
proximal map is called soft-thresholding and can be given in closed form.

It is important to note that the methods we will derive for the composite problem (\ref{composite_optimization})
have convergence rates which only depend upon the Lipschitz constant of $F$. This is where the drastic improvement
lies, since often $G$ will not be Lipschitz at all, for instance $G = i_A$, or the Lipschitz constant of $G$
will be very large, for instance $G = \|\cdot\|_1$.

To derive the regularized dual averaging methods, we begin by noting that if $g_i\in \partial F(x_i)$, then
$$ F(x_i) + \langle g_i, x - x_i\rangle + G(x)
$$
is a lower bound for $f(x)$. Substituting this lower bound into the dual averaging iteration (\ref{generalized_dual_averaging}),
we arrive at the following iteration
\begin{equation}\label{naive_regularized_dual_averaging}
 x_{n+1} = \arg\min_x \left(\displaystyle\sum_{i = 1}^n \langle s_ig_i, x\rangle + \frac{\alpha_{n+1}}{2}\|x\|_2^2 + \gamma_{n+1} G(x)\right)
\end{equation}
where $\gamma_{n + 1} = \sum_{i = 1}^n s_i$.

Before we proceed, it is useful to write out the iteration (\ref{naive_regularized_dual_averaging}) in two steps by completing the
square
\begin{equation}
 x_{n+\frac{1}{2}} = -\frac{1}{\alpha_{n+1}}\displaystyle\sum_{i = 1}^n s_ig_i,~x_{n+1} = P_{(\gamma_{n+1}/\alpha_{n+1})G}(x_{n+\frac{1}{2}})
\end{equation}
We note that the second, i.e. proximal, step above means that
$$h_{n+1}:=\frac{\alpha_{n+1}}{\gamma_{n+1}}(x_{n+\frac{1}{2}} - x_{n+1})\in \partial G(x_{n+1})
$$
So in the process of computing the minimizer in (\ref{naive_regularized_dual_averaging}) we obtain an element
of $\partial G(x_{n+1})$ as a byproduct. We can use this element to lower bound $f$ in a different way, namely by
$$ F(x_i) + \langle g_i, x - x_i\rangle + f_i(G(x_i) + \langle h_i, x - x_i\rangle) + (1 - f_i)G(x)
$$
where $0 \leq f_i$. Here we have replaced a fraction $f_i$ of the bound $G(x)$ by its linear lower bound
based on $h_i\in \partial G(x_i)$. Writing $t_i = s_if_i$, this leads to the following method
\begin{equation}\label{argmin_formulation_generalized_RDA}
 x_{n+1} = \arg\min_x \left(\displaystyle\sum_{i = 1}^n \langle s_ig_i + t_ih_i, x\rangle + \frac{\alpha_{n+1}}{2}\|x\|_2^2 + \gamma_{n+1} G(x)\right)
\end{equation}
i.e.
\begin{align}\label{first_attempt_iteration}
 x_{n+\frac{1}{2}} = -\frac{1}{\alpha_{n+1}}\displaystyle\sum_{i = 1}^n &s_ig_i + t_ih_i = \frac{\alpha_n}{\alpha_{n+1}}x_{n-\frac{1}{2}} - \frac{s_ng_n + t_nh_n}{\alpha_{n+1}}\\
 x_{n+1} &= P_{(\gamma_{n+1}/\alpha_{n+1})G}(x_{n+\frac{1}{2}})
\end{align}
where $\gamma_{n + 1} = \sum_{i = 1}^n s_i-t_i$, $h_i = \frac{\alpha_{i}}{\gamma_{i}}(x_{i-\frac{1}{2}} - x_{i})$, and
$g_i\in \partial f(x_i)$. Additionally, we must have $t_1 = 0$ since we don't have an element in the subdifferential
of $G$ at $x_1$ (as $\gamma_1 = 0$).

By utilizing the definition of $h_i$ in (\ref{first_attempt_iteration}), we can rewrite this iteration in the following simpler way
\begin{align}\label{generalized_RDA_iteration}
 &x_{n+\frac{1}{2}} = \frac{\alpha_n}{\alpha_{n+1}}\left(\frac{t_n}{\gamma_n}x_n + \left(1 - \frac{t_n}{\gamma_n}\right)x_{n-\frac{1}{2}}\right) - \frac{s_n}{\alpha_{n+1}}g_n \\
 &x_{n+1} = \arg\min_x \left(\frac{1}{2}\|x - x_{n+\frac{1}{2}}\|_2^2 + \frac{\gamma_{n+1}}{\alpha_{n+1}}G(x)\right)
\end{align}

We have the following convergence result for the generalized RDA method.
\begin{theorem}
 Assume that $F$ is convex and Lipschitz with parameter $M$ and let $x^*\in \arg\min_x f(x)$.
 Then if $x_n$ is given by the iteration (\ref{generalized_RDA_iteration})
 with $t_i\leq \gamma_i$, $\alpha_i\leq \alpha_{i+1}$, and $s_{i+1}\leq s_i$, then we have
 \begin{align}
  &f(\bar{x}_n) - f(x^*) \leq \\
  &\left(\displaystyle\sum_{i = 1}^n s_i\right)^{-1} \left(\alpha_n\|x^*\|_2^2 + M^2\displaystyle\sum_{i = 1}^n \frac{s_i^2}{\alpha_i} + s_1(G(x_1) - \inf_x G(x))\right)
 \end{align}
 where $\bar{x}_n = (\sum_{i = 1}^n s_i)^{-1}\sum_{i = 1}^n s_ix_i$ is a weighted average of the iterates with weights $s_i$.
 We also have
 \begin{align}
  &\min_{i = 1,...,n} f(x_i) - f(x^*) \leq \\
  &\left(\displaystyle\sum_{i = 1}^n s_i\right)^{-1}\left(\alpha_n\|x^*\|_2^2 + M^2\displaystyle\sum_{i = 1}^n \frac{s_i^2}{\alpha_i} + s_1(G(x_1) - \inf_x G(x))\right)
 \end{align}
\end{theorem}

\begin{proof}
  We begin by noting that by convexity we have
 \begin{equation}
  f(\bar{x}_n) - f(x^*) \leq \left(\displaystyle\sum_{i = 1}^n s_i\right)^{-1}\displaystyle\sum_{i = 1}^n s_i(f(x_i) - f(x^*))
 \end{equation}
 and since the minimum is smaller than the average we have
 \begin{equation}
  \min_{i = 1,...,n} f(x_i) - f(x^*) \leq \left(\displaystyle\sum_{i = 1}^n s_i\right)^{-1}\displaystyle\sum_{i = 1}^n s_i(f(x_i) - f(x^*))
 \end{equation}
 Thus it suffices to prove that
 \begin{equation}
 \displaystyle\sum_{i = 1}^n s_i(f(x_i) - f(x^*)) \leq
 \frac{1}{2}\left(\alpha_n\|x^*\|_2^2 + M^2\displaystyle\sum_{i = 1}^n \frac{s_i^2}{\alpha_i}\right)+ s_1(G(x_1) - \inf_x G(x))
 \end{equation}
 For this we first use the subgradient property to get
 \begin{align}\label{eq_rda_100}
  \displaystyle\sum_{i = 1}^n s_i(f(x_i) - f(x^*)) &\leq \displaystyle\sum_{i = 1}^n \langle s_ig_i + t_ih_i, x_i - x^*\rangle \\
  &+ \displaystyle\sum_{i = 1}^n (s_i - t_i)(G(x_i) - G(x^*))
 \end{align}
 We now rewrite the first sum above as
 \begin{align}\label{eq_rda_105}
  \displaystyle\sum_{i = 1}^n \langle s_ig_i + t_ih_i, x_i - x^*\rangle &= \displaystyle\sum_{i = 1}^n \langle s_ig_i + t_ih_i, x_n - x^*\rangle \\
  &+ \displaystyle\sum_{i = 1}^{n-1} \displaystyle\sum_{j = 1}^i\langle s_jg_j + t_jh_j, x_i - x_{i+1}\rangle
 \end{align}
 We proceed by bounding the term
 \begin{equation}
  \sum_{j = 1}^i\langle s_jg_j + t_jh_j, x_i - z\rangle
 \end{equation}
 Recall from (\ref{argmin_formulation_generalized_RDA}) that $x_i = \arg\min_x f_i(x)$ where
 \begin{equation}
  f_i(x) = \displaystyle\sum_{j = 1}^{i-1} \langle s_jg_j + t_jh_j, x\rangle + \frac{\alpha_{i}}{2}\|x\|_2^2 + \gamma_i G(x_i)
 \end{equation}
 The definition of $h_i\in \partial G(x_i)$ means that
 \begin{equation}
  0 = \alpha_i x_i + \gamma_i h_i + \displaystyle\sum_{j = 1}^{i-1} s_jg_j + t_jh_j
 \end{equation}
 This means that $0\in \partial f_i^\prime(x_i)$ if we set
 $$f_i^\prime(x) = f_i(x) + \langle t_ih_i, x\rangle - t_iG(x)
 $$
 Moreover, $f_i^\prime$ is a strongly convex function with convexity parameter $\alpha_i$ as long as $t_i\leq \gamma_i$.
 Now, define $f^{\prime\prime}_i(x) = f_i^\prime(x) + \langle s_ig_i, x\rangle$. Then we have
 $s_ig_i\in \partial f^{\prime\prime}_i(x_i)$ and the strong convexity of $f^{\prime\prime}_i$ implies that
 \begin{equation}
  f^{\prime\prime}_i(x_i) \leq f^{\prime\prime}_i(z) + \frac{1}{2\alpha_i}\|s_ig_i\|_2^2
 \end{equation}
 Substituting in the definition of $f^{\prime\prime}_i$ and rearranging, we get
 \begin{align}
  \sum_{j = 1}^i\langle s_jg_j + t_jh_j, x_i - z\rangle &\leq \frac{\alpha_i}{2}(\|z\|_2^2 - \|x_i\|_2^2) + \frac{s_i^2}{2\alpha_i}\|g_i\|_2^2 \\
  &+ (\gamma_i - t_i)(G(z) - G(x_i))
 \end{align}
 Plugging this bound into (\ref{eq_rda_105}) with $z = x_{i+1}$ and $z = x^*$, recalling that
 $\gamma_n = \sum_{i = 1}^{n-1} s_i - t_i$, that $x_1 = 0$, $\gamma_1 = t_1 = 0$, and that
 $\|g_i\|_2\leq M$, and rearranging terms, we obtain
 \begin{align}
  \displaystyle\sum_{i = 1}^n \langle s_ig_i + t_ih_i, x_i - x^*\rangle& \leq \frac{\alpha_n}{2}\|x^*\|_2^2
  + \frac{1}{2}\displaystyle\sum_{i = 2}^n (\alpha_{i-1} - \alpha_i)\|x_i\|_2^2 \\
  &+ \frac{M^2}{2}\displaystyle\sum_{i = 1}^n \frac{s_i^2}{\alpha_i}
  + \displaystyle\sum_{i = 2}^n (t_i - s_{i-1})G(x_i)
 \end{align}
 Plugging this bound into (\ref{eq_rda_100}) we obtain
 \begin{align}
  \displaystyle\sum_{i = 1}^n s_i(f(x_i) - f(x^*)) &\leq \frac{\alpha_n}{2}\|x^*\|_2^2
  + \frac{M^2}{2}\displaystyle\sum_{i = 1}^n \frac{s_i^2}{\alpha_i} \\
  &+ \frac{1}{2}\displaystyle\sum_{i = 2}^n (\alpha_{i-1} - \alpha_i)\|x_i\|_2^2 \\
  &+ s_1 G(x_1) - \displaystyle\sum_{i = 2}^n (s_{i-1} - s_i)G(x_i) - s_nG(x^*)
 \end{align}
 Finally, we use the assumption that $\alpha_{i-1} - \alpha_i \leq 0$ and that $s_{i-1} - s_i \geq 0$ to conclude that
 $$\frac{1}{2}\displaystyle\sum_{i = 2}^n (\alpha_{i-1} - \alpha_i)\|x_i\|_2^2 \leq 0
 $$
 and
 $$s_1 G(x_1) - \displaystyle\sum_{i = 2}^n (s_{i-1} - s_i)G(x_i) - s_nG(x^*) \leq s_1(G(x_1) - \inf_x G(x))
 $$
 Finally, this yields
 \begin{equation}
  \displaystyle\sum_{i = 1}^n s_i(f(x_i) - f(x^*)) \leq \frac{\alpha_n}{2}\|x^*\|_2^2
  + \frac{M^2}{2}\displaystyle\sum_{i = 1}^n \frac{s_i^2}{\alpha_i}
  + s_1(G(x_1) - \inf_x G(x))
 \end{equation}
 which completes the proof.

\end{proof}

We begin by noting that the regularizer $\|x\|_2^2$ can be replaced by any strongly convex function. In particular,
if we replace it by $\|x - x_1\|_2^2$, we are free to choose $x_1$ arbitrarily. This allows us to remove the
annoying term $s_1(G(x_1) - \inf_x G(x))$ from the bound in the above theorem, by simply choosing $x_1$ to be a minimizer
of $G$.

We would also like to note two special cases of the above method. If we set $t_i = \frac{1}{\sqrt{i-1}}$ for $i > 1$,
$s_i = \frac{1}{\sqrt{i}}$, and $\alpha_i = 1$ (note that this means $\gamma_i = t_i$),
then we recover forward-backward subgradient descent.

Also, if we set $t_i = 0$, $s_i = 1$, and $\alpha_i = \sqrt{n}$ (note that this means $\gamma_i = i$), we recover the
original version of RDA.


\chapter{Splines}

Through-out this section, the regression function $f$ will depend on a
single, real-valued predictor $X$ ranging over some possibly infinite
interval of the real line, $I \subset \mathbb{R}$. Therefore, the
(mean) dependence of $Y$ on $X$ is given by
\begin{equation}
\label{fdef}
f(x) = \E(Y|X=x), x \in I \subset \mathbb{R}.
\end{equation}
For spline models, estimate definitions and their properties
are more easily characterized in the context of linear spaces.

\input{section-04-01}
\input{section-04-02}
\input{section-04-03}
\input{section-04-04}
\input{references-04}


\section{Notes for myself}

It is possible to generate random numbers
for various distributions using unit rectangular
generator, see e.g. \cite[][p.196,197]{evans2000}:

\begin{quote}
Random numbers of the Weibull variate $W$: $\gamma$,
$\eta$, $\beta$ can be generated from the unit rectangular
variate $R$ using the relationship:

\begin{displaymath}
W: \gamma, \eta, \beta \sim \gamma +
 \eta
  \left(
   - \log R
  \right)
          ^ {1/\beta}
\end{displaymath}

\end{quote}


\section{Homogeneous systems}
\label{sec:homogeneous-systems}

\begin{outcome}
  \begin{enumerate}
  \item Determine whether a homogeneous system of equations has
    non-trivial solutions from its rank.
  \item Find the basic solutions of a homogeneous system of
    equations.
  \item Understand the relationship between the general solution of a
    system of equations and that of its associated homogeneous
    system.
  \end{enumerate}
\end{outcome}

There is a special type of system of linear equations that requires
additional study. This type of system is called a {\em
  homogeneous}\footnote{The word ``homogeneous'' has 5 syllables. In
  scientific usage, it is not the same as the word
  ``\nospellcheck{\homogenous}''.}  system of equations.  Our focus in
this section is to consider what types of solutions are possible for a
homogeneous system of equations, and how the solutions of
non-homogeneous systems are related to those of their homogeneous
counterparts.

\begin{definition}{Homogeneous system of equations}{homogeneous-system}
  A system of equations is called
  \textbf{homogeneous}%
  \index{system of linear equations!homogeneous}%
  \index{homogeneous system}
  if each of the constant terms is equal to $0$. A homogeneous system
  therefore has the form
  \begin{equation*}
    \begin{array}{c}
      a_{11}x_1 + a_{12}x_2 + \ldots + a_{1n}x_n = 0 \\
      a_{21}x_1 + a_{22}x_2 + \ldots + a_{2n}x_n = 0  \\
      \vdots \\
      a_{m1}x_1 + a_{m2}x_2 + \ldots + a_{mn}x_n = 0,
    \end{array}
  \end{equation*}
  where $a_{ij}$ are coefficients and $x_j$ are variables.
\end{definition}

The first thing we note is that a homogeneous system is always
consistent. Indeed, it always has the solution $x_1=0$, $x_2=0$,
$\ldots$, $x_n=0$. This solution is called the \textbf{trivial
  solution}%
\index{trivial solution}%
\index{solution!trivial}%
\index{system of linear equations!trivial solution}.

If the system has a solution in which not all of the $x_1,\ldots, x_n$
are equal to zero, then we call this solution \textbf{non-trivial}%
\index{non-trivial solution}%
\index{solution!non-trivial}%
\index{system of linear equations!non-trivial solution}.  When working
with homogeneous systems of equations, since the trivial solution
always exists, we are usually interested in finding whether there are
non-trivial solutions.

The following theorem is a special case of
Theorem~\ref{thm:rank-consistent-solutions}. Recall that the {\em
  rank}%
\index{rank} of a system is the number of pivot variables in its
{\ef}.

\begin{theorem}{Rank and solutions of homogeneous system of equations}{rank-homogeneous-solutions}
  Consider a homogeneous system of $m$ equations in $n$ variables, and
  assume that the coefficient matrix has rank $r$. Then the system is
  consistent, and
  \begin{enumerate}
  \item if $r=n$, then the system has only the trivial solution;
  \item if $r<n$, then the system has infinitely many solutions.
  \end{enumerate}
\end{theorem}

\begin{example}{Homogeneous system with more variables than equations}{homo-more-variables}
  True or false: Suppose a homogeneous system has more variables than
  equations. Then the system has infinitely many solutions.
\end{example}

\begin{solution}
  This is true. If the system has $m$ equations and $n$ variables,
  then the rank can be at most $m$. Since $m<n$, the system has
  infinitely many solutions. Note that it is not possible for a
  homogeneous system to be inconsistent, since there is always the
  trivial solution.
\end{solution}

\begin{example}{Homogeneous system with an equal number of variables and equations}{homo-less-variables}
  True or false: Suppose a homogeneous system has the same number of
  variables as equations. Then the system has a unique solution.
\end{example}

\begin{solution}
  This is false in general. While it is possible for such a system to
  have a unique solution, it is also possible for it to have
  infinitely many. Let there be $n$ equations and $n$ variables.  Then
  depending on the {\ef}, the rank $r$ could be either equal to $n$,
  in which case there is a unique solution, or less than $n$, in which
  case there are infinitely many.
\end{solution}

We now consider an example of solving a homogeneous system of equations.

\begin{example}{Solutions to a homogeneous system of equations}{homogeneous-solution}
  Find the general solution to the following homogeneous system of
  equations. Does the system have non-trivial solutions?
  \begin{equation*}
    \begin{array}{c}
      2x + y + z + 4w = 0 \\
      x + 2y - z + 5w = 0
    \end{array}
  \end{equation*}
\end{example}

\begin{solution}
  Notice that this system has $m = 2$ equations and $n = 4$ variables,
  so $n>m$.  Therefore by our previous discussion, we expect this
  system to have infinitely many solutions. In particular, it will
  have non-trivial solutions.

  The process we use to find the solutions for a homogeneous system of
  equations is the same process we used for non-homogeneous
  equations. We construct the augmented matrix and reduce it to
  {\rref}.
  \begin{equation*}
    \begin{mymatrix}{rrrr|r}
      2 & 1 & 1 & 4 & 0 \\
      1 & 2 & -1 & 5 & 0
    \end{mymatrix}
    \roweq
    \begin{mymatrix}{rrrr|r}
      1 & 0 &  1 & 1 & 0 \\
      0 & 1 & -1 & 2 & 0
    \end{mymatrix}
  \end{equation*}
  The corresponding system of equations is
  \begin{equation*}
    \begin{array}{r}
      x + z + w = 0 \\
      y - z + 2w = 0. \\
    \end{array}
  \end{equation*}
  The free variables are $z$ and $w$. We set them equal to parameters
  $z=s$ and $w=t$. Then our general solution has the form
  \begin{equation*}
    \begin{array}{c}
      x = -s-t \\
      y = s-2t \\
      z = s \\
      w = t.
    \end{array}
  \end{equation*}
  Hence this system has infinitely many solutions, with two parameters
  $s$ and $t$.
\end{solution}

Let us write the solution of the last example in another form.
Specifically, it can be written as
\begin{equation}\label{eqn:homogeneous-solution-1}
  \begin{mymatrix}{r}
    x \\
    y \\
    z \\
    w
  \end{mymatrix}
  =
  s
  \begin{mymatrix}{r}
    -1 \\
    1 \\
    1 \\
    0
  \end{mymatrix}
  +
  t
  \begin{mymatrix}{r}
    -1 \\
    -2 \\
    0 \\
    1
  \end{mymatrix}.
\end{equation}
Notice that we have constructed a column from the coefficients of $s$
in each equation, and another column from the coefficients of $t$.  We
will discuss this notation more in later chapters. For now, consider
what happens when we choose the parameters to be $s=1$ and $t=0$. In
this case, we get the solution
\begin{equation}\label{eqn:homogeneous-solution-2}
  \begin{mymatrix}{r}
    -1 \\
    1 \\
    1 \\
    0
  \end{mymatrix},
\end{equation}
which is the same as the column of coefficients for $s$. This is
called a \textbf{basic solution}%
\index{basic solution}%
\index{solution!basic} of the
homogeneous system of equations. The other basic solution is obtained
by setting $s=0$ and $t=1$. In this case,
\begin{equation}\label{eqn:homogeneous-solution-3}
  \begin{mymatrix}{r}
    -1 \\
    -2 \\
    0 \\
    1
  \end{mymatrix}.
\end{equation}
The basic solutions of a system are columns constructed from the
coefficients on parameters in the solution. If $X_1$ and $X_2$ are the
basic solutions {\eqref{eqn:homogeneous-solution-2}} and
{\eqref{eqn:homogeneous-solution-3}}, then the general solution
{\eqref{eqn:homogeneous-solution-1}} is of the form $sX_1+tX_2$.  We
say that the general solution of the homogeneous system is a
\textbf{linear combination}%
\index{linear combination!of basic solutions} of its basic solutions.

We explore this further in the following example.

\begin{example}{Basic solutions of a homogeneous system}{basic-solutions}
  Consider the following homogeneous system of equations.
  \begin{equation}\label{eqn:basic-solutions-1}
    \begin{array}{c}
      x + 4y + 3z = 0 \\
      3x + 12y + 9z = 0.
    \end{array}
  \end{equation}
  Find the basic solutions to this system.
\end{example}

\begin{solution}
  The augmented matrix of this system and the resulting {\rref} are
  \begin{equation*}
    \begin{mymatrix}{rrr|r}
      1 & 4 & 3 & 0 \\
      3 & 12 & 9 & 0
    \end{mymatrix}
    \roweq
    \begin{mymatrix}{rrr|r}
      1 & 4 & 3 & 0 \\
      0 & 0 & 0 & 0
    \end{mymatrix}.
  \end{equation*}
  When written in equations, this system is given by
  \begin{equation*}
    x + 4y + 3z = 0.
  \end{equation*}
  Notice that $x$ is the only pivot variable, and $y$ and $z$ are free
  variables. Let $y = s$ and $z=t$ for parameters $s$ and $t$. Then the
  general solution is
  \begin{equation*}
    \begin{array}{c}
      x = -4s - 3t \\
      y = s \\
      z = t,
    \end{array}
  \end{equation*}
  which can be written as
  \begin{equation*}
    \begin{mymatrix}{r}
      x \\
      y \\
      z
    \end{mymatrix}
    =
    s
    \begin{mymatrix}{r}
      -4 \\
      1 \\
      0
    \end{mymatrix}
    +
    t
    \begin{mymatrix}{r}
      -3 \\
      0 \\
      1
    \end{mymatrix}.
  \end{equation*}
  You can see here that we have two columns of coefficients
  corresponding to parameters, specifically one for $s$ and one for $t$.
  Therefore, this system has two basic solutions! They are
  \begin{equation*}
    X_1=
    \begin{mymatrix}{r}
      -4 \\
      1 \\
      0
    \end{mymatrix},\quad X_2 = \begin{mymatrix}{r}
      -3 \\
      0 \\
      1
    \end{mymatrix}.
  \end{equation*}
\end{solution}

We can take any non-homogeneous system of equations and get a new
homogeneous system by keeping the left-hand sides the same and setting
all of the constant terms equal to $0$. This is called the
\textbf{associated homogeneous system}%
\index{homogeneous system!associated}%
\index{associated homogeneous system}
of the system of equations. We end this section by investigating how
the solutions of a system of equations are related to the solutions of
its associated homogeneous system.

\begin{example}{Non-homogeneous vs. homogeneous system}{non-homo-vs-homo}
  Solve the system of equations
  \begin{equation}\label{eqn:non-homo-vs-homo-1}
    \begin{array}{c}
      x + 4y + 3z = 2 \\
      3x + 12y + 9z = 6.
    \end{array}
  \end{equation}
  How are the solutions related to those of the associated homogeneous
  system in Example~\ref{exa:basic-solutions}?
\end{example}

\begin{solution}
  We note that the associated homogeneous system of
  {\eqref{eqn:non-homo-vs-homo-1}} is the system we saw in
  Example~\ref{exa:basic-solutions}. We solve the system
  {\eqref{eqn:non-homo-vs-homo-1}} in the usual way by reducing its
  augmented matrix to {\rref}
  \begin{equation*}
    \begin{mymatrix}{rrr|r}
      1 & 4 & 3 & 2 \\
      3 & 12 & 9 & 6
    \end{mymatrix}
    \roweq
    \begin{mymatrix}{rrr|r}
      1 & 4 & 3 & 2 \\
      0 & 0 & 0 & 0
    \end{mymatrix}
  \end{equation*}
  and then assigning parameters $y=s$, $z=t$ to the free
  variables. From the equation $x+4y+3z=2$, the general solution is
  \begin{equation*}
    \begin{array}{c}
      x = 2 -4s - 3t \\
      y = s \\
      z = t,
    \end{array}
  \end{equation*}
  which can be written as
  \begin{equation}\label{eqn:non-homo-vs-homo-2}
    \begin{mymatrix}{r}
      x \\
      y \\
      z
    \end{mymatrix}
    =
    \begin{mymatrix}{r}
      2 \\
      0 \\
      0
    \end{mymatrix}
    +
    s
    \begin{mymatrix}{r}
      -4 \\
      1 \\
      0
    \end{mymatrix}
    +
    t
    \begin{mymatrix}{r}
      -3 \\
      0 \\
      1
    \end{mymatrix}.
  \end{equation}
  We see that the general solution is almost exactly the same as that
  of the homogeneous system in Example~\ref{exa:basic-solutions}. The
  only difference is the additional column
  \begin{equation}\label{eqn:non-homo-vs-homo-3}
    \begin{mymatrix}{r}
      2 \\
      0 \\
      0
    \end{mymatrix}
  \end{equation}
\end{solution}

Note that the column {\eqref{eqn:non-homo-vs-homo-3}}, by itself, is a
solution of the non-homogeneous system. It is not the most general
solution, but rather the particular solution resulting from the
parameters $s=0$ and $t=0$. We can therefore interpret equation
{\eqref{eqn:non-homo-vs-homo-2}} as saying that the general solution%
\index{general solution} of the non-homogeneous system is equal to a
particular solution%
\index{particular solution} of the non-homogeneous system, plus the
general solution of the associated homogeneous system. The same is
true in general, and we summarize it as a theorem.

\begin{theorem}{Non-homogeneous vs. homogeneous system}{non-homo-vs-homo}
  Let $A$ be a system of equations, and let $B$ be the associated
  homogeneous system. Then
  \begin{equation*}
    \mbox{the general solution of $A$}
    = \mbox{a particular solution of $A$}
    + \mbox{the general solution of $B$}.
  \end{equation*}
\end{theorem}


\subsection{Residue systems}

\subsubsection{Least residue system modulo \(n\)}

This is the set of numbers from \(0\) to \(n-1\).

\subsubsection{Complete residue system}

This a set of numbers none of which are congruent \(\mod n\). That is, for no pair \(\{a,b\}\) does \(a \mod(n)=b mod(n)\)

\subsubsection{Reduced residue system}

This is a complete residue system where all numbers are relatively prime to \(n\).


\subsection{Functionals}

Functionals map functions to scalars. They are the \(1\)-forms of infinite-dimensional vector spaces.

If we have a function \(f\), we can write functional \(J[f]\).

\subsubsection{More}

We can define neighbourhoods around a function \(f\). For example, taking \(y\) to be \(f\) with infintesimal changes. to each of the values.

The difference between the functional at both points is

\(\delta J=J[y]-J[f]\)

\subsubsection{Extrema}

If

\(\delta J=J[y]-J[f]\)

is the same sign for all y around f, then J has an extremum at f.

\subsubsection{Functional derivatives}


\begin{Definition}{facets}
  Let $\cell\subset \R^d$ be a polyhedron. We call the lower
  dimensional polyhedra constituting its boundary \define{facet}s. A
  facet of dimension zero is called \define{vertex}, of dimension one
  \define{edge}, and a facet of codimension one is called a
  \define{face}.
\end{Definition}

\begin{Definition}{mesh}
  A \define{mesh} $\mesh$ is a nonoverlapping subdivision of the
  domain $\domain$ into polyhedral \define{cell}s denoted by $\cell$,
  for instance simplices, quadrilaterals, or hexahedra. The
  faces of a cell are denoted by $\face$, the
  vertices by $\vertex$. Cells are typically considered open sets.

  A mesh $\mesh$ is called regular, if each face
  $\face \subset \d\cell$ of the cell $\cell\in\mesh$ is either a
  face of another cell $\cell\prime$, that is,
  $\overline{\face} = \overline{\cell} \cap \overline{\cell\prime}$,
  or a subset of $\d\domain$.
\end{Definition}

\begin{remark}
  For this introduction, we will assume that indeed $\domain$ is the
  union of mesh cells, which means, that its boundary consists of a
  finite union of planar faces. The more general case of a mesh
  approximating the domain will be deferred to later discussion.
\end{remark}

\begin{Definition}{finite-element}
  With a mesh cell $\cell$, we associate a finite dimensional
  \define{shape function} space $\shapespace(\cell)$ of dimension
  $n_\cell$. The term \define{node functional} denotes linear
  functionals on this space.

  A set of node functionals $\{\nodal_\cell^i\}_{i=1,\dots,n_\cell}$ is called
  \define{unisolvent} on $\shapespace(\cell)$ if for any vector
  $\vu = (u_1,\dots,u_{n_\cell})^T$ there exists a unique
  $u\in \shapespace(\cell)$ such that
  \begin{gather}
    \nodal_\cell^i(u) = \vu_i,\quad i=1,\dots,n_\cell.
  \end{gather}

  A \define{finite element} is a set of shape function spaces
  $\shapespace(\cell)$ for all $\cell\in\mesh$ together with
  unisolvent set of node functionals.
\end{Definition}

\begin{Notation}{dofs}
  If the node functionals $\nodal^i$ are unisolvent on
  $\shapespace(\cell)$, then, there is a basis $\{p_k\}$ of $\shapespace(\cell)$
  such that
  \begin{gather}
    \nodal^i(p_k) = \delta_{ik}.
  \end{gather}
  We refer to $\{p_k\}$ as \define{shape function basis} and use the
  term \define{degrees of freedom} for both the node functionals and
  the basis functions.
\end{Notation}

\begin{Definition}{node-topology}
  Node functionals can be associated with the cell $\cell$ or with one
  of its lower dimensional boundary facets. We call this association
  the \define{topology} of the finite element.
\end{Definition}

\begin{Definition}{fe-space}
  The \define{finite element space} on the mesh $\mesh$, denoted by
  $V_\mesh$ is a subset of the concatenation of all shape function
  spaces,
  \begin{gather}
    V_\mesh \subset \bigl\{ f\in L^2(\domain) \big|
    f_{|\cell} \in \shapespace(\cell) \bigr\}.
  \end{gather}
  The \define{degrees of freedom} of $V_\mesh$ are the union of all
  node functionals, where we identify node functionals associated to
  boundary facets among all cells sharing this facet. The resulting
  dimension is
  \begin{gather}
    n = \dim V_\mesh \le \sum n_\cell.
  \end{gather}
\end{Definition}

\begin{figure}[tp]
  \begin{center}
    \includegraphics[width=.5\textwidth]{graph/concatenation.tikz}
  \end{center}
  \caption{Identification of node functionals. The node functionals on
    shared edges (separated for presentation purposes) are
    distinguished locally as belonging to their respective cells, but
    identical global indices are assigned to all nodes in a single
    circle. Thus, all associated shape functions obtain the same
    coefficient in the global basis representation of a finite element
    function $u$.}
  \label{fig:nodes-identification}
\end{figure}

\begin{Notation}{global-local}
  When we enumerate the degrees of freedom of $V_\mesh$, we obtain a
  global numbering of degrees of freedom $\nodal^i$ with
  $i=1,\dots,n$. For each mesh cell, we have a local numbering
  $\nodal_\cell^j$ with $j=1,\dots,n$. By construction of the finite
  element space, there is a unique $i$, such that
  $\nodal_\cell^j(f) = \nodal^i(f)$ for all cells $\cell$ and local
  indices $j$. The converse is not true due to the identification
  process.
\end{Notation}

\begin{Definition}{local-global}
  We refer to the mapping between $\nodal^i$ and $\nodal_\cell^j$ as
  the mapping between global and local indices
  \begin{gather}
    \iota: (\cell, j) \mapsto i.
  \end{gather}
  It induces a
  ``natural'' basis $\{v_i\}$ of $V_\mesh$ by
  \begin{gather}
    v_{i|\cell} = p_{\cell,j},
  \end{gather}
  where $\{p_{\cell,j}\}$ is the shape function basis on $\cell$. For
  each $\nodal^i$, we define $\mesh(\nodal^i)$ as the set of cells
  $\cell$ sharing the node functional $\nodal^i$, and
  \begin{gather}
    \domain\left(\nodal^i\right) = \bigcup_{\cell\in \mesh(\nodal^i)} \cell.
  \end{gather}
\end{Definition}

\begin{Lemma}{fe-support}
  The support of the basis function $v_i\in V_\mesh$ is
  \begin{gather*}
    \operatorname{supp}(v_i) \subset \domain\left(\nodal^i\right).
  \end{gather*}
\end{Lemma}

\begin{Lemma}{mesh-continuity}
  Let $\mesh$ be a subdivision of $\domain$, and let $u$ be a function
  on $\domain$, such that $u_{|\cell} \in C^1(\overline{\cell})$ for
  each $\cell\in\mesh$. Then,
  \begin{gather}
    u\in H^1(\domain)
    \quad \Longleftrightarrow\quad
    u\in C(\overline\domain).
  \end{gather}
\end{Lemma}

\begin{Lemma}{nodal-continuity}
  We have $V_\mesh\subset C(\overline{\domain})$ if and only if for
  every facet $F$ of dimension $d_F < d$ there holds that
  \begin{enumerate}
  \item the traces of the spaces $\shapespace(\cell)$ on $F$ coincide
    for all cells $\cell$ having $F$ as a facet,
  \item The node functionals associated to the facet are unisolvent on
    this trace space.
  \end{enumerate}
\end{Lemma}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Shape function spaces on simplices}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{Definition}{barycentric-coordinates}
  A simplex $\cell\in \R^d$ with vertices $\vertex_0,\dots,\vertex_d$
  is described by a set of $d+1$ \define{barycentric coordinates}
  $\vlambda = (\lambda_0,\dots,\lambda_d)^T$ such that
  \begin{xalignat}2
    0\le\lambda_i &\le 1& i&=0,\dots,d;\\
    \lambda_i(\vertex_j) &= \delta_{ij}& i,j&=0,\dots,d\\
    \sum \lambda_i(\vx) &= 1,
  \end{xalignat}
  and there holds
  \begin{gather}
    T = \Bigl\{x\in\R^d \Big| x = \sum \vertex_k\lambda_k \Bigr\}.
  \end{gather}
\end{Definition}

\begin{todo}
  Properties of simplicial coordinates, e.g. $\{\lambda_i=0\}$ is the facet opposite to $\vertex_i$.
\end{todo}

\begin{Lemma}{barycentric-affine}
  There is a matrix $B_T\in \R^{d+1\times d}$ and a vector
  $b_T\in\R^{d+1}$, such that
  \begin{gather}
    \vlambda = B_T\vx + b_T.
  \end{gather}
\end{Lemma}

\begin{Corollary}{barycentric-interpolation}
  The barycentric coordinates $\lambda_0,\dots,\lambda_d$ are the
  linear Lagrange interpolating functions for the points
  $\vertex_0,\dots,\vertex_d$. In particular, $\lambda_k \equiv 0$ on
  the facet not containing $\vertex_k$.
\end{Corollary}

\begin{example}
  We can use barycentric coordinates to define interpolating polynomials on
  simplicial meshes easily, as in
  Table~\ref{tab:barycentric-shapes}.
  \begin{table}[tp]
    \centering
    \begin{tabular}{|c|l|}
      \hline Degrees of freedom
      & Shape functions \\\hline
      \adjustbox{valign=center,margin=3pt}{\includegraphics[width=2cm]{mixed/fig/p1-p.tikz}}
      &
        {\begin{minipage}[b]{6cm}
          \begin{gather*}
            \phi_i = \lambda_i,
            \quad i=0,1,2
          \end{gather*}
        \end{minipage}}
      \\\hline
      \adjustbox{valign=center,margin=3pt}{\includegraphics[width=2cm]{mixed/fig/p2-p.tikz}}
      &
        {\begin{minipage}[b]{6cm}
          \begin{xalignat*}2
            \phi_{ii} &= 2\lambda_i^2 - \lambda_i,
            &i&=0,1,2\\
            \phi_{ij} &= 4\lambda_i\lambda_j
            &j&\neq i
          \end{xalignat*}
        \end{minipage}}
        \\\hline
      \adjustbox{valign=center,margin=3pt}{\includegraphics[width=2cm]{mixed/fig/p3-p.tikz}}
      &
        {\begin{minipage}[b]{6cm}
          \begin{xalignat*}2
          \phi_{iii} &= \tfrac12 \lambda_i(3\lambda_i-1)(3\lambda_i-2)
          &i&=0,1,2\\
          \phi_{ij} &= \tfrac92\lambda_i\lambda_j(3\lambda_j-1)
          &j&\neq i\\
          \phi_0 &= 27\lambda_0\lambda_1\lambda_2
        \end{xalignat*}
        \end{minipage}}
        \\\hline
    \end{tabular}
    \caption{Degrees of freedom and shape functions of simplicial elements
      in terms of barycentric coordinates}
    \label{tab:barycentric-shapes}
  \end{table}
\end{example}

\begin{remark}
  The functions $\lambda_i(x)$ are the shape functions of the linear
  $P_1$ element on $T$. They allow us to define basis functions on the
  cell $T$ without use of a reference element $\widehat T$.

  Note that $\lambda_i\equiv 0$ on the face opposite to the
  vertex $x_i$.
\end{remark}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Shape functions on tensor product cells}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{Definition}{tensor-product-polynomials}
  The space of \define{tensor product polynomials} of degree $k$ in
  $d$ dimensions, denoted as $\Q_k$ consists of polynomials of degree
  up to $k$ in each variable. Given a basis for one-dimensional
  polynomials $\{p_i\}_{i=0,\dots,k}$, a natural basis for $\Q_k$ is
  the \define{tensor product basis}
  \begin{gather}
    \label{eq:fem-intro:1}
    p_{i_1,\dots,i_d}(\vx)
    = p_{i_1}\otimes \dots\otimes p_{i_d}(\vx)
    = \prod_{k=1}^d p_{i_k}(x_k).
  \end{gather}
\end{Definition}

\begin{remark}
  Note that the basis functions of $\Q_k$ can be denoted as products
  of univariate polynomials, but that general polynomials in this
  space as linear combinations of these basis functions do not have
  this structure.
\end{remark}

\begin{Lemma}{tensor-product-node-functionals}
  Let $\{\nodal_j\}$ be a set of one-dimensional node functionals dual
  to the one-dimensional basis $\{p_i\}$ such that
  \begin{gather}
    \nodal_j(p_i) = \delta_{ij}.
  \end{gather}
  Then, a dual basis for $\{p_{i_1,\dots,i_d}\}$ is obtained by
  defining on the tensor product basis of $\Q_k$
  \begin{gather}
    \label{eq:fem-intro:2}
    \nodal_{j_1,\dots,j_d}(p_{i_1,\dots,i_d})
    = \nodal_{j_1} \otimes \dots\otimes \nodal_{j_d}(p_{i_1}\otimes\dots\otimes p_{i_d})
    = \prod_{k=1}^d \nodal_{j_k}(p_{i_k}).
  \end{gather}
\end{Lemma}

\begin{proof}
  It is a theorem in linear algebra, that a linear functional on a
  vector space is uniquely defined by its values on a basis of the
  space. Thus,~\eqref{eq:fem-intro:2} uniquely defines the node
  functionals $\nodal_{j_1,\dots,j_d}$. The duality property follows
  from the fact that
  \begin{gather*}
    \nodal_{j_1,\dots,j_d}(p_{i_1,\dots,i_d}) = \prod_{k=1}^d \delta_{i_k,j_k},
  \end{gather*}
  which is one if and only if all index pairs match and zero in all
  other cases.
\end{proof}

\begin{example}
  Let a basis $\{p_i\}$ of the univariate space $\P_k$ be defined by
  Lagrange interpolation in $k+1$ points $t_j \in [0,1]$. A basis of
  the $d$-dimensional space $\Q_k$ is then obtained by all possible
  products
  \begin{gather*}
    p_{i_1,\dots,i_d}(\vx) = \prod_{k=1}^d p_{i_k}(x_k).
  \end{gather*}
  The node functionals following the construction above are obtained by
  \begin{gather*}
    \nodal_{j_1,\dots,j_d}(p_{i_1,\dots,i_d}) = \prod_{k=1}^d p_{i_k}(x_{j_k}).
  \end{gather*}
  Finally, we have to convert the term on the right into an
  expression, which can be applied to any polynomial in $\Q_k$. To
  this end, we observe that
  \begin{gather*}
    \prod_{k=1}^d p_{i_k}(x_{j_k}) = p_{i_1,\dots,i_d}(x_{j_1},\dots,x_{j_d}).
  \end{gather*}
  Therefore, we conclude that the tensor product node functionals
  resulting from this construction are
  \begin{gather*}
    \nodal_{j_1,\dots,j_d}(p) = p(x_{j_1},\dots,x_{j_d}).
  \end{gather*}
\end{example}

\begin{Example*}{q2}{The space $\Q_2$}
    \begin{center}
    \includegraphics[width=.3\textwidth]{graph/shape0}
    \includegraphics[width=.3\textwidth]{graph/shape1}
    \includegraphics[width=.3\textwidth]{graph/shape2}

    \includegraphics[width=.3\textwidth]{graph/shape3}
    \includegraphics[width=.3\textwidth]{graph/shape4}
    \includegraphics[width=.3\textwidth]{graph/shape5}

    \includegraphics[width=.3\textwidth]{graph/shape6}
    \includegraphics[width=.3\textwidth]{graph/shape7}
    \includegraphics[width=.3\textwidth]{graph/shape8}
  \end{center}
\end{Example*}

\begin{Lemma}{tensor-product-trace}
  The trace of the $d$-dimensional tensor product polynomial space
  $\Q_k$ on the $\delta$-dimensional facets of the reference cube
  $\refcell = (0,1)^d$ is the $\delta$-dimensional space $\Q_k$.

  The traces from two cells sharing the same face coincide, if the
  mapping is continuous. Therefore, continuity can be achieved by
  unisolvent sets of node functionals on the face.
\end{Lemma}

\begin{proof}
  By keeping $d-\delta$ variables constant in the tensor product basis
  in~\eqref{eq:fem-intro:1}.
\end{proof}

\begin{Example*}{cg-q2}{Continuous basis functions}
  \begin{center}
    \includegraphics[height=.20\textwidth]{graph/cgbasis1-02}
    \includegraphics[height=.20\textwidth]{graph/cgbasis1-03}
    \includegraphics[height=.20\textwidth]{graph/cgbasis1-15}
    \includegraphics[height=.20\textwidth]{graph/cgbasis1-16}

    \includegraphics[height=.20\textwidth]{graph/cgbasis1-07}
    \includegraphics[height=.20\textwidth]{graph/cgbasis1-13}
    \includegraphics[height=.20\textwidth]{graph/cgbasis1-18}
    \includegraphics[height=.20\textwidth]{graph/cgbasis1-22}

    \includegraphics[height=.20\textwidth]{graph/cgbasis1-17}
    \includegraphics[height=.20\textwidth]{graph/cgbasis1-23}
    \includegraphics[height=.20\textwidth]{graph/cgbasis1-20}
    \includegraphics[height=.20\textwidth]{graph/cgbasis1-24}
  \end{center}
\end{Example*}

\begin{Example*}{dg-q2}{Discontinuous basis functions}
  \begin{center}
    \includegraphics[height=.20\textwidth]{graph/dgbasis1-08}
    \includegraphics[height=.20\textwidth]{graph/dgbasis1-15}
    \includegraphics[height=.20\textwidth]{graph/dgbasis1-20}
    \includegraphics[height=.20\textwidth]{graph/dgbasis1-27}

    \includegraphics[height=.20\textwidth]{graph/dgbasis1-07}
    \includegraphics[height=.20\textwidth]{graph/dgbasis1-19}
    \includegraphics[height=.20\textwidth]{graph/dgbasis1-23}
    \includegraphics[height=.20\textwidth]{graph/dgbasis1-30}

    \includegraphics[height=.20\textwidth]{graph/dgbasis1-26}
    \includegraphics[height=.20\textwidth]{graph/dgbasis1-33}
    \includegraphics[height=.20\textwidth]{graph/dgbasis1-24}
    \includegraphics[height=.20\textwidth]{graph/dgbasis1-22}
  \end{center}
\end{Example*}

\begin{example}
  As a second example, we choose $d=2$ and the univariate space $\P_2$ with node functionals
  \begin{gather}
    \nodal_0(p) = p(0),
    \quad \nodal_1(p) = \int_0^1 p(t) \dt,
    \quad \nodal_2(p) = p(1),
  \end{gather}
  that is, a mixture of Lagrange interpolation and orthogonality on
  the interval $[0,1]$. The matching basis polynomials are
  \begin{gather}
    p_0(t) = 3(1-t)^2 - 2(1-t),
    \quad p_1(t) = 6 t(1-t),
    \quad p_2(t) = 3t^2-2t.
  \end{gather}
  Follwoing the construction of the previous example, we obtain
  \begin{gather}
    \begin{aligned}
    \nodal_{00}(p) &= p(0,0),
    &\nodal_{02}(p) &= p(0,1),\\
    \nodal_{20}(p) &= p(1,0),
    &\nodal_{22}(p) &= p(1,1).
    \end{aligned}
  \end{gather}
  Then,
  \begin{gather}
    \nodal_{01}(p_{01}) = \nodal_{01}(p_0\otimes p_1)
    = p_0(0)\int_0^1 p_1(y)\dy
    = \int_0^1 p(0,y) \dy.
  \end{gather}
  Thus, the node functional $\nodal_{01}$ is the integral over the
  left edge of the reference square. By the same construction,
  $\nodal_{01}$ is the integral over the right edge. $\nodal_{10}$ and
  $\nodal_{12}$ are the integrals over the bottom and top edge,
  respectively. Finally,
  \begin{gather}
    \nodal_{11}(p_{11})
    = \int_0^1 p_1(x)\dx \int_0^1 p_1(y) \dy
    = \int_0^1\int_0^1 p_{11}(x,y) \dx\dy.
  \end{gather}
  Thus, the tensor product of two line integrals becomes the integral
  over the area.
\end{example}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{The Galerkin equations and Céa's lemma}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{Definition*}{galerkin-approximation}{Galerkin approximation}
  Let $u\in V$ be determined by the weak formulation
  \begin{gather*}
    a(u,v) = f(v) \qquad\forall v\in V,
  \end{gather*}
  where $V$ is a suitable function space including boundary
  conditions. The \define{Galerkin approximation}, also called
  \define{conforming approximation} of this problem reads as follows:
  choose a subspace $V_n\subset V$ of dimension $n$ and find
  $u_n\in V_n$, such that
  \begin{gather*}
    a(u_n,v_n) = f(v_n) \qquad\forall v_n\in V_n.
  \end{gather*}
  We will refer to this equation as the \define{discrete problem}.
\end{Definition*}

\begin{Corollary*}{galerkin-equations}{Galerkin equations}
  After choosing a basis $\{v_i\}$ for $V_n$, the Galerkin equations are
  equivalent to a linear system
  \begin{gather}
    \mata \vu = \vf,
  \end{gather}
  with $\mata\in\R^{n\times n}$ and $\vf\in \R^n$ defined by
  \begin{gather}
    a_{ij} = a(v_j, v_i), \qquad f_i = f(v_i).
  \end{gather}
\end{Corollary*}

\begin{Lemma}{discrete-lax-milgram}
  If the lemma of Lax-Milgram holds for $a(.,.)$ on $V$, it holds on
  $V_n\subset V$. In particular, solvability of the Galerkin equations
  is implied.
\end{Lemma}

\begin{Lemma*}{cea}{Céa}
  Let $a(.,.)$ be a bounded and elliptic bilinear form on the Hilbert
  space $V$.  Let $u \in V$ and $u_n\in V_n \subset V$ be
  the solution to the weak formulation and its Galerkin approximation
  \begin{gather*}
    \begin{aligned}
      a(u,v) &= f(v) & \qquad\forall v&\in V,\\
      a(u_n,v_n) &= f(v_n) & \qquad\forall v_n&\in V_n,
    \end{aligned}
  \end{gather*}
  respectively. Then, there holds
  \begin{gather}
    \norm{u-u_h}_V \le \frac{M}{\alpha}
    \inf_{v_n\in V_n}\norm{u-v_h}_V.
  \end{gather}
\end{Lemma*}

\begin{Lemma}{fe-matrix}
  For a finite element discretization of Poisson's equation with the
  space $V_\mesh$, the Galerkin equations can be computed using the
  following formulas:
  \begin{alignat*}3
    a_{ij} &= \int\limits_\domain \nabla v_j \cdot \nabla v_i \dx
    &&= \int\limits_{\domain(\nodal^i)} \nabla v_j \cdot \nabla v_i \dx
    &&= \sum_{\cell\in\mesh(\nodal^i)}\int\limits_\cell \nabla v_j \cdot \nabla v_i \dx\\
    f_{i} &= \int\limits_\domain f v_i \dx
    &&= \int\limits_{\domain(\nodal^i)} f v_i \dx
    &&= \sum_{\cell\in\mesh(\nodal^i)}\int\limits_\cell f v_i \dx
  \end{alignat*}
\end{Lemma}

\begin{Algorithm*}{matrix-assembling}{Assembling the matrix}
  \begin{enumerate}
  \item Start with a matrix $\mata = 0 \in \R^{n\times n}$
  \item Loop over all cells $\cell\in\mesh$
  \item On each cell $\cell$, compute a cell matrix
    $\mata_\cell \in \R^{n_\cell\times n_\cell}$ by integrating
    \begin{gather}
      a_{\cell,ij} = \int_\cell \nabla p_{\cell,j}\cdot\nabla p_{\cell,i}\,dx,
    \end{gather}
    where $\{p_{\cell,i}\}$ is the shape function basis.
  \item Assemble the cell matrices into the global matrix by
    \begin{gather}
      a_{\iota(i),\iota(j)} = a_{\iota(i),\iota(j)} + a_{\cell,ij}
      \qquad i,j = 1,\dots,n_\cell.
    \end{gather}
  \end{enumerate}
\end{Algorithm*}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Mapped finite elements}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{Definition}{mapped-mesh}
  A mapped mesh $\mesh$ is a set of cells $\cell$, which are defined
  by a single \define{reference cell} $\refcell$ and individual
  smooth mappings
  \begin{gather}
    \begin{split}
      \Phi_\cell \colon \refcell &\to \R^d\\
      \Phi_\cell(\refcell) &= \cell.
    \end{split}
  \end{gather}
  The definition extends to small sets of reference cells, for
  instance for triangles and quadrilaterals.
\end{Definition}

\begin{Example}{mapping-linear}
  Let the reference triangle $\refcell$ be defined by
  \begin{gather}
    \refcell = \left\{
      \begin{pmatrix}
        \refx\\\refy
      \end{pmatrix}
      \middle|
      \refx,\refy >0, \refx+\refy < 1
    \right\}.
  \end{gather}
  Then, every cell $\cell$ spanned by the vertices $\vertex_0$,
  $\vertex_1$, and $\vertex_2$ is obtained by mapping $\refcell$ by
  the \putindex{affine mapping}
  \begin{gather}
    \Phi_\cell(\refvx) =
    \begin{pmatrix}
      X_1-X_0 & X_2 - X_0 \\ Y_1-Y_0 & Y_2 - Y_0
    \end{pmatrix}
    \begin{pmatrix}
      \refx \\ \refy
    \end{pmatrix}
    +
    \begin{pmatrix}
      X_0 \\ Y_0
    \end{pmatrix} =: \matb_\cell \refvx + \vb_\cell
  \end{gather}
\end{Example}

\begin{Example}{mapping-bilinear}
  The reference cell for a quadrilateral is the reference square
  $\refcell = (0,1)^2$. Every quadrilateral $\cell$ spanned by the
  vertices $\vertex_0$ to $\vertex_3$ is then obtained by the
  \putindex{bilinear mapping}
  \begin{gather}
    \Phi_\cell(\refvx)
    = \vertex_0 (1-\refx)(1-\refy)
    + \vertex_1 \refx(1-\refy)
    + \vertex_2 (1-\refx)\refy
    + \vertex_3 \refx\refy
  \end{gather}
\end{Example}

\begin{Definition}{mapped-fe}
  Mapped shape functions $\{p_i\}$ on a mesh cell $\cell$ are defined by a
  set of shape functions $\{\refp_i\}$ on the reference cell
  $\refcell$ through \define{pull-back}
  \begin{gather}
    \begin{split}
      p_i(\vx) &= \refp_i\left(\Phi^{-1}(\vx)\right) = \refp_i(\refvx),\\
      \nabla p_i(\vx) &= \nabla\Phi^{-T}(\refvx)\refgrad\refp_i(\refvx)
    \end{split}
  \end{gather}
\end{Definition}

\begin{Lemma}{mapped-norms-affine}
  Let $\refcell$ be the reference triangle and let $\cell$ be a
  triangular mesh cell with mapping
  $\vx = \Phi_\cell(\refvx) = \matb \refvx + \vb$. Let there hold
  $u(\vx) = \refu(\refvx)$. Then, $u\in H^k(\cell)$ if and only if
  $\refu\in H^k(\refcell)$ and we have with some constant $c$ the
  estimates
  \begin{gather}
    \begin{split}
      \snorm{\refu}_{k,\refcell}
      &\le c \norm{\matb}^k (\det \matb)^{-\nicefrac12}
      \snorm{u}_{k,\cell},\\
      \snorm{u}_{k,\cell}
      &\le c \norm{\matb^{-1}}^k (\det \matb)^{\nicefrac12}
      \snorm{\refu}_{k,\refcell}.
    \end{split}
  \end{gather}
\end{Lemma}

\begin{Lemma}{shape-regular-transformation}
  For a cell $\cell$, let $R$ be the radius of the circumscribed
  circle and $\rho$ the radius of the inscribed circle. Then,
  \begin{gather}
    \norm{\matb} \le c R, \qquad \norm{\matb^{-1}} \le c \rho^{-1}.
  \end{gather}
\end{Lemma}

\begin{proof}
  \begin{figure}
    \centering
    \begin{tikzpicture}
    \def\scale{3}

    %% reference cell
    \def\mid{\scale*0.2928932188}
    \def\outermid{\scale*0.5}
    \def\outerrad{\scale*0.7071067811}

    \node (mid) at (\mid,\mid) {$\refvx_0$};
    \coordinate (x1) at (\scale*1,0);
    \node[right = \scale*0.01 of x1] (x1name) {$\refvx_1$};
    \coordinate (x2) at (0,\scale*1);
    \node[above = \scale*0.01 of x2] (x2name) {$\refvx_2$};
    \coordinate (x3) at (0,0);
    \node[left = \scale*0.01 of x3] (x3name) {$\refvx_3$};
    \coordinate (xc) at (\outermid,\outermid);
    \node[above = \scale*0.01 of xc] (xcname) {$\refvx_c$};

    \draw (x1)--(x2)--(x3)--(x1);

    % incircle
    \draw (mid) circle (\mid);
    \draw (mid) --  (\mid,0);
    \node[below = \scale*0.01 of mid,xshift=0.5em] (rho) {$\refrho$};

    % circumcircle
    \draw (xc) circle (\outerrad);
    \draw (xc) -- (\outermid+\outerrad,\outermid);
    \node[above right = \scale*0.01 and \scale*0.25 of xc] (R) {$\reference{R}$};
    \end{tikzpicture}
    \caption{Visualization of the inscribed and circumscribed circle of the reference cell $\refcell$.}
  \end{figure}

  Let us define $S_{\refrho}\coloneqq \{\refvy\in\R^d ~\mid~ \abs{\refvy}=\refrho \}$.
  For $\refvy\in S_{\refrho}$ there holds
  $\refvx\coloneqq \refvxo + \refvy\in \overline{\refcell}$
  and $\vx=\matb\refvx + \vb = \matb(\refvxo+\refvy)+\vb$.
  Additionally, we have $\vxo = \matb \refvxo + \vb$
  which yields
  \begin{align*}
    \abs{\vx-\vxo} = \abs{\matb\refvy}\leq 2R.
  \end{align*}
  Now consider the operator norm of $\matb$
  \begin{align*}
    \norm{\matb} = \sup_{\refvy\in\R^d} \frac{\abs{\matb\refvy}}{\abs{\refvy}}
    = \sup_{\abs{\refvy}=1} \abs{\matb\refvy}
    = \sup_{\refvy\in S_{\refrho}} \frac{\abs{\matb\refvy}}{\refrho}
    \leq \frac{2R}{\refrho}.
  \end{align*}
  $\refrho$ is a constant only depending on $\refcell$ but
  not on $\cell$. Hence, we get $\norm{\matb}\leq cR$ with $c$ depending
  on $\refrho$.

  The same argument can be applied for the second estimate.
  We now define $S_\rho\coloneqq \{\vy\in\R^d~\mid~\abs{\vy}=\rho \}$.
  For $\vy\in S_\rho$ there holds $\vx\coloneqq\vxo+\vy\in\overline{\cell}$
  and let $\refvx=\matb^{-1}\vx-\vb,~\refvxo=\matb^{-1} \vxo - \vb$ which yields
  \begin{align*}
    \abs{\refvx - \refvxo} = \abs{\matb^{-1} \vy}\leq 2\reference{R}.
  \end{align*}
  Analogous to the first estimate we obtain
  \begin{align*}
    \norm{\matb^{-1}} = \sup_{\vy\in S_\rho} \frac{\abs{\matb^{-1} \vy}}{\rho}
    \leq \frac{2\reference{R}}{\rho}.
  \end{align*}
  Now $\reference{R}$ is only depending on $\refcell$ but not on $\cell$.
  Hence, we have $\norm{\matb^{-1}}\leq c\rho^{-1}$ where $c$ depends
  on $\reference{R}$.
\end{proof}

\begin{Assumption}{mapping-decomposition}
  For more general mappings $\Phi\colon \refcell\to \cell$, we
  make the assumption, that they can be decomposed into three factors,
  \begin{gather}
    \Phi = \Phi_O \circ \Phi_S \circ \Phi_W,
  \end{gather}
  where $\Phi_O$ is a combination of translation and rotation,
  $\Phi_S$ is a scaling with a characteristic length $h_T$, and
  $\Phi_W$ is a warping function not changing the characteristic length.
\end{Assumption}

\begin{example}
  We construct the inverse of $\Phi$ in two dimensions by the
  following three steps, using as $h_\cell$ the length of the longest
  edge of $\cell$.
  \begin{enumerate}
  \item Choose $\Phi_O$ as the rigid body movement which maps the
    longest edge to the interval $(0,h_\cell)$ on the $x$-axis and the
    cell itself to $\cell_O$ in the positive half plane. This mapping
    has the structure
    \begin{gather*}
      \Phi^{-1}_O (\vx) = \mats \vx - \mats \vertex_0,
    \end{gather*}
    where $\mats$ is an orthogonal matrix and $\vertex_0$ is the
    vertex moved to the origin.
  \item Choose the scaling
    \begin{gather*}
      \Phi^{-1}_S (\vx) = \tfrac1{h_\cell} \vx,
    \end{gather*}
    such that the longest edge of the resulting cell $\cell_S$ has the
    longest edge equal to the interval $(0,1)$ on the $x$-axis.
  \item Warp the cell $\cell_S$ into the reference cell $\refcell$ by
    the mapping $\Phi^{-1}_W$. This operation leaves the longest edge
    untouched. For triangles, it is the uniquely defined linear
    transformation mapping the vertex not on the longest edge to
    $(0,1)$. For quadrilaterals, it is a bilinear transformation.
  \end{enumerate}

  In the first step, we have assumed that the cell is convex, which is
  always true for triangles. For nonconvex quadrilaterals, it can be
  shown that the determinant of $\nabla\Phi$ changes sign inside the
  cell, such that these cells are not useful for computations.

  The idea of this decomposition is, that we separate mappings
  changing the position, size, and shape of the cells.
\end{example}

\begin{Lemma*}{scaling-1}{Scaling lemma}
  Let the typical length of a cell $\cell$ be $h_\cell$. Assume there
  are constants $0 < M_\cell, m_\cell, d_\cell, D_\cell$, such that
  \begin{gather}
    \begin{split}
      \norm{\nabla\Phi_W(\refvx)} \le M_\cell,
      \\
      \norm{\nabla\Phi_W^{-1}(\refvx)} \le m_\cell^{-1} ,
      \\
      d^2_\cell \le \det \nabla\Phi_W(\refx)) \le D^2_\cell.
    \end{split}
  \end{gather}
  for all $\refvx\in\refcell$. Then, for $k=0,1$ and a constant $c$
  \begin{gather}
    \begin{split}
      \snorm{\refu}_{k,\refcell}
      &\le c \frac{M_\cell}{d_\cell}  h_\cell^{k-\nicefrac d2}
      \snorm{u}_{k,\cell},\\
      \snorm{u}_{k,\cell}
      &\le c \frac{D_\cell}{m_\cell} h_\cell^{\nicefrac d2-k}
      \snorm{\refu}_{k,\refcell}.
    \end{split}
  \end{gather}
  This extends to higher derivatives under assumptions on higher
  derivatives of $\Phi_\cell$.
\end{Lemma*}

\begin{proof}
  By the chain rule,
  $\nabla \Phi_T = \nabla \Phi_O \nabla \Phi_S \nabla \Phi_W$. By
  construction, $\nabla\Phi_O$ is an orthogonal matrix, such that
  \begin{gather*}
    \norm{\nabla\Phi_O} = \norm{\nabla\Phi_O^{-1}} = 1.
  \end{gather*}
  Since it preserves angles and lengths, $\det \nabla\Phi_O =
  1$. Since $\Phi_S$ is a multiple of the identity, we have
  \begin{gather*}
    \norm{\nabla\Phi_S} = h_\cell,
    \quad\norm{\nabla\Phi_S^{-1}} = \frac1{h_\cell},
    \quad\det\nabla\Phi_S = h_\cell^d.
  \end{gather*}
  By change of variables, we have
  \begin{gather*}
    \int_\cell u^2 \dvx
    = \int_{\refcell} \refu^2 \abs{\det\nabla\Phi_\cell} \dvxref
    = \int_{\refcell} \refu^2 \det\nabla\Phi_S\det\nabla\Phi_O\det\nabla\Phi_W \dvxref
    ,
  \end{gather*}
  such that the case $k=0$ is proven immediately by
  \begin{gather*}
    h_T^d d_\cell^2 \int_{\refcell} \refu^2 \dvxref
    \le \int_\cell u^2 \dvx
    \le h_T^d D_\cell^2 \int_{\refcell} \refu^2 \dvxref
  \end{gather*}
  By the chain rule, we have
  \begin{gather*}
    \refgrad\refu(\refvx) = \nabla\Phi^T \nabla u(\vx)
    = \nabla\Phi_W^T \nabla\Phi_S^T \nabla\Phi_O^T \nabla u(\vx),
  \end{gather*}
  such that there holds
  \begin{gather}
    \begin{split}
      \abs{\refgrad\refu(\refvx)}
      &\le \norm{\nabla\Phi_W} h_\cell \abs{\nabla u},\\
      \abs{\nabla u(\vx)}
      &\le \norm{\nabla\Phi_W^{-1}} h_\cell^{-1} \abs{\refgrad \refu}.
    \end{split}
  \end{gather}
\end{proof}

\begin{Remark}{simple-mappings}
  We have $d_\cell = D_\cell$, if and only if the mapping is
  affine. The quotient $M_\cell/m_\cell$ measures how much the shape
  of the mesh cell deviates from the reference cell. For instance, it
  is one for squares.
\end{Remark}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:


\section{Newtonian Dynamics: The Basics}
\subsection{Particles}
\begin{definition}
    A particle is an object that has negligible size but have positive mass $m$ and electric charge $q$.
\end{definition}
Since a particle will have small size, we can describe its position by a simple position vector $\underline{r}(t)\in\mathbb R^3$ relative to the origin.
We often write the vector in terms of its Cartesian components $\underline{r}=x\underline{i}+y\underline{j}+z\underline{k}=(x,y,z)$ where $\underline{i},\underline{j},\underline{k}$ are an orthonormal basis.
The choice of the coordinate system (the origin and the basis) defines a frame of reference.\\
When the particle moves, its position is determined by a curve $\underline{r}(t)$.
The velocity of the particle is naturally its derivative $\underline{u}(t)=\underline{\dot{r}}(t)$.
Geometrically, the velocity will be the tangent to the curve (or trajectory) at time $t$.
The momentum as we know would be $\underline{p}=m\underline{u}=m\underline{\dot{r}}$.
The acceleration is defined as $\underline{a}=\underline{\dot{u}}=\underline{\ddot{r}}$.
\begin{note}
    The time derivative of a vector valued function $\underline{v}(t)$ is
    $$\underline{\dot{v}}(t)=\lim_{h\to0}\frac{\underline{v}(t+h)-\underline{v}(t)}{h}$$
    provided its existence.
    If anyone is worried, $\underline{v}\to\underline{v_0}\iff\|\underline{v}-\underline{v_0}\|\to 0$.\\
    In particular, if $\underline{v}=x\underline{i}+y\underline{j}+z\underline{k}$, then $\underline{\dot{v}}=\dot{x}\underline{i}+\dot{y}\underline{j}+\dot{z}\underline{k}$ (given that the frame of reference is invariance in time).
\end{note}
\begin{proposition}
    For scalar functions $f(t)$ and vector functions $\underline{g}(t),\underline{h}(t)$, we have\\
    1. $(f\underline{g})^\prime=f^\prime \underline{g}+f\underline{g}^\prime$.\\
    2. $(\underline{g}\cdot\underline{h})^\prime=\underline{g}^\prime\cdot\underline{h}+\underline{g}\cdot\underline{h}^\prime$.\\
    3. $(\underline{g}\times\underline{h})^\prime=\underline{g}^\prime\times\underline{h}+\underline{g}\times\underline{h}^\prime$.\\
    Note that sometimes the order matters.
\end{proposition}
\subsection{Newton's Laws of Motion}
\begin{law}[Newton's First Law]
    There exists inertial frames of reference (or inertial frames).
    That is, a particle at rest or move in constant velocity continues to do so given that it is acted by no force.
\end{law}
\begin{law}[Newton's Second Law]
    In an inertial frame, then the motion obeys the rule $\underline{\dot{p}}=\underline{F}$.
\end{law}
\begin{law}[Newton's Third Law]
    To every action there is an equal and opposite reaction.
\end{law}
The statements, albeit are made for particles, can be extended to finite bodies.
\footnote{Bounded bodies.}
\subsection{Inertial Frames and Galileo Transformation}
If we have an inertial frame, $\ddot{r}=0$ if there is no force acting on it.
There is obviously not only one inertial frame.
In particular, if $S$ is an inertial frame, then a frame $S'$ moving with uniform velocity relative to $S$ is also an inertial frame.
For example, if the frame $S'$ is moving with velocity $v$ on the $x$ direction, then
$$\begin{cases}
    x'=x-vt\\
    y'=y\\
    z'=z\\
    t'=t
\end{cases}$$
More generally, if $S'$ is moving with vector velocity $\underline{v}$ relative to $S$, we have
$$\begin{cases}
    \underline{r'}=\underline{r}-\underline{v}t\\
    t'=t
\end{cases}$$
This transformation is called a \textit{boost}.
For a partical having position vector $\underline{r}(t)$ in $S$ and $\underline{r'}(t')$ in $S'$.
So we have the velocity $\underline{u'}=\underline{u}-\underline{v}$ (note that the primes are NOT used for derivatives here) and $\underline{a'}=\underline{a}$.
\begin{definition}
    A general Galileo transformation is one which preserves inertial frames.
    It combines a boost with any of the following:\\
    1. Translation of space: $\underline{r'}=\underline{r}-\underline{r_0}$.\\
    2. Translation of time: $t'=t-t_0$.\\
    3. Rotations and reflections: $\underline{r'}=R\underline{r},R\in\operatorname{O}(3)$.\\
    This set generates the Galilean group of transformations.
\end{definition}
Note that if the acceleration is zero in one frame, so it is in another.
\begin{definition}[Principle of Galilean Relativity]
    The laws of (Newtonian) physics is unchanged in all inertial frames.
\end{definition}
That is, the laws of physics look the same in every inertial frame.
Hence the system of Newtonian physics has to be invariant under the Galilean transformations.
\subsection{Newton's Second Law}
The law postulates that $\underline{F}=\underline{\dot{p}}$.
Assume that $m$ is constant in time, then we have $\underline{F}=m\underline{\ddot{r}}$.
Easily $m$ is the measure of ``reluctance to accelerate'', that is inertia.
If we specify $\underline{F}$ as a function of $\underline{r},\underline{\dot{r}},t$, then we have a second order ODE in $\underline{r}$:
$$\underline{F}(\underline{r},\underline{\dot{r}},t)=m\underline{\ddot{r}}$$
We then need two initial conditions to solve the equation (or to determine the motion).
For example, we can specify the initial position and velocity.
With these information
\footnote{And perhaps Picard-Lindel\"of Theorem}
we can get an unique solution for the trajectory of our particle.
\subsection{Examples of Forces}
Consider $2$ particles indexed by $1,2$, then Newton tells us
\begin{law}[Newton's Law of Gravitation]
    There is an action-reaction pair on the two particles, namely
    $$\underline{F_1}=-\frac{Gm_1m_2}{|\underline{r_1}-\underline{r_2}|^3}(\underline{r_1}-\underline{r_2})=-F_2$$
\end{law}
In particular, $|\underline{F_1}|=|\underline{F_2}|\propto|\underline{r_1}-\underline{r_2}|^{-2}$.
This is known as the inverse square law.
It is quite obvious that $G$ has an unit.
It is called Newton's Gravitation Constant.\\
Another example is electromagnetic forces.
Let there be a particle with electric charge $q$ and imagine that it is moving in an electric-magnetic field $\underline{E}(\underline{r},t)$ and $\underline{B}(\underline{r},t)$.
\begin{law}[Lorentz Force Law]
    We have
    $$\underline{F}=q(\underline{E}+\underline{\dot{r}}\times\underline{B})$$
\end{law}
\begin{example}
    Take $\underline{E}=\underline{0},\underline{B}=\underline{B}(t)$, i.e. the electric field is constant and the magnetic field is constant in space.
    Hence
    $$m\underline{\ddot{r}}=q\underline{\dot{r}}\times\underline{B}(t)$$
    Choose axes such that $\underline{B}=B\underline{\hat{z}}$, then $m\ddot{z}=0\implies z=z_0+ut$.
    As for the other directions, we have
    $$\begin{cases}
        m\ddot{x}=qB\dot{y}\\
        m\ddot{y}=-qB\dot{x}
    \end{cases}$$
    which we can easily solve to get
    $$\begin{cases}
        x=x_0-\alpha\cos(\omega(t-t_0))\\
        y=y_0+\alpha\sin(\omega(t-t_0))
    \end{cases}$$
    which shall produce a helical path which is clockwise when viewed from the direction of $\underline{B}$.
    And the axis of the helix is parallel to the magnetic field.
\end{example}