\section{MgNet: a new network structure}\label{sec:mgnet}
In this section, we introduce a new neural network structure,
named as MgNet, motivated by the multigrid algorithm,
Algorithm \ref{alg:L-Slash0}, as discussed in the previous section.

%The main structure of both CNN and multigrid can be
%understood as:
%\begin{itemize}
%	\item Solving an undetermined system in grid $\ell$.
%	\item Connection of different grids by restriction and prolongation.
%\end{itemize}

First, given the data-feature equation \eqref{Auf}, we consider
its restrictions to grid $\ell$ as follows:
\begin{equation}
\label{Auf-ell}
A^\ell(u^\ell) = f^\ell, \quad \ell=1:J,
\end{equation}
where
\begin{equation}
\label{f-ell}
f^{\ell}\in\mathbb R^{m_\ell\times n_\ell\times c_\ell},
\end{equation}
and
\begin{equation}
\label{u-ell}
u^{\ell}\in\mathbb R^{m_\ell\times n_\ell\times h_\ell}.
\end{equation}
We are now in a position to state the main algorithm, namely
MgNet as:
\begin{breakablealgorithm}
	\caption{$u^J={\rm MgNet}(f; J,\nu_1, \cdots, \nu_J)$}
	\label{alg:mgnet}
	\begin{algorithmic}
		\State Initialization:  $f^1 =\theta(f)$, $u^{1,0}=0$
		%		\State Initialization $u^{1,0}$
		\For{$\ell = 1:J$}
		\For{$i = 1:\nu_\ell$}
		\State Feature extraction (smoothing):
		\begin{equation}\label{mgnet}
		u^{\ell,i} = u^{\ell,i-1} + B^{\ell,i}  ({f^\ell -  A^{\ell} (u^{\ell,i-1})}).
		\end{equation}
		\EndFor
		\State Note:
		$
		u^\ell= u^{\ell,\nu_\ell}
		$
		\State Interpolation and restriction:
		\begin{equation}
		\label{interpolation}
		u^{\ell+1,0} = \Pi_\ell^{\ell+1}u^{\ell}
		\end{equation}
		\begin{equation}
		\label{restrict-f}
		f^{\ell+1} = R^{\ell+1}_\ell(f^\ell - A^\ell(u^{\ell})) + A^{\ell+1} (u^{\ell+1,0}).
		\end{equation}
		\EndFor
	\end{algorithmic}
\end{breakablealgorithm}

The first property of MgNet is that it recovers the multigrid methods.
\begin{theorem}
	If $A^\ell$, $R_\ell^{\ell+1}$ and $B^{\ell,i} = S^{\ell}$ are all linear operations as described in multigrid method
	in \S \ref{sec:mg}.
	Then Algorithm \ref{alg:L-Slash0} is equivalent to Algorithm \ref{alg:mgnet} with any choice of $\Pi_\ell^{\ell+1}$.
\end{theorem}
\begin{proof}
	Here we replace $u^{\ell,i}$ and $f^{\ell}$ by $\tilde u^{\ell,i}$ and $\tilde f^{\ell}$ in MgNet.
	What we want to prove are
	\begin{equation}\label{eq:f-u}
	\tilde f^{\ell} =  f^{\ell} + A_\ell \tilde u^{\ell,0} \quad \text{and} \quad u^{\ell,i} = \tilde u^{\ell, i} - \tilde u^{\ell, 0},
	\end{equation}
	with $u^{\ell,i}$, $f^\ell$ in Algorithm \ref{alg:L-Slash0} and
	$\tilde u^{\ell,i}$, $\tilde f^\ell$ in Algorithm \ref{alg:mgnet} for any choice of $\Pi_\ell^{\ell+1}$.
	We prove this result by induction.
	\begin{itemize}
		\item It is easy to check that $\ell = 1$ is right by taking $\theta  = \rm{id}$.
		\item Once the above equation \eqref{eq:f-u} is right for $\ell$,
		let us prove the corresponded result for $\ell+1$.
		\begin{itemize}
			\item For $\tilde f^{\ell+1}$, as the definition in Algorithm \ref{alg:mgnet}, we have
			\begin{align*}
			\tilde f^{\ell+1} &= R_\ell^{\ell+1}(\tilde f^\ell - A^\ell \tilde u^{\ell,\nu_\ell}) + A^{\ell+1}\tilde u^{\ell+1,0}, \\
			&= R_\ell^{\ell+1}(\tilde f^\ell + A^{\ell} \tilde u^{\ell,0}- A^{\ell} \tilde u^{\ell,\nu_\ell}) + A^{\ell+1}\tilde u^{\ell+1,0} \\
			&= R_\ell^{\ell+1}(\tilde f^\ell - A^{\ell} (\tilde u^{\ell,\nu_\ell}- u^{\ell,0})) + A^{\ell+1}\tilde u^{\ell+1,0} \\
			&= R_\ell^{\ell+1}(f^\ell - A^{\ell} u^{\ell,\nu_\ell}) + A^{\ell+1}\tilde u^{\ell+1,0}, \\
			&=  f^{\ell+1} + A^{\ell+1}\tilde u^{\ell+1,0}.
			\end{align*}
			\item For $u^{\ell+1,i}$, first we have
			$$
			u^{\ell+1,0} = 0 =\tilde u^{\ell+1, 0} - \tilde u^{\ell+1, 0},
			$$
			then we prove
			\begin{equation}\label{u:i+1}
			u^{\ell+1,i} = \tilde u^{\ell+1, i} - \tilde u^{\ell+1, 0}
			\end{equation} by induction for $i$.

			We assume \eqref{u:i+1} holds for $0,1,\cdots,i-1$. Let us miner $\tilde u^{\ell+1, 0}$ in both sides of
			the smoothing process \eqref{mgnet} in Algorithm \ref{alg:mgnet}. Then we have
			\begin{align*}
			\tilde u^{\ell+1,i} - \tilde u^{\ell+1, 0} &= \tilde u^{\ell+1,i-1} - \tilde u^{\ell+1, 0} + B^{\ell+1,i} (\tilde f^{\ell+1} - A^{\ell+1} \tilde u^{\ell+1,i-1}), \\
			&= \tilde u^{\ell+1,i-1} - \tilde u^{\ell+1, 0} + B^{\ell+1,i} (f^{\ell+1} + A^{\ell+1}\tilde u^{\ell+1,0} - A^{\ell+1} \tilde u^{\ell+1,i-1} ),\\
			&= u^{\ell+1,i-1} + B^{\ell+1,i} (f^{\ell+1} - A^{\ell+1}u^{\ell+1,i-1} ).
			\end{align*}
			This is exact the smoothing process in Algorithm \ref{alg:L-Slash0} as we take $ B^{\ell+1,i} = S^{\ell+1}$.
			%			\item At last, recall that $u^{1,0} = 0$, so the output of Algorithm \ref{alg:L-Slash0}
			%			$$
			%			tilde u^{1, \nu_1} = u^{1, \nu_1},
			%			$$
			%			which is the output of Algorithm \ref{alg:L-Slash}.
		\end{itemize}
	\end{itemize}
\end{proof}

Despite of the simplicity look of Algorithm \ref{alg:mgnet}, there
are rich mathematical structures and variants which we briefly discuss below.

\subsection{Initialization: feature space channels}
Initially for $\ell=1$,  we take $m_1 = m$ and $n_1 = n$ and we may define the linear mapping
\begin{equation}
\label{eq:6}
\theta: \mathbb R^{m\times n\times c}
\mapsto \mathbb R^{m_1\times n_1\times c_1},
\end{equation}
to obtain $f^{1} = \theta(f)$ with $c$ given in \eqref{data-c} changed to the channel of the initial
data space to $c_1$.   Usually
\begin{equation}
\label{cc}
c_1\ge c.
\end{equation}
One possibility is that we choose $c_1=c$.  In this case, we choose
$\theta=$identity.   But in general, we may need to choose $c_1\gg
c$. One possible advantage of preprocessing the RGB ($c=3$) to
different color spaces is that we can better choose what kind of
features the CNN can detect, and under what
conditions those detections will be invariant.

One possibility of understanding and modifying this step
is to decompose the data $f$ into a number of more
specialized data
\begin{equation}
\label{decomp-f}
f=\sum_{k=1}^{c_1}\xi_kf^1_k  =\xi^Tf^1.
\end{equation}
We may use some knowledge from image processing or physics to
design a procedure to obtain the right decomposition of
\eqref{decomp-f}, or we can just train it.
Conceivably, we may view $f^{1} = \theta(f)$ as a special approximation solution of
\eqref{decomp-f} with the same sparsity pattern to $\xi$.

\subsection{Extracted Units: $u^{\ell}$ and channels}
The first new feature and the main new ingredient
in the proposed neural network is the introduction
of feature variables  $u^{\ell}$ in \eqref{u-ell}, which will be known
as the extracted units.

We emphasize that the extracted-units $u^\ell$ and the data $f^\ell$ can have
different numbers of channels:\
\begin{equation}
\label{uf-channels}
u^\ell\in \mathbb{R}^{m_\ell\times n_\ell \times c_{u,\ell}}, \quad
f^\ell\in \mathbb{R}^{m_\ell\times n_\ell \times c_{f,\ell} }
\end{equation}
One possibility is that the number of channels for both $u$ and $f$ remain
unchanged in different grids:
\begin{equation}
\label{cfl}
c_{f,\ell}=c_f, \quad \ell=1:J,
\end{equation}
and
\begin{equation}
\label{ufl}
c_{u,\ell}=c_{u}, \quad \ell=1:J.
\end{equation}
Both $c_f$ and $c_{u}$ are two super-parameters that need to be tuned,
and we may even take $c_u = c_f$.

\subsection{Poolings: $\Pi_{\ell+1}^\ell$ and $R_{\ell+1}^\ell$}
The pooling $\Pi_{\ell+1}^\ell$ in \eqref{restriction} and
$R_{\ell+1}^\ell$ in \eqref{restrict-f} are in general different.
They can be trained in general, but they may be a priori chosen.

There are many different possibilities to choose $\Pi_{\ell+1}^\ell$.
The simplest choice of $\Pi_{\ell+1}^\ell$ is
\begin{equation}
\label{eq:8}
\Pi_{\ell+1}^\ell=0.
\end{equation}
A more sophisticated choice can be obtained by considering an
interpolation from fine grid to coarse (that, for example preserves linear function
locally).  Namely
\begin{equation}
\label{Pi}
\Pi_{\ell+1}^\ell=\bar\Pi_{\ell+1}^\ell \otimes I_{c_\ell\times c_\ell}
\end{equation}
with $\bar\Pi_{\ell+1}^\ell$ given by~\eqref{mg-Pi}.

\subsection{Data-feature mapping: $A^{\ell}$}
The second new feature of MgNet is that this data-feature mapping
only depends on the grid ${\cal T}_\ell$, and it does not depend on layers
within the same grid.  This amounts to a significant saving of the number of
parameters.  In comparison, the existing CNN, such as ResNet, can be
interpreted as a network related to the case that $A^{\ell}$ is
replaced by $A^{\ell, i}$, namely
\begin{equation}\label{u-resnet}
u^{\ell,i} = u^{\ell,i-1} + B^{\ell,i}  (f^\ell -  A^{\ell,i} (u^{\ell,i-1}).
\end{equation}

The data-feature mapping: $A^{\ell}$ can be either linear
\eqref{linearA}, or nonlinear \eqref{nonlinearA}.  The underlying
convolution kernels can be different on different grids and they can
all be trained.

\subsection{Feature extractors: $B^{\ell,i}$}
There are some freedoms in choosing these feature extrators.
One common choice of extractors is given by \eqref{extractor}, namely
\begin{equation}
\label{extractor-ell}
B^{\ell,i}=\sigma\circ \eta^{\ell,i}\circ\sigma.
\end{equation}

Other than the level dependent extractors, the following
different strategies can be used
\begin{description}
	\item[Constant Extractors]: $B^{\ell,i}=B^{\ell}$ for   $i=1:\nu_\ell$
	\item[Scaled Extractors]:$B^{\ell,i}=\alpha_iB^{\ell}$ for   $i=1:\nu_\ell$
	\item[Variable Extractors]: $B^{\ell,i}$
\end{description}

This brief framework gives us the basic principle on designing
a CNN models for classification. All models are seen as the special
choice of data-feature mapping $A^\ell$, feature extractors $B^{\ell,i}$
and the pooling operators $\Pi_{\ell+1}^\ell$ with $R_{\ell+1}^\ell$.

\section{Some classic CNN models}\label{sec:CNNs}
In this section, we will use the notation introduced above to
give a brief description of some classic CNN models.

\subsection{LeNet-5, AlexNet and VGG}
The  LeNet-5 \cite{lecun1998gradient}, AlexNet \cite{krizhevsky2012imagenet} and VGG \cite{simonyan2014very}
can be written as:
	\begin{equation}
	\begin{cases}
	f^{1,0} &= \theta^0(f), \\
	f^{\ell,i} &= \theta^{\ell,i} \circ \sigma (f^{\ell, j-1}), \quad i = 1:\nu_\ell ~\text{and}~ \ell = 1:J,\\
	f^{\ell+1,0} &= R_\ell^{\ell+1}( f^{\ell,m+\ell}).  \\
	\end{cases}
	\end{equation}
	where $R_\ell^{\ell+1}$ can be general pooling operators and $\theta^{\ell,i}$ can be convolution with stride 1,
	or fully connected operators.
	Then the CNN model will be defined by
	\begin{equation}\label{eq:cnndefine}
	H_0(f) = f^{L,\nu_\ell}.
	\end{equation}
	In these three classic CNN models, they still need some
	extra fully connected layers in nonlinear mapping $H_0$ before the logistic regression as it contains
	a fully connected layer as in \eqref{eq:log_reg}.
	These fully connected layers are removed in ResNet to be described below.
\subsection{ResNet}
The ResNet \cite{he2016deep} can be written as
	\begin{equation}\label{ori-ResNet}
	\begin{cases}
	f^{1,0} &=R_{\rm max}\circ \sigma \circ \theta^0(f), \\
	f^{\ell,i} &= \sigma \left( f^{\ell, i-1} + \mathcal{F}^{\ell, i} (f^{\ell,i-1}) \right), \quad i = 1:\nu_\ell ~\text{and}~ \ell = 1:J ,\\
	f^{\ell+1,0} &= \sigma \left( R_\ell^{\ell+1} (f^{\ell, \nu_\ell} )+ \mathcal{F}^{\ell, 0} (f^{\ell, \nu_\ell} ) \right), \quad \ell = 1:J-1,\\
	H_0(f) &=  R_{\rm ave}(f^{L,\nu_\ell}). \\
	\end{cases}
	\end{equation}
	Here
	$$
	\mathcal{F}^{\ell,i} (f^{i-1}) = \xi^{i} \circ \sigma \circ \eta^{i} (f^{i-1}).
	$$
	Generally, $\xi^{\ell,i}$ and $\eta^{\ell,i}$ takes the form of \label{eq:conv-1} with zero padding and stride 1,
	except, $\eta^{\ell,0}$  is taken as convolution with stride 2 with the same output dimension of $R_\ell^{\ell+1}$.

\subsection{iResNet}
	The iResNet\cite{he2016identity} can be written as:
	\begin{equation}\label{eq:iResNet1}
	\begin{cases}
	f^{1,0} &=R_{\rm max}\circ \sigma \circ \theta^0(f), \\
	f^{\ell,i} &= f^{\ell, i-1} + \mathcal{F}^{\ell, i} (f^{\ell,i-1}), \quad i = 1:\nu_\ell ~\text{and}~ \ell = 1:J ,\\
	f^{\ell+1,0} &=  R_\ell^{\ell+1} (f^{\ell, \nu_\ell} )+ \mathcal{F}^{\ell, 0} (f^{\ell, \nu_\ell} ) , \quad \ell = 1:J-1,\\
	H_0(f) &=  R_{\rm ave}(f^{L,\nu_\ell}). \\
	\end{cases}
	\end{equation}
	where
	$$
	\mathcal{F}^{\ell,i} (f^{\ell,i -1}) = \xi^{\ell,i} \circ \sigma \circ \eta^{\ell,i} \sigma (f^{\ell,i-1}).
	$$
	The only difference between ResNet and iResNet can be viewed as
	putting a $\sigma$ in different places.
	The connection of those three models are often shown with next diagrams:
	\begin{figure}[!htb]
		\begin{center}
			\includegraphics[width=.6\textwidth, height=.13\textheight]{comparison-net}
		\end{center}
		\caption{Comparison of CNN Structures}
	\end{figure}

Without loss of generality, we extract the key
feedforward steps on the same grid in different CNN models as follows.
\begin{description}
	\item[Classic CNN]
	\begin{equation}\label{eq:cCNN}
	f^{\ell,i} = \xi^i \circ \sigma (f^{\ell,i-1}) \quad \text{or} \quad f^{\ell,i} = \sigma \circ \xi^{i} (f^{\ell,i-1}) .
	\end{equation}
	\item[ResNet]
	\begin{equation}\label{eq:ResNet}
	f^{\ell,i} = \sigma( f^{\ell,i-1} + \xi^{\ell,i} \circ \sigma \circ \eta^{\ell,i}(f^{\ell,i-1})).
	\end{equation}
	\item[iResNet]
	\begin{equation}\label{eq:iResNet}
	f^{\ell,i} = f^{\ell,i-1} + \xi^{\ell,i} \circ \sigma \circ \eta^{\ell,i}\circ \sigma(f^{\ell,i-1}).
	\end{equation}
\end{description}

\section{Variants and generalizations of MgNet}\label{sec:relation}

%\subsection{Some properties of MgNet}
%So, this MgNet is corresponded to the multigrid methods
%with iteration in function space. A natural idea is that there is also a dual version of
%MgNet similar with the multigrid methods with iteration in dual space.
The MgNet model algorithm is one very basic and it can be generalized
in many different ways. It can also be used as a guidance to modify and
extend many existing CNN models.

The following result show how MgNet is related to he iResNet \cite{he2016identity}.
\begin{theorem}\label{thm:mgnet1}
The MgNet model Algorithm \ref{alg:mgnet},
with $A=\xi^\ell$ and $B^{\ell,i}=\sigma \circ \eta^{\ell,i}\circ\sigma$,
admits the following identities
\begin{equation}\label{dualmgnet}
f^{\ell, i} = f^{\ell, i-1} -  \xi^{\ell} \circ \sigma \circ \eta^{\ell,i}\circ \sigma (f^{\ell,i-1}), \quad i = 1:\nu_\ell, \\
\end{equation}
where
\begin{equation}
  \label{eq:5}
	f^{\ell,i} = f^{\ell} - \xi^{\ell} (u^{\ell,i}).
\end{equation}
Furthermore, \eqref{dualmgnet} represents iResNet~\cite{he2016identity}
as shown in \eqref{eq:iResNet}.
\end{theorem}

\begin{proof}
	Because of the linearity of $\xi^\ell$ and invariant within the same grid $\ell$,
	we can apply $\xi^\ell$ on both sides of \eqref{mgnet} and minus with
	$f^\ell$, thus we have
	$$
	f^{\ell} - \xi^{\ell} (u^{\ell,i})= f^{\ell} - \xi^\ell(u^{\ell,i}) -
	\xi^{\ell} \circ \sigma \circ \eta^{\ell,i}\circ \sigma (f^\ell + \xi^\ell(u^{\ell,i})).
	$$
This finish the proof with definition in \eqref{eq:5}.
\end{proof}

The above result is very simple but critically important.
In view of Theorem \ref{thm:mgnet1}, it shows how multigrid and
CNN are intimately related. Furthermore, it provides a different version
of iResNet, which can be viewed as the dual version of the original iResNet.
This relation is quit similar with the dual relation of $u$ and $f$
in multigrid method \cite{xu2017algebraic}.
\begin{lemma}\label{thm:mgnet2}
	The ResNet~\cite{he2016deep} step
 	 as in \eqref{eq:ResNet}
%	\begin{equation}\label{resnet}
%	f^{\ell,i} = \sigma( f^{\ell, i-1} - \xi^{\ell,i} \circ \sigma \circ \eta^{\ell,i} (f^{\ell,i-1}) ).
%	\end{equation}
admits the following relation:
%(which resembles closely with \eqref{dualmgnet})
\begin{equation}\label{tilde-resnet}
\tilde f^{\ell,i} =\sigma(\tilde f^{\ell,i-1}) -
\xi^{\ell,i} \circ \sigma \circ \eta^{\ell,i}\circ \sigma( \tilde f^{\ell,i-1}),
\end{equation}
where
\begin{equation}\label{tilde-f}
\tilde f^{\ell,i} = f^{\ell, i-1} -\xi^{\ell,i} \circ \sigma \circ \eta^{\ell,i} (f^{\ell,i-1}).
\end{equation}
\end{lemma}
\begin{proof}
%	Now, we will establish the connection between classical ResNet and MgNet.
	First, we apply $ \xi^{\ell,i+1} \circ \sigma \circ \eta^{\ell,i+1}$
	on the both sides of \eqref{eq:ResNet} and get
	\begin{equation}\label{resnet1}
	\xi^{\ell,i+1} \circ \sigma \circ \eta^{\ell,i+1}( f^{\ell,i} ) =
	\xi^{\ell,i+1} \circ \sigma \circ \eta^{\ell,i+1}\circ \sigma( \tilde f^{\ell,i} ).
	\end{equation}
	Minus by $f^{\ell,i}$ on the both sides and recall the definition in \eqref{tilde-f}, we have
	\begin{equation*}
	\tilde f^{\ell,i+1} = f^{\ell,i} - \xi^{\ell,i+1} \circ \sigma \circ \eta^{\ell,i+1}\circ \sigma( \tilde f^{\ell,i}).
	\end{equation*}
	By the definition of $f^{\ell,i} = \sigma(\tilde f^{\ell,i})$, we finish this proof.
\end{proof}

We call the above form \eqref{tilde-resnet} as
$\sigma$-ResNet, similar to the MgNet we replace $\xi^{\ell,i}$ by $\xi^{\ell}$  and get
the next Mg-ResNet form as:
\begin{equation}\label{mg-resnet}
f^{\ell,i} =\sigma(f^{\ell,i-1}) -
\xi^{\ell} \circ \sigma \circ \eta^{\ell,i}\circ \sigma(f^{\ell,i-1}).
\end{equation}

If we take these pooling and prolongation operators
as discussed in the previous sections and focus on
the iterative forms on a certain grid $\ell$, we may
compare them all as:
\begin{table}[!htbp]
	\caption{Comparison for all iterative forms }
	\label{comparison-ALL}
	\begin{center}\scriptsize
		\resizebox{.8\textwidth}{!}{
			\begin{tabular}{|c|c|c|}
				\hline
				Primal-Dual & Model & Iterative form \\
				\hline
				\multirow{3}{*}{Feature space} & Abstract-MgNet & Solving $A^\ell(u^\ell) = f^\ell$ \\
				\cline{2-3}
				& General-MgNet & $u^{\ell,i} = u^{\ell, i-1} + B^{\ell,i} (f^\ell - A^{\ell}(u^{\ell,i-1}))$ \\
				\cline{2-3}
				& {MgNet} & $u^{\ell,i} = u^{\ell, i-1} + \sigma \circ \eta^{\ell,i}\circ \sigma (f^\ell - \xi^{\ell}(u^{\ell,i-1}))$ \\
				\hline
					\multirow{5}{*}{Data space} & iResNet & $ f^{\ell,i} = f^{\ell, i-1} -  \xi^{\ell,i} \circ \sigma \circ \eta^{\ell,i} \circ \sigma (f^{\ell,i-1})$ \\
				\cline{2-3}
				& Mg-iResNet & $f^{\ell,i} = f^{\ell, i-1} -  \xi^{\ell} \circ \sigma \circ \eta^{\ell,i}\circ \sigma (f^{\ell,i-1})$ \\
				\cline{2-3}
				& Mg-ResNet & $f^{\ell,i} = \sigma(f^{\ell,i-1}) - \xi^{\ell} \circ \sigma \circ \eta^{\ell,i}\circ \sigma( f^{\ell,i-1})$ \\
				\cline{2-3}
				& $\sigma$-ResNet & $f^{\ell,i} = \sigma(f^{\ell,i-1}) - \xi^{\ell,i} \circ \sigma \circ \eta^{\ell,i}\circ \sigma( f^{\ell,i-1})$ \\
				\cline{2-3}
				& ResNet & $f^{\ell,i} = \sigma(f^{\ell, i-1} -  \xi^{\ell,i} \circ \sigma \circ \eta^{\ell,i} (f^{\ell,i-1}))$ \\
				\hline
			\end{tabular}
		}
	\end{center}
\end{table}

We can have these connections for all iterative scheme in data space:
\begin{equation}
\text{ ResNet} \xleftrightarrow{\eqref{tilde-f}} \sigma\text{-ResNet } \xleftrightarrow{\xi^{\ell,i} \leftrightarrow \xi^{\ell}} \text{Mg-ResNet}
\xleftrightarrow{\sigma(f^{\ell,i-1}) \leftrightarrow f^{\ell, i-1} } \text{Mg-iResNet} \xleftrightarrow{ \xi^{\ell} \leftrightarrow \xi^{\ell,i}} \text{iResNet}.
\end{equation}

%Because of the linearity of $\xi^{\ell,i}$, the above forms of iResNet and ResNet
%are equivalent to the previous as \eqref{dualmgnet} and \eqref{tilde-resnet}.

In this sense, these MgNet related models can be understood as
 models between iResNet and ResNet. And all these models can be
 understood as iteration in the data space as a dual relationship with
 feature space as MgNet.

The rationality of replacing  $\xi^{\ell,i}$ by layer independent $\xi^{\ell}$ may
be justified by the following theorem.
\begin{theorem}\label{thm:CNN}
On each grid $\mathcal T_\ell$,
\begin{enumerate}
	\item Any CNN model with
%	CNN and Mg-ResNet]
	\begin{equation}
	\label{CNN1}
	f^{\ell,i} =   \chi^{\ell,i} \circ \sigma (f^{\ell,i-1}),
	\end{equation}
	can be written as
	\begin{equation}\label{Res-CNN1}
	f^{\ell,i} = \sigma(f^{\ell,i-1}) - \xi^{\ell} \circ \sigma \circ \eta^{\ell,i} \circ\sigma ( f^{\ell,i-1}).
	\end{equation}
	\item Any CNN model with
%	[CNN and ResNet]
	\begin{equation}
	\label{CNN2}
	f^{\ell,i} =   \sigma\circ\chi^{\ell,i} (f^{\ell,i-1}).
	\end{equation}
	can be written as
	\begin{equation}\label{Res-CNN2}
	f^{\ell,i} = \sigma\left(f^{\ell,i-1} - \xi^{\ell} \circ \sigma \circ \eta^{\ell,i}  ( f^{\ell,i-1})\right).
	\end{equation}
\end{enumerate}

%\begin{equation}
% \chi^{\ell,i}: \mathbb{R}^{n_\ell \times n_\ell \times c_\ell}
% \mapsto \mathbb{R}^{n_\ell \times n_\ell \times c_\ell},
%\end{equation}
\end{theorem}
\begin{proof}
%	Without loss of generality, consider the classical CNN with
%	\begin{equation}
%	f^{\ell,i} =   ({\rm id}  + \tilde \eta^{\ell,i} )\circ \sigma (f^{\ell,i-1}).
%	\end{equation}
Let use prove the first case as an example,
the second case can be proven with the same process.

With similar structure in MgNet, we can take
\begin{equation}
\label{xi-cnn1}
\xi^{\ell}=  [\hat \delta_1, \cdots, \hat \delta_{{c_\ell}}],
\end{equation}
and
\begin{equation}
\label{eta-ell}
\eta^{\ell,i} = [{\rm id}_{c_\ell}, -{\rm id}_{c_\ell}] \circ (\chi^{\ell,i} - {\rm id}_{c_\ell}).
\end{equation}
Here
\begin{equation}
{\rm id}_{c_\ell}: \mathbb{R}^{n_\ell \times n_\ell \times c_\ell}
\mapsto \mathbb{R}^{n_\ell \times n_\ell \times c_\ell},
\end{equation}
is the identity map and
\begin{equation}
\hat \delta_k :  \mathbb{R}^{n_\ell \times n_\ell \times 2c_\ell}
\mapsto \mathbb{R}^{n_\ell \times n_\ell},
\end{equation}
with
\begin{equation}\label{eq:hatdelta}
\hat \delta_k([X ,Y]) = -([X]_k + [Y]_k),
\end{equation}
for any $X, Y \in \mathbb{R}^{n_\ell \times n_\ell \times c_\ell}$
and $[X,Y] \in \mathbb{R}^{n_\ell \times n_\ell \times 2c_\ell} $.

	First, we see that $\eta^{\ell,i}$ with the above
	form is a convolution from $\mathbb{R}^{n_\ell \times n_\ell \times c_\ell}$
	to  $\mathbb{R}^{n_\ell \times n_\ell \times 2c_\ell}$.
	Following the identity
	\begin{equation}
	ReLU(x) + ReLU(-x) = x,
	\end{equation}
	and the definition of $\xi^{\ell}$ i.e.
	\begin{equation}
	\xi^{\ell} = \hat \delta,
	\end{equation}
	as a special case in MgNet.
	For more details, we can give a exact form of
	$\hat \delta_k$ as in \eqref{eq:hatdelta} with
	\begin{equation}
	\hat \delta_k = [0, \cdots,0, -\delta, \cdots 0;  0, \cdots,0, -\delta, \cdots 0],  \quad k = 1:{c_\ell},
	\end{equation}
	where $\delta$ is the identity kernel during one channel.

	At last, we have
	\begin{align}
	\left[\xi^{\ell} \circ \sigma \circ [{\rm id}_{c_\ell}, -{\rm id}_{c_\ell}] (x) \right]_k &=  \left[\xi^{\ell} \circ \sigma \circ [x, -x]  \right]_k, \\
	&= \hat \delta_k ( [\sigma(x), \sigma(-x)]),  \\
	&= -\delta([\sigma(x)]_k) - \delta([\sigma(-x)]_k),\\
	&=-( \sigma([x]_k)+ \sigma(-[x]_k)) , \\
	&=  -[x]_k
	\end{align}
	Thus to say,
	\begin{equation}
	\xi^{\ell} \circ \sigma \circ [{\rm id}_{c_\ell}, -{\rm id}_{c_\ell}]  = -{\rm id}_{c_\ell}.
	\end{equation}
	Then the modified dual form of MgNet in \eqref{tilde-resnet} becomes
	\begin{align}
	f^{\ell,i} &= \sigma(f^{\ell,i-1}) - \xi^{\ell,i} \circ \sigma \circ \eta^{\ell,i} \circ\sigma ( f^{\ell,i-1}) , \\
	&=  \sigma(f^{\ell,i-1}) - \left( \xi^{\ell} \circ \sigma \circ [{\rm id}_{c_\ell}, -{\rm id}_{c_\ell}] \right)
	\circ (\chi^{\ell,i} - {\rm id}_{c_\ell})\circ \sigma(f^{\ell,i-1})\\
	&=\sigma(f^{\ell,i-1}) + (\chi^{\ell,i} -{\rm id}_{c_\ell})\circ \sigma(f^{\ell,i-1}),  \\
	&=\chi^{\ell,i} \circ  \sigma (f^{\ell,i-1}).
	\end{align}
	This covers \eqref{Res-CNN1}.
\end{proof}

\begin{remark}
Theorems~\ref{thm:CNN} shows that general CNN in
the forms of either \eqref{CNN1} or \eqref{CNN2} can be written recast
as \eqref{Res-CNN1} or \eqref{Res-CNN2} with the data-feature mapping
$A^\ell=\xi^\ell$ that is not only independent of the layers, but is
actually given a priori as in \eqref{xi-cnn1}.  In
view of Theorems~\ref{thm:mgnet1} and \ref{thm:mgnet2}, the classic
CNN models can be essentially recovered from MgNet by choosing
$\xi^\ell$ a priori as in  \eqref{xi-cnn1}.  Since
the classic CNN models have been extensively tested to be successful,
the more general MgNet with more general $\xi^\ell$ (to be trained)
are expected to be more efficient than the classic CNN models.
\end{remark}

%At last we have the next relation:
%\input{MgNet-relation.tex}

\section{Numerical experiments}\label{sec:numerics}
In this section, we present some numerical results to illustrate the
efficiency and potential of MgNet as described in Algorithm
\ref{alg:mgnet}.

\subsection{Data sets and model structure }
We choose CIFAR-10 and CIFAR-100
\cite{krizhevsky2009learning}
as two data sets for numerical tests.
Here, the CIFAR-10 dataset consists of 60000 32x32 colour
images in 10 classes, with 6000 images per class.
The CIFAR-100 dataset is just like the CIFAR-10,
except it has 100 classes containing 600 images each.
We split these two data sets with 50000 training images
and 10000 test images.

We will mainly carry out
 a comparison with study between MgNet and  ResNet \cite{he2016deep}
on these two data sets, so we choose some
similar process techniques such as there will
be a average pooling before linear regression
layers:
\begin{equation}\label{eq:ave-pooling}
R_{ave}: \mathbb{R}^{m_{J-1} \times n_{J-1} \times c_{J-1}} \mapsto \mathbb{R}^{c_{J-1}}.
\end{equation}
We use the similar ideas in MgNet to take $m_{J} = 0$, thus to choose
$$
u^{J} = \Pi_{J-1}^J u^{J-1, m_{J-1}} \in \mathbb{R}^{c_{J-1}},
$$
with
$$
\Pi_{J-1}^J  = R_{ave}.
$$
This can be true also thanks to our structure that
\begin{equation}\label{eq:c_u}
c_{u,\ell} = c_{u}, \quad 1 \le \ell \le J.
\end{equation}
Given an image $f$, similar to ResNet, we apply our MgNet as follows:
\begin{equation}\label{final-mg}
y = S \circ \theta \circ u^{J}(f),
\end{equation}
where $u^J(f)$ is the output from our MgNet as described in Algorithm
\ref{alg:mgnet},  $S$ is the soft-max mapping in \eqref{softmax} and
\begin{equation}\label{final-theta}
\theta: \mathbb{R}^{c_u} \mapsto \mathbb{R}^\kappa,
\end{equation}
represents a fully linear layer with $\kappa = 10$ for CIFAR-10 and
$\kappa = 100$ for CIFAR-100.

We will make the following choice of hyper parameters
for the MgNet:
\begin{itemize}
	\item $J$: the number of grids. As all images in CIFAR-10 or CIFAR-100
	are $32\times 32 \times 3$,  we choose $J = 4$ to be consistent with ResNet.
	\item $\nu_\ell$:  the number of smoothings in each grids. To be consistent with
	ResNet-18 or ResNet-34 we choose $\nu_\ell = 2$ or $\nu_\ell = 4$.
	\item $c_u$ and $c_f$: the number of feature and data channels.
	\item $A^\ell$: the data-feature mapping. We choose the linear case in \eqref{linearA}.
	\item $B^{\ell,i}$: the feature extractor. We choose the variable extractors as in \eqref{extractor-ell}.
	\item $R_{\ell}^{\ell+1}$: the restriction operator in \eqref{restrict-f}.
	Here we choose it as a convolution with stride $2$ which need to be trained.
	\item $\Pi_\ell^{\ell+1}$: the interpolation operator in
	\eqref{interpolation}.  Here we compare these next three
	different choices:
	\begin{enumerate}
		\item {$\Pi_0$: } $\Pi_\ell^{\ell+1} = 0$;
		\item {$\Pi_1$: }convolution with stride $2$ which need to be
		trained;
		\item {$\Pi_2$: }channel-wise interpolation as in
		\eqref{Pi}, with $\bar P_{\ell}^{\ell+1}$ as a convolution
		with one channel and stride $2$ which also need to be trained.
	\end{enumerate}
\end{itemize}

\subsection{Training algorithm}
While there are many different choices of training algorithms \cite{bottou2018optimization},
in our test, we adopt the popular
stochastic gradient descent (SGD) with mith-batch and momentum for
cross-entropy loss function.
\begin{breakablealgorithm}
	\caption{SGD with mini-batch and momentum}
	\label{alg:sgd}
	\begin{algorithmic}
\State {\bf Input}: learning rate $\eta_t$, batch size $m$, parameter Initialization $ w_0$, number of epochs $K$.
\For{Epoch $k = 1:K$} \\
\State Shuffle data and get mini-batch $B_1, \cdots, B_{\frac{N}{m}}$, choose mini-batch as: $B_{i_t}$ with
$$
i_t \equiv t \mod(\frac{N}{m}),
$$
\State Compute the gradient on $B_{i_t}$:
$$
g_t = \nabla_{w} \frac{1}{m} \sum_{i \in B_{i_t}} h_i(w_{t}).
$$
\State Compute the momentum:
\begin{equation}
v_t = \alpha v_{t-1} - \eta_t g_t \quad (v_0 = 0).
\end{equation}
\State Update $w$:
\begin{equation}
w_{t+1} = w_t + v_t.
\end{equation}
\EndFor
\end{algorithmic}
\end{breakablealgorithm}

Here we have $h_i(w_t) = l(\classmap(f_i;w_t),y_i)$ as defined in \eqref{eq:3}, where $w_t$ notes all free parameters in MgNet and $\theta$ in \eqref{final-theta}.
We use the SGD with momentum of 0.9.
The mini-batch size is chosen as
$m=128$. The learning rate starts from 0.1 and is divided by $10$ for
every $30$ epochs, and the models are trained for up to $K=120$ epochs.
We adopt batch normalization (BN) after each convolution and before
activation, following \cite{ioffe2015batch}.  Initialization strategy
is the same with ResNet as in \cite{he2015delving}.  We
do not use weight decay and dropout.  The final Top-1 test accuracy is
shown in Table~\ref{comparison}.
\begin{table}[!htbp]
	\caption{ResNet and MgNet on CIFAR-10 and CIFAR-100.
	Our methods are named with $\nu_\ell$, ($c_u$, $c_f$), $\Pi_\ell^{\ell+1}$ by definition above.}
	\label{comparison}
	\vskip 0.15in
	\begin{center}
				%		\resizebox{.5\textwidth}{!}{
				\begin{tabular}{cccc}
					\hline
					Models & CIFAR-10 & CIFAR-100 & Params \\
					\hline
					ResNet-18 & 92.24 & 71.96 & 11.2M   \\
					ResNet-34 & 92.80 & 71.93 & 21.3M   \\
					\hline
					%				{$\nu_\ell$,  $(c_u, c_f)$, $\Pi_\ell^{\ell+1}$} & Accuracy & Accuracy & Numbers  \\
					$2, (256,256)$, $\Pi_0$ & 92.02 & 68.29 & 7.1M  \\
					$2, (256,256)$, $\Pi_1$ & 93.04 & 72.32 & 8.9M  \\
					$2, (256,512)$, $\Pi_1$ & 93.20 & 72.42 & 19.5M  \\
					$4, (256,512)$, $\Pi_2$ & 93.53& 74.26 & 17.7M  \\
					\hline
				\end{tabular}
		%		}
	\end{center}
	\vskip -0.1in
\end{table}

From the above numerical results, we find that the modified CNN models
based on MgNet structure have competitive and sometimes better
performance in comparison with standard ResNet models when applied to
both CIFAR-10 and CIFAR-100 data sets. Generally speaking, the more
channels the better performance you can achieve (see WideResNet
\cite{zagoruyko2016wide} for similar observation). Furthermore,
$\Pi_1$ and $\Pi_2$ work better than $\Pi_0$, and $\Pi_2$ can even
work better than $\Pi_1$ with fewer parameters for big enough
channel numbers.

\section{Concluding remarks}\label{sec:conclusion}
By carefully studying the connections between the traditional
multigrid method and the convolutional neural network (especially the
ResNet type) models, the MgNet established in this paper provides a
unified framework that connects both multigrid and CNN in a technical
level.  Comparing with other existing works that discuss the
connection between multigrid and CNN, MgNet goes beyond formal or
qualitative comparisons and identifies key model components that play
the same corresponding roles, from an abstract viewpoint, for these two different
methodologies.  As a result, how and why CNN models work can be
mathematically understood in a similar fashion as for multigrid method
which has a much more mature and better developed theory.  Motivated
from various known techniques from multigrid method, many variants and
improvements of CNN can then be naturally obtained.  For example, as
demonstrated from our preliminary numerical experiments, the resulting
modified CNN models quipped with fewer weights and hyper parameters
actually exhibit competitive and sometimes better performance than
standard ResNet models.

The MgNet framework opens a new door to the
mathematical understanding, analysis and improvements of deep learning
models.  The very preliminary results presented in
this paper have demonstrated the great potential of MgNet from both
theoretical and practical viewpoints.  Obviously many aspects of MgNet
should be further explored and expect to be much improved.  In fact, only very
few techniques from multigrid method have been tried in this paper and
many more in-depth techniques from multigrid require further study for
deep neural networks, especially CNN.
In particular, we believe that the MgNet framework will
lead to improved CNN that only has a small fraction of the number
of weights that are required by the current CNN. On the other hand,
the techniques in CNN can also be used to develop new generation of multigrid
and especially  algebraic multigrid methods \cite{xu2017algebraic} for solving
partial differential equations. Our ongoing works have
demonstrated great potentials for research in these directions and  many
more results will be reported in future papers.


\documentclass[12pt]{article}

\def\exercisename{DG notes}
\usepackage{mystyle}
\begin{document}
	\section*{\exercisename}

Equation to be solved:
\begin{equation*}
    \frac{\partial u}{\partial t} + \frac{\partial au}{\partial x} = 0
\end{equation*}
Multiplication by a test function $v(x)$ and integration over each element:
\begin{equation*}
	\int_{V_e} \frac{\partial u}{\partial t}\, v\, dV_e + \int_{V_e} \frac{\partial au}{\partial x}\, v\, dV_e = 0
\end{equation*}
The second term can be integrated by parts:
\begin{equation*}
	\int_{x_1}^{x_2} \frac{\partial u}{\partial t}\, v\, d{V_e}
	+ \left[ au\,v\right]_{x_1}^{x_2}
	  - \int_{x_1}^{x_2} au \frac{\partial v}{\partial x}\, d{V_e} = 0
\end{equation*}
The key idea of the method is to replace the flux $au$ on the boundary by a flux numerical $f^*$ that is evaluated from the values of $u$ in the neighbouring elements. Thanks to this numerical flux, all the equations will be coupled.
\begin{equation*}
	\int_{x_1}^{x_2} \frac{\partial u}{\partial t}\, v\, d{V_e}
	+ \left[ f^*\,v\right]_{x_1}^{x_2}
	- \int_{x_1}^{x_2} au \frac{\partial v}{\partial x}\, d{V_e} = 0
\end{equation*}
We can integrate by parts the last term:
\begin{equation*}
	\int_{x_1}^{x_2} \frac{\partial u}{\partial t}\, v\, dx  + \int_{x_1}^{x_2} \frac{\partial au}{\partial x}\, v\, dx
= \left[ (au - f^*)\,v\right]_{x_1}^{x_2}
\end{equation*}
Let $N_i(x)$ be the shape functions.
\begin{equation*}
	v(x) = \sum_i N_i(x)\, v_i
\end{equation*}
\begin{equation*}
	u(x,t) = \sum_j N_j(x)\, u_j(t)
\end{equation*}
The integral equation becomes:
\begin{equation*}
	\sum_i \sum_j
	\left[
	\int_{x_1}^{x_2} N_i(x)\,N_j(x)\, dx\, \frac{\partial u_j(t)}{\partial t}
	+
	\int_{x_1}^{x_2} N_i(x)\,\frac{\partial N_j(x)}{\partial x}\, dx\,  au_j(t)
	\right]
	\, v_i
\end{equation*}
\begin{equation*}
	= \sum_i \sum_j
	\left[
		\left( N_j(x)\,au_j(t) - f^{*}\right)\,N_i(x)
	\right]_{x_1}^{x_2}
	\, v_i
\end{equation*}
This equality should be verified for any $v_i$, thus, each coefficient of $v_i$ should vanish. This leads to a system of equations:
\begin{equation*}
 \sum_j M_{ij}\, u_j + S_{ij}\,u_j
 =
 (au_j(x_2)-f^*(x_2))\,N_i(x_2) - (au_j(x_1)-f^*(x_1))\,N_i(x_1)
\end{equation*}

\end{document}


\section{Content}

\subsection{Notation}

\begin{Defn}
  \label{defn:1}
  Some definitions, $\cos(x)$ and \(\sin(x)\) are
  \begin{eqnarray}
    \cos(x) \doteq \frac{e^{ix}+e^{-ix}}{2}  \label{eq:fefwe} \\
    \sin(x) \doteq \frac{e^{ix}-e^{-ix}}{2i} \label{eq:fefwe2}
  \end{eqnarray}
  A reference to a figure \ref{fig:a354}.
  A reference \eqref{eq:fefwe} and  \eqref{eq:fefwe2} to the above equations.
\end{Defn}

\begin{figure}[ht]\label{fig:a354}
  \begin{center}
    \includegraphics[width=0.5\textwidth]{F/sin}
    \caption{Graph of \(\sin(x)\).}
  \end{center}
\end{figure}

\lipsum[2]

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "paper"
%%% End:


\section{Imbalanced Data}

% ===
\emph{Cost Sensitive Classification}
Replace loss by: $\ell_{CS}(w;x,y) = c_y \ell(w;x,y)$

$\color{gray} \text{e.g. } \ell_{\pm} = c_\pm \ell(w;x,y)
\to c_-\cdot \hat R(w; \frac{c_+}{c_-},c_-)$

% ===
\emph{Metrics}

\begin{minipage}{.7\linewidth}
	\textbf{acc} $= \frac{\textrm{TP}+\textrm{TN}}{n}$,
	\textbf{prec} $= \frac{\textrm{TP}}{p_+}$

	\textbf{FPR} $= \frac{\textrm{FP}}{n_-}$,
	\textbf{Recall/TPR} $= \frac{\textrm{TP}}{n_+}$
\end{minipage}%
\begin{minipage}{.3\linewidth}
	\begin{tabular}{@{}l @{ }l | @{ }l}
		TP & FP & $p_+$\\
		FN & TN & $p_-$\\\hline
		$n_+$ & $n_-$ & $n$
	\end{tabular}
\end{minipage}

\textbf{Fβ score}: $F_\beta = \frac{(1+\beta)^2}{\frac{1}{\textrm{prec}} + \frac{\beta^2}{\textrm{rec}}}$,
\enskip
\textbf{ROC:} FPR vs. TPR


\section{Other Algorithms}
  \subsection{2SAT}
    Build the implication graph of the input by converting ORs $A \vee B$ to
    $!A \rightarrow B$ and $!B \rightarrow A$. This forms a bipartite graph.
    If there exists $X$ such that both $X$ and $!X$ are in the same strongly connected component,
    then there is no solution. Otherwise, iterate through the literals, arbitrarily assign
    a truth value to unassigned literals and propagate the values to its neighbors.
\begin{comment}
    \code{other/two_sat.cpp}
\end{comment}
  \subsection{DPLL Algorithm}
    A SAT solver that can solve a random 1000-variable SAT instance within a second.
    \code{other/dpll.cpp}
  \subsection{Stable Marriage}
    \ifverbose
    The Gale-Shapley algorithm for solving the stable marriage problem.
    \fi
    \code{other/stable_marriage.cpp}
\begin{comment}
  \subsection{Algorithm X}
    \ifverbose
    An implementation of Knuth's Algorithm X, using dancing links. Solves the Exact Cover problem.
    \fi
    \code{other/algorithm_x.cpp}
  \subsection{Matroid Intersection}
    Computes the maximum weight and cardinality intersection of two
    matroids, specified by implementing the required abstract methods, in
    $O(n^3(M_1+M_2))$.
    \code{other/matroid_intersection.cpp}
\end{comment}
  \subsection{Cycle-Finding}
    \ifverbose
    An implementation of Floyd's Cycle-Finding algorithm.
    \fi
    \code{other/floyds_algorithm.cpp}
  \subsection{Longest Increasing Subsequence}
    \code{other/lis.cpp}
  \subsection{Dates}
    \ifverbose
    Functions to simplify date calculations.
    \fi
    \code{other/dates.cpp}
  \subsection{Simulated Annealing}
    An example use of Simulated Annealing to find a permutation of length $n$
    that maximizes $\sum_{i=1}^{n-1}|p_i - p_{i+1}|$.
    \code{other/simulated_annealing.cpp}
  \subsection{Simplex}
    \begin{verbatim}
// Two-phase simplex algorithm for solving linear programs
// of the form
//     maximize     c^T x
//     subject to   Ax <= b
//                  x >= 0
// INPUT: A -- an m x n matrix
//        b -- an m-dimensional vector
//        c -- an n-dimensional vector
//        x -- a vector where the optimal solution will be
//             stored
// OUTPUT: value of the optimal solution (infinity if
//                   unbounded above, nan if infeasible)
// To use this code, create an LPSolver object with A, b,
// and c as arguments.  Then, call Solve(x).
    \end{verbatim}
    \code{other/simplex.cpp}
  \subsection{Fast Input Reading}
    If input or output is huge, sometimes it is beneficial to optimize the
    input reading/output writing. This can be achieved by reading all input
    in at once (using fread), and then parsing it manually. Output can also
    be stored in an output buffer and then dumped once in the end (using
    fwrite). A simpler, but still effective, way to achieve speed is to use
    the following input reading method.
    \code{tricks/fast_input.cpp}
  \subsection{128-bit Integer}
    GCC has a 128-bit integer data type named \texttt{\_\_int128}. Useful
    if doing multiplication of 64-bit integers, or something needing a
    little more than 64-bits to represent. There's also
    \texttt{\_\_float128}.
  \subsection{Bit Hacks}
    \code{tricks/snoob.cpp}


\section{Downsampling}

The analysed data was recorded at 32 kHz. For faster processing, the data was downsampled. The Nyquist frequency after downsampling needs to be sufficiently high for sharp-wave ripple detection.\footnotemark{} As ripples have a frequency of up to 250 Hz (\cref{tab:bands}), the new sampling frequency needs to be larger than 500 Hz. I chose a new sampling frequency of 1000 Hz. This allows integer subsampling of the original data, by a factor 32.

\footnotetext{The Nyquist frequency is the maximum discernible frequency in a digital signal, equal to half the sampling frequency.}

To counter aliasing, the 32 kHz data was low-pass filtered before subsampling.\footnotemark{} The cutoff frequency was chosen to be 80\% of the new Nyquist frequency (i.e. at 400 Hz for a new sampling frequency of 1000 Hz). The digital filter was applied both in forwards and backwards directions over the signal, to cancel out distortions due to filter delays that vary per frequency component. The filter type and parameters were chosen to yield both low processing time and low-distortion, high quality filtering, as was verified visually (see for example \cref{fig:downsample}).

\footnotetext{A type I Chebyshev filter was used, which is an infinite impulse response filter, with relatively steep roll-off. The filter was of order 8 and had a passband ripple of 0.05 dB. For this filter, the cutoff frequency is defined to be where the gain starts dropping below the passband ripple.}

\begin{figure}
\img[1]{downsample}
\captionn{Downsampling the data}{Blue: the recorded data at 32 kHz. Orange: the same data, after downsampling to 1000 Hz. The arrow indicates a possible neural spike, removed by the anti-aliasing filter.}
\label{fig:downsample}
\end{figure}


\section{Start with two points}
$f(s)$ is the direct connecting line between the vectors  $\vec a$ and $\vec b$. The output of $f$ is a two-vector $(\theta, \varphi)^T$
\begin{equation}\label{line}
    f(s)=(1-s)\cdot\vec{a}+s\cdot\vec{b}\qquad\forall\, s \in [0,1]
\end{equation}
\begin{equation}
    f'(s)=\vec{b}-\vec{a}
\end{equation}
K: Spherical coordinates $\rightarrow$ euclidean coordinates\\
given:
\begin{equation}\label{euklid}
    K \begin{pmatrix}\theta \\ \varphi\end{pmatrix}=
    \begin{pmatrix}
        \sin\theta\cdot\cos\varphi \\
        \sin\theta\cdot\sin\varphi \\
        \cos\theta
    \end{pmatrix}
\end{equation}
so that:
\begin{equation}
    K'=
    \begin{pmatrix}
       \cos\theta\cdot\cos\varphi && -\sin\theta\cdot\sin\varphi \\
       \cos\theta\cdot\sin\varphi && \sin\theta\cdot\cos\varphi \\
       -\sin\theta && 0
    \end{pmatrix}
\end{equation}
\subsection{Numerical solution}
\begin{equation}
    \frac{dK}{d\theta}\dbinom{\theta}{\varphi}=
    \begin{pmatrix}
        \sin\theta\cdot\cos\varphi \\
        \sin\theta\cdot\sin\varphi \\
        \cos\theta
    \end{pmatrix}
\end{equation}
See \eqref{euklid}.
\begin{equation}
    c_1=\frac{1}{2}
\end{equation}
\begin{equation}
    v_1=\underbrace{K
    \begin{pmatrix}
        \begin{matrix}
            \theta_a \\
            \varphi_a
        \end{matrix}
        +
        \begin{pmatrix}
            \theta_b-\theta_a\\
            \varphi_b-\varphi_a
        \end{pmatrix}
        \cdot c
    \end{pmatrix}
    }_{\alpha}
    -\vec{a}
\end{equation}
\begin{equation}
    c_2=\frac{c}{2}
\end{equation}
\begin{equation}
    v_2=\cdots
\end{equation}
if
\begin{equation}\label{epsilon}
    \frac{\|v_2-v_1\|}{\|v_1\|}<\varepsilon
\end{equation}
or
\begin{equation}
    \frac{\langle v_2, v_1\rangle}{\|v_2\|\cdot\|v_1\|}<\varepsilon
\end{equation}
\begin{lstlisting}[escapechar=@]
    while(@equation: \eqref{epsilon}@ > @$\varepsilon$@) {
        v1 = K(@… $\cdot$ @c) - @$\vec a$@
        v2 = K(@… $\cdot$ @c/2) - @$\vec a$@
        c = c/2;
    }
\end{lstlisting}
\begin{equation}
    \cos(\alpha)=\frac{\langle\mathrm{iter}(a,b), \mathrm{iter}(a,c)\rangle}{\|\mathrm{iter}(a,b)\|\cdot \|\mathrm{iter}(a,b)\|}
\end{equation}


\documentclass[../../main.tex]{subfiles}

\begin{document}

\subsection{Motivation}

The aim of lab is use a generalized multilinear regression model, termed the Higher-Order Partial Least Squares (HOPLS) \cite{zhao2012higher}, to predict a tensor $\underline{Y}$ from a tensor $\underline{X}$ through projecting the data onto the latent space and performing regression on the corresponding latent variables. Specifically, We have problem of recognition human body position. On video with person we need to mark the position of the main points of the body (face, shoulders, hips, elbows, lap, ankle). The experiment is carried out on JHMDB (Joint-annotated Human Motion Data Base) \cite{Jhuang:ICCV:2013}.

\subsection{Problem statement}

Given video with person and skeleton's coordinates for each frame of this video:
\[
\label{eq:example:1}
\begin{aligned}
    \mathfrak{D} = \left\{\underline{X_i}, ~\underline{Y_i}\right\}_{i=1}^{K},
\end{aligned}
\]
where ~$\underline{X_i} \in \mathbb{R}^{T \times W\times H\times C}, ~\underline{Y_i} \in \mathbb{R}^{T \times D \times I}$,\\
~$T$ - number of framres,  ~{W} - width, ~{H} - height, ~{C} - channel,\\ ~{I} - number of points in skeleton, ~{D} - picture dimension.\\
\\
In our case, $T \leqslant 40, ~W = 320, ~H = 240, ~C = 3, ~I = 15, ~D = 2.$\\
\\
The n-mode product of a tensor
$ X \in \mathbb{R}^{I_1\times ... \times I_n \times ... \times I_N}$
and matrix $A \in \mathbb{R}^{J_n\times I_n}$
is denoted by $\underline{Y} = \underline{X} \times_{i}A \in \mathbb{R}^{I_1\times ... \times I_{n-1} \times J_n \times I_{n+1} \times ... \times I_N}$
and defined as:
$$y_{i_1...i_{n-1}j_ni_{n+1}...i_N } = \sum_{i_n} x_{i_1...i_n...i_N }a_{j_ni_n}$$
$$\underline{Y} = \underline{G} \times_{1} A^{(1)} \times_{2} A^{(2)} \times_{3} ...\times_{N} A^{(N)} + \underline{E} = [[G; A^{(1)}, A^{(2)}, ..., A^{(N)}]] + \underline{E}$$

We assume $\underline{X}$ is decomposed as a sum of $rank-(1, L_2, . . . , L_N )$ Tucker blocks, while $\underline{Y}$ is decomposed as a sum of $rank-(1, K_2, . . . , K_M)$ Tucker blocks, which can be expressed as

$$\underline{X} = \sum_{r=1}^{R}\underline{G_r}\times_{1} t_r \times_{2} P^{(1)}_r \times_{3} P^{(2)}_r \times_{4} ...\times_{N} P^{(N-1)}_r + \underline{E}_r$$
$$\underline{Y} = \sum_{r=1}^{R}\underline{D_r}\times_{1} t_r \times_{2} Q^{(1)}_r \times_{3} Q^{(2)}_r \times_{4} ...\times_{N} Q^{(N-1)}_r + \underline{F}_r$$
where R is the number of latent vectors, $t_r \in \mathbb{R}^{I_1}$ is the r-th latent vector $P_r^{n} \in \mathbb{R}^{I_{n+1} \times L_{n+1}}$ and $Q_r^{m} \in \mathbb{R}^{J_{n+1} \times K_{n+1}}$ are loading matrices on mode-n and mode-m respectively, and $\underline{G_r} \in \mathbb{R}^{1 \times L_{2}\times ...\times L_{N}}$ and $\underline{D_r}\in \mathbb{R}^{1 \times K_{2}\times ...\times K_{M}}$ are core tensors.\\
\\
However the Tucker decompositions are not unique due to the permutation, rotation, and scaling issues. To alleviate this problem, additional constraints should be imposed such that the core tensors $\underline{G_r}$ and $\underline{D_r}$ are all-orthogonal, a sequence of loading matrices are column-wise orthonormal, i.e., $P^{(n)T}_rP^{(n)}_r = I$ and $Q^{(n)T}_rQ^{(n)}_r = I$, the latent vector is of length one, i.e. $\|t_r\|_F=1$.\\
\\
The main aim is to reduce Frobeniuses norm of residuals $\underline{E_r}, \underline{F_r}$. With a few propositions the following optimization problems arise

\[
\label{eq:example:2}
\begin{aligned}
   \min_{P^{(n)}, Q^{(m)}} \| [[C; P^{(1)T}, P^{(2)T}, ..., P^{(N-1)T}, Q^{(1)T}, Q^{(2)T}, ..., Q^{(M-1)T}]]\|_F^2,\\
\end{aligned}
\]
$$s.t. ~P^{(n)T}_rP^{(n)}_r = I, ~Q^{(n)T}_rQ^{(n)}_r = I$$
where $\underline{C} = COV_{\{1;1\}}(\underline{X}, \underline{Y})$.\\
Next, we can find latent vector ~$t$ from
\[
\label{eq:example:2}
\begin{aligned}
    \min_{t}\| \underline{X} - [[\underline{G};t,P^{(1)},P^{(2)}, ...,P^{(N-1)}]]\|_F^2
\end{aligned}
\]
Now we found the first latent decomposition. Next step is repiting it for $\underline{X} = \underline{E_1}, ~\underline{Y} = \underline{F_1}$.
\subsection{Problem solution}
Full algorithm is described in [1]. Now we can find solution:
$${Y}^{(new)_{(1)}} \approx T^{(new)}Q^{*T} = X^{(new)}_{(1)}WQ^{*T}$$
where ~$W$ and ~$Q^*$ have R columns, represented by
$$w_r = (P_r^{(N-1)}\otimes ...\otimes P_r^{(1)})\underline{G_{r(1)}^+}$$
$$q_r^* = \underline{D_{r(1)}}(Q_r^{(M-1)}\otimes ...\otimes Q_r^{(1)})^T.$$

\subsection{Code analysis}

The code was taken from \cite{zhao2012higher}.

\subsection{Experiment}

The experiment is carried out on JHMDB (Joint-annotated Human Motion Data Base) \cite{Jhuang:ICCV:2013}.
In our case, $T \leqslant 40, ~W = 320, ~H = 240, ~C = 3, ~I = 15, ~D = 2.$\\
The dataset has 928 videos from 21 class.
\\
Define function of similarity of $\underline{Y}^{(true)}$ and $\underline{Y}^{(predict)}$ as
$Q^2$:
$$Q^2 = 1 - \frac{\|\underline{Y}^{(true)} - \underline{Y}^{(predict)}\|_F^2}{\|\underline{Y}^{(true)}\|_F^2}$$
\\
The first experiment. We divided video into train and test part. Train has 32 frames, test has 8 frames.
\\
\begin{figure}[h!]
\centering
\includegraphics[width=1\textwidth]{figures/fig1}
\caption{Experiment 1. Dependence $Q^2$ on number of latent vectors R.}
\end{figure}
\\
The second experiment. In our dataset we have very similar videos, so we can lear our model on the first and test on the second video. It will show how robust this method.
\\
\begin{figure}[h!]
\centering
\includegraphics[width=1\textwidth]{figures/fig2}
\caption{Experiment 2. Dependence $Q^2$ on number of latent vectors R.}
\end{figure}

\end{document}


%!TEX root =  ../main.tex

\subsection{Like Powers}

\objective{Simply log expressions}

Whether we use the Triangle of Power or not, it is very powerful to recognize
that logs are simply the inverse of exponents.  Without this insight, expressions
like $\log_{10}{100} + \log_{10}{1000}$ would be very intimidating.  But with this
insight, we can paraphrase as we go in our minds: ``There is some exponent
to put on 10 and get 100.  Add to that, some exponent we put on 10 to get
1000.  Adding exponents comes from multiplying bases, so this is the same as
$\log_{10}10000$.  That is asking the question, what exponent do we put on 10
to get 10,000?  5!''

We think it is even easier in Triangles, but we might show you both styles for
the time being:

$$
\tripow{10}{}{100} + \tripow{10}{}{1000} = \tripow{10}{}{10000} = 5
$$

In other words, two logs added, is the same as one long of a multiplication.  This
works for the inverse operation of subtraction: two logs subtracted is the same
as one log of a division.  Lastly, it also works for an exponent: the log of a number
with an exponent is the same as the exponent multiplied against the log of number.

Here, Triangle notation really shines superior, because notation like $\log_2{64^2}$
is confusing.  Does it mean $\log_2{64} \cdot \log_2{64}$ or $\log_2{4096}$, which
is the difference between 36 and 12?  How much clearer is

$$
\tripow{2}{}{\tripow{64}{2}{}} vs \tripow{\tripow{2}{}{64}}{2}{}
$$

Even more consequential is the two-fold possibilities for $2^{3^4}$.  Is that $(2^3)^4$
or $2^{(3^4)}$?  That is the difference between 4096 and $2^81$, the latter of which
is a 25 digit number!  But no one can make the same mistake with

$$
\tripow{\tripow{2}{3}{}}{4}{} vs \tripow{2}{\tripow{3}{4}{}}{}
$$

\subsection{Names}
There are some logs which occur so commonly, that they base is not written.  Normally,
the word ``log'' with no base written means $\log_{10}$.  Log base $e$ is very common,
and has its own symbol $\ln$.  This acronym comes from the French, \textit{log
natural}, since $e$ is the natural number.\footnote{In French, as in Spanish, adjective
most often follow their noun, not precede it.}

Computer scientists most often used $\log_2$, which is called the binary log,
written lb.  In fact
in many disciplines, their flavor of log is the only one used, so it is assumed and the name
``log'' is written, which can fool outsiders into assuming $\log_{10}$.  For example,
Wolfram Alpha is fantastically powerful website for mathematics and other fields,
and so they use ``log'' to mean ``ln''!  We will not be so tricky, and you may assume
the solution to $\log{x}=2$ is 100.

\subsection{Change of Base}
Logs are amazing.  But given the rarity of most bases, it become necessary to ask,
Is there a way to convert them?  If we know how to re-write a log as an exponent,
and take an exponent as a multiplier, then we can:

\begin{align*}
	\log_b{a} & = c \\
	b^c &= a\\
	\ln{b^c} &= \ln{a}\\
	c \cdot \ln{b} &= \ln{a} \\
	c = \frac{\ln{a}}{\ln{b}}
\end{align*}

Notice that it does not matter what base we chose in the third line.  $\ln$ ($\log_e$)
works just as well as $\log_{\pi}$.  This rearrangement to have an arbitrary
base is called the the Change of Base Formula.

\begin{derivation}{Change of Base Formula}
$$
\tripow{b}{}{a} = \frac{\tripow{c}{}{a}}{\tripow{c}{}{b}} \quad \text{a.k.a.} \quad
\log_b{a} = \frac{\log_c{a}}{\log_c{b}}
$$
\end{derivation}

\begin{derivation}{Negative Logarithm}
$$
\tripow{b}{}{1/y} = -\tripow{b}{}{y}  = \tripow{1/b}{}{y}
\quad \text{a.k.a.} \quad
\log_b{\frac{1}{y}} = -\log_b{y} = \log_{\frac{1}{b}}{y}
$$
\end{derivation}

Students often confusing ``the sum of logs'' with ``the log of a sum''.  What can we say
about $\log_b{(a+c)}$?  On the face of it, not much.  But through some creative manipulation,
one identity can be made, which is used in Probability Theory:

\begin{align*}
	\log_b{(a+c)} 	&= + \log_b{a} - \log_b{a} + \log_b{(a+c)}\\
				&= \log_b{a} + \log_b{(a+c)} - \log_b{a}\\
				&= \log_b{a} + \log_b{\frac{a+c}{a}} \\
				&= \log_b{a} + \log_b{1 + \frac{c}{a}}\\
	\tripow{b}{}{a+c} &= \tripow{b}{}{a} + \tripow{b}{}{\left(1 + \frac{c}{a}\right)}
\end{align*}

Lastly, we have already made with the Triangle of Power:

\begin{derivation}{P-Plus Logs}
$$
\tripow{m}{}{x} \pplus \tripow{n}{}{x} = \tripow{m\cdot{}n}{}{x}
\quad \text{a.k.a.} \quad
\log_m{x} \oplus \log_n{x} = \log_{mn}{x}
$$

$$
\tripow{m}{}{x} \pminus \tripow{n}{}{x} = \tripow{m\div{}n}{}{x}
\quad \text{a.k.a.} \quad
\log_m{x} \pminus \log_n{x} = \log_{\frac{m}{n}}{x}
$$

$$
\tripow{\tripow{m}{n}{}}{}{x} = \frac{\tripow{m}{}{x}}{n}
\quad \text{a.k.a.} \quad
\log_{m^n}{x} = \frac{1}{n}\log_m{x}
$$
\end{derivation}


\section{Fundamental Theorem of Calculus}
\subsection{Integration}
Assume below that all functions under consideration are nice enough to let the integral exist.\\
Consider the following sum
$$\sum_{n=0}^{N-1}f(x_n)\Delta x$$
where $\Delta x=(b-a)/N, x_n=a+n\Delta x$.\\
The question is, how close is the finite sum above to the area under the curve of $f$, when $N$ is large?
How big is the difference between the difference between the area and the discrete area in the sum?
\begin{theorem}[Mean-vaue Theorem on Definite Integrals]
    For a continuous function $f(x)$,
    $$\int_{x_n}^{x_n+1}f(x)\,\mathrm dx=f(x_c)(x_{n+1}-x_n)$$
    for some $x_c\in(x_n,x_{n+1})$.
\end{theorem}
Expand $f(x)$ about $x\to x_n$ and evaluate it at $x_c$.
As $\Delta x\to 0$,
$$f(x_c)=f(x_n)+O(x_c-x_n)=f(x_n)+O(x_{n+1}-x_n)$$
as $|x_c-x_n|<|x_{n+1}-x_n|$.
Hence by the mean-value theorem:
\begin{align*}
    \int_{x_n}^{x_n+1}f(x)\,\mathrm dx
    &=f(x_n)(x_{n+1}-x_n)+O(x_{n+1}-x_n)(x_{n+1}-x_n)\\
    &=\Delta xf(x_n)+O(\Delta x^2)
\end{align*}
So $\epsilon=O(\Delta x^2)$.
It follows that
$$\int_a^bf(x)\,\mathrm dx=\lim_{N\to\infty}\sum_{n=0}^{N-1}f(x_n)\Delta x+\epsilon_n$$
By our bounds above, the error terms $\sum_n\epsilon_n=O(N\Delta x^2)=O((b-a)^2/N)$ vanish as $N\to\infty$.
Therefore,
$$\int_a^bf(x)\,\mathrm dx=\lim_{N\to\infty}\sum_{n=0}^{N-1}f(x_n)\Delta x$$
\begin{theorem}[Fundamental Theorem of Calculus]
    Let
    $$F(x)=\int_a^xf(t)\,\mathrm dt$$
    then $\mathrm dF/\mathrm dx=f(x)$.
\end{theorem}
\begin{proof}
    We try to evaluate the derivative of $F$,
    \begin{align*}
        \frac{\mathrm dF}{\mathrm dx}
        &=\lim_{h\to0}\frac{1}{h}\int_{x}^{x+h}f(t)\,\mathrm dt\\
        &=\lim_{h\to0}\frac{1}{h}(f(x)h+O(h^2))\\
        &=f(x)
    \end{align*}
    So $\mathrm dF/\mathrm dx=f(x)$.
\end{proof}
\begin{corollary}
    $$\frac{\mathrm d}{\mathrm dx}\int_x^bf(t)\,\mathrm dt=-f(x)$$
\end{corollary}
\begin{corollary}
    $$\int_a^{g(x)}f(t)\,\mathrm dt=f(g(x))g^\prime(x)$$
\end{corollary}
\begin{corollary}
    Let
    $$F(x)=\int f(x)\,\mathrm dx$$
    be the indefinite integral (or anti-derivative) of $f$, then
    $$\int_a^bf(t)\,\mathrm dt=F(b)-F(a)$$
\end{corollary}
\subsection{Some Integration Techniques}
The first one is integration by substitution.
\begin{example}
    $$\int\frac{1-2x}{\sqrt{x-x^2}}\,\mathrm dx=\int\frac{\mathrm du}{\sqrt u}=2\sqrt{u}+C$$
\end{example}
Trigonometric substitution
\begin{example}
    If we see something like $\sqrt{a^2-x^2}$, then we can substitute $x=a\sin\theta$.\\
    If we see something like $x^2+a^2$, we can use $x=a\tan\theta$.\\
    If we see $\sqrt{x^2-a^2}$, we can use $x=a\cosh\theta$ or $x=a\sec\theta$.\\
    If we see $\sqrt{x^2+a^2}$, we can use $x=a\sinh\theta$ or $x=a\tan\theta$.\\
    If we see $a^2-x^2$, we can use $x=a\tanh\theta$
\end{example}
And, of course, we have integration by part:
$$\int uv^\prime=uv-\int u^\prime v$$
from product rule.


\section{The \binarysearch algorithm}
\Label{sec:binarysearch}
\Label{sec:binarysearchii}

The \binarysearch algorithm is one of the four binary search
algorithms of the \cxx Standard Library \cite[\S 28.7.3.4]{cxx-17-draft}.
For our purposes we have modified
the generic implementation
to that of an array of type \valuetype.
The signature now reads:

\begin{lstlisting}[style = acsl-block]
  bool binary_search(const value_type* a, size_type n, value_type  v);
\end{lstlisting}

Again, \binarysearch requires that its input array is in increasing order.
It will return \inl{true} if there exists an index~\inl{i}
in~\inl{a} such that \inl{a[i] == v} holds.\footnote{%
   To be more precise: The \cxx Standard Library requires that
   \inl{(a[i] <= v)  && (v <= a[i])} holds.
   For our definition of \valuetype (see \S\ref{sec:frequentPattern}) this
   means that \inl{v} equals \inl{a[i]}.
}

\begin{figure}[hbt]
\centering
\includegraphics[width=0.60\textwidth]{Figures/binary_search.pdf}
\caption{\Label{fig:binarysearch}Some examples for \binarysearch}
\end{figure}

\FloatBarrier

In Figure~\ref{fig:binarysearch} we do not need to use arrows to visualize the
effects of \binarysearch.
The colors orange and grey of the sought-after values indicate whether the algorithm
returns true or false, respectively.

\subsection{Formal specification of \binarysearch and \binarysearchii}

The \acsl specification of \specref{binarysearch} is shown in the following listing.

\input{Listings/binary_search.h.tex}

Note that instead of the somewhat lengthy existential quantification
of \specref{binarysearch} we can use our previously introduced predicate
\logicref{SomeEqual} in order to achieve the following more concise
formal specification \specref{binarysearchii}.

\input{Listings/binary_search2.h.tex}

It is interesting to compare the specification of \specref{binarysearch}
with that of \specref{findii}.
Both algorithms allow to determine whether a value is contained in an array.
The fact that the \cxx Standard Library requires that \find has
\emph{linear} complexity whereas \binarysearch must have a
\emph{logarithmic} complexity can currently not be expressed with \acsl.

\subsection{Implementation of \binarysearch}

Our implementation \implref{binarysearchii} first calls \specref{lowerbound}.
Remember that if the latter returns an index \inl{0 <= i < n},
then we can be sure that \inl{v <= a[i]} holds.

\input{Listings/binary_search2.c.tex}


% based on example 7 in pythontex_gallery
% https://github.com/gpoore/pythontex/

\documentclass[12pt]{pylatex}
\usepackage{examples}

\begin{document}

\section*{A table of derivatives and anti-derivatives}

This example is based upon a nice example in the Pythontex gallery, see
\ \url{https://github.com/gpoore/pythontex/}.
It uses a tagged block to capture the {\tt\small Sympy} output for later use
in the body of the LaTeX table.

\lstset{numbers=left}

\begin{minipage}[t]{0.72\textwidth}
\begin{python}
   from sympy import *

   var('x')

   # Create a list of functions to include in the table
   funcs = [['sin(x)',r'\\'],       ['cos(x)',r'\\'],       ['tan(x)',r'\\'],
            ['asin(x)',r'\\[5pt]'], ['acos(x)',r'\\[5pt]'], ['atan(x)',r'\\[5pt]'],
            ['sinh(x)',r'\\'],      ['cosh(x)',r'\\'],      ['tanh(x)',r' ']]

   # pyBeg (CalculusTable)
   for func, eol in funcs:
       myddx = 'Derivative(' + func + ', x)'
       myint = 'Integral(' + func + ', x)'
       print(latex(eval(myddx)) + '&=' + latex(eval(myddx + '.doit()')) + r'\quad & \quad')
       print(latex(eval(myint)) + '&=' + latex(eval(myint + '.doit()')) + eol)
   # pyEnd (CalculusTable)
\end{python}
\end{minipage}
\hskip 1cm
\begin{minipage}[t]{0.28\textwidth}
\begin{latex}
   \begin{align*}
      \py {CalculusTable}
   \end{align*}
\end{latex}
\end{minipage}

\clearpage

\begin{align*}
   \py {CalculusTable}
\end{align*}

\end{document}


% !TeX root = ../solution.tex

\hypertarget{he22.24}{%
\chapter{[HE22.24] C2 Traffic}\label{he22.24}}

\begin{marginfigure}
	\includegraphics[width=49mm]{level6/challenge24.jpg}
\end{marginfigure}
\section{Intro}
We have detected C2 payloads on one of our servers! The blue team have
extracted its communications from the traffic logs, and Operations have dumped
the payload code from the running process.

Find out what the actors have exfiltrated!

File: \verb+c2traffic.zip+

\section{Solution}\label{hv22.24solution}

The payload shows that three values, $p$, $g$, $A$, are passed in, then a
random number $b$ is selected from the interval $(0,p]$ and then another value
$B = A^b \bmod p$ is calculated and returned.  From the log, we know the four
values and can calculate $b$, which is then used in the encryption of the
traffic.

There must be a simpler way, but I just bruteforced the constant $b$ and used
the result to decode the traffic.  The exchanged messages contain the flag and
all constants are:

\begin{verbatim}
p: 2272978429
g: 2
A: 1116819144
B: 1042188408
found b:  620620105
b'ls'
b'cat sensitive.txt'
b'sensitive.txt\n'
b'he2022{wh4dy4_m3an_32_b1t5_1s_1n53cur3}\n'
\end{verbatim}


\section{Logistic hypothesis using logistic regression}

\subsection{Implementation}
For logistic regression the hypothesis function is a sigmoid function given by equation \ref{eq:logistic_regression}.
As before a learning rate($\alpha$) = 0.01 was used and gradient descent approach was used for parameter optimization.

\begin{equation}
\label{eq:logistic_regression}
h_{\theta}(X) = \frac{1}{e^{-\theta_0 - \theta_1 * X}}
\end{equation}

\subsection{Observation}
The final values of $\theta$s for a run are as follows:

% Theta 0 final value
\begin{equation}
\theta_0 = -3.5327129208000407
\end{equation}

% Theta 1 final value
\begin{equation}
\theta_1 = 1.3214831636264568
\end{equation}

\begin{figure}[!ht]
  \includegraphics[width=\textwidth,height=0.4\textheight,keepaspectratio]{logistic_regression_curve_0_01.png}
  \caption{Logistic regression curve}
  \label{fig:logistic_regression}
\end{figure}

The logistic curve corresponding to above $\theta$s is shown in figure \ref{fig:logistic_regression}

\subsection{Source Code}
\lstinputlisting[language=python]{task_2.py}


\chapter{A Template chapter}
\label{chapter:example_chapter}

This chapter contains some examples of how to use equations, tables and figures in the template. The last section shows how it all comes together in the final document.

\section{Equations}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Single line equations}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\vspace{0.4cm}

\begin{equation}
	l = \int ds =  \int_t \sqrt{\left(\frac{dx}{dt} \right)^2 + \left(\frac{dy}{dt} \right)^2 + \left(\frac{dz}{dt} \right)^2} dt \; .
	\label{eq:curve_length_3D_cartesian}
\end{equation}

\vspace{0.4cm}

\begin{equation}
	ds^2 = -\left( 1 - \frac{2 M}{ r}  \right)  dt^2 + \left( 1 - \frac{2 M}{ r}  \right)^{-1} dr^2 + r^2 d \theta^2 + r^2 \sin^2 \theta d \phi^2
	\label{eq:schwarzschild_solution}
\end{equation}

\vspace{0.4cm}

\begin{equation}
	\rho = \rho_0 + \epsilon \rho_1 \qquad p = p_0 + \epsilon p_1 \qquad \psi = \psi_0 + \epsilon \psi_1 \; .
	\label{eq:linear_perturbations}
\end{equation}

\vspace{0.4cm}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Single line equations with text}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\vspace{0.4cm}

\begin{equation}
	\lambda_B \sim \SI{0.5}{kpc} \left( \frac{ 10^{-22} \; \text{eV} }{m} \right) \left( \frac{ 250 \; \text{km/s} }{v} \right) \;.
\end{equation}

\vspace{0.4cm}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Single line equations with box}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\vspace{0.4cm}

\begin{equation}
		\boxed{
				T_{\text{Scalar}}^{\mu \nu} =
				(g^{\mu \alpha}g^{\nu \beta} + g^{\mu \beta}g^{\nu \alpha} - g^{\mu \nu}g^{\alpha \beta}) \partial_\alpha \Psi \partial_\beta \Psi - g^{\mu \nu} \mu^2 \Psi^2
		}
	\label{eq:superradiant_condition}
\end{equation}

\begin{equation}
	\boxed{
		\boxed{
			\omega < m \Omega_H
		}
	}
	\label{eq:superradiant_condition_2}
\end{equation}

\vspace{0.4cm}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Multiline equations}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\vspace{0.4cm}

\begin{equation}
	\begin{aligned}
		a 		&= J/M \\
		\rho 	&= r^2 + a^2 \cos^2 \theta  \\
		\Delta	&= r^2 - 2 M r + a^2 \; .
	\end{aligned}
\end{equation}

\vspace{0.4cm}

\begin{equation}
	\begin{aligned}
		\left[  \frac{(r^2 + a^2)}{\Delta} - a^2 \sin^2 \theta\right]\frac{\partial^2 \Psi}{\partial t^2} + \frac{4M a r}{\Delta} &\frac{\partial^2 \Psi}{\partial t \partial \varphi} + \left[ \frac{a^2}{\Delta} - \frac{1}{\sin^2 \theta}\right] \frac{\partial^2 \Psi}{\partial \varphi^2} + \\[10pt] & - \frac{\partial}{\partial r} \left( \Delta \frac{\partial  \Psi}{\partial r}\right) - \frac{1}{\sin \theta} \frac{\partial}{\partial t}\left( \sin \theta \frac{\partial \Psi}{\partial \theta}\right) = 0 \; .
	\end{aligned}
\end{equation}

\vspace{0.4cm}

\begin{equation}
	\begin{aligned}
		\sum_{m} \mathcal{A}_{+}^{m}&\left[ \phi_m^+ - i \tilde{Z}_0 \left(\phi_m^+\right)'   \right] + \mathcal{A}_{-}^{m}\left[ \phi_m^- - i \tilde{Z}_0 \left(\phi_m^-\right)'   \right] = \\
		=&\sum_{m} \epsilon \left(\frac{i  \tilde{Z}}{2}\right) \left[ \mathcal{A}_{+}^{m} \left(\phi_m^+\right)' + \mathcal{A}_{+}^{m+2} \left(\phi_{m+2}^+\right)' + \mathcal{A}_{+}^{m-2} \left(\phi_{m-2}^+\right)'  \right] + \\
		& \qquad +\sum_{m} \epsilon \left(\frac{i  \tilde{Z}}{2}\right) \left[ \mathcal{A}_{-}^{m} \left(\phi_m^-\right)' + \mathcal{A}_{-}^{m+2} \left(\phi_{m+2}^-\right)' + \mathcal{A}_{-}^{m-2} \left(\phi_{m-2}^-\right)'  \right] \; ,
	\end{aligned}
	\label{eq:infinite_set_of_equations}
\end{equation}
%

\vspace{0.4cm}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Matrix Equations}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\vspace{0.4cm}

\begin{equation}
	\arraycolsep = 5pt \def\arraystretch{1.1}
	\left[
	\begin{array}{ccccccc}
		\ddots    &  \beta_+^4  &               &               &               &               &             \\
		\beta_+^6 & \Lambda_+^4 &  \beta_+^2  &               &               &               &             \\
		&  \beta_+^4  & \Lambda_+^2 &  \beta_+^0  &               &               &             \\
		&               &  \beta_+^2  & \Lambda_+^0 &  \beta_+^2  &               &             \\
		&               &               &  \beta_+^0  & \Lambda_+^2 &  \beta_+^4  &             \\
		&               &               &               &  \beta_+^2  & \Lambda_+^4 & \beta_+^6 \\
		&               &               &               &               &  \beta_+^4  &   \ddots
	\end{array}
	\right]
	\left[
	\begin{array}{c}
		\vdots \\
		\mathcal{A}_{+}^{-4} \\
		\mathcal{A}_{+}^{-2} \\
		\mathcal{A}_{+}^{0} \\
		\mathcal{A}_{+}^{2} \\
		\mathcal{A}_{+}^{4} \\
		\vdots
	\end{array}
	\right]
	=
	\left[
	\begin{array}{c}
		\vdots \\
		0 \\
		-\beta_-^0 \\
		-\Lambda_-^0 \\
		-\beta_-^0 \\
		0 \\
		\vdots
	\end{array}
	\right]
	\label{eq:matrix_eigenvalue_equation}
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Conditional equation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\vspace{0.4cm}

\begin{equation}
	\alpha(t , \mathbf{r}) =
	\begin{cases}
		\alpha_0 \qquad \text{if} \qquad (\mathbf{r} -\mathbf{R_{orbit})}^2 < R_a^2 \\
		\alpha_0 \qquad \text{if} \qquad (\mathbf{r} +\mathbf{ R_{orbit})}^2 < R_a^2 \\
		0   \; \;\qquad \text{otherwise}
		\label{eq:alpha_binary_bh}
	\end{cases} \; ,
\end{equation}

\vspace{0.4cm}

\section{Tables}

\begin{table}[h]
	\caption{Example table number 1}
	\label{table:an_example_table}
	\centering
	\begin{tabular}{clllll}
		\toprule
		\midrule[0.4pt]
		ID&$A$  & $r_0$  &  $\sigma$ &  $\omega$  &  $m$  \\
		\midrule
		S1 & 3.5 & 40.0 & 4.0 & 1.28 & 0 \\
		S2 & 3.5 & 15.0 & 2.0 & 0.1 & 2 \\
		\bottomrule
	\end{tabular}
\end{table}

\begin{table}[h!]
	\centering
	\caption{Example table number 2}
	\begin{tabular}{SSSSSSS}
		\toprule
		\midrule[0.4pt]
		{$k$} & {$J_0(x)$} & {$J_1(x)$} & {$J_2(x)$} & {$J_3(x)$} & {$J_4(x)$} & {$J_5(x)$} \\
		\midrule
		1     & 2.4048     & 3.8317     & 5.1356     & 6.3802     & 7.5883     & 8.7715     \\
		2     & 5.5201     & 7.0156     & 8.4172     & 9.7610     & 11.0647    & 12.3386    \\
		3     & 8.6537     & 10.1735    & 11.6198    & 13.0152    & 14.3725    & 15.7002    \\
		4     & 11.7915    & 13.3237    & 14.7960    & 16.2235    & 17.6160    & 18.9801     \\
		\midrule[0.4pt]
		\bottomrule
	\end{tabular}
	\label{table:bessel_zeros}
\end{table}

\section{Figures}

Generating figures in a consistent manner is usually challenge. Especially when the work to be presented is made by a team of several elements. The graphics presented in this template are generated with the \href{"aa"}{\texttt{Jlop}} template in python.

\subsection*{Single Figure}

\begin{figure}[!ht]
	\centering
	\includegraphics{figures/chapter_B/jlop_sc_sqroot.png}
	\caption{An example figure}
	\label{fig1:single_image}
\end{figure}

\pagebreak
\subsection*{Side By Side}

\begin{figure}[!ht]
	\centering
	\includegraphics{figures/chapter_B/jlop_dc_golden.png}
	\caption{An example figure with two elements. The two plots where generated as a single image with the correct document width. If this is not possible, follow the procedure done to generate figure \ref{fig1:side_by_side_image}.}
	\label{fig1:side_by_side_single}
\end{figure}

\begin{figure}[h!]
	\centering
	\includegraphics[width=.48\linewidth]{example-image-a}
	\includegraphics[width=.48\linewidth]{example-image-b}
	\caption{An example figure with two elements. It is usually better to make a single image with both plots as to input a single file. When that is not possible, one should resort to this method}
	\label{fig1:side_by_side_image}
\end{figure}

\vfill
\pagebreak
\section{Putting it al together}

\lipsum[5]

\begin{equation}
	\lambda_B \sim \SI{0.5}{kpc} \left( \frac{ 10^{-22} \; \text{eV} }{m} \right) \left( \frac{ 250 \; \text{km/s} }{v} \right) \;.
\end{equation}

\lipsum[10]

\begin{table}[h]
	\caption{Example table number 1}
	\label{table:example_table_2}
	\centering
	\begin{tabular}{clllll}
		\toprule
		\midrule[0.4pt]
		ID&$A$  & $r_0$  &  $\sigma$ &  $\omega$  &  $m$  \\
		\midrule
		S1 & 3.5 & 40.0 & 4.0 & 1.28 & 0 \\
		S2 & 3.5 & 15.0 & 2.0 & 0.1 & 2 \\
		\bottomrule
	\end{tabular}
\end{table}

\lipsum[2]

\begin{figure}[t!]
	\centering
	\includegraphics[width=.48\linewidth]{example-image-a}
	\includegraphics[width=.48\linewidth]{example-image-b}
	\caption{An example figure with two elements. It is usually better to make a single image with both plots as to input a single file. When that is not possible, one should resort to this method}
	\label{fig1:side_by_side_image_2}
\end{figure}

\lipsum


% based on example 7 in pythontex_gallery
% https://github.com/gpoore/pythontex/

\documentclass[12pt]{mpllatex}
\usepackage{examples}

\begin{document}

\section*{A table of derivatives and anti-derivatives}

This example is based upon a nice example in the Pythontex gallery, see
\ \url{https://github.com/gpoore/pythontex/}.
It uses a tagged block to capture the Maple output for later use
in the body of the LaTeX table.

\lstset{numbers=left}

\begin{minipage}[t]{0.75\textwidth}
\begin{maple}
   # Create a list of functions to include in the table
   funcs := [[sin(x),"\\\\"],         [cos(x),"\\\\"],         [tan(x),"\\\\"],
             [arcsin(x),"\\\\[5pt]"], [arccos(x),"\\\\[5pt]"], [arctan(x),"\\\\[5pt]"],
             [sinh(x),"\\\\"],        [cosh(x),"\\\\"],        [tanh(x)," "]]:

   # mplBeg (CalculusTable)
   for foo in funcs do
       func := foo[1]:
       eol  := foo[2]:
       myddx := ''diff''(func,x):
       myint := ''int''(func,x):
       Print(cat(Latex(myddx),"&=",Latex(diff(func,x)),"\\quad & \\quad")):
       Print(cat(Latex(myint),"&=",Latex(int(func,x)),eol)):
   end do:
   # mplEnd (CalculusTable)
\end{maple}
\end{minipage}
\hskip 1cm
\begin{minipage}[t]{0.25\textwidth}
\begin{latex}
   \begin{align*}
      \mpl {CalculusTable}
   \end{align*}
\end{latex}
\end{minipage}

\clearpage

\begin{align*}
   \mpl {CalculusTable}
\end{align*}

\end{document}


\subsubsection{The Fundamental Theorem of Linear Algebra}

In order to proof the pending \cref{thm:SVD2} from previous
section, we need to present the four subspaces that an arbitrary
matrix $A$ introduces. But before that, a few important
definitions and remarks: \\

\begin{itemize}
    \item Matrix application $A\vec{x}$ can be seen as a linear
      combination of the columns of $A$:
      \[
      A\vec{x} =
      \begin{bmatrix}
        \vec{A_1} \mid \vec{A_2} \mid \cdots \mid \vec{A_n}
      \end{bmatrix} \vec{x} =
      \sum_{i=1}^n x_i \vec{A_i}
      \]
  \item Subspace: A subset of a vector space, which is itself a
    vector space (that is, contains the \vec{0} and is closed under
    the addition and multiplication by an scalar).
  \item Dimension: Is the size of any basis of a vector space (an
    important result in Linear Algebra, shows that all the basis must
    have the same number of elements; hence, the dimension is a property of the
    space itself). The dimension of a vector space (or subspace) $V$
    is denoted as $\dim{V}$.
  \item Given vector space $V$ and a subspace $W \subset V$ , the
    subspace $\ortc{W} \subset V$ consists of all
    the vectors of $V$ which are orthogonal to all the vectors of
    $W$. $\ortc{W}$ is called the orthogonal complement of $W$.
\end{itemize}
\hfill

Now is right time to talk about the subspaces: we already established
that each matrix of $m \times n$, can be seen as the operational
representation of a linear transformation with signature $\R{n}
\fromto \R{m}$. The action of $A$, in transforming the vectors from
one space to the other, has an interesting effect on each side: both
domain (\R{n}) and codomain (\R{m}), are broken in two orthogonal
pieces. Those pieces actually, happen to be subspaces and their basis
are contained on the matrices $V$ and $U$ of the SVD factorization! But
let us explain piece by piece; a good start, is the column space. \\

The column space of a matrix $A$, is pretty much the concept of the
image of a function; that is, the set of all vectors in
$A\vec{x} \in \R{m} \suchthat \vec{x} \in \R{n}$, and it is denoted as
$\C{A}$. Another way of looking at it (per one of the remarks above),
is that each application
of the linear transformation $A$ (that is, each $A\vec{x}$), converts
the input vector \vec{x} into a linear combination of the columns of
A; therefore, $\C{A}$ is the spanning set generated by the
columns of $A$. It can be proved that this subset is actually a
subspace of \R{m}. \\

The next subspace is also clearly understood, is the so called
null space of $A$. It consists of all the solutions to the homogeneous
system $A\vec{x} = \vec{0}$ and is denoted as $\N{A}$. In the
language of transformations, is the set
of all vectors $\vec{x} \in \R{n}$ that function $A$ compresses into the zero
vector of \R{m}. This subset at least contains the vector \vec{0}, but
in general it will contain much more vectors (only the non-singular
matrices have $\N{A} = \{\vec{0}\}$). Again, it can be proved
that this subset is also a subspace (though this one belongs to \R{n}). \\

The next two subspaces, are not that intuitive to introduce; unless we
think now in terms of the transformation represented by matrix
\trans{A}. This matrix represents a linear transformation that goes
into the opposite direction of $A$, that is, from \R{m} to \R{n}. If
we think in the image of this function $\{\trans{A}\vec{y} \in \R{n} \mid
y \in \R{m} \}$, an interesting realization comes to the picture: what if we
apply the same idea as before, that $\trans{A}\vec{y}$ is a linear
combination of the columns of $\trans{A}$: \\

\[
\trans{A}\vec{y} =
\begin{bmatrix}
  \vec{(\trans{A})_1} \mid \vec{(\trans{A})_2} \mid \cdots \mid \vec{(\trans{A})_m}
\end{bmatrix} \vec{y} =
\sum_{i=1}^m y_i \vec{(\trans{A})_i} =
\sum_{i=1}^m y_i (\text{row $i$ of $A$})
\]
\hfill

In order words, columns of \trans{A} are the rows of $A$, therefore;
the column space of \trans{A} is precisely the row space of original
matrix $A$; this is denoted as \C{\trans{A}} and it can also
be proved that it is a subspace of \R{m}. The last and fourth
subspace, comes from considering the null space of \trans{A}; that is,
those vectors \vec{y} in \R{m} which are compressed into the zero
vector of \R{n}. Is not hard to prove that this is a subspace as
well; it is called the left null space of $A$, and denoted as
\N{\trans{A}}. \\

Summarizing, the four subspaces associated to any
matrix $A$ of $m \times n$ are the following (intuitive proofs that
all of them are indeed subspaces can be found in \cite{strang88}): \\

\begin{itemize}
\item \C{\trans{A}}: row space, lives in \R{n}
\item \N{A}: null space, lives in \R{n}
\item \C{A}: column space, lives in \R{m}
\item \N{\trans{A}}: left null space, lives in \R{m}
\end{itemize}
\hfill

The first thing we note is the intentional grouping of these subspaces;
while \C{\trans{A}} and \N{A} belong to \R{n} [the domain of $A$],
\C{A} and \N{\trans{A}} belong to \R{m} [the codomain of $A$]. These
pairs of subspaces have more in common than merely sharing same
hosting space, they are orthogonal with each other! This is the time
to meet what Strang calls the Fundamental Theorem of Linear Algebra
(part II\footnote{Strang presents the theorem parts in the opposite order
  in \cite{strang88};
  but we preferred to keep our own order, aiming to match better the
  flow of deductions presented in this work.}): \\

\begin{theorem}[Fundamental Theorem of Linear Algebra (part II)]
\label{thm:Fund1}
Let $A$ be a real matrix of $n \times m$, then

\[
\C{\trans{A}} = \ortc{\N{A}} \ds{\land} \C{A} = \ortc{\N{\trans{A}}}
\]
\end{theorem}
\hfill

\begin{proof}
Let $\vec{x} \in \N{A}$, then \vec{x} satisfies the equation $A\vec{x}
= \vec{0}$; but the resulting vector in \R{m} has as entries the dot product
of \vec{x} with the rows $r_i$ of $A$, therefore, the equation $A\vec{x} =
\vec{0}$ can be rewritten as $m$ equations of the form:

\[
r_i \cdot \vec{x} = 0, \ds{for } 1 \le i \le m
\]

which is essentially saying that the vector \vec{x} is orthogonal to
all the rows $r_i$ of $A$; therefore, it is orthogonal to every linear
combination of them. But those linear combinations are precisely the
row space \C{\trans{A}}; thus $\C{\trans{A}} \perp \N{A}$, or,
reusing previously introduced terminology (see remarks section), we
can say that the row space is the orthogonal complement of the null
space (which is written as $\C{\trans{A}} = \ortc{\N{A}}$).  \\

An analogous argument can be constructed for \C{A} and \N{\trans{A}},
using the equation $\trans{A}\vec{y} = \vec{0}$ (details are in
\cite{strang88}). Thus, we can also conclude that the column space is
the orthogonal complement of the left null space (which can be written as $\C{A} =
\ortc{\N{\trans{A}}}$).
\end{proof}

Having established the orthogonality of these subspaces, allow us to
introduce a secret weapon that will finally help us prove the pending
\cref{thm:SVD2}. This weapon is another theorem that establishes a
relationship between the dimension of any subspace and its orthogonal
complement \footnote{The
  name was provided by us, as Lang does not name it in his book
  \cite{lang04}.}: \\

\begin{theorem}[Orthogonal Complement Dimension Theorem]
\label{thm:ortdim}
Let $W$ any subspace of \R{n}, then is the case that

\[
\dim{W} + \dim{\ortc{W}} = n
\]
\end{theorem}
\hfill

An even more generic version of this theorem is proved by Lang in
\cite{lang04} (theorem 2.3 in that book), and it basically says that
if we take any subspace and its orthogonal complement together, they
form the entirety of the host space! Another way of seeing this
result, is saying that the host space $V$ is the direct sum of the
subspace $W$ and its orthogonal  complement (denoted as $V = W \oplus
\ortc{W}$). Intuitively, the notion
of a direct sum tells us that there is nothing out of the union of the
subspace $W$ and its orthogonal complement \ortc{W}; every vector in
the original space can be expressed as a sum $x_1 + x_2$ (where
each $\vec{x_1} \in W$ and $\vec{x_2} \in \ortc{W}$, and $W$), and
\ortc{W} do not share anything other than zero vector ($W \cap
\ortc{W} = \{\vec{0}\}$). \\

Using this weapon, we can finally prove the pending
\cref{thm:SVD2}, which was about proving that the last $n-r$ vectors
of $V$ actually belong to \N{A}. \\

\svdtwo*

\begin{proof}
By hypothesis we know that

\[
A\vec{v_i} = \sigma_i\vec{u_i}, \ds{}\forall i=1 \dots r, \ds{}\text{where
} r = rank(A).
\]
\\

Since the vectors \vec{u_i}\apos{s} form a basis of \R{m}, that
implies none of them can be zero; therefore $A\vec{v_i} \ne \vec{0}
\ds{}\forall i=1 \dots r$. By definition, such condition implies that
those vectors $v_i \notin \N{A}$, but since $\R{n} = \C{\trans{A}} \oplus \N{A}$,
then the only other option for those vectors
$\{\vec{v_1},\vec{v_2},\dots,\vec{v_r}\}$ is to belong to the row
space \C{\trans{A}}. Actually, since they are all orthogonal, they form
a basis of \C{\trans{A}} (because $\dim{\C{\trans{A}}} = \func{rank}(A) =
r$). Let us call this basis $B_r$. \\

Let $\vec{v_i} \in \R{n}$, which also belongs to
$\{\vec{v_{r+1}},\vec{v_{r+2}},\dots,\vec{v_n}\}$; since \vec{v_i}
is orthogonal to every vector in $B_r$ (as all the \vec{v}\apos{s}
form an orthonormal basis of \R{n}), then \vec{v_i} can not belong
to the subspace generated by $B_r$, which happens to be
\C{\trans{A}}. Using again the fact that $\R{n} = \C{\trans{A}} \oplus
\N{A}$, we can tell that the only other option for $\vec{v_i}$ is to
belong to the nullspace \N{A}. And by definition of nullspace:

\[
A\vec{v_i} = \vec{0}, \ds{}\forall i=(r+1) \dots n
\]
\hfill

which completes the proof. Mirroring the reasoning about basis $B_r$
of the row space, we can also tell that the vectors
$\{\vec{v_{r+1}},\vec{v_{r+2}},\dots,\vec{v_n}\}$ form a basis of the
null space \N{A}.
\end{proof}

Since we established already the orthogonality between the four
subspaces of matrix $A$, we can simply apply \cref{thm:ortdim}
to them in pairs (depending on whether they are hosted on same space),
and derive the following equations: \\

\begin{enumerate}
\item In \R{n}: $\C{\trans{A}} = \ortc{\N{A}} \implies
  \dim{\N{A}} + \dim{\C{\trans{A}}} = n$
\item In \R{m}: $\C{A} = \ortc{\N{\trans{A}}} \implies
  \dim{\N{\trans{A}}} + \dim{\C{A}} = m$
\end{enumerate}
\hfill

These two equations are what Strang calls the Fundamental Theorem of
Linear Algebra (Part I); further references are \cite{strang88} and
\cite{strang93}. The two parts of such theorem together, basically
describe what are the subspaces generated by matrix $A$, what is the
relationship among them (orthogonality) and what are their
dimensions. The \cref{fig:fund} below (taken from
\cite{strang88}), summarizes both parts of this important theorem:
\\

\begin{figure}[h]
  \centering
  \includegraphics[width=14cm]{fund}
  \caption{Visualization of the Fundamental Theorem of Linear Algebra}
  \label{fig:fund}
\end{figure}
\hfill

Strang goes further in contextualizing the SVD factorization in the
above diagram (\cite{strang93}), by noting that the columns of the
matrices $V$ and $U$, actually contain the basis of these four
subspaces:

\begin{itemize}
\item The orthogonal matrix $V$ contains a basis for the row space
  \C{\trans{A}} in the first $r$ columns, and a basis for the null
  space \N{A} in the last $n-r$ columns. We showed this already while
  proving \cref{thm:SVD2}.
\item The orthogonal matrix $U$ contains a basis for the column space
  \C{A} in the first $r$ columns, and a basis for the left null
  space \N{\trans{A}} in the last $m-r$ columns.
\end{itemize}
\hfill

The last observation makes the SVD factorization even more astonishing
and intriguing: not only it allows one to understand the true nature of
an arbitrary matrix $A$, by explicitly giving the two change of basis
that make $A$ a positive diagonal matrix $\Sigma$ (having just
compressions and expansions). Also, if we consider a basis as a
representation of a vector space; then the matrices $V$ and $U$ of the
SVD factorization can be
considered a representation of the four subspaces generated by that
particular matrix $A$. Putting together the three matrices as in $A =
U\Sigma\trans{V}$, gives the truly complete picture about the effects
of transformation $A$.


\documentclass{standalone}
\begin{document}
	\chapter{Quadratic Equations}
	\section{Definition}
	\quad A quadratic equation is of the form $ax^2+bx+c=0$ where $a,b,c \in \mathbb{R},\quad a\neq0$. These can be solved algebraically using one of the following methods:
	\begin{itemize}
		\item{Fractions}
		\item{Completing the square}
		\item{The quadratic formula\footnote[3]{\qe}}
	\end{itemize}
	\section{Nature of roots of the Quadratic Equation}
	\quad Any quadratic equation has in general two roots, namely \qe.
	The quantity $b^2-4ac$ determines the nature of these roots.
	\begin{itemize}
		\item{$b^2-4ac > 0\colon$ Equation holds two real and distinct roots.}
		\item{$b^2-4ac = 0\colon$ Equation holds two equal\footnote[4]{It is implied that they are real.} roots.}
		\item{$b^2-4ac > 0\colon$ Equation holds two complex roots.}
	\end{itemize}
	\quad Thus, the quantity $b^2-4ac$ discriminates among the type of roots that a quadratic equation may have. Therefore it is called the discriminant.
	\begin{example}
		Determine, without solving the nature of the following function.
	\end{example}
	\begin{alignat*}{2}
		& Let        & f(x) =                 & 2x^2+3x-17                              \\
		& \implies   & b^2-4ac                & =3^2-4(2)(-17)                          \\
		&            &                        & = 145                                   \\
		&            &                        & > 0                                     \\
		& \therefore & \text{Roots of $f(x)$} & \in \mathbb{R} \text{ and are distinct}
	\end{alignat*}
	\hrulefill
	\begin{example}
		Determine the value of $p$ if $ px^2-10x+1 = 0 $ has two equal roots
	\end{example}

	Given that the equation has two equal roots:

	\begin{alignat*}{2}
		&          & b^2-4ac                        & = 0 \\
		& \implies & 100-4p                         & = 0 \\
		&          & \boxed{\therefore \quad p = 5} &
	\end{alignat*}

	\newpage
	\section{Roots and Coefficients of a Quadratic Equation}
	\subsection{Proof}
	\emph{Consider a general quadratic equation:}
	\begin{alignat*}{2}
		&            & ax^2+bx+c                            & = 0 \tag{1..}                              \\
		& \implies   & x^2 +\frac{bx}{a} + \frac{c}{a}      & = 0                                        \\
		\intertext{Let $\alpha$  and $\beta$  be the roots:}
		& \implies   & (x-\alpha)(x-\beta)                  & = 0                                        \\
		& \implies   & x^2 -\beta x -\alpha x + \alpha\beta & = 0                                        \\
		& \implies   & x^2-(\alpha + \beta)x + \alpha\beta  & = 0\tag{2..}                               \\
		\intertext{Since (1..) and (2..) are identical:}
		& \implies   & x^2 +\frac{bx}{a} + \frac{c}{a}      & \equiv x^2-(\alpha + \beta)x + \alpha\beta \\
		& \therefore & \alpha + \beta                       & = \frac{-b}{a}                             \\
		&            & \alpha\beta                          & = \frac{c}{a}
	\end{alignat*}
	\hrulefill

	\begin{example}
		Write down the quadratic equation whose roots have a sum of 7 \& a product of 5.
	\end{example}
	\begin{alignat*}{2}
		&   &        & x^{2} - (\alpha + \beta) + (\alpha\beta) \\
		&   & =\quad & x^2 - 7x + 7
	\end{alignat*}
	\hrulefill
	\begin{example}
		The roots of the equation $2x^2 + 5x -1$ are $\alpha$ and $\beta$. Find the equation whose roots are $\frac{1}{\alpha} $ \& $\frac{1}{\beta}$
	\end{example}
	\begin{alignat*}{2}
		&   & \alpha + \beta & = \frac{-5}{2} \\
		&   & \alpha\beta    & = \frac{-1}{2}
	\end{alignat*}
	\newpage
	\begin{multicols}{2}
		\textit{Sum of roots:}
		\begin{alignat*}{2}
			&   &   & \frac{1}{\alpha} + \frac{1}{\beta} \\
			&   & = & \frac{\alpha+\beta}{\alpha\beta}   \\
			&   & = & \frac{-5}{2} \div \frac{-1}{2}     \\
			&   & = & 5
		\end{alignat*}
		\textit{Product of roots:}

		\begin{alignat*}{2}
			\\&&&\frac{1}{\alpha\beta}\\
			&   & = & \frac{-2}{1} \\
			&   & = & -2
		\end{alignat*}
	\end{multicols}
	\[
	\therefore \quad f(x) = x^2-5x-2
	\]
	\newpage

	\end{document}


\subsection{Tuning to phase shifts on the lattice}\label{sec:tuning}

Observables in a FV lattice theory must converge against their physical counterpart in the IV and continuum limit.
Because the hamiltonian is implmented in a discrete finite volume, we must match the interactions to a discrete finite observable.
The implementation is well behaved if, after the matching of one observable (input), the IV and continuum limit for other observales (prediction) converges against the physical counterpart.
The implementation of the hamiltonian includes only one parameter, the strength of the contact interaction, which we match against scattering information.
The goal is to reproduce the entire phase shifts as a function of scattering momenta by matching the interactions to just one scattering momentum point.
The FV scattering spectrum can be exactly computed by using L\"{u}scher's formalism \textit{in reverse}: we compute the intersections of the phase shifts with the zeta function to extract the finit volume spectrum (see also \Figref{tuning})
\begin{equation}
    S^\spherical_D(x_i)
    =
    \frac{1}{\pi^2 L^{D-2}}
    \begin{cases}
        -\frac{1}{a_{03} \pi}   & D=3\\
        \frac{1}{\pi}\log(x_i) + \frac{2}{\pi} \log \left(\frac{ 2 \pi R_{02}}{L}\right) & D=2\\
        2 a_{01}   & D=1
    \end{cases}
	\, .
\end{equation}

We compute the FV discrete spectrum $\{E_i(\epsilon)\}$, adjust the contact interaction such that the ground state energy $E_0(\epsilon)$ matches the first zero of the zeta function $x_0 = 2 \mu E_0(\epsilon) L^2 / (2 \pi)^2$  and repeat the procedure for different lattice spacings.