\section{Simplicial Homology}
\subsection{Oriented Simplices and Boundary Homomorphism}
We slightly modify our definition of simplex by associating each simplex with an orientation.
For a simplex $\langle v_0,\ldots,v_n\rangle$, we can shuffle the vertices around which defines an action of $S_{n+1}$ on it.
Write $A_{n+1}\unlhd S_{n+1}$ as the alternating group, then it also acts on the simplex by restriction.
An orientation on $\sigma=\langle v_0,\ldots,v_n\rangle$ is a choice of ordering defined up to the action of $A_{n+1}$.
So we can now view $\langle v_0,\ldots,v_n\rangle$ as a simplex with an orientation.
\begin{example}
    $0$-simplex does not have a notion of orientation.\\
    There are two possible orderings of $1$-simplex, namely $\langle v_0,v_1\rangle$ and $\langle v_1,v_0\rangle$ which have different orientation.\\
    For $2$-simplices, again there are two orientations and they are $\langle v_0,v_1,v_2\rangle=\langle v_1,v_2,v_0\rangle=\langle v_2,v_0,v_1\rangle$ and $\langle v_1,v_0,v_2\rangle=\langle v_0,v_2,v_1\rangle=\langle v_2,v_1,v_0\rangle$.
\end{example}
So afterwards when we mention simplex we mean oriented simplex.
\begin{definition}
    Let $K$ be a simplicial complex, we define the group $C_n(K)$ of $n$-chains to be the free abelian group generated by the simplices of dimension $n$ in $K$, i.e.
    $$C_n(K)=\bigoplus_{\sigma\in K,\dim\sigma=n}\langle\sigma\rangle$$
    where $\langle\sigma\rangle$ is the free group generated by $\sigma$.
\end{definition}
We will convention that we have made our choice of some orientation.
For a simplex $\sigma$ in this orientation, the same simplex in the other orientation is denoted $\bar\sigma$.
In $\langle\sigma\rangle$, we identify $\bar\sigma$ by $-\sigma$.
\begin{definition}
    The $n^{th}$ boundary homomorphism is a homomorphism $\partial=\partial_n:C_n(K)\to C_{n-1}(K)$ induced by
    $$\partial_n\langle v_0,\ldots,v_n\rangle = \sum_{i=0}^n(-1)^i\langle v_1,\ldots,\hat{v}_i,\ldots,v_n\rangle$$
    where $\langle v_1,\ldots,\hat{v}_i,\ldots,v_n\rangle=\langle v_0,\ldots,v_{i-1},v_{i+1},\ldots,v_n\rangle$.
\end{definition}
\begin{example}
    $\partial\langle v_0,v_1\rangle=\langle v_1\rangle-\langle v_0\rangle$.
    $\partial\langle v_0,v_1,v_2\rangle=\langle v_1,v_2\rangle-\langle v_0,v_2\rangle+\langle v_0,v_1\rangle=\langle v_1,v_2\rangle+\langle v_2,v_0\rangle+\langle v_0,v_1\rangle$ which is indeed the (oriented) topological boundary of the simplex $\langle v_0,v_1,v_2\rangle$.
\end{example}
\begin{remark}
    We have $\partial\bar\sigma=-\partial\sigma$.
\end{remark}
\subsection{The Homology Groups of Simplicial Complexes}
\begin{definition}
    Let $K$ be a simplicial complex and $n\in\mathbb Z$, the group of $n$-cycles of $K$ is $Z_n(K)=\ker\partial_n$.
    The group of $n$-boundaries is $B_n(K)=\operatorname{Im}\partial_{n+1}$.
\end{definition}
\begin{lemma}
    $B_n(K)\subset Z_n(K)$, or in other words $\partial_{n-1}\circ\partial_n=0$.
\end{lemma}
\begin{proof}
    Pick an $n$-simplex $\langle v_0,\ldots,v_n\rangle$, then
    \begin{align*}
        \partial_{n-1}\circ\partial_n(\langle v_0,\ldots,v_n\rangle)&=\partial_{n-1}\left( \sum_{i=0}^n(-1)^i\langle v_0,\ldots,\hat{v}_i,\ldots,v_n\rangle \right)\\
        &=\sum_{i=0}^n(-1)^i\partial_{n-1}(\langle v_0,\ldots,\hat{v}_i,\ldots,v_n\rangle)\\
        &=\sum_{j<i}(-1)^j(-1)^i\langle v_0,\ldots,\hat{v}_j,\ldots,\hat{v}_i,\ldots,v_n\rangle\\
        &\quad+\sum_{j>i}(-1)^{j-1}(-1)^i\langle v_0,\ldots,\hat{v}_i,\ldots,\hat{v}_j,\ldots,v_n\rangle\\
        &=0
    \end{align*}
    as desired.
\end{proof}
\begin{definition}
    The $n^{th}$ (simplicial) homology group of the simplicial complex $K$ is defined as $H_n(K)=Z_n(K)/B_n(K)$.
\end{definition}
\begin{example}
    Take the simplicial complex $K$ generated by boundary of a $2$-simplex (which can be taken as a triangulation of $S^1$).
    Then $C_0(K)\cong C_1(K)\cong\mathbb Z^3$ and $C_n(K)=0$ for all $n>1$.
    So we only need to understand $\partial_1$.
    Say its $0$-simplices are $\langle v_0\rangle,\langle v_1\rangle,\langle v_2\rangle$ and $1$-simplices are $\langle v_0,v_1\rangle,\langle v_1,v_2\rangle,\langle v_2,v_0\rangle$, then $\partial_1:C_1(K)\to C_0(K)$ maps
    \begin{align*}
        \langle v_0,v_1\rangle &\mapsto \langle v_1\rangle-\langle v_0\rangle\\
        \langle v_1,v_2\rangle &\mapsto \langle v_2\rangle-\langle v_1\rangle\\
        \langle v_2,v_0\rangle &\mapsto \langle v_0\rangle-\langle v_2\rangle
    \end{align*}
    So if we take the free abelian groups as free $\mathbb Z$-modules generated by the simplices, then $\partial_1$ has the matrix
    $$\begin{pmatrix}
        -1&0&1\\
        1&-1&0\\
        0&1&-1
    \end{pmatrix}$$
    So $Z_1(K)=\ker\partial_1=\langle(1,1,1)\rangle=\langle \langle v_0,v_1\rangle +\langle v_1,v_2\rangle + \langle v_2,v_0\rangle\rangle$ and $B_1(K)=\ker\partial_2=0$, therefore $H_1(K)\cong\mathbb Z$.\\
    As for the $H_0(K)$, we have $Z_0(K)=\ker\partial_0=C_0(K)\cong\mathbb Z^3$ and $B_0(K)=\operatorname{Im}\partial_1\cong\langle (-1,1,0),(0,-1,1)\rangle$, hence $H_0(K)\cong\mathbb Z^3/\langle (-1,1,0),(0,-1,1)\rangle\cong\mathbb Z$.
    And $H_n(K)=0$ for $n>1$.
\end{example}
\begin{example}
    Take $L$ the simplicial complex generated by the $2$-simplex (so it is a solid triangle which is a triangulation of the closed unit disk).
    Then $C_0(L)=C_0(K),C_1(L)=C_1(K)$ but $C_2(L)=\langle\langle v_0,v_1,v_2\rangle\rangle$ where $K$ is as in the previous example.
    Now $\partial_2(\langle v_0,v_1,v_2\rangle)=\langle v_0,v_1\rangle+\langle v_1,v_2\rangle+\langle v_2,v_0\rangle$ which is the generator of $\ker\partial_1$, therefore $Z_1(L)=B_1(L)$ and hence $H_1(L)=0$.
    Note that $L,K$ coincides on $0$ and $1$-simplices they contain, so $H_0(L)\cong H_0(K)\cong\mathbb Z$.
    Now easily $Z_2(L)=\ker\partial_2=0$ and therefore $H_2(L)=0$, therefore $H_0(L)\cong\mathbb Z$ and $H_n(L)=0$ for any $n\neq 0$.
\end{example}
\begin{lemma}
    Let $K$ be a simplicial complex.
    If $d$ is the number of path components of $|K|$, then $H_0(K)\cong\mathbb Z^d$.
\end{lemma}
\begin{proof}
    Denote by $\pi_0(K)$ the set of path-connected components of $|K|$.
    Write $\mathbb Z[A]$ as the free abelian group generated by a set $A$, then $\mathbb Z[\pi_0(K)]\cong\mathbb Z^d$.
    Consider a $q:C_0(K)\to\mathbb Z[\pi_0(K)]$ sending a vertex $\langle v\rangle$ to the path component containing $v$ and extend it to a homomorphism.
    Then $q$ is surjective.
    Note that $\partial_0=0$, so $Z_0(K)=C_0(K)$, therefore $H_0(K)\cong C_0(K)/B_0(K)$, therefore it suffices to show that $\ker q=B_0(K)$.
    If $\langle v_0,v_1\rangle\in C_1(K)$, then $q\circ\partial_1(\langle v_0,v_1\rangle)=q(\langle v_1\rangle-\langle v_1\rangle)=0$ since $v_0$ and $v_1$ are joined by a path.
    This means that $B_0(K)\subset\ker q$.
    Conversely, $\ker q$ is generated by elements of $C_0(K)$ of the form $\langle w\rangle-\langle v\rangle$ where $v,w$ are in the same path components of $|K|$.
    But then there is a sequence of vertices $v=v_1,v_2,\ldots,v_n=w$ in $K$ with $\langle v_i,v_{i+1}\rangle\in K$.
    Then
    $$\langle w\rangle-\langle v\rangle=(\langle v_n\rangle-\langle v_{n-1}\rangle)+(\langle v_{n-1}\rangle-\langle v_{n-2}\rangle)\cdots +(\langle v_2\rangle-\langle v_1\rangle)\in\operatorname{Im}\partial_1=B_0(K)$$
    Hence $\ker q\subset B_0(K)$.
    This completes the proof.
\end{proof}
\begin{remark}
    There is a very rough analogy between $\pi_1(|K|)$ and $H_1(K)$ as $B_1(K)$ is kind of homotopies between the loops in $Z_1(K)$.
    They are of course not the same as $H_1(K)$ is always abelian, but in fact, there does exist a certain connection as one can show that $H_1(K)\cong\pi_1(|K|)^{\operatorname{ab}}$.
\end{remark}
\subsection{Chain Maps and Homotopies}
We want to understand the maps on homology that is induced by maps of simplicial complexes.
\begin{definition}
    A chain complex $C_\bullet$ is a sequence of abelian groups $C_n,n\in\mathbb Z$ with homomorphisms $\partial_n:C_n\to C_{n-1}$ such that $\partial_{n-1}\circ\partial_n=0$ for all $n$.\\
    A chain map $f_\bullet:C_\bullet\to D_\bullet$ between chain complexes is a collection of homomorphisms $f_n:C_n\to D_n$ indexed by $n\in\mathbb Z$ such that
    \[
        \begin{tikzcd}
            C_n\arrow{r}{\partial_n}\arrow[swap]{d}{f_n}&C_{n-1}\arrow{d}{f_{n-1}}\\
            D_n\arrow[swap]{r}{\partial_n}&D_{n-1}
        \end{tikzcd}
    \]
    commutes for any $n$.
\end{definition}
Homology usually deals with the cases where nontrivial groups only occur at nonnegative $n$.
In these situations, we can just define $C_n$ for $n\ge 0$ and $\partial_n$ for $n\ge 1$ -- because those are what we care about -- and leave the rest of the groups and boundary maps to be zero.
This will be the case for the simplicial complexes.
\begin{definition}
    Given a chain complex $C_\bullet$, we define the group of $n$-cycles to be $Z_n(C_\bullet)=\ker\partial_n$ and the group of $n$-boundaries to be $B_n(C_\bullet)=\operatorname{Im}\partial_n$.
    Then $B_n(C_\bullet)\unlhd Z_n(C_\bullet)$, so we define the $n^{th}$ homology group is then $H_n(C_\bullet)=Z_n(C_\bullet)/B_n(C_\bullet)$.
\end{definition}
\begin{lemma}
    If $f_\bullet:C_\bullet\to D_\bullet$ is a chain map, then for any $n\in\mathbb Z$, we have a well-defined homomorphism $f_\ast:H_n(C_\bullet)\to H_n(D_\bullet)$ via $[c]\mapsto [f_n(c)]$ for $c\in Z_n(C_\bullet)$.
\end{lemma}
\begin{proof}
    Suffices to show that the map is well-defined.
    For $c\in Z_n(C_\bullet)$, we have $\partial_n\circ f_n(c)=f_{n-1}\circ\partial_n(c)=0$, so indeed $f_n(c)\in Z_n(D_\bullet)$.
    Also, if $c\in B_n(C_\bullet)$, then there is some $c'\in C_{n-1}$ such that $c=\partial_{n+1}(c')$, so $f_n(c)=f_n\circ\partial_{n+1}(c')=\partial_{n+1}\circ f_{n+1}(c')$, therefore $f_n(c)\in B_n(D_\bullet)$.
    Hence $f_\ast$ is well-defined.
\end{proof}
\begin{lemma}
    A simplicial map $f:K\to L$ induces a chain map $f_\bullet:C_\bullet(K)\to C_\bullet(L)$ via
    $$f_n:\sigma\mapsto\begin{cases}
        f(\sigma)\text{, if $\dim f(\sigma)=n$}\\
        0\text{, otherwise}
    \end{cases}$$
    for $\sigma\in K,\dim\sigma=n$.
    Hence for each $n\in\mathbb N$, $f_\bullet$ induces a homomorphism $f_\ast:H_n(K)\to H_n(L)$.
\end{lemma}
\begin{proof}
    We need to show that $\partial_n\circ f_n=f_{n-1}\circ\partial_n$.
    Suffices to demonstrate this on generators.
    Let $\sigma=\langle v_0,\ldots,v_n\rangle$.
    If $\dim f(\sigma)=n$, then $f(\sigma)=\langle f(v_0),\ldots,f(v_n)\rangle$, then there is a one-to-one correspondence between faces of $\sigma$ and fases of $f(\sigma)$, hence necessarily $f_{n-1}\circ\partial_n(\sigma)=\partial_n\circ f_n(\sigma)$.
    If $\dim f(\sigma)\le n-2$, then $f_{n-1}\circ\partial_n(\sigma)=0=\partial_n\circ f_n(\sigma)$.
    We are left with the case $\dim f(\sigma)=n-1$.
    Assume $f(v_0)=f(v_1)$ and $f(v_1),\ldots,f(v_n)$ are all distinct.
    In that case $f(\langle v_0,\ldots,v_n\rangle)=f(\langle v_1,\ldots,v_n\rangle)$.
    We know that $f_n(\sigma)=0$, so $\partial_n\circ f_n(\sigma)=0$.
    Now
    \begin{align*}
        f_{n-1}\circ\partial_n(\sigma)&=f_{n-1}\left(\sum_{i=0}^n(-1)^i\langle v_0,\ldots,\hat{v}_i,\ldots,v_n\rangle\right)\\
        &=\sum_{i=0}^n(-1)^if_{n-1}(\langle v_0,\ldots,\hat{v}_i,\ldots,v_n\rangle)\\
        &=f_{n-1}(\langle v_1,\ldots,v_n\rangle)-f_{n-1}(\langle v_0,v_2,\ldots,v_n\rangle)\\
        &=0
    \end{align*}
    Therefore $\partial_n\circ f_n(\sigma)=0=f_{n-1}\circ\partial_n(\sigma)$ too, which means $f_\bullet$ is indeed a chain map.
\end{proof}
\begin{remark}
    If $f:K\to L$ and $g:L\to M$ are simplicial maps, then $(g\circ f)_\ast=g_\ast\circ f_\ast$.
    Also, if $K$ is a simplicial complex, then $(\operatorname{id}_K)_\ast=\operatorname{id}_{H_n(K)}$.
\end{remark}
A natural question is that when do chain maps induce the same maps on homology.
\begin{definition}
    Let $f_\bullet,g_\bullet:C_\bullet\to D_\bullet$ be chain maps.
    A chain homotopy $h_\bullet$ between $f_\bullet$ and $g_\bullet$ is a collection of homomorphisms $h_n:C_n\to D_{n+1}$ such that $g_n(c)-f_n(c)=\partial_{n+1}\circ h_n(c)+h_{n-1}\circ\partial_n(c)$.
    We say $f_\bullet$ and $g_\bullet$ are chain homotopic, written as $f_\bullet\simeq g_\bullet$ if such $h_\bullet$ exists.
\end{definition}
\begin{lemma}
    If $f_\bullet\simeq g_\bullet:C_\bullet\to D_\bullet$, then $f_\ast=g_\ast$.
\end{lemma}
\begin{proof}
    Let $c\in Z_n(C_\bullet)$, then $g_n(c)-f_n(c)=\partial_{n+1}\circ h_n(c)+h_{n-1}\circ\partial_n(c)=\partial_{n+1}\circ h_n(c)\in B_n(D_\bullet)$, therefore $[g_n(c)]=[f_n(c)]$.
\end{proof}
\begin{example}
    Consider the triangle $K$ and the line segment $L$, both as simplicial complexes in the obvious way.
    Say the vertices in $K$ are $e_0,e_1,e_2$ and those in $L$ are $e_0,e_1$ and let $i:L\to K$ be the natural inclusion $e_0\mapsto e_0,e_1\mapsto e_1$, $r$ be the simplicial retraction $e_0\mapsto e_0,e_1\mapsto e_1,e_2\mapsto e_0$, both as simplicial maps.
    Now $r\circ i=\operatorname{id}_L$, but $i\circ r\neq \operatorname{id}_K$.
    However, we can define a chain homotopy between $(i\circ r)_\bullet$ and $\operatorname{id}_{C_\bullet(K)}$.
    This would be given by $h_\bullet$ which is everywhere zero except $h_0(\langle e_2\rangle)=\langle e_2,e_0\rangle$ and $h_1(\langle e_1,e_2\rangle)=-\langle e_0,e_1,e_2\rangle$ which works since
    \begin{align*}
        (\partial_1\circ h_0+h_{-1}\circ\partial_0)(\langle e_2\rangle)&=\partial_1(\langle e_2,e_0\rangle)\\
        &=\langle e_0\rangle-\langle e_2\rangle\\
        &=(i_0\circ r_0-\operatorname{id}_{C_0(K)})(\langle e_2\rangle)\\
        (\partial_2\circ h_1+h_0\circ\partial_1)(\langle e_1,e_2\rangle)&=\partial_2(-\langle e_0,e_1,e_2\rangle)+h_0(\langle e_2\rangle-\langle e_1\rangle)\\
        &=-\langle e_0,e_1\rangle-\langle e_1,e_2\rangle-\langle e_2,e_0\rangle+\langle e_2,e_0\rangle\\
        &=(i_1\circ r_1-\operatorname{id}_{C_1(K)})(\langle e_1,e_2\rangle)\\
        (\partial_3\circ h_2+h_1\circ\partial_2)(\langle e_0,e_1,e_2\rangle)&=h_1(\langle e_0,e_1\rangle+\langle e_1,e_2\rangle+\langle e_2,e_0\rangle)\\
        &=-\langle e_0,e_1,e_2\rangle\\
        &=(i_2\circ r_2-\operatorname{id}_{C_2(K)})(\langle e_0,e_1,e_2\rangle)
    \end{align*}
    Therefore $(i\circ r)_\ast=i_\ast\circ r_\ast$ would just be the identity on $H_n(K)$.
    In particular, $r_\ast$ is an isomorphism on the homology groups.
\end{example}
\begin{definition}
    A simplicial complex $K$ is a cone if there is a vertex $x_0$ such that for all other simplices $\tau\in K$, there exists $\sigma\in K$ such that $x_0\in\sigma$ and $\tau\le\sigma$.
\end{definition}
Perhaps nonsurprisingly,
\begin{lemma}
    If $K$ is a cone, then $H_0(K)\cong\mathbb Z$ and $H_n(K)=0$ if $n\neq 0$.
\end{lemma}
\begin{proof}
    Let $i:\{\langle x_0\rangle\}\to K$ be the obvious inclusion and $r:K\to\{\langle x_0\rangle\}$ be constant.
    Then $r\circ i=\operatorname{id}_{\{\langle x_0\rangle\}}$, therefore $r_\ast\circ i_\ast=\operatorname{id}_{H_n(\{\langle x_0\rangle\})}$.
    We shall show that $i_\ast\circ r_\ast=\operatorname{id}_{H_n(K)}$ which implies $H_n(K)\cong H_n(\{\langle x_0\rangle\})$ from where the result follows.\\
    We will build a chain homotopy between $\operatorname{id}_{C_\bullet(K)}$ and $i_\bullet\circ r_\bullet$ where $i_\bullet$ and $r_\bullet$ are the induced chain maps.
    Let $\sigma=\langle v_0,\ldots,v_n\rangle\in K$, then we define
    $$h_n(\sigma)=\begin{cases}
        0\text{, if $x_0\in\sigma$}\\
        \langle x_0,v_0,\ldots,v_n\rangle\text{, otherwise}
    \end{cases}$$
    which is well-defined as $K$ is a cone with vertex $x_0$.
    We want to show that $\partial_{n+1}\circ h_n+h_{n-1}\circ\partial_n=\operatorname{id}_{C_n(K)}-i_n\circ r_n$.
    Suppose $n>0$ and $x_0\notin\sigma$, then
    \begin{align*}
        &\quad(\partial_{n+1}\circ h_n+h_{n-1}\circ\partial_n)(\sigma)\\
        &=\partial_{n+1}(\langle x_0,v_0,\ldots,v_n\rangle)+h_{n-1}\left( \sum_{i=0}^n(-1)^i\langle v_0,\ldots,\hat{v}_i,\ldots,v_n \rangle\right)\\
        &=\langle v_0,\ldots,v_n\rangle+\sum_{i=0}^n(-1)^{i+1}\langle x_0,v_0,\ldots,\hat{v}_i,\ldots,v_n \rangle\\
        &\quad+\sum_{i=0}^n(-1)^i\langle x_0,v_0,\ldots,\hat{v}_i,\ldots,v_n \rangle\\
        &=\langle v_0,\ldots,v_n\rangle=\sigma\\
        &=(\operatorname{id}_{C_n(K)}-i_n\circ r_n)(\sigma)
    \end{align*}
    which works.
    If $n>0$ but $x_0\in\sigma$, then $x_0=v_j$ for some $j$.
    Consequently,
    \begin{align*}
        &\quad(\partial_{n+1}\circ h_n+h_{n-1}\circ\partial_n)(\sigma)\\
        &=0+h_{n-1}\left( \sum_{i=0}^n(-1)^i\langle v_0,\ldots,\hat{v}_i,\ldots,v_n \rangle\right)\\
        &=(-1)^j\langle x_0=v_j,v_0,\ldots,\hat{v}_j,\ldots,v_n\rangle\\
        &=\langle v_0,\ldots,v_n\rangle=\sigma\\
        &=(\operatorname{id}_{C_n(K)}-i_n\circ r_n)(\sigma)
    \end{align*}
    The case $n=0$ is trivial.
    Therefore $h_\bullet$ is indeed a chain homotopy as desired.
\end{proof}
\begin{example}
    Any $n$-simplex $K$ is a cone, so by the lemma
    $$H_n(K)\cong\begin{cases}
        \mathbb Z\text{, if $n=0$}\\
        0\text{, otherwise}
    \end{cases}$$
    Take $L=\partial\sigma_n$ be the boundary of the $n$-simplex, then $|L|\cong S^{n-1}$ for $n\ge 2$.
    The obvious inclusion $L\hookrightarrow K$ of simplicial complexes induces a chain map
    \[
        \begin{tikzcd}
            0\arrow{r}&0\arrow{r}\arrow{d}&C_{n-1}(L)\arrow{r}{\partial_{n-1}}\arrow[equal]{d}&\cdots\arrow{r}{\partial_1}&C_0(L)\arrow{r}\arrow[equal]{d}&0\\
            0\arrow{r}&C_n(K)\arrow[swap]{r}{\partial_n}&C_{n-1}(K)\arrow[swap]{r}{\partial_{n-1}}&\cdots\arrow[swap]{r}{\partial_1}&C_0(K)\arrow{r}&0
        \end{tikzcd}
    \]
    So evidently $H_d(L)\cong H_d(K)$ for $d\le n-2$.
    Since $H_{n-1}(K)=0$, we have $Z_{n-1}(L)=Z_{n-1}(K)=B_{n-1}(K)$.
    Also $B_{n-1}(L)=0$, therefore $H_{n-1}(L)\cong Z_{n-1}(L)=B_{n-1}(K)$.
    But this is easy enough to calculate:
    $K$ only has one $n$-simplex $\sigma$, so $C_n(K)=\mathbb Z\sigma$ and hence $B_{n-1}\cong\mathbb Z$ as $\partial_n$ has to be injective.
    Thus
    $$H_d(L)=\begin{cases}
        \mathbb Z\text{, if $d=0$ or $d=n-1$}\\
        0\text{, otherwise}
    \end{cases}$$
    This means homology can actually detect ``higher dimensional holes''.
\end{example}
\subsection{Continuous Maps and Homotopies}
For a map $\phi:|K|\to|L|$, we want to associate to it a homomorphism $\phi_\ast:H_n(K)\to H_n(L)$.
Note that a chief difficulty in this is that $\phi$ may not contain much information about the structures of $K,L$ as simplicial complexes, since it is just a continuous map between topological spaces.
The idea is to use a simplicial approximation.
That is, instead of looking for $\phi_\ast:H_n(K)\to H_n(L)$ directly, we seek $\phi_\ast:H_n(K^{(r)})\to H_n(L^{(r)})$ for sufficiently large $r$ and show that $H_n(K^{(r)})\cong H_n(K)$ for any simplicial complex $K$ and $r\in\mathbb N$.
Eventually, we will show that $H_n(K)$ only depends on $|K|$.\\
First step on that journey is the notion of homotopy on simplicial maps.
\begin{definition}
    Two simplicial maps $f,g:K\to L$ are contiguous if, for every $\sigma\in K$, there exists some $\tau\in L$ such that $f(\sigma)$ and $g(\sigma)$ are both faces of $\tau$.
\end{definition}
\begin{remark}
    Suppose given $\phi:|K|\to |L|$ with $f,g:K\to L$ different simplicial approximations to $\phi$.
    Then choose any $x\in\sigma^\circ,\phi(x)\in\tau^\circ$, we know that $f(\sigma)\le\tau$ and $g(\sigma)\le\tau$ and hence $f,g$ are contiguous.
\end{remark}
\begin{lemma}
    If $f,g:K\to L$ are contiguous, then $f_\ast=g_\ast:H_n(K)\to H_n(L)$ for all $n$.
\end{lemma}
\begin{proof}
    We shall construct a chain homotopy.
    Fix a total order $<$ on vertices of $K$ and use the convention that $\sigma=\langle v_0,\ldots,v_n\rangle$ is oriented in such a way that $v_0<\ldots<v_n$.
    We write $\langle v_0,\ldots,v_n\rangle=0$ if $v_0,\ldots,v_n$ are not in general position.
    Easy to see this is compatible with everything.
    Define $h_n:C_n(K)\to C_{n+1}(L)$ by
    $$h_n(\langle v_0,\ldots,v_n\rangle)=\sum_{i=0}^n(-1)^i\langle f(v_0),\ldots,f(v_i),g(v_i),\ldots,g(v_n)\rangle$$
    Note that whenever the summand is nonzero, it would be an $(n+1)$-simplex of $L$ since $f,g$ are contiguous.
    We have some calculations to do.
    \begin{align*}
        (\partial\circ h+h\circ\partial)(\sigma)&=\partial\left( \sum_{i=0}^n(-1)^i\langle f(v_0),\ldots,f(v_i),g(v_i),\ldots,g(v_n)\rangle \right)\\
        &\quad+h\left( \sum_{i=0}^n(-1)^i\langle v_0,\ldots,\hat{v}_i,\ldots,v_n\rangle \right)\\
        &=\sum_{i\le j}(-1)^{i+j}\langle f(v_0),\ldots,\widehat{f(v_i)},\ldots,f(v_j),g(v_j),\ldots,g(v_n)\rangle\\
        &\quad-\sum_{i\ge j}(-1)^{i+j}\langle f(v_0),\ldots, f(v_j),g(v_j),\ldots,\widehat{g(v_i)},\ldots,g(v_n)\rangle\\
        &\quad+\sum_{j<i}(-1)^{i+j}\langle f(v_0),\ldots, f(v_j),g(v_j),\ldots,\widehat{g(v_i)},\ldots,g(v_n)\rangle\\
        &\quad-\sum_{j>i}(-1)^{i+j}\langle f(v_0),\ldots,\widehat{f(v_i)},\ldots,f(v_j),g(v_j),\ldots,g(v_n)\rangle\\
        &=\sum_{i=0}^n\langle f(v_0),\ldots,f(v_{i-1}),g(v_i),\ldots,g(v_n)\rangle\\
        &\quad-\sum_{i=0}^n\langle f(v_0),\ldots,f(v_i),g(v_{i+1}),\ldots,g(v_n)\rangle\\
        &=\langle g(v_0),\ldots,g(v_n)\rangle-\langle f(v_0),\ldots,f(v_n)\rangle\\
        &=g(\sigma)-f(\sigma)
    \end{align*}
    as desired.
\end{proof}
\begin{lemma}
    Let $K$ be a simplicial complex and $K'$ be its barycentric subdivision.
    A simplicial map $s:K'\to K$ is a simplicial approximation to $\operatorname{id}_{|K|}$ iff for every $\sigma\in K$, $s(\hat\sigma)$ is a vertex of $\sigma$.
    Also, such $s$ always exists.
\end{lemma}
\begin{proof}
    Let $s:K'\to K$ be a simplicial approximation to $\operatorname{id}_{|K|}$, which just means $\operatorname{id}_{|K|}(\operatorname{St}_{K'}(\hat\sigma))\subset\operatorname{St}_K(s(\hat\sigma))$.
    In particular, $\sigma\circ\subset\operatorname{St}_K(s(\hat\sigma))$, therefore $s(\hat\sigma)$ is a vertex of $\sigma$.\\
    Conversely, suppose $s(\hat\sigma)$ be a vertex of $\sigma$ for any $\sigma\in K$.
    Choose any $\tau'\in K'$ with $\tau'^\circ\subset\operatorname{St}_{K'}(\hat\sigma)$ (i.e. $\hat\sigma$ is a vertex of $\tau'$).
    Then $\tau'^\circ$ is contained in the interior of a simplex $\tau\in K$ such that $\sigma\le\tau$.
    Thus $s(\hat\sigma)$ is also a vertex of $\tau$.
    But $\tau'^\circ\subset\tau^\circ\subset\operatorname{St}_K(s(\hat\sigma))$.
    But such $\tau'^\circ$ necessarily cover $\operatorname{St}_{K'}(\hat\sigma)$, therefore $\operatorname{id}_{|K|}(\operatorname{St}_{K'}(\hat\sigma))\subset\operatorname{St}_K(s(\hat\sigma))$ as desired.\\
    To see such an $s$ exists, we simply just need to send $\hat\sigma$ to an arbitrarily chosen vertex of $\sigma$ which, as one can verify, works.
\end{proof}
\begin{proposition}\label{barycentric_iso_homol}
    Let $s:K'\to K$ be the simplicial approximaion to the identity obtained like in the proof above, then $s_\ast:H_n(K')\to H_n(K)$ is an isomorphism for all $n$.
\end{proposition}
We will postpone this proof until more machinery is developed.
But let us see some implications first.
\begin{corollary}
    Let $K$ be a simplicial complex, then for all $r$, there is a canonical isomorphism $H_n(K)\cong H_n(K^{(r)})$.
\end{corollary}
\begin{proof}
    Suffices to make the choice for $r=1$.
    Choose a simplicial approximation $s:K'\to K$ to the identity on $|K|$ which induces an ismorphism $s_\ast:H_n(K')\to H_n(K)$ by the preceding proposition.
    To see this is canonical, we shall show that this isomorphism is independent of the choice of $s$.
    But this is obvious since any other choice $s'$ is contiguous with $s$.
\end{proof}
We write $\nu_{K,r,s}:H_n(K^{(r)})\to H_n(K^{(s)})$ as the canonical ismorphism for $r\ge s$ and write $\nu_{K,r}=\nu_{K,r,0}$.
Then $\nu_{K,r_2,r_3}\circ \nu_{K,r_1,r_2}=\nu_{K,r_1,r_3}$.
\begin{proposition}
    To each continuous map $f:|K|\to|L|$, there is an associated homomorphism $f_\ast:H_n(K)\to H_n(L)$ given by $f_\ast=s_\ast\circ\nu_{K,r}^{-1}$ where $s:K^{(r)}\to L$ is a simplicial approximation to $f$.
    This homomorphism does not depend on the choice of $r$ or $s$.
    Furthermore, if $g:|M|\to |K|$ is continuous for some other simplicial complex $M$, then $(f\circ g)_\ast=f_\ast\circ g_\ast$.
\end{proposition}
\begin{proof}
    We already know that the homomorphism does not depend on the choice of $s$.
    If $s:K^{(r)}\to L,t:K^{(q)}\to L$ are both simplicial approximations to $f$ where WLOG $r\ge q$, then let $a:K^{(r)}\to K^{(q)}$ be a simplicial approximation to the identity on $|K|=|K^{(q)}|$.
    Now $s,t\circ a:K^{(r)}\to L$ are both simplicial approximations to $f$, hence are contiguous and induces the same homomorphism $s_\ast=(t\circ a)_\ast=t_\ast\circ a_\ast=t_\ast\circ\nu_{K,r,q}$, so $s_\ast\circ\nu_{K,r}^{-1}=t_\ast\circ\nu_{K,r,q}\circ\nu_{K,r}^{-1}=t_\ast\circ\nu_{K,q}$ as desired.\\
    Now let $s:K^{r}\to L$ and $t:M^{q}\to K^{r}$ be simplicial approximations to $f,g$ respectively (here we used $|K|=|K^{(r)}|$).
    Then $s\circ t$ is a simplicial approximation of $f\circ g$, so
    $$(f\circ g)_\ast=(s\circ t)_\ast\circ\nu_{M,q}^{-1}=s_\ast\circ t_\ast\circ \nu_{M,q}^{-1}=(s_\ast\circ\nu_{K,r}^{-1})\circ(\nu_{K,r}\circ t_\ast\circ\nu_{M,q}^{-1})=f_\ast\circ g_\ast$$
    as desired.
\end{proof}
And now we finally arrive at:
\begin{corollary}
    If $|K|\cong|L|$, then $H_n(K)\cong H_n(L)$.
\end{corollary}
\begin{proof}
    Immediate.
\end{proof}
We can do even better.
\begin{lemma}
    If $L$ is a simplicial complex residing in $\mathbb R^m$, then there exists $\epsilon=\epsilon(L)>0$ such that if $f,g:|K|\to|L|$ satisfies $|f(x)-g(x)|<\epsilon$ for any $x\in |K|$, then $f_\ast=g_\ast$.
\end{lemma}
\begin{proof}
    The set $\{\operatorname{St}_L(w):w\in L\}$ forms an open cover of $|L|$.
    So by Lebesgue number lemma, there exists $\epsilon>0$ such that each ball of radius $2\epsilon$ in $L$ lies in some $\operatorname{St}_L(w)$.
    We take $\epsilon(L)=\epsilon$.
    Let $f,g:|K|\to|L|$ as in the statement and consider the open cover of $|K|$ given by $\{f^{-1}(B_\epsilon(y)):y\in L\}$ which admits $\delta>0$ such that each $B_\delta(x)$ is contained in some member of this cover again by Lebesgue number lemma.
    This would mean that $f(B_\delta(x))\subset B_\epsilon(x)$, so $g(B_\delta(x))\subset B_{2\epsilon}(y)$.
    Choose some large $r$ such that $\operatorname{mesh}(K^{(r)})<\delta/2$, then for each vertex $v\in K^{(r)}$, the diameter of $\operatorname{St}_{K^{(r)}}(v)$ is strictly less than $\delta$, so both $f(\operatorname{St}_{K^{(r)}}(v))$ and $g(\operatorname{St}_{K^{(r)}}(v))$ are contained in some $\operatorname{St}_L(w)$.
    Set $s(v)=w$, then $s$ is a simplicial approximation to both $f$ and $g$, hence $f_\ast=s_\ast\circ \nu_{K,r}^{-1}=g_\ast$.
\end{proof}
\begin{theorem}
    If two maps $f,g:|K|\to|L|$ are homotopic, then $f_\ast=g_\ast$.
\end{theorem}
\begin{proof}
    Let $H:|K|\times I\to |L|$ be the homotopu between $f,g$, so $H(\cdot,0)=f,H(\cdot,1)=g$.
    As $|K|\times I$ is compact, $H$ is uniformly continuous.
    Thus for $\epsilon=\epsilon(L)$ as in the preceding lemma, there is some $\delta>0$ such that $|H(x,s)-H(x,t)|<\epsilon$ whenever $|s-t|<\delta$.
    Now choose $0=t_0<t_1<\ldots<t_k=1$ such that $t_i-t_{i-1}<\delta$ for any $i$ and let $f_i(x)=H(x,t_i)$.
    By construction $|f_i(x)-f_{i-1}(x)|<\epsilon$ for any $x\in |K|$, therefore $(f_i)_\ast=(f_{i-1})_\ast$ for all $i$.
    In particular, $f_\ast=(f_0)_\ast=\cdots=(f_k)_\ast=g_\ast$.
\end{proof}
\begin{corollary}
    If $|K|,|L|$ are homotopy equivalent, then $H_n(K)\cong H_n(L)$ for all $n$.
\end{corollary}
\begin{proof}
    Follows directly.
\end{proof}
\begin{definition}
    We write $H_n(X)=H_n(K)$ if $X=|K|$.
\end{definition}


\label{sec:Averaging}

A {\it mesh catalog} is a set $\{ \M_0 \ldots \M_{n-1} \}$
of $n$ meshes which are all embeddings
of the same simplicial complex
(that is, they have the same vertices, edges, and faces,
but the vertex positions may be different).

\subsection{Linear weights}
\label{sec:Linear-weights}

We can compute a {\it weighted average mesh} by averaging the vertex positions:
\begin{equation}
\p_i(\w) = \sum_{j=0}^{n-1} w_j \p_{ij}
\end{equation}
where $\p_{i}; (i=0 \ldots m-1)$ is the position of the $i$th vertex of the average mesh,
and $\p_{ij}$ is the position of the $i$th vertex of the $j$th catalog element.

In {\it catalog fitting} we fit a registered weighted average mesh,
to a set of data, $\{ \d_k \in \Reals^3; k=0 \ldots p-1 \}$
by minimizing
\begin{equation}
f(\M(\Tr,\w)) = \sum_{k=0}^{p-1} \| \d_k - \Pr_{\M(\Tr,\w)} (\d_k) \|^2 ,
\end{equation}
over a family of registration transforms $\{\Tr\}$
and vectors of weights $\w \in \Reals^n$.
The vertex positions $\p(\Tr,\w) = (\p_0(\Tr,\w) \ldots  \p_{n-1}(\Tr,\w))$
of the registered average mesh, $\M(\Tr,\w)$, are given by:
\begin{equation}
\p_i(\Tr,\w) = \Tr ( \sum_{j=0}^{n-1} w_j \p_{ij} )
\end{equation}

The partial derivative with respect to $\Tr$
is:
\begin{eqnarray}
\De{\Tr}{f(\M(\Tr,\w))}{\Tr^0,\w^0}
& = &
\De{\p}{f(\M))}{\M(\Tr^0,\w^0)}
\circ
\De{\Tr}{\p(\Tr,\w)}{\Tr^0,\w^0}
\\
& = &
\De{\p}{f(\M))}{\M(\Tr^0,\w^0)}
\circ
\De{\Tr}{\Tr\p(\w^0)}{\Tr^0}
\nonumber
\end{eqnarray}

$\De{\p}{f(\M))}{\M(\Tr^0,\w^0)}$ is the derivative of $f$ with respect to
the vertex positions of of the registered average mesh.
It's value for data fitting distance functions
is given in \autoref{sec:data-fitting}.

$\Df{\Tr}{\Tr\p}$ is given in \autoref{sec:Transforms}
for various families of transforms $\{\Tr\}$: affine, euclidean, and rigid.

Using the chain rule, the partial derivative with respect to $\w$ is:
\begin{eqnarray}
\De{\w}{f(\M(\Tr,\w))}{\Tr^0,\w^0}
& = &
\De{\w}{f(\Tr(\M(\w)))}{\Tr^0,\w^0}
\\
& = &
\De{\p}{f(\M))}{\M(\Tr^0,\w^0)}
\circ
\De{\Tr}{\Tr\p(\w^0)}{\Tr^0}
\circ
\De{\w}{\p(\w)}{\w^0}
\nonumber
\end{eqnarray}

$\Df{\p}{f}$ and $\Df{\Tr}{\Tr\p}$ are already known
so we need only determine $\Df{\w}{\p(\w)}$.
Using reasoning similar
to equation \ref{eq:total-registration-transform-derivative},
we have $\p(\w) = \bigoplus_{j=0}^{m-1} \p_i(\w)$,
and
$\Df{\w}{\p(\w)}
=
\Df{\w}{\bigoplus_{j=0}^{m-1} \p_i(\w)}
=
\bigoplus_{j=0}^{m-1} \Df{\w}{\p_i(\w)}$,
so we can restrict ourselves to
$\Df{\w}{\p(\w)}$, where $\p(\w) = \sum_{i=0}^{n-1} w_i \p_i$,
and $\p, \p_i \in \Reals^3$.

Note that $\p(\w)$ is a linear transform from $\Reals^n \mapsto \Reals^3$,
which can be expressed as $\P = \sum_{i=0}^{n-1} \p_i \otimes \e_i$,
where $\e_i$ are the canonical basis vectors of $\Reals^n$.
($\P$ can be written as a matrix whose $i$th column is the vector $\p_i$.)
Thus it follows that
\begin{equation}
\Df{\w}{\p(\w)} = \Df{\w}{\P\w} = \P
\end{equation}

\subsection{Convex weights}
\label{sec:Convex-weights}

It may be desireable to restrict the weights to be convex,
that is, $0 \leq w_i \leq 1; \sum w_i = 1$.
To use unconstrained optimization methods with convex weights,
we re-parameterize:
\begin{equation}
u_i(\w) = {{w_i^2} \over {\| \w \|^2}}
\end{equation}
where, as usual, $\| \w \|^2 = \sum_{j=0}^{n-1} w_j^2$.

Then we need to compute
\begin{eqnarray}
\De{\w}{\p(\u(\w))}{\w^0}
& = &
\De{\u}{\p(\u)}{\u(\w^0)}
\circ
\De{\w}{\u(\w)}{\w^0}
\\
& = &
\P
\circ
\De{\w}{\u(\w)}{\w^0}
\nonumber
\end{eqnarray}

The partial derivatives are
\begin{eqnarray}
\Df{w_j}{u_i(\w))}
& = &
\Df{w_j}{{w_i^2} \over {\| \w \|^2}}
\\
& = &
{{2 w_j} \over {\| \w \|^4}} \left( \delta_{ij} \| \w \|^2 - w_i^2 \right)
\nonumber
\end{eqnarray}


\section{Hyperparameter Tuning}\label{section:background-hyperparameter}
Hyperparameter tuning is the process of optimizing a model to find the hyperparameters that will perform optimally given a specified metric. This metric is usually either the validation loss or the validation accuracy.
Hyperparameters are generally chosen by hand or by some automated process, but are not learned as a part of the model itself as opposed to the model weights.

Methods to find the optimal hyperparameters include manual tuning, grid search, random search, genetic algorithms, bayesian optimization, and hyperband, among others.
Random and grid searches are very time-consuming and computationally expensive, as they potentially run less than optimal parameters and do not use knowledge of previous runs to select the next set of parameters.
Bayesian optimization attempts to curtail this problem by taking previous runs into account, greatly speeding up the learning process and not experimenting with less than optimal hyperparameters~\cite{bayesian}.
Hyperband is a bandit-based approach to hyperparameter optimizations that can abandon or reduce the budget of an experimental run if the network performance is below with a chosen parameter set is below a certain threshold~\cite{hyperband}.

\subsection{Bayesian Optimization}\label{subsection:background-bayesian}
The idea behind Bayesian optimization is that the hyperparameters of a network follow a probabilistic distribution $\text{P}(y | x)$ where $y$ is the score determined by some objective function and $x$ is a set of hyperparameters.
Based on this probabilistic model, an optimal set of hyperparameters can then be calculated.
Applying these hyperparameters to the model, the objective function can then be used to calculate the scores.
The probabilistic distribution is then updated and the process is repeated until the maximum number of iterations is reached.

Bayesian optimization relies on the following~\cite{bayesiantds}:
\begin{enumerate}
	\itemsep-1em
	\item A clearly defined hyperparameter domain of possible configurations $\chi$ over which to search,
	\item An objective function which calculates a score that we wish to minimize using a set of hyperparameters,
	\item A surrogate model that represents the probabilistic distribution of the hyperparameters,
	\item A selection function to evalute which hyperparameters should be evaluated next, and
	\item A history of score-hyperparameter pairs used to update the surrogate model.
\end{enumerate}
The domain must be chosen by the researcher and is usually defined with respect to previous knowledge or based on related works.
The objective function is evaluated by running the model and calculating its error using a predetermined loss function.

The surrogate model used is essentially a mapping of hyperparameters to scores from the objective function.
An example for the surrogate model is the Tree-structured Parzen Estimator (TPE)~\cite{tpe}.
TPE takes advantage of Bayes' rule and represents the probability function $\text{P}(y~|~x)$ as
\begin{equation}\label{eq:bohb-1}
	\begin{split}
		\text{P}(y~|~x) &= \frac{\text{P}(x~|~y) \cdot \text{P}(x)}{\text{P}(y)}\\
		\text{with } \text{P}(x~|~y) &=
		\begin{cases}
		l(x) &\text{when } y < y^* \\
		g(x) &\text{when } y \geq y^*
		\end{cases}
	\end{split}
\end{equation}
where $y^*$ is a threshold value. $l(x)$ is thus a probability density function formed from the set of observations using hyperparameters $x^i \in \chi$ which have been performed such that the loss of the network is less than $y^*$.
Conversely, $g(x)$ is the density function formed from the remaining observations.
$y^*$ is chosen by the algorithm such that $\text{P}(y<y^*) = \gamma$ with $\gamma$ being a value chosen by the researcher.

The selection function chooses which hyperparameters $x \in \chi$ should be chosen for each successive experiment and is commonly based on the expected improvement~\cite{bayesiantds}:
\begin{align}
	EI_{y^*}(x) &= \int_{-\infty}^{y^*} (y^* - y) p(y~|~x)dy
\end{align}
Here, $y^*$ is again a threshold value, $x$ is the proposed set of hyperparameters, $y$ is the actual value of the objective function using hyperparameters $x$ and $p(y~|~x)$ is the surrogate probability model expressing the probability of $y$ given $x$.
Substituting the values of $p(y~|~x)$ from the surrogate function we get
\begin{equation}
	\begin{split}
		EI_{y^*}(x) &= \frac{\gamma y^* l(x) - l(x) \int_{-\infty}^{y^*}p(y)dy}{\gamma l(x) + (1 - \gamma)g(x)}\\
		&\propto \left(\gamma + \frac{g(x)}{l(x)} (1 - \gamma)\right)^{-1}
	\end{split}
\end{equation}
We can see from this function that the expected improvement is proportional to $\frac{l(x)}{g(x)}$ and thus, to maximize the expected improvement, samples should be drawn from the maximum value in $l(x)$.
The selected hyperparameters are then evaluated with regards to the objective function and the results used to update the surrogate model.

\subsection{Hyperband}\label{section:background-hyperband}
Hyperband is a learning strategy that "relies on a principled early-stopping strategy to allocate resources"~\cite{hyperband}.
It is a variation of successive halving, and in fact uses successive halving in its inner loop~\cite{successivehalving}.

Successive halving randomly samples $n$ hyperparameter sets in the search domain $\chi$.
It then evaluates all of these sets for $B$ iterations to calculate the validation loss.
The lowest performing half is then discarded and the remaining evaluated for a further $B$ iterations.
This is repeated until only one hyperparameter set remains.

Successive halving suffers from the $n \text{ vs } \frac{B}{n}$ problem, which is whether to train more configurations $n$ or to explore fewer, but with more resources $B$.
If a larger $n$ is chosen, configurations which are slower to converge might be killed off too quickly.
If a larger $\frac{B}{n}$ is chosen, then lower performing configurations may be given more resources, thus wasting resources that could otherwise be spent on higher performing configurations~\cite{hyperband}.

Hyperband attempts to solve this issue by considering several possible values of $n$ for a fixed $B$.
The pseudocode in Algorithm \ref{algorithm:hyperband} shows how Hyperband uses an outer loop, performing essentially a grid search with multiple values of $n$ and an inner loop utilizing a method similar to successive halving but with the top $\left\lfloor n_i/\eta\right\rfloor$ instead of the top half.

\input{algorithms/hyperband}

\subsection{Bayesian Optimization with Hyperband}\label{section:background-bohb}
"BOHB: Robust and Efficient Hyperparameter Optimization at Scale" by Stefan Falkner, Aaron Klein, and Frank Hutter proposes a hyperparameter optimization approach which seeks to combine the benefits of both Bayesian optimization and Hyperband~\cite{bohb}.
This approach seeks to have strong anytime performance, strong final performance, effectively use parallel resources, be easily scalable, and be robust and flexible.

BOHB works by using Hyperband to determine the number of configurations to evaluate with a given budget, but replaces the random selection of configurations with a Bayesian model based search.
The process of using Bayesian optimization for configuration sampling can be seen in Algorithm \ref{algorithm:bohb}. In this algorithm, $l'(x)$ is the $l(x)$ component of the newly updated probability density function.

Even though Hyperband also provides strong anytime performance compared to random search and Bayesian optimization, Fakner et al. observed an improvement of over 55x against a random search at larger budgets, converging to the global optimum much faster than either Hyperband or Bayesian optimization alone~\cite{bohb}.

\input{algorithms/bohb}


\documentclass[12pt]{cdblatex}
\usepackage{examples}

\begin{document}

\section*{Elementary maths}

\vspace{-5pt}

This example is based on a similar example in the \href{../../python/examples/example-01.pdf}{Python collection}. Its purpose is to show that Cadabra is fluent in Python -- which is not surprisng since the Cadabra language is based on Python (and a subset of LaTeX).

\vspace{-8pt}

\begin{minipage}[t]{0.65\textwidth}
\begin{cadabra}
   from sympy import *
   x, y, z = symbols('x y z')
   a, b, c = symbols('a b c')
   ans = expand((a+b)**3)                                  # cdb (ans.101,ans)
   ans = factor(-2*x+2*x+a*x-x**2+a*x**2-x**3)             # cdb (ans.102,ans)
   ans = solve(x**2-4, x)                                  # cdb (ans.103,ans)
   ans = solve([2*a-b - 3, a+b+c - 1,-b+c - 6],[a,b,c])    # cdb (ans.104,ans)
   ans = N(pi,50)                                          # cdb (ans.105,ans)
   ans = apart(1/((1 + x)*(5 + x)))                        # cdb (ans.106,ans)
   ans = together((1/(1 + x) - 1/(5 + x))/4)               # cdb (ans.107,ans)
   ans = simplify(tanh(log(x)))                            # cdb (rhs.108,ans)
   ans = simplify(tanh(I*x))                               # cdb (rhs.109,ans)
   ans = simplify(sinh(3*x) - 3*sinh(x) - 4*(sinh(x))**3)  # cdb (rhs.110,ans)
   ans = tanh(log(x))                                      # cdb (lhs.108,ans)
   ans = tanh(UnevaluatedExpr(I*x))                        # cdb (lhs.109,ans)
   ans = sinh(3*x) - 3*sinh(x) - 4*(sinh(x))**3            # cdb (lhs.110,ans)
\end{cadabra}
\end{minipage}
\hskip 1cm
\begin{minipage}[t]{0.35\textwidth}
\begin{latex}
   \begin{align*}
      &\cdb*{ans.101}\\
      &\cdb*{ans.102}\\
      &\cdb*{ans.103}\\
      &\cdb*{ans.104}\\
      &\cdb*{ans.105}\\
      &\cdb*{ans.106}\\
      &\cdb*{ans.107}\\
      \cdb{lhs.108} &= \Cdb{rhs.108}\\
      \cdb{lhs.109} &= \Cdb{rhs.109}\\
      \cdb{lhs.110} &= \Cdb{rhs.110}
   \end{align*}
\end{latex}
\end{minipage}

\vspace{-8pt}

\begin{align*}
   &\cdb*{ans.101}\\
   &\cdb*{ans.102}\\
   &\cdb*{ans.103}\\
   &\cdb*{ans.104}\\
   &\cdb*{ans.105}\\
   &\cdb*{ans.106}\\
   &\cdb*{ans.107}\\
   \cdb{lhs.108} &= \Cdb{rhs.108}\\
   \cdb{lhs.109} &= \Cdb{rhs.109}\\
   \cdb{lhs.110} &= \Cdb{rhs.110}
\end{align*}

\clearpage

\section*{Linear Algebra}

\vspace{-10pt}

\begin{minipage}[t]{0.65\textwidth}
\begin{cadabra}
   from sympy import linsolve
   lamda = Symbol('lamda')
   mat  = Matrix([[2,3], [5,4]])                   # cdb (ans.201,mat)
   eig1 = mat.eigenvects()[0][0]                   # 1st eigenvalue
   eig2 = mat.eigenvects()[1][0]                   # 2nd eigenvalue
   v1   = mat.eigenvects()[0][2][0]                # 1st eigenvector
   v2   = mat.eigenvects()[1][2][0]                # 2nd eigenvector
   eig  = simplify(Matrix([eig1,eig2]))            # cdb (ans.202,eig)
   vec  = simplify(5*Matrix([]).col_insert(0,v1)
                               .col_insert(1,v2))  # cdb (ans.203,vec)
   det  = expand((mat - lamda * eye(2)).det())     # cdb (ans.204,det)
   rhs  = Matrix([[3],[7]])                        # cdb (ans.205,rhs)
   ans  = list(linsolve((mat,rhs),x,y))[0]         # cdb (ans.206,ans)
\end{cadabra}
\end{minipage}
\hskip 1cm
\begin{minipage}[t]{0.35\textwidth}
\begin{latex}
   \begin{align*}
      &\cdb*{ans.201}\\
      &\cdb*{ans.202}\\
      &\cdb*{ans.203}\\
      &\cdb*{ans.204}\\
      &\cdb*{ans.205}\\
      &\cdb*{ans.206}
   \end{align*}
\end{latex}
\end{minipage}

\begin{align*}
   &\cdb*{ans.201}\\
   &\cdb*{ans.202}\\
   &\cdb*{ans.203}\\
   &\cdb*{ans.204}\\
   &\cdb*{ans.205}\\
   &\cdb*{ans.206}
\end{align*}

\clearpage

\section*{Limits}

\vspace{-10pt}

\begin{minipage}[t]{0.65\textwidth}
\begin{cadabra}
   n, dx = symbols('n dx')
   ans = limit(sin(4*x)/x,x,0)                  # cdb (ans.301,ans)
   ans = limit(2**x/x,x,oo)                     # cdb (ans.302,ans)
   ans = limit(((x+dx)**2 - x**2)/dx, dx,0)     # cdb (ans.303,ans)
   ans = limit((4*n + 1)/(3*n - 1),n,oo)        # cdb (ans.304,ans)
   ans = limit((1+(a/n))**n,n,oo)               # cdb (ans.305,ans)
\end{cadabra}
\end{minipage}
\hskip 1cm
\begin{minipage}[t]{0.35\textwidth}
\begin{latex}
   \begin{align*}
      &\cdb*{ans.301}\\
      &\cdb*{ans.302}\\
      &\cdb*{ans.303}\\
      &\cdb*{ans.304}\\
      &\cdb*{ans.305}
   \end{align*}
\end{latex}
\end{minipage}

\begin{align*}
   &\cdb*{ans.301}\\
   &\cdb*{ans.302}\\
   &\cdb*{ans.303}\\
   &\cdb*{ans.304}\\
   &\cdb*{ans.305}
\end{align*}

\section*{Series}

\vspace{-10pt}

\begin{minipage}[t]{0.65\textwidth}
\begin{cadabra}
   ans = series((1 + x)**(-2), x, 1, 6)         # cdb (ans.401,ans)
   ans = series(exp(x), x, 0, 6)                # cdb (ans.402,ans)
   ans = Sum(1/n**2, (n,1,50)).doit()           # cdb (ans.403,ans)
   ans = Sum(1/n**4, (n,1,oo)).doit()           # cdb (ans.404,ans)
\end{cadabra}
\end{minipage}
\hskip 1cm
\begin{minipage}[t]{0.35\textwidth}
\begin{latex}
   \begin{align*}
      &\cdb*{ans.401}\\
      &\cdb*{ans.402}\\
      &\cdb*{ans.403}\\
      &\cdb*{ans.404}
   \end{align*}
\end{latex}
\end{minipage}

\begin{align*}
   &\cdb*{ans.401}\\
   &\cdb*{ans.402}\\
   &\cdb*{ans.403}\\
   &\cdb*{ans.404}
\end{align*}

\clearpage

\section*{Calculus}

This example shows how {\tt\small\verb|\Cdb|} can be used to set the equation tag on the far right hand side.

\begin{minipage}[t]{0.65\textwidth}
\begin{cadabra}
   ans = diff(x*sin(x),x)                                    # cdb (ans.501,ans)
   ans = diff(x*sin(x),x).subs(x,pi/4)                       # cdb (ans.502,ans)
   ans = integrate(2*sin(x)**2, (x,a,b))                     # cdb (ans.503,ans)
   ans = Integral(2*exp(-x**2), (x,0,oo))                    # cdb (lhs.504,ans)
   ans = ans.doit()                                          # cdb (ans.504,ans)
   ans = Integral(Integral(x**2 + y**2, (y,0,x)), (x,0,1))   # cdb (lhs.505,ans)
   ans = ans.doit()                                          # cdb (ans.505,ans)
\end{cadabra}
\end{minipage}
\hskip 1cm
\begin{minipage}[t]{0.35\textwidth}
\begin{latex}
   \begin{align*}
      &\cdb*{ans.501}\\
      &\cdb*{ans.502}\\
      &\cdb*{ans.503}\\
       \cdb{lhs.504}&=\Cdb{ans.504}\\
       \cdb{lhs.505}&=\Cdb{ans.505}
   \end{align*}
\end{latex}
\end{minipage}

\begin{align*}
   &\cdb*{ans.501}\\
   &\cdb*{ans.502}\\
   &\cdb*{ans.503}\\
    \cdb{lhs.504}&=\Cdb{ans.504}\\
    \cdb{lhs.505}&=\Cdb{ans.505}
\end{align*}

\clearpage

\section*{Differential equations}

\vspace{-10pt}

\begin{minipage}[t]{0.70\textwidth}
\begin{cadabra}
   y = Function('y')
   C1, C2 = symbols('C1 C2')

   ode = Eq(y(x).diff(x) + y(x), 2*a*sin(x))
   sol = expand(dsolve(ode,y(x)).rhs)                              # cdb (ans.601,sol)
   cst = solve([sol.subs(x,0)],dict=True)
   sol = sol.subs(cst[0])                                          # cdb (ans.602,sol)

   ode = Eq(y(x).diff(x,2) + y(x), 0)
   sol = expand(dsolve(ode,y(x)).rhs)                              # cdb (ans.603,sol)
   cst = solve([sol.subs(x,0),sol.diff(x).subs(x,0)-1],dict=True)
   sol = sol.subs(cst[0])                                          # cdb (ans.604,sol)

   ode = Eq(y(x).diff(x,2) + 5*y(x).diff(x) - 6*y(x), 0)
   sol = expand(dsolve(ode,y(x)).rhs)                              # cdb (ans.605,sol)
   sol = sol.subs({C1:2,C2:3})                                     # cdb (ans.606,sol)
\end{cadabra}
\end{minipage}
\hskip 1cm
\begin{minipage}[t]{0.30\textwidth}
\begin{latex}
   \begin{align*}
      &\cdb*{ans.601}\\
      &\cdb*{ans.602}\\
      &\cdb*{ans.603}\\
      &\cdb*{ans.604}\\
      &\cdb*{ans.605}\\
      &\cdb*{ans.606}
   \end{align*}
\end{latex}
\end{minipage}

\begin{align*}
   &\cdb*{ans.601}\\
   &\cdb*{ans.602}\\
   &\cdb*{ans.603}\\
   &\cdb*{ans.604}\\
   &\cdb*{ans.605}\\
   &\cdb*{ans.606}
\end{align*}

\end{document}


\gotosection{0}{4}
\subsection{Functions}

\begin{exercise}{9}
  \begin{enumerate}
    \item $\begin{aligned}[t]
            (f \circ g \circ h)(a) &= f(g(h(a))) = f(g(h(3))) \\
                                   &= f(g(-3+2)) = f(g(-1)) \\
                                   &= f(3 \times (-1)) = f(-3) \\
                                   &= (-3)^2-1 = 8
          \end{aligned}$
    \item $\begin{aligned}
            (f \circ g \circ h)(a) &= f(g(h(a))) = f(g(h(1))) \\
                                   &= f(g(1-3)) = f(g(-2)) \\
                                   &= f(-2-3) = f(-5) \\
                                   &= (-5)^2 = 25
          \end{aligned}$
  \end{enumerate}
\end{exercise}

\begin{exercise}{10}
  \begin{enumerate}
    \item Let the function \FunSS{f}{B}{C} and \FunSS{g}{A}{B} be onto.
          Then the composition $f \circ g$ is onto.

    \Proof{} For every $c \in C$: since \FunSS{f}{B}{C} is surjective, there
    exists $b \in B$ such that $f(b) = c$; since \FunSS{g}{A}{B} is surjective,
    there exists $a \in A$ such that $g(a) = b$. Therefore, for every $c \in C$,
    there exists $a \in A$ such that $(f \circ g)(a) = f(g(a)) = c$.
    Thus, $f \circ g$ is surjective, or onto. \QED

    \item Let the function \FunSS{f}{B}{C} and \FunSS{g}{A}{B} be one to one.
          Then the composition $f \circ g$ is one to one.

    \Proof{} For every $c \in C$ such that there exists $a \in A$ such that
    $f(g(a)) = c$, suppose there do exist $x, y \in A$ such that both $f(g(x)) = c$
    and $f(g(y)) = c$. Since $f$ is injective, $g(x) = g(y)$. Since $g$ is injective
    , $x = y$. Therefore, there is at most one $x$ such that $(f \circ g)(x) =
    f(g(x)) = c$. Thus, $f \circ g$ is injective, or one to one. \QED
  \end{enumerate}
\end{exercise}


\documentclass[paper.tex]{subfiles}

\usepackage{tikz}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{tabularx}
\usepackage{multicol}
\usepackage{algpseudocode}
\usepackage{algorithm}

% Add vertical spacing to tables
\renewcommand{\arraystretch}{1.4}

% Begin Document
\begin{document}

\section{A System of Linear Equations}

This problem can be solved by utilizing a system of linear equations.
Each equation represents the input and output of a specific router node.
First we must construct a table of the input and output flow of each node.
The input and output must be equivalent for each node.
Solving for variables for each node gives us a system of linear equations.

\begin{center}
\begin{tabular}{c | l | l}

    \textbf{Node} & \textbf{Input} & \textbf{Output} \\ \hline

    A & 100                         & $2x_1 + x_2$ \\
    B & $x_1 + x_2$                 & $x_3 + x_5$  \\
    C & $50 + x_1$                  & $x_3 + x_5$  \\
    D & $x_4 + x_5$                 & $x_2 + 120$  \\
    E & $x_2 + x_3 + x_5$ \hspace{8mm} & $x_4$

\end{tabular}

\begin{tabular}{c | l}
    \textbf{Node} & \textbf{Equations} \\ \hline
    A & $2x_1 + x_2 = 100$ \\
    B & $x_1 + x_2 - x_3 - x_5 = 0$    \\
    C & $x_1 - x_3 - x_5 = -50$        \\
    D & $-x_2 + x_4 + x_5 = 120$       \\
    E & $x_2 + x_3 - x_4 + x_5 = 0$
\end{tabular}
\end{center}

Utilizing this system of equations, we can construct a matrix equation of the form $Ax = b$ representing it.
This becomes equation (1) below:

\begin{center}
\begin{equation}
    \begin{bmatrix}
        2 & 1 & 0 & 0 & 0   \\
        1 & 1 & -1 & 0 & -1 \\
        1 & 0 & -1 & 0 & -1 \\
        0 & -1 & 0 & 1 & 1  \\
        0 & 1 & 1 & -1 & 1
    \end{bmatrix}
    \begin{bmatrix}
        x_1 \\
        x_2 \\
        x_3 \\
        x_4 \\
        x_5
    \end{bmatrix}
    =
    \begin{bmatrix}
        100 \\
        0 \\
        -50 \\
        120 \\
        0
    \end{bmatrix}
\end{equation}
\end{center}

\end{document}


%!TEX root = paper.tex
\subsection{Linear standing wave}
The linear standing wave test illustrates the different linear dispersion relations and the ability of the numerical method to represent them in an accurate manner.
Both, the linearized systems of the hydrostatic and dispersive have an analytic standing wave solution
\begin{align}
\xi(\bx,t)&=-a\sin\left(\kappa x \right)\cos\left(\kappa ct\right), \\
 u(\bx,t) &=a \frac{c}{d}\cos\left(\kappa x\right)\sin\left(\kappa ct\right), \\
 v(\bx,t) &=0, \qquad \forall \ \bx=(x, y)^T \in \Omega, \forall t \in \mathbb{R},
\end{align}
where the phase velocity $c=\frac{\omega_{\text{nh,}\fnh}}{\kappa}$ is chosen for the dispersive equation set and $c=\csw$ for the hydrostatic equation set, respectively. For the \nh\ model, the analytic vertical velocity has to be defined as
\[
w(\bx,t)=-\frac{d}{2}\partial_x u=\frac{1}{2}ac\kappa\sin\left(\kappa x \right)\sin\left(\kappa ct\right),
\]
which is derived directly from \eqref{eq:nh_closure}.
In different model runs, we vary the depth $d$ while keeping the wave length $\lambda=\frac{2\pi}{\kappa}=20$ m and the amplitude $a=0.01$m constant to get ratios for $\frac{d}{\lambda}$ between $0.05$ and $1.0$, and ratios for $\frac{a}{d}$ between $0.01$ and $0.005$, respectively.
We impose double periodic boundary conditions on a grid length of one wave length. The simulation time is chosen long enough to measure one wave period.

\subsubsection{Results of \nh\ model}
The resulting normalized phase velocities for the shallow water model and the \nh\ equation set with either the linear or the quadratic vertical pressure are displayed in figure \ref{fig:nh_standingwave}. Furthermore, they are compared to their analytical reference phase velocities and the full reference phase velocity as derived in section \ref{sec:dispersion}.
In each case, the numerical dispersion relation matches the corresponding analytical one precisely.
% The coincidence between numerical and analytical phase velocities is very good.
In a close neighborhood of the shallow water assumption (i.e., in the limit $\frac{d}{\lambda} \rightarrow 0$), the quadratic vertical pressure profile gives a better phase velocity compared to the full reference solution than the linear profile, as expected from series expansions around this state used in \Bt\ models.
% \svnote{Can we have an additional detail plot of this neighborhood?} \ajnote{Then only thinner lines would help. I could also provide the whole image with different plotting style (thinner lines and ) One could also use absolute errors between numerical values and reference phase velocities. }\jbnote{What are you expecting to see?}\svnote{I expect to see what we describe in the text, i.e., that the quadratic vertical pressure profile gives a better phase velocity than the linear one near $d/\lambda=0$, see added figure.}
However, for ratios $\frac{d}{\lambda} > 0.25$ approximately, the linear profile matches better.

\begin{figure}[htbp]
        \includegraphics*[width=0.95\textwidth]{standingwave_nh.eps}
        \caption{Standing wave: Comparison of simulated hydrostatic and \nh\ phase velocities with analytic reference values for all simulations (left) and a zoom onto the close neighborhood of the long wave limit (right)}
        \label{fig:nh_standingwave}
\end{figure}


\section{Juncai}
\subsection{Classification Problems for DNN}
Now we want to make some rigorous mathematical statement about classification problem for DNN. What we want to state and prove is about, what's the necessary and sufficient conditions for $\{x_i, y_i\}_{i \in I}$ such that, DNN model(with one hidden layer, when we talks about DNN model in this subsection, we always mean one hidden layer model.) can ``separate" them.

Let us note:
\begin{align}
A_k = \{x_i ~ | ~  y_i = e_k, i \in I\} \quad k = 1:c.
\end{align}
We need the next two assumptions for data first:
\begin{assumption}[Boundedness]We say those data $\{x_i, y_i\}$ for $i \in I$ are bounded if there exist a $M < \infty$ such that
\begin{align}
\|x_i\| < M \quad \forall i \in I.
\end{align}
\end{assumption}
\begin{assumption}[Consistence]We say those data $\{x_i, y_i\}$ for $i \in I$ are consistent for classification if
\begin{align}
A_k \cap A_k =  \emptyset \quad \forall k\neq l.
\end{align}
\end{assumption}
Then we can have the next definition for what means ``DNN can separate  $\{A_k\}_{k=1}^c$":
\begin{definition}
We say that DNN model can separate $\{A_k\}_{k=1}^c$ or DNN model can fix the classification problem for $\{x_i ,y_i\}_{i \in I}$ means that there is a function $f: \mathbb{R}^n \to \mathbb{R}^c$ such that:
\begin{align}
f(x) = e_k \quad \forall x \in A_k, ~~ k = 1:c,
\end{align}
with
\begin{align}
f(x) = a(W_2 \sigma(W_1 x + b_1) + b_2),
\end{align}
for some
\begin{align}
W_1 \in \mathbb{R}^{m \times n}, \quad b_1 \in \mathbb{R}^m \quad W_2 \in \mathbb{R}^{c\times m}, \quad b_2 \in \mathbb{R}^c, \quad \epsilon > 0,
\end{align}
and $\sigma, a: \mathbb{R} \to \mathbb{R}$.
Here $\sigma$ can be ``Sigmoid", ``ReLU", ``Heaviside" or $a$ function, with
\begin{equation}
\sigma(x) = \frac{1}{1 + e^{-x}}, \quad r(x) = \max\{0,x\}, \quad H(x) = \begin{cases}
1 ~~ &x > 0 \\
0 ~~ &x \le 0.
\end{cases}
\end{equation}
and $a$ is a variant form of Heaviside function or combine of two ReLU functions like:
%\begin{equation}
%\hat{H}_{\epsilon}(x) = \begin{cases}
%0 \quad & x < \epsilon, \\
%\frac{x}{1-2\epsilon} - \frac{\epsilon}{1-2\epsilon} \quad & \epsilon \le x < 1-\epsilon, \\
%1 \quad & x \ge 1-\epsilon.
%\end{cases}
%\end{equation}
%By the way, $H_{\epsilon}(x) = \frac{1}{1-2\epsilon}r(x - \epsilon) +  \frac{1}{1-2\epsilon}r(x - 1 + \epsilon).$
\begin{equation}
a(x) = \begin{cases}
0 \quad & x < 0, \\
x \quad & 0 \le x < 1, \\
1 \quad & x \ge 1.
\end{cases}
\end{equation}
By the way, $a(x) = r(x ) - r(x - 1).$

\end{definition}

Here we define what means ``$\{A_k\}_{k=1}^c$ are separable with positive distance":
\begin{definition}
We say that $\{A_k\}_{k=1}^c$ are separable with positive distance when there exists a small positive number $\epsilon$, such that
\begin{align}
d(\bar{A}_k, \bar{A}_l) > \epsilon, \quad \forall k \neq l,
\end{align}
where $\bar{A}_k$ means the closure of $A_k$ and $d(\bar{A}_k, \bar{A}_l)$ define the distance of two sets as:
\begin{align}
d(\bar{A}_k, \bar{A}_l) = \inf_{x \in A_k, y \in A_l} d(x,y).
\end{align}
\end{definition}

As for the ``separable with positive distance" property, we have the next lemma:
\begin{lemma}If $\{A_k\}_{k=1}^c$ are separable with positive distance $\epsilon$, then there exist a smooth feature map $\phi: \mathbb{R}^n \to \mathbb{R}^c$ such that:
\begin{align}
\phi(x) = e_k \quad \forall x \in \bar{A}_k,\quad k = 1:c.
\end{align}
\end{lemma}
\begin{proof}This proof is based on the partition of unity theorem. Because of the positive distance with $\epsilon$ property, we define
\begin{equation}
\mathcal{M}_k = \{x ~|~ d(x, \bar{A}_k) < \frac{\epsilon}{2}\}.
\end{equation}
So, $\mathcal{M}_k$ is an open set as the continuity for $d(\cdot, \bar{A}_k)$. Let
\begin{equation}
\mathcal{M}_0 = (\bigcup_{k=1:c} \bar{A}_k)^c,
\end{equation}
so $\mathcal{M}_0$ is an open set too. And
\begin{equation}
\mathbb{R}^n = (\bigcup_{k=0:c} \mathcal{M}_k).
\end{equation}
So we can apply the partition of unity for the above decomposition, getting $\{\phi_k\} \in C^{\infty}(\mathbb{R}^n)$ for $k = 0:c$ and
\begin{equation}
1 = \sum_{k=0}^c \phi_k \quad \text{with} \quad supp(\phi_k) \subset \mathcal{M}_k.
\end{equation}
Then we will see that:
\begin{equation}
\phi_k(x_l) = \delta_{kl} \quad \forall x_l \in \bar{A}_l, \quad k, l = 1:c,
\end{equation}
because of the definition of $\mathcal{M}_k$ and the results of partition of unity.

At last, we can get $\phi$ as:
\begin{align}
\phi: \mathbb{R}^n &\to \mathbb{R}^c \\
 x &\mapsto (\phi_1(x), \cdots, \phi_c(x)),
\end{align}
and it satisfy the condition.
\end{proof}

Now we can have those next theorem:
\begin{theorem}
If $I$ is a finite index set, then DNN can always separate $\{A_k\}_{k=1}^c$.
\end{theorem}

\begin{theorem}
If $\{A_k\}_{k=1}^c$ are separable with positive distance, then it can be separated by DNN.
\end{theorem}
\begin{proof}
We know that
\begin{equation}
a(e_k) = e_k \quad k = 1:c.
\end{equation}

Because of the universal approximation property for ``Sigmoid", ``ReLu", ``Heaviside" or $a(x)$ type Neural Network, we can take
\begin{align}
\hat{W}_2\sigma(W_1 x + b_1) + \hat{b}_2,
\end{align}
to approximate $\phi$ in $B_M$ such that,
\begin{align}
\|\hat{W}_2\sigma(W_1 x + b_1) + \hat{b}_2 - \phi(x) \|_{L^{\infty}} < \delta < \frac{1}{4}, \quad \forall \|x\| \le M.
\end{align}
{\bf Here $\delta$ is independent with the distant of training data, we can choose any one in $(0, \frac{1}{2})$ in fact.
This is because of the linearly separable of $\{e_k\}_{k=1}^c$ points.
}

Now we can take the DNN model as:
\begin{align}
f(x) = a(\frac{1}{1-2\delta}\hat{W}_2\sigma(W_1 x + b_1) + (\hat{b}_2 -\frac{\delta}{1-2\delta}\bm{1})).
\end{align}
So we only need to check that $f(x) = e_k$ if $x\in A_k$ with $k=1:c$.

If $x \in A_k$ for any $k=1:c$, we know that
\begin{align}
|(\hat{W}_2\sigma(W_1 x + b_1) + \hat{b}_2 - e_k)_i| < \delta, \quad  i = 1:c,
\end{align}
which means that
\begin{equation}
\begin{cases}
(\hat{W}_2\sigma(W_1 x + b_1) + \hat{b}_2 )_i < \delta \quad &i\neq k \\
(\hat{W}_2\sigma(W_1 x + b_1) + \hat{b}_2 )_i  > 1- \delta \quad &i = k.
\end{cases}
\end{equation}
Thus by the definition of $a(x)$, we know:
\begin{equation}
a(\frac{x}{1-2\delta} - \frac{\delta}{1-2\delta}) = \begin{cases}
0 \quad & x < \delta, \\
\frac{x}{1-2\delta} - \frac{\delta}{1-2\delta} \quad & \delta \le x < 1 - \delta, \\
1 \quad & x \ge 1 - \delta.
\end{cases}
\end{equation}
So we have
$f(x) = e_k$ if $x\in A_k$ with $k=1:c$, for
\begin{align}
f(x) = a(W_2\sigma(W_1 x + b_1) + b_2),
\end{align}
with
\begin{equation}
W_2 = \frac{1}{1-2\delta}\hat{W}_2, ~ \text{and} ~ b_2 = \hat{b}_2 -\frac{\delta}{1-2\delta}\bm{1}.
\end{equation}
This finishes this proof.
\end{proof}

\begin{remark}
Here $W_2$ have a special structure like as
\begin{align}
W_2 = diag\{\omega_1, \cdots, \omega_c\},
\end{align}
where $\omega_i \in \mathbb{R}^{1\times n_i}$ with $\sum_{i=1}^c n_i = m$.
\end{remark}

\begin{theorem}
If $\{A_k\}_{k=1}^c$ can be separated by DNN, then $\{A_k\}_{k=1}^c$ are separable with positive distance.
\end{theorem}
\begin{proof}First we know that,
\begin{align}
A_k \subset L_k := \{x ~|~ f(x) = e_k\}.
\end{align}
Here $L_k$ is a closed set and $d(L_k, L_l) \ge \delta > 0$ because of the continuity of $f$. So,
$\{A_k\}_{k=1}^c$ are separable with positive distance.
\end{proof}

\subsection{Linearly Separable and Global Minima}

Here we note those data sets $\{x_i, y_i\}_{i=1}^N$ as $\{A_k\}_{k=1}^c$ with $y = e_k$ for any $x \in A_k$. The simple NN model or perceptron with Heaviside function or $H_{\epsilon}$ like:
\begin{equation}
H_{\epsilon}(x) = \begin{cases}
0 \quad & x < -\epsilon, \\
\frac{x}{2\epsilon} - \frac{1}{2} \quad & -\epsilon \le x < \epsilon, \\
1 \quad & x \ge \epsilon.
\end{cases}
\end{equation}
can classify those data means there exist $W \in \mathbb{R}^{c\times n}$ and $b \in \mathbb{c}$ such that
\begin{align}
H(Wx + b) = e_k \quad x \in A_k, ~ k =1:c.
\end{align}

Note the loss function as:
\begin{align}
L(\theta) = \sum_{i=1}^N\|H(Wx_i + b) - y_i\|^2.
\end{align}

\begin{properties}Those next two statements are equivalent:
\begin{enumerate}
\item The perceptron can classify those data.
\item $\min L(\theta) = 0$.
\end{enumerate}
\end{properties}

\begin{theorem}\label{theo:linearconvex}
Let
\begin{align}
\Theta := \mathop{\arg\min}_{\theta}L(\theta).
\end{align}
Then $\Theta$ is a convex set if the perceptron can classify those data.
\end{theorem}

\begin{proof}
If the perceptron can classify those data, then $\Theta \neq \emptyset$. So, let $\theta_1, \theta_2 \in \Theta$, let us consider $\theta = t\theta_1 + (1-t)\theta_2$ with $t \in [0,1]$.
Then for any data $(x_i, y_i)$ and $\theta = (W, b)$ we have
\begin{align}
H(Wx_i + b) = H(t(W_1x_i + b_1) + (1-t)(W_2x_i + b_2)),
\end{align}
and
\begin{align}
H(W_1x_i + b_1) = H(W_2x_i + b_2) = y_i = e_k,
\end{align}
for some $1 \le k \le c$.

Because of the properties of Heaviside function(its level sets are convex sets), we know that:
\begin{align}
(W_1 x_i + b_1)_j, (W_2 x_i + b_2)_j  \le 0 \quad &\text{if}~ j \neq k \\
(W_1 x_i + b_1)_j, (W_2 x_i + b_2)_j    > 0 \quad &\text{if}~ j = k.
\end{align}
This means that $H(Wx_i + b) = e_k = y_i$, so $\theta \in \Theta$.
\end{proof}

%\begin{remark}
%This result also holds for $H_\epsilon$ and $\hat{H}_{\epsilon}$, and in fact
%\begin{align}
%\hat{H}_{\epsilon}(x) = H_{\frac{1-2\epsilon}{2}}(x -1).
%\end{align}
%\end{remark}

\begin{theorem}\label{theo:linearinfinity}
If $A_i$ are compact sets, then
there are infinity global minima for $L(\theta)$ if $\{A_i\}$ are also linearly separable.
\end{theorem}
\begin{proof}If $\{A_i\}$ are also linearly separable, then there exist $\theta=(W,b)$ such that
\begin{align}
H(Wx + b) = e_k \quad x \in A_k, ~ k =1:c.
\end{align}
Let us note that:
\begin{align}
(Wx + b)_k = \omega_k^T x + b_k,
\end{align}
so we have
\begin{align}
\omega_k^T x + b_k > 0 \quad &\text{if}~ x \in A_k \\
\omega_k^T x + b_k \le 0 \quad &\text{if}~ x \in A \setminus A_k.
\end{align}
By using the compactness for $A_k$, we have
\begin{align}
\min_{x\in A_k} \omega_k^T x + b_k = \epsilon > 0.
\end{align}
So, there at leat have those different $\theta_{k,\delta}$ with
\begin{align}
\theta_{k,\delta} = (W, \hat{b}),
\end{align}
and
\begin{align}
\begin{cases}
\hat{b}_i = b_i \quad &\text{if}~ i \neq k \\
\hat{b}_i = b_i - \delta \quad &\text{if}~ i = k,
\end{cases}
\end{align}
for any $0 \le \delta < \epsilon$.

Now, we next prove that, we can not only change $b$ but also $W$. First, we can assume that
\begin{align}
\omega_k^T x + b_k > 0 \quad &\text{if}~ x \in A_k \\
\omega_k^T x + b_k >0 \quad &\text{if}~ x \in A \setminus A_k.
\end{align}
If this is not true, we can change $b_k$ such that this $\theta$ can satisfy the above conditions with the above trick. Using the compactness again,
\begin{align}
\min_{x\in A_k} \omega_k^T x + b_k &\ge \epsilon > 0 \\
\max_{x\in A\setminus A_k} \omega_k^T x + b_k &\le -\epsilon < 0,
\end{align}
and $\|x\| < M $ for all $x \in A$.
So we may construct $\theta_{k,\delta}$ like:
\begin{align}
\theta_{k,\delta} = (\hat{W}, b),
\end{align}
and
\begin{align}
\begin{cases}
\hat{\omega}_i = \omega_i \quad &\text{if}~ i \neq k \\
\hat{\omega}_i = \omega_i + \omega \quad &\text{if}~ i = k,
\end{cases}
\end{align}
with $\omega \in B(0, \frac{\delta}{M}) \subset \mathbb{R}^n$ for any $0 < \delta < \epsilon$. Then we can check that
\begin{align}
\hat{\omega}_k^T x + b_k > 0 \quad &\text{if}~ x \in A_k \\
\hat{\omega}_k^T x + b_k >0 \quad &\text{if}~ x \in A \setminus A_k.
\end{align}
This means $\theta_{k,\delta}$ with above form also belongs to $\Theta$.
This shows that, $\Theta$ is infinity set and have both degree of freedom on $W$ and $b$.
\end{proof}

\begin{theorem}
If the perceptron can classify those data, then the SGD method for $\min L(\theta)$ will convergence with small positive learning rate.
\end{theorem}

\subsection{Separable and Global Minima}
In simple terms, we assume that those $\{A_k\}_{k=1}^c$ are compact sets. Then the positive distance can be got from the consistency.  Here we note that, we want to use general DNN model with just one hidden layer to separate those data, so the loss function will be:
\begin{align}
L(\theta) = \sum_{i=1}^N\|H(W_2\sigma(W_1x_i + b_1) + b_2) - y_i\|^2,
\end{align}
with $H$ could be $\hat{H}_{\epsilon}$ or $H_\epsilon$.

\begin{theorem}Let
\begin{align}
\Theta := \mathop{\arg\min}_{\theta}L(\theta).
\end{align}
For $(\theta_1, \theta_2) \in \Theta$, we note that
\begin{equation}
\Theta_2(\theta_1) = \{\theta ~|~ (\theta_1, \theta) \in \Theta \}.
\end{equation}
Then $\Theta_2(\theta_1)$ is a convex set if the DNN model can classify those data with compactness assumption.
\end{theorem}

\begin{proof}
Because of the continuity of $g(x) = \sigma(W_1x + b_1)$, so we have that $\{g(A_k)\}_{i=k}^c$ are also compact sets, and the consistency of $A_k$ also shows the consistency of $g(A_k)$.

Then because DNN model can classify those data, so $g(A_k)$ are also linear separable w.r.t to $\hat{H}_\epsilon$.
So, for fixed $\theta_1$, we can finish this proof by using Theorem \ref{theo:linearconvex}.
\end{proof}

\begin{theorem}If $A_i$ are compact sets, then
$\Theta_2(\theta_1)$ is a infinity set if $\{A_i\}$ can be separated by DNN model.
\end{theorem}
\begin{proof}Use the analysis in the theorem above and we can get a similar poof with Theorem \ref{theo:linearinfinity}.
\end{proof}

Next, we want to talk about some properties about $\theta_1$ part. We can also get some infinity property for
\begin{equation}
\Theta_1(\theta_2) = \{\theta ~|~ (\theta, \theta_2) \in \Theta \}
\end{equation} with some special $\theta_2$.

\subsection{Discussion with Prof. Xu on 3/10/2017}
Prof. Xu proposed those next points about vanishing gradient and random strategies in training algorithms:
\begin{enumerate}
\item ``Vanishing Gradient' is a mathematical problem.

\item For deep neural network models, ReLU activation function is better than Sigmoid in many cases just likes that singular system can be solved more easily than nearly singular system.

\item For some components in $\frac{\partial f^J_i}{\partial \theta^k_{st}} \approx 0$, we may just take those as 0.

\item For classification problems, if the class is not big, like $c = 10$, and we may have $100000$ data, and mini-batch size is $100$, which means for every mini-batch, we have $10$ samples in every class. One way for good "mini-batch" is to choose those $10$ samples in the same class as separable as possible. {\bf And in some degree, SGD is doing this. }

If the class is too big, like $c = 1000$ but mini-bath size is still $100$, we can change our mind to choose those mini-batch just comes from one class and also as separable as possible.

Here is a suggested strategy for how to choose those mini-batch  as separable as possible from Prof. Xu. For example, we have $N$ data for one class, and we want to choose $m$ data from this data set  as separable as possible. We may use next steps:
\begin{itemize}
\item Use KNN method for those data with $k = m$.
\item Use the uniform sampling strategy to choose one data in every cluster, then we get $m$ samples.
\end{itemize}

\end{enumerate}

\subsection{Train good smoother for multigrid methods}
We are going to using the idea of deconvolution to get some good smoother in multigrid method with fixing the restriction and interpolation method.

Data $\{u_k, f_k\}_{k=1}^N$ such that
\begin{equation}
A u_k = f_k,
\end{equation}
with $A$ is the linear FEM matrix with uniform grids.

Then we have stander nested multi-grids like $\{\mathcal{G}_j \}_{j=1}^J$, and $P_{j}^{j+1}$ is the interpolation operator form $\mathcal{G}_j$ to $\mathcal{G}_{j+1}$. We may have $S_j$ as the smoother in grid $\mathcal{G}$, there are also what we will trained. Under those set up, we may note $B(S_j)$ as the multigrid solve for $B \approx A^{-1}$.

So we optimization problem is:
\begin{equation}
\min_{S_j} \sum_{k=1}^N\| u_k - B(S_j)f_k\|^2.
\end{equation}

There are two points we need to take care of:
\begin{itemize}
\item How to choose those data $\{u_k, f_k\}_{k=1}^N$?

For this problem, Prof. Xu, suggested that we need to choose good data by choosing fine $f_k$ not $u_k$.

\item How to solve those above optimization problem?

One way to train this model is to compute
\begin{equation}
\sum_{k=1}^N\| u_k - B(S_j)f_k\|^2 = \sum_{k=1}^N \| (I - B(S_j)A)u_k\|^2.
\end{equation}
And then, we may use ``BP" for $I - B(S_j)A$ for $S_j$ from $j = J: -1 :1$? But, how to implement this, we may need to check more details.
\end{itemize}

\subsubsection{Another viewpoint for this problem}
We can use the idea for best convergence rate with
\begin{equation}
\min_{S_j} \|I - B(S_j)A\|^2,
\end{equation}
then the above question is the next $min-max$ problem:
\begin{equation}
\min_{S_j} \max_{\|u\| = 1}\|(I - B(S_j)A)u\|^2.
\end{equation}
And then we may take this method as:
\begin{equation}
 \sum_{k=1}^N \| (I - B(S_j)A)u_k\|^2 \approx \max_{\|u\| = 1}\|(I - B(S_j)A)u\|^2.
\end{equation}

\subsection{Seminar notes on 11/31/2017}
\begin{itemize}
\item Runge phenomenon and overfitting.
\item How about sample points and interpolation and Runge phenomenon.
\end{itemize}

\subsection{Discussion on 12/13/2017}
\subsubsection{Statistical learning based regression(classification)
  problems}

Here we suppose that for all image with label and task we want to solve, we have the next joint distribution in real world as:
\begin{equation}\label{eq:jointdit}
(X,Y) \sim p_{\rm data},
\end{equation}
as
\begin{equation}\label{eq:defPr}
\rm{Pr}(X \in I_x, Y \in I_y) = \int_{I_x \times I_y}p_{\rm data}(x,y)dxdy,
\end{equation}
with $X \in \mathbb{R}^{n\times n}$ and $Y \in \mathbb{R}^c$.

So for regression problem, the function that we want to interpolate is:
\begin{equation}\label{eq:deff}
f^*(x) = \mathbb{E}_Y (Y | X = x).
\end{equation}
This is in fact that
\begin{equation}\label{eq:f*}
f^* = \mathop{\arg\min}_{f ~\text{is a distribution}}\mathbb{E}_{X\times Y} \|y - f(x)\|^2.
\end{equation}

For classification problem, the values of $Y$ should be some discrete values such as $\{e_1, \cdots, e_c\}$. So the reals problem that we may try to solve is to use some function family to approximate the $f^*$ by:
\begin{equation}\label{eq:stat_opt}
\mathbb{E}_{X} (\|f^*(x) - f(x;\Theta)\|^2) = \mathbb{E}_{X\times Y}(\|y - f(x;\Theta)\|^2).
\end{equation}

\hrule
\paragraph{Jinchao:}  In general, we assume that the label $y$  of a given
data $x$ is also a random variable depending on $x$.  For example, for
$c$-classification problem,
$$
\sum_{k=1}^cP\{Y=e_k | X=x\}=1.
$$

$$
Y: \{e_1,\ldots e_k\}\mapsto \mathbb R^c.
$$

There are at least five different concepts:
\begin{enumerate}
\item The original expected loss
\item The empirical loss with $N$-sampled data
\item SGD directly from the original data [training]
\item SGD from the $N$-sampled data [training]
\item SGD with epoch from $N$-sampled data [training]
\end{enumerate}

Jinchao's questions:
\begin{enumerate}
\item From pure math point of view, generalization error is a concept
  for model, not for training algorithms
\item In the literature, people often relate the generalization error
  with a training algorithm.   This would only make sense if we treat
  a training algorithm as a modeling procedure.
\item In other words, the training algorithm, when finished in
  practice, has not really solved the original model problem
  accurately, namely a global minimizer or a ``good local minimizer''
\end{enumerate}

Traditionally, generalization error = test error.

For expected loss, the concept of ``training data'' and ``test data''
should not apply.

``test data'' are those data from the original data space (for
expected loss) that are not in the ``training data''

In practice, we should never aim to accurately find a global minimizer
of the empirical loss function, which is why we need ``validation
data''.  The role of the validation data

\begin{quote}
{\bf Jinchao's conjecture:}  Under reasonable assumptions, the exact
global minimizer of the empirical loss function would for sure lead to
over-fitting.
\end{quote}

In this case, we can find a DNN so that the emirical lost function is
zero.

We also note that a global minimizer of the expected loss function is
also a global minimizer of the empirical loss function.  But in
practice, it is hard to find a global minimizer of the expected loss
function.  The practical approximation of the global minimizer would
lead to over-fitting.

The above conclusions are not valid for cross-entry with soft-max.

There are many kinds of data
\begin{enumerate}
\item All feasible data:
\item training data
\item validation data
\item test data
\end{enumerate}

The labelled data can be divided to, subjectively or statistically,
into
\begin{enumerate}
\item training data
\item validation data
\item test data
\end{enumerate}
\bigskip
\hrule
\bigskip

\subsubsection{Manifold distance}
For classification problem, we may state that
\begin{equation}\label{def:manifoldAk}
A_k = \{x: f(x) = e_k\},\quad 1 \le k \le c,
\end{equation}
with $f$ is defined above \eqref{eq:deff}. Then the manifold distance means that
\begin{equation}\label{key}
\rm{dist}(A_k, A_l) \ge \delta > 0, \quad k \neq l.
\end{equation}