\lab{Algorithms}{Speeding up Python Functions: Vectorization}{Vectorization}

\objective{Demonstrates the importance of vectorization.}

This lab introduces two applied topics (Information Entropy and Chebyshev Polynomials) to explore how we can make functions faster through vectorization.  There are two primary motivations behind vectorizing code.  First, vectorized code is often much clearer and more concise than non-vectorized code.  The second, much larger motivation is performance.  Vectorized code is often magnitudes faster than non-vectorized code.

The concept behind vectorizing code is move away from operating on one element at a time to operating on entire collections of elements at once.  This section will focus specifically on reducing the number of for loops in our program.  Let us look at two examples of functions that would be really useful if could vectorize them to make them execute faster.

\section*{Information Entropy}

Information entropy is loosely defined as the amount of information that is encoded in each bit of a signal. It is directly related to amount of compression that a signal can undergo without loss.

For example, suppose that our signal is an infinite string ``AAAAAAA..." Since the signal never changes, each bit additional bit after the first add no new information. Thus the entropy in this case is zero.

On the other hand, we assign a truly random uniform binary source an entropy of one (such a source may be related to radioactive decay). This is since we have absolutely no clue what the next symbol we read will be.

We can empirically calculate the entropy of a given signal by the following:
\[
H = -\sum_k{p_k log_2(p_k)}
\]
where $p_k$ is the probability of the kth symbol. Note that this fits the two conventions we set above since
\[
-(1log_2(1) + 0 log_2(0)) = 0
\]
\[
-(1/2log_2(1/2) + 1/2log_2(1/2)) = 1
\]

\begin{problem}
Write a function that finds the entropy of a array (you will need to flatten multidimensional arrays to a single dimension). You will also need to empirically calculate the probability of a given symbol (start by letting your symbols be integers). To find the probability of a symbol, you need to find the number of times it occurs in the sequence and divide it by the length of the sequence (watch out for integer division).
\end{problem}

We can test the entropy function that you just wrote against bit sources to see how ``random" they are. For example, we can read an image into Python and view it using the following code:
\begin{lstlisting}[style=python]
: from scipy import misc
: img = misc.imread('cameraman.tif'); img
\end{lstlisting}

\begin{figure}[h!]
\begin{center}
\includegraphics{./Figures/CameramanClean.pdf}
\end{center}
\caption{The MIT camerman, a classic image used in image processing}
\label{fig:cameramanclean}
\end{figure}
You should see the image shown in Figure \ref{fig:cameramanclean}. Now pass this image to our entropy function to calculate the entropy of the image data. We get a value of $7.009$. If this image were uniformily random, the entropy would be $8$, since each pixel is an $8$ bit number, and a uniform distribution of integers between $1$ and $128 = 2^8$ will have an entropy of $8$ (you can verify this fairly easily using the equation we defined above). This implies that the image has potential for compression.

\begin{problem}
Now rewrite your function without using loops. This is an example of using vectorization to speed up algorithms. Use the cameraman image to test how much faster this function is than your original.
\end{problem}

\section*{Chebyshev Polynomials}

One special type of polynomials are the Chebyshev Polynomials. They are useful in a variety of applications including:
\begin{itemize}
\item Evaluating integrals using Gaussian quadrature
\item Solving PDE's
\item Speeding up iterative matrix methods (finding eigenvalues and solving large systems)
\item minimizing interpolation error
\end{itemize}

We will explain many of the these applications in subsequent chapters. The purpose of this example is primarily to give you a real opportunity to investigate ways to speed a process up on your own.

The value of the n-th Chebyshev Polynomials can be described by the following mathematical formulae\footnotemark :
\[
T_n(x) = cos(n cos^{-1}(x))
\]

\[
T_n(x) = 2xT_{n-1}(x) - T_{n-2}(x), T_0(x) = 1, T_1(x) = x
\]

\footnotetext{Technically the first formula only works on the interval $[-1,1]$. There are more general formulae, piecewise defined using $\cosh$ outside of $[-1,1]$, but for this exercise we'll stick to the above definition.}

\begin{problem}
Write a function that accepts a vector $x_0$ of values in $[-1,1]$ and a degree $n$.  The function should return an array evaluting each entry of $x_0$ at every chebeshev polynomial of degree $n$ or less. Essentially, the n-th column of the output should contain the $n-1$ Chebyshev Polynomial evaluated at $x_0$.

You will be writing four slightly different versions of this function.
\begin{enumerate}
\item Use only for loops and Python's \li{math} library (you will need two for loops).
\item Same as previous, but replacing one for loop with a list comprehension.
\item Use \li{sp.vectorize} to vectorize Python's \li{math} functions.  Keep the list comprehension from previous part.
\item Use SciPy's optimized $\cos$ and $\arccos$ functions.
\end{enumerate}

Make this function as fast as possible (this is the point of the exercise, writing a function solving this problem shouldn't be too difficult anymore). You should investigate both mathematical definitions, and how the number of {\tt for} loops affects the speed. Explain how you optimized your function, and why other implementations would be slower.
\end{problem}


\subsection{Gaussian Process Regression}
\label{gp}

Evaluating the production curve for a specific kind of facility using
full fuel cycle simulations is relatively expensive, even in the
computationally cheapest case of low-fidelity simulations. This is because a
fuel cycle realization
typically computes many features that, though coupled to the production
curve, are not directly the production curve. For example, the mass balance of
the fuel cycle is physically bound the electricity production. However,
in the particular once-through demonstration case used throughout this paper,
the mass balances are not explicitly taken into account when trying to meet
a power demand curve.

Alternatively, surrogate models that predict the production curve directly
have many orders-of-magnitude fewer operations by virtue of not computing
implicit physical characteristics. This is not to say that the surrogate
models are correct.  Rather, they are simply good enough to drive a demand
curve
optimization. Surrogate models are used here inform a simulator about where
in the parameter space to look next. Truth about production curves should
still be derived from the fuel cycle simulator and not the surrogate model.
In the WORG algorithm, Gaussian processes are used to form the model.

Gaussian processes are more fully covered elsewhere
\cite{rasmussen2006gaussian}. Using Gaussian process for optimization has
also been previously explored \cite{osborne2009gaussian}, though such studies
tend not to
investigate the integral problems posed by facility deployment. As with
dynamic time warping, a minimal but sufficient introduction to GP regression
is presented
for the purposes of the deployment optimization.
Consider the case of $Z$ simulations indexed by $z$ that each have a
$\Theta_z$ deployment schedule and $g_z(t, \Theta_z)$ production curve.

A Gaussian process of these $Z$ simulations is set by its mean and
covariance functions. The mean function is denoted as $\mu(t, \Theta)$ and
is the expectation value $\E$ of
the series of $G$ inputs:
\begin{equation}
\label{G}
G = \left\{g_1(t, \Theta_1), g_2(t, \Theta_2), \ldots,
           g_Z(t, \Theta_Z)\right\}
\end{equation}
The covariance function is denoted $k(t, \Theta, t^\prime, \Theta^\prime)$
and is the expected value of the input to the mean. The mean and
covariance can be expressed as
in Equations \ref{mean-func} \& \ref{covar-func} respectively.
\begin{equation}
\label{mean-func}
\mu(t, \Theta) = \E G
\end{equation}
\begin{equation}
\label{covar-func}
k(t, \Theta, t^\prime, \Theta^\prime) =
    \E\left[(g_z(t, \Theta) - \mu(t, \Theta))
            (g_z(t^\prime, \Theta^\prime) - \mu(t^\prime, \Theta^\prime))
      \right]
\end{equation}
Note that in the above, the Gaussian process is itself $P+1$ dimensional,
since the means and covariance are a function of both the deployment
schedule ($P$) and time ($+1$).

The Gaussian process $\GP$ approximates the production curve
given $Z$ simulations. Allow $*$ to indicate that the a quantity comes from
the model as opposed to coming from the results of the simulator. A model
production curve can then be written using either functional or operator
notation, as appropriate:
\begin{equation}
\label{gp-def-approx}
g_*(t, \Theta) \approx \GP\left(\mu(t, \Theta),
                                 k(t, \Theta, t^\prime, \Theta^\prime)\right)
                \equiv \GP G
\end{equation}
In machine learning terminology, $G$ serves as the training set for the
GP model. Furthermore, if the individual $g_z$ return a
multivariate result, then the model $g_*$ will also compute a result of
the same rank.

Now, when performing a regression on Gaussian processes,
the nominal functional form for the covariance must be given.
Such a functional form is also known as the the kernel function.
The kernel contains the \emph{hyperparameters} that are solved for to
obtained a best-fit Gaussian process. The hyperparameters themselves are
defined based on the definition of the kernel function. Hyperparameter
values are found via a regression of the maximal likelihood of
the production curve. Any functional form could potentially serve as a kernel
function. However, a generally useful form is the is the exponential
squared. This kernel can be seen in Equation \ref{exp2-kernel} with
hyperparameters $\ell$ and $\sigma^2$ for a vector of parameters $r$:
\begin{equation}
\label{exp2-kernel}
k(r, r^\prime) = \sigma^2 \exp\left[-\frac{1}{2\ell}(r - r^\prime)^2 \right]
\end{equation}
However, other kernels such as the Mat\'ern $3/2$ kernel and Mat\'ern $5/2$
kernel \cite{paciorek2004nonstationary} were observed to be more robust for
the WORG method. These can be seen in Equations \ref{matern-32} and
\ref{matern-52} respectively.
\begin{equation}
\label{matern-32}
k(r, r^\prime) = \sigma^2
                 \left(1 + \frac{\sqrt{3}}{\ell}|r - r^\prime|\right)
                 \exp\left(-\frac{\sqrt{3}}{\ell}|r - r^\prime|\right)
\end{equation}
\begin{equation}
\label{matern-52}
k(r, r^\prime) = \sigma^2
                 \left(1 + \frac{\sqrt{5}}{\ell}|r - r^\prime|
                         + \frac{5}{3\ell^2}|r - r^\prime|^2\right)
                 \exp\left(-\frac{\sqrt{5}}{\ell}|r - r^\prime|\right)
\end{equation}

From here, say that $\K$ is a covariance matrix
such that the element at the $r$-th row and $r^\prime$-th column is
given by whichever kernel is chosen from
Equations \ref{exp2-kernel}-\ref{matern-52}. Then the
log likelihood $\log q$ of obtaining the training set production curves
$G$ for a given time grid $\mathbf{t}$ and deployment schedule is as
seen in Equation \ref{log-q}.
\begin{equation}
\label{log-q}
\log q(G|\mathbf{t}, \Theta)
    = -\frac{1}{2}G^\top\left(\K + \tau^2\I\right)^{-1}G
      -\frac{1}{2}\log\left|\K + \tau^2\I\right|
      -\frac{ZTP}{2}\log 2\pi
\end{equation}
Here, $\tau$ is the uncertainty in the production curves coming from the
simulations themselves. As most simulators do not report such uncertainties,
$\tau$ may be set to floating point precision. $\I$ is the usual identity
matrix. The hyperparameters $\ell$ and $\sigma^2$ are then adjusted via
standard real-valued optimization methods such that Equation \ref{log-q} is
as close to zero as possible.
This regression of the Gaussian process itself yields the most likely
model of the production curve knowing only a limited number of simulations.

However, the purpose of such a Gaussian process regression is to evaluate
the production curve at points in time and for deployment schedules that
have not been simulated. Take a time grid $\mathbf{t_*}$ and a hypothetical
deployment schedule $\Theta_*$. Now call the covariance vector between
the training set and the model evaluation
$\mathbf{k}_* = \mathbf{k}(\mathbf{t_*}, \Theta_*)$.
The production curve predicted by this Gaussian process is then given by
the following:
\begin{equation}
\label{metric-model}
\mathbf{g}_*(\mathbf{t}_*, \Theta_*) =
    \mathbf{k}_*^\top \left(\K + \tau^2\I\right)^{-1}G
\end{equation}
Equations \ref{mean-func}-\ref{metric-model} are derived and discussed fully
in \cite{rasmussen2006gaussian}.

Implementing the above Gaussian process mathematics for the specific
case of the WORG algorithm
is not needed.  Free and open source Gaussian process modeling software
libraries already exist and are applicable to the regression problem here.
Scikit-learn v0.17 \cite{scikit-learn} and George v0.2.1 \cite{hodlr}
implement such a method and have a Python interface. George is specialized
around Gaussian processes, and thus is preferred for WORG over scikit-learn,
which is a general purpose machine learning library.

As an example, consider a Gaussian process between two power production
curves similar to the example used in \S\ref{dtw}. The first is a nominal 1\% growth
in GWe for 50 years starting at
90 GWe in 2016. The second curve under-produces the first curve by 10\%
for the first 25 years and over-produces by 10\% for the last 25 years.
Additionally, assume that there is a 10\% error on the training set data.
This will produce a model of the mean and covariance that splits the
difference between these two curves. This example may be seen graphically
in Figure \ref{gwe-model-}.

\begin{figure}[htb]
\centering
\includegraphics[width=0.9\textwidth]{gwe-model-.eps}
\caption{The Gaussian process model of a 1\% growth curve along with the
an initial 10\% under production followed by a 10\% under production.
The model is represented by the black line that runs between the red
training points. Two standard deviations form the model are displayed as the
gray region.}
\label{gwe-model-}
\end{figure}

The simple example above does not take advantage of an important
feature of Gaussian processes. Namely, it is not limited to two production
curves in the training set.  As many as desirable may be used.  This will
allow the WORG algorithm to dynamically adjust the number of $Z$ simulations
which are used to predict the next deployment schedule. WORG is thus capable
of
effortlessly expanding $Z$ when new and useful simulations yield valuable production
curves.  However, it also enables $Z$ to contract to discard production
curves that would drive the deployment schedule away from an optimum.

Now that the Gaussian process regression and dynamic time warping tools have
been added to the toolbox, the architecture of the WORG algorithm can
itself be described.

\clearpage


\section{Types}
Types in Amulet are composed of functions, tuples and primitives. You can store multiple types together in a tagged union (or sum type). A tagged union with one entry is a useful way of boxing a type to ensure that it cannot be accidentally used as an object of the same type but different semantics.

Types can also be generic over one or more parameters. When inferring the type for an expression, the most generic type will always be used. Constraints can be put on generic parameters by specifying type class implementations that must exist for this set of parameters. There can be multiple constraints, each applying to multiple parameters.

There is no subtyping in Amulet, \textit{however} type constraints can be loosened through the upcast operator: this converts an existing value to a more general type: for instance \texttt{[a]} can be upcast to \texttt{Show b}.

\subsection{Types and kinds}
\begin{quote}
A kind is the type of a type constructor or, less commonly, the type of a higher-order type operator. A kind system is essentially a simply typed lambda calculus "one level up", endowed with a primitive type, denoted $*$ and called \texttt{Type}, which is the kind of any data type which does not need any type parameters.\cite{kindsWikipedia}
\end{quote}

The base lambda calculus in Amulet is that which operates on values, namely the input program. Every value must have a corresponding type. 0 might have type \texttt{int} and the empty list type \texttt{[a]}. These have kinds $*$ and $* \to *$ respectively, where $*$ is a shorthand notation for \texttt{Type}.

However $*$ is not the only kind in Amulet. Constraints, such as \texttt{Show Int}, also have a kind called, imaginatively, \texttt{Constraint}.  However type classes generally take one or more parameters and so have kind  $* \to \mathtt{Constraint}$.

\subsubsection{Unions and rows}
In order to build more complex types Amulet adds row and union kinds. These are polymorphic over kinds and so have the signature $\square \to \square$. Here the $\square$ symbol represents all kinds such as types, constraints, rows and unions. These can then be used to create a compound type: $\mathtt{Row} \to *$

Union kinds represent a unique, unordered set of a given kind. They are denoted by \texttt{Union} or $\%$. Unions are written using basic set notation: $\left \{  \mathtt{Int}, \mathtt{Bool} \right \}$ is a set containing the \texttt{Int} and \texttt{Boolean} types.

Row kinds represent a map of string keys to kinds (like a disjoint union but with strings). This is denoted by \texttt{Row} or $\#$. Rows are written with the key value pairs within parenthesis: $\left ( \mathtt{name : String}, \mathtt{age : Int} \right )$.

Unions and rows share similar semantics as they allow being combined with another object of the same kind. For instance you can add \texttt{String} to the union \texttt{a} with  $\left \{ \mathtt{String} | \mathtt{a} \right \}$. This also highlights the fact that unions and rows can be polymorphic (this case is $\mathtt{Row} \to \mathtt{Row}$). This simple fact allows emulating subtyping of these kinds. When creating a union of row objects where both sides have the same key the value must be unifiable.

It is also possible to take the union of two polymorphic kinds such as $\left \{ { a | b }\right \}$ (however, type inference is not guaranteed).

\subsection{Universal quantification}
Universal quantification, or $\forall$ is a way of marking a type that can exists for all types matching its criteria. For instance the identity function would be universally quantified as any type can be applied to it.

\subsubsection{Constraints}
For universal quantification to be useful, constraints must be added to the type. Constraints come in two forms: a type class constraint, or a unification requirement.

The former requires that a particular type class instance exists: $\forall a . \mathtt{Num}\ a \Rightarrow a \rightarrow a$ states that $\mathtt{Num}\ a$ must exist. It is worth noting that any number of variables (or constant types) can be give
n to the type class: $\forall a. \mathtt{Num\ Int} \Rightarrow a \rightarrow a$ is equally valid, though meaningless as the only valid implementation would be the identity function.

A unification requirement states that two types must unify for this type to exist. An simple example might be $\forall a. a \sim \mathtt{Int} \Rightarrow a$. This type is pointless as the only value of $a$ which will satisfy this requirement is \texttt{Int}. However, this can be useful with more complex types such as rows or unions.

\subsection{Record types}
Record types express a way of handling key-value pairs where the keys are known at compile time. Records are implemented using rows, with a hidden polymorphic argument, allowing for converting records with many fields to records with less. Strictly records are defined as:

\begin{minted}{haskell}
data Record p = forall a . Record ( a |  p )
\end{minted}

\subsection{Tagged unions}
Tagged unions, also known as \textit{sum-of-product types} or \textit{Algebraic Data Types}, are data structures that can take on several variations. Only one of these variants may be in use at once, and a tag field explicitly indicates so.

Tagged unions are represented using a row, where each entry is the type of the corresponding entry. For example:

\begin{minted}{haskell}
data Expr = Var Name
          | Abs Name Expr
          | App Expr Expr
\end{minted}
could be represented as
\begin{minted}{haskell}
type ExprTotal = (
    Var : Name -> Expr,
    Abs : Name -> Expr -> Expr,
    App : Expr -> Expr -> Expr,
)

data Expr = Expr (Union ExprTotal)
\end{minted}

\texttt{Union} is an internal Amulet type which is understood by the compiler to form a union type.

\paragraph{Using the union kind}
It would be nice to replace this with a union as it doesn't \textit{need} to have the types of the constructor. However this is hard as the union is over types rather than strings.

\subsubsection{Narrowed functions}
The main benefit of using rows to represent ADTs is that we can define functions which only take a subset of the values: for instance `head` could only accept non-nil lists.

For example, require a function which accepts \texttt{Abs} or \texttt{App} but not \texttt{Var}. Lets us consider the type $\mathtt{ExprPartial}\ p$ which represents a subset of \texttt{Expr}. We can define these requirements for this function:

You should be able to pass in anything of the form:
\begin{itemize}
\item \texttt{ExprPartial (App : _, Abs : _)}
\item \texttt{ExprPartial (App : _)}
\item \texttt{ExprPartial (Abs : _)}
\end{itemize}

But not of types:

\begin{itemize}
\item \texttt{ExprPartial ()}
\item \texttt{ExprPartial (Var : _ | p)}
\end{itemize}

Namely the arguments passed to this required function must be a subset of \texttt{(App : _, Abs : _)} and not an empty list. The first is trivial to solve: we can define a constraint $\mathtt{RowSubset}\ p a$ which requires that $a \subset p$.

\begin{minted}{haskell}
type RowSubset p a = (a | p) ~ p
\end{minted}

More difficult is defining a type which refuses the empty row. This could either be done with a non-empty-row constraint, or a more general "does not unify" requirement. For now we will consider the former and call it $\mathtt{NonEmpty}\ a$.

We can then give the definition of $\mathtt{ExprPartial}\ p$ and of our function:

\begin{minted}{haskell}
data ExprPartial a = ExprPartial (RowSubset ExprTotal a)

type Partial p = forall a . NonEmpty a => RowSubset p a => p
func : ExprPartial (Partial (App : _, Abs : _))
\end{minted}

Of course \texttt{Expr} can now be generalized to:

\begin{minted}{haskell}
type Expr = ExprPartial ExprTotal
\end{minted}

\subsubsection{Pattern matching}
This form of narrowing can also be used in pattern matching. Each case can remove one of the possible values of `p` from `ExprPartial`:

\begin{minted}{haskell}
case (x : Expr) of
| Abs name _ = name
| y = ... -- y : ExprPartial (Var : _, App : _) as it cannot be Abs if you hit this point
\end{minted}

\subsubsection{GADTs}
It is also possible to define GADTs in the same way, though obviously the resulting type is not just \texttt{Expr}. For instance, the GADT

\begin{minted}{haskell}
data Expr a where
    I   : Int  -> Expr Int
    B   : Bool -> Expr Bool
    Add : Expr Int -> Expr Int -> Expr Int
\end{minted}

would be represented:

\begin{minted}{haskell}
type ExprTotal a = (
    I   : Int -> Expr Int
    B   : Bool -> Expr Bool
    Add : Expr Int -> Expr Int -> Expr Int
)
\end{minted}

\subsection{Type classes}
Type classes are a way to ensure different types have a consistent interface, and to constrain polymorphic types on having an instance of a type class. A fully instantiated type class resolves to a type of \texttt{Constraint}.

A constraint section on a type is given as a fat-arrow behind the type. For example, the following snippet says that ``for all types for which there is an instance of \texttt{a}, there is a function \texttt{show} transforming it into a \texttt{string}.''

\begin{minted}{haskell}
show : forall a. Show a => a -> String
\end{minted}

The type-class mechanism poses a way to do name overloading based on types. In contrast to, say, Java interfaces, type classes allow the user to be parametric on the return type, not only the parameter types.

\begin{minted}{haskell}
class Read a where
  read : String -> a
\end{minted}

When resolving type classes the type checking algorithm will unify the current type across all instances. If more than one instance matches then an error will be created.

Type classes can also be named and a specific implementation can be used at compile time. Implementations are passed in to the type lambda. For instance using the above show definitions we get:

$$
\mathtt{show} = \Lambda (a : \mathtt{Type})\ (x : \mathtt{Num}\ a) \to \lambda (x : a)\ (y : a) \to function\ body
$$

Type classes can take have multiple parameters. In the case where some parameters define others, functional dependencies\cite{funcDeps} can be used to prevent conflicting definitions.

Some type classes will be auto-generated by the compiler (such as \texttt{Eq} or \texttt{Show}) if the criteria are matched (for instance all child types also implement \texttt{Show}). Auto-generated instances can safely be overridden or masked: there will not be a compile error if both a auto-generated and normal instance match.

\paragraph{Named implementations}
All type class instances require an explicit name. This allows us to explicitly specific implementations to functions.

\paragraph{Type families}
Type classes can also include type definitions\cite{typeFams}, allowing for type aliases and data definitions to exist and be instantiated within a type class.


%---------------------------Area-----------------------------
\section{Area\label{s:quad-area}}

Signed area, defined as
\[
q = \frac {1} {4} \sum_{i=0}^3 \alpha_i
\]
is useful for two purposes: first, the sign can indicate elements
that have vertices ordered incorrectly or arranged in a concave
pattern; and second, the magnitude can be used to identify
elements that are too small for accurate analysis.
Figure~\ref{f:quad-areas} shows how each vertex area contributes to
the total area of a quadrilateral.

\begin{figure}[htb]
  \centering
  \includegraphics[width=2in]{quad-areas}
  \caption{The areas associated with each vertex may be summed and
           weighted to get the area of the entire quadrilateral.%
                                                                  \label{f:quad-areas}}
\end{figure}

\quadmetrictable{area}%
{$L^2$}%                                    Dimension
{$[0,DBL\_MAX]$}%                           Acceptable range
{$[0,DBL\_MAX]$}%                           Normal range
{$[-DBL\_MAX,DBL\_MAX]$}%                   Full range
{$1$}%                                      Unit square
{--}%                                       Citation
{v\_quad\_area}%                            Verdict function name


\subsection{Topology of Euclidean spaces}\label{subsec:topology_of_euclidean_spaces}

\begin{definition}\label{def:euclidean_space}
  \todo{Define euclidean spaces}.
\end{definition}

\begin{theorem}\label{thm:real_metric_and_order_topologies_coincide}
  For the real numbers, the \hyperref[def:metric_topology]{metric} and \hyperref[def:order_topology]{order topologies} coincide.
\end{theorem}
\begin{proof}
  The metric topology \( \mscrT_M \) is generated by the \hyperref[def:topological_base]{base}
  \begin{equation*}
    \mathcal{B} \coloneqq \{ B(x, r) \colon x \in \BbbR, r \in \BbbR_{>0} \}
  \end{equation*}
  and the order topology \( \mscrT_O \) is generated by the \hyperref[def:topological_subbase]{subbase}
  \begin{equation*}
    \mathcal{P} \coloneqq \{ (a, \infty) \colon a \in \BbbR \} \cup \{ (\infty, b) \colon b \in \BbbR \}.
  \end{equation*}

  The inclusion \( \mathcal{B} \subseteq FI(\mathcal{P}) \) is obvious since any ball \( B(x, r) \) is the intersection of the two rays
  \begin{equation*}
    B(x, r) = (x - r, \infty) \cap (-\infty, x + r).
  \end{equation*}

  Thus, \( \mscrT_M \subseteq T_O \). We now only need to show that \( \mathcal{B} \) is a base for \( \mscrT_O \).

  Let \( U \in T_O \). Since \( FI(\mathcal{P}) \) is a base for \( \mscrT_O \), there \hyperref[def:topological_base/union]{exists} a family \( \{ U_i \}_{i \in I} \subseteq FI(\mathcal{P}) \) such that
  \begin{equation*}
    U = \bigcup_{i \in I} U_i.
  \end{equation*}

  We only need to express every \( U_i \) as a union of balls from \( \mathcal{B} \). There are several possibilities:
  \begin{itemize}
    \item if \( U_i \) is the open interval \( (a, \infty) \),
          \begin{equation*}
            (a, \infty) = \bigcup_{i=1}^\infty B(a + i, 1).
          \end{equation*}

    \item if \( U_i \) is the open interval \( (-\infty, b) \),
          \begin{equation*}
            (-\infty, b) = \bigcup_{i=1}^\infty B(b - i, 1).
          \end{equation*}

    \item if \( U_i \) is the intersection \( (a, \infty) \cap (-\infty, b), a < b \),
          \begin{equation*}
            (a, \infty) \cap (-\infty, b) = B(\tfrac {a + b} 2, \tfrac {b - a} 2)
          \end{equation*}

    \item if \( U_i \) is the empty set,
          \begin{equation*}
            \varnothing = \bigcup \varnothing \text{ (see \fullref{def:basic_set_operations/union})}.
          \end{equation*}
  \end{itemize}

  Thus, \( U_i \) is the union of an at most countable amount of balls. The countable union of countable sets is again countable, hence by \fullref{def:topological_base/union}, \( \mathcal{B} \) is a base for \( \mscrT_O \).
\end{proof}

\begin{proposition}\label{thm:rn_bounded_iff_totally_bounded}
  A set in \( \BbbR^n \) is totally \hyperref[def:totally_bounded_set]{bounded} if and only if it is \hyperref[def:metric_space/bounded_set]{bounded}.
\end{proposition}
\begin{proof}
  \SufficiencySubProof Follows from \fullref{thm:totally_bounded_sets_are_bounded}.
  \NecessitySubProof Let \( A \) be a bounded set in \( \BbbR^n \) and let \( B(x, r) \) be a ball containing \( A \). Fix \( \varepsilon > 0 \).

  Denote by \( e_1, \ldots, e_n \) the \hyperref[def:hamel_basis]{basis} of \( \BbbR^n \). Denote by \( m \) the smallest integer such that \( m \varepsilon \geq r \).

  We can create a grid around \( B(x, r) \) as follows:

  Define the set
  \begin{equation*}
    \left\{ x + \sum_{i=1}^n [k_i \varepsilon] e_i \colon \forall i = 1, \ldots, n: k_i = 1, \ldots, m \right\}.
  \end{equation*}

  is finite. Furthermore, it is an \( \varepsilon \)-net of \( A \). Indeed, let \( y \in A \). Denote its coordinates along \( e_1, \ldots, e_n \) by \( y_1, \ldots, y_n \). Then \( y \) is contained in the ball
  \begin{equation*}
    B\left(x + \sum_{i=1}^n [\ceil(y_i) \varepsilon] e_i, \varepsilon \right).
  \end{equation*}
\end{proof}

\begin{theorem}[Heine-Borel theorem]\label{thm:heine_borel}
  A set in \( \BbbR^n \) is compact in the sense of \fullref{def:compact_space} if and only if it is closed and bounded.
\end{theorem}
\begin{proof}
  Follows from \fullref{thm:rn_bounded_iff_totally_bounded} and \fullref{thm:complete_metric_space_compact_conditions/closed_totally_bounded}.
\end{proof}

\begin{proposition}\label{thm:real_supremum_of_closure}
  The supremum (resp. infimum) of a set \( A \subseteq \BbbR \), if it exists, is equal to the supremum (resp. infimum) of \( \cl A \).
\end{proposition}
\begin{proof}
  \SufficiencySubProof Denote by \( M \) the supremum of \( A \). Assume that it is not a supremum of \( \cl A \), that is, there exists an upper bound \( M' \) of \( \cl A \) such that \( M' < M \). But this is impossible because \( A \subseteq \cl A \).

  Therefore, \( M \) is a supremum of \( \cl A \).

  \NecessitySubProof Denote by \( M \) the supremum of \( \cl A \). Assume that it is not a supremum of \( A \), that is, there exists an upper bound \( M' \) of \( A \) such that \( M' < M \).

  Let \( \{ x_i \}_{i=1}^\infty \subseteq A \) be a sequence that converges to \( M \). Then
  \begin{equation*}
    x_i < M' < M.
  \end{equation*}

  By \fullref{thm:squeeze_lemma/sequences}, we have \( M' = M \), which contradicts our choice of \( M' \). Thus, \( M \) is the supremum of \( A \).
\end{proof}

\begin{proposition}\label{thm:real_bounded_set_has_supremum}
  Every nonempty \hyperref[def:metric_space/bounded_set]{bounded set} in \( \BbbR \) has a supremum and infimum.
\end{proposition}
\begin{proof}
  Let \( A \subseteq \BbbR \) be a nonempty bounded set. By \fullref{thm:heine_borel}, the set \( \cl A \) is compact. By \fullref{thm:weierstrass_extreme_value_theorem}, the identity function \( \id: \BbbR \to \BbbR \) attains its minimum \( m \) and maximum \( M \) on \( \cl A \). Note that both \( m \) and \( M \) do not have to belong to \( A \), but \( m \) is a lower bound and \( M \) is an upper bound of the set \( A \).

  If we take any other upper bound \( M' \) of \( A \), then by \fullref{thm:real_supremum_of_closure},
  \begin{equation*}
    M' \geq \sup A = \sup \cl A = M.
  \end{equation*}

  Hence, \( M \) is the least upper bound of \( A \).

  We can analogously prove that \( m \) is the greatest lower bound of \( A \).
\end{proof}


\section{Universes}

\index{universe|(}
\index{universal family|(}
To complete our specification of dependent type theory, we introduce type theoretic \emph{universes}. Universes are types that consist of types. In other words, a universe is a type $\UU$ that comes equipped with a type family $\Ty$ over $\UU$, and for any $X:\UU$ we think of $X$ as an \emph{encoding}\index{encoding of a type in a universe} of the type $\Ty(X)$. We call this type family the \emph{universal type family}.

There are several reasons to equip type theory with universes. One reason is that it enables us to define new type families over inductive types, using their induction principle. For example, since the universe is itself a type, we can use the induction principle of $\bool$ to obtain a map $P:\bool\to\UU$ from any two terms $X_0,X_1:\UU$. Then we obtain a type family over $\bool$ by substituting $P$ into the universal type family:
\begin{equation*}
  x:\bool\vdash \Ty(P(x))~\type
\end{equation*}
satisfying $\Ty(P(0_\bool))\jdeq \Ty(X_0)$ and $\Ty(P(1_\bool))\jdeq \Ty(X_1)$.

We use this way of defining type families to define many familiar relations over $\N$, such as $\leq$ and $<$. We also introduce a relation called \emph{observational equality} $\EqN$ on $\N$, which we can think of as equality of $\N$. This relation is reflexive, symmetric, and transitive, and moreover it is the least reflexive relation. Furthermore, one of the most important aspects of observational equality $\EqN$ on $\N$ is that $\EqN(m,n)$ is a type for every $m,n:\N$, unlike judgmental equality. Therefore we can use type theory to reason about observational equality on $\N$. Indeed, in the exercises we show that some very elementary mathematics can already be done at this early stage in our development of type theory.

A second reason to introduce universes is that it allows us to define many types of types equipped with structure. One of the most important examples is the type of groups\index{group}, which is the type of types equipped with the group operations satisfying the group laws, and for which the underlying type is a set\index{set}. We won't discuss the condition for a type to be a set until \cref{chap:hierarchy}, so the definition of groups in type theory will be given much later. Therefore we illustrate this use of the universe by giving simpler examples: pointed types, graphs, and reflexive graphs.

One of the aspects that make universes useful is that they are postulated to be closed under all the type constructors. For example, if we are given $X:\UU$ and $P:\Ty(X)\to \UU$, then the universe is equipped with a term
\begin{equation*}
  \check{\Sigma}(X,P):\UU
\end{equation*}
satisfying the judgmental equality $\Ty(\check{\Sigma}(X,P)\jdeq\sm{x:\Ty(X)}\Ty(P(x))$. We will similarly assume that any universe is closed under $\Pi$-types and the other ways of forming types. However, there is an important restriction: it would be inconsistent to assume that the universe is contained in itself. One way of thinking about this is that universes are types of \emph{small} types, and it cannot be the case that the universe is small with respect to itself. We address this problem by assuming that there are many universes: enough universes so that any type family can be obtained by substituting into the universal type family of some universe.

\subsection{Specification of type theoretic universes}

In the following definition we already state that universes are closed under identity types. Identity types will be introduced in \cref{chap:identity}.

\begin{defn}
  A \define{universe} in type theory is a closed type $\UU$\index{U, V, W@{$\UU$, $\mathcal{V}$, $\mathcal{W}$}|see {universe}} equipped with a type family $\Ty$\index{T@{$\Ty$}|see {universal family}} over $\UU$ called the \define{universal family}\index{family!universal family}, equipped with the following structure:
  \begin{enumerate}
  \item $\UU$ is closed under $\Pi$, in the sense that it comes equipped with a function
    \begin{equation*}
      \check{\Pi} :\prd{X:\UU}(\Ty(X)\to\UU)\to\UU
    \end{equation*}
    for which the judgmental equality
    \begin{equation*}
      \Ty\big(\check{\Pi}(X,P)\big)\jdeq \prd{x:\Ty(X)}\Ty(P(x)).
    \end{equation*}
    holds, for every $X:\UU$ and $P:\Ty(X)\to\UU$.
  \item $\UU$ is closed under $\Sigma$ in the sense that it comes equipped with a function
    \begin{equation*}
      \check{\Sigma} :\prd{X:\UU}(\Ty(X)\to\UU)\to\UU
    \end{equation*}
    for which the judgmental equality
    \begin{equation*}
      \Ty\big(\check{\Sigma}(X,P)\big) \jdeq \sm{x:\Ty(X)}\Ty(P(x))
    \end{equation*}
    holds, for every $X:\UU$ and $P:\Ty(X)\to\UU$.
  \item $\UU$ is closed under identity types, in the sense that it comes equipped with a function
    \begin{equation*}
      \check{\mathrm{I}} : \prd{X:\UU}\Ty(X)\to(\Ty(X)\to\UU)
    \end{equation*}
    for which the judgmental equality
    \begin{equation*}
      \Ty\big(\check{\mathrm{I}}(X,x,y)\big)\jdeq (\id{x}{y})
    \end{equation*}
    holds, for every $X:\UU$ and $x,y:\Ty(X)$.
  \item $\UU$ is closed under coproducts, in the sense that it comes equipped with a function
    \begin{equation*}
      \check{+}:\UU\to (\UU\to\UU)
    \end{equation*}
    that satisfies $\Ty\big(X\check{+}Y\big)\jdeq \Ty(X)+\Ty(Y)$.
  \item $\UU$ contains terms $\check{\emptyt},\check{\unit},\check{\N}:\UU$
    that satisfy the judgmental equalities
    \begin{align*}
      \Ty\big(\check{\emptyt}\big) & \jdeq \emptyt \\
      \Ty\big(\check{\unit}\big) & \jdeq \unit \\
      \Ty(\check{\N}) & \jdeq \N.
    \end{align*}
  \end{enumerate}
  Given a universe $\UU$, we say that a type $A$ in context $\Gamma$ is \define{small}\index{small type}\index{universe!small types} with respect to $\UU$ if it occurs in the universe, i.e., if it comes equipped with a term $\check{A}:\UU$ in context $\Gamma$, for which the judgment
  \begin{equation*}
    \Gamma\vdash\Ty\big(\check{A}\big)\jdeq A~\type
  \end{equation*}
  holds. If $A$ is small with respect to $\UU$, we usually write simply $A$ for $\check{A}$ and also $A$ for $\Ty(\check{A})$. In other words, by $A:\UU$ we mean that $A$ is a small type.
\end{defn}

\begin{rmk}
  Since ordinary function types are defined as a special case of dependent function types, we don't have to assume that universes are closed under ordinary function types. Similarly, it follows from the assumption that universes are closed under dependent pair types that universes are closed under cartesian product types.
\end{rmk}

\subsection{Assuming enough universes}

\index{enough universes|(}
\index{universe!enough universes|(}
  Most of the time we will get by with assuming one universe $\UU$, and indeed we recommend on a first reading of this text to simply assume that there is one universe $\UU$. However, sometimes we might need a second universe $\mathcal{V}$ that contains $\UU$ as well as all the types in $\UU$. In such situations we cannot get by with a single universe, because the assumption that $\UU$ is a term of itself would lead to inconsistencies like the Russel's paradox.

  Russel's paradox is the famous argument that there cannot be a set of all sets. If there were such a set $S$, then we could consider Russel's subset
  \begin{equation*}
    R:=\{x\in S\mid x\notin x\}.
  \end{equation*}
  Russell then observed that $R\in R$ if and only if $R\notin R$, so we reach a contradiction. A variant of this argument reaches a similar contradiction when we assume that $\UU$ is a universe that contains a term $\check{\UU}:\UU$ such that $\mathcal{T}\big(\check{\UU}\big)\jdeq \UU$. In order to avoid such paradoxes, Russell and Whitehead formulated the \emph{ramified theory of types} in their book \emph{Principia Mathematica}. The ramified theory of types is a precursor of Martin L\"of's type theory that we are studying in this course.

  Even though the universe is not a term of itself, it is still convenient if every type, including any universe, is small with respect to \emph{some} universe. Therefore we will assume that there are sufficiently many universes: we will assume that for every finite list of types
\begin{align*}
  \Gamma_1 & \vdash A_1~\type \\
  & ~\vdots \\
  \Gamma_n & \vdash A_n~\type,
\end{align*}
there is a universe $\UU$ that contains each $A_i$ in the sense that $\UU$ comes equipped with a term
\begin{align*}
  \Gamma_i\vdash \check{A}_i:\UU
\end{align*}
for which the judgment
\begin{equation*}
  \Gamma_i\vdash \Ty\big(\check{A}_i\big)\jdeq A_i~\type
\end{equation*}
holds. With this assumption it will rarely be necessary to work with more than one universe at the same time.

\begin{rmk}\label{rmk:universe-constructions}
  Using the assumption that for any finite list of types in context there is a universe that contains those types, we obtain many specific universes:
  \begin{enumerate}
  \item There is a \emph{base universe} $\UU_0$ that we obtain using the empty list of types in context. This is a universe, but it isn't specified to contain any further types.
  \item Given a finite list
    \begin{align*}
      \Gamma_1 & \vdash A_1~\type \\
      & ~\vdots \\
      \Gamma_n & \vdash A_n~\type,
    \end{align*}
    of types in context, and a universe $\UU$ that contains them, there is a universe $\UU^+$ that contains all the types in $\UU$ as well as $\UU$. More precisely, it is specified by the finite list
    \begin{align*}
      & \vdash \UU~\type \\
      X:\UU & \vdash \mathcal{T}(X)~\type.
    \end{align*}
    The universe $\UU^+$ therefore contains the type $\UU$ as well as every type in $\UU$, in the following sense
    \begin{align*}
      & \vdash \check{\UU}:\UU^+ & & \vdash \mathcal{T}^+(\check{\UU})\jdeq\UU~\type \\
      X:\UU & \vdash \check{\mathcal{T}}(X) :\UU^+ & X:\UU & \vdash \mathcal{T}^+(\check{\mathcal{T}}(X))\jdeq \mathcal{T}(X)~\type.
    \end{align*}
    In particular, we obtain a function $i:\UU\to\UU^+$ that includes the $\UU$-small types into $\UU^+$.

    Note that since the universe $\UU^+$ contains all the types in $\UU$, it also contains the types $A_1,\ldots,A_n$. To see this, we derive that there is a code for $A_i$ in $\UU^+$.
    \begin{prooftree}
      \AxiomC{$\Gamma_i\vdash\check{A}_i:\UU$}
      \AxiomC{$X:\UU\vdash\check{\mathcal{T}}(X):\UU^+$}
      \UnaryInfC{$\Gamma_i,X:\UU\vdash\check{\mathcal{T}}(X):\UU^+$}
      \BinaryInfC{$\Gamma_i\vdash\check{\mathcal{T}}(\check{A}_i):\UU^+$}
    \end{prooftree}
    We leave it as an exercise to derive the judgmental equality
    \begin{equation*}
      \mathcal{T}^+(\check{\mathcal{T}}(\check{A}_i))\jdeq A_i.
    \end{equation*}
  \item Given two finite lists
    \begin{align*}
      \Gamma_1 & \vdash A_1~\type & \Delta_1 & \vdash B_1~\type \\
      & ~\vdots & & ~\vdots \\
      \Gamma_n & \vdash A_n~\type & \Delta_m & \vdash B_m~\type
    \end{align*}
    of types in context, and two universes $\mathcal{U}$ and $\mathcal{V}$ that contain $A_1,\ldots,A_n$ and $B_1,\ldots,B_m$ respectively, there is a universe $\UU\sqcup\mathcal{V}$ that contains the types of both $\UU$ and $\mathcal{V}$. The universe $\UU\sqcup\mathcal{V}$ is specified by the finite list
    \begin{align*}
      X:\UU & \vdash \mathcal{T}_{\mathcal{U}}(X)~\type \\
      Y:\mathcal{V} & \vdash \mathcal{T}_{\mathcal{V}}(Y) ~\type.
    \end{align*}
    With an argument similar to the previous construction of a universe, we see that the universe $\UU\sqcup\mathcal{V}$ contains the types $A_1,\ldots,A_n$ as well as the types $B_1,\ldots,B_m$.

    Note that we could also directly obtain a universe $\mathcal{W}$ that contains the types $A_1,\ldots,A_n$ and $B_1,\ldots,B_m$. However, this universe might not contain all the types in $\UU$ or all the types in $\mathcal{V}$.
  \end{enumerate}
  Since we don't postulate any relations between the universes, there are indeed very few of them. For example, the base universe $\UU_0$ might contain many more types than it is postulated to contain. Nevertheless, there are some relations between the universes. For instance, there is a function $\UU\to\UU^+$, since we can simply derive
  \begin{prooftree}
    \AxiomC{$X:\UU\vdash \check{\mathcal{T}}(X):\UU^+$}
    \UnaryInfC{$\vdash \lam{X}\check{\mathcal{T}}(X) : \UU\to\UU^+$}
  \end{prooftree}
  Similarly, there are functions $\UU\to \UU\sqcup\mathcal{V}$ and $\mathcal{V}\to \UU\sqcup\mathcal{V}$ for any two universes $\UU$ and $\mathcal{V}$.
\end{rmk}
\index{enough universes|)}
\index{universe!enough universes|)}

\subsection{Pointed types}

\index{pointed type|(}
\begin{defn}
  A \define{pointed type} is a pair $(A,a)$ consisting of a type $A$ and a term $a:A$. The type of all pointed types in a universe $\UU$ is defined to be\index{UU*@{$\UU_\ast$}}
  \begin{equation*}
    \UU_\ast \defeq \sm{X:\UU}X.
  \end{equation*}
\end{defn}

\begin{defn}
  Consider two pointed types $(A,a)$ and $(B,b)$. A \define{pointed map}\index{pointed map} from $(A,a)$ to $(B,b)$ is a pair $(f,p)$ consisting of a function $f:A\to B$ and an identification $p:f(a)=b$. We write
  \begin{equation*}
    A\to_\ast B \defeq \sm{f:A\to B}f(a)=b
  \end{equation*}
  for the type of all pointed maps from $(A,a)$ to $(B,b)$, leaving the base point implicit.
\end{defn}

Since we have a type $\UU_\ast$ of \emph{all} pointed types in a universe $\UU$, we can start defining operations on $\UU_\ast$. An important example of such an operation is to take the loop space of a pointed type.

\begin{defn}
  We define the \define{loop space}\index{loop space}\index{Omega@{$\Omega$}|see {loop space}} operation $\Omega : \UU_\ast \to \UU_\ast$
  \begin{equation*}
    \Omega(A,a)\defeq \big((a=a),\refl{a}\big).
  \end{equation*}
\end{defn}

We can even go further and define the \emph{iterated loop space} of a pointed type. Note that this definition could not be given in type theory if we didn't have universes.

\begin{defn}
  Given a pointed type $(A,a)$ and a natural number $n$, we define the $n$-th loop space $\Omega^n(A,a)$ by induction on $n:\N$, taking\index{iterated loop space}\index{Omega^n@{$\Omega^n$}|see {iterated loop space}}
  \begin{align*}
    \Omega^0(A,a) & \defeq (A,a) \\
    \Omega^{n+1}(A,a) & \defeq \Omega(\Omega^n(A,a)).
  \end{align*}
\end{defn}
\index{pointed type|)}

\subsection{Families and relations on the natural numbers}

As we have already seen in the case of the iterated loop space, we can use the universe to define a type family over $\N$ by induction on $\N$. For example, we can define the finite types in this way.

\begin{defn}\label{defn:fin}
We define the type family $\Fin:\N\to\UU$ of finite types\index{Fin@{$\Fin$}}\index{finite types} by induction on $\N$\index{family!of finite types}, taking
\begin{align*}
\Fin(\zeroN) & \defeq \emptyt \\
\Fin(\succN(n)) & \defeq \Fin(n)+\unit
\end{align*}
\end{defn}

Similarly, we can define many relations on the natural numbers using a universe. We give here the example of \emph{observational equality} on $\N$. This inductively defined equivalence relation is very important, as it can be used to show that equality on the natural numbers is \emph{decidable}, i.e., there is a program that decides for any two natural numbers $m$ and $n$ whether they are equal or not.

\begin{defn}\label{defn:obs_nat}
We define the \define{observational equality}\index{observational equality!on N@{on $\N$}}\index{natural numbers!observational equality} on $\N$ as binary relation $\EqN:\N\to(\N\to\UU)$\index{Eq N@{$\EqN$}} satisfying
\begin{align*}
\EqN(\zeroN,\zeroN) & \jdeq \unit & \EqN(\succN(n),\zeroN) & \jdeq \emptyt \\
\EqN(\zeroN,\succN(n)) & \jdeq \emptyt & \EqN(\succN(n),\succN(m)) & \jdeq \EqN(n,m).
\end{align*}
\end{defn}

\begin{constr}
We define $\EqN$ by double induction on $\N$. By the first application of induction it suffices to provide
\begin{align*}
E_0 & : \N\to\UU \\
E_S & : \N\to (\N\to\UU)\to(\N\to\UU)
\end{align*}
We define $E_0$ by induction, taking $E_{00}\defeq \unit$ and $E_{0S}(n,X,m)\defeq \emptyt$. The resulting family $E_0$ satisfies
\begin{align*}
E_0(\zeroN) & \jdeq \unit \\
E_0(\succN(n)) & \jdeq \emptyt.
\end{align*}
We define $E_S$ by induction, taking $E_{S0}\defeq \emptyt$ and $E_{S0}(n,X,m)\defeq X(m)$. The resulting family $E_S$ satisfies
\begin{align*}
E_S(n,X,\zeroN) & \jdeq \emptyt \\
E_S(n,X,\succN(m)) & \jdeq X(m)
\end{align*}
Therefore we have by the computation rule for the first induction that the judgmental equality
\begin{align*}
\EqN(\zeroN,m) & \jdeq E_0(m) \\
\EqN(\succN(n),m) & \jdeq E_S(n,\EqN(n),m)
\end{align*}
holds, from which the judgmental equalities in the statement of the definition follow.
\end{constr}

\begin{lem}\label{ex:obs_nat_least}\index{observational equality!on N@{on $\N$}!is least reflexive relation}
  Suppose $R:\N\to(\N\to\UU)$ is a reflexive relation on $\N$\index{reflexive relation}\index{relation!reflexive}, i.e., $R$ comes equipped with
  \begin{equation*}
    \rho : \prd{n:\N}R(n,n).
  \end{equation*}
  Then there is a family of maps
  \begin{equation*}
    \prd{m,n:\N}\EqN(m,n)\to R(m,n).
  \end{equation*}
\end{lem}

\begin{proof}
  We will prove by induction on $m,n:\N$ that there is a term of type
  \begin{equation*}
    f_{m,n}:\prd{e:\EqN(m,n)}\prd{R:\N\to(\N\to\UU)}\Big(\prd{x:\N}R(x,x)\Big)\to R(m,n)
  \end{equation*}
  The dependent function $f_{m,n}$ is defined by
  \begin{align*}
    f_{\zeroN,\zeroN} & \defeq \lam{\ttt}\lam{r}\lam{\rho}\rho(\zeroN) \\
    f_{\zeroN,\succN(n)} & \defeq \indempty \\
    f_{\succN(m),\zeroN} & \defeq \indempty \\
    f_{\succN(m),\succN(n)} & \defeq \lam{e}\lam{R}\lam{\rho}f_{m,n}(e,R',\rho'),
  \end{align*}
  where $R'$ and $\rho'$ are given by
  \begin{align*}
    R'(m,n) & \defeq R(\succN(m),\succN(n)) \\
    \rho'(n) & \defeq \rho(\succN(n)).\qedhere
  \end{align*}
\end{proof}

We can also define observational equality for many other kinds of types, such as $\bool$ or $\Z$. In each of these cases, what sets the observational equality apart from other relations is that it is the \emph{least} reflexive relation.

\begin{exercises}
\exercise \label{ex:obs_nat_eqrel}Show that observational equality on $\N$\index{observational equality!on N@{on $\N$}!is an equivalence relation} is an equivalence relation\index{equivalence relation!observational equality on N@{observational equality on $\N$}}, i.e., construct terms of the following types:
  \begin{align*}
    & \prd{n:\N} \EqN(n,n) \\
    & \prd{n,m:\N} \EqN(n,m)\to \EqN(m,n) \\
    & \prd{n,m,l:\N} \EqN(n,m)\to (\EqN(m,l)\to \EqN(n,l)).
  \end{align*}
\exercise \index{observational equality!on N@{on $\N$}!is preserved by functions}Show that every function $f:\N\to \N$ preserves observational equality in the sense that
  \begin{equation*}
    \prd{n,m:\N} \EqN(n,m)\to \EqN(f(n),f(m)).
  \end{equation*}
\exercise \label{ex:order_N}
  \begin{subexenum}
  \item Define the \define{order relations}\index{relation!order}\index{order relation!leq on N@{$\leq$ on $\N$}}\index{order relation!le on N@{$<$ on $\N$}}\index{leq@{$\leq$}!on N@{on $\N$}}\index{le@{$<$}!on N@{on $\N$}} $\leq$ and $<$ on $\N$.
  \item Show that $\leq$ is reflexive\index{reflexive!leq on N@{$\leq$ on $\N$}} and that $<$ is \define{irreflexive}\index{irreflexive}\index{relation!irreflexive}, i.e., show that
    \begin{equation*}
      (n\leq n)\qquad\text{and}\qquad (n\nless n).
    \end{equation*}
  \item Show that $n\leq\succN(n)$ and $n<\succN(n)$.
  \item Show that both $\leq$ and $<$ are transitive.
  \item Show that $\leq$ is \define{antisymmetric}, i.e., show that $m=n$ whenever both $m\leq n$ and $n\leq m$ hold. Also show that if $m<n$, then $n\nless m$.
  \item Show that
    \begin{equation*}
      (m\leq n)\leftrightarrow (m=n)+(m<n)
    \end{equation*}
    holds for all $m,n$.
  \item Show that
    \begin{equation*}
      (m\leq n)+(n<m)
    \end{equation*}
    holds for all $m,n$.
  \item Show that $k\leq \min(m,n)$ holds if and only if both $k\leq m$ and $k\leq n$ hold, and show that $\max(m,n)\leq k$ holds if and only if both $m\leq k$ and $n\leq k$ hold.
  \end{subexenum}
\exercise \label{ex:obs_bool}\index{observational equality!on 2@{on $\bool$}}
  \begin{subexenum}
  \item Define observational equality $\EqBool$\index{Eq 2@{$\EqBool$}} on the booleans.
  \item Show that $\EqBool$ is reflexive.\index{observational equality!on 2@{on $\bool$}!is reflexive}
  \item Show that for any reflexive relation $R:\bool\to(\bool\to \UU)$ one has\index{observational equality!on 2@{on $\bool$}!is least reflexive relation}
    \begin{equation*}
      \prd{x,y:\bool} \EqBool(x,y)\to R(x,y).
    \end{equation*}
  \end{subexenum}
\exercise \label{ex:int_order}
  \begin{subexenum}
  \item Define the order relations\index{relation!order}\index{order relation} $\leq$ and $<$ on and $\Z$.
  \item Show that $\leq$ is reflexive, transitive, and antisymmetric.
  \item Show that $<$ is irreflexive and transitive.
  \end{subexenum}
\begin{comment}
\item
  \begin{subexenum}
  \item For each $i:\Fin(\succN(n))$, define a function
    \begin{equation*}
      \skipFin_i : \Fin(n)\to\Fin(\succN(n))
    \end{equation*}
    that includes $\Fin(n)$ in $\Fin(\succN(n))$ by skipping $i$.
  \item For each $i:\Fin(n)$, define a function
    \begin{equation*}
      \doubleFin_i : \Fin(\succN(n))\to\Fin(n)
    \end{equation*}
    that projects $\Fin(\succN(n))$ onto $\Fin(n)$ by doubling at $i$.
  \end{subexenum}
\end{comment}
\end{exercises}
\index{universe|)}
\index{universal family|)}


%\chapter{Higher homotopy pushouts}

\section{$2$-spans and $2$-pushouts}

\section{Applications}

\begin{thm}
The join is an associative operation.
\end{thm}

\begin{thm}
the 3-by-3-lemma
\end{thm}

\begin{exercises}
\item In this exercise we study the \define{reflexive coequalizer}. Let $R:A\to A\to\UU$ be a relation, and $\rho:\prd{x:A}R(x,x)$ be a witness of reflexivity.
\begin{subexenum}
\item Formulate the induction principle and computation rules for the higher inductive type $\mathsf{rcoeq}(A,R,\rho)$ with constructors
\begin{align*}
\pts{\eta} &: A \to \mathsf{rcoeq}(A,R,\rho) \\
\edg{\eta} &: \prd*{x,y:A} R(x,y)\to \id{\pts{\eta}(x)}{\pts{\eta}(y)} \\
\rfx{\eta} &: \prd{x:A} \edg{\eta}(\rho(x))=\refl{\pts{\eta}(x)}.
\end{align*}
\item Show that
\begin{equation*}
\begin{tikzcd}
\sm{x,y:A} R(x,y) \arrow[r,"\pi_2"] \arrow[d,swap,"\pi_1"] & A \arrow[d,"\pts{\eta}"] \\
A \arrow[r,swap,"\pts{\eta}"] & \mathsf{rcoeq}(A,R,\rho)
\end{tikzcd}
\end{equation*}
commutes, and is a pushout square.
\item Compute $\mathsf{rcoeq}(A,\idtypevar{A},\refl{})$ and $\mathsf{rcoeq}(A,(\lam{x}{y}\unit),(\lam{x}\ttt))$. Furthermore, consider a pointed type $\pairr{X,x_0}$ as a reflexive relation on the unit type, and compute $\mathsf{rcoeq}(\unit,X,x_0)$.
\end{subexenum}
\end{exercises}


\chapter{Intuition for the Damping Terms in the Stiff String PDE}\label{app:intuitionSigma1}
This appendix will delve deeper into the damping terms in the PDE of the stiff string presented in Chapter \ref{ch:stiffString}, especially the frequency-dependent damping term $2\sigma_1\pt\pxx u$. Recalling the compact version of the PDE of the stiff string in Eq. \eqref{eq:stiffStringPDECompact}:
\begin{equation}
    \ptt u = c^2 \pxx u - \kappa^2 \pxxxx u - 2 \sz \pt u + 2 \so \pt \pxx u
\end{equation}
Consider first the frequency-independent damping term $-2\sz\pt u$. The more positive the velocity $\partial_tu$ is, i.e., the string is moving upwards the more this term applies a negative, or downwards force (/effect) on the string. Vice versa, a more negative velocity will make this term apply a more positive force on the string.

As for the frequency-dependent damping term, apart from the obvious $\sigma_1$, the effect of the term increases with an increase of $\partial_t\partial_x^2u$ which literally describes the `rate of change of the curvature' of the string.

Let's first talk about positive and negative curvature, i.e., when $\partial_x^2u > 0$ or $\partial_x^2u < 0$. Counterintuitively, in the positive case, the curve points downwards. Think about the function $f(x) = x^2$. It has a positive curvature (at any point), but has a minimum. This can be proven by taking $x=0$ and setting grid spacing $h=1$.
\begin{equation}
  \begin{aligned}
  \delta_{xx}f(x) &= \frac{1}{h^2} \left(f(-1)-2f(0)+f(1)\right), \\
  &= \frac{1}{1^2} \left((-1)^2-2\cdot0^2+1^2\right),\\
  &= \left(1-0+1\right) = 2.
  \end{aligned}
\end{equation}
In other words, the second derivative of the function $f(x)=x^2$ around $x=0$ is positive.

As the term does not only include a second-order spatial derivative but also a first-order time derivative, this is referred to as a a positive or negative \textit{rate of change} of the curvature, i.e., when $\partial_t\partial_x^2u>0$ or $\partial_t\partial_x^2u<0$. A positive rate of change of curvature means that the string either has a positive curvature and is getting more positive, i.e., the string gets more curved over time, or that the string has a negative curvature and is getting less negative, i.e., the string gets less curved or 'loosens up' over time.  In the same way, a negative rate of change of curvature means that the string either has a negative curvature and is getting more negative, or that the string has a positive curvature and is getting less positive.

Let's see some examples. Take the same function described before, but now $f$ changes over time, fx. $f(x, t)=tx^2$. When $t$ increases over time, the curvature gets bigger. Repeating the above with $x=0$ and grid spacing $h = 1$, but now with $t=2$ and step size $k=1$, but now with a backwards time derivative yields:
  \begin{alignat*}{3}
    \delta_{t-}\delta_{xx}f(x,t) &= \frac{1}{kh^2}\bigg(&&f(-1, 2) - 2f(0, 2) + f(1, 2) \\
    & &&- \Big(f(-1, 1) - 2f(0, 1) + f(1, 1)\Big)\bigg),\\
    & = \frac{1}{1\cdot 1^2}\bigg(&&2\cdot(-1)^2-2\cdot2\cdot(0)^2+2\cdot1^2\\
    & &&-\Big(1\cdot(-1)^2-2\cdot1\cdot(0)+1\cdot(1^2)\Big)\Bigg),\\
    &=2+2-&&(1+1)=2.
  \end{alignat*}
So the rate of change of the curvature is positive, i.e., the already positively curved function $x^2$ gets more curved over time.

If the curvature around a point along a string gets more positive (or less negative) over time, the force applied to that point will be positive, effectively `trying' to reduce the curvature. Vice versa, if the curvature around a point along a string gets more negative (or less positive) over time, the force applied will be negative, again `trying' to reduce the curvature.

From an auditory point of view, higher curvature generally means higher frequency. As the frequency-dependent damping term reduces curvature along the string it effectively damps higher frequencies.

% The fact the frequency dependent term to be added rather than subtracted, is caused by the fact that a location along the string with a positive curvature implies that its neighbouring locations have relatively more positive displacement than itself. This translated to the force/effect this term has on the scheme means...


\chapter{The Language}

\par The MalbolgeLISP implements a Lisp dialect which is described in this chapter. The main differences from Lisp is emphasis on functional programming, immutability and minimisation of the state (MalbolgeLISP doesn't implement \verb|setcar!| or \verb|setcdr!|). MalbolgeLISP borrows it's core features from Scheme, Haskell and APL, while using Lisp as a foundation or framework for constructing a language.

\section{Arithmetic}

\par MalbolgeLISP supports the following arithmetic operations:

\begin{itemize}
    \item \verb|+|, \verb|+'| - unsigned bounded addition\footnote{obeying to the laws of modular arithmetic; the result is truncated to 26 bits} and unbounded addition\footnote{on standards-compliant interpreters}
    \item \verb|-|. \verb|-'| - unsigned bounded subtraction and unbounded subtraction.
    \item \verb|*|, \verb|/|, \verb|%| - unsigned bounded multiplication, division and modulus.
    \item \verb|~| - boolean negation - $f(x) = 0$ for $x > 0$ and $f(x) = 1$ otherwise.
    \item \verb|!| - factorial function - $f(x) = x!$ computed using a lookup table for $x \leq 11$ and $f(x) = 0$ otherwise, since $!12$ exceeds the word size.
    \item \verb|^| - $n$-th power function assuming $0^0=1$.
    \item \verb|&| - logical AND function - $f(x, y) = 1$ for $x > 0 \wedge y > 0$, $f(x, y) = 0$ otherwise.
    \item \verb=|= - logical AND function - $f(x, y) = 0$ for $x = 0 \wedge y = 0$, $f(x, y) = 1$ otherwise.
    \item \verb|=|, \verb|/=| - equality and inequality.
    \item \verb|<|, \verb|>|, \verb|<=|, \verb|>=| - correspondingly: lesser than, greater than, lesser or equal to, greater or equal to.
    \item \verb|gcd|, \verb|lcm| - greatest common divisor and least common multiple.
    \item \verb|min|, \verb|max| - pick the smaller or larger value (\textit{minimum} or \textit{maximum}).
\end{itemize}

\par Demonstration of the arithmetic functions follows:

\begin{verbatim}
% (+ 2 1)
..............|.....
3
% (* 3 3)
..............|.....
9
% (% 5 2)
..............|.....
1
% (> 5 6)
..............|.....
0
% (< 5 6)
..............|.....
1
% (= 6 6)
..............|......
1
% (= (~ (= 6 7)) (/= 6 7))
......................|.........
1
% (/ 25 5)
..............|.....
5
% (- 4 3)
..............|.....
1
% (& (= 2 2) (= 3 3))
....................................|...............
1
% (| (= 2 3) (= 3 2))
................|..........
0
% (lcm 14 29)
......|....
406
% (max 5 (max 8 10))
...........|.......
10
% (^ 2 8)
......|....
256
% (! 6)
.....|...
720
\end{verbatim}

\section{Conditional execution}

\par In MalbolgeLISP, conditional evaluation is accomplished using \verb|if| and \verb|cond|\footnote{also \verb|iterate|, \verb|iterateN|, etc..., but these functions weren't designed for conditional evaluation and it only happened so that they can be used for this purpose, so they won't be covered in this section.}. \verb|&| and \verb=|= don't short-circuit, meaning that their operands are always evaluated. \verb|if| is very similar in it's structure to \verb|if..else| statements\footnote{or ternary operator expressions} known from languages like C++ or Java - \verb|(if condition valueIfTrue valueIfFalse)|. The short-circuiting aspect of \verb|if| can be demonstrated - If both cases of the \verb|if| condition were evaluated, the interpreter would print \verb|yes| and \verb|no| before yielding the correct result:

\begin{verbatim}
% (def noisy_eq (dyad (if (= x y) (print yes) (print no))))
.............................|..
(lambda (x y) (if (= x y) (print yes) (print no)))
% (noisy_eq 5 6)
......|............
no

no
% (noisy_eq 6 6)
......|............
yes

yes
\end{verbatim}

\par The \verb|cond| function is essentially a chain of \verb|if| expressions. To demonstrate the convenience of using \verb|cond| over multiple \verb|if| expressions, two implementations of three-way comparison are given:

\begin{verbatim}
; An implementation that uses if.
% (def <=> (dyad (
    if (= x y) eq (
        if (> x y) gt (
            if (< x y) lt unreachable)))))
...........................................|..
(lambda (x y) (if (= x y) eq (if (> x y) gt (if (< x y) lt unreachable))))
% (<=> 5 5)
......|..........
eq
% (<=> 6 7)
......|....................
lt
% (<=> 7 6)
......|...............
gt

; An implementation that uses cond.
% (def <=> (dyad (cond
    ((= x y) eq)
    ((> x y) gt)
    ((< x y) lt)
    (unreachable))))
...............................................|..
(lambda (x y) (cond ((= x y) eq) ((> x y) gt) ((< x y) lt) (unreachable)))
% (<=> 6 6)
......|...........
eq
% (<=> 6 7)
......|...................
lt
% (<=> 7 6)
......|...............
\end{verbatim}

\par It's worth mentioning that an \verb|if| expression must include a truthy and falsy clause (since the expression must evaluate to something and it's impossible to guarantee that the condition is always true). This isn't the case with the \verb|cond| expression. If one is certain that a \verb|cond| expression is already exhaustive, it doesn't have to provide a default case. The interpreter will error if the \verb|cond| expression isn't exhaustive and no default case was provided, though.

\section{Let bindings and the scope}

\par In MalbolgeLISP, there is a single way to bind variables without scope (via macros), and two ways of binding lexically scoped variables. Variables are usually bound using \verb|lambda| and all the covers over it (\verb|dyad|, \verb|monad|, \verb|defun|), but there is a way to bind variables without explicitly using a lambda expression - \verb|let| bindings. \verb|let| accepts a list of atoms to be bound interleaved with their values and the code to execute in the freshly created scope. The main difference from \verb|let| present in other lisps is that the binding list is flattened:

\begin{verbatim}
% (let (x 5 y 6) (print [x + y]))
.....................|............
11

11
\end{verbatim}

\par When evaluated, \verb|let| is desugared to a lambda expression with reversed parameters. The conversion schema follows:

\begin{verbatim}
(let (x x0 y y0 z z0 ...) (...)) => ((lambda (z y x ...) (...)) z0 y0 x0 ...)
\end{verbatim}

\par The real parameter order is reversed, since the interpreter (for the sake of simplicity) iterates over the input list and uses \verb|cons| to create the resulting desugared expression. Single-variable \verb|let| bindings could illustratively also be implemented as a macro:

\begin{verbatim}
% (defmacro let' (x y c) ((lambda (x) (eval c)) y)))
............................|.
macro/syn
% (let' x 5 (print x))
...........|...........
5

5
\end{verbatim}

\par The current scope is bound to the lambda expression at the time of it's creation. Since lazily evaluated values utilise closures which are lexically scoped, this property applies to them as well. Consequently, everything inside the lazily evaluated expression has access to the lexical ancestor's bound variables. Generally speaking, the global scope can be modified using \verb|def|, \verb|defun| and \verb|defmacro|, regardless of where these functions appear. Modifying the local scope is possible using only lambda expressions and the \verb|let| function. It's discouraged to use \verb|def|, \verb|defun| and \verb|defmacro| outside of the top-level code scope (i.e. inside lambda expressions, iterated callables, etc...), since it introduces global and mutable state.

\section{Lisp-style list processing}

\par MalbolgeLISP supports Lisp-style list processing (usually distinguished by extensive use of primitives mentioned below and recursion), even though the more sophisticated and concise Haskell and APL-like functionality is also supported.

\par \verb|car| is used to query a list's head (\verb|1∘↑|). It doesn't perform any copies and returns the atom the list's head points to. \verb|car| of an empty list is \textit{NULL}.

\begin{figure}[H]
\centering
\includegraphics[width=0.3\textwidth]{figures/x1.png}
\caption{The result of car invocation}
\end{figure}

\par \verb|cdr| is used to query a list's tail (\verb|1∘↓|), i.e. everything besides it's head, which is why it's sometimes called \textit{beheading}. \verb|cdr| of a single-element or empty list is \textit{NULL}.

\begin{figure}[H]
\centering
\includegraphics[width=0.3\textwidth]{figures/x2.png}
\caption{The result of car invocation}
\end{figure}

\par \verb|cons| is used to prepend a node (head and a tail) to a list. The tail of the newly created node with given head is set to the provided list. For instance, the list demonstrated in previous examples could be made using \verb|(cons A (cons B null))|. A more concise version of this expression is \verb|(tie A B)|. \verb|tie| behaves like a chain of \verb|cons| invocation, where the last invocation prepends to a \textit{NULL} list. A parallel could be drawn between \verb|cons & tie| and \verb|if & cond|.

\section{Functional list processing}

\par In MalbolgeLISP, the preferred way to process data involves functional devices introduced with MalbolgeLISP v1.2. Most of them were borrowed from Haskell\footnote{\verb|intersperse|, \verb|filter|, \verb|zipwidth|, etc..} and APL\footnote{to name a few: \verb|where|, \verb|take|, \verb|drop|, \verb|map|, \verb|replicate|, \verb|rev|}.

\subsection{iota}

\par The \verb|iota| function exhibits the same behavior as it's C++ counterpart\footnote{\url{https://en.cppreference.com/w/cpp/algorithm/iota}}, although the concept of an index generator has been introduced by APL. \verb|iota| assumes the index origin of 0, so it generates indices in range $[0, n)$. Unlike in APL, \verb|iota| isn't ambivalent nor doesn't support taking a list as it's only argument:

\begin{verbatim}
; MalbolgeLISP
% (iota 5)
.....|...
(0 1 2 3 4)
% (iota '(3 3))
..........|...E030
% (iota '(1 2 3) '(4 5 6))
..................|..E029

      ⍝ APL
      ⎕io←0
      ⍳ 3 3
┌───┬───┬───┐
│0 0│0 1│0 2│
├───┼───┼───┤
│1 0│1 1│1 2│
├───┼───┼───┤
│2 0│2 1│2 2│
└───┴───┴───┘
      ⍳ 5
0 1 2 3 4
      'ABCDEF'⍳'ACF'
0 2 5
\end{verbatim}

\subsection{size}

\par The \verb|size| functions yields the length of a list (in $O(n)$ time complexity). It traverses the list shallowly and doesn't account for it's depth. For instance, a very inefficient identity function on numbers could be implemented to demonstrate it's behavior:

\begin{verbatim}
; MalbolgeLISP
% (def f (atop size iota))
...........|......
bind/syn
% (f 6)
.....|.........
6
% (f 3)
.....|.........
3

      ⍝ APL
      f←≢⍳
      f 6
6
      f 3
3
\end{verbatim}

\subsection{n-th}

\par Picking arbitrary elements from a list is done using \verb|nth|, or a constant $n$-th (the \verb|#| prefix). For example, to pick $n$-th element from the end of a list, the following function might be used:

\begin{verbatim}
; MalbolgeLISP
% (def nl (dyad (nth [(size y) - [x + 1]] y)))
.............................|..
(lambda (x y) (nth (- (size y) (+ x 1)) y))
% (nl '(1 2 3 4 5) 2)
..............|................
3

      ⍝ APL
      nl←⊃⌽
      2 nl 1 2 3 4 5
3
\end{verbatim}

\par \textit{Constant n-th} is an alternative way to query the $n$-th (where $n$ is constant) element of a list. \verb|#N| is equivalent to \verb|(bind nth N)|. \textit{constant n-th} is in many ways similar to a tack - it could even be implemented using a tack and \verb|lift|. For instance:

\begin{verbatim}
; Demonstration of constant n-th
% (print ((atop #1 map) (bind lazy ~) '(0 1 1 1 1 0)))
.............................|................................................
0

0

; Correspondence between constant n-th and tack/lift:
#N <=> (bind lift $N)
\end{verbatim}

\subsection{map}

\par \verb|map| is an ubiquitous higher-order function originating from functional languages present in C++\footnote{\url{https://en.cppreference.com/w/cpp/algorithm/transform}}, APL\footnote{\url{https://help.dyalog.com/latest/index.htm#Language/Primitive\%20Operators/Each\%20with\%20Monadic\%20Operand.htm}} or Java\footnote{\url{https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html#map-java.util.function.Function-}}. \verb|map| is reponsible for transforming (\textit{mapping}) one list into another of equal length, by calling a functor on every element of the original list. It takes any callable first argument, and a list second argument. For example:

\begin{verbatim}
% (map (bind selfie *) (iota 6))
...............|..........................................................
(0 1 4 9 16 25)
% (map ~ '(1 0))
...........|..........
(0 1)
% (map ~ null)
......|....
null
\end{verbatim}

\subsection{filter}

\par \verb|filter| is a higher order function that conditionally removes elements of a list. First, the list is mapped, and then the elements that correspond to falsy values in the list obtained by mapping are removed, and truthy values are kept. \par \verb|filter| behaves in the same way as \verb|map| when the second argument is \textit{NULL}. To demonstrate:

\begin{verbatim}
; MalbolgeLISP
% (filter (bind' % 2) (iota 10))
...............|......................................................
(1 3 5 7 9)
% (filter (bind' % 2) null)
...........|..
null

      ⍝ APL
      filter←{⍵/⍨⍺⍺¨⍵}
      2∘| filter ⍳10
┌→────────┐
│1 3 5 7 9│
└~────────┘
      2∘| filter ⍬
┌⊖┐
│0│
└~┘
\end{verbatim}

\subsection{rev}

\par \verb|rev| is an ambivalent built-in function inspired by APL's \textit{reverse/rotate}\footnote{\url{https://aplwiki.com/wiki/Reverse} and \url{https://aplwiki.com/wiki/Rotate}}. The monadic case simply reverses a list, while the dyadic case rotates it. Since MalbolgeLISP doesn't support rotation with a negative argument, but the dyadic rotation with negative parameter can be defined in an alternative way.

\begin{verbatim}
; MalbolgeLISP
% (rev (iota 5))
.........|.....
(4 3 2 1 0)
% (rev 3 (iota 6))
..........|......
(3 4 5 0 1 2)
% (def rev' (dyad (rev [(size y) - x] y)))
........................|..
(lambda (x y) (rev (- (size y) x) y))
% (rev' 2 (iota 6))
..........|...............
(4 5 0 1 2 3)

      ⍝ APL
      ⌽⍳5
┌→────────┐
│4 3 2 1 0│
└~────────┘
      3⌽⍳6
┌→──────────┐
│3 4 5 0 1 2│
└~──────────┘
      ¯2⌽⍳6
┌→──────────┐
│4 5 0 1 2 3│
└~──────────┘
\end{verbatim}

\subsection{any, every}

\par \verb|any| and \verb|every| are closely tied to each other. \verb|any| returns \verb|1| if its functor returned a truthy value for \textbf{any} element (and \verb|0| otherwise), and \verb|every| returns \verb|1| if its functor returned a truthy value for \textbf{every} element (and \verb|0| otherwise). \verb|any| and \verb|every| short-circuit (otherwise, they would be easy to implement using \verb|filter|). To demonstrate:

\begin{verbatim}
; MalbolgeLISP
% (every (bind = 2) '(2 2 2 2))
..................|................................
1
% (any (bind = 2) '(2 2 2 2))
..................|................................
1
% (every (bind = 3) '(3 3 2))
.................|..............
0
% (any (bind = 3) '(0 1 2))
.................|..............
0

      ⍝ APL
      ⍝ Note: This implementation doesn't short-circuit
      any←{∨/⍺⍺¨⍵}
      every←{∧/⍺⍺¨⍵}
      2∘= every 2 2 2
1
      2∘= any 1 2 3
1
      3∘= every 3 3 2
0
      3∘= any 1 2 2
0
\end{verbatim}

\subsection{zip, zipwith}

\par \verb|zip| juxtaposes elements from two lists to form a list of pairs of corresponding elements from them. The pairs can be further processed to produce a flat result if \verb|zipwith| is used. \verb|zip| and \verb|zipwith| will return a list of size \verb|⌊⍥≢|\footnote{when end of list is encountered when the pairs are formed, the operation finishes}.

\begin{verbatim}
; MalbolgeLISP
% [zip selfie (iota 3)]
..........|..........
((0 0) (1 1) (2 2))
% [+ zipwith (iota 3) (iota 3)]
...............|.....................
(0 2 4)

      ⍝ APL
      zip←,¨
      zipwith←{⍺⍺/¨⍺,¨⍵}
      zip⍨ ⍳3
┌→──────────────────┐
│ ┌→──┐ ┌→──┐ ┌→──┐ │
│ │0 0│ │1 1│ │2 2│ │
│ └~──┘ └~──┘ └~──┘ │
└∊──────────────────┘
      (⍳ 3) (+ zipwith) (⍳ 3)
┌→────┐
│0 2 4│
└~────┘
\end{verbatim}

\subsection{flatten, flatmap}

\par \verb|flatten| and \verb|flatmap|'s mutual relations somewhat resemble \verb|zip| and \verb|zipwith|. While the first operation is a function that simply flattens a list, the second function is a \verb|map| which result is flattened afterwards. List flattening isn't deep:

\begin{verbatim}
; MalbolgeLISP
% (flatten '((1 2) (3 4)))
..................|...
(1 2 3 4)
% (flatten '(((1 2) (3 4)) (3 4)))
..........................|...
((1 2) (3 4) 3 4)
; Simple deep flattening
% (def deepf (bind iterate = flatten))
............|........
bind/syn
% (deepf '(((1 2) (3 4)) (3 4)))
..........................|............................
(1 2 3 4 3 4)
; Successor and predecessor of a number
% (def sp (monad (tie [x + 1] [x - 1])))
.........................|..
(lambda (x) (tie (+ x 1) (- x 1)))
% (flatmap sp '(1 2 3 4))
.............|........................................................
(2 0 3 1 4 2 5 3)
; As opposed to...
% (map sp '(1 2 3 4))
.............|........................................................
((2 0) (3 1) (4 2) (5 3))

      ⍝ APL
      f←⊃,/
      f ((1 2) (3 4))
1 2 3 4
      f (((1 2) (3 4)) (3 4))
┌───┬───┬─┬─┐
│1 2│3 4│3│4│
└───┴───┴─┴─┘
      ⍝ two choices for deep flattening
      ⍝ MalbolgeLISP port:
      g←f⍣≡
      ⍝ Idiomatic APL:
      h←∊
      g (((1 2) (3 4)) (3 4))
1 2 3 4 3 4
      h (((1 2) (3 4)) (3 4))
1 2 3 4 3 4
      sp←(+,-)∘1
      fm←{⊃,/⍺⍺¨⍵}
      sp fm 1 2 3 4
2 0 3 1 4 2 5 3
      sp¨ 1 2 3 4
┌───┬───┬───┬───┐
│2 0│3 1│4 2│5 3│
└───┴───┴───┴───┘
\end{verbatim}

Illustratively, flattening a list alters it in the following way:

\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{figures/beforeFlatten.png}
\caption{A list before flattening}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{figures/afterFlattening.png}
\caption{A list after flattening}
\end{figure}

\subsection{folds}

\par Folding is a way to transform an entire list by putting a binary operation between each element of it. Since folding must yield a result, and the list might be empty, \verb|fold| takes an additional argument which specifies the identity element, which semantically means that the value is left unchanged\footnote{for instance, the identity element of $+$ is \verb|0|, since $x + 0 = x$ and the identity element of $\times$ is \verb|1|, since $x \times 1 = x$}. Since APL supports only reductions, the comparison between MalbolgeLISP and APL is not demonstrated in this example.

\begin{verbatim}
% (fold 0 + '(1 2 3 4 5))
...............|.........................
15
% [[[[[0 + 1] + 2] + 3] + 4] + 5]
..........................|................
15
% (fold 0 + null)
.......|.....
0
\end{verbatim}

\par If it is known that the list contains at least one element, \verb|fold'| can be used, which is equivalent to APL's reductions:

\begin{verbatim}
; MalbolgeLISP
% (fold' + '(1 2 3 4 5))
...............|.........................
15

      ⍝ APL
      +/ 1 2 3 4 5
15
\end{verbatim}

\subsection{where}

\par \verb|where| is a function borrowed from APL - \verb|⍸|\footnote{\url{http://help.dyalog.com/16.0/Content/Language/Primitive\%20Functions/Where.htm}}. It returns the indices on which the input array contains truthy values. If the truthy value is greater than one, it's repeated that amount of times. For example:

\begin{verbatim}
; MalbolgeLISP
% (where '(1 0 1 0 1 1 1 0))
................|...
(0 2 4 5 6)
% (where '(1 2 3 4))
............|...
(0 1 1 2 2 2 3 3 3 3)

      ⍝ APL
      ⍸1 0 1 0 1 1 1 0
0 2 4 5 6
      ⍸1 2 3 4
0 1 1 2 2 2 3 3 3 3
\end{verbatim}

\subsection{count}

\par \verb|count| counts the amount of times it's functor returned a truthy value for each element of a list. It can be trivially expressed as a \verb|fold| and \verb|map|. An example of \verb|count| usage follows.

\begin{verbatim}
; MalbolgeLISP
% (count (bind = 2) '(2 2 3 2 2 3))
....................|............................................
4
% (def count' (dyad (fold 0 + (map [[< bind 0] atop x] y))))
...............................|..
(lambda (x y) (fold 0 + (map (atop (bind < 0) x) y)))
% (count' (bind = 2) '(2 3 2 3))
..................|..............................................
..........................................................
2

      ⍝ APL
      count←{+/(0<⍺⍺)¨⍵}
      2∘= count 2 3 2 3 2 2
4
\end{verbatim}

\subsection{replicate}

\par \verb|replicate| is one of the most overloaded functions in MalbolgeLISP. It takes three different forms depending on the argument types:
\begin{itemize}
    \item \verb|replicate list list| - copy elements from list 2 according to the masks in list 1. comparable to \verb|filter|\footnote{a filter-like function is implemented in APL using replicate}, but it takes a pre-mapped array instead of a functor.
    \item \verb|replicate num list| - duplicate the list specified amount of times
    \item \verb|replicate num any| - make a list out of any atom repeated specified amount of times
\end{itemize}

\begin{verbatim}
; MalbolgeLISP
% (replicate 5 hello)
......|....
(hello hello hello hello hello)
% (replicate 5 '(1 2))
...........|....
(1 2 1 2 1 2 1 2 1 2)
% (replicate '(1 0 2) '(1 2 3))
..................|....
(1 3 3)

      ⍝ APL
      replicate1←⍴∘⊂
      5 replicate1 'hello'
┌─────┬─────┬─────┬─────┬─────┐
│hello│hello│hello│hello│hello│
└─────┴─────┴─────┴─────┴─────┘
      replicate2←{↑,/⍺⍴⊂⍵}
      5 replicate2 1 2
1 2 1 2 1 2 1 2 1 2
      1 0 2/1 2 3
1 3 3
\end{verbatim}

\subsection{scan}

\par \verb|scan| performs a \verb|fold| with partial results. The standard behavior diverges from APL (since the identity element is excluded), but it can be accomplished using \verb|scan'|, which assumes that the input list has at least a single element.

\begin{verbatim}
; MalbolgeLISP
% (scan' * '(1 2 3 4 5))
..............|....................
(1 2 6 24 120)
% (scan * 1 '(1 2 3 4 5))
..............|........................
(1 1 2 6 24 120)

      ⍝ APL
      ×\1 2 3 4 5
1 2 6 24 120
\end{verbatim}

\subsection{uniq}

\par \verb|uniq| returns unique elements of a list using the \textit{formal definition of equality}. It uses an algorithm which makes $O(n^2)$ equality checks in the pessimistic case. To demonstrate:

\begin{verbatim}
; MalbolgeLISP
% (uniq '(1 6 2 5 2 6 2 3))
................|...
(1 6 2 5 3)
% (uniq '((1 2) (1 2) (1 2 3) (4 5) 6 (4 5)))
...................................|...
((1 2) (1 2 3) (4 5) 6)

      ⍝ APL
      ∪ 1 6 2 5 2 5 6 2 3
1 6 2 5 3
      ∪ ((1 2) (1 2) (1 2 3) (4 5) 6 (4 5))
┌───┬─────┬───┬─┐
│1 2│1 2 3│4 5│6│
└───┴─────┴───┴─┘
\end{verbatim}

\subsection{sort}

\par \verb|sort| is a function that sorts a list with an arbitrary comparator (or assumes a default comparator for sorting numeric lists, if none was provided). MalbolgeLISP utilises the insertion sort algorithm\footnote{because of it's simplicity and performance on small lists, which are going to realistically be the main use case for it in MalbolgeLISP}. It has a time complexity of $O(n^2)$ and auxiliary space requirement of $O(1)$. It takes maximum time to sort a list if elements are sorted in reverse order, and it takes minimum time when the elements are already sorted. For example:

\begin{verbatim}
; MalbolgeLISP
% (sort '(1 9 5 8 2 3 9 5))
................|...
(1 2 3 5 5 8 9 9)
% (sort '(1 9 5 8 2 3 9 5))
................|...
(1 2 3 5 5 8 9 9)
% (sort > '(1 9 5 8 2 3 9 5))
.................|......................................
........................................................
......................
(1 2 3 5 5 8 9 9)

      ⍝ APL
      ⍝ Implementation of an insertion sort.
      sortn←((≥⊢⍤/⊢),⊣,<⊢⍤/⊢)/
      sortn 32 4 1 34 95 3 2 120 ¯38
┌────────────────────────┐
│¯38 1 2 3 4 32 34 95 120│
└────────────────────────┘
      ⍝ A more idiomatic way of solving the problem:
      sortn1←{⍵[⍋⍵]}
      sortn1 32 4 1 34 95 3 2 120 ¯38
¯38 1 2 3 4 32 34 95 120
      ⍝ Sorting with a comparator
      sortc←{p←⍺⍺⋄{r←/∘⍵⋄c←⍺p⍵⋄(r c),⍺,r ~c}/⍵}
      > sortc 32 4 1 34 95 3 2 120 ¯38
┌────────────────────────┐
│¯38 1 2 3 4 32 34 95 120│
└────────────────────────┘
\end{verbatim}

\subsection{take, take', drop, drop'}

\par \verb|take| and \verb|take'| extract $n$ elements from the front (\verb|take|) or from the back (\verb|take'|) of a list. They correspond to \verb|↑| and \verb|((-⊣)↑⊢)| in APL. \verb|take| triggers a copy, while \verb|take'| doesn't:

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{figures/take.png}
\caption{Taking the first three elements from a list}
\end{figure}

\par The list must be cloned, since setting the tail of the list node which holds \verb|C| to \textit{NULL} (to yield a new list) would invalidate existing references. This doesn't apply to \verb|take'|, since it doesn't have to modify the memory it's operating on.

\begin{verbatim}
; MalbolgeLISP
% (take 3 '(1 2 3 4 5))
..............|....
(1 2 3)
% (take' 3 '(1 2 3 4 5))
..............|....
(3 4 5)

      ⍝ APL
      3↑1 2 3 4 5
1 2 3
      ¯3↑1 2 3 4 5
3 4 5
\end{verbatim}

\par \verb|drop| and \verb|drop'| exhibit the same behavior. They drop $n$ elements from the front (\verb|drop|) or the back (\verb|drop'|) of a list. Dropping elements from the back requires a copy, while dropping them from the front does, for the reasons outlined above.

\begin{verbatim}
; MalbolgeLISP
% (drop 3 '(1 2 3 4 5))
..............|....
(4 5)
% (drop' 3 '(1 2 3 4 5))
..............|....
(1 2)

      ⍝ APL
      3↓1 2 3 4 5
4 5
      ¯3↓1 2 3 4 5
1 2
\end{verbatim}

\section{Iteration and recursion}

\par This section is intended to demonstrate various implementations of the Fibonacci series in MalbolgeLISP\footnote{in comparison to APL} using iteration and recursion. Then, the approaches will be judged by cleaniness, conciseness and performance.

\par The naive, doubly recursive attempt timed at 1m 19s follows:

\begin{minted}{lisp}
(defun fib1 (n) (
    if [n < 2]
        n
        [(fib1 [n - 1]) + (fib1 [n - 2])]))
(fib1 6) ; => 8
; APL: {1≥⍵:⍵ ⋄ (∇⍵-2)+∇⍵-1} 6
\end{minted}

\par A singly-recursive attempt which keeps track of the accumulator tuple. There exist two versions of it - a port of the APL solution and the idiomatic MalbolgeLISP attempt, which is much faster than the port, since it takes advantage of MalbolgeLISP's support of functions of arity 3 or higher.

\begin{minted}{lisp}
; APL version: {⍺←0 1 ⋄ 0=⍵:⊃⍺ ⋄ (1↓⍺,+/⍺)∇⍵-1} 6
; Direct port at 1m 6s:
(defun fib2 (n) ((lambda (a w) (
    if [w = 0]
        (#0 a)
        ((bruijn 0) (tie (#1 a) (lift + a)) [w - 1]))) '(0 1) n))
; A more idiomatic solution at 54s:
(defun fib2 (n) ((lambda (x y w) (
    if [w = 0]
        x
        ((bruijn 0) y [x + y] [w - 1]))) 0 1 n))
\end{minted}

\par An iterative attempt timed at 43s:

\begin{minted}{lisp}
; APL version: {⊃+\∘⌽⍣⍵⍳2} 6
(defun fib3 (n) (#0 (
    iterateN n (lambda (x) (
        tie (#1 x) [(#0 x) + (#1 x)])) '(0 1))))
\end{minted}

\par It should be noted that even though the second approach is generally faster, it's not as clean as the first or third one. The first approach is the slowest, while the second and third approaches are faster. Generally, the most concise, clean and the fastest attempt is the iterative attempt.


\documentclass[12pt]{article}
\usepackage{times}
\usepackage{indentfirst}
\usepackage{graphicx}
\usepackage{multirow}
\usepackage{float}
\usepackage{amsmath}
\graphicspath{ {./img/} }
%Setup the layout of the pages
\topmargin 0.0cm
\oddsidemargin 0.2cm
\textwidth 16cm
\textheight 21cm
\footskip 1.0cm

\title{
    Data Science\\
}
\author{Molin Liu}

\begin{document}

\maketitle

\section*{Introduction}

This is a review work for the Data Science course in
\textit{Univeristy of Glasgow}.

\section{Linear Algebra}

\subsection{Vector}
Vectors can be composed (via addition),
compared (via norms/inner products)
and weighted (by scaling).

\subsubsection{Basic Vector Operations}
(Ommited)
\subsubsection{Inner Product}
$$\cos \theta=\frac{\mathbf{x} \bullet \mathbf{y}}{\|\mathbf{x}\|\|\mathbf{y}\|}$$
 - The inner product of two orthogonal vectors is 0;

\subsubsection{Norm}

- Norm 0: Count of non-zero values;

- Norm 1: Sum of absolute values;

- Norm 2: Euclidean distance;

\subsubsection{High Dimensional Vector Space}

\subsection{Matrix}

\subsubsection{Operations}
(Ommited)

\subsubsection{Anatomy of Matrix}

\subsection{Polynomial Regression}

The \textbf{Polynomial Regression} can be written as:
$$
t=w_{0}+w_{1} x+w_{2} x^{2}+w_{3} x^{2}+\ldots+w_{K} x^{K}=\sum_{k=0}^{\kappa} w_{k} x^{k}
$$

Define the loss funcion:
$$
\mathcal{L}=\frac{1}{N}(\mathbf{t}-\mathbf{X} \mathbf{w})^{\top}(\mathbf{t}-\mathbf{X} \mathbf{w})
$$
\subsubsection{Generalization \& Overfitting}

We can find out that the \textbf{loss} will always decrease
as the model is made more complex.

How to choose the right model complexity?
\textbf{Cross-validation}

\subsubsection{Cross-Validation}

\section{Classification}

The \textbf{Classification} task is to classify
a set of \textit{N} objects $x_i$ with attributes.
Each object has an associated label $t_i$

\textbf{Probabilistic classifier} produce a probability of class membership:

$P(t_{\text {new}}=k | \mathbf{x}_{\text {new }}, \mathbf{X}, \mathbf{t})$

\textbf{non-Probabilistic classifier} produce a hard assignment:

$t_{new}=1$ or $t_{new}=0$

\subsection{KNN}

K-Nearest Neighbours(KNN)

- Non-probabilistic classifier;

- Supervised trainning;

- Fast;

- We can use CV to find the right $K$;

\subsubsection{Problem}
- As $K$ increases, the small classes will disppear.

\subsection{Logistic Regression}

\subsection{SVM}
\subsubsection{Hard Margin}
If the training data is linearly seperable,
we can select two parallel hyperplanes that separate the two classes of data,
so that the distance between them is as large as possible.

These hyperplanes can be described by the equations:
$$
\vec{w} \cdot \vec{x}_{i}-b \geq 1, \text { if } y_{i}=1
$$
or
$$
\vec{w} \cdot \vec{x}_{i}-b \leq-1, \text { if } y_{i}=-1
$$

We can easily infer that
$$
y_{i}\left(\vec{w} \cdot \vec{x}_{i}-b\right) \geq 1, \quad \text { for all } 1 \leq i \leq n
$$

We want to maximise $\gamma=\frac{1}{\|\mathbf{w}\|}$,
equivalent to minimising $\|\mathbf{w}\|$

Note: $y_i$ is the label in the data, which is in $\{-1, 1\}$,
rather than vertical axis value of the data.
\subsubsection{Soft Margin}
Soft-margin function for the data are not linearly seperable.

\subsubsection{Inner Product}

\section{ROC}
Sensitivity/Recall

$$
S_{e}=\frac{T P}{T P+F N}
$$

Specificity

$$
S_{p}=\frac{T N}{T N+F P}
$$

\section{Unsupervised Learning}
\subsection{K-Means}

\subsection{Gaussion Mixture Model}
\end{document}