% \cleardoublepage
% \chapter*{Introduction}
% \markboth{Introduction}{Introduction}
% \addcontentsline{toc}{chapter}{Introduction}

\chapter{Introduction}

\section{Problem and Motivation}

Suppose we are given samples of data say $\mathbf{x}$ and $\mathbf{y}$, s.t.

\begin{align*}
    \mathbf{x} = x_1, ..., x_n  \\
    \mathbf{y} = y_1, ..., y_n
\end{align*}

For example, we may be measuring the blood pressure and heart rate of Alice at time $k$,
say $x_k$ and $y_k$ respectively, with $x_k, y_k \in \R$. Further, suppose we are unaware of her context, for example,
Bob hacked into Alice's Apple watch and so can only read $\mathbf{x}$ and $\mathbf{y}$ --- he has no idea of
anything she might be up to.

Bob then observes the following trend:

\begin{figure}[htb]
    \centering
    \begin{tikzpicture}
        \node (img)  {\includegraphics[width=.45\textwidth]{correlated_hr.png}};
        \node[below=of img, node distance=0cm, yshift=1cm,font=\color{black}] {time};
        \node[left=of img, node distance=0cm, rotate=90, anchor=center,yshift=-0.7cm,font=\color{black}] {bpm/mmHg};
    \end{tikzpicture}
    \caption{Heart rate (HR) and Blood pressure (BP) measured in bbp and mmHg resp. Day 1.}
\end{figure}

Bob, having studied data science, is well aware of the fallacy of the law of small numbers\footnote{
    The law of small numbers is the error of concluding too much from too few data (many such examples
    can be found in the book "fooled by randomness" by \cite{taleb}).}.
 He therefore checks again the data the next day at a slightly different time. He again observes
a similar trend, and is now more confident in the existence of a causal relation and --- having neglected biology ---
makes the conjecture that either blood pressure causes heart rate, or
perhaps the other way around.

Given this strong correlation, Bob asserts that he may either model $x$ as a function of $y$ or the other way
around. He proceeds to find some $f$ s.t. $f(x) \approx y$. The next day, to his dismay, he notices that his
model has terrible performance when evaluated on new data. He then proceeds to see what is going on, and
observes the following:

\begin{figure}[htb]
    \centering
    \begin{tikzpicture}
        \node (img)  {\includegraphics[width=.45\textwidth]{uncorrelated_hr.png}};
        \node[below=of img, node distance=0cm, yshift=1cm,font=\color{black}] {time};
        \node[left=of img, node distance=0cm, rotate=90, anchor=center,yshift=-0.7cm,font=\color{black}] {bpm/mmHg};
    \end{tikzpicture}
    \caption{Heart rate (HR) and Blood pressure (BP) measured in bbp and mmHg resp. Day 3.}
\end{figure}

As it turns out, in the last few days, Alice was working hard on finishing her thesis and the deadline had been
the previous day. But how, Bob wondered, could this have changed the relationship between BP and HR?
Finally, admitting to himself that curve fitting alone is not enough to understand the world;
Bob spends some time learning about the heart. It turns out, that fear triggers a "flight or fight"
response that increases both the heart rate and blood pressure; Interestingly your heart rate and blood pressure
won’t always rise and fall in sync.

So what did Bob learn\footnote{
    Note that heart rate and blood pressure are intimately linked, and the story between them is more complicated.
    The plots were randomly generated using a Gaussian process, however they do resemble some real examples that
    can be found in the internet.
}?

\begin{enumerate}
    \item When we train a model with some data, and then use it on some newly acquired data, we might
    face a \textbf{covariate-shift}\footnote{Another way to describe the covariate-shift is as follows,
    suppose we observe some samples, but
    these samples only represent a small portion of the data space, then when we acquire new data, this new data might be very
    different from our original acquisition. } --- that is, the distribution might change due to the context changing.
    \item When we see correlation it might be spurious due to a \textbf{confounder} --- fear was the \textbf{confounder}
    of the heart rate and blood pressure.
\end{enumerate}

If Bob was able to incorporate these notions into his machine learning models, then it might have been more
robust to the covariate-shift. To give a more concrete example, there is a "neural net tank urban legend"\footnote{
    More about this story here: https://www.gwern.net/Tanks.
}
, where a neural network accurately predicts if there is a tank or not in an image, but it turns out it uses the
weather as a predictor. From this it is clear that the model will perform badly under covariate shift, and indeed
it makes the case that incorporating causality to a model should make it more robust as
\cite{scholkopf2019causality} argues. Note that this is in effect the issue with generalization in machine learning:
how can we ensure that we learn \textit{meaningful} representations (features about the tanks) rather than just
correlations (the weather) useful for training accuracy.

As for confounders, it is impossible to say anything in general\footnote{
    For most of the 20th century, a huge debate took place to determine the question of whether or not
    smoking caused cancer. A clever argument against a causal relation was that there existed a gene that
    made a person both want to smoke and more prone to cancer; even the father of modern statistics
    himself thought this explanation more plausible (For a good read on how science is and was used
    for wrong see the excellent book of \cite{NaomiMerchants}).

}. We must therefore specify a causal model, and
then see what guarantees we can give under what assumptions. Even in the absence of confounders it is highly
non trivial to determine causality.

As this simple example illustrates, causality is related to many interesting questions; perhaps, one of the most
simple questions we can ask --- and the one that we will explore --- is, given that either X causes Y, or Y causes X
(we assume no confounders) then, when
can we predict the direction of causality? If yes, how?

In the figure bellow (figure \ref{fig:simple_bivariate_example}) can you tell if $X$ causes $Y$? Or perhaps
it is the other way around? The right answer is that $X$ causes $Y$, and we will show algorithms that
can accurately predict causality in such settings with as few as 75 samples.

\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{bivariate_causal_example.png}
    \caption{75 samples of data $X$, $Y$.  The samples are generated independently as follows:
    $y_i = f(x_i) + n_i$ where $x_i$ is drawn from an exponential distribution and $n_i$ is drawn
    independently from a Gaussian one and $f(x) = 10 \tanh(x) + 4\sin(x) + x + x^2$}
    \label{fig:simple_bivariate_example}
\end{figure}

\section{Causality}

\subsection{Causal models: FCM}

We can model any causal model by using a \textbf{Function Causal Model (FCM)} which can be constructed as follows
(\cite{Peters2017}):

We \textit{generate} a random vector\footnote{That is, each $X_i$ is a random variable.}
$X = (X_1, ..., X_d)$ by using a graph $\mathcal{G}$ (encoding the relationships),
a set of functions $f = (f_1, ..., f_d)$ (encoding the type of relationship) and a noise distribution $\mathcal{E}$
(the randomness generator).

For each $i = 1, ..., d$

$$
    X_i \leftarrow f_i(X_{\text{Pa}(i)}, E_i), \quad E_i \thicksim \mathcal{E}
$$

where $\text{Pa}(i)$ is the set of parents of $i$; and so $X_{\text{Pa}(i)}$ is the set of random variables
for the parents of $X_i$. For example in Figure \ref{fig:fcm_1}, $\text{Pa}(4) = \{1, 2\}$,
and so $X_{\text{Pa}(4)} = \{X_1, X_2\}$.

\begin{figure}[!h]
    \centering

    \begin{tikzpicture}[
        > = stealth, % arrow head style
        shorten > = 1pt, % don't touch arrow head to node
        auto,
        node distance = 2.5cm, % distance between nodes
        thick % line style
    ]

    \tikzstyle{every state}=[
        draw = black,
        thick,
        fill = white,
        minimum size = 4mm
    ]

    \node[state] (a) {$X_1$};
    \node[state] (b) [right of=a] {$X_2$};
    \node[state] (c) [below left of=a] {$X_3$};
    \node[state] (d) [below right of=a] {$X_4$};

    \path[->] (a) edge node {} (c);
    \path[->] (a) edge node {} (d);
    \path[->] (b) edge node {} (d);

    \end{tikzpicture}
    \caption{ Example for FCM with $X = X_1, ... X_4$, with $E_i \thicksim \mathcal{E}$,
    with $X_1 = f_1(E_1)$, $X_2 = f_2(E_2)$, $X_3 = f_3(X_1, E_3)$, and
    $X_4 = f_4(X_1, X_2, E_4)$}
    \label{fig:fcm_1}

\end{figure}

Note that causal relations can also be \textit{cyclic}, i.e. $X$ causes $Y$ which in turn causes
$X$ ad infinitum; while this deserves consideration as many systems have feedback loops we will
not look at such settings for simplicity.

\subsection{Interventions}

To make precise the meaning of causality, suppose that we are given
two random variables $X$, $Y$ with joint distribution $p_{x, y}$. Intuitively we
would say that $X$ causes $Y$, or $X \rightarrow Y$, if we intervene on $X$ and then see an effect on $Y$. In particular
we will denote $\operatorname{do}(x)$ --- short for $\operatorname{do}(X = x)$ --- as an intervention
that forces the variable $X$ to have the value $x$, and leaves the rest of the system untouched.
Following the convention inspired by \cite{pearl2000causality},
we define the resulting distribution as $p_{y|do(x)}$.

This motivates the following definition:

\begin{definition}
    We say that $X$ \textbf{causes} $Y$ if $p_{y|do(x)} \neq p_{y|do(x^\prime)}$ for some
    $x \neq x^\prime$
\end{definition}

When we talk about $p_{y|x}$, we often say, "The chance of $y$ given that $x$ happened". This sounds similar
to $p_{y|do(x)}$; note however that "$x$ happened" and "force $X = x$" are very different. Imagine that
there indeed was a gene that made people both prone to smoking and cancer; then if we forced someone at
random to smoke, he would on average be less likely to have cancer than someone who smoked because he wanted to.
This also illustrates
one of the limitations of causality: some interventions are not possible due to ethical issues.

You might have heard about randomized trials or A/B testing, these are both common ways to estimate $p_{y|do(x)}$.
For example, when developing cures, the idea of a random trial is to give experimental drugs to participants
at random. When designing new UIs to maximize user participation in apps, developers implement A/B testing,
they assign new versions to people at random to estimate engagement. Note that in both of these, we are able
to avoid a potential confounder by picking $x$'s at random, and "forcing" them to "do$(x)$".

Since we will restrict ourselves to the observational setting, we will not be able to perform any interventions,
which would allow us to estimate $p_{y|do(x)}$. In this setting however, in order to perform any meaningful
inference, we will need to make concessions; in particular, we will make some assumptions about
the causal structure. If we are able to infer the causal structure in such a setting, we
shall call it \textbf{identifiable}.

% f we have the random variables $X$, $Y$, and $Z$, then depending on how they are causaly linked, then
% we will have different relationships between the marginals and the Pearl's "do" conditional. For example,
% if $X \rightarrow Z \rightarrow Y$, then we will have that $p_X = p_{X|do(y)}$, but $p_Y \neq p_{Y|do(x)}$.

% Indeed, for all 6 different relations of $X$, $Y$, and $Z$ we get a different set of relationships between
% the interventional and observational conditions. Then, assuming that we get both interventional and observational data it suffices to check to which of the 6
% possible structure the data corresponds to (at least in theory).

We restrict ourselves also to the bivariate case; one big difference worth noting is that in the
multivariate setting we can test conditional independence. Using conditional independence tests
is a very powerful method for causal inference. Suppose
we have random variables $X$, $Y$ and $Z$, then if we can estimate that $X \bigCI Y | Z$, then it must be
that all information between $X$ and $Y$ must flow through $Z$ (See Figure \ref{fig:conditionalIndep}).
This test alone has has now greatly reduced the space of possible causal directions between these random variables.

One can in fact generalize the conditional independence such that $X$ and the other variables are
a collection of random variables, which gives a lot of flexibility to devise clever algorithms.
The theory comes from graphical models, which tries to understand the relationship between
distributions and their graphical counterparts, such as Figure \ref{fig:conditionalIndep}.
The key difference is that in graphical models we do not care about the causal direction.
Since we will not be using any of this theory, we will not go into any detail
(a good overview can found in \cite{bishop2006pattern}).

\begin{figure}[!h]
    \centering

    \begin{tikzpicture}[
        > = stealth, % arrow head style
        shorten > = 1pt, % don't touch arrow head to node
        auto,
        node distance = 2.5cm, % distance between nodes
        thick % line style
    ]

    \tikzstyle{every state}=[
        draw = black,
        thick,
        fill = white,
        minimum size = 4mm
    ]

    \node[state] (a) {$Z$};
    \node[state] (b) [below left of=a] {$X$};
    \node[state] (c) [below right of=a] {$Y$};

    \path[-] (a) edge node {} (b);
    \path[-] (a) edge node {} (c);

    \end{tikzpicture}
    \caption{An example of FCM with random variables $X$, $Z$ and $Y$; we leave it undirected}
    \label{fig:conditionalIndep}

\end{figure}

In some sense the two variable case is hard because we cannot use conditional independence. As we will see,
it is not possible to distinguish causality in the general bivariate setting when only observing
observational data; we will thus need to restrict
the class of such models. The underlying structure behind such causal models is what is known as
Structural Equation Models (SEM) (\cite{Peters2017}). Essentially
it is a model specification; and the key insight is that it should not be reversible.

\section{Proposed Methods}

We propose a new type of inference method which exploits the i.i.d
noise assumption; the idea is to split the data in different intervals, and then to regress each
interval separately. In the causal direction one would expect that the residuals of each
interval will be more homogenous\footnote{By homogenous residuals we mean that the residuals will be similar
to each other, for example if we plot their respective histograms, these will asymptotically converge to the
same distribution}  --- in the ANM settings, asymmetries introduced
by inversion will break the i.i.d assumption. We will conclude by
showing that this intuition is indeed correct, by proving that the method is consistent ---
assuming that causal discovery is indeed possible and that the additive noise is i.i.d. We will
also propose a second method, however while also consistent, it requires knowledge of the
noise distribution, which is not a good assumption in practice.

\section{Outline}

We begin by formalizing further the causal framework that we will explore; we will then present an overview of
popular methods for inference on the bivariate causal setting. Next, we will overview some notions in statistical distance
--- as these play a central role in causal inference --- such as maximum mean discrepancy (MMD) and f-divergences.
For the convenience of the reader we will also review some background material that will help gain a better
intuition on the MMD.

We will end by going over the proposed methods, and showing both theoretical guarantees and experimental results.


% This is part of the TFTB Reference Manual.
% Copyright (C) 1996 CNRS (France) and Rice University (US).
% See the file refguide.tex for copying conditions.

\markright{fmsin}
\section*{\hspace*{-1.6cm} fmsin}

\vspace*{-.4cm}
\hspace*{-1.6cm}\rule[0in]{16.5cm}{.02cm}
\vspace*{.2cm}

{\bf \large \sf Purpose}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
Signal with sinusoidal frequency modulation.
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Synopsis}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
\begin{verbatim}
[y,iflaw] = fmsin(N)
[y,iflaw] = fmsin(N,fmin)
[y,iflaw] = fmsin(N,fmin,fmax)
[y,iflaw] = fmsin(N,fmin,fmax,period)
[y,iflaw] = fmsin(N,fmin,fmax,period,t0)
[y,iflaw] = fmsin(N,fmin,fmax,period,t0,f0)
[y,iflaw] = fmsin(N,fmin,fmax,period,t0,f0,pm1)
\end{verbatim}
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Description}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
        {\ty fmsin} generates a sinusoidal frequency modulation, whose
minimum frequency value is {\ty fmin} and maximum is {\ty fmax}.  This
sinusoidal modulation is designed such that the instantaneous frequency at
time {\ty t0} is equal to {\ty f0}, and the ambiguity between increasing or
decreasing frequency is solved by {\ty pm1}.\\

\hspace*{-.5cm}\begin{tabular*}{14cm}{p{1.5cm} p{8.5cm} c}
Name & Description & Default value\\
\hline
        {\ty N}       & number of points\\
        {\ty fmin}    & smallest normalized frequency          & {\ty 0.05}\\
        {\ty fmax}    & highest normalized frequency           & {\ty 0.45}\\
        {\ty period}  & period of the sinusoidal frequency modulation  & {\ty N}  \\
        {\ty t0}      & time reference for the phase           & {\ty N/2} \\
        {\ty f0}      & normalized frequency at time {\ty t0}     & {\ty 0.25}\\
        {\ty pm1}     & frequency direction at {\ty t0} (-1 or +1)& {\ty +1}  \\
  \hline {\ty y}       & signal\\
        {\ty iflaw}   & instantaneous frequency law \\

\hline
\end{tabular*}

\end{minipage}
\vspace*{1cm}

{\bf \large \sf Example}
\begin{verbatim}
         z=fmsin(140,0.05,0.45,100,20,0.3,-1.0);
         plot(real(z));
\end{verbatim}
\vspace*{.5cm}

{\bf \large \sf See Also}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
\begin{verbatim}
fmconst, fmlin, fmodany, fmhyp, fmpar, fmpower.
\end{verbatim}
\end{minipage}


\section{Cycles}

%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Cycle detection}
  \begin{exampleblock}{Cycle detection (Problem 5.24--1)}
	\begin{table}[ht]
	  \centering
	  \renewcommand{\arraystretch}{1.1}
	  \begin{tabular}{c||c|c}
	    \hline
		& Digraph 			& Undirected graph  \\ \hline \hline
		DFS & {\uncover<2->{back edge $\iff$ cycle}}
		& {\uncover<3->{back edge $\iff$ cycle}}
		\\ \hline
		BFS & {\uncover<5->{\begin{tabular}[c]{@{}l@{}}back edge $\implies$ cycle\\ cycle \textcolor{red}{$\centernot\implies$} back edge \end{tabular}}}
		& {\uncover<4->{cross edge $\iff$ cycle}}
		\\ \hline
	  \end{tabular}
	\end{table}
  \end{exampleblock}

  \begin{columns}
	\column{0.50\textwidth}
	  \uncover<6->{\fignocaption{width = 0.30\textwidth}{figs/bfs-digraph-cycle-without-back.png}}
	\column{0.50\textwidth}
	  \uncover<7->{%
		\begin{alertblock}{Remark}
		  How to identify back edges?
		\end{alertblock}}
  \end{columns}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Evasiveness of acyclicity}
  \begin{exampleblock}{Evasiveness of acyclicity (Problem 5.24--2)}
	\[
	  \text{Evasiveness } \triangleq \text{ check } \binom{n}{2} \text{ edges (adjacency matrices)}
	\]

	\pause
	\centerline{Is acyclicity evasive?}
  \end{exampleblock}

  \pause
  \vspace{0.50cm}
  \fignocaption{width = 0.20\textwidth}{figs/adversary-logo.png}{\centerline{\pause Hint: Kruskal}}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Evasiveness of connectivity}
  \begin{exampleblock}{Evasiveness of connectivity (Additional Problem)}
	\[
	  \text{Evasiveness } \triangleq \text{ check } \binom{n}{2} \text{ edges}
	\]

	\centerline{Is connectivity evasive?}
  \end{exampleblock}

  \pause
  \vspace{0.50cm}
  \fignocaption{width = 0.20\textwidth}{figs/adversary-logo.png}{\centerline{Hint: Anti-Kruskal}}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Edge deletion}
  \begin{exampleblock}{Edge deletion (Problem 5.20)}
    \begin{itemize}
      \item connected, undirected graph $G$
      \item $\exists? e \in E: G \setminus e$ is connected?
      \item $O(|V|)$
    \end{itemize}
  \end{exampleblock}

  \pause
  \[
	\exists \text{ cycle} \iff \exists \text{ such } e
  \]

  \pause
  \[
	O(m + n)
  \]

  \pause
  \[
	\text{tree: } |E| = |V| - 1 \implies \text{ check } |E| \ge |V|
  \]
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Orientation of undirected graph}
  \begin{exampleblock}{Orientation of undirected graph (Problem 5.9)}
	\begin{itemize}
      \item undirected (connected) graph $G$
	  \item edges oriented \emph{s.t.}
		\[
		  \forall v, \text{in}[v] \ge 1
		\]
	\end{itemize}
  \end{exampleblock}

  \pause
  \[
	\text{orientation} \iff \exists \text{ cycle } C
  \]

  \pause
  \[
	\text{BFS/DFS from } v \in C
  \]
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Shortest cycle of undirected graph}
  \begin{exampleblock}{Shortest cycle of undirected graph (Problem 5.8)}
	Shortest cycle of $G$:
	\begin{itemize}
	  \item DFS on $G$
	  \item $\forall v: \text{level}[v]$
	  \item back edge $u \to v: \text{level}[u] - \text{level}[v] + 1$
	\end{itemize}
  \end{exampleblock}

  \pause
  \vspace{0.50cm}
  \begin{alertblock}{Question}
	What about digraphs?
  \end{alertblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%


\section{Optimization objective}\label{optimization-objective}

In support vector machine, we don't directly have $\lambda$. Instead, C
is used, which penalizes the training set instead of the $\theta$. In
reality, it's almost the same, as we can set a small value for C and
then we're giving more importance to the regularization term. We could
think of $C = \frac{1}{\lambda}$.

Informally, the C parameter is a positive value that controls the
penalty for misclassified training examples. A large C parameter tells
the SVM to try to classify all the examples correctly. \smallskip

A part from that, the objective function (minimizing cost function) is
almost the same as the logistic regression one:

\begin{equation}
min_{\theta} C \sum_{i=1}^m [y^{(i)}cost_1(\theta^T x^{(i)}) + (1 - y^{(i)})cost_0(\theta^T x^{(i)})] + \frac{1}{2} \sum_{j=1}^n \theta_j^2
\end{equation}

Where $cost_0$ and $cost_1$ are functions that are applied when $y = 0$
and $y = 1$, respectively. \smallskip

SVMs don't output probabilities, they output directly the class they
classificate into, using the following hyphotesis:

\begin{equation}
h_\theta(x) =
\begin{dcases}
    1, & \text{if\,} \theta^Tx \geq 0 \\
    0, & \text{otherwise}
\end{dcases}
\end{equation}

\section{Large Margin Intuition}\label{large-margin-intuition}

We can see the SVM as a large margin classifier, meaning that if we have
linearly separable data, it will separate it optimizing the boundary
distance between the data (fig \ref{fig:large_margin}).

\begin{figure}
\centering
\includegraphics[width=\textwidth]{img/large_margin.eps}
\caption{Example of linearly separable data correctly classified by an SVM.}
\label{fig:large_margin}
\end{figure}

\section{Kernels I}\label{kernels-i}

In logistic regression, when we had a non-linear problem, we had to use
polynomial features. Kernels are another way of getting more features
out of the training sample. \smallskip

Kernels are similarity functions. In the video, he selects 3 landmarks
and uses a gaussian kernel. He then uses those 3 landmarks to compute 3
features, using the gaussian to know the similarity between x and each
of the landmark.

Doing that, we can create a decision boundary around the landmarks,
where everything near them gets classified as 1 and everything that is
far from them gets classified as 0 (following a gaussian shape).

\section{Kernels II}\label{kernels-ii}

\subsection{Choosing Landmarks}\label{choosing-landmarks}

For every training example, get a landmark exactly at that location.
Then, features will be based around the similarity of a point in our
data to other points. This way we have one feature per datapoint, so we
have m features. \smallskip

Basically, we just substitute the features we had and we use this ones.

Mathematical implemenation note:

\begin{equation}
\sum_{j=1}^n \theta_j^2 = \theta^t M \theta
\end{equation}

Where M is a matrix that depends on the kernel. This is done to scale to
bigger training sets.

\subsection{SVM Parameters}\label{svm-parameters}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  Large C: Lower bias, high variance (small $\lambda$).
\item
  Small C: Higher bias, low variance (large $\lambda$).
\end{itemize}

\section{Using an SVM}\label{using-an-svm}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  Use SVM software package to solve for paraemters $\theta$.
\item
  You still need to specify the choice of parameter C and the kernel.
\item
  No kernel is sometimes called ``linear kernel''.
\item
  Do perfrom feature scaling before using the kernel.
\end{itemize}

Not all similarity functions make valid kernels. They need to satisfy a
technical condition called ``Mercer's Theorem''.

\subsection{Multiclass classification}\label{multiclass-classification}

Many SVM packages already have built-in multi-class classification
functionality. Otherwise, use one-vs-all.

\subsection{Logistic regression vs
SVMs}\label{logistic-regression-vs-svms}

Let n = number of features and m = number of training examples.
\smallskip

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  If n is large relative to m -\textgreater{} Use logistic regression or
  SVM without kernel.
\item
  If n is small, m is intermediate -\textgreater{} Use SVM with Gaussian
  kernel.
\item
  If n is small, m is large -\textgreater{} Create/Add features and then
  use logistic regression or SVM without kernel.
\end{itemize}

Neural networks is likely to work well for mos of these settings, but
they're way slower to train.

SVMs don't have to worry about local optima, because they solve a convex
problem. NN may have problems with that, though.

theory/07.01 - Support Vector Machines/07.01 - Support Vector Machines.tex


\documentclass[a4paper,11pt]{article}

\usepackage{physics}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{amsthm, mathtools}
%\usepackage{hyperref}
\usepackage{color}
\usepackage{jheppub}
\usepackage[T1]{fontenc} % if needed

\newcommand{\be}{\begin{equation}}
\newcommand{\ee}{\end{equation}}
\newcommand{\bes}{\begin{equation*}}
\newcommand{\ees}{\end{equation*}}
\newcommand{\bea}{\begin{flalign*}}
\newcommand{\eea}{\end{flalign*}}

%\linespread{1.0}
%\setlength{\parindent}{0em}
%\setlength{\parskip}{0.8em}

\title{\textbf{Some Information about Gravitational Waves}}
\author{Aditya Vijaykumar}
\affiliation{International Centre for Theoretical Sciences, Bengaluru, India.}
\emailAdd{aditya.vijaykumar@icts.res.in}
\abstract{One needs to get accustomed with what gravitational waves are before jumping in and working on it. One hopes the two tomes by Maggiore will aid one's exploration.}

\begin{document}
\maketitle

\section{Gravitational Waves as Geometry}

\textit{Though Maggiore, the main reference for this tome, prefers not to use geometric units, I shall unapologetically use them.}

The gravitational action is a sum of the Einstein-Hilbert action and the matter action; $S = S_E + S_M$ where,

$$S_E = \frac{1}{16 \pi} \int d^4 x \sqrt{-g}R$$

The \textit{energy-momentum tensor} $T_{\mu \nu}$ is defined from the variation of the matter action under a metric change $g_{\mu \nu} \rightarrow g_{\mu \nu} + \delta g_{\mu \nu}$ as,

$$\delta S_M = \frac{1}{2}  \int d^4 x \sqrt{-g} \ T^{\mu \nu} \delta g_{\mu \nu}$$

Variation of the total action with respect to $g_{\mu \nu}$ yields Einstein's equations,
$$\boxed{R_{\mu \nu} - \frac{1}{2}g_{\mu \nu}R = 8 \pi T_{\mu \nu}}$$

General relativity is invariant under a huge groups of transformations $x^\mu \rightarrow x'^\mu(x)$, as long as $x'$ is invertible, differentiable, and has a differentiable inverse. Such transformations are called \textit{diffeomorphisms}. Under diffeomorphisms, $g_{\mu \nu}$ transforms as,
$$g_{\mu \nu}(x') = \pdv{x^\rho}{x'^\mu} \pdv{x^\sigma}{x'^\nu} g_{\mu \nu}(x)$$

\subsection{Expansions around flat space}
Lets imagine that our system is a perturbation around the flat space metric,
$$g_{\mu \nu} = \eta_{\mu \nu} + h_{\mu \nu}$$
The magnitude of $h_{\mu \nu}$ is infinitesmal, and we only consider equations upto first order in $h_{\mu \nu}$.

Lets stare at the above equation for a few seconds. We know that the numerical values of the tensor depends on the choice of reference frame. So on what basis are we calling $h_{\mu \nu}$ infinitesmal? Actually, what we merely wish to say is that for some choice of reference frame, the above equation will be valid and we can do all our calculations. While this choice helps us simplify our calculations, it should be noted that choosing a specific reference frame breaks the diffeomorphism invariance.

Even after choosing the above transformation, we still have a residual gauge freedom left. Consider $$x'^\mu \rightarrow x^\mu + \xi^\mu (x)$$ where the derivatives of $\xi^\mu$ as as inifinitesmal as $h_{\mu \nu}$. By using the diffeomorphism property,
$$g'_{\mu \nu}(x') = \pdv{x^\rho}{x'^\mu} \pdv{x^\sigma}{x'^\nu} g_{\rho \sigma}(x)$$ and substituting for the partial derivatives, one gets,
\begin{flalign*}
	g'_{\mu \nu}(x') &= \left(\delta^\rho_\mu - \pdv{\xi^\rho}{x'^\mu}\right)  \left(\delta^\sigma_\nu - \pdv{\xi^\sigma}{x'^\nu}\right)  (\eta_{\rho \sigma} + h_{\rho \sigma}(x)) \\
	\eta_{\mu \nu} + h'_{\mu \nu}(x') &= \left(\delta^\rho_\mu \delta^\sigma_\nu - \pdv{\xi^\rho}{x'^\mu}\delta^\sigma_\nu - \pdv{\xi^\sigma}{x'^\nu} \delta^\rho_\mu \right)  (\eta_{\rho \sigma} + h_{\rho \sigma}(x))\\
	h'_{\mu \nu}(x') &= h_{\mu \nu}(x) - \pdv{\xi_\nu}{x'^\mu} - \pdv{\xi_\mu}{x'^\nu}
\end{flalign*}

\end{document}\grid


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{hyperref}
\usepackage{mathrsfs}
\usepackage{mathtools}
\usepackage{textcomp}
\usepackage{fullpage}

\newtheorem{theorem}{Theorem}[section]
\newtheorem{lemma}[theorem]{Lemma}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}

\newcommand{\defterm}[1]{\textit{#1}}

\title{A Compressed Introduction to Compressed Sensing}
\author{Benjamin Peterson}

\begin{document}
\maketitle

\begin{abstract}
We attempt to convey a sense of compressed sensing. Specifically, we discuss how $\ell_1$ minimization and the restricted isometry property for matrices can be used for sparse recovery of underdetermined linear systems even in the presence of noise.
\end{abstract}

\section{Introduction}
Suppose $A \in \mathcal{M}_{m\times n}(\mathbb{R})$ is a matrix and we obtain a measurement $y \in \mathbb{R}^m$, which we know to be the image of some $x \in \mathbb{R}^n$ under $A$. If the system $Ax = y$ is underdetermined (i.e., $m < n$), then $x$ is not unique. Suppose however that we want a single solution and we know $x$ is ``small''. We might try to pick an $x$ which solves the equation with the least $\ell_2$ norm by solving the convex program
\begin{equation}\label{eq:l2-prog}
\min_{x \in \mathbb{R}^n}\|x\|_{\ell_2}\qquad Ax = y.\tag{L}
\end{equation}
This program has the advantage that a unique solution is guaranteed. Unfortunately, the solution to \eqref{eq:l2-prog} is often not the best one for applications.

Many real world signals are known to be sparse (i.e.\ having few nonzero entries) in some basis. Thus, we might alternatively seek a sparse solution for $Ax = y$. The sparsest solution to the linear equation is given by the program
\begin{equation}\label{eq:l0-prog}
\min_{x\in \mathbb{R}^n} \|x\|_{\ell_0}\qquad Ax = y.\tag{S}
\end{equation}
Here the ``norm'' $\|x\|_{\ell_0}$ counts the number of nonzero entries in a vector $x$. Solving \eqref{eq:l0-prog} is a difficult NP-hard problem—the subset-sum problem may be reduced to it—in combinatorial optimization \cite{sparse-np}. We will replace \eqref{eq:l0-prog} with the convex program
\begin{equation}\label{eq:l1-prog}
\min_{x \in \mathbb{R}^n}\|x\|_{\ell_1}\qquad Ax = y.\tag{P}
\end{equation}
Solving this program is often called ``basis pursuit'' and can be efficiently done with linear programming or specialized algorithms \cite[ch. 15]{foucart-intro}. Relating the solutions of \eqref{eq:l0-prog} and \eqref{eq:l1-prog} is major goal in compressed sensing which we will explore in this paper.

There is an intuitive geometrical reason that we might expect the $\ell_1$ norm to be better at finding sparse solutions than the $\ell_2$ norm. The program~\eqref{eq:l2-prog} finds smallest ball around the origin that intersects the solution subspace of $Ax = y$. The level sets of the $\ell_1$ norm are polyhedra, which emphasize the axes. Therefore, it seems ``likely'' that the \eqref{eq:l1-prog} finds a sparse solution in the solution subspace.

We now introduce a restriction on matrices that allows sparse solutions to be easily recovered. Recall that a vector is said to be $k$-sparse if it has at most $k$ nonzero entries.
\begin{definition}[\cite{ct-decoding}]
A matrix $A \in \mathcal{M}_{m\times n}(\mathbb{R})$ satisfies the \defterm{restricted isometry property} (RIP) of order $k$ if there exists a $\delta_k \geq 0$ such that for all $k$-sparse vectors $x \in \mathbb{R}^n$, $$(1 - \delta_k)\|x\|_{\ell_2}^2 \leq \|Ax\|_{\ell_2}^2 \leq (1 + \delta_k)\|x\|_{\ell_2}^2.$$
The smallest such $\delta_k$ is called $A$'s \defterm{restricted isometry constant} of order $k$.
\end{definition}
Notice that $\delta_1 \leq \delta_2 \leq \dotsb$. The RIP is related to vector space frames \cite{ole-frames} and the Johnson-Lindenstrauss lemma \cite{rip-jl}. Unfortunately, checking whether a matrix has the restricted isometry property is NP-Hard in general \cite{rip-np}. On the other hand, many families of random matrices (e.g.\ having Gaussian or Bernoulli entries) satisfy the RIP with high probability \cite[\S 1.3]{ctr-stable}.

\begin{theorem}[{\cite[lemma 1.2]{ct-decoding}}]\label{thm:unique-sparse}
Suppose $A \in \mathcal{M}_{m\times n}$ satisfies RIP with $\delta_{2s} < 1$. Then the equation $Ax = y$ has an unique $s$-sparse solution given by \eqref{eq:l0-prog}.
\end{theorem}
\begin{proof}
Suppose $x, x\in \mathbb{R}^m$ are $s$-sparse and $Ax = Ax'$. By the RIP,
$$
0 \leq (1 - \delta_{2s})\|x - x'\|_{\ell_2} \leq \|A(x - x')\|_{\ell_2} = 0.
$$
Necessarily, $x = x'$.
\end{proof}

Before stating the main theorem, we introduce the notion of error. In reality, we do not know $Ax$ exactly. Instead, we measure $y = Ax + z$, where $z \in \mathbb{R}^m$ is a small error vector satisfying $\|z\|_{\ell_2} \leq \epsilon$. We would like our recovery of $x$ to be robust against error. To deal with this, we generalize \eqref{eq:l1-prog} to the (still convex) program
\begin{equation}\tag{P\textquotesingle}\label{eq:l1-approx}
\min_{x \in \mathbb{R}^n}\|x\|_{\ell_1}\qquad \|Ax - y\|_{\ell_2} \leq \epsilon.
\end{equation}

Finally, we can state the main theorem of this lecture. We denote the vector containing only the $s$ largest entries of $x$ by $x_s$.
\begin{theorem}[{\cite[1.3]{candes-rip}}]\label{thm:noisy-recovery}
Suppose $A \in \mathcal{M}_{m\times n}(\mathbb{R})$ satisfies RIP with $\delta_{2s} < \sqrt{2} - 1$. Let $x^*$ denote the solution to \eqref{eq:l1-approx}. Then, there are constants $C_0, C_1$ depending only on $\delta_{2s}$ such that $$\|x^* - x\|_{\ell_2} \leq C_0s^{-1/2}\|x - x_s\|_{\ell_1} + C_1\epsilon.$$
\end{theorem}

Theorem~\ref{thm:noisy-recovery} tells us that when RIP holds, basis pursuit recovers solutions very close to sparse solutions of the equation.

\section{Proof of Theorem~\ref{thm:noisy-recovery}}
We closely follow \cite{candes-rip}.

Set $x = x^* + h$. Our goal is to show $\|h\|_{\ell_2}$ is small. The RIP for $A$ only gives us control over sparse vectors, so we start by breaking $h$ up into $s$-sparse vectors. Let $T_0$ be the indexes of the $s$ largest entires of $x$. Let $T_1$ be the indexes of the $k$ largest entries of $h_{T_0^c}$. Let $T_2$ be the indexes of the $s$ largest entries of $h_{(T_0\cup T_1)^c}$ and so on. We observe that $h$ can be written as the sum of $s$-sparse vectors $h_{T_0} + h_{T_1} + h_{T_2} + \dotsb$. Using the triangle inequality,
\begin{equation}\label{eq:main-triang}
\|x - x^*\|_{\ell_2} = \|h\|_{\ell_2} \leq \|h_{T_0\cup T_1}\|_{\ell_2} + \|h_{(T_0\cup T_1)^c}\|_{\ell_2}.
\end{equation}
We will estimate the two terms of \eqref{eq:main-triang} separately and then combine them to prove the theorem. We start by showing the $\|h_{(T_0\cup T_1)^c}\|_{\ell_2}$ term can be bounded in terms of the first term $\|h_{T_0\cup T_1}\|_{\ell_2}$. Several useful intermediate inequalities are obtained along the way.

\begin{lemma}[Tail estimates]\label{lem:tail-estimate}
\begin{align}\label{eq:small-h1}
\|h_{T_0^c}\|_{\ell_1} &\leq \|h_{T_0}\|_{\ell_1} + 2\|x_{T_0^c}\|_{\ell_1}
\\
\label{eq:tail-sum}
\sum_{j \geq 2}\|h_{T_j}\|_{\ell_2} &\leq s^{-1/2}\|h_{T_0^c}\|_{\ell_1} \\
\label{eq:small-h2}
\|h_{(T_0\cup T_1)^c}\|_{\ell_2} &\leq \|h_{T_0}\|_{\ell_2} + 2s^{-1/2}\|x - x_s\|_{\ell_1}
\end{align}
\end{lemma}
\begin{proof}
Since $x$ is feasible for \eqref{eq:l1-approx} and $x^*$ is a minimum,
$$
\|x\|_{\ell_1} \geq \|x + h\|_{\ell_1} = \sum_{i \in T_0}|x_i + h_i| + \sum_{i \in T_0^c}|x_i + h_i| \geq \|x_{T_0}\|_{\ell_1} - \|h_{T_0}\|_{\ell_1} + \|h_{T_0^c}\|_{\ell_1} - \|x_{T_0^c}\|_{\ell_1}.
$$
The last step uses the triangle inequality twice. Rewriting and applying the reverse triangle inequality proves \eqref{eq:small-h1}.
$$
\|h_{T_0^c}\|_{\ell_1} \leq \|x\|_{\ell_1} - \|x_{T_0}\|_{\ell_1} + \|x_{T_0^c}\|_{\ell_1} + \|h_{T_0}\|_{\ell_1} \leq \|h_{T_0}\|_{\ell_1} + 2\|x_{T_0^c}\|_{\ell_1}
$$

If $j \geq 2$,
$$
\|h_{T_j}\|_{\ell_2} \leq s^{1/2}\|h_{T_j}\|_{\ell_\infty} \leq s^{-1/2}\|h_{T_{j - 1}}\|_{\ell_1}.
$$
Summing yields \eqref{eq:tail-sum}.
$$
\sum_{j \geq 2}\|h_{T_j}\|_{\ell_2} \leq s^{-1/2}\sum_{j \geq 1}\|h_{T_j}\|_{\ell_1}= s^{-1/2}\|h_{T_0^c}\|_{\ell_1}
$$
From this, we immediately obtain
\begin{equation}\label{eq:small-h3}
\|h_{(T_0\cup T_1)^c}\|_{\ell_2} = \left\|\sum_{j \geq 2}h_{T_j}\right\|_{\ell_2} \leq \sum_{j \geq 2}\|h_{T_j}\|_{\ell_2} \leq s^{-1/2}\|h_{T_0^c}\|_{\ell_1}.
\end{equation}

By the Cauchy-Schwarz inequality,
\begin{equation}\label{eq:lem1-cs}
\|h_{T_0}\|_{\ell_1} \leq s^{1/2}\|h_{T_0}\|_{\ell_2}.
\end{equation}
Applying \eqref{eq:small-h3}, \eqref{eq:small-h1}, and \eqref{eq:lem1-cs} proves \eqref{eq:small-h2}.
$$
\|h_{(T_0\cup T_1)^c}\|_{\ell_2} \leq s^{-1/2}\|h_{T_0^c}\|_{\ell_1} \leq s^{-1/2}(\|h_{T_0}\|_{\ell_1} + 2\|x_{T_0^c}\|_{\ell_1}) \leq \|h_{T_0}\|_{\ell_2} + 2s^{-1/2}\|x_{T_0^c}\|_{\ell_1}
$$
\end{proof}

Next, we want to bound the main part of the error term $\|h_{T_0\cup T_1}\|_{\ell_2}$. We begin with a lemma.

\begin{lemma}\label{lem:innner-control}
Suppose $x$ and $x'$ are $s$-sparse and $s'$-sparse respectively and supported on disjoint sets. Then
$$
|\langle Ax, Ax'\rangle| \leq \delta_{s + s'}\|x\|_{\ell_2}\|x'\|_{\ell_2}
$$
\end{lemma}
\begin{proof}
We may assume without a loss of generality that $\|x\|_{\ell_2} = \|x'\|_{\ell_2} = 1$. The RIP tells us that
$$
2(1 - \delta_{s + s'}) = (1 - \delta_{s + s'})\|x + x'\|_{\ell_2}^2 \leq \|Ax \pm Ax'\|_{\ell_2}^2 \leq (1 + \delta_{s + s'})\|x + x'\|_{\ell_2}^2 = 2(1 - \delta_{s+s'}).
$$
By the polarization identity,
$$
|\langle Ax, Ax'\rangle| \leq \frac{1}4\left|\|Ax + Ax'\|_{\ell_2}^2 - \|Ax - Ax'\|_{\ell_2}^2\right| \leq \delta_{s + s'}.
$$
\end{proof}

\begin{lemma}[Main term estimate]\label{lem:main-estimate}
\begin{equation}\label{eq:main-term-est}
\|h_{T_0\cup T_1}\|_{\ell_2} \leq (1 - \rho)^{-1}(\alpha\epsilon + 2\rho s^{-1/2}\|x - x_s\|_{\ell_1})
\end{equation}
where
$$
\alpha \equiv \frac{2\sqrt{1 - \delta_{2s}}}{1 - \delta_{2s}},\quad \rho \equiv \frac{\sqrt{2}\delta_{2s}}{1 - \delta_{2s}}.
$$
\end{lemma}
\begin{proof}
RIP allows us to control the size of $\|h_{T_0\cup T_1}\|_{\ell_2}$ with $\|Ah_{T_0\cup T_1}\|_{\ell_2}$, and so we start by bounding the latter. We break $\|Ah_{T_0\cup T_1}\|_{\ell_2}$ into parts using properties of the inner product.
\begin{equation}\label{eq:main-bound}
\|Ah_{T_0\cup T_1}\|_{\ell_2}^2 = \langle A h_{T_0\cup T_1}, Ah \rangle - \sum_{j \geq 2}\left(\langle A h_{T_0}, Ah_{T_j}\rangle + \langle Ah_{T_1}, Ah_{T_j}\rangle\right)
\end{equation}
From the triangle inequality and hypothesis,
\begin{equation}\label{eq:Ah-bound}
\|Ah\|_{\ell_2} = \|A(x - x^*)\|_{\ell_2} \leq \|Ax^* - y\|_{\ell_2} + \|y - Ax\|_{\ell_2} \leq 2\epsilon.
\end{equation}
The first term of \eqref{eq:main-bound} can be bounded using Cauchy-Schwarz, the RIP, and \eqref{eq:Ah-bound}.
\begin{equation}\label{eq:first-term-est}
|\langle Ah_{T_0\cup T_1}, Ah\rangle| \leq \|Ah_{T_0\cup T_1}\|_{\ell_2}\|Ah\|_{\ell_2} \leq 2\epsilon \sqrt{1 + \delta_{2s}}\|h_{T_0\cup T_1}\|_{\ell_2}
\end{equation}
Since $T_0$ and $T_1$ are disjoint Cauchy-Schwarz gives
$$
\|h_{T_0}\|_{\ell_2} + \|h_{T_1}\|_{\ell_2} \leq \sqrt{2}\|h_{T_0\cup T_1}\|_{\ell_2}.
$$
To estimate the sum term of \eqref{eq:Ah-bound}, we apply lemma~\ref{lem:innner-control} and \eqref{eq:tail-sum}.
\begin{align}
\sum_{j \geq 2}|\langle Ah_{T_0}, Ah_{T_j}\rangle| + |\langle Ah_{T_1}, Th_{T_j}\rangle| &\leq \delta_{2s}(\|h_{T_0}\|_{\ell_2} + \|h_{T_1}\|_{\ell_2})\sum_{j \geq 2}\|h_{T_j}\|_{\ell_2}
\nonumber\\&\leq \delta_{2s}\sqrt{2}s^{-1/2}\|h_{T_0\cup T_1}\|_{\ell_2}\|h_{T_0^c}\|_{\ell_1} \label{eq:second-term-est}
\end{align}
We now have
$$
(1 - \delta_{2s})\|h_{T_0\cup T_1}\|_{\ell_2}^2 \leq \|Ah_{T_0\cup T_1}\|_{\ell_2}^2\leq \|h_{T_0\cup T_1}\|_{\ell_2}(2\epsilon\sqrt{1 +\delta_{2s}} + \sqrt{2}\delta_{2s}s^{-1/2}\|h_{T_0^c}\|_{\ell_1})
$$
from RIP, \eqref{eq:first-term-est}, and \eqref{eq:second-term-est}. We divide by $(1 - \delta_{2s})\|h_{T_0\cup T_1}\|_{\ell_2}$.
$$
\|h_{T_0\cup T_1}\|_{\ell_2} \leq \alpha\epsilon + \rho s^{-1/2}\|h_{T_0^c}\|_{\ell_1}
$$
From \eqref{eq:small-h1} and \eqref{eq:lem1-cs}, we obtain
$$
\|h_{T_0 \cup T_1}\|_{\ell_2} \leq \alpha\epsilon + \rho s^{-1/2}(\|h_{T_0}\|_{\ell_1} + 2\|x - x_s\|_{\ell_1}) \leq \alpha\epsilon + \rho \|h_{T_0\cup T_1}\|_{\ell_2} + 2\rho s^{-1/2}\|x - x_s\|_{\ell_1}.
$$
By the hypothesis of theorem~\ref{thm:noisy-recovery}, $\rho < 1$, and we get
$$
\|h_{T_0\cup T_1}\|_{\ell_2} \leq (1 - \rho)^{-1}(\alpha\epsilon + 2\rho s^{-1/2}\|x - x_s\|_{\ell_1}),
$$
which completes the lemma's proof.
\end{proof}

Applying our estimates \eqref{eq:small-h2} and lemma~\ref{lem:main-estimate} to the two terms of \eqref{eq:main-triang}, we have
$$
\|h\|_{\ell_2} \leq 2\|h_{(T_0\cup T_1)}\|_{\ell_2} + 2s^{-1/2}\|x - x_s\|_{\ell_1} \leq 2(1 - \rho)^{-1}(\alpha\epsilon + (1 + \rho)s^{-1/2}\|x - x_s\|_{\ell_1}).
$$
This completes the proof of theorem~\ref{thm:noisy-recovery}.

\section{Remarks}
A huge amount of research in compressed sensing has appeared since the publication of the original papers \cite{ctr-stable,donoho-cs} around 2004-2006. Among other things, researchers have investigated techniques for sparse recovery besides basis pursuit e.g.\ matching pursuit \cite{tg-matching}. There is also replacement condition for RIP called the nullspace property, which is a necessary and sufficient condition for \eqref{eq:l0-prog} and \eqref{eq:l1-prog} to have the same solutions \cite{nullspace}.

As far as applications are concerned, compressed sensing is being used in areas as diverse as tomography, astronomy, machine linearing, linear coding, and experiment design \cite{cs-science,dantzig-selector}. It is particularly useful in situations where minimizing the work done in sensors is important such as space probes. Gimmicks like single-pixel cameras \cite{single-pixel} have captured the public imagination.

Finally, for readers still curious, there are a lot of other (possibly more palatable) introductions to compressed sensing \cite{cw-intro,cs-book,foucart-intro,users-guide,kutyniok-cs,qaisar-compressive,romberg-intro}.

\bibliographystyle{amsplain}
\bibliography{compressed_sensing}

\end{document}


\section{Algorithms}

In this section we will describe the randomized algorithms in detail, provide the
corresponding computational complexity analysis, and state the main theoretical results
that guarantee the accuracy of the approximation.

We split this section in three main parts: Stage 1 \ref{sec:stage1} and Stage 2
\ref{sec:stage2} that will study in depth the algorithmic details of both stages,
and Full Algorithms \ref{sec:full-algos} in which full algorithms will be presented
for a diversity of matrices with certain properties.

\subsection{Stage 1}
\label{sec:stage1}
In the introductory section, we provided some intuition on the randomized procedure
and we developed a general Proto-Algorithm \ref{alg:proto-algorithm} to find
the range-approximating matrix $\mtx{Q}$. However, Proto-Algorithm \ref{alg:proto-algorithm} is very general and can be
tunned depending on the problem requirements.
The number $T_{\rm basic}$ of flops required by Proto-Algorithm \ref{alg:proto-algorithm} satisfies
\begin{equation}
\label{eq:cost_basic}
T_{\rm basic} \sim \ell n \, T_{\rm rand} + \ell\,T_{\rm mult} + \ell^{2}m %\,T_{\rm flop},
\end{equation}
where $T_{\rm rand}$ is the cost of generating a Gaussian random number
and $T_{\rm mult}$ is the cost of multiplying $\mtx{A}$ by a vector.
The last term comes from the orthonormalization procedure of $\mtx{Y}$.

We will now describe some specific realizations of Proto-Algorithm \ref{alg:proto-algorithm}
that will be intended for problems with different requirements.

%%%%%%%%%%%%%%%%%% RANDOMIZED RANGE FINDER %%%%%%%%%%%%%%%%%%%%%%%%

\subsubsection{Randomized Range Finder} \label{alg:randomized-range-finder}
This is the most naive and simplest implementation of Proto-Algorithm \ref{alg:proto-algorithm}.
Given an oversampling parameter $p$, the \textit{Randomized Range Finder}
performs Proto-Algorithm \ref{alg:proto-algorithm} with a gaussian test matrix
$\mtx{\Omega}\in\Rspace{n\times\ell}$ with $\ell=k+p$ and $k$ being a pre-specified
target rank. Then, it orthogonalizes the rows of the resulting matrix $\mtx{Y}$ by computing
a QR decomposition. A numerical issue arises when computing the orthogonalization
procedure due to the fact that the columns of $\mtx{Y}$ are almost linearly
dependent. The authors in ~\cite{halko2011finding} found that
using the \textit{double orthogonalization}
~\cite{bjorck1994numerics} was enough to guarantee stability
of the procedure.

\begin{figure}[ht]
\begin{center}
\fbox{
\begin{minipage}{.9\textwidth}
\begin{center}
\textsc{Randomized Range Finder}
\end{center}
\begin{tabbing}
\hspace{5mm} \= \hspace{5mm} \= \hspace{5mm} \= \hspace{5mm} \= \kill
\anum{1} \>Draw an $n\times \ell$ Gaussian random matrix $\mtx{\Omega}$.\\
\anum{2} \>Form the $m\times \ell$ matrix $\mtx{Y} = \mtx{A}\mtx{\Omega}$.\\
\anum{3} \>Construct an $m \times \ell$ matrix $\mtx{Q}$ whose columns form an orthonormal\\
         \> basis for the range of $\mtx{Y}$, e.g., using the QR factorization $\mtx{Y} = \mtx{Q}\mtx{R}$.
\end{tabbing}
\end{minipage}}
\end{center}
\end{figure}

The complexity analysis of the Algorithm \ref{alg:randomized-range-finder}
gives

\begin{equation}\label{eq:analysis-rand-finder}
T_{\text{Randomized Range Finder}}\sim \mathcal{O}(mn\ell)
\end{equation}

this is because generating a gaussian random number is $\mathcal{O}(1)$
and computing a matrix vector multiplication is $\mathcal{O}(mn)$
\footnote{Note that parallel computation can be extremely helpful here
to drastically reduce the effective complexity of the product $\mtx{A}\mtx{\Omega}$.}.
One important observation is that the complexity $\mathcal{O}(mn\ell)$
can be prohibitive for massive datasets \footnote{Obviously, we can't hope
to do better than $\mathcal{O}(mn)$ which is the size of the matrix
we want to approximate. The term that we want to address is the linear term
in $\ell$. Classical algorithms to compute the $\ell$-SVD cost
$\mathcal{O}(mn\ell)$, by first computing s rank-revealing QR factorization
~\cite{gu1996efficient} and then manipulating the terms to get to the desired
factorization.}. A variant of this procedure
will be studied in \ref{alg:fast-randomized-range-finder}
to get around this problem,

%%%%%%%%%%%%%%%%%% ADAPTIVE RANDOMIZED RANGE FINDER %%%%%%%%%%%%%%%%%%%%%%%%

\subsubsection{Adaptive Randomized Range Finder}
\label{alg:adaptive-randomized-range-finder}

One important pitfall of the \textit{Randomized Range Finder}
\ref{alg:randomized-range-finder} is that it
requires to know in advance the target rank $k$. However, if we intend
to solve the \textit{fixed-precision approximation problem}, we need a scheme to estimate
the error $\|\mtx{A}-\mtx{Q}\mtx{Q}^\adj\mtx{A}\|$ during the algorithm
in order to match the requited tolerance $\epsilon$.

This scheme is possible and it is direct consequence of the following lemma.
\begin{lemma}
\label{thm:aposteriori}
Let $\mtx{B}$ be a real $m\times n$ matrix.
Fix a positive integer $r$ and a real number $\alpha > 1$.
Draw an independent family $\{ \vct{\omega}^{(i)} : i = 1, 2, \dots, r \}$
of standard Gaussian vectors.  Then
\begin{equation*}
\norm{\mtx{B}}
    \leq \alpha \sqrt{\frac{2}{\pi}} \max_{i = 1, \dots, r}
    \smnorm{}{\mtx{B}\vct{\omega}^{(i)} }
\end{equation*}
except with probability $\alpha^{-r}$.
\end{lemma}

Lemma \ref{thm:aposteriori} says that we can bound the error with high
probability using inexpensive computations in an online manner.
The Lemma \ref{thm:aposteriori} applied to our problem reads

\begin{equation}
\label{eq:errorest}
\norm{ (\Id - \mtx{Q}\mtx{Q}^{\adj})\mtx{A}}
    \leq 10 \sqrt{\frac{2}{\pi}} \max_{i = 1, \dots, r}
    \smnorm{}{ (\Id - \mtx{Q}\mtx{Q}^{\adj}) \mtx{A}\vct{\omega}^{(i)} }
\end{equation}
with probability at least $1 - 10^{-r}$.

The high probability bound \ref{eq:errorest} gives a simple online scheme
to decide when we have a good enough $\mtx{Q}$ that matches the pre-specified
tolerance. The goal here is to find an integer $l$ and a $m\times l$
orthonormal matrix $\mtx{Q}^{(l)}$ such that
\begin{equation} \label{eqn:err_est_err_bd}
\smnorm{}{ \big(\Id - \mtx{Q}^{(\ell)} (\mtx{Q}^{(\ell)})^{\adj} \big)\mtx{A} } \leq \eps.
\end{equation}

We call \textit{Adaptive Randomized Range Finder}
\ref{alg:adaptive-randomized-range-finder} to the
algorithm derived from Lemma \ref{thm:aposteriori} that solves this
problem.

\begin{figure}[ht]
\begin{center}
\fbox{
\begin{minipage}{.9\textwidth}
\begin{center}
\textsc{Adaptive Randomized Range Finder}
\end{center}
\begin{tabbing}
\hspace{5mm} \= \hspace{5mm} \= \hspace{5mm} \= \hspace{5mm} \= \kill
\anum   {1} \> Draw standard Gaussian vectors $\vct{\omega}^{(1)}, \dots, \vct{\omega}^{(r)}$ of length $n$.\\
\anum{2} \> For $i = 1,2,\dots,r$, compute $\vct{y}^{(i)} = \mtx{A}\vct{\omega}^{(i)}$.\\
\anum{3} \> $j=0$.\\
\anum{4} \> $\mtx{Q}^{(0)} = [\ ]$, the $m\times 0$ empty matrix. \\
\anum{5} \> \textbf{while} $\displaystyle
         \max\left\{\smnorm{}{\vct{y}^{(j+1)}},\smnorm{}{\vct{y}^{(j+2)}},\dots,\smnorm{}{\vct{y}^{(j+r)}} \right\} >
\varepsilon/(10\sqrt{2/\pi})$,\\
\anum{6} \> \> $j = j + 1$.\\
\anum{7} \> \> Overwrite $\vct{y}^{(j)}$ by $\bigl(\Id - \mtx{Q}^{(j-1)}(\mtx{Q}^{(j-1)})^{\adj}\bigr)\vct{y}^{(j)}$.\\
\anum{8} \> \> $\vct{q}^{(j)} = \vct{y}^{(j)}/\norm{\vct{y}^{(j)}}$.\\
\anum{9} \> \> $\mtx{Q}^{(j)} = [\mtx{Q}^{(j-1)}\ \vct{q}^{(j)}]$.\\
\anum{10} \> \> Draw a standard Gaussian vector $\vct{\omega}^{(j+r)}$ of length $n$.\\
\anum{11} \> \> $\vct{y}^{(j+r)} = \left(\Id - \mtx{Q}^{(j)}(\mtx{Q}^{(j)})^{\adj}\right)\mtx{A}\vct{\omega}^{(j+r)}$.\\
\anum{12} \> \> \textbf{for} $i = (j+1),(j+2),\dots,(j+r-1)$,\\
\anum{13} \> \> \> Overwrite $\vct{y}^{(i)}$ by $\vct{y}^{(i)} - \vct{q}^{(j)}\ip{\vct{q}^{(j)}}{\vct{y}^{(i)}}$.\\
\anum{14} \> \> \textbf{end for}\\
\anum{15} \> \textbf{end while}\\
\anum{16} \> $\mtx{Q} = \mtx{Q}^{(j)}$.
\end{tabbing}
\end{minipage}}
\end{center}
\end{figure}

One important question regarding Algorithm \ref{alg:adaptive-randomized-range-finder}
 is how good the bound given by Lemma \ref{thm:aposteriori}
is in practice. If there is a significant gap between theory and practice
the optimal $l$ will be overestimated. This question will be addressed in the experimental section
\ref{sec:gaussian-matrices}.

%%%%%%%%%%%%%%%%%% RANDOMIZED POWER ITERATION %%%%%%%%%%%%%%%%%%%%%%%%

\subsubsection{Randomized Power Iteration}
\label{alg:randomized-power-iteration}

The \textit{Randomized Range Finder} \ref{alg:randomized-range-finder}
algorithm assumes that the singular values
of the matrix decay fast. This can be intuitively seen from equation \ref{eq:iter-range},
where the small singular
values interfere with the calculation of the range. This intuition is made precise
in Theorem \ref{thm:avg-frob-error-gauss}, where the error of the approximation depends on
the $\sigma_{k+1}$ and $\sum_{j>k}\sigma_j^2$.

The goal here is to reduce the weight of the
small singular values by taking powers of the matrix whose range we want
to approximate. Instead of applying the sampling scheme to $\mtx{A}$, we will
apply it to $\mtx{B}=(\mtx{A}\mtx{A}^\adj)^q\mtx{A}$ where $q>0$ is a small
integer.

The matrix $\mtx{B}$ has the same singular vectors than $\mtx{A}$ (hence,
the same range), but its singular values decay much more quickly.

\begin{equation}\label{eq:sing-values-power}
\sigma_j(\mtx{B}) = \sigma_j(\mtx{A})^{2q+1},
\hspace{0.5cm} j=1,2,3,\ldots
\end{equation}

The \textit{Randomized Power Iteration}
\ref{alg:randomized-power-iteration} algorithm is the same
as the \textit{Randomized Range Finder} \ref{alg:randomized-range-finder}
 but replacing the formula
$\mtx{Y}=\mtx{A}\mtx{\Omega}$ by $\mtx{Y}=\mtx{B}\mtx{\Omega}$.

\begin{figure}[ht]
\begin{center}
\fbox{
\begin{minipage}{.9\textwidth}
\begin{center}
\textsc{Randomized Power Iteration}
\end{center}
\begin{tabbing}
\hspace{5mm} \= \hspace{5mm} \= \hspace{5mm} \= \hspace{5mm} \= \kill
\anum{1} \>Draw an $n\times \ell$ Gaussian random matrix $\mtx{\Omega}$.\\
\anum{2} \>Form the $m\times \ell$ matrix $\mtx{Y} = (\mtx{A}\mtx{A}^{\adj})^{q}\mtx{A}\mtx{\Omega}$ via alternating application\\
         \>of $\mtx{A}$ and $\mtx{A}^{\adj}$.\\
\anum{3} \>Construct an $m \times \ell$ matrix $\mtx{Q}$ whose columns form an orthonormal\\
         \> basis for the range of $\mtx{Y}$, e.g., via the QR factorization $\mtx{Y} = \mtx{Q}\mtx{R}$.
\end{tabbing}
\end{minipage}}
\end{center}
\end{figure}

The computational complexity of the Algorithm \ref{alg:randomized-power-iteration}
is essentially the same because
it only requires $2q+1$ as many matrix-multiplications as Algorithm
\ref{alg:randomized-range-finder}
but the number $q$ is in practice 2,3 or 4. This can be seen from
Corollary \ref{cor:power-method-spec-gauss},
which shows that the power iteration drives the approximation
gap to 1 exponentially fast as $q$ increases.

%%%%%%%%%%%%%%%%%% FAST RANDOMIZED RANGE FINDER %%%%%%%%%%%%%%%%%%%%%%%%

\subsubsection{Fast Randomized Range Finder}
\label{alg:fast-randomized-range-finder}

A simple inspection to equation \ref{eq:cost_basic} reveals the computational
bottleneck of the sampling procedure. This is the matrix multiplication
$\mtx{Y}=\mtx{A}\mtx{\Omega}$ that takes $\mathcal{O}(mn\ell)$ operations
for dense matrices, which is the same as the $\ell$-SVD.

The key idea is to use a \textit{structured} random matrix that allows us
to compute the product in $\mathcal{O}(mn\log(\ell))$ operations.

The simplest structured random matrix that meets our goals is the so-called
\textit{subsampled random Fourier transform} (SRFT).

An SRFT is an $n \times \ell$ matrix of the form
\begin{equation}
\label{eq:def_srft}
\mtx{\Omega} = \sqrt{\frac{n}{\ell}} \, \mtx{DFR},
\end{equation}
where
\lsp
\begin{itemize}
\item   $\mtx{D}$ is an $n \times n$ diagonal matrix whose entries are
independent random variables uniformly distributed on the complex unit circle.

\item   $\mtx{F}$ is the $n \times n$ unitary discrete Fourier transform (DFT),
whose entries take the values $f_{pq} = n^{-1/2} \, \econst^{-2\pi\iunit (p-1)(q-1)/n}$ for $p, q = 1, 2, \dots, n$

\item   $\mtx{R}$ is an $n \times \ell$ matrix that samples $\ell$ coordinates
from $n$ uniformly at random, i.e., its $\ell$ columns are drawn randomly
without replacement from the columns of the $n \times n$ identity matrix.
\end{itemize}
\lsp

Now, via a subsampled FFT ~\cite{woolfe2008fast}, we can compute the
sample matrix $\mtx{Y}=\mtx{A}\mtx{\Omega}$ with
$\mathcal{O}(mn\log(\ell))$ operations.

The total number of operations required by this procedure is reduced to
\begin{equation}
\label{eq:cost_SRFT}
T_{\rm struct} \sim mn \log(\ell) + \ell^2 n
\end{equation}

Hence, the computational complexity of the approach is essentially
$\mathcal{O}(mn \log(\ell))$.

% If $\ell$ is substantially larger than the numerical rank $r(\mtx{A})$
% \ref{eq:num-rank}, then the orthogonalization can be done in
% $\mathcal{O}(k\ell n)$ instead of $\mathcal{O}(\ell^2n)$.

\begin{figure}[ht]
\begin{center}
\fbox{
\begin{minipage}{.9\textwidth}
\begin{center}
\textsc{Fast Randomized Range Finder}
\end{center}
\begin{tabbing}
\hspace{5mm} \= \hspace{5mm} \= \hspace{5mm} \= \hspace{5mm} \= \kill
\anum{1} \>Draw an $n\times \ell$ SRFT test matrix $\mtx{\Omega}$, as defined by \eqref{eq:def_srft}. \\
%\pgnotate{Added reference.}\\
\anum{2} \>Form the $m\times \ell$ matrix $\mtx{Y} = \mtx{A}\mtx{\Omega}$ using a (subsampled) FFT.\\
\anum{3} \>Construct an $m \times \ell$ matrix $\mtx{Q}$ whose columns form an orthonormal\\
         \> basis for the range of $\mtx{Y}$, e.g., using the QR factorization $\mtx{Y} = \mtx{Q}\mtx{R}$.
\end{tabbing}
\end{minipage}}
\end{center}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% STAGE 2 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Stage 2} \label{sec:stage2}
 The output of the Stage 1 \ref{sec:stage1} produces an orthonormal matrix $\mtx{Q}$ whose
 range captures the action of the matrix $\mtx{A}$. The goal of Stage 2
 \ref{sec:stage2} is
 to produce standard approximate matrix factorizations of $\mtx{A}$
 using this $\mtx{Q}$.

This subsection is divided into three parts; first, we will show how to compute
standard approximate matrix factorizations (SVD and QR) from a general
approximate low-rank factorization. Recall that taking $\mtx{B}=\mtx{Q}$
and $\mtx{C}=\mtx{Q}^\adj\mtx{A}$ we readily have a factorization that
satisfies $\|\mtx{A}-\mtx{B}\mtx{C}\|\leq\varepsilon$.
Then, we will describe in detail the \textit{Direct SVD} algorithm, which will
consist in constructing an SVD from $\mtx{B}$ and $\mtx{C}$. Finally, we will
comment on other more involved methods that avoid computing the expensive
product $\mtx{C}=\mtx{Q}^\adj\mtx{A}$.

\subsubsection{Compute standard QR and SVD from a general factorization}
Now we will specify how we can compute the standards SVD and QR decompositions
from a general low rank decomposition $\|\mtx{A} - \mtx{B}\mtx{C}\|\leq\varepsilon$
maintaining the tolerance $\epsilon$ from Stage 1 \ref{sec:stage1}.

\begin{itemize}
  \item \label{itm:SVD-from-C} \textit{SVD decomposition:} $\|\mtx{A}-\mtx{U}\mtx{\Sigma}\mtx{V}^\adj\|\leq\varepsilon$
\lsp
\begin{enumerate}
\item Compute a QR factorization of $\mtx{B}$ so that $\mtx{B} = \mtx{Q}_{1}\mtx{R}_{1}$.
\item Form the product $\mtx{D} = \mtx{R}_{1}\mtx{C}$, and compute an SVD:
      $\mtx{D} = \mtx{U}_{2}\mtx{\Sigma}\mtx{V}^{\adj}$.
\item Form the product $\mtx{U} = \mtx{Q}_{1}\mtx{U}_{2}$.
\end{enumerate}
\lsp
% The result is a diagonal matrix $\mtx{\Sigma}$ and orthonormal matrices $\mtx{U}$ and $\mtx{V}$
% such that $\norm{\mtx{A} - \mtx{U}\mtx{\Sigma}\mtx{V}^{\adj}} \leq \varepsilon$.

\item \label{itm:QR-from-C} \textit{QR decomposition:} $\|\mtx{A}-\mtx{Q}\mtx{R}\|\leq\varepsilon$
\lsp
\begin{enumerate}
\item Compute a QR factorization of $\mtx{B}$ so that $\mtx{B} = \mtx{Q}_{1}\mtx{R}_{1}$.
\item Form the product $\mtx{D} = \mtx{R}_{1}\mtx{C}$, and compute a QR factorization: $\mtx{D} = \mtx{Q}_{2}\mtx{R}$.
\item Form the product $\mtx{Q} = \mtx{Q}_{1}\mtx{Q}_{2}$.
\end{enumerate}
% The result is an orthonormal matrix  $\mtx{Q}$ and a weakly upper-triangular matrix $\mtx{R}$ such
% that $\norm{\mtx{A} - \mtx{Q}\mtx{R}} \leq \varepsilon$.

\end{itemize}

\subsubsection{Direct SVD} \label{alg:direct-svd}

The procedure described in \ref{itm:SVD-from-C} to compute the approximate
SVD decomposition without sacrificing error, defines what we call
the \textit{Direct SVD} \ref{alg:direct-svd}.

\begin{figure}[ht]
\begin{center}
\fbox{\begin{minipage}{.9\textwidth}
\begin{center}
\textsc{Direct SVD}
\end{center}
\anum{1}    Form the matrix
$
\mtx{B} = \mtx{Q}^{\adj}\mtx{A}.
%\mtx{B} = \bigl(\mtx{A}^{\adj}\mtx{Q}\bigr)^{\adj}.
$

\anum{2}    Compute an SVD of the small matrix:
$
\mtx{B} = \widetilde{\mtx{U}}\mtx{\Sigma}\mtx{V}^{\adj}.
$

\noindent
\anum{3}    Form the orthonormal matrix
$
\mtx{U} = \mtx{Q}\widetilde{\mtx{U}}.
$
\end{minipage}}
\end{center}
%\caption{trash}
%\label{alg:Atranspose}
\end{figure}

Although using \textit{Direct SVD} \ref{alg:direct-svd} algorithm for
Stage 2 \ref{sec:stage2} does not incur additional errors, the computation
of $\mtx{C}=\mtx{Q}^\adj\mtx{A}$ is in general too expensive for dense matrices.

More concretely, the product costs $\mathcal{O}(mn\ell)$, even more
expensive that the cost of Stage 1 \ref{sec:stage1} when using
the accelerated Algorithm
\ref{alg:fast-randomized-range-finder}.

\subsubsection{Different Procedures} \label{sec:dif-app}

\begin{itemize}
    \item \textit{Match the cost} $\mathcal{O}(mn\log(\ell))$ \textit{of accelerated
    Stage 1} \ref{alg:fast-randomized-range-finder}.

    In order to match the complexity $\mathcal{O}(mn\log(\ell))$ from
Stage 1 \ref{sec:stage1} we must avoid the product $\mtx{Q}^\adj\mtx{A}$.

In \cite{halko2011finding}, the authors propose algorithms based
on row extraction of $\mtx{Q}$ via its \textit{Interpolative Decomposition}
$\mtx{Q}=\mtx{X}\mtx{Q}_{(J,:)}$ \cite{cheng2005compression}.
Now, $\mtx{Q}_{(J,:)}$ is a $k\times k$ matrix. The proposed
algorithm takes $\mtx{Q}$ as input and constructs a rank-$k$ matrix
factorization

\begin{equation}\label{eq:via-row-extraction}
\mtx{A}\approx\mtx{X}\mtx{B}
\end{equation}
where $\mtx{B}$ is a $k \times n$ matrix consisting of $k$ rows
extracted from $\mtx{A}$.

The key here is that \ref{eq:via-row-extraction} can be produced without any large
matrix-matrix multiplication resulting in a total of $\mathcal{O}(k^2(m+n))$
operations. The drawback is that the initial error is larger than the one
incurred by $\mtx{Q}^\adj\mtx{Q}\mtx{A}$
\footnote{In \cite{halko2011finding}, they prove that the error is increased at
most by $\left(\sqrt{1+4k(n-k)}\right)\varepsilon$, which can be large
when $\varepsilon$ is not that small or $kn$ is large.}.

\item \textit{Single-Pass algorithms}

The previously described algorithms require revisit the matrix $\mtx{A}$
multiple times. In \cite{halko2011finding} they propose single-pass
algorithms relying on the observation that all the information
you need to compute the decomposition is in the matrices $\mtx{\Omega},\mtx{Y}$
and $\mtx{Q}$. They call the Algorithm \textit{Eigenvalue Decomposition in
One-pass}. However, this algorithm too adds additional error to the Stage 1
\footnote{Authors in \cite{halko2011finding} argue that this issue can be
addressed with extra oversampling.}.
\end{itemize}

\subsection{Full Algorithms}\label{sec:full-algos}
Let's now propose a full algorithm depending on the input matrix properties.

\begin{itemize}
\item \textit{General Matrices That Fit in Core Memory}

In this case, the appropiate method for Stage 1 is the accelerated
Algorithm \ref{alg:fast-randomized-range-finder} that uses structured random
matrices. For Stage 2, use the \textit{row-extraction technique}
described in \ref{sec:dif-app}. The overall cost $T_{\rm random}$ reduces to

\begin{equation} \label{eq:cost-1}
\boxed{T_{\rm random} \sim mn \log(k) + k^2 (m+n)}
\end{equation}
The overall approximation error satisfies
\footnote{This can be derived by combining the error analysis
of range approximation with SRFT (which can be found in \cite{halko2011finding})
and the additional error incurred by the \textit{row-extraction technique}.}
\begin{equation} \label{eq:totalerror-1}
\norm{ \mtx{A} - \mtx{U\Sigma V}^\adj }
    \lesssim n \cdot \sigma_{k+1},
\end{equation}

\item \textit{Matrices for which Matrix-Vector Products are Cheap.}

In this set of matrices we include for example sparse and structured matrices for
which the matrix-vector multiplication is substantially cheaper than the general case.

The appropiate method for Stage 1 is Algorithm \ref{alg:randomized-range-finder}
with $p$ constant \footnote{If we are dealing
with the \textit{fixed-precision problem}, then we use
Algorithm \ref{alg:adaptive-randomized-range-finder}.}, or more generally
Algorithm \ref{alg:randomized-power-iteration} with $q>0$. For Stage 2, we apply
the \textit{Direct SVD} \ref{alg:direct-svd}. The total cost $T_{\rm sparse}$ satisfies
\begin{equation}
\label{eq:sparsealg-cost}
\boxed{T_{\rm sparse} = (2q+2)\,(k + p) \, T_{\rm mult} + \bigO(k^2 (m + n))}
\end{equation}

The overall approximation error satisfies (see Corollary \ref{cor:power-method-spec-gauss}).
\begin{equation}
\label{eq:sparsealg-error2}
\norm{ \mtx{A} - \mtx{U\Sigma V}^\adj } \lesssim (kn)^{1/2(2q+1)} \cdot \sigma_{k+1}.
\end{equation}

\end{itemize}

\newpage


\section{Comparison with empirical portfolio returns}
\label{sec:comparison_returns}

With the definition of the four different distributions we proceed to compare
them with empirical data from 200 stocks traded in the S\&P 500 stock market
index.

As described in Sect. \ref{sec:exact_distributions}, we obtain normalized all
the time series in the market data, the we compute the covariance matrix and
obtain the corresponding eigenvalues and eigenvectors. Using them we rotate and
scale the returns, and finally, we aggregate the rotated and scaled returns
into a single univariate distribution.

In Fig. \ref{fig:gg_dist} we plot four Gaussian-Gaussian distributions with
fit parameter $N = 2, 3, 4, 5$. The best fit seems to be the distributions
with $N = 3$ and $N = 4$.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.7\columnwidth]
    {figures/08_gg.png}
    \caption{Gaussian-Gaussian probability density
             $\left\langle p \right\rangle_{GG'}^{\left(k\right)}$, in the
             Markovian case, normalized to unit standard deviation for
             different values of $N$, and aggregated distribution returns
             $\left(\tilde{r}\right)$ for fixed covariance for $200$ companies
             selected from the S\&P 500 dataset and $\Delta t = 1d$. (left)
             semilog scale and (right) loglog scale.}
    \label{fig:gg_dist}
\end{figure}

For the Gaussian-Algebraic distribution and Algebraic-Gaussian distribution,
we have to add an additional shape parameter $L$ and $l$, respectively. In
Fig. \ref{fig:ga_dist} we plot four Gaussian-Algebraic distributions and in
Fig. \ref{fig:ag_dist} we plot four Algebraic-Gaussian distributions with fit
parameters $N = 2, 3, 4, 5$. In both cases, a good agreement when $N = 4$ can
be seen.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.7\columnwidth]
    {figures/08_ga.png}
    \caption{Gaussian-Algebraic probability density
             $\left\langle p \right\rangle_{GA'}^{\left(k\right)}$, in the
             Markovian case, normalized to unit standard deviation for
             different values of $N$, and aggregated distribution returns
             $\left(\tilde{r}\right)$ for fixed covariance for $200$ companies
             selected from the S\&P 500 dataset and $\Delta t = 1d$. (left)
             semilog scale and (right) loglog scale.}
    \label{fig:ga_dist}
\end{figure}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.7\columnwidth]
    {figures/08_ag.png}
    \caption{Algebraic-Gaussian probability density
             $\left\langle p \right\rangle_{AG'}^{\left(k\right)}$, in the
             Markovian case, normalized to unit standard deviation for
             different values of $N$, and aggregated distribution returns
             $\left(\tilde{r}\right)$ for fixed covariance for $200$ companies
             selected from the S\&P 500 dataset and $\Delta t = 1d$. (left)
             semilog scale and (right) loglog scale.}
    \label{fig:ag_dist}
\end{figure}

Finally, in Fig. \ref{fig:aa_dist} we plot four Algebraic-Algebraic
distributions with fit parameters $N = 2, 3, 4, 5$. In this case we use the
shape parameters $L$ and $l$. Again, a good agreement when $N = 4$ can be seen.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.7\columnwidth]
    {figures/08_aa.png}
    \caption{Algebraic-Algebraic probability density
             $\left\langle p \right\rangle_{AA'}^{\left(k\right)}$, in the
             Markovian case, normalized to unit standard deviation for
             different values of $N$, and aggregated distribution returns
             $\left(\tilde{r}\right)$ for fixed covariance for $200$ companies
             selected from the S\&P 500 dataset and $\Delta t = 1d$. (left)
             semilog scale and (right) loglog scale.}
    \label{fig:aa_dist}
\end{figure}


\section{List processing}

This Add-on contains several primitives which operate on Lists.

\subsection*{fput} \index{\verb+fput+}

{\tt fput} {\it thing list}
\newline\newline
Output a list formed by the concatenation of the first input and the second input.
\begin{verbatimtab}
@> show fput 34.56 ['foo' 345]
[34.56 'foo' 345]
\end{verbatimtab}

\subsection*{gseq} \index{\verb+gseq+}

{\tt gseq} {\it from to (count)}
\newline\newline
Output a list of the numbers from {\it from} to {\it to}. If {\it count} is specified, numbers will be equally spaced and the size of the list will be {\it count}.
\begin{verbatimtab}
@> print gseq 5 10
5 6 7 8 9 10
@> print gseq 1 10 20
1 1.47 1.95 2.42 2.89 3.37 3.84 4.32 4.79 5.26
5.74 6.21 6.68 7.16 7.63 8.11 8.58 9.05 9.53 10
\end{verbatimtab}

\subsection*{lappend} \index{\verb+lappend+}

{\tt lappend} {\it list thing1 thing2 thing3 ...}
\newline\newline
Append the inputs to the list specified as the first input. This primitive mutates the list.
\begin{verbatimtab}
@> make "a [1 2 3 4]
@> lappend :a 5 6 7 8 [9]
@> show :a
[1 2 3 4 5 6 7 8 [9]]
\end{verbatimtab}

\subsection*{lempty} \index{\verb+lempty+}

{\tt lempty} {\it list}
\newline\newline
Output {\it true} if the list is empty, and {\it false} otherwise.
\begin{verbatimtab}
@> print lempty [1 2 3 4]
false
\end{verbatimtab}

\subsection*{lfind} \index{\verb+lfind+}

{\tt lfind} {\it list thing}
\newline\newline
Output the position in the list given as first input, of the first occurrence of the second input. Output {\tt 0} if not found.
\begin{verbatimtab}
@> make "l [23 56 "hello 'this is it !']
@> print lfind :l "hello
3
\end{verbatimtab}

\subsection*{lindex} \index{\verb+lindex+}

{\tt lindex} {\it list index (thing)}
\newline\newline
Output the value at the position {\tt index} in the list, or change the value at this position if a third input is specified. This primitive mutates the list.
\begin{verbatimtab}
@> print lindex [1 2 3 4 5] 3
3
@> make "a [1 2 3 4 5]
@> make "b :a
@> lindex :a 5 99
@> show :b
[1 2 3 4 99]
\end{verbatimtab}

\subsection*{list} \index{\verb+list+}

{\tt list} {\it thing1 thing2 thing3 ...}
\newline\newline
Output a list whose members are in the inputs.
\begin{verbatimtab}
@> make "mylist list 2 5.6 'hello' "foo
@> show :mylist
[2 5.6 'hello' "foo]
\end{verbatimtab}

\subsection*{ljoin} \index{\verb+ljoin+}

{\tt ljoin} {\it list (thing)}
\newline\newline
Output the concatenation of all the elements of the list {\it list} separated by {\it thing} if specified in a string.
\begin{verbatimtab}
@> print ljoin [1 2 3 4 5] ';'
1;2;3;4;5
\end{verbatimtab}

\subsection*{llength} \index{\verb+llength+}

{\tt llength} {\it list}
\newline\newline
Output the length of the list.
\begin{verbatimtab}
@> print llength [2 5 1 76 8]
5
\end{verbatimtab}

\subsection*{lput} \index{\verb+lput+}

{\tt lput} {\it thing list}
\newline\newline
Output a list formed by the concatenation of the second input and the first input.
\begin{verbatimtab}
@> show lput 'end' ['foo' 345]
['foo' 345 'end']
\end{verbatimtab}
\cbstart
\subsection*{lremove} \index{\verb+lremove+}

{\tt lremove} {\it list thing}
\newline\newline
Remove the first occurence of the second input from the first input.
\begin{verbatimtab}
@> lremove :list 4
\end{verbatimtab}
\cbend
\subsection*{lscan} \index{\verb+lscan+}

{\tt lscan} {\it list "var1 "var2 "var3 ...}
\newline\newline
Store the element of the list starting from the first one into each given variable name, and output the rest of the list. If there are less
elements than variables in the list, the variable values will be set to an empty list.
\begin{verbatimtab}
@> make "rest lscan [1 2 3 4 5 6] "a "b "c
@> show :rest
[4 5 6]
@> show :a :b :c
1 2 3
\end{verbatimtab}

\subsection*{lsub} \index{\verb+lsub+}

{\tt lsub} {\it list from (to)}
\newline\newline
Ouput a sublist of the list given as first input, from the index {\it from} to the index {\it to} if given, else to the end of the list.
\begin{verbatimtab}
@> show lsub ["a "b "c "d "e "f] 3 5
["c "d "e]
\end{verbatimtab}

\subsection*{reverse} \index{\verb+reverse+}

{\tt reverse} {\it list}
\newline\newline
Output a list in the reverse order from that given as input.
\begin{verbatimtab}
@> show reverse [1 2 3 4 5]
[5 4 3 2 1]
\end{verbatimtab}

\subsection*{sentence} \index{\verb+sentence+}

{\tt sentence} {\it thing1 thing2 thing3 ...}
\newline\newline
Output a list whose members are in the inputs. Lists in the inputs are flattened.
\begin{verbatimtab}
@> make "mylist sentence 2 5.6 [1 2 3 4]
@> show :mylist
[2 5.6 1 2 3 4]
\end{verbatimtab}


\chapter{Network Theory}
\label{ch:networktheory}

\begin{chapterabstract}
The theory underpinning complex networks is discussed, covering the representation of atomic systems as networks and the relationship of the dual network to ring structure.
The laws which govern the topological properties of physical networks are also introduced, namely Euler's law (mean node degree), \lm's law (node degree distribution) and the \aw{} law (node degree correlations).
\end{chapterabstract}

\section{Network Theory}
\label{sec:networktheory}

The scope of what constitutes a complex network is extremely broad, covering everything from the tangible (\eg{} computational clusters) to the more abstract (\eg{} social interactions). Yet part of the appeal and power of network science is the ability to quantify and relate these highly disparate systems with the same underlying theory.
A network is simply a collection of components termed \textit{nodes} and the connections between them termed \textit{links}, an example of which is given in figure \ref{fig:smallnet}.
There are then two fundamental classes of network based on the nature of the connections.
Networks in which the links between nodes are mutual are termed undirected, whereas those in which the links are one\--way are termed directed \cite{barabasi2016n}.
At the risk of dating this thesis, this is the difference between Facebook (an undirected social network of friends) and Twitter (a directed social network of followers).
All the networks considered in this work are undirected and all the theory assumes this property.

\begin{figure}[ht]
     \centering
      \includegraphics[width=0.45\textwidth]{./figures/methods/small_periodic_net.pdf}
     \caption{Example of a periodic \td{} network where nodes are represented by circles and links as lines. Nodes are coloured similarly according to their degree, whilst periodic images are faded to highlight the central repeating unit.}
     \label{fig:smallnet}
\end{figure}

\subsection{Node Degree and Probability Distributions}
\label{s:theorynodeprob}

A key concept in network science is the the node degree, defined as the number of links associated with each node.
A node with $k$ links is then said simply to have degree $k$, where $k\in\mathbb{N}$.
This is illustrated in figure \ref{fig:smallnet}, which consists of 5\--, 6\-- and 7\-- degree nodes (coloured blue, grey and red respectively).
The occurrence and correlations of nodes of given degrees can then be described by a range of probability distributions.

The probability of a randomly selected node having degree $k$ is given by the node degree distribution, denoted $p_k$.
This is a normalised discrete distribution such that
\begin{equation}
	\label{eq:pknorm}
	\sumk p_k = 1.
\end{equation}
The $n$\th{} moments of this distribution are then given by:
\begin{equation}
	\label{eq:pkmoment}
	\langle k^{n} \rangle = \sumk k^np_k.
\end{equation}
Alternatively, one can also calculate the probability that a randomly selected link has a $k$\--degree node at the end, denoted $q_k$.
This is not the same as the distribution above, as there is greater chance of selecting links which emanate from high degree nodes, in a manner which is proportional to the node degree.
As this distribution is normalised, this leads to the relations:
\begin{align}
	\sumk q_k &= 1, \label{eq:qknorm} \\
	q_k &= \frac{kp_k}{\langle k \rangle} \label{eq:qkpk}.
\end{align}
In addition, one can also evaluate the probability that a randomly chosen link has nodes of degree $j,k$ at either end.
This is the edge joint degree distribution, denoted $e_{jk}$.
Once again this is normalised and satisfies the following relationships:
\begin{align}
	\sumjk e_{jk} &= 1, \label{eq:ejknorm} \\
	\sumk e_{jk} &= q_j, \label{eq:ejkqk} \\
	e_{jk} &= e_{kj} \label{eq:ejkekj},
\end{align}
where the final result arises from reciprocal nature of the links in an undirected network.
As an illustration of these different distributions, examples %for all three probability distributions
are provided below for the network in figure \ref{fig:smallnet}:%, where the adjacent numbers correspond to the node degrees:
\begin{align}
	\mathbf{p} =  \frac{1}{16} \, \begin{blockarray}{*{1}{c} l}
	\begin{block}{[*{1}{c}]>{$\footnotesize}l<{$}}
	4 \: \bigstrut[t]& 5\\
	8 & 6 \\
	4 & 7 \\
	\end{block}
	\end{blockarray},
	\qquad
	\mathbf{q} =  \frac{1}{96} \, \begin{blockarray}{*{1}{c} l}
	\begin{block}{[*{1}{c}]>{$\footnotesize}l<{$}}
	20 \: \bigstrut[t]& 5\\
	48 & 6 \\
	28 & 7 \\
	\end{block}
	\end{blockarray},
	\qquad
	\mathbf{e} = \frac{1}{96}\: \begin{blockarray}{*{3}{c} l}
	\begin{block}{*{3}{>{$\footnotesize}c<{$}} l}
	5 & 6 & 7 \\
	\end{block}
	\begin{block}{[*{3}{c}]>{$\footnotesize}l<{$}}
	2 & 9 & 9 \: \bigstrut[t]& 5\\
	9 & 22 & 17 & 6 \\
	9 & 17 & 2 & 7\\
	\end{block}
	\end{blockarray}.
%	\textbf{p} = \frac{1}{96}\: \begin{blockarray}{*{5}{c} l}
%	\begin{block}{*{5}{>{$\footnotesize}c<{$}} l}
%	3 & 6 & 7 & 8 & 9 \\
%	\end{block}
%	\begin{block}{[*{5}{c}]>{$\footnotesize}l<{$}}
%	0 & 0 & 5 & 10 & 1\: \bigstrut[t]& \:3\\
%	0 & 0 & 1 & 1 & 0 & \:6 \\
%	5 & 1 & 14 & 14 & 1 & \:7\\
%	10 & 1 & 14 & 14 & 1 & \:8\\
%	1 & 0 & 1 & 1 & 0 & \:9\\
%	\end{block}
%	\end{blockarray}
\end{align}

\subsection{Atomic and Ring Networks}
\label{s:atomringnetworks}

To see how network theory relates to atomic materials, consider the amorphous graphene configuration in figure \ref{fig:graphdualgraph}.
In this network the nodes represent carbon atoms and the links sp$^2$ bonds.
The node degree in the atomic network for all nodes is then equal to three, being equivalent to the atomic coordination number (which throughout this thesis will be denoted by $c$).
This is problematic, because whilst there is clear disorder in the system, it is not well captured by the atomic network.
Due to the fact that the local environment around the atoms is identical, when examining the node degree distribution, any information about the glassy structure is lost.
This network is to at least first order indeterminable from a crystalline hexagonal lattice.

\begin{figure}[bt]
     \centering

     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth]{./figures/methods/graph.pdf}
         \caption{Atomic network}
         \label{fig:graphdualgraph}
     \end{subfigure}
     \hfill
	\begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth]{./figures/methods/dual.pdf}
         \caption{Ring network}
         \label{fig:graphdualdual}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth]{./figures/methods/graph_dual.pdf}
         \caption{Dual relationship}
         \label{}
     \end{subfigure}
     \hfill

     \caption{Panel (a) gives an example of a 3\--coordinate periodic atomic network with disordered ring structure. Nodes and links represent atoms and bonds respectively where rings are coloured by size. Panel (b) gives the corresponding ring network where nodes and links represent rings and their adjacencies, where nodes are coloured by degree. Panel (c) shows the dual relationship between the atomic and ring networks, where the node degree in the ring network is equal to the ring size in the atomic network.}
     \label{fig:graphdual}
\end{figure}

Observing figure \ref{fig:graphdualgraph} one can see there is another level of structure in the network, namely that of the ring structure.
A ring is strictly any closed path of sequentially linked nodes in a network, but this thesis will use the term in reference only to the primitive rings \ie{} those which cannot be subdivided into two smaller rings \cite{Yuan2002}.
A ring of size $k$ (or $k$\--ring) is then defined as a ring with $k$ constituent nodes.
It is clear that finding and counting the number of rings of each size, often termed calculating the ring statistics, is a method of quantifying the disorder in the system \cite{Kumar2014}.
The ring statistics can be summarised by the normalised probability distribution, $p_k$.

However, there is a more efficient way of representing and quantifying the ring structure in the system, and that is by constructing the dual network \cite{Aboav1984}.
The dual is generated by placing a node at the centre of each ring and linking the nodes of adjacent (\ie{} edge\--sharing) rings, as in figure \ref{fig:graphdualdual}.
This will be referred to as the ring network.
The ring network is a reciprocal lattice in which the node degree, $k$, is equivalent to the ring size in the atomic network.
Similarly, it consists solely of triangles, reflecting the 3\--coordinate nature of the underlying atomic network.
Hence, the disorder is captured directly in the node properties of the ring network.
These characteristics make the ring network preferable for manipulating and analysing the systems in this thesis.

\section{Topological Laws}
\label{s:topolaws}

There are a number of laws which govern the topological properties of \td{} network\--forming materials.
These laws constrain the ring structure, influencing the network properties in a manner that makes physical networks unique in the field of network science.
These laws act on a number of ``levels'': Euler's law controls the overall mean ring size, \lm's{} law the ring size distribution and the \aw{} law the ring\--ring correlations.

\subsection{Euler's Law}
\label{s:eulerslaw}

Euler's law constrains the mean ring size, $\ki$, in an atomic network,  equivalent to the mean node degree of the ring network.
The atomic networks studied in this work are all \td{}, connected (\ie{} there is a path between any two nodes) and planar (\ie{} they have no overlapping links) and so are subject to Euler's formula which states:
\begin{equation}
	\label{eq:eulerformula}
	N + V - E = \chi,
\end{equation}
where $N$, $V$, $E$ are the number of rings, vertices and edges in the network and $\chi$ in an integer termed the Euler characteristic, which is dependent on the global topology of the system.
Each vertex represents an atom and the number of edges emanating from each vertex is then the coordination number.

For generality, consider an atomic network with atoms of assorted coordination numbers, $c$.
If the proportion of each coordination type is $x_c$, then the mean coordination number is given by $\langle c \rangle = \sum\limits_c cx_c$.
This allows the number of edges to be written in terms of the number of vertices as $E=\frac{V}{2}\langle c \rangle$.
In turn the mean ring size is simply the total number of vertices per ring, allowing for multiple counting, such that $\ki=\frac{V}{N}\langle c \rangle$.
Substituting these two expressions into equation \eqref{eq:eulerformula} leads to the expression:
\begin{equation}
	\label{eq:avdegree}
	\ki = \frac{2\langle c \rangle\left(1-\chi/N\right)}{\langle c \rangle - 2}.
\end{equation}
Hence the average node degree in the ring network (equivalent to the mean ring size of the physical network), is simply related to the average degree of the physical network (\ie{} local coordination environment), the topology of the system and the number of rings.

Although equation \eqref{eq:avdegree} may appear simple, it is a very powerful constraint.
To demonstrate this consider a two\--dimensional lattice with two possible coordination environments $c=3,4$.
The planar case with periodic boundary conditions (mimicking an infinite planar lattice) maps onto the torus with $\chi=0$, and so:
\begin{equation}
	\label{eq:2dplanarcases}
	\ki = \begin{cases}
		6, \quad x_3 = 1 \\
		4, \quad x_4 = 1 \\
		5, \quad x_3 = 2/3,\, x_4 =1/3
	\end{cases}.
\end{equation}
To reiterate in plain terms, this means that if there is a material consisting of atoms all forming exactly three bonds (as for amorphous carbon), the mean ring size \textit{must} be equal to six.
Similarly if all atoms form four bonds the mean ring size is four, and if there is a two\--thirds to one\--third mixture of coordination environments the mean ring size is five.
The simplest illustrations of these are the hexagonal, square and Cairo  tilings, shown in figure \ref{fig:lattices}, but this law holds equally well for amorphous configurations.
For aperiodic systems strictly $\chi=1$, but as $N\rightarrow \infty$, the proportion of vertices with unsatisfied coordination on the sample perimeter become negligible overall, hence does the term in $\chi$.
Therefore in reality these relationships hold, and remain as applicable to amorphous graphene as the basalt columns in Fingal's Cave, and the Penrose tiling \cite{Goehring2014,Ressouche2009}.

This analysis also extends to spherical topology where $\chi=2$, and so:
 \begin{equation}
 	\label{eq:2dsphericalcases}
	\ki = \begin{cases}
		\frac{6N-12}{N}, \quad x_3 = 1 \\
		\frac{4N-8}{N}, \quad x_4 = 1
	\end{cases}\,.
\end{equation}
These relationships are the origin of the 12 pentagon rule for 3-coordinate fullerenes (the ``football problem''), or equivalently an ``8 triangle rule'' in the 4\--coordinate case, as this is the only way to satisfy these equations if the allowed ring sizes are limited to $k=5,6$ and $k=3,4$ respectively (as in figures \ref{fig:latticesfull92} and \ref{fig:latticesfull98}) \cite{Fowler1996}.
Much of the richness in the structural behaviour of \td{} physical networks stems from this fundamental constraint on the network average degree.

\begin{figure}[bt]
     \centering

     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[height=2.4cm]{./figures/methods/hex.pdf}
         \caption{Hexagonal}
         \label{fig:latticeshex}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[height=2.4cm]{./figures/methods/cai.pdf}
         \caption{Cairo}
         \label{fig:latticescairo}
     \end{subfigure}
     \hfill
	\begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[height=2.4cm]{./figures/methods/sq.pdf}
         \caption{Square}
         \label{fig:latticessq}
     \end{subfigure}
     \hfill
     \vspace{0.5cm}

       \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[height=2.4cm]{./figures/methods/full92.pdf}
         \caption{3\--coordinate fullerene}
         \label{fig:latticesfull92}
     \end{subfigure}
     \hspace{1cm}
     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[height=2.4cm]{./figures/methods/full98.pdf}
         \caption{4\--coordinate fullerene}
         \label{fig:latticesfull98}
     \end{subfigure}

     \caption{Panels (a)\--(c) give planar tilings of 6\--, 5\-- and 4\-- rings, where the ring size is related to the underlying atomic coordination. Panels (d) and (e) show the 3\-- and 4\-- coordinate tilings in spherical topology, where the mean ring size is reduced due to the change in the Euler characteristic.}
     \label{fig:lattices}
\end{figure}

\subsection{\lm's Law}
\label{s:lemaitre}

Knowing that the mean node degree is fixed by Euler's law, the next level of available information is the form of the underlying degree distribution, $p_k$.
Interestingly, the degree distributions found in physical ring networks seem relatively well defined.
For instance, it has been noted in models and realisations of \td{} silica glass that the ring statistics looked to follow a lognormal distribution \cite{Shackelford1981,Buchner2017}.
\lm{} \etal{} demonstrated that the distribution in 3-coordinate networks systems can be well described by a maximum entropy distribution \cite{Gervois1992}.
\lm's{} maximum entropy method is summarised here, trivially extended to arbitrary coordination.

The entropy of a probability distribution is defined as
\begin{equation}
	\mathcal{S}=-\sumk p_k\log p_k.
\end{equation}
In addition, the degree distribution has the following constraints:
\begin{align}
		\sumk p_k &=1, \\
		\sumk kp_k&=\ki,  \label{con:lm2}\\
		\sumk \frac{p_k}{k}&=\text{constant} \label{con:lm3},
\end{align}
where the first two constraints correspond to the normalisation condition and the fixed mean ring size, and the final constraint will be discussed below.
The entropy can then be maximised using Lagrange's method of undetermined multipliers to yield the result:
\begin{equation}
	\label{eq:mepk}
	p_k = \frac{e^{-\lambda_1 k - \lambda_2 / k}}{\sumk e^{-\lambda_1 k - \lambda_2 / k}},
\end{equation}
which can be solved numerically by substitution into equations \eqref{con:lm2} and \eqref{con:lm3}.
By allowing the chosen constant to vary, a family of maximum entropy curves can be generated, as in figure \ref{fig:lm1}.
The resulting distributions can be summarised by relating the variance, $\mu_2=\kii-\ki^2$, to a single chosen node degree probability, leading to the plot known as \lm's law, given in figure \ref{fig:lm2}.
It is usually framed in the context of the proportion of hexagons in a system, $p_6$, for the precise reason that most networks have $\ki=6$ and $p_6$ as the largest contribution.
Many experimental and theoretical studies have shown good agreement to this law \cite{Caer1993,Cerisier1996,Miklius2012}.

Simple extensions of the classic law are however possible, by modifying the mean degree or the permitted degree range.
For instance, $k$ is usually taken in the interval $k\geq3$ (as the triangle, $k=3$, is the smallest polygon), but there can be manifestations of physical systems where only certain degrees are accessible \cite{Rivier1988}.
Good examples of such systems are procrystalline lattices, which are explored in chapter \ref{ch:procrystals}, in which the underlying lattice constraints may preclude the formation of odd membered rings or rings greater than a certain size.
\lm{} curves for a selection of $\ki$ and $k$\--range modifications are given in figure \ref{fig:lm3}.
The maximum values of these curves, as indicated by circular points, are determined by removing constraint \eqref{con:lm3}, equivalent to setting $\lambda_2=0$ in equation \eqref{eq:mepk}.
One can see that these modifications can lead to marked differences in the maximum entropy solutions.
In particular, removing the possibility of small rings ($k<\ki$) forming reduces the maximum variance, as these are required to offset very large rings ($k\gg\ki$) in order to maintain the fixed mean ring size.

\begin{figure}[bt]
     \centering

     \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{./figures/methods/lm_1.pdf}
         \caption{Maximum entropy distributions}
         \label{fig:lm1}
     \end{subfigure}
     \hfill
      \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{./figures/methods/lm_2.pdf}
         \caption{\lm's law}
         \label{fig:lm2}
     \end{subfigure}
     \hfill

     \vspace{2mm}
      \begin{subfigure}[b]{0.45\textwidth}
         \centering
         \includegraphics[width=\textwidth]{./figures/methods/lm_3.pdf}
         \caption{Extensions to \lm's law}
         \label{fig:lm3}
     \end{subfigure}
     \hfill

     \caption{Illustration of \lm's maximum entropy method. Panel (a) gives examples of explicit maximum entropy distributions with different values of $p_6$. Panel (b) shows how these distributions can be summarised in a plot of $p_6$ \vs{} $\mu_2$ (\lm's law). Panel (c) provides extensions to the law by modifying the underlying constraints of the mean ring size and allowable $k$\--range, with circles indicating maximal values.}
     \label{fig:lm}
\end{figure}

The only somewhat puzzling aspect of this successful theory is the choice of constraint \eqref{con:lm3}.
It was originally rationalised on the basis that the areas of rings of a given size, $A_k$, can be well fit by an expression $A_k = ak+b+c/k$, where $a,b,c$ are constants.
As noted at the time, this is by no means true for all systems and in fact is contrary to the widely known Lewis law, which states that $A_k$ is linear in $k$ for many observable networks \cite{Lewis1928,Fortes1995,Kim2014}.
Overall, there does not appear to be a strong physical basis for this constraint, its justification relying on the perceived universality of the \lm{} law, and  hence it can be viewed as a tool that engineers the correct behaviour in the underlying distributions.

%Despite this, the universality of the \lm{} law suggests that there must be a physical basis to \eqref{con:lm3}, and in the section \ref{s:assortativity} it will be demonstrated that it can be regenerated by considering ring adjacencies.

\subsection{\aw{} Law}
\label{s:awlaw}

The ring statistics given by \lm's law are an important measure for physical networks, but they do not provide a complete characterisation of the ring structure, as they say nothing about the ring adjacencies.
This is important because whilst with the same ring statistics it is theoretically possible to organise the rings in many different arrangements, it is well known experimentally that only a subset of these are observed.
The vast majority of physical systems have a preference for small rings ($k<\ki$) be adjacent to large rings ($k>\ki$).
This effect was first noted in the grains of polycrystals by Aboav \cite{Aboav1970}.
Aboav quantified these ring correlations by measuring the mean ring size about a $k$\--ring, denoted $m_k$, and found empirically that $m_k \approx 5 + 8/k$.

In an attempt to explain this observation, Weaire formulated the %following
relationship:
\begin{equation}
	\label{eq:weairesumrule}
	\sumk km_kp_k = \sumk k^2p_k = \mu_2 + \ki^2,
\end{equation}
known as Weaire's sum rule \cite{Weaire1974}.
Although Weaire's original proof was not rigorous, the sum rule has since been shown to be exact, and formulations have also been constructed for use in higher dimensions \cite{Chiu1994}.
From this he suggested the modification of $m_k=5+\left(6+\mu_2\right)/k$ which satisfied this rule.
Aboav's original equation then became a special case when $\mu_2=2$, which is close to the expected value for a random collection of Voronoi polygons (see section \ref{s:voronoiintro}).
Aboav proposed that if a generic form of $m_k = A + B/k$ was used in conjunction with Weaire's sum rule then
\begin{equation}
	m_k = A+\frac{\mu_2+\ki^2-A\ki}{k}.
\end{equation}
This is now more commonly expressed in the linear form \cite{Chiu1995}:
\begin{equation}
	\label{eq:aboavweaire}
	km_k = \mu_2+\ki^2+\ki\left(1-\alpha\right)\left(k-\ki\right).
\end{equation}
Equation \ref{eq:aboavweaire} is known as the \aw{} law and relates the mean ring size about a given central ring to a single fitting parameter, $\alpha$.
The value of $\alpha$ describes the strength of the ring correlations, with a larger positive value indicating a greater tendency for small\--large ring adjacencies.
More specifically, the random limit can be deduced by evaluating $\frac{\partial{m_k}}{\partial{k}}=0$ as \cite{Delannay1994}:
\begin{equation}
	\label{eq:awrandlim}
	\alpha=-\frac{\mu_2}{\ki^2}.
\end{equation}
Hence all systems with $\alpha>-\mu_2/\ki^2$ have more small\--large ring  adjacencies than would be expected from chance whilst conversely those with $\alpha<-\mu_2/\ki^2$ have more small\--small and large\--large pairings.

\begin{figure}[tb]
     \centering
      \includegraphics[width=0.45\textwidth]{./figures/methods/aw_demo.pdf}
     \caption{Calculation of an \aw{} fit for three configurations (shown in figures \ref{fig:zach_high}\--\ref{fig:zach_low}). The value of the $\alpha$ parameter quantifies the tendency of small rings to be adjacent to large rings, with a larger value indicating stronger small\--large ring correlations.}
     \label{fig:awdemo}
\end{figure}

Despite the \aw{} law being purely empirical and there being no topological requirement for $m_k$ to vary systematically with $k$, the law does seem to hold well for a diverse set of physical systems.
The law is used %for example
in studies of materials, emulsions, biological tissues as well as in planetary science \cite{LeRoux2013,Roy2018,Noever1992,Mombach1993,Pedro2008}.
As an example of the calculation% of the \aw{} parameter
, the plots of the fits for the systems in figure \ref{fig:zach} are presented in figure \ref{fig:awdemo}, along with the corresponding $\alpha$ parameters.
This demonstrates two contrasting aspects of the \aw{} law.
Firstly the law holds very well, especially given the fact that these samples consist of just 20 rings each.
However, it also demonstrates that the law is by no means exact and that some uncertainty is inevitably introduced during the linear regression.
This dichotomy will be discussed in detail in section \ref{s:awdeficiencies}, where it will be shown that the \aw{} law can be replaced by another measure from network theory, namely the assortativity.