\section{Methods}

\subsection{Notation}

In the following, we will use definitions and notations introduced by \citep{rougier:2011} where a neural map is defined as the projection from a manifold $\Omega \subset \mathbb{R}^d$ onto a set $\mathcal{N}$ of $n$ {\em  neuron}s which is formally written as $\Phi : \Omega \rightarrow \mathcal{N}$. Each neuron $i$ is associated with a code word $\mathbf{w}_i \in \mathbb{R}^d$, all of which establish the set  $\{\mathbf{w}_i\}_{i \in   \mathcal{N}}$ that is referred as the code book. The mapping from $\Omega$ to $\mathcal{N}$ is a closest-neighbor winner-take-all rule such that any vector $\mathbf{v} \in \Omega$ is mapped to a neuron $i$ with the code $\mathbf{w}_\mathbf{v}$ being closest to the actual presented stimulus vector $\mathbf{v}$,
\begin{equation}
\Phi : \mathbf{v} \mapsto argmin_{i \in \mathcal{N}} (\lVert \mathbf{v} -
\mathbf{w}_i \rVert).
\label{eq:psi}
\end{equation}
The neuron $\mathbf{w}_\mathbf{v}$ is named the best matching unit (BMU) and the set $C_i = \{x \in \Omega | \Phi(x) = \mathbf{w}_i \}$ defines the {\em receptive field} of the neuron $i$.

%Before we present our new SOM learning algorithm, we introduce the notation  and terminology we are using throughout the present work. We borrow the notation from a previous work \citep{rougier:2011}.  A neural map is defined to be the projection from a manifold $\Omega \subset \mathbb{R}^d$ onto a set $\mathcal{N}$ of $n$ {\em neuron}s $\Phi : \Omega \rightarrow \mathcal{N}$. Each neuron $i$ is associated with a code word $\mathbf{w}_i \in \mathbb{R}^d$, all of which establish the set  $\mathcal{W} = \{\mathbf{w}_i, i \in \mathcal{N}\}$ that is referred as the code book. The mapping from $\Omega$ to $\mathcal{N}$ is a closest-neighbor winner-take-all rule such that any vector $\mathbf{v} \in \Omega$ is mapped to a neuron $i$ with the code $\mathbf{w}_\mathbf{v}$ being closest to the current input vector $\mathbf{v}$,
%\begin{equation}
%\Phi : \mathbf{v} \mapsto argmin_{i \in \mathcal{N}} (\lVert \mathbf{v} -
%\mathbf{w}_i \rVert).
%\label{eq:psi}
%\end{equation}
%The neuron $\mathbf{w}_\mathbf{v}$ is named the best matching unit (BMU) and the set $C_i = \{x \in \Omega | \Phi(x) = \mathbf{w}_i \}$ defines the {\em receptive field} of neuron $i$.

\subsection{Spatial distribution} % \& Centroidal Voronoi Tesselation}
\label{sec:spatial_dist}

The SOM space is usually defined as a two-dimensional region where nodes are arranged in a regular lattice (rectangular or hexagonal). Here, we consider instead the random placement of neurons with a specific spectral distribution (blue noise). As explained in \citep{Zhou:2012}, the spectral distribution property of noise patterns is often described in terms of the Fourier spectrum color. White noise corresponds to a flat spectrum with equal energy distributed in all frequency bands while blue noise has weak low-frequency energy, but strong high-frequency energy. In other words, blue noise has intuitively good properties with points evenly spread without visible structure (see figure~\ref{fig:sampling} for a comparison of spatial distributions).
%%
\begin{figure}[htbp]
  \includegraphics[width=\textwidth]{figure-blue-noise.pdf}
  \caption{\textbf{Spatial distributions.}
    \textbf{\textsf{A.}} Uniform sampling (n=1000) corresponding to white noise.
    \textbf{\textsf{B.}} Regular grid (n=32$\times$32) + jitter (2.5\%).
    \textbf{\textsf{C.}} Poisson disc sampling (n=988) corresponding to blue noise.}
  \label{fig:sampling}
\end{figure}
%%
There exists several methods \citep{Lagae:2008} to obtain blue noise sampling that have been originally designed for computer graphics (e.g. Poisson disk sampling, dart throwing, relaxation, tiling, etc.). Among these methods, the fast Poisson disk sampling in arbitrary dimensions \citep{Bridson:2007} is among the fastest ($\mathcal{O}(n)$) and easiest to use. This is the one we retained for the placement of neurons over the normalized region $[0,1]\times[0,1]$. Such Poisson disk sampling guarantees that samples are no closer to each other than a specified minimum radius. This initial placement is further refined by applying a LLoyd relaxation \citep{Lloyd:1982} scheme for 10 iterations, achieving a quasi centroidal Voronoi tesselation.

%The SOM space is usually defined as a two-dimensional manifold where nodes are arranged in a regular lattice (rectangular or hexagonal). Here, we follow a  different approach and, instead of the regular lattice, we place the neurons randomly by sampling a specific spectral distribution. More specifically, we assign neurons positions by drawing samples from a blue noise distribution. \citep{Zhou:2012}. have shown that the spectral distribution property of noise patterns is often described in terms of the Fourier spectrum color. For instance, white noise corresponds to a flat spectrum with signal's energy equally distributed to all frequency bands while blue  noise has weak low-frequency energy and strong high-frequency energy. An interesting property of the blue noise distribution is that  the resulting positions of neurons drawn are evenly spread without any apparent structure (see figure~\ref{fig:sampling} for a comparison of spatial distributions).
%%
%\begin{figure}[htbp]
%  \includegraphics[width=\textwidth]{figures/blue-noise.pdf}
%  \caption{\textbf{Spatial distributions.}
%    \textbf{\textsf{A.}} Uniform sampling (n=1000) corresponding to white noise.
%    \textbf{\textsf{B.}} Regular grid (n=32$\times$32) + jitter (2.5\%).
%    \textbf{\textsf{C.}} Poisson disc sampling (n=988) corresponding to blue noise.}
%  \label{fig:sampling}
%\end{figure}
%%

%Blue noise distributions have been used in the field of computer graphics for many years and there are manych different techniques for computing them: Poisson disk sampling, dart throwing, relaxation, tiling, and other applications (see \citep{Lagae:2008} for a review). One of the fastest and easiest to implement method for generating blue noise samples is the  fast Poisson disk sampling. This method, introduced by \citep{Bridson:2007}, can be used on arbitrary dimensions in linear time ($\mathcal{O}(n)$). In this work, we propose to use this method for placing neurons over a normalized  region of $[0,1]\times[0,1]$. Such Poisson disk sampling guarantees that samples are no closer to each other than a specified minimum radius. This initial placement is further refined by applying a Lloyd relaxation~\cite{Lloyd:1982} scheme for $10$ iterations to achieve a quasi centroidal Voronoi tesselation.

\subsection{Topology}
\label{sec:topo}

Considering a set of $n$ points $P = \{P_i\}_{i \in [1,n]}$ on a finite region,
we first compute the Euclidean distance matrix $E$, where $e_{ij} = \lVert P_i - P_j \rVert$
and we subsequently define a connectivity matrix $G^{p}$
%= \{G^{p}_{ij}\}_{i,j \in [1,n]}$ \gid{$G^{p} = g^{p}_{ij},\, \text{where } i,j \in [1,n]$}
such that only the $p$ closest points
are connected. More precisely, if $P_j$ is among the $p$ closest neighbours of
$P_i$ then $g^p_{ij} = 1$ else we have $g^p_{ij} = 0$.
From this connectivity
matrix representing a graph, we compute the length of the shortest path between
each pair of nodes and stored them into a distance matrix $D^p$. Note that
lengths are measured in the number of nodes between two nodes such that two
nearby points (relatively to the Euclidean distance) may have a corresponding
long graph distance as illustrated in figure \ref{fig:topology}. This matrix
distance is then normalized by dividing it by the maximum distance between two
nodes such that the maximum distance in the matrix is 1. In the singular case
when two nodes cannot be connected through the graph, we recompute a spatial
distribution until all nodes can be connected.
%%
\begin{figure}
  \includegraphics[width=\columnwidth]{figure-distances.pdf}
  \caption{\textbf{Influence of the number of neighbours on the graph
    distance.} The same initial set of 1003 neurons has been equiped with
    2-nearest neighbors, 3 nearest neighbors and 4-nearest neighbors induced
    topology (panels \textbf{A}, \textbf{B} and \textbf{C} respectively). A
    sample path from the the lower-left neuron to the upper-right neuron has
    been highlighted with a thick line (with respective lengths of 59, 50 and
    46 nodes).}
  \label{fig:topology}
\end{figure}

%Consider a set $P$ of $n$ points on a finite region of $[0, 1] \times [0, 1]$. The steps we follow to determine the topology of the SOM are: First, we compute the Euclidean distance matrix ${\bf E} \in \mathbb{R}^{n\times n}$, where $e_{ij} = \lVert p_i - p_j \rVert$ and $i, j=1, \ldots, n$. Subsequently  we define a connectivity matrix ${\bf G}_m$ with elements $g_{ij} = 1$ if $p_j$ belongs to the $m$ closest points to $p_i$ and $g_{ij} = 0$ otherwise. This definition implies that the matrix ${\bf G}_m$ carries information about connected neurons within a predetermined vicinity. Once we compute matrix ${\bf G_m}$, which essentially represents a graph, we compute the shortest path between each pair of nodes on the graph and we store them into a new matrix called ${\bf D}_m$. Note that lengths are measured in number of nodes (hops) required to reach two nodes such that the two corresponding Euclidean points (represented by the nodes) may have a graph distance as illustrated in figure \ref{fig:topology}.  \gid{This matrix distance is then normalized by dividing it by the maximum distance between two nodes. NOT VERY CLEAR}. In the degenerative case where two nodes are not connected on the graph, we resample from a spatial distribution until all nodes have degree greater than one (are connected at least with one other node \gid{IS THIS CORRECT?}).
%%
%\begin{figure}
%  \includegraphics[width=\columnwidth]{figures/distances.pdf}
%\caption{\textbf{Influence of the number of neighbours on the graph distance.} The same initial set of 1003 neurons has been equipped with 2-nearest neighbors, 3 nearest neighbors and 4-nearest neighbors induced topology (panels \textbf{A}, \textbf{B} and \textbf{C} respectively). A sample path from the the lower-left neuron to the upper-right neuron has been highlighted with a thick line (with respective lengths of 59, 50 and 46 nodes).}
%  \label{fig:topology}
%\end{figure}

\subsection{Learning}

The learning process is an iterative process between time $t=0$ and time $t=t_f \in \mathbb{N}^+$ where vectors $\mathbf{v} \in \Omega$ are sequentially presented to the map. For each presented vector $\mathbf{v}$ at time $t$, a winner $s \in \mathcal{N}$ is determined according to equation (\ref{eq:psi}). All codes $\mathbf{w}_{i}$ from the code book are shifted towards $\mathbf{v}$ according to
\begin{equation}
  \Delta\mathbf{w}_{i} = \varepsilon(t)~h_\sigma(t,i,s)~(\mathbf{v} -
  \mathbf{w}_i)
  \label{eq:som-learning}
\end{equation}
with $h_\sigma(t,i,j)$ being a neighborhood function of the form
\begin{equation}
  h_\sigma(t,i,j) = e^{- \frac{{d^p_{ij}}^2}{\sigma(t)^2}}
  \label{eq:som-neighborhood}
\end{equation}
where $\varepsilon(t) \in \mathbb{R}$ is the learning rate and $\sigma(t) \in \mathbb{R}$
is the width of the neighborhood defined as
\begin{equation}
  \sigma(t) =
  \sigma_i\left(\frac{\sigma_f}{\sigma_i}\right)^{t/t_f}, \text{ with } \varepsilon(t) =
  \varepsilon_i\left(\frac{\varepsilon_f}{\varepsilon_i}\right)^{t/t_f},
\end{equation}
while $\sigma_i$ and $\sigma_f$ are respectively the initial and final neighborhood width and $\varepsilon_i$ and $\varepsilon_f$ are respectively the initial and final learning rate. We usually have $\sigma_f \ll \sigma_i$ and $\varepsilon_f \ll \varepsilon_i$.

%The learning algorithm we propose in this work relies on the standard SOM algorithm~\cite{Kohonen:1982}. Once we have define the topology of the map following the steps we described in paragraph~\ref{sec:topo}, we can start the learning process. Learning is iterative and starts at a time $t_0=0$ and runs until some predetermined final time step, $t_f \in \mathbb{N}^+$, has been reached. At every iteration input vectors $\mathbf{v} \in \Omega$ are sequentially given to the map with respect to the probability density function $f$ \gid{Where is defined?}. For each vector $\mathbf{v}$ at time $t$, a winner neuron with index  $s \in \mathcal{N}$ is determined according to equation (\ref{eq:psi}). This means that at time $t$ neuron $s$ is closer to the input vector ${\bf v}$, in the sense of Euclidean distance, than any other neuron. Once the winner neuron has been identified all codes $\mathbf{w}_{i}$ from the current code book are shifted towards $\mathbf{v}$ according to
%\begin{align}
%\label{eq:som-learning}
%    \Delta\mathbf{w}_{i} &= \varepsilon(t)~h(t,i,s;\sigma)~(\mathbf{v} - \mathbf{w}_i),
%\end{align}
%where $s$ is the index of the winner neuron, $i$ is the index of code words in the code book and $t$ is the current time step. $h_\sigma(t,i,j;\sigma)$ is a neighborhood function of the form
%\begin{equation}
%  h(t,i,j; \sigma) = \exp\Big(-\frac{{d_{ij}}^2}{\sigma(t)^2}\Big)
%  \label{eq:som-neighborhood}
%\end{equation}
%where $\varepsilon: \mathbb{R} \rightarrow \mathbb{R}$ is the learning rate time-dependent function given by $\varepsilon(t) = \varepsilon_i\left(\frac{\varepsilon_f}{\varepsilon_i}\right)^{t/t_f}$, where $\varepsilon_i$ and $\varepsilon_f$ are the initial and final learning rates, respectively. $\sigma: \mathbb{R} \rightarrow \mathbb{R}$ is determines the width of the  neighborhood function~\eqref{eq:som-neighborhood} and it is reads $\sigma(t) = \sigma_i\left(\frac{\sigma_f}{\sigma_i}\right)^{t/t_f}$, where $\sigma_i$ and $\sigma_f$ are the initial and final neighborhood widths, respectively. We usually assume $\sigma_f \ll \sigma_i$ and  $\varepsilon_f \ll \varepsilon_i$. The entire learning procedure is summarized by Algorithm~\ref{algo:vsom}.

%% \input{algorithm}

\subsection{Analysis Tools}
In order to analyze and compare the results of RSOM and SOM, we used a spectral method and persistence diagram analysis on the respective codebooks. These analysis tools are detailed below but roughly, the spectral method allows to estimate the distributions of eigenvalues in the activity of the maps while the persistence diagram allows to check for discrepancies between the topology of the input space and the topology of the map.

% To analyze the results of both the Kohonen SOM and VSOM algorithms and to make any comparison between the two algorithms we use a spectral method and persistence diagram on codebooks. The spectral method estimates the distributions of eigenvalues of the activity of neurons. The persistence diagram is a topological-geometrical  approach, more precisely is a tool coming from the field of topological data analysis (TDA). TDA provides the tools to investigate the topology of the maps and the input space and spot differences between the topology of the input space and the neural space of the SOM algorithms (Kohonen and VSOM).

%\subsubsection{Topological Data Analysis}
\label{sec:tda}

Topological Data Analysis (TDA) \citep{Carlsson:2009} provides methods and tools to study topological structures of data sets such as point cloud and is useful when geometrical or topological information is not apparent within a data set. Furthermore, TDA tools are insensitive to dimension reduction and noise which make them well suited to analyze high-dimensional self-organized maps and their corresponding input data sets. In this work, we use the notion of persistent barcodes and diagrams \citep{Edelsbrunner:2008} to spot any differences between the topology of the input and neural spaces. Furthermore, we can apply some metrics from TDA such as the Bottleneck distance and measure how close two persistent diagrams are.
% qualify the quality of the representations of a map.
Since the exact manifold (or distribution) of the input space is not known in general and the SOM algorithms only approximate it, we simplify these manifolds by retaining their original topological structure.
Here we approach the manifolds of input and neural spaces using the Alpha complex. Before diving into more details regarding TDA, we provide here a few definitions and some notation. A $k$-simplex $\sigma$ is the convex hull of $k+1$ affinely independent points (for instance a $0$-simplex is a point, a $1$-simplex is an edge, a $2$-simplex is a triangle, etc). A simplicial complex with vertex set $\mathcal{V}$ is a set $\mathcal{S}$ of finite subsets of $\mathcal{V}$ such that the elements of $\mathcal{V}$ belong to $\mathcal{S}$ and for any $\sigma \in \mathcal{S}$ any subset $\sigma$ belongs to $\mathcal{S}$. Said differently, a simplicial complex is a space that has been  constructed out of intervals, triangles, and other higher dimensional simplices.

In our analysis we let $\mathcal{S}(\mathcal{M}, \alpha)$ be a Alpha simplicial complex with $\mathcal{M}$ being a point cloud, either the input space or the neural one, and $\alpha$ is the ``persistence'' parameter. More specifically, $\alpha$ is a threshold (or radius as we will see later) that determines if the set $X$ spans a $k$-simplex if and only if $d(x_i, x_j) \leq \alpha$ for all $0 \leq i, j \leq k$. From a practical point of view, we first define a family of thresholds $\alpha$ (or radius) and for each $\alpha$, we center a ball of radius $\alpha$ on each data point and look for possible intersections with other balls. This process is called filtration of simplicial complexes. We start from a small $\alpha$ where there are no intersecting balls (disjoint set of balls) and steadily we increase the size of $\alpha$ up to a point where a single connected blob emerges. As $\alpha$ varies from a low to a large value, holes open and close as different balls start intersecting. Every time an intersection emerges we assign a {\em birth} point $b_i$ and as the $\alpha$ increases and some new intersections of larger simplicies emerge some of the old simplicies die (since they merge with  other smaller simplicies to form larger ones). Then we assign a {\em death} point $d_i$. A pair of a birth and death points $(b_i, d_i)$ is plotted on a Cartesian two-dimensional plane and indicates when a simplicial complex was created and when it died. This two-dimensional diagram is called persistent diagram and the pairs (birth, death) that last longer reflect significant topological properties. The longevity of birth-death pairs is more clear in the persistent barcodes where the lifespan of such a pair is depicted as a straight line.

In other words, for each value of $\alpha$ we obtain new simplicial complexes and thus new topological properties such as  homology are revealed. Homology encodes the number of  points, holes, or voids in a space. For more thorough reading we refer the reader to \citep{Chazal:2017,Ghrist:2008,Zomorodian:2005}. In this work, we used the Gudhi library~\citep{Maria:2014} to compute the Alpha simplicial complexes, the filtrations and the persistent diagrams and barcodes. Therefore, we compute the persistent diagram and persistent barcode of the input space and of the maps and we calculate the Bottleneck distance between the input and SOM and RSOM maps diagrams. The bottleneck distance provides a tool to compare two persistent diagrams in a quantitative way. The Bottleneck distance between two persistent diagrams $\text{dgm}_1$ and $\text{dgm}_2$ as it is described in~\cite{Chazal:2017}
%%
\begin{align}
    \label{eq:bottle}
    d_b(\text{dgm}_1, \text{dgm}_2) &= \inf_{\text{matching }m}\{ \max_{(p, q) \in m} \{||p - q||_{\infty} \} \},
\end{align}
%%
where $p \in \text{dgm}_1 \backslash \Delta$, $q \in \text{dgm}_2 \backslash \Delta$, $\Delta$ is the diagonal of the persistent diagram (the diagonal $\Delta$ represents all the points that they die the very moment they get born, $b = d$). A matching between two  diagrams $\text{dgm}_1$ and $\text{dgm}_2$ is a subset $m \subset \text{dgm}_1 \times \text{dgm}_2$ such that every point in $\text{dgm}_1 \backslash \Delta$ and $\text{dgm}_2 \backslash \Delta$ appears exactly once in $m$.

% In a similar way the Wasserstein distance is  defined by
% %%
% \begin{align}
%     \label{eq:wasser}
%     W_p(\text{dgm}_1, \text{dgm}_2)^p &= \inf_{\text{matching } m} \{ \sum_{(p, q) \in m}^{} ||p - q||^p_{\infty} \}.
% \end{align}
% %%

\subsection{Simulation Details}

Unless specified otherwise, all the models were parameterized using values given in table \ref{table:parameters}. These values were chosen to be simple and do not really impact the performance of the model. All simulations and figures were produced using the Python scientific stack, namely, SciPy \citep{Jones:2001}, Matplotlib \citep{Hunter:2007}, NumPy \citep{Walt:2011}, Scikit-Learn \citep{Pedregosa:2011}. Analysis were performed using Gudhi \citep{Maria:2014}).
Sources are available at \href{https://github.com/rougier/VSOM}{github.com/rougier/VSOM}.
%%
\begin{table}[!ht]
  \begin{center}
    \begin{tabular}{ll}
        \textbf{Parameter} & \textbf{Value} \\
        \hline
        Number of epochs      ($t_f$)           & 25000\\
        Learning rate initial ($\varepsilon_i$) & 0.50\\
        Learning rate final   ($\varepsilon_f$) & 0.01\\
        Sigma initial         ($\sigma_i$)      & 0.50\\
        Sigma final           ($\sigma_f$)      & 0.01\\
    \end{tabular}
      \caption{\textbf{Default parameters} Unless specified otherwise, these are
        the parameters used in all the simulations.}
      \label{table:parameters}
  \end{center}
\end{table}

%We conduct all the experiments using the parameters provided by Table~\ref{table:parameters}. In all the experiments the input space is the Cartesian product $[0, 1] \times [0, 1]$ and neurons positions drawn from a blue noise distribution using the fast Poisson disk sampling algorithm~\cite{Bridson:2007} (see paragraph~\ref{sec:spatial_dist} for more details).  The source code of the proposed algorithm is written in the Python programming language (SciPy~\cite{Jones:2001}, Matplotlib~\cite{Hunter:2007} and NumPy~\cite{Walt:2011}, Scikit-Learn~\cite{Pedregosa:2011}, Gudhi~\cite{Maria:2014}). Sources are available at \href{https://github.com/rougier/VSOM}{github.com/rougier/VSOM}.

%% Considering a set of $n$ points $P = \{P_i\}_{i \in [1,n]}$ on a finite domain
%% $D \in \mathbb{R}^2$, the Voronoi tesselation $V(P) = \{V_i\}_{i \in [1,n]}$ of
%% $P$ is defined as:
%% %
%% \begin{equation}
%%   \forall i \in [1,n], V_i = \{x \in D \mid
%%   \lVert x - P_i \rVert \leq \lVert x - P_j \rVert, \forall j \neq i\}
%% \end{equation}
%% %
%% Reciprocally, the (unique) Delaunay triangulation $T(P) = \{T_i\}_{i \in
%%   [1,n]}$ of $P$ is the dual graph of the Voronoi diagram and defined such that
%% no point in $P$ is inside the circumcircle of any triangles in $T(P)$. The
%% centers of the circumcircles are equivalent to the Voronoi diagram, i.e. a
%% partition of $D$ into Voronoi cells. For each of the cell $V_i$, we can compute
%% its centroid $C_i$ which is the center of mass of the cell. A Voronoi
%% tesselation is said to be centroidal when we have $\forall i \in [1,n], C_i =
%% P_i$ (see figure~\ref{fig:CVT}).\\

%% For an arbitrary set of points, there is no guarantee that the corresponding
%% Voronoi tesselation is centroidal but different methods can be used to
%% generate a centroidal tesselation from an arbitrary set of points. One of the
%% most straightforward and iterative methods is the Lloyd relaxation scheme
%% \cite{Lloyd:1982}:
%% \begin{enumerate}
%%   \item The Voronoi diagram of the $n$ points is computed
%%   \item The centroid of each of the $n$ Voronoi cell is computed.
%%   \item Each point is moved to the corresponding centroid of its Voronoi cell
%%   \item The method terminates if criterion is met (see below), else go to 1
%% \end{enumerate}
%% The algorithm finishes when the maximum distance between points and centroids
%% is less than a given threshold as illustrated in figure~\ref{fig:CVT}. It is
%% to be noted that because of numerical imprecisions, there is no guarantee that
%% an arbitrary small threshold can be reached.

%% \begin{figure}[htbp]
%%   \includegraphics[width=\textwidth]{figures/CVT.pdf}
%%   \caption{\textbf{Centroidal Voronoi Tesselation.}  \textbf{\textsf{A.}}
%%     Voronoi diagram of a uniform distribution (n=100) where red dots represent
%%     the uniform distribution and white circles represent the centroids of each
%%     Voronoi cell. \textbf{\textsf{B.}} Centroidal Voronoi diagram where the
%%     point distribution matches the centroid distribution which constitutes a
%%     blue noise distribution (i.e. {\em a distribution that is roughly uniformly
%%       random with no preferred inter-point directions or distances} according
%%     to the definition of \cite{Ebeida:2014}). This figure has been obtained
%%     from the initial distribution on the left after 50 iterations of the Lloyd
%%     relaxation algorithm. }
%%   \label{fig:CVT}
%% \end{figure}
%


%!TEX root = ../main.tex
\section{Pure preferential attachment}\label{section:pure-preferential-attachment}

\subsection{Degree distribution: Theory}\label{subsection:ppa-degree-distribution}
The master equation that describes the evolution of the BA model is given by

\begin{equation}
	n(k, t+1) = n(k, t) + m \Pi(k-1, t)n(k-1, t) - m \Pi(k, t)n(k, t) + \delta_{k,m}
	\label{eq:master}
\end{equation}
where $k$ is the total degree of a vertex, $n(k, t)$ is the number of nodes at time $t$ with total degree $k$, and the probability $\Pi$ for choosing the existing vertex depends on the model.

In the pure preferential attachment model, we choose an existing edge with probability $\Pi_{pa} = k/ 2E(t)$ where $E(t)$ is the number of edges and $2E(t)$ is the normalization constant corresponding to the total degree of the network. Assuming $E(0) = mN(0)$, the number of edges at a given time $t$ is given by $E(t) = mN(t)$, so we get $\Pi = k / 2mnN(t)$. Since we are concerned with the degree distribution of the model at large $t$, we consider the long-time ansatz $n(k, t) \rightarrow N(t) p_{\infty}(k)$.

Substituting these terms into the master equation, we obtain
\begin{equation}
	p_{\infty}(k) = \frac{1}{2}[(k-1)p_{\infty}(k-1) - kp_{infty}(k)] + \delta_{k,m}
	\label{eq:degree-distribution-p-infinity}
\end{equation}

It is clear that $p_{\infty}(k < m) = 0$, since $m$ edges are added at every stage. So there are 2 cases to consider when solving for the above equation: $k = m$ and $k > m$.

In the first case of $k > m$, $\delta_{k,m} = 0$ and we can rearrange \autoref{eq:degree-distribution-p-infinity} to get

\begin{equation}
	\frac{p_{\infty}(k)}{p_{\infty}(k+1)} = \frac{k-1}{k+2}
	\label{eq:p-infinity-k-greater-m}
\end{equation}

We can substitute in a trial solution of the form

\begin{equation}
	f(z) = A \frac{\Gamma(z+1+a)}{\Gamma(z+1+b)}
	\label{eq:trial-solution}
\end{equation}
where $\Gamma(z)$ is the Gamma function. Its central property is that
\begin{equation}
	\Gamma(z+1) = z \Gamma(z),\,\, \Gamma(1) = 1.
	\label{eq:gamma-function-property}
\end{equation}

Substituting the trial solution in \autoref{eq:trial-solution} gives
\begin{equation}
	\frac{A \Gamma(z+1+a)}{\Gamma(z+1+b)} \times \frac{\Gamma(z+b)}{A \Gamma(z+a)}
\end{equation}
which simplifies to give $(z+a) / (z+b)$, using the the property in \autoref{eq:gamma-function-property}.

Substituting $a = -1$ and $b=2$, we get the solution for \autoref{eq:p-infinity-k-greater-m} in terms of $A$ and the Gamma function:

\begin{equation}
	p_{\infty}(k) = A \frac{\Gamma(k)}{\Gamma(k+3)}
\end{equation}
which simplifies to
\begin{equation}
	p_{\infty}(k) = \frac{A}{k(k+1)(k+2)}.
	\label{eq:p-infinity-solution-unknown-A}
\end{equation}

For the second case of $k = m$, \autoref{eq:degree-distribution-p-infinity} becomes
\begin{equation}
	p_{\infty}(m) = \frac{1}{2}[(m+1)p_{\infty}(m-1) - mp_{\infty}(m)] + 1.
	\label{eq:p-infinity-k-equals-m}
\end{equation}
However, we already know that $p_{\infty}(k < m) = 0$, that is, $p_{\infty}(m-1) = 0$. Using this, and rearranging \autoref{eq:p-infinity-k-equals-m}, we get

\begin{equation}
	p_{\infty}(m) = \frac{2}{m+2}.
	\label{p-infinity-normalization}
\end{equation}

Substituting $k = m$ and \autoref{p-infinity-normalization} into \autoref{eq:p-infinity-solution-unknown-A}, we get

\begin{equation}
	\frac{A}{m(m+1)(m+2)} = \frac{2}{m+2},
\end{equation}
giving us the constant $A$ as
\begin{equation}
	A = 2m(m+1).
	\label{eq:normalization-constant}
\end{equation}

For this constant to be physically reasonable, we need to check that the probability satisfies normalization, that is, we need to prove
\begin{equation}
	\sum_{k=m}^\infty p_{\infty}(k) = 2m(m+1)\sum_{k=m}^\infty \frac{1}{k(k+1)(k+2)} = 1.
	\label{eq:normalization-criteria}
\end{equation}

The term in the summation of \autoref{eq:normalization-criteria} can be expanded as a partial fraction:
\begin{equation}
	\sum_{k=m}^\infty \frac{1}{k(k+1)(k+2)} = \sum_{k=m}^\infty \frac{1}{2k} - \sum_{k=m}^\infty \frac{1}{k+1} + \sum_{k=m}^\infty \frac{1}{2(k+2)}
	\label{eq:partial-fractions}
\end{equation}

By writing out the first few terms of each summation, we can see that most terms cancel:

\begin{equation}
\setlength{\arraycolsep}{0pt}% no padding
\newcolumntype{B}{>{{}}c<{{}}}
\begin{array}{ B l B l B l B l B l B}
	\frac{1}{2m} & {}-{} &\frac{1}{m+1} & {}+{} & \cancel{\frac{1}{2(m+2)}} & \\
	& {}+{} & \frac{1}{2(m+1)} & {}-{} &\cancel{\frac{1}{m+2}} & {}+{} &\cancel{\frac{1}{m+3}} \\
	& & & {}+{} & \cancel{\frac{1}{2(m+2)}} & {}-{} & \cancel{\frac{1}{m+3}} & {}+{} &\frac{1}{2(m+4)} \\
	& & & & & {}+{}& \cancel{\frac{1}{2(m+3)}} & {}-{} & \frac{1}{m+4} & {}+{} & \frac{1}{2(m+5)} \\
	& & & & & & & {}+{}& ...\\
\end{array}
\label{eq:summation-cancel}
\end{equation}
and from the remaining terms we get the relation in \autoref{eq:normalization-criteria}
\begin{equation}
	 \sum_{k=m}^\infty p_{\infty}(k) = 2m(m+1)\left ( \frac{1}{2m} - \frac{1}{m} + \frac{1}{2(m+1)} \right ) = 2m(m+1) \frac{1}{2m(m+1)} = 1
	 \label{eq:normalization-satisfied}
\end{equation}

Hence, we can confirm that the complete exact solution for the probability distribution in the long time limit is
\begin{equation}
	p_{\infty}(k) = \frac{2m(m+1)}{k(k+1)(k+2)}.
	\label{eq:p-infinity-solution}
\end{equation}

\subsection{Degree distribution: Numerical analysis}\label{subsection:ppa-numerical-analysis}

To leverage its speed, \texttt{c++} code was used to generate graph data, while \texttt{python} was used for data analysis due to its wide range of data analysis tools, such as \texttt{numpy}, \texttt{scipy}, and \texttt{pandas}.

There were two main concerns when doing the numerical simulation:

\begin{enumerate}
	\item What should the initial graph $G_0$ be?
	\item Should self loops and multiple edges be allowed?
	\item Of what order should $m$ and $N$ be respectively?
\end{enumerate}

The initial graph should be negligible if $N \rightarrow \infty$. In our simulations, we assumed that the $N$ chosen was large enough for this limit to apply, so we for convenience we chose our initial graph to be an empty graph.

Computational efficiency of our algorithm is important considering that bigger datasets give better and more reliable statistical results. To yield statistically significant results and optimize efficiency, the following algorithm was used. In this algorithm, the array $M$ holds the list of edges represented by pairs of vertices, for example, the vertices at $M[0]$ and $M[1]$ are connected, $M[2]$ and $M[3]$ are connected, and so on. In this list, the number of occurences of a vertex is equal to its degree, so it can be used as a sample pool to achieve preferential attachment. To choose $m$ neighbours for each new vertex, we then sample from $M$. This is also equivalent to choosing an edge at random and then choosing a vertex at random from the edge.

\begin{algorithm}
\caption{Algorithm for preferential attachment}\label{alg:pa}
\begin{algorithmic}[1]
\Require{number of vertices $N$, minimum degree $m$}\Comment{$N > m$}
\State Initialize our graph $g$
\State Initialize $M$ as an empty array of length $2Nm$
\For{$v$ in [0, ..., n-1]}
	\State add new vertex to $g$
	\For{$i$ in [0, ..., m-1]}
		\State $M[2(vm + i)] \gets v$
		\State draw $r$ uniformly at random
		\State from $[0, ..., 2(vm + i)]$\Comment{Choose random vertex from $M$}
		\State $M[2(vm + i)+1] \gets M[r]$ \Comment{Add edge between vertex $M[r]$ and $v$}
	\EndFor
\EndFor
\State
\For{$i$ in [0, ..., nm-1]}\Comment{Add all edges stored in M into the graph}
	\State Add edge $(M[2i], M[2i+1])$ to graph $g$
\EndFor
\end{algorithmic}
\end{algorithm}

Clearly this approach produces self loops and multiple edges. However, since we are concerned with the limit of $N \rightarrow \infty$, these effects are insignificant. Also, while unsatisfactory, multiple edges and self-loops do not affect our theoretical result. In the large $N$ limit, the probability of getting multiple edges or self loops is small, and so for simplicity this algorithm was used without modification.

To check that the model was implemented correctly, the degree distribution generated by the model compared checked against the \texttt{networkx} implementation of the BA model, since it is an established software package. It was verified that the degree distribution for $N = 10^5$ and different values of $m$ generated through our algorithm is highly similar to the \texttt{networkx} degree distributions. This gives confidence that the algorithm is working as expected. At $N = 10^5$, it is already big enough to achieve a stationary degree distribution.

To investigate the degree distribution, the model was run for fixed $N$ but varying $m$. $N$ was mostly limited by efficiency and storage space to be $10^7$, and $m$ was chosen to vary logarithmically between $1$ and $32$. As $N$ is finite, there is bound to be a noisy tail at large $k$, where these degrees appeared once. To reduce the noise, 100 simulations were run for a single $m$ and the degree distribution was averaged over all runs. This can be seen in \autoref{fig:pa-fixed-n-degree-dist}. The log-binning technique \citep{Christensen:2005} was adopted to to collapse the noisy data.

\begin{figure}
    \centering
    \includegraphics[height=0.5\linewidth]{img/pa-fixed-n-degree-dist}
    \caption{Raw degree distribution averaged over 100 runs for $m = 1, 2, 4, 8, 16, 32$. A noisy tail can still be seen at large $k$ due to finite sized effects. }
    \label{fig:pa-fixed-n-degree-dist}
\end{figure}

Visually, as can be seen from \autoref{fig:pa-fixed-n-logbin}, the numerical results seem to agree with the theoretical model after log-binning, until finite sized effects begin to kick in at large $k$. Alternatively, we can look at the complementary cumulative distribution function (ccdf) to observe the behaviour of the fat tail more clearly. This is shown in \autoref{fig:ccdf}. We can see that near the fat tail, the numerical ccdf goes slightly higher than the theoretical, before falling off. This is consistent across different $m$ values.

\begin{figure}
    \centering
    \includegraphics[height=0.5\linewidth]{img/pa-fixed-n-logbin}
    \caption{The solid lines show log-binned degree distributions for $m = 1, 2, 4, 8, 16, 32$. The dashed lines show the values predicted by the theoretical model. There is good agreement for small $k$ finite sized effects kick in. }
    \label{fig:pa-fixed-n-logbin}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[height=0.7\linewidth]{img/ccdf}
    \caption{CCDF of $m=2, 4, 8, 16$, where the solid line is the numerical CCDF and the dotted line is the CCDF predicted by theory.}
    \label{fig:ccdf}
\end{figure}

A Kolmogorov-Smirnov (KS) test was used to quantify the goodness-of-fit between theoretical and numerical results. It is a non-parametric test that measures how far apart two distributions are, and returns a KS-statistic that can be converted into a $p$-value. A smaller KS-statistic implies that differences between data and model are more likely to be attributed to statistical differences, while a large KS-statistic implies that the model is unlikely to have been the generating function for the data.

The table below shows the results of the KS test on $m = 1$ to $m=32$:

\begin{center}
\begin{tabular}{ c | c }
 m &  KS-statistic \\
 \hline
 1 & 0.000414 \\
 2 & 0.000978 \\
 4 & 0.000744 \\
 8 & 0.000662 \\
 16 & 0.001002 \\
 32 & 0.001132 \\
\end{tabular}
\label{table:ks-test}
\end{center}

However, these values are meaningless unless we know what kind of deviation from theoretical is considered acceptable, and at what value we should reject the null hypothesis. To answer that, we generate synthetic datasets governed by the theoretical distribution in \autoref{eq:p-infinity-solution} to measure how far they fluctuate from the reference theoretical distribution in a similar way as described by \citet{Clauset2009}, and compare the results with the simulated data.

The synthetic datasets were generated with a lower bound of $m$ and an upper bound given by the maximum degree in the simulated dataset. If the simulated data is much further from the theoretical distribution than the typical synthetic data, then we would have grounds to reject the null hypothesis.

For each of the synthetic datasets, we compare it to the theoretical distribution and calculate its KS-statistic. Then we define our $p$-value as the fraction of the time the resulting statistic is larger than the value for the numerical data. We can then reject the null hypothesis if $p \leq 0.1$ \citep{Clauset2009}, that is, if there is a 1 in 10 probability that we would, by chance, get data that agree as poorly with the model as the current data.

Another issue to consider is the number of synthetic datasets to generate. Again, \citet{Clauset2009} suggests a useful rule: to have p-values accurate to within about $\epsilon$, we need at least $\frac{1}{4}\epsilon^{-2}$ datasets. In this project, 100 datasets were generated for each $m$, to get an accuracy of about $0.05$.

The resulting p-values are given in the table below:
\begin{center}
\begin{tabular}{ c | c }
 m &  p-value\\
 \hline
 1 & 0.71 \\
 2 & 0.45 \\
 4 & 0.50 \\
 8 & 0.58 \\
 16 & 0.71 \\
 32 & 0.46 \\
\end{tabular}
\captionof{table}{The list of $p$-values for each $m$ for preferential attachment when compared with synthetic datasets.}
\label{table:ks-test-all}
\end{center}

Since the $p$-values are all larger than $0.1$, we can say it is plausible that the numerical data was drawn from the theoretical distribution.

A point to note is that the KS test is used for continuous distributions, while our degree distribution is discrete. However, it was assumed that for large $N$, there will be values spanning a large range of $k$, hence the change in $k$ can be considered small, and the distribution can be approximated as a continuous distribution.

A chi-squared test was also considered. The chi-squared test is a categorical test, suitable for discrete distributions, however, the test becomes invalid when the observed or expected frequencies for each category is too small, with a typical rule being that the frequencies should be at least 5 \citep{Lawrence1997}. In our numerical data, there many large values of $k$ that only appeared once, and hence this test was determined to be not suitable.

\subsection{Largest expected degree: Theory}
The finite size of the system imposes a structural cutoff on the largest expected degree. For scale free networks, \citet{Aiello2001a} defined the maximum degree to be approximately the value above which there is less than one vertex of that degree in the graph on average, that is, $N \sum_{k = k_1}^\infty p_\infty(k) = 1$.

Generally, it is shown \citep{Boguna2004} that for a scale free network with $p_{\infty}(k) \propto k^{\gamma}$, the largest expected degree will be

\begin{equation}
	k_1(N) \sim N^{1 / (\gamma -1)}.
	\label{eq:largest-expected-degree-research}
\end{equation}

Starting with the equation

\begin{equation}
	N \sum_{k=k_1}^\infty p_{\infty}(k) = 1,
	\label{eq:largest-expected-degree-criteria}
\end{equation}
we can see that this is almost identical to \autoref{eq:normalization-criteria}, just with a different factor and lower limit. Hence we have

\begin{equation}
	2m(m+1) \frac{1}{2k_1(k_1+1)} = \frac{1}{N}.
	\label{eq:largest-expected-degree-derivation}
\end{equation}
We can then rearrange this to give us an expression for $k_1$:
\begin{equation}
	k_1 = \frac{-1 + \sqrt{1 + 4Nm(m+1)}}{2}
	\label{eq:pa-k1-expression}
\end{equation}
where the other negative solution is rejected as it is unphysical, and verifying that $k \propto N^{0.5}$.

\subsubsection{Numerical analysis: Largest expected degree}\label{subsection:pa-numerical-largest-degree}

As can be seen from the \autoref{fig:pa-numerical-theoretical-k1}, the numerical value seems to be consistently lower than the theoretical $k_1$ values. This is reasonable since numerical simulations are for finite $N$, and hence there will be an upper limit to the possible degrees that a vertex can take, while in the theoretical derivation, there is no upper limit to the possible values of $k$ that a vertex can have. By looking at the ratios of $ k_1 \text{(numerical)} / k_1 \text{(theoretical)} $ as shown in \autoref{table:pa-numerical-theoretical-ratio}, we can see that deviations are generally constant.

We can estimate the error on $k_1$ by calculating the standard deviation for the sample and using the following formula to estimate population standard deviation:

\begin{equation}
	\sigma = \sqrt{\frac{1}{n} \sum_{i=1}^n (k_{1, i} - \bar{k}_1)^2}
	\label{eq:population-std}
\end{equation}
where $n$ is the number of repeats. In this case, $n = 100$. The values can be seen in Table 2.

\begin{figure}
    \centering
    \includegraphics[height=0.5\linewidth]{img/pa-numerical-theoretical-k1}
    \caption{This shows the difference in $k_1$ for different values of $N$ ranging from $100$ to $10^7$ for preferential attachment. There is a constant offset of the numerical values from the theoretical, indicating a systematic bias instead of statistical fluctuations. }
    \label{fig:pa-numerical-theoretical-k1}
\end{figure}

\begin{center}
\begin{tabular}{ ||c | c | c | c ||}
\hline
N & $k_1^{\text{theory}}$ & $k_1^{\text{numerical}}$ & $k_1^{\text{numerical}} / k_1^{\text{theory}} $\\
\hline
$10^2$ & 45    & 39.3  $\pm$ 0.1 & 0.889 \\
$10^3$ & 141   & 124.9 $\pm$ 0.3 & 0.886 \\
$10^4$ & 447   & 396   $\pm$  1  & 0.887 \\
$10^5$ & 1414  & 1248  $\pm$  3  & 0.883 \\
$10^6$ & 4472  & 3924  $\pm$  9  & 0.878 \\
$10^7$ & 14142 & 12558 $\pm$ 30  & 0.888 \\
\hline
\end{tabular}
\label{table:pa-numerical-theoretical-ratio}
\captionof{table}{This table shows the theoretical and numerical values for the largest expected degree as defined in \autoref{eq:pa-k1-expression} for preferential attachment. The errors on $k_1$ are rounded off to 1 significant figure. }
\end{center}

To produce a data collapse, we need to find the function $f$ such that

\begin{equation}
	p_N(k) = f(k) \mathcal{G}\left ( k / k_1 \right )
	\label{eq:data-collapse}
\end{equation}

To find $f(k)$, we know that in the limit of $N \rightarrow \infty$, $p$ must have no dependence on $L$, and in the large $N$ limit, $p_N(k) = f(k)$ which is just $p_{\infty}(k)$. This implies
\begin{equation}
	\frac{p_N(k)}{p_{\infty}(k)} = G \left ( k / k_1 \right )
\end{equation}
which means that plotting $p_N(k) / p_{\infty}(k)$ against $k / k_1$ will produce a data collapse, as shown in \autoref{fig:pa-data-collapse}.

\begin{figure}
    \centering
    \includegraphics[height=0.5\linewidth]{img/pa-data-collapse}
    \caption{Data collapse of the degree distribution for networks of size $N=10^2, 10^3, 10^4, 10^5, 10^6, 10^7$}
    \label{fig:pa-data-collapse}
\end{figure}


\section{Dress spectral sequence, Leray-Hirsch}\label{section-leray-hirsch}
I think I have to be doing something tomorrow, so no office hours then.
The new pset is up, and there'll be one more problem up.
There are two more things about spectral sequences, and specifically the multiplicative structure, that I have to tell you about.
The construction of the Serre sseq isn't the one that we gave.
He did stuff with simplicial homology, but as you painfully figured out, $\Delta^s\times\Delta^t$ isn't another simplex.
Serre's solution was to not use simplices, but to use cubes.
He defined a new kind of homology using the $n$-cube.
It's more complicated and unpleasant, but he worked it out.
\subsection{Dress' sseq}
Dress made the following variation on this idea, which I think is rather beautiful.
We have a trivial fiber bundle $\Delta^t \to \Delta^s\times\Delta^t\to \Delta^s$.
Let's do with this what we did with homology in the first place.
Dress started with some map $\pi:E\to B$ (not necessarily a fibration), and he thought about the set of maps from $\Delta^s\times\Delta^t\to \Delta^s$ to $\pi:E\to B$.
This set is denoted $\Sin_{s,t}(\pi)$.
This forgets down to $S_s(B)$.
Altogether, this $\Sin_{\ast,\ast}(\pi)$ is a functor $\Deltab^{op}\times\Deltab^{op}\to \set$, forming a ``bisimplicial set''.

The next thing we did was to take the free $R$-module, to get a bisimplicial $R$-module $R\Sin_{\ast,\ast}(\pi)$.
We then passed to chain complexes by forming the alternating sum.
We can do this in two directions here!
(The $s$ is horizontal and $t$ is vertical.)
This gives us a double complex.
We now get a spectral sequence!
I hope it doesn't come as a surprise that you can compute the horizontal -- you can compute the vertical differential first, and then taking the horizontal differential gives the homology of $B$ with coefficients in something.
Oh actually, the totalization $tR\Sin_{\ast,\ast}(\pi) \simeq R\Sin_\ast(E) = S_\ast(E)$.
We'll have
$$
E^2_{s,t} = H_s(B;\text{crazy generalized coefficients}) \Rightarrow H_{s+t}(E)
$$
These coefficients may not even be local since I didn't put any assumptions on $\pi$!
This is like the ``Leray'' sseq, set up without sheaf theory.
If $\pi$ is a fibration, then those crazy generalized coefficients is the local system given by the homology of the fibers.
This gives the Serre sseq.

This has the virtue of being completely natural.
Another virtue is that I can form $\Hom(-,R)$, and this gives rise to a multiplicative double complex.
Remember that the cochains on a space form a DGA, and that's where the cup product comes from.
The same story puts a bigraded multiplication on this double complex, and that's true \emph{on the nose}.
That gives rise a multiplicative cohomology sseq.

This is very nice, but the only drawback is that the paper is in German.
That was item one in my agenda.
\subsection{Leray-Hirsch}
This tells you condition under which you can compute the cohomology of a total space.
Anyway.
We'll see.

Let's suppose I have a fibration $\pi:E\to B$.
For simplicity suppose that $B$ is path connected, so that gives meaning to the fiber $F$ which we'll also assume to be path-connected.
All cohomology is with coefficients in a ring $R$.
I have a sseq
$$
E_2^{s,t} = H^s(B;\underline{H^t F}) \Rightarrow H^{s+t}(E)
$$
If you want assume that $\pi_1(B)$ acts trivially so that that cohomology in local coefficients is just cohomology with coefficients in $H^\ast F$.
I have an algebra map $\pi^\ast:H^\ast(B)\to H^\ast(E)$, making $H^\ast(E)$ into a module over $H^\ast(B)$.
We have $E^{\ast,t}_2 = H^\ast(B;H^t(F))$, and this is a $H^\ast(B)$-module.
That's part of the multiplicative structure, since $E_2^{\ast,0} = H^\ast B$.
This row acts on every other row by that module structure.

Everything in the bottom row is a permanent cycle, i.e., survives to the $E_\infty$-page.
In other words
$$
H^\ast(B) = E^{\ast,0}_2 \fib E^{\ast,0}_3 \fib \cdots \fib E^{\ast,0}_\infty
$$
Each one of these surjections is an algebra map.

What the multiplicative structure is telling us is that $E^{\ast,0}_r$ is a graded algebra acting on $E^{\ast,t}_r$.
Thus, $E^{\ast,t}_\infty$ is a module for $H^\ast(B)$.

Really I should be saying that it's a module for $H^\ast(B;\underline{H^0(F)})$.
Can I guarantee that the $\pi_1(B)$-action on $F$ is trivial.
We know that $F\to \ast$ induces an iso on $H^0$ (that's part of being path-connected).
So if you have a fibration whose fiber is a point, there's no possibility for an action.
This fibration looks the same as far as $H^0$ of the fiber is concerned.
Thus the $\pi_1(B)$-action is trivial on $H^0(F)$, so saying that it's a $H^\ast(B)$-module is fine.

Where were we?
We have module structures all over the place.
In particular, we know that $H^\ast(E)$ is a module over $H^\ast(B)$ as we saw, and also $E^{\ast,t}_\infty$ is a $H^\ast(B)$-module.
These better be compatible!

Define an increasing filtration on $H^\ast(E)$ via $F_t H^n(E) = F^{n-t} H^n(E)$.
For instance, $F_0 H^n(E) = F^n H^n(E)$.
What is that?
In our picture, we have the associated quotients along the diagonal on $E^{s,t}_\infty$ given by $s+t = n$.
In the end, since we know that $F^{n+1} H^n(E) = 0$, it follows that
$$F_0 H^n(E) = F^n H^n(E) = E^{n,0}_\infty = \img(\pi^\ast:H^n(B)\to H^n(E))$$
With respect to this filtration, we have
$$
\gr_t H^\ast(E) = E^{\ast,t}_\infty
$$
I learnt this idea from Dan Quillen.
It's a great idea.
This increasing filtration $F_\ast H^\ast(E)$ is a filtration by $H^\ast(B)$-modules, and $\gr_t H^\ast(E) = E^{\ast,t}_\infty$ is true as $H^\ast B$-modules.
It's exhaustive and bounded below.

This is a great perspective.
Let's use it for something.
Let me give you the Leray-Hirsch theorem.
\begin{theorem}[Leray-Hirsch]\label{leray-hirsch}
    Let $\pi:E\to B$.
    \begin{enumerate}
	\item Suppose $B$ and $F$ are path-connected.
	\item Suppose that $H^t(F)$ is free\footnote{Everything is coefficients in $R$} of finite rank as a $R$-module.
	\item Also suppose that $H^\ast(E)\fib H^\ast(F)$.
    That's a big assumption; it's dual is saying that the homology of the fiber injects into the homology of $E$.
    This is called ``totally non-homologous to zero'' -- this is a great phrase, I don't know who invented it.
    \end{enumerate}
    Pick an $R$-linear surjection $\sigma:H^\ast(F)\to H^\ast(E)$; this defines a map $\overline{\sigma}:H^\ast(B)\otimes_R H^\ast(F)\to H^\ast(E)$ via $\overline{\sigma}(x\otimes y) = \pi^\ast(x)\cup \sigma(y)$.
    This is the $H^\ast(B)$-linear extension.
    Then $\overline{\sigma}$ is an isomorphism.
\end{theorem}
\begin{remark}
    It's not natural since it depends on the choice of $\sigma$.
    It tells you that $H^\ast(E)$ is free as a $H^\ast(B)$-module.
    That's a good thing.
\end{remark}
\begin{proof}
    I'm going to use our Serre sseq
    $$
    E^{s,t}_2 = H^s(B;\underline{H^t F}) \Rightarrow H^{s+t}(E)
    $$
    Our map $H^\ast(E)\to H^\ast(F)$ is an edge homomorphism in the sseq, which means that it factors as $H^\ast(E)\to E^{0,\ast}_2 = H^0(B;\underline{H^\ast(F)}) \subseteq H^\ast(F)$.
    Since $H^\ast(E)\to H^\ast(F)$, we have $H^0(B;\underline{H^\ast(F)}) \simeq H^\ast(F)$.
    Thus the $\pi_1(B)$-action on $F$ is trivial.
    \begin{question}
	What's this arrow $H^\ast(E)\to E^{0,\ast}_2$?
	We have a map $H^\ast(E)\to H^\ast(E)/F^1 = E^{0,\ast}_\infty$.
	This includes into $E^{0,\ast}_2$.
    \end{question}
    Now you know that the $E_2$-term is $H^s(B;H^t(F))$.
    By our assumption on $H^\ast(F)$, this is $H^s(B)\otimes_R H^t(F)$, as algebras.
    What do the differentials look like?
    I can't have differentials coming off of the fiber, because if I did then the restriction map to the fiber wouldn't be surjective, i.e., that $d_r|_{E^{0,\infty}_r} = 0$.
    The differentials on the base are of course zero.
    This proves that $d_r$ is zero on every page by the algebra structure!
    This means that $E_\infty = E_2$, i.e., $E_\infty^{\ast,t} = H^\ast(B)\otimes H^t(F)$.

    Now I can appeal to the filtration stuff that I was talking about, so that $E^{\ast,t}_\infty = \gr_t H^\ast(E)$.
    Let's filter $H^\ast(B)\otimes H^\ast(F)$ by the degree in $H^\ast(F)$, i.e., $F_q = \bigoplus_{t\leq q} H^\ast(B)\otimes H^t(F)$.
    The map $\overline{\sigma}:H^\ast(B)\otimes H^\ast(F)\to H^\ast(E)$ is filtration preserving, and it's an isomorphism on the associated graded.
    This is the identification $H^\ast(B)\otimes H^t(F) = E^{\ast,t}_\infty = \gr_t H^\ast(E)$.
    Since the filtrations are exhaustive and bounded below, we conclude that $\overline{\sigma}$ itself is an isomorphism.
\end{proof}


\documentclass[12pt,pdftex]{article}

\usepackage[usenames,dvipsnames]{color}
\newcommand{\jake}[1]{{\color{blue}\it[JTV: #1]}}
\newcommand{\zeljko}[1]{{\color{ForestGreen}\it[ZI: #1]}}
\newcommand{\problem}[1]{{\color{red}\it[problem: #1]}}

\title{Creating Robust Periodograms}
\author{Jake Vanderplas, \v{Z}eljko Ivezi\'{c}}

\begin{document}

\maketitle

\begin{abstract}
The Lomb-Scargle periodogram is a well-known method of analyzing periodicity in unevenly-sampled time-series data. The problem with the LS approach is that it is not robust to outliers in data. In this paper we propose and compare three approaches to computing robust periodograms: a procedural iterative drop-out approach, a frequentist approach based on robust loss functions, and a Bayesian approach based on marginalization over nuisance parameters.
\end{abstract}

\section{The Plan}
\begin{enumerate}
  \item Derive classic (generalized) Lomb-Scargle
  \item Demonstrate the non-robustness of the classic result
  \item Show the Bayesian formulation \& the robust mixture-model version
  \item Show the M-Estimator robust version (Huber's loss function)
  \item Show the iterative dropout engineering solution
  \item Compare results for robustness, computational complexity, etc.
\end{enumerate}

\section{The Lomb-Scargle Periodogram}
The Lomb-Scargle Periodogram is fundamentally a measure of a normalized chi-squared for fitting a sinusoidal model to data. Given data $\{t_j, y_j\}$ with homoscedastic Gaussian errors $\sigma$, and assuming that the mean was subtracted from ``raw'' data
values $\{y_j\}$, we can choose a single-term linear sinusoidal model defined by the frequency $\omega$ and amplitudes $a$ and $b$:

\begin{equation}
  f(t|\omega, a, b) \equiv a\sin(\omega t) + b\cos(\omega t)
\end{equation}

Given this model, we can write the likelihood

\begin{equation}
\label{eq:dataL}
 L \equiv p(\{y_j\} |~\omega, a, b, \{t_j\}, \sigma) =
  \prod_{j=1}^{N} \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(
  \frac{-[y_j - f(t_j|a, b, \omega)]^2}{2\sigma^2} \right)
\end{equation}

For any choice of $\omega$, we can find values $[a_0(\omega), b_0(\omega)]$ which maximize this likelihood. The goodness-of-fit can be determined by evaluating $\chi^2$ at this maximum:

\begin{equation}
  \chi^2(\omega) = \frac{1}{\sigma^2}\sum_{j=1}^N[y_j - f(t|\omega, a_0(\omega), b_0(\omega))]^2
\end{equation}

If we define $\chi_0^2 = \sigma^{-2}\sum y_j^2$, then we can write the {\it Lomb-Scargle Periodogram} as

\begin{equation}
\label{eq:PLS}
  P_{LS}(\omega) \equiv 1 - \frac{\chi^2(\omega)}{\chi_0^2}
\end{equation}

This periodogram is a normalized measure of the goodness-of-fit of a sinusoidal model with frequency $\omega$, as compared to the null hypothesis of a pure-noise constant model, and lies in the range $0 \le P_{LS} \le 1$.

\subsection{Computing $P_{LS}$}
We can compute $P_{LS}$ quite easily via the above formalism.
For later convenience, let's re-express our model in the form of a matrix-vector product:
\begin{equation}
  f(t|\omega, \theta) = X_\omega \theta,
\end{equation}
where in the simple case above, $\theta = [a, b]^T$ and
\begin{equation}
  X_\omega = \left[\begin{array}{lll}
    \sin\omega t_1 && \cos\omega t_1\\
    \sin\omega t_2 && \cos\omega t_2\\
     & \vdots &\\
    \sin\omega t_N && \cos\omega t_N
  \end{array}\right]
\end{equation}
Letting $y = [y_1, y_2\cdots y_N]^T$ be the vector of amplitudes, the $\chi^2$ expression can be concisely written
\begin{equation}
\chi^2(\omega, \theta) = \frac{1}{\sigma^2}||y - X_\omega \theta||^2
\end{equation}
Assuming a fixed $\omega$, this $\chi^2$ can be minimized by standard means to find the best-fit parameters:
\begin{equation}
\label{eq:thetaML}
  \theta_0(\omega) = (X_\omega^TX_\omega)^{-1}X_\omega^Ty.
\end{equation}
Plugging this result back in to the expression for $\chi^2$ gives
\begin{equation}
\label{eq:chi2}
  \chi^2(\omega) = \frac{1}{\sigma^2}\left[
    y^Ty - y^TX_\omega(X_\omega^TX_\omega)^{-1}X_\omega^Ty
    \right]
\end{equation}
The reference model (with $X=0$) gives $\chi_0^2 = \sigma^{-2}y^Ty$, so we see that
\begin{equation}
  P_{LS}(\omega) = \frac{y^TX_\omega(X_\omega^TX_\omega)^{-1}X_\omega^Ty}{y^Ty}
\end{equation}
In the standard treatment, $P_{LS}$ is defined as some recipe of products of sines and cosines (e.g. Scargle 1982; Zechmeister \& Kurster 2009, ICVG2014): all of that is contained in the above expression, and we won't repeat it here. Further, it is possible to use some tricks involving fast Fourier transforms to quickly compute the above expression for many frequencies $\omega$, but we'll not get into those algorithmic considerations here.

The important point here is that fundamentally, the Lomb-Scargle periodogram is simply a normalized measure of the $\chi^2$ for a {\bf maximum-likelihood model fit} of a particular single-frequency periodic model. Further, it's clear that by changing the definition of $X_\omega$ and adding more columns, we can quite easily account for an arbitrary offset $\mu$ (as in the {\it floating-mean Lomb-Scargle method} proposed by Zechmeister \& Kurster 2009), include non-uniform errors $\sigma_j$, compute the periodogram for arbitrary multi-harmonic and multi-frequency models, etc. For detailed discussion, see Section 10.3 in ICVG2014.

\jake{Show an example periodogram \& folded light curve here}

\jake{Mention generalizations to multiple Fourier terms: Bretthorst 2003; Zechmeister \& Kurstur 2009. These are included in our formalism here!}

\jake{This all needs to be slightly modified for heteroscedastic errors. Should we do this from the beginning, or mention it separately?}

\subsection{Bayesian View of the Lomb-Scargle Periodogram}
\jake{Mention Bretthorst's work in this area}

The Bayesian view of this also starts with the likelihood.
We'll specify this in terms of $\omega$, which is our model for a given frequency $\omega$. In this case, the likelihood is:

\begin{equation}
  p(D|\omega,\theta) =
  (2\pi\sigma^2)^{-N/2} \exp\left(
  \frac{-||y - X_\omega\theta||^2}{2\sigma^2}
  \right)
\end{equation}
applying Bayes' rule and marginalizing over $\theta$ gives:
\begin{equation}
  p(\omega|D) = \frac{1}{p(D)}\int p(D|\omega,\theta)p(\omega, \theta){\rm d}^N\theta
\end{equation}
If we assume uniform priors (i.e. $p(\omega, \theta) \propto 1$) then it can be shown (see Appendix) that the Bayesian odds ratio between the model $\omega$ and the null model $M_0$ is given by:

\begin{equation}
  O_\omega \equiv \frac{p(\omega|D)}{p(M_0|D)} \propto \exp\left(\frac{\chi^2(\omega)}{2\sigma^2}\right) \propto \exp\left(\frac{\chi_0^2}{2\sigma^2}P_{LS}(\omega)\right)
\end{equation}

This is our Bayesian alternative to the frequentist Lomb-Scargle periodogram.
Maximizing $O_\omega$ across multiple values of $\omega$ gives us the best period in the Bayesian sense.
The beauty here is that, unlike the Lomb-Scargle result which is predicated on $\chi^2$ as a goodness-of-fit, our Bayesian model allows the insertion of any likelihood, including ones which may be robust to outliers. We'll consider this below.

\jake{should we consider floating-mean LS from the beginning? in this case, the null model is not just $||y||^2$, but is $||y - \mu||^2$ where we solve for $\mu$: that is, $X_0 = [1, 1, \cdots 1]^T$. Nothing else needs to be modified, which is the nice part of this linear algebra approach!}

\section{Non-Robustness of the Classic Lomb-Scargle}

\jake{Show a figure here, perhaps using LINEAR sample 1004849}

\section{Robust Periodogram: Huber Loss}
{\it Follow Huber 1981 and use a general $M$-estimator corresponding to the Huber Loss (Huber 1963).}

Huber (1981) proposed the $M$-estimator, which minimizes the generalized loss function
\begin{equation}
  \sum_i \rho(y_i|y).
\end{equation}
In the special case of the standard maximum-likelihood this loss function is proportional to the log of the likelihood:
\begin{equation}
  \rho(y_i|y) = \frac{(y_i - y)^2}{2\sigma_i^2}.
\end{equation}
This makes it clear that any outliers contribute quadratically to the loss, which is why they will have such a large effect on the fit.
One way to address this is to soften these tails for distant points. Huber (1963) proposed a softer loss function, usually known as the {\it Huber Loss}:
\begin{equation}
  \rho_H(y_i|y,c) = \left\{
  \begin{array}{ll}
    \frac{1}{2}t^2; & |t| \le c \\
    c|t| - \frac{1}{2}c^2; & |t| > c
  \end{array}
  \right.
\end{equation}
where we've defined $t_i \equiv (y_i - y) / \sigma_i$. Here $c$ is a free parameter which gives the cutoff (in units of $\sigma$) beyond which the loss function turns over.

If we replace the standard $\chi^2$ goodness-of-fit with an adjusted goodness-of-fit based on this Huber loss, the result is a more robust estimate of the periodogram. Unlike the standard solution above, there is no closed-form version of this. Instead we must do this in a two-step process:

\begin{equation}
  \theta_0(\omega, c) = \arg\min_\theta \sum_i\rho_H(y_i|X_\omega\theta, c)
\end{equation}

Once this is computed, the loss is simply $\sum\rho_H(y_i|X_\omega\theta_0,c)$, and the robust periodogram is

\begin{equation}
  P_{LS,H}(\omega|c) = 1 - \frac{\sum_i\rho_H(y_i|X_\omega\theta_0,c)}{\sum_i\rho_H(y_i|0,c)}
\end{equation}

\section{Robust Periodogram: Iterative Drop-outs}
ad-hoc procedural solution. Fit a model, drop outliers, repeat until it converges.

\jake{I just realized an issue: the drop-outs will be different for each $\omega$! That is, for the wrong $\omega$ the $\chi^2$ should be very large. Should there be a limit on the number of dropped points?}

\section{Robust Periodogram based on Bayesian Approach}

A well-known problem with the Lomb-Scargle periodogram is its lack of robustness to outliers: the Gaussian form of the likelihood expression means that if the errors $\sigma_j$ are mis-specified, the outlying point(s) might have a large effect on the final fit.
What is required is to replace the above $\chi^2$ computation with a robust model that can account for these errors.

If we knew which points were outliers, then we would simply exclude them and
apply standard Gaussian results to the remaining points (assuming that outliers
represent a small fraction of the data set). We will assume that we do not have this
information. Bayesian analysis enables a formal treatment of this problem, as well
as the ability to estimate which points are likely outliers using an objective framework.

First, given $\{t_j, y_j, \sigma_j\}$, how do we assess whether non-Gaussianity is important?
In case of no outliers, we expect that
\begin{equation}
           \chi^2_{\rm dof} = {1 \over N-1} \chi^2(\omega_0) \approx 1,
\end{equation}
where $\chi^2(\omega_0)$ is given by eq.~\ref{eq:chi2}, and evaluated at $\omega=\omega_0$ which
minimizes its value. If $\chi^2_{\rm dof}-1$ is a few times larger than $\sqrt{2/(N-1)}$, then it is unlikely
(as given by the cumulative pdf for $\chi^2_{\rm dof}$ distribution) that our data set $\{t_j, y_j\}$ was
drawn from a distribution specified by the chosen model and Gaussian error distribution with
the claimed $\{\sigma_j\}$.

\subsection{Bayesian Periodogram}

We start by reformulating the data likelihood from eq.~\ref{eq:dataL}  as
\begin{eqnarray}
\label{eq:dataL2}
    p(\{y_j, g_j\} |~\omega, \theta, \{t_j, \sigma_j\}) = \nonumber \\
  \prod_{j=1}^{N} \left[ \frac{g_j}{\sqrt{2\pi\sigma_j^2}} \exp\left(
  \frac{-[y_j - f(t_j|\omega, \theta)]^2}{2\sigma_j^2}\right) +
       (1-g_j) p_{\rm bad}(y_j|I) \right].
\end{eqnarray}
Here $g_j$ is 1 if the data point is ``good'' and 0 if it came from the distribution
of outliers, $p_{\rm bad}(y_j|I)$. In this model $p_{\rm bad}(y_j|I)$ applies to all
outliers. Again, if we knew $g_j$ this would be an easy problem to solve.

Since $\{g_j\}$ represent hidden variables, we shall treat them as model parameters and then
marginalize over them to get $p(\theta|\{y_j, t_j, \sigma_j\} ,I)$. With a separable prior,
which implies that the reliability of the measurements is decoupled from the true value
of the quantity we are measuring,
\begin{equation}
        p(\theta,\{g_j\}|I) = p(\theta|I)  \, p(\{g_j\}|I),
\end{equation}
we get
\begin{equation}
   p(\theta,\{g_j\}|~\{y_j, t_j, \sigma_j\}, I) \propto \prod_{j=1}^{N} \left[ g_j p_{\rm good}(y_j)   + (1-g_j) p_{\rm bad}(y_j|I) \right] p(\{g_j\}|I),
\end{equation}
where we assumed uniform priors for parameters $\theta$ and introduced for notational simplicity
\begin{equation}
       p_{\rm good}(y_j) = \frac{1}{\sqrt{2\pi\sigma_j^2}} \exp\left(
                  \frac{-[y_j - f(t_j|\omega, \theta)]^2}{2\sigma_j^2}\right).
\end{equation}
Finally, marginalizing over $g_j$ gives
\begin{equation}
  p(\theta|~\{y_j, t_j, \sigma_j\}, I) \propto \int  p(\theta,\{g_j\}|~\{y_j, t_j, \sigma_j\}, I) \, d^N g_j.
\end{equation}

Following Section 5.6.7 in ICVG2014, in case of uniform priors for all $g_j$, marginalization over
$g_j$ effectively replaces every $g_j$ by 1/2 and leads to
\begin{equation}
\label{eq:Btheta}
   p(\theta|~\{y_j, t_j, \sigma_j\}, I) \propto \prod_{j=1}^{N} \left[ p_{\rm good}(y_j)  + p_{\rm bad}(y_j|I) \right].
\end{equation}

\jake{Show the expression for $p(\omega|~\{y_j, t_j, \sigma_j\}, I)$; this is what we compute via MCMC}

\section{Discussion}
\begin{enumerate}
  \item compare the three approaches
  \item compare results on several LINEAR curves
  \item discuss computational issues
\end{enumerate}

\subsection{QA and Visualization}

An obvious plot is to compare Lomb-Scargle and Bomb-Scargle periodograms in the same figure.

One could make a 2D plot where the x axis is $\omega$ and the y axis is the $j$, the data point index.
Each ($\omega, j$) pixel gets colored by its MAP value of $g_j$.

\section{Conclusion}
This is what we did

\section*{References}

\appendix
\section{Deriving the Bayesian Expression}
Here is the calculation of the Bayes factor for our model.

For separable priors $p(\omega,\theta) = p(\omega)p(\theta)$ the posterior for our linear model is:

\begin{equation}
  p(\omega|D) = \frac{p(\omega)}{p(D)}\int{\rm d}^N\theta(2\pi\sigma^2)^{-N/2}\exp\left(\frac{-||y - X_\omega\theta||^2}{2\sigma^2}\right)
\end{equation}

We can compute this integral by completing the square in $\theta$. Let's look at the argument of the exponent:

\begin{equation}
  ||y - X_\omega\theta||^2 = \theta^TX_\omega^TX_\omega\theta - 2\theta^TX_\omega^Ty + y^Ty
\end{equation}

If we now define the hermitian matrix $C = X_\omega^TX_\omega$ and find its Cholesky decomposition $U^TU = C$, then we can rewrite this as

\begin{equation}
  ||y - X_\omega\theta||^2 = ||v - U\theta||^2 + y^Ty - v^Tv
\end{equation}

where the length-N array $v$ satisfies $U^Tv = X_\omega^Ty$. Given this, we can rewrite the expression

\begin{equation}
  ||y - X_\omega\theta||^2 = ||v - U\theta||^2 + y^Ty - y^TX_\omega C^{-1}X_\omega^Ty
\end{equation}

The expression $\phi = v - U\theta$ is now an $\mathbf{R}^N\to\mathbf{R}^N$ mapping with a Jacobian given by $U$, so we can change variables to $\phi$ in the above integral to simplify its evaluation:

\begin{equation}
  p(\omega|D) = \frac{p(\omega)}{p(D)}(2\pi\sigma^2)^{-N/2}
\int\frac{{\rm d}^N\phi}{\det|U|}\exp\left(\frac{-||\phi||^2 - y^Ty + y^TX_\omega(X_\omega^TX_\omega)^{-1}X_\omega^Ty}{2\sigma^2}\right)
\end{equation}

Observing that $\det|U|^2 = \det|C| = \det|X_\omega^TX_\omega|$ and evaluating the (now straightforward) integral gives

\begin{equation}
  p(\omega|D) = \frac{p(\omega)}{p(D)}\frac{1}{\sqrt{\det|X_\omega^TX_\omega|}}
  \exp\left(\frac{y^TX_\omega(X_\omega^TX_\omega)^{-1}X_\omega^Ty - y^Ty}{2\sigma^2}\right)
\end{equation}

Recalling that

\begin{equation}
  P_{LS}(\omega) = \frac{y^TX_\omega(X_\omega^TX_\omega)^{-1}X_\omega^Ty}{y^Ty}
\end{equation}

we see that we can express this

\begin{equation}
  p(\omega|D) = \frac{p(\omega)}{p(D)}\frac{1}{\sqrt{\det|X_\omega^TX_\omega|}}
  \exp\left(\frac{y^Ty(P_{LS}(\omega) - 1)}{2\sigma^2}\right)
\end{equation}

We can compare two models using the odds ratio:
\begin{equation}
  O_{\omega_1\omega_2} \equiv \frac{p(M_{\omega_2}|D)}{p(M_{\omega_1}|D)}
  =\frac{p(M_{\omega_2})}{p(M_{\omega_1})}\sqrt{\frac{\det|X_{\omega_1}^TX_{\omega_1}|}{\det|X_{\omega_2}^TX_{\omega_2}|}}\exp\left(\frac{y^Ty}{2\sigma^2}\left[P_{LS}(\omega_2) - P_{LS}(\omega_1)\right]\right)
\end{equation}

For the special case of $M_0$, where $X = 0$, we can show
\begin{equation}
\frac{p(M_{\omega}|D)}{p(M_0|D)} =
\frac{p(M_{\omega})}{p(M_0)}\sqrt{\frac{2\pi\sigma^2}{\det|X_\omega^TX_\omega|}}\exp\left(\frac{y^Ty}{2\sigma^2}\left[P_{LS}(\omega) - 1\right]\right)
\end{equation}

\end{document}


\documentclass[12pt, titlepage, oneside]{article}

\input{settings}

\begin{document}

	\textbf{ELECENG 3TQ3}\\
	\textbf{Elston A.}

\section{Lecture 5}

\subsection{Bernoulli Trials}

From the last example in Lecture 4, we talked about repeated trails with a coin toss.

Consider tossing a coin 5 times, recording the outcomes, and repeating the whole process 10 times. We can say that one outcome would lead to heads and the other outcome would lead to tails. This is in a way a "success" and "failure" type experiment. We would like to know how to deal with such an event.

Lets continue to a probably more cooler scenario. Consider Intel's manufacturing process in which 100 CPUs are selected for testing.
Assume that the probability of a CPU manufactured at that location failing the test is 0.02. What is the probability that of the 100 CPUs tested that k will pass the test.

The answer is simply
\begin{align}
{100\choose n} (0.98)^n (0.02)^{100-n}
\end{align}
Since we do not care about which order the CPUs failed or passed, we need to only consider the combination rather than the permutation. We know that $(0.98)^n(0.02)^{100-n}$ is the probability that if you picked a single CPU if it would pass, but since we are picking 100 CPUs we need to multiply by the total combination to see the actual pass rate.

\subsection{Sub Experiment}

Instead of two outcomes, we may have multiple outcomes for every trial shown as $s_1,s_2,\dots,s_m$. We denote the probability of outcome $s_i$ as $p_i$. If we have $n$ trials, then the sum of all the $m$ outcomes during an experiment $n_1 + n_2 + \dots + n_m = n$. The probability of the outcome is given as
\begin{align}
{n \choose n_1,n_2,\dots,n_m} p_1^{n_1} p_2^{n_2} \dots p_m^{n_m}
\end{align}
The coefficient in the bracket is called the multinomial coefficient and represent permutation with repetition. This is because we can have multiple outcomes being the same.

\b{note}: The multinomial coefficient is calculated as follows
\begin{align}
{n \choose n_1, n_2, \dots, n_m} = \frac{n!}{n_1! n_2! \dots n_m!}
\end{align}

\ex Let us consider the set $\{a,a,b,b\}$. To final all the distinct permutations of this set, we can see it is as follows:
${aabb,abab,abba,baab,baba,bbaa}$ so the answer is $6$ NOT $4!=24$. To find the distinct permutations, we need to divide the number of permutations that are considered as repetitions
\begin{align}
{4! \choose 2,2} = \frac{4!}{2!2!} = 6
\end{align}
\ex How many 7 digit numbers can you create using $0,0,0,0,1,2,3$ if we ignore numbers that begin with zero
\begin{align}
{7 \choose 4} = 7!/4! = 210
\end{align}

\subsection{Reliability}
Deals with the problem in which we want to find the probability that a particular system consisting of multiple components is all working properly.

The two main types to consider are series and parallel systems. Series systems needs every component to work whereas parallel needs at least one component to work.

We \b{ALWAYS} consider the probabilities to be independent unless specified otherwise

\ex
\begin{figure}[h]
	\centering
	\includegraphics[width=0.7\linewidth]{../Lecture4/images/series}
	\caption{Series systems}
	\label{fig:series}
\end{figure}

Given the probability of each component $W_i$ working is $P(W_i)$.

In this case we see the probability of the system working would be
\begin{align}
P[W_1W_2W_3] = P[W_1]P[W_2]P[W_3]
\end{align}
The probability of failure would be
\begin{align}
P[W_1 \u W_2 \u W_3] = 1-P[W_1]P[W_2]P[W_3]
\end{align}
We can see this is a direct result from our talks about independent in the previous lecture.

\ex
\begin{figure}[h]
	\centering
	\includegraphics[width=0.4\linewidth]{../Lecture4/images/parallel}
	\caption{Parallel systems}
	\label{fig:parallel}
\end{figure}

With the same notation as the previous example, we can see the probability of this system working is given as follows:
\begin{align}
&P[W_1 \u W2 \u W3] = P[W_1] + P[W_2 \u W_3] - P[W_1 \n (W_2 \u W_3)]\\
&= P[W_1] + P[W_2] + P[W_3] - P[W_1\u W_2] - P[W_1 \u W_3] - P[W_2 \u W_3] + P[W_1 \u W_2 \u W_3]
\end{align}
This way is also too tedious and wastes too much time. Since we know the events are independent, we can use the following to get the same result
\begin{align}
P[W_1 \u W_2 \u W_3] = 1- P[W_1 W_2 W_3] = 1-P[W_1]P[W_2]P[W_3]
\end{align}
\end{document}


%!TEX root = ../TTK18-Summary.tex
\section{Sum of squares programming}
SoS decomposition can prove non-negativity of polynomials. This can be used for control analysis and design for polynomial nonlinear systems.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%% SUBSECTION %%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Polynomials and monomials}
A (multivariate) polynomial is
%
\begin{equation}
  f(x) = \sum_k c_k
  \underbrace{x_1^{a_{k1}} \dots x_n^{a_{kn}}}_{\text{a monomial}},\quad a_{ki} \in \mathbb{Z},
\end{equation}
%
and a monomial is one term in a polynomial, without the coefficient:
%
\begin{equation}
  m_k(x) = x_1^{a_{k1}} \dots x_n^{a_{kn}}.
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%% SUBSECTION %%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Sum of squares decomposition}
A \emph{polynomial} $f(x)$ is a SoS if it can be written as a sum of squares (lol):
%
\begin{equation}\label{eq:sos-decomp}
  f(x) =
  \sum_{i=1}^N h_i^2(x) =
  \sum_{i=1}^N \left(q_i\tp v(x)\right)^2 =
  v\tp(x) Q v(x)
\end{equation}
%
where $v(x)$ is a vector of monomials, and $Q \geq 0$.

A symmetric \emph{polynomial matrix} $M(x)$ is an SoS matrix if it can be written
%
\begin{equation}
  M(x) = H\tp(x) H(x)
\end{equation}
%
for some polynomial matrix $H(x)$.

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Existence of SoS decomposition}
An SoS decomposition is a sufficient condition for non-negativity of a polynomial. Thus, some non-negative polynomials don't have an SoS decomposition.

The existence of an SoS decomposition is only guaranteed for these nonnegative polynomials:
%
\begin{itemize}
  \item in two variables,
  \item in quadratic forms, or
  \item in three variables and of fourth order.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Uniqueness of SoS decompositions}
The matrix $Q$ in \eqref{eq:sos-decomp} and thus the SoS decomposition is not unique, because the elements of $v(x)$ are not independent:
%
\begin{equation}
  \underbrace{2x_1^4 + 2x_1^3x_2 - x_1^2x_2^2 + 5x_2^4}_{f(x)}
  =
  \underbrace{\bmat{x_1^2 \\ x_2^2 \\ x_1x_2}\tp}_{v(x)\tp}
  \underbrace{\bmat{2 & 0 & 1 \\ 0 & 5 & 0 \\ 1 & 0 & -1}}_{Q_1}
  \underbrace{\bmat{x_1^2 \\ x_2^2 \\ x_1x_2}}_{v(x)}
\end{equation}
%
and
%
\begin{equation}
  \underbrace{2x_1^4 + 2x_1^3x_2 - x_1^2x_2^2 + 5x_2^4}_{f(x)}
  =
  \underbrace{\bmat{x_1^2 \\ x_2^2 \\ x_1x_2}\tp}_{v(x)\tp}
  \underbrace{\bmat{2 & -\lambda & 1 \\ -\lambda & 5 & 0 \\ 1 & 0 & 2\lambda -1}}_{Q_2}
  \underbrace{\bmat{x_1^2 \\ x_2^2 \\ x_1x_2}}_{v(x)}
\end{equation}

We have $\det(Q_1) = -15$, and therefore $Q_1 \ngeq 0$. Therefore, $Q_1$ does not imply that $f(x)$ is a SoS polynomial. However, we can find a $\lambda$ in $Q_2$ that gives $Q_2 \geq 0$, such as $\lambda = 1 \Rightarrow \det(Q_2) = 4 \Rightarrow Q_2 \geq 0$. Thus $f(x)$ is a SoS polynomial after all.

It turns out that in general the set of parameters that make a symmetric matrix $Q \geq 0$ is convex when it depends linearly on its parameters, and thus SoS constraints are convex.

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Solving}
SoS constraints are usually convex, and can be formulated as a convex semi-definite program. Can be solved with e.g. YALMIP or SOSTOOLS.

Sometimes $Q$ is very ill-conditioned, with tiny eigenvalues. Can be improved by
\begin{itemize}
  \item removing unused monomials from $v(x)$
  \item making $Q$ block-diagonal,
  \item altering $Q$ based on an initial solution,
  \item using software that pre- and post-processes automatically.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Validity of solution}
Primal SoS optimization always yields a P.D. $Q$, but the solution may be so inaccurate that the polynomial is still not SoS. With
%
\begin{gather}
  \begin{split}
    \lambda_{\min}(Q) &= \min \eig(Q) \\
    v(x) &\in \mathbb{R}^M \\
    r &= f(x) - v\tp(x) Q v(x)
  \end{split}
\end{gather}
%
we can guarantee positivity of $f(x)$ given
%
\begin{equation}
  \begin{split}
    \lambda_{\min}(Q) &\geq 0 \\
    \lambda_{\min}(Q) &\geq M \cdot \abs(r)
  \end{split}
\end{equation}

Sometimes, a dual solver will find a valid solution that the primal does not find.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%% SUBSECTION %%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{LMIs and SoS}
SoS problems are a special case of LMIs, but only some LMI ``tricks'' can be applied to SoSs:

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Linearizing change of variables}
If a problem depends linearly on $P>0$ and $PK$ (no standalone $K$s), then we can use $L = PK$ to make the problem linear in $L$ and $P$.

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Congruence transform}
For a matrix SoS with $M(x) \geq 0$ and a full rank $W(x)$:
\begin{equation}
  M(x) \geq 0 \quad \Leftrightarrow \quad W\tp(x) M(x) W(x) \geq 0
\end{equation}
Note: Cannot be used with scalarized Schur complement.

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Scalarization}
For an LMI, positive definiteness has the relation
%
\begin{equation}
  L > 0 \Leftrightarrow x\tp L x > 0 \quad \forall x
\end{equation}
%
while for an SoS it is instead
%
\begin{equation}
  M(x) > 0 \Leftrightarrow z\tp M(x) z > 0 \quad \forall x, \forall z.
\end{equation}
%
As it turns out, it's better to formulate scalar-valued SoS problems.

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Scalarized Schur complement}
With
%
\begin{equation}
  M(x) = \bmat{E(x) & F\tp(x) \\ F(x) & P(x)},\quad P(x) \mbox{ symmetric and invertible}
\end{equation}
%
then
%
\begin{equation}
  \bmat{x \\ z}\tp M(x) \bmat{x \\ z} > 0 \quad \forall\{x,z\} \neq \{0,0\}
\end{equation}
%
is equivalent to (all matrices are functions of $x$)
%
\begin{equation}
  \begin{split}
    x\tp \left( E - F\tp P\inv F \right) x &> 0 \quad \forall x \neq 0 \\
    z\tp P z &> 0 \quad \forall z \neq 0, \forall x
  \end{split}
\end{equation}
%
because of
%
\begin{equation}
  M(x) =
  \bmat{I & F\tp P\inv \\ 0 & I}
  \bmat{E - F\tp P\inv F & 0 \\ 0 & P}
  \bmat{I & 0 \\ P\inv F & I}.
\end{equation}

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Scalarized Schur product}
If we have
%
\begin{equation}
  \bmat{x \\ w} =
  \bmat{I & 0 \\ P\inv(x) F(x) & I}
  \bmat{x \\ z}
\end{equation}
%
then $w$ can take any value regardless of $x$, by choosing $z$. The same applies in reverse: $z$ can take any value regardless of $x$, by choosing $w$.

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{S-procedure}
We want to prove $f(x) > 0$ when $g(x) < 0$. Can be written
%
\begin{equation}
  f(x) + s(x) g(x) > 0
\end{equation}
%
for an arbitrary SoS polynomial $s(x)$. With this formulation, optimizing over parameters in $s(x)$ and $g(x)$ gives a bilinear (nonconvex) problem.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%% SUBSECTION %%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Bilinear discrete systems}
The bilinear\footnote{Bilinear because the next state is a function with a term that is a product of the state and the input. In a linear system, the $B_x$ term would not be present.} system
%
\begin{equation}
  x_{k+1} = A x_k + \sum_{i=1}^m (B_i x_k + b_i) u_{i,k} = A x_k + (B_x + B) u_k
\end{equation}
%
is stable under state feedback if
%
\begin{gather}
  x_k\tp P x_k - x_{k+1}\tp P x_{k+1} > 0 \\
  x_k\tp P x_k
  -
  \big(
    A x_k + (B_x + B) u_k(x_k)
  \big)\tp
  P
  \big(
    A x_k + (B_x + B) u_k(x_k)
  \big)
  > 0
\end{gather}

If the system is open-loop unstable, $u_k(x_k)$ must be a ratio of same-order polynomials to achieve global quadratic stability, such as:
%
\begin{equation}
  u_k(x_k) = \frac{C(x_k) x_k}{c_0(x_k) + 1}
\end{equation}
%
where $C(x_k)$ is a polynomial matrix and $c_0(x_k)$ is an SoS polynomial.

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Region of convergence}
With a quadratic LF $V(x_k) = x_k\tp P x_k$, a polynomial matrix $C(x_k)$, and SoS polynomials $c_0(x_k)$, $s_1(x_k,z)$, then the closed loop system is stable for all $x_k | x_k\tp P x_k < \gamma$ given
%
\begin{equation}
  \bmat{x_k \\ z}\tp
  M(x)
  \bmat{x_k \\ z}
  -
  s_1(x_k,z) (\gamma - x_k\tp P x_k) > 0
\end{equation}
%
where (omitting arguments for brevity)
%
\begin{equation}\label{eq:polynomial-matrix-convergence}
  M(x_k) =
  \begin{bmatrix}
    (c_0 + 1) P & \big( (c_0 + 1)A + (B_x + B)C \big)\tp P \\
    P \big( (c_0 + 1)A + (B_x + B)C \big) & (c_0 + 1)P
  \end{bmatrix}
\end{equation}
%
which comes from $(c_0 + 1)$ being strictly positive, and using the scalarized Schur complement and the S-procedure.

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Input saturation}
The input constraints
%
\begin{equation}
  -u_{i,\max} \leq u_i \leq u_{i,\max}
\end{equation}
%
are satisfied for all $x_k | x_k\tp P x_k < \gamma$ given
\begin{equation}
  \begin{bmatrix}
    \big( c_0(x_k) + 1 \big)u_{i,\max}^2 - q_i(x_k)(\gamma - x_k\tp P x_k) & c_i(x_k) \\
    c_i(x_k) & c_0(x_k) + 1
  \end{bmatrix}
  > 0
\end{equation}
%
where
%
$c_i(x_k)$ are rows of $C(x_k)$, and $c_0(x_k)$ and $q_i(x_k)$ are SoS polynomials.

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Rate of convergence}
We can change $M(x)$ from \eqref{eq:polynomial-matrix-convergence} to
\begin{equation}
  M(x_k) =
  \begin{bmatrix}
    (1 - \alpha)(c_0 + 1) P & \big( (c_0 + 1)A + (B_x + B)C \big)\tp P \\
    P \big( (c_0 + 1)A + (B_x + B)C \big) & (c_0 + 1)P
  \end{bmatrix}
\end{equation}
to guarantee an exponential convergence rate defined by $\alpha$. There will be a tradeoff between maximizing the region of convergence and rate of convergence. Using this $M(x)$ allows us to determine the tradeoff.

%%%%%%%%%%%%%%%%%%%%%% SUBSUBSECTION %%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Optimization formulation}
\begin{enumerate}
  \item If $\alpha$, $\gamma$, and $P$ are known, then $C$, $c_0$, $q_i$, $s_1$ enter linearly into inequalities above, and they can be solved as is.
  \item If $\gamma$ and $P$ are known, we can find $C$, $c_0$, $q_i$, $s_1$ by formulating a feasibility problem (optimization without objective).
  \item If $C$, $c_0$, $q_i$, and $s_1$ are known, we can maximize $\gamma$ with $P$ as a free variable. Must normalize $P$, i.e. by a constraint $\trace(P) = k$, for some constant $k$.
  \item Can iterate between the two points directly above to create a controller with gradually larger region of convergence.
\end{enumerate}

TTK18 Optimaliseringsbasert reguleringsdesign og analyse/tex/sec-sos.tex


\section{First compute cost tests}

For now plots are with SSE and the San Miguel scene (16,852,353 Vertices and 5,600,315 Triangles)

The function that is optimised:

\[totalTime = memoryTime * m + computeTime * n \]

while $m$ is the amount of memory batches we load (one batch is 4 in SSE or 8 in AVX) and $n$ the amount of batches we compute.

An example with SSE and a N8 with the data we collected from San Miguel:
\[0.665495 = memoryTime * 8  + computeTime * 8 \]
\[0.749434 = memoryTime * 12 + computeTime * 8 \]
\[0.825299 = memoryTime * 16 + computeTime * 8 \]
\[0.943323 = memoryTime * 20 + computeTime * 8 \]

After a linear least squares approximation it results in $memoryTime = 0.0226$  and $computationTime = 0.0612$. For the graphs below the values are normalized.

Im not sure if it is correct to multiply with the computeTime with 8 and the MemoryTime with 8. The reason i choose it like this is to have a reasonable way to do the calculation with configurations like N2, N3, N5 ...

\newpage

\pgfplotsset{
	every axis/.append style={colorbar = false},
}
\begin{minipage}[t]{0.8\textwidth}
\begin{tikzpicture}
\pgfplotsset{ymin=0, ymax=1}
\begin{axis}[
ybar stacked,
legend style={at={(0.99,0.2)}, anchor = north east},
xlabel = \leafs,
ylabel = factor,
]

\addplot+[ybar]table[y = leafComputeCostNorm, x = leafSize, col sep=comma]{Data/sanMiguelSSESeqMemoryLeafComputeCostTable.txt};
\addplot+[ybar]table[y = memoryCostNorm, x = leafSize, col sep=comma]{Data/sanMiguelSSESeqMemoryLeafComputeCostTable.txt};
\legend{\strut triangle compute time, \strut triangle memory load time}
\end{axis}
\end{tikzpicture}
\end{minipage}

\begin{minipage}[t]{0.8\textwidth}
\begin{tikzpicture}
\pgfplotsset{ymin=0, ymax=1}
\begin{axis}[
ybar stacked,
legend style={at={(0.99,0.2)}, anchor = north east},
xlabel = \nodes,
ylabel = factor,
]

\addplot+[ybar]table[y = nodeComputeCostNorm, x = branchFactor, col sep=comma]{Data/sanMiguelSSESeqMemoryNodeComputeCostTable.txt};
\addplot+[ybar]table[y = memoryCostNorm, x = branchFactor, col sep=comma]{Data/sanMiguelSSESeqMemoryNodeComputeCostTable.txt};
\legend{\strut node compute time, \strut node memory load time}
\end{axis}
\end{tikzpicture}
\end{minipage}
\newpage
This plot shows the computed time spent on computation or loading memory. The results of the approximation are multiplied with the m and n of the function. (for an example of N3 this would lead to n3 and m=4 since we always pad the memory to multiples of 4 for SSE)

\begin{minipage}[t]{0.8\textwidth}
\begin{tikzpicture}
\begin{axis}
[
%view={90}{0} for x, view={0}{0} for y restriction
%view={90}{0},
xlabel = \leafs,
ylabel = time in seconds,
cycle list name=exotic,
legend style={at={(0.25,0.95)}, anchor = north west},
xtick = {2, 3, ..., 16},
%xticklabels={$8$,$16$,$24$,$32$,$40$,$48$,$56$,$64$,$72$,$80$,$88$,$96$,$104$,$112$,$120$,$128$},
]
\addplot+[thick, mark=none]table[y expr = \thisrowno{4} *\thisrowno{2} , x = branchFactor, col sep=comma]{Data/sanMiguelSSESeqMemoryNodeComputeCostTable.txt};
\addplot+[thick, mark=none]table[y expr = \thisrowno{3} *\thisrowno{0} , x = branchFactor, col sep=comma]{Data/sanMiguelSSESeqMemoryNodeComputeCostTable.txt};
%\addplot+[thick, mark=none]table[y = nodeComputeCost, x = branchFactor, col sep=comma]{Data/sanMiguelSSESeqMemoryNodeComputeCostTable.txt};
%\addplot+[thick, mark=none]table[y = memoryCost, x = branchFactor, col sep=comma]{Data/sanMiguelSSESeqMemoryNodeComputeCostTable.txt};
\legend{node memory loading time, node computation time}
\end{axis}
\end{tikzpicture}
\end{minipage}

\newpage
Relative memory cost:

\begin{minipage}[t]{0.8\textwidth}
	\begin{tikzpicture}
	\begin{axis}
	[
	%view={90}{0} for x, view={0}{0} for y restriction
	%view={90}{0},
	xlabel = \leafs,
	ylabel = ,
	cycle list name=exotic,
	legend style={at={(0.05,1.05)}, anchor = north west},
	xtick = {2, 3, ..., 16},
	%xticklabels={$8$,$16$,$24$,$32$,$40$,$48$,$56$,$64$,$72$,$80$,$88$,$96$,$104$,$112$,$120$,$128$},
	]
	\addplot+[thick, mark=none]table[y = memoryRelative, x = branchFactor, col sep=comma]{Data/sanMiguelSSESeqMemoryNodeComputeCostTable.txt};

	\legend{ memory factor in relation to compute cost}
	\end{axis}
	\end{tikzpicture}
\end{minipage}


\section{Buckley Leverett Equations with/without Gravity}

\subsection{Nomenclature}
\begin{tabular}{c l}
%\hline
%
$f_{j}$: & fractional flow of phase $j$ \\
%
$f_{ij}$: & $\partial f_{i}/\partial S_{j}$ \\
%
$g$: & gravitational acceleration constant \\
%
$j$: & phase \\
%
$k$: & absolute permeability of the porous medium in 1D \\
%
$k_{rj}$:& relative permeability of phase $j$ \\
%
$k^{o}_{rj}$: & endpoint relative permeability of phase $j$ \\
%
$n_{j}$: & Corey exponent of phase $j$\\
%
$n_{p}$: & number of phases present\\
%
$S_{j}$: & saturation of phase $j$\\
%
$S_{j}^{\star}$: & normalized saturation of phase $j$\\
%
$S_{rj}$: & residual saturation of phase $j$\\
%
$v$: & flow velocity in 1D\\
%
$\mu_{j}$: & viscosity of phase $j$\\
%
$\rho_{mj}$: & mass density of phase $j$\\
%
$\lambda$: & eigenvalue (characteristic velocity)\\
%
$\Lambda$: & shock velocity\\
%
$\theta$: & dip angle of porous medium from horizontal\\
%
%\hline
\end{tabular}

\subsection{Equations}

Using the notation and derivation of \cite{Orr_2007}, the conservation equations for purely convective one-dimensional flow of two immiscible phases is

\begin{eqnarray}
&&\frac{{\partial S_j }}{{\partial \tau }} +  \frac{{\partial f_j }}{{\partial \xi }}  = 0 \label{eqn:BL}\\
&&j = g,w,o \nonumber
\end{eqnarray}

\noindent where dimensionless time $\tau = \frac{v t}{\phi L}$ and dimensionless distance $\xi = \frac{x}{L}$ .  For the two-phase water/gas system only one of these equations is independent as $S_g + S_w = 1$.  All variables are defined in the Nomenclature section.

The fractional flow of a component is defined as

\begin{eqnarray}
&&f_j = \frac{\frac{k_{rj}}{\mu_j}}{\sum_{n=1}^{n_p}{\frac{k_{rn}}{\mu_n}}} \left( 1-\frac{kg \sin \theta}{v} \sum_{n=1}^{n_p}{\frac{k_{rn}}{\mu_n}\left( \rho_{mj}-\rho_{mn} \right)} \right) \label{eqn:ff}
\end{eqnarray}

\noindent where

\begin{eqnarray}
&&k_{rj} = k_{rj}^o {\left(S_j^*\right)^{n_j}}\\
&&S_j^* = \frac{S_j - S_{rj}}{1-\sum_{n=1}^{n_p}{S_{rn}}} \nonumber
\end{eqnarray}

\noindent which can be written entirely in terms of $S_g$ for the Buckley-Leverett problem using $S_w = 1-S_g$.  Equation \ref{eqn:BL} can be solved as an eigenvalue problem by ...

The exact analytical form for the derivative of the fractional flow is

\begin{eqnarray}
\frac{df_g}{dS_g}&=& \frac{\left(\frac{\frac{d k_{rg}}{d S_g}}{\mu_g}\right) \frac{k_{rw}}{\mu_w}-\left(\frac{\frac{d k_{rw}}{d S_g}}{\mu_w}\right)\frac{k_{rg}}{\mu_g}}{\left(\frac{k_{rg}}{\mu_g}+\frac{k_{rw}}{\mu_w}\right)^2}\left( 1-\frac{kg \sin \theta}{v} \frac{k_{rw}}{\mu_w}\left( \rho_{mg}-\rho_{mw} \right) \right)\nonumber \\
&&-f_g\frac{kg \sin \theta}{v} \frac{\frac{d k_{rw}}{d S_g}}{\mu_w}\left(\rho_{mg}-\rho_{mw} \right)
\end{eqnarray}

\noindent where

\begin{eqnarray}
\frac{d k_{rg}}{d S_g} &=& k_{rg}^o n_g \frac{\left(S_g - S_{rg}\right)^{n_g-1}}{\left(1-S_{rg}-S_{rw}\right)^{n_g}}\nonumber \\
\frac{d k_{rw}}{d S_g}&=& -k_{rw}^o n_w \frac{\left(1-S_g - S_{rw}\right)^{n_w-1}}{\left(1-S_{rg}-S_{rw}\right)^{n_w}}\nonumber
\end{eqnarray}

\subsection{Parameters in example problems}
All parameters in Equation \ref{eqn:BL} are assumed to be constant in the Buckley-Leverett problem.  The fractional flow parameters are given in Table \ref{table:paras}.

\begin{table}[h]
\begin{center}
\begin{tabular}{lrrrr}
\hline
Phase&$S_{rj}$&$k_{rj}^o$&$n_j$& $\mu_j$($cp$)\\
%%\lcline{1-1}\rlcline{2-5}
Gaseous&0.1&1.0&2.0&0.1\\
%Oleic&-&-&-&-\\
Aqueous&0.3&1.0&2.0&1.0\\
\hline
\end{tabular}
\caption{Fractional flow parameters for the benchmark Cases I-III.}
\label{table:paras}
\end{center}
\end{table}

\subsection{Solutions}

\subsubsection{Case I: Solution without gravity}
The first benchmark solution is for the case of no gravity, i.e. a horizontal displacement.  The initial (right state) and injection (left state) conditions are given in Table \ref{table:BCs}.

\begin{table}[h]
\begin{center}
\begin{tabular}{lrr}
\hline
BC & $S_g$ & $S_w$\\
%%\lcline{1-1}\rlcline{2-3}
Initial& 0.0 & 1.0\\
(right state)&  & \\
Injection&1.0 & 0.0\\
(left state)&  & \\
\hline
\end{tabular}
\caption{Boundary conditions for benchmark Cases I-III. }
\label{table:BCs}
\end{center}
\end{table}

\begin{figure}[h]
\centering
%\includegraphics[width=0.75\textwidth,viewport=10 10 420 420,clip]{./figures/BL_no_gravity}
\includegraphics[width=10.0cm,height=15.cm]{./figures/BL_no_gravity}
\vspace{-4.cm}
\caption{Top: Comparison the analytical solutions to Case I without gravity with the upstream weighted DG simulated solution with for 3 (green), 10 (blue) and 50 (red) grid blocks.  Bottom: Comparison the analytical solutions to Case I without gravity with the mixed weighted DG simulated solution for 3 (green), 10 (blue) and 50 (red) grid blocks.}
\label{fig:BL_no_grav}
\end{figure}

The displacement consists of a fast moving shock wave from the initial condition, with no gas, to a gas saturation of $S_g = 0.3255$, followed by a rarefaction wave to the residual water saturation. As there is no transfer of components between phases, it is impossible to move the residual water from the system.

The DG simulator was run in 2 modes, upstream weighting and 80\% upstream weighting.  A comparison of the analytical solution for various levels of refinement is shown for upstream weighting on the top of Figure \ref{fig:BL_no_grav} and for mixed weighting on the bottom.

\paragraph{Displacements with gravity}

In this case the flux in the conservation law is slightly different.  The fractional flow curve as computed from Eq. \ref{eqn:ff} may be larger than one or less than zero, which is physically meaningless in one dimension as $f_g$ is defined as the fraction of the total flow that is taken up by the gas phase.  The impossible fluxes are caused by the inability of a 1D model to capture counter-current gravity-driven flow.  A 2D benchmark case to study this effect will be dealt with later.  Anytime Eq. \ref{eqn:ff} gives $f_g > 1.0$ it must be that in 1D $f_g = 1.0$, similarly if $f_g < 0.0$ in Eq. \ref{eqn:ff}, $f_g = 0.0$.  The additional parameters necessary for the cases including gravity are in Table \ref{table:grav_const}.  The phase densities are calculated using the method from \cite{spycher_2003}.

\begin{table}[h]
\begin{center}
\begin{tabular}{lr}
\hline
$\rho_g$ ($\frac{kg}{m^3}$)& 710 \\
%$\rho_o$ ($\frac{kg}{m^3}$)&-\\
$\rho_w$ ($\frac{kg}{m^3}$)& 1050\\
g ($\frac{m^2}{s}$)& 9.8 \\
k ($ m^2$)& $9.87E-13$\\
v ($ \frac{m}{s}$)& $5.00E-06$\\
\hline
Case II &\\
$\theta$ (degrees) & -90 \\
\hline
Case III &\\
$\theta$  (degrees)& 90  \\
\hline
\end{tabular}
\caption{Additional parameters needed for benchmark Cases II and III with gravity. Phase densities for water and super-critical CO$_2$ are at 80$^oC$ and 27$MPa$. }
\label{table:grav_const}
\end{center}
\end{table}

\paragraph{Case II: Displacements with gravity slowing displacement}

The first benchmark solution including gravity is the injection of oil into the bottom of a vertical core or sand pack. In this case analytical solution in the presence of gravity is physically meaningful, though there is a small region at low $S_g$ where $f_g < 0.0$.  The solution structure is the same as in the displacement without gravity.  Because the more dense oil is being pushed into the core against the force of gravity the leading shock front and rarefaction wave have both slowed down substantially.

\begin{figure}[h]
\vspace{-1cm}
\centering
\includegraphics[width=15.0cm,height=20.cm]{./figures/BL_gravity_up}
%\includegraphics[width=0.75\textwidth,viewport=10 0 390 300,clip]{./figures/BL_gravity_up}
\vspace{-5.5cm}
\caption{Analytical solution to Case II, the stable benchmark case with gravity (black) slowing the displacement.  A fine-grid FD solution (red) and the analytical solution for the case without gravity (blue) are shown for comparison purposes.}
\label{fig:BL_grav_up}
\end{figure}

\paragraph{Case III: Displacements with gravity accelerating displacement}

The second benchmark solution including gravity is the injection of oil into the top of a vertical core or sand pack. In reality this displacement is gravity unstable and needs to be modelled in two-dimensions, but for the purposes of benchmarking the code it is still useful.

The solution with gravity is a shock wave from the right state to the saturation $S_g = 0.2960$ followed by a rarefaction wave to $S_g = 0.3340$.  Then there is a constant state to the injection composition.  This constant state is caused by the fact that $f_g > 1.0$ for large gas saturations and no longer tapers off uniformly as the residual water saturation is approached.

The analytical solution with gravity is shown in Figure \ref{fig:BL_grav_down} with a fine-grid FD solution and the analytical solution without gravity for comparison.

\begin{figure}[h]
\vspace{-1cm}
\centering
\includegraphics[width=15.0cm,height=20.cm]{./figures/BL_gravity_down}
\vspace{-5.5cm}
%\includegraphics[width=0.75\textwidth,viewport=10 0 390 300,clip]{./figures/BL_gravity_down}
\caption{Analytical solution to Case III, the unstable benchmark case with gravity (black) accelerating the displacement.  A fine-grid FD solution (red) and the analytical solution for the case without gravity (blue) are shown for comparison purposes.}
\label{fig:BL_grav_down}
\end{figure}