% -----------------------------*- LaTeX -*------------------------------
\documentclass[12pt]{report}
\usepackage{scribe_hgen486}
\usepackage{tikz}
\usetikzlibrary{bayesnet}
\usepackage{hyperref}
\begin{document}

\lecturer{John Novembre}		% optional
\scribe{Matt Bonakdarpour}		% required
\lecturenumber{2}			% required, must be a number
\lecturedate{January 7}		        % required, omit year

\maketitle

\framebox[.95\textwidth]{\parbox{.93\textwidth}{ {{\bf Note:}} These
lecture notes are still rough, and have only have been mildly
proofread.  }}
\vspace*{.1in}
\setlength{\parindent}{0cm}

% ----------------------------------------------------------------------
\section{Graphical Models: Introduction}
Quite often, we are confronted with the task of understanding a complex system
with many dependent components. In this context, probability theory
gives us a framework to reason about a collection of possible outcomes
and their associated likelihoods. For example, if we are trying to
diagnose a patient, there are many potential diseases this patient
could have. Also, associated with each patient, we may have hundreds
of data points related to their symptoms, personal traits, and
diagnostic tests. Each of these characteristics could be thought of as
a random variable. Our goal is to reason about this patient given the
values of one or more of these random variables. In the framework of
probabilistic reasoning, we need to construct a joint distribution
over the space of possible assignments to this set of random
variables.

If we have five realizations of \emph{binary} random variables for a patient, we must specify a joint
distribution over $2^5$ possible values -- a potentially daunting task! Graphical models provide a
framework to compactly encode a joint distribution in high
dimensions. They also provide other benefits, including but not
limited to:
\begin{itemize}
\item A tool for visualizing the structure of a probabilistic model
\item Provides insights, such as conditional independence properties, by inspection of the graph
\item Complex computations can be expressed as operations on the graph
\end{itemize}

In this lecture, we will consider directed graphical models, sometimes
known as Bayesian networks.

\subsection{Joint probabilities and independence}
Using the definition of conditional probability, we have that the
joint distribution between $X_i$ and $X_j$ is:
\begin{align*}
p(X_i,X_j) = P(X_i | X_j)P(X_j)
\end{align*}

In general, if we have any $n$ random variables $X_1,\ldots,X_n$, we can
use the \emph{chain rule} to factor the joint distribution as follows:
\begin{align*}
p(X_1,\ldots,X_n) = p(X_1)p(X_2|X_1)p(X_3|X_1,X_2)\cdots
p(X_n|X_1,\ldots ,X_{n-1})
\end{align*}

Recall that $X_i$ is independent of $X_j$ if knowledge of $X_j$ does
not change our knowledge of $X_i$. That is, $p(X_i | X_j) =
p(X_i)$. Plugging this in to the expression for the joint distribution
above, we have that $X_i$ is independent of $X_j$ if and only if
$p(X_i,X_j)=p(X_i)p(X_j)$.

We say that $X_i$ is \emph{conditionally independent} of $X_j$ given
$X_k$ if
\begin{align*}
p(X_i,X_j|X_k) = p(X_i|X_k)p(X_j|X_k)
\end{align*}
which means
\begin{align*}
p(X_i,X_j,X_k) = p(X_i|X_k)p(X_j|X_k)p(X_k)
\end{align*}

Notice how the joint distribution factorizes nicely when we have
conditional independence. Idea: when faced with a large number of
features, use our prior knowledge of independence to simplify the
joint distribution.

\subsection{Some basics}
A directed graph is a set of vertices (or nodes) along with a set of
directed edges (or arrows) between nodes. Here is a simple graph with
two vertices and one directed edge:

\begin{center}
\begin{tikzpicture}

  % Define nodes
  \node[latent]                             (x1) {$X_1$};
  \node[latent,right=of x1]                 (x2) {$X_2$};

  \edge {x1} {x2} ; %
\end{tikzpicture}
\end{center}

Here, each vertex represents a random variable. When a specific random
variable in the graph is observed, we shade that particular
vertex. For example, this is what the graph above would look like if
we observed $X_1$, but $X_2$ was still latent:

\begin{center}
\begin{tikzpicture}

  % Define nodes
  \node[obs]                             (x1) {$X_1$};
  \node[latent,right=of x1]                 (x2) {$X_2$};

  \edge {x1} {x2} ; %
\end{tikzpicture}
\end{center}

A directed graphical model encodes conditional independence in the
following way. A specific random variable (or vertex) is conditionally
independent of all other nodes, given its parents (i.e. all nodes that
points to it). For example, in the following graph $X_1$ is
independent of $X_3$ given $X_2$:

\begin{center}
\begin{tikzpicture}

  % Define nodes
  \node[obs]                                (x2) {$X_2$};
  \node[latent,below left=of x2]                 (x1) {$X_1$};
  \node[latent,below right=of x2, right=of x1]                 (x3) {$X_3$};

  \edge {x2} {x1} ; %
  \edge {x2} {x3} ; %
\end{tikzpicture}
\end{center}
The general form for the joint distribution for a directed graphical
model is therefore:
\begin{align}
p(X_1,\ldots,X_n) = \prod_{i=1}^n p(X_i |\text{pa}_i) \label{eq:gm_prob}
\end{align}
where $\text{pa}_i$ denotes the parents of $X_i$.
\subsection{A motivating example}
Assume we have 3 possible genotypes $\{$AA,Aa,aa$\}$. To each person, we will assign a
random variable $X_i$ according to their genotype:
\begin{center}
\begin{tabular}{l|l }
  Genotype & $X_i$ \\
  \hline
  AA & 0 \\
  Aa & 1 \\
  aa & 2
\end{tabular}
\end{center}
Let's also assume that the genotype of each person is binomially
distributed, so that:
\begin{align*}
X_i|p &\sim \text{Bin}(2,p) \\
p(X_i=2|p) &= p^2 \\
p(X_i=1|p) &= 2p(1-p) \\
p(X_i=0|p) &= (1-p)^2
\end{align*}
In this context, we represent inheritance in a 3-generation
pedigree with the following graphical model. In what follows, we're
assuming $X_8 = p$.

\begin{center}
\begin{tikzpicture}

  % Define nodes
  \node[obs]                               (x8) {$X_8$};
  \node[latent,below left=of x8, right of=x1]                 (x2)
  {$X_2$};
  \node[latent,below left =of x8, left=of x2]                 (x1) {$X_1$};
  \node[latent,below right=of x8, right=of x2]                 (x3) {$X_3$};
  \node[latent,below right=of x8, right=of x3]                 (x4)
  {$X_4$};

  \node[latent,below right=of x1, below left=of x2]                 (x5) {$X_5$};
  \node[latent,below right=of x3, below left=of x4]                 (x6)
  {$X_6$};

  \node[latent,below right=of x5, below left=of x6]                 (x7) {$X_7$};

  % Connect the nodes
  \edge {x8} {x1,x2,x3,x4} ; %
  \edge {x1,x2} {x5} ; %
  \edge {x3,x4} {x6} ; %
  \edge {x5,x6} {x7} ; %
\end{tikzpicture}
\end{center}

Assume $X_8$ is observed and that we are interested in
$p(X_5|X_1,X_2)$. We could represent this conditional probability as a
table, enumerating all possibly quantities that $X_1,X_2$ and $X_5$
take. In general, we can represent $P(X_i | X_j,X_k)$ where $X_i$ is
the child of $X_j$ and $X_k$ using the basic laws of Mendelism:
\begin{center}
\begin{tabular}{|c c|c c c|}
\hline
Father & Mother & \multicolumn{3}{c|}{Child ($X_i$)} \\
$X_j$ & $X_k$ &           0 &           1 &           2 \\
\hline
0     & 0     &           1 &           0 &           0 \\
0     & 1     & $\frac{1}{2}$ & $\frac{1}{2}$ &           0 \\
0     & 2     &           0 &           1 &           0 \\
1     & 0     & $\frac{1}{2}$ & $\frac{1}{2}$ &           0 \\
1     & 1     & $\frac{1}{4}$ & $\frac{1}{2}$ & $\frac{1}{4}$ \\
1     & 2     &           0 & $\frac{1}{2}$ & $\frac{1}{2}$ \\
2     & 0     &           0 &           1 &           0 \\
2     & 1     &           0 & $\frac{1}{2}$ & $\frac{1}{2}$ \\
2     & 2     &           0 &           0 &           1 \\
\hline
\end{tabular}
\end{center}

From the graphical model above, and using equation~\ref{eq:gm_prob},
we can factorize the joint distribution in the following way, in which
the values of the table above can be used to compute the joint distribution:
\begin{align*}
p(X_1,\ldots,X_8) = p(X_1|X_8)p(X_2|X_8)p(X_3|X_8)p(X_4|X_8)\times \\p(X_5|X_1,X_2)p(X_6|X_3,X_4)p(X_7|X_5,X_6)p(X_8)
\end{align*}

\subsection{Some classes of 3-node graphs}

With the linear chain graph, we have that $X_1 \perp X_3 | X_2$:

\begin{center}
\begin{tikzpicture}

  % Define nodes
  \node[obs]                                (x2) {$X_2$};
  \node[latent,left=of x2]                 (x1) {$X_1$};
  \node[latent,right=of x2]                 (x3) {$X_3$};

  \edge {x1} {x2} ; %
  \edge {x2} {x3} ; %
\end{tikzpicture}
\end{center}

As we saw in a previous section, in this multiple offspring graph, we
have that $X_1 \perp~ X_3 | X_2$:

\begin{center}
\begin{tikzpicture}

  % Define nodes
  \node[obs]                                (x2) {$X_2$};
  \node[latent,below=of x2]                 (x1) {$X_1$};
  \node[latent,right=of x1]                 (x3) {$X_3$};

  \edge {x2} {x1} ; %
  \edge {x2} {x3} ; %
\end{tikzpicture}
\end{center}

The v-structure graph has the following structure:
\begin{center}
\begin{tikzpicture}

  % Define nodes
  \node[obs]                                (x2) {$X_2$};
  \node[latent,above=of x2]                 (x1) {$X_1$};
  \node[latent,right=of x1]                 (x3) {$X_3$};

  \edge {x1} {x2} ; %
  \edge {x3} {x2} ; %
\end{tikzpicture}
\end{center}
In this case, it is \emph{not true} that $X_1 \perp X_3 | X_2$. For example, assume
$X_2$ was the event that your house alarm was going off, $X_1$ was the
event that there was an earthquake, and $X_3$ was the event that there
was a burglar in your house. Assuming your alarm was going off, the
additional information about whether a burglar was in your house would
affect your knowledge of whether or not an earthquake was
happening.

\subsection{D-separation}
Take any undirected path (ignoring arrows) in the graph $G$. This path
is called an \emph{active trail} for observed variables $O \subset
\{X_1,\ldots,X_n\}$, if for every consecutive triple of variables
$X,Y,Z$ on the path:
\begin{itemize}
\item $X \rightarrow Y \rightarrow Z$ and $Y\not\in O$
\item $X \leftarrow Y \leftarrow Z$ and $Y\not\in O$
\item $X \leftarrow Y \rightarrow Z$ and $Y\not\in O$
\item $X\rightarrow Y \leftarrow Z$ and $Y$ or any of $Y$'s
  descendants are in $O$
\end{itemize}

Any two variables $X_i$ and $X_j$ for which there does not exist an
active trail for observations $O$ are called \emph{d-separated} by
$O$, written d-sep($X_i;X_j|O$). Two sets of vertices $A$ and $B$ are
d-separated by $O$ if d-sep($X,Y|O$) for all $X\in A$ and $Y\in B$.

The key result is the following: if $X$ and $Y$ are d-separated by a
set $O$, then $X \perp Y | O$. In words, $X$ is conditionally
independent of $Y$ given $O$ if there does not exist any active trail
between $X$ and $Y$ for observations $O$. Intuition: active trails allow the
dependencies to flow.

\subsection{Elimination Algorithm}
Assume we have a graph $G$, and we take two disjoint subsets of
nodes $X_E$ and $X_Q$. Our goal is to calculate $p(X_Q|X_E)$. In this
section, we will focus on the case when $X_Q$ is one node called the
\emph{query node}, and the set of nodes $X_E$ are called
\emph{evidence nodes}.

Assume we have the following graph:
\begin{center}
\begin{tikzpicture}

  % Define nodes
  \node[latent]                              (x1) {$X_1$};
  \node[latent, above right=of x1]              (x2) {$X_2$};
  \node[latent, below right=of x3, below=of x2]                              (x3) {$X_3$};
  \node[latent, above right=of x2]                              (x4) {$X_4$};
  \node[latent, right=of x3]                              (x5) {$X_5$};
  \node[obs, above right=of x5, below=of x4, right=of x5]                              (x6) {$X_6$};

  \edge {x1} {x2} ; %
  \edge {x1} {x3} ; %
  \edge {x2} {x4} ; %
  \edge {x2} {x6} ; %
  \edge {x3} {x5} ; %
  \edge {x5} {x6} ; %
\end{tikzpicture}
\end{center}
Let our evidence set be $\{X_6\}$ and our query set be $\{X_1\}$. We
want to compute:
\begin{align*}
p(X_1|X_6) = \frac{p(X_1,X_6)}{p(X_6)} = \frac{p(X_1,X_6)}{\sum_{x_1}p(X_1=x_1,X_6)}
\end{align*}
The elimination algorithm provides an effective computational method
for making these calculations.

From the expression above, it seems that to compute the conditional distribution, we just need to be able to
compute the joint distribution $p(X_1,X_6)$. We could start off by
marginalizing the full joint distribution for all variables:
\begin{align*}
p(X_1,X_6) = \sum_{x_2}\sum_{x_3}\sum_{x_4}\sum_{x_5}p(X_1,X_2,\ldots,X_6)
\end{align*}
In this case,if every $X_i$ can take on one of $k$ values, this expression would
have $k^4$ terms. However, using the conditional independence
relations from the graph, we get:
\begin{align*}
p(X_1,X_6) &=
\sum_{x_2}\sum_{x_3}\sum_{x_4}\sum_{x_5}p(X_1)p(X_2|X_1)p(X_3|X_1)p(X_4|X_2)p(X_5|X_3)p(X_6|X_2,X_5)
\\
&=p(X_1)p\sum_{x_2}(X_2|X_1)\sum_{x_3}p(X_3|X_1)\sum_{x_4}p(X_4|X_2)\sum_{x_5}p(X_5|X_3)p(X_6|X_2,X_5)
\end{align*}
We can further simplify this summation by introducing the following
notation. Let $m_i(x_{S_i})$ denote the expression that arises when
performing the sum $\sum_{x_i}$ where $x_{S_i}$ are the variables,
other than $x_i$ that appear in the summand. Employing this notation,
we get:
\begin{align*}
p(X_1,X_6) &=
p(X_1)p\sum_{x_2}(X_2|X_1)\sum_{x_3}p(X_3|X_1)\underbrace{\sum_{x_4}p(X_4|X_2)}_{m_4(x_2)}\underbrace{\sum_{x_5}p(X_5|X_3)p(X_6|X_2,X_5)}_{m_5(x_2,x_3)}
\\
&=
p(X_1)p\sum_{x_2}(X_2|X_1)m_4(x_2)\underbrace{\sum_{x_3}p(X_3|X_1)m_5(x_2,x_3)}_{m_3(x_1,x_2)}
= p(X_1)p\sum_{x_2}(X_2|X_1)m_4(x_2)m_3(x_1,x_2) \\
&= p(x_1)m_2(x_1) \\
\end{align*}
Using this result, we get the desired conditional probability:
\begin{align*}
\Rightarrow p(X_1|X_6) &= \frac{p(x_1)m_2(x_1)}{\sum_{x_1}p(x_1)m_2(x_1)}
\end{align*}
This exercise gives rise to a general algorithm for computing marginal
probabilities. To see the details of this algorithm, refer to the
handout given in class entitled \emph{The Elimination Algorithm}.

In statistical genetics, Felsenstein's tree-pruning algorithm (1981, JME) computes
the likelihood of an evolutionary tree from nucleic acid sequence
data. This pruning algorithm was an early instance of the elimination
algorithm. As a simplified illustration, we can represent a phylogeny
as a tree in which the leaves correspond to the \emph{observed} values
of a site across different species. The non-leaves are values of the site
for the ancestors. Each edge of the tree can represent the
\emph{evolutionary distance} to be estimated which can be encoded as the
conditional probability of a state given its ancestral state. Using
Felsenstein's pruning algorithm, we can obtain the joint probability
across all sites. This can be used in conjunction with the EM
algorithm to find the maximum likelihood tree.

Relevant online course notes:
\begin{itemize}
\item \url{http://www.inf.ed.ac.uk/teaching/courses/pmr/slides/elim-2x2.pdf}
\item \url{https://www.cs.cmu.edu/~aarti/Class/10701/readings/graphical_model\_Jordan.pdf}
\item \url{http://www.cs.columbia.edu/~blei/fogm/2015F/notes/inference.pdf}
\item \url{http://www.cs.berkeley.edu/~jordan/courses/281A-fall04/}
\end{itemize}
For drawing graphical models
\begin{itemize}
\item \url{https://github.com/jluttine/tikz-bayesnet}
\end{itemize}

\end{document}


\documentclass[a4paper,10pt]{article}
%\pdfoutput=1
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage{graphicx}
\usepackage[a4paper,margin=2.5cm]{geometry}
%\usepackage{palatino}
\usepackage[math]{kurier}
%\usepackage[sc]{mathpazo} % consider options: osf, sc
\usepackage{xspace}
%\PassOptionsToPackage{latex2html}{hyperref}
%\usepackage{html}
\usepackage{url}
%\usepackage{booktabs}
%\usepackage{hyperref}
%\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsfonts}
\usepackage{rotating}
%\usepackage{srctex}
\usepackage{color}
\usepackage{enumerate}
%\usepackage{afterpage}

\usepackage{arydshln}
\usepackage{etoolbox}
\usepackage{blkarray}
\usepackage{multirow}

\usepackage[normalem]{ulem}

\usepackage{listings}
\usepackage[vlined]{algorithm2e}
\usepackage{marginnote}
\reversemarginpar

\usepackage{fancyvrb}
\RecustomVerbatimEnvironment{Verbatim}{Verbatim}{xleftmargin=5mm}

\graphicspath{{Figures/}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{mylstset}
\usepackage{mystyle}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newif\ifshow
%\showtrue
\showfalse
\usepackage{exercises}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{tikz}
\usetikzlibrary{arrows,automata}
\usepackage[all]{xy}

\newcommand{\row}[1]{
%HEVEA \begin{comment}
\ensuremath \overrightarrow{\mathbf #1}
%HEVEA \end{comment}
%HEVEA \overrightarrow{\mathbf #1}
}

\graphicspath{{Figures/}}

\title{%
\begin{flushleft}
  DM561 -- Linear Algebra and Applications\\
  DM562 -- Scientific Programming\\[0.3cm]
{\Large Obligatory Assignment 5, Autumn 2018} %
% \ahref{\url{http://www.imada.sdu.dk/~marco/DM559/Assignments-LA/sheet2.pdf}}{\tiny [pdf format]}
\\
\hrulefill
\\[-1.8cm]
\end{flushleft}
}

%\hrulefill \\[-1.8cm]
\author{}
\date{}

\begin{document}

\maketitle

\begin{solution}
Contains Solutions!
\end{solution}

\color{red}In red the modifications after publication.\color{black}

\begin{center}
%\color{red}
%  {\bf Submission Deadline: Friday, December 21, 2018, at noon}
\marginnote{---}\sout{{\bf Submission Deadline: Wednesday, January 9, 2019, at noon}}

\color{red}
  {\bf Submission Deadline: Monday, January 14, 2019, at noon}
\color{black}
  %\color{black}
\end{center}

In the period between Christmas and New Years Eve, issues with the
automatic grading or other type of assistance is not guaranteed.

\medskip
In this assignment it is not allowed to use the module Networkx.

\medskip
This document is associated with the files \texttt{asg5.py} and the data files:\\
\texttt{web\_stanford.txt}, \texttt{psh-uefa-2017-2018.csv}, which are available
in the git repository. The file \texttt{asg5.py} is the only one that needs to be
edited and submitted.

\medskip
\color{red}
\marginnote{---}On line 38 of \texttt{asg5.py} you must change \lstinline{linsolve} into
\lstinline{eigensolve}.
\color{black}

\section*{Task 1}

\subparagraph{Subtask a.}\label{subtask-a.}

Write a class for representing directed graphs via their adjacency
matrices. The constructor should accept an \(n\times n\) adjacency
matrix \(A\) and a list of node labels (such as \([\)a, b, c, d\(]\))
defaulting to \texttt{None}. Modify \(A\) into \(\widetilde{A}\) as
shown in the slides so that there are no sinks in the corresponding
graph, then calculate the \(\widehat{A}\):

\[
\widehat{A}_{ij} = \frac{\widetilde{A}_{ij}}{\sum_{k=1}\widetilde{A}_{kj}}.
\]

Save \(\widehat{A}\) and the list of labels as attributes of the DiGraph
objects. Use
\([0,1,\ldots,n-1]\) as the labels if none are provided. Finally, raise
a \texttt{ValueError} if the number of labels is not equal to the number
of nodes in the graph.~(Hint: use array broadcasting to compute
\(\widehat{A}\) efficiently. The Numpy function \texttt{where} could
help you.)

In the docstring example you find the same graph that was used in
slides.

\paragraph{Subtask b.}\label{subtask-b.}

Add the following methods to your class from the previous Subtask. Each
should accept a damping factor \(\epsilon\) (defaulting to 0.85),
compute the PageRank vector \(\mathbf{x}\), and return a dictionary
mapping label \(i\) to its PageRank value \(x_i\).

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
  \texttt{linsolve()}: solve for \(\mathbf{x}\) in slide 32 after
  substituting the fact that
  \(\mathbf{1}\mathbf{1}^{\mathsf{T}}\mathbf{x}=\mathbf{1}\) since
  \(\sum_i x_t(i)=1\):

  \[
      \begin{array}{rl}
      \left(I -\epsilon\widehat{A}-(1-\epsilon)\frac{1}{n}\mathbf{1}\mathbf{1}^{\mathsf{T}}\right)\mathbf{x}&=\mathbf{0}\\
      \left(I - \epsilon\widehat{A}\right)\mathbf{x}&=\frac{1-\epsilon}{n}\mathbf{1}
      \end{array}
  \]
\item
  \texttt{eigensolve()}: solve \(\bar{A}\mathbf{x} = \mathbf{x}\) from
  slide 33 for \(\mathbf{x}\). Normalize the resulting eigenvector so that
  its entries sum to \(1\).
\item
  \texttt{itersolve()}: in addition to \(\epsilon\), accept an
  integer \texttt{maxiter} and a float \texttt{tol}. Iterate until\\
  $||\mathbf{x}_t - \mathbf{x}_{t-1}||_1 < \texttt{tol}$ or
  $t \textgreater{} $ \texttt{maxiter}.
  Use \(\mathbf{x}_{0}=[\frac{1}{n},\frac{1}{n},\ldots,\frac{1}{n}]^{\mathsf{T}}\)
  as the initial vector (any positive vector that sums to \(1\) will do,
  but this assumes equal starting probabilities).
\end{enumerate}

In the docstring example you can test that the three methods yield the
same results. For the graph of the lecture with \(\epsilon=0.85\), you
should get the following dictionary mapping labels to PageRank values.

\begin{lstlisting}
{'a': 0.095758635, 'b': 0.274158285, 'c': 0.355924792, 'd': 0.274158285}
\end{lstlisting}

\subparagraph{Subtask c.}\label{subtask-c.}

Write a function that accepts a dictionary mapping labels to PageRank
values, like the outputs in the previous Subtask and returns a list of
labels sorted \textbf{from highest to lowest} rank.~(Hint: look for the
built-in function \texttt{sorted()}.)

For the graph in the slides with \(\epsilon=0.85\), the list is \([\)c,
b, d, a\(]\) (or \([\)c, d, b, a\(]\), since b and d have the same
PageRank value).

\color{red}
\marginnote{---}Round the steady state probabilities to the 8th decimal
and break ties by lexicographic order (for example, the ranking for the
graph in the slides with \(\epsilon=0.85\) must be \([\)c, b, d, a\(]\)).
\color{black}

Looking at the graph from the slides, it is easy to see why a has the
lowest PageRank value: the only other node that points to it is b. It
also makes sense that c has the highest ranking, since c and d both have
edges from the other three nodes pointing to them, but d only has one
edge (pointing to c), while c points to both b and d. In other words, at
each step d distributes all of its importance to c, while c splits its
importance between b and d.

Of course, constructing rankings is much more difficult to do by hand
when there are more than just a few nodes in the graph.

\section*{Task 2}

The file \texttt{web\_stanford.txt} contains a subset of the information
on \href{http://snap.stanford.edu/data/web-Stanford.html}{Stanford
University webpages} and the hyperlinks between them, gathered in 2002.
Each line of the file is formatted as \texttt{a/b/c/d/e/f...}, meaning
the webpage with ID \texttt{a} has hyperlinks to webpages with IDs
\texttt{b}, \texttt{c}, \texttt{d}, and so on.

Write a function that accepts a damping factor \(\epsilon\) defaulting
to 0.85. Read the data and get a list of the \(n\) unique page IDs in
the file (the labels). Construct the \(n\times n\) adjacency matrix of
the graph where node \(j\) points to node \(i\) if webpage \(j\) has a
hyperlink to webpage \(i\). Use your class from Subtask 1.a and its
\texttt{itersolve()} method from Subtask 1.b to compute the PageRank
values of the webpages, then rank them with your function from Subtask
1.c. Return the ranked list of webpage IDs. (Hint: after constructing
the list of webpage IDs, make a dictionary that maps a webpage ID to its
index in the list. The values are the row/column indices in the
adjacency matrix for each label.)

With \(\epsilon=0.85\), the top three ranked webpage IDs are \(98595\),
\(32791\), and \(28392\).

\section*{Task 3}

The file \texttt{psh-uefa-2018-2019.csv} contains data for men's
football teams in Europe for the current season from the main leagues in
Italy, England, France, Scotland, Spain, Germany, Greece, Belgium,
Holland, Portugal, Turkey, together with Champions League and Europa
League. Each line represents a different game, formatted
\texttt{home\_team,away\_team,home\_goals,away\_goals}. Write a function that accepts a
filename and a damping factor \(\epsilon\) defaulting to 0.85. Read the
specified file and get a list of the \(n\)
unique teams in the file. Construct the \(n\times n\) adjacency matrix
of the graph where node \(j\) points to node \(i\) with weight \(w\) if
team \(j\) was defeated by team \(i\) in \(w\) games. That is,
\textbf{edges point from losers to winners}. Ignore draw games. Use your class from Subtask 1.c
 and its \texttt{itersolve()} method to compute the PageRank values of
the teams, then rank them with your function from Subtask 2.c. Return
the ranked list of team names.

Using \texttt{psh-uefa-2018-2019.csv} with \(\epsilon=0.85\),
the top three ranked teams in the current season should be
Liverpool, Athletic Madrid, Paris SG.

The damping factor \(\epsilon\) acts as an ``upset'' factor: a
larger \(\epsilon\) puts more emphasis on win history; a
smaller \(\epsilon\) allows more randomness in the system, giving
underdog teams a higher probability of defeating a team with a better
record.

Note that we did not take into account weights derived from home-away
situations, goal difference, importance of the match and aggregate
results from Europa and Champions league.

It is also worth noting that the sink-fixing procedure is still
reasonable for this model because it gives every other team
\textbf{equal} likelihood of beating an undefeated team. That is, the
additional arcs don't provide an extra advantage to any one team.

\end{document}


\documentclass{article}
\bibliographystyle{unsrt}
\usepackage{amssymb,mathtools,graphicx,algorithm,amsmath,algpseudocode,hyperref}
\usepackage[margin=1in]{geometry}
\begin{document}

\title{Maximum Flow\\Comparative Analysis of Ford-Fulkerson and Edmonds-Karp}

\author{Pedro Rodriguez\\
University of Colorado at Boulder
}

\maketitle
\section{Introduction}
The first research into network flow algorithms began in the early and mid 20th century. Initial research by A.N. Tolstoi sought to optimize production networks such as railroads in the Soviet Union\cite{Schrijver:2002dj}. The subject later picked up interest amongst the American scientists Ford and Fulkerson who devised the first known algorithm for computing the maximum flow through a network\cite{Ford:1956vc}.

The maximum flow problem is defined by: Finding the maximum flow through a network from a source vertex $s$ to a sink vertex $t$ in a directed graph $G$ which has edges labeled with flow capacities.

Twenty years after Ford-Fulkerson published their results Edmonds and Karp independently improved upon the algorithm by replacing the depth-first search strategy used in Fork-Fulkerson with breadth-first search\cite{Edmonds:1972ht}. This change improved the algorithmic efficiency from $O(E|f|)$ where $f$ is the maximum flow to $O(VE^2)$. This idea was further improved upon by Dinic's who exploited the property that the flow paths were monotonically increasing to achieve an even better $O(V^2E)$ running time\cite{Dinic:lsM40ti7}. All of these algorithms share a common strategy: find an augmenting path from $s$ to $t$ to push flow across and repeat until there is no such path. At this point the algorithm terminates and returns the maximum flow value and paths.

Beyond the natural applications of maximum flow algorithms to optimizing supply chains, routing, and others it has gained popularity as a sub-algorithm call in more advanced problems. For example, it was recently used to solve the k-clique densest subgraph problem\cite{tsourakakis2015k}, and various computer vision such as object category segmentation, image deconvolution, super resolution, texture restoration, character completion, and 3D segmentation\cite{Verma:2012gs}. It has also been used to solve other problems such as minimum path cover in directed acyclic graphs, maximum cardinality bipartite matching, and in the "real-world" baseball elimination, airline scheduling, and circulation-demand\cite{wiki:maxflow}.

\section{Problem Statement}
\subsection{Formal Problem Statement}
\label{formal-problem-statement}
Now we proceed to formally defining the maximum flow problem. The problem is defined as given a graph with:
\begin{enumerate}
	\item Vertex set $V$
	\item Directed edge set $E$
	\item Source vertex $s\in V$ such that no edges are incident
	\item Sink vertex $t\in V$ such that no edges leave it
	\item Integer edge capacities $c_{uv}\in \mathbb{R}_{\ge 0}, \forall (u,v)\in E$
\end{enumerate}

And subject to the conditions:
\begin{enumerate}
	\item $f_{uv}\coloneqq$ variable indicating flow from $u$ to $v$
	\item  $f_{uv}\le c_{uv},\forall (u,v)\in E$
	\item $\forall v\in V\text{ such that }v\ne s,t:\sum_{(u,v)\in E}f_{uv}=\sum_{(v, u)\in E} f_{vu}$
\end{enumerate}

Find the flow assignments $f_{uv}$ such that network flow $|f|\coloneqq\sum_{(s,v)\in E}f_{sv}$ is maximized.

\subsection{Graphical Depiction}
To fix ideas we will continue by describing a concrete instance of the problem. For example, consider the example in figure \ref{simple-flow}\cite{wiki:fordfulk}. In the figure, each edge is labeled by a pair of numbers which are presented as $f_{uv}/c_{uv}$. This represents how much flow $f_{uv}$ is passing through an edge with capacity $c_{uv}$. By inspection, it is easy to deduce that the maximum flow is $2000$ with $1000$ flowing through the top and $1000$ flowing through the bottom. This representation makes the meaning of the conditions (1) and (2) clear. Condition (3) expresses that the flow into any vertex which is not a source or sink must be the flow out of that vertex. Finally, the objective can be explained by noting (with proof later) that the maximum flow from $A$ to $D$ is equivalent to taking a cut across $A$.

Before moving on, lets consider a simple algorithm: find a path through the network and push the maximum possible flow through until pushing flow is impossible. One can see this may result in the optimal solution, but if one path found is the one cross $BC$, then the optimal flow will never be found. This hints that the simple graph construction is insufficient to solve the maximum flow problem.

\begin{figure}[!ht]
	\caption{Simple flow network}
	\centering
	\includegraphics[width=0.5\textwidth]{images/simple-flow.png}
	\label{simple-flow}
\end{figure}

\section{Algorithm Analysis}
In this section we will first present the very similar Ford-Fulkerson and Edmonds-Karp algorithms, prove their correctness, and analyze their asymptotic behavior in both time and space complexity.

\subsection{Algorithm}
Before proceeding to the algorithm itself, we must make a few additional definitions and constructions. First we will define a $G_f(V,E_f)$ which we will call the residual graph. In this graph, the capacities are defined as $c^f_{uv}=c_{uv}-f_{uv}$ and there is zero flow. We additionally define that $\forall (u,v)\in E,\ f_{uv}=-f_{vu}$. Combining these two loosens the constraints from the original graph to allow cases where if $f_{uv}>0$ and $c_{uv}=0$ that $c^f_{vu}=c_{vu}-f_{vu}=f_{uv}>0$. As will be proven, this allows for algorithm to function correctly. We now proceed to stating the main algorithm as Algorithm \ref{ff-ek-algorithm}.

\begin{algorithm}
\label{ff-ek-algorithm}
\caption{Ford-Fulkerson and Edmonds-Karp}
\begin{algorithmic}[1]
\Function{max\_flow(G,c,s,t)}{}
\State Input: $G,c,s,t$ as described in \ref{formal-problem-statement}
\State Output: Flow $f_{max}$ and paths to achieve it
\Loop
\State $path\gets augmenting\_path(G_f,s,t)$
\If {$path = None$}
\State break
\EndIf
\State $f_{min}=\min_{(u,v)\in path} c^f_{uv}$
\For{$(u,v)\in path$}
\State $f_{uv}\gets f_{uv}+f_{min}$
\State $f_{vu}\gets f_{vu}-f_{min}$
\EndFor
\EndLoop
\State $f_{max}\gets \sum_{(s,v)\in E}f_{sv}$
\State $paths\gets$ paths/flows from $s$ to $t$ in created graph
\State \Return $f_{max},paths$
\EndFunction
\end{algorithmic}
\end{algorithm}

The algorithm proceeds in two alternating phases: finding an augmenting path from $s$ to $t$ in $G_f$ and pushing the maximal viable flow across that path. When an augmenting path cannot be found, then the algorithm terminates and returns the current maximal flow.

We define an augmenting path as any path from $s$ to $t$ in $G_f$ such that $c^f_{uv}>0\ \forall (u,v)\in path$. The algorithm was not previously specified because its definition determines whether the algorithm is known as Ford-Fulkerson or Edmonds-Karp. If the augmenting path is found using depth first search then the algorithm is called Fork-Fulkerson. If the augmenting path is found using breadth first search then the algorithm is called Edmonds-Karp. With that in mind, we now define the two possible augmenting path algorithms. Note that the $0th$ element of a list is considered the back and the last element of a list is considered the front. Additionally the notation $vec[a;b]$ is short for creating a vector of length $b$ with values initialized to $a$.

\begin{algorithm}
\label{ap-algorithm}
\caption{Augmenting Path Algorithm}
\begin{algorithmic}[1]
\Function{augmenting\_path($G_f$, s, t, search)}{}
\State $V, E_f\gets G_f$
\State $search\_list \gets list()$
\If {$search = BFS$}
\State $pop \gets list.pop\_front$
\State $push \gets list.push\_back$
\Else
\State $pop \gets list.pop\_front$
\State $push \gets list.push\_front$
\EndIf
\State $push(search\_list, s)$
\State $distances\gets vec[\infty; |V|]$
\State $parents\gets vec[\infty; |V|]$
\State $u\gets pop(search\_list)$
\While{$u \ne None\ and\ u\ne t$}
\For{$v$ in $neighbors(u)$}
\If{$distances[v] =\infty\ and\ c^f_{uv}>0$}
\State $distances[v]\gets distances[u] + 1$
\State $parents[v]\gets u$
\State $push(search\_list, v)$
\EndIf
\EndFor
\State $u\gets pop(search\_list)$
\EndWhile
\State\Return $u == t$, reconstruct\_path(s, t, parents)
\EndFunction
\Statex
\Function{reconstruct\_path(s, t, parents)}{}
\State $path\gets list()$
\State $node\gets t$
\Loop
\State $path.append(node)$
\If{$node == s$}
\State \textbf{break}
\EndIf
\State $path\gets parents[node]$
\EndLoop
\State\Return $reverse(path)$
\EndFunction
\end{algorithmic}
\end{algorithm}

Algorithm \ref{ap-algorithm} is essentially a breadth first search or depth first search from the source to the sink with the existence of an edge being defined as when $c^f_{uv}>0$. The output of this search is whether a path exists followed by a vector containing back pointers. This is then used to reconstruct the original path from $s$ to $t$ in $reconstruct\_path$.

Combining Algorithms \ref{ff-ek-algorithm} and \ref{ap-algorithm} yields Ford-Fulkerson when depth first search is used and Edmonds-Karp when breadth first search is used.

\subsection{Correctness}
In this section we first prove that the maximum flow algorithm is correct assuming augmenting paths is correct, then prove that the augmenting paths algorithm is correct to complete the proof.
\subsubsection{Maximum Flow}
Consider the following lemmas:
\begin{enumerate}
	\item \label{lemma:initial-flow} The initial flow through the network $f_{uv}=0,\forall (u,v)\in E$ is a valid flow since $c_{uv}$ is at least $0$ for all edges present in the graph.
	\item Since the original graph contains no parallel edges or cycles, if $c_{uv}>0$ then $c_{vu}=0$. Otherwise this would imply that a cycle would exist in the original graph which is prohibited.
	\item \label{lemma:pos-flow} Given an edge $(u,v)$ by definition $c_{uv}^f=c_{uv}-f_{uv}>0$. If $c_{uv}>0$ then an additional flow up to and including $c_{uv}^f$ can be pushed across $(u,v)$ while preserving the capacity constraint that $c_{uv}\ge f_{uv}$. This is true because the new flow $f_{uv}'=c_{uv}^f+f_{uv}=c_{uv}-f_{uv}+f_{uv}=c_{uv}$ which satisfies the capacity constraint.
	\item \label{lemma:neg-flow} Consider the case where $c_{uv}=0$. If $c_{uv}^f>0$ then $c_{uv}^f=c_{uv}-f_{uv}=-f_{uv}>0$. The maximal value of the new flow would be $f_{uv}'=c_{uv}^f+f_{uv}=c_{uv}-f_{uv}+f_{uv}=0$. This shows that for any path augmentation when $c_{uv}=0$ that flow is conserved since $f_{uv}'\le 0\le c_{uv}$
	\item \label{lemma:valid-flow} The augmented flow pushed across the path is $\min c_{uv}^f=c_{uv}-f_{uv}>0$ which implies that for the minimal flow edge $f_{uv}$ in the path, it is brought up to the full capacity $c_{uv}$. This guarantees that the flow function $f$ is valid since it does not violate the capacity constraint (flow cannot exceed capacity). Since all other edges have equal or greater capacity, then the flow function $f$ stays valid across the entire network after any flow augmentation.
	\item \label{lemma:source-zero} Consider the source edges $(u,v)\in E$ where $u=s$ and $c_{uv}=0$. Since $s$ is a source, it can have no incident edges with $c_{uv}>0$ otherwise it would be an invalid flow network. Therefore $c_{uv}^f=0$ at all times which implies that source edges with $c_{uv}=0$ will never have any flow across them.
	\item \label{lemma:increasing-flow} Consider the source edges $(u,v)\in E$ where $u=s$ and $c_{uv}>0$. Since $c_{uv}\ne 0$ by lemma \ref{lemma:pos-flow} the flow is positive. Therefore, at every iteration the flow across the source edges increases by at least $1$.
	\item \label{lemma:saturated} Define $V_s$ to be all the vertexes reachable from $s$ when the algorithm terminates. By definition, this occurs when no augmenting paths can be found which is equivalent to stating that $c_{uv}^f\ngtr 0$ for all edges from $V_s$ to $V-V_s$. This implies that for all edges across these sets that either $c_{uv}=0$ or $c_{uv}=f_{uv}$. In either case, this is equivalent to saying that all the edges are saturated.
\end{enumerate}

With these lemmas in mind, lets stitch them together for a complete proof. By lemma \ref{lemma:initial-flow} the initial zero flow is a valid flow. Lemmas \ref{lemma:pos-flow} and \ref{lemma:neg-flow} show that for any edge and its capacity, the augmented flow is a valid flow for the edge with minimal $c_{uv}^f$. By lemma \ref{lemma:valid-flow} the augmented flow is valid for the entire path, not just the minimal edge. By lemmas \ref{lemma:source-zero} and \ref{lemma:increasing-flow} the total flow across the network is increasing by at least $1$ at each iteration. Since the initial flow is valid, this proves that every iteration yields an increase in flow across the graph that creates a new valid flow. Since the maximum flow is finite and the flow increases by at least $1$ at each iteration, this proves that the algorithm terminates after at most $f_{max}$ iterations with a valid flow.

We will now show that the flow is maximal through a proof by contradiction. Suppose that the flow returned is not maximal. This would imply that there exists some path $A$ from $s$ to $t$ along which flow can still be pushed. From lemma \ref{lemma:saturated} we know that $V_s$ includes only vertexes reachable from $s$, but that cannot reach $t$. Analogously, the edges from $V_s$ to $V-V_s$ form a cut along the graph. Therefore, any path from $s$ to $t$ including $A$ must contain an edge in the cut. Since flow can be pushed from $s$ to $t$, the edge is not saturated. However, from lemma \ref{lemma:saturated} we know that all edges that are part of the cut are saturated. Since this is a contradiction, then it is impossible for any path to exist where flow can be pushed. Therefore, this proves that the flow returned is maximal.

\subsubsection{Augmenting Path}
In this section we aim to prove that the augmenting paths algorithm will return a path from $s$ to $t$ if one exists or return that one does not exist. Through a general framework, this will be proven for both breadth first search and depth first search. In algorithm \ref{ap-algorithm} we first initialize some list structure to empty. Also initialized are a distances vector and parents (backpointer) vector. When distance is $\infty$ it implies that the given vertex has not been visited. Similarly, the parent of a vertex being set to $\infty$ implies that the vertex does not have a parent in the search. The final part of the algorithm initialization is to push the source on top of the list structure.

Consider an arbitrary vertex $u$ which is pulled from the list structure. We will show that all nodes reachable from $u$ are marked as visited. In the loop, the neighbors of $v$ are fetched. Either they have already been visited, are not in the residual graph, or are processed. When a vertex is processed, it is pushed onto the list, marked as visited, and has its distance set. Consider that $|V|$ is finite, so the number of nodes that can be marked is also finite which implies that the algorithm terminates after a finite number of iterations. Note that the algorithm only terminates when the list structure is empty which can only be empty if exactly zero vertexes need to be processed (since processing a vertex adds it to the list structure). Since none of the vertexes are marked as processed anymore, they must be marked as visited.

Now by induction we will show that all the ancestors of $u$ are marked as visited. The base case considers when the $u$ from above is processed. Therefore, all of the children of $u$ are either already visited or processed. By the reasoning above, we know that eventually all the children of $u$ are processed. The inductive step will consider when the children of $u$ are processed. When they are, all their children will be marked visited or to be processed. Therefore by induction, all the ancestors of $u$ will be processed. Finally, since all vertexes are eventually processed then by elimination all the ancestors of $u$ will be marked as visited.

To complete the proof, consider when $u=s$. Since all ancestors of $s$ will be marked as visited, if $t$ is an ancestor of $s$ then it will be visited. If it is not, then it will not be found. By following the back pointers, the original path from $s$ to $t$ will be found and returned if it exists. This shows that for any list structure, the augmenting paths algorithm will return a path from $s$ to $t$ if it exists. Therefore, any augmenting path algorithm is correct and therefore the max flow algorithm is correct. The analysis of breadth first search and depth first search as augmenting path algorithms will be left to the complexity analysis sections since that is what changes between the two.

\subsection{Ford-Fulkerson Complexity}
Let the list structure in the augmenting paths algorithm is a stack. Consider the order of vertexes visited: the deepest are visited first since vertex children are put on a stack. This is precisely the definition of depth first search. When the max flow algorithm uses depth first search it is called Ford-Fulkerson.

\subsubsection{Time Complexity}
By the max flow correctness proof, we know that at each iteration flow is increased by at least $1$. This implies that the number of iterations is bounded by the maximum flow we call $f_{max}$. Depth first search in the worst case visits all vertexes and edges which is $O(|V|+|E|)$. In the worst case if there are $n$ vertexes the number of edges is $\frac{n(n-1)}{2}=O(|V|^2)$. This implies that the worst case time complexity of depth first search is $O(|E|)$. Finally, since depth first search is run maximally $f_{max}$ times the time complexity is $O(f|E|)$.

\subsubsection{Space Complexity}
The storage used by Ford-Fulkerson is defined by the size of the stack, parent pointer vector, and distances vector. The size of the parent pointer vector and distance vector is $|V|$ since there is an entry per vertex. The maximum size of the stack is all the vertexes which is $|V|$. Therefore, the overall space complexity is $O(|V|)$.

\subsection{Edmonds-Karp Complexity}
Let the list structure in the augmenting paths algorithms is a queue. Consider the order of vertexes visited: the shallowest are visited first since the vertex children are put on a queue. This is precisely the definition of breadth first search. When the max flow algorithm uses breadth first search it is called Edmonds-Karp.

\subsubsection{Time Complexity}
Now we will show that the runtime of Edmonds-Karp is $O(VE^2)$. Like with Ford-Fulkerson, the augmenting paths algorithm runtime is also $O(|V|+|E|)=O(|E|)$ so it remains to be shown that it is called $O(VE)$ times.

First, consider the levels of the graph $G_f$ to be all vertexes reachable at consecutive distances. For example, in figure \ref{simple-flow} $A$ is at level $0$ since it is reachable at $0$ distance, $B$ and $C$ are at level $1$ since they are reachable at $1$ distance, and $D$ is at level $2$ since it is reachable at $2$ distance. For any BFS path there will be exactly one vertex from each level, otherwise it wouldn't be a shortest path.

Consider a path $P$ found by breadth first search along which flow will be augmented, and the edge changes that could occur from an iteration of Edmonds-Karp. Consider an edge $(u,v)\in P$ and the changes which may occur to it. From the level definition, if the distance to $u$ is $d$ then the distance to $v$ must be $d+1$. If flow is pushed across $(u,v)$ this means that an edge is created in the opposite direction, or more precisely from distance $d+1$ to $d$. This shows that any edge added by Edmonds-Karp must be from level $d+1$ to $d$, therefore it does not decrease the distance from $s$ to $t$. Therefore the paths found from BFS are monotonically increasing.

We call edges which were minimal in $\min c_{uv}^f$ bottlenecks. We will now bound the number of times that a given bottleneck edge can be in a shortest path. Consider a bottleneck edge $(u,v)$ which by definition exist in level $d$ and $d+1$. The new edge created must point from $v$ which is at $d+1$ to $u$ which is at $d$. In order for the new bottleneck edge $(v,u)$ to show up again the level of  $v$ and $u$ must be $d$ and $d+1$, but are currently $d+1$ and $d$. Therefore, at least two iterations are needed to increase the level of $u$ from $d$ to $d+2$. Therefore, the bound on the number of times the edge $(u,v)$ or $(v,u)$ is $1/2$ the number of iterations. Note that every time a bottleneck edge appears again the path length must have increased, and the maximum path length is $|V|$. Therefore the maximum number of times that a a bottleneck edge can appear again is $|V|/2$. At each iteration at least one bottleneck edge is guaranteed to be found by the correctness proof. Since the maximum number of edges is $O(E)$ and the maximum number of iterations a given edge can appear in is $O(V)$ the total number of calls to the algorithm is $O(EV)$. Since each iteration takes $O(E)$ the total running time is $O(VE^2)$

\subsubsection{Space Complexity}
As with Ford-Fulkerson, the only data structures used in Edmonds-Karp are a list with maximal size $V$, and two vectors with length $|V|$. Therefore the space complexity is $O(V)$.

\section{Runtime}
In this section we will analyze the performance of Ford-Fulkerson and Edmonds-Karp on specific graphs which showcase the situations in which each performs badly and thus near the asymptotic bound, and perform well and thus much better than the asymptotic bound.

\subsection{Worst Case}
The pathologically worse case for Ford-Fulkerson would be to increase the flow by $1$ at each iteration. Consider the graph from \ref{simple-flow}, and suppose that depth first search returned the alternating paths $ABCD$ and $ACBD$. Since the flow across $BC$ is $1$, this would result in $O(f)$ iterations where $f=2000$. More generally, the degenerate case for Ford-Fulkerson is when paths with low $c_{uv}^f$ have low or even $1$ integer values, and get selected often or always.

This motivates the desire to have a max flow algorithm whose runtime is independent of the maximum flow. In this report we discuss Edmonds-Karp in detail which achieves this goal, but also has inputs where it performs near its asymptotic bound. This occurs when bottleneck edges re-appear from the breadth first search the maximum number of times $|V|$. To get a glimpse at this case consider the graph in figure \ref{ek-flow}. Suppose that the first path found is $suvt$ along which $5$ flow is pushed. It should be obvious that the optimal would be to reserve $v$ for the flow comping from $a$. This however cannot occur until flow is pushed along $sueft$ and $sacdt$ since they are shortest paths. In order to fix the flow path $suvacdt$ needs to be chosen which would occur after two iterations at which time the flow could finally be fixed to optimal

\begin{figure}[!ht]
	\caption{Edmonds Karp Worst Case}
	\centering
	\includegraphics[width=0.5\textwidth]{images/edmonds-karp.png}
	\label{ek-flow}
\end{figure}

\subsection{Average Case}
The performance of both algorithms is highly dependent on the input graph structure which makes characterizing an average case difficult or impossible. Additionally, artifacts of common implementations (including this one), can cause widely varying performance. For example, in Ford-Fulkerson the ordering of edges in the depth first search can lead to widely varying behavior, but can be addressed by randomization methods \cite{graph-paths}. For these reasons it is not reasonable to report an average case since it is so dependent on input and specific implementation.

\subsection{Best Case}
The best case performance for both of these algorithms is when the minimum number of iterations occur. The trivial case for both graphs is that a single flow push results in termination of the algorithm. In both of these algorithms, as the number of times a single edge appears in the augmenting path decreases the better performing it becomes up to $O(E)$ for Ford-Fulkerson and $O(E)$ for Edmonds-Karp.

\section{Experiments}
The experiments in the following section were performed with the code attached to this project which is also available at \url{https://github.com/EntilZha/max_flow}. The source code for Ford-Fulkerson and Edmonds-Karp can be found in the "src/lib.rs", the running harness in "src/bin.rs", and various python and bash scripts used for formatting input data in the root directory. The algorithms were implemented using the Rust programming language version 1.8 \cite{rust}. All experiments were run on Amazon Web Services Elastic Cloud Compute with m4.2xlarge instances which have 8 2.4GHz Intel Xeon CPU cores and 32GB of memory\cite{aws}. Data sets used either the DIMACS format or an adjacency list representation\cite{dimacs}.

\subsection{Random Input Generation}
For testing, a randomized flow graph generator was created. Since the flow algorithm is highly sensitive to the graph structure, it is not representative for all inputs, but it is sufficient to verify asymptotic correctness. As input the generator takes an integer $l$ representing number of layers, integer $k$ representing layer size, $f$ the maximum flow through an edge, and interconnect ratio $r\ge 1$. The output will be a graph which has $l$ layers, and each layer having $k$ vertexes. Between layer $i$ and layer $i+1$, $k$ edges with $f$ flow are added such that the two layers form a bipartite graph with all nodes connected. Following this, edges are randomly added between layer $i$ and $i+1$ until $\lfloor r\cdot k\rfloor\le |E_{i,i+1}|$ where $E_{i,i+1}$ are the edges going from layer $i$ to layer $i+1$. The flow across these edges are sampled uniformly from $[1,f]$. All the layers were connected in this fashion while the first and last layers were fully connected to a source $s$ and sink $t$ with all edges having randomized flow like above. The experiments in the following section vary $l$ and $k$ while running $r,f$ through a wide range of values. The result of this generation process is a many layers of bipartite graphs with a randomized distribution of edges, but an easily verifiable maximum flow.

\subsection{Scaling}
To test these algorithms $f=100$, and $l,k,r$ varied. For each pair of values $(l,k)$ $r$ ranged from $1$ to $100$ by $10$ with each experiment being repeated five times. In these experiments $l=k$ for values of $50,100,150,220$. The cap value of $220$ was chosen since this resulted in 20GB/32GB of memory being used. In total this resulted in $4\cdot 10\cdot 5=200$ experimental results for each of Ford-Fulkerson and Edmonds-Karp (400 total). In terms of $|V|$, $|E|$, and $f$, the input size ranged from $|V|=2500$ to $|V|=48400$, $|E|=5100$ to $|E|=8866000$, and $f=~3500$ to $f=25000$. It should be clear that these values provide a more than adequate range of values to test against since they span many orders of magnitude in the absolute sense, but in the total asymptotic runtime as well since they are $O(f|E|)$ and $O(E^2V)$.

\subsubsection{Ford-Fulkerson}
Now we proceed to analyzing the asymptotic performance of Ford-Fulkerson by considering figure \ref{ff-performance}.

\begin{figure}[!ht]
	\caption{Ford-Fulkerson Performance}
	\centering
	\includegraphics[width=0.5\textwidth]{images/ff-performance.png}
	\label{ff-performance}
\end{figure}

This figure plots $Ef$ versus the algorithm running time with the increasing blackness of circles indicating the experiments that achieved the highest maximum flow across all experiments. Since the algorithm is $O(Ef)$ one would expect that the runtime plot as a function of these variables should be a linear function since the asymptotic order only allows it to vary by some constant. Figure \ref{ff-performance} clearly shows a linear relationship which is verified by linear regression with good statistical values of $r=.738,r^2=.54$. One can also see that as the graph and flow increases so does the spread of the data points. This can be attributed to the fact that there is high variance in the impact of $f$ in the runtime depending on the specific graph structure. These results indicate that this implementation of Ford-Fulkerson achieves the correct asymptotic performance.

\subsubsection{Edmonds-Karp}
Now we proceed to analyzing the asymptotic performance of Edmonds-Karp by considering figure \ref{ek-performance}.

\begin{figure}[!ht]
	\caption{Ford-Fulkerson Performance}
	\centering
	\includegraphics[width=0.5\textwidth]{images/ek-performance.png}
	\label{ek-performance}
\end{figure}

Figure \ref{ek-performance} plots the asymptotic performance $O(E^2V)$ against runtime. Like before, if this achieves linear or better performance then the asymptotic bound is upheld. Linear regression analysis of the data shows good statistical values of $r=.845,r^2=.714$. Discounting the tail end, this upholds the hypothesis that the data has a linear correlation. The tail end is sublinear therefore the empirical results agree with a runtime of $O(E^2V)$.

\subsubsection{Comparison Discussion}
Interestingly enough, performance results indicate that in practice Ford-Fulkerson outperformed Edmonds-Karp. Considering the trellis random graph input this is not surprising since edges within a layer are not interconnected which avoids some of the pathological performance issues of Ford-Fulkerson. A better comparison would include graphs where this behavior does occur. The other primary difference is that the variance of Ford-Fulkerson seemed to be higher than Edmonds-Karp.

\subsection{Computer Vision Datasets}
To analyze some real world performance of maximum flow algorithms computer vision data sets were used \cite{Verma:2012gs}. Although there were many data sets available, experiments were ran against the deconvolution data set with Ford-Fulkerson, Edmonds-Karp, NetworkX Edmonds-Karp, and NetworkX Push-Relabel \cite{networkx}. Experiments were conducted against two datasets, one called vision3 with $|E|=47872,|V|=2002,f=5668675$ and another called vision5 with $|E|=139600,|V|=2002,f=6849322$.

\begin{figure}[!ht]
	\caption{Deconvolution Performance}
	\label{vision}
	\begin{tabular}{c | c | c}
		Dataset & Algorithm & Runtime in Seconds\\
		\hline
		vision3 & FF & 26\\
		vision3 & EK & 2.6\\
		vision3 & NX-EK & 35\\
		vision3 & NX-PP & .55\\
		vision5 & FF & 109\\
		vision5 & EK & 13.3\\
	\end{tabular}
\end{figure}

The results of these experiments are in figure \ref{vision}. From these several conclusions can be drawn. First, on this dataset Ford-Fulkerson seems to perform much worse than Edmonds-Karp. Second, interestingly enough this Rust implementation of Edmonds-Karp for this project outperforms the NetworkX implementation by 10x. Finally, the Push Relabel maximum flow algorithm by Goldberg and Tarjan significantly outperforms all other algorithms which is consistent with its much better runtime asymptotic performance of $O(V^2E)$\cite{goldberg1988new}.

\section{Related Work}
	There is an abundant amount of work in this field some of which has already been mentioned. Maximum flow was first solved by Ford-Fulkerson in $O(Ef)$ time which was inconvenient since $f$ was unknown before the algorithm start\cite{Ford:1956vc}. This motivated the Edmonds-Karp algorithm which attained a runtime guarantee independent of the maximum flow $O(VE^2)$\cite{Edmonds:1972ht}. Although better, since $E=O(V^2)$ the runtime in terms of $V$ for a dense graph could approach $O(V^5)$ which is expensive. Although not discussed, Dinic's blocking flow algorithm followed shortly after which iterated on the shortest paths idea of using Breadth First Search to improve the running time to $O(V^2E)$\cite{Dinic:lsM40ti7}. This was further improved using dynamic trees to $O(EV\log V)$. The algorithm in most wide use today (as in NetworkX) is Push-relabel with runtime $O(V^2E)$\cite{goldberg1988new}. Most recently the flow problem has been solved in $O(VE)$ time by using Orlin's algorithm and the KRT algorithm\cite{orlin2013max}.

\section{Conclusion}
Through this report several goals are accomplished. First, the historical motivation for solving maximum flow is explored. Second, the correctness of both Ford-Fulkerson and Edmonds-Karp is shown in finding the maximal flow if there is one. Third, the asymptotic runtime behavior of these is analyzed. Fourth, experimental results show that the Rust implementation achieves the appropriate runtime and outperforms the implementation in the Python library NetworkX. Additionally, performance tests were done using computer vision datasets. Finally, Ford-Fulkerson and Edmonds-Karp are discussed in the context of other common and more advanced maximum flow algorithms.

\section{Resources}
In the course of learning about these algorithms the following sources were very helpful: \cite{cornell-ek},\cite{umass-ek}. Additionally, I used integration tests from Comp 572 at Hong Kong University to check correctness (in addition to many of my own tests)\cite{UstProject}.

\bibliographystyle{abbrv}
\bibliography{references}

%\balancecolumns

\end{document}


\subsubsection{Riding Bow}\label{weapon:ridingBow}
Weapon, Bow, Two-Handed, Regular, Ranged\\
Size: M\\
Ammunition: Arrows\\
Cost: 100 Gold

\textbf{Draw/Sheath}\\
6AP draw, 10AP sheath

\textbf{Shoot}\\
2 AP, DE to hit, \passus{60} Reach\\
2d8 + \sfrac{1}{2}\texttimes DE Piercing Damage

\textbf{Reload}\\
3 AP from a back quiver\\
2 AP from a belt quiver\\
The Riding Bow has to be reloaded after every shot.


\documentclass[letterpaper]{article}
\usepackage{typearea}
\typearea{12}
\usepackage{here}
\usepackage{bm}
\usepackage{amsmath, amsfonts}
\usepackage[top=20truemm,bottom=20truemm,left=25truemm,right=25truemm]{geometry}
\usepackage[dvipdfmx]{hyperref,graphicx}

\begin{document}
\title{The Art of Linear Algebra\\
\vspace{5pt}
\large{
-- Graphic Notes on ``Linear Algebra for Everyone" --
}
}

\author{Kenji Hiranabe
\thanks{twitter: @hiranabe, k-hiranabe@esm.co.jp, \url{https://anagileway.com}} \\
with the kindest help of Gilbert Strang
\thanks{Massachusetts Institute of Technology, \url{http://www-math.mit.edu/\~gs/}}
}

\date{September 1, 2021/updated \today}

\maketitle

\vspace{-5pt}

\begin{abstract}
I tried intuitive visualizations of important concepts introduced
in ``Linear Algebra for Everyone".\footnote{``Linear Algebra for Everyone":
\url{http://math.mit.edu/everyone/} with Japanese translation started by Kindai Kagaku.}\linebreak
This is aimed at promoting understanding of vector/matrix calculations
and algorithms from the\linebreak perspectives of matrix factorizations.
They include Column-Row ($\bm{CR}$), Gaussian Elimination ($\bm{LU}$),
Gram-Schmidt Orthogonalization ($\bm{QR}$), Eigenvalues and Diagonalization ($\bm{Q \Lambda Q^\mathbf{T}}$),
and Singular Value Decomposition ($\bm{U \Sigma V^\mathbf{T}}$).
\end{abstract}

\section*{Foreword}
I am happy to see Kenji Hiranabe's pictures of matrix operations in linear algebra !
The pictures are an excellent way to show the algebra.  We can think of matrix
multiplications by row $\bm{\cdot}$ column dot products, but that is not all --  it is ``linear combinations"
and ``rank 1 matrices" that complete the algebra and the art.
I am very grateful to see the books in Japanese translation
and the ideas in Kenji's pictures.
\begin{flushright}
-- Gilbert Strang \\ Professor of Mathematics at MIT
\end{flushright}

\tableofcontents

\section{Viewing a Matrix -- 4 Ways}

A matrix ($m \times n$) can be seen as $1$ matrix, $mn$ numbers, $n$ columns and $m$ rows.

\begin{figure}[H]
  \centering
  \includegraphics[keepaspectratio, width=7cm]{ViewingMatrix-4Ways.png}
  \caption{Viewing a Matrix in 4 Ways}
\end{figure}

\begin{equation*}
  A= \begin{bmatrix}
    a_{11} & a_{12}\\
    a_{21} & a_{22}\\
    a_{31} & a_{32}
  \end{bmatrix}
  =
  \begin{bmatrix}
    | & |\\
    \bm{a_1} & \bm{a_2}\\
    | & |
  \end{bmatrix}
  =
  \begin{bmatrix}
    - \bm{a_1^*} -\\
    - \bm{a_2^*} -\\
    - \bm{a_3^*} -
  \end{bmatrix}
\end{equation*} \\

Here, the column vectors are in bold as $\bm{a_1}$.
Row vectors include $\bm{*}$ as in $\bm{a_1^*}$.
Transposed vectors and matrices are indicated by $\mathrm{T}$ as
in $\bm{a}^{\mathrm{T}}$ and $A^{\mathrm{T}}$.

\section{Vector times Vector -- 2 Ways}

Hereafter I point to specific sections of ``Linear Algebra for Everyone"
and present graphics which illustrate the concepts with short names
in colored circles.

\begin{itemize}
  \item Sec. 1.1 (p.2) Linear combination and dot products
  \item Sec. 1.3 (p.25) Matrix of Rank One
  \item Sec. 1.4 (p.29) Row way and column way
\end{itemize}

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{VectorTimesVector.png}
  \caption{Vector times Vector - (v1), (v2)}
\end{figure}

\section{Matrix times Vector -- 2 Ways}

A matrix times a vector creates a vector of three dot products (Mv1)
as well as a linear combination (Mv2) of the column vectors of $A$.

\begin{itemize}
  \item Sec. 1.1 (p.3) Linear combinations
  \item Sec. 1.3 (p.21) Matrices and Column Spaces
\end{itemize}

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{MatrixTimesVector.png}
  \caption{Matrix times Vector - (Mv1), (Mv2)}
\end{figure}

At first, you learn (Mv1). But when you get used to viewing it as (Mv2),
you can understand $Ax$ as a linear combination of the columns of $A$.
Those products fill the column space of $A$  denoted as $\mathbf{C}(A)$.
The solution space of $Ax=0$ is the nullspace of $A$ denoted as $\mathbf{N}(A)$.

The four subspaces consists of $\mathbf{N}(A)$ + $\mathbf{C}(A^\mathbf{T})$
(which are perpendicular to each other) in $\mathbb{R}^n$ and
$\mathbf{N}(A^\mathbf{T})$ + $\mathbf{C}(A)$ in $\mathbb{R}^m$
(which are perpendicular to each other).

\begin{itemize}
  \item Sec. 3.5 (p.124) Dimentions of the Four Subspaces
\end{itemize}

\begin{figure}[H]
  \centering
  \includegraphics[keepaspectratio, width=8cm]{4-Subspaces.png}
  \caption{The Four Subspaces}
\end{figure}

See $A=CR$ (Sec 6.1) for the rank $r$.

\clearpage

\section{Matrix times Matrix -- 4 Ways}

``Matrix times Vector" naturally extends to ``Matrix times Matrix".

\begin{itemize}
  \item Sec. 1.4 (p.35) Four Ways to Multiply $\bm{AB=C}$
  \item Also see the back cover of the book
\end{itemize}

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{MatrixTimesMatrix.png}
  \caption{Matrix times Matrix - (MM1), (MM2), (MM3), (MM4)}
\end{figure}

\clearpage

\section{Practical Patterns}

Here, I show some practical patterns which allow you to capture
the coming factorizations more intuitively.

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{Pattern12.png}
  \caption{Pattern 1, 2 - (P1), (P1)}
\end{figure}

Pattern 1 is a combination of (MM2) and (Mv2).
Pattern 2 is an extention of (MM3). Note that Pattern 1 is a column operation (multiplying a matrix from right),
whereas Pattern 2 is a row operation (multiplying a matrix from left).

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{Pattern11-22.png}
  \caption{Pattern 1$^\prime$, 2$^\prime$ - (P1$^\prime$), (P2$^\prime$)}
\end{figure}

(P1$^\prime$) multipies the diagonal numbers to the columns of the matrix,
whereas (P2$^\prime$) multipies the diagonal numbers to the row of the matrx.
Both are variants of (P1) and (P2).

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{Pattern3.png}
  \caption{Pattern 3 - (P3)}
\end{figure}

This pattern appears when you solve differential equations and recurrence equations:

\begin{itemize}
  \item Sec. 6 (p.201) Eigenvalues and Eigenvectors
  \item Sec. 6.4 (p.243) Systems of Differential Equations
\end{itemize}

\begin{align*}
  \frac{d \bm{u}(t) }{dt} &= A \bm{u}(t), \quad \bm{u}(0)=\bm{u}_0\\
  \bm{u}_{n+1} &= A \bm{u}_n, \quad \bm{u_0} = \bm{u}_0
\end{align*}

In both cases, the solutions are expressed with
eigenvalues ($\lambda_1, \lambda_2, \lambda_3$),
eigenvectors $X=\begin{bmatrix} \bm{x}_1 & \bm{x}_2 & \bm{x}_3 \end{bmatrix}$ of $A$, and
the coefficients $c=\begin{bmatrix} c_1 & c_2 & c_3 \end{bmatrix}^\mathbf{T}$
which are the coordinates of the initial condition $\bm{u}(0)=\bm{u}_0$ in terms of
the eigenvectors $X$.

\begin{equation*}
  \bm{u}_0 = c_1 \bm{x}_1 + c_2 \bm{x}_2 + c_3 \bm{x}_3
\end{equation*}
\begin{equation*}
  \bm{c} =
  \begin{bmatrix}
    c_1\\
    c_2\\
    c_3
  \end{bmatrix} = X^{-1} \bm{u}_0
\end{equation*}

and the general solution of the two equations are:

\begin{align*}
  \bm{u}(t) &= e^{At} \bm{u}_0 = X e^{\Lambda t} X^{-1} \bm{u_0} &= X e^{\Lambda t} \bm{c} &= c_1 e^{\lambda_1 t} \bm{x}_1 + c_2 e^{\lambda_2 t} \bm{x}_2 + c_3 e^{\lambda_3 t} \bm{x}_3\\
  \bm{u}_n &= A^n \bm{u}_0 = X \Lambda^n X^{-1} \bm{u_0} &= X \Lambda^n \bm{c} &= c_1 \lambda_1^n \bm{x}_1 + c_2 \lambda_2^n \bm{x}_2 + c_3 \lambda_3^n \bm{x}_3
\end{align*}

See Figure 8: Pattern 3 (P3) above again for $XDc$.

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{Pattern4.png}
  \caption{Pattern 4 - (P4)}
\end{figure}

This pattern (P4) works in both eigenvalue decomposition and singular value decomposition.
Both decompositions are expressed as a product of three matrices with a diagonal matrix in the middle,
and also a sum of rank 1 matrices with the eigenvalue/singular value coefficients.

More details are discussed in the next section.

\clearpage

\section{The Five Factorizations of a Matrix}

\begin{itemize}
  \item Preface p.vii, The Plan for the Book.
\end{itemize}
$A=CR, A=LU, A=QR, A=Q \Lambda Q^\mathbf{T}\, A=U \Sigma V^\mathbf{T}$ are
illustrated one by one.

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{5-Factorizations.png}
  \caption{The Five Factorizations}
\end{figure}

\clearpage

\subsection{$\boldsymbol{A=CR}$}

\begin{itemize}
  \item Sec.1.4 Matrix Multiplication and $\bm{A=CR}$ (p.29)
\end{itemize}

All general rectangular matrices $A$ have the same row rank as the column rank.
This factorization is the most intuitive way to understand this theorem.
$C$ consists of independent columns of $A$, and $R$ is the row reduced echelon form of $A$.
$A=CR$ reduces to $r$ independent columns in $C$ times $r$ independent rows in $R$.

\begin{equation*}
  \begin{split}
    A &= CR\\
  \begin{bmatrix}
    1 & 2 & 3 \\
    2 & 3 & 5
  \end{bmatrix}
  & =
  \begin{bmatrix}
    1 & 2 \\
    2 & 3
  \end{bmatrix}
  \begin{bmatrix}
    1 & 0 & 1 \\
    0 & 1 & 1
  \end{bmatrix}
\end{split}
\end{equation*}

Procedure: Look at the columns of $A$ from left to right. Keep independent ones,
discard dependent ones which can be created by the former columns.
The column 1 and the column 2 survive, and the column 3 is discarded
because it is expressed as a sum of the former two columns.
To rebuild $A$ by the independent columns 1, 2, you find a row echelon form $R$
appearing in the right.

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{CR1.png}
  \caption{Column Rank in $CR$}
\end{figure}

Now you see the column rank is two because there are only two independent columns in $C$
and all the columns of $A$ are linear combinations of the two columns of $C$.

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{CR2.png}
  \caption{Row Rank in $CR$}
\end{figure}

And you see the row rank is two because there are only two independent rows in $R$
and all the rows of $A$ are linear combinations of the two rows of $R$.

\clearpage

\subsection{$\boldsymbol{A=LU}$}

Solving $Ax=b$  via Gaussian elimination can be expressed as a $LU$ factorization.
Usually, you apply elementary row operation matrices ($E$) from left of $A$ to make upper trianglar $U$.

\begin{align*}
  EA &= U\\
  A &= E^{-1}U\\
\text{let} \; L = E^{-1}, \quad  A &= LU
\end{align*}

Now solve $Ax=b$ in 2 steps: 1) forward $Lc=b$ and 2) back $Ux=c$.

\begin{itemize}
  \item Sec.2.3 (p.57) Matrix Computations and $\bm{A=LU}$
\end{itemize}

Here, we directly calculate $L$ and $U$ from $A$.

\begin{equation*}
  A =
      \begin{bmatrix}
        |\\
        \bm{l}_1\\
        |
      \end{bmatrix}
      \begin{bmatrix}
        -  \bm{u}^*_1  -
      \end{bmatrix}
  +  \begin{bmatrix}
      0 & \begin{matrix} 0 & 0 \end{matrix}\\
      \begin{matrix} 0 \\ 0 \end{matrix} & A_2
    \end{bmatrix}
  =
  \begin{bmatrix}
    |\\
    \bm{l}_1\\
    |
  \end{bmatrix}
  \begin{bmatrix}
    - \bm{u}^*_1 -
  \end{bmatrix}
  +
  \begin{bmatrix}
    |\\
    \bm{l}_2\\
    |
  \end{bmatrix}
  \begin{bmatrix}
    - \bm{u}^*_2  -
  \end{bmatrix}
  +  \begin{bmatrix}
  0 & 0 & 0\\
  0 & 0 & 0 \\
  0 & 0 & A_3
  \end{bmatrix} = LU
\end{equation*}

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{LU1.png}
\caption{Recursive Rank 1 Matrix Peeling from $A$}
\end{figure}

To find $L$ and $U$, peel off the rank 1 matrix made of
the first row and the first column of $A$.
This leaves $A_2$. Do this recursively and decompose $A$ into the sum of rank 1 matrices.

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{LU2.png}
\caption{$LU$ rebuilds $A$}
\end{figure}

To rebuild $A$ from $L$ times $U$ is easy.

\clearpage

\subsection{$\boldsymbol{A=QR}$}

$A=QR$ changes the columns of $A$ into perpendicular columns of $Q$, keeping $\bm{C}(A) = \bm{C}(Q)$.

\begin{itemize}
  \item Sec.4.4 Orthogonal matrices and Gram-Schmidt (p.165)
\end{itemize}

In Gram-Schmidt, the normalized $\bm{a}_1$ is picked up as $\bm{q}_1$ first and then
$\bm{a}_2$ is adjusted to be perpendicular to $\bm{q}_1$ to create $\bm{q}_2$, and this
procedure goes on.

\begin{align*}
  \bm{q}_1 &= \bm{a}_1/|\bm{a}_1| \\
  \bm{q}_2 &= \bm{a}_2 - (\bm{q}_1^\mathbf{T} \bm{a}_2)\bm{q}_1 , \quad \bm{q}_2 = \bm{q}_2/|\bm{q}_2| \\
  \bm{q}_3 &= \bm{a}_3 - (\bm{q}_1^\mathbf{T} \bm{a}_3)\bm{q}_1 - (\bm{q}_2^\mathbf{T} \bm{a}_3)\bm{q}_2, \quad \bm{q}_3 = \bm{q}_3/|\bm{q}_3|
\end{align*}

or you can write:

\begin{align*}
  \bm{a}_1 &= r_{11}\bm{q}_1\\
  \bm{a}_2 &= r_{12}\bm{q}_1 + r_{22} \bm{q}_2\\
  \bm{a}_3 &= r_{13}\bm{q}_1 + r_{23} \bm{q}_2 + r_{33} \bm{q}_3
\end{align*}

The original $A$ becomes $QR$: orthogonal times triangular.

\begin{gather*}
  A =
  \begin{bmatrix}
    | & | & |\\
    \bm{q}_1 & \bm{q}_2 & \bm{q}_3\\
    | & | & |
  \end{bmatrix}
  \begin{bmatrix}
    r_{11} & r_{12} & r_{13}\\
           & r_{22} & r_{23}\\
           &        & r_{33}
  \end{bmatrix} = QR\\
  \\
  Q Q^\mathbf{T}=Q^\mathbf{T} Q = I
\end{gather*}

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{QR.png}
  \caption{$A=QR$}
\end{figure}

The column vectors of $A$ can be adjusted into an orthonormal set: the column vectors of $Q$.
Each column vector of $A$ can be rebuilt from $Q$ and the upper triangular matrix $R$ .

See Pattern 1 (P1) again for the graphic interpretation.

\clearpage

\subsection{$\boldsymbol{S=Q \Lambda Q^\mathbf{T}}$}

All symmetric matrices $S$ must have real eigenvalues and orthogonal eigenvectors.
The eigenvalues are the diagonal elementes of $\Lambda$ and the eigenvectors are in $Q$.

\begin{itemize}
  \item Sec.6.3 (p.227) Symemtric Positive Definite Matrices
\end{itemize}

\begin{align*}
  S = Q \Lambda Q^\mathbf{T}
&= \begin{bmatrix}
    | & | & |\\
    \bm{q}_1 & \bm{q}_2 & \bm{q}_3\\
    | & | & |
  \end{bmatrix}
  \begin{bmatrix}
    \lambda_1 \\
           & \lambda_2 & \\
           & & \lambda_3
  \end{bmatrix}
  \begin{bmatrix}
  - \bm{q}_1^\mathbf{T} -\\
  - \bm{q}_2^\mathbf{T} -\\
  - \bm{q}_3^\mathbf{T} -
  \end{bmatrix}\\
  \\
  &=
  \lambda_1 \begin{bmatrix}
    |\\
    \bm{q}_1\\
    |
  \end{bmatrix}
  \begin{bmatrix}
    - \bm{q}_1^\mathbf{T} -
  \end{bmatrix}
  +
  \lambda_2 \begin{bmatrix}
  |\\
  \bm{q}_2\\
  |
  \end{bmatrix}
  \begin{bmatrix}
  - \bm{q}_2^\mathbf{T} -
  \end{bmatrix}
  +
  \lambda_3 \begin{bmatrix}
    |\\
    \bm{q}_3 \\
    |
  \end{bmatrix}
  \begin{bmatrix}
    - \bm{q}_3^\mathbf{T} -
  \end{bmatrix} \\
&= \lambda_1 P_1 + \lambda_2 P_2 + \lambda_3 P_3
\end{align*}

\begin{equation*}
  P_1=\bm{q}_1 \bm{q}_1^\mathbf{T}, \quad P_2=\bm{q}_2 \bm{q}_2^\mathbf{T}, \quad P_3=\bm{q}_3 \bm{q}_3^\mathbf{T}
\end{equation*}

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{EVD.png}
  \caption{$S=Q \Lambda Q^\mathbf{T}$}
\end{figure}

A symmetric matrix $S$ is diagonalized into $\Lambda$  by an orthogonal matrix $Q$
and its transpose. And it is broken down into a combination of rank 1 projection matrices $P=qq^\mathbf{T}$.
This the spectral theorem.

Note that Pattern 4 (P4) is working for the decomposition.

\begin{gather*}
  S=S^\mathbf{T} = \lambda_1 P_1 + \lambda_2 P_2 + \lambda_3 P_3\\
  QQ^\mathbf{T} = P_1 + P_2 + P_3 = I \\
  P_1 P_2 = P_2 P_3 = P_3 P_1 = O\\
  P_1^2 =P_1=P_1^\mathbf{T}, \quad P_2^2=P_2=P_2^\mathbf{T}, \quad P_3^2=P_3=P_3^\mathbf{T}
\end{gather*}

\clearpage

\subsection{$\boldsymbol{A=U \Sigma V^\mathbf{T}}$}

All matrices including rectangular ones have a singular value decomposition (SVD).

$A=U \Sigma V^\mathbf{T}$ has the singular vectors of $A$ in $U$ and $V$.
And its singular values in $\Lambda$.

\begin{itemize}
  \item Sec.7.1 (p.259) Singular Values and Singular Vecrtors
\end{itemize}

\begin{figure}[H]
  \includegraphics[keepaspectratio, width=\linewidth]{SVD.png}
  \caption{$A=U \Sigma V^\mathbf{T}$}
\end{figure}

You can find $V$ as an orthonormal basis of $\mathbb{R}^n$ (eigenvectors of $A^\mathbf{T}A$),
and $U$ as an orthonormal basis of $\mathbb{R}^m$ (eigenvectors of $AA^\mathbf{T}$).
Together they diagonalize $A$ into $\Sigma$.
This is also expressed as a combination of rank 1 matrices.

\begin{align*}
  A = U \Sigma V^\mathbf{T} =
  \begin{bmatrix}
    | & | & |\\
    \bm{u}_1 & \bm{u}_2 & \bm{u}_3\\
    | & | & |
  \end{bmatrix}
  \begin{bmatrix}
    \sigma_1 \\
           & \sigma_2 \\
           & &
  \end{bmatrix}
  \begin{bmatrix}
  - \bm{v}_1^\mathbf{T} -\\
  - \bm{v}_2^\mathbf{T} -
  \end{bmatrix}
  & =
  \sigma_1 \begin{bmatrix}
    |\\
    \bm{u}_1\\
    |
  \end{bmatrix}
  \begin{bmatrix}
    - \bm{v}_1^\mathbf{T} -
  \end{bmatrix}
  +
  \sigma_2 \begin{bmatrix}
  |\\
  \bm{u}_2\\
  |
  \end{bmatrix}
  \begin{bmatrix}
  - \bm{v}_2^\mathbf{T} -
  \end{bmatrix} \\
& = \sigma_1 \bm{u}_1 \bm{v}_1^\mathbf{T} + \sigma_2 \bm{u}_2 \bm{v}_2^\mathbf{T}
\end{align*}

Note that:

\begin{align*}
  U U^\mathbf{T} &= I_m \\
  V V^\mathbf{T} &= I_n
\end{align*}

See Pattern 4 (P4) for the graphic notation.

\section*{Conclusion and Acknowledgements}

I presented systematic visualizations of matrix/vector multiplication and
their application to the Five Matrix Factorizations. I hope you
enjoyed them and will use them
in your understanding of Linear Algebra.

Ashley Fernandes helped me with beautifying this paper in typesetting
and made it much more consistent and professional.

To conclude this paper, I'd like to thank Prof. Gilbert Strang for
publishing ``Linear Algebra for Everyone". It guides us
through a new vision to these beautiful landscapes in Linear Algebra.
Everyone can reach a fundamental understanding of its underlying ideas
in a practical manner that introduces us to contemporary and also
traditional data science and machine learning. An important part of the matrix world.

\section*{References and Related Works}
\begin{enumerate}
  \item
  Gilbert Strang(2020),\emph{Linear Algebra for Everyone}, Wellesley Cambridge Press.,
  \href{http://math.mit.edu/everyone}{http://math.mit.edu/everyone}
  \item
  Gilbert Strang(2016), \emph{Introduction to Linear Algebra},Wellesley Cambridge Press, 5th ed.,
  \href{http://math.mit.edu/linearalgebra}{http://math.mit.edu/linearalgebra}
  \item Kenji Hiranabe(2020), \emph{Matrix World}, An Agile Way(blog),
  \href{https://anagileway.com/2020/09/29/matrix-world-in-linear-algebra-for-everyone/}{https://anagileway.com/2020/09/29/matrix-world-in-linear-algebra-for-everyone/}
  \item Kenji Hiranabe(2021), \emph{Map of Eigenvalues}, An Agile Way(blog),
  \href{https://anagileway.com/2021/10/01/map-of-eigenvalues/}{https://anagileway.com/2021/10/01/map-of-eigenvalues/}
\end{enumerate}

\end{document}


\documentclass{beamer}

\mode<presentation>
{
  \usetheme{Hawke}
  \setbeamercovered{transparent}
}

\usepackage[english]{babel}
\usepackage[latin1]{inputenc}
\usepackage{times}
\usepackage[T1]{fontenc}
\usepackage{multimedia}

%%%%%%
% My Commands
%%%%%%

\newcommand{\bb}{{\boldsymbol{b}}}
\newcommand{\bx}{{\boldsymbol{x}}}
\newcommand{\by}{{\boldsymbol{y}}}
\newcommand{\bfm}[1]{{\boldsymbol{#1}}}
\newcommand{\pda}[2]{\frac{\partial{#1}}{\partial{#2}}}
\newcommand{\pdb}[2]{\frac{\partial^2{#1}}{\partial{#2}^2}}
\newcommand{\pdc}[3]{\frac{\partial^2{#1}}{\partial{#2}\partial{#3}}}

%%%%

\title[Lecture 27] % (optional, use only with long paper titles)
{Lecture 27 - Von Neumann stability}

\author[I. Hawke] % (optional, use only with lots of authors)
{I.~Hawke}
\institute[University of Southampton] % (optional, but mostly needed)
{
  School of Mathematics, \\
  University of Southampton, UK
}

\date[Semester 1] % (optional, should be abbreviation of conference name)
{MATH3018/6141, Semester 1}

\subject{Numerical methods}

\pgfdeclareimage[height=0.5cm]{university-logo}{mathematics_7469}
\logo{\pgfuseimage{university-logo}}

\AtBeginSection[]
{
  \begin{frame}<beamer>
    \frametitle{Outline}
    \tableofcontents[currentsection]
  \end{frame}
}

\begin{document}

\begin{frame}
  \titlepage
\end{frame}

\section{Stability}

\subsection{Evidence}

\begin{frame}
  \frametitle{Partial differential equations}

  Partial differential equations (PDEs) involve derivatives of
  functions of more than one variable, say $u(x, y)$ or $y(t,
  x)$. Hence more complex behaviour and more interesting
  physics. \pause

  \vspace{1ex}

  Only look at linear problems.  Also only consider finite difference
  methods: simple to analyse but not always competitive. \pause

  \vspace{1ex}

  Previously looked at simple finite difference algorithms applied to
  heat and advection equations. At present have no explanation for
  which worked and which did not; for that we need stability theory.

\end{frame}

\begin{frame}
  \frametitle{The results}

  \begin{center}
    \begin{tabular}{c|c c}
      Algorithm & Heat equation & Advection equation \\ \hline
      FTCS & Stable $s < 1/2$ & Unstable \\
      BTCS & Stable $\forall s$ & ?? \\
      FTBS & ?? & Stable $c < 1$
    \end{tabular}
  \end{center}

  Here $s = k \delta / h^2$ for the heat equation
  \begin{equation*}
    \partial_t y - k \partial_{x x} y = 0,
  \end{equation*}
  and $c = v \delta / h$ for the advection equation
  \begin{equation*}
    \partial_t y + v \partial_x y = 0.
  \end{equation*}

\end{frame}

\subsection{Lax Theorem}

\begin{frame}
  \frametitle{Consistency}

  \begin{overlayarea}{\textwidth}{0.8\textheight}
    \only<1|handout:1>
    {
      Theory for multistep methods for IVPs related
      \emph{consistency}, \emph{stability} and \emph{convergence}.
      Have similar concepts and theory for PDEs.
    }
    \only<2-|handout:1->
    {
      Finite difference equation \emph{consistent} if it tends to the
      PDE as grid sizes go to zero \emph{independently}.
    }
    \only<3-|handout:1->
    {

      \vspace{1ex}
      Local error must vanish as $h, \delta \rightarrow 0$ in any way.
    }
    \only<4-|handout:1>
    {

      \vspace{1ex}
      All the methods defined previously are clearly consistent.
    }
    \only<5|handout:2>
    {

      \vspace{1ex}
      The Lax method for the advection equation,
      \begin{equation*}
        y_i^{n+1} = \frac{1}{2} \left( y_{i+1}^n + y_{i-1}^n \right) -
        \frac{c}{2} \left( y_{i+1}^n - y_{i-1}^n \right)
      \end{equation*}
      can be Taylor expanded to give
      \begin{equation*}
        \partial_t y + v \partial_x y = \tfrac{1}{2} \left(
          h^2 / \delta - v^2 \delta \right) \partial_{x x} y + \dots
      \end{equation*}
      which is consistent if, for example, $h / \delta$ is constant, but
      not in general.
    }
  \end{overlayarea}

\end{frame}

\begin{frame}
  \frametitle{Stability}

  As before a finite difference equation is \emph{stable} if it
  produces a bounded solution and unstable otherwise. \pause

  \vspace{1ex}

  Have seen that BTCS is stable for the heat equation, whilst FTCS was
  stable only when $s < 1/2$. For heat equation FTCS is
  \emph{conditionally} stable, whilst BTCS is \emph{unconditionally}
  stable. \pause Have also seen that FTCS is unstable for advection
  equation; it is \emph{unconditionally unstable}. \pause

  \vspace{1ex}

  As for IVPs there may exist genuine unbounded solutions, and
  spurious solutions that swamp genuine solutions.  Often closely
  related to the complex concept of well-posedness: shall ignore this
  issue here.

\end{frame}

\begin{frame}
  \frametitle{Convergence}

  Key concept is \emph{convergence}: in the limit of zero grid spacing
  the finite difference solution converges to the exact
  solution. \pause

  \vspace{1ex}

  Exceptionally difficult to directly prove in general. \pause Instead
  rely on

  \vspace{2ex}

  {\bf Lax's Theorem (1954)}: Given a properly posed linear
  initial-value problem and a finite difference approximation to it
  that is consistent, stability is necessary and sufficient for
  convergence. \pause

  \vspace{2ex}

  Can be extended (with some caveats) to nonlinear problems with
  non-trivial boundary conditions, even when discontinuities form.

\end{frame}

\section{Von Neumann stability theory}

\subsection{Von Neumann stability theory}

\begin{frame}
  \frametitle{Determining stability}

  Von Neumann stability method for linear algorithms and equations:
  \begin{enumerate}
  \item<2-> Make ansatz that solutions of the finite difference
    equation have form
    \begin{equation*}
      y_{\ell}^k =  e^{\alpha j \ell h} q^k, \qquad (\, j = \sqrt{-1}, \alpha \in {\mathbb R}, q \in {\mathbb C} ).
    \end{equation*}
  \item<3-> Substitute this solution into the finite difference
    equation.
  \item<4-> Determine $q$ as a function of $\alpha$, the grid spacings
    $h,\delta$, and known numbers (such as the advection velocity
    $v$).
  \item<5-> Determine under what conditions we have stability, i.e.\ when
    \begin{equation*}
      |q| < 1 \quad \forall \alpha.
    \end{equation*}
  \end{enumerate} \pause[6]
  Can apply method blindly to produce stability result; but how does
  it come about?

\end{frame}

\begin{frame}
  \frametitle{Von Neumann stability}

  \begin{overlayarea}{\textwidth}{0.8\textheight}
    \only<1-2|handout:1>
    {
      Consider a linear initial-value problem, such as heat or
      advection equation.  Assume we have a consistent finite
      difference algorithm (easily checked).
    }
    \only<2|handout:1>
    {

      \vspace{1ex}

      To prove convergence need to show algorithm is
      stable: i.e., numerical result is bounded.
    }
    \only<3-|handout:1->
    {
      Assume we can separate variables and write
      \begin{equation*}
        y = \sum_{m = 0}^{\infty} A_{m}(t) \cos (m x) +
        B_{m}(t) \sin (m x).
      \end{equation*}
    }
    \only<4-|handout:2>
    {
      Implies assumptions about e.g.\ spatial domain (ignore here).
    }
    \only<5-|handout:2>
    {

      \vspace{1ex}

      Use exponential form of trigonometric terms to
      write as
      \begin{equation*}
        y = \sum_{m = -\infty}^{\infty} a_{m}(t) e^{j m x}
      \end{equation*}
      where $a$ is \emph{complex} function (containing the
      two degrees of freedom in $A,B$), and $j = \sqrt{-1}$.
    }
    \only<6-|handout:2>
    {

      \vspace{1ex}

      As problem is linear can superpose solutions, so each
      $a_{m}$ is independent. Hence problem stable \emph{only
        if} $a_{m}$ does not grow with $t$ for \emph{every}
      $m$.  }
  \end{overlayarea}

\end{frame}

\begin{frame}
  \frametitle{Linear stability}

  % \begin{overlayarea}{\textwidth}{0.8\textheight}
  %   \only<1-4>
  %   {
      From linearity consider just the single mode ``solution''
      \begin{equation*}
        y = a_{m}(t) e^{j m x}, \qquad m \in \mathbb{Z}.
      \end{equation*}
      Just need to show that $y$ bounded for general $m$. \pause
    % }
    % \only<2-4>
    % {

      \vspace{1ex}

      Consider a point on grid $\{ x_{\ell} \} = \ell h$, so that
      \begin{equation*}
        y_{\ell} = a_{m}(t) e^{j m \ell h}.
      \end{equation*} \pause
    % }
    % \only<3-4>
    % {
      At first two steps $t^0, t^1 = t^0 + \delta$, have values
      \begin{equation*}
        y_{\ell}^0 = a_{m}(t^0) e^{j m \ell h}, \quad y_{\ell}^1 =
        a_{m}(t^1) e^{j m \ell h}.
      \end{equation*} \pause
    % }
    % \only<4>
    % {
      \emph{Define} the growth from one timestep to the next as
      \begin{equation*}
        q \equiv \frac{y_{\ell}^{k+1}}{y_{\ell}^k} =
        \frac{a_{m}(t^{k+1})}{a_{m}(t^k)} \quad \therefore \quad
        \text{Stability} \Leftrightarrow | q | < 1.
      \end{equation*}
  %   }
  % \end{overlayarea}

\end{frame}

\section{Von Neumann stability in practise}

\subsection{Von Neumann stability in practise}

\begin{frame}
  \frametitle{FTCS for the heat equation}

  Our finite difference algorithm is
  \begin{equation*}
    y_i^{n+1} = ( 1 - 2 s ) y_i^n + s \left( y_{i+1}^n + y_{i-1}^n
    \right).
  \end{equation*} \pause
  By identifying $i \leftrightarrow \ell$ and $n \leftrightarrow k$ we
  can substitute in our ansatz
  \begin{align*}
    && y_{\ell}^k & =  e^{\alpha j \ell h} q^k. \\
    & \implies &
    e^{\alpha j \ell h} q^{k+1} &= (1 - 2 s) e^{\alpha j \ell h} q^k +
    s \left( e^{\alpha j (\ell + 1) h} q^k + e^{\alpha j (\ell - 1) h}
      q^k \right).
  \end{align*} \pause
  Removing the common factor of $e^{\alpha j \ell h} q^k$ gives
  \begin{equation*}
    q = (1 - 2 s) + s \left( e^{\alpha j h} +  e^{-\alpha j h} \right).
  \end{equation*}
  Relates $q$ to (undetermined) $\alpha$ and (free) $h$ and $s$.

\end{frame}

\begin{frame}
  \frametitle{FTCS for the heat equation: 2}

  \begin{equation*}
    y_i^{n+1} = ( 1 - 2 s ) y_i^n + s \left( y_{i+1}^n + y_{i-1}^n
    \right).
  \end{equation*}
  \begin{equation*}
    q = (1 - 2 s) + s \left( e^{\alpha j h} +  e^{-\alpha j h} \right).
  \end{equation*}
  \begin{overlayarea}{\textwidth}{0.7\textheight}
    \only<1-2|handout:1>
    {
      Now bound $|q|$ for \emph{any} $\alpha$
      (may depend on grid spacing).
    }
    \only<2|handout:1>
    {
      More obvious by writing
      \begin{equation*}
        q = 1 - 4 s \sin^2 \left(\frac{\alpha h}{2} \right).
      \end{equation*}
    }
    \only<3-|handout:2>
    {
      \vspace{-1ex}
      \begin{equation*}
        q = 1 - 4 s \sin^2 \left(\frac{\alpha h}{2} \right)
      \end{equation*}
      See that, for arbitrary $\alpha$, $\sin$ term
      bounded between $0$ and $1$.
    }
    \only<4-|handout:2>
    {
      Therefore the limiting values for $q$ are
      \begin{align*}
        q & \le 1 & (\sin^2(\dots) & = 0) \\
        q & \ge 1 - 4 s & (\sin^2(\dots) & = 1).
      \end{align*}
    }
    \only<5-|handout:2>
    {
      Therefore stability, and hence convergence, is given when \vspace{-.5ex}
      \begin{equation*}
        |q| < 1 \Leftrightarrow s < \tfrac{1}{2} \Leftrightarrow
        \frac{\delta}{h^2} < \tfrac{1}{2}.
      \end{equation*}
    }
  \end{overlayarea}

\end{frame}

\section{Summary}

\subsection{Summary}

\begin{frame}
  \frametitle{Summary}

  \begin{itemize}
  \item Von Neumann stability analysis is the key tool for
    understanding the results seen ``experimentally'' in the last
    lecture.
  \item By looking at the growth or decay of individual frequency
    modes we can check the stability of the method; Lax's theorem then
    proves convergence.
  \item For complex equations and / or algorithms, it is often very
    difficult or impossible to apply even Von Neumann stability
    analysis.
  \item If an algorithm does not work in the linear case then it is
    highly unlikely to work for more complex problems!
  \end{itemize}

\end{frame}

\end{document}


\documentclass[11pt, letterpaper]{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{array}
\usepackage{wrapfig}
\usepackage{multirow}
\usepackage{tabu}
\title{Lecture 2: Definition of a fluid and the Continuum Hypothesis}
\begin{document}
\maketitle
In this lecture, we want to answer 2 key questions-
\begin{itemize}
\item What is a fluid?
\item When can we treat it as a continuum?
\end{itemize}

\textbf{Answer}\\

A fluid is a material that cannot sustain an applied shear and we can treat it
as a continuum at length scales and time scales sufficiently larger than the
following

\begin{center}
\begin{tabular}{ c|c|c| }
 & Gas & Liquid \\
\hline
 length & $10^{-7}$ m & $10^{-10}$ m \\
 time & $10^{-10}$ s & $10^{-12}$ s
\end{tabular}
\end{center}

\rule{\textwidth}{0.8pt}

\section{The definition}

Let us try to understand these answers better. We have given an operational definition of fluid - that it cannot sustain shear. Solids can sustain shear because the sheared configuration of the lattice has higher intermolecular potential energy than that at the equilibrium. This increase in potential energy provides a restoring force. Absence of such a force in fluids indicates the absence of a lattice structure and hence fluids flow under shear. However, we must also consider the time for which shear is being imposed.
\begin{itemize}
\item Silly-putty: It bounces off a hard surface indicating that it behaves like a solid if duration of applied shear is $~0.1$s. However, it flows if that duration is of a few hours.

\item Pitch-drop: It takes nearly a decade to form a drop. The viscosity of the pitch is $~10^{7}$ Pa-s. By contrast, $\nu$ for water is just about $10^{-3}$
Pa-s.

\item Glaciers: Deformations occur over hundreds of years. They can flow faster if the ice at the base of the glacier changes into water due to the high
pressure.

\item Earth: The Earth shows a solid like behaviour for processes which occur at small time scales (e.g. earthquakes) and fluid like behaviour for very slow processes (e.g. continental drift)
\end{itemize}

\textbf{P-waves and S-waves}\\

The solid like behaviour of Earth allows propagation of two kinds of waves
during an earthquake- the P-waves and the S-waves. P-waves are longitudinal and travel faster whereas the S-waves are transverse and hence cannot travel through a fluid.

In the event of an earthquake, the inability of the S-waves to travel through fluid creates a zone shielded from the S-waves. This is an important observation because it asserts that the core is in fact liquid. P-waves, however, can travel through the fluid in the inner core but their direction is changed due to refraction. This leads to a P-wave shadow zone.

Another wave that travels through the surface is the Rayleigh wave which is neither transverse not longitudinal but causes the material packets to move in closed orbits. This is quite akin to water waves.\\
\\
\textbf{In essence}\\
\\
These examples show that given enough time, everything flows under shear.
Whether we consider the material as a fluid or as a solid depends on the time scale of the physical process (and the intrinsic time scales of the material) under examination.

\section{The Continuum Hypothesis}

The conditions under which we can treat a fluid as a continuum differ for
liquids and gases. We begin with gases

\subsection{A qualitative picture}
For gases, we have in mind a picture of the gas molecules represented as rigid spheres moving ballistically and undergoing elastic collisions. The speed of these gas molecules is given by

$$
C_g \sim \bigg(\frac{\gamma k T}{m}\bigg)^{1/2} \sim 300m/s
$$

The force between molecules is attractive at large distances(universal dependence of $-1/r^6$) and strongly repulsive at very short distances(often
used Lennard-Jones potential given by $1/r^{12}$). Based on this potential $V(r)$, we can compare the kinetic and potential energy balances in solids, liquids and gases.

For gases the kinetic energy$(=mC_g^2)$ is much greater than potential energy
$$
KE \gg PE \Rightarrow kT \gg V(r)
$$
Whereas for solids
$$
kT \ll V(r)
$$
And in liquids
$$
kT \sim V(r)
$$

In solids the molecules undergo oscillations about their mean positions (known as phonons) due to thermal fluctuations. Likewise, in liquids, they oscillate about their mean positions but since the kinetic and potential energy are comparable in liquids, every once in a while a molecule is able to jump over the potential well and settle into a nearby well. This random walk of the molecule leads to diffusion. At low temperatures, the molecules can no longer hop out of the well and the liquid forms an amorphous solid.

\subsection{Gases}

The characteristic microscopic length for gases is the mean free path of the gas. How do we find it? Let a molecule traveling in a straight path trace a cylinder until it collides with another molecule. The length of this cylinder is the mean free path $\lambda_{MFP}$. If the no. of molecules per unit volume times the volume of the cylinder equals 1, then there must be a collision. Therefore
$$
\pi a^2 n \lambda_{MFP} = 1
$$
$$
\Rightarrow \lambda_{MFP} = \frac{1}{\pi a^2 n}
$$

Using the Ideal gas equation of state to estimate the molecular number density at STP ($n = 2.5 * 10^{19}/cm$ - the Loschimdt number) and the radius of the nitrogen atom($a = 3.6*10^{-10}m$), we obtain $\lambda_{MFP} \sim 100 nm$, a slight overestimate from the actual value of $70nm$.

Using $\lambda_{MFP}$ and $C_g$ we can immediately get the time between two successive collisions - $T_{MFP} = \lambda_{MFP} / C_g \sim 20 ns $

Another quantity of interest that can be estimated here is the diffusive coefficient. It has the unit of $m^2/s$ or $(m/s)$ times $m$. Hence,
$$
D \sim C_g * \lambda_{MFP} \sim 10^{-5} m^2/s
$$

It should be borne in mind that while gas molecules undergo ballistic motions for $l \sim \lambda_{MFP}$, at lengths much larger than the mean free path, even the gases will exhibit diffusion due to the random walk of molecules.

\subsection{Liquids}

The liquid has two scales - that due to oscillations of molecules in the potential well and due to hopping of molecules out of the potential well. During one oscillation, the structure of the liquid remains nearly unchanged and thus we can treat it as a crystal for this short time. This allows us to use the idea of phonons. The phonons are of quantum origin and the energy of these oscillations is therefore of the order of $h\nu$. This implies
$$
h \nu \sim kT \Rightarrow \nu \sim kT/h = 10^{-13} /s
$$
at room temperature. Therefore the time scale becomes $\sim 0.1 ps$. The length scale is given by intermolecular separation which is in this case approximately equal to the molecular diameter $\sim 10^{-10}$. Finally, we can estimate the diffusive constant of liquids. Diffusion in liquids occurs due to hopping of molecules out of potential well. This hopping is an order of magnitude slower than the rattling. Hence, $T_{hopping}\sim 1ps$. In this time, a molecule hops into a nearby cage and hence travels a distance of the order of molecular diameter ($10^{-10}m$). Hence, its speed is nearly $10^{-10}/10^{-12} = 100m/s$. Therefore, the diffusive constant becomes

$$
D \sim 10^{-10}*100 \sim 10^{-8}m^2/s
$$

which is much less than that for a gas, as we might intuitively expect.

\end{document}