\section{Probability Distributions}

    \subsection{What is a probability distribution?}
    A probability distribution is a method of displaying data that displays the probabilities of every possible outcome. We'll need to define a few types of distributions first though, which are uniform and non-uniform distributions.
    \begin{definition}
        Uniform Distribution: A distribution is called uniform when all of the outcomes are equally likely, just like a single die roll.
    \end{definition}
    \begin{definition}
        Non-Uniform Distribution: A distribution is called non-uniform when not all of the outcomes are equally likely, just like rolling the sum of a pair of dice.
    \end{definition}
    The variable we measure when calculating these, is called the random variable, and is denoted as $X$. The various possibilities for $X$ are the outcomes, denoted $x$. We'll typically find the probability distributions that rely on two different types of variables: discrete and continuous variables.
    \begin{definition}
        Discrete Variables: These are values that are typically separate from each other, and have a finite number of possibilities.
    \end{definition}
    \begin{definition}
        Continuous Variables: These are values that have an infinite number of possibilities, such as a length of time.
    \end{definition}

    \subsection{Sigma Notation \& Regular Probability Distributions}
    Sigma Notation, or $\sum_{}^{}$, may look terrifying, but it simply means the sum of a lot of numbers. We use this when finding an expected value of something, which we'll denote as $E(x)$. To find this, we'll use the equation below.
    \begin{equation*}
        E(x) = \sum_{i=1}^{n}x_{i}P(x_{i})
    \end{equation*}
    To break it down, the $i=1$ is the start, the $n$ is the stop, $x_{i}$ is the outcome, and $P(x_{i})$ is the possibility of that outcome.
    To show this in action, let's suppose that in a game, you'll get the amount of points you'll roll on a die.
    To find that, we'll need to use find $E(x)$.
    We know that the probability of rolling each number is $\frac{1}{6}$, and we know all the outcomes, so let's fill out the equation. First, we'll need to set our limits though.
    \begin{equation*}
        E(x) = \sum_{i=1}^{6}
    \end{equation*}
    This means that we must repeat the $x_{i}P(x_{i})$ starting at one, going to six, which would be six times.
    Now we'll expand that below.
    \begin{equation*}
        E(x) = 1(\frac{1}{6}) + 2(\frac{1}{6}) + 3(\frac{1}{6}) + 4(\frac{1}{6}) + 5(\frac{1}{6}) + 6(\frac{1}{6})
    \end{equation*}
    We can also simplify this, as we know it is a uniform distribution. Shown below is the updated equation for this uniform distribution.
    \begin{equation*}
        E(x) = \frac{1}{6}(1 + 2 + 3 + 4 + 5 + 6)
    \end{equation*}

    \subsection{Binomial Distribution}
    Remember binomials from combinations? They look like this $\binom{x}{y}$, and come in handy when working with distributions.
    We have to mention something called a Bernoulli Trial first.
    \begin{definition}
        Bernoulli Trials: This refers to repeated, independent trials that are measured in terms of success, or failure (or other Boolean outcomes).
    \end{definition}
    The probability distribution of the number of successes in Bernoulli Trials is called a binomial distribution, and is found using the equation below.
    \begin{equation*}
        P(x) = \binom{n}{x}p^{x}q^{n-x}
    \end{equation*}
    The equation solves for the probability of exactly $x$ successful trials out of $n$ total trials. In this equation, $p$ represents the probability of success, and $q$ represents the probability of failure.
    To find the expected value for a binomial distribution, we'll resort to using the formula below.
    \begin{equation*}
        E(x) = n\cdot p
    \end{equation*}

    \subsection{Geometric Distributions}
    A geometric distribution is found when an experiment is repeated just \textbf{until the first success}.
    It differs from binomial distributions, as the number of trials is not known at the start.
    Just before the success, there is something called a "waiting period".
    \begin{definition}
        Waiting Period: This is the number of failures before the first and only success.
    \end{definition}
    If you have 8 failures before your success, then you have a waiting period of 8.
    To solve for the probability of success after $x$ failures, we can use the formula below.
    Remember, $x$ represents the waiting period.
    \begin{equation*}
        P(x) = q^{x}p^{1}
    \end{equation*}
    In this equation, the variable $p$ \textbf{must have a max of one} as you can never have more than one success in a Geometric Distribution. To find your expected waiting period, you must solve for $E(x)$, which you can do by using the formula below.
    \begin{equation*}
        E(x) = \frac{q}{p}
    \end{equation*}
    \subsection{Hyper\textemdash Geometric Distribution}
    Hyper-Geometric Distribution are quite similar to Geometric Distributions, yet they are dependent, rather than independent.
    Repeated sampling without replacement exhibits this type of process.
    To find the probability of this, we must use the equation below, where $n$ represents the total number of choices,
    $r$ represents the total number of items chosen,
    $a$ represents the number of successes to choose from, and
    $x$ represents the successes chosen.\\
    \begin{equation*}
        P(x) = \frac{\binom{a}{x}\cdot \binom{n-a}{r-x}}{\binom{n}{r}}
    \end{equation*}
    We can also find the number of expected successes chosen by using the formula below, using the same variables as in the equation above.
    \begin{equation*}
        E(x) = r \cdot \frac{a}{n}
    \end{equation*}


\chapter{ODE Example Revisited}\label{ext:ode-example}
One way by which modelers can improve the quality of parameter estimates is by collecting more data.
We consider what would happen if our measurement equipment were able to capture twice as many observations.
Furthermore, we verify computationally that using more precise measurement equipment improves the precision of the MUD estimate.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Different Measurement Equipment}
Instead of using equipment that operates at $100$Hz, we take $200$ measurements every second, resulting in 400 equispaced observations for $t \in (1,3)$.
All other choices involved in the experiment (assumed equipment tolerance, number of trials, parameter samples), are kept the same.
We refer to this setup wit faster measurement equipment as the ``alternative'' design.

We show the resulting predictions for the signal using MUD points from twenty repeated trials in Figure~\ref{fig:ode-alt-reference} using the first twenty (middle) and all of the measurements (bottom).
The top of the figure shows the solution from the original setup \ref{fig:ode-reference} for visual comparison.
The true signal is well-recovered even with a small subset of the data collected.
By the time all measurements are used, the stability of the solutions\---with respect to the noise that may have polluted them\---is evidenced by the fact that the red lines representing solutions disappear against the backdrop of the true signal in black.

\begin{figure}[htbp]
  \includegraphics[width=\linewidth]{figures/ode/ode_20_reference_solution.png}
  \includegraphics[width=\linewidth]{figures/ode/ode-alt_20_reference_solution.png}
  \includegraphics[width=\linewidth]{figures/ode/ode-alt_400_reference_solution.png}
  \caption{Gray lines are the initial parameter samples.
  The true signal (black) is well-recovered by the MUD estimates (red).
  (Top): First twenty measurements used to solve the original problem.
  (Middle): First twenty measurements used to solve the alternative problem.
  (Bottom): The entire value $\param_i$ ($1\leq i \leq N$), in the sampled parameter set.
  }
  \label{fig:ode-alt-reference}
\end{figure}

To quantify accuracy and stability of the MUD solutions, we solve the problem for the same choices of $S$ as the original problem (with the addition of $S=400$).
We show the resulting error plots for convergence in the right half of Figures~\ref{fig:ode-convergence-alt}, juxtaposed against the original experimental design with $100$Hz equipment.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.475\linewidth]{figures/ode/ode_convergence_mud_obs_mean_comp.png}
  \includegraphics[width=0.425\linewidth]{figures/ode/ode_convergence_mud_obs_var_comp.png}

  \caption{Convergence of the MUD point given $N=1E3$ model evaluations for increasing numbers of observations for randomly placed sensors.
  Convergence rates are estimated using first-order linear regressions in $\text{log}_{10}$-space.
  $100$Hz equipment demonstrates a reduction of uncertainty and improvement in precision as $S$ increases towards $200$.
  We observe the same rates of convergence for the alternative equipment and note the (slightly) lower overall error for equal numbers of measurements ($S=200$ corresponding to $t\in (1,2)$ in this formulation).
  }
  \label{fig:ode-convergence-alt}
\end{figure}

The convergence rates are similar (shown in the legend of \ref{fig:ode-convergence}), and reduction in uncertainty is almost negligible at a given $S$.
However, we note that in the alternative setup, for an equal number of measurements, the time elapsed is half of that in the original due to the different equipment being used.
To this end, we estimate convergence rates with respect to the time elapsed in the experiment rather than number of measurements used, and notice that the alternative setup (orange) exhibits much lower error at a given point of time.
This implies that we can achieve similar results with a shorter observational window by using equipment that allows for faster observations.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\FloatBarrier
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Impact of Equipment Precision}
To achieve higher precision in the estimate of the MUD point, one can use more precise measurement equipment.
We expect that a method designed to address parameter estimation would see an improvement in accuracy  if the data is collected with more precise instruments.
Here we show that this is indeed the case for the time-series example introduced earlier by considering choices of $\tau = 0.1, 0.05, 0.01, \text{ and } 0.005$ for $\mathbb{P}( \abs{\xi} < \tau ) = 99\%$ to select our $\sigma$ in our normal additive noise model.
We sequentially incorporate $S=5, 10, 15, 20, 25, 50, 100, \text{ and } 200$ measurements and study the error in our estimate of $\paramref$.

\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.475\linewidth]{figures/ode/ode_convergence_mud_std_mean.png}
  \includegraphics[width=0.425\linewidth]{figures/ode/ode_convergence_mud_std_var.png}

  \caption{Convergence of the MUD point given $N=1E3$ model evaluations incorporating measurements at a fixed point in time.
  As more precise measurements are incorporated, the accuracy and precision of the MUD solution improves.
  }
  \label{fig:ode-convergence-std}
\end{figure}

In Figure~\ref{fig:ode-convergence-std}, we study the absolute error's mean and variance as our measurement equipment gets more precise (lower tolerance), for both the $100$Hz (ode) and $200$Hz (ode-alt) variants of sensors we are simulating.
In the left half of the figure, we find that the convergence rates for the two designs are nearly identical but the equipment which records twice as many measurements has a persistent reduction in error.
The right half shows the convergence in variance of the absolute error, and the vertical displacement between the two designs is visually difficult to distinguish.
However, as evidenced in the legend annotations of Fig~\ref{fig:ode-convergence-std}, the alternative design (faster equipment) exhibits an increase in the rate of convergence from 1.87 to 1.92.

We have shown that the Data--Consistent approach to solving parameter identification problems manages to generalize to problems involving time-series data from a single Quantity of Interest.
We now turn our attention to an example where instead of temporal measurements, we incorporate spatial data to solve another 1-D parameter identification problem.


\chapter{Markov Chain Monte Carlo}
\section{Monte Carlo}
Monte Carlo is a general term for computational techniques that use random
numbers.
Monte Carlo can be used in classical and Bayesian statistics. A special kind
of Monte Carlo
called Markov Chain Monte Carlo (MCMC) was one of the main reasons
for the revival of Bayesian statistics in the second half of the 20th century.
Before MCMC became popular, one of the major drawbacks of the Bayesian approach
was that some of the calculations were too hard to do. MCMC enables us to solve
a wide range of Bayesian problems which cannot be solved using analytical
methods.

\subsection{Summaries}
So far, we have represented our probability distributions (prior and posterior)
in a computer by using a vector of possible parameter values and a corresponding
vector of probabilities. For example, suppose we have a single parameter $\theta$
and we have worked out the posterior distribution by using a Bayes' Box. This
will give us a vector {\tt theta} of possible $\theta$ values and a corresponding
vector {\tt post} containing the posterior probabilities. Well, one thing we could
do is plot the posterior distribution, resulting in a plot like the one in
Figure~\ref{fig:normal}.
\begin{verbatim}
plot(theta, post, xlab="Theta", ylab="Posterior Probability")
\end{verbatim}

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.5]{Figures/normal.pdf}
\caption{\it A posterior distribution can be represented in a computer by a discrete
set of possible parameter values, and the corresponding probabilities.\label{fig:normal}}
\end{center}
\end{figure}

If we want to obtain some summaries, we could do it like so:
\begin{verbatim}
post_mean = sum(theta*post)
post_sd = sqrt(sum(theta^2*post) - post_mean^2)
\end{verbatim}

However,
there is an alternative way of representing this posterior distribution in a
computer. It may not be immediately obvious why this is a good idea,
because there is nothing wrong with the tried and true method we have used
so far. But this second method has the advantage that it continues to
work well on much bigger problems, such as when we have more than one parameter.
With more than one parameter, the `vector of possible solutions'' approach
can fail very dramatically.

Our new way of representing a probability distribution in a computer will be
via {\it Monte Carlo} samples.
Instead of having two vectors (one of $\theta$ values and one of the
corresponding probabilities),
imagine we had some method to compute a random sample of $\theta$ values, drawn
from the posterior distribution in Figure~\ref{fig:normal}.
There would only be one vector {\tt theta}. So how would we
know there is greater probability around $\theta=1$? Well, {\it more elements
of the {\tt theta} vector would be near 1}.
Instead of carrying around a second vector of
probabilities, we understand that more probable regions will simply contain
more points. Say our vector of random
samples is called {\tt theta}. Then we can look at the posterior distribution
by plotting a histogram of samples:
\begin{verbatim}
hist(theta, breaks=100)
\end{verbatim}
The histogram looks something like the one in Figure~\ref{fig:normal2}.
\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.5]{Figures/normal2.pdf}
\caption{\it The posterior distribution for a parameter $\theta$ can also be
represented by a random sample of $\theta$ values drawn from the posterior
distribution. Some $\theta$ values are more probable than others, which is
encoded by certain values appearing more frequently in the sample.
\label{fig:normal2}}
\end{center}
\end{figure}
We can also get our summaries, but the code looks different
(it's actually easier than before!):
\begin{verbatim}
post_mean = mean(theta)
post_sd = sd(theta)
\end{verbatim}
Because of the randomness involved in generating the {\tt theta} values,
the summaries aren't exact. For example, I know the actual posterior mean
and standard deviation in this example were both 1, but the values obtained
from the Monte Carlo samples were 0.9604 and 1.0008, respectively. However,
this doesn't matter much because the results indicate $\theta$ is probably
somewhere around 1, with an uncertainty of about 1. The error introduced by
using random samples is much smaller than the amount of uncertainty inherent
in the posterior distribution itself. For example, if I summarised the posterior
distribution by saying $\theta = 0.9604 \pm 1.0008$, for almost all practical
purposes the conclusion is exactly the same as the true version of the summaries
$\theta = 1 \pm 1$.

In this discussion, we haven't answered the question of how to actually generate
random samples of $\theta$ from the posterior distribution. This is the job
of Markov Chain Monte Carlo.

\begin{framed}
{\bf
The purpose of Markov Chain Monte Carlo is to generate random samples of
parameter values drawn from the posterior distribution. This makes it very easy
to compute summaries even if you have more than one unknown parameter.}
\end{framed}

\section{Multiple Parameters}
MCMC becomes extremely useful when we begin to look at Bayesian models involving
more than one unknown parameter.
Having posterior samples makes the process of {\it marginalisation}
much easier. Imagine we wanted to infer two parameters, called $a$ and $b$,
from
data $x$. Bayes' rule (parameter estimation version) would give us the posterior
distribution:
\begin{eqnarray}
p(a, b | x) \propto p(a, b)p(x|a,b)
\end{eqnarray}

However, what if you didn't really care about the value of $b$ but
only really wanted to measure $a$? The terminology for this is that $b$ is a
{\it nuisance parameter}: you need it to define the model, but ultimately you
are not really interested in knowing its value.
What you need in this case is the
{\it marginal} posterior distribution for $a$ (that is, the posterior
distribution for $a$ on its own, not the joint distribution with $b$).
This can be obtained using the sum rule. The result is:
\begin{eqnarray}
p(a | x) &=& \int_b p(a, b|x) \, db
\end{eqnarray}
or
\begin{eqnarray}
p(a | x) &=& \sum_{b} p(a, b|x)
\end{eqnarray}
depending on whether the set of possible $b$ values is continuous or discrete.
Before MCMC, these integrals or sums usually couldn't be done without making
certain choices purely for mathematical convenience (e.g. choosing the prior
to be a certain kind of distribution only because it will make the maths work
out, rather than it being a good description of your prior beliefs).

Samples of parameter values drawn from the posterior distribution
(achieved using MCMC) make this hard problem much easier. We no longer need
to worry about mathematical convenience.
See Figure~\ref{fig:marginalisation} for an example showing how Monte Carlo
sampling makes marginalisation trivial.

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.5]{Figures/marginalisation.pdf}
\caption{\it An example of a posterior distribution for two parameters, $a$ and
$b$. The left panels show the joint posterior distribution (which has a
correlation) and the marginal posterior distribution for $a$, obtained by
integrating over all the possible $b$ values. The top right panel contains
random samples
(points) drawn from the posterior distribution for $a$ and $b$.
The only step needed to get the
marginal distribution for $a$ (lower right panel) is to ignore the $b$-values of
the points!
\label{fig:marginalisation}}
\end{center}
\end{figure}

\section{The Metropolis Algorithm}
The Metropolis algorithm is the most basic MCMC method.
The ideas behind it are fairly simple, yet Metropolis forms the basis of a large
number of more advanced MCMC methods. In STATS 331 we will study the basic ideas
behind how the Metropolis algorithm works, which will involve a small amount
of Markov chain theory. We will also look at a small amount of R code
which implements the Metropolis algorithm, but for solving practical problems
it is more convenient to use the JAGS program\footnote{JAGS uses a number
of MCMC methods internally, including Metropolis, ``Gibbs Sampling'', and
``Slice Sampling'', which we will not study in this course.}.

The Metropolis algorithm was invented in the 1950s by physicists
(including Nicholas Metropolis, for whom the algorithm is named), who used it
to do calculations in the field of statistical mechanics. This intriguing field
focuses on calculating the macroscopic (large scale) properties of matter from
knowledge of the small-scale properties. For example, if you know water is
composed of H$_2$O molecules, you can use statistical mechanics to figure out
what will happen if you have a lot
of water molecules. For example, it will freeze at 0 degrees Celsius and boil at 100 degrees Celsius.

It took
many decades before people started to realise the Metropolis algorithm was
useful in Bayesian
statistics as well as statistical mechanics. The Bayesian approach
seemed very elegant and
useful to many people, but it could always be criticised as unworkable,
because you usually had to do difficult or
impossible integrals when solving practical problems
(to summarise the posterior, or to get rid of nuisance
parameters). MCMC changed all that, and is one of the reasons for the
explosion in the popularity of Bayesian statistics beginning in the 1990s.

The basic idea of MCMC is that we want a method which will travel between
different possible states (such as the possible hypotheses/parameter values in
a Bayesian analysis). We want the amount of time spent in
any particular state to be proportional to the posterior probability of the state.
The computer ``explores'' the set of possible parameter values, spending a lot
of time in the regions with high posterior probability, and only rarely visiting regions
of low posterior probability. Figure~\ref{fig:mcmc} shows an example of MCMC
applied to a problem with only two possible hypotheses or parameter values.
Nobody would actually use MCMC on such a small problem, but it is helpful for
explaining how MCMC works.

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.65]{Figures/mcmc.pdf}
\caption{\it An illustration of the basic idea behind MCMC. Imagine we had a
Bayes' Box with two possible hypotheses, and we knew the
{\tt prior $\times$ likelihood} values. The amount of time the MCMC program will
spend in each state is proportional to the posterior probability of the state.
In this example, the MCMC algorithm was in State 1 three times and in State 2
seven times. Using this, we could estimate the posterior probability of State 2
as being 0.7. This estimate would become more accurate if we ran the MCMC for
more iterations.\label{fig:mcmc}}
\end{center}
\end{figure}

\subsection{Metropolis, Stated}
The Metropolis algorithm is given below. The first thing to do is start
{\it somewhere} in the ``parameter space'' (set of possible parameter values).
You then
{\it propose} to move somewhere else. There is an acceptance probability $\alpha$
that determines whether to accept the proposal. If the proposal is better
($h$, the prior times likelihood value is higher), then you accept it, and the
proposed state becomes the new state of the algorithm.
If the
proposal is worse, you can also accept it, but the probability of accepting is
given by the ratio of the unnormalised posterior probabilities, i.e. $h'/h$.
For example, if the proposed point is 1/3 as good as the current point, the
acceptance probability is 1/3. If the proposed point is rejected, the original
point remains the state of the algorithm, and gets counted again in the results.

The Metropolis algorithm works because it makes transitions towards low
probability
states rare, while transitions towards high probability states are common, because
of the acceptance probability. It is hard to move into an improbable state, so
not much time will be spent there. Over time, the fraction of time spent in
any given state is equal to the posterior probability of the corresponding
hypothesis.

\begin{framed}
\begin{itemize}
\item Start in some state $\theta$
\item Generate a ``proposal'' state $\theta'$ from a {\it proposal distribution} $q$ (assumed symmetric so that $q(\theta'|\theta)=q(\theta|\theta')$)
\item With probability $\alpha = \textnormal{min}(1, h'/h)$, replace the current
state with the proposed state
\item Repeat
\end{itemize}
\end{framed}

\section{A Two State Problem}
We will now study how the Metropolis algorithm works on a very simple
example, namely the two-ball problem from the beginning of the notes. There
are two hypotheses and the posterior probabilities are $1/3$ and $2/3$.
It is important to note that MCMC is not actually needed for a problem this
simple, but it is a good test case to see precisely how an MCMC algorithm works.
When we solve real data analysis problems with JAGS,
we won't have to think too
much about how the MCMC works, but can concentrate instead on the Bayesian
statistics problem at hand.

Let's
call the less probable hypothesis ``State 1'' and the more probable hypothesis
``State 2'' for the purposes of this section.
What we need is a Markov process that will spend 1/3 of the time in State 1 and
2/3 of the time in State 2. The Metropolis algorithm described above will
do what we need. The main thing we need to compute is the acceptance probability
$\alpha_{ij}$ for a proposed transition from state $i$ to state $j$ where
$i, j \in \{1, 2\}$.
The acceptance probability $\alpha$ for a proposed move from state $i$ to
state $j$ is given by
\begin{eqnarray}
\alpha_{ij} = \textnormal{min}\left(1, \frac{h_j}{h_i}\right)
\end{eqnarray}
where $h_i$ and $h_j$ are proportional to the posterior probabilities of
states $i$ and $j$ respectively. If the proposal is to move
to an equal or better (higher posterior probability)
state ($h_j \geq h_i$) then the acceptance probability is 1.
If the proposal is to move to a less probable state then the acceptance probability
is $h_j/h_i$, the ratio of the two posterior probabilities\footnote{Note that
this algorithm can be used even if the marginal likelihood is unknown, because
only ratios of posterior probabilities are needed. This is useful because the
marginal likelihood is sometimes very hard to calculate in multi-parameter
problems.}.

The transition probability is the probability
of being in state $j$ at the next iteration given that you are in state $i$
at the current iteration. The transition probability is given by the product
rule:
\begin{eqnarray}
p_{ij} = q_j \alpha_{ij}\label{eq:transition}
\end{eqnarray}
for $i \neq j$.
The transition matrix of the Markov chain is a matrix with all the different
$p_{ij}$ values in it:
\begin{eqnarray}
\mathbf{P} &=&
\left[
\begin{array}{cc}
p_{11} & p_{12}\\
p_{21} & p_{22}
\end{array}
\right]
\end{eqnarray}
In our particular case, we can work out the off-diagonal elements of
$\mathbf{P}$ using Equation~\ref{eq:transition}:
\begin{eqnarray}
\mathbf{P}
&=&
\left[
\begin{array}{cc}
 & \frac{1}{2} \times 1\\
\frac{1}{2}\times\frac{1}{2} &
\end{array}
\right]
\end{eqnarray}
The diagonal elements can be found by knowing the rows of $\mathbf{P}$ must
sum to 1. We must be in {\it some} state at the next iteration. Therefore the
transition matrix of our Markov chain in this two-state problem is:
\begin{eqnarray}
\mathbf{P}
&=&
\left[
\begin{array}{cc}
\frac{1}{2} & \frac{1}{2}\\
\frac{1}{4} & \frac{3}{4}
\end{array}
\right]\label{eq:p_matrix}
\end{eqnarray}

A Markov chain with a small number of possible states can be represented
graphically using a {\it transition diagram}, as in
Figure~\ref{fig:transitions}.

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.65]{Figures/transitions.pdf}
\caption{\it A transition diagram for a Markov Chain with two possible states.
The states (which correspond to hypotheses in Bayesian inference) are drawn
as circles labelled ``1'' and ``2''. When the algorithm is in a particular state
(e.g. State 1), and we apply one step of the Metropolis algorithm,
the probability it will be in State 1 is $p_{11}$ and the probability it
will be in State 2 is $p_{12}$.
MCMC works by making it easy to move into states with high
posterior probability, and hard to move out of them.
\label{fig:transitions}}
\end{center}
\end{figure}

\section{The Steady-State Distribution of a Markov Chain}
Once we have the transition matrix, we can work out the {\it steady state
distribution} of the Markov chain. Imagine, instead of starting the MCMC
from an arbitrary initial state, you have a probability distribution for the
initial state. After applying one iteration of the Metropolis algorithm, the
probability distribution for the updated state will usually be different from
the probability distribution for the initial state.

However, one special probability distribution, called the {\it steady state
distribution}, does not change after you apply an MCMC update. If your initial
point was drawn from the steady state distribution, then subsequent points will
also be drawn from the steady state distribution. In MCMC the steady state
distribution should be the same as the posterior distribution.

Imagine we are using MCMC on our two-state problem, and our initial position
is State 1 with probability $v_1$ and State 2 with probability $v_2$. What
is the probability of being in State 1 at the next iteration? There are two
ways for that to happen: by starting in State 1 and then making a transition
from $1\to 1$, or by starting in State 2  and making a transition from $2\to 1$.
The total probability is then:
\begin{eqnarray}
P(\textnormal{State 1 after iteration}) &=& v_1p_{11} + v_2p_{21}.
\end{eqnarray}
Similarly for State 2:
\begin{eqnarray}
P(\textnormal{State 2 after iteration}) &=& v_1p_{12} + v_2p_{22}.
\end{eqnarray}
If $v_1$ and $v_2$ happened to be the steady state distribution, then these
probabilities would also be $v_1$ and $v_2$. This gives us two simultaneous
equations
\begin{eqnarray}
v_1 &=& v_1p_{11} + v_2p_{21}\\
v_2 &=& v_1p_{12} + v_2p_{22}
\end{eqnarray}
which can also be written in matrix form as $\mathbf{v}\mathbf{P} = \mathbf{v}$
where $\mathbf{v} = \left(v_1, v_2\right)$ and $\mathbf{P}$ is the transition
matrix\footnote{Mathematics students might recognise this equation, which says
$\mathbf{v}$ is the left-eigenvector of $\mathbf{P}$, with eigenvalue 1.
An alternative is to write the stationary distribution as a column vector and
use $\mathbf{P}^T\mathbf{v} = \mathbf{v}$.}.
These two simultaneous equations can be solved for $v_1$ and $v_2$.
However there is a third constraint, that is $v_1 + v_2 = 1$. Since there
are three equations but two unknowns, it seems like the problem might be
``over-determined'', but that is not actually the case because the transition
matrix rows are not all linearly independent.
It is left as an exercise for the reader to show the steady state
distribution for our Markov chain (Equation~\ref{eq:p_matrix}) is in fact
equal to the posterior distribution, so
$\mathbf{v} = \left(\frac{1}{3}, \frac{2}{3}\right)$.

\section{Tactile MCMC}
In class we will do ``Tactile MCMC'', which is an implementation of the
Metropolis algorithm using coins and dice instead of the random number generators
provided in computer software such as R. This is a good way to get a feel for
the flow of MCMC algorithms.

In STATS 331, when we want to use MCMC in practice, we will use the JAGS program
rather than using Metropolis directly. JAGS is a general purpose MCMC program
which allows you to solve fairly complex problems without a large amount of
programming.


\section{Time Complexity of MST Algorithms}

%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Prim \emph{vs.} Kruskal (Problem 6.4)}
  \begin{exampleblock}{Prim \emph{vs.} Kruskal (Problem 6.4)}
	\begin{itemize}
	  \item Array \emph{vs.} heap
	  \item $m = O(n)$ \emph{vs.} $m = \Omega(n^2)$
	\end{itemize}

	\[
	  T(n,m) = O(n + n T(\text{deleteMin}) + m T(\text{decreaseKey}))
	\]
  \end{exampleblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{MST on special graphs (Problem 6.3)}
  \begin{exampleblock}{MST on special graphs (Problem 6.3)}
	\begin{enumerate}
	  \item $K$-bounded degree graph
	  \item Planar graph
	\end{enumerate}
  \end{exampleblock}

  \[
	(1)\; m \le \frac{nk}{2}
  \]

  \[
	(2)\; m \le 3n - 6
  \]

  \begin{alertblock}{Reference}
	\begin{itemize}
	  \item ``Finding Minimum Spanning Trees'' by David Cheriton and Robert Tarjan, 1976 (linear on planar graph).
	\end{itemize}
  \end{alertblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Prim on special graphs (Problem 6.1)}
  \begin{exampleblock}{Prim on special graphs (Problem 6.1)}
	\[
	  E = \set{v_1v_i \mid i = 2 \dots n}, W(v_1v_i) = 1
	\]
  \end{exampleblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Prim on special graphs (Problem 6.2)}
  \begin{exampleblock}{Prim on special graphs (Problem 6.2)}
	\begin{enumerate}
	  \item $G = K_n$
	  \item $W(v_iv_j) = n + 1 -i, 1 \le i < j \le n$
	\end{enumerate}
  \end{exampleblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%


\documentclass[12pt]{article}

\usepackage{graphicx}
\usepackage{latexsym,algorithm,algorithmic}
\usepackage{times}
\usepackage{amsmath,amssymb,mathtools}
\usepackage{mathptmx}      % use Times fonts if available on your TeX system

\setlength{\textwidth}{7in}
\setlength{\textheight}{9in}
\setlength{\oddsidemargin}{-0.5in}
\setlength{\evensidemargin}{-0.5in}
\setlength{\topmargin}{-0.5in}

\newcommand{\weightvec}{\textbf{w}}
\newcommand{\edge}[2]{{u\rightarrow{}v}}
\newcommand{\edgeuv}{{\edge{u}{v}}}

\newcommand{\vek}[1]{\textbf{#1}}
\newcommand{\ddw}{\frac{\partial}{\partial\vek{w}}}
\newcommand{\M}{\textrm{M}}
\newcommand{\dM}{\textrm{dM}}
\newcommand{\df}{\textbf{df}}
\newcommand{\dt}{\textbf{dt}}
\newcommand{\vphi}{\vec{\phi}}

\begin{document}

\section{Derivation: PPR and its derivative}

Notation: $\vek{s}$ is seeds, $\M$ is transition matrix, $\vek{w}$ are
parameters, and $\vek{p}^\infty$ is the PPR stationary distribution,
$\vek{p}^t$ is a point in the power-rule iteration for computating
PPR, and $\vek{d}^t$ is the partial derivative of $\vek{p}^t$ wrt the
parameters $\vek{w}$.

\begin{eqnarray}
\vek{p}^{t+1} & \equiv & \alpha \vek{s} + (1-\alpha) \M^\top \vek{p}^t \\
\vek{d}^t & \equiv &  \ddw \vek{p}^t
\end{eqnarray}

Note $\vek{p}^t$ and $\vek{d}^t$ have different dimensions: while
$\vek{p}^t_u$ is a scalar score for $u$ under PPR, $\vek{d}^t_u$ is a
vector, giving the sensitivity of that score to each parameter in
$\vek{w}$.

$\M$ is defined as follows.  There is a weight vector $\vek{w}$, and
for an edge $\edgeuv$, there is a feature vector $\vphi_{uv}$, which is
used to define a basic score $s_{uv}$ for the edge, which is passed
thru a squashing function $f$, e.g., $f(x)\equiv e^x$, and then
normalized to form $\M$.

\begin{eqnarray}
 s_{uv}   & \equiv & \vphi_{uv} \cdot \vek{w}  \\
 t_u      & \equiv & \sum_{v'} f(s_{uv'}) \\
 \M_{u,v} & \equiv & \frac{ f(s_{uv}) }{ t_u }
\end{eqnarray}

We can define $\vek{d}^t$ recursively:

\begin{eqnarray}
\vek{d}^{t+1}  & =  &  \ddw \vek{p}^{t+1} \\
               & =  &  \ddw \left( \alpha \vek{s} + (1-\alpha) \M^\top \vek{p}^t \right) \\
               & =  &  (1-\alpha) \ddw \M^\top \vek{p}^t \\
               & =  &  (1-\alpha) \left( (\ddw \M^\top) \vek{p}^t  + \M^\top \ddw\vek{p}^t \right) \\
               & =  &  (1-\alpha) \left( (\ddw \M^\top) \vek{p}^t  + \M^\top \vek{d}^t \right)
\end{eqnarray}

Now let's look at $\ddw\M$, which I'll denote $\dM$ below.  Note that
each $\dM_{uv}$ is a vector, again giving the sensitivity of the
weight $\M_{uv}$ to each parameter in $\vek{w}$.

\begin{eqnarray}
\dM_{uv}   & = & \ddw \frac{ f(s_{uv}) }{ t_u } \\
           & = & \frac{1}{t_u ^2} \left( t_u \ddw f(s_{uv}) - f(s_{uv}) \ddw t_u \right)
\end{eqnarray}

To continue this, define $\df$ and $\dt$ as the vectors

\begin{eqnarray}
 \df_{uv} & \equiv & \ddw f(s_{uv}) = f'(s_{uv}) \vphi_{uv} \label{eqn:fprime} \\
 \dt_{u}  & \equiv & \ddw t_u = \sum_{v'} \df_{uv'}
\end{eqnarray}

Note that $\df_{uv}$ has no more non-zero components than
$\vphi_{uv}$, so it is sparse, and $\dt_{u}$ is also sparse, but
somewhat less so.

We can continue the derivation as

\begin{eqnarray}
\dM_{uv}   & = & \frac{1}{t_u ^2} \left( t_u \ddw f(s_{uv}) - f(s_{uv}) \ddw t_u \right) \\
           & = & \frac{1}{t_u ^2} \left( t_u \df_{uv} - f(s_{uv}) \dt_u \right)
\end{eqnarray}

This is a little different from the old algorithm: there is always an
(implicit) reset with probability $\alpha$, and that reset probability
can be increased by learning by weighting the features for the
(explicit) reset links that are present already in the graph.  No min
$\alpha$!

\section{Computation}

\begin{table} % Computing M and dM
\hrule
\begin{enumerate}
\item For each node/row $u$
  \begin{enumerate}
  \item  $t_u = 0$
  \item  $\dt_u = \vek{0}$, an all-zeros vector
  \item For each neighbor $v$ of $u$
    \begin{enumerate}
    \item $s_{uv} = \vek{w} \cdot \vphi_{uv}$,  a scalar.

      In detail: For $i\in \vphi_{uv}$: increment $s_{uv}$ by $\vek{w}_i\vphi_i$

    \item $t_u  +\!\!= f(s_{uv})$, a scalar
    \item $\df_{uv} = f'(s_{uv}) \vphi_{uv}$, a vector, as sparse as $\vphi_{uv}$

      In detail: For $i\in \vphi_{uv}$: set $\df_{uv,i} = \vphi_f * c$, where $c=f'(s_{uv})$

    \item  $\dt_u +\!\!= \df_{uv}$, a vector, as sparse as $\sum_{v'} \vphi_{uv'}$

      In detail: For $i\in \df_{uv}$: increment $\dt_{u,i}$ by $\df_{uv,i}$

    \end{enumerate}
  Now $t_u = \sum_{v'} f(s_{uv'})$ and $\dt_{u} = \sum_{v'} \df_{uv'}$
  \item For each neighbor $v$ of $u$ create the vector

\[ \dM_{uv}  = \frac{1}{t_u ^2} \left( t_u \df_{uv} - f(s_{uv}) \dt_u \right)
\]

Or in detail: For $i\in\dt_u$: $\dM_{uv,i} = \frac{1}{t_u ^2} \left( t_u \df_{uv,i} - f(s_{uv}) \dt_u,i \right)$.

There aren't any dimensions $i$ that are present in $\df_{uv}$ but not
$\dt_u$, since $\dt_u$ is a summation.  Also create the scalar

\[ \M_{uv}  = \frac{f(s_{uv})}{t_u}
\]

  You can now discard the intermediate values like $t_u$, $\dt_u$, $\df_{uv}$.

  \end{enumerate}
\end{enumerate}
\caption{Computing $\M$ and $\dM$} \label{alg:mat}
\hrule
\end{table}

Since $\dM$ and $M$ are reused many times, we should compute and store
$M$ and $\dM$ first.  This will expand the size of the graph somewhat:
in particular, we will need to store, not only the active edges $u,v$
and their features $\vphi_{uv}$, but also $\dM_{uv}$, which includes
weights for all features of vertexes $v'$ that are siblings of $v$
(i.e., there is an edge $u,v'$).  I'm not sure how bad this will be in
practice: perhaps we should estimate it for some of our test cases.
After this, operations should run in time linear in the size of the
new, less sparse graph encoded in $\dM$.  I believe that this scheme
also makes operations like the APR learning more modular.

Computing $M$ and $\dM$ is one pass over the graph, shown in Table
\ref{alg:mat}.

Then you can start with $\vek{p}^0 = \vek{d}^0 = \vek{0}$ and iterate

\begin{eqnarray}
\vek{p}^{t+1}  & =  & \alpha \vek{s} + (1-\alpha) \M^\top \vek{p}^t \\
\vek{d}^{t+1}  & =  & (1-\alpha) \left( \dM^\top \vek{p}^t  + \M^\top \vek{d}^t \right)
\end{eqnarray}

In more detail, the iteration for the updates on $\vek{p}$ are shown
in Table~\ref{alg:updates}.

\begin{table} % updating p and d
\hrule

Updating $\vek{p}$:
\begin{enumerate}
\item $\vek{p}^{t+1} = \vek{0}$
\item For each node $u$
  \begin{enumerate}
  \item $\vek{p}^{t+1}_u +\!\!= \alpha \vek{s}_u$
  \item For each neighbor $v$ of $u$
    \begin{enumerate}
    \item $\vek{p}^{t+1}_v +\!\!= (1-\alpha) \vek{p}^t_u \M_{uv} $
    \end{enumerate}
  \end{enumerate}
\end{enumerate}

Updating $\vek{d}$:
\begin{enumerate}
\item $\vek{d}^{t+1} = \langle \vek{0}, \ldots, \vek{0} \rangle$ --- i.e., for each node $u$ there is an all-zeros
  vector of weights.
\item For each node $u$
  \begin{enumerate}
  \item For each neighbor $v$ of $u$
    \begin{enumerate}
      \item For each $i$ in $\dM_{vu}$

      \[ \vek{d}^{t+1}_{v,i} +\!\!= (1-\alpha) \vek{p}^t_u \dM_{uv,i} \]
      \item For each $i$ in $\vek{d}^t_{v}$

      \[ \vek{d}^{t+1}_{v,i} +\!\!= (1-\alpha) \vek{d}^t_{u,i} \M_{uv} \]
    \end{enumerate}
  \end{enumerate}
\end{enumerate}
\caption{Updates for $\vek{d}$ and $\vek{p}$} \label{alg:updates}
\hrule
\end{table}

\section{Loss functions and lazy regularization}

For SRW each example is a triple $(\vek{s},P,N)$ where $\vek{s}$ is
the seed distribution, $P=\{a^1,\ldots,a^I\}$ are the positive (a-ok?)
examples, and $N=\{b^1,\ldots,b^J\}$ are the negative (bad?) examples.
I use $\vek{p}$ for the PPR distribution starting at $\vek{s}$, and
write $\vek{p}[u]$ for $\vek{p}_u$ if I run out of space for
subscripts.

The loss function is
\begin{equation}
L(\vek{w}) \equiv - \Bigg (\sum_{k=1}^I \log \vek{p}[a^k] + \sum_{k=1}^J \log (1 - \vek{p}[b^k]) \Bigg) + \mu R(\vek{w})
\end{equation}
where $R(\vek{w})$ is the regularization, eg
$R(\vek{w})\equiv||\vek{w}||^2_2$. This means loss increases as the objective function decreases, where the objective function is proportional to the probability of either hitting a positive solution or not-hitting a negative solution.  Once we have $\vek{d}$ it's easy
to compute the gradient of this as

\begin{eqnarray}
\ddw L(\vek{w}) & \equiv & - \Bigg (\sum_{k=1}^I \frac{1}{\vek{p}[a^k]}\vek{d}[a^k]
                                  - \sum_{k=1}^J \frac{1}{1 - \vek{p}[b^k]}\vek{d}[b^k] \Bigg) + \mu \ddw R(\vek{w})
\end{eqnarray}

We can split this into two parts: the empirical loss gradient, which is

\[
-\Bigg (\sum_{k=1}^I \frac{1}{\vek{p}[a^k]}\vek{d}[a^k]
     - \sum_{k=1}^J \frac{1}{1 - \vek{p}[b^k]}\vek{d}[b^k] \Bigg)
\]
and the regularization gradient,
\[
 \mu \ddw R(\vek{w})
\]
If an example doesn't contain all features then the empirical gradient
will be a sparse vector, but the regularization gradient will be
dense.  So the following code might be more efficient for SGD than
just computation of the full gradient and taking a step in that
direction.

\begin{enumerate}
\item Maintain a ``clock'' counter $m$ which is incremented when each
  example is processed. Also maintain a history $\vek{h}_i$ which
  says, for each feature $i$, the last time $t$ an example containing
  $i$ was processed.
\item When a new example $(\vek{s},P,N)$ arrives at time $t$
  \begin{enumerate}
    \item For each feature active in the example, initialize it, if
      necessary, and them perform the regularization-loss gradient
      update $t-\vek{h}_i$ times.
    \item Peform the empirical-loss update, using
      the new weights.
  \end{enumerate}
\item When you finish learning at final time $T$, consider every
  feature $i$, and perform the regularization-loss gradient update
  $T-\vek{h}_i$ times.  Then write out the final parameters.
\end{enumerate}

\section{Inference: PPR and APR}

Inference using requires only $\M$, which at theorem-proving time is
computed on-the-fly.  That is, whenever you need a row of $\M$ (the
set of values $\M[u,v]$ for all $v$ near $u$) you simply compute the
normalized outlinks of $u$.

The power-iteration version of PPR, which in the codebase is called
the \texttt{PPRProver}, simply iterates this step until convergence
(or for a fixed number of iterations), starting with
$\vek{p}^0=\vek{0}$.

\begin{eqnarray}
\vek{p}^{t+1} & \equiv & \alpha \vek{s} + (1-\alpha) \M \vek{p}^t \\
\end{eqnarray}

Breaking this down, the one-step update is the following.

\begin{enumerate}
\item $\vek{p}^{t+1} = \vek{0}$
\item For each key $u$ with non-zero weight in $\vek{s}$:
  \begin{enumerate}
  \item $\vek{p}^{t+1}[u]$ += $\alpha \vek{s}[u]$
  \end{enumerate}
\item For each key $u$ with non-zero weight in $\vek{p}^t$:
  \begin{enumerate}
  \item For each node $v$ near $u$ in $\M$:
\(
       \vek{p}^{t+1}[v] \mbox{~+=~} (1-\alpha) \M[u,v] \vek{p}^t[u]
\)
  \end{enumerate}
\end{enumerate}

The approximate PageRank is based on a more primitive \texttt{push}
operation.  The full approximate PageRank algorithm also uses
$\vek{d}$, a vector of node degrees.

\begin{enumerate}
\item Let $\vek{p}= \vek{0}$ and $\vek{r}= \vek{s}$
\item While there is some vertex $u$ such that $r(u)\geq{}\epsilon
  \vek{d}(u)$:
  \begin{enumerate}
  \item Perform the \texttt{push}($u$) operation:
    \begin{enumerate}
      \item Save the current value of $u$ in $\vek{r}$: $ru = \vek{r}[u]$
      \item $\vek{p}[u]$ += $\alpha \vek{r}[u]$
      \item $\vek{r}[u]$ *= $(1-\alpha)$
      \item For each node $v$ near $u$ in $\M$:
\(
       \vek{r}[v] \mbox{~+=~} (1-\alpha) \M[u,v] ru
\)
    \end{enumerate}
  \end{enumerate}
\end{enumerate}

In the old implementation, $\vek{d}[u]$ was computed and cached each
time a new key $u$ was added to $\vek{r}$.  In the new implementation
of the prover, this caching is done by the proof graph, so there's no
need to do it in the prover.

The old DPR prover implementation computed this approximation by
finding $u$'s with large $\vek{r}[u]$ using a variation of the
standard depth-first prover: essentially, you traversed the tree
depth-first, and whenever you hit a node with $\vek{r}[u]$ below
threshold, you stopped.  Factoring this in the algorithm used the same
initial values of $\vek{p}$ and $\vek{r}$ but then called a recursive
routine \texttt{proveState}$(u_0)$, where $u_0$ is the initial query
node (aka the initial state of the proof graph).

William Wang's got a new version which calls this iteratively with
smaller and smaller $\epsilon$'s, which seems to work faster.  That
suggests that it's helpful to do pushes on the nodes with larger
$\vek{r}$ values first.  With that in mind, another thing we could
consider would be storing $u$'s in a heap/priority queue, ordered by
$\vek{r}$ values.  I know this idea has been used in the past but I
don't know how much it helps.  Obviously there's an overhead for the
heap but my guess is that's not going to dominate.

\bigskip

\noindent Function \texttt{proveState}$(u)$:
\begin{enumerate}
\item If $r(u)\geq{}\epsilon \vek{d}(u)$:
  \begin{enumerate}
  \item Perform the \texttt{push}$(u)$ operation
  \item For each $v$ near $u$ (i.e., each subgoal of $u$): \texttt{proveState}$(v)$
  \end{enumerate}
\end{enumerate}

\section{Architectural Comments}

\emph{ [Katie's comments inline and signed with -k]}

\noindent
The routines that are suggested are:
\begin{itemize}
\item Loading: Compute $\M$ and $\dM$ from a graph and $\vek{w}$.  It
  seems necessary, altho a little non-modular, to initialize and
  create new features while the graph is read in. \emph{ [Do this here, or in SGD? -k]} We should be able
  to estimate performance pretty well after this step.

Inputs: graph and current values of parameters $\vek{w}$.

Outputs: $\M$ and $\dM$, and a list of all active features $i$.

Parameters: $f$, $f'$; method to init new features $i$ for $f$, clock
time and lazy-regularization function.

\item Extended inference: compute $\vek{p}$ and $\vek{d}$.

Inputs: $\vek{s}$, $\vek{w}$, $\M$ and $\dM$.

Outputs: $\vek{p}$ and $\vek{d}$.

\item SGD: Initialize new features $i$, use clock time and
  lazy-regularization function to update their weights, and update
  $\vek{w}$ using the empirical (non-gradient) loss.

Inputs: $\vek{p}$, $\vek{d}$, and $P$, $N$.

Outputs: Modified $\vek{w}$.

\end{itemize}

\begin{table}
\hrule
\begin{verbatim}
class Graph {

  // names of features

  String[] featName;

  // space for labels for feature weights on edges, each is a
  // feature index and a weight

  int[] label_featIndex;
  double[] label_featWeight;

  // space for edges, each is a destination node and a
  // list of labels, which point into the label space

  int[] edge_dst;
  int[] edge_labels_lo;
  int[] edge_labels_hi;

  // space for list of neighbors of nodes, each
  // points into the edge space

  int [] node_near_lo;
  int [] node_near_hi;

  // list of nodes in graph
  int node_lo, node_hi;

  // space for weight derivatives, which are structurally
  // just like labels.  these maybe should be vectors since
  // we don't know the size of this space in advance...?

  int[] deriv_featIndex;
  double[] deriv_featDeriv;

  // list of derivative features for M[u][euv]
  int[][] dM_lo;
  int[][] dM_hi;
}
\end{verbatim}
\caption{Proposed data structure for graph} \label{alg:data}
\hrule
\end{table}

\begin{table}
\hrule
\begin{verbatim}
for (int u = g.node_lo; u < g.node_hi; u++) {
  // euv is edge from u to v
  for (euv = g.node_near_lo[u]; euv < g.node_near_hi[u]; euv++) {
    int v = g.edge_dst[euv];
    // luvk is k-th label on edge euv
    for (luvk = g.edge_labels_lo[euv]; luvk = g.edge_labels_hi[euv]; luvk++) {
      int i = g.label_featIndex[luvk];
      double w = g.label_featWeight[luvk];
      ...
    }
  }
}
for (int d = g.dM_lo[u][euv]; d < g.dM_hi[u][euv]; d++) {
  i = g.deriv_featIndex[d];
  double v = g.deriv_featDeriv[d];
  // v is partial/(partial feature i) if M[u][v]
  ...
}
\end{verbatim}
\caption{Accessing the proposed data structure for graph} \label{alg:access}
\hrule
\end{table}

A proposed data structure which would be very efficient is shown in
Table~\ref{alg:data} and \ref{alg:access}.  Some things that aren't
clear now are:
\begin{itemize}
\item Where does the feature-index $\leftrightarrow$ feature-name
  symbol table go?  Probably not in the graph as shown here.
\item Is $\M$ and $\dM$ part of the graph or a different structure?
\end{itemize}
Also a clarification: \texttt{M} and \texttt{dM\_lo,dM\_hi}, are not
dense 2-matrixes, of size quadratic in the number of nodes, $n$.
Instead \texttt{M} is a length-$n$ array of variable-size arrays, and
\texttt{M[u]}, for node index \texttt{u}, is an array of size $m_u$,
where $m_u$ is the number of neighbors (edges away from) \texttt{u}.
And \texttt{dM\_lo,dM\_hi} are parallel structures.

A final note: the squashing function $f$ and its derivative $f'$ are
only used in computing $\df$ in Equation~\ref{eqn:fprime}.  One future
extension to ProPPR might be to have facts with weights defined by
parameters: e.g., facts like \texttt{sim(a,b)} which when used would
lead to an edge with a computed weight $s_{uv}$ based on an internal
set of parameters $\vec{\lambda}$ (think of a learned similarity
subroutine).  A modular way of dealing with this might be worth
thinking through.  Conceptually, $\vec{\lambda}$ is a part of
$\vek{w}$: one could have an interface which given an edge $u,v$ with
weight $s_{uv}$ returns $f'(s_{uv})\ddw s_{uv}$ in terms of components
of $\vec{\lambda}$.  For training you would also need to pass in an
update of the $\vec{\lambda}$ features to the learning subroutine....

\clearpage
\begin{appendix}
\section{Linear Algebra proof of imperative procedure for p and d updates}

\subsection{Computing the p update}
\begin{align*}
\vek{p}^{t+1} =& \alpha \vek{s} + (1-\alpha)\vek{p}^t \M\\
\vek{p}^t \M =& \left[ \begin{array}{ccccc}
p_0 & p_1 & p_2 & \ldots & p_n
\end{array} \right] \left[ \begin{array}{cccc}
\\M_{00} & \M_{01} & \M_{02} & \ldots \\
\\M_{10} & \M_{11} & \M_{12} & \ldots \\
\M_{20} & \M_{21} & \M_{22} & \ldots \\
\vdots & \vdots & \vdots
\end{array} \right]\\
=& \left[ \begin{array}{cccc} p^{t+1}_0 & p^{t+1}_1 & p^{t+1}_2 & \ldots \end{array} \right]\\
p^{t+1}_0 =& p_0 \M_{00} + p_1 \M_{10} + p_2 \M_{20} + \ldots\\
p^{t+1}_1 =& p_0 \M_{01} + p_1 \M_{11} + p_2 \M_{21} + \ldots\\
\end{align*}

If we set $p^{t+1}_u$, we must iterate across, over all terms of the first element, then all terms of the second element, etc:

$$p^{t+1}_u += (1-\alpha) p_v \M_{vu}$$

Given our data structure for $\M$ however, constructing $\M_{vu}$ is difficult. To use $\M_{uv}$ instead, we can set $p^{t+1}_v$ and iterate down, computing the first term of all elements, then the second term of all elements, etc:

$$p^{t+1}_{v} += (1-\alpha) p_u \M_{uv}$$

\subsection{Computing the d update}

Computing $d$ proceeds in much the same way:

\begin{align*}
\vek{d}^{t+1}_i =& (1-\alpha)(\vek{p}^t \dM_i + \vek{d}^t_i \M)
\end{align*}

We'll start by deconstructing the first term only:

\begin{align*}
\vek{p}^t \dM_i =& \left[ \begin{array}{cccc} p_0 & p_1 & p_2 & \ldots \end{array} \right] \
\left[ \begin{array}{cccc}
\dM_{00,i} & \dM_{01,i} & \dM_{02,i} & \ldots \\
\dM_{10,i} & \dM_{11,i} & \dM_{12,i} & \ldots \\
\dM_{20,i} & \dM_{21,i} & \dM_{22,i} & \ldots \\
\vdots & \vdots & \vdots
\end{array} \right]\\
=& \left[ \begin{array}{cccc} d^{t+1}_{0,i} & d^{t+1}_{1,i} & d^{t+1}_{2,i} & \ldots \end{array} \right]\\
d^{t+1}_{0,i} =& p_0 \dM_{00,i} + p_1 \dM_{10,i} + p_2 \dM_{20,i} + \ldots\\
p^{t+1}_{1,i} =& p_0 \dM_{01,i} + p_1 \dM_{11,i} + p_2 \dM_{21,i} + \ldots\\
\end{align*}

We have the same choice of setting $d^{t+1}_{u,i}$ or $d^{t+1}_{v,i}$, with the same dependence on using $\dM_{vu,i}$ or $\dM_{uv,i}$. Since we must use $\dM_{uv,i}$ due to the way our data structure is constructed, we must update $d^{t+1}_{v,i}$:

$$ d^{t+1}_{v,i} += (1-\alpha) \vek{p}^t_u \dM_{uv,i} $$

The same principles hold for the second term:

$$ d^{t+1}_{v,i} += (1-\alpha) \vek{d}^t_{u,i} \M_{uv} $$

\end{appendix}

\end{document}


\documentclass{article}
\usepackage{amsmath, amsfonts, graphicx}
\begin{document}
\section{Problem Statement}
We consider a special type of piecewise affine system, that the boundaries of the hybrid modes intersect at the origin. Mathematically, the system has this form
\begin{align}
	\dot{x} = f(x) = A_ix \text{ if } P_ix\le 0
\end{align}
Notice that the righ-hand side of $P_ix\le0$ is zero vector, which guarantees that all mode boundaries pass through the origin. This also implies that $f(kx) = kf(x)$ if $k>0$.

We want to find a Lyapunov function satisfying
\begin{align}
	V(x) > 0 \forall x\neq 0\\
	\dot{V}(x) < 0 \forall x\neq 0\\
	V(x) = 0
\end{align}

\textbf{Goal}: We want to prove that if there exists a piecewise affine Lyapunov function in the form
\begin{align}
	V(x) = g_i^Tx + h_i \text{ if } C_ix\le d_i \label{eq:piecewise_affine_lyapunov}
\end{align}
Namely in the $i'th$ piece $\mathcal{P}_i=\{x | C_ix \le d_i\}$, the Lyapunov function is an affine function of state $x$. Then there has to exist a piecewise linear Lyapunov function in the form
\begin{align}
	\bar{V}(x) = \bar{g}_i^Tx \text{ if } \bar{C}_i x \le 0
\end{align}
Namely the piecewise linear Lyapunov function $\bar{V}$ is linear inside each piece, and each piece is a conic region originating from the origin.

\section{Proof}
The first step is to prove that the origin is at the vertex of some pieces $\mathcal{P}_i$ in Lyapunov function $V(x)$. To prove this, suppose that the origin lies in the $i'th$ piece $\mathcal{P}_i$. Since the Lyapunov function is linear within each piece, and $V(x)$ obtains its global minimal at the origin (implied by the strict positivity of Lyapunov function), we know that the origin must be a vertex of the polyhedron region $\mathcal{P}_i$ (This is because the minimal of an LP must be obtained at the vertex of the polyhedron region). Hence the origin is at the vertices of many neighbouring pieces, as shown in the plot \ref{fig:global_convergence1}, it cannot be in the strict interior of any polyhedron piece $\mathcal{P}_i$.
\begin{figure}
	\centering
	\includegraphics[width=0.4\textwidth]{./global_convergence1.eps}
	\caption{The origin (black dot) has to be the shared vertex of the neighbouring pieces ($\mathcal{P}_1, \mathcal{P}_2, ..., \mathcal{P}_6$ in the figure), it cannot be in the strict interior of any piece.}
	\label{fig:global_convergence1}
\end{figure}
As $V(0)=0$, we know that $h_i = 0$ if $\mathcal{P}_i$ has the origin as a vertex.

Next we show that we can construct a piecewise linear Lyapunov function $\bar{V}$ using the piecewise affine Lyapunov function $V$. The procedure is as follows
\begin{enumerate}
	\item Select each of the piece $\mathcal{P}_i$ that neighbours the origin.
	\item For each of the piece $\mathcal{P}_i$ in step 1, compute the cone of $\mathcal{P}_i = \{x | C_ix\le d_i\}$, by keeping the boundaries of $\mathcal{P}_i$ that contain the origin, and remove the boundaries that don't contain the origin. We denote this conic region as $\bar{\mathcal{P}}_i=\{x|\bar{C}_i\le 0\}$.
	\item Within each of the conic region $\bar{\mathcal{P}}_i$, the new Lyapunov function is defined as $\bar{V}(x) = g_i^Tx \text{ if } \bar{C}_i \le 0$.
\end{enumerate}
This procedure is shown in Fig.\ref{fig:global_convergence2}.
\begin{figure}
	\centering
	\includegraphics[width=0.4\textwidth]{./global_convergence2.eps}
	\caption{We construct the new Lyapunov function, by first forming the conic region of each piece that neighbours the origin. Hence we extend the region $\mathcal{P}_1, ...\mathcal{P}_6$ to infinity, and remove the pieces that don't neighbour the origin ($\mathcal{P}_7, ...\mathcal{P}_{11}$).}
	\label{fig:global_convergence2}
\end{figure}
Now we need to prove that $\bar{V}$ is a valid Lyapunov function. Notice that in the polyhedron piece $\mathcal{P}_i$, the Lyapunov function $V(x)$ is $V(x) = g_i^Tx$. Now for any state $x$ in $\mathcal{P}_i$, we shoot a ray starting from the origin and passing $x$. We denote the state on this ray as $kx, k > 0$. Obviously this entire ray is in the conic piece $\bar{\mathcal{P}}_i$ (by the definition of the cone). We also know that $\bar{V}(kx) = k\bar{V}(x) = kV(x)>0$. The first equality is because the function $\bar{V}$ is linear inside each piece, the second equality is because $\bar{V}(x) = V(x) \text{ if } x\in\mathcal{P}_i$. Finally, we know that $\dot{\bar{V}}(kx) = k\dot{\bar{V}}(x) = k\dot{V}(x) < 0$. The first equality is because inside each conic piece $\partial V/\partial x$ is a constant, and the dynamics scales proportionally w.r.t $k$ ($f(kx) = kf(x)$).The second equality is because $\bar{V}(x)$ is the same as $V(x)$ in the polyhedron region $\mathcal{P}_i$. As a result, we prove that this new function $\bar{V}(x)$ is a valid Lyapunov function. This piecewise linear function $\bar{V}$ can be represented by neural network without bias term.
\end{document}