\subsection{Data} \label{subsec:data_linear_regression}
In this project, we are fitting $2D$ polynomials to the Franke function with gaussian noise. The Franke function is defined as
\begin{equation}
    \begin{split}
        f(x_1,x_2) := \frac{3}{4}\exp{\left(-\frac{(9x_1-2)^2}{4}-\frac{(9x_2-2)^2}{4}\right)} \\ + \frac{3}{4}\exp{\left(-\frac{(9x_1+1)^2}{49}-\frac{(9x_2+1)}{10}\right)} \\+ \frac{1}{2}\exp{\left(-\frac{(9x_1-7)^2}{4}-\frac{(9x_2-3)^2}{4}\right)} \\ - \frac{1}{5}\exp{\left(-(9x_1-4)^2-(9x_2-7)^2 \right)}
    \end{split}
\end{equation}
To this Franke function, we add noise sampled from a Gaussian distribution $\mathcal{N}(0, var)$ to get a noisy Franke function:  $\tilde{f}(x_1,x_2)$. Figure \ref{fig:franke_function} shows a plot of the Franke function with and without noise.
\begin{figure}[H]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/franke_function.png}
  \caption{}
  \label{fig:franke}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/franke_function_noise.png}
  \caption{}
  \label{fig:franke_noise}
\end{subfigure}
\caption{Plot of the Franke function with noise (b) and without noise (a). The variance of the added noise is 0.1.}
\label{fig:franke_function}
\end{figure}


At Stage~3, an approximation of the quantity of interest is constructed using
the interpolation algorithm presented in this section. The algorithm constitutes
the core of the framework proposed in this chapter, and it features a sparse
structure, hierarchical construction, and hybrid adaptivity. The benefits of
these characteristics are interconnected and can be summarized as follows:
\one~the ability to efficiently address multidimensional problems, \two~the
ability to progressively refine the approximation, and \three~the ability to
perform this refinement strategically by virtue of fine-grained error control.

Hierarchical interpolation is introduced in \xref{hierarchical-interpolation},
and here we rely heavily on the results discussed in that section. The
mathematics presented in \xref{hierarchical-interpolation} and below are based
on the development in \cite{klimke2006, ma2009, jakeman2012}.

Consider the quantity of interest \g as a function of \vz via $\transform$ as
shown in \sref{frame-transformation}. Assume that \g belongs to $\continuous{[0,
1]^\nz}$, the space of continuous functions on $[0, 1]^\nz$; the assumption is
not limiting in practice. As shown in \xref{hierarchical-interpolation}, \g can
be approximated by means of the following interpolant:
\begin{equation} \elab{frame-interpolant}
  \g \approx \interpolant{\nz}{\ls}(\g)
  = \interpolant{\nz}{\ls - 1}(\g) + \sum_{\vi \in \Delta\sparseindex{\nz}{\ls}} \sum_{\vj \in \Delta\tensorindex{\nz}{\vi}} \Delta(\g \circ \transform)(\vx_{\vi \vj}) e_{\vi \vj}
\end{equation}
where $\vi \in \natural^\nz$ is called a level index; $\vj \in \natural^\nz$ is
called an order index; $\set{\vx_{\vi \vj}}$ and $\set{e_{\vi \vj}}$ are
collocation nodes and basis functions, respectively; $\set{\Delta(\g \circ
\transform)(\vx_{\vi \vj})}$ are hierarchical surpluses defined in
\eref{interpolant-sparse-surplus}; and $\Delta\sparseindex{\nz}{\ls}$ and
$\Delta\tensorindex{\nz}{\vi}$ are index sets defined in
\eref{interpolant-sparse-index-delta} and \eref{interpolant-tensor-index-delta},
respectively.

Due to the reasons clarified in \sref{frame-adaptivity}, the interpretation of
\eref{frame-interpolant} used in this chapter is different from the one used in
\xref{hierarchical-interpolation}. Specifically, $\ls \in \natural$ no longer
represents an interpolation level but rather an interpolation step. Accordingly,
$\interpolant{\nz}{\ls}(\g)$ represents the interpolant obtained by a certain
interpolation step. In addition, all the index sets discussed here are generally
subsets of their full-fledged counterparts defined in
\xref{hierarchical-interpolation}.

Let us now turn to the choice of collocation nodes and basis functions.

\subsection{Collocation Nodes}
\slab{frame-grid}

In order to gain computational efficiency, the integration grid should be fully
nested, which is explained in \xref{hierarchical-interpolation}. Such a grid can
be constructed using the family of Newton--Cotes rules \cite{ma2009}. In one
dimension, a Newton--Cotes rule is a set of equidistant nodes on $[0, 1]$. There
are two types of Newton--Cotes rules: open and closed. The only difference
between the two types is that the latter includes the endpoints (zero and one),
whereas the former does not.

Technically, in order to be able to proceed to hierarchical interpolation, the
chosen rules have to fulfill a certain condition, which is discussed in
\xref{hierarchical-interpolation} and can be seen in
\eref{interpolant-tensor-exactness}. Closed Newton--Cotes rules satisfy this
condition, and they are the ones used in the original version of local
adaptivity presented in \cite{ma2009}. The open rules, on the other hand, do not
fulfill the condition close to the boundaries of the unit interval. However,
according to our experience, open Newton--Cotes rules are a viable option, since
they perform well in practice, which is also noted in \cite{klimke2006}. In
fact, we are able to obtain better results with open rules and, therefore,
present them here.

\inputfigure{frame-grid}
The open Newton--Cotes rule of level $i \in \natural$ is
\[
  \X^1_i = \set{\x_{ij}}{j \in \tensorindex{1}{i}}
\]
where
\begin{align*}
  & \x_{ij} = \frac{j + 1}{n_i + 1}, \\
  & \tensorindex{1}{i} = \set{i - 1}_{i = 1}^{n_i}, \text{ and} \\
  & n_i = 2^{i + 1} - 1.
\end{align*}
\fref{frame-grid} depicts the first three levels of this rule. It can be seen
that the number of nodes grows as 1, 3, 7, and so on, and that the rule is fully
nested. In multiple dimensions, collocation nodes are formed as shown in
\eref{interpolant-tensor}.

\subsection{Basis Functions}
\slab{frame-basis}

The basis functions that go hand in hand with open Newton--Cotes rules are
piecewise linear functions. For $i = 0$ and $j = 0$,
\[
  e_{00}(\x) = 1.
\]
For $i > 0$ and $j = 0$ (close to the left endpoint),
\[
  e_{i0}(\x) =
  \begin{cases}
    2 - \left(n_i + 1\right) \x, & \text{if } \x < \frac{2}{n_i + 1}; \\
    0, & \text{otherwise}.
  \end{cases}
\]
For $i > 0$ and $j = n_i - 1$ (close to the right endpoint),
\[
  e_{i, n_i - 1}(\x) =
  \begin{cases}
    \left(n_i + 1\right) \x - n_i + 1, & \text{if } \x > \frac{n_i - 1}{n_i + 1}; \\
    0, & \text{otherwise}.
  \end{cases}
\]
In other cases,
\[
  e_{ij}(\x) =
  \begin{cases}
    1 - \left(n_i + 1\right)|\x - \x_{ij}|, & \text{if } |\x - \x_{ij}| < \frac{1}{n_i + 1}; \\
    0, & \text{otherwise}.
  \end{cases}
\]
The basis functions that correspond to the first three levels of one-dimensional
interpolation are depicted in \fref{frame-basis}. In multiple dimensions, basis
functions are formed as shown in \eref{interpolant-tensor}, which results in
\[
  e_{\vi \vj}(\vx) = \prod_{k = 1}^n e_{i_k j_k}(x_k).
\]

\inputfigure{frame-basis}
Additionally, let us calculate the volumes (the integrals over the whole domain)
of the aforementioned piecewise linear functions; these volumes are required in
the continuation. For $i = 0$ and $j = 0$,
\[
  w_{00} = 1.
\]
For $i > 0$ and $j \in \set{0, n_i - 1}$,
\[
  w_{ij} = \frac{2}{n_i + 1}.
\]
In other cases,
\[
  w_{ij} = \frac{1}{n_i + 1}.
\]
The volumes of multidimensional basis functions are products of the
corresponding one-dimensional volumes as follows:
\begin{equation} \elab{frame-volume}
  w_{\vi \vj} = \prod_{k = 1}^\nz w_{i_k j_k}.
\end{equation}

Now, imagine a function that is nearly flat on the first half of $[0, 1]$ and
rather irregular on the other. Under these circumstances, it is natural to
expect that, in order to attain the same accuracy, the first half should require
many fewer collocation nodes than the other one; recall the example given in
\fref{frame-motivation}. However, if we followed the usual construction
procedure described in \xref{hierarchical-interpolation}, we would not be able
to benefit from this idiosyncrasy. Both sides would be treated equally, and all
the nodes of each interpolation level would be added to the interpolant, which
is wasteful. The solution to this problem is to make the interpolation algorithm
adaptive, which we discuss next.

\subsection{Hybrid Adaptivity}
\slab{frame-adaptivity}

In order to make the algorithm adaptive, we first need to decide on a criterion
used for measuring the accuracy of the interpolant $\interpolant{\nz}{\ls}(\g)$
in \eref{frame-interpolant} at any point in $[0, 1]^\nz$. Then, when refining
the interpolant, instead of evaluating the quantity of interest \g at all
possible nodes, we choose only those that are located in the regions with poor
accuracy as indicated by the criterion.

We already have a good foundation for building the above-mentioned criterion.
Hierarchical surpluses, which are introduced in
\xref{hierarchical-interpolation} and defined in
\eref{interpolant-sparse-surplus}, are natural indicators of the interpolation
error: they are the difference between the true values of \g and those estimated
by $\interpolant{\nz}{\ls}(\g)$ at the nodes of the underlying integration grid.
Therefore, hierarchical surpluses can be recycled in order to effectively
identify problematic regions.

We proceed as follows. First, a score is assigned to each node $\vx_{\vi \vj}$
or, equivalently, to each pair of a level index \vi and an order index \vj as
follows:
\begin{equation} \elab{frame-score}
  s_{\vi \vj} = \absolute{\Delta(\g \circ \transform)(\vx_{\vi \vj}) w_{\vi \vj}}
\end{equation}
where $\Delta(\g \circ \transform)(\vx_{\vi \vj})$ is the surplus at the node as
defined in \eref{interpolant-sparse-surplus}, and $w_{\vi \vj}$ is the volume of
the corresponding basis function as shown in \eref{frame-volume}. The above
score is utilized for guiding the algorithm as explained below.

Each hierarchical interpolant $\interpolant{\nz}{\ls}$ is characterized by a set
of level indices $\sparseindex{\nz}{\ls}$, and each level index $\vi \in
\sparseindex{\nz}{\ls}$ by a set of order indices
$\Delta\tensorindex{\nz}{\vi}$. At each interpolation step $\ls \in \natural$, a
single level index
\[
  \vi_{\ls} \in \sparseindex{\nz}{\ls - 1}
\]
is chosen where $\sparseindex{\nz}{-1} = \set{\v{0}}$. This index gives birth to
$\Delta\sparseindex{\nz}{\ls}$ and $\set{\Delta\tensorindex{\nz}{\vi}}{\vi \in
\Delta\sparseindex{\nz}{\ls}}$, forming the increment given on the right-hand
side of \eref{frame-interpolant}.

The level index set $\Delta\sparseindex{\nz}{\ls}$ contains so-called admissible
forward neighbors of the chosen $\vi_{\ls}$. The forward neighbors of an
index~\vi are given by
\[
  \set{\vi + \v{1}_k}_{k = 1}^\nz
\]
where $\v{1}_k \in \set{0, 1}^\nz$ is a vector whose elements are zero except
for element~$k$, which is equal to unity. An index~\vi is called admissible if
its inclusion into the index set $\sparseindex{\nz}{\ls}$ under consideration
keeps the set admissible. Finally, $\sparseindex{\nz}{\ls}$ is admissible if it
satisfies the following condition \cite{klimke2006}:
\[
  \vi - \v{1}_k \in \sparseindex{\nz}{\ls}
\]
for $\vi \in \sparseindex{\nz}{\ls}$ and $k \in \set{\range{1}{\nz}}$ where the
cases with $i_k = 0$ need no check.

Let us now turn to the content of $\Delta\tensorindex{\nz}{\vi}$ where $\vi =
\vi_{\ls} + \v{1}_k$ for some $k$. It also contains admissible forward
neighbors; however, they are order indices, and their construction is different
from the one used in the case of $\Delta\sparseindex{\nz}{\ls}$. These indices
are identified by inspecting the backward neighborhood of \vi, which is
analogous to the forward one but in the other direction. For each backward
neighbor $\vi - \v{1}_m$ and each $\vj \in \Delta\tensorindex{\nz}{\vi -
\v{1}_m}$, we first check the condition
\[
  s_{\vi - \v{1}_m, \vj} \geq \error{s}
\]
where \error{s} is a user-defined constant referred to as the score error. If
the condition holds, the forward neighbors of \vj in dimension~$k$ are added to
$\Delta\tensorindex{\nz}{\vi}$. This procedure is illustrated in
\fref{frame-grid} for open Newton--Cotes rules in one dimension. The arrows
emerging from a collocation node connect the node with its forward neighbors. In
general, each node has two forward neighbors in each dimension. The order
indices of these nodes are
\begin{align*}
  & (j_1, \dots, 2 j_k \phantom{{} + 2}, \dots, j_{\nz}) \text{ and} \\
  & (j_1, \dots, 2 j_k + 2,              \dots, j_{\nz}).
\end{align*}
The above refinement procedure should be performed for each level index $\vi \in
\Delta\sparseindex{\nz}{\ls}$ with respect to each dimension $k \in
\set{\range{1}{\nz}}$.

The choice of $\vi_{\ls} \in \sparseindex{\nz}{\ls - 1}$ at each step \ls in
\eref{frame-interpolant} is made as follows. First, each index can be picked at
most once. The rest is resolved by prioritizing the candidates. It is reasonable
to compute the priority of a level index~\vi based on the scores of the order
indices associated with this level index, that is, based on the scores of
$\tensorindex{\nz}{\vi}$. We set the priority to the average score
\[
  s_{\vi} = \frac{1}{\cardinality{\Delta\tensorindex{\nz}{\vi}}} \sum_{\vj \in \Delta\tensorindex{\nz}{\vi}} s_{\vi \vj}.
\]
Thus, at each step \ls, the index~\vi with the highest $s_{\vi}$ is promoted to
$\vi_{\ls}$.

The final question to answer is the stopping condition of the approximation
process in \eref{frame-interpolant}. Apart from the natural constraints on the
maximum number of function evaluations and the maximum interpolation level (the
original \ls in \xref{hierarchical-interpolation}), we rely on the following
criterion. Given two additional user-defined constants \error{a} and \error{r},
which are referred to as the absolute and relative errors, respectively, the
process is terminated as soon as
\begin{equation} \elab{frame-stop}
  \max_{\vi \vj} \absolute{\Delta(\g \circ \transform)(\vx_{\vi \vj})} \leq \max \set{\error{a}, \error{r} (\g_\maximum - \g_\minimum)}
\end{equation}
where $\g_\minimum$ and $\g_\maximum$ are the minimum and maximum observed
values of \g, respectively. The left-hand side of \eref{frame-stop} corresponds
to the largest surplus whose level index has not been refined yet, that is, has
not been considered as $\vi_{\ls}$ at some step \ls. The above criterion is an
adequate technique for curtailing the interpolation process, since it is based
on the actual progress.

The adaptivity presented in this subsection is referred to as hybrid, since it
combines features of global and local adaptivity \cite{jakeman2012}. Local
adaptivity operates on the level of individual nodes \cite{ma2009} and is
discussed in \sref{frame-motivation}. By contrast, global adaptivity operates on
the level of individual dimensions \cite{klimke2006}. The intuition behind
global adaptivity is that, in general, the input variables manifest themselves
(impact \g) differently, and the interpolation algorithm is likely to benefit by
prioritizing those variables that are the most influential.

\conclusioncut
So far, we have formalized an efficient algorithm for adaptive hierarchical
interpolation in multiple dimensions. The main equation is
\eref{frame-interpolant} where $\Delta(\g \circ \transform)(\vx_{\vi \vj})$,
$\vx_{\vi \vj}$, and $e_{\vi \vj}$ are the ones in
\eref{interpolant-sparse-surplus}, \sref{frame-grid}, and \sref{frame-basis},
respectively; and the interpolation is undertaken according to the rules in
\sref{frame-adaptivity}. Next, we discuss a few implementation details.

\subsection{Implementation}

The life cycle of interpolation has roughly two stages: construction and usage.
The construction stage invokes \g at a set of collocation nodes and produces
certain artifacts. The usage stage estimates the values of \g at a set of
arbitrary points by manipulating these artifacts. In this subsection, we provide
the pseudocode for the two stages in order to give a better sense of the
technique.

Let us first make a general note. According to our experience, it is beneficial
to the clarity and ease of implementation to collapse the two sums in
\eref{frame-interpolant} into one. This requires storing a level index $\vi =
(i_k) \in \natural^\nz$ and an order index $\vj = (j_k) \in \natural^\nz$ for
each node. It is also advantageous to encode each pair $(i_k, j_k)$ as a single
unsigned integer, which, in particular, eliminates excessive memory usage. In
multiple dimensions, this results in a vector $\v{\iota} = (\iota_k) \in
\natural^\nz$, which we simply call an index. Our encoding is
\[
  \iota_k = i_k \lor (j_k \ll \n{\mathrm{bits}})
\]
where $\lor$ and $\ll$ stand for the bitwise \up{OR} and logical left shift,
respectively, and \n{\mathrm{bits}} is the number of bits reserved for storing
level indices, which can be adjusted according to the maximum permitted depth of
interpolation.

\inputalgorithm{frame-construction}
The pseudocode for the construction stage is given in \aref{frame-construction}.
The input is a subroutine called Algorithm~G that evaluates $\g \circ
\transform$. The output is a structure \texttt{interpolant} that contains the
artifacts of the interpolation. These artifacts are a set of tuples
$\set{(\v{\iota}_k, \Delta\g(\vx_{\v{\iota}_k}))}$, which is a comprehensive
description of a hierarchical interpolant. The pseudocode works as follows.

Line 1: Each iteration of the loop corresponds to an interpolation step \ls in
\eref{frame-interpolant}. The progress is captured by a structure
\texttt{state}. The \texttt{strategy} object represents the adaptation strategy
utilized, and it operates in accordance with \sref{frame-adaptivity}. The
\texttt{Continue?} method of \texttt{strategy} checks if any of the stopping
conditions is satisfied, in which case the process is terminated.

Line 2: The \texttt{Next} method of \texttt{strategy} consumes the previous
state and returns the initial state of the ongoing interpolation step. In
particular, it populates the \texttt{indices} field of \texttt{state} with the
indices of the step. The rest of the loop's body populates the rest of
\texttt{state}'s fields so that \texttt{strategy} can adequately execute its
functionality at the beginning of the next iteration.

Line 3: The \texttt{grid} object represents the interpolation grid that is being
utilized, and its \texttt{Compute} method calculates the collocation nodes
$\set{\vx_{\v{\iota}_k}}$ that correspond to the given indices
$\set{\v{\iota}_k}$; see \sref{frame-grid}.

Line 4: Algorithm~G evaluates $\g \circ \transform$ at the collocation nodes.
This is by far the most time consuming operation of the algorithm, as \g is
generally expensive to execute. This operation is also a prominent candidate for
parallelization, since the algorithm does not impose any particular evaluation
order.

Line 5: \aref{frame-evaluation} is a subroutine that exercises the interpolant
constructed so far at the collocation nodes and thereby approximates the values
obtained on line~4. This algorithm will be discussed separately later on.

Line 6: The \texttt{Subtract} subroutine computes the difference between the
actual and approximated values of \g, which yields the hierarchical surpluses
$\set{\Delta(\g \circ \transform)(\vx_{\v{\iota}_k})}$ (see
\eref{interpolant-sparse-surplus}) of the current interpolation step.

Line 7: The \texttt{Score} method of \texttt{strategy} calculates the scores of
the collocation nodes based on their surpluses as described in
\sref{frame-adaptivity}.

Line 8: The \texttt{Append} method of \texttt{interpolant} refines the
interpolant by extending it with the indices and surpluses of the completed
iteration.

\inputalgorithm{frame-evaluation}
We now turn our attention to the usage stage of an interpolant. The pseudocode
is given in \aref{frame-evaluation}. This subroutine is also used in
\aref{frame-construction} on line~5. Let us make a couple of observations
regarding this subroutine.

Line 3: The inner loop corresponds to an unfolded version of
\eref{frame-interpolant}; that is, there is no separation into the individual
interpolation steps taken.

Line 4: The \texttt{basis} object represents the interpolation basis that is
being used, and its \texttt{Compute} method evaluates the basis functions
$\set{e_{\v{\iota}_k}}$ that correspond to the given indices $\set{\v{\iota}_k}$
at arbitrary points; see \sref{frame-basis}.

It is worth noting that the \texttt{strategy}, \texttt{grid}, and \texttt{basis}
objects conform to certain interfaces and can be easily swapped out. This makes
the two algorithms very general and reusable with different configurations. In
particular, the adaptation strategy can be fine-tuned for each particular
problem.

\conclusioncut
To recapitulate, in this section, the approximation engine of our framework for
probabilistic analysis of electronic systems under workload variation has been
presented. It is consolidated in \aref{frame-construction} and
\aref{frame-evaluation}.


\documentclass[main.tex]{subfiles}
\begin{document}

\marginpar{Wednesday\\ 2021-11-17}

The conjugate basis is often denoted as \(\widetilde{\omega}^{(J)} = \dd{\widetilde{x}}^{(J)}\).

The application of a one-form on a vector, \(\widetilde{q}(\vec{V}) = q_J V^{J}\),
can be computed as a \emph{contraction}.

A one-form transforms with \(\pdv*{x^{\mu }}{x^{\prime \alpha }}\),
a vector transforms with \(\pdv*{x^{\prime \mu }}{x^{\alpha }}\).

Since a basis vector transforms like
%
\begin{align}
\vec{e}_{(\alpha ')} = \Lambda^{\mu }{}_{\alpha '} \vec{e}_{(\mu) }
\,
\end{align}
%
we have
%
\begin{align}
q_{J} = \widetilde{q}(\vec{e}_{(J)}) = \widetilde{q}( \vec{e}_{(k')} \Lambda^{k'}{}_J )
= \Lambda^{k'}{}_J \widetilde{q}( \vec{e}_{(k')}) = \Lambda^{k'}_J q_{k'}
\,.
\end{align}

This shows that the one-form transforms with the inverse of the Jacobian.

An example of a one-form is the gradient of a scalar function \(\phi \):
%
\begin{align}
\pdv{\phi }{x^{(J')}} = \pdv{\phi }{x^{k}} \pdv{x^{k}}{x^{\prime J'}} = \pdv{\phi }{x^{k}} \Lambda^{k}{}_{J'}
\,.
\end{align}

\paragraph{Tensors}

A tensor of type \((N, R)\) is a function which takes \(N\) one-forms, \(R\) vectors, and yields a number.
We require that it is multilinear (linear in all its arguments),

One-forms are \((0, 1)\) tensors, vectors are \((1, 0)\) tensors.

The components of the tensor are defined by its application to the basis vectors / covectors of the vector space. For a \((0, 2)\) tensor, we have
%
\begin{align}
F_{\alpha \beta } = F( \vec{e}_{(\alpha )}, \vec{e}_{(\beta )})
\,,
\end{align}
%
from which we can recover by linearity any application of the tensor: \(F(\vec{A}, \vec{B}) = A^{\alpha } B^{\beta } F_{\alpha \beta }\).

We can then try to construct a basis for the tensor space such that
%
\begin{align}
F = F_{\alpha \beta } \widetilde{\omega}^{(\alpha , \beta )}
\,.
\end{align}

We do this by writing
%
\begin{align}
F (\vec{A}, \vec{B})
= \qty(F_{\alpha \beta } \widetilde{\omega}^{(\alpha , \beta )}) (\vec{A}, \vec{B})
= F_{\alpha \beta } \widetilde{\omega}^{\alpha }(\vec{A}) \widetilde{\omega}^{(\beta )}(\vec{B})
\,,
\end{align}
%
therefore we can write the \((0, 2)\) basis as an outer product:
%
\begin{align}
\widetilde{\omega}^{(\alpha , \beta )} = \widetilde{\omega}^{(\alpha)} \otimes \widetilde{\omega}^{(\beta )}
\,.
\end{align}

We can iterate this procedure to get a basis for any tensor space. How do tensors transform?
%
\begin{align}
F = F_{\alpha ' \beta '} \Lambda^{\alpha '}_{\gamma } \widetilde{\omega}^{(\gamma )} \otimes
\widetilde{\omega}^{(\rho )} \Lambda^{\beta '}_{\gamma }
= F_{\gamma \rho } \widetilde{\omega}^{(\gamma )} \otimes \widetilde{\omega}^{(\rho )}
\,.
\end{align}
%
\todo[inline]{this looks wrong\dots}

The transformation law is therefore
%
\begin{align}
F_{\alpha ' \beta '} \Lambda^{\alpha '}{}_\gamma \Lambda^{\beta '}{}_\delta = F_{\gamma \delta }
\,,
\end{align}
%
or equivalently
%
\begin{align}
F_{\alpha ' \beta ' } = \Lambda^{\rho }{}_{\alpha '} \Lambda^{\sigma }{}_{\beta '} F_{ \rho \sigma }
\,.
\end{align}

Tensors of the same kind can be added, and tensors of any order can be multiplied and contracted.

Tensors can be symmetric: in an algebraic sense, \(F\) is symmetric if \(F(\vec{A}, \vec{B}) = F(\vec{B}, \vec{A})\).
In terms of its indices, this means \(F_{\alpha \beta } = F_{\beta \alpha }\).

We can symmetrize a tensor by \(F_{(\alpha \beta )} = (F_{\alpha \beta } + F_{\beta \alpha }) / 2\).

Symmetric \(n\)-dimensional  tensors have \(n(n+1) / 2\) independent components.

Tensors can be antisymmetric: in an algebraic sense, \(F\) is antisymmetric if \(F(\vec{A}, \vec{B}) = - F(\vec{B}, \vec{A})\).
In terms of its indices, this means \(F_{\alpha \beta } = - F_{\beta \alpha }\).

We can antisymmetrize a tensor by \(F_{[\alpha \beta ]} = (F_{\alpha \beta } - F_{\beta \alpha }) / 2\).

Antisymmetric \(n\)-dimensional tensors have \(n(n-1) / 2\) independent components.

The \textbf{metric tensor}!  We call its application to two vectors their scalar product, \(g(\vec{A}, \vec{B}) = \vec{A} \cdot \vec{B}\).
It is symmetric, therefore it has 10 free components.

The distance between two points separated by an infinitesimal \(\dd{s}\) is
%
\begin{align}
\dd{s^2} = \dd{s} \cdot \dd{s} = g_{\mu \nu } \dd{x^{\mu }} \dd{x^{\nu }}
\,.
\end{align}

A curve \(\gamma \colon [a, b] \to \mathcal{M}\) can be measured: its length will be
%
\begin{align}
s = \int_{a}^{b} \dd{s} = \int_{a}^{b} \dd{\lambda } \underbrace{\sqrt{g_{\mu \nu } \dv{x^{\mu }}{\lambda } \dv{x^{\nu }}{\lambda }}}_{\dv*{s}{\lambda }}
\,.
\end{align}

This gives the length of the \textbf{path}, which is invariant with respect to the parametrization.

Thanks to the metric tensor we can lower or raise indices:
the map \(\omega _V \colon U \to g(U, V)\) is a one-form by the properties we require of the metric, so we can say it is the ``dual'' form of \(V\), typically just denoted as \(V_\mu = g_{\mu \nu } V^{\nu }\).

In order to do the inverse, we need the inverse of the metric tensor, \(g^{\mu \nu }\), which satisfies \(g^{\mu \nu } g_{\nu \rho }= \delta^{\mu }_{\rho }\).

\begin{extracontent}
Consider the metric tensor
%
\begin{align}
\eta_{\alpha \beta } = \left[\begin{array}{ccc}
-1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{array}\right]
\,.
\end{align}

Let us rotate the coordinate system:
%
\begin{align}
x^{0} &= x^{0 \prime}  \\
x^{1} & = r \cos \theta  \\
x^{2} & = r \sin \theta
\,,
\end{align}
%
where \(r \) and \(\theta \) are polar coordinates for the unprimed system.

How does the metric transform?
%
\begin{align}
g_{0'0'} &= \Lambda^{\mu }_{0'} \Lambda^{\nu }_{0'} \eta_{\mu \nu } = \eta_{00} = -1   \\
g_{0'i'} &= \Lambda^{\mu }_{0'} \Lambda^{\nu }_{i '} \eta_{\mu \nu } = 0 \\
g_{1'1'} &= \Lambda^{\mu }_{1'} \Lambda^{\nu }_{1'} \eta_{\mu \nu } = \cos^2 \theta + \sin^2 \theta = 1\\
g_{2'2'} &= \Lambda^{\mu }_{2'} \Lambda^{\nu }_{2'} \eta_{\mu \nu } = r^2 (\cos^2 \theta + \sin^2 \theta) = r^2
\,.
\end{align}

So, we find the line element in polar coordinates
%
\begin{align}
g_{\alpha ' \beta '} = \left[\begin{array}{ccc}
-1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & r^2
\end{array}\right]
\,.
\end{align}
\end{extracontent}

\end{document}


\documentclass[12pt]{cdblatex}
\usepackage{exercises}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% --------------------------------------------------------------------------------------------
\section*{Exercise 3.9 Ricci in terms of the metric and its derivatives}

\begin{cadabra}
   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u#}::Indices(position=independent).

   \partial{#}::PartialDerivative;

   g_{a b}::Metric;
   g^{a b}::InverseMetric;

   dgab := \partial_{c}{g^{a b}} -> - g^{a e} g^{b f} \partial_{c}{g_{e f}}.      # cdb (ex-0309.dgab,dgab)

   Gamma := \Gamma^{a}_{b c} ->
            (1/2) g^{a e} (   \partial_{b}{g_{e c}}
                            + \partial_{c}{g_{b e}}
                            - \partial_{e}{g_{b c}}).                             # cdb (ex-0309.Gamma,Gamma)

   Rabcd := R^{a}_{b c d} ->
            \partial_{c}{\Gamma^{a}_{b d}} + \Gamma^{a}_{e c} \Gamma^{e}_{b d}
          - \partial_{d}{\Gamma^{a}_{b c}} - \Gamma^{a}_{e d} \Gamma^{e}_{b c}.   # cdb (ex-0309.Rabcd,Rabcd)

   FourRab := 4 R^{c}_{a c b}.                      # cdb (ex-0309.101,FourRab)

   substitute     (FourRab, Rabcd)                  # cdb (ex-0309.102,FourRab)
   substitute     (FourRab, Gamma)                  # cdb (ex-0309.103,FourRab)

   product_rule   (FourRab)                         # cdb (ex-0309.104,FourRab)
   distribute     (FourRab)                         # cdb (ex-0309.105,FourRab)

   substitute     (FourRab, dgab)                   # cdb (ex-0309.106,FourRab)

   sort_product   (FourRab)                         # cdb (ex-0309.107,FourRab)
   rename_dummies (FourRab)                         # cdb (ex-0309.108,FourRab)
   canonicalise   (FourRab)                         # cdb (ex-0309.109,FourRab)

   # sort so that g to appeares before dg

   substitute     (FourRab, $g^{a b} -> A^{a b}$)
   sort_product   (FourRab)
   rename_dummies (FourRab)
   substitute     (FourRab, $A^{a b} -> g^{a b}$)   # cdb (ex-0309.110,FourRab)
\end{cadabra}

\clearpage

\begin{dgroup*}
   \Dmath*{4 R_{ab} = \Cdb*{ex-0309.101}
                    = \Cdb*{ex-0309.102}
                    = \Cdb*{ex-0309.103}
                    = \Cdb*{ex-0309.104}
                    = \Cdb*{ex-0309.105}
                    = \Cdb*[\hskip 2cm\hfill]{ex-0309.106}
                    = \Cdb*[\hskip 2cm\hfill]{ex-0309.107}
                    = \Cdb*[\hskip 2cm\hfill]{ex-0309.108}
                    = \Cdb*{ex-0309.109}
                    = \Cdb*{ex-0309.110}}
\end{dgroup*}

% ============================================================================================
% export to json format

\CdbSetup{action=hide}

\begin{cadabra}
   import cdblib
   cdblib.create ('ex-0309.json')
   cdblib.put ('FourRab',FourRab,'ex-0309.json')
\end{cadabra}

\end{document}


\section{Case detection and isolation} \label{cdr}

\subsection{Determining the proportion of cases detected}
We calculate a time-varying case detection rate, being the proportion of all symptomatic cases (the second and third clinical strata, described in Section \ref{clin}) that are detected (the third clinical stratum only).
This proportion is informed by the number of tests performed using the following formula:

\[CDR(time)=1-(1-floor)\times e^{-shape \times tests(time)}\]

$time$ is the calendar date and $tests(time)$ is the number of tests per capita done on that date. To determine the value of the shape parameter, we solve this equation based on the assumption that a certain daily testing rate $tests(t)$ is associated with a certain $CDR(t)$.
$floor$ is the minimum case detection rate possible, which would theoretically occur when zero tests are conducted.
Solving for $shape$ yields:

\[shape = \frac{-log(\frac{1 - CDR(t)}{1 - floor})}{tests(t)}\]

That is, if it is assumed that a certain daily per capita testing rate is associated with a certain proportion of symptomatic cases detected, we can determine $shape$.
As this relationship is not well understood and unlikely to be consistent across all settings, we vary the $CDR$ that is associated with a certain per capita testing rate during calibration.
This approach allows us to both vary the $CDR(\cdot)$ relationship through calibration,
while also varying the specific $CDR(time)$ to reflect historical changes in testing capacity with time.

\subsection{Isolation of detected cases}
As described in the clinical stratification section above, as infected persons progress from the early to the late stage of active COVID-19,
infectiousness is reduced for those detected to reflect case isolation.


\documentclass[../main.tex]{subfiles}
\begin{document}
\chapter{Abstract Integration}

\begin{exercise}
  Does there exist an infinte $ \sigma $-algebra which has only countably many members?
  \paragraph{Solution. }
  If $ \mathcal{F} $ is a $ \sigma $-algebra and $ A $ a set, define $ \mathcal{F} \cap A := \{A \cap F: F \in \mathcal{F}\}$. Hence if $ A \in \mathcal{F} $, then $ \mathcal{F} \cap A $ is a $ \sigma $-subalgebra of $ \mathcal{F} $. Further, if $ \mathcal{F} $ is infinte, either $ A \cap \mathcal{F} $ or $ A^c \cap \mathcal{F}$  is infinte.

  In other words, if $ \mathcal{F}_n $ is infinte, then there is $ A _{n+1} \in \mathcal{F}_n $ such that $ \mathcal{F} _{n+1} = \mathcal{F}_n \cap A_n^c $ is infinte. Take $ \mathcal{F}_0 = \mathcal{M} $, by induction we get a disjoint sequence of sets $ A_n \in \mathcal{M}$.
  Since $ \mathcal{M} $ must also contain any union of sets of $ A_n $, and each union is different, this is to say $ \mathcal{M} $ contains a embedding of all subsets of $ \mathbb{N} $, therefore uncountable.
\end{exercise}

\begin{exercise}
  Prove an analogue of Theorem 1.8 for $ n $ functions.
  \paragraph{Solution. }
  It suffices to prove
  \begin{align*}
    f = f_1 \times f_2 \times ... \times f_n
  \end{align*}
  is measurable is each $ f_i $ is measurable. Similarly as in the theorem, take $ R $ be any rectangles in $ \mathbb{R}^n $. Notice
  \begin{align*}
    f ^{-1}(R) = \bigcap f_i(I_i)
  \end{align*}
  is measurable, if $ R = I_1 \times I_2 \times ... \times I_n $. The rest of the proof is a repeat of Theorem 1.8.
\end{exercise}

\begin{exercise}
  Prove that if $ f$ is a real function on a measurable space $ X $ such that $ \{x: f(x) < r\} $ is measurable for every rational $ r $, then $ f $ is measurable.
  \paragraph{Solution. }
  Take $ \Omega $ as all $ E \subset \mathbb{R} $ such that $ f ^{-1}(E) $ is measurable. By Theorem 1.2, $ \Omega $ is a $ \sigma $-algebra. Notice
  \begin{align*}
    \{f > a\} = \bigcup _{q \in \mathbb{Q} \cap (-\infty, a)} \{f > q\}
  \end{align*}
  for all $ a \in \mathbb{R} $. Therefore, $ \{f > a\} $ is measurable. By Theorem 1.2 again, $ f $ is measurable.

\end{exercise}

\begin{exercise}
  Let $ \{a_n\} $ and $ \{b_n\} $ be sequences in $ [-\infty, \infty] $, and prove the following assertions:
  \begin{enumerate}
    \item $$ \limsup -a_n = -\liminf a_n. $$
    \item $$ \limsup (a_n + b_n) \le \limsup a_n + \limsup b_n $$ provided none of the sums is of the form $ \infty - \infty $.
    \item If $ a_n \le b_n $ for all $ n $, then
    \begin{align*}
      \liminf a_n \le \liminf b_n
    \end{align*}

    Show by example that strict inequality can hold in b).

  \end{enumerate}
  \paragraph{Solution. }
  \begin{enumerate}
    \item First prove $ \sup -A = - \inf A $, where $ A \in \mathbb{\bar R} $. Notice $ -a \le -\inf a $ implies $ \sup -A \le -\inf A $. Similarly $ -\inf A \le \sup -A $.
    \begin{align*}
      \limsup (-a_n) = \inf _{n \ge 1} \sup _{k \ge n} (-a_k) = - \sup \inf a_k = -\limsup a_n
    \end{align*}

    \item Notice if $ \limsup a_n + \limsup b_n $ is defined, $ \sup _{k \ge n} a_k + \sup _{k \ge n} b_k $ is defined for sufficiently large $ n $. Notice
    \begin{align*}
      a_n + b_n \le \sup _{k \ge n} a_k + \sup _{k \ge n} b_k
    \end{align*}
    implies
    \begin{align*}
      \sup _{k \ge n} a_n + b_n \le \sup _{k \ge n} a_k + \sup _{k \ge n} b_k
    \end{align*}
    for sufficiently large $ n $. Taking the limits, whose existence indicated by the question, on both sides gives the result (any decreasing sequence has its limit the same as its infimum).

    \item Clearly $ \inf _{k \ge n} a_k \le \inf _{k \ge n} b_k $ for any $ n $. This completes the proof.
  \end{enumerate}

  Define
  \begin{align*}
    a_n &= 1, -1, 1, -1, 1, ...\\
    b_n &= -1, 1, -1, 1, -1, ...
  \end{align*}
  Obviously, $ \limsup a_n = \limsup b_n = 1 $, while $ \limsup a_n + b_n = 0 $.
\end{exercise}

\begin{exercise}
  \begin{enumerate}
    \item Suppose $ f: X \to [-\infty, \infty] $ and $ g: X \to [-\infty, \infty] $ are measurable. Prove that the sets
    \begin{align*}
      \{x: f(x) < g(x) \}, \{x: f(x) = g(x) \}
    \end{align*}
    are measurable.

    \item Prove that the set of points at which a sequence of measurable real-valued functions converges (to a finite limit) is measurable.

  \end{enumerate}

  \paragraph{Solution. }
  \begin{enumerate}
    \item Notice
    \begin{align*}
      \{f < g\} = \bigcup _{r \in \mathbb Q} \{f < r\} \cap \{r < g\}
    \end{align*}
    therefore measurable. By symmetry, $ \{f > g\} $ is measurable as well. Now $ \{f = g\} $ is the complement of $ \{f > g\} \cup \{f < g\} $ thus must be measurable.

    \item For any sequence of functions $ f_n $, the set of points the sequence converges to a finite limit is
    \begin{align*}
      \{\limsup f_n = \liminf f_n \} \cap \{\limsup f_n < \infty \}.
    \end{align*}
    Since the upper and lower limits of a sequence of functions are measurable functions, the above set is measurable.
  \end{enumerate}

\end{exercise}

\begin{exercise}
  Let $ X $ be an uncountable set, let $ \mathcal{M} $ be the collection of all sets $ E \subset X $ such that either $ E $ or $ E^c $ is at most countable, and define $ \mu(E) = 0 $ in the first case, $ mu(E) = 1 $ in the second. Prove that $ \mathcal{M} $ is a $ \sigma $-algebra in $ X $ and that $ \mu $ is a measure on $ \mathcal{M} $. Describe the corresponding measurable functions and their integrals.

  \paragraph{Solution. }
  It suffices to prove $ \mathcal{M} $ is closed under countable union and $ \mu $ is a measure. Notice for any sequence of sets $ E_n $ in $ \mathcal{M} $, if all of them are at most countable, so must be their union. Therefore $ \bigcup E_n \in \mathcal{M} $, and $ \mu(\bigcup E_n) = 0 = \sum \mu(E_n) $. If $ E_i $ have its complement at most countable, $ \left (\bigcup E_n \right )^c = \bigcap E_n^c \subset E_i^c$ is at most countable. Hence $ \bigcup E_n \in \mathcal{M} $.
  Further, if $ E_n $ are disjoint, for any set $ E_j, j \ne i $, $ X \subset E_i^c \cup E_j^c $, showing $ E_j^c $ must be uncountable, therefore $ E_j $ must be at most countable and $ \sum \mu(E_n) = 1 = \mu \left (\bigcup E_n \right ) $.

  Only describe integrals of non-negative functions, the general case follows is of no interest. Since any singalton is Borel, for any $ r \in [0, \infty] $, a measurable function $ f: X \to [0, \infty] $ would have $ f ^{-1}(r) $ either at most countable or have a at most countable complement. Take $ A $ as the set of $ r $ such that $ f ^{-1}(r)  $ has an at most countable complement. Then if $ A $ is countable, it is easy to check $ \int f = \sum _{r \in A} r  $. If $ A $ is uncountable, then $ \int f = \infty $. 21

\end{exercise}

\end{document}


% !TeX TXS-program:compile = txs:///pdflatex/
% !TeX TXS-program:bibliography = txs:///biber
% !TeX program = pdflatex
% !BIB program = biber

\subsection{Structural Estimation}
\label{sec:structural}

\blindmathfalse
\Blindtext[1]

\[
	\mathbfup{\Sigma} \coloneqq
	\mathup{Cov}(\mathbf{X}) =
	\begin{bmatrix}
		\mathup{Var}(X_1)      & \cdots & \mathup{Cov}(X_1, X_n) \\[-2.5pt]
		\vdots                 & \ddots & \vdots                 \\
		\mathup{Cov}(X_n, X_1) & \cdots & \mathup{Var}(X_n)
	\end{bmatrix}
\]


% This LaTeX was auto-generated from an M-file by MATLAB.
% To make changes, update the M-file and republish this document.

\subsection*{augment\_unicycle.m}

\begin{par}
\textbf{Summary:} The function computes the $(x,y)$ velocities of the contact point in both absolute and unicycle coordinates as well as the the unicycle coordinates of the contact point themselves.
\end{par} \vspace{1em}

\begin{verbatim}     function r = augment(s)\end{verbatim}
    \begin{par}
\textbf{Input arguments:}
\end{par} \vspace{1em}

\begin{lstlisting}
%		s     state of the unicycle (including the torques).             [1 x 18]
%         The state is assumed to be given as follows:
%         dx      empty (to be filled by this function)
%         dy      empty (to be filled by this function)
%         dxc     empty (to be filled by this function)
%         dyc     empty (to be filled by this function)
%         dtheta  roll angular velocity
%         dphi    yaw angular velocity
%         dpsiw   wheel angular velocity
%         dpsif   pitch angular velocity
%         dpsit   turn table angular velocity
%         x       x position
%         y       y position
%         xc      empty (to be filled by this function)
%         yc      empty (to be filled by this function)
%         theta   roll angle
%         phi     yaw angle
%         psiw    wheel angle
%         psif    pitch angle
%         psit    turn table angle
%
% *Output arguments:*
%
%   r     additional variables that are computed based on s:          [1 x 6]
%         dx    x velocity of contact point (global coordinates)
%         dy    y velocity of contact point (global coordinates)
%         dxc   x velocity of contact point (unicycle coordinates)
%         dyc   y velocity of contact point (unicycle coordinates)
%         xc    x position of contact point (unicycle coordinates)
%         yc    y position of contact point (unicycle coordinates)
%
%
% Copyright (C) 2008-2013 by
% Marc Deisenroth, Andrew McHutchon, Joe Hall, and Carl Edward Rasmussen.
%
% Last modified: 2013-03-27

function r = augment_unicycle(s)
\end{lstlisting}

\subsection*{Code}

\begin{lstlisting}
rw = 0.225;                               % wheel radius in meters

% x velocity of contact point (global coordinates)
r(1) = rw*cos(s(15))*s(7);
% y velocity of contact point (global coordinates)
r(2) = rw*sin(s(15))*s(7);
% (x,y) velocities of contact point (unicycle coordinates)
A = -[cos(s(15)) sin(s(15)); -sin(s(15)) cos(s(15))];
dA = -s(6)*[-sin(s(15)) cos(s(15)); -cos(s(15)) -sin(s(15))];
r(3:4) = A*r(1:2)' + dA*s(10:11)';
% (x,y) coordinates of contact point (unicycle coordinates)
r(5:6) = A*s(10:11)';
\end{lstlisting}


\title{Supervised Learning (Classification)}

\subsection{Supervised Learning (Classification)}

In supervised learning, the task is to infer hidden structure from
labeled data, comprised of training examples $\{(x_n, y_n)\}$.
Classification means the output $y$ takes discrete values.

We demonstrate with an example in Edward.
An interactive version with Jupyter notebook is available
\href{http://nbviewer.jupyter.org/github/blei-lab/edward/blob/master/notebooks/supervised_classification.ipynb}{here}.

\subsubsection{Data}

We use the
\href{https://stat.ethz.ch/R-manual/R-devel/library/MASS/html/crabs.html}
{crabs data set},
which consists of morphological measurements on a crab species. We
are interested in predicting whether a given crab has the color form
blue (encoded as 0) or orange (encoded as 1). We use all the numeric features
in the dataset.
\begin{lstlisting}[language=Python]
from observations import crabs

data, metadata = crabs("~/data")
X_train = data[:100, 3:]
y_train = data[:100, 1]

N = X_train.shape[0]  # number of data points
D = X_train.shape[1]  # number of features

print("Number of data points: {}".format(N))
print("Number of features: {}".format(D))
\end{lstlisting}

\begin{lstlisting}
Number of data points: 100
Number of features: 5
\end{lstlisting}

\subsubsection{Model}

A Gaussian process is a powerful object for modeling nonlinear
relationships between pairs of random variables. It defines a distribution over
(possibly nonlinear) functions, which can be applied for representing
our uncertainty around the true functional relationship.
Here we define a Gaussian process model for classification
\citep{rasmussen2006gaussian}.

Formally, a distribution over functions $f:\mathbb{R}^D\to\mathbb{R}$ can be specified
by a Gaussian process
\begin{align*}
  p(f)
  &=
  \mathcal{GP}(f\mid \mathbf{0}, k(\mathbf{x}, \mathbf{x}^\prime)),
\end{align*}
whose mean function is the zero function, and whose covariance
function is some kernel which describes dependence between
any set of inputs to the function.

Given a set of input-output pairs
$\{\mathbf{x}_n\in\mathbb{R}^D,y_n\in\mathbb{R}\}$,
the likelihood can be written as a multivariate normal
\begin{align*}
  p(\mathbf{y})
  &=
  \text{Normal}(\mathbf{y} \mid \mathbf{0}, \mathbf{K})
\end{align*}
where $\mathbf{K}$ is a covariance matrix given by evaluating
$k(\mathbf{x}_n, \mathbf{x}_m)$ for each pair of inputs in the data
set.

The above applies directly for regression where $\mathbb{y}$ is a
real-valued response, but not for (binary) classification, where $\mathbb{y}$
is a label in $\{0,1\}$. To deal with classification, we interpret the
response as latent variables which is squashed into $[0,1]$. We then
draw from a Bernoulli to determine the label, with probability given
by the squashed value.

Define the likelihood of an observation $(\mathbf{x}_n, y_n)$ as
\begin{align*}
  p(y_n \mid \mathbf{z}, x_n)
  &=
  \text{Bernoulli}(y_n \mid \text{logit}^{-1}(z_n)).
\end{align*}

Define the prior to be a multivariate normal
\begin{align*}
  p(\mathbf{z})
  &=
  \text{Normal}(\mathbf{z} \mid \mathbf{0}, \mathbf{K}),
\end{align*}
with covariance matrix given as previously stated.

Let's build the model in Edward. We use a radial basis function (RBF)
kernel, also known as the squared exponential or exponentiated
quadratic. It returns the kernel matrix evaluated over all pairs of
data points; we then Cholesky decompose the matrix to parameterize the
multivariate normal distribution.
\begin{lstlisting}[language=Python]
from edward.models import Bernoulli, MultivariateNormalTriL
from edward.util import rbf

X = tf.placeholder(tf.float32, [N, D])
f = MultivariateNormalTriL(loc=tf.zeros(N), scale_tril=tf.cholesky(rbf(X)))
y = Bernoulli(logits=f)
\end{lstlisting}
Here, we define a placeholder \texttt{X}. During inference, we pass in
the value for this placeholder according to data.

\subsubsection{Inference}

Perform variational inference.
Define the variational model to be a fully factorized normal.
\begin{lstlisting}[language=Python]
qf = Normal(loc=tf.Variable(tf.random_normal([N])),
            scale=tf.nn.softplus(tf.Variable(tf.random_normal([N]))))
\end{lstlisting}

Run variational inference for \texttt{500} iterations.
\begin{lstlisting}[language=Python]
inference = ed.KLqp({f: qf}, data={X: X_train, y: y_train})
inference.run(n_iter=500)
\end{lstlisting}
In this case
\texttt{KLqp} defaults to minimizing the
$\text{KL}(q\|p)$ divergence measure using the reparameterization
gradient.
For more details on inference, see the \href{/tutorials/klqp}{$\text{KL}(q\|p)$ tutorial}.
(This example happens to be slow because evaluating and inverting full
covariances in Gaussian processes happens to be slow.)

% \subsubsection{Criticism}

\subsubsection{References}\label{references}


\chapter{Latent variable models for discrete data}

\section{Introduction}
In this chapter, we are concerned with latent variable models for discrete data, such as bit vectors, sequences of categorical variables, count vectors, graph structures, relational data, etc. These models can be used to analyse voting records, text and document collections, low-intensity images, movie ratings, etc. However, we will mostly focus on text analysis, and this will be reflected in our terminology.

Since we will be dealing with so many different kinds of data, we need some precise notation to keep things clear. When modeling variable-length sequences of categorical variables (i.e., symbols or \textbf{tokens}), such as words in a document, we will let $y_{il} \in \{1,\cdots,V\}$ represent the identity of the $l$'th word in document $i$,where $V$ is the number of possible words in the vocabulary. We assume $l=1:L_i$, where $L_i$ is the (known) length of document $i$, and $i=1:N$, where $N$ is the number of documents.

We will often ignore the word order, resulting in a \textbf{bag of words}. This can be reduced to a fixed length vector of counts (a histogram). We will use $n_{iv} \in \{0,1,\cdots,Li\}$ to denote the number of times word $v$ occurs in document $i$, for $v=1:V$. Note that the $N \times V$ count matrix $\vec{N}$ is often large but sparse, since we typically have many documents, but most words do not occur in any given document.

In some cases, we might have multiple different bags of words, e.g., bags of text words and bags of visual words. These correspond to different “channels” or types of features. We will denote these by $y_{irl}$, for $r=1:R$(the number of responses) and $l=1:L_{ir}$. If $L_{ir} =1$,it means we have a single token (a bag of length 1); in this case, we just write $y_{ir} \in \{1,\cdots,V_r\}$ for brevity. If every channel is just a single token, we write the fixed-size response vector as $y_{i,1:R}$; in this case, the $N \times R$ design matrix \vec{Y} will not be sparse. For example, in social science surveys, $y_{ir}$ could be the response of personito the $r$'th multi-choice question.

Out goal is to build joint probability models of $p(\vec{y}_i)$ or $p(\vec{n}_i)$ using latent variables to capture the correlations. We will then try to interpret the latent variables, which provide a compressed representation of the data. We provide an overview of some approaches in Section 27.2 TODO, before going into more detail in later sections.

\section{Distributed state LVMs for discrete data}


\section{Decision Trees}

%%%%%%%%%%%%%%%%%%%%
\begin{frame}{$K$-sorted (Problem 2.9)}
  \[
	1,\;2,\;4,\;3;\quad 7,\;6,\;8,\;5;\quad 10,\;11,\;9,\;12;\quad 15,\;13,\;16,\;14
  \]

  \pause
  \vspace{0.20cm}

  \begin{center}
	$1\text{-sorted}$? \\ \pause
	$2\text{-sorted}$? \\ \pause
	$n\text{-sorted}$?
  \end{center}

  \pause
  \[
	1\text{-sorted} \to 2\text{-sorted} \to 4\text{-sorted} \to \cdots \to n\text{-sorted}
  \]

  \pause
  \centerline{Quicksort (with median) stops after the $\log k$ recursions.}

  \pause
  \[
	O(n \log k)
  \]
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{$K$-sorted (Problem 2.9)}
  \[
    \Omega(n \log k)
  \]

  \pause
  \[
	L = \binom{n}{n/k, \ldots, n/k} \pause = \frac{n!}{\left( (\frac{n}{k})! \right)^{k}}
  \]

  \pause
  \[
	H \ge \log \left(\frac{n!}{\left( (\frac{n}{k})! \right)^{k}} \right)
  \]
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{$K$-sorted (Problem 2.9)}
  \centerline{Sorting the $k$-sorted array.}

  \pause
  \[
	O(n \log \frac{n}{k})
  \]

  \pause
  \[
	L = ((\frac{n}{k})!)^k
  \]

  \pause
  \[
	H \ge \log ((\frac{n}{k})!)^k = \Omega(n \log \frac{n}{k})
  \]
\end{frame}
%%%%%%%%%%%%%%%%%%%%


\documentclass[a4paper]{article}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage[fleqn]{amsmath}
\usepackage{hyperref}

\begin{document}

\section{Math stuff}

Cubic interpolation for one segment $[x_k, x_{k+1}]$ can be described as:

\begin{equation*}
	\begin{aligned}
		f(t) &= c_{oef1}t^3 + c_{oef2}t^2 + c_{oef3}t + c_{oef4}  \hspace{1cm} with \\
		t(x) &= \frac{x - x_k}{x_{k+1} - x_k}
	\end{aligned}
\end{equation*}

\hspace{1cm} and

\begin{equation*}
	\begin{aligned}
		c_{oef1} &= 2p_0 - 2p_1 - m_0 - m_1 \\
		c_{oef2} &= -3p_0 + 3p_1 - 2m_0 - m_1 \\
		c_{oef3} &= m_0 \\
		c_{oef4} &= p_0
	\end{aligned}
\end{equation*}

(see Wikipedia-Links below)\\
\\
If we rewrite this as function of $d = x - x_k$ we get

\begin{equation*}
	\begin{aligned}
		f'(d) &= c_{oef1}' d^3 + c_{oef2}' d^2 + c_{oef3}' d + c_{oef4}' \hspace{1cm} with \\
		c_{oef1}' &= \frac{c_{oef1}}{(x_{k+1} - x_k)^3} \\
        c_{oef2}' &= \frac{c_{oef2}}{(x_{k+1} - x_k)^2} \\
        c_{oef3}' &= \frac{c_{oef3}}{x_{k+1} - x_k} \\
        c_{oef4}' &= c_{oef4}
    \end{aligned}
\end{equation*}
\\
The implemented algorithm uses two helper variables to calculate the coefficients of $f'$ efficiently:

\begin{equation*}
	\begin{aligned}
		common = m_k + m_{k+1} - 2 \frac{p_{k+1} - p_k}{x_{k+1} - x_k} \\
		invLength = \frac{1}{x_{k+1} - x_k}
    \end{aligned}
\end{equation*}
\\
We use $p_0 = p_k$, $p_1 = p_{k+1}$, $m_0 = m_k (x_{k+1} - x_k)$, $m_1 = m_{k+1} (x_{k+1} - x_k)$ and $s = \frac{p_{k+1}-p_k}{x_{k+1}-x_k}$. The tangents are scaled with the length of the segment. \\
\\
If we insert this into the equations for the coefficients we get the formulas that are used in the algorithm:

\begin{equation*}
\begin{aligned}
	c_{oef1}' &= \frac{c_{oef1}}{(x_{k+1} - x_k)^3} \\
	&= \frac{2p_0 - 2p_1 + m_0 + m_1}{(x_{k+1} - x_k)^3}\\
	&= (2p_k - 2p_{k+1} + m_k (x_{k+1} - x_k) + m_{k+1} (x_{k+1} - x_k) / (x_{k+1} - x_k)^3 \\
	&= \frac {(2p_k - 2p_{k+1} + m_k (x_{k+1} - x_k) + m_{k+1} (x_{k+1} - x_k)}{x_{k+1} - x_k} / (x_{k+1} - x_k)^2 \\
	&= (\frac {2p_k - 2p_{k+1}}{x_{k+1} - x_k} + m_k + m_{k+1}  ) * invLength^2 \\
	&= (-2\frac {p_{k+1}- p_k}{x_{k+1} - x_k} + m_k + m_{k+1}  ) * invLength^2 \\
	&= common * invLenght^2
\end{aligned}
\end{equation*}

\begin{equation*}
\begin{aligned}
	c_{oef2}' &= \frac{c_{oef2}}{(x_{k+1} - x_k)^2} \\
	&= (-3p_0 + 3p_1 - 2m_0 - m_1) / (x_{k+1} - x_k)^2 \\
	&= (-3p_k + 3p_{k+1} - 2* m_k (x_{k+1} - x_k) - m_{k+1} (x_{k+1} - x_k)) / (x_{k+1} - x_k)^2 \\
	&= (\frac{-3p_k + 3p_{k+1}}{x_{k+1} - x_k} - 2m_k - m_{k+1}) * invLenght \\
	&= (3\frac{p_{k+1} - p_k}{x_{k+1} - x_k} - 2m_k - m_{k+1}) * invLenght \\
	&=(\frac{p_{k+1} - p_k}{x_{k+1} - x_k} + 2\frac{p_{k+1} - p_k}{x_{k+1} - x_k} - m_k - m_{k+1} - m_k) * invLenght \\
	&= (s - common - m_k) * invLenght
\end{aligned}
\end{equation*}

\begin{equation*}
\begin{aligned}
	c_{oef3}' &= \frac{c_{oef3}}{x_{k+1} - x_k} \\
	&= \frac{m_0}{x_{k+1} - x_k} \\
	&= \frac{m_k (x_{k+1} - x_k)}{x_{k+1} - x_k} \\
 	&= m_k
\end{aligned}
\end{equation*}

\begin{equation*}
\begin{aligned}
	c_{oef4}' &= c_{oef4} = p_0 = p_k
\end{aligned}
\end{equation*}

\section{Useful Links}

\url{http://de.wikipedia.org/w/index.php?title=Kubisch_Hermitescher_Spline&oldid=130168003)}\\
\url{http://en.wikipedia.org/w/index.php?title=Monotone_cubic_interpolation&oldid=622341725}\\
\url{http://math.stackexchange.com/questions/45218/implementation-of-monotone-cubic-interpolation}\\
\url{http://math.stackexchange.com/questions/4082/equation-of-a-curve-given-3-points-and-additional-constant-requirements#4104}

\end{document}


\documentclass[letterpaper,12pt,leqno]{article}
\usepackage{paper,math,notes}
\newcommand{\pdf}{phasediagrams.pdf}
\available{https://www.pascalmichaillat.org/t3.html}
\hypersetup{pdftitle={Differential Equations}}

\begin{document}

\title{Differential Equations}
\author{Pascal Michaillat}
\date{}

\begin{titlepage}
\maketitle
\tableofcontents
\end{titlepage}

\section{First-Order Differential Equations}\label{sec:one}

\subsection{Constant Growth Rate}

Let $x(t)$ be a function of time $t\in\R$. Let \[\dot{x}(t)\equiv dx/dt\] denote the derivative of $x(t)$ with respect to time.

Consider the equation
\begin{equation}
\dot{x}(t) -\l\cdot x(t) = 0
\label{eq:FODE1}\end{equation}
where $\l\in\R$ is a constant. Equation~\eqref{eq:FODE1} is a first-order differential equation (FODE), because it involves $x(t)$ and the first-order derivative of $x(t)$ with respect to time: $\dot{x}(t)$.  Equation~\eqref{eq:FODE1} is a functional equation: the unknown is the function $x(t)$ rather than a number or a vector. Solving equation~\eqref{eq:FODE1} means finding the functions $x(t)$ that, together with their derivative $\dot{x}(t)$, satisfy equation~\eqref{eq:FODE1} for all $t \in \R$.

Equation~\eqref{eq:FODE1} is an especially simple differential equation. It can be rewritten as
\[\frac{\dot{x}(t)}{x(t)}= \l,\]
so it imposes that $x(t)$ has a constant growth rate $\l$ over time. It admits a simple class of functions as solution:
\begin{equation}
x(t) =A\cdot  e^{\l \cdot t} \label{eq:FODE1sol},
\end{equation}
for any constant $A\in \R$. Furthermore, the constant $A$ can be determined by an additional boundary condition because
\[A=x(0)=x(t_{0})\cdot e^{-\l \cdot t_{0}}\]
for any date $t_{0}\in \R$.

It is clear that functions of the type~\eqref{eq:FODE1sol} satisfy equation~\eqref{eq:FODE1}. We now show that if a function $x(t)$ solves equation~\eqref{eq:FODE1}, it is necessarily of the type~\eqref{eq:FODE1sol}.
Observe that
\begin{equation*}
\frac{\dot{x}(t)}{x(t)}=\od{\ln{x(t)}}{t},
\end{equation*}
which allows us to rewrite differential equation~\eqref{eq:FODE1} as
\begin{equation*}
\frac{d\ln x(t)}{dt}=\l
\end{equation*}
Let $t_{0}\in\R$. Integrating the equation from $t_{0}$ to $t$, $x(t)$ necessarily satisfies
\begin{align*}
\int_{t_{0}}^{t} d\ln x(t)&=\int_{t_{0}}^{t} \l\cdot dt\\
\ln{x(t)} -\ln{ x\bp{t_{0}}}& =\l \bp{t-t_{0}}\\
x(t)&=x(t_{0})\cdot  e^{\l \cdot(t-t_{0})}=\bs{x(t_{0})\cdot e^{-\l \cdot t_{0}}}\cdot e^{\l \cdot t}.
\end{align*}
Therefore, if $x(t)$ solves equation~\eqref{eq:FODE1}, it is necessarily of the type~\eqref{eq:FODE1sol}.

\subsection{Constant Coefficient}

We have solved the simplest FODE, which takes the form of equation~\eqref{eq:FODE1}. We now study a more general FODE:
\begin{equation}
\dot{x}(t)-\l \cdot x(t) =f(t),\label{eq:FODE2}
\end{equation}
where $f(t)\in \R$. Equation~\eqref{eq:FODE1} is the special case of equation~\eqref{eq:FODE2} with $f(t)=0$ for all $t$.

Equation~\eqref{eq:FODE2} admits a simple class of functions as solution:
\begin{equation}
x(t)=e^{\l \cdot t} \cdot \bs{A+\int_{0}^{t}f(z) \cdot e^{-\l \cdot z} dz} \label{eq:FODE2sol}
\end{equation}
for any constant $A\in \R$. Furthermore, the constant $A$ can be determined by an additional boundary condition because
\[A=x(0)=x(t_{0})\cdot e^{-\l \cdot t_{0}}-\int_{0}^{t_{0}}f(z) \cdot e^{-\l \cdot z} dz\]
for any date $t_{0}\in \R$.

It is clear that functions of the type~\eqref{eq:FODE2sol} satisfy equation~\eqref{eq:FODE2}. We now show that if a function $x(t)$ solves equation~\eqref{eq:FODE2}, it is necessarily of the type~\eqref{eq:FODE2sol}.

To be able to solve the FODE, we manipulate a certain function $x(t)\cdot \mu(t)$ instead of manipulating $x(t)$ directly. The auxiliary function $\mu(t) $ is called the \textit{integrating factor}. The integrating factor for this problem is
\[\mu(t) =e^{-\l \cdot t}.\]
This integrating factor $\mu(t)$ has the desirable property that $\dot{\mu}(t) =-\l \cdot \mu(t)$.

We multiply both sides of the differential equation~\eqref{eq:FODE2} by the integrating factor to obtain
\begin{align*}
\dot{x}(t) \cdot \mu(t) -\l \cdot x(t)\cdot  \mu(t) & =f(t) \cdot \mu(t)\\
\dot{x}(t) \cdot\mu(t) +x(t) \cdot\dot{\mu}(t) &=f(t)\cdot \mu(t)\\
\od{\bs{x(t) \cdot \mu(t)}}{t}& =f(t) \cdot\mu(t).
\end{align*}
Integrating the equation from $t_{0}\in \R$ to $t$ we obtain
\begin{align}
\int_{t_{0}}^{t} d\bs{x(t) \cdot \mu(t)}&=\int_{t_{0}}^{t}f(z) \cdot \mu(z) dz\nonumber\\
x(t)\cdot \mu(t)-x(t_{0})\cdot \mu\bp{t_{0}}&=\int_{t_{0}}^{t}f(z)\cdot  \mu(z) dz\nonumber\\
x(t)& =\frac{x\bp{t_{0}} \cdot \mu\bp{t_{0}}+\int_{t_{0}}^{t}f(z) \cdot \mu(z) dz}{\mu(t)}.\label{eq:INTER}
\end{align}
Given the definition of the integrating factor $\mu(t)$,
\begin{align*}
x(t)& =e^{\l \cdot t} \cdot \bs{x\bp{t_{0}} \cdot e^{-\l\cdot t_{0}}+\int_{t_{0}}^{t}f(z) \cdot e^{-\l \cdot z} dz}.
\end{align*}
Therefore, there exists $A\in \R$ such that
\begin{align*}
x(t)& =e^{\l \cdot t} \cdot \bs{A+\int_{0}^{t}f(z) \cdot e^{-\l \cdot z} dz} .
\end{align*}

\subsection{General Case}

We now generalize~\eqref{eq:FODE2} to allow the coefficient $\l$ to vary with time $t$. We solve
\begin{equation}
\dot{x}(t) -\l(t) \cdot x(t) =f(t),\label{eq:FODE3}
\end{equation}
with $\l(t)\in \R$ and  $f(t)\in \R$.

Equation~\eqref{eq:FODE3} admits the following class of functions as solution:
\begin{equation}
x(t)=\exp{\int_{0}^{t}\l(s) ds} \cdot \bs{A+\int_{0}^{t}f(z) \cdot \exp{-\int_{0}^{z}\l(s)ds} dz} \label{eq:FODE3sol}
\end{equation}
for any constant $A\in \R$. Furthermore, the constant $A$ can be determined by an additional boundary condition because
\[A=x(0)=x(t_{0})\cdot \exp{-\int_{0}^{t_{0}}\l(s)ds}-\int_{0}^{t_{0}}f(z) \cdot \exp{-\int_{0}^{z}\l(s)ds} dz\]
for any date $t_{0}\in \R$.

Some algebra shows that functions of the type~\eqref{eq:FODE3sol} satisfy equation~\eqref{eq:FODE3}. We now show that if a function $x(t)$ solves equation~\eqref{eq:FODE3}, it is necessarily of the type~\eqref{eq:FODE3sol}. As above, we introduce an integrating factor. The integrating factor for this problem is
\begin{equation*}
\mu(t) =\exp \bp{-\int_{0}^{t}\l(s) ds}.
\end{equation*}
This integrating factor $\mu(t)$ has the desirable property that
\[\dot{\mu}(t) =-\l(t) \cdot \mu(t).\]

We multiply both sides of equation~\eqref{eq:FODE3} by the integrating factor to obtain
\begin{align*}
\dot{x}(t) \cdot \mu(t) -\l(t) \cdot  \mu(t) \cdot x(t)& =f(t) \cdot \mu(t)\\
\dot{x}(t) \cdot\mu(t) +x(t) \cdot \dot{\mu}(t) &=f(t)\cdot \mu(t)\\
\od{\bs{x(t) \cdot \mu(t)}}{t}& =f(t) \cdot\mu(t).
\end{align*}
Integrating the equation from $t_{0}\in \R$ to $t$ we obtain as earlier equation~\eqref{eq:INTER}. Therefore the solution to equation~\eqref{eq:FODE3} is necessarily of the type~\eqref{eq:FODE3sol}.

\subsection{Initial-Value Problem}

Often, an initial condition for $x(t) $ is given:
\begin{equation}
x\bp{t_{0}} =x_{0}.  \label{eq:ic}
\end{equation}
Equation~\eqref{eq:FODE3} together with equation~\eqref{eq:ic} form an initial-value problem. The constant $A$ in~\eqref{eq:FODE3sol} must satisfy
\[A=x_{0}\cdot \exp{-\int_{0}^{t_{0}}\l(s)ds}-\int_{0}^{t_{0}}f(z) \cdot \exp{-\int_{0}^{z}\l(s)ds} dz.\]
Hence the solution to the initial-value problem is
\begin{equation}
x(t) =x_{0}\cdot \exp{\int_{t_{0}}^{t}\l(s)ds}+\int_{t_{0}}^{t}f(z)\cdot \exp{\int_{z}^{t}\l(s)ds}dz.\label{eq:icsol}
\end{equation}

\section{Linear Systems of First-Order Differential Equations}\label{sec:two}

We often encounter dynamic systems with several variables that move together over time. For example in the consumption-saving problem that we solved with optimal control methods, in the case of CRRA utility, the solution is characterized by two FODEs:
\begin{align*}
\dot{a}(t) &=r\cdot a(t)-c(t), \\
\dot{c}(t) &=\frac{r-\rho}{\g}\cdot  c(t).
\end{align*}
The first FODE is the asset accumulation equation and the second FODE is the Euler equation that characterizes optimal consumption over time. To solve explicitly for the optimal consumption path, we need to solve the two FODEs
simultaneously. This section presents a method to solve systems of FODEs.

We consider a system of $n$ FODEs with constant coefficients:
\begin{align*}
\dot{x}_{1}(t) &=A_{11}\cdot x_{1}(t)+A_{12}\cdot x_{2}(t)+\ldots+A_{1n}\cdot x_{n}(t) +f_{1}(t) \\
\dot{x}_{2}(t) &=A_{21}\cdot x_{1}(t)+A_{22}\cdot x_{2}(t)+\ldots+A_{2n}\cdot x_{n}(t) +f_{2}(t) \\
&. \\
&. \\
&. \\
\dot{x}_{n}(t) &=A_{n1}\cdot x_{1}(t)+A_{n2}\cdot x_{2}(t)+\ldots+A_{nn}\cdot x_{n}(t) +f_{n}(t).
\end{align*}
Our goal is to solve for the $n$ functions $x_{1}(t)$, $x_{2}(t),\ldots, x_{n}(t)$.

An alternative way of expressing the system is to write it in matrix form:
\begin{equation}
\bm{\dot{x}}(t) =\bm{A}  \bm{x}(t) +\bm{f}(t), \label{eq:FODEsys}
\end{equation}
where $\bm{\dot{x}}(t) \in \R^{n}$, $\bm{x}(t) \in \R^{n}$, and $\bm{f}(t) \in \R^{n}$ are column vectors with $n$ elements. $\bm{A}\in \R^{n\times n}$ is a constant $n\times n$ matrix. The system of FODEs is linear because it can be written in matrix form: it involves a linear relationship between the vector $\bm{\dot{x}}(t)$ and the vector $\bm{x}(t)$.

If $\bm{A}$ is diagonal ($A_{ij}=0$ for all $i\neq j$), the system would reduce to a collection of $n$ FODEs---one FODE for each  $x_{i}(t)$---that can be solved independently using the techniques from Section~\ref{sec:one}.
If $\bm{A}$ is not diagonal, the different entries in $\bm{x}(t)$ interact and we must solve the system of FODEs simultaneously.

\subsection{General Solution}

Assume that $\bm{A}$ is diagonalizable. There exists $\bm{V}\in \R^{n\times n}$ such that
\begin{equation}
\bm{A}=\bm{V}\bm{\Lambda}\bm{V}^{-1},\label{eq:DECO}
\end{equation}
where $\bm{\Lambda}\in \R^{n\times n}$ is a diagonal matrix. The diagonal entries of $\bm{\Lambda}$ are the $n$ eigenvalues $\l_{1},\ldots,\l_{n}$ of $\bm{A},$ and $\bm{V}$ is the matrix whose
columns are the eigenvectors $\bm{z}_{1},\ldots,\bm{z}_{n}$ of $\bm{A}$.

By definition, $\l_{1},\ldots,\l_{n}$ are the $n$ roots of the polynomial equation
\begin{equation*}
\det( \bp{\bm{A}-\l \bm{I}} =0.
\end{equation*}
For any $i=1,\dots,n$, the eigenvector $\bm{z}_{i}$ associated with the eigenvalue $\l_{i}$ satisfies
\begin{equation*}
\bp{\bm{A}-\l_{i} \bm{I}} \bm{z}_{i} =\bm{0}.
\end{equation*}

Using the decomposition~\eqref{eq:DECO}, we rewrite the system \eqref{eq:FODEsys} as
\begin{align}
\bm{V}^{-1}\bm{\dot{x}}(t) &=\bm{\Lambda} \bm{V}^{-1}\bm{x}(t) +\bm{V}^{-1}\bm{f}(t)\nonumber\\
\bm{\dot{y}}(t) &=\bm{\Lambda} \bm{y}(t) +\bm{g}(t) ,  \label{eq:FODEsyst}
\end{align}
where we define
\begin{align*}
\bm{y}(t) &\equiv \bm{V}^{-1}\bm{x}(t)\\
\bm{g}(t) &\equiv \bm{V}^{-1}\bm{f}(t).
\end{align*}
Since the matrix $\bm{\Lambda}$ is diagonal, the system is reduced to a collection of $n$ independent FODEs---one for each $y_{i}(t)$. Once we have solved for $\bm{y}(t)$, we can recover $\bm{x}(t)$ by
\begin{equation*}
\bm{x}(t) =\bm{V} \bm{y}(t) .
\end{equation*}
The nature of the eigenvalues and corresponding eigenvectors determines the dynamics of the solution.

\subsection{Homogenous Systems}

If $\bm{f}(t) =\bm{0}$, the system \eqref{eq:FODEsys} is homogenous; otherwise it is nonhomogenous.

For homogenous systems,
\begin{equation}
\bm{\dot{x}}(t) =\bm{A} \bm{x}(t).\label{eq:FODEsysh}
\end{equation}
So the transformed system~\eqref{eq:FODEsyst} becomes
\begin{equation*}
\bm{\dot{y}}(t) =\bm{\Lambda}\bm{y}(t) ,
\end{equation*}
which leads to $n$ independent FODEs:
\begin{equation*}
\dot{y}_{i}(t) -\l_{i} \cdot y_{i}(t)=0
\end{equation*}
for $i=1,\ldots,n$. In other words, each $y_{i}(t) $ is growing at constant rate $\l_{i}$. The analysis of Section~\ref{sec:one} shows that the solution to the $i^{th}$ FODE is
\begin{equation*}
y_{i}(t) =A_{i}\cdot e^{\l_{i}\cdot t}
\end{equation*}
where $A_{i}\in \R$ is a constant. Finally, $x_{1}(t),\ldots,x_{n}(t)$ are given by
\begin{equation*}
\bm{x}(t) =\bm{V} \bm{y}(t) .
\end{equation*}
The columns of $\bm{V}$ are the eigenvectors $\bm{z}_{1},\ldots,\bm{z}_{n}$ corresponding to the
eigenvalues $\l_{1},...\l_{n}$. Hence the solution of the homogenous system~\eqref{eq:FODEsysh} is
\begin{equation}
\bm{x}(t) =A_{1}\cdot \bm{z}_{1} \cdot e^{\l_{1}\cdot t}+\ldots+A_{n}\cdot \bm{z}_{n}\cdot e^{\l_{n}\cdot t}.\label{eq:SOLEV}
\end{equation}
The nature of the eigenvalues and the corresponding eigenvectors determines
the dynamics of the solution.

\subsection{Closed-Form Solution to a Two-Variable Homogenous System}

As an example, we consider a two-variable homogenous system:
\begin{align*}
\dot{x}_{1}(t) &=a\cdot x_{1}(t)+b\cdot x_{2}(t) \\
\dot{x}_{2}(t) &=c\cdot x_{1}(t)+d\cdot x_{2}(t).
\end{align*}
We can write it in matrix form
\begin{equation*}
\bm{\dot{x}}(t) =\bm{A} \bm{x}(t)
\end{equation*}
where the matrix $\bm{A}$ is
\begin{equation*}
\bm{A}=\bs{
\begin{array}{ll}
a & b \\
c & d
\end{array}}.
\end{equation*}
Assume $\det(\bm{A}) =a\cdot d-b\cdot c\neq 0. $

Equation~\eqref{eq:SOLEV} implies that to determine a closed-form solution of this homogenous system, we need to find the eigenvalues and eigenvectors of the matrix $\bm{A}$.

The eigenvalues are solutions to
\begin{align*}
\det( \bp{\bm{A}-\l \bm{I}} &=0\\
\bp{a-\l} \cdot \bp{d-\l} -b\cdot c &=0 \\
\l ^{2}-\bp{a+d}\cdot  \l +\bp{a\cdot d-b\cdot c} &=0.
\end{align*}
Note that the product of the two eigenvalues is equal to the determinant of $
\bm{A}$:
\begin{equation}
\l_{1}\cdot \l_{2}=a\cdot d-b\cdot c=\det(\bm{A}).\label{eq:DETL}
\end{equation}

Let $\bs{
\begin{array}{l}
\a_{1} \\
\b_{1}
\end{array}
} $ be the eigenvector correspond to $\l_{1}$ and $\bs{
\begin{array}{l}
\a_{2} \\
\b_{2}
\end{array}} $ be the eigenvector correspond to $\l_{2}$. These vectors are solutions to
\begin{equation*}
\bp{\bm{A}-\l_{i}\bm{I}} \bs{
\begin{array}{l}
\a_{i} \\
\b_{i}
\end{array}} =0
\end{equation*}
which yields the system
\begin{align*}
\bp{a-\l_{i}}\cdot  \a_{i}+b\cdot \b _{i} &=0 \\
c\cdot \a_{i}+\bp{d-\l_{i}}\cdot  \b _{i} &=0.
\end{align*}

Consider the cases where the eigenvalues are real and distinct, the general
solution~\eqref{eq:SOLEV} implies
\begin{align*}
x_{1}(t) &=A_{1}\cdot \a_{1}\cdot e^{\l_{1}\cdot t}+A_{2}\cdot \a_{2}\cdot e^{\l_{2}\cdot t}\\
x_{2}(t) &=A_{1}\cdot \b_{1}\cdot e^{\l_{1}\cdot t}+A_{2}\cdot \b_{2}\cdot e^{\l_{2}\cdot t}
\end{align*}
where $A_{1}$ and $A_{2}$ are arbitrary constants.

Note that in the case in which $\l_{1}=\l_{2}=\l$, the system $\bs{x_{1}(t),x_{2}(t)}$ above is still the general solution of the system of FODEs as long as the two eigenvectors $\bs{\a_{1},\b_{1}}$ and $\bs{\a_{2},\b_{2}}$ are linearly independent.

Also note that any nonhomogenous system with constant terms $\bs{\k_{1},\k_{2}}$:
\begin{align*}
\dot{x}_{1}(t) &=a\cdot x_{1}(t)+b\cdot x_{2}(t)+\k_{1} \\
\dot{x}_{2}(t) &=c\cdot x_{1}(t)+d\cdot x_{2}(t)+\k_{2},
\end{align*}
can be transformed into an homogenous system.

\subsection{Stability of a Two-Variable Homogenous System}

Now that we have found a closed-form solution to the system, we can analyze its stability. There are three cases.

\paragraph{Sink: $\l_{1}<0$ and $\l_{2}<0$} As shown by~\eqref{eq:DETL}, since $\l_{1}$ and $\l_{2}$ have the same sign, $\det(\bm{A}) >0$. As $t\to +\infty$, $x_{1}(t)\to 0$ and $x_{2}(t)\to 0$. The system is a \textit{sink}.

\paragraph{Source: $\l_{1}>0$ and $\l_{2}>0$} As shown by~\eqref{eq:DETL}, since $\l_{1}$ and $\l_{2}$ have the same sign, $\det(\bm{A}) >0$. As $t\to +\infty$, $|x_{1}(t)|\to +\infty$ and $|x_{2}(t)|\to +\infty$. The system is a \textit{source}.

\paragraph{Saddle: $\l_{1}$ and $\l_{2}$ have opposite sign} As shown by~\eqref{eq:DETL}, since $\l_{1}$ and $\l_{2}$ have opposite sign, $\det(\bm{A}) <0$. One part of the solution is stable (it converges to 0 at $t\to +\infty$), the other is unstable (it converges to $\infty$ at $t\to +\infty$). The system is a \textit{saddle}.

\section{Phase Diagrams}\label{sec:three}

Without solving for eigenvalues and eigenvectors explicitly, we can study the properties of a linear system of FODEs by drawing its phase diagram.

Here we construct the phase diagram for the following linear nonhomogenous system of two FODEs:
\begin{align}
\dot{x}(t) &=a\cdot x(t)+b\cdot y(t)+\k_{1}\label{eq:nonh1}\\
\dot{y}(t) &=c\cdot x(t)+d\cdot y(t)+\k_{2}\label{eq:nonh2}
\end{align}
with $a<0$, $b<0$, $c<0$, $d>0$, $\k_{1}>0$, and $\k_{2}>0$. Since $a\cdot d-b\cdot c<0$, the eigenvalues of the system are of opposite sign. Hence the dynamical system is a saddle.

Drawing the phase diagram of a two-variable system is useful to understand the main features of the dynamic system without solving for $x(t) $ and $y(t)$ explicitly. The phase diagram is represented in figure~\ref{f:phase}.

\begin{figure}[p]
\subcaptionbox{Nullclines \label{f:phase1}}{\includegraphics[scale=\sfig,page=1]{\pdf}}\hfill
\subcaptionbox{Steady state \label{f:phase2}}{\includegraphics[scale=\sfig,page=2]{\pdf}}\vfig
\subcaptionbox{Directional arrows \label{f:phase3}}{\includegraphics[scale=\sfig,page=3]{\pdf}}\hfill
\subcaptionbox{Trajectories \label{f:phase4}}{\includegraphics[scale=\sfig,page=4]{\pdf}}
\caption{Phase diagram for the dynamical system \eqref{eq:nonh1}--\eqref{eq:nonh2}}
\label{f:phase}\end{figure}

\subsection{Nullclines}

We first plot the nullclines, which are the loci $\dot{x}=0$ and $\dot{y}=0$ (figure~\ref{f:phase1}).

The locus for $\dot{x}=0$ is given by
\begin{align*}
y=-\frac{a}{b}\cdot x-\frac{\k_{1}}{b}.
\end{align*}
The locus is a straight line with a negative slope in the $(x,y)$ plan.

The locus for $\dot{y}=0$ is given by
\begin{align*}
y=-\frac{c}{d}\cdot x-\frac{\k_{2}}{d}.
\end{align*}
The locus is a straight line with positive slope in the $(x,y)$ plan.

\subsection{Steady state}

Next we place the system's steady state(figure~\ref{f:phase2}). The steady state is given by the intersection of the two nullclines. Denote the intersection of the two nullclines as $\bp{x^{*},y^{*}} $. These two nullclines divide the $(x,y)$ plane into four areas.\footnote{In some other fields the steady state of the system is called \textit{critical point} of the system.}

\subsection{Directional arrows}

Then we place on the diagram the directional arrows. These arrows determine the direction of the system's trajectories over time anywhere on the phase diagram (figure~\ref{f:phase3}).

From~\eqref{eq:nonh1}, we see that $\dot{x}$ is decreasing in $y$ because $b<0$. Thus any point above the $\dot{x}=0$ line must have $\dot{x}<0$ and any point below the $\dot{x}=0$ line must have $\dot{x}>0$. We represent these properties by an horizontal arrow pointing west for any point above the $\dot{x}=0$ line and an horizontal arrow pointing east for any point below the $\dot{x}=0$ line.

Similarly, from \eqref{eq:nonh2}, $\dot{y}$ is increasing in $y$ because $d>0$. Thus any point above the $\dot{y}=0$ line must have $\dot{y}>0$ and any point below the $\dot{y}=0$ line must have $\dot{y}<0$. We represent these properties by a vertical arrow pointing north for any point above the $\dot{y}=0$ line and a vertical arrow pointing south for any point below the $\dot{y}=0$ line.

\subsection{Trajectories}

Using the directional arrows, we can draw trajectories that satisfy the system of differential equations (figure~\ref{f:phase4}). These are solutions to the system. To select a specific solution among all possible solutions, we will need to specify either an initial condition or a final condition.

Among all the trajectories, we highlight the saddle path for the system. We know that such a saddle path exist because the eigenvalues of the system have opposite sign. The saddle path is the straight line that goes through the steady state.\footnote{The saddle path is also sometimes called a \textit{stable line} of the system. There is also an unstable line, which goes through the steady state but moves away from it.}

\begin{figure}[p]
\subcaptionbox{Initial phase diagram \label{f:news1}}{\includegraphics[scale=\sfig,page=5]{\pdf}}\hfill
\subcaptionbox{New phase diagram \label{f:news2}}{\includegraphics[scale=\sfig,page=6]{\pdf}}\vfig
\subcaptionbox{Jump upon news \label{f:news3}}{\includegraphics[scale=\sfig,page=7]{\pdf}}\hfill
\subcaptionbox{Movement after the jump \label{f:news4}}{\includegraphics[scale=\sfig,page=8]{\pdf}}
\caption{Response to a shock in a phase diagram with a state variable}
\label{f:news}\end{figure}

\subsection{Using Phase Diagram with State Variable and Control Variable}

Suppose $x$ is a state variable: information revealed at $t$ does not influence its value at $t$. Suppose $y$ is a control variable: information revealed at $t$ may influence its value at $t$. Suppose that we are in the steady state $\bp{x^{*},y^{*}}$ of the previous phase diagram.

Assume that there is an exogenous, unanticipated increase in $\k_{2}$. This increase is revelation of news because it is an unanticipated change to one of the parameters or variables of the system. The response to the news in the phase diagram is represented in figure~\ref{f:news}.

As $\k_{2}$ increases, the $\dot{y}=0$ locus shifts down, so the new steady state $\bp{x^{* *},y^{* *}} $ is to the south-east of the previous steady state: $x^{* *}>x^{*}$ and $y^{* *}<y^{*}.$ There is also a new saddle path passing through this new steady state.

Where do we start after the news is revealed at $t=t_{r}$? That is, what are $x\bp{t_{r}}$ and $y\bp{t_{r}}$? Since $x$ is the state variable, it cannot respond to the news, and $x\bp{t_{r}} =x^{*}$. For the system to converge to the new steady state, it must arrive at the steady-state level $x^{* *}$ of the state variable along the new saddle path. So $y\bp{t_{r}} $ must be on the new saddle path at $x^{*}$, and over time both $x(t) $ and $y(t) $ move along the saddle path until they converge to the new steady state. To sum up, the system jumps from $\bp{x^{*},y^{*}}$ to $\bp{x^{*},y\bp{t_{r}}} $, and then moves along the saddle path until it reaches $\bp{x^{* *},y^{* *}}$.

\section{Nonlinear Systems of First-Order Differential Equations}

Unlike the systems of FODEs studied in sections~\ref{sec:two} and~\ref{sec:three}, which were linear, systems of FODEs in macroeconomics are often nonlinear. For example the typical growth model is characterized by the following nonlinear system of FODEs:
\begin{align}
\dot{k}(t) &=f\bp{k(t)} -c(t)-\d \cdot k(t),  \label{eq:growth1} \\
\dot{c}(t) &=\bs{f'\bp{k(t)} -\bp{\d+\rho}}\cdot c(t), \label{eq:growth2}
\end{align}
where $\rho >0$,and $\d \in \bp{0,1}$ are parameters, the capital stock $k(t)$ is a state variable
with $k_{0}$ given, and the production function $f$ satisfies the Inada conditions:
\begin{align*}
f\bp{0}=0,\; f'>0,\;f''<0,\;\lim_{k\to +\infty}f'(k)=0,\;\lim_{k\to 0}f'(k) =+\infty.
\end{align*}
It is difficult to solve this system explicitly. But without solving it explicitly, we can characterize its properties by constructing its phase diagram. This is what we do in this section.

We draw the phase diagram in a plane with the state variable $k$ on the x-axis and the control variable $c$ on the y-axis (figure~\ref{f:growth}).

\begin{figure}[p]
\subcaptionbox{Nullclines \label{f:growth1}}{\includegraphics[scale=\sfig,page=9]{\pdf}}\hfill
\subcaptionbox{Steady state \label{f:growth2}}{\includegraphics[scale=\sfig,page=10]{\pdf}}\vfig
\subcaptionbox{Directional arrows \label{f:growth3}}{\includegraphics[scale=\sfig,page=11]{\pdf}}\hfill
\subcaptionbox{Trajectories\label{f:growth4}}{\includegraphics[scale=\sfig,page=12]{\pdf}}
\caption{Phase diagram for a simple growth model}
\label{f:growth}\end{figure}

\subsection{Nullclines}

We first draw the nullclines (figure~\ref{f:growth1}). We draw the $\dot{k}=0$ curve defined by
\begin{align*}
c=f\bp{k} -\d\cdot k
\end{align*}
and the $\dot{c}=0$ curve defined by
\begin{align*}
f^{\prime}\bp{k} =\d +\rho.
\end{align*}
In the $(k,c)$ plane, the $\dot{k}=0$ curve is concave and the $\dot{c}=0$ curve is a vertical line .

\subsection{Steady State}

The intersection of these two loci is the steady state $(k^{*},c^{*})$ of the system (figure~\ref{f:growth2}).

\subsection{Directional Arrows}

Next we construct the directional arrows (figure~\ref{f:growth3}). To do that, we partially differentiate equations~\eqref{eq:growth1} and~\eqref{eq:growth2}:
\begin{align*}
\pd{\dot{k}}{c} &=-1<0 \\
\pd{\dot{c}}{k} &=c\cdot f''(k)<0.
\end{align*}
Therefore as $c$ increases, $\dot{k}$ decreases. So, the horizontal arrows point eastward below the $\dot{k}=0$ curve and westward above it. Similarly as $k$ increases, $\dot{c}$ decreases. So the vertical arrows point northward to the left of the $\dot{c}=0$ curve and southward to the right of it.

\subsection{Trajectories}

The directional arrows drawn describe a saddle around the steady state (figure~\ref{f:growth4}). The only way for the economy to converge to the steady state is on the saddle path leading to it. This means that given any initial capital $k_{0}$, initial consumption $c_{0}$ is such that the pair $\bp{k_{0},c_{0}} $ lies on the saddle path.

\subsection{Linearization}

The phase diagram indicates that the system is a saddle around the steady state (figure~\ref{f:growth}). We can also obtain this result by linearizing the nonlinear system~\eqref{eq:growth1}--\eqref{eq:growth2} using a first-order Taylor expansion around the steady state:
\begin{align*}
\dot{k} &=\dot{k}^{*} +\bp{k-k^{*}} \cdot \pd{\dot{k}}{k}+\bp{c-c^{*}}\cdot \pd{\dot{k}}{c} \\
\dot{c} &=\dot{c}^{*} +\bp{k-k^{*}}\cdot  \pd{\dot{c}}{k}+\bp{c-c^{*}}\cdot \pd{\dot{c}}{c}.
\end{align*}
Given that $\dot{k}^{*} =\dot{c}^{*} =0,$ we have
\begin{equation*}
\bs{\begin{array}{l}
\dot{k}\\
\dot{c}
\end{array}} =\bm{J}^{*}\bs{
\begin{array}{l}
k-k^{*} \\
c-c^{*}
\end{array}},
\end{equation*}
where $\bm{J}^{*}$ is the Jacobian matrix evaluated at the steady state:
\begin{equation*}
\bm{J}^{*}=\bs{
\begin{array}{ll}
\pdw{\dot{k}}{k}{(k^{*},c^{*})}  & \pdw{\dot{k}}{c}{(k^{*},c^{*})} \\
\pdw{\dot{c}}{k}{(k^{*},c^{*})} & \pdw{\dot{c}}{c}{(k^{*},c^{*})}
\end{array}}.
\end{equation*}
This system is a two-variable nonhomogenous system of FODEs
for \[\bm{x}=\bs{\begin{array}{l}
k\\
c
\end{array}}.\]

But it is a two-variable homogenous system for the transformed variable $\bm{y}$, where
\begin{equation*}
\bm{y}=\bm{x}-\bm{x}^{*}=\bs{
\begin{array}{l}
k-k^{*}\\
c-c^{*}
\end{array}}.
\end{equation*}
The constant matrix $A$ of Section~\ref{sec:two} is $\bm{J}^{*}$. The analysis of Section~\ref{sec:two} shows that the properties of the steady state depend on the eigenvalues of $\bm{J}^{*}$. The four partial derivatives are
\begin{align*}
\pdw{\dot{k}}{k}{(k^{*},c^{*})}&=f'(k^{*}) -\d =\rho >0 \\
\pdw{\dot{k}}{c}{(k^{*},c^{*})}&=-1<0 \\
\pdw{\dot{c}}{k}{(k^{*},c^{*})}&=c\cdot f''\bp{k^{*}} <0 \\
\pdw{\dot{c}}{c}{(k^{*},c^{*})}&=f'(k^{*}) -\bp{\d +\rho} =0
\end{align*}
It follows that the Jacobian matrix can be written
\begin{equation*}
\bm{J}^{*}=\bs{
\begin{array}{ll}
\rho  & -1  \\
c\cdot f''(k) & 0
\end{array}}
\end{equation*}
As shown by~\eqref{eq:DETL}, the product of the two eigenvalues is the determinant of $\bm{J}^{*}$:  \[\det(\bm{J}^{*})=c\cdot f''(k) <0.\] Therefore, the two eigenvalues have opposite sign. This property confirms that around the steady state the system is a saddle.

\end{document}


\section{Singh Combustion Model}

The calculation of a chemical source term is explained in detail in Anderson. \cite{gen:anderson2}.
A brief summary of that exaplanation follows here.
Each species conservation equation has a source term for each species $k$ given by:

\begin{displaymath}
\dot{W}_k = {\cal M}_k \sum_{l=1}^{N_e} \left( \nu''_{l,k} - \nu'_{l,k} \right)
\left[ k_{f,l} \prod_{m=1}^{N_l} \left[ \chi_m \right]^{\nu'_{l,m}}
- k_{b,l} \prod_{m=1}^{N_l} \left[ \chi_m \right]^{\nu''_{l,m}} \right]
\end{displaymath}

$N_e$ is the number of elementary reactions involving species $k$.
$N_l$ is the number of species involved in the elementary reaction.
${\cal M}_k$ is the molecular weight of species $k$.
$\chi_m$ is the mole fraction of species $m$.
$\nu'_{l,k}$ is the stoichiometric coefficient for the reactants of reaction $l$.
$\nu''_{l,k}$ is the stoichiometric coefficient for the products of reaction $l$.
$k_b$ is the backward reaction rate constant and
$k_f$ is the forward reaction rate constant, given by the
modified Arrhenius equation:
\begin{displaymath}
k_f = AT^n \exp(-E/{\cal R}T)
\end{displaymath}
where the coefficients $A$, $n$ and $E$ are germane to a particular combustion
model. In this study, the Singh hydrogen-air combustion model is used
\cite{chem:singh} and the coefficients are given in table \ref{model}
below.

The backward and the forward reaction rate constants are related by the
equilibrium constant, $K_c$
\begin{displaymath}
\frac{k_f}{k_b} = K_c
\end{displaymath}
The equilibrium constant, $K_c$ is given by:
\begin{displaymath}
K_c = ({\cal R} T)^{-\Delta \nu}
       \exp \left( \frac{-\Delta G^0}{{\cal R} T} \right)
\end{displaymath}

where
\begin{displaymath}
\Delta \nu = \sum_{i=1}^{N_l} (\nu''_i - \nu'_i)
\end{displaymath}
The Gibbs free energy for species $i$
is defined as $G_i = H_i - T S_i$, where $H_i$ is the
enthalpy, $T$ is the temperature and $S_i$ is the entropy.
The difference in Gibbs free energy $\Delta G^0$ of products and reactants is
\begin{displaymath}
\Delta G^0 = \sum_{i=1}^{N_l} (\nu''_i - \nu'_i) \left( h^0_i - T s^0_i \right)
\end{displaymath}
where $h^0_i$ is the molar enthalpy (including the heat of formation at a
reference temperature of 298 K)
of the species $i$, and $s^0_i$ is its the molar entropy,
each calculated at a reference pressure (one atmosphere).\\
\\

\begin{table}
\begin{threeparttable}
\caption{Singh Hydrogen-Air Combustion Model with Nitrogen Inert}
\begin{tabular}{|cc|c|c|c|} \hline
\multicolumn{2}{|c|}{Reaction} & A & n & E  \\ \hline \hline
(1) & H$_{2}$ + O$_{2} \rightarrow$ OH + OH & 1.70 $\times$ 10$^{13}$ & 0 & 48 000 \\
(2) & H + O$_{2} \rightarrow$ OH + O & 1.42 $\times$ 10$^{14}$ & 0 & 16 400 \\
(3) & O + H$_{2} \rightarrow$ OH + H & 2.07 $\times$ 10$^{14}$ & 0 & 13 750 \\
(4) & OH + H$_{2} \rightarrow$ H$_{2}$O + H & 3.16 $\times$ 10$^{7}$ & 1.80 & 3 030 \\
(5) & OH + OH $\rightarrow$ H$_{2}$O + O & 5.50 $\times$ 10$^{13}$ & 0 & 7 000 \\
(6) & H + OH + M $\rightarrow$ H$_{2}$O + M & 2.21 $\times$ 10$^{22}$ & -2.00 & 0 \\
(7) & H + H + M $\rightarrow$ H$_{2}$ + M & 6.53 $\times$ 10$^{17}$ & -1.00 & 0 \\
\hline
\end{tabular}
\label{model}
\end{threeparttable}
\end{table}

The units for $A$ are in $[\frac{cm^{3b}}{(mole-s)^b}]$ where $b=1$ for two body reactions and
$b=2$ for three body reactions. $E$ is in $[\frac{cal}{mole}]$.\\
\\

The symbol $M$ denotes a third-body collision partner, a species acting as a catalyst only.
The concentration of $M$ is simply determined from the equation :\\

\begin{displaymath}
X_M = \sum_{k=1}^{n_s} \eta_k X_k
\end{displaymath}

where $\eta_k$ is the third-body efficiency. $\eta_k$ is unity
for all species and reactions in this case.