\chapter{Signals \& Systems}

\section{What is a system?}

A system is defined as a grouping of elements to be
analysed together. They can be categorised as linear or
non-linear, depending on the equations used to
describe them.
Linear systems are considered to be idealised systems,
whilst non-linear systems are those representing real-world
conditions.

Systems may also be categorised based on the
\emph{order} of their differential equations. Some examples
of categorised systems are shown in table \ref{table:
examples of systems}.

\begin{table}[h!]
\centering
\begin{tabular}{c|cc}
 & Linear & Non-Linear\\
 \hline
1st order & RC circuit & Population growth\\
2nd order & Spring Mass Damper & Pendulum\\
3rd order & - & Chaotic Systems\\
... & ...  & ... \\
Nth order & Wave Equation & General Relativity\\
\hline
\end{tabular}
\label{table: examples of systems}
\caption{Examples of linear and non-linear systems.}
\end{table}

\section{Linear Systems}
A system is considered linear if they meet the following
two principles:

\begin{itemize}
 \item Proportionality. Given an input $t$, the system will
return an output $f(t)$, and if given an input $\alpha t$
the system will return an output $f(\alpha t) = \alpha
f(t)$.
 \item Superposition. Given inputs $t_1$ and $t_2$, the
system will return outputs $f(t_1)$ and $f(t_2)$
respectively. If given an input $t_1 + t_2$ the system will
return an output $f(t_1 + t_2) = f(t_1) + f(t_2)$.
\end{itemize}

\subsection{Exercise: Linear systems}
Given the following equations, determine if they belong to
linear systems or not.

\textbf{Case 1:} $f(s) = 3s$. Evaluating the function to
test proportionality, we obtain the following:

\begin{equation*}
 \begin{split}
  f(2s) & =3(2s)\\
  & = 6 s\\
  2 f(s) &= 2 (3s)\\
  &= 6s\\
  f(2s)&=2f(s)
 \end{split}
\end{equation*}

For the superposition principle, we have:
\begin{equation*}
 \begin{split}
  f(s_1) & =3s_1\\
  f(s_2) & =3s_2\\
  f(s_1+s_2) & =3(s_1+s_2)\\
   &= 3s_1 + 3s_2\\
  f(s_1 + s_2) & = f(s_1) + f(s_2)
 \end{split}
\end{equation*}

We conclude that case 1 is linear.

% \vspace*{1cm}

An equation can be tested to meet both principles
simultaneously. Let $s=\alpha(s_1+s_2)$ and $f(s)$ be the
output of the system. A system is linear if the following
condition is met:

\begin{equation}
  f(\alpha(s_1+s_2))  = \alpha f(s_1) + \alpha f(s_2)
  \label{eq: proportionality and superposition}
\end{equation}

Evaluating \ref{eq: proportionality and superposition} for
case 1 as an example:

\begin{equation*}
\begin{split}
 f(\alpha (s_1 + s_2)) & = 3 (\alpha (s_1 + s_2))\\
 & = 3 \alpha s_1 + 3 \alpha s_2\\
 f(\alpha s) & = 3 \alpha s \\
 f(\alpha (s_1 + s_2)) & = \alpha f(s_1) + \alpha f(s_2)
\end{split}
\end{equation*}

\textbf{Case 2:} $f(s) = 3s + 1$.

\begin{equation*}
 \begin{split}
  f(\alpha s) & = 3 \alpha  s +1\\
  \alpha f(s) & = 3 \alpha s + \alpha\\
  f(\alpha s) & \neq \alpha f(s)\\
  f(s_1) + f(s_2) & = 3 s_1 + 3 s_2 + 2\\
  f(s_1 + s_2) & = 3 (s_1+s_2) + 1\\
  f(s_1 +s_2) & \neq f(s_1) + f(s_2)
 \end{split}
\end{equation*}

Case 2 is non-linear.

% \vspace*{1cm}

\textbf{Case 3:} $f(s) = 0.5 \cos (0.1 s) $

\begin{equation*}
 \begin{split}
  f(\alpha s) & = 0.5 \cos (0.1 \alpha s)\\
  \alpha f(s) & = 0.5 \alpha \cos (0.1 s)\\
  f(\alpha s) & \neq \alpha f(s)\\
  f(s_1) + f(s_2) & = 0.5 \cos (0.1 s_1) + 0.5 \cos (0.1
s_2) \\
  f(s_1 + s_2) & = 0.5 \cos (0.1 s_1 + 0.1 s_2)\\
  f(s_1 + s_2) & \neq f(s_1) + f(s_2)
 \end{split}
\end{equation*}

Case 3 is non-linear.

% \vspace*{1cm}

\textbf{Case 4:} $f(s) = 1.2 \mathrm{e}^{0.1 s}$

\begin{equation*}
 \begin{split}
f(\alpha (s_1+s_2))  & = 1.2 \mathrm{e}^{0.1 \alpha (s_1 +
s_2)}\\
\alpha f(s_1) & = 1.2 \alpha \mathrm{e}^{0.1 s_1}\\
\alpha f(s_1) + \alpha f(s_2) & = 1.2 \alpha
(\mathrm{e}^{0.1 s_1} + \mathrm{e}^{0.1 s_2})\\
f(\alpha (s_1+s_2))  & \neq \alpha f(s_1) + \alpha f(s_2)
 \end{split}
\end{equation*}

Case 4 is non-linear.

% \vspace*{1cm}

\textbf{Case 5:} $f(s) = \int\limits_0^t s(t)dt$
\begin{equation*}
 \begin{split}
f(\alpha (s_1 + s_2))  & = \alpha \int\limits_0^t
\Bigl( s_1(t)+s_2(t) \Bigr)  dt\\
& = \alpha \int\limits_0^t
s_1(t)dt + \alpha \int\limits_0^t
s_2(t)dt\\
\alpha f(s_1) + \alpha f(s_2) & = \alpha \int\limits_0^t
s_1(t)dt + \alpha \int\limits_0^t
s_2(t)dt\\
f(\alpha (s_1+s_2))  & = \alpha f(s_1) + \alpha f(s_2)
 \end{split}
\end{equation*}

Case 5 is linear.

% \vspace*{1cm}

\textbf{Case 6:} $f(s) = \frac{ds(t)}{dt}$

\begin{equation*}
 \begin{split}
f(\alpha (s_1 + s_2))  & =  \alpha
\frac{d \Bigl (s_1(t)+s_2(t) \Bigr)}{dt}\\
& = \alpha \frac{d s_1(t)}{dt} + \alpha \frac{d
s_2(t)}{dt} \\
\alpha f(s_1) + \alpha f(s_2) & = \alpha \frac{d s_1(t)}{dt}
+ \alpha \frac{d s_2(t)}{dt} \\
f(\alpha (s_1+s_2))  & = \alpha f(s_1) + \alpha f(s_2)
 \end{split}
\end{equation*}

Case 6 is linear.

Note that for all equations $\alpha \neq 1$ because using
the identity is not valid for proving compliance.

\subsection{Order of a system}

The order of a system is dependent on the highest order of
derivatives in the equations that describe it. A system
with no derivative terms is a \emph{zero order} system, a
system with a differential equation of order 1 is a
\emph{first-order} system, and so on.

Revisiting the cases shown in Exercise 1, we can classify
the systems depending on their order as well. Table
\ref{table: exercise 1 order} contains the classification
of each case.

\begin{table}[t]
\centering
\begin{tabular}{c|cc}
Equation & Type & Order\\
\hline
$3s$ & Linear & 0\\
$3s + 1$ & Non-linear & 0\\
$0.5 \cos (0.1 s)$ & Non-Linear & 0\\
$1.2 \mathrm{e}^{0.1s}$ & Non-Linear & 0\\
$\int\limits_0^{t} s(t)dt$ & Linear & 0\\
$\frac{d s(t)}{dt}$ & Linear & 1\\
\hline
\end{tabular}
\label{table: exercise 1 order}
\caption{Classification of systems.}
\end{table}

\section{Example: Spring-Mass-Damper System}

The Spring-Mass-Damper System (abbreviated SMD) is the most
commonly used abstraction for systems. Depending on the
initial conditions of the systems, it may or may not be
linear. An example is presented below.

Given the forces interacting in the Free Body Diagram, the
system's equation is obtained as follows:

\begin{equation}
 \begin{split}
  \sum F & = 0\\
  F_s(t) + F_d(t) - F(t) & = 0\\
 \end{split}
 \label{eq: sum of forces - week01}
\end{equation}

Recall the equations for springs and dampers, substituting
them in \eqref{eq: sum of forces - week01}.

\begin{equation*}
 \begin{split}
  F_s & = b \dot{x}\\
  F_d & = k x(t)\\
  b \dot{x} + k x(t) & = F(t)
 \end{split}
\end{equation*}

Rewriting the equation into the form $\dot x + f(t) x(t) =
g(t)$.

\begin{equation}
 \dot{x} + \frac{k}{b} x(t) = \frac{1}{b} F(t)
 \label{eq: system equation}
\end{equation}

\subsection{Linear System Case}

Solving the system via Laplace Transforms to obtain an
equation for x(t). Consider $x(0) = 0$ as the initial
condition.

\begin{equation}
 \begin{split}
  \mathfrak{L} \Bigl \{ \dot{x} + \frac{k}{b} x(t)  =
\frac{1}{b} F(t) \Bigr \} & \xrightarrow{} s \mathcal{X} -
x(0) + \frac{k}{b} \mathcal{X} = \frac{1}{b} \mathcal{F}
 \end{split}
 \label{eq: general laplace transform}
\end{equation}

Evaluating $x(0)=0$ gives us the system equation in the
frequency domain.

\begin{equation}
 s \mathcal{X} + \frac{k}{b} \mathcal{X} = \frac{1}{b}
\mathcal{F}
\label{eq: laplace transform}
\end{equation}

Solve \ref{eq: laplace transform} for $\mathcal{X}$ and
then obtain the inverse Laplace transform of the equation.

\begin{equation*}
 \begin{split}
  \mathcal{X} (s + \frac{k}{b}) & = \frac{1}{b}
\mathcal{F}\\
\mathcal{X} & = \frac{1}{b} \frac{1}{s + \frac{k}{b}}
\mathcal{F}\\
 \end{split}
\end{equation*}

\begin{equation}
 \begin{split}
  \mathfrak{L}^{-1} \Bigl \{ \mathcal{X}  = \frac{1}{b}
\frac{1}{s + \frac{k}{b}}
\mathcal{F}   \Bigr\} & \xrightarrow{} x(F(t)) =
\frac{1}{b} \mathrm{e}^{-\frac{k}{b}t} F(t)
 \end{split}
 \label{eq: linear system equation}
\end{equation}

Testing the principles of proportionality and superposition
on \ref{eq: linear system equation} we have

\begin{equation*}
 \begin{split}
  x(\alpha (F_1 + F_2)) & = \frac{\alpha}{b}
\mathrm{e}^{-\frac{k}{b}t} (F_1 + F_2)\\
\alpha x(F_1) & = \frac{\alpha}{b}
\mathrm{e}^{-\frac{k}{b}t} F_1\\
\alpha x(F_1) + \alpha x(F_2) & = \frac{\alpha}{b}
\mathrm{e}^{-\frac{k}{b}t} (F_1 + F_2)\\
x(\alpha (F_1 + F_2)) & = \alpha x(F_1) + \alpha x(F_2)
 \end{split}
\end{equation*}

We confirm that the system is linear for an initial
condition of $x(0)=0$.


\section{\textbf{MMThreadTest}}

\subsection{Particular Case (problem)}
The problem is that of matrix multiplication, but done in parallel. \\

Note: this exercise is exactly the same as the homonym from chapter
14; we believe the authors of the book made a mistake and ended up
with this repetition. Still we are repeating here for the sake of
completeness.

\subsection{Solution}
The solution creates one thread per cell on the result matrix, and for
each one it creates a new worker thread; which in turn calculates the
cell $(i,j)$ value by computing the dot product of row $i$ of left
matrix by column $j$ of right matrix (according to the definition of
the matrix product). Below the most relevant sections of the code: \\

\begin{lstlisting}[style=numbers]
  public MMThread(double[][] a, double[][]  b) {
    n = a.length;
    this.a = a;
    this.b = b;
    this.c = new double[n][n];
  }

  void multiply() {
    Worker[][] worker = new Worker[n][n];
    // create one thread per matrix entry
    for (int row = 0; row < n; row++) {
      for (int col = 0; col < n; col++) {
        worker[row][col] = new Worker(row,col);
      }
    }
    ...
  }

  class Worker extends Thread {
    int row, col;
    Worker(int row, int col) {
      this.row = row; this.col = col;
    }
    public void run() {
      double dotProduct = 0.0;
      for (int i = 0; i < n; i++) {
        dotProduct += a[row][i] * b[i][col];
      }
      c[row][col] = dotProduct;
    }
  }
\end{lstlisting}
\hfill

\subsection{Experiment Description}
The test consists of multiplying two identify matrices of dimensions
$3 \times 3$, and compare the result with one of the operands (it
should be identity matrix again).

\subsection{Observations and Interpretations}
The test runs fine on 2 and 24 core machines, below a sample output: \\

\begin{verbatim}
[oraadm@gdlaa008 orig]$ junit steal.MMThreadTest
.run

Time: 0.004

OK (1 test)
\end{verbatim}
\hfill


\section{Partial Differentiation}\label{sec:PartialDifferentiation}

The derivative of a function of a single variable tells us how quickly
the value of the function changes as the value of the independent variable
changes. Intuitively, it tells us how ``steep'' the graph of the function is.
We might wonder if there is a similar idea for graphs of functions
of two variables, that is, surfaces. It is
not clear that this has a simple answer, nor how we might proceed. We
will start with what seem to be very small steps toward the goal.
Surprisingly, it turns out that these simple ideas hold the keys to a
more general understanding.

\begin{figure}[H]
%\texonly
\centerline{
\vbox{\beginpicture
\normalgraphs
%\ninepoint
\setcoordinatesystem units <3truecm,3truecm>
\setplotarea x from 0 to 1.1, y from 0 to 1.1
\put {\hbox{\epsfxsize8cm\epsfbox{images/parabolic_crosssection.eps}}} at 0 0
\endpicture}}
%\endtexonly
\caption{The intersection of a plane $x+y=1$ and the surface $f(x,y)=x^2+y^2$}
\label{fig:parabolic bowl cross-section}
\end{figure}

The derivative of a single-variable function $f(x)$ tells us how much $f(x)$
changes as $x$ increases. The obvious analogue for a function of two
variables $g(x,y)$ would be something that tells us how quickly $g(x,y)$
increases as $x$ and $y$ increase. However, in most cases this will depend on
how quickly $x$ and $y$ are changing relative to each other.

\begin{example}{}{}
Analyze $f(x,y)=y^2$.
\end{example}
\begin{solution}
If we look at a point $(x,y,y^2)$ on this surface, the value of a function
does not change at all if we fix $y$ and let $x$ increase, but increases like $y^2$
if we fix $x$ and let $y$ increase.
\end{solution}

Now let us consider what happens to $f(x,y)$ when both $x$ and $y$ are
increasing, perhaps at different rates. We can think of this as being a
movement in a certain direction of a point in the $x,y$-plane. A point and a
direction defines a line in the $x,y$-plane, and so we are asking how the function
changes as we move along this line.

Let us then imagine a plane perpendicular to the $x,y$-plane that intersects
the the $x,y$-plane along this line. This plane will intersect the surface of
$f$ in a curve, so we can just look at the behaviour of this curve in the given plane.

Figure~\ref{fig:parabolic bowl cross-section} shows the plane $x+y=1$, which
is the plane perpendicular to the line $x+y=1$ in the $x,y$-plane.
Observe that its intersection with the surface of $f$ is a curve, in fact, a parabola.
We will refer to such a curve as the cross-section of the surface above the line in
the $x,y$-plane.

We can now look at the rate of change (or slope) of $f$ in a particular
direction by looking at the slope of a curve in a plane --- something we
already have experience with.

Let's start by looking at some particularly
easy lines: Those parallel to the $x$ or $y$ axis. Suppose we are
interested in the cross-section of $f(x,y)$ above the line $y=b$. If
we substitute $b$ for $y$ in $f(x,y)$, we get a function in one
variable, describing the height of the cross-section as a function of
$x$. Because $y=b$ is parallel to the $x$-axis, if we view it from a
vantage point on the negative $y$-axis, we will see what appears to be
simply an ordinary curve in the $x$-$z$ plane.

\begin{figure}[H]
%\texonly
\centerline{
\vbox{\beginpicture
\normalgraphs
%\ninepoint
\setcoordinatesystem units <3truecm,3truecm>
\setplotarea x from 0 to 4, y from 0 to 1
\put {\hbox{\epsfxsize8cm\epsfbox{images/parabolic_crosssection2.eps}}} at 0 0
\put {\hbox{\epsfxsize5cm\epsfbox{images/parabolic_crosssection3.eps}}} at 2.5 0
\endpicture}}
%\endtexonly
\caption{$f(x,y)=x^2 + y^2$, cut by the plane $y=2$}
\label{fig:parabolic bowl cross-section two}
\end{figure}

Consider again the parabolic surface $f(x,y)=x^2+y^2$. The
cross-section above the line $y=2$ consists of all points
$(x,2,x^2+4)$. Looking at this cross-section we see what appears to be just the curve
$f(x)=x^2+4$. At any point on the cross-section, $(a,2,a^2+4)$, the
slope of the surface \emph{in the direction of the line} $y=2$ is
  simply the slope of the curve $f(x)=x^2+4$, namely $2x$.
Figure~\ref{fig:parabolic bowl cross-section two} shows the same
parabolic surface as before, but now cut by the plane $y=2$. The left
graph shows the cut-off surface, the right shows just the
cross-section.

If, for example, we're interested in the point $(-1,2,5)$ on the surface, then
the slope in the direction of the line $y=2$ is $2x=2(-1)=-2$. This
means that starting at $(-1,2,5)$ and moving on the surface, above the
line $y=2$, in the direction of increasing $x$ values, the surface
goes down; of course moving in the opposite direction, toward
decreasing $x$ values, the surface will rise.

If we're interested in some other line $y=k$, there is really no
change in the computation. The equation of the cross-section above
$y=k$ is $x^2+k^2$ with derivative $2x$. We can save ourselves the
effort, small as it is, of substituting $k$ for $y$: all we are in
effect doing is temporarily assuming that $y$ is some constant. With
this assumption, the derivative ${d\over dx}(x^2+y^2)=2x$. To
emphasize that we are only temporarily assuming $y$ is constant, we
use a slightly different notation: ${\partial\over \partial
  x}(x^2+y^2)=2x$; the ``$\partial$'' reminds us that there are more
variables than $x$, but that only $x$ is being treated as a variable.
We read the equation as ``the partial derivative of $(x^2+y^2)$ with
respect to $x$ is $2x$.'' A convenient alternate notation for the
partial derivative of $f(x,y)$ with respect to $x$ is
is $f_x(x,y)$.

\begin{example}{Partial Derivative with respect to $x$}{PartialDerOne}
Find the partial derivative with respect to $x$ of $x^3+3xy$.
\end{example}
\begin{solution}
The partial derivative with respect to $x$ of
$x^3+3xy$ is $3x^2+3y$. Note that the partial derivative includes the
variable $y$, unlike the example $x^2+y^2$. It is somewhat unusual for
the partial derivative to depend on a single variable; this example is
more typical.
\end{solution}

Of course, we can do the same sort of calculation for lines parallel
to the $y$-axis. We temporarily hold $x$ constant, which gives us the
equation of the cross-section above a line $x=k$. We can then compute
the derivative with respect to $y$; this will measure the slope of
the curve in the $y$ direction.

\begin{example}{Partial Derivative with respect to $y$}{PartialDerTwo}
Find the partial derivative with respect to $y$ of $f(x,y)=\sin(xy)+3xy$.
\end{example}
\begin{solution}
The partial derivative with respect to $y$ of
$f(x,y)=\sin(xy)+3xy$ is
$$f_y(x,y)={\partial\over\partial y}\sin(xy)+3xy=\cos(xy){\partial\over\partial
  y}(xy)+ 3x=x\cos(xy)+3x.
$$
\end{solution}

So far, using no new techniques, we have succeeded in measuring the
slope of a surface in two quite special directions. For functions of
one variable, the derivative is closely linked to the notion of
tangent line. For surfaces, the analogous idea is the tangent
plane---a plane that just touches a surface at a point, and has the
same slope as the surface in all directions. Even though we
haven't yet figured out how to compute the slope in all directions, we
have enough information to find tangent planes. Suppose we want the
plane tangent to a surface at a particular point $(a,b,c)$. If we compute the
two partial derivatives of the function for that point, we get enough
information to determine two lines tangent to the surface, both
through $(a,b,c)$ and both tangent to the surface in their respective
directions. These two lines determine a plane, that is, there is
exactly one plane containing the two lines: the tangent
plane. Figure~\ref{fig:sphere with tangent plane}
shows (part of) two tangent lines at a point,
and the tangent plane containing them.

\begin{figure}[H]
%\texonly
\centerline{
\vbox{\beginpicture
\normalgraphs
%\ninepoint
\setcoordinatesystem units <3truecm,3truecm>
\setplotarea x from -1 to 3, y from 0 to 1
\put {\hbox{\epsfxsize4cm\epsfbox{images/tangent_vectors.eps}}} at 0 -0.15
\put {\hbox{\epsfxsize5.8cm\epsfbox{images/tangent_plane.eps}}} at 2.5 0
\endpicture}}
%\endtexonly
\caption{Tangent vectors and tangent plane.}
\label{fig:sphere with tangent plane}
\end{figure}

How can we discover an equation for this tangent plane? We know a
point on the plane, $(a,b,c)$; we need a vector normal to the
plane. If we can find two vectors, one parallel to each of the tangent
lines we know how to find, then the cross product of these vectors
will give the desired normal vector.

\begin{figure}[H]
%\texonly
\centerline{
\vbox{\beginpicture
\normalgraphs
%\ninepoint
\setcoordinatesystem units <1.5truecm,1.5truecm>
\setplotarea x from 0 to 4.1, y from 0 to 3.1
\axis left ticks numbered from 0 to 3 by 1 /
\axis bottom ticks numbered from 0 to 4 by 1 /
\put {$z$} [b] <0pt,3pt> at 0 3.1
\put {$x$} [l] <3pt,0pt> at 4.1 0
\put {$f_x(2,b)$} [l] <3pt,0pt> at 3 1.35
\putrule from 2 1.17 to 3 1.17
\putrule from 3 1.17 to 3 1.5
\setlinear
%\plot 2 1.17 3 1.5 /
\arrow <5pt> [0.17, 0.5] from 2 1.17 to 3 1.5
\setquadratic
\plot 0.000 1.167 0.100 1.135 0.200 1.107 0.300 1.082 0.400 1.060
0.500 1.042 0.600 1.027 0.700 1.015 0.800 1.007 0.900 1.002
1.000 1.000 1.100 1.002 1.200 1.007 1.300 1.015 1.400 1.027
1.500 1.042 1.600 1.060 1.700 1.082 1.800 1.107 1.900 1.135
2.000 1.167 2.100 1.202 2.200 1.240 2.300 1.282 2.400 1.327
2.500 1.375 2.600 1.427 2.700 1.482 2.800 1.540 2.900 1.602
3.000 1.667 3.100 1.735 3.200 1.807 3.300 1.882 3.400 1.960
3.500 2.042 3.600 2.127 3.700 2.215 3.800 2.307 3.900 2.402
4.000 2.500 /
%0.000 1.333 0.100 1.270 0.200 1.213 0.300 1.163 0.400 1.120
%0.500 1.083 0.600 1.053 0.700 1.030 0.800 1.013 0.900 1.003
%1.000 1.000 1.100 1.003 1.200 1.013 1.300 1.030 1.400 1.053
%1.500 1.083 1.600 1.120 1.700 1.163 1.800 1.213 1.900 1.270
%2.000 1.333 2.100 1.403 2.200 1.480 2.300 1.563 2.400 1.653
%2.500 1.750 2.600 1.853 2.700 1.963 2.800 2.080 2.900 2.203
%3.000 2.333 3.100 2.470 3.200 2.613 3.300 2.763 3.400 2.920
%3.500 3.083 3.600 3.253 3.700 3.430 3.800 3.613 3.900 3.803
%4.000 4.000 /
\endpicture}}
%\endtexonly
\caption{A tangent vector.}
\label{fig:tangent vector}
\end{figure}

How can we find vectors parallel to the tangent lines? Consider first
the line tangent to the surface above the line $y=b$. A vector
$\langle u,v,w\rangle$ parallel to this tangent line must have $y$
component $v=0$, and we may as well take the $x$ component to be
$u=1$. The ratio of the $z$ component to the $x$ component is the
slope of the tangent line, precisely what we know how to compute. The
slope of the tangent line is $f_x(a,b)$, so
$$ f_x(a,b)={w\over u} ={w\over1} = w.$$
In other words, a vector parallel to this tangent line is
$\langle 1,0,f_x(a,b)\rangle$, as shown in Figure~\ref{fig:tangent vector}.
If we repeat the reasoning for the
tangent line above $x=a$, we get the vector $\langle
0,1,f_y(a,b)\rangle$.

Now to find the desired normal vector we compute the cross product,
$\langle 0,1,f_y\rangle\times\langle 1,0,f_x\rangle=
\langle f_x,f_y,-1\rangle$. From our earlier discussion of planes, we
can write down the equation we seek: $f_x(a,b)x+f_y(a,b)y-z=k$, and
$k$ as usual can be computed by substituting a known point:
$f_x(a,b)(a)+f_y(a,b)(b)-c=k$. There are various more-or-less nice
ways to write the result:
\begin{align*}
&f_x(a,b)x+f_y(a,b)y-z=f_x(a,b)a+f_y(a,b)b-c	\\
&f_x(a,b)x+f_y(a,b)y-f_x(a,b)a-f_y(a,b)b+c=z	\\
&f_x(a,b)(x-a)+f_y(a,b)(y-b)+c=z	\\
&f_x(a,b)(x-a)+f_y(a,b)(y-b)+f(a,b)=z	\\
\end{align*}

\begin{example}{Tangent Plane to a Sphere}{TangentPlaneSphere}
Find the plane tangent to $x^2+y^2+z^2=4$ at
$(1,1,\sqrt2)$.
\end{example}
\begin{solution}
The point $(1,1,\sqrt2)$ is on the upper hemisphere, so
we use $\ds f(x,y)=\sqrt{4-x^2-y^2}$. Then
$\ds f_x(x,y)=-x(4-x^2-y^2)^{-1/2}$ and $\ds
f_y(x,y)=-y(4-x^2-y^2)^{-1/2}$, so $f_x(1,1)=f_y(1,1)=-1/\sqrt2$
and the equation of the plane is
$$z=-{1\over\sqrt2}(x-1)-{1\over\sqrt2}(y-1)+\sqrt2.$$
The hemisphere and this tangent plane are pictured in
Figure~\ref{fig:sphere with tangent plane}.
\end{solution}

So it appears that to find a tangent plane, we need only find two
quite simple ordinary derivatives, namely $f_x$ and $f_y$. This is
true {\em if the tangent plane exists}. It is, unfortunately, not
always the
case that if $f_x$ and $f_y$ exist there is a tangent plane.
Consider the function  $xy^2/(x^2+y^4)$ with $f(0,0)$ defined to be 0, pictured in
Figure~\ref{fig:weird limit}. This function has value 0 when $x=0$
or $y=0$. Now it's clear that $f_x(0,0)=f_y(0,0)=0$, because in the
$x$ and $y$ directions the surface is simply a horizontal line. But
it's also clear from the picture that this surface does not have
anything that deserves to be called a tangent plane at the origin,
certainly not the $x$-$y$ plane containing these two tangent lines.

When does a surface have a tangent plane at a particular point? What
we really want from a tangent plane, as from a tangent line, is that
the plane be a ``good'' approximation of the surface near the
point. Here is how we can make this precise:

\begin{definition}{Tangent Plane}{TangentPlaneDefinition}
Let $\Delta x=x-x_0$, $\Delta y=y-y_0$, and $\Delta z=z-z_0$
where $z_0=f(x_0,y_0)$. The
function $z=f(x,y)$ is differentiable\index{differentiable} at
$(x_0,y_0)$ if
$$\Delta z=f_x(x_0,y_0)\Delta x+f_y(x_0,y_0)\Delta y+\epsilon_1\Delta
x + \epsilon_2\Delta y,$$ where both $\epsilon_1$ and $\epsilon_2$
approach 0 as $(x,y)$ approaches $(x_0,y_0)$.\index{tangent plane}
\end{definition}

This definition takes a bit of absorbing. Let's rewrite the central
equation a bit:
\begin{equation}\label{eq:f is differentiable}
z=f_x(x_0,y_0)(x-x_0)+f_y(x_0,y_0)(y-y_0)+f(x_0,y_0)+
\epsilon_1\Delta x + \epsilon_2\Delta y.
\end{equation}
The first three terms on
the right are the equation of the tangent plane, that is,
$$f_x(x_0,y_0)(x-x_0)+f_y(x_0,y_0)(y-y_0)+f(x_0,y_0)$$
 is the $z$-value
of the point on the plane above $(x,y)$.
Equation~\ref{eq:f is differentiable} says that
the $z$-value of a point on the surface is equal to the $z$-value of a
point on the plane plus a ``little bit,'' namely $\epsilon_1\Delta x +
\epsilon_2\Delta y$. As $(x,y)$ approaches $(x_0,y_0)$, both $\Delta
x$ and $\Delta y$ approach 0, so this little bit $\epsilon_1\Delta x +
\epsilon_2\Delta y$ also approaches 0, and the $z$-values on the
surface and the plane get close to each other. But that by itself is
not very interesting: since the surface and the plane both contain the
point $(x_0,y_0,z_0)$, the $z$ values will approach $z_0$ and hence
get close to each other whether the tangent plane is ``tangent'' to
the surface or not. The extra condition in the definition says that as
$(x,y)$ approaches $(x_0,y_0)$, the $\epsilon$ values approach
0---this means that $\epsilon_1\Delta x + \epsilon_2\Delta y$
approaches 0 much, much faster, because $\epsilon_1\Delta x$ is much
smaller than either $\epsilon_1$ or $\Delta x$. It is this extra
condition that makes the plane a tangent plane.

We can see that the extra condition on $\epsilon_1$ and $\epsilon_2$
is just what is needed if we look at partial derivatives. Suppose we
temporarily fix $y=y_0$, so $\Delta y=0$. Then the equation from the
definition becomes
$$\Delta z=f_x(x_0,y_0)\Delta x+\epsilon_1\Delta x$$
or
$${\Delta z\over\Delta x}=f_x(x_0,y_0)+\epsilon_1.$$
Now taking the limit of the two sides as $\Delta x$ approaches 0, the
left side turns into the partial derivative of $z$ with respect to
$x$ at $(x_0,y_0)$, or in other words $f_x(x_0,y_0)$, and the right
side does the same, because as $(x,y)$ approaches $(x_0,y_0)$,
$\epsilon_1$ approaches 0. Essentially the same calculation works for
$f_y$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:PartialDifferentiation}}

\begin{enumialphparenastyle}

\begin{ex}
Find $f_x$ and $f_y$ where $\ds f(x,y)=\cos(x^2y)+y^3$.
\begin{sol}
$-2xy\sin(x^2y)$, $-x^2\sin(x^2y)+3y^2$
\end{sol}
\end{ex}

\begin{ex}
Find $f_x$ and $f_y$ where $\ds f(x,y)={xy\over x^2+y}$.
\begin{sol}
$(y^2-x^2y)/(x^2+y)^2$, $x^3/(x^2+y)^2$
\end{sol}
\end{ex}

\begin{ex}
Find $f_x$ and $f_y$ where $\ds f(x,y)=e^{x^2+y^2}$.
\begin{sol}
$2xe^{x^2+y^2}$, $2ye^{x^2+y^2}$
\end{sol}
\end{ex}

\begin{ex}
Find $f_x$ and $f_y$ where $\ds f(x,y)=xy\ln(xy)$.
\begin{sol}
$y\ln(xy)+y$, $x\ln(xy)+x$
\end{sol}
\end{ex}

\begin{ex}
Find $f_x$ and $f_y$ where $\ds f(x,y)=\sqrt{1-x^2-y^2}$.
\begin{sol}
$-x/\sqrt{1-x^2-y^2}$, $-y/\sqrt{1-x^2-y^2}$
\end{sol}
\end{ex}

\begin{ex}
Find $f_x$ and $f_y$ where $\ds f(x,y)=x\tan(y)$.
\begin{sol}
$\tan y$, $x\sec^2 y$
\end{sol}
\end{ex}

\begin{ex}
Find $f_x$ and $f_y$ where $\ds f(x,y)={1\over xy}$.
\begin{sol}
$-1/(x^2y)$, $-1/(xy^2)$
\end{sol}
\end{ex}

\begin{ex}
Find an equation for the plane tangent to
$\ds 2x^2+3y^2-z^2=4$ at
$(1,1,-1)$.
\begin{sol}
$z=-2(x-1)-3(y-1)-1$
\end{sol}
\end{ex}

\begin{ex}
Find an equation for the plane tangent to
$\ds f(x,y)=\sin(xy)$ at
$(\pi,1/2,1)$.
\begin{sol}
$z=1$
\end{sol}
\end{ex}

\begin{ex}
Find an equation for the plane tangent to
$\ds f(x,y)=x^2+y^3$ at
$(3,1,10)$.
\begin{sol}
$z=6(x-3)+3(y-1)+10$
\end{sol}
\end{ex}

\begin{ex}\label{ex:ln tan plane}
Find an equation for the plane tangent to
$\ds f(x,y)=x\ln(xy)$ at
$(2,1/2,0)$.
\begin{sol}
$z=(x-2)+4(y-1/2)$
\end{sol}
\end{ex}

\begin{ex}
Find an equation for the line normal to
$\ds x^2+4y^2=2z$ at
$(2,1,4)$.
\begin{sol}
${\bf r}(t)=\langle 2,1,4\rangle+t\langle 2,4,-1\rangle$
\end{sol}
\end{ex}

\begin{ex}
Explain in your own words why, when taking a partial derivative
of a function of multiple variables, we can treat the variables not
being differentiated as constants.
\end{ex}

\begin{ex}
Consider a differentiable function, $f(x,y)$.  Give physical
  interpretations of the meanings of $f_x(a,b)$ and $f_y(a,b)$ as they
  relate to the graph of $f$.
\end{ex}

\begin{ex}
In much the same way that we used the tangent line to
  approximate the value of a function from single variable calculus,
  we can use the tangent plane to approximate a function from
  multivariable calculus.  Consider the tangent plane found in
  Exercise~\ref{ex:ln tan plane}. Use this plane to approximate
  $f(1.98, 0.4)$.
\end{ex}

\begin{ex}
	The volume of a cylinder is given by $V=\pi r^2 h$.
	Suppose that the current values of $r$ and $h$ are $r=7$ cm and $h=3$ cm. Is
	the volume more sensitive to a small change in radius or the same amount of
	change in height? Why?
	\begin{sol}
		height
	\end{sol}
\end{ex}

\begin{ex}
Suppose that one of your colleagues has calculated the partial
  derivatives of a given function, and reported to you that
  $f_x(x,y)=2x+3y$ and that $f_y(x,y)=4x+6y$.  Do you believe them?
  Why or why not?  If not, what answer might you have accepted for
  $f_y$?
\end{ex}

\begin{ex}
Suppose $f(t)$ and $g(t)$ are single variable differentiable
  functions.  Find $\partial z/\partial x$ and
  $\partial z/\partial y$ for each of the following two variable functions.
\begin{enumerate}
	\item $z=f(x)g(y)$
	\item $z=f(xy)$
	\item $z=f(x/y)$
\end{enumerate}
\end{ex}

\end{enumialphparenastyle}


%!TEX root=ClassNotes.tex
\section{Limits \& Continuity}

We'll now use the proof techniques we've learned so far to study functions.

We'll start with rigorously defining limits. At first this might seem unnecessarily complicated, but having precise definitions will allow us to make more and more sophisticated constructions and later on in the course enable us to prove statements about derivatives and integrals.

Time permitting, towards the end of the semester, we'll construct the Weierstrass function, a function defined on real numbers which is continuous everywhere but differentiable nowhere! But it all starts with the definition of a limit.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.6\textwidth]{WeierstrassFunction.jpg}
	% \captionsetup{labelformat=empty}
	\caption*{Weierstrass function (Image from Wikipedia)}
\end{figure}

\subsection{Limits}
At the heart of all analysis (and hence calculus) lies the notion of a limit. Almost every single concept that we'll define will be a limit of some kind.

\begin{definition}[Provisional definition of limit]
	\label{def:provisional_definition_limit}
	For a function $f$ and a real number $a$, we say that the function $f$ \textbf{approaches a limit $L$ at $a$}, for some real number $L$, if we can make $f(x)$ as close to $L$ as we like by requiring that $x$ be sufficiently close to, but unequal to, $a$. This is denoted
	\begin{align*}
		\lim \limits _ {x \rightarrow a} f(x) = L
	\end{align*}
\end{definition}
\noindent There are 3 problems with this definition:
\begin{enumerate}
	\item The terms {\it ``as close to $L$ as we like''} and {\it``sufficiently close''} are not precise. (How close is sufficiently close?)
	\item The order of implication is unclear.
	\item The definition does not give us a way to come up with the limit $L$.
\end{enumerate}

First, we'll make the terms precise.  We can measure the distance between two real numbers $x$, $y$ by taking their difference $x - y$, however, the difference can be negative so we use the absolute value $|x - y|$ instead.  So that
\begin{center}
	\begin{tabular}{l l l}
		{\it $x$ sufficiently close to, but unequal to, $a$} & translates to & {\it $0 < |x - a| < \delta$}                            \\
		{\it $f(x)$ as close to $L$ as we like}              & translates to & {\it for every $\epsilon > 0$, $|f(x) - L| < \epsilon$}\end{tabular}
\end{center}
and the definition of limit becomes\\
\begin{indentPara}
	\dots $f$ \textbf{approaches a limit $L$ at $a$}, for some real number $L$, if we can make $|f(x) - L| < \epsilon$ for every $\epsilon > 0$ by requiring that $0 < |x - a | < \delta$ for some $\delta > 0$.\\
\end{indentPara}

Next, to clarify the order of implication we rephrase the statement in the more standard {\it ``If ... then''} form and put all the quantifiers in the front,\\
\begin{indentPara}
	\dots $f$ \textbf{approaches a limit $L$ at $a$}, for some real number $L$, if for every $\epsilon > 0$, there exists a $\delta > 0$ such that, for all $x$, if $0 < |x - a | < \delta$ then $|f(x) - L| < \epsilon$.\\
\end{indentPara}

The third problem cannot be fixed! There is no systematic way of finding the limit of a general function, the only thing we can do is {\it guess} the limit and then use the definition to {\it prove} that it is indeed the limit. For special functions like polynomials, trig functions, exponential, and logarithms, we can find limits explicitly, which is what makes these functions useful for approximating and estimating more complicated functions.

\begin{definition}[Formal definition of limit]
	\label{def:formal_definition_limit}
	We say that the function $f$ \textbf{approaches a limit $L$ at $a$}, for some real number $L$, if for every $\epsilon > 0$, there exists a $\delta > 0$ such that, for all $x$, if $0 < |x - a | < \delta$ then $|f(x) - L| < \epsilon$.
\end{definition}

\begin{exercise}
	Come up with formal definitions for the following:
	\begin{enumerate}
		\item The function $f$ approaches a limit $L$ at $a$ {\bf from the right}, for some real number $L$, if we can make $f(x)$ as close to $L$ as we like by requiring that $x$ be sufficiently close to, but strictly greater than $a$. This is denoted
		      \begin{align*}
			      \lim \limits _ {x \rightarrow a^+} f(x) = L.
		      \end{align*}
		\item The function $f$ approaches a limit $L$ at $a$ {\bf from the left}, for some real number $L$, if we can make $f(x)$ as close to $L$ as we like by requiring that $x$ be sufficiently close to, but strictly smaller than $a$. This is denoted
		      \begin{align*}
			      \lim \limits _ {x \rightarrow a^-} f(x) = L.
		      \end{align*}
		\item The function $f$ \textbf{approaches the limit $\infty$ at $a$}, if $f(x)$ can be made as large as we like by requiring that $x$ be sufficiently close to, but unequal to, $a$. This is denoted \footnote{
		In mathematics $\infty$ means several things, in fact, there are infinitely many infinities. For us, $\infty$ is a {\it placeholder} for a limit of a function that grows very large. $\infty$ is not a real number and hence cannot be used in equations.}
		      \begin{align*}
			      \lim \limits _ {x \rightarrow a} f(x) = \infty.
		      \end{align*}

		\item The function $f$ \textbf{approaches a limit $L$ at $\infty$}, for some real number $L$ if we can make $f(x)$ as close to $L$ as we like by requiring that $x$ be sufficiently large. This is denoted
		      \begin{align*}
			      \lim \limits _ {x \rightarrow \infty} f(x) = L.
		      \end{align*}
		% \item The function $f$ \textbf{does not approaches a limit $L$ at $a$}. Further generalize this to: the function $f$ does not approaches a limit $L$ at $a$, {\it for any $L$}. In this case, we say that the {\bf limit of $f$ at $a$ does not exist}.
	\end{enumerate}
\end{exercise}

\begin{exercise}
	For each of the following, first guess the limit, then use the formal definition of limit to prove that it is indeed the limit.
	\begin{enumerate}
		\item $\lim \limits_{x \rightarrow 1^+} 2x + 1$
		\item $\lim \limits_{x \rightarrow 1^+} x^2$
		\item $\lim \limits_{x \rightarrow \infty} x^{10} + x$
		\item $\lim \limits_{x \rightarrow 0^+} 1/x$
		% \item $\lim \limits_{x \rightarrow 0^+} \dfrac{\sin x}{x}\qquad$  (You can assume that $\sin x < x - \dfrac{x^3}{6}$ for all $0 < x < 1$.)
		\item $\lim \limits_{x \rightarrow 0} f(x)$ where $f(x) = \begin{cases}
				      x & \mbox{ if $x$ is rational}   \\
				      0 & \mbox{ if $x$ is irrational}
			      \end{cases}$
	\end{enumerate}
\end{exercise}

\subsubsection*{Optional Problems}

\begin{exercise} Give examples to show that the following definitions of $\lim \limits _ {x \rightarrow a} f(x) = L$ are not correct.
	\begin{enumerate}
		\item For every $ \delta > 0$, there exists an $ \epsilon > 0$ such that, for all $x$, if $ 0<|x-a|< \delta$ then $ |f(x) - L|< \epsilon$.
		\item For every $\epsilon > 0$, there exists a $\delta > 0$ such that, for all $x$, if $|f(x) - L| < \epsilon$ then $0 < |x - a| < \delta$.
	\end{enumerate}
\end{exercise}

\begin{exercise}
	Prove that if $\lim \limits _ {x \rightarrow a} f(x) = L$ and $ L \neq 0$ then $\lim \limits _ {x \rightarrow a} 1/f(x) = 1/L$ .
\end{exercise}

\subsubsection{Triangle Inequality}

To prove abstract theorems involving absolute values we will need the following very important inequality called the {\bf triangle inequality}.
\begin{align*}
	|x - y| \le |x| + |y|
\end{align*}
If you think of the points $0$, $x$, $y$ (on the real axis) as being the three vertices of a (degenerate) triangle, then $|x|$, $|y|$, and $|x - y|$ are the lengths of the three sides and the triangle inequality is saying that: {\it the sum of the lengths of two sides of a triangle is greater than or equal to the length of the third side.}\\\\
There are other forms in which the triangle inequality is commonly used, e.g.
\begin{align*}
	|x + y|       & \le |x| + |y|   \\
	|x + y| - |y| & \le |x|         \\
	|x|           & \le |x-y| + |y|
\end{align*}
These inequalities are central to a lot of analysis proofs.

\begin{exercise}$ $
	\begin{enumerate}
		\item Prove that $|x| + |x - 1| \ge 1$ for any real number $x$.
		\item Prove that for any real number $x$, at least one of $|x|$ and $|x-1|$ is $\ge 1/2$.\hint{Proof by Contradiction.}
	\end{enumerate}
\end{exercise}

We'll use these to prove inequalities about the non-existence of limits.

\begin{definition}
	If $f$ does not approach the limit $L$, for any real number or $\infty$ or $-\infty$, then we say that the limit of $f$ at $a$ {\bf does not exist}.
\end{definition}

\begin{exercise}
	\label{q:formal_definition_non_existence_limit}
	Negate the formal definitions of limits and come up with an $\epsilon, \delta$ definition for the following.
	\begin{enumerate}
		\item $f$ {\bf does not} approach the limit $L$, where $L$ is a real number, at $a$.
		\item $f$ {\bf does not} approach the limit $\infty$ at $a$. (Similarly for $-\infty$.)
	\end{enumerate}
\end{exercise}

\begin{exercise}
	\label{q:rational_irrational_discontinuity}
	Let $
		f(x) = \begin{cases}
			1 & \mbox{if $x$ is rational,}   \\
			0 & \mbox{if $x$ is irrational.}
		\end{cases}
	$
	\begin{enumerate}
		\item Prove that for every real number $L$, $\lim \limits_{x \rightarrow 0} f(x) \neq L$.
		\item Prove that $\lim \limits_{x \rightarrow 0} f(x) \neq \infty$. (Similarly for $-\infty$.)
	\end{enumerate}
	Hence the limit of $f$ at $0$ does not exist.
\end{exercise}

\begin{exercise}$ $
	\begin{enumerate}
		\item Prove that $\lim \limits_{x \rightarrow a} f(x) = L$ if and only if $\lim \limits_{x \rightarrow a^+} f(x) = L$ and $\lim \limits_{x \rightarrow a^-} f(x) = L$.\hint{This is very easy, don't overthink! Simply write down the formal definitions.}
		      (Similarly for $\infty$, $-\infty$.)

		\item Prove that $\lim \limits_{x \rightarrow 0} \dfrac{1}{x}$ does not exist.
	\end{enumerate}
\end{exercise}

\subsection{Non-existence of limits}

Let's move backwards and try and understand what it means {\it intuitively} for a function to not approach a limit $L$ near $a$. The solution to Exercise \ref{q:formal_definition_non_existence_limit} is the following:\\

\begin{indentPara}
	{\it (Formal definition)} The function $f$ \textbf{does not approach thelimit $L$ at $a$} if there exists an $\epsilon > 0$, such that for all $\delta > 0$, there exists an $x$ such that, $0 < |x - a | < \delta$ and $|f(x) - L| \ge \epsilon$.\\
\end{indentPara}

Going back to how to we came up with the Formal Definition of Limit 	\ref{def:formal_definition_limit}
from the Provisional Definition of Limit \ref{def:provisional_definition_limit} and replacing the absolute values by the distance between points, we can work backwards and get the following provisional definition:\\

\begin{indentPara}
	{\it (Provisional definition)}
	The function $f$ \textbf{does not approach the limit $L$ at $a$} if no matter how close we are to $a$, there is some $x$ for which $f(x)$ is not close to $L$.
\end{indentPara}

% \subsubsection*{Optional Problems}
% \begin{exercise} $ $
% 	\begin{enumerate}
% 		\item Explain how the provisional definition captures the notion of the failure of the function to approach a limit $L$ at $a$.
% 		\item Starting from the provisional definition try to systematically come up with the formal definition using arguments similar to the ones we made to go from Definition  \ref{def:provisional_definition_limit} to Definition \ref{def:formal_definition_limit}.
% 	\end{enumerate}
% \end{exercise}
%
% \begin{exercise}
% 	It is easy to show that for the function
% 	\begin{align*}
% 		f(x) = \begin{cases}
% 			0 & \mbox{if } x \le 0 \\
% 			1 & \mbox{if } x > 0
% 		\end{cases}
% 	\end{align*}
% 	the limit $\lim \limits_{x \rightarrow 0} f(x)$ does not exist by computing the limits from the left and the right.
%
% 	Try to prove that, $\lim \limits_{x \rightarrow 0} f(x) \neq L$ for some real numbers, say $L = 0$, 1, 1/2, and more generally for any real number, using the formal definition and try to explain what the variables $\epsilon$, $\delta$, $x$ in your proof mean {\it geometrically}.
% \end{exercise}

\subsection{Continuity}

\begin{definition}
  We say that a function $f$ is {\bf continuous} at $a$ if
  \begin{align*}
    \lim \limits_{x \rightarrow a} f(x) = f(a)
  \end{align*}
  We say that a function $f$ is {\bf continuous on an interval} $(a,b)$ it $f$ is continuous at every $x$ in $(a,b)$. \footnote{If we use closed intervals $[a,b]$ instead, then we have to use $\lim \limits_{x \rightarrow a^+}$ and $\lim \limits_{x \rightarrow b^-}$ in the definition of continuity.}
\end{definition}

Thus continuous functions have {nice} limits. Further, we have the following theorem which makes continuous functions easy to manipulate.

\begin{theorem}
  \label{theorem:continuous_functions}
  If the functions $f$, $g$ are continuous at $x=a$ then so are the functions
  \begin{enumerate}
    \item $ f + g$
    \item $f \cdot g$
    \item $f / g$, if $g(a) \neq 0$
  \end{enumerate}
\end{theorem}

\noindent The full proof of this theorem is just a tricky application of the triangle inequalities. We'll only prove the first part which is relatively manageable.

\begin{exercise}
  \label{q:for_later_1}
  Using the formal definition of limits and the triangle inequality $|x+y| \le |x| + |y|$, prove that
  \begin{indentPara}
    {\it if the functions $f$, $g$ are continuous at $x=a$ then so is the function $f + g$.}
  \end{indentPara}
  Is the converse true?
\end{exercise}

\begin{exercise}$ $
  \label{q:for_later_2}
  \begin{enumerate}
    \item Prove that the constant function $f(x) = c$ is continuous everywhere, where $c$ is some real number.
    \item  Prove that the function $f(x) = x$ is continuous everywhere.
    \item Argue that these two facts along with Theorem \ref{theorem:continuous_functions} imply that polynomials are continuous everywhere.
  \end{enumerate}
\end{exercise}

\begin{exercise}
  \label{q:for_later_3}
  Let $f$ be a continuous function on $[a,b]$, and let $g$ be a continuous function on $[b,c]$, such that \begin{align*}
    f(b) = g(b).
  \end{align*}
  Show that the function $h$ defined as
  \begin{align*}
    h(x) :=
    \begin{cases}
      f(x) & \mbox{if } x \le b \\
      g(x) & \mbox{if } x > b
    \end{cases}
  \end{align*}
  is continuous at $b$ (i.e. we can {\it glue} continuous functions).
\end{exercise}


\section{Diagonalization}

There are two questions for a linear operator $T$:
\begin{enumerate}
    \item Is there an ordered basis $\beta$ that $\coordinate{T}_\beta$ is a diagonal matrix?
    \item If such basis exists, how can it be found?
\end{enumerate}

\subsection{Eigenvalue and Eigenvectors}

\begin{definition}
    A linear operator $T$ on $V$ is \cindex{diagonalizable} if there is an ordered basis $\beta$ of $V$ that $\coordinate{T}_\beta$ is a diagonal matrix. A matrix is \cindex{diagonalizable} if $L_A$ is diagonalizable.

If an operator $T$ is diagonalizable, for $\beta = \{v_i\}$, we have
\begin{equation*}
    T(v_j) = \sum_{i=1}^n D_{ij} v_j = D_{jj} v_j = \lambda_j v_j
\end{equation*}
So to prove a  linear operator $T$ is diagnolizable is to find a basis $\beta = \{ v_i \}$ and $\{ \lambda_j \}$ that $T(v_i) = \lambda_i v_i$.
\qed
\end{definition}

\begin{definition}
    A \emph{non-zero} vector $v \in V$ is called an \cindex{eigenvector} of linear operator $T$ if $\exists \lambda : T(v) = \lambda v$. $\lambda$ is called \cindex{eigenvalue} corresponding to eigenvector $v$. Eigenvector is also called \cindex{characteristic vector}. Eigenvalue is also called \cindex{characteristic value}.

    A eigenvalue could be $0$, but eigenvector could not be $\vec{0}$. An eigenvector is an invariant subspace of dimension $1$.
\end{definition}

\begin{theorem}
    A linear operator $T$ is diagonalizable if there exists an ordered basis consisting of eigenvectors of $T$.
\end{theorem}

\begin{theorem}
    $\lambda$ is an eigenvalue of $A$ $\iff$ $\determinate{A - \lambda I_n} = 0$.
\end{theorem}

\begin{proof}
    If $\lambda$ is an eigenvalue of $A$, $\exists v \in F^n, v \neq 0$ that $A v = \lambda v$, which is $(A - \lambda I_n)(v)= 0$, which means $A - \lambda I_n$ is not invertible because $v \neq 0$, so $\determinate{A - \lambda I_n} = 0$.
\end{proof}

\begin{theorem}
    Every eigenvalue has at least one eigenvector.
\end{theorem}
\begin{proof}
    Since $\determinate{A - \lambda I_n} = 0$, $(A - \lambda I_n) x = 0$ is a homogeneous equation with $\dimension{A - \lambda I_n} < n$.
\end{proof}

\begin{definition}
    For $A = \coordinate{T}_\beta$ the polynomial $f_A(t) = \determinate{A - t I_n}$ is called the \cindex{characteristic polynomial} of $A$ and $T$.
\end{definition}

\begin{theorem}
    For all eigenvalues $\lambda_i$ of $A$, define
    \begin{equation}
        S_k(A) = \sum_{1\leq j_1 \leq j_2 \leq \dots \leq j_k} \prod_{j=1}^k \lambda_{i_j}
    \end{equation}
    that is $S_k(A)$ is the sum of the product of all $k$ eigenvalues, which is the coefficient of characteristic polynomial of $f_A(t)$:
    \begin{equation}
        f_A(t) = (-1)^n t^n + (-1)^{n-1} S_{1}(\lambda)t^{n-1} + \dots + (-1)^{n-k} S_{k} t^{n-1} + \dots + S_{n}
    \end{equation}
    Define the sum of all\footnote{There are $\binom{n}{k}$ of them.} principal minor of size $k$ of $A$ as $E_k(A)$. We have
    \begin{equation}
        E_k(A) = S_k(A)
    \end{equation}
    So
    \begin{equation}
        \text{tr} A = \sum \lambda_i
    \end{equation}
    and
    \begin{equation}
        \determinate{A} = \prod \lambda_i
    \end{equation}
\end{theorem}
\begin{proof}
    calculate the coefficient by $\dfrac{1}{k!} \eval{\dod[k]{f_A(t)}{t}}_{t = 0}$
\end{proof}

\begin{theorem}
    The choice of basis $\beta$ did not change the eigenvalue of $T$.
\end{theorem}
\begin{proof}
    \begin{equation*}
        \absolutevalue{\coordinate{T}_\beta - \lambda I} = \absolutevalue{Q^{-1} \left( \coordinate{T}_\alpha - \lambda I \right) Q} = \absolutevalue{Q^{-1}} \times \absolutevalue{\coordinate{T}_\alpha - \lambda I } \times \absolutevalue{Q} = \absolutevalue{\coordinate{T}_\alpha - \lambda I }
    \end{equation*}
\end{proof}

\begin{theorem}
Similar matrices have the same characteristic function.
\end{theorem}
\begin{proof}
    Assume $A$ is similar to $B$: $A = P^{-1} B P$. We have
    \begin{equation*}
    f_A(\lambda) = \determinate{Ax - \lambda I} = \determinate{P^{-1} B P - \lambda P^{-1} P} = \determinate{P^{-1}} \times \determinate{B - \lambda I } \times \determinate{P} = \determinate{B - \lambda I } = f_B(\lambda)
    \end{equation*}
\end{proof}

\begin{theorem}
    if $Q$ is a matrix with columns of eigenvectors of $\beta$, then according to \thmref{specialchangeofcoordinates} , $Q^{-1} A Q$ is a diagonal matrix with eigenvalue.
\end{theorem}

\subsection{Diagonalizability}

\begin{theorem}
    Let ${\lambda_i}$ be distinct eigenvalue of $T$. If $\set{v_i}$ are eigenvector that corresponding to $\lambda_i$, then $\set{v_i}$ is \emph{linearly independent}.
\end{theorem}
\begin{proof}
    suppose it works for $k - 1 \geq 1$ and we have $k$ eigenvector $\{ v_i\}$. Suppose
    \begin{equation*}
        a_1 v_1 + a_2 v_2 + \dots + a_k v_k = 0
    \end{equation*}

    multiply $T - \lambda_k I$ to both sides, we have
    \begin{equation*}
        a_1(\lambda_1 - \lambda_k) v_1 + a_1(\lambda_2 - \lambda_k) v_2 +  \dots + a_1(\lambda_{k-1} - \lambda_k) v_{k-1} +  = 0
    \end{equation*}

    because $\set{v_1, v_2, \dots, v_{k-1} }$ are linearly independent, we have
    \begin{equation*}
        a_1(\lambda_1 - \lambda_k) = a_1(\lambda_2 - \lambda_k) =  a_1(\lambda_{k-1} - \lambda_k) = 0
    \end{equation*}

    because $\lambda_i$ are different, we have $a_i = 0$.
\end{proof}

\begin{theorem}
    if $T$ has $n$ distinct eigenvalues, then $T$ is diagonalizable. If $T$ is diagonalizable, it may not have $n$ distinct eigenvalues, for example the identity matrix $I_V$.
\end{theorem}

\begin{definition}
    A polynomial $f(t)$ in $P(F)$ \cindex{split over} $F$ if there are scalars $c, a_1, \dots, a_n$ (not necessarily distinct) in $F$ that
    \begin{equation*}
        f(t) = c(t - a_1)(t - a_2) \dots (t-a_n)
    \end{equation*}

    the \cindex{multiplicity} of $\lambda$ is the largest positive integer $k$ for which $(t - \lambda)^k$ is a factor of $f(t)$.
\end{definition}

\begin{theorem}
    the characteristic polynomial of any diagonalizable linear operator splits.
\end{theorem}

\begin{proof}
    choose a basis $\beta$ of eigenvectors. $[\mathrm{T}]_\beta$ is a diagonal matrix $D$. The characteristic polynomial of $T$ is $|D - tI|$ splits.
\end{proof}

Be careful that the characteristic polynomial splits does not mean the matrix is diagonalizable. The eigenvectors need to form a basis.

\begin{definition}
    let $\lambda$ be an eigenvalue of $T$. Let $E_\lambda = \nullspace{T - \lambda I_V}$. the set $E_\lambda$ is called the \cindex{eigenspace} of $T$ corresponding to eigenvalue $\lambda$. So is it for matrix.
\end{definition}

\begin{theorem}
    let $\lambda$ be an eigenvalue of $T$ having multiplicity $m$. then $1 \leq \dimension{E_\lambda} \leq m$.
\end{theorem}
\begin{proof}
    choose ordered basis $\set{v_1, v_2, \dots, v_p}$ for $E_\lambda$, and extend it to ordered basis $\beta =\set{v_1, v_2, \dots, v_p, v_{p+1}, \dots, v_n}$ for $V$, and let $A = \coordinate{T}_\beta$. let $v_i (1 \leq i \leq q)$ be an eigenvector of $T$ corresponding to $\lambda$, we have
    \begin{equation*}
        A = \begin{pmatrix}
            \lambda I_p & B \\
            0 & C
        \end{pmatrix}
    \end{equation*}
    so \begin{equation*}
        \begin{aligned}
            f(t) &= \absolutevalue{A - t I_n} \\
            &= \absolutevalue{\begin{bmatrix}
                (\lambda - t) I_p & B \\
                0 & C - t I_{n-p}
            \end{bmatrix}} \\
            &= \absolutevalue{(\lambda - t)I_p} \times \absolutevalue{C - t I_{n-p}} \\
            &= (\lambda - t)^p g(t)
        \end{aligned}
        \end{equation*}
    So $(\lambda - t)^p$ is a factor of $f(t)$, and the multiplicity of $\lambda$ is at least $p = \text{dim}(E_\lambda)$, so $\text{dim}(E_\lambda) \leq m$
\end{proof}

\begin{theorem}
    let $\set{\lambda_1, \lambda_2, \dots, \lambda_k}$ be distinct eigenvalue of $T$. let $S_i$ be a finite linearly independent subset of eigenspace $E_{\lambda_i}$. then $S_1 \cup S_2 \cup \dots \cup S_k$ is a linearly independent subset of $V$.
\end{theorem}

\begin{theorem}
    let $\lambda_1, \lambda_2, \dots, \lambda_k$ be distinct eigenvalue of $T$, then
    \begin{enumerate}
        \item $T$ is diagonalizable $\iff$ the multiplicity of $\lambda_i$ is equal to $\dimension{E_{\lambda_i}}$ for all $i$.
        \item If $T$ is diagonalizable and $\beta_i$ is an ordered basis for $E_{\lambda_i}$ for each $i$, then $\beta = \beta_1 \cup \beta_2 \cup \dots \cup \beta_k$ is an ordered basis for $V$ consisting of eigenvectors of $T$.
    \end{enumerate}
\end{theorem}

\begin{theorem}
    $T$ is diagonalizable $\iff$ both of the following holds:
    \begin{enumerate}
        \item the characteristic polynomial of $T$ splits.
        \item for each eigenvalue $\lambda$ of $T$, the multiplicity of $\lambda$ equals $n - \rank{T - \lambda I}$.
    \end{enumerate}
\end{theorem}

\begin{definition}
    Let $W_i$ be subspaces of a vector space $V$. The \cindex{sum} of these subspaces is defined as:
    \begin{equation}
        \sum_{i=1}^k W_i = \set{v_1 + v_2 + \dots + v_k : v_i \in W_i \text{ for } 1 \leq i \leq k }
    \end{equation}
\end{definition}

\begin{definition}
    let $W_i$ be subspace of $V$. $V$ is the \cindex{direct sum} of subspace $\set{W_1, W_2, \dots, W_k}$, or $V = W_1 \oplus W_2 \oplus \dots \oplus W_k$ if
    \begin{equation*}
        V = \sum_{i=1}^k W_i
    \end{equation*}
    and
    \begin{equation*}
        W_j \cap \sum_{i \neq j} W_i = \emptyset, (1 \leq j \leq k)
    \end{equation*}
\end{definition}

\begin{theorem}
    $T$ is diagonalizable $\iff$ $V$ is the direct sum of eigenspaces of $T$.
\end{theorem}

\subsection{Invariant Subspaces}

\begin{definition}
    A subspace $W$ of $V$ is $T$-\cindex{invariant subspace} of $V$ if $T(W) \subseteq W$.
    Common $T$-invariant subspaces are: $\emptyset$, $V$, $R(T)$, $N(T)$.
    \qed
\end{definition}

\begin{theorem}
    A subspace $W$ with basis $\alpha = \set{v_1, v_2, \dots, v_k}$ is $T$-invariant. Let $\beta = \alpha \cup \gamma$ as the expanded basis of $V$. Then
    \begin{equation}
        \coordinate{T}_\beta = \begin{bmatrix}
            A_{k \times k} & B \\
            0 & C
        \end{bmatrix}
    \end{equation}
    The reverse is true. If $\coordinate{T}_\beta$  has such representation, the first $k$ basis of $\beta$ is $T$-invariant.
\end{theorem}

\begin{definition}
    A $T$-\cindex{cyclic subspace} of $V$ generated by $x$ is defined as $W=\text{span} \left(  \set{x, T(x), T^2(x), \dots} \right)$.
\end{definition}

\begin{theorem}
    Let $T$ be a linear operator on finite-dimensional vector space $V$, and let $W$ be a $T$-invariant subspace of $V$. Then the characteristic polynomial of $T_W$ divides the characteristic polynomial of $T$.
\end{theorem}

\begin{proof}
    Choose ordered basis $\gamma$ for $W$ and expand it to $\beta$ for $V$. Calculate $\coordinate{T}_\beta$ and $\coordinate{T}_\gamma$.
\end{proof}

\begin{theorem}
    Let $T$ be a linear operator on finiate-dimensional vector space $V$, and let $W$ be a $T$-cyclic subspace of $V$ generated by nonzero vector $v \in V$. Let $k = \dimension{W}$. Then:
    \begin{enumerate}
        \item $\set{v, T(v), T^2(v), \dots, T^{k-1}(v)}$ is a basis for $W$.
        \item If $a_0 v + a_1 T(v) + a_2 T^2(v) + \dots + a_{k-1} T^{k-1}(v) + T^k(v) = 0$, then the characteristic polynomial of $T_W$ is $f(t) = (-1)^k \left( a_0 + a_1 t + \dots + a_{k-1} t^{k-1} + t^k \right)$.
    \end{enumerate}
\end{theorem}

\begin{proof}
    Let $\beta = \set{v, T(v), T^2(v), \dots, T^{k-1}(v)}$, and let $a_i$ be the scalars that
    \begin{equation*}
            a_0 v + a_1 T(v) + a_2 T^2(v) + \dots + a_{k-1} T^{k-1}(v) + T^k(v) = 0
    \end{equation*}

    Fors basis $\set{v, T(v), T^2(v), \dots, T^{k-1}(v)}$, $\coordinate{T(v)}_\beta = \coordinate{0,1,\dots, 0}$, $T\left( T(v)\right)_\beta = \coordinate{0,0,1,\dots, 0}$, etc, we have:
    \begin{equation*}
        [T_W]_\beta = \begin{bmatrix}
            0 & 0 & \dots & 0 & - a_0 \\
            1 & 0 & \dots & 0 & -a_1\\
            \vdots & \vdots  & & \vdots & \vdots \\
            0 & 0 & \dots & 1 & - a_{k-1}
        \end{bmatrix}
    \end{equation*}
    which has characteristic polynomial
    \begin{equation*}
        f(t) = (-1)^k (a_0 + a_1 t + \dots + a_{k-1} t^{k-1} + t^k)
    \end{equation*}
\end{proof}

\begin{theorem}[\cindex{Cayley-Hamilton}]
    Let $T$ be linear operator on a finite-dimensional vector space $V$, and let $f(t)$ be the characteristic polynomial of $T$. Then $f(T) = 0$.
\end{theorem}

\begin{proof}
    Suppose $v \neq 0$. Let $W$ be the $T$-cyclic subspace generated by $v$, and suppose the $\dimension{W} =k$. So there exists scalars $\set{a_i}$ that
    \begin{equation*}
        a_0 v + a_1 T(v) + a_2 T^2(v) + \dots + a_{k-1} T^{k-1}(v) + T^k(v) = 0
    \end{equation*}
    which implies the characteristic polynomial of $T_W$ is
    \begin{equation*}
        g(t) = (-1)^k \left(a_0 + a_1 t + \dots + a_{k-1} t^{k-1} + t^k \right)
    \end{equation*}
    We have
    \begin{equation*}
        g(T)(v) = (-1)^k \left(a_0 I + a_1 T + \dots + a_{k-1} T^{k-1} + T^k \right)(v) = 0
    \end{equation*}
    Because $g(t)$ divides $f(t)$, $\exists q(t)$ that $f(t) = g(t) q(t)$. So
    \begin{equation*}
        f(T)(v) = q(T)g(T)(v) = q(T) \left(g(T)(v)\right) = q(T)(0) = 0
    \end{equation*}
\end{proof}

\begin{definition}
    Let $B_1 \in M_{m \times m}(F)$, and $B_2 \in M_{n \times n} (F)$. The \cindex{direct sum} of $B_1$ and $B_2$, denoted as $B_1 \oplus B_2$, as the $(m+n) \times (m+n)$ matrix $A$ that
    \begin{equation*}
        A = \begin{bmatrix}
            B_1 & 0 \\
            0 & B_2
        \end{bmatrix}
    \end{equation*}
\end{definition}

\begin{theorem}
    Suppose $V=W_1 \oplus W_2 \oplus \dots \oplus W_k$, where $W_i$ is a $T$-invariant subspace of $V$. Suppose $f_i(t)$ is the characteristic polynomial of $T_{W_i}$, Then $\displaystyle \prod_{i=1}^k f_i$ is the characteristic polynomial of $T$. Let $\beta_i$ be an ordered basis for $W_i$, and let $\displaystyle \beta = \bigcup_{i=1}^k \beta_i$. Let $A=\coordinate{T}_\beta$, and $B_i=[T_{W_i}]_\beta$. Then $A = B_1 \oplus B_2 \oplus \dots \oplus B_k$.
\end{theorem}

\subsection{Limit of Markov Chain Matrix}

\begin{definition}
    A sequence $\set{A_1, A_2, \dots}$ \cindex{converge} to \cindex{limit} $L$ if $\displaystyle \lim_{m \rightarrow \infty} (A_m)_{ij} = L_{ij}$.
\end{definition}

\begin{theorem}
    If $A_i \rightarrow L$, them for any $P$ and $Q$, $\displaystyle \lim_{m \rightarrow \infty} P A_m = PL$ and $\displaystyle \lim_{m \rightarrow \infty} A_m Q = LQ$.
\end{theorem}

\begin{theorem}
    Let $Q$ be invertible and $A_i \rightarrow L$. Then $\displaystyle \lim_{m \rightarrow \infty} (Q A Q^{-1})^m = Q A Q^{-1}$.
\end{theorem}

\begin{definition}
    Define a set $S$ which consists of the interior of unit disk and $1$:
    \begin{equation}
        S = \set{\lambda \in C :  \absolutevalue{\lambda} < 1   \vee \lambda = 1}
    \end{equation}
\end{definition}

\begin{theorem}
    Let $A$ be square matrix in $C$. $\displaystyle \lim_{m \rightarrow \infty} A^m$ exists if and only if:
    \begin{enumerate}
        \item Every eigenvalue of $A$ is in $S$.
        \item If $1$ is an eigenvalue of $A$, then the dimension of its eigenspace equals its multiplicity.
    \end{enumerate}
\end{theorem}
\begin{proof}
    use Jordan canonical form.
\end{proof}

\begin{theorem}
    For square matrix $A$ in $C$, if
    \begin{enumerate}
        \item Every eigenvalue of $A$ is in $S$.
        \item $A$ is diagonalizable.
    \end{enumerate}
    Then $\displaystyle \lim_{m \rightarrow \infty} A^m$ exists.
\end{theorem}
\begin{proof}
    Since $A$ is diagonalizable, $\exists Q: A = Q D Q^{-1}$. So $A^m = Q D^m Q^{-1}$. This is used to calculate $A^m$.
\end{proof}

\begin{definition}
    \cindex{transition matrix} or \cindex{stochastic matrix} is a square matrix $A$ that $A_{ij} \geq 0 \wedge \forall j \left(\sum_{i} A_{ij} = 1 \right)$.
\end{definition}

\begin{definition}
    $P$ is a \cindex{probability vector} if its entries are all non-negative and sum to $1$.
\end{definition}

\begin{definition}
    \cindex{$\vec{1_n}$} is a column vector that each coordinate is $1$.
\end{definition}

\begin{theorem}
    Let $M$ be a square matrix with non-negative real entries, and $v$ a column vector with real non-negative coordinates. Then
    \begin{enumerate}
        \item $M$ is a transition matrix if and only if $M^\top \vec{1_n} = \vec{1_n}$.
        \item $v$ is a probability vector if and only if $\vec{1_n}^\top v = 1$.
        \item The product of two transition matrix is transition matrix.
        \item The product of a transition matrix and probability vector is a probability vector.
    \end{enumerate}
\end{theorem}

\begin{definition}
    A transition matrix is \cindex{regular} if some power of the matrix contains only positive entries. It may contain zero entries.
\end{definition}

\begin{definition}
    For square matrix $A$, define $\displaystyle \rho_i (A) = \sum_j \absolutevalue{A_{ij}}$ and $\displaystyle v_j(A) = \sum_i \absolutevalue{A_{ij}}$. The \cindex{row sum} $\rho (A) = \max \rho_i$ and \cindex{column sum} $v(A) = \max v_j$.
\end{definition}

\begin{definition}
    For square matrix $A_{n \times n}$, the \cindex{Gerschgorin disk} $C_i$ is defined as:
    \begin{equation}
        C_i = \set{z \in C: \absolutevalue{z - A_{ii}} < \rho_i (A) - \absolutevalue{A_{ii}}}
    \end{equation}
    So the disk center is the diagonal entry, and the radius is the sum of the absolute values of all rest row entries.
\end{definition}

\begin{theorem}
    Every eigenvalue of $A$ is contained in a Gerschgorin disk.
\end{theorem}
\begin{proof}
    Let $\lambda$ be a eigenvalue with eigenvector $v$. So $\displaystyle \sum_{j=1}^n A_{ij} v_j = \lambda v_i$. Assume $v_k$ is the coordinate of $v$ that has the largest absolute value. Then $v_k \neq 0$ because $v \neq 0$. We have
    \begin{equation*}
        \begin{aligned}
            \absolutevalue{\lambda v_k - A_{kk} v_k} = \absolutevalue{\sum_{j=1}^n A_kj v_j - A_{kk} v_k} = \absolutevalue{\sum_{j \neq k} A_{kj} v_j} \leq \sum_{j \neq k} \absolutevalue{A_{kj}} \absolutevalue{v_j} \leq \sum_{j \neq k} \absolutevalue{A_{kj}} \absolutevalue{v_k} = \absolutevalue{v_k} \left(\rho_i (A) - \absolutevalue{A_{kk}} \right)
        \end{aligned}
    \end{equation*}
    So $\absolutevalue{v_k} \times \absolutevalue{\lambda - A_{kk} } \leq \absolutevalue{v_k} \left(\rho_i (A) - \absolutevalue{A_{kk}} \right)$ and $\absolutevalue{\lambda - A_{kk} } \leq \left(\rho_i (A) - \absolutevalue{A_{kk}} \right)$.
\end{proof}

\begin{theorem}
    Let $\lambda$ be any eigenvalue of $A$. Then $\absolutevalue{\lambda} \leq \rho(A)$.
\end{theorem}
\begin{proof}
    $\displaystyle \absolutevalue{\lambda} = \absolutevalue{(\lambda - A_{kk}) + A_{kk}} \leq \absolutevalue{\lambda - A_{kk}} + \absolutevalue{A_{kk}} \leq \rho_i (A) - \absolutevalue{A_{kk}}  + \absolutevalue{A_{kk}}  = \rho_i (A)$
\end{proof}

\begin{theorem}
    Let $\lambda$ be any eigenvalue of $A$. Then $\absolutevalue{\lambda} \leq \min \set{\rho(A), v(A)}$.
\end{theorem}
\begin{proof}
    $\lambda$ is an eigenvalue of $A^\top$.
\end{proof}

\begin{theorem}
    If $\lambda$ is an eigenvalue of transition matrix, then $\absolutevalue{\lambda} \leq 1$.
\end{theorem}

\begin{theorem}
    Every transition matrix has $1$ as eigenvalue.
\end{theorem}
\begin{proof}
    $A^\top \times  \vec{1_n} = \vec{1_n}$.
\end{proof}

\begin{theorem}
    Let $A$ be a matrix with positive entries, and let $\lambda$ be an eigenvalue of $A$ that $\absolutevalue{\lambda} = \rho(A)$. Then $\lambda = \rho(A)$ and $\vec{1_n}$ is a basis for $E_\lambda$.
\end{theorem}
\begin{proof}
    Let $v$ be an eigenvector for $\lambda$, and $v_k$ is the coordinate that has the largest absolute value $b = \absolutevalue{v_k}$. Then
    \begin{equation*}
        \absolutevalue{\lambda} b = \absolutevalue{\lambda v_k} = \absolutevalue{\sum_{j=1}^n A_{kj} v_j} \leq \sum_{j=1}^n \absolutevalue{A_{kj} v_j} = \sum_{j=1}^n \absolutevalue{A_{kj}} \absolutevalue{v_j} \leq \sum_{j=1}^n \absolutevalue{A_{kj}} b = \rho_k(A) b \leq \rho(A) b
    \end{equation*}
    Since $\absolutevalue{\lambda} = \rho(A)$, all inequalities are equalities, so
    \begin{enumerate}
        \item \label{transitionmatrixproperty1}$\displaystyle \absolutevalue{\sum_{j=1}^n A_{kj} v_j} = \sum_{j=1}^n \absolutevalue{A_{kj} v_j}$
        \item \label{transitionmatrixproperty2}$\displaystyle \absolutevalue{A_{kj}} \absolutevalue{v_j} = \sum_{j=1}^n \absolutevalue{A_{kj}} b$
        \item \label{transitionmatrixproperty3}$\rho_k(A) \leq \rho(A)$
    \end{enumerate}

    For Item \ref{transitionmatrixproperty1} to hold, $A_{kj} v_j$ are non-negative multiplies of a common complex number $z$. Assume $\absolutevalue{z}=1$. Then $\left(\exists \set{c_j} \subset R^+ \right) (A_{kj} v_j = c_j z)$.

    For item \ref{transitionmatrixproperty2}, since $b = \max \absolutevalue{v_j}$, $\absolutevalue{v_j} = b$. So $\displaystyle b = \absolutevalue{v_j} = \absolutevalue{\frac{c_j}{A_{kj}} z} = \frac{c_j}{A_{kj}}$, and $\displaystyle v_j = \frac{c_j}{A_{kj}} z = bz$, and $v = bz \vec{1_n}$.

    Since $A$ and $\vec{1_n}$ are all positive, $A \vec{1_n} = \lambda \vec{1_n}$, so $\lambda > 0$.
\end{proof}

\begin{theorem}
    Let $A$ be a transition matrix that each entry is positive, and let $\lambda$ be any eigenvalue of $A$ other than $1$. Then $\absolutevalue{\lambda} < 1$. Moreover, the eigenspace of eigenvalue $1$ has dimension $1$.
\end{theorem}

\begin{theorem}
    Let $A$ be a regular transition matrix, and $\lambda$ be one of its eigenvalue, then
    \begin{enumerate}
        \item $\absolutevalue{\lambda} \leq 1$.
        \item If $\absolutevalue{\lambda} = 1$, then $\lambda = 1$ and $\dimension{E_\lambda} = 1$.
    \end{enumerate}
\end{theorem}

\begin{theorem}
    Let $A$ be a disagonalizable regular transition matrix, then $\displaystyle \lim_{m \rightarrow \infty} A^m$ exists.
\end{theorem}

\begin{theorem}
    Let $A$ be a regular transition matrix, then
    \begin{enumerate}
        \item the multiplicity of eigenvalue $1$ is $1$.
        \item $\displaystyle \lim_{m \rightarrow \infty} A^m$ exists.
        \item $L = \displaystyle \lim_{m \rightarrow \infty} A^m$ is a transition matrix.
        \item $AL = LA = L$.
        \item The column of $L$ are identical vector $v$ which is the probability vector in $E_1$.
        \item For any probability vector $w$, $\displaystyle \lim_{m \rightarrow \infty} A^m w = v$.
    \end{enumerate}
\end{theorem}
\begin{proof}
    Since $AL = L$, $L$ are columns of eigenvector for eigenvalue $1$. Let $y = \displaystyle \lim_{m \rightarrow \infty} A^m w = Lw$, $Ay = ALw = Lw = y$. So $y$ is an eigenvector for eigenvalue $1$, and $y = v$.
\end{proof}


\section{Introducing differentiable binary arithmetic operations}
\label{sec:Nalu}
We define our problem as learning a set of static arithmetic operations between selected elements of a vector. E.g. for a vector $\mathbf{x}$ learn the function ${(x_5 + x_1) \cdot x_7}$. The approach taking in this paper, is to develop layers around specific operations, and then let each layer decide which inputs to include using backpropagation.

We develop these layers by taking inspiration from a theoretical analysis of Neural Arithmetic Logic Unit (NALU) by \citet{trask-nalu}.

\subsection{Introducing NALU}
The Neural Arithmetic Logic Unit (NALU) consists of two sub-units; the $\text{NAC}_{+}$ and $\text{NAC}_{\bullet}$. The sub-units represent either the $\{+, -\}$ or the $\{\times, \div \}$ operations. The NALU then assumes that either $\text{NAC}_{+}$ or $\text{NAC}_{\bullet}$ will be selected exclusively, using a sigmoid gating-mechanism.

The $\text{NAC}_{+}$ and $\text{NAC}_{\bullet}$ are defined accordingly,
\begin{align}
W_{h_\ell, h_{\ell-1}} &= \tanh(\hat{W}_{h_\ell, h_{\ell-1}}) \sigma(\hat{M}_{h_\ell, h_{\ell-1}}) \label{eq:weight}\\
\textrm{NAC}_+:\ z_{h_\ell} &= \sum_{h_{\ell-1}=1}^{H_{\ell-1}} W_{h_{\ell}, h_{\ell-1}} z_{h_{\ell-1}} \label{eq:naca}\\
\textrm{NAC}_\bullet:\ z_{h_\ell} &= \exp\left(\sum_{h_{\ell-1}=1}^{H_{\ell-1}} W_{h_{\ell}, h_{\ell-1}} \label{eq:nacm}\log(|z_{h_{\ell-1}}| + \epsilon) \right)
\end{align}
where $\hat{\mathbf{W}}, \hat{\mathbf{M}} \in \mathbb{R}^{H_{\ell} \times H_{\ell-1}}$ are weight matrices and $z_{h_{\ell-1}}$ is the input. The matrices are combined using a tanh-sigmoid transformation to bias the parameters towards a $\{-1,0,1\}$ solution. Having $\{-1,0,1\}$ allows $\text{NAC}_{+}$ to perform exact $\{+, -\}$ operations between elements of a vector.
The $\text{NAC}_{\bullet}$ uses an exponential-log transformation to create the $\{\times, \div \}$ operations within $\epsilon$ precision and for positive inputs only.

The NALU combines these units with a gating mechanism $\mathbf{z} = \mathbf{g} \odot \text{NAC}_{+} + (1 - \mathbf{g}) \odot \text{NAC}_{\bullet}$ given $\mathbf{g} = \sigma(\mathbf{G} \mathbf{x})$. Thus allowing NALU to decide between all of $\{+, -, \times, \div\}$ using backpropagation.

\subsection{Weight matrix construction  and the Neural Addition Unit}\label{sssec:weight}

\citet{glorot-initialization} show that $E[z_{h_\ell}] = 0$ at initialization is a desired property, as it prevents explosion of both the output and the gradients.
To satisfy this property with $W_{h_{\ell-1},h_\ell} = \tanh(\hat{W}_{h_{\ell-1},h_\ell}) \sigma(\hat{M}_{h_{\ell-1},h_\ell})$, an initialization must satisfy $E[\tanh(\hat{W}_{h_{\ell-1},h_\ell})] = 0$.
In NALU, this initialization is unbiased as it samples evenly between $+$ and $-$, or $\times$ and $\div$.
Unfortunately, this initialization also causes the expectation of the gradient to become zero, as shown in \eqref{eq:nac-weight-gradient}.

\begin{equation}
E\left[\frac{\partial \mathcal{L}}{\partial \hat{M}_{h_{\ell-1},h_\ell}}\right] = E\left[\frac{\partial \mathcal{L}}{\partial W_{h_{\ell-1},h_\ell}}\right] E\left[\tanh(\hat{W}_{h_{\ell-1},h_\ell})\right] E\left[\sigma'(\hat{M}_{h_{\ell-1},h_\ell})\right] = 0
\label{eq:nac-weight-gradient}
\end{equation}

Besides the issue of initialization, our empirical analysis (table \ref{tab:function-task-static-defaults}) shows that this weight construction \eqref{eq:weight} do not create the desired bias for $\{-1, 0, 1\}$ for the addition and subtraction problem. This bias is desired as it restricts the solution space to exact addition, and in section \label{sec:method:nmu} also exact multiplication, which is an intrinsic property of an underlying arithmetic function. However, it does not necessarily restrict the output space as a plain linear transformation will always be able to scale values accordingly. The bias also adds interpretability which is important for being confident in a model’s ability to extrapolate.

To solve these issues, we add a sparsifying regularizer to the loss function ($\mathcal{L} = \hat{\mathcal{L}} + \lambda_{\mathrm{sparse}} \mathcal{R}_{\ell,\mathrm{sparse}}$) and use simple linear weight construction, where $W_{h_{\ell-1},h_\ell}$ is clamped to $[-1, 1]$ in each iteration.

\begin{align}
W_{h_{\ell-1},h_\ell} &= \min(\max(W_{h_{\ell-1},h_\ell}, -1), 1), \\
\mathcal{R}_{\ell,\mathrm{sparse}} &= \frac{1}{H_\ell \cdot H_{\ell-1}} \sum_{h_\ell=1}^{H_\ell} \sum_{h_{\ell-1}=1}^{H_{\ell-1}} \min\left(|W_{h_{\ell-1},h_\ell}|, 1 - \left|W_{h_{\ell-1},h_\ell}\right|\right) \\
\textrm{NAU}:\ z_{h_\ell} &= \sum_{h_{\ell-1}=1}^{H_{\ell-1}} W_{h_{\ell}, h_{\ell-1}} z_{h_{\ell-1}}
\end{align}

\subsection{Challenges of division} \label{sssec:nac-mul}

The $\text{NAC}_{\bullet}$, as formulated in equation \ref{eq:nacm}, has the ability to perform exact multiplication and division, or more precisely multiplication of the inverse of elements from a vector, when a weight in $W_{h_{\ell-1},h_\ell}$ is $-1$.

However, this flexibility creates critical optimization challenges. Expanding the exp-log-transformation, $\text{NAC}_{\bullet}$ can be expressed as
\begin{equation}
\textrm{NAC}_\bullet:\ z_{h_\ell} = \prod_{h_{\ell-1}=1}^{H_{\ell-1}} (|z_{h_{\ell-1}}| + \epsilon)^{W_{h_{\ell}, h_{\ell-1}}}\ .
\label{eq:division:nac-mul-rewrite}
\end{equation}

In equation \eqref{eq:division:nac-mul-rewrite}, if $|z_{h_{\ell-1}}|$ is near zero ($E[z_{h_{\ell-1}}] = 0$ is a desired property when initializing \cite{glorot-initialization}), $W_{h_{\ell-1},h_\ell}$ is negative, and $\epsilon$ is small, then the output will explode. This issue is present even for a reasonably large $\epsilon$ value (such as $\epsilon = 0.1$), and just a slightly negative $W_{h_{\ell-1},h_\ell}$, as visualized in figure \ref{fig:nac-mul-eps-issue}. Also note that the curvature can cause convergence to an unstable area.

This singularity issue in the optimization space also makes multiplication challenging, which further suggests that supporting division is undesirable. These observations are also found empirically in \citet[table 1]{trask-nalu} and Appendix \ref{sec:appendix:comparison-all-models}.

%However, backpropagation through the $\text{NAC}_{\bullet}$ unit (equation \ref{eq:dz}, derivation in Appendix \ref{sec:appendix:gradient-derivatives:gradient-nac-mul}) reveals that if $|z_{h_{\ell-1}}|$ is near zero, $W_{h_{\ell-1},h_\ell}$ is negative and $\epsilon$ is small, the gradient term will explode and oscillate between large positive and large negative values, which can be problematic in optimization \cite{adam-optimization}, as visualized in figure \ref{fig:nac-mul-eps-issue}.
%\begin{align}
%\frac{\partial \mathcal{L}}{\partial W_{h_{\ell}, h_{\ell - 1}}} &= \frac{\partial \mathcal{L}}{\partial z_{h_\ell}} \frac{\partial z_{h_\ell}}{\partial W_{h_{\ell}, h_{\ell - 1}}} = \frac{\partial \mathcal{L}}{\partial z_{h_\ell}} z_{h_\ell} \log(|z_{h_{\ell-1}}| + \epsilon) \label{eq:dw}\\
%\frac{\partial \mathcal{L}}{\partial z_{h_{\ell-1}}} &= \sum_{h_\ell = 1}^{H_\ell} \frac{\partial \mathcal{L}}{\partial z_{h_\ell}} \frac{\partial z_{h_\ell}}{\partial z_{h_{\ell-1}}} = \sum_{h_\ell = 1}^{H_\ell} \frac{\partial \mathcal{L}}{\partial z_{h_\ell}} z_{h_\ell} W_{h_\ell, h_{\ell-1}} \frac{\mathrm{sign}(z_{h_{\ell-1}})}{|z_{h_{\ell-1}}| + \epsilon}\label{eq:dz}
%\end{align}

%This is not an issue for positive values of $W_{h_{\ell-1},h_\ell}$ (multiplication), as $z_{h_{\ell}}$ and $z_{h_{\ell-1}}$ will be correlated causing the terms $z_{h_\ell}$ and $\frac{\mathrm{sign}(z_{h_{\ell-1}})}{|z_{h_{\ell-1}}| + \epsilon}$ to partially cancel out.

% This gradient can be particular problematic when considering that $E[z_{h_{\ell-1}}] = 0$ is a desired property when initializing \cite{glorot-initialization}.
% A desired multiplication unit should not explode for $z_{h_{\ell-1}}$ near zero, which is why supporting division is likely infeasible.

\begin{figure}[h]
\centering
\begin{subfigure}{.33\textwidth}
  \centering
  \includegraphics[width=\linewidth,trim={0 0 0 4.35cm},clip]{graphics/nac-mul-eps-1em7.png}
  \caption{$\mathrm{NAC}_{\bullet}$ with $\epsilon = 10^{-7}$}
\end{subfigure}%
\begin{subfigure}{.33\textwidth}
  \centering
  \includegraphics[width=\linewidth,trim={0 0 0 4.35cm},clip]{graphics/nac-mul-eps-1em1.png}
  \caption{$\mathrm{NAC}_{\bullet}$ with $\epsilon = 0.1$}
\end{subfigure}
\begin{subfigure}{.33\textwidth}
  \centering
  \includegraphics[width=\linewidth,trim={0 0 0 4.35cm},clip]{graphics/nac-mul-eps-1.png}
  \caption{$\mathrm{NAC}_{\bullet}$ with $\epsilon = 1$}
\end{subfigure}
%\begin{subfigure}{.33\textwidth}
%  \centering
%\includegraphics[width=\linewidth]{graphics/nac-mul-nmu.png}
%  \caption{Our NMU solution}
%\end{subfigure}

\caption{RMS loss curvature for a $\mathrm{NAC}_{+}$ layer followed by a $\mathrm{NAC}_{\bullet}$. The weight matrices are constrained to $\mathbf{W}_1 = \left[\protect\begin{smallmatrix}
w_1 & w_1 & 0 & 0 \\
w_1 & w_1 & w_1 & w_1
\protect\end{smallmatrix}\right]$, $\mathbf{W}_2 = \left[\protect\begin{smallmatrix}
w_2 & w_2
\protect\end{smallmatrix}\right]$. The problem is $(x_1 + x_2) \cdot (x_1 + x_2 + x_3 + x_4)$ for $x = \left(1, 1.2, 1.8, 2\right)$.
The solution is $w_1 = w_2 = 1$ with many unstable alternatives.}
\label{fig:nac-mul-eps-issue}
\end{figure}

\subsection{Initialization of \texorpdfstring{$\mathrm{NAC}_{\bullet}$}{NAC-mul}}
Initialization is important to consider for fast and consistent convergence.
One desired property is that weights can be initialized such that $E[z_{h_\ell}] = 0$ \cite{glorot-initialization}. Using second order Taylor approximation and assuming all $z_{h_{\ell-1}}$ are uncorrelated; the expectation of $\mathrm{NAC}_{\bullet}$ can be estimated as
\begin{equation}
E[z_{h_\ell}] \approx \left(1 + \frac{1}{2} Var[W_{h_\ell, h_{\ell-1}}] \log(|E[z_{h_{\ell-1}}]| + \epsilon)^2\right)^{H_{\ell-1}} \Rightarrow E[z_{h_\ell}] > 1.
\label{eq:nac-mul:expectation}
\end{equation}
As shown in equation \ref{eq:nac-mul:expectation}, satisfying $E[z_{h_\ell}] = 0$ for $\mathrm{NAC}_{\bullet}$ is likely impossible. The variance cannot be input-independently initialized and is expected to explode (proofs in Appendix \ref{sec:appendix:moments:nac-mul}).

\subsection{Neural multiplication unit}
\label{sec:method:nmu}
To solve the the gradient and initialization challenges for $\mathrm{NAC}_{\bullet}$ we propose a new unit for multiplication: the Neural Multiplication Unit (NMU)

\begin{align}
W_{h_{\ell-1},h_\ell} &= \min(\max(W_{h_{\ell-1},h_\ell}, 0), 1), \\
\mathcal{R}_{\ell,\mathrm{sparse}} &= \frac{1}{H_\ell \cdot H_{\ell-1}} \sum_{h_\ell=1}^{H_\ell} \sum_{h_{\ell-1}=1}^{H_{\ell-1}} \min\left(W_{h_{\ell-1},h_\ell}, 1 - W_{h_{\ell-1},h_\ell}\right) \\
\textrm{NMU}:\ z_{h_\ell} &= \prod_{h_{\ell-1}=1}^{H_{\ell-1}} \left(W_{h_{\ell-1},h_\ell} z_{h_{\ell-1}} + 1 - W_{h_{\ell-1},h_\ell} \right) \label{eq:nmu-defintion}
\end{align}
The NMU is regularized similar to the NAU and has a multiplicative identity when $W_{h_{\ell-1},h_\ell}=0$.
The NMU do not support division by design.
%Previous experiments using the NALU for division do not work well on division hence very little is lost with this modification \cite{trask-nalu}.
As opposed to the $\mathrm{NAC}_{\bullet}$, the NMU can represent input of both negative and positive values and is not $\epsilon$ bounded, which allows the NMU to extrapolate to $z_{h_{\ell-1}}$ that are negative or smaller than $\epsilon$. Its gradients are derived in Appendix \ref{sec:appendix:gradient-derivatives:gradient-nmu}.

\subsection{Moments and initialization}
\label{sec:methods:moments-and-initialization}
The NAU is a linear layer and can be initialized using \citet{glorot-initialization}. The $\mathrm{NAC}_{+}$ unit can also achieve an ideal initialization, although it is less trivial (details in Appendix \ref{sec:appendix:moments:weight-matrix-construction}).

The NMU is initialized with $E[W_{h_{\ell}, h_{\ell - 1}}] = \nicefrac{1}{2}$. Assuming all $z_{h_{\ell-1}}$ are uncorrelated, and $E[z_{h_{\ell-1}}] = 0$, which is the case for most neural units \cite{glorot-initialization}, the expectation can be approximated to
\begin{equation}
E[z_{h_\ell}] \approx \left(\frac{1}{2}\right)^{H_{\ell-1}},
\end{equation}
which approaches zero for $H_{\ell-1} \rightarrow \infty$ (see Appendix \ref{sec:appendix:moments:nmu}). The NMU can, assuming $Var[z_{h_{\ell-1}}] = 1$ and $H_{\ell-1}$ is large, be optimally initialized with $Var[W_{h_{\ell-1},h_\ell}] = \frac{1}{4}$ (proof in Appendix \ref{sec:appendix:moments:nmu:initialization}).

\subsection{Regularizer scaling}
We use the regularizer scaling as defined in \eqref{eq:regualizer-scaling}. We motivate this by observing optimization consists of two parts: a warmup period, where $W_{h_{\ell-1},h_\ell}$ should get close to the solution, unhindered by the sparsity regularizer, followed by a period where the solution is made sparse.
\begin{equation}
\lambda_{\mathrm{sparse}} = \hat{\lambda}_{\mathrm{sparse}} \max\left(\min\left(\frac{t - \lambda_{\mathrm{start}}}{\lambda_{\mathrm{end}} - \lambda_{\mathrm{start}}}, 1\right), 0\right)
\label{eq:regualizer-scaling}
\end{equation}

\subsection{Challenges of gating between addition and multiplication}
\label{sec:methods:gatting-issue}
The purpose of the gating-mechanism is to select either $\text{NAC}_{+}$ or $\text{NAC}_{\bullet}$ exclusively.
This assumes that the correct sub-unit is selected by the NALU, since selecting the wrong sub-unit leaves no gradient signal for the correct sub-unit.

Empirically we find this assumption to be problematic.
We observe that both sub-units converge in the beginning of training whereafter the gating-mechanism, seemingly randomly, converge towards either the addition or multiplication unit. Our study shows that gating behaves close to random for both NALU and a gated NMU/NAU variant. However, when the gate correctly selects multiplication our NMU converge much more consistently. (in-depth empirical analysis in appendix \ref{sec:appendix:nalu-gate-experiment}, with results for both a shared and non-shared weight matrix between $\text{NAC}_{+}$ and $\text{NAC}_{\bullet}$, and a gated version of NAU/NMU).

As the problem size grows, randomly choosing the correct gating value becomes an exponential increasing problem. Because of these challenges we leave solving the problem of sparse gating for future work and focus on improving the sub-units $\text{NAC}_{+}$ and $\text{NAC}_{\bullet}$.


% !TEX root =  manual.tex
\section{Introduction}\label{sec:intro}

Numerous problems in both the natural and social sciences now involve large scale models characterized by numerous parameters and assumptions. These models are then often compared to heterogenous data sets, which in some cases are distilled from peta-scale observations. In addition to the investment usually required in obtaining such data sets, running the models is often expensive and time-consuming.

Perhaps the most common method for determining model parameters, $x_1\cdots x_P$ from comparing to a set of measurements $y_1\cdots y_M$ is to calculate the implausibility, $\chi^2$, as a function of ${\bf x}$ where $\chi^2({\bf x})\ge 0$ describes the ``goodness'' of the fit for a specific point in parameter space and is zero for a perfect fit,
\begin{equation}
\chi^2({\bf x})\equiv
\sum_a \frac{\left(y_a^{\rm(mod)}({\bf x})-y_a^{\rm(exp)}\right)^2}{\sigma_a^2},
\end{equation}
where $\sigma_a$ is a measure for the uncertainty for comparing the model to the experiment, and can come from uncertainties from either the model or from the experiment. If the uncertainty involved in comparing the model to the data is normally distributed, and if there is no prior information about the parameters (a flat prior distribution), Bayes theorem tells us that the likelihood that the point in parameter space could reproduce the data is
\begin{equation}
\mathcal{L}({\bf x})\sim \exp\left\{-\frac{\chi^2({\bf x})}{2}\right\}.
\end{equation}
In some cases, one is interested in only the point of minimum $\chi^2$, but a much more appropriate goal is to understand the entire distribution $\mathcal{L}({\bf x})$, so that one knows not only the most likely value, but also understands the range and distribution of likely values of ${\bf x}$.

The textbook method for exploring and determining $\mathcal{L}({\bf x})$ is to perform Markov-chain Monte Carlo (MCMC) calculations. MCMC codes ergodically explore the $P-$dimension parameter space, and provide a sampling of the parameter space weighted by the likelihood. For a higher dimension parameter space, the exploration might require millions of sampling points. At each point ${\bf x}$, one would run the full model, and determine $\chi^2({\bf x})$. However, if the model is expensive to run, this procedure is untenable.

In the last decade strategies based on {\it model emulators} (a.k.a. surrogate models or meta-models or interpolators) have been developed to handle this problem. Emulators are functions that estimate the observables $y_i({\bf x})$ given $N$ sampling runs, $Y_i({\bf x}_\alpha),~\alpha=1\cdots N$. When running the MCMC code, one calls the emulator to provide not only an interpolated value $y_i^{\bf(emu)}({\bf x})$, but an estimate of the emulator's error. Emulators are typically very fast, which makes the MCMC procedure plausible. The number of sampling runs is on the order of hundreds or thousands, and the calculations remain expensive, but the procedure is much less expensive than the alternative of running the calculations at millions of points during the MCMC exploration.

The software described in this manual is designed mainly to assist scientists in comparing models to data using emulators. This manual provides installation instructions, a summary of the statistical theory, directions for organizing the data, and instructions for creating the emulator and running the MCMC. The package includes both command-line elements and visualization tools. A separate tutorial is also part of the package.


\section*{Integration technics}
\cm{HAVE NO IDEA HOW TO START THIS SECTION}
Euler first order explicit
\begin{align}\label{eqn:euler}
    \begin{split}
        &v(t+\Delta{t}) = v(t) + a(t) * \Delta{t}\\
        &x(t+\Delta{t}) = x(t) + v(t) * \Delta{t}
    \end{split}
\end{align}
\par
The central difference scheme, also known as velocity Verlet is a very widely
used integration method of second order. Velocities in the upcoming time step $t
+ \Delta{t}/2 $ and positions at $t + \Delta{t}$ are calculated as
\begin{align}\label{eqn:verlet}
    \begin{split}
        &v(t + \Delta{t}/2) = v(t + \Delta{t}/2) +a(t - \Delta{t}/2) * \Delta{t}\\
        &x(t + \Delta{t}) = x(t) +v(t + \Delta{t}/2) * \Delta{t}\\
    \end{split}
\end{align}
\par
The fourth order Gear’s method (GPC4). In the prediction step positions and
their higher derivatives are calculated based on Taylor series expansions as
\begin{equation}\label{eqn:gearP}
    \begin{split}
        b(t+\Delta{t}, p) =& b(t)\\
        a(t+\Delta{t}, p) =& a(t) + b(t) *\Delta{t}^2\\
        v(t+\Delta{t}, p) =& v(t) + a(t) *\Delta{t} + \frac{1}{2} * b(t) *\Delta{t}^2\\
        x(t+\Delta{t}, p) =& x(t) + v(t) * \Delta{t} + \frac{1}{2} * a(t) *\Delta{t}^2 +\\
        &\frac{1}{6} * b(t) *\Delta{t}^3\\
    \end{split}
\end{equation}
with the first and second derivative of the accelerations for GPC3 calculated as:
\begin{align}\label{eqn:gearJerk3}
    b(t) = \frac{\Delta{a(t)}}{\Delta{t}}, c(t) = 0
\end{align}\par
In the evaluation step the difference in the accelerations calculated based on
the acceleration $a(t+ \Delta{t}, p)$ and the acceleration $a(t+\Delta{t})$
calculated from positions $x(t+\Delta{t}, p)$ and velocities $v(t+\Delta{t}, p)$
is obtained by
\begin{align}\label{eqn:gearDa}
    \Delta{a} = a(t + \Delta{t}) - a(t + \Delta{t}, p)
\end{align}
\par
In the following, correction step positions and their higher derivatives are
calculated based on their values from the previous time step and the obtained
difference in acceleration as
\begin{align}\label{eqn:gearCorrector}
    \begin{split}
        &x(t+\Delta{t}) = x(t+\Delta{t}, p) + k1 * \Delta{a} * \Delta{t}^2\\
        &v(t+\Delta{t}) = v(t+\Delta{t}, p) + k2 * \Delta{a} * \Delta{t}\\
        &a(t+\Delta{t}) = a(t+\Delta{t}, p) + k3 * \Delta{a}\\
        &b(t+\Delta{t}) = b(t+\Delta{t}, p) + k4 * \frac{\Delta{a}}{\Delta{t}}\\
    \end{split}
\end{align}
\par
Gear’s scheme parameters k1-k5 for GPC3:
$k_1=1/12$, $k_2=5/12$, $k_3=1$, $k_4=1$, $k_5=0$\par