\lab{Interior Point 1: Linear Programs}{Interior Point 1: Linear Programs}

% Bold mu, lambda, and beta symbols for this lab. TODO: move to command.tex.
\def\Mu{\boldsymbol{\mu}}
\def\Lamb{\boldsymbol{\lambda}}
\def\Beta{\boldsymbol{\beta}}

\objective{For decades after its invention, the Simplex algorithm was the only competitive method for linear programming.
The past 30 years, however, have seen the discovery and widespread adoption of a new family of algorithms that rival--and in some cases outperform--the Simplex algorithm, collectively called \emph{Interior Point methods}.
One of the major shortcomings of the Simplex algorithm is that the number of steps required to solve the problem can grow exponentially with the size of the linear system.
Thus, for certain large linear programs, the Simplex algorithm is simply not viable.
Interior Point methods offer an alternative approach and enjoy much better theoretical convergence properties.
In this lab we implement an Interior Point method for linear programs, and in the next lab we will turn to the problem of solving quadratic programs.}

\section*{Introduction}

Recall that a linear program is a constrained optimization problem with a linear objective function and linear constraints.
The linear constraints define a set of allowable points called the \emph{feasible region}, the boundary of which forms a geometric object known as a \emph{polytope}.
The theory of convex optimization ensures that the optimal point for the objective function can be found among the vertices of the feasible polytope. The Simplex Method tests a sequence of such vertices until it finds
the optimal point.
Provided the linear program is neither unbounded nor infeasible, the algorithm is certain to produce the correct answer after a finite number of steps, but it does not guarantee an efficient path along the polytope toward the minimizer.
Interior point methods do away with the feasible polytope and instead generate a sequence of points that cut through the interior (or exterior) of the feasible region and converge iteratively to the optimal point.
Although it is computationally more expensive to compute such interior points, each step results in significant progress toward the minimizer.
See Figure \ref{fig:intPath}.
In general, the Simplex Method requires many more iterations (though each iteration is less expensive computationally).

\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/interiorPath.pdf}
\caption{A path traced by an Interior Point algorithm.}
\label{fig:intPath}
\end{figure}

\section*{Primal-Dual Interior Point Methods} % ALGORITHM =====================
Some of the most popular and successful types of Interior Point methods are known as Primal-Dual Interior Point methods.
Consider the following linear program:
\begin{align*}
\text{minimize }\qquad &\c\trp \x\\
\text{subject to }\qquad &A\x = \b\\
&\x \succeq \0.
\end{align*}
Here, $\x, \c \in \mathbb{R}^n$, $\b \in \mathbb{R}^m$, and $A$ is an $m \times n$ matrix with full row rank.
% By $x \succeq 0$, we simply mean that each coordinate of $x$ is nonnegative.
% Note that this formulation is quite general, as any linear program can be posed in this manner, after appropriate transformations.
This is the \emph{primal} problem, and its \emph{dual} takes the form:
\begin{align*}
\text{maximize }\qquad &\b\trp \Lamb\\
\text{subject to }\qquad &A\trp \Lamb + \Mu = \c\\
&\Mu,\Lamb \succeq \0,
\end{align*}
where $\Lamb \in \mathbb{R}^m$ and $\Mu \in \mathbb{R}^n$.
% Changed s to \mu to match the textbook.

\subsection*{KKT Conditions}

The theory of convex optimization gives us necessary and sufficient conditions for the solutions to the primal and dual problems via the Karush-Kuhn-Tucker (KKT) conditions.
The Lagrangian for the primal problem is as follows:
\[
\CalL(\x, \Lamb, \Mu)
= \c\trp \x + \Lamb\trp (\b - A\x) - \Mu\trp \x
\]
The KKT conditions are
\begin{align*}
A\trp \Lamb + \Mu &= \c\\
A\x &= \b\\
x_i\mu_i &= 0, \quad i = 1,2,\ldots,n,\\
\x, \Mu &\succeq 0.
\end{align*}
It is convenient to write these conditions in a more compact manner, by defining an almost-linear function $F$ and setting it equal to zero:
\begin{align*}
F(\x,\Lamb,\Mu) :=
\begin{bmatrix}
A\trp \Lamb + \Mu - \c\\
A\x - \b\\
M\x\\
\end{bmatrix}
&= \0,\\
(\x,\Mu &\succeq \0),
\end{align*}
where $M = \text{diag}(\mu_1,\mu_2,\ldots,\mu_n)$.
Note that the first row of $F$ is the KKT condition for dual feasibility, the second row of $F$ is the KKT condition for the primal problem, and the last row of $F$ accounts for complementary slackness.

\begin{comment} % Derivation overkill vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
First, we calculate the so-called \emph{Lagrangian function} for our linear program.
Recall that the Lagrangian function for a general constrained optimization problem of the form
\begin{align*}
\text{minimize }\qquad &f(x)\\
\text{subject to }\qquad &c_i(x) = 0, \quad i = 1,2,\ldots,k,\\
&d_i(x) \geq 0, \quad i = 1,2,\ldots,j,
\end{align*}
is defined to be
\[
\mathcal{L}(x, \lambda, s) := f(x) - \displaystyle\sum_{i=1}^k\lambda_ic_i(x) - \displaystyle\sum_{i=1}^js_id_i(x).
\]
From here, the KKT conditions are
\begin{align*}
\nabla_x\mathcal{L}(x,\lambda,s) &= 0,\\
c_i(x) &= 0,\quad i = 1,2,\ldots,k,\\
s_id_i(x) &= 0,\quad i=1,2,\ldots,j,\\
d_i(x) &\geq 0,\quad i=1,2,\ldots,j,\\
s_i &\geq 0,\quad i=1,2,\ldots,j.
\end{align*}
For our linear program, $f(x) = c\trp x$, $c_i(x) = A_i\trp x - b_i$ (where $A_i$ denotes the $i$-th row of $A$), and $d_i(x) = x_i$.
Hence, you can check that our Lagrangian function is
\[
\mathcal{L}(x,\lambda,s) = c\trp x - \lambda\trp (Ax - b) - s\trp x.
\]
We then calculate
\[
\nabla_x\mathcal{L}(x,\lambda,s) = (c\trp  - \lambda\trp A - s\trp )\trp  = c - A\trp \lambda - s.
\]
The KKT conditions can now be expressed as follows:
\begin{align*}
A\trp \lambda + s &= c\\
A\x &= \b\\
x_is_i &= 0, \quad i = 1,2,\ldots,n,\\
\x, \mathbf{s} &\succeq 0.
\end{align*}
In fact, it is convenient to write these conditions in a more compact manner, by defining an almost-linear function $F$ and setting it equal to zero:
\begin{align*}
F(\x,\Lamb,\Mu) :=
\begin{bmatrix}
A\trp \lambda + s - c\\
Ax - b\\
XSe\\
\end{bmatrix}
&= 0,\\
(\x,\mathbf{s} &\succeq \0),
\end{align*}
where $X = \text{diag}(x_1,x_2,\ldots,x_n)$, $S = \text{diag}(s_1,s_2,\ldots,s_n)$, and $e = (1,1,\ldots,1)\trp $.
\end{comment} % Derivation overkil ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

\begin{problem} % First step: define F(x,lamb,mu) -----------------------------
Define a function \li{interiorPoint()} that will be used to solve the complete interior point problem.
This function should accept $A$, $\b$, and $\c$ as parameters, along with the keyword arguments \li{niter=20} and \li{tol=1e-16}.
The keyword arguments will be used in a later problem.

For this problem, within the \li{interiorPoint()} function, write a function for the vector-valued function $F$ described above.
This function should accept $\x$, $\Lamb$, and $\Mu$ as parameters and return a 1-dimensional NumPy array with $2n+m$ entries.
\end{problem}

\subsection*{Search Direction}

A Primal-Dual Interior Point method is a line search method that starts with an initial guess $(\x_0\trp , \Lamb_0\trp , \Mu_0\trp )$ and produces a sequence of points that converge to $({\x^*}\trp , {\Lamb^*}\trp , {\Mu^*}\trp )$, the solution to the KKT equations and hence the solution to the original linear program.
%We now describe how to select the search direction and step length at each iteration of the algorithm.
The constraints on the problem make finding a search direction and step length a little more complicated than for the unconstrained line search we have studied previously.

In the spirit of Newton's Method, we can form a linear approximation of the system $F(\x,\Lamb,\Mu) = \0$ centered around our current point $(\x, \Lamb, \Mu)$, and calculate the direction $(\triangle \x\trp , \triangle \Lamb\trp , \triangle \Mu\trp )$ in which to step to set the linear approximation equal to $\0$.
This equates to solving the linear system:
\begin{align}
DF(\x,\Lamb,\Mu)
\begin{bmatrix}\triangle \x\\ \triangle \Lamb\\ \triangle \Mu\end{bmatrix}
= - F(\x,\Lamb,\Mu) \label{eq:naiveNewton}
\end{align}
Here $DF(\x,\Lamb,\Mu)$ denotes the total derivative matrix of $F$.
We can calculate this matrix block-wise by obtaining the partial derivatives of each block entry of $F(\x,\Lamb, \Mu)$ with respect to $\x$, $\Lamb$, and $\Mu$, respectively.
We thus obtain:
\[
DF(\x,\Lamb,\Mu) = \left[\begin{array}{ccc}
0 & A\trp  & I \\
A & 0 & 0 \\
M & 0 & X
\end{array}\right]
\]
where $X = \text{diag}(x_1,x_2,\ldots,x_n).$

\begin{comment} % More derivation overkill vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
For example, consider the topmost block entry of $F(\x,\Lamb,\Mu)$, given by
\[
A\trp \lambda + s - c.
\]
We calculate the partial derivatives of this expressions as follows:
\begin{align*}
\frac{\partial}{\partial x}[A\trp \lambda + s - c] &= 0,\\
\frac{\partial}{\partial \lambda}[A\trp \lambda + s - c] &= A\trp ,\\
\frac{\partial}{\partial s}[A\trp \lambda + s - c] &= I.\\
\end{align*}
Hence, the topmost block row of $DF(\x,\Lamb,\Mu)$ is given by
\[
\begin{bmatrix}
0 \quad A\trp  \quad I\\
\end{bmatrix}.
\]
Continuing in this manner, we can calculate the entire derivative matrix, which leads us to the following linear system:
\end{comment} % More derivation overkill ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Unfortunately, solving Equation \ref{eq:naiveNewton} often leads to a search direction that is too greedy.
Even small steps in this direction may lead the iteration out of the feasible region by violating one of the constraints.
To remedy this, we define the \emph{duality measure} $\nu$\footnote{$\nu$ is the Greek letter for $n$, pronounced ``nu.''} of the problem: \[\nu = \frac{\x\trp \Mu}{n} \]
The idea is to use Newton's method to identify a direction that strictly decreases $\nu$.
Thus instead of solving Equation \ref{eq:naiveNewton}, we solve:
\begin{align}
DF(\x,\Lamb,\Mu)
\begin{bmatrix}\triangle \x\\ \triangle \Lamb\\ \triangle \Mu\end{bmatrix}
= - F(\x,\Lamb,\Mu) +
\begin{bmatrix} \0 \\ \0 \\ \sigma\nu\e \end{bmatrix}
\label{eq:newNewton}
\end{align}
where $\e = (1,1,\ldots,1)\trp $ and $\sigma \in [0,1)$ is called the \emph{centering parameter}.
The closer $\sigma$ is to 0, the more similar the resulting direction will be to the plain Newton direction.
The closer $\sigma$ is to 1, the more the direction points inward to the interior of the of the feasible region.

\begin{problem} % Search direction --------------------------------------------
Within \li{interiorPoint()}, write a subroutine to compute the search direction $(\triangle \x\trp , \triangle \Lamb\trp , \triangle \Mu\trp )$ by solving Equation \ref{eq:newNewton}.
Use $\sigma = \frac{1}{10}$ for the centering parameter.

Note that only the last block row of $DF$ will need to be changed at each iteration (since $M$ and $X$ depend on $\Mu$ and $\x$, respectively).
Consider using the functions \li{lu_factor()} and \li{lu_solve()} from the \li{scipy.linalg} module to solving the system of equations efficiently.
\end{problem}

\begin{comment} % Old strategy for search direction vvvvvvvvvvvvvvvvvvvvvvvvvvv
\begin{equation}
\begin{bmatrix}
0 & A\trp  & I\\
A & 0 & 0\\
S & 0 & X
\end{bmatrix}
\begin{bmatrix}
\triangle \x\\
\triangle \Lamb\\
\triangle \Mu
\end{bmatrix}
=
\begin{bmatrix}
-r_c\\
-r_b\\
-XSe
\end{bmatrix},
\label{eq:affine}
\end{equation}
where $r_b = Ax - b$ and $r_c = A\trp \lambda + s - c$.
This Newton direction is too greedy, however, and even small steps in this direction may cause us to violate the nonnegativity condition (the last line of the KKT conditions).
We need to find a new search direction.

Choosing an appropriate search direction is a tricky task, and various approaches exist.
We will follow a popular strategy known as the \emph{Predictor-Corrector Algorithm}.
The idea is to actually calculate two directions: we first calculate
the standard Newton direction described above; that is, we obtain the solution to Equation \ref{eq:affine}.
This is known as the \emph{predictor step}, since it gives us a direction in which the objective does indeed decrease, hence predicting our
final search direction.
Denote the solution to this system by $(\triangle \x^a, \triangle \Lamb^a, \triangle \Mu^a)$.
As discussed, however, we must deviate from this direction somewhat, and so we additionally solve the following linear system of equations:
\begin{equation}
\begin{bmatrix}
0 & A\trp  & I\\
A & 0 & 0\\
S & 0 & X
\end{bmatrix}
\begin{bmatrix}
\triangle \x\\
\triangle \Lamb\\
\triangle \Mu
\end{bmatrix}
=
\begin{bmatrix}
-r_c\\
-r_b\\
-XSe - \triangle X^a\triangle S^ae + \sigma \mu e
\end{bmatrix}.
\label{eq:centering}
\end{equation}
The solution to this system, which we denote by $(\triangle \x, \triangle \Lamb, \triangle \Mu)$, is our final search direction.

We now describe the new pieces of Equation \ref{eq:centering}.
Note that
\begin{align*}
\triangle X^a &= \text{diag}(\triangle x_1^a,\ldots,\triangle x_n^a),\\
\triangle S^a &= \text{diag}(\triangle \Mu_1^a,\ldots,\triangle \Mu_n^a).
\end{align*}
Further, define $\mu := x\trp s/n$.
This quantity, called the \emph{duality measure}, tells us roughly how close we are to the optimal solution, with values closer to 0 being more desirable.
It is present in Equation \ref{eq:centering} for the purpose of obtaining a search direction that leads to a decrease towards 0 in the duality measure.

The formula for $\sigma$ is somewhat more complicated, and is based on a heuristic approach.
First, make the following calculations (which give the maximum allowable step lengths in the Newton search direction for $x$ and $s$ before the nonnegativity condition is violated):
\begin{align*}
\alpha_a^p &:= \min\left(1, \displaystyle\min_{i : \triangle x_i^a < 0}-\frac{x_i}{\triangle x_i^a}\right)\\
\alpha_a^d &:= \min\left(1, \displaystyle\min_{i : \triangle \mu_i^a < 0}-\frac{s_i}{\triangle \mu_i^a}\right).
\end{align*}
Next, define
\[
\mu_a := \frac{1}{n}(x+\alpha_a^p\triangle \x^a)\trp (s+\alpha_a^d\triangle \Mu^a),
\]
which is simply the duality measure for the point obtained by taking a full step in the Newton direction.
Finally, calculate $\sigma$ by the formula
\[
\sigma = \left(\frac{\mu_a}{\mu}\right)^3.
\]
\end{comment} % Old strategy for search direction ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

\subsection*{Step Length}

Now that we have our search direction, it remains to choose our step length.
We wish to step nearly as far as possible without violating the problem's constraints, thus remaining in the interior of the feasible region.
First, we calculate the maximum allowable step lengths for $\x$ and $\Mu$, respectively:
\begin{align*}
\alpha_{\max} &= \min\left\{1, \min\{-\mu_i/\triangle \mu_i\ |\ \triangle \mu_i < 0 \}\right\}\\
\delta_{\max} &= \min\left\{1, \min\{-x_i/\triangle x_i\ |\ \triangle x_i < 0 \}\right\}
\end{align*}
Next, we back off from these maximum step lengths slightly:
\begin{align*}
\alpha &= \min(1, 0.95\alpha_{\max})\\
\delta &= \min(1, 0.95\delta_{\max}).
\end{align*}
These are our final step lengths.
Thus, the next point in the iteration is given by:
\begin{align*}
\x_{k+1} &= \x_k + \delta \triangle \x_k\\
(\Lamb_{k+1}, \Mu_{k+1}) &= (\Lamb_k, \Mu_k) + \alpha(\triangle \Lamb_k, \triangle \Mu_k).
\end{align*}

\begin{comment} % Old strategy for step length vvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
\begin{align*}
\beta^p &:= \displaystyle\min_{i : \triangle x_i < 0}-\frac{x_i}{\triangle x_i}\\
\beta^d &:= \displaystyle\min_{i : \triangle \mu_i < 0}-\frac{s_i}{\triangle \mu_i}.
\end{align*}
Next, we back off from these maximum step lengths slightly:
\begin{align*}
\alpha^p &:= \min(1, 0.95\beta^p)\\
\alpha^d &:= \min(1, 0.95\beta^d).
\end{align*}
These are our final step lengths.
That is, our next point $(x', \lambda', s')$ is given by
\begin{align*}
x' &= x + \alpha^p\triangle \x\\
(\lambda', s') &= (\lambda, s) + \alpha^d(\triangle \Lamb, \triangle \Mu).
\end{align*}

We summarize the entire procedure in Algorithm \ref{alg:predcorr}.
\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{Predictor-Corrector Algorithm}{}
    \State \textrm{Choose initial point } $(x_0, \lambda_0, s_0)$.
    \For{$k = 0, 1, 2, \ldots$}
        \State \textrm{Solve for } $(\triangle \x^a, \triangle \Lamb^a, \triangle \Mu^a)$.
        \State \textrm{Calculate } $\alpha_a^p, \alpha_a^d, \mu_a$, \textrm{and} $\sigma$.
        \State \textrm{Solve for } $(\triangle \x, \triangle \Lamb, \triangle \Mu)$.
        \State \textrm{Calculate the step lengths } $\alpha^p, \alpha^d$.
        \State $x_{k+1} = x_k + \alpha^p\triangle \x$,\\
        $\qquad\quad(\lambda_{k+1}, s_{k+1}) = (\lambda_k, s_k) + \alpha^d(\triangle \Lamb, \triangle \Mu)$.
    \EndFor
\EndProcedure
\end{algorithmic}
\caption{Predictor-Corrector Algorithm}
\label{alg:predcorr}
\end{algorithm}
\end{comment} % Old strategy for step length ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
\begin{comment} % Unnecessary / confusing hints vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
A few notes on the implementation of this algorithm are in order.
In each iteration, by far the most expensive operations are solving Equations \ref{eq:affine} and \ref{eq:centering}.
Fortunately, the only difference between the two equations is the right-hand side.
Thus, we can avoid repetitious calculation by first factorizing the block matrix
\[
\begin{bmatrix}
0 & A\trp  & I\\
A & 0 & 0\\
S & 0 & X
\end{bmatrix}
\]
once at the beginning of the iteration, and then using the factorization twice to solve both equations.
For convenience, consider using the functions \li{lu_factor()} and \li{lu_solve()} from the \li{scipy.linalg} module.
It is possible to speed up these calculations further, but for simplicity we won't pursue the issue further.

Next, as mentioned above, the duality measure $\mu$ tells us in some sense how close our current point is to the minimizer.
The closer $\mu$ is to 0, the closer we are to the optimal point.
Thus, by printing the value of $\mu$ at each iteration, you can track how your algorithm is progressing and detect when you have converged.

Another potentially tricky calculation that comes up in each iteration has the following form:
\[
\min\left(1, \displaystyle\min_{i : u_i < 0}-\frac{v_i}{u_i}\right),
\]
where $\mathbf{u} = (u_1, \ldots, u_n)\trp $ and $\mathbf{v} = (v_1, \ldots, v_n)\trp $ are vectors.
This can be done in a vectorized fashion in Python as follows (assuming that the arrays \li{u} and \li{v} have already been initialized):
\begin{lstlisting}
>>> mask = u < 0
>>> if np.any(mask):
>>>     myMin = min(1, (-v/u)[mask].min())
>>> else:
>>>     myMin = 1
\end{lstlisting}
We need the \li{if} statement to deal with the case where no entry of $u$ is negative.
\end{comment} % Unnecessary / confusing hints ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

\begin{problem} % Step size ---------------------------------------------------
Within \li{interiorPoint()}, write a subroutine to compute the step size after the search direction has been computed.
Avoid using loops when computing $\alpha_{\max}$ and $\beta_{\max}$ (use masking and NumPy functions instead).
\end{problem}

\subsection*{Initial Point}
Finally, the choice of initial point $(\x_0, \Lamb_0, \Mu_0)$ is an important, nontrivial one.
A na\"{i}vely or randomly chosen initial point may cause the algorithm to fail to converge.
The following function will calculate an appropriate initial point.

% TODO: change name from startingPoint() to starting_point() in spec / solution
\begin{lstlisting}
def starting_point(A, b, c):
    """Calculate an initial guess to the solution of the linear program
    min c\trp  x, Ax = b, x>=0.
    Reference: Nocedal and Wright, p. 410.
    """
    # Calculate x, lam, mu of minimal norm satisfying both
    # the primal and dual constraints.
    B = la.inv(A @ A.T))
    x = A.T @ B @ b
    lam = B @ A @ c
    mu = c - (A.T @ lam)

    # Perturb x and s so they are nonnegative.
    dx = max((-3./2)*x.min(), 0)
    dmu = max((-3./2)*mu.min(), 0)
    x += dx*np.ones_like(x)
    mu += dmu*np.ones_like(mu)

    # Perturb x and mu so they are not too small and not too dissimilar.
    dx = .5*(x*mu).sum()/mu.sum()
    dmu = .5*(x*mu).sum()/x.sum()
    x += dx*np.ones_like(x)
    mu += dmu*np.ones_like(mu)

    return x, lam, mu
\end{lstlisting}

\begin{problem} % Put it all together -----------------------------------------
Complete the implementation of \li{interiorPoint()}.

Use the function \li{starting_point()} provided above to select an initial point, then run the iteration \li{niter} times, or until the duality measure is less than \li{tol}.
Return the optimal point $\x^*$ and the optimal value $\c\trp \x^*$.

The duality measure $\nu$ tells us in some sense how close our current point is to the minimizer.
The closer $\nu$ is to 0, the closer we are to the optimal point.
Thus, by printing the value of $\nu$ at each iteration, you can track how your algorithm is progressing and detect when you have converged.

To test your implementation, use the following code to generate a random linear program, along with the optimal solution.
% TODO: update the spec file with this version of randomLP().
\begin{lstlisting}
    """Generate a linear program min c\trp  x s.t. Ax = b, x>=0.
    First generate m feasible constraints, then add
    slack variables to convert it into the above form.
    Inputs:
        m (int >= n): number of desired constraints.
        n (int): dimension of space in which to optimize.
    Outputs:
        A ((m,n+m) ndarray): Constraint matrix.
        b ((m,) ndarray): Constraint vector.
        c ((n+m,), ndarray): Objective function with m trailing 0s.
        x ((n,) ndarray): The first 'n' terms of the solution to the LP.
    """
    A = np.random.random((m,n))*20 - 10
    A[A[:,-1]<0] *= -1
    x = np.random.random(n)*10
    b = np.zeros(m)
    b[:n] = A[:n,:] @ x
    b[n:] = A[n:,:] @ x + np.random.random(m-n)*10
    c = np.zeros(n+m)
    c[:n] = A[:n,:].sum(axis=0)/n
    A = np.hstack((A, np.eye(m)))
    return A, b, -c, x
\end{lstlisting}
\begin{lstlisting}
>>> m, n = 7, 5
>>> A, b, c, x = randomLP(m, n)
>>> point, value = interiorPoint(A, b, c)
>>> np.allclose(x, point[:n])
<<True>>
\end{lstlisting}
\end{problem}

% TODO: In the future, time interiorPoint() against a Simplex method (CVXOPT?).
% Have a problem to demonstrate the advantages of each algorithm.

\section*{Least Absolute Deviations (LAD)} % APPLICATION ======================
We now return to the familiar problem of fitting a line (or hyperplane) to a set of data.
We have previously approached this problem by minimizing the sum of the squares of the errors between the data points and the line, an approach known as \emph{least squares}.
The least squares solution can be obtained analytically when fitting a linear function, or through a number of optimization methods (such as Conjugate Gradient) when fitting a nonlinear function.

The method of \emph{least absolute deviations} (LAD) also seeks to find a best fit line to a set of data, but the error between the data and the line is measured differently.
In particular, suppose we have a set of data points $(y_1, \x_1), (y_2, \x_2), \ldots, (y_m, \x_m)$, where $y_i \in \mathbb{R}$, $\x_i \in \mathbb{R}^n$ for $i = 1, 2, \ldots, m$.
Here, the $\x_i$ vectors are the \emph{explanatory variables} and the $y_i$ values are the \emph{response variables}, and we assume the following linear model:
\[
y_i = \Beta\trp \mathbf{x}_i + b, \qquad i = 1, 2, \ldots, m,
\]
where $\Beta\in\mathbb{R}^n$ and $b \in \mathbb{R}$.
The error between the data and the proposed linear model is given by
\[
\sum_{i=1}^n |\Beta\trp \mathbf{x}_i + b - y_i|,
\]
and we seek to choose the parameters $\Beta, b$ so as to minimize this error.

\subsection*{Advantages of LAD}
The most prominent difference between this approach and least squares is how they respond to outliers in the data.
Least absolute deviations is robust in the presence of outliers, meaning that one (or a few) errant data points won't severely affect the fitted line.
Indeed, in most cases, the best fit line is guaranteed to pass through at least two of the data points.
This is a desirable property when the outliers may be ignored (perhaps because they are due to measurement error or corrupted data).
Least squares, on the other hand, is much more sensitive to outliers, and so is the better choice when outliers cannot be dismissed.
See Figure \ref{fig:leastAbsDev}.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/leastAbsDev.pdf}
\caption{Fitted lines produced by least absolute deviations (top) and least squares (bottom). The presence of an outlier accounts for the
stark difference between the two lines.}
\label{fig:leastAbsDev}
\end{figure}

While least absolute deviations is robust with respect to outliers, small horizontal perturbations of the data points can lead to very different fitted lines.
Hence, the least absolute deviations solution is less stable than the least squares solution.
In some cases there are even infinitely many lines that minimize the least absolute deviations error term.
However, one can expect a unique solution in most cases.

The least absolute deviations solution arises naturally when we assume that the residual terms $\Beta\trp \mathbf{x}_i + b - y_i$ have a particular statistical distribution (the Laplace distribution).
Ultimately, however, the choice between least absolute deviations and least squares depends on the nature of the data at hand, as well as your own good judgment.

% =============================================================================
% =============================================================================
% EDITED TO HERE ==============================================================
% =============================================================================
% =============================================================================

\subsection*{LAD as a Linear Program}
We can formulate the least absolute deviations problem as a linear program, and then solve it using our interior point method.
For $i = 1, 2, \ldots, m$ we introduce the artificial variable $u_i$ to take the place of the error term $|\Beta\trp \mathbf{x}_i + b - y_i|$, and we require this variable to satisfy $u_i \geq |\Beta\trp \mathbf{x}_i + b - y_i|$.
This constraint is not yet linear, but we can split it into an equivalent set of two linear constraints:
\begin{align*}
u_i &\geq \Beta\trp \mathbf{x}_i + b - y_i,\\
u_i &\geq y_i - \Beta\trp \mathbf{x}_i - b.
\end{align*}
The $u_i$ are implicitly constrained to be nonnegative.

Our linear program can now be stated as follows:
\begin{align*}
\text{minimize }\qquad &\sum_{i=1}^m u_i\\
\text{subject to }\qquad &u_i \geq \Beta\trp \mathbf{x}_i + b - y_i,\\
&u_i \geq y_i - \Beta\trp \mathbf{x}_i - b.
\end{align*}
Now for each inequality constraint, we bring all variables ($u_i, \Beta, b$) to the left hand side and introduce a nonnegative slack variable to transform the constraint into an equality:
\begin{align*}
u_i  - \Beta\trp \mathbf{x}_i - b - s_{2i-1}&= -y_i,\\
u_i +\Beta\trp \mathbf{x}_i + b - s_{2i}&= y_i,\\
s_{2i-1}, s_{2i}&\geq 0.
\end{align*}

Notice that the variables $\Beta, b$ are not assumed to be nonnegative, but in our interior point method, all variables are assumed to be nonnegative.
We can fix this situation by writing these variables as the difference of nonnegative variables:
\begin{align*}
  \Beta &= \Beta_1 - \Beta_2,\\
  b &= b_1 - b_2,\\
  &\Beta_1, \Beta_2\succeq \0; b_1, b_2 \geq 0.
\end{align*}
Substituting these values into our constraints, we have the following system of constraints:
\begin{align*}
u_i  - \Beta_1\trp \mathbf{x}_i + \Beta_2\trp \mathbf{x}_i - b_1 + b_2 - s_{2i-1}&= -y_i,\\
u_i + \Beta_1\trp \mathbf{x}_i - \Beta_2\trp \mathbf{x}_i + b_1 - b_2 - s_{2i}&= y_i,\\
\Beta_1, \Beta_2 \succeq \0; u_i, b_1, b_2, s_{2i-1}, s_{2i}&\geq 0.
\end{align*}
Writing $\mathbf{y} = (-y_1, y_1, -y_2, y_2, \ldots, -y_m, y_m)\trp $ and $\Beta_i = (\beta_{i,1}, \ldots, \beta_{i,n})\trp $ for $i = \{1, 2\}$, we can aggregate all of our variables into one vector as follows:
\[
\mathbf{v} = (u_1,\ldots, u_m, \beta_{1,1},\ldots, \beta_{1,n}, \beta_{2,1},\ldots, \beta_{2,n}, b_1, b_2, s_1,\ldots,s_{2m})\trp .
\]
Defining $\c = (1, 1, \ldots, 1, 0, \ldots, 0)\trp $ (where only the first $m$ entries are equal to 1), we can write our objective function as
\[
\sum_{i=1}^m u_i = \mathbf{c}\trp \mathbf{v}.
\]
Hence, the final form of our linear program is:
\begin{align*}
  \text{minimize }\qquad &\mathbf{c}\trp \mathbf{v}\\
  \text{subject to }\qquad A\mathbf{v} &= \mathbf{y},\\
  \mathbf{v} &\succeq \0,
\end{align*}
where $A$ is a matrix containing the coefficients of the constraints.
Our constraints are now equalities, and the variables are all nonnegative, so we are ready to use our interior point method to obtain the solution.

\subsection*{LAD Example}

Consider the following example.
We start with an array \li{data}, each row of which consists of the values $y_i, x_{i,1},\ldots,x_{i,n}$, where $\mathbf{x}_i = (x_{i,1}, x_{i,2}, \ldots, x_{i,n})\trp $.
We will have $3m + 2(n+1)$ variables in our linear program.
Below, we initialize the vectors $\mathbf{c}$ and $\mathbf{y}$.
\begin{lstlisting}
>>> m = data.shape[0]
>>> n = data.shape[1] - 1
>>> c = np.zeros(3*m + 2*(n + 1))
>>> c[:m] = 1
>>> y = np.empty(2*m)
>>> y[::2] = -data[:, 0]
>>> y[1::2] = data[:, 0]
>>> x = data[:, 1:]
\end{lstlisting}

The hardest part is initializing the constraint matrix correctly.
It has $2m$ rows and $3m + 2(n+1)$ columns.
Try writing out the constraint matrix by hand for small $m, n$, and make sure you understand why the code below is correct.
% TODO: Do this with NumPy stacking.
\begin{lstlisting}
>>> A = np.ones((2*m, 3*m + 2*(n + 1)))
>>> A[::2, :m] = np.eye(m)
>>> A[1::2, :m] = np.eye(m)
>>> A[::2, m:m+n] = -x
>>> A[1::2, m:m+n] = x
>>> A[::2, m+n:m+2*n] = x
>>> A[1::2, m+n:m+2*n] = -x
>>> A[::2, m+2*n] = -1
>>> A[1::2, m+2*n+1] = -1
>>> A[:, m+2*n+2:] = -np.eye(2*m, 2*m)
\end{lstlisting}

Now we can calculate the solution by calling our interior point function.
\begin{lstlisting}
>>> sol = interiorPoint(A, y, c, niter=10)[0]
\end{lstlisting}

The variable \li{sol}, however, holds the value for the vector
\[
\mathbf{v} = (u_1,\ldots, u_m, \beta_{1,1},\ldots, \beta_{1,n}, \beta_{2,1},\ldots, \beta_{2,n}, b_1, b_2, s_1,\ldots,s_{2m+1})\trp .
\]
We extract values of $\Beta = \Beta_1-\Beta_2$ and $b = b_1 - b_2$ with the following code:
\begin{lstlisting}
>>> beta = sol[m:m+n] - sol[m+n:m+2*n]
>>> b = sol[m+2*n] - sol[m+2*n+1]
\end{lstlisting}

\begin{problem} % Least Total Deviations Problem ------------------------------
The file \li{simdata.txt} contains two columns of data.
The first gives the values of the response variables ($y_i$), and the second column gives the values of the explanatory variables ($\x_i$).
Find the least absolute deviations line for this data set, and plot it together with the data.
Plot the least squares solution as well to compare the results.
\begin{lstlisting}
>>> from scipy.stats import linregress
>>> slope, intercept = linregress(data[:,1], data[:,0])[:2]
>>> domain = np.linspace(0,10,200)
>>> plt.plot(domain, domain*slope + intercept)
\end{lstlisting}
% \begin{figure}[H] % solution to the problem with new simdata.
% \centering
% \includegraphics[width=\textwidth]{figures/LADprob.pdf}
% \label{fig:LADprob}
% \end{figure}
\end{problem}


\lab{Complex Integration}{Integration in the Complex Plane}

\objective{Understand some simple uses of residues and singularities in the complex plane.}

In the previous lab, we looked a visual representations of the roots and singularities of complex functions. Here we look more at singularities and what they can be used to compute.

\section*{Laurent Series and Singular Points}

We will now introduce another form of series representation of functions.
A Laurent series of a function is a series of the form
\[\sum_{n= -\infty}^{\infty} a_n (z-z_0)^n\]
It can be proven that
\[a_n = \frac{1}{2\pi i} \int_C \frac{f(z)}{(z-z_0)^{n+1}} dz\]
where $C$ is a contour which passes counterclockwise around the singularity exactly once.
When $f$ does not have a singularity at $z_0$ this representation degenerates to a normal Taylor Series (with the derivatives evaluated by the formula for the $n$th derivative of an analytic function).
These sorts of series are considered in greater detail in the text.
The built in function \li{sympy.series} can evaluate the series expansion of a function at a singularity, for example
\begin{lstlisting}
import sympy as sy
z = sy.Symbol('z')
(1/sy.sin(z)).series(z,0,8)
\end{lstlisting}

The Laurent series representation provides a simple way to classify singularities in the complex plane.
Isolated singular points can be classified as removable singular points, poles, or essential singular points.
These definitions will also be discussed in greater detail in the text.
Here we will show another method for visualizing complex functions.
In Lab \ref{Lab:complex_intro} we presented color plots as a useful method for visualizing functions in the complex plane.
Here we will also show how to use surface plots to visualize the modulus of complex functions.

Now you have to take care when a surface plot is about a singularity.  We must account for the fact that the function is not going to be defined at all of the points we use in our graph.
We will also have to limit the $z$ axis on the plot to avoid creating a plot that is dominated exclusively by the extremely large and extremely small values of the function.
Plotting libraries like Mayavi and Matplotlib do allow thresholding of 3D plots via the use of floating point values of \li{nan}, but doing this may result in graphs having jagged edges where they have been cut.
We can avoid the jagged edges by artificially adding a small lip of constant values to the plot as well.
This can be done with Matplotlib as follows:
\begin{lstlisting}
import numpy as np
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D #needed to create 3d plots

x_bounds, y_bounds = (-1.,1.), (-1.,1.)
res = 400
# Set the threshold at which to cut the 'z' values
threshold = 2.
# space in 'z' values to use to create a lip on the plot
lip = .5
x = np.linspace(x_bounds[0], x_bounds[1], res)
y = np.linspace(y_bounds[0], y_bounds[1], res)
X, Y = np.meshgrid(x, y, copy=False)
Z = 1 / (X + 1.0j * Y)
Z = np.abs(Z)
# Set the values between threshold and
# threshold + lip to be equal to threshold.
# This forms a somewhat more concrete
# edge at the top of the plot.
Z[(threshold+lip>Z)&(Z>threshold)] = threshold
# Do the same thing for the negative restriction on 'z'.
Z[(-threshold-lip<Z)&(Z<-threshold)] = -threshold
# Set anything that is larger to np.nan so it doesn't get plotted.
Z[np.absolute(Z) >= threshold + lip] = np.nan
# Now actually plot the data.
fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(X, Y, Z, cmap="coolwarm")
plt.show()
\end{lstlisting}
You may notice that this still leaves some very jagged edges around the singularity.
Much more detailed code would be needed to obtain good surface plots around a singular point.

Figures \ref{fig:inv_surfaces} show surface plots of some relatively well-behaved singular points.

\begin{figure}
\begin{subfigure}{.5\textwidth}
\includegraphics[width=\textwidth]{absinvz.png}
\end{subfigure}
\begin{subfigure}{.5\textwidth}
\includegraphics[width=\textwidth]{absinvz2.png}
\end{subfigure}
\caption{Surface plots of the absolute value of $\frac{1}{z}$ and $\frac{1}{z^2}$ about the origin.}
\label{fig:inv_surfaces}
\end{figure}

\begin{problem}
Write a function that takes in a function, x and y bounds, and a resolution and plots the modulus of the function as a 3d plot. Remember to take in account that it may be around a singularity.
\end{problem}

\begin{comment}
Color plots, when used in their default form suffer from similar limitations.
On the other hand, we can do things like taking the absolute value, then scaling the values of a function logarithmically to mitigate some of the effect that the rapid growth of the plot values has on the colors in the plot.
We can also allow the color map used by the plotting library to repeat itself as the values of the function get larger and larger.
This can be done by taking the sine of the values.
Combining these things together, we obtain expressions like $\sin\left(\log\left|\text{Re}\left(Z\right)\right|\right)$.

\begin{warn}
When generating color plots like this, you will have to check for floating point values of \li{nan} and \li{inf}.
One easy way to get rid of them is to set those values to zero.
\end{warn}

\begin{problem}
Write a function that produces color plots of $\sin\left(\log\left|f\left(z\right)\right|\right)$ for a given function $f$.
Have it accept bounds for the interval where you want the plot generated.
Also have an argument that tells the function whether to use the real part, imaginary part, or the absolute value of the function.
\end{problem}
\end{comment}

\section*{Residues}
The number $a_{-1}$ in the Laurent expansion for a function $f$ at a point $z_0$ is called the Residue of $f$ at $z_0$.
The formula for the coefficients of the Laurent series provides one way to compute residues.
It is sometimes easier to evaluate a residue using limits or some other formula.
One good way to do this is to use the following formula:

Let $f(z)=\frac{p(z)}{q(z)}$ and let $p$ and $q$ be holomorphic at $z_0$. Let $p(z_0) \neq 0$ and $q(z_0)=0$.
Suppose that $z_0$ is a pole of order $1$ of $f$.
Then
\[\Res{z=z_0} f(z) = \frac{p(z_0)}{q'(z_0)}\]

A natural consequence of the Laurent series expansion of $f(z)$ and $f'(z)$ at a pole $z_0$ is that, where $d$ is the degree of the pole at $z_0$,
\[- \Res{z=z_0} \frac{f'(z)}{f(z)} = d\]
This is a useful bit of information that may be used to simplify computation of residues or of the Laurent series expansion of a function since it allows us to avoid evaluating a long series of integrals we already know will evaluate to $0$.

There is also a natural relationship between the residues of a function at its poles and its partial fraction decomposition.
Let $f = \frac{p}{q}$ where $p$ and $q$ are polynomials, $q$ has no repeated roots, and the degree of $p$ has degree less than the degree of $q$.
This means that $f$ has a partial fraction representation
\[f = \sum \frac{c_i}{z - z_i}\]
where the $c_i$ are appropriately chosen coefficients and $z_i$ are the distinct zeros of $q$.
Consider what happens when we take the residue of this sum.
Since integrals distribute over sums, so do residues, so we may say
\[\Res{z=z_i} f = \sum \Res{z=z_i} \frac{c_i}{z - z_i}\]
Since the zeros of $q$ are distinct, and the function $\frac{1}{z - z_i}$ is holomorphic wherever $z \neq z_i$, we see that
\[\Res{z=z_i} f = c_i \Res{z=z_i} \frac{1}{z-z_i} = c_i\]
This means that we can use residues to compute partial fraction decompositions, so long as the function in the denominator does not have repeated roots.

\begin{problem}
Write a Python function that, given polynomial objects $p$ and $q$, computes the partial fraction decomposition of the function $\frac{p}{q}$.
Assume that the degree of $p$ is less than the degree of $q$ and that $q$ has no repeated roots.
Return two arrays.
The first should contain the coefficients in the partial fraction decomposition.
The second should contain the corresponding zeros of $q$.

The \li{poly1d} object in NumPy makes it easy to work with polynomials:
\begin{lstlisting}
>>> from numpy import poly1d
# To represent the polynomial y = x^2 - 1, we pass in a list of its coefficients, highest power to lowest power.
>>> my_polynomial = poly1d([1., 0., -1])
>>> my_polynomial.roots
array([ 1., -1.]) #The roots are 1 and -1
>>> my_polynomial.deriv()
poly1d([ 2.,  0.]) #The derivative is the polynomial y = 2x
\end{lstlisting}
\end{problem}

\section*{Evaluating Indefinite Integrals Using Residues}

One convenient use of residues is the evaluation of integrals that are difficult to evaluate symbolically in other ways.
Often, when we cannot directly assign a value to one of these integrals, residues can still help us evaluate the Cauchy principal value of the integral.
Recall that, for an integral $\int_{-\infty}^{\infty} f(x)dx$, the Cauchy principal value is $\lim_{r\to \infty} \int_{-r}^{r} f(x) dx$.
This limit may exist, even though the integral itself may not.
The methods for using residues to evaluate such integrals will be discussed in greater details in the text.
Here we consider one example.

\begin{figure}
\includegraphics[width=\textwidth]{contour1.pdf}
\caption{A contour used for integration using residues.}
\label{complexint:c1}
\end{figure}

Consider the integral $\int_{-\infty}^{\infty}\frac{z^2}{z^4+1}$.
Let $f(z)=\frac{z^2}{z^4+1}$.
Notice that this function has poles at $e^{\frac{\pi i}{4}}$, $e^{\frac{3\pi i}{4}}$, $e^{\frac{5\pi i}{4}}$, and $e^{\frac{7\pi i}{4}}$.
For notation, let these be $p_0$, $p_1$, $p_2$, and $p_3$.
For some real $R>1$, consider the contour $C$ from $-R$ to $R$ and counterclockwise along the circle centered at $0$ of radius $R$ back to -$R$.

This contour (a semi circle) is shown in Figure \ref{complexint:c1} with $R = 2$.
Let $A$ be this second portion of $C$.
Since $R>1$, $p_0$ and $p_1$ lie inside the contour, and we have
\[\int_C f(z)dz = 2\pi i (\Res{z=p_0} f(z) +\Res{z=p_1} f(z))\]
So, rewriting, we have
\[\int_{-R}^R f(z) dz = 2\pi i (\Res{z=p_0} f(z) +\Res{z=p_1} f(z)) - \int_A f(z) dz\]
so
\[\int_{-\infty}^{\infty} f(z) dz = \lim_{R\to \infty} \int_{-R}^R f(z) dz = 2\pi i (\Res{z=p_0} f(z) +\Res{z=p_1} f(z)) - \lim_{R\to \infty} \int_A f(z) dz\]
We would like to show that $\lim_{R\to\infty} \int_A f(z) dz = 0$, so note that on $A$, $\abs{z}=R$.
With some effort, it follows from the triangle inequality that $\abs{z^4+1}\geq \abs{\abs{z}^4-1} = R^4 -1$, so we have that
\[\abs{\int_A f(z) dz}\leq \int_A \abs{f(z)} dz \leq \int_A \frac{R^2}{R^4 -1}dz = \pi R \frac{R^2}{R^4-1}\]
so $\lim_{R\to\infty} \int_A f(z) dz = 0$ as desired.
This then implies that
\[\int_{-\infty}^{\infty} f(z) dz = 2\pi i (\Res{z=p_0} f(z) +\Res{z=p_1} f(z))\]
Evaluating the residues at $p_0$ and $p_1$ we have
\[\int_{-\infty}^{\infty} f(z) dz = \frac{\pi}{\sqrt{2}}\]

\begin{comment}
If a function has a singularity on the real line, we can often still evaluate the value of $\int_{-\infty}^{\infty} f(z) dz$ using a similar argument as before, but we must now indent the path along the real axis around the singularity, then, as we take the limit as our outer contour moves out toward infinity, we can let the small contour around the singularity approach the singularity.
An example of a contour like this is shown in Figure \ref{complexint:c2}.
It is centered at the origin with $R=2$ on the outer circle and $R=\frac{1}{2}$ on the inner circle.

\begin{figure}
\includegraphics[width=\textwidth]{contour2.pdf}
\caption{An example of a contour that has been modified to avoid a singularity at the origin.}
\label{complexint:c2}
\end{figure}

The following is a useful theorem involving these ``indented path" methods.
\begin{theorem}
Consider a function $f$ with a pole of order $1$ at $z=x_0$ with a Laurent series representation in a punctured disk of radius $R$ about $x_0$ and residue $B_0$ at $x_0$.
Let $C_r$ be the upper half of a circle $\abs{z-x_0}=r$ where $r<R$ oriented in the clockwise direction, then
\[\lim_{r\to 0} \int_{C_r} f(z) dz = - B_0 \pi i\]
\end{theorem}
As a consequence of this,
\end{comment}
We will not consider functions that have a singularity on the real line. So for a function $f$ on $\mathbb{C}$ with only zeros of at most order $1$ on $\mathbb{R}$, where $A$ is the sum of the residues of $f$ on the upper half plane,
\[\int_{-\infty}^{\infty} f(z) dz = 2\pi i A\]
whenever this integral exists.
When we can say that the integral of $f$ over the upper half of a circle centered at $0$ goes to $0$ as the radius of the circle increases and we can also say that the Cauchy principal value of the integral of $f$ over $\mathbb{R}$ exists, this formula will give us a proper numerical value for the Cauchy principal value of the integral of $f$ over $\mathbb{R}$.

\begin{problem}
Write a function that computes the sum $2\pi i A$ (where $A$ is defined as above) for a function $f$ of the form $\frac{p}{q}$ where $p$ and $q$ are polynomials of the same form as in Problem 2.
\end{problem}

Integration techniques using residues can also be extended to integration around branch points, some types of integrals involving sines and cosines, inverse Laplace transforms, and many other difficult integration problems.

\begin{problem}
In the text, the zero and pole counting formula was stated and proved.
An immediate consequence of that theorem is that, for a meromorphic function $f$ and a positively oriented simple closed curve $\gamma$ that does not pass through any roots of $f$,
\[\int_\gamma \frac{f'\left(z\right)}{f\left(z\right)} dz = 2 \pi i \left(a - b\right)\]
where $a$ is the number of zeros on the interior of $\gamma$ (counting multiplicities) and $b$ is the number of poles of $f$ on the interior of $\gamma$ (also counting multiplicities).

This formula gives another possible way to count the number of zeros of a polynomial on the interior of a given contour along with looking at the color plots and manually counting them as taught in the previous lab.

Write a Python function that counts the number of zeros of a polynomial on the interior of the unit circle then plots the color plot of the function on the unit square.
\end{problem}


% Created 2021-11-08 Mon 08:13
% Intended LaTeX compiler: pdflatex
\documentclass[presentation,aspectratio=169]{beamer}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{grffile}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{textcomp}
\usepackage{amssymb}
\usepackage{capt-of}
\usepackage{hyperref}
\usepackage{khpreamble}
\usepackage{amssymb}
\usepackage{tcolorbox}
\usepackage{pgfplots}
\usepgfplotslibrary{groupplots}
\DeclareMathOperator{\shift}{q}
\DeclareMathOperator{\diff}{p}
\usetheme{default}
\author{Kjartan Halvorsen}
\date{\today}
\title{Stability}
\hypersetup{
 pdfauthor={Kjartan Halvorsen},
 pdftitle={Stability},
 pdfkeywords={},
 pdfsubject={},
 pdfcreator={Emacs 26.3 (Org mode 9.4.6)},
 pdflang={English}}
\begin{document}

\maketitle

\section{Intro}
\label{sec:orgd9a4d2f}

\begin{frame}[label={sec:orgfa2845f}]{Hard disk drive arm model}
\begin{center}
\includegraphics[width=0.4\linewidth]{../../figures/diskdrive.png}
\end{center}

{\tiny "Laptop-hard-drive-exposed" by Evan-Amos - Own work. Licensed under CC BY-SA 3.0 via Commons }
\end{frame}

\begin{frame}[label={sec:orgf561952}]{Sampling the hard disk drive arm model}
\footnotesize
\[ H(z) = \frac{z-1}{z} \ztrf{\mathcal{L}^{-1}\{ \frac{G(s)}{s} \}} \]
\begin{center}
\includegraphics[height=0.25\textheight]{../../figures/diskdrive.png}
\begin{tikzpicture}[scale=0.7, node distance=2.2cm, block/.style={rectangle, draw, minimum height=12mm, minimum width=12mm}, sumnode/.style={circle, draw, inner sep=1pt}]
\footnotesize
  \node[coordinate] (input) {};
  \node[block, right of=input] (plant) {$\frac{1}{Js^2}$};
  \node[coordinate, right of=plant] (output) {};
  \draw[->] (input) -- node[above] {$u$} (plant);
  \draw[->] (plant) -- node[above] {$y$} (output);

  \node at (-2, 1) {$J\ddot{y} = u$};
  \end{tikzpicture}
\end{center}

\pause

Which of the below graphs show the sampled \alert{step-response} of the system?

\begin{center}
\begin{tikzpicture}[scale=0.85]
       \footnotesize

       \begin{groupplot}[group style={group size=2 by 2, vertical sep=1.2cm, horizontal sep=1.2cm, vertical sep=4mm},
       width=7cm,
       height=2.6cm,
       %xlabel={$t$},
       ylabel={$y(kh)$},
       xmin=-0.5,
       xmax=8.5,
       ytick = \empty,
       xtick = 0,
       ]
       \nextgroupplot
       \addplot[red, thick, ycomb, mark=*,domain=0:8, samples=9]  {x};

       \nextgroupplot
       \addplot[red, thick, ycomb, mark=*, domain=0:8, samples=9]  {1};

       \nextgroupplot
       \addplot[red, thick, ycomb, mark=*, domain=0:8, samples=9]  {2*exp(-x/2)};

       \nextgroupplot
       \addplot[red, thick, ycomb, mark=*,domain=0:8, samples=9]  {pow(x,2)};

     \end{groupplot}
     \node[blue!60] at (group c1r1.center) {\huge 1};
       \node[blue!60] at (group c2r1.center) {\huge 2};
       \node[blue!60] at (group c1r2.center) {\huge 3};
       \node[blue!60] at (group c2r2.center) {\huge 4};
       \end{tikzpicture}

     \end{center}
\end{frame}

\begin{frame}[label={sec:org848a1ab}]{Sampling the hard disk drive arm model}
\footnotesize
\[ H(z) = \frac{z-1}{z} \ztrf{\mathcal{L}^{-1}\{ \frac{G(s)}{s} \}} \]
\begin{center}
\includegraphics[height=0.25\textheight]{../../figures/diskdrive.png}
\begin{tikzpicture}[scale=0.7, node distance=2.2cm, block/.style={rectangle, draw, minimum height=12mm, minimum width=12mm}, sumnode/.style={circle, draw, inner sep=1pt}]
\footnotesize
  \node[coordinate] (input) {};
  \node[block, right of=input] (plant) {$\frac{1}{Js^2}$};
  \node[coordinate, right of=plant] (output) {};
  \draw[->] (input) -- node[above] {$u$} (plant);
  \draw[->] (plant) -- node[above] {$y$} (output);

  \node at (-2, 1) {$J\ddot{y} = u$};
  \end{tikzpicture}
\end{center}

Sampled step-response: \(y(kh) = \frac{1}{2J}(kh)^2\)
\[ k^2 \qquad \overset{\mathcal{Z}}{\longleftrightarrow} \qquad \frac{z(z+1)}{(z-1)^3}\]

\pause

Which of the below pulse-transfer functions corresponds to the discretized hard disk drive model?
\begin{center}
\begin{tabular}{rrr}
1 & 2 & 3\\
\(H(z)=\frac{h^2z}{2J(z+1)^2}\) & \(H(z)=\frac{h^2(z+1)}{2Jz^2}\) & \(H(z)=\frac{h^2(z+1)}{2J(z-1)^2}\)\\
\end{tabular}
\end{center}
\end{frame}

\section{Block-diagram algebra}
\label{sec:org029e8b2}
\begin{frame}[label={sec:orgbb5598b}]{Block-diagram algebra}
\alert{Same rules as in the continuous-time case!}
\begin{center}
\begin{tikzpicture}
\tikzset{node distance=2cm,
    block/.style={rectangle, draw, minimum height=12mm, minimum width=14mm},
    sumnode/.style={circle, draw, inner sep=2pt}
}

  \node[coordinate] (input) {};
  \node[block, right of=input] (TR) {$F_f(z)$};
  \node[sumnode, right of=TR, node distance=30mm] (sum) {\tiny $\sum$};
  \node[block,right of=sum, node distance=30mm] (plant) {$H(z)$};
  %\node[sumnode, right of=plant, node distance=30mm] (sumdist) {$\sum$};
  %\node[coordinate, above of=sumdist, node distance=15mm] (dist) {};
  %\node[coordinate, right of=sumdist, node distance=15mm] (measure) {};
  \node[coordinate, right of=plant, node distance=30mm] (output) {};
  \node[coordinate, right of=plant, node distance=22mm] (measure) {};
  %\node[sumnode,below of=measure, node distance=25mm] (sumnoise) {$\sum$};
  %\node[coordinate, right of=sumnoise, node distance=15mm] (noise) {};
  \node[block,below of=plant, node distance=20mm] (SR) {$F_b(z)$};
  \draw[->] (input) -- node[above, pos=0.2] {$y_{ref}(k)$} (TR);
  \draw[->] (TR) -- node[above] {$u_1(k)$} (sum);
  \draw[->] (sum) -- node[above] {$u(k)$} (plant);
  \draw[->] (plant) -- node[at end, above] {$y(k)$} (output);
  \draw[->] (measure) |- (SR);
  \draw[->] (SR) -| (sum) node[right, pos=0.8] {$u_2(k)$} node[left, pos=0.96] {$-$};
\end{tikzpicture}
\end{center}
With \[U(z) = U_1(z) - U_2(z) = F_f(z)Y_{ref}(z) - F_b(z)Y(z), \quad \text{and}\]
\[ Y(z) = H(z)U(z), \quad \text{we obtain} \]
\[ Y(z) = \underbrace{\frac{F_f(z)H(z)}{1 + F_b(z)H(z)}}_{H_c{z}} Y_{ref}(z). \]
\end{frame}

\begin{frame}[label={sec:org4ace140}]{Block-diagram algebra - steps in detail}
With \[U(z) = U_1(z) - U_2(z) = F_f(z)Y_{ref}(z) - F_b(z)Y(z), \quad \text{and}\]
\[ Y(z) = H(z)U(z), \quad \text{we obtain} \]
\[ Y(z) = H(z)U(z) = H(z)\left(F_f(z)Y_{ref}(z) - F_b(z)Y(z)\right)\]
Move all terms with \(Y\) to the left side:
\[ Y(z) + H(z)F_b(z)Y(z) = H(z)F_f(z)Y_{ref}(z)\]
\[ Y(z)\big(1 + H(z)F_b(z)\big) = H(z)F_f(z)Y_{ref}(z)\]
\[ Y(z) = \frac{H(z)F_f(z)}{1 + H(z)F_b(z)}Y_{ref}(z)\]
\end{frame}

\begin{frame}[label={sec:org7f81524}]{Stability for the closed-loop system}
\[ Y(z) = \frac{F_f(z)H(z)}{1 + F_b(z)H(z)} Y_{ref}(z). \]

\begin{tcolorbox}
Stability requires that all poles of the system, i.e. all solutions to the characteristic equation
\[ 1 + F_b(z)H(z) = 0\]
are located inside the unit circle of the  z-plane.
\end{tcolorbox}
\end{frame}

\begin{frame}[label={sec:orgd4efd26}]{Stability for the disk drive arm}
\small
Case \(\frac{h^2}{J} = 1\). Suggested controller:

\begin{center}
\begin{tikzpicture}
\tikzset{node distance=2cm,
    block/.style={rectangle, draw, minimum height=12mm, minimum width=14mm},
    sumnode/.style={circle, draw, inner sep=2pt}
}

  \node[coordinate] (input) {};
  \node[block, right of=input] (TR) {$F_f(z) = 0.2K$};
  \node[sumnode, right of=TR, node distance=30mm] (sum) {\tiny $\sum$};
  \node[block,right of=sum, node distance=30mm] (plant) {$H(z) = \frac{z+1}{2(z-1)^2}$};
  %\node[sumnode, right of=plant, node distance=30mm] (sumdist) {$\sum$};
  %\node[coordinate, above of=sumdist, node distance=15mm] (dist) {};
  %\node[coordinate, right of=sumdist, node distance=15mm] (measure) {};
  \node[coordinate, right of=plant, node distance=30mm] (output) {};
  \node[coordinate, right of=plant, node distance=22mm] (measure) {};
  %\node[sumnode,below of=measure, node distance=25mm] (sumnoise) {$\sum$};
  %\node[coordinate, right of=sumnoise, node distance=15mm] (noise) {};
  \node[block,below of=plant, node distance=20mm] (SR) {$F_b(z)=K\frac{z-0.8}{z}$};
  \draw[->] (input) -- node[above, pos=0.2] {$y_{ref}(k)$} (TR);
  \draw[->] (TR) -- node[above] {$u_1(k)$} (sum);
  \draw[->] (sum) -- node[above] {$u(k)$} (plant);
  \draw[->] (plant) -- node[at end, above] {$y(k)$} (output);
  \draw[->] (measure) |- (SR);
  \draw[->] (SR) -| (sum) node[right, pos=0.8] {$u_2(k)$} node[left, pos=0.96] {$-$};
\end{tikzpicture}
\end{center}

\alert{Characteristic equation}
\begin{align*}
1 + H(z)F_b(z) &= 0\\
1 + \frac{z+1}{2(z-1)^2}K\frac{z-0.8}{z} &= 0\\
(z-1)^2z + \frac{K}{2}(z+1)(z-0.8) &= 0
\end{align*}

\pause

\alert{Is the system stable for all gains \(K\)?}
\end{frame}

\begin{frame}[label={sec:org5deaacc}]{Stability for the disk drive arm}
\alert{Pair activity} Complete the root-locus below
   \[(z-1)^2z + \frac{K}{2}(z+1)(z-0.8) = 0\]
\begin{center}
  \begin{tikzpicture}[scale=2.5]
    \draw[->] (-1.2, 0) -- (1.2,0);
    \draw[->] (0, -1.2) -- (0,1.2);
    \node[red, pin=45:{2 plant poles}] at (1,0) {\large $\times$};
    \node[red, pin=135:{controller pole}] at (0,0) {\large $\times$};
    \node[green!70!black, pin=-145:{controller zero}] at (0.8,0) {\Large $\circ$};
    \node[green!70!black, pin=-145:{plant cero}] at (-1,0) {\Large $\circ$};
    \node at (0.8, -0.2) {$0.8$};
    \node at (1, -0.2) {$1$};
    \draw[domain=0:360, samples=361, dashed] plot ({cos(\x)}, {sin(\x)});
    \node[coordinate, pin=60:{$|z|=1$}] at (0.5, 0.87) {};
  \end{tikzpicture}
\end{center}
\end{frame}

\section{Estabilidad para el control del brazo del disko duro}
\label{sec:orgb4848e4}
\end{document}


\chapter{Derivation of twist averaging efficiency}
\label{sec:app_ta_efficiency}
In this appendix we derive the relative statistical efficiency of
twist averaging with an irreducible (weighted) set of k-points
versus using uniform weights over an unreduced set of k-points
(e.g., a full Monkhorst-Pack mesh).

Consider the weighted average of a set of statistical variables
$\{x_m\}$ with weights $\{w_m\}$:
\begin{align}
  x_{TA} = \frac{\sum_mw_mx_m}{\sum_mw_m}\:.
\end{align}
If produced by a finite QMC run at a set of
twist angles/k-points $\{k_m\}$, each variable mean $\mean{x_m}$
has a statistical error bar $\sigma_m$, and we can also obtain
the statistical error bar of the mean of the twist-averaged
quantity $\mean{x_{TA}}$:
\begin{align}
  \sigma_{TA} = \frac{\left(\sum_mw_m^2\sigma_m^2\right)^{1/2}}{\sum_mw_m}\:.
\end{align}
The error bar of each individual twist $\sigma_m$ is related to the
autocorrelation time $\kappa_m$,  intrinsic variance $v_m$, and the number
of postequilibration MC steps $N_{step}$ in the following way:
\begin{align}
  \sigma_m^2=\frac{\kappa_mv_m}{N_{step}}\:.
\end{align}
In the setting of twist averaging, the autocorrelation time and
variance for different twist angles are often very similar across
twists, and we have
\begin{align}
  \sigma_m^2=\sigma^2=\frac{\kappa v}{N_{step}}\:.
\end{align}
If we define the total weight as $W$, that is, $W\equiv\sum_{m=1}^Mw_m$,
for the weighted case with $M$ irreducible twists, the error bar is
\begin{align}
  \sigma_{TA}^{weighted}=\frac{\left(\sum_{m=1}^Mw_m^2\right)^{1/2}}{W}\sigma\:.
\end{align}
For uniform weighting with $w_m=1$, the number of twists is $W$ and
we have
\begin{align}
  \sigma_{TA}^{uniform}=\frac{1}{\sqrt{W}}\sigma\:.
\end{align}
We are interested in comparing the efficiency of choosing weights
uniformly or based on the irreducible multiplicity of each twist angle
for a given target error bar $\sigma_{target}$.  The number of MC
steps required to reach this target for uniform weighting is
\begin{align}
  N_{step}^{uniform} = \frac{1}{W}\frac{\kappa v}{\sigma_{target}^2}\:,
\end{align}
while for nonuniform weighting we have
\begin{align}\label{eq:weighted_step}
  N_{step}^{weighted} &= \frac{\sum_{m=1}^Mw_m^2}{W^2}\frac{\kappa v}{\sigma_{target}^2} \nonumber\:,\\
                  &=\frac{\sum_{m=1}^Mw_m^2}{W}N_{step}^{uniform}\:.
\end{align}
The MC efficiency is defined as
\begin{align}
  \xi = \frac{1}{\sigma^2t}\:,
\end{align}
where $\sigma$ is the error bar and $t$ is the total CPU time required
for the MC run.

The main advantage made possible by irreducible twist weighting is to
reduce the equilibration time overhead by having fewer twists and,
hence, fewer MC runs to equilibrate.  In the context of twist
averaging, the total CPU time for a run can be considered to be
\begin{align}
  t=N_{twist}(N_{eq}+N_{step})t_{step}\:,
\end{align}
where $N_{twist}$ is the number of twists, $N_{eq}$ is the number of MC steps required to reach equilibrium, $N_{step}$ is the number
of MC steps included in the statistical averaging as before,
and $t_{step}$ is the wall clock time required to complete a single
MC step. For uniform weighting $N_{twist}=W$; while for irreducible
weighting $N_{twist}=M$.

We can now calculate the relative efficiency ($\eta$) of irreducible vs.
uniform twist weighting with the aim of obtaining a target error bar
$\sigma_{target}$:
\begin{align}
  \eta &= \frac{\xi_{TA}^{weighted}}{\xi_{TA}^{uniform}} \nonumber\:, \\
       &= \frac{\sigma_{target}^2t_{TA}^{uniform}}{\sigma_{target}^2t_{TA}^{weighted}} \nonumber\:, \\
       &= \frac{W(N_{eq}+N_{step}^{uniform})}{M(N_{eq}+N_{step}^{weighted})} \nonumber\:, \\
       &= \frac{W(N_{eq}+N_{step}^{uniform})}{M(N_{eq}+\frac{\sum_{m=1}^Mw_m^2}{W}N_{step}^{uniform})} \nonumber\:, \\
       &= \frac{W}{M}\frac{1+f}{1+\frac{\sum_{m=1}^Mw_m^2}{W}f}\:.
\end{align}
In this last expression, $f$ is the ratio of the number of usable
MC steps to the number that must be discarded during equilibration
($f=N_{step}^{uniform}/N_{eq}$); and as before, $W=\sum_mw_m$, which is the number of
twist angles in the uniform weighting case.  It is important to recall
that $N_{step}^{uniform}$ in $f$ is defined relative to uniform weighting and is
the number of MC steps required to reach a target accuracy in the
case of uniform twist weights.

The formula for $\eta$ in the preceding can be easily changed with the help of
Equation~\ref{eq:weighted_step} to reflect the
number of MC steps obtained in an irreducibly weighted run
instead.  A good exercise is to consider runs that have already completed
with either uniform or irreducible weighting and calculate the
expected efficiency change had the opposite type of weighting been used.

The break even point $(\eta=1)$ can be found at a usable step fraction of
\begin{align}
  f=\frac{W-M}{M\frac{\sum_{m=1}^Mw_m^2}{W}-W}\:.
\end{align}

The relative efficiency $(\eta)$ is useful to consider in view of certain
scenarios.  An important case is where the number of required sampling
steps is no larger than the number of equilibration steps (i.e.,
$f\approx 1$).  For a very simple case with eight uniform twists with
irreducible multiplicities of $w_m\in\{1,3,3,1\}$ ($W=8$, $M=4$), the
relative efficiency of irreducible vs. uniform weighting is
$\eta=\frac{8}{4}\frac{2}{1+20/8}\approx 1.14$.  In this case,
irreducible weighting is about $14$\% more efficient than uniform weighting.

Another interesting case is one in which the number of sampling steps you can
reach with uniform twists before wall clock time runs out is small
relative to the number of equilibration steps ($f\rightarrow 0$).
In this limit, $\eta\approx W/M$.  For our eight-uniform-twist example, this would
result in a relative efficiency of $\eta=8/4=2$, making irreducible
weighting twice as efficient.

A final case of interest is one in which the equilibration time is short
relative to the available sampling time $(f\rightarrow\infty)$,
giving $\eta\approx W^2/(M\sum_{m=1}^Mw_m^2)$.  Again, for our simple example
we find $\eta=8^2/(4\times 20)\approx 0.8$, with uniform weighting being
$25$\% more efficient than irreducible weighting. For this example, the crossover point for irreducible weighting being
more efficient than uniform weighting is $f<2$, that is, when the
available sampling period is less than twice the length of the equilibration
period.  The expected efficiency ratio and crossover point should be checked
for the particular case under consideration to inform the choice between
twist averaging methods.


\subsection{Completeness of zero-order logic}

A theory is complete if all true formulae are included.

Note that there are three types of formulae in a theory.

\begin{itemize}
\item Tautologies (always true)
\item Refutable formulae (always false)
\item Satisfiable formulae which are not tautologies (true in some, but not all, interpretations).
\end{itemize}


\title{One-Bit Maximal Matching on a Ring}
%\author{}
\date{}

\begin{document}

\href{http://en.wikipedia.org/wiki/Matching_(graph_theory)}{Matching} is well-known problem from graph theory.
A matching is a set of edges that do not share any common vertices.
For a matching to be \textit{maximal}, it must be impossible to add another edge to the set without breaking the matching property.

\tableofcontents

\section{1-Bit Maximal Matching on a Ring}
\label{sec:MatchRing}

\quicksec{MatchRing}
(\href{\examplespec/MatchRing.prot}{spec},
\href{\examplesett/MatchRing.args}{args},
\href{\examplesynt/MatchRing.prot}{synt}
\href{\examplesoln/MatchRingOneBit.prot}{soln})

In the specification, the $e$ variables denote whether an edge is in the matching.
The invariant specified as a maximal matching, which can be restated as the following two conditions for the special case of a ring:
\begin{enumerate}
\item No two adjacent edges can be selected.
\item At least one of every three consecutive edges must be selected.
\end{enumerate}

\quicksec{Synthesis}
Processes cannot realistically write to edge variables, therefore the $e$ variables are marked as \ilcode{shadow} and their values must be derived from $x$ values owned by processes.
For an instructive look at how this works see \href{#sec:MatchRingOneBit}{the next section}, which derives the same protocol from a slightly different way of specifying the maximal matching property.

\quicksec{Stabilization Proof}
It is fairly easy to show that the 1-bit matching protocol is stabilizing.
First we will show that all executions terminate.
Then we will show that all silent states belong to the invariant.
From the \href{\examplesoln/MatchRingOneBit.prot}{protocol}, we see that each $P[i]$ has the following actions:
\begin{code}
(              x[i]==1 && x[i+1]==1 --> x[i]:=0; )
( x[i-1]==0 && x[i]==0 && x[i+1]==0 --> x[i]:=1; )
\end{code}

We can analyze the actions to see that the protocol is livelock-free.
The first action of $P[i]$ removes cases of $2$ consecutive $1$ values by changing the left value to $0$.
This may enable the second action of $P[i-1]$.
The second action of $P[i]$ removes cases of $3$ consecutive $0$ values by changing the middle value to $1$.
If $P[i]$ executes its section action neither it or its neighbors is enabled!
Therefore, actions may propagate by changing consecutive $1$s to $0$s, and some of the resulting $0$s may toggle back to $1$s, but the $1$s will not be consecutive.

Clearly the silent states are those where no $2$ consecutive $1$s exist and no $3$ consecutive $0$s exist.
That means a $1$ must occur at least every $3$ values and will be followed by a $0$.
We can therefore interpret a $1$ value followed by a $0$ value to mean that the edge between the two values is selected.
These edges are not consecutive and at least on of every $3$ will be selected, therefore it is a maximal matching.
\begin{code}
forall i <- Nat % N :
   x[i-1]==1 && x[i]==0               // P[i] matched with P[i-1]
|| x[i-1]==0 && x[i]==0 && x[i+1]==1  // P[i] is not matched
||              x[i]==1 && x[i+1]==0  // P[i] matched with P[i+1]
\end{code}

\section{3-State Maximal Matching on a Ring}
\label{sec:MatchRingThreeState}

\quicksec{MatchRingThreeState}
(\href{\examplespec/MatchRingThreeState.prot}{spec},
\href{\examplesett/MatchRingThreeState.args}{args},
\href{\examplesoln/MatchRingThreeState.prot}{soln})

Matching can also be reasoned about in terms of processes.
Allow each process $P[i]$ in a ring to point to $P[i-1]$, itself, or $P[i+1]$.
Let these directions be denoted by having $P[i]$'s variable $m[i]$ have a value of $L$, $S$, and $R$ respectively.
The processes form a maximal matching when they point to each other or themselves, but no two neighboring processes can both point to themselves.
\begin{code}
forall i <- Nat % N :
   m[i-1]==R && m[i]==L               // P[i] pointing to P[i-1] and P[i-1] pointing back
|| m[i-1]==L && m[i]==S && m[i+1]==R  // P[i] pointing to itself and neighbors pointing away
||              m[i]==R && m[i+1]==L  // P[i] pointing to P[i+1] and P[i+1] pointing back
\end{code}

One stabilizing protocol has the actions:
\begin{code}
( m[i-1]==2 && m[i]!=0 && m[i+1]!=0 --> m[i]:=0; )
( m[i-1]!=2 && m[i]!=1 && m[i+1]==2 --> m[i]:=1; )
( m[i-1]!=2 && m[i]!=2 && m[i+1]!=2 --> m[i]:=2; )
\end{code}

\subsection{Deriving 1-Bit Protocol from 3-State Protocol}
\label{sec:MatchRingOneBit}

\quicksec{MatchRingOneBit}
(\href{\examplespec/MatchRingOneBit.prot}{spec},
\href{\examplesynt/MatchRingOneBit.prot}{synt},
\href{\examplesoln/MatchRingOneBit.prot}{soln})

This section explains shadow/puppet synthesis as a special case of superposition.
In the \href{#sec:MatchRing}{previous section}, we saw that a protocol could achieve a matching using only 1 bit per process.
How could this be derived?
From the above 3-state protocol, it seems that each process needs to be able to point in $3$ directions.

Give each process $P[i]$ a binary $x[i]$ variable to perform the protocol along with a ternary $m[i]$ variable used to specify the invariant.
Furthermore, $P[i]$ is given read access to $x[i-1]$ and $x[i+1]$, but it cannot read $m[i-1]$ or $m[i+1]$.

We use the previous section's invariant on the underlying $m$ variables.
Since processes only know their own $m$ values, the protocol is forced to use $x$ values to negotiate appropriate $m$ values.
Our invariant style (the \ilcode{((future & silent) % puppet)} style) allows closure to be violated for some (but not all) valuations of $x$ variables.
\codeinputlisting{../../../examplespec/MatchRingOneBit.prot}

From synthesis, \href{\examplesynt/MatchRingOneBit.prot}{one of the protocols} we get is the following.
\begin{code}
( x[i-1]==1 && x[i]==1 && x[i+1]==1 --> x[i]:=0; m[i]:=L; )
( x[i-1]==0 && x[i]==1 && x[i+1]==1 --> x[i]:=0; m[i]:=S; )
( x[i-1]==0 && x[i]==0 && x[i+1]==0 --> x[i]:=1; m[i]:=R; )

( x[i-1]==1 && x[i]==0              && m[i]!=L --> m[i]:=L; )
( x[i-1]==0 && x[i]==0 && x[i+1]==1 && m[i]!=S --> m[i]:=S; )
( x[i-1]==0 && x[i]==1 && x[i+1]==0 && m[i]!=R --> m[i]:=R; )
\end{code}

From here, we can create the 1-bit matching protocol on the $x[i]$ variables without the $m[i]$s.
The first three actions of the synthesized protocol change $x[i]$ and are therefore used as actions in our 1-bit matching protocol, discarding changes to $m[i]$.
\begin{code}
( x[i-1]==1 && x[i]==1 && x[i+1]==1 --> x[i]:=0; )
( x[i-1]==0 && x[i]==1 && x[i+1]==1 --> x[i]:=0; )
( x[i-1]==0 && x[i]==0 && x[i+1]==0 --> x[i]:=1; )
\end{code}

The invariant is all states where the $x[i]$ values don't change (see the last three actions above).
\begin{code}
forall i <- Nat % N :
   x[i-1]==1 && x[i]==0               // P[i] pointing to P[i-1] and P[i-1] pointing back
|| x[i-1]==0 && x[i]==0 && x[i+1]==1  // P[i] pointing to itself and neighbors pointing away
|| x[i-1]==0 && x[i]==1 && x[i+1]==0  // P[i] pointing to P[i+1] and P[i+1] pointing back
\end{code}
Each of these cases in the disjunction corresponds to $P[i]$ pointing to $P[i-1]$, itself, and $P[i+1]$ respectively.
We know this by looking at how $m[i]$ is changed to be \ilcode{m[i]:=L}, \ilcode{m[i]:=S}, and \ilcode{m[i]:=R} in the synthesized protocol.

Note that the third case in the disjunction can be simplified from \ilcode{x[i-1]==0 && x[i]==1 && x[i+1]==0} to \ilcode{x[i]==1 && x[i+1]==0} since if the formula holds for $P[i]$ and the system is in the invariant, then the first or second cases in the disjunction must hold for $P[i-1]$ (hence, $x[i-1]=0$).

Putting this all together, we get:
\codeinputlisting{../../../examplesoln/MatchRingOneBit.prot}

\subsection{Using Shadow/Puppet Variables}

Shadow variables are variables that cannot be used in the guard of any actions.
Therefore, to reliably obtain a protocol that is free of $m$ variables, we must mark them as shadow variables by replacing \ilcode{direct} with \ilcode{shadow} in the \href{\examplespec/MatchRingOneBit.prot}{specification}.
This is essentially what is done in the \href{#sec:MatchRing}{previous section}, but a more convenient invariant is used.

\section{Segmented Ring}
\label{sec:SegmentRing}

\quicksec{SegmentRing}
(\href{\examplespec/SegmentRing.prot}{spec},
\href{\examplesett/SegmentRing.args}{args},
\href{\examplesynt/SegmentRing.prot}{synt})

This is a problem similar to matching where a ring is segmented into chains.

\end{document}


\section{Types}\label{sec:types}
In JavaScript there are six primitive data types.
\begin{itemize}
\item Number
\item Boolean
\item String
\item Null
\item Undefined
\item Symbol (new in ES6)\footnote{We do not discuss this type here but readers intereseted in knowing more about this new type please take a look at \url{http://blog.keithcirkel.co.uk/metaprogramming-in-es6-symbols/}}
\item Object
\end{itemize}

We also have functions, but in JavaScript functions are said to be "callable" objects and are therefore objects. We discuss functions in section~\ref{sec:functions}.

There is nothing to wait for, let us go through these types and get familiar with them.

\subsection{Numbers}\label{sub:numbers}
In JavaScript there is only one number type, namely \texttt{Number}. If you open the REPL and type in a number, you will get the number back as following listing indicates.

\begin{lstlisting}
> 42
42
> 34.3
34.3
\end{lstlisting}

There is no difference between float, doable or integers in JavaScript. They are all represented by the same type, and that type is \texttt{Number}. Numbers are represented with 64 bits and  some of these bits are used for the fractional part of the number, even if the value does not have a fractional part. Also bit is taken for representing negative numbers. Thus, the highest value to store in a number is $2^{31} - 1$.

This is a common pitfall in JavaScript when developers are trying to represent 64 bits values as numbers.

In \texttt{Number} there is a constant called \texttt{MAX\_VALUE} that can be used to view the maximum value that can be stored in a \texttt{Number} type.

\begin{lstlisting}
> Number.MAX_VALUE
1.7976931348623157e+308
\end{lstlisting}

In JavaScript you can look up the type of a given value using the `typeof` operator.

\begin{lstlisting}
> typeof 1337.42
'number'
\end{lstlisting}

We can apply all the well known mathematical operators on Numbers. They are similar to the once we have in C or Java.

\begin{lstlisting}
> 42 + 4.2
46.2
> 2 * 34
68
> 7 / 23
0.30434782608695654
> 2 - (-3)
5
\end{lstlisting}

The precedence of operators are also similar as in C or Java. You can view a table of the precedence of operators at \url{https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Operator_Precedence}.

In JavaScript you have access to a global object named \texttt{Math}. In \texttt{Math} you get access to many mathematical constants and functions to use in your calculations. There is a good overview at \url{https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Math}

\begin{lstlisting}
> Math.sqrt(25)
5
> Math.pow(2, 4)
16
\end{lstlisting}

There is a global constant in the \texttt{Number} object called \texttt{NaN}. NaN stands for ``Not a Number''. If you are doing some funky calculations or you are mixing types that cannot be interpreted as numbers in your calculations, you will get \texttt{NaN} as the result.

\begin{lstlisting}
> 34 / "a"
NaN

> 34 + NaN / 34
NaN
\end{lstlisting}

\texttt{NaN} is contagious in expressions. If any part of the expression is \texttt{NaN}, the whole expression becomes \texttt{NaN}.

A funny joke in the JavaScript community: the type of \texttt{NaN} is \texttt{Number}.

\begin{lstlisting}
> typeof NaN
'number'
\end{lstlisting}

There is another global value that you might see if you are doing shady calculations, and that is \texttt{Infinity}

\begin{lstlisting}
> typeof Infinity
'number'
\end{lstlisting}

If you interested in math, then this statement will probably not make your day.

\begin{lstlisting}
> 5 / 0
Infinity
\end{lstlisting}

Don't drink and drive -- and don't create a programming language under the same influence.

There are two functions that you can use to check if value is \texttt{NaN} and \texttt{Infinity}, \texttt{Number.isNaN} and \texttt{Number.isFinite} respectively.

\begin{lstlisting}
> isNaN(10)
false
> isNaN(3 / "bob")
true
> isFinite(5/0)
false
> isFinite(30 + 2)
true
\end{lstlisting}

Word of advice. JavaScript is not the ``best'' programming language for scientific calculations. It was not create for that purpose. Validate your calculations, write tests for them and watch out for large numbers. Keep it safe.

\subsection{Boolean}
JavaScript has a \texttt{boolean} type. There are exactly two values that \texttt{boolean} can have, that is \texttt{true} and \texttt{false}.

\begin{lstlisting}
    > true
    true
    > false
    false
    > typeof true
    'boolean'
    > typeof false
    'boolean'
\end{lstlisting}

JavaScript supports the same boolean operators as C and Java.

\begin{lstlisting}
    > true && false
    false
    > false || true
    true
    > (false || true) && true
    true
\end{lstlisting}

The boolean operators always return a \texttt{boolean} value.

As in similar programming languages, \texttt{boolean} can be used when controlling the flow in our programs when we use \texttt{if}, \texttt{while} and such. We look the flow control in section n.

\subsection{String}
A stream of characters in JavaScript within single quotes, or double quotes, are of the type \texttt{string}.

\begin{lstlisting}
    > "hello js"
    'hello js'
    > typeof 'hello js'
    'string'
\end{lstlisting}

Strings in JavaScript are immutable. When a string has been created it cannot be changed. All methods that work with strings return a new string.

\begin{lstlisting}
    > "Hello" + "world"
    'Helloworld'
\end{lstlisting}

In this example we are concatenating two strings, the result is a new string.

Oh yeah, while we are at it. It is time for some strange drunken ``programming language decisions'' rant. When you add together a string and a number, you get a string back.

\begin{lstlisting}
    > 2 + "2"
    '22'
\end{lstlisting}

JavaScript is trying to be smart, and typecasts the number into a string. This happens when a language is dynamically- and loosely typed with an ambiguous `+` operator dangling around. Well, we can forgive that. But in subtraction it is different.

\begin{lstlisting}
    > 2 - "3"
    -1
    > "3" - 2
    1
\end{lstlisting}

Here it is the opposite. The string is represented as a number. So be careful mixing types when you are doing calculations and when concatenating strings. You can always parse the values yourself using the global functions \texttt{parseInt}, \texttt{parseFloat}. Also, Numbers can be represented as Strings with `.toString()`

\begin{lstlisting}
    > (12).toString()
    '12'
\end{lstlisting}

The \texttt{string} type has many functions that you are likely familiar with from other programming languages. The MDN entry at \url{https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String} is comprehensive and describes them well. You can look for string functions there when you need them. The following listing shows couple of them in action.

\begin{lstlisting}
    > "hello world today".split(' ')
    [ 'hello', 'world', 'today' ]
    > 'bobby cakes'.indexOf('c')
    6
    > 'hlysig'.startsWith('h')
    true
    > 'hlysig!'.toUpperCase()
    'HLYSIG!'
\end{lstlisting}

\subsection{Null and Undefined}
There are two types to represent ``nothing'' in JavaScript and that is \texttt{null} and \texttt{undefined}. They are not the same and they have different semantics in the language.

First, \texttt{null} is actually not a type in JavaScript but a global object.

\begin{lstlisting}
> typeof null
'object'
\end{lstlisting}

Whereas \texttt{undefined} is a type.

\begin{lstlisting}
> typeof undefined
'undefined'
\end{lstlisting}

\texttt{undefined} means that you have declare a variable that has not been assigned yet, or the return value from a function that has not return value. We introduce variables and functions in sections n and n, but here are two examples that helps us understand \texttt{undefined}.

\begin{lstlisting}
> let b;
undefined
> b
undefined
> b === undefined
true
\end{lstlisting}

\begin{lstlisting}
> function noReturnValue() {console.log('hello');}
undefined
> noReturnValue() === undefined
hello
true
\end{lstlisting}

\texttt{null} on the other hand is a value that you can sign to a variable or return from a function that represents ``nothing''. You will only see \texttt{null} if you use it yourself.

\begin{lstlisting}
> let a = null;
undefined
> a
null
> a === null
true
\end{lstlisting}

The semantics here is that you are stating that this variables points to nothing. But behind the scenes, this variable is pointing to the global object \texttt{null}.

You should always try to avoid using \texttt{null} as much as you avoid drinking gasoline. To back that sentence up we give a citation to Sir Charles Antony Richard Hoare and his famous sentence ``Null References, The Billion Dollar Mistake''. We know that he apologised later, but functional programmers still have that sentence tattooed on their forehead. Just be careful when using \texttt{null}.

\subsection{Objects}
JavaScript is a Prototype-based programming language\footnote{\url{https://en.wikipedia.org/wiki/Prototype-based_programming}}. That is a different approach to Object Oriented Programming \footnote{\url{https://en.wikipedia.org/wiki/Object-oriented_programming}} where object behaviour is reused through Prototype chains\footnote{\url{http://javascriptissexy.com/javascript-prototype-in-plain-detailed-language/}}. In this course we will not deal so deep with JavaScript objects nor do much OOP programming in it. If anything, we will do more functional style programming.

This does not mean that we will not use objects, far from it. We will mostly be creating objects, using the object literal notation, for encapsulating data, grouping common data together and as a associative array.

If you are interested in learning more about Object-Oriented programming in JavaScript

We encourage students who are interested in learning more about Object-Oriented programming in JavaScript to do so. The internet is full of great material on that subject. We have added some handpicked articles in section~\ref{sec:references}.

Now let's look at objects. To define an object using literal notation we use \texttt{\{\}}.

\begin{lstlisting}
const entry = {
  title: 'Concerning Hobbits',
  content: 'They have hairy toes'
};
\end{lstlisting}

In this example we define variable which points to an instance of an object. Objects have many simularities to associative arrays and can be used as such. Objects have collections of keys, in this example we have the keys \texttt{title} and \texttt{content} and properties which they point to.

Properties can be fetched with the `.` operator on the object using the key.

\begin{lstlisting}
> entry.title
'Concerning Hobbits'
\begin{lstlisting}

We can also place the key into brackets.

\begin{lstlisting}
> entry['title']
'Concerning Hobbits'
\end{lstlisting}

This can be useful when the keys have spaces or strange characters in them.

Properties in objects can be of any type even another object or a function.

JSON (JavaScript Object Notation) is a formal description how JavaScript object can be serialized back and forth from objects to strings. There is a global object named `JSON` that we can use for that purpose, as following listing indicates.

\begin{lstlisting}
> var x = {'data': 'hello world'}
> let jsdata = JSON.stringify(x)
> jsdata
'{"data":"hello world"}'
> JSON.parse(jsdata)
{ data: 'hello world' }
\end{lstlisting}

In this example we create an object. We serialize it to JSON and then we take that JSON and we serialize it back to object. We will be doing this heavily in this course where our web services will be returing JSON data and accepting JSON data.


\documentclass[]{book}
\usepackage{graphicx}
\usepackage{amsmath,amssymb,amsthm}

%opening
\title{Probability and Statistics}
\author{}

\newcommand{\distas}[1]{\mathbin{\overset{#1}{\kern\z@\sim}}}%
\newsavebox{\mybox}\newsavebox{\mysim}
\newcommand{\distras}[1]{%
	\savebox{\mybox}{\hbox{\kern3pt$\scriptstyle#1$\kern3pt}}%
	\savebox{\mysim}{\hbox{$\sim$}}%
	\mathbin{\overset{#1}{\kern\z@\resizebox{\wd\mybox}{\ht\mysim}{$\sim$}}}%
}

\begin{document}

\maketitle

\setcounter{tocdepth}{1}
\tableofcontents
\newpage

\chapter{Discrite Distributions}

\section{Generic Formulas}

\subsection{Expected Value}
X is discrete random variable\\
$g : \mathbf{R} \rightarrow \mathbf{R}$\\
$\Omega X = Im(X)$\\

\begin{align}
	E[g(X)] &= \sum_{x \in \Omega X} {P(X=x)*g(x)}
\end{align}

\subsection{Variance}

\begin{align}
	var(g(X)) &= E([g(X) - E(g(X))]^2)\\
	&= \sum_{x\in \Omega X} {[g(X) - E(g(X))]^2*P(X=x)]}\\
	&= \sum_{x\in \Omega X} {[g(X)^2 - 2*g(X)*E(g(X)) + E(g(X))^2]*P(X=x)}\\
	&= \sum_{x\in \Omega X} {g(X)^2*P(X=x)} \\&- \sum_{x\in \Omega X} {2*g(X)*E(g(X))*P(X=x)} \\&+ \sum_{x \in \Omega X} {E(g(X))^2*P(X=x)}\\
	&= \sum_{x\in \Omega X} {g(X)^2*P(X=x)} \\&- 2*E(g(X))*\sum_{x\in \Omega X} {g(X)*P(X=x)} \\&+ E(g(X))^2*\sum_{x \in \Omega X} {P(X=x)}\\
	&= E(g(X)^2) - 2*E(g(X))*E(g(X)) + E(g(X))^2*1\\
	&= E(g(X)^2) - 2*E(g(X))^2 + E(g(X))^2\\
	&= E(g(X)^2) - E(g(X))^2\\
	\qed
\end{align}

\subsection{Covariance Matrix}

\begin{align}
cov(X) &= E[(X-E(X)]^2\\
&= \sum_{x\in \Omega X} {(X-E(X))^2*P(X=x)}\\
&= \sum_{x\in \Omega X} {[X^2-2XE(X)+E(X)^2]*P(X=x)}\\
&= \sum_{x\in \Omega X} {X^2P(X=x)} \\&- \sum_{x\in \Omega X} {2XE(X)*P(X=x)} \\&+ \sum_{x \in \Omega X} {E(X)^2P(X=x)}\\
&= \sum_{x\in \Omega X} {X^2P(X=x)} \\&- 2E(X)*\sum_{x\in \Omega X} {XP(X=x)} \\&+ E(X)^2*\sum_{x \in \Omega X} {P(X=x)}\\
&= E(X^2)-2E(X)E(X)+E(X)^2*1\\
&= E(X^2)-2*E(X)^2+E(X)^2\\
&= E(X^2)-E(X)^2\\
&= E(X^tX)-\mu^t\mu\\
\qed
\end{align}

\subsection{Variance of the Sample Mean}

\begin{align}
Var\left({\overline{X}}\right)&=Var\left( \frac{1}{n}\sum_{i=1}^nX_i\right)\\&= \frac{1}{n^2}Var\left( \sum_{i=1}^nX_i\right)\\&=\frac{1}{n^2}\sum_{i=1}^nVar(X_i), \text{ by independence}\\&= \frac{1}{n^2}\left[Var(X_1)+Var(X_2)+\ldots+Var(X_n) \right]\\&=\frac{1}{n^2}\left[\sigma^2+\sigma^2+\ldots+\sigma^2  \right], \text{ since the }X_i \text{ are identically distributed }\\&= \frac{1}{n^2}(n\sigma^2)\\&=\frac{\sigma^2}{n}
\end{align}

\subsection {Law of Iterated Expectation}
\begin{align}
	E[X] = E[ E[X|Y] ]
\end{align}

\subsection {Law of Total Variance}
\begin{align}
var(X) = E[ var(X|Y) ] + var( E[X|Y] )
\end{align}

\subsection{MSE}

\begin{align}
	\hat{\theta} &= \hat{\theta}(X) \text{ Random Variable}\\
	E[\hat{\theta}] &= \text{constant}\\
	\theta &= \text{true value, constant}\\
	\\
	MSE(\hat{\theta}) &= E[(\hat{\theta} - \theta)^2]\\
	&= E[(\hat{\theta} - E[\hat{\theta}] + E[\hat{\theta}]) - \theta)^2]\\
	&= E[([\hat{\theta} - E[\hat{\theta}]] + [E[\hat{\theta}] - \theta])^2]\\
	&= E[(A + B)^2]\\
	\\
	& A = \hat{\theta} - E[\hat{\theta}]\\
	& B = E[\hat{\theta}] - \theta\\
	\\
	&= E[A^2 + 2AB + B^2]\\
	&= E[(\hat{\theta} - E[\hat{\theta}])^2 + 2(\hat{\theta} - E[\hat{\theta}])(E[\hat{\theta}] - \theta) + (E[\hat{\theta}] - \theta)^2]\\
	&= E[(\hat{\theta} - E[\hat{\theta}])^2] + E[2(\hat{\theta} - E[\hat{\theta}])(E[\hat{\theta}] - \theta)] + E[(E[\hat{\theta}] - \theta)^2]\\
	\\
	& C = E[\hat{\theta}]  - \theta \text{ is a constant}\\
	\\
	&= E[(\hat{\theta} - E[\hat{\theta}])^2] + E[2(\hat{\theta} - E[\hat{\theta}])(C)] + E[C^2]\\
	&= E[(\hat{\theta} - E[\hat{\theta}])^2] + 2CE[\hat{\theta} - E[\hat{\theta}]] + C^2\\
	&= E[(\hat{\theta} - E[\hat{\theta}])^2] + 2C(E[\hat{\theta}] - E[E[\hat{\theta}]]) + C^2\\
	&= E[(\hat{\theta} - E[\hat{\theta}])^2] + 2C(E[\hat{\theta}] - E[\hat{\theta}]) + C^2\\
	&= E[(\hat{\theta} - E[\hat{\theta}])^2] + 2C(0) + C^2\\
	&= E[(\hat{\theta} - E[\hat{\theta}])^2] + C^2\\
	&= E[(\hat{\theta} - E[\hat{\theta}])^2] + (E[\hat{\theta}]  - \theta)^2\\
	\\
	& var(\hat{\theta}) = E[(\hat{\theta} - E[\hat{\theta}])^2]\\
	& bias(\hat{\theta}, \theta) = E[\hat{\theta}]  - \theta\\
	\\
	&= var(\hat{\theta}) + (bias(\hat{\theta}, \theta))^2\\
\end{align}

\section {Uniform Random Variable}

\section {Bernoulli Distribution}

The Bernoulli Distribution is a special case of the Binomial Distribution, where $$n=1$$.

\subsection{PMF}
\begin{align}
P(X=k) &= \binom{1} {k} p^k (1-p)^{1-k}\\
&=p^k (1-p)^{n-k}
\end{align}

\subsection {Expected Value}
\begin{align}
E(x) &= \sum_{k \geqslant 1} [\binom{n}{k} p^k (1-p)^{n-k}] * k\\
&= np && \text{see Binomial Distribution E[X]}\\
&= 1*p\\
&= p
\end{align}

\subsection {Variance}
\begin{align}
Var(X) &= np*(1-p)\\
&= p*(1-p)\\
\end{align}

\subsection {Likelihood of IID Bernoulli}

\begin{align}
x_i \overset{iid}{\sim} Bernoulli(p)\\
L(x_i|p)&=p(x_1,x_2,...,x_n|p)\\&=\prod_{n=1}^np(x_i|p)\\
&=p^S*(1-p)^{n-S}
\end{align}

\subsection {Maximun Likelihood}

\begin{align}
\frac{d[L(x_i|p)]}{dp} &=\frac{d[p^S*(1-p)^{n-S}]}{dp}\\
\frac{d[log(L(x_i|p))]}{dp} &=\frac{d[log(p^S*(1-p)^{n-S})]}{dp}\\
&= \frac{d}{dp}*[log(p^S*(1-p)^{n-S})]\\
&= \frac{d}{dp}[log(p^S)+log((1-p)^{n-S})]\\
&= \frac{d}{dp}[S*log(p)+(n-S)*log(1-p)]\\
&= S*\frac{d}{dp}[log(p)]+(n-S)*\frac{d}{dp}[log(1-p)]\\
&= S*[\frac{1}{p}]+(n-S)*\frac{d}{dp}[log(1-p)] && \text{chain rule}\\
&= S*\frac{1}{p}+(n-S)*\frac{1}{p-1}\\
&= \frac{S}{p}+\frac{n-S}{p-1}\\
&= \frac{S*(p-1)}{p*(p-1)}+\frac{p*(n-S)}{p*(p-1)}\\
&= \frac{S*(p-1)+p*(n-S)}{p*(p-1)}\\
&= \frac{S*p-S+p*n-p*S}{p*(p-1)}\\
&= \frac{-S+p*n}{p*(p-1)}\\
0 &= \frac{-S+p*n}{p*(p-1)}\\
0*(p*(p-1)) &= -S+p*n\\
0 &= -S+p*n\\
S &= p*n\\
\frac{S}{n} &= p\\
p &= \frac{S}{n}\\
\end{align}

\subsection{MGF}

\begin{align}
	M(t) &= E[e^{tX}]\\
	&= (1-p)e^{t*0} + pe^{t*1}\\
	&= (1-p) + pe^{t}\\
\end{align}

\subsubsection{E[X] using MGF}

\begin{align}
E[X] &= \frac{d^1}{dt^1}[M(t)](0)\\
\\
M(t) &= (1-p) + pe^{t}\\
\\
E[X] &= \frac{d^1}{dt^1}[(1-p) + pe^{t}](0)\\
&= [\frac{d^1}{dt^1}[(1-p)] + \frac{d^1}{dt^1}[pe^{t}]](0)\\
&= [0 + \frac{d^1}{dt^1}[pe^{t}]](0)\\
&= [\frac{d^1}{dt^1}[pe^{t}]](0)\\
&= [pe^{t}](0)\\
&= [pe^{0}]\\
&= p*1\\
&= p\\
\end{align}

\subsubsection{$E[X^2]$ using MGF}

\begin{align}
E[X^2] &= \frac{d^2}{dt^2}[M(t)](0)\\
\\
M(t) &= (1-p) + pe^{t}\\
\\
E[X^2] &= \frac{d^2}{dt^2}[(1-p) + pe^{t}](0)\\
&= \frac{d^1}{dt^1}[pe^{t}](0) && \text{see E[X] using MGF}\\
&= [pe^{t}](0)\\
&= [pe^{0}]\\
&= p*1\\
&= p\\
\end{align}

And these values can be checked calculating the $var[X]$.

\begin{align}
	var[X] &= E[X^2] - E[X]^2\\
	&= p - p^2\\
	&= p*(1-p)\\
	\qed
\end{align}

\subsubsection{$E[X^n]$ using MGF}

\begin{align}
E[X^2] &= \frac{d^2}{dt^2}[M(t)](0)\\
\\
M(t) &= (1-p) + pe^{t}\\
\\
E[X^2] &= \frac{d^2}{dt^2}[(1-p) + pe^{t}](0)\\
\end{align}

\section{Binomial Distribution}

\subsection{PMF}
\begin{align}
P(X=k) &= \binom{n} {k} p^k (1-p)^{n-k}
\end{align}

%prove that the summation is 1
%prove the expected value
%\paragraph{https://proofwiki.org/wiki/Expectation_of_Binomial_Distribution}

\subsection {Expected Value}

$E[g(X)]$ when $g(X) = X$.

\begin{align}
E(X) &= \sum_{k \geqslant 0}P(x=k)*k\\
&= \sum_{k \geqslant 0}[\binom{n} {k} p^k (1-p)^{n-k}] * k\\
\end{align}

when $$k=0$$, the formula $$[\binom{n} {k} p^k (1-p)^{n-k}] * k = [\binom{n} {0} p^k (1-p)^n] * 0 = 0$$, so the index of the summation can be increased by 1.

\begin{align}
E(X) &= \sum_{k \geqslant 1} \binom{n}{k} p^k (1-p)^{n-k} * k\\
&= \sum_{k \geqslant 1} \frac{n}{k} * \binom{n-1}{k-1} p^k (1-p)^{n-k} * k && \text{see BinomialCoefficient}\\
&= \sum_{k \geqslant 1} \frac{n*k}{k} * \binom{n-1}{k-1} p^k (1-p)^{n-k}\\
&= \sum_{k \geqslant 1} n * \binom{n-1}{k-1} p^k (1-p)^{n-k}\\
&= \sum_{k \geqslant 1} n*p * \binom{n-1}{k-1} p^{k-1} (1-p)^{n-k}\\
&= np * \sum_{k \geqslant 1} \binom{n-1}{k-1} p^{k-1} (1-p)^{n-k}\\
u = n-1\\
z = k-1\\
u-z&=(n-1)-(k-1)\\
&=n-1-k+1\\
&=n-k\\
k>1&=(z+1)>1\\
&=z>0\\
&= np * \sum_{z>0} \binom{u}{z} p^{z} (1-p)^{u-z}\\
&= np * 1 && see BinomialDistributionProofEquals1\\
&= np\\
\qedsymbol
\end{align}

\subsection {Variance}

\begin{align}
	Var(X) &= E(X^2) - E(X)^2 && \text{see Variance}\\
	&= \sum_{k \geqslant 0} {[\binom{n} {k} p^k (1-p)^{n-k}]*k^2} - np && \text{see Binomial Expected Value}
\end{align}

when $$k=0$$, the formula $$[\binom{n} {k} p^k (1-p)^{n-k}] * k = [\binom{n} {0} p^k (1-p)^n] * 0 = 0$$, so the index of the summation can be increased by 1.

\begin{align}
&= \sum_{k \geqslant 1} {[\binom{n} {k} p^k (1-p)^{n-k}]*k^2} - (np)^2\\
&= \sum_{k \geqslant 1} {\frac{n}{k} [\binom{n-1} {k-1} p^k (1-p)^{n-k}]*k^2} - (np)^2\\
&= \sum_{k \geqslant 1} {\frac{n*k^2}{k} [\binom{n-1} {k-1} p^k (1-p)^{n-k}]} - (np)^2\\
&= \sum_{k \geqslant 1} {[nk*\binom{n-1} {k-1} p^k (1-p)^{n-k}]} - (np)^2\\
&= \sum_{k \geqslant 1} {[nkp*\binom{n-1} {k-1} p^{k-1} (1-p)^{n-k}]} - (np)^2\\
&= np*\sum_{k \geqslant 1} {[k*\binom{n-1} {k-1} p^{k-1} (1-p)^{n-k}]} - (np)^2\\
u = n-1\\
z = k-1\\
u-z&=(n-1)-(k-1)\\
&=n-1-k+1\\
&=n-k\\
k >= 1 &= (z+1) >=1\\
&= z >= 0\\
&= np*\sum_{z \geqslant 0} {[(z+1)* \binom{u} {z} p^{z} (1-p)^{u-z}]} - (np)^2\\
&= np*[\sum_{z \geqslant 0} {[z*\binom{u} {z} p^{z} (1-p)^{u-z}]} + \sum_{z \geqslant 0} {[\binom{u} {z} p^{z} (1-p)^{u-z}]}] - (np)^2\\
&= np*[\sum_{z \geqslant 0} {[z*\frac{u}{z}*\binom{u-1} {z-1} p^{z} (1-p)^{u-z}]} + \sum_{z \geqslant 0} {[\binom{u} {z} p^{z} (1-p)^{u-z}]}] - (np)^2\\
&= np*[u*\sum_{z \geqslant 0} {[\binom{u-1} {z-1} p^{z} (1-p)^{u-z}]} + \sum_{z \geqslant 0} {[\binom{u} {z} p^{z} (1-p)^{u-z}]}] - (np)^2\\
\end{align}
\begin{align}
&= np*[up*\sum_{z \geqslant 0} {[\binom{u-1} {z-1} p^{z-1} (1-p)^{u-z}]} + \sum_{z \geqslant 0} {[\binom{u} {z} p^{z} (1-p)^{u-z}]}] - (np)^2\\
&= np*[up*\sum_{z \geqslant 1} {[\binom{u-1} {z-1} p^{z-1} (1-p)^{(u-1)-(z-1)}]} + \sum_{z \geqslant 0} {[\binom{u} {z} p^{z} (1-p)^{u-z}]}] - (np)^2\\
&= np*[up*\sum_{z \geqslant 0} {[\binom{u} {z} p^{z} (1-p)^{u-z}]} + \sum_{z \geqslant 0} {[\binom{u} {z} p^{z} (1-p)^{u-z}]}] - (np)^2\\
&= np*[up*(p+q)^{u-1} + (p+q)^u] - (np)^2\\
&= np*[(n-1)*p*(p+q)^{n-1-1}+(p+q)^(n-1)] -(np)^2\\
&= np*[(n-1)*p*(p+q)^{n-2}+(p+q)^(n-1)] - (np)^2\\
&= np*([(n-1)*p*(p+q)^{n-2}+(p+q)^(n-1)] - np)\\
&= np*([(n-1)*p+1] - np) && \text{p+q=1}\\
&= np*([(n-1)*p+1] - np)\\
&= np*([np-p +1] - np)\\
&= np*(np-p +1- np)\\
&= np*(-p +1)\\
&= np*(1-p)\\
\qed
\end{align}

\section {Geometric Distribution}

\subsection {CDF}

\begin{align*}
	P(X \le x) &=\\
	CDF(X=x) &= \sum_{i=0}^{x}{(1-p)^ip} &\text{by geometric summation}\\
	&= p\frac{1-(1-p)^x}{1-(1-p)}\\
	&= p\frac{1-(1-p)^x}{1-1+p}\\
	&= p\frac{1-(1-p)^x}{p}\\
	&= 1-(1-p)^x\\
	\qed\\
	\\
	P(X > x) &= 1 - CDF(X=x)\\
	&= 1 - (1-(1-p)^x)\\
	&= 1 - 1 +(1-p)^x\\
	&= (1-p)^x\\
	\qed
\end{align*}

\section {Normal Distribution}

\subsection {Definition}

\begin{align}
pdf(x) &= \frac{1}{\sigma\sqrt{2\pi}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)
\end{align}

\chapter{Continuous Distributions}

\section{Uniform Distribution}

\subsection{PDF}
\begin{align*}
\int_{a}^{b} {kdx} &= 1\\
&= k\int_{a}^{b} {dx}\\
&= k[x|_{a}^{b}]\\
&= k[b-a]\\
\\
k[b-a] &= 1\\
k &= \frac{1} {b-a}\\
\end{align*}

\subsection{Expected Value}
\begin{align*}
\int_{a}^{b} {x\big( \frac{1}{b-a} \big)dx} &= \\
&= \int_{a}^{b} {x\big( \frac{1}{b-a} \big)dx}\\
&= \frac{1}{b-a}\int_{a}^{b} {xdx}\\
&= \frac{1}{b-a}\big[ \frac{x^2}{2}|_{a}^{b} \big]\\
&= \frac{1}{b-a}\big[ \frac{b^2}{2} - \frac{a^2}{2} \big]\\
&= \frac{1}{b-a}\big[ \frac{b^2-a^2}{2} \big]\\
&= \frac{1}{b-a}\big[ \frac{(b+a)(b-a)}{2} \big]\\
&= \frac{(b+a)(b-a)}{2(b-a)}\\
&= \frac{b+a}{2}\\
\end{align*}

\subsection{Variance}
\begin{align*}
var(X) &= E[X^2] - E[X]^2\\
&= [\int_{a}^{b} {x^2\big( \frac{1}{b-a} \big)dx}] - (\frac{b+a}{2})^2\\
&= [\frac{1}{b-a}\int_{a}^{b} {x^2dx}] - (\frac{b+a}{2})^2\\
&= [\frac{1}{b-a}\frac{x^3}{3}|_{a}^{b}] - (\frac{b+a}{2})^2\\
&= \frac{1}{b-a}(\frac{b^3}{3}-\frac{a^3}{3}) - (\frac{b+a}{2})^2\\
&= \frac{b^3-a^3}{3(b-a)} - \frac{(b+a)^2}{4}\\
&= \frac{(b-a)(b^2-ab+a^2)}{3(b-a)} - \frac{(b+a)^2}{4}\\
&= \frac{(b^2-ab+a^2)}{3} - \frac{(b+a)^2}{4}\\
&= \frac{4(b^2-ab+a^2)}{12} - \frac{3(b+a)^2}{12}\\
&= \frac{4b^2-4ab+4a^2}{12} - \frac{3(b^2+2ab+a^2)}{12}\\
&= \frac{4b^2-4ab+4a^2}{12} - \frac{3b^2+6ab+3a^2}{12}\\
&= \frac{4b^2-3b^2-4ab-6ab+4a^2-3a^2}{12}\\
&= \frac{b^2-2ab+a^2}{12}\\
&= \frac{(b-a)^2}{12}\\
\end{align*}

\pagebreak
\section {Exponential Distribution}

\subsection{PDF}
\begin{align*}
f_x(x) &= \lambda e^{-\lambda x} && x >= 0
\end{align*}

\subsection{CDF}
\begin{align*}
	CDF(x) = \int_{-\infty}^{x} {f_xdx}\\
	CDF(x) = \int_{-\infty}^{x} {\lambda e^{-\lambda x}dx}\\
	\\
	u = e^{-\lambda x}\\
	\frac{du}{dx} = d[e^{-\lambda x}]\\
	du = d[e^{-\lambda x}]*dx\\
	du = [-\lambda * e^{-\lambda x}]*dx\\
	du = -\lambda e^{-\lambda x}dx\\
	\\
	CDF(x) = \int_{-\infty}^{x} {\lambda e^{-\lambda x}dx}\\
	CDF(x) = -1*\int_{-\infty}^{x} {-1*\lambda e^{-\lambda x}dx}\\
	CDF(x) = -1*\int_{-\infty}^{x} {-\lambda e^{-\lambda x}dx}\\
	CDF(x) = -1*\int_{-\infty}^{x} {du}\\
	CDF(x) = -1*\int_{0}^{x} {du} && \text{by x bounds}\\
	CDF(x) = -1*[u]_{0}^{x}\\
	CDF(x) = -1*[e^{-\lambda x}]_{0}^{x}\\
	CDF(x) = -1*[e^{-\lambda x} - e^{-\lambda 0}]\\
	CDF(x) = -1*[e^{-\lambda x} - e^{0}]\\
	CDF(x) = -1*[e^{-\lambda x} - 1]\\
	CDF(x) = [1 - e^{-\lambda x}]\\
	CDF(x) = 1 - e^{-\lambda x}\\
\end{align*}

\subsection{Expected Value}
\begin{align*}
f_x(x) &= \lambda e^{-\lambda x}\\
\\
E[x] &= \int {x*f_x(x)dx}\\
E[x] &= \int {x*\lambda e^{-\lambda x}dx}\\
\\
	u = x\\
	du = 1*dx\\
	v = e^{-\lambda x}\\
	dv = e^{-\lambda x}*-\lambda * dx\\
	dv = -\lambda e^{-\lambda x}dx\\
	\\
E[x] &= \int {[x] [\lambda e^{-\lambda x}dx]}\\
E[x] &= -1*\int {[x] [-1* \lambda e^{-\lambda x}dx]}\\
E[x] &= -1*\int {[x] [-\lambda e^{-\lambda x}dx]}\\
E[x] &= -1*\int {udv}\\
E[x] &= -1*[uv|_{-\infty}^{\infty} - \int {vdu}]\\
E[x] &= -1*[uv|_{0}^{\infty} - \int {vdu}] && \text{because x bounds}\\
E[x] &= -1*[[xe^{-\lambda x}]_{0}^{\infty} - \int {e^{-\lambda x}dx}]\\
\end{align*}
\begin{align*}
[xe^{-\lambda x}]_{0}^{\infty} &= \lim_{x->\infty}{[xe^{-\lambda x} - (0)e^{-\lambda (0)}]}\\
&= \lim_{x->\infty}{[xe^{-\lambda x} + 0e^{0}]}\\
&= \lim_{x->\infty}{[xe^{-\lambda x}]}\\
&= 0\\
\\
E[x] &= -1 * - \int {e^{-\lambda x}dx}\\
E[x] &= \frac{1}{\lambda} \int {\lambda e^{-\lambda x}dx}\\
E[x] &= \frac{1}{\lambda}\\
\end{align*}

\begin{figure}[h]
	\centering
	\includegraphics[width=\textwidth]{Random Variables Relationships}
\end{figure}

\end{document}