\newpage
\section{Systems of Ordinary Differential Equations}
\subsection{}
Systems of ordinary differential equations have 1 dependent variable and 1 independent variable. Let's look at an example.

\begin{eg}
  Here is a system of 3 first order ordinary differential equations of dimension 3
  \begin{align*}
    x'&=4x-7y+z^2\\
    y'&=2t-e^{t}y-z\\
    z'&=z^2+y^2-yx
  .\end{align*}
\end{eg}

We can also turn higher order ODE's into systems of first-order ODEs.
\begin{eg}
  Consider the equation $x''+7x''+7tx'+6x=t^2$. If we let $y=x',z=x''=y',x'''=z'$, then using the ODE,
  \[
  z'=t^2-6x+7ty-7z
  ,\]
  we can make a system of nonlinear, nonautonomous, nonhomogeneous ordinary differential equations.\footnote{After this system we went to chapter 2.6}
  \begin{align*}
    x'&=y\\
    y'&=z\\
    z'&=t^2-6x+ty-7z
  .\end{align*}
\subsection{Matrix Algebra}
A matrix is a rectangular array of numbers.
\begin{eg}
  Consider $A=\begin{pmatrix} 2&3&1\\4&5&0 \end{pmatrix} $. $A$ has 2 rows and 3 columns, so its size is $2\times 3$. The $a_{ij}$ is an entry in $A$'s $i^{th}$ row and $j^{th}$ column.
\end{eg}
\begin{eg}
  Consider $B=\begin{pmatrix} 5&0&0\\0&-1&0\\0&0&3 \end{pmatrix} $, which is a $3\times 3$ matrix. This is also a square matrix, due to the size. $B$ is also a diagonal matrix, because it's only nonzero entries are on the main diagonals.
\end{eg}
An upper triangular matrix looks like
\[
  \begin{pmatrix} 5&2&0\\0&-1&3\\0&0&3 \end{pmatrix}
.\]
A lower triangular matrix looks like
\[
  \begin{pmatrix} 5&0&0\\ \pi&-1&0\\-1&-\frac{1}{2}&3 \end{pmatrix}
.\]
\end{eg}
We can add and subtract matrices element-wise, but only if the matrices have the same size. We can also multiply a matrix by a scalar
\[
  3\begin{pmatrix} 4&5\\6&7 \end{pmatrix} =\begin{pmatrix} 12&15\\18&21 \end{pmatrix}
.\]
A row vector is $\begin{pmatrix} 2&3&1 \end{pmatrix} $, in this case the size would be $1\times 3$. A column vector looks like $\begin{pmatrix} 2\\3\\1 \end{pmatrix} $ with the size $3\times 1$. The zero vector is of size $n\times n$, and has $0$ in every index in the matrix. The transpose of a matrix reverses all of the columns and rows. Consider
\[
  \begin{pmatrix} 2&3&1\\4&5&6 \end{pmatrix} ^{T}=\begin{pmatrix} 2&4\\3&5\\1&6 \end{pmatrix}
.\]
Here are the properties of a transpose:
\begin{itemize}
  \item $\left( A^{T} \right)^{T}=A $
  \item $\left( A+B \right) ^{T}=A^{T}+B^{T}$
  \item $(kA)^{T}=kA^{T}$
  \item $\left( AB \right) ^{T}=B^{T}A^{T}$
\end{itemize}
Now we are going to start talking about matrix multiplication. Matrix multiplication is like the dot product. The $(i,j)^{th}$ entry of $AB$ is the dot product of the $i^{th}$ row of $A$ w/ $j^{th}$ column of $B$. Consider the following
\[
  \begin{pmatrix} 1&2&3 \end{pmatrix} \begin{pmatrix} 4\\5\\6 \end{pmatrix} =(1\times 4)+(2\times 5)+(3\times 6)= 32
.\]
In order to do matrix multiplication, rows of $A$ must match the columns of $B$ and vice versa. Matrix multiplication is not commutative.
\begin{eg}
  If we let $A=\begin{pmatrix} 1&4\\5&10\\8&12 \end{pmatrix}$ and $B=\begin{pmatrix} -4&7&-3\\1&-3&2 \end{pmatrix} $. Does $AB=BA$? This is not true, because matrices are equal if and only if all entries are equal. If we just multiply $BA$, we get \[
  \begin{pmatrix} -4&7&-3\\1&-3&2 \end{pmatrix}\begin{pmatrix} 1&4\\5&10\\7&12 \end{pmatrix}  = \begin{pmatrix}2&8\\2&-2\end{pmatrix}
  .\]
\end{eg}
Here are some properties of matrices.
\begin{note}
  We must maintain the order from left to right.
\end{note}
\begin{itemize}
  \item $A(B+C)=AB+BC$
  \item $(A+B)C=AC+BC$
  \item $k(AB)=(kA)B=A(kB)$
  \item $ABC=(AB)C=A(BC)$
\end{itemize}
The identity matrix is a square diagonal matrix with ones on the diagonal. For example, the identity matrix of dimension 3 and 4 are the following
\begin{align*}
  I_3&=\begin{pmatrix} 1&0&0\\0&1&0\\0&0&1 \end{pmatrix} \\
  I_4&=\begin{pmatrix} 1&0&0&0\\0&1&0&0\\0&0&1&0\\0&0&0&1 \end{pmatrix}
.\end{align*}
With the identity matrix, if we multiply any matrix by the identity matrix, then we get the same answer. Back in lesson 2, we were given the general solution to a Cauchy-Euler ordinary differential equation as \[
  x(t)=C_1(1)+C_2t^3+C_3 \frac{1}{t^2}
.\] Let's find a particular solution satisfying $x(1)=2,x'(1)=4,x'(1)=0$. We can do this by taking the derivative and second derivative of our functions to get
\begin{align*}
  x'(t)=3C_2t^2-2C_3t ^{-3}\\
  x''(t)=6C_2t+6C_3t ^{-4}
.\end{align*}
Now using our initial conditions
\begin{align*}
  x(1)=C_1+C_2+C_3=2\\
  x'(1)=3C_2-2C_3=4\\
  x''(1)=6C_2+6C_3=0
.\end{align*}
We can rewrite this in matrix form as \[
  \begin{pmatrix} 1&1&1\\0&3&-2\\0&6&6 \end{pmatrix} \begin{pmatrix} C_1\\C_2\\C_3 \end{pmatrix} =\begin{pmatrix} 2\\4\\0 \end{pmatrix}
.\] We will call the first matrix, $A$, the second matrix, $\vec{x}$, and the third matrix, $\vec{b}$, so we must solve $A\vec{x}=\vec{b}$ for $\vec{x}$. We can solve this by doing the following techniques
\begin{enumerate}
  \item Cramer's Rule (See book appendix)
  \item Gaussian elimination
  \item (Left)-multiply by $A^{-1}$
  \item MATLAB: rref or $A / B$
\end{enumerate}

\begin{theorem}
  This is the invertible matrix theorem. If $A$ is square, either all of case 1 or all of case 2 statements are true. Either $A$ is non-singular [good] or $A$ is singular [bad]. \newline Considering case 1
  \begin{itemize}
    \item $det(A)\neq 0$
    \item $A\vec{x}=\vec{b}$ has exactly one solution $\vec{x}$ for each $\vec{b}$.
    \item $A\vec{x}=\vec{0}$ has only the trivial solution, $\vec{x}=\vec{0}$.
    \item The rows (and columns) are linearly independent.
    \item $A^{-1}$ exists.
  \end{itemize}
  Considering Case 2
  \begin{itemize}
    \item $det(A)=0$
    \item $A\vec{x}=\vec{b}$ has zero or infinite solutions
    \item $A\vec{x}=\vec{0}$ has infinite solutions.
    \item The rows (and columns) are linearly dependent.
    \item $A^{-1}$ does not exist.
  \end{itemize}
\end{theorem}
$A^{-1}$ is a square matrix such that \[
A^{-1}A=A A^{-1}=I
.\] We find $A^{-1}$ through the following methods
\begin{enumerate}
  \item $rref(A|I)\to(I|A^{-1})$
  \item $A^{-1}=\frac{1}{det(A)}\cdot adj(A)$ where $adj(A)$ is the transpose of the cofactor matrix.
\end{enumerate}
How do we use $A^{-1}$? Given $A\vec{x}=\vec{b}$, we must solve for $x$. In order to do this, we left multiply by $A^{-1}$ to get
\begin{align*}
  A^{-1}A\vec{x}&=A^{-1}\vec{b}\\
  I\vec{x}&=A^{-1}\vec{b}\\
  \vec{x}=A^{-1}\vec{b}
.\end{align*}
If given a general $2\times 2$ matrix, $A=\begin{pmatrix} a&b\\c&d \end{pmatrix} $, we can find the inverse by doing
\begin{align*}
  A^{-1}&=\frac{1}{ad-bc}\begin{pmatrix} d&-c\\-b&a \end{pmatrix} ^{T}\\
        &=\frac{1}{ad-bc}\begin{pmatrix} d&-b\\-c&a \end{pmatrix}
.\end{align*}
The trace of $A$ ($trace(A)$) is the sum along the diagonal.
\newline
Let's write a linear system of first order differential equations in matrix form. If the equation is not first order, use Sect (4.1) technique to turn into 1st order system.
\begin{eg}
  Consider the following system
  \begin{align*}
    x'&=x-y+z\\
    y'&=4x+y-5z+7\\
    z'&=7x-8y-9z
  .\end{align*}
  The $+7$ in the second equation makes this a nonhomogeneous ordinary differential equation with constant coefficients. We can rewrite this as $\vec{x}'(t)=A(t)\vec{x}+\vec{b}(t)$, where $\vec{x}(t)=$ a vector of dependent variables. In order to do this, let $\vec{x}(t)=\begin{bmatrix} x(t)\\y(t)\\z(t) \end{bmatrix} $. Using the equation from before, we can set \[
  \begin{bmatrix} x'\\y'\\z'\\ \end{bmatrix} =\begin{bmatrix} 1&-1&1\\4&1&-5\\7&-8&-9 \end{bmatrix} +\begin{bmatrix} 0\\7\\0 \end{bmatrix}
.\] We can notice that all of our nonhomogeneous terms get moved into the $\vec{b}(t)$ matrix.
\begin{note}
  We might have $\begin{bmatrix} c \end{bmatrix} \vec{x}'=A\vec{x}+\vec{b}$. From here we would need to do the following
  \begin{align*}
    c^{-1}c\vec{x}'=c^{-1}(A\vec{x}+\vec{b})\\
    \vec{x}'=c^{-1}A\vec{x}+c^{-1}\vec{b}
  .\end{align*}
\end{note}
\end{eg}
\begin{eg}
  Consider the following ordinary differential equation and put it into matrix form.
  \begin{align*}
    x''+4x'-4x+4y&=0\\
    y''-4y'+5y-2x'+3x&=\sin(t)
  .\end{align*}
  In order to solve this we need to let $u=x'$, which makes $u'=x''$, and we need to let $v=y'$, which lets $u'=-4x'+4x-4y$ and $v'=y''=4y'-5y+2u-3x+\sin(t)$. This makes our system
  \begin{align*}
    u'&=-4u+4x-4y\\
    v'&=4v-5y+2u-3x+\sin(t)\\
    x'&=u\\
    y'&=v
  .\end{align*}
  Now we can turn this into our $\vec{x'}=A\vec{x}+\vec{b}$. This makes our equation \[
    \begin{bmatrix} u'\\v'\\x'\\y'\\ \end{bmatrix} =\begin{bmatrix} -4&0&4&-4\\2&4&-3&-5\\1&0&0&0\\0&1&0&0 \end{bmatrix} +\begin{bmatrix} 0\\ \sin(t)\\0\\0 \end{bmatrix}
  .\]
\end{eg}

\subsection{Eigenvalues and Eigenvectors of Square Matrices}
We are going to be using $\lambda$ for the eigenvalues and $\vec{v}$ for the eigenvectors.
\begin{definition}
  $\lambda$ and $\vec{v}$ are an eigenvalue/eigenvector of $A$ if \[
  A\vec{v}=\lambda\vec{v}, v\neq 0
  .\]
\end{definition}

Our steps to find the eigenvector are the following:
\begin{enumerate}
  \item Find $\lambda$'s with $det(A-\lambda I)=0$ (polynomial in $\lambda$).
  \item For each $\lambda$ find its eigenvector $\vec{v}$ using $A\vec{v}=\lambda\vec{v}$ or $(A-\lambda I)\vec{v}=0$
\end{enumerate}
We need to make sure that $\vec{v}$ is a non-zero solution to the equation. This means that we will have infinite solutions because every multiple of an eigenvector is also an eigenvector.
\begin{eg}
  For $A=\begin{bmatrix} 4&2\\5&1 \end{bmatrix}$, find eigenvalues and eigenvectors. In order to find the eigen vector we take the deteminant of $A-\lambda I$ and set it equal to zero.
  \begin{align*}
    det(\begin{bmatrix} 4&2\\5&1 \end{bmatrix} -\lambda\begin{bmatrix} 1&0\\0&1 \end{bmatrix} )&=0\\
    \left| \begin{matrix} 4-\lambda&2\\5&1-\lambda \end{matrix} \right| &=0\\
    (4-\lambda)(1-\lambda)-10&=0\\
    4-5\lambda+\lambda^2-10&=0\\
    \lambda^2-5\lambda-6&=0\\
    \lambda&=6,-1
  .\end{align*}
  Now we need to solve for the eigenvectors. First, let $\lambda=6$. Let $\vec{v}=\begin{bmatrix} v_1\\v_2 \end{bmatrix}$.
  \begin{align*}
    \begin{bmatrix} 4-6&2\\5&1-6 \end{bmatrix} \begin{bmatrix} v_1\\v_2 \end{bmatrix} &=\begin{bmatrix} 0\\0 \end{bmatrix} \\
    \begin{bmatrix} -2&2\\5&-5 \end{bmatrix} \begin{bmatrix} v_1\\v_2 \end{bmatrix} =\begin{bmatrix} 0\\0 \end{bmatrix}
  .\end{align*}
  If we put this back into equation form we get
  \begin{align*}
    -2v_1+2v_2&=0\\
    5v_1-5v_2&=0\\
    \to v_1=v_2
  .\end{align*}
  So this means that $v_2\begin{bmatrix} 1\\1 \end{bmatrix} $ is an eigenvalue for $\lambda=6$ for all $v_2\neq 0$. Let's now solve for $\lambda=1:$
  \begin{align*}
    \begin{bmatrix} 5&2\\5&2 \end{bmatrix} \begin{bmatrix} v_1\\v_2 \end{bmatrix} &=\begin{bmatrix} 0\\0 \end{bmatrix} \\
    5v_1-1v_2=0\to v_2=\frac{-5}{2}v_1
  .\end{align*}
  This means that any multiple of the vector $\begin{bmatrix} 1\\-\frac{5}{2} \end{bmatrix}$ is an eigenvector for $\lambda=1$.
\end{eg}

Let's show that if $\vec{v}$ is an eigenvector for $A$ with $\lambda$, then $c\vec{v}$ is an eigenvector for $A$ with $\lambda$ if $c\neq 0.$. We know that $A\vec{v}=\lambda\vec{v}$. Is $A(c\vec{v})=\lambda(c\vec{v})$? Yes.
Let $X(t)=\begin{bmatrix} x(t)\\y(t) \end{bmatrix} $. If we solve for $\vec{X}'=A\vec{x}$,
\begin{align*}
  \begin{bmatrix} x\\y \end{bmatrix} '=\begin{bmatrix} 4&2\\5&1 \end{bmatrix} \begin{bmatrix} x\\y \end{bmatrix} \\
  x'=4x+2y\\
  y'=5x+y
.\end{align*}
We can now write down our solutions as
\begin{align*}
  \vec{X}_1(t)=e^{6t}\begin{bmatrix} 1\\1 \end{bmatrix} =\begin{bmatrix} e^{6t}\\e^{6t} \end{bmatrix} \\
  \vec{X}_2(t)=e^{-t}\begin{bmatrix} -2\\5 \end{bmatrix} =\begin{bmatrix} -2e^{-t}\\5e^{-t} \end{bmatrix}
.\end{align*}
This means that we can determine that \[
  \vec{X}(t)=c_1\vec{x}_1+c_2\vec{x}_2
\] is also a solution for all $c_1,c_2$. This means that our general solution is going to be \[
\vec{X}=c_1e^{6t}\begin{bmatrix} 1\\1 \end{bmatrix} +c_2e^{-t}\begin{bmatrix} -2\\5 \end{bmatrix}
.\] We can also take our $c_1,c_2$ and distribute them to get \[
\vec{X}(t)=\begin{bmatrix} c_1e^{6t}-2c_2e^{-2}\\c_2e^{6t}+5c_2e^{-t} \end{bmatrix}
,\] with $x(t)$ being the top equation and $y(t)$ being the bottom equation. We can verify the solutions in order to convince ourselves that this works. We can verify by using our $\begin{bmatrix} -2e^{-t}\\5e^{-t} \end{bmatrix} $. Our left hand side of the equation is going to be \[
\vec{X}_2'=-e^{-t}\begin{bmatrix} -2\\5 \end{bmatrix} =e^{t}\begin{bmatrix} 2\\-5 \end{bmatrix}
.\] Our right hand side of the original equation is going to be \[
A\vec{X}_2 = \begin{bmatrix} 4&2\\5&1 \end{bmatrix} e^{t}\begin{bmatrix} -2\\5 \end{bmatrix}
.\] After simplification of this equation, we get that the result is $e^{t}\begin{bmatrix} 2\\-5 \end{bmatrix} $.
\begin{eg}
  Find the particular solution satisfying $x(0)=2,y(0)=-2$ for the general solution above. Now we just go through the following steps.
  \begin{align*}
    \vec{X}(0)=\begin{bmatrix} 2\\-2 \end{bmatrix} =c_1\begin{bmatrix} 1\\1 \end{bmatrix} +c_2\begin{bmatrix} -2\\5 \end{bmatrix} \\
    2&=c_1-2c_2\\
    -2&=c_1+5c_2
  .\end{align*}
  Now we just solve for $c_1,c_2$ to get $c_1=\frac{6}{7},c_2=-\frac{4}{7}$, so our general solution is \[
    \vec{X}(t)=\frac{6}{7}e^{6t}\begin{bmatrix} 1\\1 \end{bmatrix} -\frac{4}{7}e^{-t}\begin{bmatrix} -2\\5 \end{bmatrix}
  .\]
\end{eg}

How can we verify that two solutions are linearly independent\footnote{This is a 4.4 idea but we are not quite there yet.}? We can solve this by using the wronskian. For the last equation we can check this by doing \[
  w(t)=\left| \begin{matrix} e^{6t}&-2e^{-t}\\e^{6t}&5e^{-t} \end{matrix} \right| = 5e^{5t}+2e^{5t}\neq 0
.\] Because the wronskian does not equal zero, the solutions are linearly independent.\newline

\begin{eg}
  \begin{enumerate}
    \item Is $\vec{v}=\begin{bmatrix} 3\\2 \end{bmatrix}$ an eigenvector for $A=\begin{bmatrix} 2&3\\2&1 \end{bmatrix} $. This can be tested by checking to see if $A\vec{v}=\lambda\vec{v}$ for some $\lambda$?\[
      A\vec{v}=\begin{bmatrix} 2&3\\2&1 \end{bmatrix} \begin{bmatrix} 3\\2 \end{bmatrix} =\begin{bmatrix} 12\\8 \end{bmatrix} =\lambda\begin{bmatrix} 3\\2 \end{bmatrix}
    .\] This is true for all $\lambda=4$
    \item Is $\lambda=-1$ and eigenvalue for $A=\begin{bmatrix} 2&3\\2&1 \end{bmatrix} $? We can check this by asking if $det(A-(-1)I)=0$?\[
      \left| \begin{matrix} 2+1&3\\2&1+1 \end{matrix} \right| =\left| \begin{matrix} 3&3\\2&2 \end{matrix} \right| =6-6=0
    .\] Because this determinant equals zero, $\lambda=-1$ is an eigenvalue of $A$
  \end{enumerate}
\end{eg}

For repeated (real) eigenvalues, we can have many different options that can come out. For a $2\times 2, \lambda_1=\lambda_2$. Here are our options
\begin{enumerate}
  \item Only one eigenvector (and its multiples).
  \item 2 linearly independent eigenvectors.
\end{enumerate}
For option 2, an example would be
\begin{align*}
  \vec{X}_1=e^{\lambda t}\begin{bmatrix} 2\\3 \end{bmatrix} \\
  \vec{X}_2=e^{\lambda t}\begin{bmatrix} 4\\0 \end{bmatrix}
.\end{align*}
Our general solution is going to be \[
  \vec{X}(t)=C_1\vec{X}_1+C_2\vec{X}_2=e^{2t}\begin{bmatrix} 2C_1+4C_2\\3C_1 \end{bmatrix}
.\] This means that every vector is an eigenvector. This only occurs if our eigenvector is a multiple of the identity matrix.
 \begin{eg}
  Find all eigenpairs of the matrix $A=\begin{bmatrix} 4&5\\-2&6 \end{bmatrix} $. First we need to find the eigenvalues by doing
  \begin{align*}
    \left| \begin{matrix} 4-\lambda&5\\-2&6-\lambda \end{matrix} \right| &=(4-\lambda)(6-\lambda)+10\\
                                   &=24-10\lambda+\lambda^2+10\\
                                   &=\lambda^2-10\lambda+34=0
  .\end{align*}
  We can solve this by completing the square by doing
  \begin{align*}
    (\lambda^2-10\lambda+25)&=-34+25\\
    (\lambda-5)^2&=-9\\
    |\lambda-5|&=3i\\
    \lambda-5&=\pm 3i\\
    \lambda &= 5\pm 3i
  .\end{align*}
  Now we need to find the eigenvectors by solving $A\vec{v}=\lambda\vec{v}$ or doing $(A-\lambda I)\vec{v}=\vec{0}$. Now solving for $\lambda_1=5+3i$.
  \begin{align*}
    \begin{bmatrix} 4-(5+3i)&5\\-2&6-(5+3i) \end{bmatrix} \begin{bmatrix} v_1\\v_2 \end{bmatrix} &=\begin{bmatrix} 0\\0 \end{bmatrix} \\
    \begin{bmatrix} -1-3i&5\\-2&1-3i \end{bmatrix} \begin{bmatrix} v_1\\v_2 \end{bmatrix} &=\begin{bmatrix} 0\\0 \end{bmatrix} \\
    (-1-3i)v_1+5v_2=0\\
    -2v_1+(1-3i)v_2=0
  .\end{align*}
  The two last equations above are equivalent so we can work with either of them, we are going to be working with $5v_2=(1+3i)v_1$. If we choose $v_1=5,v_2=1+\lambda i$ we can solve $v_2=\frac{1+3i}{5}v_1$ for all $v_1$.
\end{eg}
\begin{eg}
  Show that $\vec{v}=\begin{bmatrix} 2+2i\\-1 \end{bmatrix} $ is an eigenvector of $A=\begin{bmatrix} 2&8\\-1&-2 \end{bmatrix} $ We can set $A\vec{v}=\lambda\vec{v}$ for some $\lambda$ to get \[
  \begin{bmatrix} 2&8\\-1&-2 \end{bmatrix} \begin{bmatrix} 2+2i\\-1 \end{bmatrix} =\begin{bmatrix} 4+4i-8\\-2-2i+3 \end{bmatrix} =\begin{bmatrix} 4i-4\\-2i \end{bmatrix} \stackrel{?}{=}\lambda\begin{bmatrix} 2+2i\\-1 \end{bmatrix}
  .\]
  We can check to see if $\lambda=2i$ by doing
  \begin{align*}
    2i&=(2+2i)\stackrel{?}{=}4i-4\\
    4i-4&=4i-4
  .\end{align*}
  This means that $\vec{v}$ is an eigenvector for $A$ with $\lambda=2i$.
\end{eg}
How can we turn a complex eigenpair of $A$ into a solution of $\vec{x}'=A\vec{x}$? Last time we found that $\lambda_1=5+3i$ had $\vec{v}_1=\begin{bmatrix} 5\\1+3i \end{bmatrix} \text{ or }\begin{bmatrix} 1-3i\\2 \end{bmatrix} $
\begin{note}
  We know that $\vec{x}_1(t)=e^{(5+3i)t}\begin{bmatrix} 5\\1+3i \end{bmatrix} $ is a solution, and so is $\vec{x}_1(t)=e^{5-3i)t}\begin{bmatrix} 5\\1-3i \end{bmatrix} $ is a solution, so \[
  \vec{x}(t)=C_1\vec{x}_1(t)=C_2\vec{x}_2(t)
  .\]
\end{note}
How are we able to turn these into real-valued solutions? The real and imaginary parts of a solution must also be solutions. We can do the following calculations
\begin{align*}
  \vec{x}_1(t)&=e^{5t}e^{3it}\begin{bmatrix} 5\\1+3i \end{bmatrix} \\
              &=e^{5t}\left( \cos(3t)+i\sin(3t) \right)\begin{bmatrix} 5\\1+3i \end{bmatrix} \\
              &=e^{5t}\begin{bmatrix} 5\cos(3t)+5i\sin(3t)\\ \cos(3t)+i\sin(3t)+3i\cos(3t)-3\sin(3t) \end{bmatrix}\\
              &=e^{5t}\begin{bmatrix} 5\cos(3t)\\ \cos(3t)-3\sin(3t) \end{bmatrix} +ie^{5t}\begin{bmatrix} 5\sin(3t)\\3\cos(3t)+\sin(3t) \end{bmatrix}
.\end{align*}
The two sides of the addition are considered $\vec{x}_3(t),\vec{x}_4(t)$ respectively. ($x_4$ does not include the $i$). This makes our general solution \[
  \vec{x}(t)=C_3\vec{x}_3(t)+C_4\vec{x}_4(t)
.\]
\begin{eg}
  Find the eigenvalues of $A=\begin{bmatrix} 2&-7&0\\5&10&4\\0&5&2 \end{bmatrix} $. First we need to find the eigenvalues by doing $det(A-\lambda I)=0.$
  \begin{align*}
    \left| \begin{matrix} 2-\lambda&-7&0\\5&10-\lambda&4\\0&5&2-\lambda \end{matrix} \right| &= (2-\lambda)\left| \begin{matrix} 10-\lambda&4\\5&2-\lambda \end{matrix} \right| + (7) \left| \begin{matrix} 5&4\\0&2-\lambda \end{matrix} \right| \\
                                   &=(2-\lambda)[(10-\lambda)(2-\lambda)-20]+7.5(2-\lambda)\\
                                   &=(2-\lambda)[20-12\lambda+\lambda^2-20+35]\\
                                   &=(2-\lambda)(\lambda^2-12\lambda+35)\\
                                   &=(2-\lambda)(\lambda-5)(\lambda-7)=0\\
    \lambda&=2,5,7
  .\end{align*}
\end{eg}
How can we find a second linearly independent solution to $\vec{x}'=A\vec{x}$ if $A$ only had 1 eigenpair?
\begin{eg}
  Consider the equation $B=\begin{bmatrix} 3&-18\\2&-9 \end{bmatrix} $, which has the eigenvalues $\lambda=-3,-3$ with an eigenvector of $\vec{v}=\begin{bmatrix} 3\\1 \end{bmatrix} $. We know that $\vec{x}_1=(t)e^{-3t}\begin{bmatrix} 3\\1 \end{bmatrix} $ is one solution. We need to find $\vec{x}_2$. We can say that $\vec{x}_2(t)=te^{\lambda t}\vec{v}+e^{\lambda t}\vec{w}$, where $(B-\lambda I)\vec{w}=\vec{v}$ Let's find our $\vec{w}$.
  \begin{align*}
    \begin{bmatrix} 3-(-3)&-18\\2&-9-(-3) \end{bmatrix} \begin{bmatrix} w_1\\w_2 \end{bmatrix} =\begin{bmatrix} 3\\1 \end{bmatrix} \\
    \begin{bmatrix} 6&-18\\2&-6 \end{bmatrix} \begin{bmatrix} w_1\\w_2 \end{bmatrix} =\begin{bmatrix} 3\\1 \end{bmatrix} \\
    6w_1-18w_2&=3\\
    2w_1-6w_2&=1
  .\end{align*}
  We can choose any of the $w_1,w_2$ combinations that work. If we choose that $w_2=0,w_1=\frac{1}{2}$. This makes our general solution \[
    \vec{x}(t)=Ce^{-3t}\begin{bmatrix} 3\\1 \end{bmatrix} +C_2e^{-3t}\left( t\begin{bmatrix} 3\\1 \end{bmatrix} +\begin{bmatrix} 0\\\frac{1}{2} \end{bmatrix}  \right)
  .\]
\end{eg}
\begin{eg}
  Let's solve $\vec{x}'=\begin{bmatrix} 0&1\\0&6 \end{bmatrix} \vec{x}$ for $\vec{x}(t)$. The first step we need to take is calculating the eigenvalues of the matrix.
  \begin{align*}
    det(A-\lambda I)=\left| \begin{matrix} 0-\lambda&1\\0&6-\lambda \end{matrix} \right| =0\\
    -\lambda(6-\lambda)=0\\
    \lambda=0,6
  .\end{align*}
  For $\lambda=0$:
  \begin{align*}
    \begin{bmatrix} 0&1\\0&6 \end{bmatrix} \begin{bmatrix} v_1\\v_2 \end{bmatrix} &=\begin{bmatrix} 0\\0 \end{bmatrix} \\
    v_2&=0\\
    6v_2&=0\\
    \vec{v}=\begin{bmatrix} v_1\\0 \end{bmatrix} &=v_1\begin{bmatrix} 1\\0 \end{bmatrix}
  .\end{align*}
  For $\lambda=6$
  \begin{align*}
    \begin{bmatrix} -6&1\\0&0 \end{bmatrix} \begin{bmatrix} v_1\\v_2 \end{bmatrix} &=\begin{bmatrix} 0\\0 \end{bmatrix} \\
    -6v_1+v_2&=0\\
    0&=0\\
    v_2&=6v_1\\
    \vec{v}&=\begin{bmatrix} v_1\\6v_1 \end{bmatrix}=v_1\begin{bmatrix} 1\\6 \end{bmatrix}
  .\end{align*}
  This will make our general solution be \[
    \vec{x}(t)=C_1e^{0t}\begin{bmatrix} 1\\0 \end{bmatrix} +C_2e^{6t}\begin{bmatrix} 1\\6 \end{bmatrix} = \begin{bmatrix} C_1+C_2e^{6t}\\6C_2e^{6t} \end{bmatrix}
  .\]
\end{eg}
\subsection{Undetermined Coefficients for Linear Systems}
\begin{eg}
  Let's take a look at the differential equation \[
    \vec{x}'=\begin{bmatrix} 3&-18\\2&-9 \end{bmatrix} \vec{x}+\begin{bmatrix} 2t\\0 \end{bmatrix}
  .\]
  This ordinary differential equation has the homogeneous solution of \[
    \vec{x}_h(t)=C_1e^{-3t}\begin{bmatrix} 3\\1 \end{bmatrix} +C_2\left[ te^{-3t}\begin{bmatrix} 3\\1 \end{bmatrix} +e^{-3t}\begin{bmatrix} \frac{1}{2}\\0 \end{bmatrix}  \right]
  .\]
  If we were to make our initial $\vec{x}_p(t)$ guess, we should guess $\begin{bmatrix} At+B\\Ct+D \end{bmatrix} $. Now we need to find $\vec{x}_p'=\begin{bmatrix} A\\C \end{bmatrix}$ and plug it into the left hand side of the equation. The right hand side will now equal \[
  \begin{bmatrix} 3&-18\\2&-9 \end{bmatrix} \begin{bmatrix} At+B\\Ct+D \end{bmatrix} +\begin{bmatrix} 2t\\0 \end{bmatrix}
  .\] This means that this should be equal \[
  \begin{bmatrix} A\\C \end{bmatrix} =\begin{bmatrix} 3At+3B-18Ct-18D+2t\\2At+2B-9Ct-9D \end{bmatrix}
  .\] So
  \begin{align*}
    A=3At+3B-18Ct-18D+2t\\
    C=2At+2B-9Ct-9D\\
  .\end{align*}
  We can separate our coefficients with 1's and t's like so
  \begin{align*}
    1:A&=3B-18D\\
    C&=2B-9D\\
    t: 0=3A-18C+2\\
    0=2A-9C
  .\end{align*}
  Aftering doing algebra and simplifying we get that $A=2,C=\frac{4}{9},B=-\frac{10}{9},D=-\frac{8}{27}$, so \[
    \vec{x}_p=\begin{bmatrix} 2t&-\frac{10}{9}\\\frac{4}{9}t&-\frac{8}{27} \end{bmatrix}
  .\]
\end{eg}
\subsection{Modeling with Systems}
Let's start off by looking at an example.
\begin{eg}
  \begin{figure}[h!]
    \centering
    \includegraphics[width=0.8\textwidth]{resource/images/4.4.2 Example 1.jpg}
    \caption{}
    \label{fig:}
  \end{figure}
  Tanks $A$ and $B$ both have 100 gallons of brine. $A$ concentration $0.2 \frac{lb}{gal}$ salt, $B$ concentration is $0.3 \frac{lb}{gal}$. Let $x(t)$ and $y(t)$ be the amounts of salt in tanks $A$ and $B$ in pounds at time $t$(min).
  \begin{align*}
    x'=0+\left( 2 \frac{gal}{min} \right) \left( \frac{y}{100} \frac{lb}{gal} \right)-5 \frac{x}{100} \\
    y'=3 \frac{x}{100}- \frac{3y}{100}\\
    x(0)=20\\
    y(0)=30
  .\end{align*}
  This gives us that \[
    \begin{bmatrix} x\\y \end{bmatrix} '=\begin{bmatrix} -\frac{1}{20}&\frac{1}{50}\\\frac{3}{100}&-\frac{3}{100} \end{bmatrix} \begin{bmatrix} x\\y \end{bmatrix}
  .\]
\end{eg}


\section{Complex LMS and Widely Linear Modelling}

\begin{enumerate}[label=\alph*), leftmargin=*]

%% a)
\item
%

A Wide Linear Moving Average order 1, WLMA(1), process $y(n)$ is generated, whose \texttt{real} and \texttt{imag} parts are provided in figure \ref{fig:4_1_a}.
As expected, the process is not circular, and we expect simple Complex Least Mean Squared (CLMS) algorithm to fail to model it.

The mean squared prediction error curve in figure \ref{fig:4_1_a} agrees with our assumption, though the Augemented Complex Least Mean Squared (ACLMS) algorithm
successfully learns the process parameters. In more detail, the ACLMS achieves an $MPSE = -300dB$ in steady-state, while CLMS saturates at $MPSE = 7.6$. This is
the consequence of the inability of the CLMS to adapt to non-circular signals, due to its limited capacity.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/a/comparison}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/a/learning_curves}
    \end{subfigure}
    \caption{CLMS vs ACLMS: non-circularity and (A)CLMS learning curves.}
    \label{fig:4_1_a}
\end{figure}

%% b)
\item
%

Figures \ref{fig:4_1_b_1}, \ref{fig:4_1_b_2}, \ref{fig:4_1_b_3} illustrate the circularity plots (\texttt{real}-\texttt{imag} scatter plots) along with the learning curves
of the ACLMS and CLMS algorithms for different model orders, on the \texttt{wind-dataset}.

Despite its non-obviously symmetric shape, the low regime wind data has lowest circularity coefficient, $\rho = 0.159$, while the medium and high regimes have
$\rho = 0.454$ and $\rho = 0.624$, repsectively. A complex-valued random variable is said to be circular if its probability distribution is not dependent on the angle, that is,
the distribution is rotationally invariant. In other words the variable’s probability distribution function should only depend on the Euclidean distance from the origin in the complex domain.
By definition, the higher the circularity coefficient, $\rho$, the less circular the data is.

Hence, as shown in the previous part, ACLMS algorithm outperforms CLMS on non-circular data. This is verified again on the \texttt{wind-dataset}, where the ACLMS has lower
mean squared prediction error (MPSE) for any wind regime. More interesting is the fact that at high regime data, the least circular, the ACLMS has a larger margin in performance
($0.1 dB$). Lastly, we observe that the MPSE is minimised for small model orders, $M \in [3, 6]$, since over-modelling leads to over-fitting the noise and fail to generalise, despite
the excess degrees of freedom. For larger model order, $M > 10$, we observe that the ACLMS performs worse than the CLMS, due to its extra capacity (more parameters), which lead to
overfitting.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/b/circularity_1}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/b/learning_curves_1}
    \end{subfigure}
    \caption{Low Regime:circularity plot and (A)CLMS learning curves.}
    \label{fig:4_1_b_1}
\end{figure}

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/b/circularity_2}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/b/learning_curves_2}
    \end{subfigure}
    \caption{Medium Regime:circularity plot and (A)CLMS learning curves.}
    \label{fig:4_1_b_2}
\end{figure}

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/b/circularity_3}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/b/learning_curves_3}
    \end{subfigure}
    \caption{High Regime:circularity plot and (A)CLMS learning curves.}
    \label{fig:4_1_b_3}
\end{figure}

%% c)
\item
%

Complex voltages are synthetically generated for various magnitude, angle combinations leading to balanced and unbalanced configurations, while their illustrations in the complex plane
are provided in figures \ref{fig:4_1_c_1}, \ref{fig:4_1_c_2}. Overall, we notice that balanced systems have a circular shape, while unbalanced don't.
This is also collaborated by the fact that the balanced system has circularity coefficient $\rho = 0$, while the unbalanced system has a high circularity coefficient, $\rho = 0.757$.
Finally, the impact of the angle and magnitude distortions, $\Delta_{b} \neq \Delta_{c} \neq 0$ and $V_{a} \neq V_{b} \neq V_{c}$, respectively, is depicted in figure \ref{fig:4_1_c_2}.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/c/balanced}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/c/unbalanced}
    \end{subfigure}
    \caption{Complex Voltage: balanced and unbalanced voltages.}
    \label{fig:4_1_c_1}
\end{figure}

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/c/unbalanced_angle}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/c/unbalanced_magnitude}
    \end{subfigure}
    \caption{Complex Voltage: unbalanced voltages angle and magnitude distortions.}
    \label{fig:4_1_c_2}
\end{figure}

%% d)
\item
%

Balanced complex $\alpha\ -\ \beta$ voltages satisfy:

\begin{equation}
    u(n) = \sqrt{\frac{3}{2}} V e^{j(2\pi \frac{f_{o}}{f_{s}} n + \phi)}
\end{equation}

For time index $n+1$:

\begin{align}
    u(n+1)  &= \sqrt{\frac{3}{2}} V e^{j(2\pi \frac{f_{o}}{f_{s}} (n+1) + \phi)} \\
            &= \sqrt{\frac{3}{2}} V e^{j(2\pi \frac{f_{o}}{f_{s}} n + \phi)} e^{j 2 \pi \frac{f_{o}}{f_{s}}} \\
            &= u(n) e^{j 2 \pi \frac{f_{o}}{f_{s}}}
\label{eq:u_n+1}
\end{align}

Using the strictly linear autoregressive model of order 1, equation, satisfying:

\begin{equation}
    u(n+1) = u(n) h^{*}(n)
\end{equation}

we express the complex exponential in (\ref{eq:u_n+1}) as a function of the model parameter $h(n)$:

\begin{align}
    e^{j 2 \pi \frac{f_{o}}{f_{s}}} &= h^{*}(n) \\
    e^{-j 2 \pi \frac{f_{o}}{f_{s}}}&= h(n) \\
                                    &= | h(n) | e^{j\big( arctan \big(\frac{\Im\{h(n)\}}{\Re\{h(n)\}} \big) \big)}
\end{align}

Two complex numbers are equal if and only of both their magnitudes and their angles are equal, therefore we set the angles to be equal, obtaining:

\begin{align}
    2 \pi \frac{f_{o}}{f_{s}} = arctan \big(\frac{\Im\{h(n)\}}{\Re\{h(n)\}} \big)
\end{align}

Solving for $f_{o}$ concludes the proof:

\begin{equation}
    f_{o} = \frac{f_{s}}{2\pi} arctan \big(\frac{\Im\{h(n)\}}{\Re\{h(n)\}} \big)
\label{proof:fo_CLMS}
\end{equation}

The unbalanced system satisfies:

\begin{equation}
    u(n) = A(n) e^{j(2\pi \frac{f_{o}}{f_{s}} n + \phi)} + B(n) e^{-j(2\pi \frac{f_{o}}{f_{s}} n + \phi)}
\label{eq:u_unbalanced}
\end{equation}

Using the widely linear autoregressive model of order 1, equation, satisfying:

\begin{equation}
    u(n+1) = h^{*}(n) u(n) + g^{*}(n) u^{*}(n)
\end{equation}

we substitute (\ref{eq:u_unbalanced}) terms:

\begin{align}
    u(n+1) =\
        &h^{*}(n) \bigg[ A(n) e^{j(2\pi \frac{f_{o}}{f_{s}} n + \phi)} + B(n) e^{-j(2\pi \frac{f_{o}}{f_{s}} n + \phi)} \bigg] + \nonumber\\
        &g^{*}(n) \bigg[ A^{*}(n) e^{-j(2\pi \frac{f_{o}}{f_{s}} n + \phi)} + B^{*}(n) e^{j(2\pi \frac{f_{o}}{f_{s}} n + \phi)} \bigg]
\label{eq:u_n+1_ar}
\end{align}

For time index $n+1$, the complex voltage satisfies:

\begin{equation}
    u(n+1) = A(n+1) e^{j(2\pi \frac{f_{o}}{f_{s}} (n+1) + \phi)} + B(n+1) e^{-j(2\pi \frac{f_{o}}{f_{s}} (n+1) + \phi)}
\label{eq:u_n+1_clarke}
\end{equation}

Equating (\ref{eq:u_n+1_ar}) and (\ref{eq:u_n+1_clarke}), collecting common exponential terms:

\begin{align}
    A(n+1) e^{j(2\pi \frac{f_{o}}{f_{s}} (n+1) + \phi)} &= \bigg[ h^{*}(n) A(n) + g^{*}(n) B^{*}(n) \bigg] e^{j(2\pi \frac{f_{o}}{f_{s}} n + \phi)} \\
    B(n+1) e^{-j(2\pi \frac{f_{o}}{f_{s}} (n+1) + \phi)} &= \bigg[ h^{*}(n) B(n) + g^{*}(n) A^{*}(n) \bigg] e^{-j(2\pi \frac{f_{o}}{f_{s}} n + \phi)}
\end{align}

Assuming that the amplitude change over time is negligible, or equivalently $A(n+1) \approx A(n)$ and $B(n+1) \approx B(n)$, the equations simplify as:

\begin{align}
    e^{j 2\pi \frac{f_{o}}{f_{s}}} = \frac{h^{*}(n) A(n) + g^{*}(n) B^{*}(n)}{A(n+1)} \approx h^{*}(n) + g^{*}(n) \frac{B^{*}(n)}{A(n)} \label{eq:e_A}\\
    e^{-j 2\pi \frac{f_{o}}{f_{s}}} = \frac{h^{*}(n) B(n) + g^{*}(n) A^{*}(n)}{B(n+1)} \approx h^{*}(n) + g^{*}(n) \frac{A^{*}(n)}{B(n)} \label{eq:e_B}
\end{align}

Note that (\ref{eq:e_A}) is the complex conjugate of (\ref{eq:e_B}, we reach:

\begin{align}
    h^{*}(n) + g^{*}(n) \frac{B^{*}(n)}{A(n)} = h(n) + g(n) \frac{A(n)}{B^{*}(n)}
\end{align}

Multiplying with $\frac{B^{*}(n)}{A(n)}$ both sides:

\begin{align}
    \bigg( h^{*}(n) - h(n) \bigg) \frac{B^{*}(n)}{A(n)} + g^{*}(n) \bigg( \frac{B^{*}(n)}{A(n)} \bigg)^{2} - g(n) = 0
\label{eq:quad}
\end{align}

We note that (\ref{eq:quad}) is quadratic in $\frac{B^{*}(n)}{A(n)}$, hence solving it we obtain:

\begin{align}
    \frac{B^{*}(n)}{A(n)}   &= \frac{- \bigg( h^{*}(n) - h(n) \bigg) \pm \sqrt{\bigg( h^{*}(n) - h(n) \bigg)^{2} + 4 g^{*}(n) g(n)}}{2g^{*}(n)} \\
                            &= \frac{2 \Im\{h(n)\}j \pm j\sqrt{-4 \Im\{h(n)\}^{2} + 4 |g(n)|^{2}}}{2g^{*}(n)} \\
                            &= \frac{\Im\{h(n)\}j \pm j\sqrt{\Im\{h(n)\}^{2} - |g(n)|^{2}}}{g^{*}(n)}
\end{align}

Substitution in (\ref{eq:e_A}), yields:

\begin{align}
    e^{j 2\pi \frac{f_{o}}{f_{s}}}  &= h^{*}(n) + \Im\{h(n)\}j \pm j\sqrt{\Im\{h(n)\}^{2} - |g(n)|^{2}} \\
                                    &= \Re\{h(n)\} \pm j\sqrt{\Im\{h(n)\}^{2} - |g(n)|^{2}} \\
\end{align}

Keeping one solution ($+$ sign), since $f_{s} \gg f_{o} > 0$:

\begin{align}
    e^{j 2\pi \frac{f_{o}}{f_{s}}}  &= \Re\{h(n)\} + j\sqrt{\Im\{h(n)\}^{2} - |g(n)|^{2}} \\
                                    &= \rho e^{j\big(\frac{\sqrt{\Im\{h(n)\}^{2} - |g(n)|^{2}}}{\Re\{h(n)\}}\big)}
\end{align}

where $\rho > 0$. Set the angles to be equal and solving for $f_{o}$ we complete the proof:

\begin{equation}
    f_{o} = \frac{f_{s}}{2 \pi} arctan \bigg(\frac{\sqrt{\Im\{h(n)\}^{2} - |g(n)|^{2}}}{\Re\{h(n)\}}\bigg)
\label{proof:fo_ACLMS}
\end{equation}

%% e)
\item
%

First order CLMS and ACLMS filters are trained on both the balanced and unbalanced (magnitude and angle) complex voltage synthetic data.
The derived formulae (\ref{proof:fo_CLMS}) and (\ref{proof:fo_ACLMS}) are used along with the learnt filter parameters, $h_{CLMS}$, $h_{ACLMS}$ \& $g_{ACLMS}$,
while the frequency estimates over time and the error curves are provided in figures \ref{fig:4_1_e_1}, \ref{fig:4_1_e_2} for the balanced and unbalanced complex voltage, respectively.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/e/balanced_error}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/e/balanced_frequency}
    \end{subfigure}
    \caption{Balanced Complex Voltage: mean squared prediction error (MPSE) and frequency estimation.}
    \label{fig:4_1_e_1}
\end{figure}

In case of the balanced grid, the CLMS filter excels, since it both converges faster to the nominal frequency $f_{o} = 50 Hz$, without overshooting (transient behaviour).
On the other hand, the ACLMS oscillates for the first $300$ timesteps and finally converges to the true frequency after that.
This is an expected result, since in figure \ref{fig:4_1_c_1} we showed that the balanced system has a circular complex voltage, which can be sufficiently modelled by a CLMS filter.
The extra degrees of ACLMS freedom, put a burden on the model, which needs more time to converge.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/e/unbalanced_error}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/complex-LMS-and-widely-linear-modelling/assets/e/unbalanced_frequency}
    \end{subfigure}
    \caption{Unbalanced Complex Voltage: mean squared prediction error (MPSE) and frequency estimation.}
    \label{fig:4_1_e_2}
\end{figure}

Figure \ref{fig:4_1_e_2} depicts the mean squared prediction error and frequency estimates for the unbalanced system,
with $(V_{a}, V_{b}, V_{c}) = (0.1, 1.0, 1.9)$ and $(\Delta_{b}, \Delta_{c}) = (2.0, 0.5)$.
In this scenario, the ACLMS converges to the true nominal frequency value after 450 timesteps, though the CLMS filter oscillates around $\hat{f}_{o} = 37 Hz$ and never adapts to the true $f_{o}$.
Unsurprisingly, CLMS does not have the capacity to model non-circular distributions and hence fails to model unbalanced configurations ($\rho = 0.757$).

%
\end{enumerate}


\subsection{Special Case: 1-hidden Layer}
First, let us define the so-called 1-hidden layer (shallow) neural network.
\begin{definition}The 1-hidden layer (shallow) neural network is defined as:
	$$
	{\rm DNN}_1 = \lbrace \phi~:~ \phi(x) = \sum_{i=1}^N a_i \sigma(w_i x + b_i ) + c, ~~ N \in \mathbb{N}^+\rbrace.
	$$
\end{definition}

To consistent with above notation, we can write it as:
$$
\phi = W^1 x^1 + b^1 = (a_1,\cdots,a_N) \sigma(W^0 x + b^0) + c \in {\rm DNN}_1.
$$

The first question about ${\rm DNN}_1$ is about the approximation properties for any continuous functions. Here we have the next theorem:
\begin{theorem}[Universal Approximation Property of Shallow Neural Networks] Let $\Omega$ be bounded, if any $f\in C(\bar \Omega)$, there exists a sequence $\phi_n \in {\rm DNN}_1$ such that
	$$
	\max_{x\in \bar \Omega} |\phi_n(x) - f(x)| \to 0, \quad n \to \infty.
	$$
	This provide that $\sigma$ is not a polynomial.
	On the other hand, let $\sigma$ be a non-polynomial Riemann integrable function and $\sigma\in L_{loc}^\infty(\mathbb{R})$ then we have
	$$
	\overline{\rm DNN}_1 = C(\bar \Omega).
	$$
\end{theorem}
Before the proof, let us give some notation.
We use $\mathbb{P}_m(\mathbb{R}^d)$ to define the polynomials
of $d$-variables of degree less than $m$.
Let $\alpha = (\alpha_1,\cdots,\alpha_d)$( $\alpha_i $ non-negative integers), we note $|\alpha| = \sum_{i=1}^d \alpha_i$ and
$$
x^{\alpha} = x_1^{\alpha_1}x_2^{\alpha_2}\cdots x_d^{\alpha_d}.
$$
\begin{lemma}\label{lemm:dsigma}
	Let $\sigma \in C^{\infty}(\Omega)$ (i.e. $\sigma$ is infinitely differentiable) and is not a polynomial, then for any $k \ge 0$ there exists $t_k \in \mathbb{R}$ such that
	$$
	\sigma^{(k)}(t_k) \neq 0.
	$$
\end{lemma}

Now we are going to give the proof of
\begin{proof}
	If $\sigma$ is a polynomial, say $\sigma \in \mathbb{P}_m(\mathbb{R})$, then we have that
	$$
	{\rm DNN}_1 \subset \mathbb{P}_m(\mathbb{R}^d).
	$$
	Thus, ${\rm DNN}_1$ cannot approximate polynomial of degree bigger than $m+1$. This implies that $\sigma$ cannot be polynomial if ${\rm DNN}_1$ has the approximation property.

	Now we prove that $\overline{\rm DNN}_1 = C(\bar \Omega)$ if $\sigma$ is not a polynomials.

	\begin{description}
		\item[Case 1] First, let us assume that $\sigma \in C^{\infty}(\Omega)$.

		\item[Fact 1:] We have the next relation:
		$$
		\frac{\partial}{\partial [w]_i} \left(\sigma(wx + b) \right) |_{w=0}= \sigma'(wx+b)\frac{\partial}{\partial [w]_i} (wx+b) = \sigma'(wx+b)x_i |_{w=0}.
		$$
		That is to say,
		$$
		\frac{\partial}{\partial [w]_i} \left(\sigma(wx + b) \right) |_{w=0} = \sigma'(b)x_i.
		$$
		By the lemma \ref{lemm:dsigma}, there exists a $b \in \mathbb{R}$ such that
		$$
		\sigma'(b) \neq 0.
		$$
		\item[Fact 2:] By using the definition of derivative, we have
		$$
		\frac{\partial}{\partial [w]_i} \left(\sigma(wx + b) \right) |_{w=0}=
		\lim_{n\to \infty} \frac{\sigma((0 + \frac{1}{n}e_i )\cdot x + b) - \sigma(b)}{\frac{1}{n}} = \lim_{n\to\infty} \phi_n(x),
		$$
		where
		$$
		\phi_n(x) = n \left(\sigma(\frac{1}{n}e_i \cdot x + b) - \sigma(b)\right) \in {\rm DNN}_1.
		$$
		This leads to the result that
		$$
		\sigma'(b)x_i = \lim_{n\to\infty} \phi_n(x) \in \overline{\rm DNN}_1,
		$$
		because of the definition that
		$$
		\overline{\rm DNN}_1 = {\rm DNN}_1 \cup \{f ~:~f = \lim_{n\to\infty} \phi_n(x), \phi_n \in {\rm DNN}_1 \}.
		$$
	\end{description}
	Follow there facts, we know that
	$$
	\sigma'(b)x_i \in {\rm DNN}_1,
	$$
	this leads to
	$$
	x_i \in 	\overline{\rm DNN}_1,
	$$
	because  $[\sigma'(b)]^{-1} \phi_n(x) \to x_i$.

	Thus we have
	$$
	\frac{\partial^2}{\partial [w]_1 \partial [w]_2} \left(\sigma(wx+b)\right) |_{w=0} = \sigma^{(2)}(b)x_1x_2.
	$$
	Using the Lemma \ref{lemm:dsigma} again, there exists a $b \in \mathbb{R}$ such that
	$$
	\sigma^{(2)}(b) \neq 0.
	$$
	This leads to
	$$
	x_1x_2 \in \overline{\rm DNN}_1 .
	$$
	Similarly, we can prove that
	$$
	x_1^{\alpha_1} x_2^{\alpha_2} \cdots x_d^{\alpha_d} \in \overline{\rm DNN}_1.
	$$
	This proves that ${\rm DNN}_1$ can approximate any polynomials.
	Combine with the Weierstrass theorem, ${\rm DNN}_1$ can approximate any continuous functions.

	Then we will finish the proof for any $\sigma$ as a non-polynomial Riemann integrable function and $\sigma\in L_{loc}^\infty(\mathbb{R})$.
\end{proof}

\iffalse
Given an activation function
\begin{equation}
  \label{activation}
\sigma: \mathbb R^1\mapsto \mathbb R^1
\end{equation}
We consider the following shallow neural network function class
\begin{equation}
  \label{ShallowNN}
	\Sigma_d(\sigma)=\mathrm{span}\left\{\sigma(\omega\cdot x+\theta):\omega\in\mathbb{R}^d,\theta\in\mathbb{R}\right\}.
\end{equation}

In this chapter, we will two theorems as follows.
\begin{theorem}
Let $\sigma$ be a non-polynomial Riemann integrable function and
$\sigma\in L_{loc}^\infty(\mathbb{R})$. Then $\Sigma_d(\sigma)$ in dense in
$C(\mathbb R^d)$.
\end{theorem}
\fi

\begin{theorem}
Let $\Omega\subset \mathbb{R}^d$ be a bounded set, $\sigma\in W^{m,\infty}(\mathbb R)$ that has
compact support such that
\begin{equation}
  \label{eq:1}
  \hat\sigma(a)\neq 0, \mbox{ for some } a\neq 0.
\end{equation}
Then
\begin{equation}
  \label{eq:2}
\inf_{f_n\in\Sigma_n}\|f-f_n\|_{H^m( \Omega)}
\le C(d,m)\;n^{-{1\over2}}\int_{\mathbb R^d}(1+|\omega|)^{m+1}|\hat f(\omega)|
\end{equation}
\end{theorem}

\iffalse
	\begin{lemma}
		Let $\sigma\in C^\infty(\mathbb{R})$ and assume $\sigma$ is not a polynomial. Then $\Sigma_n(\sigma)$ is dense in $C(\mathbb{R}^n)$.
	\end{lemma}
	\begin{proof}
		Since $\sigma\in C^\infty(\mathbb{R})$, and $[\sigma((\omega+h e_j)\cdot x+\theta)-\sigma(\omega\cdot x+\theta)]/h\in\Sigma_n(\sigma)$ for every $\omega,\theta$ and $h\ne0$, it follows that
		$\frac{\partial}{\partial \omega_j}\sigma(\omega\cdot x+\theta)\in\overline{\Sigma}_n(\sigma)$ for all $j=1:n$. By the same argument $\frac{\partial^k}{\partial \omega^k_j}\sigma(\omega\cdot x+\theta)\in\overline{\Sigma}_n(\sigma)$ for all $k\in\mathbb{N}$, $j=1:n$, $\omega\in\mathbb{R}^n$ and $\theta\in\mathbb{R}$.

		Now $\frac{\partial^k}{\partial \omega^k_j}\sigma(\omega\cdot x+\theta)=x_j^k\sigma^{(k)}(\omega\cdot x+\theta)$, and since $\sigma$ is not a polynomial there exists a $\theta_k\in\mathbb{R}$ such that $\sigma^{(k)}(\theta_k)\ne0$. Take $\omega=0$ and $\theta=\theta_k$, we then have $x_j^k\in\overline{\Sigma}_n(\sigma)$. Similarly, for all polynomials of the form $x_1^{k_1}\cdots x_n^{k_n}$, we can get them by taking the corresponding partial derivatives.

		This implies that $\overline{\Sigma}_n(\sigma)$ contains all polynomials. By Weierstrass's Theorem it follows that $\overline{\Sigma}_n(\sigma)$ contains $C(K)$ for each compact $K\subset\mathbb{R}^n$. That is $\Sigma_n(\sigma)$ is dense in $C(\mathbb{R}^n)$.
		\end{proof}

	\fi

The first proof for this lemma above can be found in \cite{leshno1993multilayer} and summarized in \cite{pinkus1999approximation}.
The next theorem plays an important role in the proof of above lemma, which is first proved
in \cite{leshno1993multilayer} with several steps. Here we can present a more direct and simple version.
%
%\begin{proposition}
%	If $\Sigma_1$ is dense in $C(\mathbb{R})$, then $\Sigma_n$ is dense in $C(\mathbb{R}^n)$.
%	\end{proposition}
%
%
%
%From now on, we will focus on $\mathbb{R}$.

\begin{theorem}
	\label{prop:conti}
Let $\sigma$ be a non-polynomial Riemann integrable function and $\sigma\in L_{loc}^\infty(\mathbb{R})$. Then $\Sigma_1(\sigma)$ in dense in $C(\mathbb{R})$.
\end{theorem}
\begin{proof}
	Consider the mollifier $\eta$
	\begin{equation*}\eta(x)=\left\{
	\begin{aligned}
	Cexp(\frac{1}{|x|^2-1}),\quad&|x|<1,\\
 	0,\qquad\qquad\qquad\quad  &|x|\ge1.
	\end{aligned}
	\right.
	\end{equation*}
	here C is selected so that $\int_{\mathbb{R}}\eta dx=1.$

	Set $\eta_\epsilon=\frac{1}{\epsilon}\eta(\frac{x}{\epsilon})$. Then consider $\sigma_{\eta_\epsilon}$
\begin{equation}
	\sigma_{\eta_\epsilon}(x):=\sigma\ast{\eta_\epsilon}(x)=\int_{\mathbb{R}}\sigma(x-y){\eta_\epsilon}(y)dy
\end{equation}

It can be seen that $\sigma_{\eta_\epsilon}\in
C^\infty(\mathbb{R})$. Following the proof in the previous
proposition, we want to show that $\overline{\Sigma}_1(\sigma)$
contains all polynomials.  The first step is to show that
$\overline{\Sigma}_1(\sigma_{\eta_\epsilon})\subset\overline{\Sigma}_1(\sigma)$,
which can be done easily by checking the Riemann sum of
$\sigma_{\eta_\epsilon}(x)=\int_{\mathbb{R}}\sigma(x-y){\eta_\epsilon}(y)dy$
is in $\overline{\Sigma}_1(\sigma)$.

Then it suffices to show that there exists $\theta_k$ and
$\sigma_{\eta_\epsilon}$ such that
$\sigma_{\eta_\epsilon}^{(k)}(\theta_k)\ne0$ for each k. If not, then
there must be $k_0$ such that
$\sigma_{\eta_\epsilon}^{(k_0)}(\theta)=0$ for all
$\theta\in\mathbb{R}$ and all $\epsilon>0$.  Thus
$\sigma_{\eta_\epsilon}$'s are all polynomials with degree at most
$k_0-1$.  In particular, It is known that $\eta_\epsilon\in
C_0^\infty(\mathbb{R})$ and $\sigma\ast\eta_\epsilon$ uniformly
converges to $\sigma$ on compact sets in $\mathbb{R}$ and
$\sigma\ast\eta_\epsilon$'s are all polynomials of degree at most
$k_0-1$. Polynomials of a fixed degree form a closed linear subspace,
therefore $\sigma$ is also a polynomial of degree at most $k_0-1$,
which leads to contradiction.
\end{proof}


\documentclass[12pt]{article}
\usepackage[margin=2cm]{geometry}
\usepackage{amsmath}
\usepackage{slashed}
\usepackage{tikz}

\begin{document}

\noindent
Bhabha scattering is the result of interactions between positrons and electrons.
The following diagram represents a collider experiment with
collinear electron and positron beams.
\begin{center}
\begin{tikzpicture}
\draw[dashed] (0,0) circle (0.5cm);
\draw[thick,->] (2,0) node[anchor=west] {$e^-$} -- (0.6,0);
\draw[thick,->] (-2,0) node[anchor=east] {$e^+$} -- (-0.6,0);
\draw[thick,->] (0.40,0.40) -- (1.3,1.3) node[anchor=south west] {$e^+$};
\draw[thick,->] (-0.4,-0.4) -- (-1.3,-1.3) node[anchor=north east] {$e^-$};
\draw (1,0.5) node {$\theta$};
\end{tikzpicture}
\end{center}

\noindent
Here is the same diagram with momentum and spinor labels.

\begin{center}
\begin{tikzpicture}
\draw[dashed] (0,0) circle (0.5cm);
\draw[thick,->] (2,0) node[anchor=west] {$p_2, u_2$} -- (0.6,0);
\draw[thick,->] (-2,0) node[anchor=east] {$p_1, v_1$} -- (-0.6,0);
\draw[thick,->] (0.40,0.40) -- (1.3,1.3) node[anchor=south west] {$p_3, v_3$};
\draw[thick,->] (-0.4,-0.4) -- (-1.3,-1.3) node[anchor=north east] {$p_4, u_4$};
\draw (1,0.5) node {$\theta$};
\end{tikzpicture}
\end{center}

\noindent
In a typical collider experiment the momentum vectors are
\begin{equation*}
\underset{\text{inbound positron}}
{p_1=\begin{pmatrix}E\\0\\0\\p\end{pmatrix}}
\quad
\underset{\text{inbound electron}}
{p_2=\begin{pmatrix}E\\0\\0\\-p\end{pmatrix}}
\quad
\underset{\text{outbound positron}}
{p_3=\begin{pmatrix}
E\\
p\sin\theta\cos\phi\\
p\sin\theta\sin\phi\\
p\cos\theta
\end{pmatrix}}
\quad
\underset{\text{outbound electron}}
{p_4=\begin{pmatrix}
E\\
-p\sin\theta\cos\phi\\
-p\sin\theta\sin\phi\\
-p\cos\theta
\end{pmatrix}}
\end{equation*}

\noindent
Symbol $p$ is incident momentum,
$E$ is total energy $E=\sqrt{p^2+m^2}$,
and $m$ is electron mass.
Polar angle $\theta$ is the observed scattering angle.
Azimuth angle $\phi$ cancels out in scattering calculations.

\bigskip
\noindent
The spinors are
\begin{gather*}
\underset{\text{inbound positron, spin up}}
{v_{11}=\begin{pmatrix}p\\0\\E+m\\0\end{pmatrix}}
\quad
\underset{\text{inbound positron, spin down}}
{v_{12}=\begin{pmatrix}0\\-p\\0\\E+m\end{pmatrix}}
\quad
\underset{\text{inbound electron, spin up}}
{u_{21}=\begin{pmatrix}E+m\\0\\-p\\0\end{pmatrix}}
\quad
\underset{\text{inbound electron, spin down}}
{u_{22}=\begin{pmatrix}0\\E+m\\0\\p\end{pmatrix}}
\\
\underset{\text{outbound positron, spin up}}
{v_{31}=\begin{pmatrix}p_3^z\\p_3^x+ip_3^y\\E+m\\0\end{pmatrix}}
\quad
\underset{\text{outbound positron, spin down}}
{v_{32}=\begin{pmatrix}p_3^x-ip_3^y\\-p_3^z\\0\\E+m\end{pmatrix}}
\quad
\underset{\text{outbound electron, spin up}}
{u_{41}=\begin{pmatrix}E+m\\0\\p_4^z\\p_4^x+ip_4^y\end{pmatrix}}
\quad
\underset{\text{outbound electron, spin down}}
{u_{42}=\begin{pmatrix}0\\E+m\\p_4^x-ip_4^y\\-p_4^z\end{pmatrix}}
\end{gather*}

\noindent
The spinors shown above are not individually normalized.
Instead, a combined spinor normalization constant $N=(E+m)^4$
will be used.

\bigskip
\noindent
The following formula computes a probability density $|\mathcal{M}_{abcd}|^2$
for Bhabha scattering where $abcd$ are spin states.
\begin{equation*}
|\mathcal{M}_{abcd}|^2=\frac{e^4}{N}
\left|
-\frac{1}{t}(\bar{v}_{1a}\gamma^\mu v_{3c})(\bar{u}_{4d}\gamma_\mu u_{2b})
+\frac{1}{s}(\bar{v}_{1a}\gamma^\nu u_{2b})(\bar{u}_{4d}\gamma_\nu v_{3c})
\right|^2
\end{equation*}

\noindent
Symbol $e$ is electron charge.
Symbols $s$ and $t$ are Mandelstam variables
\begin{align*}
s&=(p_1+p_2)^2
\\
t&=(p_1-p_3)^2
\end{align*}

\noindent
Let
\begin{equation*}
a_1=(\bar{v}_{1a}\gamma^\mu v_{3c})(\bar{u}_{4d}\gamma_\mu u_{2b})
\qquad
a_2=(\bar{v}_{1a}\gamma^\nu u_{2b})(\bar{u}_{4d}\gamma_\nu v_{3c})
\end{equation*}

\noindent
Then
\begin{align*}
|\mathcal{M}_{abcd}|^2
&=
\frac{e^4}{N}\left|{-\frac{a_1}{t}} + \frac{a_2}{s}\right|^2\\
&=
\frac{e^4}{N}\left(-\frac{a_1}{t} + \frac{a_2}{s}\right)\left(-\frac{a_1}{t} + \frac{a_2}{s}\right)^*\\
&=
\frac{e^4}{N}
\left(
\frac{a_1a_1^*}{t^2} - \frac{a_1a_2^*}{st} -
\frac{a_1^*a_2}{st} + \frac{a_2a_2^*}{s^2}
\right)
\end{align*}

\noindent
The expected probability density $\langle|\mathcal{M}|^2\rangle$ is computed
by summing $|\mathcal{M}_{abcd}|^2$ over all spin states and then dividing by the number of inbound states.
There are four inbound states.
\begin{align*}
\langle|\mathcal{M}|^2\rangle
&=
\frac{1}{4}\sum_{a=1}^2\sum_{b=1}^2\sum_{c=1}^2\sum_{d=1}^2
|\mathcal{M}_{abcd}|^2\\
&=
\frac{e^4}{4N}\sum_{a=1}^2\sum_{b=1}^2\sum_{c=1}^2\sum_{d=1}^2
\left(
\frac{a_1a_1^*}{t^2} - \frac{a_1a_2^*}{st} -
\frac{a_1^*a_2}{st} + \frac{a_2a_2^*}{s^2}
\right)
\end{align*}

\noindent
Use the Casimir trick to replace sums over spins with matrix products.
\begin{align*}
f_{11}&=\frac{1}{N}\sum_{abcd}a_1a_1^*=
\mathop{\rm Tr}\left(
(\slashed{p}_1-m)\gamma^\mu(\slashed{p}_3-m)\gamma^\nu
\right)
\mathop{\rm Tr}\left(
(\slashed{p}_4+m)\gamma_\mu(\slashed{p}_2+m)\gamma_\nu
\right)
\\
f_{12}&=\frac{1}{N}\sum_{abcd}a_1a_2^*=
\mathop{\rm Tr}\left(
(\slashed{p}_1-m)\gamma^\mu(\slashed{p}_2+m)\gamma^\nu
(\slashed{p}_4+m)\gamma_\mu(\slashed{p}_3-m)\gamma_\nu
\right)
\\
f_{22}&=\frac{1}{N}\sum_{abcd}a_2a_2^*=
\mathop{\rm Tr}\left(
(\slashed{p}_1-m)\gamma^\mu(\slashed{p}_2+m)\gamma^\nu
\right)
\mathop{\rm Tr}\left(
(\slashed{p}_4+m)\gamma_\mu(\slashed{p}_3-m)\gamma_\nu
\right)
\end{align*}

\noindent
Hence
\begin{equation*}
\langle|\mathcal{M}|^2\rangle
=\frac{e^4}{4}
\left(
\frac{f_{11}}{t^2} - \frac{f_{12}}{st} -
\frac{f_{12}^*}{st} + \frac{f_{22}}{s^2}
\right)
\end{equation*}

\noindent
Run ``bhabha-scattering-1.txt'' to verify the Casimir trick.

\bigskip
\noindent
The following formulas are equivalent to the Casimir trick.
(Recall that $a\cdot b=a^\mu g_{\mu\nu}b^\nu$)
\begin{align*}
f_{11}&=
32(p_1\cdot p_2)^2
+32(p_1\cdot p_4)^2
-64 m^2(p_1\cdot p_3)
+64 m^4
\\
f_{12}&=
-32 (p_1\cdot p_4)^2
-32 m^2 (p_1\cdot p_2)
+32 m^2 (p_1\cdot p_3)
-32 m^2 (p_1\cdot p_4)
-32 m^4
\\
f_{22}&=
32(p_1\cdot p_3)^2
+32(p_1\cdot p_4)^2
+64 m^2(p_1\cdot p_2)
+64 m^4
\end{align*}

\noindent
Using Mandelstam variables
\begin{align*}
s&=(p_1+p_2)^2
\\
t&=(p_1-p_3)^2
\\
u&=(p_1-p_4)^2
\end{align*}
the formulas are
\begin{align*}
f_{11} &= 8 s^2 + 8 u^2 - 64 s m^2 - 64 u m^2 + 192 m^4
\\
f_{12} &= -8 u^2 + 64 u m^2 - 96 m^4
\\
f_{22} &= 8 t^2 + 8 u^2 - 64 t m^2 - 64 u m^2 + 192 m^4
\end{align*}

\subsection*{High energy approximation}
When $E\gg m$ a useful approximation is to set $m=0$ and obtain
\begin{align*}
f_{11}&= 8 s^2 + 8 u^2\\
f_{12}&= -8 u^2\\
f_{22}&= 8 t^2 + 8 u^2
\end{align*}

\noindent
Hence
\begin{align*}
\langle|\mathcal{M}|^2\rangle
&=\frac{e^4}{4}
\left(
\frac{f_{11}}{t^2} - \frac{f_{12}}{st} -
\frac{f_{12}^*}{st} + \frac{f_{22}}{s^2}
\right)
\\
&=\frac{e^4}{4}
\left(
\frac{8s^2+8u^2}{t^2} - \frac{-8u^2}{st} - \frac{-8u^2}{st} + \frac{8t^2+8u^2}{s^2}
\right)
\\
&=2e^4
\left(
\frac{s^2+u^2}{t^2} + \frac{2u^2}{st} + \frac{t^2+u^2}{s^2}
\right)
\end{align*}

\noindent
Combine terms so $\langle|\mathcal{M}|^2\rangle$ has a common denominator.
\begin{equation*}
\langle|\mathcal{M}|^2\rangle
=2e^4
\left(\frac{s^2\left(s^2+u^2\right)+2stu^2+t^2\left(t^2+u^2\right)}{s^2t^2}\right)
\end{equation*}

\noindent
For $m=0$ the Mandelstam variables are
\begin{align*}
s&=4E^2
\\
t&=2E^2(\cos\theta-1)
\\
u&=-2E^2(\cos\theta+1)
\end{align*}

\noindent
Hence
\begin{align*}
\langle|\mathcal{M}|^2\rangle
&=2e^4
\left(
\frac{32E^8\cos^4\theta+192E^8\cos^2\theta+288E^8}{64E^8(\cos\theta-1)^2}
\right)
\\
&=e^4
\left(
\frac{\cos^4\theta+6\cos^2\theta+9}{(\cos\theta-1)^2}
\right)
\\
&=e^4
\left(
\frac{\cos^2\theta+3}{\cos\theta-1}
\right)^2
\end{align*}

\noindent
Run ``bhabha-scattering-2.txt'' to verify.

\subsection*{Cross section}
The differential cross section is
\begin{equation*}
\frac{d\sigma}{d\Omega}
=\frac{\langle|\mathcal{M}|^2\rangle}{64\pi^2s}
=\frac{e^4}{64\pi^2s}
\left(
\frac{\cos^2\theta+3}{\cos\theta-1}
\right)^2,\quad s\gg m
\end{equation*}

\noindent
Substituting $e^4=16\pi^2\alpha^2$ yields
\begin{equation*}
\frac{d\sigma}{d\Omega}
=\frac{\alpha^2}{4s}
\left(
\frac{\cos^2\theta+3}{\cos\theta-1}
\right)^2
\end{equation*}

\noindent
We can integrate $d\sigma$ to obtain a cumulative distribution function.
Recall that
\begin{equation*}
d\Omega=\sin\theta\,d\theta\,d\phi
\end{equation*}

\noindent
Hence
\begin{equation*}
d\sigma=\frac{\alpha^2}{4s}
\left(
\frac{\cos^2\theta+3}{\cos\theta-1}
\right)^2
\sin\theta\,d\theta\,d\phi
\end{equation*}

\bigskip
\noindent
Let $I(\theta)$ be the following integral of $d\sigma$.
\begin{align*}
I(\theta)
&=\frac{4s}{2\pi\alpha^2}
\int_0^{2\pi}\int d\sigma
\\
&=\int
\left(
\frac{\cos^2\theta+3}{\cos\theta-1}
\right)^2
\sin\theta\,d\theta,
\quad a\le\theta\le\pi
\end{align*}

\noindent
Angular support is limited to an arbitrary $a>0$ because $I(0)$ is undefined.
Assume that $I(\theta)-I(a)$ is computable given $\theta$ by either symbolic or numerical integration.

\bigskip
\noindent
Let $C$ be the normalization constant
\begin{equation*}
C=I(\pi)-I(a)
\end{equation*}

\noindent
Then the cumulative distribution function $F(\theta)$ is
\begin{equation*}
F(\theta)=\frac{I(\theta)-I(a)}{C},
\quad a\le\theta\le\pi
\end{equation*}

\noindent
The probability of observing scattering events in the interval $\theta_1$ to $\theta_2$
can now be computed.
\begin{equation*}
P(\theta_1\le\theta\le\theta_2)=F(\theta_2)-F(\theta_1)
\end{equation*}

\noindent
Probability density function $f(\theta)$ is the derivative of $F(\theta)$.
\begin{equation*}
f(\theta)=\frac{dF(\theta)}{d\theta}
=\frac{1}{C}
\left(
\frac{\cos^2\theta+3}{\cos\theta-1}
\right)^2
\sin\theta
\end{equation*}

\noindent
Run ``bhabha-scattering-3.txt'' to draw $f(\theta)$ for $a=\pi/4=45^\circ$.

\begin{center}
\includegraphics[scale=0.5]{bhabha-scattering.png}
\end{center}

\noindent
Probability distribution for $45^\circ$ bins ($a=45^\circ$).
\begin{center}
\begin{tabular}{|c|c|c|}
\hline
$\theta_1$ & $\theta_2$ & $P(\theta_1\le\theta\le\theta_2)$\\
\hline
$0^\circ$ & $45^\circ$ & -- \\
$45^\circ$ & $90^\circ$ & 0.83 \\
$90^\circ$ & $135^\circ$ & 0.13 \\
$135^\circ$ & $180^\circ$ & 0.04 \\
\hline
\end{tabular}
\end{center}

\subsection*{Data from SLAC SPEAR experiment}
\noindent
The following Bhabha scattering data is adapted from SLAC-PUB-1501.

\begin{center}
\begin{tabular}{|lr|c|c|}
\hline
& Bin & $x_k$, $x_{k+1}$ & $y$\\
\hline
(Smallest $\theta$) & $k=1$ & $0.6, 0.5$ & 4432\\
& 2 & $0.5, 0.4$ & 2841\\
& 3 & $0.4, 0.3$ & 2045\\
& 4 & $0.3, 0.2$ & 1420\\
& 5 & $0.2, 0.1$ & 1136\\
& 6 & $0.1, 0.0$ & 852\\
& 7 & $0.0, -0.1$ & 656\\
& 8 & $-0.1, -0.2$ & 625\\
& 9 & $-0.2, -0.3$ & 511\\
& 10 & $-0.3, -0.4$ & 455\\
& 11 & $-0.4, -0.5$ & 402\\
(Largest $\theta$) & 12 & $-0.5, -0.6$ & 398\\
\hline
\end{tabular}
\end{center}

\noindent
Data column $x$ is $\cos\theta$ of scattering angle $\theta$,
data column $y$ is number of scattering events.

\bigskip
\noindent
To compute predicted values $\hat{y}$, start by integrating
the probability density function over each bin.
\begin{equation*}
I_k=\int_{\arccos(x_k)}^{\arccos(x_{k+1})}
\left(
\frac{\cos^2\theta+3}{\cos\theta-1}
\right)^2\sin\theta\,d\theta
\end{equation*}

\noindent
Normalize $I_k$ to obtain scattering probability $P_k$.
\begin{equation*}
P_k=\frac{I_k}{N},\quad N=\sum_{j=1}^{12}I_j
\end{equation*}

\noindent
Multiply $P_k$ by total counts to obtain $\hat{y}_k$.
\begin{equation*}
\hat{y}_k=P_k T,\quad T=\sum_{j=1}^{12}y_j=15773
\end{equation*}

\begin{center}
\begin{tabular}{|r|c|c|c|}
\hline
Bin & $x_k$, $x_{k+1}$ & $y$ & $\hat{y}$ \\
\hline
1 & $0.6, 0.5$ & 4432 & 4598\\
2 & $0.5, 0.4$ & 2841 & 2880\\
3 & $0.4, 0.3$ & 2045 & 1955\\
4 & $0.3, 0.2$ & 1420 & 1410\\
5 & $0.2, 0.1$ & 1136 & 1068\\
6 & $0.1, 0.0$ & 852 & 843\\
7 & $0.0, -0.1$ & 656 & 689\\
8 & $-0.1, -0.2$ & 625 & 582\\
9 & $-0.2, -0.3$ & 511 & 505\\
10 & $-0.3, -0.4$ & 455 & 450\\
11 & $-0.4, -0.5$ & 402 & 411\\
12 & $-0.5, -0.6$ & 398 & 382\\
\hline
\end{tabular}
\end{center}

\noindent
The coefficient of determination $R^2$ measures how well predicted values fit the real data.
\begin{equation*}
R^2=1-\frac{\sum(y-\hat{y})^2}{\sum(y-\bar{y})^2}=0.997
\end{equation*}

\noindent
The result indicates that the model $d\sigma$ explains
99.7\% of the variance in the data.

\bigskip
\noindent
Run ``bhabha-scattering-4.txt'' to verify.

\subsection*{Data from DESY PETRA experiment}
See www.hepdata.net/record/ins191231, Table 3, 14.0 GeV.

\begin{center}
\begin{tabular}{|c|c|}
\hline
$x$ & $y$\\
\hline
$-0.73\phantom{00}$ & 0.10115\\
$-0.6495$ & 0.12235\\
$-0.5495$ & 0.11258\\
$-0.4494$ & 0.09968\\
$-0.3493$ & 0.14749\\
$-0.2491$ & 0.14017\\
$-0.149\phantom{0}$ & 0.1819\phantom{0}\\
$-0.0488$ & 0.22964\\
$\phantom{+}0.0514$ & 0.25312\\
$\phantom{+}0.1516$ & 0.30998\\
$\phantom{+}0.252\phantom{0}$ & 0.40898\\
$\phantom{+}0.3524$ & 0.62695\\
$\phantom{+}0.4529$ & 0.91803\\
$\phantom{+}0.5537$ & 1.51743\\
$\phantom{+}0.6548$ & 2.56714\\
$\phantom{+}0.7323$ & 4.30279\\
\hline
\end{tabular}
\end{center}

\noindent
Data $x$ and $y$ have the following relationship
with the cross section model.
\begin{equation*}
x=\cos\theta
\qquad
y=\frac{d\sigma}{d\Omega}
\end{equation*}

\noindent
The cross section formula for Bhabha scattering is
\begin{equation*}
\frac{d\sigma}{d\Omega}
=\frac{\alpha^2}{4s}
\left(\frac{\cos^2\theta+3}{\cos\theta-1}\right)^2
\end{equation*}

\noindent
To compute predicted values $\hat{y}$,
substitute $x$ for $\cos\theta$,
multiply by $(\hbar c)^2$ to convert to SI,
and multiply by $10^{37}$ to convert square meters to nanobarns.
\begin{equation*}
\hat{y}
=\frac{\alpha^2}{4s}
\left(\frac{x^2+3}{x-1}\right)^2
\times(\hbar c)^2
\times10^{37}
\end{equation*}

\noindent
The following table shows $\hat{y}$ for $s=(14.0\,\text{GeV})^2$.

\begin{center}
\begin{tabular}{|c|c|c|}
\hline
$x$ & $y$ & $\hat{y}$\\
\hline
$-0.73\phantom{00}$ & 0.10115 & 0.110296\\
$-0.6495$ & 0.12235 & 0.113816\\
$-0.5495$ & 0.11258 & 0.120101\\
$-0.4494$ & 0.09968 & 0.129075\\
$-0.3493$ & 0.14749 & 0.141592\\
$-0.2491$ & 0.14017 & 0.158934\\
$-0.149\phantom{0}$ & 0.1819\phantom{0} & 0.182976\\
$-0.0488$ & 0.22964 & 0.216737\\
$\phantom{+}0.0514$ & 0.25312 & 0.264989\\
$\phantom{+}0.1516$ & 0.30998 & 0.335782\\
$\phantom{+}0.252\phantom{0}$ & 0.40898 & 0.44363\phantom{0}\\
$\phantom{+}0.3524$ & 0.62695 & 0.615528\\
$\phantom{+}0.4529$ & 0.91803 & 0.9077\phantom{00}\\
$\phantom{+}0.5537$ & 1.51743 & 1.45175\phantom{0}\\
$\phantom{+}0.6548$ & 2.56714 & 2.60928\phantom{0}\\
$\phantom{+}0.7323$ & 4.30279 & 4.61509\phantom{0}\\
\hline
\end{tabular}
\end{center}

\noindent
The coefficient of determination $R^2$ measures how well predicted values fit the real data.
\begin{equation*}
R^2=1-\frac{\sum(y-\hat{y})^2}{\sum(y-\bar{y})^2}=0.995
\end{equation*}

\noindent
The result indicates that the model $d\sigma$ explains 99.5\% of the variance in the data.

\bigskip
\noindent
Run ``bhabha-scattering-5.txt'' to verify.

\subsection*{Notes}
Here are a few notes about how the Eigenmath scripts work.
In component notation the trace operators of the Casimir trick become sums over the repeated index $\alpha$.
\begin{align*}
f_{11}&=
\left(
(\slashed{p}_1-m)^\alpha{}_\beta
\gamma^{\mu\beta}{}_\rho
(\slashed{p}_3-m)^\rho{}_\sigma
\gamma^{\nu\sigma}{}_\alpha
\right)
\left(
(\slashed{p}_4+m)^\alpha{}_\beta
\gamma_\mu{}^\beta{}_\rho
(\slashed{p}_2+m)^\rho{}_\sigma
\gamma_\nu{}^\sigma{}_\alpha
\right)
\\
f_{12}&=
(\slashed{p}_1-m)^\alpha{}_\beta
\gamma^{\mu\beta}{}_\rho
(\slashed{p}_2+m)^\rho{}_\sigma
\gamma^{\nu\sigma}{}_\tau
(\slashed{p}_4+m)^\tau{}_\delta
\gamma_\mu{}^\delta{}_\eta
(\slashed{p}_3-m)^\eta{}_\xi
\gamma_\nu{}^\xi{}_\alpha
\\
f_{22}&=
\left(
(\slashed{p}_1-m)^\alpha{}_\beta
\gamma^{\mu\beta}{}_\rho
(\slashed{p}_2+m)^\rho{}_\sigma
\gamma^{\nu\sigma}{}_\alpha
\right)
\left(
(\slashed{p}_4+m)^\alpha{}_\beta
\gamma_\mu{}^\beta{}_\rho
(\slashed{p}_3-m)^\rho{}_\sigma
\gamma_\nu{}^\sigma{}_\alpha
\right)
\end{align*}

\noindent
To convert the above formulas to Eigenmath code,
the $\gamma$ tensors need to be transposed
so that repeated indices are adjacent to each other.
Also, multiply $\gamma^\mu$ by the metric tensor to lower the index.
\begin{align*}
\gamma^{\beta\mu}{}_\rho\quad&\rightarrow\quad
\text{\tt gammaT = transpose(gamma)}\\
\gamma^\beta{}_{\mu\rho}\quad&\rightarrow\quad
\text{\tt gammaL = transpose(dot(gmunu,gamma))}
\end{align*}

\noindent
Define the following $4\times4$ matrices.
\begin{align*}
(\slashed{p}_1-m)\quad&\rightarrow\quad\text{\tt X1 = pslash1 - m I}\\
(\slashed{p}_2+m)\quad&\rightarrow\quad\text{\tt X2 = pslash2 + m I}\\
(\slashed{p}_3-m)\quad&\rightarrow\quad\text{\tt X3 = pslash3 - m I}\\
(\slashed{p}_4+m)\quad&\rightarrow\quad\text{\tt X4 = pslash4 + m I}
\end{align*}

\noindent
Then for $f_{11}$ we have the following Eigenmath code.
The contract function sums over $\alpha$.
\begin{align*}
(\slashed{p}_1-m)^\alpha{}_\beta
\gamma^{\mu\beta}{}_\rho
(\slashed{p}_3-m)^\rho{}_\sigma
\gamma^{\nu\sigma}{}_\alpha
\quad&\rightarrow\quad
\text{\tt T1 = contract(dot(X1,gammaT,X3,gammaT),1,4)}\\
(\slashed{p}_4+m)^\alpha{}_\beta
\gamma_\mu{}^\beta{}_\rho
(\slashed{p}_2+m)^\rho{}_\sigma
\gamma_\nu{}^\sigma{}_\alpha
\quad&\rightarrow\quad
\text{\tt T2 = contract(dot(X4,gammaL,X2,gammaL),1,4)}
\end{align*}

\noindent
Next, multiply then sum over repeated indices.
The dot function sums over $\nu$ then the contract function
sums over $\mu$. The transpose makes the $\nu$ indices adjacent
as required by the dot function.
$$
f_{11}=
\mathop{\rm Tr}(\cdots\gamma^\mu\cdots\gamma^\nu)
\mathop{\rm Tr}(\cdots\gamma_\mu\cdots\gamma_\nu)
\quad\rightarrow\quad
\text{\tt f11 = contract(dot(T1,transpose(T2)))}
$$

\noindent
Follow suit for $f_{22}$.
\begin{align*}
(\slashed{p}_1-m)^\alpha{}_\beta
\gamma^{\mu\beta}{}_\rho
(\slashed{p}_2+m)^\rho{}_\sigma
\gamma^{\nu\sigma}{}_\alpha
\quad&\rightarrow\quad
\text{\tt T1 = contract(dot(X1,gammaT,X2,gammaT),1,4)}
\\
(\slashed{p}_4+m)^\alpha{}_\beta
\gamma_\mu{}^\beta{}_\rho
(\slashed{p}_3-m)^\rho{}_\sigma
\gamma_\nu{}^\sigma{}_\alpha
\quad&\rightarrow\quad
\text{\tt T2 = contract(dot(X4,gammaL,X3,gammaL),1,4)}
\end{align*}

\noindent
Hence
$$
f_{22}=
\mathop{\rm Tr}(\cdots\gamma^\mu\cdots\gamma^\nu)
\mathop{\rm Tr}(\cdots\gamma_\mu\cdots\gamma_\nu)
\quad\rightarrow\quad
\text{\tt f22 = contract(dot(T1,transpose(T2)))}
$$

\noindent
The calculation of $f_{12}$ begins with
\begin{multline*}
(\slashed{p}_1-m)^\alpha{}_\beta
\gamma^{\mu\beta}{}_\rho
(\slashed{p}_2+m)^\rho{}_\sigma
\gamma^{\nu\sigma}{}_\tau
(\slashed{p}_4+m)^\tau{}_\delta
\gamma_\mu{}^\delta{}_\eta
(\slashed{p}_3-m)^\eta{}_\xi
\gamma_\nu{}^\xi{}_\alpha
\\
\rightarrow\quad
\text{\tt T = contract(dot(X1,gammaT,X2,gammaT,X4,gammaL,X3,gammaL),1,6)}
\end{multline*}

\noindent
Then sum over repeated indices $\mu$ and $\nu$.
$$
f_{12}=\mathop{\rm Tr}(\cdots\gamma^\mu\cdots\gamma^\nu\cdots\gamma_\mu\cdots\gamma_\nu)
\quad\rightarrow\quad
\text{\tt f12 = contract(contract(T,1,3))}
$$

%F(theta) = -37/8 cos(theta) - 1/4 cos(2 theta) - 1/24 cos(3 theta) - 4 / sin(theta/2)^2 - 16 log(sin(theta/2))

%\subsection*{Epilogue}
%This is the closed form of $I(\theta)$.
%\begin{equation*}
%I(\theta)=-\tfrac{37}{8}\cos\theta - \tfrac{1}{4}\cos(2\theta)
%- \tfrac{1}{24}\cos(3\theta) - \frac{4}{\sin(\theta/2)^2} - 16\log(\sin(\theta/2))
%\end{equation*}

\end{document}


\subsection{Generative and discriminative models}

\subsubsection{Recap}

For parametric models without dependent variables we have a form:

\(P(y| \theta )\)

And we have various ways of estimating \(\theta \).

We can write this as a likelihood function:

\(L(\theta ;y )=P(y|\theta)\)

\subsubsection{Discriminative models}

In discriminative models we learn:

\(P(y|X, \theta )\)

Which we can write as a likelihood function:

\(L(\theta ;y, X )=P(y| X, \theta)\)

\subsubsection{Generative models}

In generative models we learn:

\(P(y, X| \theta )\)

Which we can write as a likelihood function:

\(L(\theta ;y, X )=P(y, X|\theta)\)

We can use the generative model to calculate dependent probabilities.

\(P(y| X, \theta )=\dfrac{P(y, X| \theta )P(\theta )}{P(X, \theta )}\)

\(P(y| X, \theta )=\dfrac{P(y, X| \theta )}{P(X| \theta )}\)


\section{Update equations}
\label{sec:GRU_eqs}

The RNN type that we choose for SWR detection is the so called \emph{gated recurrent units} RNN, or \emph{GRU} \cite{Cho2014}. It is closely related to the popular ``long short-term memory'' (LSTM) RNN \cite{Hochreiter1997,Greff2017}. The update equations for the GRU are simpler than those of the LSTM however, while achieving comparable performance \cite{Chung2014}.

The state-update function $f$ and the readout function $g$ are defined as follows for a GRU RNN:
%
\begin{align}
f:\ \h_t     &= \update \had \h\prev
                + (1-\update) \had \hcand, \label{eq:hnew}\\
g:\ n_t  &= \sigma( \vb{w}_{hn}\trans \h_t + b_n )      \label{eq:out}
\end{align}
with
\begin{align}
\hcand  &= \tanh( \reset \had \W_{hh} \h\prev
                   + \W_{zh} \z_t + \bias_h ),  \label{eq:hcand}\\
\update &= \sigma( \W_{hu} \h\prev
                   + \W_{zu} \z_t + \bias_u),   \label{eq:update}\\
\reset  &= \sigma( \W_{hr} \h\prev
                   + \W_{zr} \z_t + \bias_r ).  \label{eq:reset}
\end{align}
$\tanh$ is the hyperbolic tangent, and $\sigma$ is the logistic sigmoid, defined as $\sigma(x) = 1 / (1 + \exp(-x))$. They are applied point-wise, i.e. separately to each element of their argument vector. Similarly, ``$\had$'' denotes point-wise multiplication.  $\sigma$ compresses the real number line to $(0, 1)$, while $\tanh(x) = 2\ \sigma(x) - 1$ maps it to $(-1, 1)$. Therefore, for this RNN, $\h_t \in (-1, 1)^M$ and the output $n_t \in (0, 1)$.

The coefficients in the matrices $\W_\_$ and in the vectors $\bias_\_$ are called the \emph{weights} and \emph{biases} of the RNN, respectively.\footnotemark{} They are parameters of the algorithm, determined a priori (i.e. before online SWR detection), through an optimisation procedure described in \cref{sec:RNN-optim}.

\footnotetext{Often collectively referred to simply as the weights. The vector $\vb{w}_{hn}$ and the scalar $b_n$ are also part of the weights and biases. They may be regarded as a one-row matrix and a one-dimensional vector, respectively.}

From \cref{eq:hnew}, each element of the hidden state $\h_t$ is thus a weighted average of its existing value in $\h\prev$, and a candidate new value in $\hcand$. The weighting, in $\update$, depends on both the current input $\z_t$, and the full previous hidden state $\h\prev$ (\cref{eq:update}).

Similarly, each element of the candidate new hidden state $\hcand$ is a function of both the current input $\z_t$, and all elements of the previous hidden state $\h\prev$ (\cref{eq:hcand}). A so called ``reset'' multiplier, in $\reset$ (\cref{eq:reset}), determines how much the existing memory values in $\h\prev$ influence the candidate new memory value, compared to the influence of the current input $\z_t$. (When an element of $\reset$ is $0$, the corresponding hidden unit in $\hcand$ is only a function of the current input, and is thus decoupled from its past. The hidden unit in $\h_t$ can therefore be `reset').

Finally, the output signal $n_t$ is a linear projection of the hidden state, translated and squashed to be in $(0, 1)$ (\cref{eq:out}).

The above equations \labelcref{eq:hnew} to \labelcref{eq:reset} for the GRU RNN define a so called one-layer network.\footnotemark{} An RNN with $L$ layers has $L$ corresponding hidden unit vectors $\h^{(1)}, \h^{(2)}, \tdots, \h^{(L)}$, that may each have a different dimension. The update functions for these hidden unit vectors can be formulated as follows. If $f^{(1)}(\h^{(1)}\prev, \z_t)$ is the function defined by \cref{eq:update,eq:hcand,eq:reset,eq:hnew}, we can define similar functions $f^{(2)}(\h^{(2)}\prev, \h^{(1)}_t), \  f^{(3)}(\h^{(3)}\prev, \h^{(2)}_t)$, and so forth. These functions all have the same form, but do not share weights and biases (i.e. each layer has separately learnable weights and biases). The output is then: $n_t = \sigma( \vb{w}_{hn}\trans \h^{(L)}_t + b_n )$; i.e. a readout of the last layer.

\footnotetext{That is, one layer per timestep. A recurrent neural network can also be regarded as a very deep feedforward neural network in which all the layers share the same weights.}


\documentclass{article}
\usepackage[utf8]{inputenc}

\title{Infinite Radical Limit}
\author{Shreenabh Agrawal }
\date{\today}
\usepackage{amsmath}
\usepackage{geometry}
\geometry{a4paper, portrait, margin=1in}
\usepackage{natbib}
\usepackage{graphicx}

\begin{document}

\maketitle

\section{Question}
Find $x$ for which
$$\lim_{x\to\infty}\sqrt{1+\sqrt{x+\sqrt{x^{2}+...\ +\ \sqrt{x^{n}}}}}=2 $$
\section{Solution}
The key observation here is that
$$t+1\ =\ \sqrt{t^{2}+2t+1}$$
$$=\sqrt{t^{2}+\sqrt{\left(2t+1\right)^{2}}}$$
$$=\ \sqrt{t^{2}+\sqrt{4t^{2}+4t+1}}$$
$$=\ \sqrt{t^{2}+\sqrt{4t^{2}+\sqrt{\left(4t+1\right)^{2}}}}$$
$$=\ \sqrt{t^{2}+\sqrt{4t^{2}+\sqrt{16t^{2}+\sqrt{64t^{2}+...}}}}$$
Now putting $t=1$, we get,
$$2=\ \sqrt{1+\sqrt{4+\sqrt{16+\sqrt{64+...}}}}$$
Hence, the answer is
$$\boxed{x=4}$$

\end{document}

Calculus/mathelonn's questions/Infinite Radical Limit.tex


\documentclass{report}

\usepackage[latin1]{inputenc}
\usepackage[danish]{babel}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amssymb,bm,mathtools}
\usepackage{enumitem}
\usepackage{listings}
\usepackage{forest}
\usepackage[margin=1in]{geometry}

\begin{document}
\section*{AD - assignment 2}
\subsection*{Task 1}
Let $N(C, i)$ denote the number the number of ways to spend exactly $C$ DKK with prices $p_1,\ldots, p_i$. This formula can be written recursively as
$$
N(C,i) =
	\begin{cases}
		1 &, C = 0\\
		0 &, C < 0\text{ or } c\neq 0, i = 0 \\
		N(C, i-1) + N(C - p_i, i -1 ) &, \text{else}
	\end{cases}
$$
\subsection*{Task 2}
The intuition about the the recursion, is that if an extra $i+1$ item is added, then all of the solution that existed when there were only $i$ items, must still be valid. As we already know how to spend exactly $C$ with $i$ items, then when one new item is added, we only have to figure out how to include this in the budget as all other possibilities where this item is not included is already accounted for. Hence the last part of the recursion $N(C-p_i, i-1)$.

Proof by induction:

\noindent
Base case: $i=1$
$$
N(C,1) = N(C, 0) + N(\overbrace{C - p_i}^{C'}, 0 ) =
	\begin{cases}
		1 &, C' = 0\\
		0 &, \text{else}
	\end{cases}
$$
So with one product the only way to spend exactly $C$ is if the price of the product equals $C$, which is the same as the first parameter in $N(C,i)$ equal 0. Hence the base case holds.

\noindent
Assume it hold for $i$ then we want to show that it also holds for $i+1$.
$$
N(C, i+1) = N(C, i) + N(C - p_{i+1}, i)
$$
which is exactly what we want to show.

Per induction the formula is correct. It is not difficult to see that the problem exhibits optimal substructure and overlapping problems, as the $i-1, i-2, \ldots, 0$ problems are all part of problem $i$ and the solution to those problem does not depend on $i$ there is optimal substructure. As problem $i-1$ contains the problems for $i-2,\ldots,0$ it share all but one problem of the same problems as the original problem $i$ hence there are overlapping problems.

\subsection*{Task 3}
Below is an dynamic programming memorization  algorithm with an $O(nC)$ runtime. There are made som assumtions that $C$ is an integer, $p$ is a list of positive integers prices. Furthermore; we also assume the list $p$ have some properties $p$.last which takes the last element in the list in constant time, and $p$.notLast which return a sublist with all but the last element, this also can be done in constant time.  $m$ is a two dimenaional (zero indexed) array, with $C+1$ rows and $p\text{.length}+1$ columns and it is initialized with the values $-1$ at all index.
\begin{lstlisting}[frame=single]
N(C, p, m)
1   if C < 0
2     return 0
3   i = p.length
4   if m[C, i] != -1
5     return m[C, n]
6   if C == 0
7     m[C, i] = 1
8     return 1
9   if i == 0
10    m[C, i] = 1
11    return 1
12  m[C, i] = N(C, p.notLast, m) + N(C - p.last, p.notLast, m)
13  return m[C, i]
\end{lstlisting}

\subsection*{Task 4}
\emph{Show correctness: Show that for every input it will halt with the correct output.}

The running time if the algorithm is $O(iC)$ as lines 1-11, 13 all take constant time. The recursion in line 12 all terms are at most calculated once and there are at most $ic$ subproblems to solve, hence the algorithm takes $O(iC)$ time.

The memory usage of the algorithm is also $O(iC)$ as this is the size of the $m$ array which is the larges data we need to save.
\end{document}


%!TEX root=report.tex
\subsection{Gaussian Mixture Model (GMM)}
The GMM is a more advanced clustering model than K-Means.
The main advantage is that GMM allows for hyper elliptical clusters. This uses gaussian kernels with its the shape described in a covariance matrix. A result similar to K-means could be obtained by forcing this covariance matrix to be the identity matrix.

Restrictions on the covariance matrix (i.e. shared covariance, diagonal covariance, spherical covariance etc.) can easily be applied in GMM and is quite common. In this analysis however, no covariance restrictions will be used.

In the GMM the assumption is that data comes from a single density function.
The density function is assumed to be a combination (mixture) of $K$ Gaussian PDFs where K is finite and denotes the number of mixture components (i.e. clusters).

Each mixture component has a centroid (the mean), a covariance matrix and a mixing weight.
The sum of mixing weights across components has to be one for the GMM to constitute an actual pdf.

To estimate the model parameters several different methods exists. The most common and is the expectation maximization (EM) algorithm which is quite complex and thus wont be described here. To the curious reader we recommend \cite[p.~214,272,463]{statistical-learning}).

In practice if K is large and the vector space $X$ is high dimensional, estimation of the model parameters will take too much computing power, and even if one get model parameters to converge, the degrees of freedom will be low.
In our case the input space would be 341-dimensional (341 observations in time per location) and thus a dimensionality reduction of some kind is needed.

\subsubsection{Dimensionality reduction}
In many cases when dealing with high dimensional data, most of the data lies on a lower dimensional manifold.
Different methods exists to try to identify such manifolds, but in this analysis the previously described technique PCA will be used.
This is done by selecting only the most important principal components from the PCA, thus forcing the data onto a lower dimensional manifold.
The GRACE data contains quite a bit of noise and one might hope that the noise will be primarily contained in its own principal components. Hopefully those PCs will only account for a small amount of the variance in the data.
Thus when selecting only the most significance PCs some of the noise will be "lost". It should be noted that the standard PCA method wasn't used, instead a more complex method called Kernel PCA is used.

Using kernel PCA combined with the more flexible GMM over K-Means, will hopefully lead to better clustering.


\documentclass{article}
\usepackage[margin=1in]{geometry}
\setlength{\parindent}{0in}
\usepackage{amsmath,mathtools,amsfonts,xfrac,hyperref}

\title{Paper Title\\[1ex] \normalsize AMATH 383 Term Paper}
\date{}
\author{Saransh Kacharia, Abishek Hariharan, Alex Chkodrov}
\begin{document}
\maketitle

% Instructions
\subsection*{Introduction} Hello. \\
\hrule

% Projections
\subsection*{Section 2}

% Exercise 1
\begin{enumerate}
\item Define the orthogonal projection of $\mathbf{u} \in \mathbb{R}^n$ onto a linear space $S$ as
\[ \text{proj}_{S} (\mathbf{u}) = \sum_{i=1}^k \frac{\mathbf{v}_i \cdot \mathbf{u}}{\| \mathbf{v}_i \|_2^2} \, \mathbf{v}_i, \]
where $\{ \mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_k \}$ forms an orthogonal (but not necessarily orthonormal) basis for $S$.
	\begin{enumerate}
	\item Show $\mathbf{u} - \text{proj}_S(\mathbf{u})$ is orthogonal to $\text{proj}_S(\mathbf{u})$. \\
	\\
	To show this we have to show that the dot product between $\mathbf{u} - \text{proj}_S(\mathbf{u})$ and $\text{proj}_S(\mathbf{u})$ is zero.
	\begin{align*}
		\mathbf{u} - \text{proj}_{S} (\mathbf{u}) = \sum_{i=1}^k \frac{\mathbf{v}_i \cdot \mathbf{u}}{\| \mathbf{v}_i \|_2^2} \, \mathbf{v}_i \\
		\\
		|\mathbf{u} - \sum_{i=1}^k \frac{\mathbf{v}_i \cdot \mathbf{u}}{\| \mathbf{v}_i \|_2^2} \, \mathbf{v}_i| \cdot |\sum_{i=1}^k \frac{\mathbf{v}_i \cdot \mathbf{u}}{\| \mathbf{v}_i \|_2^2} \, \mathbf{v}_i|  &= \\
		\mathbf{u} \cdot \sum_{i=1}^k \frac{\mathbf{v}_i \cdot \mathbf{u}}{\| \mathbf{v}_i \|_2^2} \, \mathbf{v}_i - \sum_{i=1}^k \bigg(\frac{\mathbf{v}_i \cdot \mathbf{u}}{\| \mathbf{v}_i \|_2^2} \, \mathbf{v}_i\bigg) \cdot \bigg(\frac{\mathbf{v}_i \cdot \mathbf{u}}{\| \mathbf{v}_i \|_2^2} \, \mathbf{v}_i\bigg) \\
	\end{align*}
	$\frac{\mathbf{v}_i \cdot \mathbf{u}}{\| \mathbf{v}_i \|_2^2} \, \mathbf{v}_i$ can be split into a unit vector and a scalar. \\
	$\frac{\mathbf{v}_i \cdot \mathbf{u}}{\| \mathbf{v}_i \|_2} = \beta$, a scalar and $\frac{\mathbf{v}_i}{\| \mathbf{v}_i \|_2}$ is a unit vector.\\
	\begin{align*}
		\mathbf{u} \cdot \sum_{i=1}^k \frac{\mathbf{v}_i \cdot \mathbf{u}}{\| \mathbf{v}_i \|_2^2} \, \mathbf{v}_i - \sum_{i=1}^k \bigg(\frac{\mathbf{v}_i \cdot \mathbf{u}}{\| \mathbf{v}_i \|_2^2} \, \mathbf{v}_i\bigg) \cdot \bigg(\frac{\mathbf{v}_i \cdot \mathbf{u}}{\| \mathbf{v}_i \|_2^2} \, \mathbf{v}_i\bigg) &= \\
		\mathbf{u} \cdot \sum_{i=1}^k \beta \frac{\mathbf{v}_i}{\| \mathbf{v}_i \|_2} - \sum_{i=1}^k \beta \frac{\mathbf{v}_i}{\| \mathbf{v}_i \|_2} \cdot \beta \frac{\mathbf{v}_i}{\| \mathbf{v}_i \|_2} &= \\
		\mathbf{u} \cdot \sum_{i=1}^k \beta \frac{\mathbf{v}_i}{\| \mathbf{v}_i \|_2} - \beta^2
	\end{align*}
	Because $\sum_{i=1}^k \beta \frac{\mathbf{v}_i}{\| \mathbf{v}_i \|_2}$ is a projection of $\mathbf{u}$.
	\begin{align*}
		\mathbf{u} \cdot \sum_{i=1}^k \beta \frac{\mathbf{v}_i}{\| \mathbf{v}_i \|_2} = \beta^2 \\
		\mathbf{u} \cdot \sum_{i=1}^k \beta \frac{\mathbf{v}_i}{\| \mathbf{v}_i \|_2} - \beta^2 &= \\
		\beta^2 - \beta^2 &= 0
	\end{align*}
	\item Show that $\| \mathbf{u} \|_2^2 = \| \text{proj}_S(\mathbf{u)} \|_2^2 + \| \mathbf{u} - \text{proj}_S(\mathbf{u}) \|_2^2$
	\end{enumerate}

% Exercise 2
\item In this exercise, we prove the Cauchy-Schwarz inequality, which states that
\[ | \mathbf{u} \cdot \mathbf{v} | \leq \| \mathbf{u} \|_2 \| \mathbf{v} \|_2 \]
for any vectors $\mathbf{u, v} \in \mathbb{R}^n$.
	\begin{enumerate}
	\item Prove that $\| \text{proj}_{\mathbf{v}}(\mathbf{u}) \|_2 \leq \| \mathbf{u} \|_2$.
	\item Use part (a) to prove the Cauchy-Schwarz inequality.
	\item Show that $| \mathbf{u} \cdot \mathbf{v} | = \| \mathbf{u} \|_2 \| \mathbf{v} \|_2$ if and only if $\mathbf{u} = \alpha \mathbf{v}$ for some $\alpha \in \mathbb{R}$. That is, equality of the Cauchy-Schwarz inequality holds if and only if $\mathbf{u}$ is a scalar multiple of $\mathbf{v}$.
	\end{enumerate}
\end{enumerate}

% QR Factorization
\subsection*{QR Factorization}

% Exercise 3
\begin{enumerate}\setcounter{enumi}{2}
\item Compute (by hand) the reduced QR factorization of the given matrix $\mathbf{A}$, and use it to solve (by hand) the linear system of equations $\mathbf{Ax = b}$, where
\[ \mathbf{A} = \begin{bmatrix} 1 & 6 & 14 \\ 1 & -2 & -8 \\ 1 & 6 & 2 \\ 1 & -2 & 4 \end{bmatrix} \quad \text{and} \quad \mathbf{b} = \begin{bmatrix} 7 \\ -1 \\ 7 \\ -1 \end{bmatrix}. \]

% Exercise 4
\item Compute (by hand) the full QR factorization for the following matrix:
\[ \mathbf{A} = \begin{bmatrix} 1 & 4 & 7 \\ 2 & 5 & 8 \\ 3 & 6 & 9 \end{bmatrix} \]
Note that we allow the diagonal of $R$ to have zeros if $\mathbf{A}$ is not full rank.

% Exercise 5
\item Consider $\mathcal{P}_2$, the space of polynomials of degree at most 2. For any functions $f,g \in \mathcal{P}_2$ define their inner product as
\[ \langle f, g \rangle = \int_{-1}^1 f(t) g(t) \, dt. \]
(You need not show that this is an inner product.) Starting from the basis $\{ 1, x, x^2 \}$, use the Gram-Schmidt process (Algorithm 2 on page 94) to build an orthonormal basis for $\mathcal{P}_2$.

The resulting polynomials are scalar multiples of what are known as the Legendre polynomials, $P_j$, which are conventionally normalized so that $P_j(1) = 1$. Computations with such polynomials form the basis of spectral methods, one of the most powerful techniques for the numerical solution of partial differential equations.
\end{enumerate}

% Matlab
\subsection*{Matlab}

% Exercise 6
\begin{enumerate}\setcounter{enumi}{5}
\item In this exercise, we will learn how to create functions in matlab to study the stability of different algorithms for QR factorization. Consider the matrix
	\begin{enumerate}
\[ \mathbf{A} = \begin{bmatrix} 1 & 1 & 1 & 1 & 1 \\ \epsilon & \epsilon & 0 & 0 & 0 \\ \epsilon & 0 & \epsilon & 0 & 0 \\ \epsilon & 0 & 0 & \epsilon & 0 \\ \epsilon & 0 & 0 & 0 & \epsilon \end{bmatrix} \quad \text{with} \quad \epsilon = 10^{-6}. \]
	\item Use the command {\tt cond} to find the condition number of $\mathbf{A}$ in the 1-norm. Would you say that this matrix is ill-conditioned? Save this result as {\tt A1.dat}.
	\item Watch the video tutorial on creating functions in matlab found at \\ {\tt \href{https://www.youtube.com/watch?v=qo3AtBoyBdM}{https://www.youtube.com/watch?v=qo3AtBoyBdM}}.
	\item Create the function {\tt classicalGS} (inside a new m-file named {\tt classicalGS.m}) to compute the QR factorization using the classical Gram-Schmidt algorithm (Algorithm 1, page 90). This function should take as its input a matrix $\mathbf{A}$ and return as its output the matrices $\mathbf{Q}$ and $\mathbf{R}$. Use this function to compute the QR factorization of our given matrix $\mathbf{A}$, then compute the 1-norm of the error $\| \mathbf{Q}^T\mathbf{Q} - \mathbf{I} \|_1$ and save it as {\tt A2.dat}. Below is some code to get you started:
	\item Create the function {\tt modifiedGS} to compute the QR factorization using the modified Gram-Schmidt algorithm (Algorithm 4, page 106). Use this function to compute the QR factorization of our given matrix $\mathbf{A}$, then compute the 1-norm of the error and save it as {\tt A3.dat}.
	\item Create the function {\tt twostepGS} to compute the QR factorization using two steps of the Gram-Schmidt algorithm (Algorithm 5, page 106). Use this function to compute the QR factorization of our given matrix $\mathbf{A}$, then compute the 1-norm of the error and save it as {\tt A4.dat}.
	\item Compute the QR factorization of our given matrix $\mathbf{A}$ using the matlab command {\tt qr}. Compute the 1-norm of the error and save it as {\tt A5.dat}.
	\item Which algorithms appeared to be the most stable for computing the QR factorization of $\mathbf{A}$? In the table below, fill in the error values and rank the algorithms from most stable (1) to least stable(4). Include this table in your written homework.

	\begin{center}
	\begin{tabular}{| l | l | l |}
	\hline
	Algorithm & Error & Rank \\
	\hline
	Classical Gram-Schmidt & & \\
	Modified Gram-Schmidt & & \\
	Two Steps Gram-Schmidt & & \\
	Matlab QR & & \\
	\hline
	\end{tabular}
	\end{center}

	\end{enumerate}

	\bigskip
	For this problem, you must submit four matlab files to Scorelator: the functions {\tt classicalGS.m}, {\tt modifiedGS.m}, and {\tt twostepGS.m}, and your main script which generates your data files. Before clicking submit, {\bf you must highlight your main script} by clicking on it. If you highlight one of the function files, Scorelator will give you a 0.
\end{enumerate}
\end{document}


\chapter{Hierarchical Clustering}
\label{ch:hierarchical_clustering}

\newthought{We are interested in finding clusters in our data}. That is, we would like to identify groups of data instances that are close together, similar to each other. Consider a simple, two-featured data set (see the side note) and plot it in the \widget{Scatter Plot}. How many clusters do we have? What defines a cluster? Which data instances should belong to the same cluster? How does the clustering algorithm actually work?

\begin{marginfigure}
    \includegraphics[scale=0.4]{workflow_scatterplot.png}
    \caption{We will introduce clustering with a simple data set on students and their grades in English and Algebra.
Load the data set from \url{http://file.biolab.si/text/grades.tab}.}
\end{marginfigure}

First, we need to define what we mean by "similar". We will assume that all our data instances are described (profiled) with continuous features. One simple measure of similarity is the Euclidean distance. So, we would like to group data instances with small Euclidean distances.

\begin{figure*}[h]
    \vspace{1cm}
    \centering
    \infinitewidthbox{
    \includegraphics[scale=0.4]{grades_table.png}
    \includegraphics[scale=0.4]{grades_scatterplot.png}
    }
    \caption{There are different ways to measure the similarity between clusters. The estimate we have described is called average linkage. We could also estimate the distance through the two closest points in each clusters (single linkage), or through the two points that are furthest away (complete linkage).}
\end{figure*}

Next, we need to define a clustering algorithm. Say that we start with each data instance being its own cluster, and then, at each step, we join the clusters that are closest together. We estimate the distance between the clusters with, say, the average distance between all their pairs of data points. This algorithm is called hierarchical clustering.

\clearpage

One possible way to observe the results of clustering on our small data set with grades is with the following workflow:

\begin{marginfigure}
    \centering
    \includegraphics[scale=0.6]{distances.png}
\end{marginfigure}

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.4]{workflow_clustering.png}
    \caption{$\;$} % empty caption for proper pagesetting
\end{figure}

Couldn’t be simpler. Load the data, measure the distances, use them in hierarchical clustering, and visualize the results in a scatter plot. The \widget{Hierarchical Clustering} widget allows us to cut the hierarchy at a certain distance score and output the corresponding clusters:

\begin{figure*}[h]
    \centering
    \newcommand{\clustering}{\includegraphics[scale=0.4]{hierarchical_clustering.png}}
    \newcommand{\plot}{\includegraphics[scale=0.4]{scatterplot_clustered.png}}
    \infinitewidthbox{
    \stackinset{r}{-0.5\linewidth}{t}{+0.3\linewidth}{\plot}{\clustering}\hspace{8cm}
    }
\end{figure*}