%\chapter{Limits of Random Variables}

\chapter{Limit Laws of Statistics}\label{S:LimitLawsStats}

\section{Convergence of Random Variables}\label{S:ConvOfRVs}

This important topic is concerned with the limiting behavior of sequences of RVs.
We want to understand what it means for a sequence of random variables $\{X_n\}_{n=1}^{\infty} := X_1,X_2,\ldots$ to converge to another random variable $X$, when all RVs are defined on the same probability space $(\Omega,\mathcal{F},\P)$.
\[
\{X_i \}_{i=1}^n := X_1,X_2,X_3, \ldots X_{n-1}, X_n \qquad \text{as  $n \rightarrow \infty$ .}
\]
From a statistical or decision-making viewpoint, as you will see in Inference Theory I course, $n \rightarrow \infty$ is associated with the amount of data or information $\rightarrow \infty$.
More abstractly, we are interested in what happens to the limiting RV $X := \lim_{n\to \infty} X_n$ when given the DFs $F_n(x)$ for each $X_n$.

We need different notions of convergence to characterize such a behavior: two simplest behaviors are that the sequence eventually takes a constant value $\theta$,
i.e. $X_n$ approaches $X \sim \pointmass(\theta)$ RV, or that values in the sequence continue to change but can be described by an unchanging probability distribution, i.e., $X_n$ approaches $X \sim F(x)$. See \url{https://en.wikipedia.org/wiki/Convergence_of_random_variables}.

Let us first refresh ourselves with notions of convergence, limits and continuity in the real line (\hyperref[S:AnalysisRefresher]{Sec.~\ref*{S:AnalysisRefresher}}) before proceeding further.

Can the sequences of $\{\pointmass(\theta_i=17)\}_{i=1}^{\infty}$ and $\{\pointmass(\theta_i=1/i)\}_{i=1}^{\infty}$ RVs be the same as the two sequences of real numbers $\{ x_i \}_{i=1}^{\infty} = 17, 17, 17, \ldots$ and $\{ x_i \}_{i=1}^{\infty} = \frac{1}{1},\frac{1}{2},\frac{1}{3}, \ldots$ we saw in Examples~\ref{EX:limOf17s} and \ref{EX:limin1overi}?

Yes why not -- just move to space of distributions over the reals! See Figure~\ref{F:SequenceOfPointMassRVS17And1Byi}.

\begin{figure}[htpb]
\caption{Sequence of $\{\pointmass(17)\}_{i=1}^{\infty}$ RVs (left panel) and $\{\pointmass(1/i)\}_{i=1}^{\infty}$ RVs (only the first seven are shown on right panel) and their limiting RVs in red.\label{F:SequenceOfPointMassRVS17And1Byi}}
\centering   \makebox{\includegraphics[width=6.5in]{figures/SequenceOfPointMassRVS17And1Byi}}
\end{figure}

\begin{classwork}[Convergence of $X_i \sim \normal(0,1/i)$]\label{CW:Normal01bynConvToPointMass0}
Suppose you are given an independent sequence of RVs $\{X_i \}_{i=1}^n$, where $X_i \sim \normal(0,1/i)$.  How would you talk about the convergence of $X_n \sim \normal(0,1/n)$ as $n$ approaches $\infty$ ?  Take a look at \hyperref[F:PlotNormal01bynConvToPointMass0]{Figure \ref*{F:PlotNormal01bynConvToPointMass0}} for insight.  The probability mass of $X_n$ increasingly concentrates about $0$ as $n$ approaches $\infty$ and the variance $1/n$ approaches $0$, as depicted in \hyperref[F:PlotNormal01bynConvToPointMass0]{Figure \ref*{F:PlotNormal01bynConvToPointMass0}}.  Based on this observation, can we expect $\lim_{n \rightarrow \infty} X_n = X$, where the limiting RV $X \sim \pointmass(0)$ ?

The answer is {\bf no}.  This is because $\P(X_n=X)=0$ for any $n$, since $X \sim \pointmass(0)$ is a discrete RV with exactly one outcome $0$ and $X_n \sim \normal(0,1/n)$ is a continuous RV for every $n$, however large.  In other words, a continuous RV, such as $X_n$, has $0$ probability of realizing any single real number in its support, such as $0$.
\begin{figure}[htpb]
\caption{Distribution functions of several $\normal(\mu,\sigma^2)$ RVs for $\sigma^2 = 1,\frac{1}{10},\frac{1}{100},\frac{1}{1000}$.\label{F:PlotNormal01bynConvToPointMass0}}
\centering   \makebox{\includegraphics[width=6.5in]{figures/PlotNormal01bynConvToPointMass0}}
\end{figure}
\end{classwork}

Thus, we need more sophisticated notions of convergence for sequences of RVs.  Two such notions are formalized next as they are minimal prerequisites for a clear understanding of two basic propositions in Statistics :
\begin{enumerate}
\item Law of Large Numbers,
\item Central Limit Theorem,
%\item Gilvenko-Cantelli Theorem.
\end{enumerate}

\begin{definition}[Convergence in Distribution (or Weakly, or in Law)]\label{D:ConvInDist}
Let $X_1,X_2,\ldots,$ be a sequence of RVs and let $X$ be another RV.  Let $F_n$ denote the DF of $X_n$ and $F$ denote the DF of $X$.  The we say that $X_n$ converges to $X$ in distribution, and write:
\[
X_n \rightsquigarrow X
\]
if for any real number $t$ at which $F$ is continuous,
\[
\lim_{n \rightarrow \infty} F_n(t) = F(t) \qquad \text{[in the sense of \hyperref[D:LimitofRealFunction]{Definition \ref*{D:LimitofRealFunction}}].}
\]
The above limit, by \eqref{E:DF} in our \hyperref[D:DF]{Definition \ref*{D:DF}} of a DF, can be equivalently expressed as follows:
\begin{eqnarray}
& & \lim_{n \rightarrow \infty} \P( \ \{\omega: X_n(\omega) \leq  t \} \ )=
\P( \ \{\omega: X(\omega) \leq  t \} \ ), \notag \\
&\text{i.e.~} & \P( \ \{\omega: X_n(\omega) \leq t \} \ ) \rightarrow \P( \ \{\omega: X(\omega) \leq  t \} \ ), \quad \text{as} \quad n \rightarrow \infty \notag \ .
\end{eqnarray}
\end{definition}

Let us revisit the problem of convergence in \hyperref[CW:Normal01bynConvToPointMass0]{Classwork \ref*{CW:Normal01bynConvToPointMass0}} armed with our new notions of convergence.
\begin{example}[Convergence in distribution]\label{EX:Normal01bynConvinDistToPointMass0}
Suppose you are given an independent sequence of RVs $\{X_i \}_{i=1}^n$, where $X_i \sim \normal(0,1/i)$ with DF $F_n$ and let $X \sim \pointmass(0)$ with DF $F$.  We can formalize our observation in \hyperref[CW:Normal01bynConvToPointMass0]{Classwork \ref*{CW:Normal01bynConvToPointMass0}} that $X_n$ is concentrating about $0$ as $n \to \infty$ by the statement:
\[
\text{$X_n$ is converging in distribution to $X$, ie,} \qquad X_n \rightsquigarrow X \ .
\]
{\normalsize
\begin{proof}
To check that the above statement is true we need to verify that the definition of convergence in distribution is satisfied for our sequence of RVs $X_1,X_2,\ldots$ and the limiting RV $X$.  Thus, we need to verify that for any continuity point $t$ of the $\pointmass(0)$ DF $F$, $\lim_{n \to \infty} F_n(t)=F(t)$.  First note that
\[
X_n \sim \normal(0,1/n) \implies Z := \sqrt{n} X_n \sim \normal(0,1) \ ,
\]
and thus
\[
F_n(t) = \P(X_n < t) = \P(\sqrt{n} X_n < \sqrt{n} t) = \P(Z < \sqrt{n} t) \ .
\]
The only discontinuous point of $F$ is $0$ where $F$ jump from $0$ to $1$.

When $t < 0$, $F(t)$, being the constant $0$ function over the interval $(-\infty,0)$, is continuous at $t$.  Since $\sqrt{n} t \to -\infty$, as $n \to \infty$,
\[
\lim_{n \to \infty} F_n(t)  = \lim_{n \to \infty} \P(Z < \sqrt{n} t) = 0 = F(t) \ .
\]
And, when $t >0$, $F(t)$, being the constant $1$ function over the interval $(0,\infty)$, is again continuous at $t$.  Since $\sqrt{n} t \to \infty$, as $n \to \infty$,
\[
\lim_{n \to \infty} F_n(t)  = \lim_{n \to \infty} \P(Z < \sqrt{n} t) = 1 = F(t) \ .
\]
Thus, we have proved that $X_n \rightsquigarrow X$ by verifying that for any $t$ at which the $\pointmass(0)$ DF $F$ is continuous, we also have the desired equality: $\lim_{n \to \infty} F_n(t)=F(t)$.
\end{proof}
However, note that
\[
F_n(0)=\frac{1}{2} \neq F(0)=1 \ ,
\]
and so convergence fails at $0$, i.e.~$\lim_{n \to \infty}F_n(t) \neq F(t)$ at $t=0$.  But, $t=0$ is not a continuity point of $F$ and the definition of convergence in distribution only requires the convergence to hold at continuity points of $F$.
}
\end{example}

\begin{figure}[htbp]
\begin{center}
\includegraphics[width=10cm]{figures/PDFOf1minusCos2PinxOn01OscillatesIndefinitely.png}
\includegraphics[width=5cm]{figures/DFOf1minusCos2PinxOn01To1ApproachesDFOfUniform01.png}
\caption{PDF $f_{X_n}(x) := \BB{1}_{(0,1)}(x)(1-\cos(2\pi n x))$ of the RV $X_n$ [the left sub-figure] and its DF $F_n(x) := \int_{-\infty}^x\BB{1}_{(0,1)}(v)(1-\cos(2\pi n v))dv$ [the right sub-figure], for $n =1$ [red '- -'], $n=10$ [blue '-.'], and $n=100$ [green '-'], respectively. One can see clear convergence of the DFs $F_n$ to $\BB{1}_{(0,1)}(x)x$, the DF of the $\uniform(0,1)$ RV, while the corresponding PDFs $f_n(x)$ keep oscillating wildly with $n$ across $[0,2]$ about $\BB{1}_{(0,1)}(x)$, the PDF of the $\uniform(0,1)$ RV $X$. Thus giving a counter-example to the claim that convergence in DFs does not imply convergence in PDFs.\label{F:ScheffesThmCounterExample}}
\end{center}
\end{figure}
Convergence in distribution does not in general imply that the sequence of corresponsing probability density functions will also converge.
Consider for example RV $X_n$ with density $\BB{1}_{(0,1)}(x)(1-\cos(2\pi n x))$.
These RVs converge in distribution to $X \sim \uniform(0,1)$, but their densities (PDFs) do not converge at all as evident in Figure~\ref{F:ScheffesThmCounterExample}.

\begin{prop}[Scheff\'e's Theorem]
According to {\bf Scheff\'e's Theorem} convergence of the probability density function (for a continuous RV) or probability mass function (for a discrete RV) implies convergence in distribution.
\begin{proof}
We will state this without a Proof here as Proof of the Theorem requires measure theory in generality \footnote{See \url{https://en.wikipedia.org/wiki/Scheff\%C3\%A9\%27s_lemma}.}. However, you should be able to see why convergence of PMFs $f_n(x)$ for discrete RVs $X_n$, to $f(x)$, the PMF of another discrete RV $X$, implies convergence in their corresponding DFs, i.e., $F_n(x) \to F(x)$ for each $x$ as $n \to \infty$.
\end{proof}
\end{prop}

Since $F(x) = \P(X \leq x)$, convergence in distribution means that the probability for $X_n$ to be in a given range is approximately equal to the probability that the value of the limiting RV $X$ is in that range, provided $n$ is sufficiently large.

Thus, for a discrete sequence of RVs $X_n$`n to converge in distribution to another discrete RV $X$ taking values in $\mathbb{Z}_+ = \{0,1,2,\ldots\}$, it is sufficient to show that $\lim_{n \to \infty}\P(X_n = x) = \P(X=x)$ for each $x \in \mathbb{Z_+}$.
We will use this fact to prove why we can approximate $\binomial$ RVs by a $\poisson$ under some limiting conditions.

\begin{example}[$\binomial(n,\lambda/n) {\rightsquigarrow} \poisson(\lambda)$]\label{EgBinomialConvergesInDistToPoisson}
In several situations, as we saw already, it becomes cumbersome to model the events using the $\binomial(n,\theta)$ RV, especially when when the parameter $\theta \propto 1/n$ and the events become rare.

\begin{center}
\begin{frame}
{$\binomial(n,\lambda/n)$ converges in distribution to $\poisson(\lambda)$ as $n \to \infty$, $\theta=\lambda/n \to 0$}
\end{frame}
\end{center}

However, for some real parameter $\lambda>0$, the $\binomial(n,\lambda/n)$ RV with probability of the number of successes in $n$ trials, with per-trial success probability $\lambda/n$, approaches the Poisson distribution with expectation $\lambda$, as $n$ approaches $\infty$ (actually, it converges in distribution).
The $\poisson(\lambda)$ RV is much simpler to work with than the combinatorially laden $\binomial(n,\theta=\lambda/n)$ RV.  We sketch the details of this next.

Let $X_n \sim \binomial(n,\theta=\lambda/n)$ and $Y \sim \poisson(\lambda)$ and let $\lambda=n\theta$ remain constant as  $n \to \infty$, $\theta \to 0$.
We need to show that $\lim_{n \to \infty} \P(X_n=x) = \P(Y=x) = e^{-\lambda}\lambda^x/x!$ for any $x \in \{0,1,2,3,\ldots,n\}$.
%{\normalsize
\begin{eqnarray}
\P(X=x)
&=&
\binom{n}{x} \left( \frac{\lambda}{n} \right)^x \left( 1- \frac{\lambda}{n} \right)^{n-x} \notag \\
&=& \frac{n(n-1)(n-2)\cdots(n-x+1)}{x(x-1)(x-2)\cdots (2)(1)}
\left( \frac{\lambda^x}{n^x} \right)
\left( 1- \frac{\lambda}{n} \right)^n
\left( 1- \frac{\lambda}{n} \right)^{-x} \notag \\
&=&
\overbrace{\left( \frac{n}{n} \right) \left( \frac{n-1}{n} \right) \left( \frac{n-2}{n} \right) \cdots \left( \frac{n-x+1}{n} \right)}
\overbrace{\left( \frac{\lambda^x}{x!} \right)}
\underbrace{\left( 1- \frac{\lambda}{n} \right)^n}
\underbrace{\left( 1- \frac{\lambda}{n} \right)^{-x}}  \notag \\
\end{eqnarray}

As $n \to \infty$, the expression below the first overbrace $\to 1$, while that below the second overbrace, being independent of $n$ remains the same.  By the elementary examples of limits
%\remove{
\ref*{EX:LimitExpofLambda} and \ref*{EX:Limit1MinusLambdaOverNToMinusK}%}
, as $n \to \infty$, the expression over the first underbrace approaches $e^{-\lambda}$ while that over the second underbrace approaches $1$.  Finally, we get the desired limit:

\[
\lim_{n \to \infty} \P(X=x)
= \frac{ e^{-\lambda} \lambda^x}{x!}  \ .
\]
%}
\end{example}

The second notion of convergence of RVs is convergence in probability.

\begin{definition}[Convergence in Probability]\label{D:ConvInProb}
Let $X_1,X_2,\ldots,$ be a sequence of RVs and let $X$ be another RV.  Let $F_n$ denote the DF of $X_n$ and $F$ denote the DF of $X$.  The we say that $X_n$ converges to $X$ in probability, and write:
\[
X_n \overset{\P}{\longrightarrow} X
\]
if for every real number $\epsilon > 0$,
\[
\lim_{n \rightarrow \infty} \P(|X_n-X|> \epsilon) = 0 \qquad \text{[in the sense of \hyperref[D:LimitofRealFunction]{Definition \ref*{D:LimitofRealFunction}}].}
\]
Once again, the above limit, by \eqref{E:ProbOfRV} in our \hyperref[D:RV]{Definition \ref*{D:RV}} of a RV, can be equivalently expressed as follows:
\[
\lim_{n \rightarrow \infty} \P( \ \{\omega: |X_n(\omega) - X(\omega)| > \epsilon\} \ )=0, \qquad \text{ie,} \qquad \P( \ \{\omega: |X_n(\omega) - X(\omega)| > \epsilon\} \ ) \rightarrow 0, \quad \text{as} \quad n \rightarrow \infty \ .
\]
\end{definition}

For the same sequence of RVs in  \hyperref[CW:Normal01bynConvToPointMass0]{Classwork \ref*{CW:Normal01bynConvToPointMass0}} and \hyperref[EX:Normal01bynConvinDistToPointMass0]{Example \ref*{EX:Normal01bynConvinDistToPointMass0}} we are tempted to ask whether $X_n \sim \normal(0,1/n)$ converges in probability to $X \sim \pointmass(0)$, i.e.~whether $X_n \overset{\P}{\longrightarrow} X$.  We need some elementary inequalities in Probability to help us answer this question.  We visit these inequalities next.

\begin{prop}[Markov's Inequality]
Let $(\Omega,\C{F},P)$ be a probability triple and let $X=X(\omega)$ be a non-negative RV.  Then,
\begin{equation}\label{E:MarkovNeq}
\P(X \geq \epsilon) \leq \frac{\E(X)}{\epsilon}, \qquad \text{for any} \quad \epsilon > 0 \ .
\end{equation}
{\normalsize
\begin{proof}
\begin{eqnarray}
X &=& X \BB{1}_{ \{y: y \geq \epsilon \} } (x) + X \BB{1}_{ \{y: y < \epsilon \} } (x) \notag \\
&\geq& X \BB{1}_{ \{y: y \geq \epsilon \} } (x) \notag \\
&\geq& \epsilon  \BB{1}_{ \{y: y \geq \epsilon \} } (x) \notag \\
\end{eqnarray}
Finally, taking expectations on both sides of the above inequality and then using the fact that the expectation of an indicator function of an event is simply the probability of that event \eqref{E:ExpectationofIndicator}, we get the desired result:
\[
\E(X) \geq \epsilon \E( \BB{1}_{ \{y: y \geq \epsilon \} } (x)) = \epsilon \P(X \geq \epsilon) \ .
\]
\end{proof}
}
\end{prop}
Let us look at some immediate consequences of Markov's inequality.
\begin{prop}[Chebychev's Inequality]
For {\bf any} RV $X$ and any $\epsilon > 0$,
\begin{eqnarray}
\P(|X| > \epsilon) &\leq& \frac{\E(|X|)}{\epsilon} \label{E:ChebychevNeq1} \\
\P(|X| > \epsilon) = \P(X^2 \geq \epsilon^2) &\leq& \frac{\E(X^2)}{\epsilon^2} \label{E:ChebychevNeq2} \\
\P(|X-\E(X)| \geq \epsilon) = \P((X-\E(X))^2 \geq \epsilon^2) &\leq& \frac{\E(X-\E(X))^2}{\epsilon^2}  = \frac{\V(X)}{\epsilon^2}  \label{E:ChebychevNeq3}
\end{eqnarray}
{\normalsize
\begin{proof}
All three forms of Chebychev's inequality are mere corollaries (careful reapplications) of Markov's inequality.
\end{proof}
}
\end{prop}

Armed with Markov's inequality we next enquire the convergence in probability for the sequence of RVs in \hyperref[CW:Normal01bynConvToPointMass0]{Classwork \ref*{CW:Normal01bynConvToPointMass0}} and \hyperref[EX:Normal01bynConvinDistToPointMass0]{Example \ref*{EX:Normal01bynConvinDistToPointMass0}}.

\begin{example}[Convergence in probability]\label{EX:Normal01bynConvinProbToPointMass0}
Does the the sequence of RVs $\{X_n\}_{n=1}^{\infty}$, where $X_n \sim \normal(0,1/n)$, converge in probability to $X \sim \pointmass(0)$, i.e.~does $X_n \overset{\P}{\longrightarrow} X$ ?

To find out if $X_n \overset{\P}{\longrightarrow} X$, we need to show that for any $\epsilon >0$, $\lim_{n \to \infty} \P(|X_n-X|>\epsilon)=0$.

Let $\epsilon$ be any real number greater than $0$, then
\begin{eqnarray}
\P(|X_n|>\epsilon) &=& \P(|X_n|^2 > \epsilon^2) \notag \\
&\leq& \frac{\E(X_n^2)}{\epsilon^2} \qquad \text{[by Markov's Inequality \eqref{E:MarkovNeq}]} \notag \\
&=& \frac{\frac{1}{n}}{\epsilon^2} \to 0, \quad \text{as} \quad n \to \infty \qquad \text{[in the sense of \hyperref[D:LimitofRealFunction]{Definition \ref*{D:LimitofRealFunction}}].} \notag
\end{eqnarray}
Hence, we have shown that for any $\epsilon >0$, $\lim_{n \to \infty} \P(|X_n-X|>\epsilon)=0$ and therefore by \hyperref[D:ConvInProb]{Definition \ref*{D:ConvInProb}}, $X_n \overset{\P}{\longrightarrow} X$ or $X_n \overset{\P}{\longrightarrow} 0$.

{\normalsize
{\bf Convention:} When $X$ has a $\pointmass(\theta)$ distribution and $X_n \overset{\P}{\longrightarrow} X$, we simply write $X_n \overset{\P}{\longrightarrow} \theta$.
}
\end{example}

\begin{definition}[Convergence Almost Surely (or with Probability $1$)]
To say that the sequence of RVs $\{X_n\}_{n=1}^{\infty}$ converges almost surely (or with probability $1$ or strongly) towards another RV $X$ on the same probability space $(\Omega,\mathcal{F},\P)$, as denoted by
\[
X_n \overset{a.s.}{\to} X
\]
means that
\[
\P \left( \{ \lim_{n \to \infty} X_n = X \} \right) = 1 \quad \iff \quad \P \left( \{\omega \in \Omega : \lim_{n \to \infty} X_n(\omega) = X(\omega)\} \right) = 1.
\]
This means that the values of $X_n$ approach the value of $X$, in the sense that events for which $X_n$ does not converge to $X$ have probability $0$.
\end{definition}

Other notions of convergence are termed sure convergence or pointwise convergence, such as convergence in mean.
But the above three types of convergence are elementary. % and enough to appreciate the subtle issues with convergence of random variables.

\subsection{Properties of Convergence of RVs$^{**}$}

We will merely state some properties (without proofs that are hyper-linked for the curious student as they are advanced for this course) and relations between the three notions of convergence with some examples to better appreciate the subtleties among them.
You will study the proofs of these statements in Probability Theory II.
Just remember that subtle implication relations exist between the three notions.

%Now that we have been introduced to three notions of convergence for sequences of RVs we can begin to appreciate the  construction of limiting random variables from existing ones. % We will see a limiting sum of $n$ independent $\bernoulli(\theta)$ RVs as $n \to \infty$ and $\theta \to 0$ such that $n \theta = \lambda$. %  statements of the basic limit theorems of Statistics.
%But first we need some analytic tools.

%But first we formally define a statistic.

\bit
\item Convergence almost surely implies convergence in probability\footnote{{\tiny \url{https://en.wikipedia.org/wiki/Proofs\_of\_convergence\_of\_random\_variables\#Convergence\_almost\_surely\_implies\_convergence\_in\_probability}}}
\[
\boxed{X_n \overset{a.s.}{\to} X \implies X_n \overset{\P}{\to} X \enspace .}
\]
\item By the Borel-Cantelli Lemma \footnote{{\tiny \url{https://en.wikipedia.org/wiki/Borel\%E2\%80\%93Cantelli\_lemma}}}, convergence in probability does not imply almost sure convergence in the discrete case \footnote{{\tiny \url{https://en.wikipedia.org/wiki/Proofs_of_convergence_of_random_variables\#Convergence_in_probability_does_not_imply_almost_sure_convergence_in_the_discrete_case}}}
\item Convergence in probability implies convergence in distribution \footnote{{\tiny \url{https://en.wikipedia.org/wiki/Proofs_of_convergence_of_random_variables\#Convergence_in_probability_implies_convergence_in_distribution}}}
\[
\boxed{X_n \overset{\P}{\to} X \implies X_n \rightsquigarrow X \enspace .}
\]
\item Convergence in distribution to a constant $\theta$ implies convergence in probability to $\theta$: \footnote{{\tiny \url{https://en.wikipedia.org/wiki/Proofs_of_convergence_of_random_variables\#Convergence_in_distribution_to_a_constant_implies_convergence_in_probability}}}
\[
X_n \rightsquigarrow \pointmass(\theta) \implies X_n \overset{\P}{\to} \pointmass(\theta) \enspace .
\]
\item In general, convergence in distribution does not imply convergence in probability.  %\footnote{{\tiny \url{}}}
\eit

\section{Law of Large Numbers}

\begin{prop}[Law of Large Numbers (LLN): $\overline{X}_n \overset{\P}{\longrightarrow} \E(X_1)$]
If we are given a sequence if independent and identically distributed RVs, $X_1,X_2,\ldots \overset{\IID}{\sim} X_1$ and if $\E(X_1)$ exists, as per \eqref{E:ExpectationExists}, i.e., $\E(\abs(X_1))< \infty$, and the variance is finite, i.e., $\V(X_1) < \infty$, then the sample mean $\overline{X}_n$ converges in probability to the expectation of any one of the IID RVs, say $\E(X_1)$ by convention.  More formally, we write:
\[
\text{If} \quad X_1,X_2,\ldots \overset{\IID}{\sim} X_1 \ \text{and if } \ \E(X_1) \ \text{exists, then } \ \overline{X}_n \overset{\P}{\longrightarrow} \E(X_1) \ .
\]
{\normalsize
\begin{proof}
Because $\V(X_1) < \infty$, we have:
\begin{eqnarray}
\P(| \overline{X}_n - \E(\overline{X}_n) | \geq \epsilon)
&=& \frac{\V(\overline{X}_n)}{\epsilon^2} \qquad \text{{\scriptsize [by applying Chebychev's inequality \eqref{E:ChebychevNeq3} to the RV $\overline{X}_n$]}} \notag \\
&=& \frac{\frac{1}{n}\V(X_1)}{\epsilon^2} \qquad \text{{\scriptsize [by the IID assumption of $X_1,X_2,\ldots$ we can apply \eqref{E:VarOfSampleMeanOfIIDSeq}]}} \notag
\end{eqnarray}
Therefore, for any given $\epsilon>0$,
\begin{eqnarray}
\P(| \overline{X}_n - \E(X_1) | \geq \epsilon)
&=&  \P(| \overline{X}_n - \E(\overline{X}_n) | \geq \epsilon) \qquad \text{{\scriptsize [by the IID assumption of $X_1,X_2,\ldots$,  $\E(\overline{X}_n)=\E(X_1)$, as per \eqref{E:ExpOfSampleMeanOfIDSeq}]}} \notag \\
&=&  \frac{\frac{1}{n}\V(X_1)}{\epsilon^2} \to 0, \quad \text{as} \quad n \to \infty \ , \notag
\end{eqnarray}
or equivalently, $\lim_{n \to \infty} \P(| \overline{X}_n - \E(X_1) | \geq \epsilon) = 0$.  And the last statement is the definition of the claim made by the law of large numbers (LLN), namely that $\overline{X}_n \overset{\P}{\longrightarrow} \E(X_1)$ .
\end{proof}
}

\begin{prop}[Weak Law of Large Numbers (WLLN): $\overline{X}_n \rightsquigarrow \pointmass(\E(X_1))$]
If we are given a sequence of independently and identically distributed (IID) RVs, $X_1,X_2,\ldots \overset{\IID}{\sim} X_1$ and if $\E(X_1)$ exists, i.e.~$\E(\abs(X)) < \infty$, then the sample mean $\overline{X}_n$ converges in distribution to the expectation of any one of the IID RVs, say $\pointmass(\E(X_1))$ by convention.  More formally, we write:
\[
\text{If} \quad X_1,X_2,\ldots \overset{\IID}{\sim} X_1 \ \text{and if } \ \E(X_1) \ \text{exists, then } \ \overline{X}_n \rightsquigarrow \pointmass(\E(X_1)) \ \text{ as } n\to\infty \enspace .
\]
\end{prop}

{\normalsize
\begin{proof}
Our proof now is based on the convergence of characteristic functions (CFs) pointwise to the CF of the limiting RV, as this implies, by L\'evy's Continuity Theorem on CFs \footnote{\url{https://en.wikipedia.org/wiki/L\%C3\%A9vy\%27s_continuity_theorem}}, the convergence of the corresponding distribution functions (DFs).

First, the CF of $\pointmass(\E(X_1))$ is
\[
\E(e^{\imath t \E(X_1)})=e^{\imath t \E(X_1)} \enspace,
\]
since $\E(X_1)$ is just a constant, i.e., a $\pointmass$ RV that puts all of its probability mass at $\E(X_1)$.

Second, the CF of $\ol{X}_n$ is
\begin{align*}
\E \left( e^{\imath t \ol{X}_n}\right)
&= \E \left( e^{\imath t \frac{1}{n} \sum_{k=1}^n X_k} \right) = \E \left( \prod_{k=1}^n e^{\imath t X_k/n} \right) = \prod_{k=1}^n \E \left( e^{\imath t X_k/n} \right) = \prod_{k=1}^n \cf_{X_k}(t/n)\\
&= \prod_{k=1}^n \cf_{X_1}(t/n) =\left(\cf_{X_1}(t/n)\right)^n \enspace .
\end{align*}

Let us recall Landau's ``small o'' notation for the relation between two functions.
We say,  $f(x)$ is {\bf small o} of $g(x)$ if $f$ is dominated by $g$ as $x \to \infty$, i.e., $\frac{|f(x)|}{|g(x)|} \to 0$ as $x \to \infty$.
More formally, for every $\epsilon > 0$, there exists an $x_{\epsilon}$ such that for all $x > x_{\epsilon}$ $|f(x)| < \epsilon |g(x)|$.
For example, $\log(x)$ is $o(x)$, $x^2$ is $o(x^3)$ and $x^m$ is $o(x^{m+1})$ for $m\geq 1$.

Third, we can expand any CF whose expectation exists as a Taylor series with a remainder term that is $o(t)$ as follows:
\[
\cf_X(t) = 1 + \imath t \E(X) + o(t) \enspace .
\]
Hence,
\[
\cf_{X_1}(t/n) = 1 + \imath \frac{t}{n} \E(X_1) + o\left(\frac{t}{n}\right)
\]
and
\[
E \left( e^{\imath t \ol{X}_n}\right) = \left( 1 + \imath \frac{t}{n} \E(X_1) + o\left(\frac{t}{n}\right) \right)^n
\to e^{\imath t \E(X_1)} \text{ as } n \to \infty \enspace .
\]
For the last limit we have used $\left( 1+\frac{x}{n}\right)^n \to e^x$ as $n \to \infty$.

Finally, we have shown that $E \left( e^{\imath t \ol{X}_n}\right)$, the CF of the $n$-sample mean RV $\ol{X}_n$, converges to $\E(e^{\imath t \E(X_1)})=e^{\imath t \E(X_1)}$, the CF of the $\pointmass(\E(X_1))$ RV, as the sample size $n$ tends to infinity.
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%--too heavy for Inference Course - not introduced to conv in prob, Markov ineq or Chebychev ineq
%\begin{proof}
%For simplicity, we will prove a slightly weaker result by assuming finite variance of $X_1$.  Suppose $\V(X_1) < \infty$, then:
%\begin{eqnarray}
%\P(| \overline{X}_n - \E(\overline{X}_n) | \geq \epsilon)
%&=& \frac{\V(\overline{X}_n)}{\epsilon^2} \qquad \text{{\scriptsize [by applying Chebychev's inequality \eqref{E:ChebychevNeq3} to the RV $\overline{X}_n$]}} \notag \\
%&=& \frac{\frac{1}{n}\V(X_1)}{\epsilon^2} \qquad \text{{\scriptsize [by the IID assumption of $X_1,X_2,\ldots$ we can apply \eqref{E:VarOfSampleMeanOfIIDSeq}]}} \notag
%\end{eqnarray}
%Therefore, for any given $\epsilon>0$,
%\begin{eqnarray}
%\P(| \overline{X}_n - \E(X_1) | \geq \epsilon)
%&=&  \P(| \overline{X}_n - \E(\overline{X}_n) | \geq \epsilon) \qquad \text{{\scriptsize [by the IID assumption of $X_1,X_2,\ldots$,  $\E(\overline{X}_n)=\E(X_1)$, as per \eqref{E:ExpOfSampleMeanOfIDSeq}]}} \notag \\
%&=&  \frac{\frac{1}{n}\V(X_1)}{\epsilon^2} \to 0, \quad \text{as} \quad n \to \infty \ , \notag
%\end{eqnarray}
%or equivalently, $\lim_{n \to \infty} \P(| \overline{X}_n - \E(X_1) | \geq \epsilon) = 0$.  And the last statement is the Definition of the claim made by the weak law of large numbers (LLN), namely that $\overline{X}_n \overset{\P}{\longrightarrow} \E(X_1)$ .
%\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
}
\end{prop}

\subsubsection{Heuristic Interpretation of LLN}
The distribution of the sample mean RV $\overline{X}_n$ obtained from an independent and identically distributed sequence of RVs $X_1,X_2,\ldots$ {\scriptsize [i.e.~all the RVs $X_i$'s are independent of one another and have the same distribution function, and thereby the same expectation, variance and higher moments]}, concentrates around the expectation of any one of the RVs in the sequence, say that of the first one $\E(X_1)$ {\scriptsize [without loss of generality]}, as $n$ approaches infinity.  See Figure~\ref{F:RunningMeansFairDieFairCoinUnif01Exp1By10} for examples of 20 replicates of the sample mean of IID sequences from four RVs.  All the sample mean trajectories converge to the corresponding population mean.

\begin{figure}[htpb]
\caption{Sample mean $\ol{X}_n$ as a function of sample size $n$ for 20 replications from independent realizations of a fair die (blue), fair coin (magenta), $\uniform(0,30)$ RV (green) and $\exponential(0.1)$ RV (red) with population means $(1+2+3+4+5+6)/6=21/6=3.5$, $(0+1)/2=0.5$, $(30-0)/2=15$ and $1/0.1=10$, respectively.\label{F:RunningMeansFairDieFairCoinUnif01Exp1By10}}
\centering   \makebox{\includegraphics[width=6.5in]{figures/RunningMeansFairDieFairCoinUnif01Exp1By10}}
\end{figure}

\begin{example}[Bernoulli WLLN and Galton's Quincunx]\label{EgBernoulliWLLN}
We can appreciate the WLLN for $\overline{X}_n = n^{-1} S_n = \sum_{i=1}^{n} X_i$, where $X_1,X_2,\ldots,X_n \overset{\IID}{\sim} Bernoulli(p)$ using the paths of balls dropped into Galton's Quincunx of Sec.~\ref{S:Quincunx}.
\end{example}

\subsubsection{$\cauchy$ whose expectations does not exist has no Law of Large Numbers}

 Recall that the mean of the $\cauchy$ RV $X$ does not exist since $\int \left|x\right|\,dF(x) = \infty$ \eqref{E:CauchyMeanDoesNotExist}.  We will investigate this in \hyperref[LW:RunningMeanCauchy]{Labwork~\ref*{LW:RunningMeanCauchy}}.

 \begin{labwork}[Running mean of the Standard Cauchy RV]\label{LW:RunningMeanCauchy}
Let us see what happens when we plot the running sample mean for an increasing sequence of IID samples from the Standard Cauchy RV $X$ by implementing the following script file:
 \VrbMf[label=PlotStandardCauchyRunningMean.m]{scripts/PlotStandardCauchyRunningMean.m}

%}%end remove

\begin{figure}[htpb]
\caption{Unending fluctuations of  the running means based on $n$ IID samples from the Standard $\cauchy$ RV $X$ in each of five replicate simulations (blue lines).  The running means, based on $n$ IID samples from the $\uniform(0,10)$ RV, for each of five replicate simulations (magenta lines). \label{F:plot5RunningMeansStandardcauchyUnif010}}
\centering   \makebox{\includegraphics[width=5.5in]{figures/plot5RunningMeansStandardcauchyUnif010}}
\end{figure}

The resulting plot is shown in \hyperref[F:plot5RunningMeansStandardcauchyUnif010]{Figure~\ref*{F:plot5RunningMeansStandardcauchyUnif010}}.  Notice that the running means or the sample mean of $n$ samples as  a function of $n$, for each of the five replicate simulations, never settles down to a particular value.  This is because of the ``thick tails'' of the density function for this RV which produces extreme observations.  Compare them with the running means, based on $n$ IID samples from the $\uniform(0,10)$ RV, for each of five replicate simulations (magenta lines).  The latter sample means have settled down stably to the mean value of $5$ after about $700$ samples.
%\remove{
\end{labwork}
%}%end remove

\subsection{Application: Point Estimation of $E(X_1)$}
LLN gives us a method to obtain a {\bf point estimator} that gives ``the single best guess'' for the possibly unknown population mean $E(X_1)$ based on $\ol{X}_n$, the sample mean, of a simple random sequence (SRS) or independent and identically distributed (IID) sequence of $n$ RVs $X_1,X_2,\ldots, X_n \overset{IID}{\sim} X_1$.

\begin{example}\label{Eg:LLNExponential}
Let $X_1,X_2,\ldots, X_n \overset{IID}{\sim} X_1$, where $X_1$ is an $\exponential(\lambda^*)$ RV, i.e., let
\[
X_1,X_2,\ldots, X_n \overset{IID}{\sim} \exponential(\lambda^*) \enspace .
\]
Typically, we do not know the ``true'' parameter $\lambda^* \in \BB{\Lambda} = (0,\infty)$ or the population mean $E(X_1) = 1/\lambda^*$.
But by LLN, we know that
\[
\ol{X}_n \rightsquigarrow \pointmass(E(X_1)) \enspace ,
\]
and therefore, we can use the sample mean $\ol{X}_n$ as a point estimator of $E(X_1)= 1/\lambda^*$.

Now, suppose you model seven waiting times in nearest minutes between Orbiter buses at Balgay street as follows:
\[
X_1,X_2,\ldots, X_7 \overset{IID}{\sim} \exponential(\lambda^*) \enspace ,
\]
and have the following realization as your observed data:
\[
(x_1,x_2,\ldots,x_7) = (2,12,8,9,14,15,11) \enspace .
\]
Then you can use the observed sample mean $\ol{x}_7=(2+12+8+9+14+15+11)/7=71/7 \approxeq 10.14$ as a {\bf point estimate} of the population mean $E(X_1)=1/\lambda^*$.
By the rearrangement $\lambda^*=1/E(X_1)$, we can also obtain a point estimate of the ``true'' parameter $\lambda^*$ from $1/\ol{x}_7=7/71 \approxeq 0.0986$.
\end{example}

\begin{rem}[Point estimates are realizations of the Point Estimator]
We say the statistic $\ol{X}_n$, which is a random variable that depends on the data \rv~$(X_1,X_2,\ldots,X_n)$, is a {\bf point estimator} of $E(X_1)$.
But once we have a realization of the data \rv~, i.e., our observed data vector $(x_1,x_2,\ldots,x_n)$ and its corresponding realization as observed sample mean $\ol{x}_n$, we say $\ol{x}_n$ is a {\bf point estimate} of $E(X_1)$.  In other words, the point estimate $\ol{x}_n$ is a realization of the the random variable $\ol{X}_n$ called the point estimator of $E(X_1)$.
Therefore, when we observe a new data vector $(x'_1,x'_2,\ldots,x'_n)$  that is different from our first data vector $(x_1,x_2,\ldots,x_n)$, our point estimator of $E(X_1)$ is still $\ol{X}_n$ but the point estimate $n^{-1}\sum_{i=1}^nx'_i$ may be different from the first point estimate $n^{-1}\sum_{i=1}^nx_i$.  The sample means from $n$ samples for 20 replications (repeats of the experiment) are typically distinct especially for small $n$ as shown in Figure~\ref{F:RunningMeansFairDieFairCoinUnif01Exp1By10}.
\end{rem}

\begin{example}\label{Eg:LLNBernoulli}
Let $X_1,X_2,\ldots, X_n \overset{IID}{\sim} X_1$, where $X_1$ is an $\bernoulli(\theta^*)$ RV, i.e., let
\[
X_1,X_2,\ldots, X_n \overset{IID}{\sim} \bernoulli(\theta^*) \enspace .
\]
Typically, we do not know the ``true'' parameter $\theta^* \in \BB{\Theta} = [0,1]$, which is the same as the population mean $E(X_1) = \theta^*$.
But by LLN, we know that
\[
\ol{X}_n \rightsquigarrow \pointmass(E(X_1)) \enspace ,
\]
and therefore, we can use the sample mean $\ol{X}_n$ as a point estimator of $E(X_1)= \theta^*$.

Now, suppose you model seven coin tosses (encoding {\sf Heads} as $1$ with probability $\theta^*$ and {\sf Tails} as $0$ with probability $1-\theta^*$) as follows:
\[
X_1,X_2,\ldots, X_7 \overset{IID}{\sim} \bernoulli(\theta^*) \enspace ,
\]
and have the following realization as your observed data:
\[
(x_1,x_2,\ldots,x_7) = (0,1,1,0,0,1,0) \enspace .
\]
Then you can use the observed sample mean $\ol{x}_7=(0+1+1+0+0+1+0)/7=3/7 \approxeq 0.4286$ as a {\bf point estimate} of the population mean $E(X_1)=\theta^*$.
Thus, our ``single best guess'' for $E(X_1)$ which is the same as the probability of {\sf Heads} is $\ol{x}_7=3/7$.
\end{example}

Of course, if we tossed the same coin in the same IID manner another seven times or if we observed another seven waiting times of orbiter buses at a different bus-stop or on a different day we may get a different point estimate for $E(X_1)$.
See the intersection of the twenty magenta sample mean trajectories for simulated tosses of a fair coin from IID $\bernoulli(\theta^*=1/2)$ RVs and the twenty red sample mean trajectories for simulated waiting times from IID $\exponential(\lambda^*=1/10)$ RVs in Figure~\ref{F:RunningMeansFairDieFairCoinUnif01Exp1By10} with $n=7$.
Clearly, the point estimates for such a small sample size are fluctuating wildly!
However, the fluctuations in the point estimates settles down for larger sample sizes.

The {\em next natural question is how large should the sample size be} in order to have a small interval of width, say $2 \epsilon$, ``contain'' $E(X_1)$, the quantity of interest, with a high probability, say $1-\alpha$?  If we can answer this then we can make probability statements like the following:
\[
P( \mathsf{error} < \mathsf{tolerance}) = P( |\ol{X}_n - E(X_1)| < \epsilon ) = P(-\epsilon < \ol{X}_n - E(X_1) < \epsilon) = 1-\alpha \enspace .
\]

In order to ensure the $\mathsf{error}=|\ol{X}_n - E(X_1)|$ in our estimate of $E(X_1)$ is within a required $\mathsf{tolerance}=\epsilon$ we need to know the full distribution of $\ol{X}_n - E(X_1)$ itself.  The Central Limit Theorem (CLT) helps us here.

\section{Central Limit Theorem}

What if we scale the sum of $X_i$`s by $\sqrt{n}$ instead of $n$?

\begin{Exercise}[title={What if we scale by $\sqrt{n}$},label={xSumOFUnifminu1To1DividedBySqrtn}]
After reading Sec.~\ref{S:ConvOfRVs} up to now, think carefully about what you need to be able to show that $Z_n := 1/\sqrt{n}\sum_{i=1}^n X_i$ converges in distribution to the $\normal(0,1/3)$ RV, where $X_i \overset{IID}{\sim} \uniform(-1,1)$. {Hint: Characteristic functions}
%\ExePart
%\Question
%\subQuestion Show that...
%\subQuestion In this question...
%\subsubQuestion Show that...
%\subsubQuestion Conclude...
%\subQuestion Conclude.
%\Question Show that if $b > 1$...
%\ExePart
%\Question What happens to if $b=1$?
\end{Exercise}

\remove{ % removing CSE book CLT without proof
\begin{prop}[Central Limit Theorem (CLT)]
Let $X_1,X_2,\ldots \overset{\IID}{\sim} X_1$ and suppose $\E(X_1)$ and $\V(X_1)$ exists, then
{\small
\begin{eqnarray}
%1
\overline{X}_n = \frac{1}{n} \sum_{i=1}^n X_i
& \rightsquigarrow &
X \sim \normal \left( \E(X_1),\frac{\V(X_1)}{n}  \right) \ , \\
%2
\overline{X}_n -\E(X_1)
& \rightsquigarrow &
X-\E(X_1) \sim \normal \left( 0,\frac{\V(X_1)}{n}  \right) \ , \\
%3
\sqrt{n} \left( \overline{X}_n -\E(X_1) \right)
& \rightsquigarrow &
\sqrt{n} \left( X-\E(X_1) \right) \sim \normal \left( 0,\V(X_1)  \right) \ , \\
%4
Z_n :=  \frac{\overline{X}_n-\E(\overline{X}_n)}{\sqrt{\V(\overline{X}_n)}}
= \frac{\sqrt{n} \left( \overline{X}_n -\E(X_1) \right)}{\sqrt{\V(X_1)}}
& \rightsquigarrow &
Z  \sim \normal \left( 0,1  \right) \ , \\
%5
\lim_{n \to \infty} P\left( \frac{\overline{X}_n-\E(\overline{X}_n)}{\sqrt{\V(\overline{X}_n)}}  \leq z \right)
= \lim_{n \to \infty} \P(Z_n \leq z)
&=&
\Phi(z) := \int_{- \infty}^z \left( \frac{1}{\sqrt{2 \pi}} \ \exp \left( \frac{-x^2}{2} \right) \right) dx \ .
\end{eqnarray}
}
Thus, for sufficiently large $n$ (say $n>30$) we can make the following approximation:
\begin{equation}\label{E:CLTApprox}
P\left( \frac{\overline{X}_n-\E(\overline{X}_n)}{\sqrt{\V(\overline{X}_n)}}  \leq z \right)
\approxeq
\P(Z \leq z)
=
\Phi(z) := \int_{- \infty}^z \left( \frac{1}{\sqrt{2 \pi}} \ \exp \left( \frac{-x^2}{2} \right) \right) dx \ .
\end{equation}
{\scriptsize
\begin{proof}
See any intermediate to advanced undergraduate text in Probability.  Start from the index looking for ``Central Limit Theorem'' to find the page number for the proof \ldots .
\end{proof}
}
{\bf Heuristic Interpretation of CLT:}  Probability statements about the sample mean RV $\overline{X}_n$ can be approximated using a Normal distribution.
\end{prop}
Here is a simulation showing CLT in action.
\begin{VrbM}
>> % a demonstration of Central Limit Theorem --
>> % the sample mean of a sequence of n IID Exponential(lambda) RVs
>> % itself a Gaussian(1/lambda,lambda/n) RV
>> lambda=0.1; Reps=10000; n=10; hist(sum(-1/lambda * log(rand(n,Reps)))/n)
>> lambda=0.1; Reps=10000; n=100; hist(sum(-1/lambda * log(rand(n,Reps)))/n,20)
>> lambda=0.1; Reps=10000; n=1000; hist(sum(-1/lambda * log(rand(n,Reps)))/n,20)
\end{VrbM}

Let us look at an example that makes use of the CLT next.
\begin{example}[Errors in computer code (Wasserman03, p.~78)]\label{EX:CLTPoisson}
Suppose the collection of RVs $X_1,X_2, \ldots, X_n$ model the number of errors in $n$ computer programs named $1,2,\ldots,n$, respectively.  Suppose that the RV $X_i$ modeling the number of errors in the $i$-th program is the $Poisson(\lambda=5)$ for any $i=1,2,\ldots,n$.  Further suppose that they are independently distributed.  Succinctly, we suppose that
\[
X_1,X_2,\ldots,X_n \overset{\IID}{\sim} \poisson(\lambda=5) \ .
\]
Suppose we have $n=125$ programs and want to make a probability statement about $\overline{X}_n$ which is the average error per program out of these $125$ programs.  Since $\E(X_i) = \lambda=5$ and $\V(X_i)=\lambda=5$, we may want to know how often our sample mean $\overline{X}_{125}$ differs from the expectation of $5$ errors per program.  Using the CLT we can approximate $\P(\overline{X}_n < 5.5)$, for instance, as follows:
\begin{eqnarray}
\P(\overline{X}_n < 5.5)
&=& P \left( \frac{\sqrt{n}(\overline{X}_n - \E(X_1))}{\sqrt{\V(X_1)}} < \frac{\sqrt{n}(5.5-\E(X_1))}{\sqrt{\V(X_1)}} \right) \notag \\
&\approxeq& P \left( Z < \frac{\sqrt{n}(5.5-\lambda)}{\sqrt{\lambda}} \right) \qquad \text{{\scriptsize [by \eqref{E:CLTApprox}, and $\E(X_1)=\V(X_1)=\lambda$]}} \notag \\
&=& P \left( Z < \frac{\sqrt{125}(5.5-5)}{\sqrt{5}} \right) \qquad \text{{\scriptsize [Since, $\lambda=5$ and $n=125$ in this Example]}} \notag \\
&=& \P(Z \leq 2.5) = \Phi(2.5) =  \int_{- \infty}^{2.5} \left( \frac{1}{\sqrt{2 \pi}} \ \exp \left( \frac{-x^2}{2} \right) \right) dx \approxeq 0.993790334674224 \ . \notag
\end{eqnarray}
The last number above needed the following:
\begin{labwork}[Numerical approximation of $\Phi(2.5)$]
The numerical approximation of $\Phi(2.5)$ was obtained via the following call to our $\erf$-based {\tt NormalCdf} function. % from \ref*{Mf: NormalCdfPdf}. MATLABback
\begin{VrbM}
>> format long
>> disp(NormalCdf(2.5,0,1))
   0.993790334674224
\end{VrbM}
\end{labwork}
\end{example}
The CLT says that if $X_1,X_2,\ldots \overset{\IID}{\sim} X_1$, then $Z_n := \sqrt{n}(\overline{X}_n-\E(X_1))/\sqrt{\V(X_1)}$ is approximately distributed as $\normal(0,1)$.  In \hyperref[EX:CLTPoisson]{Example \ref*{EX:CLTPoisson}}, we knew $\sqrt{\V(X_1)}$.   However, in general, we may not know $\sqrt{\V(X_1)}$.  The next proposition says that we may estimate $\sqrt{\V(X_1)}$ using the sample standard deviation $S_n$ of $X_1,X_2,\ldots,X_n$, according to \eqref{E:SampleStdDevRV}, and still make probability statements about the sample mean $\overline{X}_n$ using a Normal distribution.
\begin{prop}[CLT based on Sample Variance]
Let $X_1,X_2,\ldots \overset{\IID}{\sim} X_1$ and suppose $\E(X_1)$ and $\V(X_1)$ exists, then
\begin{equation}\label{E:CLTApproxSn}
\frac{\sqrt{n} \left( \overline{X}_n - \E(X_1) \right)}{S_n} \rightsquigarrow \normal(0,1) \ .
\end{equation}
\end{prop}

We will use \eqref{E:CLTApproxSn} for statistical estimation in the sequel.
}% end remove

%The next proposition is often referred to as the fundamental theorem of statistics and is at the heart of non-parametric inference, empirical processes, and computationally-intensive bootstrap techniques.
%\begin{prop}[Gilvenko-Cantelli Theorem]\label{P:Gilvenko-Cantelli}
%Let $X_1,X_2,\ldots,X_n \overset{\IID}{\sim} F$.  Then,
%\[
%\sup_x { | \widehat{F}_n(x) - F(x) | } \overset{\P}{\longrightarrow} 0 \ .
%\]
%{\scriptsize
%\begin{proof}
%Proof to be seen in STAT 318 or another advanced Statistics course.
%\end{proof}
%}
%\end{prop}
%{\bf Heuristic Interpretation of Gilvenko-Cantelli Theorem}:  As the sample size $n$ increases the empirical distribution function $\widehat{F}_n$ converges to the true DF $F$ in probability.

%\begin{figure}[htpb]
%\caption{Plots of ten distinct ECDFs $\widehat{F}_n$ based on $10$ sets of $n$ IID samples from $\uniform(0,1)$ RV $X$, as $n$ increases from $10$ to $100$ to $1000$.  The DF $F(x)=x$ over $[0,1]$ is shown in red.  The script of \hyperref[Mf:GilvenkoCantelliUnif01n10n100n100ECDFs]{Labwork \ref*{Mf:GilvenkoCantelliUnif01n10n100n100ECDFs}} was used to generate this plot.   \label{F:GilvenkoCantelliUnif01n10n100n100ECDFs}}
%\centering   \makebox{\includegraphics[width=7.0in]{figures/GilvenkoCantelliUnif01n10n100n100ECDFs}}
%\end{figure}

%\begin{prop}[The Dvoretzky-Kiefer-Wolfowitz (DKW) Inequality]
%Let $X_1,X_2,\ldots,X_n \overset{\IID}{\sim} F$.  Then, for any $\epsilon>0$,
%\begin{equation}\label{E:DKWNeq}
%P \left( \sup_x | \widehat{F}_n(x) - F(x) | > \epsilon  \right) \leq 2 \exp {(-2 n \epsilon^2)}
%\end{equation}
%\end{prop}

\begin{prop}[Central Limit Theorem (CLT)]\label{P:CLT}

If we are given a sequence of independently and identically distributed (IID) RVs, $X_1,X_2,\ldots \overset{\IID}{\sim} X_1$ and if $E(X) < \infty$ and $V(X_1)<\infty$, then the sample mean $\overline{X}_n$ converges in
%probability
distribution to the Normal RV with mean given by any one of the IID RVs, say $\E(X_1)$ by convention, and variance given by $\frac{1}{n}$ times the variance of any one of the IID RVs, say $V(X_1)$ by convention.  More formally, we write:
\begin{multline}\label{E:CLT}
\text{If} \quad X_1,X_2,\ldots \overset{\IID}{\sim} X_1 \ \text{and if } \ \E(X_1)<\infty, V(X_1) < \infty \\ \text{ then } \ \overline{X}_n \rightsquigarrow \normal\left(E(X_1), \frac{V(X_1)}{n}\right) \ \text{ as } n\to\infty \enspace ,
\end{multline}
or equivalently after standardization:
\begin{multline}\label{E:CLTStd}
\text{If} \quad X_1,X_2,\ldots \overset{\IID}{\sim} X_1 \ \text{and if } \ \E(X_1)<\infty, V(X_1) < \infty \\ \text{ then } \ \frac{\overline{X}_n - E(X_1)}{\sqrt{V(X_1)/n}} \rightsquigarrow Z \sim \normal\left(0,1 \right) \ \text{ as } n\to\infty \enspace .
\end{multline}

{\normalsize
\begin{proof}
{
Our proof is based on the convergence of characteristic functions (CFs).
We will prove the standardized form of the CLT in Equation~\eqref{E:CLTStd} by showing that the CF of
\[
U_n := \frac{\ol{X}_n-E(X_1)}{\sqrt{V(X_1)/n}}
\]
converges to the CF of $Z$, the $\normal(0,1)$ RV.
First, note from Equation~\eqref{E:cfStandardNormal} that the CF of $Z \sim \normal(0,1)$ is:
\[
\cf_Z(t) = E \left( e^{\imath t Z}\right) = e^{-t^2/2} \enspace .
\]
Second,
\[
U_n := \frac{\ol{X}_n-E(X_1)}{\sqrt{V(X_1)/n}}
= \frac{\sum_{k=1}^n X_k - n E(X_1)}{\sqrt{nV(X_1)}}
= \frac{1}{\sqrt{n}} \sum_{k=1}^n \left(\frac{X_k - E(X_1)}{\sqrt{V(X_1)}}\right)\enspace .
\]
Therefore, the CF of $U_n$ is
\begin{align*}
\cf_{U_n}(t)
&= E\left( \exp\left({\imath t U_n}\right)\right)
= E \left( \exp\left({\imath \frac{t}{\sqrt{n}}\sum_{k=1}^n \frac{X_k - E(X_1)}{\sqrt{V(X_1)}} }\right)\right)
= \prod_{k=1}^n E \left( \exp \left( \imath \frac{t}{\sqrt{n}} \frac{X_k - E(X_1)}{\sqrt{V(X_1)}} \right) \right)\\
&= \left( E \left( \exp \left( \imath \frac{t}{\sqrt{n}} \frac{X_1 - E(X_1)}{\sqrt{V(X_1)}} \right) \right)\right)^n \enspace .
\end{align*}
Now, if we let
\[
Y = \frac{X_1 - E(X_1)}{\sqrt{V(X_1)}}
\]
then
\[
E(Y) = 0 \, , \quad E(Y^2)=1 \, , \text{and} \quad V(Y)=1 \enspace .
\]
So, the CF of $U_n$ is
\[
\cf_{U_n}(t) = \left( \cf_Y\left(\frac{t}{\sqrt{n}}\right)\right)^n \enspace ,
\]
and since we can Taylor expand $\cf_Y(t)$ as follows:
\[
\cf_Y(t) = 1 + \imath t E(Y) + \imath^2 \frac{t^2}{2} E(Y^2) + o(t^2) \enspace,
\]
which implies
\[
\cf_Y\left(\frac{t}{\sqrt{n}}\right) = 1 + \frac{\imath t}{\sqrt{n}} E(Y) + \frac{\imath^2 t^2}{2n}  E(Y^2) + o\left(\frac{t^2}{n}\right) \enspace,
\]
we finally get
\[
\cf_{U_n}(t)
= \left( \cf_Y \left(\frac{t}{\sqrt{n}}\right) \right)^n
= \left( 1 +  \frac{\imath t}{\sqrt{n}} \times 0 + \frac{\imath^2 t^2}{2n}  \times 1 + o\left(\frac{t^2}{n}\right) \right)^n
= \left( 1 - \frac{t^2}{2n} + o\left(\frac{t^2}{n}\right) \right)^n
\to e^{-t^2/2} = \cf_Z(t) \enspace .
\]
For the last limit we have used $\left( 1+\frac{x}{n}\right)^n \to e^x$ as $n \to \infty$.
Thus, we have proved Equation~\eqref{E:CLTStd} which is equivalent to Equation~\eqref{E:CLT} by a standardization argument that if $W \sim \normal(\mu,\sigma^2)$ then $Z=\frac{W-\mu}{\sigma} \sim \normal(0,1)$ through the linear transformation $W=\sigma Z + \mu$ of Example~\ref{Eg:LinearTransfStdGaussianGaussian}.
}
\end{proof}
}
\end{prop}

\subsection{Application: Tolerating Errors in our estimate of $E(X_1)$}
Recall that we wanted to ensure the $\mathsf{error}=|\ol{X}_n - E(X_1)|$ in our estimate of $E(X_1)$ is within a required $\mathsf{tolerance}=\epsilon$ and make the following probability statement:
\[
P( \mathsf{error} < \mathsf{tolerance}) = P( |\ol{X}_n - E(X_1)| < \epsilon ) = P(-\epsilon < \ol{X}_n - E(X_1) < \epsilon) = 1-\alpha \enspace .
\]
To be able to do this we needed to know the full distribution of $\ol{X}_n - E(X_1)$ itself.

Due to the Central Limit Theorem (CLT) we now know that (assuming $n$ is large)
\begin{align*}
P(-\epsilon < \ol{X}_n - E(X_1) < \epsilon)
&\approxeq P\left(-\frac{\epsilon}{\sqrt{V(X_1)/n}} < \frac{\ol{X}_n - E(X_1)}{\sqrt{V(X_1)/n}} < \frac{\epsilon}{\sqrt{V(X_1)/n}}\right)\\
&= P\left(-\frac{\epsilon}{\sqrt{V(X_1)/n}} < Z < \frac{\epsilon}{\sqrt{V(X_1)/n}} \right) \enspace ,
\end{align*}
where $Z \sim \normal(0,1)$.

\begin{example}\label{EgProbOfErrorInMeanEstimateFromKnownVar}
Suppose an IID sequence of observations $(x_1,x_2,\ldots,x_{80})$ was drawn from a distribution with variance $V(X_1)=4$.
What is the probability that the error in $\ol{x}_n$ used to estimate $E(X_1)$ is less than $0.1$?

By CLT,
\[
P (\mathsf{error} < 0.1) \approxeq P \left( -\frac{0.1}{\sqrt{4/80}} < Z < \frac{0.1}{\sqrt{4/80}} \right)
= P(-0.447 < Z < 0.447)= 0.345 \enspace .
\]
\end{example}

Suppose you want the $\mathsf{error}$ to be less than $\mathsf{tolerance}=\epsilon$ with a certain probability $1-\alpha$.
Then we can use CLT to do such {\bf sample size calculations}.
Recall the DF $\Phi(z) = P(Z < z)$ is tabulated in the standard normal table and now we want
\[
P \left(-\frac{\epsilon}{\sqrt{V(X_1)/n}} < Z < \frac{\epsilon}{\sqrt{V(X_1)/n}} \right) = 1 -\alpha \enspace .
\]
We know,
\[
P \left(-z_{\alpha/2} < Z < z_{\alpha/2} \right) = 1 -\alpha \enspace ,
\]

\vspace{1.5cm}

{\tiny make the picture here of $f_Z(z) = \Phi'(z)$ to recall what $z_{\alpha/2}$, $z_{-\alpha/2}$, and the various areas below $f_Z(\cdot)$ in terms of $\Phi(\cdot)$ from the table really mean... (See Example~\ref{Eg:UsingNormalTables}).}

\vspace{1.5cm}

where, $\Phi(z_{\alpha/2}) = 1-\alpha/2$ and $\Phi(z_{-\alpha/2}) = 1- \Phi(z_{\alpha/2}) = \alpha/2$.
So, we set
\[
\frac{\epsilon}{\sqrt{V(X_1)/n}} = z_{\alpha/2}
\]
and rearrange to get
\begin{equation}\label{E:SampleSizeCalcForPopMean}
n = \left( \frac{\sqrt{V(X_1)} z_{\alpha/2}}{\epsilon}\right)^2
\end{equation}
for the needed sample size that will ensure that our $\mathsf{error}$ is less than our $\mathsf{tolerance}=\epsilon$ with probability $1-\alpha$.  Of course, if $n$ given by Equation~\eqref{E:SampleSizeCalcForPopMean} is not a natural number then we naturally round up to make it one!

A useful $z_{\alpha/2}$ value to remember: If $\alpha=0.05$ when the probability of interest $1-\alpha=0.95$ then $z_{\alpha/2} = z_{0.025} = 1.96$.

\begin{example}\label{EgHowLargeASampleSizeToEstimateMeanWithinTOL}
How large a sample size is needed to make the $\mathsf{error}$ in our estimate of the population mean $E(X_1)$ to be less than $0.1$ with probability $1-\alpha=0.95$ if we are observing IID samples from a distribution with a population variance $V(X_1)$ of $4$?

Using Equation~\eqref{E:SampleSizeCalcForPopMean} we see that the needed sample size is
\[
n =  \left( \frac{\sqrt{4} \times 1.96}{0.1} \right)^2 \approxeq 1537
\]
Thus, it pays to check the sample size needed in advance of experimentation, provided you already know the population variance of the distribution whose population mean you are interested in estimating within a given $\mathsf{tolerance}$ and with a high probability.
\end{example}

\subsection{Application: Set Estimation of $E(X_1)$}

A useful byproduct of the CLT is the $\mathbf{(1-\alpha)}$ {\bf confidence interval}, a random interval (or bivariate \rv) that contains $E(X_1)$, the quantity of interest, with probability $1-\alpha$:
\begin{equation}\label{E:ConfIntForPopMean}
\left( \ol{X}_n \pm z_{\alpha/2} \sqrt{V(X_1)/n} \right) :=
\left( \ol{X}_n - z_{\alpha/2} \sqrt{V(X_1)/n} \, , \, \ol{X}_n + z_{\alpha/2} \sqrt{V(X_1)/n} \right) \enspace .
\end{equation}

{\scriptsize
We can easily see how Equation~\eqref{E:ConfIntForPopMean} is derived from CLT as follows:
\begin{align*}
P \left( -z_{\alpha/2} < Z < z_{\alpha/2} \right)
&= 1-\alpha \\
P \left( -z_{\alpha/2} < \frac{\ol{X}_n-E(X_1)}{\sqrt{V(X_1)/n}} < z_{\alpha/2} \right)
&= 1-\alpha \\
P \left( -\ol{X}_n - z_{\alpha/2}\sqrt{V(X_1)/n}  < -E(X_1) < -\ol{X}_n + z_{\alpha/2} \sqrt{V(X_1)/n} \right)
&= 1-\alpha \\
P \left( \ol{X}_n + z_{\alpha/2}\sqrt{V(X_1)/n}  > E(X_1) > \ol{X}_n - z_{\alpha/2} \sqrt{V(X_1)/n} \right)
&= 1-\alpha \\
P \left( \ol{X}_n - z_{\alpha/2}\sqrt{V(X_1)/n}  < E(X_1) < \ol{X}_n + z_{\alpha/2} \sqrt{V(X_1)/n} \right)
&= 1-\alpha \\
P \left( E(X_1) \in \left(\ol{X}_n - z_{\alpha/2}\sqrt{V(X_1)/n} , \ol{X}_n + z_{\alpha/2} \sqrt{V(X_1)/n} \right) \right)
&= 1-\alpha \enspace .
\end{align*}
}

\begin{rem}[Heuristic interpretation of the $(1-\alpha)$ confidence interval]
If we repeatedly produced samples of size $n$ to contain $E(X_1)$ within a $\left( \ol{X}_n \pm z_{\alpha/2} \sqrt{V(X_1)/n} \right)$, say $100$ times, then on average, $(1-\alpha) \times 100$ repetitions will actually contain $E(X_1)$ within the random interval and $\alpha \times 100$ repetitions will fail to contain $E(X_1)$.
\end{rem}

So far, we have assumed we know the population variance $V(X_1)$ in an IID experiment with $n$ samples and tried to estimate the population mean $E(X_1)$.
But in general, we will not know $V(X_1)$.  We can still get a point estimate of $E(X_1)$ from the sample mean due to LLN but we won't be able to get a confidence interval for $E(X_1)$.
Fortunately, a more elaborate form of the CLT tells us that even when we substitute the sample variance $S_n^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i-\ol{X}_n)^2$ for the population variance $V(X_1)$ the following $1-\alpha$ confidence interval for $E(X_1)$ works!
\begin{equation}\label{E:ConfIntForPopMeanUsingSampleVariance}
\left( \ol{X}_n \pm z_{\alpha/2} S_n/\sqrt{n} \right) :=
\left( \ol{X}_n - z_{\alpha/2} S_n/\sqrt{n} \, , \, \ol{X}_n + z_{\alpha/2} S_n/\sqrt{n} \right) \enspace ,
\end{equation}
where, $S_n=\sqrt{S_n^2}$ is the sample standard deviation.

Let's return to our two examples again.

\begin{example}\label{EgExponentialCIOfMeanWaitingTimesWithSampleMeanAndVar}
We model the waiting times between Orbiter buses with unknown $E(X_1)=1/\lambda^*$ as
\[
X_1,X_2,\ldots,X_n \overset{IID}{\sim} \exponential(\lambda^*)
\]
and observed the following data, sample mean, sample variance and sample standard deviation:
\[
(x_1,x_2,\ldots,x_7) = (2,12,8,9,14,15,11), \, \ol{x}_7=10.143, \,  s^2_7 = 19.143, \, s_7 = 4.375 \enspace ,
\]
respectively.
Our point estimate and $1-\alpha=95\%$ confidence interval for $E(X_1)$ are:
\[
\ol{x}_7 = 10.143 \quad \text{ and } \quad (\ol{x}_7 \pm z_{\alpha/2} s_7/\sqrt{7}) = (10.143 \pm 1.96 \times 4.375/\sqrt{7}) = (6.9016,13.3841) \enspace ,
\]
respectively.  So with $95\%$ probability the true population mean $E(X_1)=1/\lambda^*$ is contained in $(6.9016,13.3841)$ and since the mean waiting time of $10$ minutes promised by the Orbiter bus company is also within $(6.9016,13.3841)$ we can be fairly certain that the company sticks to its promise.
\end{example}

\begin{example}\label{EgBernoulliCIWithSampleMeanAndVar}
We model the tosses of a coin with unknown $E(X_1)=\theta^*$ as
\[
X_1,X_2,\ldots,X_n \overset{IID}{\sim} \bernoulli(\theta^*)
\]
and observed the following data, sample mean, sample variance and sample standard deviation:
\[
(x_1,x_2,\ldots,x_7) = (0,1,1,0,0,1,0), \, \ol{x}_7=0.4286, \,  s^2_7 = 0.2857, \, s_7 = 0.5345 \enspace ,
\]
respectively.
Our point estimate and $1-\alpha=95\%$ confidence interval for $E(X_1)$ are:
\[
\ol{x}_7 = 0.4286 \quad \text{ and } \quad (\ol{x}_7 \pm z_{\alpha/2} s_7/\sqrt{7}) = (0.4286 \pm 1.96 \times 0.5345/\sqrt{7}) = (0.0326,0.8246) \enspace ,
\]
respectively.  So with $95\%$ probability the true population mean $E(X_1)=\theta^*$ is contained in $(0.0326,0.8246)$ and since $1/2$ is contained in this interval of width $0.792$ we cannot rule out that the flipped coin is not fair with $\theta^*=1/2$.
\end{example}

\begin{rem}
The normal-based confidence interval for $\theta^*$ (as well as $\lambda^*$ in the previous example) may not be a valid approximation here with just $n=7$ samples.  After all, the CLT only tells us that the point estimator $\widehat{\Theta}_n$ can be approximated by a normal distribution for large sample sizes.
When the sample size $n$ was increased from $7$ to $100$ by tossing the same coin another $93$ times, a total of $57$ trials landed as Heads.  Thus the point estimate and confidence interval for $E(X_1)=\theta^*$ based on the sample mean and sample standard deviations are:
\[
\widehat{\theta}_{100} = \frac{57}{100} = 0.57 \qquad \text{and} \qquad
 (0.57 \pm 1.96 \times 0.4975/\sqrt{100})
 =(0.4725, 0.6675) \enspace .
\]
Thus our confidence interval shrank considerably from a width of $0.792$ to $0.195$ after an additional $93$ Bernoulli trials.  Thus, we can make the width of the confidence interval as small as we want by making the number of observations or sample size $n$ as large as we can.
\end{rem}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\input{ExsInLimitsOfRVs.tex}


\section{Carrier--Greenspan transient solution}

A transient solution for flows on a sloping beach was proposed by Carrier and Greenspan~\cite{CG1958}. The water moves to the shore at an early time, then it becomes still when time is large.

Consider the dimensionless shallow water equations, as presented in the Carrier--Greenspan periodic solution.

The analytical solution is:
\begin{equation}
w = - \frac{u^2}{2} + \epsilon {Re}
\left[1- 2 \frac{5/4 - i\lambda}{\left\{(1-i\lambda)^2 + \sigma^2 \right\}^{3/2}}
+ \frac32 \frac{(1-i\lambda)^2}{\left\{ (1-i\lambda)^2 + \sigma^2 \right\}^{5/2}} \right],
\end{equation}
\begin{equation}
u = \frac{8\epsilon}{a} {Im} \left[ \frac{1}{\left\{(1-i\lambda)^2 + \sigma^2 \right\}^{3/2}}
- \frac34 \frac{1-i\lambda}{\left\{ (1-i\lambda)^2 + \sigma^2 \right\}^{5/2}}    \right],
\end{equation}
where
\begin{equation}
t = \frac12 a\lambda -u\,, \quad c = \frac14 a\sigma\,,
\end{equation}
in which $c=\sqrt{gh}$ is the wave propagation speed.
Here $\sigma \geq 0$ and we take $a=1.5(1+0.9\epsilon)^{1/2}$. Carrier and Greenspan~\cite{CG1958} observed that the waves do not break if $\epsilon$ is very small, namely $\epsilon \leq 0.23$. Setting $\sigma=0$ into this solution, we get the motion of the shoreline.

The initial condition is given by setting time $t=0$ in this analytical solution. Note that this analytical solution is defined in the dimensionless space. To implement this in the numerical test, we just need to scale it back to the dimensional space.

\subsection{Results}

We consider $\epsilon=0.2$. The following three figures show the stage, $x$-momentum, and $y$-momentum at several instants in time. We should see excellent agreement between the analytical and numerical solutions.

\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{stage_plot.png}
\end{center}
\caption{Stage results}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{xmom_plot.png}
\end{center}
\caption{Xmomentum results}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{xvel_plot.png}
\end{center}
\caption{Xvelocity results}
\end{figure}

\endinput


\lab{The Finite Element method}{The Finite Element method}
\label{lab:finite_element}

Recall that the heat equation is given by
\begin{align*}
y_t = \epsilon y_{xx} + f(x) %\label{FEM:diffusion}
\end{align*}
where $f(x)$ represents any heat sources in the system, and $\epsilon y_{xx}$ models the diffusion of heat.
Suppose we wish to study the distribution of heat in a fluid that is moving at some constant speed $a$.
We can model this by adding an advection term to the heat equation, giving us
\begin{align*}
y_t + ay_x = \epsilon y_{xx} + f(x).
\end{align*}
This fluid flows through a pipe from $x = 0$ to $x = 1$ at a speed $a = 1$, and as it travels it is warmed a constant rate $f= 1$.
Suppose also that $y = 2$ at $x = 0$, so that the fluid is already at a constant temperature as it enters the pipe.

As time increases we expect the temperature of the fluid in the pipe to reach a steady state distribution; this heat distribution would then satisfy the ODE
\begin{align}
\begin{split}
&{ } \epsilon y'' - y' = -1,\\
&{ } y(0) = 2.
\end{split}\label{eqn:FEM_steady_state}
\end{align}
At the moment this problem is not fully defined, since the ODE is second order and there is only one boundary condition.
Suppose a device is installed on the end of the pipe that nearly instantaneously brings the heat of the water up to $y = 4$.
Physically we expect this extra heat that is introduced at $x = 1$ to diffuse backward through the water in the pipe.

We will use the finite element method to solve this boundary value problem numerically.

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{FEM_solution.pdf}
\caption{The analytic solution of \eqref{eqn:FEM_steady_state} when $\epsilon = .1$, with the additional condition that $y(1) = 4$.}
\label{fig:FEM_analytic_solution}
\end{figure}

\section*{The Weak Formulation}
Consider the equation
\begin{align}
	\begin{split}
	&{ }\epsilon y'' - y' = -1,\\
	&{ }y(0) = \alpha, \quad y(1) = \beta .
	\end{split}\label{eqn:FEM_eqn1}
\end{align}
To find the solution $y$ using the finite element method, we reframe the problem and look at what is known as its weak formulation.

%We may assume that the solution $y$ of \eqref{eqn:FEM_eqn1} lies in some appropriate function space $V$.
%; in any case, we can approximate it arbitrarily well in $C^0[0,1]$ by a piecewise linear function.
Let $w$ be a smooth function on $[0,1]$ satisfying $w(0) = w(1) = 0$.
Multiplying \eqref{eqn:FEM_eqn1} by $w$ and integrating over $[0,1]$ yields
\begin{align*}
	\int_0^1 f w &= \int_0^1 \epsilon y''w - y'w, \\
	&= \int_0^1 -\epsilon y'w' - y'w.
\end{align*}
Define a bilinear function $a$ and a linear function $l$ by
\begin{align*}
a(y,w) &= \int_0^1 -\epsilon y'w' - y'w,\\
l(w) &= \int_0^1 f w.
\end{align*}
Rather than trying to solve \eqref{eqn:FEM_eqn1}, we instead consider the problem of finding a function $y$ such that
\begin{align}
	a(y,w) &= l(w), \quad \forall \, w \in V_0,
	\label{eqn:FEM_integral_form}
\end{align}
where $V$ is some appropriate vector space that is expected to allow us to approximate the solution $y$, and $V_0 = \{w \in V|w(0) = w(1) = 0\}$.
(For example, we could consider the space of functions that are piecewise linear with vertices at a fixed set of points.
This example is discussed further below.)
This equation is called the weak formulation of \eqref{eqn:FEM_eqn1}.

Let $\mathrm{P}_n$ be some partition of $[0,1]$, $0 = x_0 < x_1< \ldots < x_{n} = 1$, and let $V_n$ be the finite-dimensional vector space of continuous functions $v$ on $[0,1]$ where $v$ is linear on each subinterval $[{x_j,x_{j+1}}]$.
These subintervals are the finite elements for which this method is named.
$V_n$ has dimension $n+1$, since there are $n+1$ degrees of freedom for continuous piecewise linear functions in $V$.
Let $V_{n0}$ be the subspace of $V_n$ of dimension $n-1$ whose elements are zero at the endpoints of $[0,1]$, and let $\triangle x_n = \max_{0 \leq j \leq n-1}|x_{j+1} - x_j|$.

Let $\{\mathrm{P}_n\}$ be a sequence of partitions that are refinements of each other, such that $\triangle x_n \to 0$ as $n \to \infty$.
Then in particular $V_1 \subset V_2 \subset \ldots \subset V_n \ldots \subset V$.
For each partition $\mathrm{P}_n$ we can look for an approximation $y_n \in V_n$ for the true solution $y$; if this is done  correctly then $y_n \to y$ as $n \to \infty$.

\section*{The Numerical Method}
Consider a partition $\mathrm{P}_5 = \{x_0, x_1, \ldots, x_5\}$.
We will define some basis functions $\phi_i$, $i = 0, \ldots, 5$ for the corresponding vector space $V_5$.
Let the $\phi_i$ be the hat functions
\[\phi_i(x) = \begin{cases}
(x - x_{i-1})/h_i \quad \quad\text{ if } x \in [x_{i-1},x_i]\\
 (x_{i+1} - x)/h_{i+1} \quad \text{ if } x \in [x_{i},x_{i+1}]\\
0 \quad \quad \quad \quad \quad \quad \quad \,\,\text{ otherwise}
\end{cases}\]
where $h_i = x_i - x_{i-1}$; see Figures \ref{fig:FEM_one_basis_function} and \ref{fig:FEM_basis_functions}.

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{one_basis_function.pdf}
\caption{The basis function $\phi_3$.}
\label{fig:FEM_one_basis_function}
\end{figure}

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{basis_functions.pdf}
\caption{Basis functions for $V_5$.}
\label{fig:FEM_basis_functions}
\end{figure}

We look for an approximation $\hat{y} = \sum_{i=0}^5 k_i \phi_i \in V_5$ of the true solution $y$; to do this we must determine appropriate values for the constants $k_i$.
We impose the condition on $\hat{y}$ that
\[a(\hat{y},w) = l(w) \quad \forall \, w \in V_{50}.\]
Equivalently, we require that
\[a \left( \sum_{i=0}^5 k_i \phi_i,\phi_j \right) = l(\phi_j) \quad \text{for } j = 1,2,3,4,\]
since $\phi_1, \phi_2, \phi_3, \phi_4$ form a basis for $V_{50}$.

Since $a$ is bilinear, we obtain
\[\sum_{i=0}^5 k_i  a ( \phi_i,\phi_j ) = l(\phi_j) \quad \text{for } j = 1,2,3,4.\]
To satisfy the boundary conditions, we also require that $k_0 = \alpha$, $k_5 = \beta$.
These equations can be written in matrix form as
\begin{align} AK = \Phi,\label{eqn:FEM_linear_system}\end{align}
where
\[A = \left[\begin{array}{cccccc}1 & 0 & 0 & 0 & 0 & 0 \\a(\phi_0,\phi_1) & a(\phi_1,\phi_1) & a(\phi_2,\phi_1) & 0 & 0 & 0 \\0 & a(\phi_1,\phi_2) & a(\phi_2,\phi_2) & a(\phi_3,\phi_2) & 0 & 0 \\0 & 0 & a(\phi_2,\phi_3) & a(\phi_3,\phi_3) & a(\phi_4,\phi_3) & 0 \\0 & 0 & 0 & a(\phi_3,\phi_4) & a(\phi_4,\phi_4) & a(\phi_5,\phi_4) \\0 & 0 & 0 & 0 & 0 &1\end{array}\right]\]
and
\[K = \left[\begin{array}{c}k_0 \\k_1 \\k_2 \\k_3 \\k_4 \\k_5\end{array}\right] , \quad\Phi =  \left[\begin{array}{c}\alpha \\l(\phi_1) \\l(\phi_2) \\l(\phi_3) \\l(\phi_4) \\\beta\end{array}\right].\]

Note that $a(\phi_i,\phi_j) = 0$ for most values of $i, j$ (that is, when the hat functions do not have overlapping domains).
Thus the finite element method results in a sparse linear system.
To compute the coefficients of \eqref{eqn:FEM_linear_system} we begin by evaluating some integrals.
Since
\[\phi_i'(x) = \begin{cases}
1/h_i \quad \quad \quad \, \text{for } x_{i-1} < x < x_i,\\
 -1/h_{i+1} \quad \text{ for } x_{i} < x < x_{i+1},\\
0 \quad \quad \quad \quad \, \text{ otherwise},
\end{cases}\]
we obtain
\begin{align*}
\int_0^1  \phi_i'\phi_j' &= \begin{cases}
- 1/h_{i+1} \quad \quad \quad \text{ if } j=i+1,\\
1/h_i + 1/h_{i+1} \quad \text{if } j=i,\\
0 \quad \quad \quad \quad \quad \quad \, \text{ otherwise},
\end{cases} \\
\int_0^1  \phi_i'\phi_j &= \begin{cases}
- 1/2 \quad \,\text{ if } j=i+1,\\
1/2 \quad \quad \text{ if } j=i-1,\\
0 \quad \quad \quad \text{ otherwise},
\end{cases} \\
a(\phi_i,\phi_j) &= \begin{cases}
\epsilon/h_{i+1} + 1/2 \quad \quad \, \text{ if } j=i+1,\\
-\epsilon/h_i -\epsilon/h_{i+1} \quad  \text{ if } j=i,\\
\epsilon/h_i - 1/2 \quad \quad \quad \, \text{ if } j=i-1,\\
0 \quad \quad \quad \quad \quad \quad \,\,\,\,\,\,\, \text{ otherwise},
\end{cases}\\
l(\phi_j) &= -(1/2)(h_j + h_{j+1}).
\end{align*}
\eqref{eqn:FEM_linear_system} may now be solved using any standard linear solver.

In this case, in order to handle the large number of elements required for Problem \ref{prob:FEM_accuracy_comparison}, you will want to use the tridiagonal algorithm provided in several of the earlier labs (for example Lab \ref{lab:finitedifference1}), or the banded matrix solver included in \li{scipy.linalg}.

\begin{problem}
Use the finite element method to solve
\begin{align}
	\begin{split}
	&{ }\epsilon y'' - y' = -1,\\
	&{ }y(0) = \alpha, \quad y(1) = \beta,
	\end{split} \label{eqn:FEM_exercise}
\end{align}
where $\alpha = 2, \beta = 4$, and $\epsilon = 0.02$.
Use $N = 100$ finite elements ($101$ grid points).
Compare your solution with the analytic solution
\[y(x) = \alpha + x + (\beta - \alpha - 1 ) \frac{e^{x/\epsilon} -1}{e^{1/\epsilon} -1}.\]

\eqref{eqn:FEM_exercise} is a singularly perturbed ODE, so-named because the parameter $\epsilon$ is a coefficient of the highest order derivative in the equation.
The character of the problem changes dramatically when $\epsilon = 0$: since the limit equation (as $\epsilon \to 0$) is first-order, it only allows for one boundary condition.
Thus as $\epsilon$ gets smaller, the rightmost boundary condition is satisfied at the `last moment',  and cannot be satisfied when $\epsilon = 0$.
\end{problem}

\begin{problem}
One of the strengths of the finite element method is the ability to generate grids that better suit the problem.
In two dimensions the finite elements are quadrilaterals and triangles, and can be used to approximate irregular domains.
The finite element method can also be used to solve PDEs where the shape and size of the domain changes over time.

The solution of \eqref{eqn:FEM_exercise} changes most rapidly near $x = 1$.
Compare the numerical solution when the grid points are unevenly spaced versus when the grid points are clustered in the area of greatest change. Specifically, use the grid points defined by
\begin{lstlisting}
even_grid = np.linspace(0,1,6)
clustered_grid = np.linspace(0,1,6)**(1./8)
\end{lstlisting}
What is the difference in accuracy?
\end{problem}

% \begin{figure}[ht]
% \centering
% \includegraphics[width=\textwidth]{FEM_singular_solution.pdf}
% \caption{The analytic solution of \eqref{FEM:exercise}.}
% \label{FEM:analytic_solution}
% \end{figure}

\begin{problem}
\label{prob:FEM_accuracy_comparison}
Higher order methods promise faster convergence, but typically require more work to code.
So why do we use them when a low order method will converge just as well, albeit with more grid points?
The answer concerns the roundoff error associated with floating point arithmetic.
Low order methods generally require more floating point operations, so roundoff error has a much greater effect.

The finite element method introduced here is a second order method, even though the approximate solution is piecewise linear.
(To see this, note that if the grid points are evenly spaced, the matrix $A$ in \eqref{eqn:FEM_linear_system} is exactly the same as the matrix for the second order centered finite difference method.)

Solve \eqref{eqn:FEM_exercise} with the finite element method using $N = 2^i$  finite elements, $i = 4, 5, \ldots, 21$.
Use a log-log plot to graph the error.
Then find the error when using the pseudospectral method and the same number of grid points.
What do you see?
\end{problem}

% \begin{align}
% \left[\begin{array}{cccccc}1 & 0 & 0 & 0 & 0 & 0 \\-\epsilon/h_1 & \epsilon/h_1 + \epsilon/h_2 & -\epsilon/h_2 & 0 & 0 & 0 \\0 & -1/h_2 & a(\phi_2,\phi_2) & a(\phi_3,\phi_2) & 0 & 0 \\0 & 0 & a(\phi_2,\phi_3) & a(\phi_3,\phi_3) & a(\phi_4,\phi_3) & 0 \\0 & 0 & 0 & a(\phi_3,\phi_4) & a(\phi_4,\phi_4) & a(\phi_5,\phi_4) \\0 & 0 & 0 & 0 & 0 &1\end{array}\right]
% \left[\begin{array}{c}k_0 \\k_1 \\k_2 \\k_3 \\k_4 \\k_5\end{array}\right] = \left[\begin{array}{c}\alpha \\l(\phi_1) \\l(\phi_2) \\l(\phi_3) \\l(\phi_4) \\\beta\end{array}\right] \label{FE:linear_system}
% \end{align}

\section*{A Comparison of Numerical Methods}

\begin{table}
  \begin{tabular}{ l |l l }
    % \hline
     & Finite Element & Finite Difference  \\ \hline
    Linear System& sparse& sparse  \\
   Derivative & approximated locally & approximated locally \\
Domain & irregular domains & fairly regular\\
% Problem Formulation & integral & derivative & derivative & integral \\
Convergence & polynomial & polynomial \\
Strengths & adaptive mesh & easier to understand \\
& refinement & and implement \\
& complex geometries & easier for higher dimensions \\
    \hline
  \end{tabular}
\end{table}

\begin{table}
  \begin{tabular}{ l |l l }
    % \hline
      & Pseudospectral & Finite Volume \\ \hline
    Linear System & dense & sparse \\
   Derivative & global & local\\
Domain  & very nice & fairly regular\\
Convergence  & exponential & polynomial\\
Strengths  & fast convergence & handling discontinuities in \\
& accurate to high precisions &the initial conditions \\
& & dealing with shock formation \\
& & and propagation \\
    \hline
  \end{tabular}
\end{table}

It is important to note that these methods are often mixed, so, for example, it is common to work with a finite element mesh in space and a finite differencing scheme in time.


\documentclass[11pt]{article}

%% WRY has commented out some unused packages %%
%% If needed, activate these by uncommenting
\usepackage{geometry}                % See geometry.pdf to learn the layout options. There are lots.
%\geometry{letterpaper}                   % ... or a4paper or a5paper or ...
\geometry{a4paper,left=2.5cm,right=2.5cm,top=2.5cm,bottom=2.5cm}
%\geometry{landscape}                % Activate for rotated page geometry
%\usepackage[parfill]{parskip}    % Activate to begin paragraphs with an empty line rather than an indent

%for figures
%\usepackage{graphicx}

\usepackage{color}
\definecolor{mygreen}{RGB}{28,172,0} % color values Red, Green, Blue
\definecolor{mylilas}{RGB}{170,55,241}
%% for graphics this one is also OK:
\usepackage{epsfig}

%% AMS mathsymbols are enabled with
\usepackage{amssymb,amsmath}

%% more options in enumerate
\usepackage{enumerate}
\usepackage{enumitem}

%% insert code
\usepackage{listings}

\usepackage[utf8]{inputenc}

\usepackage{hyperref}

%% colors
\usepackage{graphicx,xcolor,lipsum}

\usepackage{mathtools}

\usepackage{graphicx}
\newcommand*{\matminus}{%
  \leavevmode
  \hphantom{0}%
  \llap{%
    \settowidth{\dimen0 }{$0$}%
    \resizebox{1.1\dimen0 }{\height}{$-$}%
  }%
}

\title{Notes on spectral estimation, etc.}
\author{
CR \thanks{Scripps Institution of Oceanography,
University of California at San Diego, La Jolla, CA
92093--0230, USA. email:crocha@ucsd.edu. }
}
\date{\today}

\begin{document}

\include{mysymbols}

\maketitle

\section{Introduction}

The Fast Fourier transform (FFT) algorithm implemented in Python defines the one-dimensional discrete Fourier transform (DFT) as

\beq
    \label{eq:dft_dfn}
    \hat{A}[m] =  \sum_{n=0}^{\nmax-1}
    A[n]\exp \left(-2\pi i {n \,m\over \nmax}\right)\com
   \qquad m = 0, \ldots, \nmax-1\com
\eeq
where $A_n$ is a list of size $N$ that contains the data in physical domain with uniform spacing
\beq
\label{eq:x_spacing}
dx = {L \over \nmax}\com\qqand x_n = n \dd x\com
\eeq
where $L$ the length of the domain. Notice that because of the periodicity of the complex exponentials, $\hat{A}_k$ is also periodic with period $\nmax$. That is $\hat{A}[\nmax] = \hat{A}[0]$, $\hat{A}[\nmax+1] = \hat{A}[1]$, etc. If $A_n$ is real-valued, then $\hat{A}_k$ is Hermitian-symmetric about $\nmax/2$. That is, $\hat{A}[1] = \hat{A}^\star[\nmax-1]$, $\hat{A}[2] = \hat{A}^\star[\nmax-2]$, etc, where the superscript $\star$ denote complex conjugation. We can therefore shift the summation in \eqref{eq:dft_dfn} by any integer. A popular choice is to sum from $-\nmax/2$ through $\nmax/2-1$. The zeroth coefficient $\hat{A}[0]$ is a special case. Before we discuss it, let's introduce a slightly better notation. With the spectral resolution $dk$, we can define the wavenumber in \eqref{eq:dft_dfn}
\beq
\label{eq:sepc_resol}
k_m \defn m \dd k\com\qqand \dd k =\frac{1}{L} = \frac{1}{N \dd x}\per
\eeq
Hence we can rewrite  the DFT in \eqref{eq:dft_dfn} as
\beq
\label{eq:dft_dfn_2}
    \hat{A}[m] =  \sum_{n=0}^{\nmax-1}
    A[n]\ee^{-2\pi i k_m\, x}\com
   \qquad m = 0, \ldots, N-1\per
\eeq
With $m=0$ we have the zeroth Fourier coefficient
\beq
\label{eq:zeroth_fc}
\hat{A}[0] = \sum_{n=0}^{N-1} A[n]\per
\eeq
It is convenient to normalize the Fourier coefficients in \eqref{eq:dft_dfn} by  $\nmax$  so that zeroth Fourier coefficient represents the arithmetic  average of the elements of $A$. This normalization also makes the DFT defined in \eqref{eq:dft_dfn}  analogous to the continuous Fourier transform (FT) if we recognize the summation in \eqref{eq:dft_dfn}  times $dx$ as Riemann integral. Some people like \textit{obtain} DFT as a discretization of the FT. This is sometime advantageous, particularly when we are comparing data against theoretical predictions. I personally do not like that approach. I prefer to \textit{define} the DFT as \eqref{eq:dft_dfn} and show that it has analog properties to the FT; all proofs are self-consistent using discrete mathematics.

This is only a matter of normalization. Mathematically, what really matters is to define the inverse transform accordingly. The inverse DFT (iDFT) consistent with \eqref{eq:dft_dfn} is
\beq
    \label{eq:idft_dfn}
    A[n] =  \frac{1}{\nmax}\sum_{m=0}^{\nmax-1}
    \hat{A}[m]\exp \left(2\pi i {m n\over \nmax}\right)\com
   \qquad n = 0, \ldots, \nmax-1\per
\eeq
This is the iDFT implemented in Python. If we normalize \eqref{eq:dft_dfn} by $\nmax$, then we must multiply \eqref{eq:idft_dfn} by $\nmax$. Notice that $\hat{A}$  has the same units of $A$.

\section{The spectrum}
The Fourier coefficients $\hat{A}$ are complex-valued. Typically we are more interested in the relative magnitude of those coefficients. Hence we define the spectrum as the square of the absolute value of $\hat{A}$
\beq
\label{eq:spec_defn}
\hat{S}[m] \defn |A[m]|^2 = \hat{A}[m]\hat{A}^\star[m]\per
\eeq
It can be shown that $S[m]$ are the Fourier coefficients of the auto-correlation function of $A$.

\section{Parseval's theorem}
This important theorem states that
\beq
\label{eq:parseval}
\sum_{n=0}^{\nmax-1} |A[n]|^2 = \frac{1}{\nmax}\sum_{m=0}^{\nmax-1} |\hat{A}[n]|^2 = \frac{1}{\nmax}\sum_{m=0}^{\nmax-1} \hat{S}[n] \per
\eeq
This theorem is sometime quoted as ``the variance in physical space is equal to the variance in Fourier space''. For this statement to be true, we must normalize the above expression by $\nmax$
\beq
\frac{1}{\nmax}\sum_{n=0}^{\nmax-1} |A[n]|^2 = \frac{1}{\nmax^2}\sum_{m=0}^{\nmax-1} |\hat{A}[n]|^2 = \frac{1}{\nmax^2}\sum_{m=0}^{\nmax-1} \hat{S}[n] \per
\eeq
Notice that we would not have this normalization if we defined the Fourier coefficients as $\eqref{eq:dft_defn}$ normalized by $\nmax$. To be more precise, we would note include the average (zeroth coefficient). In practice, when dealing with data, we typically remove the average before applying the DFT, so that $\hat{A}[0]$ is zero within machine precision. It is sometimes useful to think of the area under spectrum $\hat{S}$ as the total variance. An estimate to this area is
\beq
\text{Area} \approx \frac{1}{\nmax^2}\sum_{m=0}^{\nmax-1} \hat{S}[n] \dd k\per
\eeq
To ensure this property while  satisfying  Parseval's relation \eqref{eq:parseval} we normalize $\hat{S}$ by $\dd k$. Thus the spectrum that we typically plot in a log$\times$log space is the square of the absolute value of the Fourier coefficients divided by $\dd k\,\nmax^2$, and it has units of $A^2/k$. Notice that if $dx = 1$, then $dk = 1/\nmax$ and we just need to normalize by $\nmax$. Another nice property of the spectrum normalized by $\dd k$ is that it is independent of the spectral resolution $\dd k$, and therefore it is useful for comparing spectra calculated from data with different sampling characteristics. Furthermore, because typically we deal with real signals, $\hat{S}$ is symmetric since $\hat{A}$ is Hermitian-symmetric as discussed above.

\section{A note on averaging many estimates}
A spectrum computed from a single realization is useless because the error in this estimate is of the same order as the spectrum (see Bendat and Piersol). Thus we average many realizations to obtain a meaningful estimate. I think it is best to normalize the spectrum by $\nmax^2$ only after averaging. If $\nmax$ is large and $\hat{A}$ is small, we can loose accuracy by normalizing single estimates.

\end{document}


\chapter{Preface}

This book is a companion to \emph{Quantum Mechanics in Simple Matrix Form} \cite{Jordan2005} by Thomas Jordan. Quantum mechanics was developed using to different, but equivalent mathematical formalisms: wave mechanics and matrix mechanics. Today many introductions to the subject follow the approach invented by Erwin Schr\"odinger and begin with waves, which requires knowledge of partial differential equiations. Using Jordan's book as a guide, we're going to follow in the footsteps of Werner Heisenberg and Max Born and develop quantum mechanics using matrix algebra.

Matrix algebra is a nice way to learn quantum mechanics for a few reasons. Firstly, it requires a lower entry to barrier on the mathematical front. The arithmetic of matrices builds on addition and multiplication of real numbers. Secondly, matrix mechanics generalizes in a particularly convenient way. Unlike real numbers, matrix products do not always commute \sidenote{Remember this fact. Commutation is one of the great big ideas in quantum mechanics.}. That is, given any two real numbers $x$ and $y$, we can multiply $x$ by $y$ or multiply $y$ by $x$. Either way, we get the same result.

$$ x\cdot y = y \cdot x $$

Given two matrices $A$ and $B$, the product $A$ applied to $B$ is not generally the same as the product $B$ applied to $A$.

$$ AB \neq BA $$

You'll notice that I wrote \emph{applied to} instead of \emph{multiplied by}, when I talked about the product of the matrices $A$ and $B$. Matrices are examples of \emph{linear operators}. Over time we will come to think of linear operators as representing observable quantites in laboratory experiments such as position, velocity, or momentum.\sidenote{You can squirrel this idea away for now, too. Linear operators represent observable quantities. I promise it will make sense later. It won't now. But we haven't even started yet.} But for now, you can think of them as a generalization of real numbers in which the order of operations matters. The operator formalism of quantum mechanics puts the commutation of observable quantities front and center. And to study quantum field theory, we'll need to graduate from state vectors to operators. Matrix mechanics will get us ready.

While Jordan wrote his book to be accessible to students with little mathematical background, it moves pretty quickly. He requires a great deal of what people often call ``mathematical maturity.''\sidenote{Said another way, there's very little that is simple in the book, even though it's written in ``simple matrix form.''} This companion text aims to fill in some of the gaps that Jordan must step over. Along the way, we'll explore some of the mathematical structures associated with the physics that Jordan doesn't have the time to introduce himself. He used his book to teach a single-semester course, and it's already packed with more than enough to fill a full semester.

I have included new exercises to flesh out some of the concepts that Jordan covers. Many of these problems are meant to provide extra practice to develop intuition with the material. Others take us too far afield ever to have been included in the original text. Exercises that appear inline with the main text serve as a means to check your understanding as you go along. Those at the end of the chapters will advance your understanding further and are meant to take you longer to complete.

Solutions to all of the problems in this book and in \emph{Quantum Mechanics in Simple Matrix Form} appear in the second part of this book. But don't peek at an answer until you've given it a good college try! Problem solving is hard work. In a research laboratory there is no answer key. Oftentimes, there is no clear answer at all. So appoach the exercises as bite-sized opportunities to become a better, thinking scientist.

So without anymore ado, let's get to it.


This work makes use of probability meshes to approximate a 4-dimensional (position,velocity) space. The meshes are then used to implement efficient forward estimation, integration of new sensor data, and integration of probabilistic rules. Finally, a pipeline is constructed from these elements to produce refined paths from raw positional input.

\section{Probability Meshes}

Each trajectory of interest $t$ has an associated 4-dimensional $n*m*i*j$ probability mesh $M_t$, with each cell $M_t(n,m,i,j)$ in the mesh representing the likelihood that the most recent position and velocity in the trajectory lie in that cell.
However $n$, $m$, $i$, and $j$ are integer values while locations and velocities are continuous.
In order to account for continuous values, an interpolation scheme can approximate points between cell centers.

Interpolation works as follows: For each dimension $i$ in a query point $q=(q_n, q_m, q_i, q_j)$, calculate $\mathit{floor}(i)$ and $\mathit{ceil}(i)$. Then collect the set of adjacent points $P$ by generating all combinations of $floor$ and $ceil$ in each dimension. Interpolation is calculated over $P$ as shown in Equation \ref{eqn:interpolation}.

\begin{equation} \label{eqn:interpolation}
    \frac{\sum_{p \in P} 1/\mathit{dist}(p, q) * M_t(p_n, p_m, p_i, p_j)}{\sum_{p \in P} (1/\mathit{dist}(p, q)}
\end{equation}

With interpolation between cells, probability meshes can approximate a bounded continuous probability. Note that the division by \textit{dist} is undefined when dist is zero. In practice, that can only occur when all indices are integers, and in that case the exact point found is returned. This interpolation algorithm has a property that will be heavily utilized in the remainder of the paper: Because all probabilities are weighted by \textit{dist}, probabilities generated from a set of points are bounded by the minimum and maximum values stored in those points. Therefore, the point of maximum probability \textit{must be an exact grid cell}.

However, the grid indices have no mapping to real-world positions and velocities. To provide mappings, boundaries and a mapping function must be specified. Given an input position and velocity $(p_x, p_y, v_x, v_y)$ and boundaries $x_{min}, x_{max}, y_{min}, y_{max}, v_{min}, v_{max}$, the appropriate grid point is calculated by Equation \ref{eqn:normalize}.

\begin{equation} \label{eqn:normalize}
    \begin{aligned}
        map(p_x, p_y, v_x, v_y) = &\\
        (&\frac{(p_x - x_{min})*n}{x_{max}-x_{min}},\\
        &\frac{(p_y - y_{min})*m}{y_{max}-y_{min}}, \\
        &\frac{(v_x - v_{min})*i}{v_{max}-v_{min}},\\
        &\frac{(v_y - v_{min})*j}{v_{max}-v_{min}})
    \end{aligned}
\end{equation}

With the inclusion of the mapping function and boundaries, a probability mesh can approximate a continuous probability over arbitrary boundaries. In addition, the mapping function can be reversed to $map_{reverse}$, calculating the corresponding point in the probability space for a particular cell in the mesh.

\subsection{Forward Propagation}

Because these probability meshes capture both position and velocity, future probability spaces can be estimated. Equation \ref{eqn:timeprop} shows how a zeroed mesh $M_t'$ can be calculated from a past mesh $M_t$.

\begin{equation} \label{eqn:timeprop}
    M_t'(n+(t'-t)i,m+(t'-t)j,i,j) += M_t(n,m,i,j)
\end{equation}

$M_t'$ represents an accurate estimate, assuming unchanging velocities. However in the real world objects can change velocity over time. To accomodate this, a blur factor can be added by averaging all adjacent points using a stencil algorithm.

\section{Probabilistic Rules}

Often in addition to sensor input, we have domain knowledge about objects whose positions and velocities are being tracked. Such rules can often be colloquially expressed as "Cars tend to drive on roads", or "Trucks and busses rarely make U-turns". Under a traditional path refinement system such as a Kalman Filter, these rules cannot be expressed, as they represent non-gaussian probabilities. However, using probability meshes, these rules can be approximated and used to refine any sensor inputs or estimates. In order for a rule to help refine paths, it need not always be true. The examples above discuss things that usually occur, which can be formally defined as a probability. By phrasing rules as probability functions in the same 4-dimensional position, velocity space as the meshes defined above, such probabilistic rules can be expressed.

A rule $R$ can be intersected with a mesh $M$ by applying the pointwise update equation -- Equation \ref{eqn:ruleupdate} -- to all cells in the mesh, producing a new mesh $M'$. Multiplication allows the probabilities to be combined to generate the probability of both the original mesh and the rule being true for each cell.

\begin{equation} \label{eqn:ruleupdate}
    M'(n, m, i, j) = M(n, m, i, j) * R(map_{reverse}(n,m,i,j))
\end{equation}

\subsection{Road Matching}

As an example of probabilistic rules, this work implements the rule \textit{"vehicles usually drive on roads"}. The exact equation for this rule is in Equation \ref{eqn:roadprob}.

\begin{equation} \label{eqn:roadprob}
    \begin{gathered}
    R_{road}(p_x, p_y, v_x, v_y) = \\
        \textit{max}(\alpha, \textit{CNDF}(\textit{distanceToRoad}(p_x, p_y)/\beta))
    \end{gathered}
\end{equation}

$\alpha$ allows for this rule being incorrect, by setting a lower bound on the probability of any cell, regardless of roadways. $\beta$ allows the allowed distance from the roadway to be scaled. Here CNDF refers to the cumulative normal distribution function, which returns a value between 0.5 and 0 for positive inputs.

The \textit{distanceToRoad} function requires additional explanation.
Road map data for the city of Edmonton was aquired from OpenStreetMap, and roadways were extracted. Then, to reduce the number of roadways inspected for a particular calculation of this rule, a uniform grid index is overlaid on the area of interest, and roads are placed into each grid cell overlapped by their minimum bounding rectangle~(MBR). Grid cells should be of a width substantially larger than $\beta$ above, so that a minimum of cells need to be inspected.

When calculating \textit{distanceToRoad} for a given point, all roads in the corresponding index cell are collected, as well as all roads in adjacent cells to account for query points near the edge of a cell. Then, the distance from each segment of each road to the query point is calculated. Finally, the minimum distance found is returned. An example of the probability space rendered from this work is shown in \figref{fig:ex:osm}.

\input{figures/fig-ex-osm}

\section{Grid-Based Trajectory Refinement}

From probability meshes and probabilistic rules, a pipeline can be assembled to refine trajectories in real-time. Consider a set of objects being tracked, $O$.
Each object $o \in O$ has an associated probability mesh $M(o)$, valid for a particular timestamp in the past $updateTime(o)$. All probability meshes are initialized to an equal uniform distribution.
As each sensor update $(o, \mathit{time}, p_x, p_y)$ comes in, meshes are updated according to the following pipeline:

\begin{enumerate}

\item The object mesh is propagated forward in time according to Equation \ref{eqn:timeprop}, using the time difference $\textit{time} - \textit{updateTime}(o)$.

\item A gaussian probability $\textit{GPS}$ is constructed around $p_x, p_y$, constructed to match the error distribution that 95\% of sensor readings are within 2 meters of accurate. The GPS probability is integrated into $M(o)$ according to Equation \ref{eqn:ruleupdate}.

\item Any applicable rules can be integrated, again using Equation \ref{eqn:ruleupdate}.

\item The new most likely position is found by finding the maximum cell in mesh $M(o)$ and reverse mapping the cell into the real-world position.

\end{enumerate}


\documentclass[11pt,twoside]{article}

\usepackage[headings]{fullpage}
\usepackage[utopia]{mathdesign}

\pagestyle{myheadings}
\markboth{Nonsmooth FD}{Nonsmooth FD}

\input{../../fncextra}

\begin{document}

\begin{center}
  \bf (Non)smooth moves
\end{center}

We derive finite difference formulas by interpolating with a polynomial, then differentiating the interpolant. This leads to, for example, the formulas
\begin{align}
   W_1(h) &=  \frac{f(h)-f(0)}{h} = f'(0) + O(h), \label{W1} \\
   W_2(h) &=  \frac{f(h)-f(-h)}{2h} = f'(0) + O(h^2), \label{W2} \\
   W_4(h) &=  \frac{-f(2h) + 8f(h)-8f(-h)+f(-2h)}{12h} = f'(0) + O(h^4). \label{W4}
\end{align}
The terms $O(h^p)$ are a statement about the accuracy of the formula, which we say has order $p$.

However, interpolation by a polynomial isn't ideal for every $f$ you might encounter. One reason is that polynomials are smooth---they have infinitely many derivatives. Consider $f(x)=|x|$, for example. It's continuous but not differentiable at $x=0$. What value should the finite difference attempt to get there: $+1$, $-1$, or something else? For a function that has one derivative but not two, there is a unique value to converge to, but the convergence rate is slowed by the singularity.

\subsection*{Goals}

You will apply finite difference methods of different orders to
functions that lack smoothness at a point, and observe the effects on
the order of accuracy.

\subsection*{Preparation}

Read sections 5.4 and 5.5. Determine the number of continuous derivatives at $x=0$ for the following functions:
\begin{align}
  g_1(x) =
  \begin{cases}
    5x+1, & x \le 0 \\ (x+1)^5, & x > 0
  \end{cases}  \label{g1} \\
  g_2(x) =
  \begin{cases}
    10x^2 + 5\sin(x) + 1, & x \le 0 \\ (x+1)^5, & x>0
  \end{cases} \label{g2}
\end{align}

\subsection*{Procedure}

Download the script template and complete it to perform the following tasks, answering all questions in the text of your script.

\begin{enumerate}
\item Define $f(x)=\exp(\sin(x+1))$. This is our ``control'' case of a function with infinitely many derivatives everywhere. For each value of $h=2^{-1},2^{-2},2^{-3},\ldots,2^{-10}$, compute $W_1(h)$, $W_2(h)$, and $W_4(h)$, storing the results in vectors. Then make a log-log plot showing all three cases of $|W_i(h)-f'(0)|$ as functions of $h$. You should see three straight lines with different slopes.

\item To the graph from step 1, add plots of the functions $h^1$, $h^2$, $h^3$, and $h^4$, as dashed lines. (Use labels, legends, etc.)  Confirm that each solid convergence curve matches well with the dashed line corresponding to the order of accuracy in the presentation of the formulas above.

\item Now let $f$ be $g_1$ from~\eqref{g1}, by defining
\begin{verbatim}
f = @(x) (5*x+1).*(x<=0) + ((x+1).^5).*(x>0);
\end{verbatim}
(This defines the function piecewise for negative and positive $x$.) Make a plot of the function over $[-1/4,1/4]$.

\item Repeat steps 1--2 for the function in step 3. Now what are the observed orders of accuracy for $W_1$, $W_2$, and $W_4$?

\item Repeat steps 1--2 for the function $g_2$ in~\eqref{g2}.

\end{enumerate}

\end{document}


\section{Polar form}

\begin{outcome}
  \begin{enumerate}
  \item Convert a complex number from standard form to polar form, and
    from polar form to standard form.
  \end{enumerate}
\end{outcome}

In the previous section, we identified a complex number $z=a+bi$ with
a point $(a, b)$ in the coordinate plane. There is
another form in which we can express the same number, called the {\em
polar form}. The polar form is the focus of this section. It will turn out to be
very useful if not crucial for certain calculations as we shall soon
see.

Suppose $z=a+bi$ is a complex number,  and let
$r=\sqrt{a^{2}+b^{2}} = |z|$. Recall that $r$ is the \textbf{modulus}\index{complex numbers!modulus}\index{modulus} of $z$. Note first that
\begin{equation*}
\paren{\frac{a}{r}} ^{2}+\paren{\frac{b}{r}} ^{2}=   \frac{a^2+b^2}{r^2}=1
\end{equation*}
and so $\paren{\frac{a}{r},\frac{b}{r}}$
is a point on the unit circle. Therefore, there exists an angle  $
\theta$ (in radians) such that
\begin{equation*}
\cos \theta =\frac{a}{r},\ \sin \theta =\frac{b}{r}
\end{equation*}
In other words $\theta$ is an angle such
that $ a = r\cos \theta$ and $b=r \sin \theta$, that is $\theta = \cos^{-1}(a/r)$ and $\theta = \sin^{-1}(b/r)$. We call
this angle $\theta$ the \textbf{argument}\index{complex numbers!argument}\index{argument} of $z$.

We often speak of the \textbf{principal argument}\index{principal argument} of $z$. This is the unique angle $\theta \in (-\pi, \pi]$ such that
\begin{equation*}
\cos \theta =\frac{a}{r},\ \sin \theta =\frac{b}{r}
\end{equation*}

The polar form of the complex number $z=a+bi = r (\cos \theta +i\sin \theta)$ is for convenience written as:
\begin{equation*}
z = r e^{i \theta}
\end{equation*}
where $\theta $ is the argument of
$z$.

\begin{definition}{Polar form of a complex number}{polar-form}
Let $z = a + bi$ be a complex number. Then the \textbf{polar form}\index{complex numbers!polar form}\index{polar form} of $z$ is written as
\[
z = re^{i\theta}
\]
where $r = \sqrt{a^2 + b^2}$ and $\theta$ is the argument of $z$.
\end{definition}

When given $z = re^{i\theta}$, the identity $e^{i\theta} = \cos\theta + i \sin\theta$ will convert $z$ back to standard form. Here we think of $ e^{i \theta}$ as a short cut for $ \cos \theta
+i\sin \theta$. This is all we will need in this course, but in
reality $e^{i \theta}$ can be considered as the complex equivalent of
the exponential function where this turns out to be a true equality.

\begin{center}
\begin{tikzpicture}
\draw(-2,0)--(2,0);
\draw(0,-2)--(0,2);
\draw[ultra thick, blue, ->](0,0)--(1.5,1.5);
\node[right] at (1.5,1.5){$z = a+bi = re^{i\theta}$};
\node[above right] at (0.5,0){$\theta$};
\node[left] at (1,1){$r$};
\node at (-1.5,1){$r = \sqrt{a^2 + b^2}$};
\end{tikzpicture}
\end{center}

Thus we can convert any complex number in the standard (Cartesian) form $z = a+bi$
into its polar form. Consider the following example.

\begin{example}{Standard to polar form}{polar-form}
Let $z = 2 + 2i$ be a complex number.
Write $z$ in the polar form
\begin{equation*}
z = re^{i \theta}
\end{equation*}
\end{example}

\begin{solution}
First, find $r$.
By the above discussion, $r=\sqrt{
a^{2}+b^{2}} = |z|$. Therefore,

\begin{equation*}
r = \sqrt{2^{2} + 2^{2}} = \sqrt{8} =2\sqrt{2}
\end{equation*}

Now, to find $\theta$, we plot the point $(2, 2)$ and
find the angle from the positive $x$ axis to the line between this
point and the origin. In this case, $\theta = 45^{\circ} =
\frac{\pi}{4}$.  That is we found the unique angle $\theta$ such that
$\theta = \cos^{-1}(1/\sqrt{2})$ and $\theta = \sin^{-1}(1/\sqrt{2})$.

Note that in polar form, we always express angles in radians, not degrees.

Hence, we can write $z$ as

\begin{equation*}
z = 2\sqrt{2} e^{i\frac{\pi}{4}}
\end{equation*}

\end{solution}

Notice that the standard and polar forms are completely equivalent. That is not only can we transform a complex number from standard form
to its polar form, we can also take a complex number in polar form and
convert it back to standard form.

\begin{example}{Polar to standard form}{polar-standard-form}
Let $z = 2 e^{ 2\pi i/3}$. Write $z$ in the standard form
\begin{equation*}
z = a+bi
\end{equation*}
\end{example}

\begin{solution}
Let $z = 2 e^{2\pi i/3}$ be the polar form of a complex number. Recall that
$e^{i\theta} = \cos \theta + i \sin \theta$. Therefore using standard values of $\sin$ and $\cos$ we get:
\begin{eqnarray*}
z = 2 e^{i 2\pi/3} &=& 2 (\cos (2\pi/3)+i\sin (2\pi/3))\\
&=& 2 \paren{-\frac{1}{2} + i \frac{\sqrt{3}}{2}} \\
&=&-1 + \sqrt{3}i
\end{eqnarray*}
which is the standard form of this complex number.
\end{solution}

You can always verify your answer by converting it back to polar form and ensuring you reach the original answer.