% !TEX root = thesis.tex
\documentclass[thesis]{subfiles}

\begin{document}
%*******************************************************************************
%*********************************** Conclusion *****************************
%*******************************************************************************

\chapter{Conclusion and Future Work}
\label{conclusion}
%********************************** %First Section  **************************************
In this dissertation, we have proposed that carefully designing networks in consideration of our prior knowledge of the task can improve the memory and computational efficiency of state-of-the art networks, and even increase accuracy through structurally induced regularization. While this philosophy defines our approach, deep neural networks\index{neural network} have a large number of degrees of freedom, and there are many facets of deep neural networks\index{neural network} that warrant such analysis. We have attempted to present each of these in isolation:

\Cref{lowrankfilters} proposed to exploit our knowledge of the low-rank nature of most filters learned for natural images by structuring a deep network to learn a collection of mostly small 1$\times$h and w$\times$1 basis filters, while only learning a few full w$\times$h filters. Our results showed similar or higher accuracy than conventional \glspl{cnn}\index{CNN} requiring much less computation. Applying our method to an improved version of VGG-11 network using \gls{gmp}, we achieve comparable validation accuracy using 41\% less computation and only 24\% of the original VGG-11 model parameters; another variant of our method gives a 1 percentage point {\em increase} in accuracy over our improved VGG-11 model, giving a top-5 \emph{center-crop} validation accuracy of 89.7\% while reducing computation by 16\% relative to the original VGG-11 model. Applying our method to the GoogLeNet architecture for \gls{ilsvrc}, we achieved comparable accuracy with 26\% less computation and 41\% fewer model parameters. Applying our method to a near state-of-the-art network for \gls{cifar10}, we achieved comparable accuracy with 46\% less computation and 55\% fewer parameters.

\Cref{deeproots} addresses the filter/channel extents of convolutional filters, by learning filters with limited channel extents. When followed by a 1$\times$1 convolution, these can also be interpreted as learning a set of basis filters, but in the channel extents.
Unlike in \cref{lowrankfilters}, the size of these channel-wise basis filters increased with the depth of the model, giving a novel sparse connection structure that resembles a tree root. This allows a significant reduction in computational cost and number of parameters of state-of-the-art deep \glspl{cnn}\index{CNN} without compromising accuracy. Our results showed similar or higher accuracy than the baseline architectures with much less computation, as measured by \gls{cpu} and \gls{gpu} timings. For example, for \gls{resnet}\index{ResNet} 50, our model has 40\% fewer parameters, 45\% fewer floating point operations, and is 31\% (12\%) faster on a \gls{cpu} (\gls{gpu}). For the deeper \gls{resnet} 200 our model has 25\% fewer floating point operations and 44\% fewer parameters, while maintaining state-of-the-art accuracy. For GoogLeNet, our model has 7\% fewer parameters and is 21\% (16\%) faster on a \gls{cpu} (\gls{gpu}).

\Cref{lowrankfilters,deeproots} proposed similar methods for reducing the computation and number of parameters in the spatial and channel (filter-wise) extents of convolutional filters respectively. Rather than approximating filters in previously-trained networks with more efficient versions, we learn a set of smaller basis filters from scratch; during training, the network learns to combine these basis filters into more complex filters that are discriminative for image classification. This means that at both training and test time our models are more efficient. Overall, the approach of learning a set of basis filters was not only effective for reducing both computation and model complexity (parameters), but in many of the results in both \cref{lowrankfilters,deeproots}, the models trained with this approach generalized better than the original state-of-the-art models they were based on.

\Cref{conditionalnetworks} presented work towards conditional computation in deep neural networks\index{neural network}. We proposed a new discriminative learning model, \emph{conditional networks},
that jointly exploits the accurate \emph{representation learning} capabilities of deep neural networks\index{neural network} with the efficient \emph{conditional computation} of decision trees and directed acyclic graphs (DAGs). In addition to allowing for faster inference, conditional networks yield smaller models, and offer test-time flexibility in the trade-off of computation \vs accuracy.

%*******************************************************************************
%*********************************** Future Work *****************************
%*******************************************************************************

\section{Future Work}  %Title of the First Chapter
\label{futurework}

%********************************** %First Section  **************************************

Research outcomes are often better evaluated by the questions borne rather than the questions answered. In this section we'll address the main research questions that this dissertation has highlighted, and propose future directions for research which we believe would have the most impact on the field.

\subsection{Learning Structural Priors}
The move towards ``end-to-end'' learning has made great strides in making learning more automatic, notably in learning complex representations rather than experts designing inferior representations themselves. There still exists however, a significant amount of hand design and manual tuning that is key to the success of any deep learning approach. We hope our work will motivate the field towards a research direction that aims to minimize this further, by working on methods of automatically structuring neural networks\index{neural network}, in a move towards a truly ``end-to-end'' learning of \gls{dnn} structure itself.

The lack of understanding or concrete rules for structuring \glspl{dnn}\index{DNN} means that in practical applications deep learning is often restricted to experts in the field, who have an intuition in network design formed from years of experience, and know which structural priors to use. The effect on deep learning research is no less profound, with a lack of understanding of the basic interplay between structure and learning in \glspl{dnn}\index{DNN}, we have little chance of understanding the limitations of deep learning or the representations learned by the networks.

The benefits of automatically structuring \glspl{dnn}\index{DNN} go further than these considerations even, as the research presented in this dissertation has shown, better structured \glspl{dnn}\index{DNN} are more computationally efficient (use fewer parameters and are faster to compute), and generalize better. Currently, training state-of-the-art \glspl{dnn}\index{DNN} for image classification requires a prohibitive amount of time and computational resources --- 3 weeks of training on 8 high-end and expensive \glspl{gpu} --- and yet we know that trained \glspl{dnn}\index{DNN} are very sparse representations and have been shown to be highly compressible. It is because we cannot appropriately understand this sparse structure well enough to fully exploit it that our current \glspl{dnn}\index{DNN} are so inefficient.

With automatic methods of learning the structure, \glspl{dnn}\index{DNN} will become markedly more efficient to train, leading to faster experimental results for research, and also allow easier deployment to embedded devices, such as mobile phones, drones and robots. It would also allow for research strides in learning networks for multiple modalities, for example a self-driving car needs to process input data from normal camera sensors, along with depth maps or point clouds, and even radar. One of the stumbling blocks in doing this is understanding how to best structure a network to deal with multiple inputs which require different structural priors.

Research on finding automatic methods of structuring neural networks\index{neural network} is not a completely new avenue of research, with a substantial effort put towards it 30 years ago when neural networks\index{neural network}, and datasets, were much smaller. This is covered in \cref{motivation}, but suffice to state that there were two main approaches:
\begin{enumerate*}[label= (\textbf{\roman*})]
	\item greedily building networks from scratch, and
	\item pruning (removing parameters) large networks
\end{enumerate*}. The proposals made for both building networks from scratch , such as that of \citet{Fahlman1989}, and pruning full networks, such as that of \citet{lecun1989optimal}, suffer drawbacks which make them unsuitable in the modern deep network of hundreds of millions of parameters. Even in neural networks\index{neural network} of contemporary size, the greedy approach of \citet{Fahlman1989} meant that learned networks were suboptimal. This proposal should also not be confused with `universal learning', or violating the no free lunch theorem (\cref{nofreelunch}), since we are interested in learning methods for the specific set of problems we as humans are interested in solving, rather than all possible input patterns.

At least three factors prevented this line of research from being successful historically, that we believe have now been overcome. Recent breakthroughs in training \glspl{dnn}\index{DNN} have given us a better understanding of how to train very large, arbitrarily structured networks, notably avoiding the so-called `vanishing gradient'~\citep{Ioffe2015,He2016}, and a better understanding of initialization~\citep{He2015b}. Extremely large and diverse datasets are now prevalent, such as ImageNet~\citep{ILSVRC2015}, whereas historically datasets were prohibitively small to be useful for automatically structuring \glspl{dnn}\index{DNN}. And finally computational resources have increased dramatically. In fact these are most of the reasons the field of deep learning itself has been more successful now than neural networks\index{neural network} were 30 years ago.

\subsection[Jointly Learning a Basis for Spatial and Channel Extents of Filters]{Jointly Learning a Basis for Spatial and\texorpdfstring{\\}{ }Channel Extents of Filters}
\label{journalplan}
In the shorter term, there is an obvious question arising from the work presented in \cref{lowrankfilters,deeproots} that explore learning more efficiently by reducing the learned parameters in the spatial and channel extents of convolutional filters respectively. These naturally lend themselves to being merged into a single effective method for training with low-rank basis filters. We plan to submit a journal article in which both methods are merged and explored in new results on state-of-the-art \glspl{dnn}\index{DNN}.

\subsection{Optimization and Structural Priors}\label{optimizationlink}
It is notable that many structural priors can be viewed as enforcing sparsity on fully-connected networks. For example, in the case of a \gls{cnn}, any learned \gls{cnn} is representable in a fully-connected network, since a \gls{cnn} can be viewed as a fully-connected network with a specific arrangement of zeroed connect weights, and some duplicated weights (shared weights) as illustrated in \cref{fig:sparseconn}.

The question arises then, why can we not learn these in fully-connected networks? Structural priors give lower training loss, and yet when we optimize fully-connected networks with an appropriate structure and capacity to learn the sparse structural priors, they do not. Another, more recent example, is that of \glspl{resnet}, as explained in \cref{residualnetworks}, these are motivated by the observation that in very deep networks the optimization fails to learn even the identity function, when it can be shown to give a lower loss.

In many ways the need for structural priors can be seen as the result of a problem with the current methods of optimization of \glspl{dnn}. As discussed in \cref{pathological}, higher order optimization might help solve this, but is not practical given the size of contemporary \glspl{dnn}.

\subsection{Parting Note}
In my PhD, I have focused on experiments which I believed would shed light on the representations being learned in \glspl{dnn}\index{DNN}. Although the overt motivation of much of the work in its publication has been efficiency, my personal motivation has always been to better understand the learned internal representation of state-of-the-art \glspl{dnn}\index{DNN} for image classification, and explain why they are so over-parameterized. Structural priors, such as those demonstrated in this dissertation do not only improve the effectiveness of a deep network, but are \emph{necessary} for good generalization.
\end{document}


\subsection{Derivatives of Inverse Trigonometric Functions}

We can apply the technique used to find the derivative of $f^{-1}$ above to find the derivatives of the inverse trigonometric functions.

In the following examples we will derive the formulae for the derivative of the inverse sine, inverse cosine and inverse tangent. The other three inverse trigonometric functions have been left as exercises at the end of this section.

\begin{example}{Derivative of Inverse Sine}{DerInvSine}
Find the derivative of $\sin^{-1}(x)$.
\end{example}
\begin{solution}
 Adopting the notation in Theorem \ref{thm:deriv_inverse_functions}, let $g(x) = \arcsin x$ and $f(x) = \sin x$. Thus $\fp(x) = \cos x$. Applying the theorem, we have
			\begin{align*}
			g\primeskip'(x) &= \frac{1}{\fp(g(x))} \\
						&= \frac{1}{\cos(\arcsin x)}.
			\end{align*}

Alternatively, we could use the technique in the justification of Theorem \ref{thm:deriv_inverse_functions}. Write $y=\sin^{-1}(x)$, so $x=\sin(y)$ and $-\pi/2\leq y\leq \pi/2$, and differentiate both sides with respect to $x$ using the chain rule.
\begin{align*}
\ds\frac{d}{dx}x&=\frac{d}{dx}\sin(y)	\\
1&=\cos(y)\frac{dy}{dx}	\\
1&=\cos\left(\sin^{-1}(x)\right)\frac{dy}{dx}	\\
\frac{dy}{dx}&=\frac{1}{\cos\left(\sin^{-1}(x)\right)}
\end{align*}

Although correct, this formula is cumbersome to use, and hardly enlightening. It can be simplified significantly with a bit of trigonometry. We have   $y =\sin^{-1}(x)$, so $\sin(\theta)=x$, and we wish to find $ \cos(y) $ in order to simplify our expression above. Construct a right angle triangle with angle $y$, opposite side length $x$ and hypotenuse $ 1 $.  The Pythagorean  Theorem gives an adjacent side length of $\sqrt{1-x^2}$.

\begin{figure}
\centering
\begin{tikzpicture}[thick]
\draw(0,0) -- (90:2cm) node[midway,left]{$x$} -- (0:4cm) node[midway,above right]{{$1$}} -- (0,0);

\node at (2,-.5) {{$\sqrt{1-x^2}$}};

%\draw[fill=lightgray, thick] (0,0) -- (0:0.8cm) arc (0:90:0.8cm) node at (45:0.5cm) {$\gamma$} -- cycle;
\draw[fill=lightgray, thick] (4,0) -- ++(180:0.8cm) arc (180:180-atan2(4,2):0.8cm) node at ($(167:0.6cm)+(4,0)$) {$y$} -- cycle;
%\draw[fill=lightgray, thick] (0,2) -- ++(-90:0.8cm) arc (-90:-90+atan2(2,4):0.8cm) node at ($(-60:0.5cm)+(0,2)$) {$\beta$} -- cycle;
\end{tikzpicture}

\caption{\label{fig:inverse3}A right triangle defined by $y=\sin ^{-1}(\frac{x}{1})$ with the length of the third leg found using the Pythagorean Theorem.}
\end{figure}

So, reading from the triangle we have $\cos\left(\sin^{-1}(x)\right)=\cos(y)=\sqrt{1-x^2}$. Note that we choose the non-negative square root $\sqrt{1-x^2}$ since $\cos(\theta)\geq 0$ when $-\pi/2\leq\theta\leq\pi/2$ (the range of $ \arcsin(x) $).

Finally, the derivative of inverse sine is
\[\left(\sin^{-1}(x)\right)'=\frac{1}{\sqrt{1-x^2}}\]
\end{solution}

\begin{example}{Derivative of Inverse Cosine}{DerInvCosine}
Find the derivative of $\cos^{-1}(x)$.
\end{example}
\begin{solution}
Let $y=\cos^{-1}(x)$, so $\cos(y)=x$ and $0\leq y\leq \pi$. Next we differentiate implicitly:
\begin{align*}
\frac{d}{dx}\left(\cos y\right)&=\frac{d}{dx}\left(x\right)	\\
-\sin y\cdot\frac{dy}{dx}&=1	\\
\frac{dy}{dx}&=-\frac{1}{\sin y}
\end{align*}

 Since $\cos y=x$, we construct a triangle with angle $y$, adjacent side length $x$ and hypotenuse $1$.

 \begin{figure}
 \centering
 \begin{tikzpicture}[thick]
 \draw(0,0) -- (90:2cm) node[midway,left]{$\sqrt{1-x^2}$} -- (0:4cm) node[midway,above right]{{$1$}} -- (0,0);

 \node at (2,-.5){$x$} ;

 %\draw[fill=lightgray, thick] (0,0) -- (0:0.8cm) arc (0:90:0.8cm) node at (45:0.5cm) {$\gamma$} -- cycle;
 \draw[fill=lightgray, thick] (4,0) -- ++(180:0.8cm) arc (180:180-atan2(4,2):0.8cm) node at ($(167:0.6cm)+(4,0)$) {$y$} -- cycle;
 %\draw[fill=lightgray, thick] (0,2) -- ++(-90:0.8cm) arc (-90:-90+atan2(2,4):0.8cm) node at ($(-60:0.5cm)+(0,2)$) {$\beta$} -- cycle;
 \end{tikzpicture}

 \caption{\label{fig:inverse3}A right triangle defined by $y=\cos ^{-1}(\frac{x}{1})$ with the length of the third leg found using the Pythagorean Theorem.}
 \end{figure}

 Solving for the opposite side length using Pythagorean Theorem we obtain $\sqrt{1-x^2}$. Using this triangle we can see that $\sin y=\sqrt{1-x^2}$ ($0\leq y\leq \pi$). Substituting this into the equation for $dy/dx$, we find that
\[\frac{d}{dx}\left(y\right)=\frac{d}{dx}\left(\cos^{-1}(x)\right)=\frac{-1}{\sqrt{1-x^2}}\].
\end{solution}

In the following example we explore an alternate method of finding the derivative.

\begin{example}{Derivative of Inverse Tangent}{DerInvTangent}
Find the derivative of $\tan^{-1}(x)$.
\end{example}
\begin{solution}
We begin with $\tan\left(\tan^{-1}(x)\right)=x$. Taking the derivative using the Chain Rule we obtain
\[\sec^2\left(\tan^{-1}(x)\right)\cdot\frac{d}{dx}\left(\tan^{-1}(x)\right)=1,\]
which we rearrange to obtain
\[\frac{d}{dx}\left(\tan^{-1}(x)\right)=\frac{1}{\sec^2\left(\tan^{-1}(x)\right)}.\]

Let $\tan^{-1}(x)=\theta$, then $\tan(\theta)=x$. We construct a triangle with angle $\theta$, adjacent side $1$ and opposite side $x$. The hypotenuse is $\sqrt{1+x^2}$ using Pythagorean theorem.

 \begin{figure}
 \centering
 \begin{tikzpicture}[thick]
 \draw(0,0) -- (90:2cm) node[midway,left]{$x$} -- (0:4cm) node[midway,above right]{{$\sqrt{1+x^2}$}} -- (0,0);

 \node at (2,-.5){$1$} ;

 %\draw[fill=lightgray, thick] (0,0) -- (0:0.8cm) arc (0:90:0.8cm) node at (45:0.5cm) {$\gamma$} -- cycle;
 \draw[fill=lightgray, thick] (4,0) -- ++(180:0.8cm) arc (180:180-atan2(4,2):0.8cm) node at ($(167:0.6cm)+(4,0)$) {$\theta$} -- cycle;
 %\draw[fill=lightgray, thick] (0,2) -- ++(-90:0.8cm) arc (-90:-90+atan2(2,4):0.8cm) node at ($(-60:0.5cm)+(0,2)$) {$\beta$} -- cycle;
 \end{tikzpicture}

 \caption{\label{fig:inverse3}A right triangle defined by $\theta=\tan ^{-1}(\frac{x}{1})$ with the length of the third leg found using the Pythagorean Theorem.}
 \end{figure}

 Then $\sec^2\left(\tan^{-1}(x)\right)=\sec^2(\theta)=\left(\sec(\theta)\right)^2=\left(\sqrt{1+x^2}\right)^2=1+x^2$. Recall that $\sec(x)=1/\cos(x)$. Finally, the derivative is
\[\frac{d}{dx}\left(\tan^{-1}(x)\right)=\frac{1}{1+x^2}.\]
\end{solution}

Using similar techniques, we can find the derivatives of the remaining inverse trigonometric functions.\\

\begin{theorem}{Derivatives of Inverse Trigonometric Functions}{deriv_inverse_trig}
{The inverse trigonometric functions are differentiable on all open sets contained in their domains (as listed in Figure \ref{fig:domain_trig}) and their derivatives are as follows:\\

\noindent	\begin{minipage}{.5\textwidth}\small
	\begin{enumerate}
	\item		$\ds \frac{d}{dx}\big(\sin^{-1}(x)\big) = \frac{1}{\sqrt{1-x^2}}$
	\item		$\ds \frac{d}{dx}\big(\sec^{-1}(x)\big) = \frac{1}{ x \sqrt{x^2-1}}$
	\item		$\ds \frac{d}{dx}\big(\tan^{-1}(x)\big) = \frac{1}{1+x^2}$
	\end{enumerate}
	\end{minipage}
	\begin{minipage}{.5\textwidth}\small
	\begin{enumerate}\addtocounter{enumi}{3}
	\item		$\ds \frac{d}{dx}\big(\cos^{-1}(x)\big) = -\frac{1}{\sqrt{1-x^2}}$
	\item		$\ds \frac{d}{dx}\big(\csc^{-1}(x)\big) = -\frac{1}{ x \sqrt{x^2-1}}$
	\item		$\ds \frac{d}{dx}\big(\cot^{-1}(x)\big) = -\frac{1}{1+x^2}$
	\end{enumerate}\index{derivative!inverse trig.}
	\normalsize
	\end{minipage}
}

\end{theorem}

Note how the last three derivatives are merely the opposites of the first three, respectively. Because of this, the first three are used almost exclusively throughout this text.\\

\subsection{Glossary of Derivatives of Elementary Functions}

In this chapter we have defined the derivative, given rules to facilitate its computation, and given the derivatives of a number of standard functions. We restate the most important of these in the following theorem, intended to be a reference for further work.

\begin{theorem}{Glossary of Derivatives of Elementary Functions}{deriv_glossary}
Let $u$ and $v$ be differentiable functions, and let $a$, $c$ and $n$ be real numbers, $a>0$, $n\neq 0$. \\
\noindent%

	\begin{minipage}{.5\textwidth}
	\begin{enumerate}
	\item		$\frac{d}{dx}\big(cu\big) = cu'$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(u\cdot v\big) = uv'+u'v$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(u(v)\big) = u'(v)v'$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(x\big) = 1$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(e^x\big) = e^x$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\ln x\big) = \frac{1}{x}$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\sin x\big) = \cos x$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\csc x\big) = -\csc x\cot x$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\tan x\big) = \sec^2x$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\sin^{-1}x\big) = \frac{1}{\sqrt{1-x^2}}$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\csc^{-1}x\big) = -\frac{1}{|x|\sqrt{x^2-1}}$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\tan^{-1}x\big) = \frac{1}{1+x^2}$\addtocounter{enumi}{1}
	\end{enumerate}
%\normalsize
\end{minipage}
\begin{minipage}{.5\textwidth}
	\begin{enumerate}\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(u\pm v\big) = u'\pm v'$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\frac uv\big) = \frac{u'v-uv'}{v^2}$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(c\big) = 0$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(x^n\big) = nx^{n-1}$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(a^x\big) = \ln a\cdot a^x$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\log_a x\big) = \frac{1}{\ln a}\cdot\frac{1}{x}$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\cos x\big) = -\sin x$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\sec x\big) = \sec x\tan x$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\cot x\big) = -\csc^2x$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\cos^{-1}x\big) = -\frac{1}{\sqrt{1-x^2}}$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\sec^{-1}x\big) = \frac{1}{|x|\sqrt{x^2-1}}$\addtocounter{enumi}{1}
	\item		$\frac{d}{dx}\big(\cot^{-1}x\big) = -\frac{1}{1+x^2}$
	\end{enumerate}
\normalsize
\end{minipage}

\end{theorem}


\section*{Rotations and Transformations in 2D Space}

A typical undergraduate linear algebra course will discuss the concept of
linear transformations. Usually rotations are referenced as one of many kinds
of linear transformations. When dealing with physical reference frames in
practical engineering, reference frames \textit{only} undergo rotational
transformations (usually). It's tempting, then, to think of the terms
``rotation' and ``transformation'' as being synonymous. However, when dealing
with physical objects rotating in space, especially in aerospace engineering,
these terms are \textit{not} usually synonymous. Here's the short of it:

\begin{itemize}
\item \textbf{Rotation} refers to rotating a vector within a single stationary
  frame. This is also referred to as the ``active'' or ``alibi'' interpretation
  of rotations.

\item \textbf{Transformation} refers to transforming one reference frame
  \textit{into} another frame. Any physical vectors in space \textit{do not}
  move with this ``passive'' or ``alias'' interpretation. Instead, the numbers
  denoting the vectors' components change to reflect the vectors' positions in
  the new frame.
\end{itemize}

Let's use the figure below for a visual example. We have two 2-dimensional
frames, labeled A and B. Each frame has two orthonormal basis vectors (that
means the two vectors are orthogonal to each other and have a length of 1,
and that every vector in the 2-dimensional plane can be defined as a sum
of some multiple of those two vectors) labeled $\hat{\bm{e}}_1$ and
$\hat{\bm{e}}_2$.

\begin{figure*}[h!]
    \centering
    \includegraphics[width=0.75\textwidth]{img/Two_2D_Frames.png}
\end{figure*}

There is also a vector, labeled $\bm{v}$. Looking between the two
frames, we see that $\bm{v}$ isn't moving in space. So the vector
$\bm{v}$ itself is some object in space with a magnitude and direction
that is independent of any reference frames. But the \textit{components} of
$\bm{v}$, which \textit{measure} its magnitude and direction with respect to
some set of basis vectors, \textit{are} dependent on the set of basis vectors.

Consider the components of $\bm{v}$ ``resolved'' in Frame A versus Frame B:
\begin{equation*}
    \bm{v}_A = \begin{bmatrix}
        -1 \\
        0
    \end{bmatrix}
    \quad\quad\quad\quad
    \bm{v}_B = \begin{bmatrix}
        -\cos(\pi/4) \\
        \sin(\pi/4)
    \end{bmatrix}
\end{equation*}

The components of $\bm{v}$ change when resolved (measured) with respect to
different basis vectors (different frames), but the \textit{actual} vector
is stationary in space.

Consider the matrix that rotates Frame A into Frame B (i.e. it rotates the
basis vectors of Frame A by $45^\circ$ anti-clockwise):
\begin{equation*}
    R_{A \to B} = \begin{bmatrix}
        \cos(\pi/4) & -\sin(\pi/4) \\
        \sin(\pi/4) & \cos(\pi/4)
    \end{bmatrix}
\end{equation*}

\textit{However}, applying this matrix to the vector $\bm{v}_A$, we do not
get $\bm{v}_B$:
\begin{equation*}
    \begin{bmatrix}
        \cos(\pi/4) & -\sin(\pi/4) \\
        \sin(\pi/4) & \cos(\pi/4)
    \end{bmatrix}
    \begin{bmatrix}
        -1 \\
        0
    \end{bmatrix}
    =
    \begin{bmatrix}
        -\cos(\pi/4) \\
        -\sin(\pi/4)
    \end{bmatrix}
    \neq
    \bm{v}_B
\end{equation*}

This disparity is because we applied a \textit{rotation} matrix to $\bm{v}_A$,
which one can think of as a function which takes as input the components of
a vector in some reference frame, rotates the vector, and outputs the components
of the rotated vector within that same frame. Effectively, applying
$R_{A \to B}$ to $\bm{v}_A$ is the equivalent of computing

\begin{figure*}[h!]
    \centering
    \includegraphics[width=\textwidth]{img/Incorrect Rotation of v.png}
\end{figure*}

You might have noticed, though, that the position of $\bm{v}$ relative to the
basis vectors of Frame B is identical to the position of $\bm{v}$ relative to
the basis vectors of Frame A \textit{had the opposite of $R_{A \to B}$ been
applied to it}. Visually:

\begin{figure*}[h!]
    \centering
    \includegraphics[width=0.75\textwidth]{img/Opposite Relative Rotation.png}
\end{figure*}

Therefore, if we want to find the components of $\bm{v}$ in Frame B, given
its components in Frame A ($\bm{b}_A$) and the rotation matrix from Frame A
to Frame B ($R_{A \to B}$), we must apply the opposite of the rotation matrix:
$\bm{v}_B = -R_{A \to B} \, \bm{v}_A$. However, this notation seems strange,
since we are applying a matrix designed to represent a rotation of a vector
within a frame to a situation in which a vector is not moving in space.
Conventionally, we tend to define a \textit{transformation} matrix which
transforms the components of a vector from one frame to another. In our case,
we can define the matrix $T_A^B = -R_{A \to B}$. The difference in the notation
between rotation and transformation matrix symbols doesn't appear to be
universally used, but it is useful for distinguishing the two when defined.

In summary,
\begin{itemize}
    \item Frames in 2D space require two basis vectors. Conventionally these
          basis vectors are orthonormal.
    \item Vectors are physical objects with a direction and magnitude.
    \item A vector's direction and magnitude can be \textit{measured}
          (or ``resolved'') with respect to the basis vectors of a frame,
          giving numerical values to its components.
    \item In applications where a physical object represented by a vector is
          moving within a single frame (e.g. a robotic arm on a factory line
          moving through a single frame), a rotation matrix aptly describes
          the motion.
    \item In applications where a physical object represented by a vector is
          \textit{not} moving, but rather the engineer wants to know its
          components in with respect to multiple frames, a transformation
          matrix aptly converts the components from one frame to another.
    \item Given the rotation matrix which rotates the basis vectors of one
          frame to another, the transformation matrix which transforms a
          vector's components from the first frame to the second is the
          negative of said rotation matrix.
\end{itemize}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{tikz}
\setlength{\parindent}{0pt}

\newtheorem*{theorem}{Theorem}
\newtheorem*{definition}{Definition}
\newtheorem*{lemma}{Lemma}
\newtheorem*{corollary}{Corollary}
\newtheorem{example}{Example}
\newtheorem*{trick}{Trick}
\newtheorem*{question}{Question}

\title{Lecture 5: Parametric Equations}
\author{}
\date{}

\begin{document}

\maketitle

\section{Equations of Lines in Space}

From previous lecture, we see that a line can be represented as the intersection
of two planes. However, as equations it is not so easy to use because we need to
solve it first.

Another representation is to treat a line as the trajectory of a moving point,
and include a parameter in the equation to describe different positions in the
line. Such representation is called \textbf{parametric equations}.

\begin{example}
  Find the parametric equation of the line through $Q_0 = (-1, 2, 2)$ and
  $Q_1 = (1, 3, -1)$.

  Given that the line can be the trajectory of a moving point, the parameter in
  the equation could be the time $t$. Suppose that at $t = 0$, the moving point
  is in $Q_0$, and at $t = 1$, the moving point is in $Q_1$, and suppose the
  point moves at a constant speed.

  With parametric equations, the coordinates of the point $Q$ in the line are
  functions of the parameter. Therefore, $Q = (x(t), y(t), z(t))$. The point $Q$
  in the line also satisfies the following equation:
  \[
    \vec{Q_0Q} = t \cdot \vec{Q_0Q_1}
  \]
  \[
    <x + 1, y - 2, z - 2> = t \cdot <2, 1, -3>
  \]
  \[
    \left\{ \begin{array}{ll}
    x(t) = 2t - 1 \\
    y(t) = t + 2 \\
    z(t) = -3t + 2 \\
    \end{array} \right.
  \]
  That is the parametric equation of the described line.
\end{example}

Notice that the coefficients of the parameter are components of a vector with
the same direction as the line (either one of the two directions a line has).
The constant terms in the parametric equation are coordinates of the point when
the parameter is equal to 0.

A line can have infinite different parametric equations. With different time
assumption and speed, the coefficients can be different, but they all express
the same line.

The reason why the parametric equation of a line only has one parameter is that
the points on a line only have one dimension (degree of freedom). It doesn't
matter whether it is a line or a curve.

\begin{example}
  Consider the plane $x + 2y + 4z = 7$, and two points $Q_0 = (-1, 2, 2)$ and
  $Q_1 = (1, 3, -1)$.

  1) Whether $Q_0$ and $Q_1$ are \underline{B}. \\
  A. on the same side. \\
  B. on the opposite sides. \\
  C. on the plane. \\
  D. cannot be decided.

  Reasons: \\
  For $Q_0 = (-1, 2, 2)$, $x + 2y + 4z = 11$. Therefore, $Q_0$ is in the plane
  $P_0$ whose equation is $x + 2y + 4z = 11$. \\
  For $Q_1 = (1, 3, -1)$, $x + 2y + 4z = 3$. Therefore, $Q_1$ is in the plane
  $P_1$ whose equation is $x + 2y + 4z = 3$. \\
  Assume the plane with the equation $x + 2y + 4z = 7$ is called $P$. \\
  The intersection point of these three planes with the $z$ axis are
  $(0, 0, \frac{11}{4})$, $(0, 0, \frac{3}{4})$, and $(0, 0, \frac{7}{4})$
  respectively, among which the intersection point of the plane $P$ are in
  between the ones of other two planes. \\
  Since the planes $P$, $P_0$, and $P_1$ are all parallel to each other, $P$ is
  in between other two planes. Hence, $P$ is in between $Q_0$ and $Q_1$.
  Therefore, $Q_0$ and $Q_1$ are on the opposite sides of the plane
  $x + 2y + 4z = 7$.

  \bigskip

  2) Does the line through $Q_0$ and $Q_1$ pass through the plane? If so, what
  is the intersection point?

  According to the previous example, we know that the parametric equation of the
  line is:
  \[
    \left\{ \begin{array}{ll}
    x(t) = 2t - 1 \\
    y(t) = t + 2 \\
    z(t) = -3t + 2 \\
    \end{array} \right.
  \]
  For the intersection point, the following equation holds:
  \[
    x(t) + 2y(t) + 4z(t) = 7
  \]
  \[
    (2t - 1) + 2(t + 2) + 4(-3t + 2) = 7
  \]
  \[
    -8t + 11 = 7
  \]
  \[
    t = \frac{1}{2}
  \]
  Therefore, the intersection point $Q = (0, \frac{5}{2}, \frac{1}{2})$.

  Notice that the coefficient of the parameter for the intersection point is
  the dot product of the direction of the line and the normal vector of the
  plane. If the coefficient is 0, from geometric point of view it means the
  direction of the line is perpendicular to the normal vector of the plane,
  which means the line is parallel to the plane; from algebraic point of view,
  the equation of the intersection point either always holds or has no
  solution, which also means the line is parallel to the plane.

\end{example}

\section{General Usage of Parametric Equations}

More generally, we can use parametric equations for arbitrary trajectory in
the plane or space.

\begin{example}
  Find the parametric equation of a cycloid, which is the curve traced by a
  fixed point on a circle as it rolls along a straight line without slipping.

  Solution:

  In this case, there is a natural parameter which is the angle $\theta$ the
  circle has rolled, because the position of the fixed point is solely decided
  by the angle. Suppose the start point of the circle is when the intersection
  between the circle and the $x$ axis is the origin $O$, which is also the
  tracing point. After rolling an angle $\theta$, the fix point is at position
  $B$, and the current intersection point is $A$. The center of the circle is
  $C$, and its radius is $a$.

  To find the parametric equation of the cycloid, we need to find the
  parametric equation of $\vec{OB}$.
  \[
    \vec{OB} = \vec{OA} + \vec{AC} + \vec{CB}
  \]
  Since the circle rolls along the $x$ axis without slipping,
  \[
    \vec{OA} = <a\theta, 0>
  \]
  Since $\vec{AC}$ is always perpendicular to the $x$ axis and the magnitude
  is the radius of the circle,
  \[
    \vec{AC} = <0, a>
  \]
  We can decompose $\vec{CB}$ into horizontal and vertical directions,
  \[
    \vec{CB} = <-a\sin\theta, -a\cos\theta>
  \]
  Notice that we can verify that the equations of the three vectors hold for
  any $\theta$, not only in $[0, 2\pi]$. Therefore,
  \[
    \begin{split}
    \vec{OB} &= \vec{OA} + \vec{AC} + \vec{CB} \\
             &= <a\theta - a\sin\theta, a - a\cos\theta> \\
    \end{split}
  \]
  Therefore, the parametric equation of a cycloid is
  \[
    \left\{ \begin{array}{ll}
    x(t) = a\theta - a\sin\theta \\
    y(t) = a - a\cos\theta \\
    \end{array} \right.
  \]

  \bigskip

  Following: What does the curve look like between two humps, i.e. around
  $\theta = 2n\pi$?

  To figure it out, we need to look at the derivative $\frac{dy}{dx}$ around
  $\theta = 2n\pi$.
  \[
    \begin{split}
      \frac{dy}{dx}|_{\theta = 2n\pi} &= (\frac{dy}{d\theta} / \frac{dx}{d\theta})|_{\theta = 2n\pi} \\
                                      &= (\frac{d(a - a\cos\theta)}{d\theta} / \frac{d(a\theta - a\sin\theta)}{d\theta})|_{\theta = 2n\pi} \\
                                      &= \frac{a\sin\theta}{a - a\cos\theta}|_{\theta = 2n\pi} \\
                                      &= \frac{0}{0} \\
    \end{split}
  \]
  According to L'Hospital rule,
  \[
    \begin{split}
      \frac{dy}{dx}|_{\theta = 2n\pi} &= \frac{a\sin\theta}{a - a\cos\theta}|_{\theta = 2n\pi} \\
                                      &= (\frac{d(a\sin\theta)}{d\theta} / \frac{d(a - a\cos\theta)}{d\theta})|_{\theta = 2n\pi} \\
                                      &= \frac{a\cos\theta}{a\sin\theta}|_{\theta = 2n\pi} \\
                                      &= \frac{a}{0} \\
                                      &= \infty
    \end{split}
  \]
  Therefore, the slope of a cycloid between two humps, i.e. around
  $\theta = 2n\pi$, is infinity, which means the tangent line in the
  intersection of two humps is vertical, perpendicular to the $x$ axis.
\end{example}

\end{document}


\section{Refinement \& Verification}

\begin{itemize}

	\item Specification statement: $ w : [P, Q] $

	\item $ P $ is the precondition, $ Q $ is the postcondition, $ w $ is the `frame' of variables that may be modified

	\item A program $ C $ satisfies $ w : [P, Q] $ if and only if

	\begin{itemize}

		\item $ \hoare{P}{C}{Q} $

		\item $ C $ only changes variables in $ w $

	\end{itemize}

	\item If $ P $ is not true when $ C $ is executed it may do anything, and it need not terminate

	\item Mixing specification statements with GCL forms a `wide-spectrum language'

	\item Refinement ($ \refsto $) -- a partial ordering on programs (similar to $ \le $ for reals)

	\begin{itemize}

		\item $ S \refsto S' $ means a user expecting program $ S $ would be satisfied with $ S' $

		\item $ S \refsto S' \iff \forall Q \cdot wp(S, Q) \entails wp(S', Q) $

		\item For a specification: $ wp(x : [P, Q], Q') \defeq P \land (\forall x \cdot Q \implies Q')[v_0 \backslash v] $

	\end{itemize}

	\item General approach to refining a program

	\begin{itemize}

		\item Start with a specification $ S = w : [P, Q] $

		\item Use rules to replace $ S $ with $ S' $ mixing specifications with GCL

		\item Each rule must preserve correctness -- i.e. every program $ C $ that satisfies $ S' $ must satisfy $ S $

		\item Eventually arrive at a pure GCL program $ C $ such that $ \hoare{P}{C}{Q} \equiv \true $

	\end{itemize}

\end{itemize}

\newpage

\subsection{Refinement Rules}

\begin{itemize}

	\item Rule 1: \textbf{Strengthen Postcondition}

	\begin{itemize}

		\item If $ P[w \backslash w_0] \land Q' \entails Q $ then $ w : [P, Q] \refsto w : [P, Q'] $ ($ P[w \backslash w_0] $ usually not needed)

	\end{itemize}

	\item Rule 2: \textbf{Weaken Precondition}

	\begin{itemize}

		\item If $ P \entails P' $ then $ w : [P, Q] \refsto w : [P', Q] $

	\end{itemize}

	\item Rule 3: \textbf{Skip}

	\begin{itemize}

		\item If $ P \entails Q $ then $ w : [P, Q] \refsto \Skip $

	\end{itemize}

	\item Rule 4: \textbf{Assignment}

	\begin{itemize}

		\item If $ P \entails Q[x \backslash E] $ then $ x : [P, Q] \refsto x := E $

	\end{itemize}

	\item Rule 5: \textbf{Composition}

	\begin{itemize}

		\item $ w : [P, Q] \refsto w : [P, M]; w : [M, Q] $ (no side condition)

	\end{itemize}

	\item Rule 6: \textbf{Following Assignment} (combined assignment and composition)

	\begin{itemize}

	\item $ w, x : [P, Q] \refsto w, x : [P, Q[x \backslash E]]; x := E $

	\end{itemize}

	\item Rule 7: \textbf{Selection}

	\begin{itemize}

		\item If $ P \entails \bigvee_{i=1}^{n} G_i $ then $ w : [P, Q] \refsto $\\
		$ ~~~~ \If ~ G_1 \rightarrow w : [G_1 \land P, Q] $\\
%		$ ~~~~ \Choice ~ G_2 \rightarrow w: [G_2 \land P, Q] $\\
		$ ~~~~ \dots $\\
		$ ~~~~ \Choice ~ G_n \rightarrow w: [G_n \land P, Q] $\\
		$ ~~~~ \Fi $

	\end{itemize}

	\item Rule 8: \textbf{Repetition}

	\begin{itemize}

		\item For repetition, a loop invariant $ I $ and loop variant (an integer expression) $ V $ are required

		\begin{itemize}

			\item Let $ V_0 $ be the value of $ V $ at the start of each iteration

			\item Then $ 0 \le V < V_0 $ is $ \true $ at the end of each iteration

			\item (i.e. $ V $ is \textit{strictly decreasing} on every iteration, and won't be negative before loop termination)

		\end{itemize}

	\item To apply the repetition rule

		\begin{enumerate}

			\item Strengthen postcondition to $ I \land \lnot G $ (side condition: $ I \land \lnot G \entails Q $)

			\item Use composition to perform $ w : [P, I \land \lnot G] \refsto w : [P, I]; w : [I, I \land \lnot G] $

			\item Refine the first half into initialisation (e.g. an assignment)

			\item Refine the second half using the repetition rule (no side conditions!)

		\end{enumerate}

	\item Rule: Let $ G \defeq \bigvee_{i=1}^{n} G_i $, then $ w : [I, I \land \lnot G] \refsto$\\
	$ ~~~~ \Do ~ G_1 \rightarrow w : [I \land G_1, I \land (0 \le V < V_0)] $\\
%	$ ~~~~ \Choice ~ G_2 \rightarrow w : [I \land G_2, I \land (0 \le V < V_0)] $\\
	$ ~~~~ \dots $\\
	$ ~~~~ \Choice ~ G_n \rightarrow w : [I \land G_n, I \land (0 \le V < V_0)] $\\
	$ ~~~~ \Od $

	\end{itemize}

	\item Rule 9: \textbf{Contract frame}

	\begin{itemize}

		\item $ w, x : [P, Q] \refsto w : [P, Q[x_0 \backslash x]] $

	\end{itemize}

	\item Rule 10: \textbf{Remove invariant}

	\begin{itemize}

		\item If $ w $ does not occur in $ I $ then $ w : [P \land I, Q \land I] \refsto w : [P, Q] $

	\end{itemize}

\end{itemize}


\subsection{Stability}
As touched on in section \ref{sec:choice_of_cont}, the controller needs to be stable for the range in which the robot operates. The stability analysis will be preformed on the kinematic equation shown in \eqref{eq:base_system_eq2}. Since the robot has two controllers, for turning left and right, the kinematic function can be rewritten as.

\begin{equation}
    \Dot{\theta} = \frac{\Delta v}{2r_b}
    \label{eq:stab1}
\end{equation}

\subsubsection{Continuous Time}

The $\Delta v$ will be seen as a constant and will be ignored in future equations. Using \eqref{eq:stab1}, the open-loop and closed-loop equation becomes.

\begin{equation}
    G_{ol}(s)=\frac{1}{2 r_b s}
\end{equation}

\begin{equation}
    G_{cl}(s)=\frac{G_{ol}(s)}{1+G_{ol}(s)}=\frac{(2r_b s)}{(2r_b s)*(2r_b s + 1)}
    \label{eq:stab12}
\end{equation}

\noindent The function \eqref{eq:stab12} has a pole and zero cancellation at origo. In Fig.\ref{fig:step1} the step response, poles and zeros are plotted for the closed loop system. Since all the poles appeare on the negative real plane, the system is stable.

\begin{figure}
    \centering
    \includegraphics[width=0.7\textwidth]{img/closed_loop_step.eps}
    \caption{The closed loop system without a controller.}
    \label{fig:step1}
\end{figure}

\noindent The \textbf{PD}-controller discussed in previous sections only need to improve the system response, since the overall system is already stable. Introducing the controller to the system gives us the closed loop function.

\begin{equation}
    G_{clPD} = \frac{2r_b s(k_p + k_d s)}{2r_b s(s(2r_b+k_d)+k_p)}
\end{equation}

\noindent From this it's easy to see that the system will remain stables for $k_d > - 2 r_b$ and that the value of $k_d$ won't have any effect on the stability of the system in continuous time.\\
\indent The values for $k_p$ and $k_d$ is therefore chosen in such a manner that $ 0 > k_d > -2 r_b$ and $k_p$ so that the greatest error input from the \textbf{MV}-node will not saturate the actuators. The result of these are shown in Fig.\ref{fig:step2} and a comparison was made, represented in table \ref{tab:with_without}.

\begin{figure}[H]
    \centering
    \includegraphics[width= 0.7\textwidth]{img/closed_loop_step_PD.eps}
    \caption{The closed loop system with a controller.}
    \label{fig:step2}
\end{figure}

\begin{table}[H]
    \centering
    \caption{Comparison between the closed loop systems.}
    \begin{tabular}{|c|ccc|}
        \hline & Without controller & \textbf{PD}-controller & Discrete Time \textbf{PD}-controller \\
        \hline Rise Time & 0.8788$s$ & 0.0879$s$ & 0.1000$s$ \\
        Settling Time & 1.5648$s$ & 0.1565$s$ & 0.2500$s$ \\
        \hline
    \end{tabular}
    \label{tab:with_without}
\end{table}

\subsubsection{Discrete Time}
Using MATLAB to move from continuous time to discrete time with a update frequency of 20 Hz\footnote{The update frequency of the camera} and an input delay of $1/30s$\footnote{Estimated input delay to the actuators}. In Fig.\ref{fig:stepD} it's clear that the pole is within the unit circle on the discrete time plane and that the responds follows the step input. In conclusion, the system is stable using the $k_p$ and $k_d$ values proposed above, even in discrete time \cite{torkel_ljung}.

\begin{figure}[H]
    \centering
    \includegraphics[width= 0.7\textwidth]{img/closed_loop_step_D.eps}
    \caption{The continuous step response, discrete step response and the poles and zeros of the discrete transfer function.}
    \label{fig:stepD}
\end{figure}


\section{Random Variable}

\subsection{Events}

\begin{definition}[sample space]
    The set of all possible outcome of an experiment is defined as \cindex{sample space} $S$.
\end{definition}

\begin{definition}[event]
    Any subset $E$ of sample space $S$ is defined as \cindex{event}.
\end{definition}

It is conventional to designate $E \cup F$ as $EF$.

\begin{definition}[mutually exclusive]
    If $EF = \emptyset $, $E$ and $F$ are said to be \cindex{mutually exclusive}.
\end{definition}

\begin{definition}
    The probabilty $P$ is defined on event $E$ of sample space $S$ which follow the following condition:
    \begin{enumerate}
        \item $0 \leq P(E) \leq 1$.
        \item $P(S) = 1$.
        \item for any sequence of mutually exclusive event $E_1, E_2, \dots$,
            \begin{equation*}
                P \left(\bigcup_{n=1}^\infty E_n \right) = \sum_{n=1}^\infty P(E_n)
            \end{equation*}

    \end{enumerate}
    \qed
\end{definition}

If the experiment is repeated over and over again, with probability $1$ the proportion of time that event $E$ will occur is $P(E)$.

\begin{theorem}[inclusion-exclusion identity]
    \begin{equation}
        \begin{aligned}
            P(\bigcup_{i=1}^n E_i) &= \sum_i P(E_i) - \sum_{i < j}P(E_i E_j) + \sum_{i < j < k} P(E_i E_j E_k) + \dots + (-1)^{n+1} P(E_1 E_2 \dots E_n)
        \end{aligned}
    \end{equation}
\end{theorem}

\begin{definition}[conditional probability]
    The \cindex{conditional probability} that $E$ occurs given that $F$ occurs is denoted by $P(E|F)$ and defined as:
    \begin{equation}
        P(E|F) = \frac{P(EF)}{P(F)}
    \end{equation}

    For conditional probability the sample space is changed.

    \qed
\end{definition}

\begin{definition}[independent]
    Two event $E$ and $F$ are \cindex{independent} if $P(EF) = P(E) P(F)$.
\end{definition}

\begin{theorem}
    $E$ and $F$ are independent if $P(E/F) = P(E)$.
\end{theorem}

\begin{definition}
    The events $\{ E_i \}$ are independent if for every subset $E_{i_j}$ ($ \forall j, 1 \leq j \leq n$), that
    \begin{equation}
        P(E_{i_1} E_{i_2} \dots E_{i_j}) = P(E_{i_1}) P(E_{i_2}) \cdots P(E_{i_j})
    \end{equation}
\end{definition}

\begin{definition}
    For mutually exclusive event $F_i$ that $\displaystyle \bigcup_{i=1}^n F_i = S$ .The \cindex{Bayes' formula} is defined as:
    \begin{equation}
    \displaystyle P(F_j | F) = \frac{\probability{EF_j}}{\sum\limits_{i=1}^n \probability{EF_i}} = \frac{\probability{E|F_j} \probability{F_j}}{\sum\limits_{i=1}^n \probability{E|F_i} \probability{F_i}}
    \end{equation}
\end{definition}

\subsection{Random Variable Definition}

\begin{definition}[random variable]
    A real value function defined on sample space is called \cindex{random variable}.
\end{definition}

\begin{definition}[discrete random variable]
    A random variable that can takes at most a countable values is said to be \cindex{discrete random variable}.
\end{definition}

\begin{definition}[probability mass function]
    The \cindex{probability mass function} $p(a)$ of discrete random variable $X$ is defined as $p(a) = \probability{X = a}$.
\end{definition}

\begin{definition}[cdf]
    The \cindex{cumulative distribution function (cdf)} of random variable $X$ for any $- \infty < b < \infty$ is defined as $F(b) = \probability{ X \leq b }$
\end{definition}

\begin{theorem}
    Some properties of cdf are:
    \begin{enumerate}
        \item $F(b)$ is nondecreasing function of $b$.
        \item $F(b)$ is continuous from the right.
        \item $\displaystyle \lim_{b \rightarrow \infty} F(b) = F(\infty) = 1$.
        \item $\displaystyle \lim_{b \rightarrow -\infty} F(b) = F(- \infty) = 0$.
    \end{enumerate}
\end{theorem}

\begin{definition}
    $\probability{X < b}$ is defined as $\probability{ X < b } = \displaystyle \lim_{h \rightarrow 0^+} \probability{ X \leq b - h } = \lim_{h \rightarrow 0^+} F(b-h)$.
\end{definition}

$\probability{X < b} \neq F(b)$ because $F(b)$ also include the probability that $X$ equals $b$.

Poisson random variable is a approximate of binomial random variable when $n$ is large and $p$ is small. Let $\lambda = np$, then:
\begin{equation*}
    \begin{aligned}
        \binom{n}{i}p^i (1-p)^{n-i} &= \frac{n!}{(n-i)! i!} \left(\frac{\lambda}{n} \right)^i \left(1-\frac{\lambda}{n} \right)^{n-i} \\
        &= \frac{n (n-1) \cdots (n - i + 1)}{n^i} \frac{\lambda^i}{i!} \frac{(1- \frac{\lambda}{n})^n}{(1- \frac{\lambda}{n})^i} \\
        & \approx 1 \times \frac{\lambda^i}{i!} \times \frac{e^{- \lambda}}{1} = e^{- \lambda} \frac{\lambda^i}{ i !}
    \end{aligned}
\end{equation*}

\begin{definition}
    $f(x)$ is called \cindex{probability density function} if $\probability{X \in B} = {\displaystyle \int_B f(x) \dif{x}} $. \qed
\end{definition}

Because $\probability{ a - \frac{\varepsilon}{2} \leq X \leq a + \frac{\varepsilon}{2} } =\displaystyle \int_{a - \frac{\varepsilon}{2}}^{a + \frac{\varepsilon}{2}} f(x) \dif{x} \approx \varepsilon f(a)  $, $f(a)$ is a measure of how likely it is that the random variable will be near $a$ within interval $\varepsilon$.

Sometimes when calculating the probability $\probability{X=a}$, we can first calculate $F(a)$ and then calculate $\probability{X=a} = \dod{F(a)}{a}$.

\begin{definition}[expectation]
    The \cindex{expectation} of $X$ is defined as:
    \begin{equation}
        \expect{X} = \begin{cases}
            \displaystyle \sum x p(x) & \text{for discrete case} \\
            \displaystyle \int_{-\infty}^{\infty} x f(x) \dif{x} & \text{for continuous case}
        \end{cases}
    \end{equation}
\end{definition}

\begin{theorem}
    The expectation of a function $g$  of a random variable $X$ is:
    \begin{equation}
        \expect{g(X)} = \begin{cases}
            \displaystyle \sum g(x) p(x) & \text{for discrete case} \\
            \displaystyle \int_{-\infty}^{\infty} g(x) f(x) \dif{x} & \text{for continuous case}
        \end{cases}
    \end{equation}
\end{theorem}

\begin{theorem}
    \begin{equation}
        \expect{aX + b} = a \expect{X} + b
    \end{equation}
\end{theorem}

\begin{theorem}
    \begin{equation}
        \expect{aX + bY} = a \expect{X} + b \expect{Y}
    \end{equation}
\end{theorem}

\begin{definition}[variance]
    The \cindex{variance} of $X$ is defined as
    \begin{equation}
        \variance{X} = \expect{(X - \expect{X})^2}
    \end{equation}
\end{definition}

\begin{theorem}
    \begin{equation}
        \variance{X} = \expect{X^2} - \expect{X}^2
    \end{equation}
\end{theorem}

\begin{theorem}
    Let $X_i$ be identically independent random variable. We need to notice that
    \begin{equation*}
        \variance{\sum_{i=1}^n X_i} = n \variance{X} \neq \variance{n \times X_i} = n^2 \variance{X}
    \end{equation*}
\end{theorem}

\subsection{Common Distributions}

\begin{center}
   \begin{tabular}{|c|c|c|c|c|}
  \hline
  name & density & $\phi(t)$ & mean & var \\
  \hline
  \cindex{binomial} & $\displaystyle \binom{n}{x} p^x (1-p)^{n-x}$  &$(p e^t + q)^n$ & np & npq \\
  \hline
  \cindex{poisson} & $\displaystyle e^{-\lambda} \frac{\lambda^x}{x!}$  & $\displaystyle e^{\lambda (e^t -1)}$& $\lambda$ & $\lambda$ \\
  \hline
  \cindex{geometric} & $p (1-p)^{x-1}$& $\displaystyle \frac{pe^t}{1-(1-p)e^t}$ & $\displaystyle \frac{1}{p}$ & $\displaystyle \frac{1-p}{p^2}$ \\
  \hline
  \cindex{uniform} &$\displaystyle \frac{1}{b-a}$ & $\displaystyle \frac{e^{tb} - e^{ta}}{t(b-a)}$ & $\displaystyle \frac{a + b}{2}$ & $\displaystyle \frac{(b-a)^2}{12}$ \\
  \hline
  \cindex{exponential} & $\lambda e^{-\lambda x}$ & $\displaystyle \frac{\lambda}{\lambda - t}$ & $\displaystyle \frac{1}{\lambda}$ & $\displaystyle \frac{1}{\lambda^2}$ \\
  \hline
  \cindex{gamma} & $\displaystyle \frac{\lambda e^{-\lambda x} (\lambda x)^{n - 1}}{(n-1)!}$ & $\displaystyle \left(\frac{\lambda }{\lambda - t} \right)^n$ & $\displaystyle \frac{n}{\lambda}$ & $\displaystyle \frac{n}{\lambda^2}$ \\
  \hline
  \cindex{normal} & $\displaystyle \frac{1}{\sqrt{2 \pi} \sigma} e^{\displaystyle - \frac{(x - \mu)^2}{2 \sigma^2}}$ & $e^{\displaystyle \mu t + \frac{\sigma^2 t^2}{2}}$&$\mu$& $\sigma^2$ \\
  \hline
\end{tabular}
\end{center}

\begin{theorem}
    If $X$ is normally distributed with $\mu$ and $\rho^2$, then $Y = aX + b$ is a normal distribution with $a\mu + b$ and $(a\rho)^2$. So $Y =\displaystyle \frac{X - \mu}{\rho}$ is normally distributed with $0$ and $1$, which is called \cindex{standard normal distribution}.
\end{theorem}

\begin{definition}
    A random variable $X$ has \cindex{gamma distribution} with parameter $\alpha$ and $\beta$ if the pdf is:
    \begin{equation*}
        f(x|\alpha, \beta) = \frac{\beta e^{-\beta x} (\beta x)^{\alpha - 1}}{\Gamma(\alpha)}
    \end{equation*}
    while $\Gamma(\alpha)$  is defined as:
    \begin{equation}
        \Gamma(\alpha) = \int_0^\infty e^{-x} x^{\alpha - 1} \dif{x}
    \end{equation}
    exponential distribution is gamma distribution with $\alpha = 1$.

    \qed
\end{definition}

\begin{theorem}
    If $X_i$ are independent gamma distribution with $\alpha_i$ and $\beta$, then the sum $\displaystyle \sum_{i=1}^n X_i$ has gamma distribution with $\displaystyle \sum \alpha_i$ and $\beta$.
\end{theorem}

\begin{theorem}
    \begin{equation}
        \Gamma \left(\frac{1}{1} \right) = \sqrt{\pi}
    \end{equation}

    \begin{equation}
        \Gamma (a + 1) = a \Gamma(a)
    \end{equation}
    \begin{equation}
        \Gamma (n) = (n-1)!
    \end{equation}
    \qed
\end{theorem}

\begin{definition}
    $X$ is a \cindex{beta distribution} if the pdf is:
    \begin{equation}
        f(x|\alpha, \beta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha) \Gamma(\beta)} x^{\alpha - 1} (1-x)^{\beta - 1}
    \end{equation}
\end{definition}

% joint distribution
\subsection{Joint Distribution}

\begin{definition}
    The \cindex{joint cumulative probability distribution function} of two discrete random variable $X$ and $Y$ is defined as
    \begin{equation}
        F(a,b) = \probability{ X \leq a, Y \leq b }
    \end{equation}

    The cumulative distribution of $Y$ is defined as
    \begin{equation}
        F_X(a) = \probability{X \leq a } =\probability{X \leq a, Y < \infty } = F(a, \infty)
    \end{equation}

    The \cindex{joint probability mass function} is defined as
    \begin{equation}
        p(x,y) =\probability{X = x, Y = y}
    \end{equation}

    The probability mass function of $X$ from $p(x,y)$ is defined as
    \begin{equation}
        p_X(x) = \displaystyle \sum_{y: p(x,y) > 0} p(x,y)
    \end{equation}
    \qed
\end{definition}

\begin{definition}
    for continuous case, $X$ and $Y$ are \cindex{jointly continuous} if there is a function $f(x,y)$ that for all sets $A$ and $B$ we have
\begin{equation}
    \probability{X \in A, Y \in B} = \int_B \int_A f(x,y) \dif{x} \dif{y}
\end{equation}

$f(x,y)$ is called \cindex{joint probability density function} of $X$ and $Y$. The probability density function of $X$ can be obtained as
\begin{equation}
    \begin{aligned}
        \probability{X \in A} &= \int_A \int_{-\infty}^\infty f(x,y) \dif{x} \dif{y} \\
        &= \int_A f_X (x) \dif{x}
    \end{aligned}
\end{equation}
where
\begin{equation}
     f_X(x) = \int_{-\infty}^\infty f(x,y) \dif{y}
\end{equation}

We have
\begin{equation}
    \dmd{F(a,b)}{2}{a}{1}{b}{1} = f(a,b)
\end{equation}
\qed
\end{definition}

For $n$ random variable $X_i$, we could construct $Y_i$ that
\begin{equation}\label{jointdenityequation}
    \begin{aligned}
        Y_1 &= g_1 (X_1, X_2, \cdots, X_n) \\
        Y_2 &= g_2 (X_1, X_2, \cdots, X_n) \\
        & \vdots \\
        Y_n &= g_n (X_1, X_2, \cdots, X_n) \\
    \end{aligned}
\end{equation}

Assume its \cindex{Jacobian} determinant $J(x_1, \dots, x_n) \neq 0$ for all $x_i$. Then the probability density function of $Y_i$ is
\begin{equation}
    f_{Y_1, \cdots, Y_n}(y_1, \cdots, y_n) = \frac{1}{\absolutevalue{J(x_1, \cdots, x_n)}} f_{X_1, \cdots, X_n}(x_1, \cdots , x_n)
\end{equation}

So the process of calculating $f_{Y_i}$ are:
\begin{enumerate}
    \item solve $X_i = h(Y_i)$ from equation (\ref{jointdenityequation} ).
    \item calculate $J(X_i)$.
    \item replace $x_i$ by $y_i$ in $f(x_i)$ and multiplicate by $\displaystyle \frac{1}{\absolutevalue{J}}$.
\end{enumerate}

\subsection{Independence}

\begin{definition}
    Two random variable $X$ and $Y$ are \cindex{independent} if $\forall a,b \in \mathbf{R}$,
    \begin{equation}
        \probability{ X \leq a , Y \leq b } = \probability{ X \leq a } \probability{ X \leq b }
    \end{equation}
    It means
    \begin{equation}
        F(a,b) = F_X(a) F_Y(b)
    \end{equation}
    When $X$ and $Y$ are discrete, it reduces to
    \begin{equation}
        p(x,y) = p_X(x) p_Y(y)
    \end{equation}

    When they are continuous, it reduces to
    \begin{equation}
        f(x,y)=f_X(x) f_Y(y)
    \end{equation}
    \qed
\end{definition}

\begin{theorem}
    If $X$ and $Y$ are indepent, for any function $h$ and $g$, we have
    \begin{equation}
        \expect{g(X) h(Y)} = \expect{g(X)} \expect{h(Y)}
    \end{equation}
\end{theorem}

\subsection{Covariance}

\begin{definition}[covariance]
    The \cindex{covariance} for $X$ and $Y$ is defined as
    \begin{equation}
    \begin{aligned}
        \covariance{X}{Y} &= \expect{(X - \expect{X})(Y - \expect{Y})} \\
        &= \expect{XY} - \expect{X} \expect{Y}
    \end{aligned}
    \end{equation}
    In general covariance means $Y$ tends to move in the same direction of $X$.
    \qed
\end{definition}

\begin{theorem}
    \begin{equation}
        \covariance{X}{Y} = \variance{X}
    \end{equation}
\end{theorem}

\begin{theorem}
    \begin{equation}
        \covariance{X}{Y} = \covariance{Y}{X}
    \end{equation}
\end{theorem}

\begin{theorem}
    \begin{equation}
        \covariance{cX}{Y} = c \covariance{X}{Y}
    \end{equation}
\end{theorem}

\begin{theorem}
    \begin{equation}
        \covariance{X}{Y + Z} = \covariance{X}{Y} + \covariance{X}{Z}
    \end{equation}
\end{theorem}

\begin{theorem}
    \begin{equation}
        \mathhilight{Cov} \left(\sum_{i=1}^n X_i, \sum_{j=1}^m Y_j \right) = \sum_{i=1}^n \sum_{j=1}^m \mathhilight{Cov}(X_i, Y_j)
    \end{equation}
\end{theorem}

\begin{theorem}
    \begin{equation}
    \begin{aligned}
        \mathhilight{Var} \left( \sum_{i=1}^n X_i \right) &= \mathhilight{Cov}\left( \sum_{i=1}^n X_i, \sum_{j=1}^n X_j \right) \\
        &= \sum_{i=1}^n \variance{X_i} + 2 \sum_{i=1}^n \sum_{j < i} \covariance{X_i}{X_j} \\
        &= \sum_{i=1}^n \variance{X_i} \text{ (if } X_i \text{,} X_j \text{ are independent)}
    \end{aligned}
    \end{equation}
    This theorem is often used to calculate the variance. \qed
\end{theorem}

\begin{definition}[sample mean]
    If $X_i$ are independent and identically distributed, then the random variable $\bar{X} =\displaystyle \frac{\sum_{i=1}^n X_i}{n}$ is called the \cindex{sample mean}.
\end{definition}

\begin{theorem}
\begin{equation}
    \expect{\bar{X}} = \mu
\end{equation}
\end{theorem}

\begin{theorem}
\begin{equation}
    \variance{\bar{X}} = \frac{\sigma^2}{n}
\end{equation}
\end{theorem}
\begin{theorem}
\begin{equation}
    \covariance{\bar{X}}{X_i - \bar{X}} = 0
\end{equation}
\end{theorem}

\begin{definition}
    Function $F_{X+Y}$ is called the \cindex{convolution} of the distribution of $F_X$ and $F_Y$, which is
    \begin{equation}
        \begin{aligned}
            F_{X+Y}(a) &= \probability{X + Y \leq a} \\
            &= \iint_{x+y \leq a} f(x) g(y) \dif{x} \dif{y} \\
            &= \int_{-\infty}^\infty \left( \int_{-\infty}^{a - y} f(x) \dif{x} \right) g(y) \dif{y} \\
            &= \int_{-\infty}^\infty F_X (a-y)g(y) \dif{y}
        \end{aligned}
    \end{equation}

    The probability density function $f_{X+Y}(a)$ is given by
    \begin{equation}
        \begin{aligned}
            f_{X+Y}(a) &= \dod{F_{X+Y}(a)}{a} \\
            &= \dod{\displaystyle \int_{-\infty}^\infty F_X (a-y)g(y) \dif{y}}{a} \\
            &= \int_{-\infty}^{\infty} f(a-y) g(y) \dif{y}
        \end{aligned}
    \end{equation} \qed
\end{definition}

% moment generating function

\begin{definition}
    The \cindex{moment generating function} $\phi(t)$ of the random variable $X$ is defined by
    \begin{equation}
        \begin{aligned}
            \phi(x) &= \expect{e^{tX}} \\
            &= \begin{cases}
                 \displaystyle \sum_x e^{tx} p(x)\\
                 \displaystyle \int_{-\infty}^\infty e^{tx} f(x) \dif{x}
            \end{cases}
        \end{aligned}
    \end{equation}

    For any $n$ random variable $X_i$, the \cindex{joint moment generating function} $\phi(t_i)$ is defined by $\phi(t_1, \dots, t_n) = \expect{ e^{ \sum_{i=1}^n t_i X_i}}$.
    \qed
\end{definition}

Because $\phi^n (t) = \expect{X^n e^{tX}}$, we have $\phi^n(0) = \expect{X^n}$.

\begin{theorem}
    \begin{equation}
        \phi_{X+Y}(t) = \phi_X(t) \phi_Y(t)
    \end{equation}
\end{theorem}

\begin{theorem}
The moment generating function uniquely determine the distribution.
\end{theorem}

% sample mean and sample variance

\subsection{Sample Mean and Sample Variance}

\begin{definition}[sample mean]
    The \cindex{sample mean} is defined as $\displaystyle \bar{X} = \frac{\sum X_i}{n}$.
    \qed
\end{definition}

\begin{definition}[sample variance]
    Let $X_i$ be independent and identically distributed random variable with mean $\mu$ and variance $\sigma^2$, the \cindex{sample variance} $S^2$ is defined by
\begin{equation}
    S^2 = \sum_{i=1}^n \displaystyle \frac{(X_i - \bar{X})^2}{n - 1}
\end{equation}
\qed
\end{definition}

\begin{theorem}
\begin{equation}
    \expect{S^2} = \sigma^2
\end{equation}
\end{theorem}

\begin{definition}[chi-squared]
    If $Z_i$ are $n$ independent standard normal random variables, the random variable $\displaystyle \sum_{i=1}^n Z_i^2$ is called \cindex{chi-squared} random variable with $n$ degrees of freedom.
\end{definition}

\begin{theorem}
If $X_i$ are independent and identically distributed normal random variable with mean $\mu$ and variance $\sigma^2$, then:
    \begin{enumerate}
        \item the sample mean $\bar{X}$ and sample variance $S^2$ are independent.
        \item $\bar{X}$ is a normal random variable with mean $\mu$ and variance $\displaystyle \frac{\sigma^2}{n}$.
        \item $\displaystyle \frac{(n-1)S^2}{\sigma^2}$ is a chi-squared random variable with $n-1$ degrees of freedom.
    \end{enumerate}
\end{theorem}

\begin{proof}
    Because $X_1$ are normal random varialbe, $\bar{X}$ is also a normal random variable. Since $\covariance{\bar{X}}{X_i - \bar{X}} = 0$, normal random variable $\bar{X}$ and $X_i - \bar{X}$ are independent. Since $S^2 = \displaystyle \sum_{i=1}^n \frac{(X_i - \bar{X})^2}{n -1}$ is a function of $X_i - \bar{X}$ which is independ from $\bar{X}$, $S^2$ is independent from $\bar{X}$.

    Because
    \begin{equation*}
        \frac{(n-1)S^2}{\sigma^2} + \left(\frac{\bar{X} - \mu}{\frac{\sigma}{\sqrt{n}}}\right)^2 = \sum_{i=1}^n \frac{(X_i - \mu)^2}{\sigma^2}
    \end{equation*}
    generate the moment generating function of $\displaystyle \frac{(n-1)S^2}{\sigma^2}$ from above formula and we have $\expect{e^{t \frac{(n-1)S^2}{\sigma^2}}} = \left(1-2t \right)^{- \frac{n-1}{2}}$ which is an chi-squared distribution with freedom $n-1$.
\end{proof}

% inequality

\subsection{Inequality}

\begin{theorem}[\cindex{Markov's Inequality}]
If $X$ is a random variable that is non-negative. Then for any $a > 0$ we have
\begin{equation}
    \probability{X \geq a} \leq \frac{\expect{X}}{a}
\end{equation}
\end{theorem}
\begin{proof}
    \begin{equation*}
        \begin{aligned}
            \expect{X} &= \int_0^\infty x f(x) \dif{x} \\
            &= \int_0^a x f(x) \dif{x} + \int_a^\infty x f(x) \dif{x} \\
            & \geq \int_a^\infty x f(x) \dif{x} \\
            & \geq \int_a^\infty a f(x) \dif{x} \\
            &= a \int_a^\infty f(x) \dif{x} \\
            &= \probability{X \geq a}
        \end{aligned}
    \end{equation*}
\end{proof}

\begin{theorem}[\cindex{Chebyshev's Inequality}]
    If $X$ is a random variable with mean $\mu$ and variance $\sigma^2$. Then for any $k > 0$ we have
    \begin{equation}
        \probability{\absolutevalue{X - \mu} \geq k } \leq \frac{\sigma^2}{k^2}
    \end{equation}
\end{theorem}
\begin{proof}
    Since $(X - \mu)^2$ is nonnegative, using Markov's Inequality we have
    \begin{equation*}
        \probability{\absolutevalue{(X - \mu)} \geq k} = \probability{(X - \mu)^2 \geq k^2} \leq \frac{\expect{(X-\mu)^2}}{k^2}
    \end{equation*}
\end{proof}

\begin{theorem}[\cindex{Strong Law of Large Numbers}]
    Let $X_i$ be a sequence of independent random variables having a common distribution. With probability 1
    \begin{equation}
        \frac{X_1 + X_2 + \dots + X_n}{n} \rightarrow \mu \text{ as } n \rightarrow \infty
    \end{equation}
\end{theorem}

\begin{theorem}[\cindex{Central Limit Theorem}]
    Let $X_i$ be a sequence of independent identical random variables with mean $\mu$ and variance $\sigma^2$, the distribution $\displaystyle \frac{X_1 + X_2 + \dots + X_n - n \mu}{\sigma \sqrt{n}}$ tends to be standard normal distribution as $n \rightarrow \infty$.
\end{theorem}

% examples
\subsection{Examples}

\begin{example}
    Suppose $m$ coupons are selected from $n$ different types of coupons. Each coupon is equally likely to be selected. What is the expected number of different types?
\end{example}

\begin{proof}
    Let $X$ denote the number of types in $m$ selection. We have $X = X_1 + ... + X_n$ where
    \begin{equation*}
        X_i = \begin{cases}
            1 & \text{ , if } i \text{ occurs in selection} \\
            0 & \text{ , otherwise}
        \end{cases}
    \end{equation*}
    We have
    \begin{equation*}
        \expect{X_i} = 1 - \left(\frac{n - 1}{n} \right)^m
    \end{equation*}
    So $\expect{X} = \expect{X_1} + \dots + \expect{X_n} = \displaystyle n \left(1 - \left(\frac{n - 1}{n} \right)^m \right)$
\end{proof}

\begin{example}
    Let $X_i$ be independent and identically distributed continuous random variable. Let $X_{(i)}$ denote the $i$th smallest of these random variables. Then $X_{(i)}$ is called \cindex{order statistics}. Note that $X_{(i)} \leq x$ if and only if at least $i$ of $X_j$ are not larger than $x$, so
    \begin{equation*}
        \probability{X_{(i)} \leq x} = \sum_{k=i}^n \binom{n}{k} \left(F(x) \right)^k \left(1 - F(x) \right)^{n-k}
    \end{equation*}

    So
    \begin{equation*}
        \begin{aligned}
            f_{X_{(i)}} (x) &= \dod{\probability{X_{(i)} \leq x}}{x} \\
            &= \frac{n!}{(n-i)!(i-1)!} f(x) \left( F(x) \right)^{i-1} \left(1-F(x) \right)^{n-i}
        \end{aligned}
    \end{equation*}
    The result could be read as there are $\left( F(x) \right)^{i-1}$ which are less than $x$, $\left(1-F(x) \right)^{n-i}$ which are larger than $x$, and $f(x)$ which equals $x$. \qed
\end{example}

\begin{example}
A particle moves along a circle of $m+1$ nodes. Each time it has equality of moving clockwise or counterclockwise. What is the probability that node $i$ is the last visited node?
\end{example}
\begin{proof}
    Suppose $i$ is the last visited nodes and consider 2 nodes $i-1$ and $i+1$ around it. If $i-1$ is visited before $i+1$, the probability is the same as gambling without losing money when it reaches $i+1$, so the probability is the same for every nodes. So the result is $\displaystyle \frac{1}{m}$.
\end{proof}


\section*{Ex.34.2-6}
\subsection*{Show that HAM-PATH $\in$ NP}

If we can verify a certificate in polynomial time, then HAM-PATH $\in$ NP.

A certificate could be a path from $u$ to $v$ in $G$:
\\
- Check that all edges on the path is in $E$, where $G=(V,E)$.
\\
- Check that that the vertices on the path covers all vertices in $V$.
\\
This can be done in polynomial time.


The key building block of the solutions that we develop in the subsequent
sections is the uncertainty-quantification technique presented in this section.
The task of this technique is to propagate uncertainty through the system, from
a set of inputs to a set of outputs. The inputs are the uncertain parameters
\vu, and the outputs are the quantities that the designer is interested in
studying. The former could be, for instance, the effective channel length, gate
oxide thickness, and threshold voltage, and the latter could be, for instance,
the temperature profile, energy consumption, and maximum temperature of the
system.

\inputfigure{chaos-overview}
The major stages of our general approach are depicted in \fref{chaos-overview}.
At Stage~1, the quantity of interest \g and the uncertain parameters \vu are
specified. The quantity is given as a ``black-box'' function that operates on a
particular outcome of the parameters. In order to evaluate \g, the designer is
implicitly required to specify the system model being considered, which also
includes a power model and a temperature model. At Stage~2, the uncertain
parameters \vu are to be transformed into independent random variables \vz,
since independence is a prerequisite of the subsequent calculations. At Stage~3,
a surrogate for \g is constructed by means of the \ac{PC} decomposition. At
Stage~4, the computed expansion is processed in order to obtain the desired
statistics about \g.

\subsection{\problemtitle}
\slab{chaos-formulation}

The problem formulation is problem specific. Thus, in this general section, we
operate on unspecified \g and \vu; more concrete discussions about Stage~1 are
postponed to later sections where we start to apply the proposed methodology to
particular problems. In addition, for convenience, \g is assumed to take values
in $\real$ here, which is generalized to $\real^n$ later on. Lastly, there is
one recurring aspect about \vu that is worth discussing before going any
further.

A description of \vu that is supplied by the designer is an input to our
probabilistic analysis. A proper way to describe a set of random variables is to
specify their joint distribution function; see \xref{probability-theory}. In
practice, however, such exhaustive information is often unavailable, which is
due mainly to the high dimensionality and intricate dependencies inherent in
real-life problems.

A more realistic assumption is the knowledge of the marginal distributions
$\set{F_i}_{i = 1}^\nu$ and correlation matrix $\correlation{\vu}$ of \vu.
However, these are not sufficient to reconstruct the joint distribution of \vu
in general. Nevertheless, it can be approximated well by accompanying the
available marginals with a carefully constructed copula \cite{nelsen2006}, which
makes \vu fully specified; see \xref{probability-theory}. Specifically, one
constructs a Gaussian copula based on $\set{F_i}_{i = 1}^\nu$ and
$\correlation{\vu}$ as a part of the Nataf transformation \cite{liu1986}, which
is introduced in \xref{probability-transformation}. This Gaussian copula is
defined in terms of an auxiliary correlation matrix.

Without loss of generality, we target the above practical scenario in our
experiments in this section and the next one. However, it should be understood
that, even though the marginal distributions and the above copula prescribe a
joint distribution for \vu, this distribution is an approximation rather than
the true one. If the joint distribution is available, it should be used instead.

\subsection{Probability Transformation}
\slab{chaos-transformation}

Mutual independence of random variables is required by \ac{PC} expansions. In
general, however, the individual variables in $\vu: \Omega \to \real^\nu$ are
dependent. Therefore, our foremost task is to transform \vu into a vector with
independent components in order to fulfill the requirement; see Stage~2 in
\fref{chaos-overview}. To this end, an adequate transformation should be
performed, depending on the available information \cite{eldred2008}. Denote such
a transformation by
\begin{equation} \elab{chaos-transformation}
  \vu = \transform{\vz}
\end{equation}
where $\vz: \Omega \to \real^\nz$ is a random vector with \nz independent
components, and $\transform: \real^\nz \to \real^\nu$. The quantity of interest
\g can now be computed as
\[
  \g(\vu) = (\g \circ \transform)(\vz) = \g(\transform(\vz)).
\]

Correlated random variables can be transformed into linearly uncorrelated ones
via the \ac{KL} decomposition described in \xref{probability-transformation} and
shown in \eref{karhunen-loeve}. In addition, if the correlated variables form a
Gaussian vector, the uncorrelated variables are mutually independent. In the
general case (non-Gaussian), the most prominent solutions for attaining
independence are the Rosenblatt transformation \cite{rosenblatt1952} and the
Nataf transformation mentioned earlier. Rosenblatt's approach is suitable when
the joint distribution of \vu is known. However, as emphasized in
\sref{chaos-formulation}, such information is rarely available. Marginal
distributions and a correlation matrix are more likely to be given, which are
sufficient for the Nataf transformation; see \xref{probability-transformation}.

Apart from the extraction of the independent variables \vz, an essential
operation at this stage is model order reduction, since the number of stochastic
dimensions---that is, the dimensionality of \vz---directly impacts the
complexity of the rest of the computations. This operation is frequently treated
as a part of the \ac{KL} decomposition, and it is also elaborated on in
\xref{probability-transformation}.

\subsection{Surrogate Construction}
\slab{chaos-construction}

In order to obtain a computationally efficient and convenient characterization
of \g, we utilize the \ac{PC} decomposition with nonintrusive spectral
projections \cite{xiu2010}. The corresponding mathematical foundation is given
in \xref{polynomial-chaos}.

Assume that \g as a function of \vz belongs to $\L{2}{\Omega, \F,
\probability}$; see \xref{probability-theory}. Then \g is expanded into the
following series at Stage~3 in \fref{chaos-overview}:
\begin{equation} \elab{chaos-expansion}
  \g \approx \chaos{\nz}{\lc}{\g}
  = \sum_{\vi \in \sparseindex{\nz}{\lc}} \hat{\g}_{\vi} \psi_{\vi}
\end{equation}
where $\lc \in \natural$ is the level of the expansion, $\vi = (i_k) \in
\natural^\nz$ is an index, $\sparseindex{\nz}{\lc}$ is an index set, and
$\set{\psi_{\vi}}{\vi \in \sparseindex{\nz}{\lc}}$ are orthonormal polynomials
in \nz variables whose orders are specified by the corresponding elements of
\vi.

It is clear that the first step toward a polynomial expansion is the choice of a
suitable polynomial basis, which is typically made based on the Askey scheme of
orthogonal polynomials \cite{xiu2010}. This step is crucial, as the rate of
convergence of \ac{PC} expansions depends on it. Although there are no rules
that guarantee the optimal choice \cite{knio2006}, there are best practices
suggesting that one should be guided by the probability distributions of the
random variables that drive the stochastic system at hand. For instance, when a
random variable follows a beta distribution, it is worth trying the Jacobi basis
first; on the other hand, the Hermite basis is preferable for Gaussian
distributions.

As shown in \eref{chaos-inner-product} and \eref{chaos-projection}, each
coefficient $\hat{\g}_{\vi}$ is an \nz-dimensional integral of the product of \g
and $\psi_{\vi}$. In general, this integral should be computed numerically as
described in \xref{numerical-integration}. Specifically, an adequate
\nz-dimensional quadrature $\quadrature{\nz}{\lq}$, which is a set of
\nz-dimensional points accompanied by a set of scalar weights, is utilized. The
result is
\begin{equation} \elab{chaos-coefficient}
  \hat{\g}_{\vi} \approx \quadrature{\nz}{\lq}{\g \psi_{\vi}}
  = \sum_{j \in \tensorindex{\nz}{\lq}} (\g \circ \transform)(\vz_j) \psi_{\vi}(\vz_j) w_j
\end{equation}
where $\lq \in \natural$ is the quadrature's level, and $\set{\vz_j} \subset
\real^\nz$ and $\set{w_j} \subset \real$ are the corresponding points and
weights, respectively, indexed by $\tensorindex{\nz}{\lq} \subset \natural$. The
operator $\quadrature{\nz}{\lq}$ is constructed via the Smolyak algorithm
\cite{smolyak1963} as shown in \eref{quadrature-sparse}. An important aspect to
note about this construction is that it is a combination of a number of
cherry-picked operators identified by a certain index set denoted by
$\sparseindex{\nz}{\lq} \subset \natural^n$. Let us discuss the content of
$\sparseindex{\nz}{\lc}$ and $\sparseindex{\nz}{\lq}$.

The standard choice of $\sparseindex{\nz}{\lc}$ in \eref{chaos-expansion} is the
isotropic total-order index set, which can be seen in
\eref{index-total-order-isotropic}. \emph{Isotropic} refers to constraining all
dimensions identically, and \emph{total-order} to the criterion used for
constraining each dimension. Since $\psi_{\vi}$ is a polynomial of total order
at most \lc, and \g is approximated by such a polynomial, the integrand in
\eref{chaos-coefficient} can be assumed to be a polynomial of total order at
least $2 \lc$. With this in mind, one typically constructs such a quadrature
that is exact for polynomials of total order up to at least $2 \lc$
\cite{eldred2008}. More generally, the index set $\sparseindex{\nz}{\lc}$, which
is used in \eref{chaos-expansion}, and the index set $\sparseindex{\nz}{\lq}$,
which implicitly determines the content of the index set
$\tensorindex{\nz}{\lq}$ used in \eref{chaos-coefficient}, should be related as
$\sparseindex{\nz}{\lc} \subseteq \sparseindex{\nz}{\lq}$.

In the case of Gaussian quadratures, which is a broad and potent class of
quadratures introduced in \xref{numerical-integration}, a quadrature of level
\lq is exact for polynomials of total order up to $2 \lq + 1$ \cite{heiss2008}.
Therefore, in this very common case, an adequate quadrature can be constructed
by ensuring that $\lq \geq \lc$.

An important generalization of the isotropic Smolyak algorithm in
\eref{quadrature-sparse} is the anisotropic Smolyak algorithm \cite{nobile2008}.
The difference between the isotropic and anisotropic versions lies in the
content of $\sparseindex{\nz}{\lq}$. In particular, the anisotropic total-order
index set is defined as follows:
\begin{equation} \elab{index-total-order-anisotropic}
  \sparseindex{\nz}{\lq} = \set{\vi}{\vi \in \natural^\nz, \innerproduct{\v{c}}{\vi} \leq \lq \, \min_{i = 1}^\nz c_i}
\end{equation}
where $\v{c} = (c_i) \in \real^\nz$ with $c_i \geq 0$ for $i = \range{1}{\nz}$
is a vector assigning importance weights to the dimensions, and
$\innerproduct{\cdot}{\cdot}$ is the standard inner product in $\real^\nz$.
\eref{index-total-order-anisotropic} plugged into \eref{quadrature-sparse}
results in a sparse grid that is exact for the polynomial subspace that is
obtained using the same index set.

The above approach allows one to exploit anisotropic behaviors that are present
in many practical problems \cite{nobile2008}. It provides fine control over the
computational cost associated with the construction of \ac{PC} expansions: a
carefully chosen importance vector $\v{c}$ in
\eref{index-total-order-anisotropic} can significantly reduce the number of
polynomial terms in \eref{chaos-expansion} and the number of quadrature points
in \eref{chaos-coefficient}, which are needed for calculating the coefficients
of those polynomial terms. The question, then, is in the choice of $\v{c}$. When
the \ac{KL} decomposition is utilized as a part of $\transform$ in
\eref{chaos-transformation}, a viable option in this regard is to rely on the
variance contributions of the dimensions given by $\set{\lambda_i}_{i = 1}^\nu$
in \eref{karhunen-loeve}. Specifically, we let
\begin{equation} \elab{chaos-anisotropic-weight}
  c_i = \left(\frac{\lambda_i}{\sum_{j = 1}^\nu \lambda_j}\right)^\gamma
\end{equation}
for $i = \range{1}{\nz}$ where $\gamma \in [0, 1]$ is a tuning parameter. The
isotropic scenario can be recovered by setting $\gamma = 0$; other values of
$\gamma$ correspond to various levels of anisotropy with the maximum attained by
setting $\gamma = 1$.

Once \vz has been identified, and \lc, \lq, and $\v{c}$ have been chosen, the
corresponding polynomial basis and quadrature stay the same for all quantities
that one might be interested in studying. This observation is very important, as
a lot of preparatory work can and should be done only once and then reused as
needed. In particular, the construction in \eref{chaos-expansion} can be reduced
to one matrix multiplication with a precomputed matrix, which we show next.

\inputalgorithm{chaos-construction}
Let $\nc = \cardinality{\sparseindex{\nz}{\lc}}$ be the cardinality of
$\sparseindex{\nz}{\lc}$, which is the number of polynomial terms and
coefficients in \eref{chaos-expansion}. Let also $\nq =
\cardinality{\tensorindex{\nz}{\lq}}$ be the cardinality of
$\tensorindex{\nz}{\lq}$, which is the number of quadrature points and weights
in \eref{chaos-coefficient}. Furthermore, assume that the index sets
$\sparseindex{\nz}{\lc}$ and $\tensorindex{\nz}{\lq}$ are given certain
orderings so that one is able to refer to their elements using one-dimensional
indices $i = \range{1}{\nc}$ and $j = \range{1}{\nq}$, respectively. Now, let
\begin{equation} \elab{chaos-projection-matrix}
  \Pi = (\psi_i(\vz_j) w_j)_{i = 1, j = 1}^{i = \nc, j = \nq}.
\end{equation}
The element of $\Pi$ on row~$i$ and column~$j$ is polynomial~$i$ evaluated at
quadrature point~$j$ and multiplied by quadrature weight~$j$. The matrix $\Pi$
is referred to as the projection matrix. Using $\Pi$, the coefficients
$\set{\hat{g}_{\vi}}{\vi \in \sparseindex{\nz}{\lc}}$ in \eref{chaos-expansion}
can now be trivially computed as follows:
\begin{equation} \elab{chaos-coefficients}
  \hat{\vg} = \Pi \vg
\end{equation}
where
\begin{align*}
  & \hat{\vg} = (\hat{\g}_i)_{i = 1}^\nc \text{ and} \\
  & \vg = ((\g \circ \transform)(\vz_i))_{i = 1}^\nq.
\end{align*}
It can be seen that this formula is a matrix version of
\eref{chaos-coefficient}. The matrix $\Pi$ is the one that should be precomputed
and stored for future use.

The pseudocode for a procedure that computes \ac{PC} expansions by leveraging
the projection matrix $\Pi$ is given in \aref{chaos-construction} where
Algorithm~G stands for a subroutine that calculates $\g \circ \transform$ for a
given \vz, which is problem specific.

Let us summarize this subsection. In order to give a probabilistic
characterization of the quantity of interest \g, we construct a polynomial
expansion of this quantity as shown in \eref{chaos-expansion}. The coefficients
of this expansion are found by means of a suitable multivariate quadrature as
shown in \eref{chaos-coefficient}. The quadrature is constructed via the Smolyak
formula given in \eref{quadrature-sparse}. The index set used in both
\eref{chaos-expansion} and \eref{quadrature-sparse} is the one given in
\eref{index-total-order-anisotropic} where the anisotropic weights are set
according to \eref{chaos-anisotropic-weight}. For computational efficiency, the
projection matrix defined in \eref{chaos-projection-matrix} is to be calculated,
stored, and used as illustrated in \aref{chaos-construction}.

\subsection{Post-Processing}
\slab{chaos-processing}

Due to the properties of the \ac{PC} decomposition---in particular, the
orthogonality of the basis discussed in \xref{polynomial-chaos}---the obtained
polynomial representation in \eref{chaos-expansion} allows various statistics
about \g to be estimated with little effort, which is the subject of Stage~4 in
\fref{chaos-overview}. The reason that this estimation is straightforward is
that the function given in \eref{chaos-expansion} is nothing more than a
polynomial; hence, it is easy to interpret and easy to evaluate.

Let us find, for example, the expectation and variance of \g. Since the first
polynomial $\psi_{\v{0}}$ in a normalized polynomial basis is unity by
definition \cite{xiu2010},
\[
  \expectation{\psi_{\v{0}}} = 1.
\]
Hence, using the orthogonality property in \eref{chaos-orthogonality}, we
conclude that
\[
  \expectation{\psi_{\vi}} = 0
\]
for $\vi \in \sparseindex{\nz}{\lc} \setminus \set{\v{0}}$. Consequently, the
expected value and variance of \g have the following straightforward
expressions:
\begin{equation} \elab{chaos-moments}
  \begin{split}
    & \expectation{\g} = \hat{\g}_{\v{0}} \text{ and} \\
    & \variance{\g} = \sum_{\vi \in \sparseindex{\nz}{\lc} \setminus \set{\v{0}}} \hat{\g}_{\vi}^2,
  \end{split}
\end{equation}
respectively. It can be seen that the \ac{PC} decomposition provides analytical
formulae, based solely on the coefficients, for these probabilistic moments.

The \ac{CDF} and \ac{PDF} of \g can be estimated by a sampling method applied to
\eref{chaos-expansion}, which is typically followed by kernel density estimation
\cite{hastie2013} or a similar technique. The sampling in this context can be
better understood by rewriting \eref{chaos-expansion}, which is given in terms
of operators, as follows:
\[
  (\g \circ \transform)(\vz) \approx \chaos{\nz}{\lc}{\g}(\vz)
  = \sum_{\vi \in \sparseindex{\nz}{\lc}} \hat{\g}_{\vi} \psi_{\vi}(\vz).
\]
Here the aforementioned operators are applied to an outcome of \vz drawn from
the corresponding distribution. Consequently, each sample is a trivial
evaluation of a polynomial; therefore, sampling methods are computationally
cheap in this case. Furthermore, probabilities of various events can be
estimated in a similar way, and global and local sensitivity analysis of
deterministic and stochastic quantities can be readily conducted on the
expansion.

\begin{remark} \rlab{chaos-multidimensional-output}
The development given in this section remains valid even when \g is a
multidimensional quantity from the standpoint of the number of outputs. In this
case, it is convenient to consider \g as a row vector with an appropriate number
of elements. All the operations that involve \g---such as those given in
\eref{chaos-expansion}, \eref{chaos-coefficient}, and
\eref{chaos-moments}---should then be performed elementwise. In
\eref{chaos-coefficients} and \aref{chaos-construction}, \vg and $\hat{\vg}$ are
treated as matrices with \nc rows, and $\g_i$ as a row vector. The output of
Algorithm~G in \aref{chaos-construction} is assumed to be automatically reshaped
into a row vector.
\end{remark}

In the following, we apply the uncertainty analysis presented here to a number
of concrete problems, namely transient and dynamic steady-state power and
temperature analysis as well as reliability analysis and optimization.


%!TEX root =  ../main.tex

\objective{Use derivative and integrals to calculate mean, max, and min}

\subsection{Extrema}
Most often, real-life models of continuously changing outputs (which graph as curved lines)
are useful to find maximums and minimum.  Zeros are important --- all the more so when
they appear in the denominator --- but extrema are typically more.  (Extrema is collective
term for both maximum and minimums.)  Calculus is extremely useful in finding these moments.

\index{derivative!test}
This about the some continuous, smooth curve.  What is the derivative measuring?  It is the
slope of the function at every point.  If it is positive, the function is increasing.  If it is
negative, the function is decreasing.  If it is constant, the function is level.  Even if this happens
only for an instant, it is noteworthy.  But there are three things which might be happening
\begin{enumerate}
\item the function might have been decreasing and is switching to increasing
\item the function might have been increasing and is switching to decreasing
\item the function may be pausing in its ascend or descend, but will resume forthwith
\end{enumerate}

Consider $x^3$ vs. $x^2$.  Both functions have a derivative of 0 at $x=0$.  Only one has
an extremity, while the other has a saddle point.  The key to telling the difference numerically
is to see what the rate of change of the rate of change is doing, i.e. the second derivative.
Just as the first derivative tells us about the slope of a function, the second derivative tells us about
concavity or convexity.  A concave region of a smooth, continuous function will have a minimum,
while a convex region will have a maximum.  So, unless we know the shape beforehand,
it is necessary to determine the second derivative too --- not just the first --- in order to know
where the extrema are.

\personfeature[-3in]{\chapdir/pics/GodfreyKneller-IsaacNewton-1689}{Isaac Newton}{1642-1726}{
was an English mathematician, astronomer, and physicist who is widely recognized as
one of the most influential scientists of all time and a key figure in the scientific revolution.
The second fundamental theorem of calculus is also called the Newton–Leibniz axiom.
\href{https://en.wikipedia.org/wiki/Isaac_Newton}{Wikipedia}}

\subsection{Average Value}
The anti-derivative of a function tells us the area under the curve.  When done over an open
interval (called a definite integral) this has a remarkable side effect: it also tells us the average
value of function, no matter how curvy.  Normally we think of averages as adding up a finite set
of values, and dividing by the number of quantities just added.  Strangely, this intuition works,
even over an infinite number of heights.

Consider the function $f(x)=-12(x^3-2x^2-11x+12)$.  If we are interested in the extrema, they
occur when the derivative is 0.  The derivative is $-36x^2-48x-11$, which requires the Q.F. to
solve.  It's zeros are $\frac{2\pm\sqrt{37}}{3}$, or approximate .  The heights of the function
are $\frac{8}{9}(37\sqrt{37}\mp55)$ respectively.  But what about the area between each
``hump'' and the $x$-axis?  For that we need the integral, which is $-3x^4+8x^3+66x^2+144x+C$.
Plugging in the zero, we find that $\int_1^4 f(x) = 297$ and $\int_{-3}^1 f(x) = -640$.

If we look at this right area, we see that we know its minimum height, it's maximum height, and
it's area.  There must exist some average height of the function, because we can imagine a rectangle
with a base of 3 and that height, which has an area of 297.  By simple division, that height must be
99.  The average value of the continuous curve $f(x)$ from 1 to 4, is 99.

\begin{figure}[h]
\begin{centering}
\includegraphics[scale=0.5]{\chapdir/pics/averageintegral}\index{integral!to the average}
\caption[Integrals and Averages]{A rectangle with height at the function's average value has the same area as the integral over the same width.}
\end{centering}
\end{figure}

How can this be?  Imagine if we cut the region up into finitely many pieces.  The average height
is equal to the sum of the heights at every point in our sample group, divided by the number of
samples, $n$, that we have cut the width up into.
In this case, let us write this sum using the integral sign.

$$
\cfrac{\text{The sum of the heights of} f(x) \text{every} \frac{4-1}{n}}{\frac{4-1}{n}}
$$

But we really ought to use the integral sign, because we don't actually want to cut up the region into
a finite number ($n$) of slices, but an infinite number, with each step being no more than $dx$ wide.
This means we should've written

$$
\cfrac{\int_1^4 f(x)}{\frac{4-1}{dx}} = \frac{\int_1^4 f(x)dx}{4-1}
$$

Because dividing twice is the same as multiplying, our infinitely many additions in the numerator
has suddenly because an infinite number of areas, with infinitely many rectangles being brought
together and summed up --- integrated, each one the height of the function times the infinitely
narrow width $dx$.  In other words, the definite integral over a region, divided by the width of
the region equals the average height of the region.

$$
\frac{\int_a^b f(x)}{b-a} = \frac{F(b)-F(a)}{b-a} = \text{average height}
$$

This is also known as the Fundamental Theorem of Calculus.\index{Fundamental Theorem of Calculus}


\section{Permutations}

    \subsection{What are Permutations?}
    Permutations are the arrangements of objects in a set order.
    For instance, how many ways can you rearrange the letters in the word "APPLE",
    or how many ways can you bring 4 people out of 20 friends to a party?

    \subsection{Factorial Notation}
    Often in permutations, we need to multiply descending consecutive numbers.
    Since these numbers are descending and not ascending, we can safely assume that \textbf{every factorial must be positive}.
    For instance,  $5\cdot4\cdot3\cdot2\cdot1$ can often be simplified and written as \textbf{5!}, or 5 factorial.\\
    The general rule for factorial notation is:
    \begin{equation*}%Factorial Equation
        n! = n\cdot(n-1)\cdot(n-2)\cdots3\cdot2\cdot1
    \end{equation*}
    The exception to this rule is 0!, which in this case would be $0! = 1$, which would follow the same thinking as $n^0=0$.

    \subsection{Rule of Product and Rule of Sum}
    How many different results can one get from flipping a coin 3 times?
    You have 2 possibilities, heads or tails, and you flip it 3 times.
    Essentially, you have three groups of two, or $2\cdot3$ different results, which turns out to be $6$ different possible results.
    This is called the \emph{Fundamental Rule of Product} or the \emph{Counting Principle}
    \begin{definition}
        The Rule of Product states that if the $1^{st}$ action can be performed in n ways, and the $2^{nd}$ action can be performed in m ways.
        They can be performed together in $n\cdot m$ ways.
    \end{definition}
    What happens if the actions cannot be performed together? Such as which car to buy? In that case we have another rule, known as the \emph{Rule of Sum}.
    \begin{definition}
        The Rule of Sum states that if the $1^{st}$ action can be performed in n ways, and the $2^{nd}$ action can be performed in m ways, and these actions cannot occur together, then there are $n+m$ ways for either of the actions to occur.
    \end{definition}

    \subsection{N-Arrangements}
    Let's say you have 5 people, and you need to arrange them in a line.
    You put one person in the first spot, and now you have 4 left. Repeat this until you are left with one person, which fits into the last spot.
    You can express this mathematically by writing out $5!$ .\\
    This is quite similar to the Rule of Product as:
    \begin{enumerate}
        \item Each placement is a choice
        \item The number of choices is reduced with each previous action.
    \end{enumerate}
    A permutation of \emph{n} objects is an arrangement of the objects in a \textbf{definite order}.
    This can be expressed with permutation notation, shown below where P(n,n) equals the number of ways to permute n objects.
    \begin{equation*}
        P(n,n) = n!
    \end{equation*}
    This equation will only work when the number of spaces to occupy and the number of objects are equal.
    In other cases we'll have to use an r-arrangement.

    \subsection{R-Arrangements}
    If you have more objects than places you can put the objects into, then we can use something called an r-arrangement.
    An r-arrangement is a permutation of \emph{n} objects taken \emph{r} at a time, as is shown below.
    \begin{equation*}
       P(n,r) = \frac{n!}{(n-r)!}
    \end{equation*}
    Where $P(n,r)$ equals the total number of arrangements you have, divided by the number of unused objects.
    This is most commonly displayed on calculators as \emph{nPr}.
    For example you have 4 subjects, but you can only study two at a time.
    The equation for that would be $\frac{4!}{(4-2)!}=12\mbox{ combinations.}$

    \subsection{Permutations with Repeating Elements}
    All previous examples will ONLY work with non-repeating elements, such as the letters in the word "MONTREAL", or 5 different books.
    They will definitely NOT work for questions with repeating elements, such as the word "FIJI".
    For all repeating elements we'll have to follow another rule shown below.
    \begin{definition}
        In general, the number of arrangements of \emph{n} objects of which \emph{a} of one kind are alike, and \emph{b} of one kind are alike and so forth is given by the expression below, Where the number of permutations can be found by dividing the total number of possible combinations by the repeating elements.
        \begin{equation*}
            \frac{n!}{a!b!c!\cdots}
        \end{equation*}
    \end{definition}

    \subsection{Problem Solving with Permutations}
    There are 3 methods for solving problems with permutations. They are called: the Indirect Method, the Case Method, and Circular Arrangements.

        \subsubsection{Indirect Method}
        The indirect method involves taking all possibilities and subtracting those which are not wanted.
        For instance, say you have 3 people that need to sit at a table, but 2 of them don't want to sit beside each other.
        There is a step-by-step process that you can use for this question that is explained below.
        \begin{enumerate}
                \item Find all possible permutations (ex. P(3,3) ),
                \item Find all unwanted permutations (in this case when they are together),
                \item Subtract the unwanted permutations from the wanted permutations.
            \end{enumerate}
        This can be written as:
        \begin{equation*}
            \mbox{\textbf{Wanted Outcomes}} = \mbox{Total Outcomes}-\mbox{Unwanted Outcomes}
        \end{equation*}

        \subsubsection{Case Method}
        The Case Method involves breaking down the equation into manageable parts, then adding them to get the final solution.
        There is a step-by-step example problem in Section 1 of the appendix, along with an explanation of why, and how you manage the "breaking into parts" aspect.

        \subsubsection{Circular Arrangements}
        Circular arrangements are no longer a part of the curriculum, and as such they will not be tested on, but they are still quite useful to know in niche cases.
        If you have seven people seated at a table, and they all move to their rights, their positions may have moved, but their order remains the same, like a circle.
        Keep in mind that if there is some fixed point, the question just ends into a typical linear arrangement, with the equation simplifying to $1 \cdot (n-1)!$.