\section{Introduction}

One of the important scientific computing problems is computing the real roots
of polynomials. It has wide applications in computer graphics. For the
polynomials with low order, like quadratic or cubic, we can use formulas to get
roots directly. However, according to the Abel–Ruffini theorem, there is no
solution in radicals to general polynomial equations of five degrees or higher
with arbitrary coefficients\cite{Abel-Ruffini}.

There are a lot of algorithms to find the real roots of polynomials. Although
most root-finding algorithms, Newton’s method for example, may produce some real
roots, the convergence of algorithms are not guaranteed. Furthermore, these
methods cannot generally certify having found all real roots. Which means if
such methods do not find any root, one cannot know if there are real roots or
not. Moreover, there are some cases where one does not need the exact roots of
polynomial equations. Take ray tracing as an example. When computing the
distance from intersection point to ray’s end point, the object with smallest
distance is important to us, instead of exact distance to point of intersection.
In these cases, computing exact roots is not very essential.

Real-root isolation is an appropriate method for the above problems. It will
generate a sequence of disjoint intervals. Each interval contains only one real
root of the polynomial. And combining these intervals together, all real roots
could be found. Besides that, isolating the roots instead of computing them out
might speed up the cases that do not need exact roots, as mentioned above.

This project aims to implement a real-root isolation program based on Budan’s
theorem and continued fraction. These two methods are based on Descartes’ rule
of signs, which describes how to get information on the number of positive real
roots of a polynomial and first introduced by René Descartes\cite{rule_of_sign}. Budan’s theorem,
developed from Descartes’ rule of signs, provides the methods to bound the
number of real roots of a polynomial in an interval\cite{Budan}. Vincent introduced the
continued fraction method in his work in 1834\cite{Vincent}. Both methods work only on
square-free polynomials. Therefore, this project takes Yun’s algorithm to
perform square free decomposition\cite{Yuns}.

The organization of this report is as follows. Section \ref{methods} will introduce the
theory basis of this project. Section \ref{implementation} describes how the program implemented
and what have been tried to optimize. Running time comparison and analysis of
how error propagation through computation will be discussed in Section
\ref{analysis}. Finally, a conclusion is drawn in Section \ref{conclusion}.


% ------------------------------------------------------ %
%  BIOE60011 Probability and Statistics For Bioengineers %
%  Formula Booklet for Practice and Exams %
% ----------------------------------------------------- %
% Author:
%           Binghuan Li  <binghuan.li19@imperial.ac.uk>,
%           Peter Xie    <peter.xie19@imperial.ac.uk>
% ------------------------------------------------------ %
%  Reviewer:
%           Zhuang Liu, Bishr Al-Badri,
%           Eliott Stoclet, Rea Tresa
% ------------------------------------------------------ %
%  Update Date: 29th December 2021  %
% ------------------------------------------------------ %
%  Graph credentials by Dr Joseph van Batenburg-Sherwood %
% ------------------------------------------------------ %
% This work is licensed under a Creative Commons Attribution 4.0 International License %
% ------------------------------------------------------ %

% keep the font size by default
\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

% set font
\usepackage[sfdefault]{arimo}

% set page margin
\usepackage[top=2cm, bottom=1cm, left=2cm, right=2cm]{geometry}

% set math env
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{mathrsfs}
\usepackage{mathtools}
\usepackage{amssymb}
\usepackage{changepage}
% set graphics env
\usepackage{float}
\usepackage{graphics}

% table scaling
\usepackage{adjustbox}

% write pdf info
\usepackage{hyperref}
%\hypersetup{hidelinks}
\hypersetup{pdfauthor={Li, Binghuan},
            pdftitle={Probs-and-Stats, Formula Sheet},
            pdfsubject={last update date: \today},
            }

% env to insert external pdf
\usepackage{pdfpages}

\usepackage{booktabs}

\usepackage{framed}

% set section title size
\usepackage{titlesec}
\titleformat*{\section}{\large\bfseries}
\titleformat*{\subsection}{\normalsize\bfseries}

% caption setting env
\usepackage{caption}
\captionsetup[table]{labelformat=empty}

% set header/footer
\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\lhead{\textit{Probability and Statistics for Bioengieering - Formula Sheet}}
\rhead{\thepage}

%% start the doc!
\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Mean, Median and Standard Deviation}

\begin{table}[H]
    \centering
    \begin{adjustbox}{width=\columnwidth,center}
    \begin{tabular}{c| c| c | c}
        Mean & Median & Standard Deviation & MAD\\ [0.7ex] \hline
        $\displaystyle \bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_{i} = \frac{x_{1}+x_{2}+...+x_{n}}{n}$ &
        $\Tilde{x} = \begin{cases}
        x(\frac{n+1}{2}) \\
        0.5(x(\frac{n}{2})+x(\frac{n}{2}+1))
        \end{cases}$ &
        $\displaystyle s = \sqrt{\frac{1}{n-1}\sum_{i=1}^{n} (x_{i}-\bar{x})^{2}}$
        & $\displaystyle MAD = k|\widetilde{x_{i} - \hat{x}}|$ \\
          &  &  &  $k = 1.4826$\\
    \end{tabular}
    \end{adjustbox}
\end{table}

\begin{itemize}
    \item[-] $s$, $s^{2}$ and $\bar{x}$ denote \underline{sample} S.D., variance and mean.
    \item[-] $\sigma$, $\sigma^{2}$ and $\mu$ denote \underline{population} S.D., variance and mean (expected value).\\

    \item[-] \textbf{Trimmed Mean at $p\%$} - Remove p/2 percentile from both ends of the data and calculate the mean
    \item[-] \textbf{Standard Error on the mean} -
    $\displaystyle S_{\bar{X}} = \frac{S_{X}}{\sqrt{n}}$, where $n$ is the sample size, $S_{X}$ is the standard deviation
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Probability Theory}
\subsection*{Set Theory}
\begin{table}[H]
    \centering
    \begin{adjustbox}{width=\columnwidth,center}
    \begin{tabular}{c| c| c |c}
        Union & Intersection & Complement & Relative Complement\\ [0.7ex] \hline \vspace{1ex}
       A \underline{OR} B & A \underline{AND} B & \underline{NOT} A & A \underline{AND} \underline{NOT} B \\
       \includegraphics[width=0.2\textwidth]{./images/AorB.pdf}&\includegraphics[width=0.2\textwidth]{./images/AandB.pdf}&\includegraphics[width=0.2\textwidth]{./images/notA.pdf}&\includegraphics[width=0.2\textwidth]{./images/Aand_notB.pdf}\\
       $P(A\cup B) = P(A)+P(B)-P(A\cap B)$ &&$P(A') = 1 - P(A)$ &
    \end{tabular}
    \end{adjustbox}
\end{table}

%%
\subsection*{Counting}
\begin{itemize}
\item Product Rule - $\displaystyle n = n_{1}\times n_{2}\times n_{3}\times...\times n_{k}$
    \item Permutations: Ordered, dependent selections (without replacement) -
    $\displaystyle P_{k,n} = \frac{n!}{(n-k)!}$
    \item Combinations: Unordered selections (without replacement) -
    $\displaystyle C_{k,n} = \frac{P_{k,n}}{k!} = {n \choose k} = \frac{n!}{k!(n-k)!}$
\end{itemize}

%%
\subsection*{Conditional Probability}
\begin{itemize}
    \item Conditional probability: knowing that A has happened, the probability of B given A - $\displaystyle P(B|A) = \frac{P(B \cap A)}{P(A)}$
    \item Bayes Theorem - $\displaystyle P(A_{j}|B) = \frac{P(B|A_{j})P(A_{j})}{\sum_{i=1}^{k}P(B|A_{i})P(A_{i})} $\\\\
    (Applies if events A are mutually exclusive and exhaustive (defines the whole set)).
\end{itemize}

%%
\subsection*{Independence}
\begin{itemize}
    \item Independent: event A is unaffected by event B
    \[P(A|B) = P(A), \quad\quad P(B|A) = P(B), \quad\quad P(A\cap B)=P(A)P(B)\]
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Probability Distributions and Discrete Random Variables}
\subsection*{Binomial and Poisson Distribution}
\begin{minipage}{.8\textwidth}
\begin{table}[H]
    \centering
    \begin{tabular}{l c c}
        & Binomial Distribution & Poisson Distribution\\ [0.7ex] \hline \vspace{1ex}
        Probability Density Function & $ \displaystyle b(x;n,p) ={ n \choose x} p^{x}(1-p)^{n-x}$  & $\displaystyle po(x,\lambda) = \frac{e^{-\lambda}\lambda^{x}}{x!}$  \\ [0.5ex]
        Expected Value, $E(x)$ & $E(x) = np$ & $E(x) = \lambda$\\ [0.5ex]
        Variance, $V(x)$ & $\displaystyle V(x) = np(1-p) = npq$ & $\displaystyle V(x) = \lambda(1-\frac{\lambda}{n})$  \\ [0.5ex]
    \end{tabular}
\end{table}
\end{minipage}
\begin{minipage}{.2\textwidth}
\begin{framed}
Note: \\\\
$V(x)$ can also be expressed as $V(x)=E(x^{2})-(E(x))^{2}$.
\end{framed}

\end{minipage}

%%
\subsection*{Discrete Random Variables}
% \begin{adjustwidth}{-100pt}{-100pt}
\begin{table}[H]
    \centering
    \begin{adjustbox}{width=\columnwidth,center}
    \begin{tabular}{c| c| c}
        Bernoulli RV & Expected Value & Variance\\ [1.0ex] \hline
        $p(x;\alpha) = \begin{cases}
        1-\alpha, & if \ x=0 \\
        \alpha, & if \ x=1\\
        0, & otherwise
        \end{cases}$
        &
        $\displaystyle E(x)=\mu_{X}=\sum_{x\in D}xp(x)$
        &
        $\displaystyle V(X) = \sigma_{X}^{2}=\sum_{x\in D}(x-\mu_{X})^{2}p(x)=E[(X-\mu_{X})^{2}]$
    \end{tabular}
    \end{adjustbox}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Continuous Random Variables and Joint Probability}
\subsection*{Random Variables}
\begin{table}[H]
    \centering
    \begin{tabular}{|l|c|}
    \hline
     \addlinespace[.3ex] Probability density function & $\displaystyle P(a \leq X \leq b) = \int_{a}^{b}f(x)dx$ \\ [.5ex] \hline
    \addlinespace[.3ex] Cumulative distribution function & $\displaystyle F(x) = P(X\leq x) = \int_{-\infty}^{x} f(y)dy \  \text{and} \ \int_{-\infty}^{+\infty} f(x)dx = 1$ \\ [.5ex] \hline
    \addlinespace[.3ex] Expected value & $\displaystyle \mu_{X} = E(X) = \int_{-\infty}^{+\infty} xf(x) dx$ \\ [.5ex] \hline
    \addlinespace[.3ex] Variance & $\displaystyle \sigma_{X}^{2} = V(X) = \int_{-\infty}^{+\infty} (x-\mu)^{2}f(x) dx$ \\ [.3ex] \hline
    \end{tabular}
\end{table}

\subsection*{Joint Probability Distributions}
\begin{itemize}
    \item Discrete Joint random variables
    \begin{table}[H]
    \centering
    % \begin{adjustbox}{width=\columnwidth,center}
    \begin{tabular}{c| c| c}
        Joint probability mass function (PMF) & Expected Value & Marginal PMFs\\ [0.7ex] \hline
        \addlinespace[.4ex] $p(x,y)=P(X=x$ and $Y=y)$
        &
        $\displaystyle E[h(X,Y)]=\sum_{X}\sum_{Y}h(x,y)p(x,y)$
        &
        $P_{X}(x)=\sum_{y}p(x,y)$\\

        $\sum_{x}\sum_{y}p(x,y)=1$
        &
        &
        $P_{Y}(y)=\sum_{x}p(x,y)$\\
    \end{tabular}
\end{table}

    \item Continuous Joint random variables
    \begin{table}[H]
    \centering
   \begin{adjustbox}{width=\columnwidth,center}
    \begin{tabular}{c| c| c}
        Joint probability mass function (PDF) & Expected Value & Marginal PDFs\\ [0.7ex] \hline
        \addlinespace[.4ex] $\displaystyle P[(X,Y)\in A]=\int\int_{A} f(x,y) dx dy$
        &
        $\displaystyle E[h(X,Y)]=\int_{-\infty}^{+\infty}\int_{-\infty}^{+\infty}h(x,y)f(x,y)dx dy$
        &
        $\displaystyle f_{X}(x)=\int_{-\infty}^{+\infty} f(x,y)dy$ \\ [2ex]

        &
        $\displaystyle P[a\leq X\leq b, c \leq Y \leq d]=\int_{c}^{d}\int_{a}^{b}f(x,y)dx dy$
        &
        $\displaystyle f_{Y}(y)=\int_{-\infty}^{+\infty} f(x,y)dx$
    \end{tabular}
    \end{adjustbox}
\end{table}
\end{itemize}

\vspace{-1cm}
\subsection*{Independence}
$X$ and $Y$ are independent if for every pair $x, y$:
\begin{itemize}
    \item Discrete: \quad $p(x,y) = p_{x}(x)p_{y}(y)$
    \item Continuous: \quad $f(x,y) = f_{x}(x)f_{y}(y)$
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Normal Distribution}

\begin{minipage}{0.7\textwidth}
\begin{itemize}
    \item Probability Density Function (pdf):\[f(x) = \frac{1}{\sqrt{2\pi \sigma^{2}}}e^{-\frac{(x-\mu)^{2}}{2\sigma^{2}}}\]
    \item Cumulative Distribution Function (cdf): \[F(X) = \frac{1}{2}\bigg[ 1+ \text{erf}\bigg( \frac{x-\mu}{\sqrt{2\sigma^{2}}} \bigg) \bigg]\]
    where \[\text{erf}(x) = \frac{2}{\sqrt{\pi}} \int_{0}^{x} e^{-t^{2}}dt\]
\end{itemize}
\end{minipage}\hfill
\begin{minipage}{0.3\textwidth}
\begin{table}[H]
    \centering
    \begin{tabular}{l c}\hline
        Discrete & Continuous \\ \hline
        $X=A$ & $A-0.5 < Y < A+0.5$\\
        $X\leq A$ & $Y<A+0.5$\\
        $X<A$ & $Y<A-0.5$ \\
        $X\geq A$ & $Y>A-0.5$ \\
        $X>A$ & $Y>A+0.5$ \\
    \end{tabular}\caption{Continuity Correction - Normal approximation of discrete distributions (e.g. Binomial)}
\end{table}
\end{minipage}

%%
\subsection*{Standard Normal Distribution}
Standard normal distribution is a special case of normal distribution with $\mu = 0$ and $\sigma =1$.
\begin{table}[H]
    \centering
    \begin{tabular}{c|c}
        Probability Density Function (PDF) &
        Cumulative Distribution Function (CDF)\\ \hline
        $\displaystyle f(Z) = \frac{1}{\sqrt{2\pi }}e^{-\frac{z^{2}}{2}}$ &
        $\displaystyle \Phi(z) = F(Z) = \frac{1}{2}\bigg[ 1+ \text{erf}\bigg( \frac{z}{\sqrt{2}} \bigg) \bigg]$ and \ $\displaystyle Z = \frac{X-\mu}{\sigma}$
    \end{tabular}
\end{table}

%%
\subsection*{Central Limit Theorem}
For sufficiently large n:
\begin{itemize}
    \item Mean sample mean is the population mean - $\displaystyle E[\bar{X}] = \mu_{\bar{X}} = \mu$
    \item Sample mean variance decreases as sample size increase - $\displaystyle  V[\bar{X}] = \sigma_{\bar{X}}^{2} = \frac{\sigma^{2}}{n}$
    \item Standard error on the mean (SEM) is the SD of sample means - $\displaystyle \sigma_{\bar{X}} = \frac{\sigma}{\sqrt{n}}$
\end{itemize}

%%
\subsection*{Margin of Error, Confidence Intervals}
Unknown $\sigma$, large $n$.
\begin{table}[H]
    \centering
    \begin{tabular}{c|c}
        Confidence interval (CI) &
        Margin of Error (ME)\\ \hline
        $\displaystyle CI = \bar{X} [\bar{X_{l}}, \bar{X_{u}}] \ =\  \bigg[\bar{X}-Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}},\ \bar{X}+Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}\bigg]$ &
        $\displaystyle ME = \bar{X}\pm ME_{\bar{X}} = \lvert Z_{\frac{\alpha}{2}} \rvert \frac{\sigma}{\sqrt{n}}$
    \end{tabular}
\end{table}

%%
\subsection*{$t$-distribution}
Unknown $\sigma$, small n. Test statistic - $\displaystyle T=\frac{\bar{X}-\mu}{S/\sqrt{n}}$.
\begin{table}[H]
    \centering
    \begin{tabular}{c|c}
        Confidence interval (CI) &
        Margin of Error (ME)\\ \hline
        $\displaystyle CI = \bar{X} [\bar{X_{l}}, \bar{X_{u}}] \ =\  \bigg[\bar{X}-t_{\lvert{\frac{\alpha}{2},n-1} \rvert}\frac{s}{\sqrt{n}},\ \bar{X}+t_{\lvert{\frac{\alpha}{2},n-1}\rvert}\frac{s}{\sqrt{n}}\bigg]$ &
        $\displaystyle ME = \bar{X}\pm ME_{\bar{X}} = \lvert Z_{\frac{\alpha}{2},n-1} \rvert \frac{s}{\sqrt{n}}$
    \end{tabular}
\end{table}
%%
\subsection*{$\chi^{2}$-distribution and $F$-distribution}
\begin{itemize}
    \item $\chi^{2}$-distribution is obtained by squaring normal distribution.
    \item For $n$ observations with unknown $\mu$ and $\sigma$, test statistic - $\displaystyle \frac{(n-1)S^{2}}{\sigma^{2}} $

    \item $F$-statistic is the ratio of two variances (see ANOVA).\[F(\nu_{1}, \nu_{2}) = \chi_{1}^{2}(\nu_{1})/ \chi_{2}^{2}(\nu_{2})\]
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Hypothesis Testing}
%%
\subsection*{NHST Types of Errors}
\begin{minipage}{0.5\textwidth}
\begin{table}[H]
    \begin{tabular}{|c|c|c|}\hline
        $\mathbf{H_{0}}$ & \textbf{True} & \textbf{False} \\\hline
        \textbf{Do not reject} & True negative, $1-\alpha$ & False negative, $\beta$\\ \hline
        \textbf{Reject} & False positive, $\alpha$ & True positive, $1-\beta$\\ \hline
    \end{tabular}
\end{table}
\end{minipage} \hfill
\begin{minipage}{0.45\textwidth} \vspace{.2cm}
\begin{itemize}
    \item $\alpha$ denotes the probability of type I error.
    \item $\beta$ denotes the probability of type II error.
\end{itemize}
\end{minipage}

\subsection*{$z$-test \& $t$-test}
\begin{itemize}
\item Assumptions:
    \begin{itemize}
        \item{Data are sampled from a normal distribution. Often lognormal distribution is more appropriate (Use SW or AD test to confirm)}
        \end{itemize}
    \item Null Hypothesis $H_{0}$: \begin{itemize}
        \item one sample: no differences between the means and a population mean.
        \item two sample: no differences between the two sample means.
        \item p-value is an estimate of the probability of getting a test-statistic more extreme. Reject $H_{0}$ if $p \leq \alpha$.
    \end{itemize}
\end{itemize}
\begin{table}[H]
    \centering
    \begin{tabular}{l | c | c}\hline
    & Expression & Notes \\ \hline
    $z$-test statistic &   $ \displaystyle Z = \frac{\bar{X}-\mu}{\sigma / \sqrt{n}}$ & $\displaystyle \frac{\text{effect}}{\text{error}}$, same below \\ [1.2em] \hline

    $t$-test one-sample statistic &   $ \displaystyle T = \frac{\bar{X}_{B}-\mu_{A}}{s_{B}/\sqrt{n}}$ & $v = n-1$\\ [1.2em] \hline

    $t$-test two-sample statistic &   $ \displaystyle T = \frac{\bar{X}_{B}-\bar{X}_{A}}{\sqrt{\frac{S_{A}^{2}}{n_{A}}+\frac{S_{B}^{2}}{n_{B}}}}$ & assume similar variances, $v = n_{A}+n_{B}-2$\\ [1.5em] \hline

    Welch-Satterthwaite Equation  & $\displaystyle \nu = \frac{(s_{\bar{X}_{1}}^{2}+s_{\bar{X}_{2}}^{2})^{2}}{\frac{s_{\bar{X}_{1}}^{4}}{n_{1}-1}+ \frac{s_{\bar{X}_{2}}^{4}}{n_{2}-1}}$ & $s_{\bar{X}_{1}}^{2} = \frac{s_{1}^{2}}{n_{1}}, \quad s_{\bar{X}_{2}}^{2} = \frac{s_{2}^{2}}{n_{2}}$
    \end{tabular}
\end{table}
\begin{figure}[H]
    \centering
    \includegraphics[width=.75\textwidth]{./images/zTest_and_tTest.pdf}
\end{figure}

%%
\vspace{-2cm}
\subsection*{ANOVA}
Assumptions:
\begin{itemize}
\item{Data are sampled from a normal distribution. Often lognormal distribution is more appropriate (Use SW or AD test to confirm)}
\item{All groups have similar variance (homoscedastic) (Use Levene’s or Bartlett’s test to confirm)}
\item{Hypothesis $H_{0}$: no differences between the means of any of the groups}
\end{itemize}
%%
\subsection*{One-way ANOVA}
\begin{itemize}
    \item Sample means and grand mean:
        \begin{table}[H]
        \centering
        \begin{tabular}{c|c}
        Sample Means & Grand Mean \\ \hline
        $\displaystyle \bar{X}_{i} = \frac{1}{J} \sum_{j=1}^{J}X_{i, j}$
        &
        $\displaystyle \bar{X} =
        \frac{1}{N}\sum^{I}_{i=1}\sum^{J}_{j=1}X_{i,j} = \frac{1}{I}\sum_{i=1}^{I}\bar{X}_{i}$\\
         \end{tabular}
        \end{table}
        \begin{itemize}
            \item $I$ denotes the number of groups
            \item $J_{i}$ denotes the number of observations for a given group.
            \item $N$ is the total number of observations, $N = \sum_{i=1}^{I}J_{i}$.
            \item $X_{i,j}$ is a random variable that denotes the $j^{th}$ observation from the $i^{th}$ group.
        \end{itemize}

    \item Sums of Squares: $SS_{T} = SS_{R}+SS_{M}$
        \begin{table}[H]
        \centering
        \begin{tabular}{c|c|c|c}
        & Sum of Squares & DoF & Notes \\ \hline
        Total & $\displaystyle SS_{T} = \sum_{i=1}^{I}\sum_{j=1}^{J_{i}}(X_{i,j}-\bar{X})^{2}=(N-1)S_{grand}^{2}$ & $\displaystyle \nu_{T}=N-1$ & $\displaystyle S^{2}_{grand} = \frac{1}{N-1}\sum^{I}_{i=1}\sum_{j=1}^{J_{i}}(X_{i,j}-\bar{X})^{2} $\\ \hline

        Model & $\displaystyle SS_{M} = \sum^{I}_{i=1}J_{i}(\bar{X}_{i}-\bar{X})^{2}$ & $\displaystyle \nu_{M}=I-1$\\ \hline

        Residuals & $\displaystyle SS_{R} = \sum_{i=1}^{I}\sum_{j=1}^{J_{i}}(X_{i,j}-\bar{X}_{i})^{2}=\sum_{i=1}^{I}S_{i}^{2}(J_{i}-1)$ & $\displaystyle \nu_{R} = N-I$ & $\displaystyle S_{i}^{2} = \frac{\sum_{j=1}^{J_{i}}(X_{i,j}-\bar{X}_{i})^{2}}{J_{i}-1}$\\
         \end{tabular}
        \end{table}

   \item Mean square values and test-statistic:
   \[MS_{M} = \frac{SS_{M}}{\nu_{M}} = \frac{1}{I-1}\sum^{I}_{i=1}J_{i}(\bar{X}_{i}-\bar{X})^{2} \quad \quad \quad  MS_{R} = \frac{SS_{R}}{\nu_{R}} = \frac{1}{N-I} \sum_{i=1}^{I}\sum_{j=1}^{J_{i}}(X_{i,j}-\bar{X}_{i})^{2}\]
   \[F = \frac{MS_{M}}{MS_{R}} \quad \quad \quad p =1-f(F, \nu_{M}, \nu_{R})\]
\end{itemize}

\subsection*{Two-way ANOVA}
        \begin{table}[H]
        \begin{adjustbox}{width=\columnwidth,center}
        \begin{tabular}{c|c|c|c|c|c}
        & Sum of Squares & DoF & Mean Square & F-Statistic & Notes \\ \hline
        Total & $\displaystyle SS_{T} = \sum_{g=1}^{G}\sum_{i=1}^{I}\sum_{j=1}^{J}(X_{g,i,j}-\bar{X})^{2}$ & $\displaystyle \nu_{T}=N-1$ & & & $N=GIJ$\\ \hline

        Model & $\displaystyle SS_{M} = \sum_{g=1}^{G}\sum^{I}_{i=1}J(\bar{X}_{gi}-\bar{X})^{2}$ & $\displaystyle \nu_{M}=GI-1$ & & &\\ \hline

        Residuals & $\displaystyle SS_{R} = \sum_{g=1}^{G}\sum_{i=1}^{I}\sum_{j=1}^{J}(X_{g,i,j}-\bar{X}_{gi})^{2}$ & $\displaystyle \nu_{R} = N-GI$ & $\displaystyle MS_{R} = \frac{SS_{R}}{\nu_{R}}$ & & \\ \hline

        $SS_{A}$  & $\displaystyle SS_{A} = IJ \sum_{g=1}^{G}(\bar{X}_{g}-\bar{X})^{2}$ & $\displaystyle \nu_{A} = G-1$ & $\displaystyle MS_{A} = \frac{SS_{A}}{\nu_{A}}$ & $\displaystyle F_{A} = \frac{MS_{A}}{MS_{R}}$ & $SS_{M}$ grouped by G\\ \hline

        $SS_{B}$  & $\displaystyle SS_{B} = GJ \sum_{i=1}^{I}(\bar{X}_{i}-\bar{X})^{2}$ & $\displaystyle \nu_{B} = I-1$ & $\displaystyle MS_{B} = \frac{SS_{B}}{\nu_{B}}$ & $\displaystyle F_{B} = \frac{MS_{B}}{MS_{R}}$ & $SS_{M}$ grouped by I\\ \hline

        $SS_{A \times B}$  & $\displaystyle SS_{A \times B} = SS_{M}-SS_{A}-SS_{B}$ & $\displaystyle \nu_{A \times B} = \nu_{M}-\nu_{A}-\nu_{B}$ & $\displaystyle MS_{A\times B} = \frac{SS_{A\times B}}{\nu_{A\times B}}$ & $\displaystyle F_{A\times B} = \frac{MS_{A\times B}}{MS_{R}}$ & left over $SS$\\
         \end{tabular}
         \end{adjustbox}
        \end{table}
\begin{figure}[H]
    \centering
    \includegraphics[width=.5\textwidth]{./images/twoWayANOVA.pdf}
\end{figure}
\subsection*{Post-Hoc Tests}
\begin{itemize}
            \item Bonferroni - pairwise t-tests. Statistically significant if $p < p_{crit}$
            \item Tukey Kramer Test (less conservative than Bonferroni).
        \end{itemize}
\subsection*{Non-parametric Tests}
\begin{itemize} \itemsep0.4cm
    \item \textbf{Sign test}: one sample, asymmetric data.
    \begin{itemize}
        \item Compare each sample to a pre-defined median value $\tilde{\mu}_{0}$: +ve/-ve difference?
        \item Make hypothesis:
        \begin{itemize}
            \item $H_{0}$: the population median $\tilde{\mu}$ from which the sample is taken is equal to $\tilde{\mu}_{0}$.
            \item $H_{1}$: $\tilde{\mu} \neq \tilde{\mu}_{0}$.
        \end{itemize}
        \item If $H_{0}$ is true, we should obtain at least $n/2$ of data points that are negative.
        \item find $P$ by taking binomial test by taking $p=0.5$.
    \end{itemize}

    \item \textbf{Wilcoxon Sign-rank test}: one sample, non-asymmetric data.
    \begin{itemize}
        \item Compare each sample to a pre-defined median value $\tilde{\mu}_{0}$. Consider both the \textbf{sign} and \textbf{magnitude} of differences.
        \item Sort and find the rank of the differences. Multiply the ranks by their sign.
        \begin{table}[H]
            \centering
            \begin{tabular}{l|c|c|c}
            rank  &  ...  &  ...&  ...\\ \hline
            $\lvert$difference$\rvert$ & ...  &  ...&  ...\\  \hline
            sign & ...  &  ...&  ...\\ \hline
            signed rank & ... &  ... &  ...\\
            \end{tabular}
        \end{table}
        \item calculate the absolute sum of the signed ranks, for +ve and -ve, respectively.
        \begin{itemize}
            \item $W^{+} = \sum_{i}^{n^{+}}R_{i}^{+}$ and $W^{-} = \sum_{i}^{n^{-}}-R_{i}^{-}$
            \item the smaller one becomes the test statistic.
        \end{itemize}
        \item Critical value $W_{crit}$ can be located from the table.
    \end{itemize}

    \item \textbf{Mann-Whitney U-test}: two samples.
    \begin{itemize}
        \item Make hypothesis:
        \begin{itemize}
            \item $H_{0}$: the mean rank of the two levels are equal. (If samples are independent are from similar underlying distributions : mean rank -> median.)
            \item $H_{1}$: the mean rank of the two levels are different.
        \end{itemize}
        \item Group both samples, rank and correct averages \textbf{in one table}.
         \begin{table}[H]
            \centering
            \begin{tabular}{l|c|c|c}
            rank  &  ...  &  ...&  ...\\ \hline
            data value & ...  &  ...&  ...\\
            \end{tabular}
        \end{table}
        \item Calculate the sum ranks for each group $W$.
        \[W_{X}=\sum_{i}^{n_{X}}R_{i}^{X} \quad \quad W_{Y}=\sum_{i}^{n_{Y}}R_{i}^{Y}\]
        \item Calculate U-statistic for each group
        \[U_{i} = W_{i}-\frac{n_{i}(n_{i}+1)}{2}\]
        \item For $n<8$: $U_{crit}$ can be found from the table. Reject $H_{0}$ if $min(U_{1}, U_{2})<u_{crit}$
        \item For $n\geq 8$: \[\mu_{U}=\frac{n_{1}n_{2}}{2} \quad \quad \sigma_{U}^{2}= \frac{n_{1}n_{2}(n_{1}+n_{2}+1)}{12}\]
        z-statistic can be calculated by \[Z_{i}=\frac{U_{i}-\mu_{U}(+continuity)}{\sigma_{U}}\]
    \end{itemize}

    \item \textbf{Kruskall-Wallis test}: more than two samples.
    \begin{itemize}
        \item $H_{0}$: the $I$ independent random samples all come from identical populations.
        \item Calculate the sum ranks for each group:
        \[H=-3(N+1)+\frac{12}{N(N+1)}\sum_{i=1}^{k}\frac{W_{i}^{2}}{J_{i}}\]
        \item For all $J_{i} > 5$, and if $H_{0}$ is true, $H$ is approximated to $\chi^{2}$ distribution with $\nu = I-1$.
    \end{itemize}
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{./images/nonparasummary.pdf}
\end{figure}

    % \item K-S test statistic: \[K = \sqrt{n}D_{n}\] p-value is $\displaystyle p = 1-KS(\sqrt{n}D_{n})$.

    % \item Shapiro-Wilk test statistic: \[W = \frac{(\sum a_{i}z_{i})^{2}}{(\sum x_{i}-\bar{x})^{2}}\]
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Correlation and Regression}
\subsection*{Correlation}
\begin{table}[H]
    \centering
    \begin{adjustbox}{width=\columnwidth,center}
    \begin{tabular}{c|c}
    Covariance (discrete and continuous)& Correlation Coefficient \\ \hline
    $\displaystyle cov(X, Y)= E[(x-\mu_{X})(y-\mu_{Y})]
    = E(XY)-E(X)E(Y)=\begin{cases}
    \displaystyle \sum_{x}\sum_{y}(x-\mu_{X})(y-\mu_{Y})p(x,y)\\
    \displaystyle \int_{-\infty}^{+\infty}\int_{-\infty}^{+\infty}  (x-\mu_{X})(y-\mu_{Y})f(x,y) dxdy %& continuous
    \end{cases} $
     &  $\displaystyle \rho_{X,Y} = \frac{cov(X,Y)}{\sigma_{X}\sigma_{Y}}$\\
    \end{tabular}
    \end{adjustbox}
\end{table}
  \begin{itemize}
     \item \textbf{Pearson's correlation coefficient}: $r$ is an point estimator of $R = \hat{\rho}$.  Assume linear relationship between $X$ and $Y$.
     \[\sigma_{X}^{2} = S_{XX} = \frac{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}}{n-1}, \quad \quad \sigma_{Y}^{2} = S_{YY} = \frac{\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}{n-1}, \quad \quad cov(x,y) = S_{XY} = \frac{\sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})^{2}}{n-1}\]
    % \[\Downarrow\]
    \[\Rightarrow \quad r_{X, Y} = \frac{S_{XY}}{\sqrt{S_{XX}}\sqrt{S_{YY}}} = \frac{\sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}}\sqrt{\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}} \quad \in [-1, 1]\]

    \item \textbf{Test statistic and DoF}
    \[T = \frac{R\sqrt{n-2}}{\sqrt{1-R^{2}}}, \quad \quad \nu = n-2\]
     \item \textbf{Spearman's rank correlation coefficient}: no linear relationship between $X$ and $Y$ -  $ \displaystyle \rho = 1- \frac{6\sum(x_{i}-y_{i})^{2}}{n(n^{2}-1)}$, where $x_{i}, y_{i}$ are ranks. Otherwise we evaluate Pearson's r on the ranks (if $x_{i}, y_{i}$ are tied).
  \end{itemize}
  %%

\subsection*{Regression}
\begin{itemize}
    \item \textbf{Regression model} - $Y = b_{0} + b_{1}X + \epsilon$, where $\epsilon$ is a normally distributed random variable with $E(\epsilon) = 0$ and $V(\epsilon) = \sigma_{\epsilon}^{2}$.
    \item \textbf{Estimate of the best fit line}

    \begin{minipage}{.45\textwidth}
    \begin{table}[H] \centering
    \begin{tabular}{c|c|c}
    \multicolumn{3}{c}{$\mathbf{Y=\hat{b}_{0}+\hat{b}_{1}X}$}\\ \hline
    slope & $\displaystyle \hat{b}_{1} = \frac{S_{xy}}{S_{xx}}$ & $\displaystyle s_{\hat{b}_{1}} = \frac{s_{\epsilon}}{\sqrt{S_{xx}}}$\\ \hline
    intercept & $\displaystyle \hat{b}_{0} = \bar{y}-\hat{b}_{1}\bar{x} $ & $\displaystyle s_{\hat{b}_{0}} = \sqrt{\frac{1}{n}+\frac{\bar{x}^{2}}{S_{xx}}}$
    \end{tabular}\end{table}
    \end{minipage}
    \begin{minipage}{.05\textwidth}\[\xrightarrow{\text{with}}\]
    \end{minipage}
    \begin{minipage}{.45\textwidth}
    \begin{tabular}{c|c|c} \centering
    $S_{xx}$ & $S_{yy}$ & $S_{xy}$ \\ \hline
    $\displaystyle \sum_{i=1}^{n}(x_{i}-\bar{x})^{2}$ & $\displaystyle \sum_{i=1}^{n}(y_{i}-\bar{y})^{2}$ & $\displaystyle \sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})^{2}$
    \end{tabular}\end{minipage}
    \begin{itemize}
        \item Best estimator for variance of residuals -  $\displaystyle \hat{\sigma}_{\epsilon}^{2} = s_{\epsilon}^{2} = \frac{SS_{R}}{\nu_{R}} = \frac{\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}{n-2}$
    \end{itemize}

    \item \textbf{Coefficient of determination}: \[r^{2} = 1-\frac{SS_{R}}{SS_{T}}, \quad \quad \text{where} \quad SS_{T}=S_{yy}, \quad SS_{R}=s_{\epsilon}^{2}\nu_{R}, \quad \quad \]

    \item \textbf{Confidence interval}:
     \begin{table}[H] \centering
    \begin{tabular}{c|c|c}

   slope & intercept & variance of residual \\ \hline
   $CI=\hat{b}_{1} + t_{1-\frac{\alpha}{2}, \nu}s_{\hat{b}_{1}}$ & $CI=\hat{b}_{0} + t_{1-\frac{\alpha}{2}, \nu}s_{\hat{b}_{0}}$ &
   $s_{\epsilon}^{2}= SS_{R}/\nu_{R}$ and $\nu_{R}=n-2$
    \end{tabular}\end{table}

    \item \textbf{Confidence bands}: uncertainty on the mean
    \[CI = \hat{y}^{*}\pm t_{\frac{\alpha}{2}, n-1}s_{\hat{y}^{*}} \quad \quad s_{\hat{y}^{*}} = s_{\epsilon}\sqrt{\frac{1}{n}+\frac{(x^{*}-\bar{x})^{2}}{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}}}\]

    \item \textbf{Prediction bands}: uncertainty on a single measurement
    \[s_{pred}^{2} = s_{\hat{y}^{*}}^{2} + s_{\epsilon}^{2} \quad \quad PI=\hat{y}^{*}\pm t_{\frac{\alpha}{2}, n-2} s^{*}_{pred}\]
\end{itemize}

\vspace*{\fill}
\framebox{\footnotesize  \href{https://www.overleaf.com/read/yrxtbmqnvzgh}{Scripted} by B Li \& P Xie. \ Last update: \today}
% \begin{flushright}
%     \begin{figure}[H]
%     \includegraphics[width=.1\textwidth]{by-nceu.eps}
%     This work is licensed under a Creative Commons Attribution 4.0 International License
% \end{figure}
% \end{flushright}

% insert data tables from the external pdf source
\includepdf[pages=-]{./images/Z_t_F_Tables.pdf}

% end the doc
\end{document}


\section{M/M/\texorpdfstring{$\infty$}{Infinity} Queues}
\label{sec:M-M-inf-Queues}

A $M/M/\infty$ is a queue where
(i) the arrival process is Poissonian with rate $\lambda$,
(ii) the service process is Exponential with rate $\mu$,
(iii) there are infinite servers,
(iv) the buffer has infinite capacity,
(v) the scheduling policy is FCFS.

%\begin{figure}[tp]
%\label{fig:M-M-inf-Queue}
%	\centering
%	\includegraphics{fig/M-M-inf-Queue}
%	\caption{An M/M/k queue and its corresponding CTMC.}
%\end{figure}

The key question in these types of systems is determining the \textit{Queue Probability} $P_{Q}$, that is the probability that an arriving job is enqueued.

\begin{theorem}[Utilization]
	\label{thm:M-M-inf-Utilization}
	For any $M/M/\infty$, the utilization is

	\begin{equation}
	\label{eqn:M-M-inf-Utilization}
	\varrho = \frac{\lambda}{\mu}
	\end{equation}
\end{theorem}

\begin{theorem}[State Probability]
\label{thm:M-M-inf-Probability-State}
	For any $M/M/\infty$, the state probability is

	\begin{equation}
	\label{eqn:M-M-inf-Probability-State}
	\pi_{i} = \Big(\frac{\lambda}{\mu})^{i} \frac{1}{i!} \pi_{0}
	\end{equation}

	with

	\begin{equation}
	\label{eqn:M-M-inf-Probability-State-Zero}
	\pi_{0} = e^{-\frac{\lambda}{\mu}}
	\end{equation}

	That is $N^{M/M/\infty} \sim Poisson(\frac{\lambda}{\mu})$.
\end{theorem}

Since, $N^{M/M/\infty} \sim Poisson(\frac{\lambda}{\mu})$, then

\begin{equation}
\label{eqn:M-M-inf-Mean-System-Jobs}
\expected{N} = \frac{\lambda}{\mu}
\end{equation}

The remaining metrics ($\expected{N_{Q}},\expected{T},\expected{T_{Q}}$) could be determined by applying the Little's Law, the basic definitions
$\expected{N}=\expected{N_{Q}}+\expected{N_{S}}$,
$\expected{N_{S}}=\varrho$,
$\expected{T}=\expected{T_{Q}}+\expected{T_{S}}$,
$\expected{T_{S}}=\frac{1}{\mu}$,
considering that $\expected{N_{Q}}=0$ and $\expected{T_{Q}}=0$, because every arrival il always served immediately.

In particular we obtain

\begin{equation}
\label{eqn:M-M-inf-Mean-Response-Time}
\expected{T} = \frac{1}{\mu}
\end{equation}


\section {Introduction to DFA Framework}
\setlength{\parindent}{0pt}
(Prepared by Namrata Priyadarshani and Shivam Bansal)

\vspace{0.3cm}

In this module, we build up common framework to express DFA. Some common characteristics of DFA are described below:

\subsection{Transfer Functions and Meet operator}

Let s be a statement and the value before statement is in[s] and after statement is out[s]. Then, for a forward DFA, \textbf{Transfer function} is defined as the function that takes in[s] as argument and converts it into out[s]. \textbf{Meet operator} expresses in[s] as the function of out[p] for all predecessor p of s. These functions operates in reverse direction in the case of Backward DFA.
\newline
Most of the time, we consider running the algorithm at the Basic Block level granularity and define Transfer function and meet operator for the values in[B] and out[B] i.e. the input and output values of the basic block B.

\subsection{Partial ordered values}

There is a partial order within the elements of Domain represented by $\geq$ operator.
Directed Acyclic Graph is one way to represent the ordering. The graph contains an edge from $a \rightarrow b$ iff $a \geq b$.
%insert image
\includegraphics[scale=0.3]{images/91_1.png}
There is one value greater than all values in Domain known as Top value (T) and one value less than all values in Domain known as Bottom value ($ \bot $).

\subsection{Constant Propagation DFA}
Different DFA has different direction, boundary condition, meet operator and transfer function. The structure and flow of algorithm remains same for all DFA. In fixed-point-algorithm first the boundary condition is specified, all other values are set to top value and then iterate until no change occur in any value.

For constant propagation the DFA is specified by:
\begin{itemize}
    \item \textbf{Domain}: set of constant definitions
    \item \textbf{Direction}: Forward
    \item \textbf{Transfer function}: $f_{B} = \lambda x. GEN[B] \cup (x-KILL[B]) $
    \item \textbf{Meet operator}: $\cap$ (set intersection)
    \item \textbf{Boundary condition}: $out[entry] = \phi$
\end{itemize}


\section*{Ex.34.4-6}
\subsection*{Formula satisfiability}

We are given an algorithm, call it $A$, which can decide formula satisfiability in polynomial time. To find a satisfying assignment with the use of $A$ do the following:

Let $x_1,\ldots,x_k$ be the $k$ input variables to the formula problem. First call $A$, to decide if the formula instance has a satisfying assignment. If not it will return 0, and no more work needs to be done. If it returns 1, there exists an assignment which we can find.

Note that if we set $x_i=1$ for some $i\in\{1,\ldots,k\}$, and then run $A$ it will tell us if there exists a solution when $x_i=1$. If there doesn't, then set $x_1 =0$. Do this for all $i\in\{1,\ldots,k\}$, and you will get a satisfying assignment. For each variable we call $A$ once, which is $O(n^m)$, and we do this $k$ times plus some constant work. This yields an algorithm that finds the satisfying assignment in $O(kn^m)=O(n^m)$ time. Which is polynomial.


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
    \DeclareGraphicsExtensions{.png, .jpeg}
\usepackage{caption}
\usepackage[top=1in, bottom=1in, left=1in, right=1in]{geometry}

\title{STAT 775: Machine Learning \\ HW 02}
\author{Terence Henriod}
\date{\today}

\begin{document}

\clearpage            % All
\maketitle            % this,
\thispagestyle{empty} % removes the page number from the title page

\begin{abstract}
A short implementation and demonstration of na\"{i}ve Bayes classifiers on hand-written digit images.
\end{abstract}

\newpage
\section{Problem Description}
The task: using the \texttt{zipcode} data from the Elements of Statistical Learning book website and Bayes rule, do the following:
\begin{enumerate}
  \item Construct Gaussian classifiers for all of the digits (0-9) using the training data set. Do so by computing the means and covariances of each class in order to fit the Gaussian distribution for the class.
  \item Use the constructed classifiers to classify the test set.
  \item Display the results in a ``confusion matrix."
\end{enumerate}
\textit{Hint}: If you find that a matrix (specifically any of the covariance matrices) is numerically or actually singular, and therefore cannot be inverted, regularize it by adding an identity matrix multiplied by a small factor to ``breathe" on it. This can help make the matrix invertible, but one needs to exercise caution as it can adversely affect the results of computations.

\section{Results}
The na\"{i}ve Bayes classifiers performed rather well, achieving nearly 94\% accuracy. While this does leave something to be desired, improving on this performance benchmark might require improving the numerical accuracy of the program (using something better than double-precision variables), finding a better regularization factor for regularizing non-invertible matrices ($0.5$ seemed to work well enough, no more effort was put forth to improve on this), application of additional techniques (like PCA), or even using this classifier as an element of a composite classifier.

\begin{figure}[h]
  \centering
  \includegraphics[width=.6\linewidth]{R_confusion_matrix_output}
  \caption{R output of a confusion matrix displaying the classification results using the na\"{i}ve Bayes classifier.}
  \label{fig:confusion_matrix}
\end{figure}

\section{Code}
After struggling for too long with R code and singular matrices, it was deemed better to switch to a familiar and high performance language - namely C++. Thus, the classification code was written in C++ and the confusion matrix was ported to R for data presentation.

\subsection{C++ Code}
\begin{verbatim}
#include <string>
#include <vector>
#include <Eigen/Dense>
#include <cmath>

#include <iostream>
#include <fstream>
#include <exception>

#define PI 3.141592653589793238462643383279502884L
#define REGULARIZATION_FACTOR 0.5

const std::string TRAINING_FILE_NAME = "zip.train"; // 2007 items
const std::string TEST_FILE_NAME = "zip.test"; // 7291 items

typedef struct {
  unsigned label;
  Eigen::VectorXd features;
} ClassificationObject;

typedef struct {
  unsigned label;
  double prior;
  Eigen::VectorXd mean;
  Eigen::MatrixXd covariance;
  double covarianceDeterminant;
  Eigen::MatrixXd covarianceInverse;
} ClassInfo;

std::vector<ClassificationObject>
ReadData(
  const std::string& fileName,
  const unsigned featureDim,
  const unsigned numItems) {

  std::ifstream fin;
  fin.clear(); fin.open(fileName.c_str());

  if (!fin.good()) {
    throw std::exception();
  }

  std::vector<ClassificationObject> data; data.reserve(numItems);

  for (unsigned i = 0; i < numItems; ++i) {
    ClassificationObject classificationObject;
    double dummy;
    fin >> dummy;
    classificationObject.label = (unsigned) dummy;

    classificationObject.features.resize(featureDim);
    for (unsigned j = 0; j < featureDim; ++j) {
      fin >> classificationObject.features(j);
    }
    data.push_back(classificationObject);
  }

  return data;
}

ClassInfo
ComputeClassInfo(
  const std::vector<ClassificationObject>& data,
  const unsigned classLabel,
  const unsigned featureDim) {

  ClassInfo classInfo;
  classInfo.label = classLabel;
  classInfo.mean.setZero(featureDim);
  classInfo.covariance.setZero(featureDim, featureDim);

  // compute mean and prior probability at once
  std::vector<ClassificationObject> subset;
  for (const ClassificationObject& classificationObject : data) {
    if (classificationObject.label == classLabel) {
      classInfo.mean += classificationObject.features;
      subset.push_back(classificationObject);
    }
  }
  classInfo.mean /= subset.size();
  classInfo.prior = (double) subset.size() / (double) data.size();

  // covariance matrix
  // Note: I use a nifty trick that is simpler in code (not sure if
  //       simpler computationally). Let $P_i = x_i - mu$, where $i$
  //       corresponds to a particular observation vector. Then
  //       $A$ is the concatenation of all the $P_i$ as column
  //       vectors ($A = [P_1 ... P_m]$). Then
  //       $\frac{1}{m} * A * A^t$ results in the same computations
  //       that create the covariance matrix as more traditional
  //       formulae.
  Eigen::MatrixXd A(featureDim, subset.size());
  for (unsigned j = 0; j < subset.size(); ++j) {
    A.col(j) = subset[j].features - classInfo.mean;
  }
  classInfo.covariance = A * A.transpose();
  classInfo.covariance /= (double) subset.size();
  if (classInfo.covariance.determinant() - 0.1 <= 0.0) {
    Eigen::MatrixXd regularizationMatrix;
    regularizationMatrix.setIdentity(featureDim, featureDim);
    regularizationMatrix *= REGULARIZATION_FACTOR;
    classInfo.covariance += regularizationMatrix;
  }
  classInfo.covarianceDeterminant = classInfo.covariance.determinant();
  classInfo.covarianceInverse = classInfo.covariance.inverse();

  return classInfo;
}

double
GaussianPdf(
  const Eigen::VectorXd& testFeatureVector,
  const ClassInfo& classSummary) {

  const Eigen::VectorXd& x = testFeatureVector;
  const Eigen::VectorXd& mu = classSummary.mean;
  const double& sigmaDet = classSummary.covarianceDeterminant;
  const Eigen::MatrixXd& sigmaInv = classSummary.covarianceInverse;

  double scalingFactor = 1.0 / sqrt(pow(2.0 * PI, mu.size()) * sigmaDet);
  double exponent = -0.5 * (((x - mu).transpose() * sigmaInv).dot((x - mu)));

  return scalingFactor * exp(exponent);
}

unsigned
ClassifyObject(const ClassificationObject& object, const std::vector<ClassInfo>& classSummaries) {
  unsigned mostLikelyClass = 0;
  double highestProbability = classSummaries.front().label;

  for (const ClassInfo& classSummary : classSummaries) {
    double probability = GaussianPdf(object.features, classSummary) * classSummary.prior;
    if (probability > highestProbability) {
      mostLikelyClass = classSummary.label;
      highestProbability = probability;
    }
  }

  return mostLikelyClass;
}

Eigen::MatrixXi
PerformClassifications(const std::vector<unsigned>& labelSet,
  const std::vector<ClassInfo>& classSummaries,
  const std::vector<ClassificationObject>& testObjects) {

  Eigen::MatrixXi confusionMatrix(labelSet.size(), labelSet.size());
  confusionMatrix.setZero(labelSet.size(), labelSet.size());

  unsigned i = 1;
  unsigned onePercent = testObjects.size() / 100;
  for (const ClassificationObject& object : testObjects) {
    if (i % onePercent == 0) {
      std::cout << i << "% processed." << std::endl;
    }
    i++;

    unsigned classifiedAs = ClassifyObject(object, classSummaries);
    confusionMatrix(object.label, classifiedAs)++;
  }

  return confusionMatrix;
}

int
main(const int argc, const char** argv) {
  std::cout << "Reading Data..." << std::endl;
  std::vector<ClassificationObject> trainingData = ReadData(
    TRAINING_FILE_NAME,
    256,
    7291
  );
  std::vector<ClassificationObject> testData = ReadData(
    TEST_FILE_NAME,
    256,
    2007
  );

  std::cout << "Training Models..." << std::endl;
  std::vector<ClassInfo> classSummaries; classSummaries.reserve(10);
  for (unsigned label = 0; label <= 9; ++label) {
    classSummaries.push_back(ComputeClassInfo(trainingData, label, 256));
  }

  std::cout << "Classifying Objects..." << std::endl;
  Eigen::MatrixXi confusionMatrix = PerformClassifications(
    {0,1,2,3,4,5,6,7,8,9},
    classSummaries,
    testData
  );

  std::cout << "Results:" << std::endl;
  std::cout << confusionMatrix << std::endl;

  return 0;
}
\end{verbatim}

\subsection{R Code}
\begin{verbatim}
############
# Plotting #
############
require(caret)

results <-data.matrix(read.table("results"))
dimnames(results) <- list(c(0,1,2,3,4,5,6,7,8,9), c(0,1,2,3,4,5,6,7,8,9))

actuals.f <- list()
predictions.f <- list()
for (i in 1:nrow(results)) {
  for (j in 1:ncol(results)) {
    actuals.f <- c(actuals.f, rep(i - 1, results[i, j]))
    predictions.f <- c(predictions.f, rep(j - 1, results[i, j]))
  }
}
actuals.f <- factor(unlist(actuals.f))
predictions.f <- factor(unlist(predictions.f))

confusionMatrix(data = predictions.f, reference = actuals.f)
\end{verbatim}

\subsection{The student would like to thank...}
The student would like to thank the authors of the Eigen C++ matrix library. This library has proved quick, easy and effective numerous times, this time being no exception. The Eigen library and more information can both be found at\hfill\\
\texttt{http://eigen.tuxfamily.org}.

\end{document}


%!TEX root = ../notes.tex
\section{February 3, 2022}

\subsection{Arithmetic Functions}

We look at arithmetic functions and how they act on prime numbers:
\begin{definition}[Arithmetic Function]
    An \ul{arithmetic function} is a function $f: \ZZ_+ \to \CC$.

    \emph{(Typically, these are integer valued.)}
\end{definition}
\begin{example}We have some examples of arithmetic functions:
    \begin{itemize}
        \item Euler $\phi$ function.
        \item $\tau(n)$, the counting function. It takes a positive integer and counts the number of positive divisors of $n$. \[\tau(n) = \sum_{d\mid n} 1\]
        \item $\sigma(n)$, the sum of divisors function. It is the sum over all the positive divisors of $n$. \[\sigma(n) = \sum_{d\mid n}d\]
    \end{itemize}
\end{example}
We have some properties of these functions, like \emph{multiplicative}, \emph{completely multiplicative}, \emph{additive}, \emph{completely additive}.

\begin{definition}[Multiplicativity]
    An arithmetic function $f$ is \ul{multiplicative} if
    \begin{align*}
        f(mn) & = f(m)f(n)\quad \text{whenever }(m, n) = 1
        \intertext{$f$ is said to be \ul{totally or completely multiplicative} if }
        f(mn) & = f(m)f(n)\quad \forall m, n\in\ZZ_+
    \end{align*}
    regardless of coprimality.
\end{definition}

If $f$ is multiplicative and $n_1, \dots, n_k$ are positive pairwise coprime integers, then \[f(n_1\dots n_k) = f(n_1)f(n_2)\dots f(n_k).\]

A particular case that is useful is when we write
\[n=p_1^{e_1}p_2^{e_2}\cdots p_k^{e_k}\]
so that assuming multiplicativity, we have that
\[f(n) = f(p_1^{e_1}) f(p_2^{e_2}) \cdots f(p_k^{e_k})\]
A common type of arithmetic function is a summatory function, namely a function $f$ of the form
\[f(n) = \sum_{d\mid n} g(d), \quad \text{where $g$ is some arithmetic function.}\]

\emph{Food for thought:} how special are summatory functions within the set of all arithmetic functions?

A special property of summatory functions is that they ``inherit multiplicativity''.
\begin{lemma}\label{lemma:summatory-preserves-multiplicativity}
    If $g$ is a multiplicative function, and \[f(n) = \sum_{d\mid n}g(d)\quad \forall n,\] then $f$ itself is multiplicative.
\end{lemma}

\begin{proof}
    Suppose $m, n\in \ZZ_+$ are coprime positive integers.

    The divisors $d$ of $mn$ are the products $a\cdot b$ where $a\mid m$ and $b\mid n$. Each such pair $a, b$ yields a uniquely determined produce $d = a\cdot b$. Conversely, since $(m, n) = 1$, each divisor $d$ of $mn$ determines a unique divisor $a=\gcd(d, m)$ and $b = \gcd(d, n)$ so that $d = a\cdot b$.

    Thus there is a \emph{bijection} between divisors of $mn$ and $m, n$ separately
    \[d\mid mn \longleftrightarrow (a\mid m, b\mid n)\]
    Thus we have
    \begin{align*}
        f(m\cdot n) & = \sum_{d\mid mn}g(d)                                              \\
                    & = \sum_{a\mid m}\sum_{b\mid n}g(ab)                                \\
                    & = \sum_{a\mid m}\sum_{b\mid n}g(a)g(b)                             \\
                    & = \left(\sum_{a\mid m} g(a)\right) \left(\sum_{b\mid n}g(b)\right)
        = f(m)\cdot f(n)
    \end{align*}
    Thus completes the proof that $f$ is multiplicative.
\end{proof}

\recall The functions introduced earlier
\[\tau(n) = \sum_{d\mid n} 1\qquad \sigma(n) = \sum_{d\mid n}d\]
So $\tau$ is the summatory function of the constant $1$ functions, and $\sigma$ is the summatory function of the identity function. We know that the constant $1$ function and the identity function are both completely multiplicative, so $\sigma$ and $\tau$ are multiplicative functions.

The implication of which is that it suffices to apply $\tau$ and $\sigma$ on prime powers and multiply.

Let $p$ be a prime. Then
\begin{align*}
    \tau(p^e)   & = e + 1\quad  \text{(from $p^0$ to $p^e$)}.          \\
    \intertext{We also have}
    \sigma(p^e) & = 1 + p + p^2 + \cdots + p^e = \frac{p^{e+1}-1}{p-1}
\end{align*}
Therefore, if $n=p_1^{e_1}p_2^{e_2}\cdots p_k^{e_k}$, then
\begin{align*}
    \tau(n)   & = \prod_{i=1}^k (e_i + 1)                                 \\
    \sigma(n) & = \prod_{i=1}^k \left(\frac{p_i^{e_i+1}-1}{p_i-1}\right).
\end{align*}
\begin{remark}
    There are higher order divisor functions
    \[\sigma_k(n) = \sum_{d\mid n}d^k\]
    so $\sigma_0 = \tau, \sigma_1 = \sigma, \dots$
\end{remark}

\subsection{Review of \texorpdfstring{$\ZZ/n\ZZ$}{Z/nZ} and its units}

\begin{definition}[Modular Congruence]
    If $a, b, m\in \ZZ$, $m\neq 0$, we say that \ul{$a$ is congruent to $b$ modulo $m$} if $m\mid b-a$. We write
    \begin{equation*}
        a\equiv b\mod{m}, \text{ or more simply } a\equiv b\ (m)
    \end{equation*}
\end{definition}
Congruence mod $m$ is an equivalence relation on $\ZZ$. If $a\in \ZZ$, $\overline{a}$ denotes the set of integers congruent to $a\mod m$, i.e. $\overline{a} = \{a + km \mid k\in \ZZ\}$.

\begin{definition}[$\ZZ/m\ZZ$, Residues mod $m$]
    The set of congruence classes mod $m$ is denoted $\ZZ/m\ZZ$. This is a quotient ring of the ring of integers $\ZZ$.

    If $\overline{a}_1, \overline{a}_2, \dots, \overline{a}_m$ form a complete set of congruence classes mod $m$, then the set of integers $\{a_1, a_2, \dots, a_m\}$ is called a \ul{complete set of residues mod $m$}.
\end{definition}

$\ZZ/m\ZZ$ can be endowed with the structure of a commutative ring by setting
\begin{align*}
    \overline{a} + \overline{b}               & = \overline{a + b} \\
    \text{and }\overline{a}\cdot \overline{b} & = \overline{ab},
\end{align*}
and proving that this is well-defined as ring operations.

\begin{proposition}
    The set of units in $\ZZ/m\ZZ$ is exactly
    \[\{\overline{a}\mid (a, m) = 1\}\]
\end{proposition}
\begin{proof}
    Let $\overline{a}\in \ZZ/m\ZZ$, then
    \begin{align*}
                              & \exists \overline{b}\in \ZZ/m\ZZ\text{ s.t. }\overline{b}\cdot \overline{a}\equiv 1\mod m \\
        \Longleftrightarrow\  & \exists b, n\in \ZZ\text{ s.t. } ba - mn = 1                                              \\
        \intertext{Then by B\'ezout's identity...}
        \Longleftrightarrow\  & (a, m) = 1
    \end{align*}
\end{proof}

\subsection{The Euler \texorpdfstring{$\phi$}{phi} Function}
For $n\in \ZZ_+$, $\phi(n)$ is defined to be the number of integers $1\leq m\leq n$ coprime to $n$.
\begin{example}
    We have some examples of the Euler $\phi$ functions:
    \begin{align*}
        \phi(1)   & = 1                                                                 \\
        \phi(p)   & = p-1 \text{ for any prime $p$}
        \intertext{Let $e\geq 1$, }
        \phi(p^e) & = p^e-p^{e-1} \text{ for prime powers, we exclude multiples of $p$}
    \end{align*}
\end{example}

\emph{Wouldn't be great if $\phi$ were multiplicative? It is!}
\begin{theorem}
    If $(m,n) = 1$, then $\phi(mn) = \phi(m)\phi(n)$.
\end{theorem}
\begin{proof}
    By the Chinese Remainder Theorem\footnote{This is an easy way to prove this assuming Math 1530 (Abstract Algebra). There is another way to prove this with one hand tied behind the back, it just takes more mental muscle to do. }, $\ZZ/mn\ZZ\cong \ZZ/m\ZZ\times \ZZ/n\ZZ$ if $(m, n) = 1$.

    Taking the unit groups on both sides, we have
    \[(\ZZ/mn\ZZ)^\times \cong (\ZZ/m\ZZ)^\times \times (\ZZ/n\ZZ)^\times\]
    and the Euler $\phi$ function is simply measuring the order of said unit groups ($\phi(n) = \left|(\ZZ/n\ZZ)^\times\right|$).
\end{proof}

Here is an important fact about the Euler $\phi$ function:
\begin{proposition}
    We have
    \begin{equation*}
        \sum_{d\mid n}\phi(d) = n.
    \end{equation*}
\end{proposition}
\begin{proof}\emph{(1: a cute, snazzy proof)}
    Consider the $n$ rational numbers
    \[\frac{1}{n}, \frac{2}{n}, \dots, \frac{n-1}{n}, \frac{n}{n} = 1\]
    and reduce all to lowest terms so that the numerator and denominator are coprime.

    Q: Given a positive divisor $d$ of $n$, how many fractions have $d$ as the denominator?

    A: We have exactly $\phi(d)$ of them.

    Conversely, every denominator $d$ is certainly a divisor of $n$. So we conclude that $\displaystyle n = \sum_{d\mid n}\phi(d)$.
\end{proof}
\begin{proof}\emph{(2: using what we've learnt)}
    We use the fact that $\phi$ is multiplicative, and that this function is a summatory function of $\phi$, so this function itself is multiplicative. We can decompose this into prime powers. So it suffices to show this for prime powers.

    Let $n = p^k$. Let
    \[f(n) = \sum_{d\mid n}\phi(d).\]
    Then we have
    \begin{align*}
        f(p^k) = \sum_{d\mid p^k} \phi(d) & = 1 + (p-1) + (p^2 - p) + \dots + (p^k-p^{k-1})
        \intertext{which is a telescoping sum which leaves}
                                          & = p^k
    \end{align*}
    which is as intended.
\end{proof}


\documentclass[../PHYS306Notes.tex]{subfiles}

\begin{document}
\section{Lecture 1}
\subsection{Lecture Notes - Lagrangian Mechanics Part 1}
\subsubsection{Newtonian Mechanics to Lagrangian Mechanics}
Recall the Newtonian formulation of classical mechanics; given the forces $\v{F}_1(t), \cdots \v{F}_n(t)$ on a particle, we can solve Newton's second law (a second order differential equation):
\[\sum_{i=1}^n \v{F}_i(t) = m \ddot{\v{r}}(t)\]
To obtain the trajectory $\v{r}(t)$, which is uniquely determined by the initial conditions $\v{r}(t_0)$ and $\dot{\v{r}}(t_0)$.

In this course, we will begin by looking at the Lagrangian formulation of classical mechanics. While this formulation contains no new physics compared to the Newtonian formulation, there are two distinct benefits:
\begin{enumerate}
    \item We can obtain EOM that do not depend on the coordinate system
    \item It is easier to treat constrained systems.
\end{enumerate}
\subsubsection{The Variational Principle Setup}
To do this, we will use a new approach, known as the \textbf{Variational principle}. To set this up, let us consider the trajectory (as well as some "wrong" paths between the same two endpoints) travelled by a particle:
\begin{center}
    \includegraphics[scale=0.4]{Lecture-1/L1-img1.png}
\end{center}
The trajectory from time $t_1$ to $t_2$ is parametrized by the generalized coordinate $q$. Examples of these are $x_1, y_1, \theta_1$.
\subsubsection{Generalized Coordinates}
For $N$ particles, a generalized coordinate $q_i$ depends on the positions of the $N$ particles:
\[q_i = q_i(\v{r}_1, \cdots, \v{r}_N)\]
In Cartesian coordinates, we have $3N$ coordinates but these are not necessarily independent. In general, we may have constraint functions $f_\alpha(\v{r}, \dot{\v{r}}, t) = 0$ where $\alpha = 1, \cdots, k$ (i.e. $k$ constraints). We then have $q_i$ independent coordinates, where $i = 1, \cdots n$ where $i = 3N - k$. In other words, for a system of $N$ particles, we have $3N - k$ independent/generalized coordinates. Generalized coordinates allow us to avoid worrying about the constrained parameters.
\subsubsection{Hamilton's Principle and the Lagrangian}
Let us consider assigning to each generalized coordinate $q(t)$ a number/value:
\[q(t) \mapsto S[q(t)] \in \RR\]
This is a functional (as denoted by the square brackets), as it takes in a function as an argument. Now, let us consider \textbf{Hamilton's principle}:
\begin{center}
    \textit{The actual path of a particle between times $t_1$ and $t_2$ is such that the line integral:
    \[ S[q] = \int_{t_1}^{t_2} \LL(q_1, \dot{q}_1, t) dt\]
    is stationary.}
\end{center}
Though the principle saysthe integral is stationary, often this correponds to a minimum (though not always). The function $\LL$ is defined as:
\[\LL = T - U\]
where $T$ is the kinetic energy and $U$ is the potential energy. This is called the \textbf{Lagrange Function} or \textbf{Lagrangian}. $S$ is called the \textbf{action}. Though we have not shown it explicitly, this Lagrange function gives the correct trajectory. Note that we have in essence replaced a second order ODE (Newton's second law) with an integral of $\LL$. Having two endpoints $t_1$ and $t_2$ is consistent with the two initial conditions for a second order ODE.
\end{document}


\documentclass{article}

\usepackage[margin=1in]{geometry}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{gensymb}

\newcommand{\norm}[1]{\lvert\lvert \, #1 \, \rvert\rvert}
\newcommand{\cross}{\times}
\newcommand{\pvec}[1]{\vec{#1}^{\,\prime}}

\title{Mobile Robot Kinematics for FTC}
\author{Ryan Brott}
\date{}

\begin{document}

\maketitle

\section{Introduction}

In FIRST Tech Challenge (FTC), kinematics are used all of the time---they're essential to robot operation. However, few teams  understand where the relationships come from or even acknowledge them explicitly, especially for holonomic drives. This paper intends to demystify the role and derivation of drive kinematics with a unified approach.

\section{Drive Kinematics}

This section systematically derives the forward and inverse kinematics for a variety of drives found in FTC (i.e., differential, mecanum, and swerve). The forward kinematics describe how the robot moves with specific wheel velocities. The inverse kinematics describe the opposite---how the wheels move during a given robot motion.

\subsection{Robots as Rigid Bodies}

Most FTC robots have a robust structure that holds all of the components together into one coherent system. When acted upon by a force, the robot does not materially deform, and its constituent parts remain essentially fixed relative to each other. Objects with this property are called rigid bodies, and their kinematics can be summarized with a single translational and angular velocity vector pair. The translational velocity describes the motion of the axis or tation, while the angular velocity magnitude points along the axis of rotation and has magnitude equal to the angular speed.\\
\\
Consider a point in the body with position $\vec{r}$. This position can be written as the sum $\vec{R} + \pvec{r}$ where $\vec{R}$ points from the origin to the axis of rotation and $\pvec{r}$ extends perpendicular to $\vec{\omega}$. Differentiation\footnote{The time derivative of a vector $\vec{v}$ in pure rotation described by $\vec{\omega}$ is $\frac{d\vec{v}}{dt} = \vec{\omega} \cross \vec{v}$.} gives the velocity relationship
\begin{align*}
    \frac{d\vec{r}}{dt} &= \frac{d\vec{R}}{dt} + \frac{d\pvec{r}}{dt} \\
    &= \vec{V} + \vec{\omega} \cross \pvec{r}.
\end{align*}
As FTC robots are generally confined to a plane, we have
\begin{align*}
    \vec{r} &= x \, \hat{\imath} + y \, \hat{\jmath}\\
    \vec{V} &= v_x \, \hat{\imath} + v_y \, \hat{\jmath}\\
    \vec{\omega} &= \omega \, \hat{k}.
\end{align*}
In this special case, the velocity is
\[
    \vec{v} = \frac{d\vec{r}}{dt} = (v_x - y \, \omega) \, \hat{\imath} + (v_y + x \, \omega) \, \hat{\jmath}.
\]
\\
Now for a given configuration $(v_x, v_y, \omega)$ it is possible to compute the two-dimensional velocity of any point on the robot.

\subsection{Differential Drives}

Perhaps the simplest practical mobile robot drive configuration is a differential drive with two normal wheels. These wheels have radius $R$ and are spaced $2l$ units apart. Let the center of rotation (i.e., the midpoint of the wheel positions) be the origin with the wheel axles laying parallel to the y-axis. With this setup, the wheel center positions are \((\pm l, 0)\).\\
\\
Using the results of the previous subsection, the desired velocities are
\begin{align*}
    \vec{v}_l &= (v_x - l \, \omega) \, \hat{\imath} + v_y \, \hat{\jmath}\\
    \vec{v}_r &= (v_x + l \, \omega) \, \hat{\imath} + v_y \, \hat{\jmath}.
\end{align*}
Furthermore, the orientation of the wheels gives us the tangential velocities in terms of the wheel angular velocities
\begin{align*}
    \pvec{v}_l &= \omega_l R \, \hat{\imath}\\
    \pvec{v}_r &= \omega_r R \, \hat{\imath}.
\end{align*}
For traction wheels, the corresponding velocities must \textit{exactly} match. This yields the equations
\begin{align*}
    v_x - l \, \omega &= \omega_l R \\
    v_x + l \, \omega &= \omega_r R \\
    v_y &= 0 \\
    v_y &= 0.
\end{align*}
From this we can solve to obtain the forward and inverse kinematics
\begin{equation*}
    \begin{aligned}[c]
        v_x &= \frac{R}{2}(\omega_l + \omega_r) \\
        v_y &= 0 \\
        \omega &= \frac{R}{2l}(\omega_r - \omega_l),
    \end{aligned}
    \qquad\qquad
    \begin{aligned}[c]
        \omega_l &= \frac{v_x - l \, \omega}{R} \\
        \omega_r &= \frac{v_x + l \, \omega}{R}.
    \end{aligned}
\end{equation*}
In matrix form, these relations are
\begin{equation*}
    \begin{bmatrix}
        v_x \\
        v_y \\
        \omega
    \end{bmatrix}
    =
    \frac{R}{2}
    \begin{bmatrix}
        1 & 1 \\
        0 & 0 \\
        -\frac{1}{l} & \frac{1}{l}
    \end{bmatrix}
    \begin{bmatrix}
        \omega_l \\
        \omega_r
    \end{bmatrix},
    \qquad\qquad
    \begin{bmatrix}
        \omega_l \\
        \omega_r
    \end{bmatrix}
    =
    \frac{1}{R}
    \begin{bmatrix}
        1 & 0 & -l\\
        1 & 0 & l
    \end{bmatrix}
    \begin{bmatrix}
        v_x \\
        v_y \\
        \omega
    \end{bmatrix}.
\end{equation*}
These same kinematics can be applied to differential drives with multiple wheels per side. However, you'll quickly realize that it is no longer possible to satisfy all the constraints. To rotate properly, the wheels must overcome static friction and slide parallel to the wheel axis. This slippage can be mathematically accounted for by treating this scrub direction as a passive degree of freedom (like an omni wheel) as we'll see in the next section. With this assumption, the kinematics are the same as above with all motors on each side sharing the same velocity.

\subsection{Mecanum Drives}

Mecanum drives consist of four mecanum wheels with rollers at $45\degree$ angles with respect to the axle positioned in a rectangle. Looking from above, the main diagonal wheels have counterclockwise-rotated rollers, while the alternate diagonal wheels have clockwise-rotated rollers. The careful combination of these two different directions enables omnidirectional movement. The track width (distance between opposing wheels) is $2l$ and the wheelbase (distance between adjacent wheels) is $2b$. Place the center of rotation at the origin as before with wheel center positions \((\pm l, \pm b)\).\\
\\
Using the same procedure as before, we obtain
\begin{align*}
    \vec{v}_{fl} &= (v_x - l \, \omega) \, \hat{\imath} + (v_y + b \, \omega) \, \hat{\jmath}\\
    \vec{v}_{bl} &= (v_x - l \, \omega) \, \hat{\imath} + (v_y - b \, \omega) \, \hat{\jmath}\\
    \vec{v}_{br} &= (v_x + l \, \omega) \, \hat{\imath} + (v_y - b \, \omega) \, \hat{\jmath}\\
    \vec{v}_{fr} &= (v_x + l \, \omega) \, \hat{\imath} + (v_y + b \, \omega) \, \hat{\jmath}.
\end{align*}
Computing the tangential velocities from the angular velocities is bit trickier for mecanum wheels. The tangential force acts $45\degree$ from the roller, resulting in a speed reduction of $\sqrt{2}$. The direction of the velocity points perpendicular to the roller axis (viewed from above). Thus, the velocities are
\begin{align*}
    \pvec{v}_{fl} &= \frac{R \, \omega_{fl}}{2} (\hat{\imath} - \hat{\jmath})\\
    \pvec{v}_{bl} &= \frac{R \, \omega_{bl}}{2} (\hat{\imath} + \hat{\jmath})\\
    \pvec{v}_{br} &= \frac{R \, \omega_{br}}{2} (\hat{\imath} - \hat{\jmath})\\
    \pvec{v}_{fr} &= \frac{R \, \omega_{fr}}{2} (\hat{\imath} + \hat{\jmath}).
\end{align*}
The vectors can't be directly equated for non-square configurations due to the passive motion of the rollers. Instead we equate the portion of each vector in the tangential velocity direction. For example, the front left constraint is
\[
    \vec{v}_{fl} \cdot \pvec{v}_{fl} = \pvec{v}_{fl} \cdot \pvec{v}_{fl}.
\]
The final system is
\begin{align*}
    \frac{R \, \omega_{fl}}{2}(v_x - l \, \omega - v_y - b \, \omega) &= \frac{R^2\omega_{fl}^2}{2}\\
    \frac{R \, \omega_{bl}}{2}(v_x - l \, \omega + v_y - b \, \omega) &= \frac{R^2\omega_{bl}^2}{2}\\
    \frac{R \, \omega_{br}}{2}(v_x + l \, \omega - v_y + b \, \omega) &= \frac{R^2\omega_{br}^2}{2}\\
    \frac{R \, \omega_{fr}}{2}(v_x + l \, \omega + v_y + b \, \omega) &= \frac{R^2\omega_{fr}^2}{2}.
\end{align*}
which yields the matrix kinematics
\begin{equation*}
    \begin{bmatrix}
        v_x \\
        v_y \\
        \omega
    \end{bmatrix}
    =
    \frac{R}{4}
    \begin{bmatrix}
        1 & 1 & 1 & 1 \\
        -1 & 1 & -1 & 1 \\
        -\frac{1}{l+b} & -\frac{1}{l+b} & \frac{1}{l+b} & \frac{1}{l+b}
    \end{bmatrix}
    \begin{bmatrix}
        \omega_{lf} \\
        \omega_{lb} \\
        \omega_{rb} \\
        \omega_{rf}
    \end{bmatrix}
    \qquad\qquad
    \begin{bmatrix}
        \omega_{lf} \\
        \omega_{lb} \\
        \omega_{rb} \\
        \omega_{rf}
    \end{bmatrix}
    =
    \frac{1}{R}
    \begin{bmatrix}
        1 & -1 & -(l + b) \\
        1 & 1 & -(l + b) \\
        1 & -1 & (l + b) \\
        1 & 1 & (l + b)
    \end{bmatrix}
    \begin{bmatrix}
        v_x \\
        v_y \\
        \omega
    \end{bmatrix}
\end{equation*}

\subsection{Swerve Drives}
Unlike the previous drives, swerve drives have actuated wheel directions. Nevertheless, swerve kinematics can still be derived within the same framework. Let each wheel have coordinates $(x_i, y_i)$, orientation $\phi_i$, tangential velocity $\vec{v}_i$, and angular velocity $\omega_i$.\\
\\
For simplicity, we will assume the swerve wheels do not slip and directly equate the tangential velocities: $v_x - y_i \, \omega = R \, \omega_i \operatorname{cos} \phi_i$ and $v_y + x_i \, \omega = R \, \omega_i \operatorname{sin} \phi_i$. This gives the following inverse kinematics\footnote{$\operatorname{atan2}$ is the standard two-argument arctangent function available in most programming environments}:
\begin{align*}
    \phi_i &= \operatorname{atan2} \big( v_y + x_i \, \omega, \; v_x - y_i \, \omega \big)\\
    \omega_i &= \frac{1}{R} \sqrt{(v_x - y_i \, \omega)^2 + (v_y + x_i \, \omega)^2}
\end{align*}

Forward kinematics for a collection of multiple wheels can be computed by solving the corresponding overdetermined system.

\section{Odometry}
The previous section focused on finding the relationship between configuration and wheel velocities. This by itself is sufficient to employ the inverse kinematics for sending the appropriate control signals. However, the forward kinematics cannot be directly used for odometry. To localize the robot, the local velocities must be integrated into positions and transformed into the global frame.

\subsection{Constant Velocity Odometry}
For simplicity, this method assumes constant translational and rotational velocity over each measurement period. In practice, this is a good assumption so long as measurements are frequent enough (additionally, estimating acceleration or other higher order derivatives robustly from wheel position data is nontrivial).\\
\\
Without loss of generality, we take the robot heading $\theta$ to be $0$ initially. A measurement is then taken $\Delta t$ time later. During this period the robot's global velocity is
$$
\begin{bmatrix}
    \dot{x}_G \\
    \dot{y}_G \\
    \dot{\theta}_G
\end{bmatrix}
=
\begin{bmatrix}
    \operatorname{cos} \theta & -\operatorname{sin} \theta & 0 \\
    \operatorname{sin} \theta & \operatorname{cos} \theta & 0 \\
    0 & 0 & 1
\end{bmatrix}
\begin{bmatrix}
    \dot{x}_R \\
    \dot{y}_R \\
    \dot{\theta}_R
\end{bmatrix}.
$$
Integrating this over time gives
$$
\begin{bmatrix}
    \Delta x_G \\
    \Delta y_G \\
    \Delta \theta_G
\end{bmatrix}
=
\begin{bmatrix}
    \frac{\operatorname{sin} \Delta \theta_R}{\Delta \theta_R} & -\frac{1 - \operatorname{cos} \Delta \theta_R}{\Delta \theta_R} & 0 \\[4pt]
    \frac{1 - \operatorname{cos} \Delta \theta_R}{\Delta \theta_R} & \frac{\operatorname{sin} \Delta \theta_R}{\Delta \theta_R} & 0 \\[4pt]
    0 & 0 & 1
\end{bmatrix}
\begin{bmatrix}
    \Delta x_R \\
    \Delta y_R \\
    \Delta \theta_R
\end{bmatrix}.
$$
This can then be rotated and added to the previous estimate to complete the update
$$
\begin{bmatrix}
    x_{G,\,t+1} \\
    y_{G,\,t+1} \\
    \theta_{G,\,t+1}
\end{bmatrix}
=
\begin{bmatrix}
    x_{G,\,t} \\
    y_{G,\,t} \\
    \theta_{G,\,t}
\end{bmatrix}
+
\begin{bmatrix}
    \operatorname{cos} \theta_{G,\,t} & -\operatorname{sin} \theta_{G,\,t} & 0 \\
    \operatorname{sin} \theta_{G,\,t} & \operatorname{cos} \theta_{G,\,t} & 0 \\
    0 & 0 & 1
\end{bmatrix}
\begin{bmatrix}
    \frac{\operatorname{sin} \Delta \theta_R}{\Delta \theta_R} & -\frac{1 - \operatorname{cos} \Delta \theta_R}{\Delta \theta_R} & 0 \\[4pt]
    \frac{1 - \operatorname{cos} \Delta \theta_R}{\Delta \theta_R} & \frac{\operatorname{sin} \Delta \theta_R}{\Delta \theta_R} & 0 \\[4pt]
    0 & 0 & 1
\end{bmatrix}
\begin{bmatrix}
    \Delta x_R \\
    \Delta y_R \\
    \Delta \theta_R
\end{bmatrix}.
$$

\subsection{Tracking Wheels}
Tracking wheels are passive omni wheels intended solely for odometry. Each wheel has an arbitrary position $(x_i, y_i)$ and orientation $\phi_i$ in the robot frame. Similar to swerve, the tangential velocities of each wheel are $\vec{v}_i = (v_x - y_i \, \omega) \, \hat{\imath} + (v_y + x_i \, \omega) \, \hat{\jmath}$ and $\pvec{v}_i = R \, \omega_i (\operatorname{cos} \phi_i \, \hat{\imath} + \operatorname{sin} \phi_i \, \hat{\jmath})$.
$$
    \omega_i = \frac{1}{R}\Big[v_x \operatorname{cos} \phi_i + v_y \operatorname{sin} \phi_i + \omega \, (x_i \operatorname{sin} \phi_i - y_i \operatorname{cos} \phi_i)\Big]
$$
If placed properly, three tracking wheels are sufficient to determine the $(v_x, v_y, \omega)$ configuration:
$$
\begin{bmatrix}
    \omega_1\\
    \omega_2\\
    \omega_3
\end{bmatrix}
=
\frac{1}{R}
\begin{bmatrix}
    \operatorname{cos} \phi_1 & \operatorname{sin} \phi_1 & x_1 \operatorname{sin} \phi_1 - y_1 \operatorname{cos} \phi_1 \\
    \operatorname{cos} \phi_2 & \operatorname{sin} \phi_2 & x_2 \operatorname{sin} \phi_2 - y_2 \operatorname{cos} \phi_2 \\
    \operatorname{cos} \phi_3 & \operatorname{sin} \phi_3 & x_3 \operatorname{sin} \phi_3 - y_3 \operatorname{cos} \phi_3 \\
\end{bmatrix}
\begin{bmatrix}
    v_x\\
    v_y\\
    \omega
\end{bmatrix}
$$
$$
\begin{bmatrix}
    \Delta \theta_1\\
    \Delta \theta_2\\
    \Delta \theta_3
\end{bmatrix}
=
\frac{1}{R}
\begin{bmatrix}
    \operatorname{cos} \phi_1 & \operatorname{sin} \phi_1 & x_1 \operatorname{sin} \phi_1 - y_1 \operatorname{cos} \phi_1 \\
    \operatorname{cos} \phi_2 & \operatorname{sin} \phi_2 & x_2 \operatorname{sin} \phi_2 - y_2 \operatorname{cos} \phi_2 \\
    \operatorname{cos} \phi_3 & \operatorname{sin} \phi_3 & x_3 \operatorname{sin} \phi_3 - y_3 \operatorname{cos} \phi_3 \\
\end{bmatrix}
\begin{bmatrix}
    \Delta x_R\\
    \Delta y_R\\
    \Delta \theta_R
\end{bmatrix}
$$
For example, take the common configuration of two tracking wheels placed parallel to the robot's $x$-axis at $(0,\pm y_0)$ and a third placed parallel to the robot's $y$-axis at $(x_0,0)$. For simplicity, all wheels point in the positive direction of their corresponding axis. This gives the following matrices:
\begin{equation*}
    \begin{bmatrix}
        \Delta \theta_1\\
        \Delta \theta_2\\
        \Delta \theta_3
    \end{bmatrix}
    =
    \frac{1}{R}
    \begin{bmatrix}
        1 & 0 & -y_0 \\
        1 & 0 & y_0 \\
        0 & 1 & x_0 \\
    \end{bmatrix}
    \begin{bmatrix}
        \Delta x_R\\
        \Delta y_R\\
        \Delta \theta_R
    \end{bmatrix}
\end{equation*}
\begin{equation*}
    \begin{bmatrix}
        \Delta x_R\\
        \Delta y_R\\
        \Delta \theta_R
    \end{bmatrix}
    =
    \frac{R}{2y_0}
    \begin{bmatrix}
        y_0 & y_0 & 0\\
        x_0 & -x_0 & 2y_0\\
        -1 & 1 & 0\\
    \end{bmatrix}
    \begin{bmatrix}
        \Delta \theta_1\\
        \Delta \theta_2\\
        \Delta \theta_3
    \end{bmatrix}
    \quad\leftrightarrow\quad
    \begin{aligned}[c]
        \Delta x_R &= \frac{R}{2}(\Delta \theta_1 + \Delta \theta_2)\\
        \Delta y_R &= R\bigg[\frac{x_0}{2y_0}(\Delta \theta_1 - \Delta \theta_2) + \Delta \theta_3\bigg]\\
        \Delta \theta_R &= \frac{R}{2y_0}(\Delta \theta_2 - \Delta \theta_1)
    \end{aligned}
\end{equation*}
Two tracking wheels and a heading sensor is also sufficient for localization.
$$
\begin{bmatrix}
    \Delta \theta_1\\
    \Delta \theta_2\\
    \Delta \theta_3
\end{bmatrix}
=
\frac{1}{R}
\begin{bmatrix}
    \operatorname{cos} \phi_1 & \operatorname{sin} \phi_1 & x_1 \operatorname{sin} \phi_1 - y_1 \operatorname{cos} \phi_1 \\
    \operatorname{cos} \phi_2 & \operatorname{sin} \phi_2 & x_2 \operatorname{sin} \phi_2 - y_2 \operatorname{cos} \phi_2 \\
    0 & 0 & R\\
\end{bmatrix}
\begin{bmatrix}
    \Delta x_R\\
    \Delta y_R\\
    \Delta \theta_R
\end{bmatrix}
$$

\end{document}