\chapter{\(K\)-Compensated de Casteljau}\label{chap:k-compensated}

\section{Introduction}

In computer aided geometric design, polynomials are usually expressed in
Bernstein form. Polynomials in this form are usually evaluated by the
de Casteljau algorithm. This algorithm has a round-off error bound
which grows only linearly with degree, even though the number of
arithmetic operations grows quadratically. The Bernstein basis is
optimally suited (\cite{Farouki1987, Delgado2015, Mainar2005})
for polynomial evaluation; it is
typically more accurate than the monomial basis, for example in
Figure~\ref{fig:horner-inferior} evaluation via Horner's method produces
a jagged curve for points near a triple root, but the de Casteljau algorithm
produces a smooth curve. Nevertheless the de Casteljau
algorithm returns results arbitrarily less accurate than the working
precision \(\mach\) when evaluating \(p(s)\) is ill-conditioned.
The relative accuracy of the computed
evaluation with the de Casteljau algorithm (\texttt{DeCasteljau}) satisfies
(\cite{Mainar1999}) the following a priori bound:
\begin{equation}\label{eq:de-casteljau-error}
  \frac{\left|p(s) - \mathtt{DeCasteljau}(p, s)\right|}{\left|p(s)\right|} \leq
  \cond{p, s} \times \bigO{\mach}.
\end{equation}
In the right-hand side of this inequality, \(\mach\) is the computing
precision and the condition number \(\cond{p, s} \geq 1\) only depends
on \(s\) and the Bernstein coefficients of \(p\) --- its expression will
be given further.

\begin{figure}
  \includegraphics{../images/k-compensated/horner_inferior.pdf}
  \centering
  \captionsetup{width=.75\linewidth}
  \caption{Comparing Horner's method to the de Casteljau method for
    evaluating \(p(s) = (2s - 1)^3\) in the neighborhood of its
    multiple root \(1/2\).}
  \label{fig:horner-inferior}
\end{figure}

For ill-conditioned problems, such as evaluating \(p(s)\) near a
multiple root, the condition number may be arbitrarily large, i.e.
\(\cond{p, s} > 1 / \mach\), in
which case most or all of the computed digits will be incorrect.
In some cases, even the order of magnitude of the computed value
of \(p(s)\) can be incorrect.

To address ill-conditioned problems, error-free transformations (EFT) can
be applied in \emph{compensated algorithms} to account for round-off.
Error-free transformations were studied in great detail in \cite{Ogita2005}
and open a large number of applications.
In \cite{langlois_et_al:DSP:2006:442}, a compensated version of Horner's
algorithm described to evaluate a polynomial in the monomial basis.
In \cite{Jiang2010},
a similar method was described to perform a compensated version of the de
Casteljau algorithm. In both cases, the \(\cond{p, s}\) factor is moved
from \(\mach\) to \(\mach^2\) and the computed value is as accurate
as if the computations were done in twice the working precision. For example,
the compensated de Casteljau algorithm (\texttt{CompDeCasteljau}) satisfies
\begin{equation}\label{eq:de-casteljau-2-error}
  \frac{\left|p(s) - \mathtt{CompDeCasteljau}(p, s)\right|}{
    \left|p(s)\right|} \leq \mach + \cond{p, s} \times
    \bigO{\mach^2}.
\end{equation}
For problems with \(\cond{p, s} < 1 / \mach^2\), the relative error
is \(\mach\), i.e. accurate to full precision, aside from rounding to the
nearest floating point number. Figure~\ref{fig:jlcs-10} shows this shift
in relative error from \texttt{DeCasteljau} to \texttt{CompDeCasteljau}.

\begin{figure}
  \includegraphics{../images/k-compensated/jlcs10_plot.pdf}
  \centering
  \captionsetup{width=.75\linewidth}
  \caption{Evaluation of \(p(s) = (s - 1)\left(s - 3/4\right)^7\)
    represented in Bernstein form.}
  \label{fig:jlcs-10}
\end{figure}

In \cite{Graillat2009}, the authors generalized the compensated Horner's
algorithm to produce a method for evaluating a polynomial as if
the computations were done in \(K\) times the working precision for
any \(K \geq 2\). This result motivates this chapter, though the
approach there is somewhat different than ours. They perform each computation
with error-free transformations and interpret the errors as coefficients of new
polynomials. They then evaluate the error polynomials, which (recursively)
generate second order error polynomials and so on. This recursive property
causes the number of operations to grow exponentially in \(K\). Here, we
instead have a fixed number of error groups, each corresponding to round-off
from the group above it. For example, when
\((1 - s) b_j^{(n)} + s b_{j + 1}^{(n)}\) is computed in floating point, any
error is filtered down to the error group below it.

As in~\eqref{eq:de-casteljau-error}, the accuracy of the compensated
result~\eqref{eq:de-casteljau-2-error} may be arbitrarily bad for
ill-conditioned
polynomial evaluations. For example, as the condition number grows in
Figure~\ref{fig:jlcs-10}, some points have relative error exactly equal to
\(1\); this indicates that \\ \(\mathtt{CompDeCasteljau}(p, s) = 0\), which is
a complete failure to evaluate the order of magnitude of \(p(s)\). For
root-finding problems \(\mathtt{CompDeCasteljau}(p, s) = 0\) when
\(p(s) \neq 0\) can cause premature convergence and incorrect results.
We describe how to defer rounding into progressively
smaller error groups and improve the accuracy of the computed result by a
factor of \(\mach\) for every error group added. So we derive
\texttt{CompDeCasteljauK}, a \(K\)-fold compensated de Casteljau algorithm
that satisfies the following a priori bound for any arbitrary integer \(K\):
\begin{equation}
  \frac{\left|p(s) - \mathtt{CompDeCasteljauK}(p, s, K)\right|}{
    \left|p(s)\right|} \leq \mach + \cond{p, s} \times
    \bigO{\mach^K}.
\end{equation}
This means that the computed value with \texttt{CompDeCasteljauK} is now
as accurate as the result of the de Casteljau algorithm performed in
\(K\) times the working precision with a final rounding back to the
working precision.

The chapter is organized as follows. In Section~\ref{sec:compensated-2},
the compensated algorithm for polynomial evaluation from \cite{Jiang2010} is
reviewed and notation is established for the expansion. In
Section~\ref{sec:compensated-k}, the \(K\)-compensated algorithm is provided
and a forward error analysis is performed. Finally, in
Section~\ref{sec:numerical} we perform two numerical experiments to
give practical examples of the theoretical error bounds.
(See Chapter~\ref{chap:preliminaries} to review notation for error analysis
with floating point operations, review results about error-free
transformations or to review the de Casteljau algorithm.)

\section{Compensated de Casteljau}\label{sec:compensated-2}

In this section we review the compensated de Casteljau algorithm
from \cite{Jiang2010}. In order to track the local errors at
each update step, we use four EFTs:
\begin{align}
\left[\widehat{r}, \rho\right] &= \mathtt{TwoSum}(1, -s) \\
\left[P_1, \pi_1\right] &= \mathtt{TwoProd}\left(
    \widehat{r}, \widehat{b}_j^{(k + 1)}\right) \\
\left[P_2, \pi_2\right] &= \mathtt{TwoProd}\left(
    s, \widehat{b}_{j + 1}^{(k + 1)}\right) \\
\left[\widehat{b}_j^{(k)}, \sigma_3\right] &= \mathtt{TwoSum}(P_1, P_2)
\end{align}
With these, we can exactly describe the local error between the exact
update and computed update:
\begin{gather}
\ell_{1, j}^{(k)} = \pi_1 + \pi_2 + \sigma_3 + \rho \cdot
  \widehat{b}_j^{(k + 1)} \label{eq:ell-j} \\
(1 - s) \cdot \widehat{b}_j^{(k + 1)} +
  s \cdot \widehat{b}_{j + 1}^{(k + 1)} =
\widehat{b}_j^{(k)} + \ell_{1, j}^{(k)}.
\end{gather}

\begin{figure}
  \includegraphics[width=0.375\textwidth]{tikz_local_err.pdf}
  \centering
  \captionsetup{width=.75\linewidth}
  \caption{Local round-off errors}
  \label{fig:loc-err-accumulate}
\end{figure}

\noindent By defining the global errors at each step
\begin{equation}
  \db{1}_j^{(k)} = b_j^{(k)} - \widehat{b}_j^{(k)}
\end{equation}
we can see (Figure~\ref{fig:loc-err-accumulate}) that the local errors
accumulate in
\(\db{1}^{(k)}\):
\begin{equation}\label{eq:err-update}
  \db{1}_j^{(k)} = (1 - s) \cdot \db{1}_j^{(k + 1)} + s \cdot
  \db{1}_{j + 1}^{(k + 1)} + \ell_{1, j}^{(k)}.
\end{equation}
When computed in exact arithmetic
\begin{equation}
  p(s) = \widehat{b}_0^{(0)} + \db{1}_0^{(0)}
\end{equation}
and by using \eqref{eq:err-update}, we can continue to compute
approximations of \(\db{1}_j^{(k)}\). The idea behind the compensated
de Casteljau algorithm is to compute both the local error and the updates
of the global error with floating point operations:

\begin{breakablealgorithm}
  \caption{\textit{Compensated de Casteljau
      algorithm for polynomial evaluation.}}
  \label{alg:comp-de-casteljau}

  \begin{algorithmic}
    \Function{\(\mathtt{result} = \mathtt{CompDeCasteljau}\)}{$b, s$}
      \State \(n = \texttt{length}(b) - 1\)
      \State \(\left[\widehat{r}, \rho\right] = \mathtt{TwoSum}(1, -s)\)
      \\
      \For{\(j = 0, \ldots, n\)}
        \State \(\widehat{b}_j^{(n)} = b_j\)
        \State \(\cdb{1}_j^{(n)} = 0\)
      \EndFor
      \\
      \For{\(k = n - 1, \ldots, 0\)}
        \For{\(j = 0, \ldots, k\)}
          \State \(\left[P_1, \pi_1\right] = \mathtt{TwoProd}\left(
              \widehat{r}, \widehat{b}_j^{(k + 1)}\right)\)
          \State \(\left[P_2, \pi_2\right] = \mathtt{TwoProd}\left(
              s, \widehat{b}_{j + 1}^{(k + 1)}\right)\)
          \State \(\left[\widehat{b}_j^{(k)}, \sigma_3\right] =
              \mathtt{TwoSum}(P_1, P_2)\)
          \State \(\widehat{\ell}_{1, j}^{(k)} = \pi_1 \oplus \pi_2 \oplus
              \sigma_3 \oplus \left(\rho \otimes
              \widehat{b}_j^{(k + 1)}\right)\)
          \State \(\cdb{1}_j^{(k)} =
              \widehat{\ell}_{1, j}^{(k)} \oplus
              \left(s \otimes \cdb{1}_{j + 1}^{(k + 1)}
              \right) \oplus
              \left(\widehat{r} \otimes
              \cdb{1}_j^{(k + 1)}\right)\)
        \EndFor
      \EndFor
      \\
      \State \(\mathtt{result} = \widehat{b}_0^{(0)} \oplus
          \cdb{1}_0^{(0)}\)
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}

\noindent  When comparing this computed error to the exact error, the
difference depends only on \(s\) and the Bernstein
coefficients of \(p\). Using a bound (Lemma~\ref{lemma:db-lemma}) on the
round-off error when computing \(\db{1}^{(0)}\), the algorithm can
be shown to be as accurate as if the computations were done in twice
the working precision:

\begin{theorem}[\cite{Jiang2010}, Theorem 5]
  If no underflow occurs, \(n \geq 2\) and \(s \in \left[0, 1\right]\)
  \begin{equation}
    \frac{\left|p(s) - \mathtt{CompDeCasteljau}(p, s)\right|}{
      \left|p(s)\right|} \leq \mach + 2 \gamma_{3n}^2 \cond{p, s}.
  \end{equation}
\end{theorem}

\begin{figure}
  \includegraphics{../images/k-compensated/compensated_insufficient.pdf}
  \centering
  \captionsetup{width=.75\linewidth}
  \caption{The compensated de Casteljau method starts to lose accuracy
    for \(p(s) = (2s - 1)^3 (s - 1)\) in the neighborhood of its
    multiple root \(1/2\).}
  \label{fig:compensated-insufficient}
\end{figure}

Unfortunately, Figure~\ref{fig:compensated-insufficient} shows how
\texttt{CompDeCasteljau} starts to break down in a region of
high condition number (caused by a multiple root with multiplicity
higher than two). For example, the point
\(s = \frac{1}{2} + 1001\mach\)
--- which is in the plotted region \(\left|s - \frac{1}{2}\right|
\leq \frac{3}{2} \cdot 10^{-11}\) --- evaluates to exactly \(0\) when
it should be \(\bigO{\mach^3}\). As shown in
Table~\ref{tab:exact-computation}, the breakdown occurs because
\(\widehat{b}_0^{(0)} = -\cdb{1}_0^{(0)} = \mach / 16\).

\begin{table}
  %% See: ``scripts/verify_table.py``.
  \centering
  \begin{adjustbox}{max width=\textwidth}
  \begin{tabular}{>{$}c<{$} >{$}c<{$} >{$}c<{$} >{$}c<{$} >{$}c<{$} >{$}c<{$}}
    \toprule
    k & j & \widehat{b}_j^{(k)} & \cdb{1}_j^{(k)} &
        \db{1}_j^{(k)} - \cdb{1}_j^{(k)} \\
    \midrule
    3 & 0 & 0.125 - 1.75 (1001 \mach) - 0.25 \mach & 0.25\mach & 0 \\
    3 & 1 & -0.125 + 1.25(1001 \mach) + 0.25 \mach & -0.25\mach & 0 \\
    3 & 2 & 0.125 - 0.75 (1001 \mach) & 0 & 0 \\
    3 & 3 & -0.125 + 0.25 (1001 \mach) & 0 & 0 \\
    \midrule
    2 & 0 & -0.5 (1001 \mach) & 3 (1001 \mach)^2 & 0 \\
    2 & 1 & 0.5(1001 \mach) + 0.125 \mach & -0.125\mach - 2 (1001 \mach)^2 &
        0 \\
    2 & 2 & -0.5 (1001 \mach) & (1001 \mach)^2 & 0 \\
    \midrule
    1 & 0 & 0.0625\mach + (1001 \mach)^2 + 239\mach^2 &
        -0.0625\mach + 0.5  (1001 \mach)^2 - 239 \mach^2 & -5 (1001\mach)^3 \\
    1 & 1 & 0.0625\mach - (1001 \mach)^2 - 239\mach^2 &
        -0.0625\mach - 0.5  (1001 \mach)^2 + 239 \mach^2 & 3 (1001\mach)^3 \\
    \midrule
    0 & 0 & 0.0625 \mach & -0.0625 \mach &
        -4 (1001 \mach)^3 + 8 (1001 \mach)^4 \\
    \bottomrule
  \end{tabular}
  \end{adjustbox}
  \captionsetup{width=.75\linewidth}
  \caption{Terms computed by \texttt{CompDeCasteljau} when evaluating
    \(p(s) = (2s - 1)^3 (s - 1)\) at the point
    \(s = \frac{1}{2} + 1001 \mach\)}
  \label{tab:exact-computation}
\end{table}

\section{\texorpdfstring{\(K\)}{K}-Compensated de Casteljau}
\label{sec:compensated-k}

\subsection{Algorithm Specified}

In order to raise from twice the working precision to \(K\) times the
working precision, we continue using EFTs when computing
\(\cdb{1}^{(k)}\). By tracking the round-off from each
floating point evaluation via an EFT, we can form a cascade of global errors:
\begin{align}
  b_j^{(k)} &= \widehat{b}_j^{(k)} + \db{1}_j^{(k)} \\
  \db{1}_j^{(k)} &= \cdb{1}_j^{(k)} + \db{2}_j^{(k)} \\
  \db{2}_j^{(k)} &= \cdb{2}_j^{(k)} +
  \db{3}_j^{(k)} \\
  %% H/T: https://tex.stackexchange.com/a/7651/32270
  &\mathrel{\makebox[\widthof{=}]{\vdots}} \nonumber
\end{align}
In the same way local error can be tracked when updating
\(\widehat{b}_j^{(k)}\), it can be tracked for updates that happen down
the cascade:
\begin{alignat}{4}
  (1 - s) \cdot \widehat{b}_j^{(k + 1)} &+
  s \cdot \widehat{b}_{j + 1}^{(k + 1)} &&  &&=
  \widehat{b}_j^{(k)} &&+ \ell_{1, j}^{(k)} \\
  (1 - s) \cdot \cdb{1}_j^{(k + 1)} &+
  s \cdot \cdb{1}_{j + 1}^{(k + 1)} &&+ \ell_{1, j}^{(k)} &&=
  \cdb{1}_j^{(k)} &&+ \ell_{2, j}^{(k)} \\
  (1 - s) \cdot \cdb{2}_j^{(k + 1)} &+
  s \cdot \cdb{2}_{j + 1}^{(k + 1)} &&+ \ell_{2, j}^{(k)} &&=
  \cdb{2}_j^{(k)} &&+ \ell_{3, j}^{(k)} \\
  &  &&  &&\mathrel{\makebox[\widthof{=}]{\vdots}} && \nonumber
\end{alignat}

\begin{figure}
  \includegraphics[width=0.875\textwidth]{tikz_filtration.pdf}
  \centering
  \captionsetup{width=.75\linewidth}
  \caption{Filtering errors}
  \label{fig:error-filtration}
\end{figure}

In \texttt{CompDeCasteljau} (Algorithm~\ref{alg:comp-de-casteljau}), after
a single stage of error filtering we
``give up'' and use \(\cdb{1}\) instead of
\(\db{1}\) (without keeping around any information about the
round-off error). In order to obtain results that are as accurate as if
computed in \(K\) times the working precision, we must continue filtering
(see Figure~\ref{fig:error-filtration})
errors down \((K - 1)\) times, and only at the final level do we accept
the rounded
\(\cdb{K - 1}\) in place of the exact
\(\db{K - 1}\).

When computing \(\cdb{F}\) (i.e. the error after
\(F\) stages of filtering)
there will be several sources of round-off. In particular, there will be
\begin{itemize}
\itemsep 0em
\item errors when computing \(\widehat{\ell}_{F, j}^{(k)}\) from the
  terms in \(\ell_{F, j}^{(k)}\)
\item an error
for the ``missing'' \(\rho \cdot \cdb{F}_j^{(k + 1)}\) in
\((1 - s) \cdot \cdb{F}_j^{(k + 1)}\)
\item an error from the product
  \(\widehat{r} \otimes \cdb{F}_j^{(k + 1)}\)
\item an error from the product
  \(s \otimes \cdb{F}_{j + 1}^{(k + 1)}\)
\item two errors from the two \(\oplus\) when combining the three
  terms in
  \(\widehat{\ell}_{F, j}^{(k)} \oplus
  \left(s \otimes \cdb{F}_{j + 1}^{(k + 1)}\right) \oplus
  \left(\widehat{r} \otimes \cdb{F}_j^{(k + 1)}\right)\)
\end{itemize}
For example, in~\eqref{eq:ell-j}:
%% H/T: https://tex.stackexchange.com/a/154333/32270
\begin{equation}
\ell_{1, j}^{(k)} =
    \underbrace{\vphantom{\rho \cdot \widehat{b}_j^{(k + 1)}} \pi_1}_{
        \vphantom{(1 - s) \widehat{b}_j^{(k + 1)}}
        P_1 = \widehat{r} \otimes \widehat{b}_j^{(k + 1)}} +
    \underbrace{\vphantom{\rho \cdot \widehat{b}_j^{(k + 1)}} \pi_2}_{
        \vphantom{(1 - s) \widehat{b}_j^{(k + 1)}}
        P_2 = s \otimes \widehat{b}_{j + 1}^{(k + 1)}} +
    \underbrace{\vphantom{\rho \cdot \widehat{b}_j^{(k + 1)}} \sigma_3}_{
        \vphantom{(1 - s) \widehat{b}_j^{(k + 1)}}
        P_1 \oplus P_2} +
    \underbrace{\rho \cdot \widehat{b}_j^{(k + 1)}}_{
        (1 - s) \widehat{b}_j^{(k + 1)}}
\end{equation}
After each stage, we'll always have
\begin{equation}
\ell_{F, j}^{(k)} = e_1 + \cdots + e_{5F - 2} + \rho \cdot
\cdb{F - 1}_j^{(k + 1)}
\end{equation}
where the terms \(e_1, \ldots, e_{5F - 2}\) come from using \texttt{TwoSum}
and \texttt{TwoProd} when computing \(\cdb{F - 1}_j^{(k)}\)
and the \(\rho\) term comes from the round-off
in \(1 \ominus s\) when multiplying \((1 - s)\) by
\(\cdb{F - 1}_j^{(k + 1)}\). With this in mind, we
can define an EFT (\texttt{LocalErrorEFT}) that computes
\(\widehat{\ell}\) and tracks all round-off errors generated in
the process:

\begin{breakablealgorithm}
  \caption{\textit{EFT for computing the local error.}}
  \label{alg:local-error-eft}

  \begin{algorithmic}
    \Function{\(\left[\eta, \widehat{\ell}\right] =
        \mathtt{LocalErrorEFT}\)}{$e, \rho, \delta b$}
      \State \(L = \texttt{length}(e)\)
      \\
      \State \(\left[\widehat{\ell}, \eta_1\right] =
          \mathtt{TwoSum}(e_1, e_2)\)
      \For{\(j = 3, \ldots, L\)}
        \State \(\left[\widehat{\ell}, \eta_{j - 1}\right] =
            \mathtt{TwoSum}\left(\widehat{\ell}, e_j\right)\)
      \EndFor
      \\
      \State \(\left[P, \eta_L\right] =
          \mathtt{TwoProd}\left(\rho, \delta b\right)\)
      \State \(\left[\widehat{\ell}, \eta_{L + 1}\right] =
          \mathtt{TwoSum}\left(\widehat{\ell}, P\right)\)
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}

\noindent With this EFT in place\footnote{And the related
\texttt{LocalError} in Algorithm~\ref{alg:local-error}}, we can
perform \((K - 1)\) error filtrations. Once we've computed the \(K\) stages
of global errors, they can be combined with
\texttt{SumK} (Algorithm~\ref{alg:sum-k}) to produce a sum that is as
accurate as if computed in \(K\) times the working precision.

\begin{breakablealgorithm}
  \caption{\(K\)-\textit{compensated de Casteljau algorithm.}}
  \label{alg:k-comp-de-casteljau}

  \begin{algorithmic}
    \Function{\(\mathtt{result} = \mathtt{CompDeCasteljauK}\)}{$b, s, K$}
      \State \(n = \texttt{length}(b) - 1\)
      \State \(\left[\widehat{r}, \rho\right] = \mathtt{TwoSum}(1, -s)\)
      \\
      \For{\(j = 0, \ldots, n\)}
        \State \(\widehat{b}_j^{(n)} = b_j\)
        \For{\(F = 1, \ldots, K - 1\)}
          \State \(\cdb{F}_j^{(n)} = 0\)
        \EndFor
      \EndFor
      \\
      \For{\(k = n - 1, \ldots, 0\)}
        \For{\(j = 0, \ldots, k\)}
          \State \(\left[P_1, \pi_1\right] = \mathtt{TwoProd}\left(
              \widehat{r}, \widehat{b}_j^{(k + 1)}\right)\)
          \State \(\left[P_2, \pi_2\right] = \mathtt{TwoProd}\left(
              s, \widehat{b}_{j + 1}^{(k + 1)}\right)\)
          \State \(\left[\widehat{b}_j^{(k)}, \sigma_3\right] =
              \mathtt{TwoSum}(P_1, P_2)\)
          \\
          \State \(e = \left[\pi_1, \pi_2, \sigma_3\right]\)
          \State \(\delta b = \widehat{b}_j^{(k + 1)}\)
          \\
          \For{\(F = 1, \ldots, K - 2\)}
            \State \(\left[\eta, \widehat{\ell}\right] =
                \mathtt{LocalErrorEFT}(e, \rho, \delta b)\)
            \State \(L = \texttt{length}(\eta)\)
            \\
            \State \(\left[P_1, \eta_{L + 1}\right] = \mathtt{TwoProd}\left(
                s, \cdb{F}_{j + 1}^{(k + 1)}\right)\)
            \State \(\left[S_2, \eta_{L + 2}\right] =
                \mathtt{TwoSum}\left(\widehat{\ell}, P_1\right)\)
            \State \(\left[P_3, \eta_{L + 3}\right] = \mathtt{TwoProd}\left(
                \widehat{r}, \cdb{F}_j^{(k + 1)}\right)\)
            \State \(\left[\cdb{F}_j^{(k)}, \eta_{L + 4}\right]
                = \mathtt{TwoSum}\left(S_2, P_3\right)\)
            \\
            \State \(e = \eta\)
            \State \(\delta b = \cdb{F}_j^{(k + 1)}\)
          \EndFor
          \\
          \State \(\widehat{\ell} =
                \mathtt{LocalError}(e, \rho, \delta b)\)
          \State \(\cdb{K - 1}_j^{(k)} =
              \widehat{\ell} \oplus
              \left(s \otimes \cdb{K - 1}_{j + 1}^{(k + 1)}
              \right) \oplus
              \left(\widehat{r} \otimes
              \cdb{K - 1}_j^{(k + 1)}\right)\)
        \EndFor
      \EndFor
      \\
      \State \(\mathtt{result} = \mathtt{SumK}\left(\left[
        \widehat{b}_0^{(0)}, \ldots, \cdb{K - 1}_0^{(0)}\right], K\right)\)
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}

\noindent Noting that \(\ell_{F, j}\) contains \(5F - 1\) terms, one can
show that \texttt{CompDeCasteljauK} (Algorithm~\ref{alg:k-comp-de-casteljau})
requires
\begin{equation}
(15K^2 + 11K - 34)T_n + 6K^2 - 11K + 11 =
\bigO{n^2 K^2}
\end{equation}
flops to evaluate a degree \(n\) polynomial, where \(T_n\) is the
\(n\)th triangular number. As a comparison, the non-compensated form of
de Casteljau requires \(3 T_n + 1\) flops. In total this will require
\((3K - 4)T_n\) uses of \texttt{TwoProd}. On hardware that supports
FMA, \texttt{TwoProdFMA} (Algorithm~\ref{alg:two-prod-fma}) can be used
instead, lowering the flop count by \(15(3K - 4)T_n\). Another way
to lower the total flop count is to just use
\(\widehat{b}_0^{(0)} \oplus \cdots \oplus \cdb{K - 1}_0^{(0)}\)
instead of \texttt{SumK}; this will reduce the total by
\(6(K - 1)^2\) flops. When using a standard sum, the results produced
are (empirically) identical to those with \texttt{SumK}. This makes
sense: the whole point of \texttt{SumK}
is to filter errors in a summation so that the final operation produces
a sum of the form \(v_1 \oplus \cdots \oplus v_K\) where each
term is smaller than the previous by a factor of \(\mach\). This
property is already satisfied for the \(\cdb{F}_0^{(0)}\) so in
practice the \(K\)-compensated summation is likely not needed.

\subsection{Error bound for polynomial evaluation}

\begin{theorem}[\cite{Ogita2005}, Proposition 4.10]\label{thm:sum-k}
A summation can be computed (\texttt{SumK}, Algorithm~\ref{alg:sum-k})
with results that are as accurate as if computed in \(K\) times the
working precision. When computed this way, the result satisfies:
\begin{equation}
\left|\mathtt{SumK}(v, K) - \sum_{j = 1}^n v_j\right| \leq
\left(\mach + 3 \gamma_{n - 1}^2\right) \left|\sum_{j = 1}^n v_j\right| +
\gamma_{2n - 2}^K \sum_{j = 1}^n \left|v_j\right|.
\end{equation}
\end{theorem}

\begin{lemma}[\cite{Jiang2010}, Theorem 4]\label{lemma:db-lemma}
The second order error \(\db{2}^{(0)}_0\) satisfies\footnote{The authors
  missed one round-off error so used \(\gamma_{3n + 1}\) where
  \(\gamma_{3n + 2}\) would have followed from their arguments.}
\begin{equation}
  \left|\db{1}^{(0)}_0 - \cdb{1}^{(0)}_0\right| =
  \left|\db{2}^{(0)}_0\right| \leq 2 \gamma_{3n + 2} \gamma_{3(n - 1)}
  \widetilde{p}(s).
\end{equation}
\end{lemma}

To enable a bound on the \(K\) order error \(\db{K}^{(0)}_0\), it's necessary
to understand the difference between the exact local errors \(\ell_{F, j}\)
and the computed equivalents \(\widehat{\ell}_{F, j}\). To do this, we define
\begin{equation}
\widetilde{\ell}_{F, j} \coloneqq \left|e_1\right| +
\cdots + \left|e_{5F - 2}\right| + \left|\rho \cdot
\cdb{F - 1}_j^{(k + 1)}\right|.
\end{equation}

\begin{lemma}\label{lemma:ell-tilde}
The local error bounds \(\widetilde{\ell}_{F, j}\) satisfy:
\begin{align}
\widetilde{\ell}_{1, j}^{(k)} &\leq
  \gamma_3 \left(
  (1 - s) \left|\widehat{b}_j^{(k + 1)}\right| +
  s \left|\widehat{b}_{j + 1}^{(k + 1)}\right|\right)
  \label{eq:ell-tilde-1} \\
\widetilde{\ell}_{F + 1, j}^{(k)} &\leq
  \gamma_3 \left(
  (1 - s) \left|\cdb{F}_j^{(k + 1)}\right| +
  s \left|\cdb{F}_{j + 1}^{(k + 1)}\right|\right) +
  \gamma_{5F} \cdot \widetilde{\ell}_{F, j}^{(k)}
  \text{ for } F \geq 1.
\end{align}
\end{lemma}

\begin{proof}
See proof in Section~\ref{proof:ell-tilde}.
\end{proof}

As we'll see soon (Lemma~\ref{lemma:k-order}), putting a bound on
sums of the form \(\sum_{j = 0}^k \ell_{F, j}^{(k)} B_{j, k}(s)\) will
be useful to get an overall bound on the relative error for
\texttt{CompDeCasteljauK}, so we define
\(L_{F, k} \coloneqq \sum_{j = 0}^k \ell_{F, j}^{(k)} B_{j, k}(s)\).

\begin{lemma}\label{lemma:L-and-D-bounds}
For \(s \in \left[0, 1\right]\), the Bernstein-type error sum defined above
satisfies the following bounds:
\begin{align}
L_{F, n - k} &\leq \left[\left(3^F \binom{k}{F - 1} + \bigO{k^{F - 1}}\right)
  \mach^F + \bigO{\mach^{F + 1}}\right] \cdot \widetilde{p}(s) \\
\sum_{k = 0}^{n - 1} \gamma_{3k + 5F} L_{F, k} &\leq
  \left[\left(3^{F + 1} \binom{n}{F + 1} + \bigO{n^F}\right)
  \mach^{F + 1} + \bigO{\mach^{F + 2}}\right] \cdot \widetilde{p}(s).
  \label{eq:L-sum-bound}
\end{align}
In particular, this means that
\(\sum_{k = 0}^{n - 1} \gamma_{3k + 5F} L_{F, k} =
\bigO{(3 n \mach)^{F + 1}} \cdot \widetilde{p}(s)\).
\end{lemma}

\begin{proof}
See proof in Section~\ref{proof:L-and-D-bounds}.
\end{proof}

\begin{lemma}\label{lemma:k-order}
The \(K\) order error \(\db{K}^{(0)}_0\) satisfies
\begin{equation}
  \left|\db{K - 1}^{(0)}_0 - \cdb{K - 1}^{(0)}_0\right| =
  \left|\db{K}^{(0)}_0\right| \leq
  \left[\left(3^{K} \binom{n}{K} + \bigO{n^{K - 1}}\right)
  \mach^{K} + \bigO{\mach^{K + 1}}\right] \cdot \widetilde{p}(s).
\end{equation}
\end{lemma}

\begin{proof}
See proof in Section~\ref{proof:k-order}.
\end{proof}

\begin{theorem}\label{thm:k-comp-result}
If no underflow occurs, \(n \geq 2\) and \(s \in \left[0, 1\right]\)
\begin{multline}
  \frac{\left|p(s) - \mathtt{CompDeCasteljau}(p, s, K)\right|}{
    \left|p(s)\right|} \leq \left[\mach + \bigO{\mach^2}
    \right] + \\
    \left[\left(3^{K} \binom{n}{K} + \bigO{n^{K - 1}}\right) \mach^K +
    \bigO{\mach^{K + 1}}\right] \cond{p, s}.
\end{multline}
\end{theorem}

\begin{proof}
See proof in Section~\ref{proof:k-comp-result}.
\end{proof}

For the first few values of \(K\) the coefficient of
\(\cond{p, s}\) in the bound is
\begin{center}
  \begin{tabular}{>{$}c<{$} c >{$}c<{$}}
    K & Method & \text{Multiplier} \\
    \midrule
    1 & \texttt{DeCasteljau} & 3 \binom{n}{1} \mach =
      3n \mach \approx \gamma_{3n} \\[0.125cm]
    2 & \texttt{CompDeCasteljau} & \left[9 \binom{n}{2} +
      15 \binom{n}{1}\right]\mach^2 = \frac{3n(3n + 7)}{2} \mach^2
      \approx \frac{1}{4} \cdot 2 \gamma_{3n}^2 \\[0.125cm]
    3 & \texttt{CompDeCasteljau3} & \left[27 \binom{n}{3} +
      135 \binom{n}{2} + 150 \binom{n}{1}\right] \mach^3 =
      \frac{3n(3n^2 + 36n + 61)}{2} \mach^3 \\[0.125cm]
    4 & \texttt{CompDeCasteljau4} & \left[81 \binom{n}{4} + 810 \binom{n}{3} +
      2475 \binom{n}{2} + 2250 \binom{n}{1}\right] \mach^4 \\[0.125cm]
  \end{tabular}
\end{center}
See the proof (Section~\ref{proof:L-and-D-bounds}) of
Lemma~\ref{lemma:L-and-D-bounds} for more details on where these
polynomials come from.

\section{Numerical experiments}\label{sec:numerical}

\begin{figure}
  \includegraphics{../images/k-compensated/de_casteljau_smooth_drawing.pdf}
  \centering
  \captionsetup{width=.75\linewidth}
  \caption{Evaluation of \(p(s) = (s - 1)\left(s - 3/4\right)^7\)
    in the neighborhood of its multiple root \(3/4\).}
  \label{fig:smooth-drawing}
\end{figure}

All experiments were performed in IEEE-754 double precision.
As in \cite{Jiang2010}, we consider the evaluation in the neighborhood
of the multiple root of \(p(s) = (s - 1)\left(s - 3/4\right)^7\),
written in Bernstein form.
Figure~\ref{fig:smooth-drawing} shows the evaluation of \(p(s)\) at
the 401 equally spaced\footnote{It's worth noting that \(0.1\) cannot
be represented exactly in IEEE-754 double precision (or any binary
arithmetic for that matter). Hence (most of) the points of the form
\(a + b \cdot 10^{-c}\) can only be approximately represented.} points
\(\left\{\frac{3}{4} + j \frac{10^{-7}}{2}\right\}_{j=-200}^{200}\)
with \texttt{DeCasteljau} (Algorithm~\ref{alg:de-casteljau}),
\texttt{CompDeCasteljau} (Algorithm~\ref{alg:comp-de-casteljau})
and \texttt{CompDeCasteljau3} (Algorithm~\ref{alg:k-comp-de-casteljau}
with \(K = 3\)). We see that \texttt{DeCasteljau} fails to get the
magnitude correct, \texttt{CompDeCasteljau} has the right shape but
lots of noise and \texttt{CompDeCasteljau3} is able to smoothly evaluate
the function. This is in contrast to a similar figure in \cite{Jiang2010},
where the plot was smooth for the 400 equally spaced points
\(\left\{\frac{3}{4} + \frac{10^{-4}}{2} \frac{2j - 399}{399}
\right\}_{j=0}^{399}\). The primary difference is that as the interval
shrinks by a factor of \(\approx \frac{10^{-4}}{10^{-7}} = 10^3\), the
condition number goes up by \(\approx 10^{21}\) and \texttt{CompDeCasteljau}
is no longer accurate.

\begin{figure}
  \includegraphics{../images/k-compensated/de_casteljau_rel_error.pdf}
  \centering
  \captionsetup{width=.75\linewidth}
  \caption{Accuracy of evaluation of \(p(s) = (s - 1)\left(s - 3/4\right)^7\)
    represented in Bernstein form.}
  \label{fig:compensated-k}
\end{figure}

Figure~\ref{fig:compensated-k} shows the relative forward errors compared
against the condition number. To compute relative errors, each input and
coefficient is converted to a fraction (i.e. infinite precision) and
\(p(s)\) is computed exactly as a fraction, then
compared to the corresponding computed values. Similar tools are used to
\emph{exactly} compute the condition number, though here we can rely
on the fact that \(\widetilde{p}(s) = (s - 1)
\left(s/2 - 3/4\right)^7\). Once the relative errors and
condition numbers are computed as fractions, they are rounded to the
nearest IEEE-754 double precision value. As in \cite{Jiang2010}, we use
values \(\left\{\frac{3}{4} - (1.3)^j\right\}_{j=-5}^{-90}\)\footnote{As with
\(0.1\), it's worth noting that \((1.3)^j\) can't be represented exactly in
IEEE-754 double precision. However, this geometric series still serves a
useful purpose since it continues to raise \(\cond{p, s}\) as \(j\) decreases
away from \(0\) and because it results in ``random'' changes in the bits of
\(0.75\) that are impacted by subtracting \((1.3)^j\).}. The curves for
\texttt{DeCasteljau} and \texttt{CompDeCasteljau} trace the same paths seen
in \cite{Jiang2010}. In particular, \texttt{CompDeCasteljau} has a relative
error that is \(\bigO{\mach}\) until \(\cond{p, s}\) reaches
\(1 / \mach\), at which point the relative error increases linearly with
the condition number until it becomes \(\bigO{1}\) when
\(\cond{p, s}\) reaches \(1 / \mach^2\).
Similarly, the relative error in \texttt{CompDeCasteljau3}
(Algorithm~\ref{alg:k-comp-de-casteljau} with \(K = 3\))
is \(\bigO{\mach}\) until \(\cond{p, s}\) reaches
\(1 / \mach^2\) at which point the relative error increases linearly
to \(\bigO{1}\) when \(\cond{p, s}\) reaches \(1 / \mach^3\)
and the relative error in \texttt{CompDeCasteljau4}
(Algorithm~\ref{alg:k-comp-de-casteljau} with \(K = 4\))
is \(\bigO{\mach}\) until \(\cond{p, s}\) reaches
\(1 / \mach^3\) at which point the relative error increases linearly
to \(\bigO{1}\) when \(\cond{p, s}\) reaches \(1 / \mach^4\).


\chapter{Miscellaneous Transformations}
\section{Fresnel Term - Schlick's approximation}
The \emph{Fresnel equations} describes the reflection and transmission of a electromagnetic wave at an interface. The Fresnel equation provides a reflection and transmission coefficients for waves. In Computer Graphics we often use the \emph{Schlick's approximation}. The specular reflection coefficient $R$ of the Fresnel equation can be approximated by:

\begin{equation}
 R(\theta) = R_0 + (1 - R_0)(1 - \cos \theta)^5
\label{eq:schlickapprox}
\end{equation}

and

\begin{equation*}
  R_0 = \left(\frac{n_1-n_2}{n_1+n_2}\right)^2
\end{equation*}

where $\theta$ is the angle between the viewing direction and the half-angle direction. This is equal to the halfway between the incident
light direction and the viewing direction, i.e. $\cos \theta = (H \cdot V)$. $n_1$ $n_2$ are the refraction indices of the two medias. $R_0$ is the reflection coefficient for light incoming parallel to the normal (i.e., the value of the Fresnel term when $\theta = 0 \degree$ or minimal reflection). In Computer Graphics one of the interfaces is usually air, meaning that $n_1$ is approximately equal to 1.

\section{Spherical Coordinates and Space Transformation}
\label{sec:sphericalcoordinates}
\begin{figure}[H]
  \centering
  \includegraphics[scale=0.35]{sphericalcoordinates.png}
  \caption[Illustration of Spherical Coordinate System]{Illustration$\footnotemark$ of Spherical coordinates $(r,θ,φ)$ radius $r$, polar (inclination) angle $\theta$, and azimuthal angle $\phi$.}
  \label{fig:sphericalcoordinatesystem}
\end{figure}
\footnotetext{image source of figure $\ref{fig:sphericalcoordinatesystem}$ \texttt{http://en.wikipedia.org/wiki/Spherical\textunderscore coordinate\textunderscore system}}

To define a \emph{spherical coordinate} system as shown in figure $\ref{fig:sphericalcoordinatesystem}$ we need two particular angles, the \emph{polar angle} $\theta$ and the \emph{azimuthal} $\phi$ plus a radius $r$. Then the Cartesian coordinates $(x,y,z)$ may be retrieved from their spherical coordinate representation as the following: $\forall \colvec[x]{y}{z} \in \mathbb{R}^3 : \exists r \in [0,\infty) \exists \phi \in [0,2\pi] \exists \theta \in [0,\pi] $ s.t.
\begin{equation*}
\colvec[x]{y}{z} = \colvec[r sin(\theta)cos(\phi)]{r sin(\theta)sin(\phi)}{r cos(\theta)}
\label{eq:sphericalcoordinates}
\end{equation*}

\label{sec:componentw}
From the definition $\ref{eq:uvw}$ of $(u,v,w)= -\omega_i - \omega_r$ and using spherical coordinates $\ref{eq:sphericalcoordinates}$, we get for $w$ the following identity:

\begin{align}
w
&= -\omega_i - \omega_r \nonumber \\
&= -(\omega_i + \omega_r) \nonumber \\
&= -\left( cos(\theta_i)+cos(\theta_r) \right)
\label{eq:sphericalomega}
\end{align}

and therefore $w^2$ is equal $(cos(\theta_i)+cos(\theta_r))^2$.

\section{Tangent Space}
\label{sec:tangentspace}
The concept of performing a transformation into the \emph{tangent space} is used in order to convert a point between the world and and its local (tangent) space.  \\

We can think of the tangent space as a bumpy surface defined on a flat plane. If the normals of a fragment were defined in a world space coordinate system, we would have to rotate these normals every time the model is rotated - even when just for a small amount. Since lights, the camera and other scene primitives usually are defined in the world space coordinate system we would to have to rotate them according to every fragment position. This would require to apply countless many object-to-world transformations at a pixel level. The workaround for this issue is to transform all vertex primitives into tangent space in the vertex shader. \\

To make this point clear: Even we would rotate the cube as shown in figure $\ref{fig:cubeintangentspace}$, its tangent space axis will remain aligned w.r.t its face. This will save us from apply many space transformations on fragments.

\begin{figure}[H]
  \centering
  \includegraphics[scale=0.6]{tangentspace.png}
  \caption[Illustration of a Tangent Space]{Cube in world space $(x,y,z)$ showing the tangent-space $(u,v,n)$ of its face $(2,1,3)$}
  \label{fig:cubeintangentspace}
\end{figure}


\chapter{数学}

\section{数学符号}

模板定义了一些正体（upright）的数学符号：
\begin{center}
\begin{tabular}{rl}
  \toprule
    符号                 & 命令 \\
  \midrule
    常数$\eu$     & \verb|\eu| \\
    复数单位$\iu$ & \verb|\iu| \\
    微分符号$\diff$ & \verb|\diff| \\
    $\argmax$         & \verb|\argmax| \\
    $\argmin$         & \verb|\argmin| \\
  \bottomrule
\end{tabular}
\end{center}

更多的例子：
\begin{equation}
  \eu^{\iu\pi} + 1 = 0
\end{equation}
\begin{equation}
  \frac{\diff^2u}{\diff t^2} = \int f(x) \diff x
\end{equation}
\begin{equation}
  \argmin_x f(x)
\end{equation}

\section{定理、引理和证明}

\begin{definition}
    If the integral of function $f$ is measurable and non-negative, we define
    its (extended) \textbf{Lebesgue integral} by
    \begin{equation}
        \int f = \sup_g \int g,
    \end{equation}
    where the supremum is taken over all measurable functions $g$ such that
    $0 \leq g \leq f$, and where $g$ is bounded and supported on a set of
    finite measure.
\end{definition}

\begin{example}
    Simple examples of functions on $\mathbb{R}^d$ that are integrable
    (or non-integrable) are given by
    \begin{equation}
        f_a(x) =
        \begin{cases}
            |x|^{-a} & \text{if } |x| \leq 1,\\
            0 & \text{if } x > 1.
        \end{cases}
    \end{equation}
    \begin{equation}
        F_a(x) = \frac{1}{1 + |x|^a}, \qquad \text{all } x \in \mathbb{R}^d.
    \end{equation}
    Then $f_a$ is integrable exactly when $a < d$, while $F_a$ is integrable
    exactly when $a > d$.
\end{example}

\begin{lemma}[Fatou]
    Suppose $\{f_n\}$ is a sequence of measurable functions with $f_n \geq 0$.
    If $\lim_{n \to \infty} f_n(x) = f(x)$ for a.e. $x$, then
    \begin{equation}
        \int f \leq \liminf_{n \to \infty} \int f_n.
    \end{equation}
\end{lemma}

\begin{remark}
    We do not exclude the cases $\int f = \infty$,
    or $\liminf_{n \to \infty} f_n = \infty$.
\end{remark}

\begin{corollary}
    Suppose $f$ is a non-negative measurable function, and $\{f_n\}$ a sequence
    of non-negative measurable functions with
    $f_n(x) \leq f(x)$ and $f_n(x) \to f(x)$ for almost every $x$. Then
    \begin{equation}
        \lim_{n \to \infty} \int f_n = \int f.
    \end{equation}
\end{corollary}

\begin{proposition}
    Suppose $f$ is integrable on $\mathbb{R}^d$. Then for every $\epsilon > 0$:
    \begin{enumerate}
        \renewcommand{\theenumi}{\roman{enumi}}
        \item There exists a set of finite measure $B$ (a ball, for example) such that
        \begin{equation}
            \int_{B^c} |f| < \epsilon.
        \end{equation}
        \item There is a $\delta > 0$ such that
        \begin{equation}
            \int_E |f| < \epsilon \qquad \text{whenever } m(E) < \delta.
        \end{equation}
    \end{enumerate}
\end{proposition}

\begin{theorem}
    Suppose $\{f_n\}$ is a sequence of measurable functions such that
    $f_n(x) \to f(x)$ a.e. $x$, as $n$ tends to infinity.
    If $|f_n(x)| \leq g(x)$, where $g$ is integrable, then
    \begin{equation}
        \int |f_n - f| \to 0 \qquad \text{as } n \to \infty,
    \end{equation}
    and consequently
    \begin{equation}
        \int f_n \to \int f \qquad \text{as } n \to \infty.
    \end{equation}
\end{theorem}

\begin{proof}
    Trivial.
\end{proof}

\section{自定义}

\newtheorem*{axiomofchoice}{Axiom of choice}
\begin{axiomofchoice}
    Suppose $E$ is a set and ${E_\alpha}$ is a collection of
    non-empty subsets of $E$. Then there is a function $\alpha
    \mapsto x_\alpha$ (a ``choice function'') such that
    \begin{equation}
        x_\alpha \in E_\alpha,\qquad \text{for all }\alpha.
    \end{equation}
\end{axiomofchoice}

\newtheorem{observation}{Observation}
\begin{observation}
    Suppose a partially ordered set $P$ has the property
    that every chain has an upper bound in $P$. Then the
    set $P$ contains at least one maximal element.
\end{observation}
\begin{proof}[A concise proof]
    Obvious.
\end{proof}


\documentclass{article}

\usepackage{amsfonts}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{textcomp}
\usepackage{fullpage}
\usepackage{setspace}
\usepackage{float}
\usepackage{cite}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage[pdfborder={0 0 0}, pdfpagemode=UseNone, pdfstartview=FitH]{hyperref}

\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}

\def\keyterm{\textit}

\newcommand{\transp}{{\scriptstyle{\mathsf{T}}}}

\begin{document}

\title{Graph SLAM Formulation}
\author{Jeff Irion}
\date{}

\maketitle
\vspace{3em}

\section{Problem Formulation}

Let a robot's trajectory through its environment be represented by a sequence of $N$ poses: $\mathbf{p}_1, \mathbf{p}_2, \ldots, \mathbf{p}_N$.  Each pose lies on a manifold: $\mathbf{p}_i \in \mathcal{M}$.  Simple examples of manifolds used in Graph SLAM include 1-D, 2-D, and 3-D space, i.e., $\mathbb{R}$, $\mathbb{R}^2$, and $\mathbb{R}^3$.  These environments are \keyterm{rectilinear}, meaning that there is no concept of orientation.  By contrast, in $SE(2)$ problem settings a robot's pose consists of its location in $\mathbb{R}^2$ and its orientation $\theta$.  Similarly, in $SE(3)$ a robot's pose consists of its location in $\mathbb{R}^3$ and its orientation, which can be represented via Euler angles, quaternions, or $SO(3)$ rotation matrices.

As the robot explores its environment, it collects a set of $M$ measurements $\mathcal{Z} = \{\mathbf{z}_j\}$.  Examples of such measurements include odometry, GPS, and IMU data.  Given a set of poses $\mathbf{p}_1, \ldots, \mathbf{p}_N$, we can compute the estimated measurement $\hat{\mathbf{z}}_j(\mathbf{p}_1, \ldots, \mathbf{p}_N)$.  We can then compute the \keyterm{residual} $\mathbf{e}_j(\mathbf{z}_j, \hat{\mathbf{z}}_j)$ for measurement $j$.  The formula for the residual depends on the type of measurement.  As an example, let $\mathbf{z}_1$ be an odometry measurement that was collected when the robot traveled from $\mathbf{p}_1$ to $\mathbf{p}_2$.  The expected measurement and the residual are computed as
%
\begin{align*}
    \hat{\mathbf{z}}_1(\mathbf{p}_1, \mathbf{p}_2) &= \mathbf{p}_2 \ominus \mathbf{p}_1 \\
    \mathbf{e}_1(\mathbf{z}_1, \hat{\mathbf{z}}_1) &= \mathbf{z}_1 \ominus \hat{\mathbf{z}}_1 = \mathbf{z}_1 \ominus (\mathbf{p}_2 \ominus \mathbf{p}_1),
\end{align*}
%
where the $\ominus$ operator indicates inverse pose composition.  We model measurement $\mathbf{z}_j$ as having independent Gaussian noise with zero mean and covariance matrix $\Omega_j^{-1}$; we refer to $\Omega_j$ as the \keyterm{information matrix} for measurement $j$.  That is,
\begin{equation}
    p(\mathbf{z}_j \ | \ \mathbf{p}_1, \ldots, \mathbf{p}_N) = \eta_j \exp \left( (-\mathbf{e}_j(\mathbf{z}_j, \hat{\mathbf{z}}_j))^\transp \Omega_j \mathbf{e}_j(\mathbf{z}_j, \hat{\mathbf{z}}_j) \right), \label{eq:observation_probability}
\end{equation}
where $\eta_j$ is the normalization constant.

The objective of Graph SLAM is to find the maximum likelihood set of poses given the measurements $\mathcal{Z} = \{\mathbf{z}_j\}$; in other words, we want to find
%
\begin{equation*}
    \argmax_{\mathbf{p}_1, \ldots, \mathbf{p}_N} \ p(\mathbf{p}_1, \ldots, \mathbf{p}_N \ | \ \mathcal{Z})
\end{equation*}
%
Using Bayes' rule, we can write this probability as
%
\begin{align}
    p(\mathbf{p}_1, \ldots, \mathbf{p}_N \ | \ \mathcal{Z}) &= \frac{p( \mathcal{Z} \ | \ \mathbf{p}_1, \ldots, \mathbf{p}_N) p(\mathbf{p}_1, \ldots, \mathbf{p}_N) }{ p(\mathcal{Z}) } \notag \\
    &\propto p( \mathcal{Z} \ | \ \mathbf{p}_1, \ldots, \mathbf{p}_N), \label{eq:bayes}
\end{align}
%
since $p(\mathcal{Z})$ is a constant (albeit, an unknown constant) and we assume that $p(\mathbf{p}_1, \ldots, \mathbf{p}_N)$ is uniformly distributed \cite{thrun2006graph}.  Therefore, we can use \eqref{eq:observation_probability} and \eqref{eq:bayes} to simplify the Graph SLAM optimization as follows:
%
\begin{align*}
    \argmax_{\mathbf{p}_1, \ldots, \mathbf{p}_N} \ p(\mathbf{p}_1, \ldots, \mathbf{p}_N \ | \ \mathcal{Z}) &= \argmax_{\mathbf{p}_1, \ldots, \mathbf{p}_N} \ p( \mathcal{Z} \ | \ \mathbf{p}_1, \ldots, \mathbf{p}_N) \\
    &= \argmax_{\mathbf{p}_1, \ldots, \mathbf{p}_N} \prod_{j=1}^M p(\mathbf{z}_j \ | \ \mathbf{p}_1, \ldots, \mathbf{p}_N) \\
    &= \argmax_{\mathbf{p}_1, \ldots, \mathbf{p}_N} \prod_{j=1}^M \exp \left( -(\mathbf{e}_j(\mathbf{z}_j, \hat{\mathbf{z}}_j))^\transp \Omega_j \mathbf{e}_j(\mathbf{z}_j, \hat{\mathbf{z}}_j) \right) \\
    &= \argmin_{\mathbf{p}_1, \ldots, \mathbf{p}_N} \sum_{j=1}^M (\mathbf{e}_j(\mathbf{z}_j, \hat{\mathbf{z}}_j))^\transp \Omega_j \mathbf{e}_j(\mathbf{z}_j, \hat{\mathbf{z}}_j).
\end{align*}
%
We define
%
\begin{equation*}
    \chi^2 := \sum_{j=1}^M (\mathbf{e}_j(\mathbf{z}_j, \hat{\mathbf{z}}_j))^\transp \Omega_j \mathbf{e}_j(\mathbf{z}_j, \hat{\mathbf{z}}_j),
\end{equation*}
%
and this is what we seek to minimize.

\section{Dimensionality and Pose Representation}

Before proceeding further, it is helpful to discuss the dimensionality of the problem.  We have:
\begin{itemize}
  \item A set of $N$ poses $\mathbf{p}_1, \mathbf{p}_2, \ldots, \mathbf{p}_N$, where each pose lies on the manifold $\mathcal{M}$
  \begin{itemize}
    \item Each pose $\mathbf{p}_i$ is represented as a vector in (a subset of) $\mathbb{R}^d$.  For example:
    \begin{itemize}
      \item[$\circ$] An $SE(2)$ pose is typically represented as $(x, y, \theta)$, and thus $d = 3$.
      \item[$\circ$] An $SE(3)$ pose is typically represented as $(x, y, z, q_x, q_y, q_z, q_w)$, where $(x, y, z)$ is a point in $\mathbb{R}^3$ and $(q_x, q_y, q_z, q_w)$ is a \keyterm{quaternion}, and so $d = 7$.  For more information about $SE(3)$ parameterizations and pose transformations, see \cite{blanco2010tutorial}.
    \end{itemize}
    \item We also need to be able to represent each pose compactly as a vector in (a subset of) $\mathbb{R}^c$.
    \begin{itemize}
      \item[$\circ$] Since an $SE(2)$ pose has three degrees of freedom, the $(x, y, \theta)$ representation is again sufficient and $c=3$.
      \item[$\circ$] An $SE(3)$ pose only has six degrees of freedom, and we can represent it compactly as $(x, y, z, q_x, q_y, q_z)$, and thus $c=6$.
    \end{itemize}
    \item We use the $\boxplus$ operator to indicate pose composition when one or both of the poses are represented compactly.  The output can be a pose in $\mathcal{M}$ or a vector in $\mathbb{R}^c$, as required by context.
  \end{itemize}
  \item A set of $M$ measurements $\mathcal{Z} = \{\mathbf{z}_1, \mathbf{z}_2, \ldots, \mathbf{z}_M\}$
  \begin{itemize}
    \item Each measurement's dimensionality can be unique, and we will use $\bullet$ to denote a ``wildcard'' variable.
    \item Measurement $\mathbf{z}_j \in \mathbb{R}^\bullet$ has an associated information matrix $\Omega_j \in \mathbb{R}^{\bullet \times \bullet}$ and residual function $\mathbf{e}_j(\mathbf{z}_j, \hat{\mathbf{z}}_j) = \mathbf{e}_j(\mathbf{z}_j, \mathbf{p}_1, \ldots, \mathbf{p}_N) \in \mathbb{R}^\bullet$.
    \item A measurement could, in theory, constrain anywhere from 1 pose to all $N$ poses.  In practice, each measurement usually constrains only 1 or 2 poses.
  \end{itemize}
\end{itemize}

\section{Graph SLAM Algorithm}

The ``Graph'' in Graph SLAM refers to the fact that we view the problem as a graph.  The graph has a set $\mathcal{V}$ of $N$ vertices, where each vertex $v_i$ has an associated pose $\mathbf{p}_i$.  Similarly, the graph has a set $\mathcal{E}$ of $M$ edges, where each edge $e_j$ has an associated measurement $\mathbf{z}_j$.  In practice, the edges in this graph are either unary (i.e., a loop) or binary.  (Note: $e_j$ refers to the edge in the graph associated with measurement $\mathbf{z}_j$, whereas $\mathbf{e}_j$ refers to the residual function associated with $\mathbf{z}_j$.)  For more information about the Graph SLAM algorithm, see \cite{grisetti2010tutorial}.

We want to optimize
%
\begin{equation*}
    \chi^2 = \sum_{e_j \in \mathcal{E}} \mathbf{e}_j^\transp \Omega_j \mathbf{e}_j.
\end{equation*}
%
Let $\mathbf{x}_i \in \mathbb{R}^c$ be the compact representation of pose $\mathbf{p}_i \in \mathcal{M}$, and let
%
\begin{equation*}
    \mathbf{x} := \begin{bmatrix} \mathbf{x}_1 \\ \mathbf{x}_2 \\ \vdots \\ \mathbf{x}_N \end{bmatrix} \in \mathbb{R}^{cN}
\end{equation*}
%
We will solve this optimization problem iteratively.  Let
%
\begin{equation}
    \mathbf{x}^{k+1} := \mathbf{x}^k \boxplus \Delta \mathbf{x}^k = \begin{bmatrix} \mathbf{x}_1 \boxplus \Delta \mathbf{x}_1 \\ \mathbf{x}_2 \boxplus \Delta \mathbf{x}_2 \\ \vdots \\ \mathbf{x}_N \boxplus \Delta \mathbf{x}_2 \end{bmatrix} \label{eq:update}
\end{equation}
%
The $\chi^2$ error at iteration $k+1$ is
\begin{equation}
    \chi_{k+1}^2 = \sum_{e_j \in \mathcal{E}} \underbrace{\left[ \mathbf{e}_j(\mathbf{x}^{k+1}) \right]^\transp}_{1 \times \bullet} \underbrace{\Omega_j}_{\bullet \times \bullet} \underbrace{\mathbf{e}_j(\mathbf{x}^{k+1})}_{\bullet \times 1}.  \label{eq:chisq_at_kplusone}
\end{equation}
%
We will linearize the residuals as:
%
\begin{align}
    \mathbf{e}_j(\mathbf{x}^{k+1}) &= \mathbf{e}_j(\mathbf{x}^k \boxplus \Delta \mathbf{x}^k) \notag \\
    &\approx \mathbf{e}_j(\mathbf{x}^{k}) + \frac{\partial}{\partial \Delta \mathbf{x}^k} \left[ \mathbf{e}_j(\mathbf{x}^k \boxplus \Delta \mathbf{x}^k) \right] \Delta \mathbf{x}^k \notag \\
    &= \mathbf{e}_j(\mathbf{x}^{k}) + \left( \left. \frac{\partial \mathbf{e}_j(\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)} \right|_{\Delta \mathbf{x}^k = \mathbf{0}} \right) \frac{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial \Delta \mathbf{x}^k} \Delta \mathbf{x}^k.  \label{eq:linearization}
\end{align}
%
Plugging \eqref{eq:linearization} into \eqref{eq:chisq_at_kplusone}, we get:
%
\small
\begin{align}
    \chi_{k+1}^2 &\approx \ \ \ \ \ \sum_{e_j \in \mathcal{E}} \underbrace{[ \mathbf{e}_j(\mathbf{x}^k)]^\transp}_{1 \times \bullet} \underbrace{\Omega_j}_{\bullet \times \bullet} \underbrace{\mathbf{e}_j(\mathbf{x}^k)}_{\bullet \times 1} \notag \\
    &\hphantom{\approx} \ \ \ + \sum_{e_j \in \mathcal{E}} \underbrace{[ \mathbf{e}_j(\mathbf{x^k}) ]^\transp }_{1 \times \bullet} \underbrace{\Omega_j}_{\bullet \times \bullet} \underbrace{\left( \left. \frac{\partial \mathbf{e}_j(\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)} \right|_{\Delta \mathbf{x}^k = \mathbf{0}} \right)}_{\bullet \times dN} \underbrace{\frac{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial \Delta \mathbf{x}^k}}_{dN \times cN} \underbrace{\Delta \mathbf{x}^k}_{cN \times 1} \notag \\
    &\hphantom{\approx} \ \ \ + \sum_{e_j \in \mathcal{E}} \underbrace{(\Delta \mathbf{x}^k)^\transp}_{1 \times cN} \underbrace{ \left( \frac{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial \Delta \mathbf{x}^k} \right)^\transp}_{cN \times dN} \underbrace{\left( \left. \frac{\partial \mathbf{e}_j(\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)} \right|_{\Delta \mathbf{x}^k = \mathbf{0}} \right)^\transp}_{dN \times \bullet} \underbrace{\Omega_j}_{\bullet \times \bullet} \underbrace{\left( \left. \frac{\partial \mathbf{e}_j(\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)} \right|_{\Delta \mathbf{x}^k = \mathbf{0}} \right)}_{\bullet \times dN} \underbrace{\frac{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial \Delta \mathbf{x}^k}}_{dN \times cN} \underbrace{\Delta \mathbf{x}^k}_{cN \times 1} \notag \\
    &= \chi_k^2 + 2 \mathbf{b}^\transp \Delta \mathbf{x}^k + (\Delta \mathbf{x}^k)^\transp H \Delta \mathbf{x}^k,  \notag
\end{align}
\normalsize
%
where
%
\begin{align*}
    \mathbf{b}^\transp &= \sum_{e_j \in \mathcal{E}} \underbrace{[ \mathbf{e}_j(\mathbf{x^k}) ]^\transp }_{1 \times \bullet} \underbrace{\Omega_j}_{\bullet \times \bullet} \underbrace{\left( \left. \frac{\partial \mathbf{e}_j(\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)} \right|_{\Delta \mathbf{x}^k = \mathbf{0}} \right)}_{\bullet \times dN} \underbrace{\frac{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial \Delta \mathbf{x}^k}}_{dN \times cN} \\
    H &= \sum_{e_j \in \mathcal{E}} \underbrace{ \left( \frac{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial \Delta \mathbf{x}^k} \right)^\transp}_{cN \times dN} \underbrace{\left( \left. \frac{\partial \mathbf{e}_j(\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)} \right|_{\Delta \mathbf{x}^k = \mathbf{0}} \right)^\transp}_{dN \times \bullet} \underbrace{\Omega_j}_{\bullet \times \bullet} \underbrace{\left( \left. \frac{\partial \mathbf{e}_j(\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)} \right|_{\Delta \mathbf{x}^k = \mathbf{0}} \right)}_{\bullet \times dN} \underbrace{\frac{\partial (\mathbf{x}^k \boxplus \Delta \mathbf{x}^k)}{\partial \Delta \mathbf{x}^k}}_{dN \times cN}.
\end{align*}
%
Using this notation, we obtain the optimal update as
%
\begin{equation}
    \Delta \mathbf{x}^k = -H^{-1} \mathbf{b}.  \label{eq:deltax}
\end{equation}
%
We apply this update to the poses via \eqref{eq:update} and repeat until convergence.

\bibliographystyle{acm}
\bibliography{graphSLAM}{}

\end{document}


\documentclass{article}
\usepackage{amsmath,amssymb,amsthm}

\newcommand{\G}{\mathbb{G}}

\begin{document}
Public parameters are supposed to consist of $g_1^{\alpha^i}$ and $g_2^{\alpha^i}$ for $i$ from $1$ to $2N$, except $N+1$ is omitted. This document describes and argues security of two procedures, one deterministic and one probabilistic, for determining whether an alleged set of parameters is indeed of this form.

\subsection*{Notation}

For $1 \le i \le 2N, i \ne N+1$, write
\begin{align*}
f_i \; &\text{to denote the $\G_1$ element that is allegedly} \; g_1^{\alpha^i} \\
h_i \; &\text{to denote the $\G_2$ element that is allegedly} \; g_2^{\alpha^i} \\
\intertext{and for notational convenience, write}
f_0 &= g_1 \\
h_0 &= g_2.
\end{align*}

A set of (alleged) parameters \[
\{f_i, h_i\}_{1 \le i \le 2N, i \ne N+1}
\]
are \textit{consistent} if $\exists \alpha \notin\{0,1\}$ such that $\forall i, f_i = g_1^{\alpha^i}$ and $h_i = g_2^{\alpha^i}$.

\subsection*{Deterministic consistency check}
Given an alleged set of parameters, the following procedure determines whether they are consistent. In the below discussion, without loss of generality, let $\alpha$ be such that $f_1 = g_1^{\alpha}$.

\begin{enumerate}
\item Check that none of the $f_i$ or $h_i$ are 1, $g_1$, or $g_2$. (This ensures $\alpha \notin \{0,1\}$.)

\item Check that \[ e(f_i, h_0) = e(f_0, h_i) \quad\text{for all $1\le i \le N$ and $N+2 \le i \le 2N$}\]
This ensures that $\log_{g_1} f_i = \log_{g_2} h_i$ for all $i$ --- in other words, the exponents of each $f_i$ and corresponding $h_i$ match. In particular we now know $h_1 = g_2^\alpha$.

\item Check that \[e(f_i, h_1) = e(f_{i+1}, h_0) \quad \text{for all $1 \le i \le N-1$}\]
This ensures $f_i = g_1^{\alpha^i}$ (and also $h_i = g_2^{\alpha^i}$ by the previous check) for $1 \le i \le N$. (To see this, consider the $i = 1$ case, where we check $e(f_1, h_1) = e(f_2, h_0)$. $e(f_1, h_1) = e(g_1^\alpha, g_2^\alpha)$, so $f_2$ must be $g_1^{\alpha^2}$. Then the $i=2$ check forces $f_3$ to be $g_1^{\alpha^3}$ and so on.)

\item Check that \[e(f_i, h_N) = e(f_{i+N}, h_0) \quad \text{for all $2 \le i \le N$}\]
In other words, check $e(f_{i+N}, g_2) = e(g_1^{\alpha^{i}}, g_2^{\alpha^{N}})$, which ensures $f_{i+N} = g_1^{\alpha^{i+N}}$ for $2\le i \le N$. (This also ensures $h_{i+N} = g_2^{\alpha^{i+N}}$ for $2 \le i \le N$ because we know the exponents of the $f$'s and $h$'s match.)

\end{enumerate}
If all checks pass, then we know that $f_i = g_1^{\alpha^i}$ and $h_i = g_2^{\alpha^i}$ for all $1 \le i \le 2N, i\ne N$, and we know $\alpha \notin \{0,1\}$. So the parameters are consistent.

\subsection*{Probabilistic consistency check}
A randomized version of the above algorithm can check consistency much more efficiently, using $O(1)$ rather than $O(N)$ pairings, with a soundness error of $O(\frac1{|\G_1|})$.

\begin{enumerate}
\item Generate $N$ random scalars $r_1, \dots, r_N$ and compute the following:
\begin{align*}
R_1 &= \prod_{i=1}^{N} f_i^{r_i} \\
R_2 &= \prod_{i=1}^{N} h_i^{r_i} \\
S & = \prod_{i=1}^{N-1} f_i^{r_i} \left( = \frac{R_1}{f_N^{r_N}}\right)\\
T &= \prod_{i=1}^{N-1} f_{i+1}^{r_i} \\
U_1 &= \prod_{i=1}^{N-1} f_{i+N+1}^{r_i} \\
U_2 &= \prod_{i=1}^{N-1} h_{i+N+1}^{r_i} \\
\end{align*}

\item Check that none of the $f$'s or $h$'s are 1, $g_1$, or $g_2$, just like in the deterministic procedure.

\item Check that
\begin{align*}
e(R_1, h_0) &= e(f_0, R_2) \\
\intertext{In other words, check that}
e(\prod_{i=1}^N f_i^{r_i}, h_0) &= e(f_0, \prod_{i=1}^N h_i^{r_i})
\end{align*}
This ensures $e(f_i, h_0) = e(f_0, h_i)$ for all $1 \le i \le N$ with soundness error $\frac1{|\G_1|}$.

\item Check that
\[
e(U_1, h_0) = e(f_0, U_2)
\]

This ensures $e(f_i, h_0) = e(f_0, h_i)$ for all $N+2 \le i \le 2N$ with soundness error $\frac1{|\G_1|}$. Combined with the previous step, this is equivalent to check 2 of the deterministic procedure.

\item Check that
\begin{align*}
e(S, h_1) &= e(T, h_0)
\intertext{or in other words,}
e(\prod_{i=1}^{N-1} f_i^{r_i}, h_1) &= e(\prod_{i=1}^{N-1} f_{i+1}^{r_i}, h_0)
\end{align*}
This ensures that $e(f_i, h_1) = e(f_{i+1}, h_0)$ for all $1 \le i \le N-1$ with soundness error $\frac1{|\G_1|}$. This is equivalent to check 3 of the deterministic procedure.

\item Check that
\begin{align*}
e(U_1, h_0) &= e(T, h_N)
\intertext{or in other words,}
e(\prod_{i=1}^{N-1} f_{i+N+1}^{r_i}, h_0) &= e(\prod_{i=1}^{N-1} f_{i+1}^{r_i}, h_N)
\end{align*}
This ensures that $e(f_{i+N+1}, h_0) = e(f_{i+1}, h_N)$ for all $1 \le i \le N-1$ (with soundness error $\frac1{|\G_1|}$). This is equivalent to check 4 of the deterministic procedure.
\end{enumerate}
If all of the above checks pass, then the parameters are consistent with high probability.

This randomized procedure is equivalent to the deterministic procedure in the previous section but with soundness error $\frac4{|\G_1|}$.
\end{document}


Again, you will need the \pkg{mosaic} package in R, which provides
some of the basic operators for constructing sampling and resampling distributions.
\begin{Schunk}
\begin{Sinput}
> require(mosaic)
\end{Sinput}
\end{Schunk}

The examples will be based on
the Cherry-Blossom 2008 data described earlier: \datasetCherryBlossomEight
\begin{Schunk}
\begin{Sinput}
> runners = fetchData("Cherry-Blossom-2008.csv")
> names( runners )
\end{Sinput}
\begin{Soutput}
[1] "position" "division" "total"    "name"     "age"      "place"
[7] "net"      "gun"      "sex"
\end{Soutput}
\end{Schunk}

\subsection{Finding a Sampling Distribution through Bootstrapping}
\label{sec:find-sampling-distribution}

Your data are typically a sample from a population.  Collecting the
sample is usually hard work.  Just to illustrate the context of
a sampling distribution, here is a simulation of selecting a sample of
size $n=100$ from the population of runners.  It's essential to keep
in mind that {\bf you do not usually} pull out your sample using the
computer in this way.  Instead, you go into the field or laboratory and collect your data.
\begin{Schunk}
\begin{Sinput}
> mysamp = deal( runners, 100 ) # A Simulation of sampling
\end{Sinput}
\end{Schunk}

Now that you have a sample, you can calculate the sample statistic
that's of interest to you.  For instance:
\begin{Schunk}
\begin{Sinput}
> mean( gun ~ sex, data=mysamp )
\end{Sinput}
\begin{Soutput}
  sex     S  N Missing
1   F 100.4 47       0
2   M  88.1 53       0
\end{Soutput}
\end{Schunk}
Note that the results are slightly different from those found above
using the whole population.  That's to be expected, since the sample
is just a random part of the population.  But ordinarily, you will not
know what the population values are; all you have to work with is your
sample.

Theoretically, the sampling distribution reflects the variation from
one randomly dealt sample to another, where each sample is taken from
the population.  In practice, your only ready access to the population
is through your sample.  So, to simulate the process of random
sampling, re-sampling is used and the re-sampling distribution is used
as a convenient approximation to the sampling distribution.

Re-sampling involves drawing from the set of cases in your sample with replacement.  To
illustrate, consider this example of a
very small, simple set:
the numbers 1 to 5:
\begin{Schunk}
\begin{Sinput}
> nums = c(1,2,3,4,5)
> nums
\end{Sinput}
\begin{Soutput}
[1] 1 2 3 4 5
\end{Soutput}
\end{Schunk}

Each resample of size $n$ consists of $n$ members from the set, but in
any one resample
each member might appear more than once and some might not appear at
all.  Here are three different resamples from \texttt{nums}:
\begin{Schunk}
\begin{Sinput}
> resample(nums)
\end{Sinput}
\begin{Soutput}
[1] 1 1 4 5 4
\end{Soutput}
\begin{Sinput}
> resample(nums)
\end{Sinput}
\begin{Soutput}
[1] 4 2 3 4 2
\end{Soutput}
\begin{Sinput}
> resample(nums)
\end{Sinput}
\begin{Soutput}
[1] 5 2 2 1 1
\end{Soutput}
\end{Schunk}

To use resampling to estimate the sampling distribution of a
statistic, you apply the calculation of the statistic to a resampled
version of your sample.  For instance, here are the group-wise means
from one resample of the running sample:
\begin{Schunk}
\begin{Sinput}
> mean( gun ~ sex, data=resample(mysamp) )
\end{Sinput}
\begin{Soutput}
  sex    S  N Missing
1   F 97.9 47       0
2   M 92.5 53       0
\end{Soutput}
\end{Schunk}
And here is another:
\begin{Schunk}
\begin{Sinput}
> mean( gun ~ sex, data=resample(mysamp) )
\end{Sinput}
\begin{Soutput}
  sex     S  N Missing
1   F 101.3 45       0
2   M  87.1 55       0
\end{Soutput}
\end{Schunk}

\index{C}{bootstrapping}

The bootstrap procedure involves conducting many such trials and examining the
variation from one trial to the next.

The \function{do} function lets
you automate the collection of multiple trials.  For instance, here
are five trials carried out using \function{do}:
\begin{Schunk}
\begin{Sinput}
> do(5) * mean( gun ~ sex, data=resample(mysamp) )
\end{Sinput}
\begin{Soutput}
   sex     S  N Missing do.rep
1    F  97.8 46       0      1
2    M  89.9 54       0      1
3    F  99.2 44       0      2
4    M  87.0 56       0      2
5    F 100.0 40       0      3
6    M  90.3 60       0      3
7    F 100.6 44       0      4
8    M  87.6 56       0      4
9    F 102.8 56       0      5
10   M  88.2 44       0      5
\end{Soutput}
\end{Schunk}

Typically, you will use several hundred trials for bootstrapping.  The
most common way to summarize the variation in bootstrap trials, you
can calculate a 95\% coverage interval.  (When applied to a sampling
distribution, the coverage interval is called a \newword{confidence interval}.)

To do the computation, give a name to  the results of the repeated
bootstrap trials, here it's called \texttt{trials}:
\begin{Schunk}
\begin{Sinput}
> trials = do(500) * mean( gun ~ sex, data=resample(mysamp) )
\end{Sinput}
\end{Schunk}
\begin{Schunk}
\begin{Sinput}
> trials
\end{Sinput}
\end{Schunk}
\begin{Schunk}
\begin{Soutput}
  sex    S  N Missing do.rep
1   F 99.6 44       0      1
2   M 87.4 56       0      1
3   F 97.1 48       0      2
4   M 88.0 52       0      2
5   F 98.0 55       0      3
6   M 88.4 45       0      3
... for 1000 rows altogether ...
\end{Soutput}
\end{Schunk}

Computing the coverage intervals can be done using \function{qdata} on
the columns, or, for convenience, use the \function{confint} function.
\begin{Schunk}
\begin{Sinput}
> confint(trials)
\end{Sinput}
\begin{Soutput}
  sex 2.5% 97.5%
1   F 96.9 104.2
2   M 83.6  92.4
\end{Soutput}
\end{Schunk}

The idea of sampling distributions is based on
drawing at random from the population, not resampling.  Ordinarily,
you can't do this calculation since you don't have the population at
hand.  But in this example, we happen to have the data for all
runners.  Here's the population-based confidence interval for the mean
running time, with sample size $n=100$, broken down by sex:

\index{P}{confint}
\index{C}{confidence intervals}

\begin{Schunk}
\begin{Sinput}
> trials = do(500) * mean( gun ~ sex, data=deal(runners,100) )
> confint(trials)
\end{Sinput}
\begin{Soutput}
  sex 2.5% 97.5%
1   F 95.1 102.5
2   M 84.1  92.2
\end{Soutput}
\end{Schunk}

\index{C}{confidence level}

The output of \function{confint} shows the lower and upper limits of
the confidence interval for each of the groups.  The labels on the
columns indicate the confidence level.  By default, the interval is at
the 95\% level, and so the interval runs from the 2.5 percentile to
the 97.5 percentile.

Historically, statisticians have been concerned with very small
samples: say $n=2$ or $n=3$.  Even in this era of huge data sets, such
small sample sizes often are encountered in laboratory experiments,
etc. Bootstrapping cannot work well with such
small samples, and other techniques are needed to simulate sampling
variability.  Many of these techniques are based in algebra and
probability theory, and give somewhat complex formulas for calculating confidence
intervals from data. The formulas are often found in textbooks, but for most of the modeling techniques you will use
in later chapters, appropriate formulas for confidence intervals have been implemented in
software.  For other modeling techniques, bootstrapping is used to
find the confidence intervals.  But keep in mind that bootstrapping
can only be effective when the sample size $n$ is one or two dozen or
larger.

\subsection{Computing Grade-Point Averages}

The grade-point average is a kind of group-wise mean, where the group
is an individual student.  This is not the usual way of looking at
things for a student, who sees only his or her own grades.  But
institutions have data on many students.

The data files \texttt{grades.csv} and \texttt{courses.csv} are drawn from
an institutional database at a college.  They give the grades for more
than 400 students who graduated in year 2005.  Another file,
\texttt{grade-to-number.csv}, gives the rules used by the institution
in converting letter grades to numbers.

The data files are part of a \newword{relational data base}, a very
important way of managing large amounts of data used by private and
public institutions, corporations and governments --- it's the basis
for a multi-billion dollar segment of the economy.  Ordinarily,
relational data bases are queried using special-purpose computer
languages that sort, extract, and combine the data.  Here are the R
commands for converting the letter grades to numbers and extracting
the data for one student:
\begin{Schunk}
\begin{Sinput}
> grades = fetchData("grades.csv")
> gp = fetchData("grade-to-number.csv")
> all.students = merge(grades, gp)
> one.student = subset( all.students, sid=="S31509" )
\end{Sinput}
\end{Schunk}
\begin{Schunk}
\begin{Sinput}
> one.student
\end{Sinput}
\end{Schunk}
\begin{Schunk}
\begin{Soutput}
     grade    sid   sessionID gradepoint
192      A S31509 session3443          4
543      A S31509 session2308          4
674      A S31509 session2851          4
1280     A S31509 session2737          4
... for 13 cases altogether ...
\end{Soutput}
\end{Schunk}

Calculating the mean grade-point for the one student is a simple matter:
\begin{Schunk}
\begin{Sinput}
> mean( gradepoint, data=one.student )
\end{Sinput}
\begin{Soutput}
[1] 3.6
\end{Soutput}
\end{Schunk}

It's equally straightforward to calculate the grade-point averages
for all students as individuals:
\begin{Schunk}
\begin{Sinput}
> mean( gradepoint ~ sid, data=all.students )
\end{Sinput}
\end{Schunk}
\begin{Schunk}
\begin{Soutput}
     sid    S  N Missing
1 S31185 2.41  8       0
2 S31188 3.02 16       2
3 S31191 3.21 14       1
4 S31194 3.36 12       0
5 S31197 3.36 13       0
... for 443 cases altogether ...
\end{Soutput}
\end{Schunk}

\index{P}{do}
\index{P}{confint}
\index{P}{mean}

Bootstrapping can be used to find the confidence interval on the
grade-point average for each student:
\begin{Schunk}
\begin{Sinput}
> trials = do(100)*mean(gradepoint ~ sid, data=resample(all.students) )
\end{Sinput}
\end{Schunk}

\begin{Schunk}
\begin{Sinput}
> confint(trials)
\end{Sinput}
\end{Schunk}

\begin{Schunk}
\begin{Soutput}
     sid 2.5% 97.5%
1 S31185 1.65  3.14
2 S31188 2.56  3.47
3 S31191 3.01  3.43
4 S31194 3.05  3.70
5 S31197 3.20  3.52
... for 443 students altogether ...
\end{Soutput}
\end{Schunk}

It's important to point out that there are other methods for calculating
confidence intervals that are based on the standard deviation of the
data.  Formulas and procedures for such methods are given in just
about every standard introductory statistics book and would certainly
be used instead of bootstrapping in a simple calculation of the sort
illustrated here.

However, such formulas don't go to the heart of the problem:
accounting for variation in the grades and the contribution from
different sources of that variation.  For example, some of the
variation in this student's grades might be due systematically to
improvement over time or due to differences between instructor's
practices.  The modeling techniques introduced in the following
chapters provide a means to examine and quantify the different sources
of variation.


\section{Dependent Type String Indexing is a Monoid}\label{sec:monoid}

\subsection{String Matching Definition}

We define the String Matching data type
@SM target@ to contain a string field @input@ and
a list of all the indices on @input@ where the
@target@ appears.
%
\begin{code}
data SM (target :: Symbol) where
  SM :: input:RString
     -> indices:[GoodIndex input target]
     -> SM target
\end{code}
%
We used type literals to parameterize the type over
the Symbol @target@.
%
This encoding is required to allow definition of
an identity element, as explained later~(\S~\ref{subsec:monoid:methods}).
%
The input field, of type @RString@, is a refined wrapper on
a String manipulation library that
1. allows for constant time string indexing, and
2. provides various string lemmata.
%
The library @RString.hs@~\footnote{
\href{https://github.com/nikivazou/verified_string_indexing/blob/master/src/String.hs}
     {\url{https://github.com/nikivazou/verified_string_indexing/blob/master/src/String.hs}}}
implements @RString@ as @ByteStrings@,
and \textbf{assumes} various String properties,
like left identity:
\begin{code}
assume leftIdentity :: s:RSTring -> { s <+> stringMempty == s }
\end{code}
%
The indices field is a list of good indices.
%
For simplicity we are using Haskell's built in lists but in the
implementation we use a user defined reflected list data type similar
to that presented in section~\ref{sec:haskell-proofs}.
%
A @GoodIndex input target@ is a refined type alias
for an integer @i@ that
1. is a natural number,
2. @target@ is a substring of @input@ appearing at position @i@.
%
\begin{code}
type GoodIndex Input Target
  = {i:Nat | isGoodIndex Input (fromString Target) i }

isGoodIndex :: RSTring -> RSTring -> Int -> Bool
isGoodIndex input target i
  =  (subString input i (stringLen target)  == target)
  && (i + stringLen target <= stringLen input)
\end{code}
%

As an example, the good indices of @"abcab"@ on @"ababcabcab"@
are @[2,5]@ while any other index is bad.
\begin{code}
goodSM :: SM "abcab"
goodSM = SM "ababcabcab" [2, 5]

badSM  :: SM "abcab"
badSM  = SM "ababcabcab" [0, 7]
\end{code}
\NV{Liquid Haskell actually will reject both the above, as the stringLen and subString functions are uninterpreted}

\subsection{Monoid Laws}
In the rest of this section we will prove that
the String Matching structure is a Monoid.
%
A structure @m@ is a monoid,
there is an identity element @mempty :: a@
and an associative appending function @(<>) :: a -> a -> a@.
That is, @mempty@ and @(<>)@ should satisfy the following laws
\begin{figure}
\begin{code}
idLeft :: x:a -> {x <> mempty == x }
idRight:: x:a -> {mempty <> x == x }
assoc  :: x:a -> y:a -> z:a -> {(x <> y) <> z == x <> (y <> z)}
\end{code}
\caption{Monoid Laws}
\label{fig:monoid:laws}
\end{figure}
%
To prove that the String Matching operator @SM@
is a monoid, we use the fact that the String and list
of indices contained in its fields are also monoids.
%
Figure~\ref{fig:monoids} defines the monoid operators used
in this section.
%
\begin{figure}
$$
\begin{array}{c c c c }
\text{Structure} & \text{Identity} & \text{Mappend} & \text{Laws} \\
\hline
\texttt{List} & \listMempty & (\listMappend) & \text{PROVEN} \\
\texttt{SM} & \mempty & (\mappend) & \text{PROVEN} \\
\texttt{RString} & \stringMempty & (\stringMappend) & \text{ASSUMED}
\end{array}
$$
\caption{Monoid Structures}
\label{fig:monoids}
\end{figure}

\subsection{Monoid Methods for String Matching}~\label{subsec:monoid:methods}
Next, we define the mappend and identity elements for string matching.
We reflect the definitions into logic, to latter use them to prove
the monoid laws.

The \textit{identity element} of @SM t@, for each target @t@, is
defined to contain the identity @RString@ (@stringMempty@) and the
identity @List@ (@listMempty@).
\begin{code}
reflect mempty
mempty:: forall (t :: Symbol). SM t
mempty = SM stringMempty listMempty
\end{code}
%
Note, that if target was encoded as a value argument, instead of a
type parameter, then there would be no way to define an identity
element for each expression argument.
%
\JP{I think the type parameter is more necessary to define mappend}

The @(mappend)@ operator,
mappends the two input strings.
The appended indices, as depicted in Figure~\ref{fig:mappend:indices},
are the concatenations of three list indices:
\begin{enumerate}
\item The indices @is1@ from the first input, casted to be good indices in the new structure,
\item the new indices @newIs@ created when concatenating the two strings, and
\item the indices @is2@ from the second input, shifted right @stringLen i2@ points.
\end{enumerate}
%
\begin{figure}
\includegraphics[scale=0.5]{makeIndices}
\caption{Mappend indices of String Matcher}
\label{fig:mappend:indices}
\end{figure}
%
The above are summarized in the Haskell definition of @(<>)@
\begin{code}
reflect mappend
(mappend)::forall (t::Symbol). KnownSymbol t => SM t -> SM t -> SM t
(SM i1 is1) mappend (SM i2 is2)
  = SM (i1 stringMappend i2) (is1' listMappend newIs listMappend is2')
  where
    tg    = fromString (symbolVal (Proxy :: Proxy t))
    is1'  = map (castGoodIndexLeft tg i1 i2) is1
    newIs = makeNewIndices i1 i2 tg
    is2'  = map (shiftStringRight tg i1 i2) is2
\end{code}

\paragraph{Step 1: Casting Good Indices}
If @is@ is a list of good indices for string @i1@ with respect to a target @tg@,
then @is@ is a list of good indices for the string @i1 stringMappend i2@, for each @i2@
for the same target.
%
To establish the above property,
we need the property from refined string
that states that
for each string @input@ and @input'@
and each integers @i@ and @j@ whose sum does not exceed the length of @input@,
the @subString@ on @input@ from @i@ with length @j@
is equal to
the @subString@ on @input stringMappend input'@ from @i@ with length @j@:
%
\begin{code}
subStringConcatLeft
    :: sl:RString -> sr:RString -> j:Int
    ->  i:{Int | i + j <= stringLen sl }
    ->  { subString sl i j == subString (sl stringMappend sr) i j }
\end{code}
%
To cast an index list of good indices in the above way,
we cast the above property to every element of the list
%
\begin{code}
reflect castGoodIndexRightList
castGoodIndexLeft
  :: tg:RString -> sl:RString -> sr:RString
  -> i:GoodIndex sl tg
  -> {v:GoodIndex (sl stringMappend sr target | v == i}

castGoodIndexLeft tg sl sr i
  = cast (subStringConcatLeft sl sr (stringLen tg) i) i
\end{code}
%
Where @cast p x@
returns @x@, after enforcing the properties of @p@ in the logic
\begin{code}
cast :: b -> x:a -> {v:a | v == x }
cast _ x = x
\end{code}
%
Moreover, the translation of @cast p x@ into the logic, is merely @x@
thus allowing random (\ie non-reflected) Haskell expressions appearing in @p@.

\paragraph{Step 2: Creation of new indices}
When concatenating two new string inputs @i1@ and @i2@,
the @stringLen tg@ last positions of @i1@ combined with the initial part of @i2@
may form new good indices.
%
@makeNewIndices s1 s2 target@ creates all such good new indices.
%
If the length of the @target@ is less than 2, then no new good indices are creates,
thus the function returns the empty list.
Otherwise, we search @concatString s1 s2@ for substring of @target@
in the range @[maxInt (stringLen s1 - (stringLen target-1)) 0, stringLen s1 - 1]@
\begin{code}
reflect makeNewIndices
makeNewIndices
  :: s1:RString -> s2:RString -> target:RString
  -> [GoodIndex {s1 stringMappend s2} target]
makeNewIndices s1 s2 target
  | lenStr target < 2
  = []
  | otherwise
  = makeIndices (s1 stringMappend s2) target lo hi
  where
    lo = maxInt (lenStr s1 - (lenStr target-1)) 0
    hi = lenStr s1 - 1
\end{code}
%
@makeIndices input target lo hi@
recursively searches for good indices on @input@ with respect to @target@
from @lo@ to @hi@.
%
\begin{code}
reflect makeIndices
makeIndices
  :: input:RString -> target:RString -> lo:Nat
  -> hi:Int -> [GoodIndex input target]
  / [hi - lo]
makeIndices input target lo hi
  | hi < lo
  = N
makeIndices input target lo hi
  | isGoodIndex input target lo
  = lo `C` rest
  | otherwise
  = rest
  where
    rest = makeIndices input target (lo + 1) hi
\end{code}
%
The notation @[hi-lo]@ declares that @makeIndices@ terminates, as the metric @hi - lo@
is a non-negative values that decreases at each recursive call.

\paragraph{Note on complexity.}
Given that the implementation of @RString@
allows substring checking via indexing, that is in constant time,
then @makeNewIndices@ and thus @(<>)@ requires @lenStr target@ checks,
that is constant checks with respect to the string input.

\paragraph{Step 3: Shift Good Indices}
If @is@ is a list of good indices for string @i2@ with respect to a target @tg@,
to get a list of good indices for the string @i1 stringMappend i2@,
we need to shift each element of @is@ @stringLen i1@ right.
%
Moreover, to persuade Liquid Haskell that the shifted indices are good indices
we need to apply the refined string theorem that states
for each string @sl@ and @sr@
and each integers @i@ and @j@,
the @subString@ on @sr@ from @i@ with length @j@
is equal to
the @subString@ on @sl stringMappend sr@ from @stringLen sl + i@ with length @j@:
%
\begin{code}
assume subStringConcatRight
  :: sl:RString -> sr:RString
  -> j:Int -> i:Int
  -> {subStr sr i j == subStr (sl stringMappend sr) (lenStr sl+i) j}
\end{code}

Thus, @shiftStringRight@ both appropriately shifts the index
and casts the shifted index using the above theorem:
\begin{code}
shiftStringRight
  :: tg:RString -> sl:RString -> sr:RString
  -> i:GoodIndex sr tg
  -> {v:(GoodIndex (sl stringMappend sr) tg) | v == i + lenStr sl}
shiftStringRight tg sl sr i
  = subStringConcatRight sl sr (stringLen tg) i)
     `cast` shift (stringLen sl) i
\end{code}

\subsection{String Matching is a Monoid}
We conclude this section by proving
that the @mempty@ and @(mappend)@
operators we defined satisfy the monoid laws.
%
In the prove, we express
\begin{itemize}
\item the monoid laws as liquid types, and
\item the monoid proofs as Haskell functions.
\end{itemize}
Then, Liquid Haskell checks that the functions indeed check the corresponding laws.

\begin{theorem}\label{theorem:monoid}
String Matching is a Monoid for the operations @mempty@ and @(mappend)@.
\end{theorem}

\begin{proof}
Based on the Monoid Definition~\ref{definition:monoid},
to prove that string maching is a monoid, we need to prove that
there exist safe implementations of the monoid laws functions.
%
We implemented the monoid laws functions
and used Liquid Haskell to (machine) check correctness of our proof.

First, we prove \textit{left identity}
\begin{code}
idLeft :: x:SM t -> {x mappend mempty == x }
idLeft (SM i is)
  =  (SM i is) mappend (mempty :: SM t)
  ==. (SM i is) mappend (SM stringMempty listMempty)
  ==. SM (i stringMappend stringMempty) (is1 listMappend isNew listMappend is2)
      ? idLeftString i
  ==. SM i (is listMappend N listMappend N)
      ? (mapCastId tg i stringMempty is &&& newIsNullLeft i tg)
  ==. SM i is
      ? idLeftList is
  ***  QED
  where
    tg    = fromString (symbolVal (Proxy :: Proxy t))
    is1   = map (castGoodIndexRight tg i stringEmp) is
    isNew = makeNewIndices i stringEmp tg
    is2   = map (shiftStringRight tg i stringEmp) N
\end{code}
The proof proceeds by rewriting and using four lemmata
\begin{itemize}
\item Left Identity on lists, that is proven by structural induction.
\begin{code}
idLeftList :: x:[a] -> {x listMappend listMempty == x}
\end{code}
\item Left Identity on strings is provided by the string library that we trust.
\begin{code}
idLeftString :: x:RString -> {x stringMappend stringMempty == x}
\end{code}
\item Identity of casting is proven by induction and identity on casts
\begin{code}
mapCastId :: tg:RString -> x:RString -> y:RString
  -> is:[GoodIndex x tg] ->
  -> {map (castGoodIndexRight tg x y) is == is}
\end{code}
\item No new indices are created by mappend
\begin{code}
newIsNullLeft :: s:RString -> t:RString
  -> {makeNewIndices s stringMempty t == listMempty }
\end{code}
The proof procceds by case splitting
on comparison of the lengths of @s@ and @t@.
At each case we prove by induction that all
the potential new indices will be out of bounds and thus
no new good indices can be created.
\end{itemize}

\NV{Actually, the right identity proof is simpler, maybe put it first}

- The proof of \textit{right identity} is similar
\begin{code}
idRight :: x:SM t -> {mempty mappend x == xs }
idRight (SM i is)
  =  (mempty :: SM t) mappend (SM i is)
  ==. (SM stringMempty listMempty) mappend (SM i is)
  ==. SM (stringMempty <+> i) (is1 ++ isNew ++ is2)
       ? idRightString i
  ==. SM i (N ++ N ++ is)
       ? (mapShiftZero tg i is &&& newIsNullRight i tg)
  ==. SM i is
       ? idRightList is
  *** QED
  where
    tg    = fromString (symbolVal (Proxy :: Proxy t))
    is1   = map (castGoodIndexRight tg i stringEmp) N
    isNew = makeNewIndices stringEmp i tg
    is2   = (map (shiftStringRight tg stringEmp i) is)
\end{code}
The proof uses again four lemmata.
\begin{itemize}
\item Right Identity on lists, that is proven by structural induction.
\begin{code}
idRightList :: x:[a] -> {listMemtpy listMappend x == x}
\end{code}
\item Right Identity on strings is provided by the string library that we trust.
\begin{code}
idRightString :: x:RString -> {stringMemepty stringMappend x == x}
\end{code}
\item Identity of shifting by an empty string is proven by induction and
the assumption that empty string has length 0
\begin{code}
mapShiftZero :: tg:RString -> i:RString -> is:[GoodIndex i target]
  -> {map (shiftStringRight tg stringMempty i) is == is }
\end{code}
\item No new indices are created by mappend
\begin{code}
newIsNullLeft :: s:RString -> t:RString
  -> {makeNewIndices stringMempty s t == listMempty }
\end{code}
The proofs relies on the fact that @makeIndices@
will be called on the range @[0, -1]@ and immediately return @listMemepty@.
\end{itemize}
- Finally we prove \textit{associativity}.
For space, we omit the detailed proof, and only give the interesting steps.
In the proof we need to show equality of two string matchers,
thus we need to show that their input and induces fields are respectively equal.
%
Equality of the input fields follows by associativity of RStrings.
%
The proof of index equlity proceeds in three steps.

\begin{enumerate}
\item Firstly, using list associativity and distribution of index shifting,
we group the indices in the five lists shown in Figure~\ref{fig:mappend:assoc}.
The indices of the input @x@,
the new indices from mappending @x@ to @y@,
the indices of the input @y@,
the new indices from mappending @x@ to @y@, and
the indices of the input @z@.
\item The representation of each group depends on the order of appending.
For example, if @zis1@ (resp. @zis2@) is the group @zis@ when
right (resp. left) mappend happened first, then we have
\begin{code}
zis1 = map (shiftStringRight tg xi (yi stringMappend zi))
       map (shiftStringRight tg yi zi) zis

zis2 = map (shiftStringRight tg (xi stringMappend yi) zi) zis
\end{code}
That is, in right first, the indices of @z@ are first shifted
by the length of @yi@ and then by the length of @xi@,
while in the left first case, the indices of @z@ are shifted by the
length of @xi stringMappend yi@.
In this second step of the proof, we prove using lemmata,
the equivalence of the group representation.
Equivalence of @xis@ and @zis@ is trivial,
but for the rest three groups, as we later discuss is more interesting,
as it depends on the relative lengths of the target and the input of @y@.
\item Finally, once we proved equivalence of representations,
we use again list associativity and distribution of casts to wrap the index groups
back in string matchers
\end{enumerate}
\begin{figure}
\includegraphics[scale=0.5]{AssociativeIndices}
\label{fig:mappend:assoc}
\end{figure}
Bellow we present the skeleton of the proof, while the detailed proof can be found online\footnote{\NV{GIVE PRoof link}}.
\begin{code}
assoc x@(SM xi xis) y@(SM yi yis) z@(SM zi zis)
  -- Step 1: unwrapping the indices
  =   x <> (y <> z)
  ==. (SM xi xis) <> ((SM yi yis) <> (SM zi zis))
                         ...
  -- via list associativity and distribution of shifts

  -- Step 3: Equivalence of representations
  ==. SM i (xis1 ++ ((xyis1 ++ yis1 ++ yzis1) ++ zis1))
  ==. SM i (xis1 ++ ((xyis1 ++ yis1 ++ yzis1) ++ zis1))
      ? castConcat tg xi yi zi xis
  ==. SM i (xis2 ++ ((xyis1 ++ yis1 ++ yzis1) ++ zis2))
      ? mapLenFusion tg xi yi zi zis
  ==. SM i (xis2 ++ ((xyis2 ++ yis2 ++ yzis2) ++ zis2))
      ? assocNewIndices y tg xi yi zi yis

  -- Step 3: Wrapping the indices
                         ...
  -- via list associativity and distribution of casts
  ==. (SM xi xis <> SM yi yis) <> SM zi zis
  =   (x <> y) <> z
  *** QED
  where
    yzis1 = map (shiftStringRight tg xi (yi <+> zi)) yzis
    yzis2 = makeNewIndices (xi <+> yi) zi tg

    yzis  = makeNewIndices yi zi tg

    i     = xi stringMappend (yi stringMappend zi)
\end{code}
Finally, we present the lemma @assocNewIndices@
that proofs equivalence of representation in the three
groups @xyis@, @yis@, and @yzis@.
The proof proceeds by case analysis in the relative size of
the target @tg@ and the middle input @yi@.
%
If the target is smaller than @yi@
then we can prove equivalence of each of the groups independently
via respective lemmata.
%
Otherwise, independent equivalence does not hold.
Thus, when @yi@ is smaller than the target,
the group @xyis1@ (similarly @yzis1@)
is not provably equal to @xyis2@ (similarly @yzis2@).
When @yi@ is smaller than the target,
when appending @x@ with @y mappend z@ new indices may occur because
of the suffix of @z@, these indices will not occur when appending @x@ with @y@.
%
Yet, the appending of the two new index groups are provably equal,
@xyis1 listMappend yzis1 == xyis2 listMappend yzis2@.
The lemma proof using append equivalence and the fact @yis1 == yis2 == []@
to conclude that the appending of all the three groups is always equivalent.

\begin{code}
-- proof that
-- xyis1 ++ yis1 ++ yzis1 == xyis2 ++ yis2 ++ yzis2
assocNewIndices y tg xi yi zi yis
  | stringLen tg <= stringLen yi
  =   xyisEquivalence xi yi zi tg
  &&&  yisEquivalence xi yi zi tg yis
  &&& yzisEquivalence xi yi zi tg
  | stringLen yi < stringLen tg
  =  xyis1 listMappend yis1 listMappend yzis1
  ==. xyis1 listMappend [] listMappend yzis1 ? emptyIndices y yis
  ==. xyis1 listMappend yzis1 ? idLeftList xyis1
  ==. xyis2 listMappend yzis2 ? shiftNewIndices xi yi zi tg
  ==. (xyis2 listMappend []) listMappend yzis2 ? idLeftList xyis2
  ==. (xyis2 listMappend yis2) listMappend yzis2 ? emptyIndices y yis
  *** QED
\end{code}

\qed\end{proof}


\subsection{Order}
One way is by this highest order derivative that appears in the equation.
\begin{definition}
	The order of a differential equation is the order of the highest derivative in the equation.
\end{definition}
\noindent
Below is a table of orders and equation numbers.
\begin{table}[H]
	\centering
	\begin{tabular}{c|c}
		Order & Equation Number \\
		\hline
		1 &  2, 5, 6, 10 \\
		2 & 1, 3, 4, 7, 8, 9 \\
	\end{tabular}
\end{table}