\lab{Value Function Iteration}{Value Function Iteration}
\newcommand\ve{\varepsilon}
\objective{This section teaches the fundamentals of Dynamic Programming using value function iteration.}

%Often it is of interest to optimize decision making in some sequential process.  For example, an oil company may need to decide
%how much oil to excavate and sell each month as prices change, a person entering retirement may need to decide how much of their
%savings to spend each year, or a model of economic growth may require a decision about how much to invest in capital versus how
%much to spend each year.  In this lab we will formulate a general dynamic optimization problem.  We will explore techniques for
%solving such a problem with both finite and infinite time horizons.

Previously we have optimizated a single decision or question.
In this lab we will optimize a sequence of decisions.
The techniques to solve these problems are called dynamic optimization.

Dynamic optimization provides different answers than optimization techniques we have studied previously.
For example, an oil company might want to know how much oil to excavate in one day in order to maximize profit.
If we only consider today, then the answer is to excavate and sell as much as possible.
If each day is considered in isolation, then the strategy to optimize profit is to maximize excavation in order to maximize profits.
However, in reality oil prices change from day to day and as supply increases or decreases, and so maximizing excavation may in fact lead to less profit.
On the other hand, if the oil company considers how production on one day will affect subsequent decisions, they may be able to maximize their profits.
In this lab we will explore techniques for solving such a problem.

\section*{The Cake Eating Problem}

Rather than maximizing oil profits, we will focus on solving a general problem that can be applied in many areas called the cake eating problem.
Given a cake of a certain size, how do we eat it to maximize our enjoyment (in other words, our utility) over time?
Some people may prefer to eat all of their cake at once, and not save any for later.
Others may prefer to only eat a little bit at a time.
These preferences will be expressed precisely using a utility function.
Our task is to find an optimal strategy given an increasing utility function, $u$.
Precisely, given a cake of size $W_0$ and some amount of consumption $c_0 \in [0, W_0]$, the utility gained is given by
\[
u(c_0).
\]

If we want to maximize utility over one time period, we will consume the entire cake.  How do we maximize utility over several days?

\subsection*{Discount Factors}

A person or firm will typically have a preference for saving or consuming.
For example, a dollar today can be invested and yield interest, whereas a dollar received next year will not include the accrued interest.
In this lab, cake now will yield more utility than cake in the future.
We can model this by multiplying future utility by a discount factor $\beta \in (0,1)$.
For example, if we were to consume $c_0$ cake at time $0$ and $c_1$ cake at time $1$, then the total utility gained will be
\[
u(c_0) + \beta u(c_1).
\]

\subsection*{The Optimization Problem}

If we are to consume a cake of size $W_0$ over $T$ time periods, then we can represent our consumption at each step as a vector
\[
(c_0, c_1, ... , c_T)
\]
where
\[
\sum_{i=1}^T c_i = W_0.
\]

We will call such a vector a policy.  Thus, our optimization problem is to

\begin{align*}
\mbox{maximize }  & \sum_{t=0}^T \beta^t u(c_t) \\
\mbox{subject to } & \sum_{t=0}^T c_t = 1, c_t \geq 0.
\end{align*}

\begin{problem}

It might seem obvious what sort of policy will yield the most utility, but the truth may surprise you.
See Figure \ref{fig:diff_pols} for some examples.
Write a function called \li{graph_policy} will accepts
\begin{itemize}
\item a policy,
\item a utility function, and
\item a discount factor;
\end{itemize}
and returns the total utility gained with the policy input.
Also display a plot of the total utility gained over time.
Ensure that the policy that the user passes in sums to 1.
Otherwise, raise an \li{InputError}.

%\begin{figure}
%\begin{subfigure}{.5\textwidth}
%    \includegraphics[width=\textwidth]{fixed_time.pdf}
%\end{subfigure}
%\begin{subfigure}{.5\textwidth}
%    \includegraphics[width=\textwidth]{fixed_w.pdf}
%\end{subfigure}
%\caption{Slices of the finite horizon value function for fixed values of $t$ and $W$, respectively.}
%\label{fig:valueslices}
%\end{figure}
\end{problem}
\begin{figure}
\includegraphics[width=\textwidth]{diff_policies.pdf}
\caption{Plots for various policies with a square root utility function.  Policy 1 eats all the cake in the first step while policy 2 eats all the cake in the last step.  Their difference in utility demonstrate the effect of the discount factor on waiting to eat.  Policy 3 eats the same amount of cake at each step, while policy 4 starts eating a lot of cake but eats less and less as time goes on and the cake runs out.}
\label{fig:diff_pols}
\end{figure}

\section*{The Value Function}

We have posed the cake eating problem as an optimization problem, but we do not have a good way to solve it yet.
To that end, we introduce the value function.
The value function  $V(a,b,W)$ is the maximal value of

\begin{align*}
\mbox{maximize } & \sum_{t=a}^b \beta^tu(c_t) \\
\mbox{subject to } & \sum_{t=a}^b c_t = W, c_t \geq 0
\end{align*}

for $t = a, a+1, ... b$.
In other words, it gives the utility gained from following an optimal policy from time $a$ to time $b$.
%The Value function gives how much utility that a policy from time $a$ to $b$ yields given a cake of size $W$.
So if we start with a cake of size $W_0$ at time $0$ and have half the cake left at time $t$, then $V(t, T, \frac{W_0}{2})$ gives how much utility we will gain by proceeding optimally from $t$.

Thus, if we know what the optimal policy in the future is, we can figure out the optimal policy for the present, since

\[
V(t, T, W_{t}) = \max_{W_{t+1}} u(W_{t} - W_{t+1}) + \beta V(t+1, T, W_{t+1}).
\]
Recall that we know what the utility function $u$ is, as well as $\beta$.
Also, note that our consumption at time $t$ is given by the amount of cake at the beginning of time $t$ minus the amount of cake at the beginning of time $t+1$, or $W_t - W_{t+1}$.

In other words, if we know how to maximize utility in the future, we only need to maximize utility in the now in order to maintain optimality.
So if we know what $V(t+1, T, W_{t+1})$ yields for all possible $W_{t+1}$, then we choose the best $W_{t+1}$ to figure out what the value function yields in the present.
Thus, if we can evaluate the value function, we can also determine the optimal policy.

\subsection*{Solving the Optimization Problem}

\begin{problem}

In this problem we will write a helper function that will help us choose the optimal policy.
We will assume our discritized cake has volume $1$ and an integer $N$ equaly-sized pieces.
Write a method that accepts $N$ and a utility function.
Create a vector whose entries correspond to possible amounts of cake given the discretization.
For example, if we wanted to consider a cake split into 4 pieces, the vector should be
\[
w = (0, 0.25, 0.5, 0.75, 1.0)
\]
Return a matrix whose $ij^{th}$ entry is the amount of utility gained by starting with $i$ cake pieces and ending with $j$ pieces.
In other words, the $ij^{th}$ entry should be $u(w_i - w_j)$.
Let the default utility function be the square root function.
Set impossible situations to 0 (i.e., eating more than you have).
We call the resulting matrix the consumption matrix.

\end{problem}

How can we find the value function if we do not know the optimal policy?
So far in our problem, we do not know how to start eating in order to maximize utility.
However, we know that we do not want to have any cake left over when time runs out.
Thus, the amount of utility we gain from having $W_{T-1}$ cake at time $T-1$ is given by $u(W_{T-1})$, and so we know $V(T-1, T, W_{T-1})$ given a choice of $W_{T-1}$.
For example, if we had a cake split into 4 pieces and wanted to eat it in $T=4$ time intervals, we could express what we know about the value function from time $T$ to $T+1$ as

\[
\begin{bmatrix}
* & * & * & * & u(1) & 0 \\
* & * & * & * & u(0.75) & 0 \\
* & * & * & * & u(0.5) & 0 \\
* & * & * & * & u(0.25) & 0 \\
* & * & * & * & :w
u(0) & 0 \\
\end{bmatrix}
\]

where the $ij^{th}$ entry gives the value of having $w_j$ cake at time $j$ and $*$ represents unknown values.
This is called the value function matrix, and note that it is dimension $N \times T+2$, since we want columns for $t = 0$ and $t = T+1$.
The last column is there to show that after time $4$, there is no value in having any cake.

\begin{problem}
Write a function that accepts
\begin{itemize}
\item $T$ (the number of time steps),
\item $N$ (the number of pieces into which we split our cake),
\item a discount factor $\beta$, and
\item a utility function;
\end{itemize}
and returns the value function matrix with the last two columns filled in.
Let the default utility function be the square root function.
\end{problem}

As we mentioned in the previous section, if we know what the value function evaluates to in the future, we can evaluate the value function in the present.
If we are at time $t$, and we know what the value function evaluates to at time $t+1$ (given by the appropriate column of the value function matrix we have built), then we can calculate
\[
V_t(W_t) = u(W_{t} - W_{t+1}) + \beta V_{t+1}(W_{t+1})
\]
for all possible values of $W_t$ and $W_{t+1}$.
This will produce an $N \times N$ matrix giving the value of all possible values of starting with a cake of size $W_t$ and finishing with size $W_{t+1}$.
We call this matrix the current value matrix.
%If we choose the maximal value of each row, then we know the optimal amount of cake to end up with a the end of time $t$, and thus how much cake to eat.
The largest entry of each row then is the optimal value that the value function can attain at this step, given that we start with the amount of cake that corresponds to the row.
Thus, the maximal values of each row become the column of the value function matrix at time $t$ and if we know one column of the value matrix, we may iterate backwards to fill the rest of it in.
Note that we can start this process with just the column of zeros that we know occupies the last column of the value function matrix.

\begin{problem}
Modify the solution to the previous problem to determine the entire value function matrix.
Initialize the matrix as zeros, and starting from the next to last column
\begin{itemize}
\item calculate the current value matrix,
\item find the largest value in each row, and
\item fill in the column with these values.
\end{itemize}
\end{problem}

Now that the value function matrix is filled in we could say that the optimization problem is solved in some sense.
However, it is not immediately apparent how to make decisions given the value function matrix only.
What we really want is the optimal policy.
It turns out that we already calculated it in the previous problem - we just didn't keep track of it.
Recall that we calculated a current value matrix for each column of the value matrix, and that the $ij^{th}$ entry of that matrix was the value of starting with $i$ pieces of cake and finishing with $j$ pieces of cake.
Thus, the policy for starting a time step with $i$ pieces of cake is $w_i - w_j$ (the vector $w = (w_1, ..., w_N)$ is the equal partition of the cake), where $j$ is the index of w that corresponds to the maximum value of row $i$ in the current value matrix.
We can use this information to fill out an $N \times T+1$ policy matrix at the same time that we fill out the value matrix (T+1 because we inlcude $t=0$).
The $ij^{th}$ entry of the policy matrix tells us how much cake to eat if we start with $w_i$ cake at time $j$.

\begin{problem}
Modify the solution to the previous problem to determine the policy matrix.
Initialize the matrix as zeros and fill it in starting from the last column at the same time that you determining the value function matrix.
\end{problem}

\begin{problem}
The $ij^{th}$ entry of the policy matrix tells us how much cake to eat at time $j$ if we start with $i$ pieces.
Use this information to find the optimal policy for starting with a cake of size 1 split into $N$ pieces, and then plot the policy.
\end{problem}

A summary of the arrays generated in this lab is given below, in the order that they were generated in the lab:

Consumption matrix: Equal to $u(u_i - w_j)$, the utility gained when you start with $i$ pieces and end with $j$ pieces.

Value Function matrix: How valuable is it to have $w_i$ pieces at time $j$.

Current Value matrix: How valuable is each possible decision at time $t$.

Policy matrix: The amount of cake we decide to eat at time $t$.

\begin{comment}
%
% This is the old version of the lab.  I am keeping it in for now.
% If we wish to expand this lab then this material may help write
% more sections, i.e. infinite horizon porblems.
%

\section*{The Sequential Problem, Finite Horizon}
Suppose there are time periods $t=0,1,\ldots, T$ and at each time period we take an action $c_t$. Furthermore, at the beginning
of each time period $t$ we are in some state $W_t$.  In many cases $W_t$ might represent an available resource, such as money.
At each time we receive some reward, $u(W_t,c_t)$, for taking action $c_t$ given state $W_t$.  We assume that rewards are worth
more now than later. We let $\beta\in (0,1)$ represent what is called the discount factor, which gives the ratio of preference for
rewards today versus rewards tomorrow.  For example, receiving a dollar today is preferable to receiving a dollar in a year
because taking a dollar today and putting it into a savings account results in having more than a dollar in a year.  Lastly, over
time our state variable $W_t$ changes according to some rule depending on the previous state and our actions,
\begin{equation}
\label{motion}
W_{t+1} = g(W_t,c_t).
\end{equation}
Equation \eqref{motion} is sometimes referred to as the law of motion, as it describes how we move from state to state.
Mathematically such a problem can be represented as follows:
\begin{equation*}
\text{maximize} \sum_{t=0}^T \beta^t u(W_t,c_t) \quad \text{s.t.} \quad W_{t+1} = g(W_t,c_t)
\end{equation*}
where our initial state, $W_0$, is given.  There may also be restrictions on our choices $c_t$.  For example, in many applications
the state $W_t$ represents the amount of some resource available, and $c_t$ represents the amount we use up in time period $t$.
 In this case we would require $c_t \in [0,W_t]$.

For simplicity, lets assume that $u$ is a function of $c_t$ only (this is often, though not always, the case in practice).
 First let's consider the case that $T=0$.  So we maximize
\begin{equation}\label{1perprob}
u(c_0)
\end{equation}
over $c_0 \in [0,W_0]$.  In most cases $u$ is increasing, which we will assume here.  In this case it will be optimal to choose
 the largest value of $c_0$ possible, that is, $c_0 = W_0$.  Thinking of $W_0$ as our available resources, this simply means
 that if we don't have future periods to consider, we will use all of it.

In fact, this is always true in the last period.  In a problem with $T$ periods we know that we will use all of our resources
remaining in period $T$.  Consider the two period problem:
\begin{equation}
\label{2perprob}
\max \, \{u(c_0) + \beta u(c_1)\}
\end{equation}
where $c_0 \in [0,W_0]$, $c_1 \in [0,W_1]$ and $W_1 = g(W_0,c_0)$.  We know that in the last period we will use all of our
remaining resources, so $c_1 = W_1=g(W_0,c_0)$.  Substituting gives
\begin{equation}
\label{2perproba}
\max \, \{u(c_0) + \beta u(g(W_0,c_0))\}.
\end{equation}
Now we need only determine $c_0$.  Taking the derivative of \eqref{2perproba} with respect to $c_0$ and setting equal to zero
gives the first order condition
\begin{equation}
\label{FOC}
u'(c_0) = -\beta u'(g(W_0,c_0))g_c(W_0,c_0)
\end{equation}
where $g_c$ is the partial derivative of $g$ with respect to $c_0$.

Given a specific form for $u$ we could solve for $W_1$ and obtain the optimal solution.  In fact, we can solve a problem of
any length $T$ in this manner, by starting at the last time period and working backward.  We know that $W_{T+1} = 0$.  Working
backward in time we obtain an equation at each time step $t<T$ by taking the derivative with respect to $c_t$ and setting the equation equal to zero.  This process is called backward induction.  The equations at each time step, such as equation \eqref{FOC}, are sometimes called the inter-temporal Euler equations.  These equations, along with $c_T = W_T$, make $T+1$ equations to go with our $T+1$ unknowns $\{c_0,c_1,c_2,\ldots,c_T\}$, where we can use the law of motion \eqref{motion} to relate the $c_t$ and $W_t$

\section*{The Recursive Problem, Finite Horizon}
Approaching the problem sequentially like this can be somewhat messy.  The dynamic programming approach we consider now is more
easily adaptable to many situations.  The key to the dynamic programming approach is to define our optimization problem in terms
 of subproblems.  Notice that if we are in time period $t$, we face a problem of exactly the same form as the problem at time
 $0$.  We are in some state $W_t$, and want to maximize the sum from $t$ to $T$.  With this idea in mind, we define a function
  $V_t(W_t)$ called the value function.  The function $V_t$ gives the value of entering time $t$ in state $W_t$ and making
  optimal decisions moving forward.  So
\begin{equation*}
V_{t-1}(W_{t-1}) = \max_{c_t} \left\{u(c_{t-1}) + \beta V_t(g(W_{t-1},c_{t-1}))\right\}.
\end{equation*}
This is called the Bellman Equation.  The key to this formulation is that we decide what to do in period $t-1$ with the
assumption that our actions in the remaining periods will be optimal.  This is called the principal of optimality.

Let us consider a specific example from economics called The Cake Eating Problem.  Suppose $W_t$ represents the amount of
cake available at time $t$.  At each time period we can choose how much to consume.  What we eat, $c_t$, gives us a reward.
What we save, $W_t-c_t$, does not give us a reward (until it is eaten in a later period).  The law of motion \eqref{motion}
becomes
\begin{equation}\label{LOM_EX}
W_{t+1} = g(W_t,c_t) = W_t-c_t
\end{equation}

Now we have completely defined the problem.  The Bellman Equation is
\begin{equation*}
V_{t-1}(W_{t-1}) = \max_{c_t} \left\{u(c_{t-1}) + \beta V_t(W_{t-1}-c_{t-1})\right\}.
\end{equation*}
Notice that by the law of motion, each $c_t$ is determined by $W_t$ and $W_{t+1}$.
In fact, rearranging \eqref{LOM_EX}, we have
\begin{equation*}
c_t = W_t - W_{t+1}.
\end{equation*}
We can therefore rewrite the value function as
\begin{equation}
V_{t-1}(W_{t-1}) = \max_{W_t} \left\{u(W_{t-1} - W_{t}) + \beta V_t(W_t)\right\}.
\label{cake_valfn}
\end{equation}

We see that determining the optimal actions $c_t$ is equivalent to determining the optimal states $W_{t}$ in the
above formulation. The solution to this
problem is often called a \emph{policy function}.  A policy function determines an action based on the current
state.  Denoting the policy function by $\psi$, this can be written as
\begin{equation*}
W_{t+1}=\psi_t \left(W_t\right).
\end{equation*}
The policy function gives the optimal amount of cake to leave for the next period (equivalent to the amount of consumption) given
the amount of cake at the start of the period.  In other words, it determines the choice of $W_t$ that satisfies the $\max$
condition in \eqref{cake_valfn}.

As before, we know that in the last time period we should not save anything.  So $V_{T+1}(W_{T+1}) = 0$, i.e. there is
no value in leaving wealth for period $T+1$. Stated in another way, our action at time $T$ should be to eat all of the
remaining cake $W_T$, so $W_{T+1} = \psi_T(W_T) = 0$.
Plugging this result into the Bellman Equation gives us $V_T(W_T) = u(W_T)$.
Now consider the value function equation for period $T-1$:
\begin{align*}
V_{T-1}(W_{T-1}) &= \max_{W_T} \left\{u(W_{T-1} - W_T) + \beta V_T(W_T)\right\} \\
                 &= \max_{W_T} \left\{u(W_{T-1} - W_T) + \beta u(W_T)\right\}.
\end{align*}
We can determine this value by optimizing over $W_T$, where $0 \leq W_T \leq W_{T-1}$.
Continuing backwards in this manner leads us to the solution of the original problem.

\begin{problem}
\label{prob:cake_prob}
We recommend reading the entire problem before beginning to work on it, as many questions may be addressed further on.  This applies to the other problems in this lab as well.

Follow the steps below to solve the problem described above.  Take $u(c_t) = \sqrt{c_t}$.
You will write a function called \li{eatCake} that takes parameters $\beta$ (the discount factor),
$N$ (the number of discrete cake values to consider), $W_{max}$ (the original size of the cake,
set to the default value of $1$), a keyword argument \li{finite} (set to default value \li{True}),
a keyword argument  $T$ (the number of time periods, set to
default value \li{None}), and a keyword argument \li{plot}, which indicates whether or not
to plot the computed results. The function should return arrays representing the value function and the
policy function (we describe how to compute these in the following steps).
\begin{enumerate}
\item Approximate the continuum of possible cake sizes by creating an array
of evenly-spaced values that range from to 0 to $W_{max}$ inclusive.
Let the number of possible cake values be given by $N$. In Python, this can be accomplished easily by
using the \li{linspace} function in NumPy. You should obtain an array (call it $w$) of the form
\[
w = (w_1, w_2, \ldots, w_N),
\]
where $w_1 = 0$ and $w_N = W_{max}$.

\item Note that in order to compute the value function, we need $u(W_{t-1}-W_t)$.
We will pre-compute all such possible values and store them in an array, as follows:
Create an $N$ by $N$ matrix that contains
all possible values of $W_{t-1} - W_t$ (where $W_{t-1}$ corresponds to rows and $W_{t}$ to columns).  Make sure that
$c_t \geq 0$ is satisfied by replacing negative entries in the matrix with zeros.  Then take the square root to get a matrix
of $u(W_{t-1}-W_t)$.  To make sure we do not choose $W_{t-1} - W_t < 0$ when maximizing, replace the corresponding entries of
the $u(W_{t-1}-W_t)$ matrix with a large negative number (e.g. $-10^{10}$). You should end up with a matrix whose
$(i,j)$-th entry is equal to $\sqrt{w_i - w_j}$ when $i \geq j$, and is equal to $-10^{10}$ when $i < j$.

\item Next, create an $N$ by $T+2$ (corresponding to $t=0,1,\ldots, T+1$) matrix representing the value function for a given
time $t$ and state $W_t$.  We can initialize it with zeros and begin filling in the columns starting with the last (which we
know is zeros), as explained below.

\item Now we are ready to iterate backward and compute the value function for each time period.  To find $V_T$, we first compute
$u(W_T - W_{T+1}) + \beta V_{T+1}(W_{T+1})$ for all values of $W_{T}$ and $W_{T+1}$.  This will result in an $N$ by $N$ matrix
where the rows correspond to values of $W_{T}$ and the columns correspond to values of $W_{T+1}$.
%Note that to compute this
%we need a matrix representing $\beta V_{T+1}(W_{T+1})$.  Because this quantity does not depend on $W_T$, its rows should be
%equal.  To do this, we want to take $\beta V_{T+1}(W_{T+1})$ as a row vector and stack this vector to create a matrix with
%equal rows.  There are multiple ways to do this.  One is the \li{np.repeat} function.  For example, if \li{b} is a row vector
%it could be used like the following.
%\begin{lstlisting}
%b = [[1, 2, 3]]
%np.repeat(b, 3, axis = 0)
%array([[1, 2, 3],
%[1, 2, 3],
%[1, 2, 3]])
%\end{lstlisting}
%In general, be careful about having the correct rows, columns, transposes, etc throughout your code.

Now we maximize over choices of $W_{T+1}$ (choosing how much to save for the next period).  Then we will have a row vector
representing the value function for period $T$ across all possible $W_{T+1}$.  Iterate this procedure to fill in the value
function for all $t=T+1,T,\ldots, 0$.

\item In each iteration, you maximize to find the value function at time $t$.  Save the values of $W_{t+1}$ that achieve the
maximum.  The result is an $N$ by $T+1$ matrix whose $(n,t)$ entry gives the optimal amount of cake to leave for period
$t+1$, given that we start period $t$ with the the $n$-th value of our vector of cake.  This is the policy function.

\item If the keyword argument \li{plot} is set to \li{True}, plot the surface of the Value and Policy functions.
This can be done by including the following import lines
\begin{lstlisting}
>>> from matplotlib import pyplot as plt
>>> from matplotlib import cm
>>> from mpl_toolkits.mplot3d import Axes3D
\end{lstlisting}
and using the following code:
\begin{lstlisting}
>>> W = np.linspace(0, Wmax, N)
>>> x = np.arange(0, N)
>>> y = np.arange(0, T+2)
>>> X, Y = np.meshgrid(x, y)
>>> fig1 = plt.figure()
>>> ax1 = Axes3D(fig1)
>>> ax1.plot_surface(W[X], Y, np.transpose(V), cmap=cm.coolwarm)
>>> plt.show()

>>> fig2 = plt.figure()
>>> ax2 = Axes3D(fig2)
>>> y = np.arange(0,T+1)
>>> X, Y = np.meshgrid(x, y)
>>> ax2.plot_surface(W[X], Y, np.transpose(psi), cmap=cm.coolwarm)
>>> plt.show()
\end{lstlisting}
where \li{W} is the vector of cake amounts, \li{V} is the value function, and \li{psi} is the policy function.

\item Return the arrays giving the value function and the policy function.
\end{enumerate}

Solve the problem using cake size $1$, discount factor $\beta = .9$, number of time periods $T = 10$, and number of
discrete cake values $N = 100$. You should also try plotting the value and policy functions for fixed time periods across
$W_t$, or for fixed $W_t$ across time,
 and make sure that these plots fit your intuition. See Figure \ref{fig:valueslices}. Your output should
agree with the figure.
\end{problem}

\begin{figure}
\begin{subfigure}{.5\textwidth}
    \includegraphics[width=\textwidth]{fixed_time.pdf}
\end{subfigure}
\begin{subfigure}{.5\textwidth}
    \includegraphics[width=\textwidth]{fixed_w.pdf}
\end{subfigure}
\caption{Slices of the finite horizon value function for fixed values of $t$ and $W$, respectively.}
\label{fig:valueslices}
\end{figure}

\section*{The Recursive Problem, Infinite Horizon}\label{SecRecProbInFHor}
Next we consider an infinite horizon problem.  For simplicity, we continue with the example from the previous section.
Suppose that rather than optimizing over $t = 0,1,\ldots,T$, we wish to optimize over an infinite time horizon:
\begin{equation*}
\text{maximize} \sum_{t=0}^\infty \beta^t u(W_t,c_t) \quad \text{s.t.} \quad W_{t+1} = g(W_t,c_t).
\end{equation*}
Since at any time $t$, there are an infinite number of periods remaining, one might suspect that the optimal policy will
not depend on the current time $t$.

\begin{problem}
\label{prob:cake_prob2}
Compute the solution to Problem \ref{prob:cake_prob} with $T = 1000$, and the rest of the inputs the same.
Plot the policy function across time for fixed $W_t = 1$.
Notice that it is the same for all time periods, except those near the end time $T$.
\end{problem}

As suggested by the results of Problem \ref{prob:cake_prob2},  the policy function for the infinite horizon problem does not
depend on the time $t$ (this can be proved).  That is, at any time $t$, the optimal decision depends only on the amount of cake
at the beginning of the period, not the value of $t$.  So everything can now be written in terms of variables today and variables
tomorrow. We will denote variables tomorrow with a ``$\:'\:$".
\begin{equation}
\label{EqBellman}
V\left(W\right) = \max_{W'\in[0,W]}\:\: \left\{u\left(W - W'\right) + \beta V\left(W'\right)\right\}
\end{equation}
Note that the value function $V$ on the left-hand-side of \eqref{EqBellman} and on the right-hand-side are the same function.

Because the problem now has an infinite horizon, the nature of the solution is a little different. The solution to \eqref{EqBellman}
is a policy function $W'=\psi(W)$ that creates a fixed point in $V$. In other words, the solution is a policy function $\psi(W)$
that makes the function $V$ on the left-hand-side of \eqref{EqBellman} equal the function $V$ on the right-hand-side.

Define $C$ as an operator on any value function $V_k\left(W\right)$. Let $C$ perform the following operation.
\begin{equation}
\label{EqContraction}
C\Bigl(V_k\left(W\right)\Bigr) \equiv \max_{W'\in[0,W]}\:\: \left\{u\left(W-W'\right) + \beta V_k\left(W'\right)\right\}.
\end{equation}
Note that the value function on the right-hand-side of \eqref{EqContraction} and on the left-hand-side are the same function $V_k$,
but have different inputs--$W$ versus $W'$. The operator $C$ takes in a function $V_k$, and gives a new
function which we will call $V_{k+1}$:
\begin{equation*}
V_{k+1}\left(W\right) \equiv C\Bigl(V_k\left(W\right)\Bigr).
\end{equation*}
The value function $V_{k+1}$ that results from the operation $C$ is not necessarily the same as the value function that the system
began with ($V_k$). However, according to equation \eqref{EqBellman} we seek a $V$ such that $C(V) = V$.  The solution, then, is the fixed point in $V$.
\begin{equation*}
C\Bigl(V_k\left(W\right)\Bigr) = V_{k+1}\left(W\right) = V_k\left(W\right) = V\left(W\right)
\end{equation*}

When trying to solve a fixed point equation, it is often very helpful to utilize the Contraction Mapping Principle, which
guarantees the existence of a fixed point of a mapping, provided that the map sends any two distinct inputs to outputs
that are strictly closer to each other than the inputs, in a controlled way. This principle also provides a constructive
way to obtain the fixed point, namely by iterating the map.
Fortunately, it can be shown that if $u(\cdot)$ is real-valued, continuous, and bounded, $\beta\in(0,1)$, and that the constraint
set $W'\in[0,W]$ is nonempty, compact-valued, and continuous, then the operator $C$ is a contraction and thus we can obtain
a solution $V$ by iteration:
\begin{equation*}
\lim_{k\rightarrow\infty}\: C^k\Bigl(V_0\left(W\right)\Bigr) = V_k(W) =  V\left(W\right)
\end{equation*}
for any $V_0$.

Remember, in the infinite horizon problem both the value and policy functions do not depend on time.  Computationally, this means that
 the value and policy functions in the infinite horizon problem are one dimensional.

\begin{problem}
Expand your \li{eatCake} function to solve the Cake Eating Problem with an infinite time horizon. If the keyword argument
\li{finite} has the value \li{True}, then your function should behave as in Problem \ref{prob:cake_prob}, solving the finite
time horizon problem. However, if \li{finite = False}, solve the infinite time horizon problem through the following steps.
Both problems will require you to pre-compute the values $u(W - W')$, where $W$ and $W'$ range over the set of discrete cake
amounts. Be sure to avoid replicating code by factoring it out.
As in Problem \ref{prob:cake_prob}, take $u(c_t) = \sqrt{c_t}$.
\begin{enumerate}
\item As in Problem \ref{prob:cake_prob}, approximate the continuum of possible
cake sizes by a column vector called $W$ that ranges from 0 to $W_{max}$ in $N$ steps.

\item \label{item:step2} Initialize the value function V as a vector of zeros of length $N$.  This is $V_0$.  Perform one iteration
of the contraction operation given in equation \eqref{EqContraction} to get a new value function $V_1$ (this should be very similar
to Problem 1).  Determine the resulting policy function $W' = \psi_1\left(W\right)$.  [HINT: The policy function should be a vector
of length $N$ of optimal future values of the cake $W'$ given the current value of the cake $W$, and $V_T$ should be an $N$-length
vector representing the value of entering a period with cake size $W$.]

\item \label{item:step3} Measure the distance between the two value functions as the sum of the
squared differences,
\begin{equation}
\label{EqDist}
\delta_1\equiv \norm{V_1\left(W\right) - V_0\left(W'\right)}_2^2 = \left(V_1 - V_0\right)^T\left(V_1 - V_0\right).
\end{equation}
Defined in this way, $\delta_1\in [0,\infty)$.

%\item \label{item:step4} Take the resulting $V_1$ from \ref{item:step2}, and perform the same contraction on it to generate $V_2$
%and $\psi_2$. That is, generate,
%\begin{equation*}
%  V_2\left(W\right) = C\Bigl(V_1\left(W\right)\Bigr) = \max_{W'\in[0,W]}\: u\left(W - W'\right) + \beta V_1\left(W'\right)
%\end{equation*}
%and the accompanying policy function $W'=\psi_2\left(W\right)$. Calculate the accompanying distance measure for $\delta_2$ using
% the formula from \eqref{EqDist} with the updated period subscripts. Compare $\delta_2$ with $\delta_1$ from \ref{item:step3}.
%
%\item \label{item:step5} Repeat \ref{item:step4} and generate $V_3$ and $\psi_2$ by performing the contraction on $V_2$. Compare
%$\delta_3$ to $\delta_2$ and $\delta_1$.

\item Write a loop that performs the contraction operation from steps \ref{item:step2} and \ref{item:step3} iteratively
until the distance measure is very small ($\delta_k < 10^{-9}$).  The distance measure $\delta_k$ being arbitrarily close to zero means
 you have converged to the fixed point $V_k = V_{k+1} = V$. (For fun, you can show that the policy function converges to the same
 function regardless of what you put in for your initial policy function value.)

\item If \li{plot = True}, plot the converged policy function vector
($y$-axis) as a function of the cake amounts ($x$-axis).

\item Return the value function and policy function arrays.

Compute the value function and policy function for the infinite time horizon problem with
cake size $1$, discount factor $\beta = .9$, and number of
discrete cake values $N = 100$. The plot you generate should agree with Figure \ref{fig:infinitePolicy}.
\end{enumerate}
\end{problem}

\begin{figure}
\includegraphics[width=\textwidth]{infiniteHorizon.pdf}
\caption{Policy function for infinite time horizon.}
\label{fig:infinitePolicy}
\end{figure}

\begin{figure}
\includegraphics[width=\textwidth]{convergence.pdf}
\caption{Due to the contraction mapping principle, $\delta_k$ decreases as we perform iterations
until it is small enough to meet our convergence tolerance.}
\end{figure}

\section*{Infinite Horizon, Stochastic, i.i.d.}\label{SecRecProbInfinHorStochiid}

In practice, dynamic programming problems often involve some level of uncertainty.
For example, as time progresses prices may fluctuate, resources may vary, or preferences themselves may change.
 In this lab, we reexamine the cake eating problem, this time allowing for uncertainty.

We consider again the problem of optimizing a sequence of decisions over an infinite time horizon.
We assume that the individual's preferences deviate each period according to some ``shock" $\ve$,
where $\ve$ is a random variable.  We assume that the shock terms $\ve$ for each time period are
independent and identically
distributed (i.i.d.).  In effect, this means the probabilities associated with the $\ve$ are the
same for any time $t$ and do not depend on each other.  We assume for now that the $\ve$ are
distributed normally with mean $\mu$ and variance $\sigma^2$.  The Bellman equation can be easily
rewritten in the following way to incorporate the uncertainty,
\begin{equation}\label{stoch_Bellman}
   V\left(W,\ve\right) = \max_{W'\in[0,W]}\: \{\ve u\left(W - W'\right) +
   \beta E_{\ve'}\left[V\left(W',\ve'\right)\right]\},
\end{equation}
where $\ve \sim N(\mu,\sigma^2)$ and
$E$ is the unconditional expectation operator over $\ve$.  Note that now the value function depends
on two variables.  It represents the value of entering the period with $W$, the amount of cake,
and a preference shock of $\ve$.  For example, in a period where the realization of $\ve$ is higher,
we will get more value from the cake eaten in the current period.  Because we do not know the value of
the shock in the next period $\ve'$, we consider only the expected value for future time.

As it turns out, we can solve this problem in a manner similar to the infinite horizon deterministic cake-eating
problem considered in the Value Function Iteration lab.  It is worth noting that in this case,
 the value and policy functions will be two dimensional, as they will depend on both $W$ and $\ve$.

In order to deal with $\ve$ computationally, we would like to represent it as a vector of possible values
it could take, along with the corresponding probabilities that it takes each of those values.  However,
$N(\mu,\sigma^2)$ is a continuous distribution, so we cannot represent every value $\ve$ could take.
We need a discrete distribution that approximates $N(\mu,\sigma^2)$.

To do this, we choose $N$ equally spaced points centered about the mean at which to approximate the
distribution. Call these values $\ve_1,\ldots,\ve_N$, and let the spacing between adjacent points be
given by $\delta$.
We can then break up the support of the distribution into $N$ bins, where adjacent bins share a common
endpoint. Call the endpoints of these bins $v_1,\ldots,v_{N+1}$. By choosing the endpoints to be
halfway between each $\ve_k$, we have the formula
\[
v_k = \ve_k - \frac{1}{2}\delta, \qquad k=1,\ldots,N
\]
and
\[
v_{N+1} = \ve_N + \frac{1}{2}\delta.
\]

\begin{figure}[h!]
\label{stoch1_fig1}
\begin{center}
\includegraphics[width = \textwidth]{discnorm.pdf}
\end{center}
\caption{Discretization of $N(\mu,\sigma^2)$.  We approximate $P(\ve = \ve_k)$ by the area of the shaded region.}
\end{figure}

We can then associate $\ve_k$ with the area under the curve from $v_k$ to $v_{k+1}$.
In Python, we can find the area using the function \li{norm.cdf} found in the \li{stats} package.
The cdf (cumulative distribution function) gives the area under the curve from $-\infty$ to a specified value.
For example, in the following code, \li{eps} is the area under the curve from 0 to 1.

\begin{lstlisting}
>>> from scipy import stats as st
>>> mu = 0
>>> sigma = 1
>>> eps = st.norm.cdf(1,loc=mu,scale=sigma) - st.norm.cdf(0,loc=mu,scale=sigma)
\end{lstlisting}

In general, it is sufficient to take our points $\ve_k$ ranging from $\mu - 3\sigma$ to $\mu + 3\sigma$, as this
range contains about 99.7\% of the probability mass.

\begin{problem}
Write a function called \li{discretenorm} that accepts an integer $K$ representing the number of discrete points
desired, a mean $\mu$, and a standard deviation $\sigma$. It should return a length-$K$ vector of equally-spaced
values ranging from $\mu - 3\sigma$ to $\mu + 3\sigma$ inclusive,
and a length-$K$ vector containing the associated probabilities.
Plot the approximation of $N(0,1)$ using different values of $K$ to check that your results are plausible.
\end{problem}

Now that we have a discrete distribution for $\ve$, we can solve for the value and policy functions
determined by \eqref{stoch_Bellman}.

\begin{problem}
Complete the following steps to solve the problem described above.
Assume that the period utility function is $u(c)=\sqrt{c}$.
Write a function \li{stochEatCake}
that accepts parameters $\beta$ (discount factor), $N$ (number of discrete cake values),
a tuple of values \li{e_params}, $W_{max}$ (the original size of the cake, set to default value 1),
a keyword argument \li{iid} (set to default value \li{True}), and
a keyword argument \li{plot} (set to default value of \li{False}). Inside the function, carry out the steps
outlined below.

The argument \li{e_params} is a tuple consisting of the values needed to generate
the discrete approximation to $\ve$. In the present case, this tuple consists (in order) of
$K$ (the number of discrete approximations of $\ve$), $\mu$ (the
mean of the shock term $\ve$), and $\sigma$ (the standard deviation of the shock term $\ve$),
since these are the arguments we need to pass to our \li{discretenorm} function.

\begin{enumerate}
\item First, compute an approximation of $\ve$ using the \li{discretenorm} function created in Problem 1.
Use $K$ equally spaced points to approximate $N(\mu,\sigma^2)$. Denote the resulting $K$-length
vector of equally-spaced values by
\[e =(e_1,\ldots,e_K),
\]
and denote the $K$-length vector of the associated probabilities
by
\[\Gamma = (\Gamma_1,\ldots,\Gamma_K).
\]
Note that $\Gamma_k$ give the probability $P(\ve = e_k)$.

Since the values needed for the \li{discretenorm} function are contained in the \li{e_params} input,
we can feed these values directly into the function in the following way:
\begin{lstlisting}
>>> e, gamma = discretenorm(*e_params)
\end{lstlisting}
The \li{*} operator essentially unpacks the values of a tuple or list.

\item As done previously, create a vector
\[w = (w_1,\ldots,w_N)
\]
of possible cake sizes. This should be
a length-$N$ vector of equally spaced values from 0 to $W_{max}$, inclusive.

\item Represent the value function as a $N \times K$ matrix $v$, satisfying
\[
v_{i,j} = V(w_i, e_j).
\]
(The rows correspond to different values of $W$ and the columns correspond to different values of $\ve$.)
Initialize each entry of the matrix to 0.

Likewise, represent the policy function as a $N \times K$ matrix $p$, satisfying
\[
p_{i,j} = \psi(w_i,e_j).
\]
Initialize all entries to 0.

\item In order to evaluate the value function equation, we need to pre-compute $\ve u(W-W')$ for all values of
$\ve,W,W'$.
Begin by computing all possible values of $u(W-W')$, and storing these values in a $N \times N$ array,
as done before. Call this array $u$. Make sure that the upper triangular
entries of this array are equal to zero, as these entries correspond to consuming more cake than is
available, which is impossible.

The values $\ve u(W-W')$ will be represented by a three-dimensional array $\hat{u}$ of size
$N\times N\times K$, satisfying
\[
\hat{u}_{i,j,k} = v_{i,j}e_k.
\]
We can compute this array easily as follows:

\begin{lstlisting}
>>> import numpy as np
>>> u_hat = np.repeat(u, K).reshape((N,N,K))*e
\end{lstlisting}

\item We also need to compute $E_{\ve'}\Bigl[V\left(W',\ve'\right)\Bigr]$ for each value of $W'$.
The expected value is simply
\begin{equation*}
E_{\ve'}\Bigl[V\left(W',\ve'\right)\Bigr] = \sum_{k=1}^K \Gamma_kV(W',e_k').
\end{equation*}
The result is a length $N$ vector, call it $E$, satisfying
\[
E_i = E_{\ve'}\Bigl[V\left(w_i,\ve'\right)\Bigr] = \sum_{k=1}^K \Gamma_kv_{i,k}
\]
This calculation can be done by multiplying $\Gamma$ element-wise to each row of the
value function matrix $v$, and then summing along the rows. Something like the following
line of code should do the trick:
\begin{lstlisting}
>>> E = (v*gamma).sum(axis=1)
\end{lstlisting}

\item We can now compute the value function contraction
\begin{equation*}\label{EqContractStochiid}
C\Bigl(V\left(W,\ve\right)\Bigr) \equiv \max_{W'\in[0,W]}\:
\Bigl\{\ve u\left(W-W'\right) + \beta E_{\ve'}\Bigl[V\left(W',\ve'\right)\Bigr]\Bigr\}.
\end{equation*}
The first task is to create an $N \times N \times K$ array $c$ satisfying
\[
c_{i,j,k} = \hat{u}_{i,j,k} + \beta E_j.
\]
This can be done in any manner of ways. Below is a one-liner that does the job.
\begin{lstlisting}
>>> c = np.swapaxes(np.swapaxes(u_hat, 1, 2) + beta*E, 1, 2)
\end{lstlisting}

Now, for any $k$, for all $i < j$, set $c_{i,j,k}$ to a large negative number, say $-10^{10}$,
so that when maximizing over this array, we do not choose to consume more cake than is available.
Again, this can be done in a variety of different ways, but the following does the job concisely:
\begin{lstlisting}
>>> c[np.triu_indices(N, k=1)] = -1e10
\end{lstlisting}

Finally, maximize over the second axis of $c$ (which corresponds to different values of $W'$)
to obtain the updated value function matrix:
\begin{lstlisting}
>>> v_new = np.max(c, axis=1)
\end{lstlisting}
You can likewise update your policy function matrix as follows:
\begin{lstlisting}
>>> max_indices = np.argmax(c, axis=1)
>>> p = w[max_indices]
\end{lstlisting}

\item We now have our updated value function matrix $v_{new}$ as well as the
previous $v$, which we refer to here as $v_{old}$. As we iterate on the value function equation, we need a norm
\begin{equation*}
\delta = \|v_{new} - v_{old}\|_2
\end{equation*}
that measures the distance between these two value functions to determine convergence.
You may compute the norm using the SciPy function \li{scipy.linalg.norm}, or by direct calculation.
At the end of each iteration, make sure to set $v$ to $v_{new}$, so that the updates carry through the
loop.
Iterate on the contraction until $\delta < 10^{-9}$.

\item If \li{plot = True}, make a 3-D surface plot of the policy function for the converged problem
$W' = \psi\left(W,\ve\right)$ which gives the value of the cake tomorrow as a
function of the cake today  and the taste shock today.  Do the same for the value function.
Example code to create the value function plot is provided below.
\begin{lstlisting}
>>> x = np.arange(0,N)
>>> y = np.arange(0,K)
>>> X,Y = np.meshgrid(x,y)
>>> fig1 = plt.figure()
>>> ax1 = Axes3D(fig1)
>>> ax1.plot_surface(w[X], Y, v.T, cmap=cm.coolwarm)
>>> plt.show()
\end{lstlisting}
Creating the policy function plot is similar.

\item Return the converged value function matrix $v$ and policy function matrix $p$.

\end{enumerate}
Test your function using values $\beta = .9$, $N = 100$, $K = 7$, $\sigma = .5$, $\mu = 4\sigma$,
and \li{plot = True}.
The proper way to set this up and call the function is as follows:
\begin{lstlisting}
>>> e_params = (7, 4*.5, .5)
>>> stuff = stochEatCake(.9, 100, e_params, plot=True)
\end{lstlisting}
\end{problem}

\begin{figure}
    \centering
    \includegraphics[width = \textwidth]{stoch_value.pdf}
    \caption{3D surface representing the value function for the Stochastic Cake-Eating problem.}
\end{figure}

\section*{Infinite Horizon, Stochastic, AR(1)}\label{SecRecProbInfinHorStochAR1}

In the previous example, we assumed that the shocks at time $t$ were independent of what happened in
previous periods.  Often a shock may depend on recent events.  We will assume now that the shocks are persistent,
meaning preferences in the current period are more likely to be close to what they were in the previous period.
We can characterize the persistence by what is called an autoregressive process of order one, denoted AR(1).
Such a process is defined as follows.
\begin{equation}\label{EqAR1shock}
\ve' = (1-\rho)\mu + \rho\ve + \nu' \quad\text{where}\quad \rho\in(0,1) \quad\text{and}\quad \nu\sim N(0,\sigma^2).
\end{equation}

Essentially, instead of allowing the shocks to have a mean which is independent of the past, the mean is now a
weighted average (weighted by $\rho$) of some $\mu$ and the previous realization of the shock, $\ve$.  As it turns
 out, we can approximate this process by thinking of it as a Markov Chain. This means we need to determine a
 discrete set of points representing possible values of $\ve$ and a Markov transition matrix that gives the
 probabilities of moving from one value of $\ve$ to another.  There are methods for determining the discrete
 approximation of $\ve$ with a Markov transition matrix.  These methods are beyond the scope of this section,
 but you can use the file \li{tauchenhussey.py} to implement them in the next problem.

The Bellman equation becomes the following, in which the only change from the i.i.d. shock case is that the
expectations operator is now conditional on the current shock $\ve$:

\begin{equation*}
   V\left(W,\ve\right) = \max_{W'\in[0,W]}\: \{\ve u\left(W - W'\right) +
   \beta E_{\ve'|\ve}\left[V\left(W',\ve'\right)\right]\},
\end{equation*}
where $\ve'$ is distributed according to \eqref{EqAR1shock}.
Let $\Gamma_{i,j}=P\left(\ve_j'|\ve_i\right)$ where $\ve_j'$ is the value of the shock in
the next period and $\ve_i$ is the value of the shock in the current period.
In other words, $\Gamma$ is the Markov transition matrix.

The solution to this problem is of the same type as that in the i.i.d. case, since the only difference is
the probability distributions of the $\ve$.

\begin{problem}
Expand your \li{stochEatCake} function to handle the case of AR(1) shock terms. The function should
handle this case for the parameter value \li{iid = False}, and should handle the previous case of
normally distributed i.i.d. shock terms for the parameter value \li{iid = True}. You will need to
add a few ``if ... else" statements, as well as implement the steps outlined below, but most of the
code will remain unchanged.

\begin{enumerate}
\item In the AR(1) case, the \li{e_params} argument should be a tuple of values needed to
generate the arrays $e$ and $\Gamma$ that approximate the values and distribution of $\ve$
as a Markov chain.
Use the file \li{tauchenhussey.py} to calculate these arrays.
The provided Python function \li{tauchenhussey} produces the vector $e$ of length $M$
and an $M\times M$ transition matrix $\Gamma$.
Thus, you simply need the following lines of code, similar to the previous case.
\begin{lstlisting}
>>> from tauchenhussey import tauchenhussey
>>> e, gamma = tauchenhussey(*e_params)
\end{lstlisting}

\item Because our values for $e$ and $\Gamma$ are different in the AR(1) case than
in the i.i.d. case, we must compute the expectation in a different manner.
In particular, we need to compute the conditional expectation
\begin{equation*}
E_{\ve'|\ve}\Bigl[V\left(W',\ve'\right)\Bigr].
\end{equation*}
We obtain a two-dimensional array, since the expectation depends on both $W'$ and on $\ve$.
The expectation can be computed by the matrix multiplication $v\Gamma^T$.
Your code should match the following.
\begin{lstlisting}
>>> E = v.dot(gamma.T)
\end{lstlisting}

\item The last difference comes in computing the array $c$. Fortunately, it is easier in this case.
Recall that $c$ gives the values for
\[
\ve u\left(W-W'\right) + \beta E_{\ve'|\ve}\Bigl[V\left(W',\ve'\right)\Bigr].
\]
The array $\hat{u}$ contains the values for the first term in the expression, and the array $E$
contains the values for the expectation term.
Hence, we obtain $c$ by simple addition. Array broadcasting makes this work without problems.
\begin{lstlisting}
>>> c = u_hat + beta*E
\end{lstlisting}
You will still need to set the upper triangular entries of $c$ to a large negative number, just as in the
previous case.
\end{enumerate}

Those are the only differences. Let the following code snippet be a guideline for how to implement
these differences.
\begin{lstlisting}
>>> if iid:
>>>     # compute E as outlined in the previous problem
>>> else:
>>>     # compute E as outlined in the current problem
\end{lstlisting}

Now test your function with $\beta = .9$, $N = 100$, \li{iid = False}, and \li{plot = True}.
As inputs to \li{tauchenhussey}, let $K=7$, the mean of the process
$\mu=4\sigma$, $\rho = 1/2$, $\sigma=1/2$, and
\[baseSigma=(0.5+\frac{\rho}{4})\sigma +
(0.5 - \frac{\rho}{4})\frac{\sigma}{\sqrt{1-\rho^2}}.
\]
Your \li{e_params} parameter will therefore be a tuple of values containing (in order)
$K$, $\mu$, $\rho$, $\sigma$, and $baseSigma$.
\end{problem}
\end{comment}


The previous chapters focused on using camera models to identify the relationship between points in a 3D scene and their projections onto the camera image, as well as how to leverage those models to reconstruct 3D scene structure from 2D images.
Alternatively, this chapter begins to look at methods for extracting other types of information through \textit{image processing}, for example to answer the question ``what object am I seeing?'' rather than ``how far away is this object?''.
Extracting this type of visual content from raw images is important for mobile robots to be able to intelligently interpret their surroundings. In fact, it can have a major impact on the ability of the robot to perform several tasks including localization and mapping or decision making. This chapter focuses on some of the more commonly used tools in image processing including image filtering, feature detection, and feature description\cite{SiegwartNourbakhshEtAl2011}\cite{Moravec1977}.

\notessection{Image Processing}
Image processing is a form of signal processing where the input signal is an image (such as a photo or a video) and the output is either an image or a set of parameters associated with the image. While a large number of image processing techniques exist, this chapter focuses on some of the more fundamental methods that are relevant for robotics. In particular, these methods will be related to image filtering, feature detection, and feature description\footnote{The software library OpenCV implements a number of useful image filtering algorithms: \url{https://docs.opencv.org}.}.

In the following methods, grayscale images are treated as functions $I$: $[a,b]\times[c,d] \rightarrow [0,L]$, where $I(x,y)$ represents the grayscale pixel intensity at $(x,y)$.
For a color image, $I$ is a vector valued function with three components, one each for the red, green, and blue color channels of the image.

\subsection{Image Filtering}
Image filtering is one of the principal tasks in image processing. The terminology ``filter'' comes from frequency domain signal processing and refers to the process of accepting or rejecting certain frequency components of a signal (e.g. eliminating high-frequency noise).

Perhaps the most common type of image filtering is \textit{spatial filtering}.
The basic principle of spatial filtering is that a particular pixel is modified in the filtered image based only on the pixels in the immediate spatial neighborhood (see Figure \ref{fig:spatial_filter_concept_fig}). To be more specific, a spatial filter for an image $I(x,y)$ consists of:
\begin{enumerate}
    \item A neighborhood $S_{xy}$ of pixels around a particular point $(x,y)$ under examination, typically rectangular.
    \item A predefined operation $F$ that is performed on the image pixels encompassed by the neighborhood $S_{xy}$.
\end{enumerate}
Once the operation $F$ has been applied to all pixels $(x,y)$ in the image $I$ a new image $I'(x,y)$ is defined.
\begin{figure}[ht]
  \centering
  \includegraphics[width=.75\textwidth]{tex/figs/ch10_figs/spatialfiltering.png}
    \caption{Illustration of the concept of spatial filtering. The spatial filter operates on a neighborhood $S_{xy}$ of each point in the original image to produce a new pixel in the filtered image.}
    \label{fig:spatial_filter_concept_fig}
\end{figure}

In general filters can be linear or nonlinear, but many of the most fundamental filters are linear and can be expressed mathematically as:
\begin{equation}
  \label{eq:correlation}
    I'(x,y) = F \circ I = \sum_{i=-N}^N \sum_{j=-M}^M F(i,j)I(x+i,y+j),
\end{equation}
where $N$ and $M$ are integers that define the width and height of a rectangular neighborhood $S_{xy}$. Based on the size of this neighborhood, it is said that this filter is of size $(2N+1) \times (2M+1)$. Additionally, the filter operation $F$ is usually called a \textit{mask} or \textit{kernel}. Broadly speaking, filters expressed by \eqref{eq:correlation} are referred to as \textit{correlation filters}.

Another type of linear filters that are commonly used are referred to as \textit{convolution} filters. Convolution filters are similar to correlation filters but use reverse image indices (in fact correlation and convolution filters are identical when the filter mask is symmetric in both the horizontal and vertical directions). In particular, these filters are expressed mathematically as:
\begin{equation}
  \label{eq:convolution}
    I'(x,y) = F \ast I = \sum_{i=-N}^N \sum_{j=-M}^M F(i,j)I(x-i,y-j).
\end{equation}
Convolution filters are associative, meaning that for two different filter masks $F$ and $G$ it is true that $F*(G*I) = (F*G)*I$. One example of how the associative property is useful is for smoothing an image \textit{before} taking applying a differentiation filter. Suppose the mask $F$ implemented a derivative filter and $G$ implemented a smoothing filter, then sequentially applying these filters would result in $F*(G*I)$. However, because of the associative property the masks can be convolved together \textit{first} such that only one filter needs to be applied to the image (i.e. $(F*G)*I$).

Note that in both the correlation and convolution filters the boundaries of the image need some special care because of the width and height of the mask. For example, Figure \ref{fig:nopaddingexample} shows how the filtered image is smaller than the original due to the width and height of the mask. Some possible options to handle this include padding the image, cropping it, extending it, or wrapping it. However, as images are generally quite large the exact approach likely won't vary the final result significantly.
\begin{figure}[ht]
  \centering
  \includegraphics[width=.65\textwidth]{tex/figs/ch10_figs/centerfilter_nopadding.png}
    \caption{Due to the width and height of the mask, the filtered image may be smaller than the original. However this can be fixed with several techniques, such as padding.}
    \label{fig:nopaddingexample}
\end{figure}

\begin{example}[Practical Considerations for Image Filtering] \label{ex:padding}
\theoremstyle{definition}
Implementation of correlation and convolution filters typically leverages some additional ``tricks'' to make things easier to implement. In this example two such tricks will be introduced: zero-padding and a change in indexing.

First, to more simply accommodate varying sizes of filters (including even and odd sized filters) the indexing is often changed such that the coordinate of interest is associated with the top-left element in the window rather than the center. In particular, for a correlation filter this would correspond to:
\begin{equation}
  \label{eq:correlation_newindex}
    I'(x,y) = F \circ I = \sum_{i=1}^K \sum_{j=1}^L F(x,y)I(x+i-1,y+j-1),
\end{equation}
where $K$ and $L$ are integers that define the width and height of the filter and the pixel $(x,y)$ is at row $x$ and column $y$. However, note that with this formulation the output image $I'$ will be shifted up and to the left. To see this consider the pixel at $x=1$ and $y=1$ in the new image $I'$, which would correspond to the top-left pixel $I'$. This new pixel value is generated by applying the filter $F$ over the pixels in the original image $I$ at rows $\{1,\dots,K\}$ and columns $\{1,\dots,L\}$ (which is not centered at $(1,1)$ in the original image $I$). Therefore it will appear as if the image has been shifted! But in practice this isn't an issue as long as you always index with respect to the top-left corner. An example of top-left indexing is shown in Figure \ref{fig:topleftfilter}
\begin{figure}[ht]
  \centering
  \includegraphics[width=.65\textwidth]{tex/figs/ch10_figs/topleftfilter_nopadding.png}
    \caption{Top-left indexing is typically easier to implement than center indexing. Notice that when top-left indexing, it appears as if the filtered image has shifted with respect to when center indexing is used.}
    \label{fig:topleftfilter}
\end{figure}

Zero-padding (also commonly referred to as \textit{same padding}) is another simple trick that can be used to ensure that the output filtered image $I'$ has the same dimension as the input image $I$. In this approach the left and right boundaries of the image are \textit{each} padded by $\lfloor K/2 \rfloor$ columns of zeros, and the top and bottom boundaries are padded by $\lfloor L/2 \rfloor$ rows of zeros ($\lfloor \cdot \rfloor$ denotes the ``floor'' operation). For example the image:
\begin{equation*}
I = \begin{bmatrix}
    1 & 2 & 3 \\
    4 & 5 & 6 \\
    7 & 8 & 9 \\
    \end{bmatrix},
\end{equation*}
would become
\begin{equation*}
I_\text{padded} = \begin{bmatrix}
    0 & 0 & 0 & 0 & 0 \\
    0 & 1 & 2 & 3 & 0 \\
    0 & 4 & 5 & 6 & 0 \\
    0 & 7 & 8 & 9 & 0 \\
    0 & 0 & 0 & 0 & 0 \\
    \end{bmatrix},
\end{equation*}
for filters $F \in \R^{3\times 3}$, $F \in \R^{2 \times 2}$, $F \in \R^{2 \times 3}$ and $F \in \R^{3 \times 2}$. When using this padding rule with the correlation filter \eqref{eq:correlation_newindex} and a filter $F$ with $K = 2,3$ and $L= 2,3$, the new image $I'$ can be defined for values $x \in \{1,2,3\}$ and $y \in \{1,2,3\}$, resulting in $I'$ being the same dimension as the original image $I$. The use of padding (along with top-left indexing) is also shown graphically in Figure \ref{fig:paddingfilter}
\begin{figure}[ht]
  \centering
  \includegraphics[width=.7\textwidth]{tex/figs/ch10_figs/topleftfilter_padding.png}
    \caption{Image padding is a commonly used technique to ensure that the size of the filtered image is the same size as the original.}
    \label{fig:paddingfilter}
\end{figure}
\end{example}

\subsubsection{Moving Average Filter}
The moving average filter returns the average of pixels in the mask, which achieves a smoothing effect (i.e. removes sharp features in the image). For example, a moving average filter with a normalized $3 \times 3$ mask is defined with the operation $F$ in \eqref{eq:correlation} chosen as:
\begin{equation*}
    F = \frac{1}{9}\begin{bmatrix}
    1 & 1 & 1 \\
    1 & 1 & 1 \\
    1 & 1 & 1 \\
    \end{bmatrix}.
\end{equation*}
Note that due to symmetry of the mask, the correlation \eqref{eq:correlation} and convolution \eqref{eq:convolution} filters will be identical. Additionally, the normalization is used to maintain the overall brightness of the image.

\subsubsection{Gaussian Smoothing Filter}
Gaussian smoothing filters are similar to the moving average filer, but instead of weighting all of the pixels evenly they are weighted by the Gaussian function:
\begin{equation*}
G_\sigma(x,y) = \frac{1}{2\pi\sigma^2} \exp \bigg(-\frac{x^2 + y^2}{2\sigma^2} \bigg).
\end{equation*}
This function is used to obtain the mask operation $F$ by sampling the function about the center pixel (i.e. for the center pixel with $i=j=0$ in \eqref{eq:correlation}, sample $G_\sigma(0,0)$). For example, for a normalized $3\times3$ mask with $\sigma$ = 0.85 this filter is approximately defined by:
\begin{equation*}
F = \frac{1}{16}
\begin{bmatrix}
1 & 2 & 1\\
2 & 4 & 2\\
1 & 2 & 1
\end{bmatrix}.
\end{equation*}
Like the moving average filter, this filter mask is symmetric and therefore yields identical results with respect to the correlation \eqref{eq:correlation} or convolution \eqref{eq:convolution} filters. The advantage of the Gaussian filter is that it provides more weight to the neighboring pixels that are closer.  An example of this filter is shown in Figure \ref{fig:gaussianfilter}.
\begin{figure}[ht]
  \centering
  \includegraphics[width=.8\textwidth]{tex/figs/ch10_figs/gaussianfilter.png}
    \caption{Example of a Gaussian smoothing filter, which produces a smoothing (blurring) effect on the filtered image.}
    \label{fig:gaussianfilter}
\end{figure}

\subsubsection{Separable Masks}
A mask $F$ is called \textit{separable} if it can be broken down into the convolution of two kernels $F = F_1 \ast F_2$. If a mask is separable into ``smaller'' masks, then it is often cheaper to apply $F_1$ followed by $F_2$, rather than by $F$ directly. One special case of this is when the mask can be represented as an outer product of two vectors (meaning it is equivalent to the 2D convolution of those two vectors). If the mask is of shape $M\times M$, and the input image has size $w\times h$, then the computational complexity of directly performing the convolution is $O(M^2wh)$. However, by separating the masks the computational cost is $O(2Mwh)$, which is linear in $M$ rather than quadratic. As an example, consider the moving average filter mask from before:
\begin{equation*}
F = \frac{1}{9}
\begin{bmatrix}
1 & 1 & 1\\
1 & 1 & 1\\
1 & 1 & 1
\end{bmatrix} = \frac{1}{9}
\begin{bmatrix}
1 \\
1 \\
1
\end{bmatrix}
\begin{bmatrix}
1 & 1 & 1\\
\end{bmatrix}.
\end{equation*}
As another example, note that the Gaussian smoothing filter mask is also separable. To see why this is, note that the Gaussian weighting function can be decomposed as:
\begin{equation*}
\begin{split}
G_\sigma(x,y) &= \frac{1}{2\pi\sigma^2} \exp \bigg(-\frac{x^2 + y^2}{2\sigma^2} \bigg), \\
    &= \frac{1}{\sqrt{2\pi}\sigma}\exp \bigg(-\frac{x^2}{2\sigma^2}\bigg)\frac{1}{\sqrt{2\pi}\sigma} \exp \bigg(-\frac{y^2}{2\sigma^2}\bigg), \\
    &= g_\sigma(x) \cdot g_\sigma(y).
\end{split}
\end{equation*}

\subsubsection{Image Differentiation Filters}
Taking the derivative of an image can be used to identify certain features, such as edges. On a basic level, the derivative of an image quantifies changes in pixel intensity in both the vertical and horizontal direction. However, since images are represented as functions defined over a discrete domain the traditional method for differentiating continuous functions can not be used. Instead it is more common to just compute differences between pixels, such as using a central difference method:
\begin{equation} \label{eq:cendiff}
\begin{split}
 \frac{\partial I} {\partial x} &= \frac{I(x+1,y) - I(x-1,y)}{2},\\
\frac{\partial I} {\partial y} &= \frac{I(x,y+1) - I(x,y-1)}{2}.
\end{split}
\end{equation}
where $\partial I/\partial x$ is the derivative in the horizontal direction and $\partial I/\partial y$ is the derivative in the vertical direction. It is of course also possible to define the derivatives using just one side, for example $\frac{\partial I}{\partial x} = I(x+1,y) - I(x,y)$.

It is also possible to differentiate an image using convolution filters. In particular, one common approach is to use a convolution filter \eqref{eq:convolution} defined with a mask $F$ called a \textit{Sobel mask} (also referred to as simply a Sobel operator). For the $x$ direction this mask is denoted as $S_x$ and for the $y$ direction as $S_y$:
\begin{equation}
S_x = \begin{bmatrix}
1 & 0 & -1\\
2 & 0 & -2\\
1 & 0 & -1
\end{bmatrix}, \quad S_y =
\begin{bmatrix}
1 & 2 & 1\\
0 & 0 & 0\\
-1 & -2 & -1
\end{bmatrix}
\end{equation}
Sobel masks are similar to the central difference method but use more neighboring pixels when calculating the derivative (i.e. they also consider the rows above and below to compute the difference). Note that Sobel masks are also separable.

\subsubsection{Similarity Measures}
Filtering can also be used to find similar features in different images, which can be useful for solving the correspondence problem in stereo vision or structure-from-motion techniques.
In particular, the similarity between the pixel $(x,y)$ in image $I_1$ and pixel $(x', y')$ in image $I_2$ can be computed by:
\begin{equation} \label{eq:similarity}
\begin{split}
SAD &= \sum_{i=-N}^N \sum_{j=-M}^M \rvert I_1(x+i,y+j)-I_2(x^\prime+i,y^\prime+j)\lvert, \\
SSD &= \sum_{i=-N}^N \sum_{j=-M}^M [I_1(x+i,y+j)-I_2(x^\prime+i,y^\prime+j)]^2,
\end{split}
\end{equation}
where SAD is an acronym for ``sum of absolute differences'', SSD is an acronym for ``sum of squared differences'', and $N$ and $M$ define the size of the window around the pixels that is considered.

\subsection{Image Feature Detection}
A local feature (also sometimes referred to as interest points, interest regions, or keypoints) in an image is a pattern that differs from its immediate neighborhood in terms of intensity, color, or texture.
Local features can generally be categorized in several ways, for example whether they provide semantic content or not. For example, features that may provide semantic content include edges or other geometric shapes (e.g. lanes of a road or blobs corresponding to blood cells in medical images). These types of features were some of the first for which feature detectors were proposed in the image processing literature. Features that do not provide semantic content may also be useful, for example in feature tracking, camera calibration, 3D reconstruction, image mosaicing, and panorama stitching. In these cases it may be more important that the feature be able to be located accurately and robustly over time. A third category of features are those that may not have semantic interpretations individually, but may have meaning as a collection.
For instance, a scene could be recognized by counting the number of feature matches between the observed scene and a query image. In this case only the number of matches is relevant and not the location or type of feature. Applications where these types of features are important include texture analysis, scene classification, video mining, and
image retrieval.

In this section several feature detection strategies will be discussed. While many strategies exist for different types of features, the focus here will be on two common features that are often useful in robotics: edges and corners.

\subsubsection{Edge Detection}
An \textit{edge} in an image is a region where there is a significant change in intensity values along one direction, and negligible change along the orthogonal direction. In one dimension an edge corresponds to a point where there is a sharp change in intensity, which mathematically can be thought of as a point of a function having a large first derivative and a small second derivative. Many edge detectors rely on this concept by differentiating images and looking for spikes in the derivative.
An edge detector can be evaluated based on several criteria for robustness and performance, including accuracy, localization, and single response. Good accuracy implies few false positives or negatives (missed edges), good localization implies that the detected edge should be exactly where the true edge is in the image, and a single response implies \textit{only} one edge is detected for each real edge. In practice, noise and discretization can make edge detection challenging.

Most edge detection methods rely on two key steps: smoothing and differentiation. Differentiation is performed in both the vertical and horizontal directions to find locations in the image with high intensity gradients. However, differentiation alone is vulnerable to false positives due to image noise, which is why many algorithms will first smooth the image.

\paragraph{Edge Detection in 1D:}
An example of how noise can corrupt image differentiation is given in Figure \ref{fig:noisy}.
\begin{figure}[ht]
  \centering
  \includegraphics[width=0.5\textwidth]{tex/figs/ch10_figs/edge_detection.png}
    \caption{Differentiation of signal (e.g. for edge detection) with noise can be particularly challenging, which can be addressed by first smoothing the signal.}
    \label{fig:noisy}
\end{figure}
Notice that in this case it is impossible to identify the jump in the signal due to the noise levels.
Smoothing filters, such as the Gaussian smoothing filter discussed earlier, can help remedy this problem. In particular, suppose the original signal in Figure \ref{fig:noisy} is defined by $I(x)$. Then a smoothed version can be defined by applying a smoothing convolution filter:
\begin{equation*}
s(x) = g_\sigma(x) \ast I(x),
\end{equation*}
where $g_\sigma(x)$ represents a Gaussian smoothing filter, and then by applying the differentiation filter:
\begin{equation*}
s'(x)=\frac{d}{dx}\ast s(x).
\end{equation*}
This process is shown in Figure \ref{fig:gauss}.
\begin{figure}[ht!]
  \centering
  \includegraphics[width=0.55\textwidth]{tex/figs/ch10_figs/edge_detection_smooth.png}
    \caption{Edge detection through convolution with a Gaussian smoothing filter, followed by a differentiation filter.}
    \label{fig:gauss}
\end{figure}
Note however that since these filters are convolutions, the associativity property can be leveraged to actually combine them into a single filter:
\begin{equation*}
s'=(\frac{d}{dx} * g_\sigma) * I.
\end{equation*}

\paragraph{Edge Detection in 2D:}
Edge detection in a two-dimensional image is quite similar to the example previously discussed in 1D. Let the smoothing filter be the Gaussian smoothing filter from before, and a differentiation filter such as the Sobel filter. The gradient of the smoothed image in both the $x$ and $y$ directions can be written as:
\begin{equation*}
\nabla S= \begin{bmatrix}
\frac{\partial}{\partial x} * G_\sigma * I \\ \frac{\partial}{\partial y} * G_\sigma * I \end{bmatrix}= \begin{bmatrix}
G_{\sigma,x} * I\\G_{\sigma,y} * I
\end{bmatrix}=\begin{bmatrix}
S_x\\S_y
\end{bmatrix},
\end{equation*}
where $I$ is the original image and the associativity properties of the smoothing and differentiation convolution filters is used to define the combined filters $G_{\sigma,x}$ and $G_{\sigma,y}$. The magnitude of the gradient can then be computed by:
\begin{equation*}
\lvert\nabla S\rvert =\sqrt{S_x^2+S_y^2},
\end{equation*}
which can be used to check against a predefined threshold value for edge detection. To guarantee thin edges it is also possible to filter out points whose gradient magnitude are above the threshold but are not local maxima. Examples of this process are shown in Figures \ref{fig:sobel} and \ref{fig:canny}.
\begin{figure}[ht]
  \centering
  \includegraphics[width=.8\textwidth]{tex/figs/ch10_figs/sobel.png}
    \caption{Edge detection using the ``Sobel'' edge detector.}
    \label{fig:sobel}
\end{figure}
\begin{figure}[ht]
  \centering
  \includegraphics[width=.8\textwidth]{tex/figs/ch10_figs/canny.png}
    \caption{Edge detection using the ``Canny'' edge detector.}
    \label{fig:canny}
\end{figure}

\subsubsection{Corner Detection}
A \textit{corner} in an image is defined as an intersection of two or more edges, and also sometimes as a point where there is a large intensity variation in every direction.
Important properties of corner detectors include repeatability and distinctiveness. Repeatability quantifies how well the same features can be found in multiple images even under geometric and photometric transformations. Distinctiveness refers to whether the information carried by the patch surrounding the feature is distinctive, which can be used to reliably produce correspondences. Both of these properties are particularly important in applications such as panorama stitching and 3D reconstruction.

Generally corner detection can be thought of in a similar way to edge detection, except that instead of looking for change along one direction there should be changes in all directions. One well known corner detector is known as the Harris detector \cite{Harris1988}, which has the useful property that the detection is invariant to rotations and linear intensity changes (i.e. geometric and photometric invariance). However the Harris detector is not invariant to scale changes or geometric affine changes, which has led to the development of scale-invariant detectors such as the Harris-Laplacian detector or the scale-invariant feature transform (SIFT) detector.

\subsection{Image Descriptors}
Image \textit{descriptors} describe features so that they can be compared across images, or used for object detection and matching. Similar to image detectors, it is also desirable for image descriptors to be repeatable (i.e. invariant with respect to pose, scale, illumination, etc.) and distinct. Perhaps the simplest example of a descriptor is an $n\times m$ window of pixel intensities centered at the feature, which can be normalized to be illumination invariant. However, such a descriptor is not invariant to pose or scale and is not distinctive, and therefore is generally not useful in practice.
Alternative detectors/descriptors that have become popular include SIFT, SURF, FAST, BRIEF, ORB, and BRISK.

\subsection{Exercises}
\subsubsection{Linear Filtering}
Complete \textit{Problem 3: Linear Filtering} located in the online repository:

\vspace{\baselineskip}

\url{https://github.com/PrinciplesofRobotAutonomy/AA274A_HW3},

\vspace{\baselineskip}

where you will explore the use of linear filters for image processing.


\section*{Ex.34.2-5}
\subsection*{Show that any language in NP can be decided by an algorithm running in time $2^{O(n^k)}$ for some constant $k$}

First note, that if we can verify a language, then we can also decide it.
\\
Now, we know that we can verify any language L $\in$ NP in polynomial time (top of page 1064, CLRS), with a two-input algorithm. As input, the algorithm takes $x\in L$ and a certificate $y$. Since the length of any certificate is bounded by $O(|x|^c)$, the input $x$ can at most be $O(|x|^c)$ long.
\\
If we run the algorithm on every possible input $x\in {0,1}^*$, with that max length, it will take time $2^{O(n^k)}$.


\documentclass[class=report, float=false, crop=false]{standalone}
\usepackage[subpreambles=true]{standalone}

\input{preamble}

\graphicspath{{figures/images/}}

% \begin{cbunit}

\begin{document}

\chapter{Ellipsoids}
\label{appendix:ellipsoids}

\section{Definition}

An ellipsoid is a surface that may be obtained from a sphere by deforming it by means of directional scalings, or more generally, of an affine transformation \cite{wiki:Ellipsoid}.\\

Within a Cartesian coordiante system in which the origin is the center of the ellipsoid and the coordinate axes are axes of the ellipsoid, a vector \(\vec{r} = \begin{pmatrix} x \\ y \\ z \end{pmatrix} \in \mathbb{R}^3\) belongs to the surface of the ellispoid if and only if
\begin{equation}
\frac{x^2}{a^2} + \frac{y^2}{b^2} + \frac{z^2}{c^2} = 1
\label{ellipsoid_cartesian}
\end{equation}
where $a$, $b$ and $c \in \mathbb{R}$ are the the semi-axes of the ellipsoid.\\

Equivalently to equation \ref{ellipsoid_cartesian}, an ellipsoid $\mathcal{A}$ can be defined by a positive definite matrix $A \in \mathcal{M}_3(\mathbb{R})$ and a vector $\vec{v} \in \mathbb{R}^3$ such that
\begin{equation}
\boxed{\forall \vec{r} \in \mathbb{R}^3, \vec{r} \in \bar{\mathcal{A}} \Leftrightarrow (\vec{r}-\vec{v})^TA(\vec{r}-\vec{v}) = 1}
\label{ellipsoid_matrix}
\end{equation}
The eigenvectors of $A$ then define the principal axes of $\mathcal{A}$ and the associated eigenvalues are the reciprocals of the squares of the semi-axes. The vector $\vec{v}$ defines the center of the ellipsoid.

\section{Homogeneous coordinates}

\subsection{Quick reminder}

A vector \(\vec{r} = \begin{pmatrix} x \\ y \\ z \end{pmatrix} \in \mathbb{R}^3\) is equivalent to its homogeneous form \(\begin{pmatrix} x \\ y \\ z \\ 1 \end{pmatrix} \in E^3\) where $E^3$ is the Euclidean 3D projective space.\\

Moreover, we have that in $E^3$, $\forall \lambda \in \mathbb{R}^*$ and $\forall (x,y,z) \in \mathbb{R}^3$, \(\begin{pmatrix} x \\ y \\ z \\ 1 \end{pmatrix}\) and \(\begin{pmatrix} \frac{x}{\lambda} \\ \frac{y}{\lambda} \\ \frac{z}{\lambda} \\ 1 \end{pmatrix}\) represent the exact same point. Therefore, we will always assume that the last coordinate of our vectors in $E^3$ is 1.\\

From the property above, you naturally have that any point in $E^3$ whose last coordinate would be $0$ is then infinitely far from the origin \(\begin{pmatrix} 0 \\ 0 \\ 0 \\ 1 \end{pmatrix}\).

\subsection{Useful affine transformations}

\subsubsection{Translation}

We define in $\mathbb{R}^3$ the translation of vector $\vec{u} \in \mathbb{R}^3$ as
\begin{align*}
\mathcal{T}_{\vec{u}} \colon &\mathbb{R}^3 \to \mathbb{R}^3\\     &\phantomarrow{\mathbb{R}^3}{\vec{v}} \vec{v} + \vec{u}
\end{align*}

In $E^3$, this function $\mathcal{T}_{\vec{u}}$ is represented by the matrix $T_{\vec{u}}$:
\begin{equation}
\boxed{T_{\vec{u}} = \begin{pmatrix} \mathbbm{1}_3 & \vec{u} \\ \vec{0}^T & 1 \end{pmatrix}}
\label{translation_matrix}
\end{equation}

\subsubsection{Dilatation}

We define in $\mathbb{R}^3$ the dilatation of factor $a \in \mathbb{R}$ as
\begin{align*}
\mathcal{X}_{a} \colon &\mathbb{R}^3 \to \mathbb{R}^3\\     &\phantomarrow{\mathbb{R}^3}{\vec{v}} a\vec{v}
\end{align*}

In $E^3$, this function $\mathcal{X}_{a}$ is represented by the matrix $X_{a}$:
\begin{equation}
\boxed{X_{a} = \begin{pmatrix} a\mathbbm{1}_3 & \vec{0} \\ \vec{0}^T & 1 \end{pmatrix}}
\label{dilatation_matrix}
\end{equation}

\subsubsection{Rotation}

We showed in part \ref{action_rotation} that every 3D rotation can be expressed as the action of some unit quaternion.\\

The product of quaternions being bilinear, we can associate the action of a quaternion to a linear function in a vectorial space and therefore to a matrix \cite{shoemake}.\\

Consider $q = [\vec{q},q_0],p = [\vec{p},p_0] \in \mathbb{H}$. According to part \ref{quat_properties}, we have $qp = [\underbrace{\vec{q}\times\vec{p}}_{Ap} + \underbrace{q_0\vec{p}}_{Bp} + \underbrace{p_0\vec{q}}_{Cp},\underbrace{q_0p_0}_{Dp}~\underbrace{-\vec{q}\cdot\vec{p}}_{Ep}]$, with
\begin{align*}
A = \begin{pmatrix} \vec{q} \times \bigcdot & \vec{0} \\ \vec{0}^T & 0 \end{pmatrix}, B = \begin{pmatrix} q_0 \mathbbm{1}_3 & \vec{0} \\ \vec{0}^T & 0 \end{pmatrix}, C = \begin{pmatrix} 0_3 & \vec{q} \\ \vec{0}^T & 0 \end{pmatrix}, D = \begin{pmatrix} 0_3 & \vec{0} \\ \vec{0}^T & q_0 \end{pmatrix}, E = \begin{pmatrix} 0_3 & \vec{0} \\ -\vec{q}^T & 0 \end{pmatrix}
\end{align*}
Therefore the left multiplication by $q$ in $\mathbb{H}$ can be represented by the matrix
\begin{align*}
L_q &= A + B + C + D + E\\
&= \begin{pmatrix} \vec{q} \times \bigcdot + q_0\mathbbm{1}_3 & \vec{q} \\ -\vec{q}^T & q_0 \end{pmatrix}
\end{align*}

Similarly, we have $pq^* = [\underbrace{\vec{q}\times\vec{p}}_{Ap} + \underbrace{q_0\vec{p}}_{Bp}~\underbrace{-p_0\vec{q}}_{-Cp},\underbrace{q_0p_0}_{Dp} + \underbrace{\vec{q}\cdot\vec{p}}_{-Ep}]$, therefore the right multiplication by $q^*$ can be represented by the matrix
\begin{align*}
R_{q^*} &= A + B - C + D - E\\
&= \begin{pmatrix} \vec{q} \times \bigcdot + q_0\mathbbm{1}_3 & -\vec{q} \\ \vec{q}^T & q_0 \end{pmatrix}
\end{align*}

Consequently, the matrix representing the action of the unit quaternion $q$ in $\mathbb{H}$ is
\begin{align*}
Q_q = L_qR_{q^*} = R_{q^*}L_q = \begin{pmatrix} \mathcal{Q}_q & \vec{0} \\ \vec{0}^T & \underbrace{N^2(q)}_{=1} \end{pmatrix}
\end{align*}
where
\begin{equation}
\boxed{\mathcal{Q}_q = (\vec{q}\times\bigcdot + q_0\mathbbm{1}_3)^2 + \vec{q}\vec{q}^T}
\label{rotation_matrix_R3}
\end{equation}
is the matrix representing the rotation associated to the unit quaternion $q$ in $\mathbb{R}^3$.\\

Therefore, the matrix reprensenting the action of $q$ in $\mathbb{H}$ and the rotation associated to $q$ in $E^3$ are the same and we will note the latter
\begin{equation}
\boxed{Q_q = \begin{pmatrix} \mathcal{Q}_q & \vec{0} \\ \vec{0}^T & 1 \end{pmatrix}}
\label{rotation_matrix}\\
\end{equation}

We can develop equation \ref{rotation_matrix_R3} to get an expression of the rotation matrix $\mathcal{Q}_q$ in $E^3$. With
\begin{align*}
  \left(\vec{e}_1 \equiv (1, 0, 0)^T, \vec{e}_2 \equiv (0, 1, 0)^T, \vec{e}_3 \equiv (0, 0, 1)^T\right)
\end{align*}
the canonical basis of $E^3$, we have
\begin{align*}
  \vec{q} \times \vec{e}_1 = \begin{pmatrix} 0 \\ q_3 \\ -q_2 \end{pmatrix}, \vec{q} \times \vec{e}_2 = \begin{pmatrix} -q_3 \\ 0 \\ q_1 \end{pmatrix}, \vec{q} \times \vec{e}_3 = \begin{pmatrix} q_2 \\ -q_1 \\ 0 \end{pmatrix}
\end{align*}
such that
\begin{align*}
  \vec{q} \times \bigcdot = \begin{pmatrix} 0 & -q_3 & q_2 \\ q_3 & 0 & -q_1 \\ -q_2 & q_1 & 0 \end{pmatrix} \text{ and } \vec{q} \times \bigcdot + q_0 \mathbbm{1}_3 = \begin{pmatrix} q_0 & -q_3 & q_2 \\ q_3 & q_0 & -q_1 \\ -q_2 & q_1 & q_0 \end{pmatrix}
\end{align*}
and then
\begin{align*}
  (\vec{q} \times \bigcdot + q_0 \mathbbm{1}_3)^2 =
  \begin{pmatrix}
    q_0^2 - q_3^2 - q_2^2 & q_1 q_2 - 2 q_0 q_3 & q_1 q_3 + 2 q_0 q_2 \\
    q_1 q_2 + 2 q_0 q_3 & q_0^2 - q_3^2 - q_1^2 & q_2 q_3 - 2 q_0 q_1 \\
    q_1 q_3 - 2 q_0 q_2 & q_2 q_3 + 2 q_0 q_1 & q_0^2 - q_1^2 - q_2^2
  \end{pmatrix}
\end{align*}
thus leading to
\begin{align*}
  \mathcal{Q}_q = (\vec{q} \times \bigcdot + q_0 \mathbbm{1}_3)^2 + \vec{q} \vec{q}^T =
  \begin{pmatrix}
    q_0^2 + q_1^2 - q_3^2 - q_2^2 & 2 q_1 q_2 - 2 q_0 q_3 & 2 q_1 q_3 + 2 q_0 q_2 \\
    2 q_1 q_2 + 2 q_0 q_3 & q_0^2 + q_2^2 - q_3^2 - q_1^2 & 2 q_2 q_3 - 2 q_0 q_1 \\
    2 q_1 q_3 - 2 q_0 q_2 & 2 q_2 q_3 + 2 q_0 q_1 & q_0^2 + q_3^2 - q_1^2 - q_2^2
  \end{pmatrix}
\end{align*}
where we can note that $q$ is an unit quaternion, \textit{i.e.} $N^2(q) = \sum_{i=0}^4 q_i^2 = 1$, and therefore
\begin{equation}
  \boxed{\mathcal{Q}_q =
    \begin{pmatrix}
      1 - 2 (q_2^2 + q_3^2) & 2 (q_1 q_2 - q_0 q_3) & 2 (q_1 q_3 + q_0 q_2) \\
      2 (q_1 q_2 + q_0 q_3) & 1 - 2 (q_1^2 + q_3^2) & 2 (q_2 q_3 - q_0 q_1) \\
      2 (q_1 q_3 - q_0 q_2) & 2 (q_2 q_3 + q_0 q_1) & 1 - 2 (q_1^2 + q_2^2)
    \end{pmatrix}
  }
\label{rotation_matrix_R3_expression}
\end{equation}
in accordance with \cite{wiki:quaternions}.

\section{Belonging matrix}

\subsection{Definition}

We want to find for any ellipsoid $\mathcal{A}$ a matrix $B$ acting on homogeneous coordinates with the following properties
\begin{equation}
\forall \vec{r} \in E^3, \begin{cases} \vec{r}^T B \vec{r} < 0 &\text{ if } \vec{r} \in \mathcal{A} \setminus \bar{\mathcal{A}} \\ \vec{r}^T B \vec{r} = 0 &\text{ if } \vec{r} \in \bar{\mathcal{A}} \\ \vec{r}^T B \vec{r} > 0 &\text{ if } \vec{r} \notin \mathcal{A} \end{cases}
\label{belonging_definition}
\end{equation}
which we will call the \textit{belonging matrix} of $\mathcal{A}$.

\subsection{Expression}
\label{belonging_exp}

We have seen with equation \ref{ellipsoid_matrix} that the belonging to the surface of an ellispoid could be expressed with a matrix whose eigenvectors define the principal axes of the ellipsoid and whose eigenvalues define the reciprocals of the squares of the semi-axes. Therefore, if we denote $(R_i)_{i=1:3}$ the semi-axes of the ellipsoid, this matrix can be written as $A = P^{-1}\text{diag}(R_1^{-2},R_2^{-2},R_3^{-2})P$.\\

The semi-axes of an ellipsoid define an orthogonal base of $\mathbb{R}^3$. Therefore P is the transition matrix from the orthogonal base formed by the principal axes of the ellipsoid to the original Euclidean base $(\vec{e_i})_{i=1:3}$, and inversely for $P^{-1}$.\\

Consequently, we have that $\forall i \in \llbracket1,3\rrbracket, \vec{v} + R_iP^{-1}\vec{e_i} \in \bar{\mathcal{A}}$ and so
\begin{align*}
\forall i \in \llbracket1,3\rrbracket,&(P^{-1}R_i\vec{e_i})^TA(P^{-1}R_i\vec{e_i}) = 1\\ \Leftrightarrow&(P^{-1}R_i\vec{e_i})^TP^{-1}\text{diag}(R_j^{-2})_{j=1:3}P(P^{-1}R_i\vec{e_i}) = 1\\
\Leftrightarrow &R_i\vec{e_i}^T(P^{-1})^TP^{-1}\text{diag}(R_j^{-2})_{j=1:3}\underbrace{PP^{-1}}_{\mathbbm{1}_3}R_i\vec{e_i} = 1\\
\Leftrightarrow &\vec{e_i}^T(P^{-1})^TP^{-1}\underbrace{\text{diag}(R_j^{-2})_{j=1:3}R_i^2\vec{e_i}}_{\vec{e_i}} = 1\\
\Leftrightarrow &\vec{e_i}^T(P^{-1})^TP^{-1}\vec{e_i} = 1
\end{align*}
Thus, the norm of every column vector of $P^{-1}$ equals to 1. Therefore, $P^{-1}$, and equivalently $P$, is an orthonormal matrix.\\

If $P$ is an orthonormal matrix, then the principal axes of the ellipsoids are derived from the original Euclidean base through rotations and permutations of axis. Without loss of generality, we can consider that there are no permutations of axis, leading to $P = \mathcal{Q}_q^{-1} = \mathcal{Q}_q^T$ with $q$ the quaternion describing the orientation of the ellipsoid. Equation \ref{ellipsoid_matrix} thus becomes:
\begin{align*}
\forall \vec{r} \in \mathbb{R}^3, \vec{r} \in \bar{\mathcal{A}} \Leftrightarrow (\vec{r}-\vec{v})^T\mathcal{Q}_q\text{diag}(R_i^{-2})_{i=1:3}\mathcal{Q}_q^T(\vec{r}-\vec{v}) = 1
\end{align*}

We can get rid of the $\vec{v}$ by using homogeneous coordinates. In $E^3$, $\vec{r} - \vec{v}$ becomes $T_{-\vec{v}}\vec{r}$ and, to conserve the scalar product, $\text{diag}(R_i^{-2})_{i=1:3}$ becomes $\text{diag}(R_i^{-2},0)_{i=1:3}$.\\

We always assume that the last coordinate of our vectors in $E^3$ is 1, therefore we can notice that $\forall \vec{u} \in E^3, \vec{u}^T\text{diag}(0,0,0,-1)\vec{u} = -1$. We can then rewrite equation \ref{ellipsoid_matrix}:
\begin{align*}
\forall \vec{r} \in E^3, \vec{r} \in \bar{\mathcal{A}} \Leftrightarrow &\vec{r}^TT_{-\vec{v}}^TQ_q\text{diag}(R_i^{-2},0)_{i=1:3}Q_q^TT_{-\vec{v}}\vec{r} = 1\\
\Leftrightarrow&\vec{r}^T\underbrace{T_{-\vec{v}}^TQ_q\text{diag}(R_i^{-2},-1)_{i=1:3}Q_q^TT_{-\vec{v}}}_{\mathcal{C}}\vec{r} = 0
\end{align*}
The matrix $\mathcal{C}$ is a good candidate for the belonging matrix of $\mathcal{A}$, we then have to understand how it works.\\

The $T_{-\vec{v}}$ translation and the $Q_q^T$ rotation bring back the principal axes of the ellipsoid to the original Euclidean base and origin. Without loss of generality, we can then consider that the principal axes of the ellipsoids are along $(\vec{e_1},\vec{e_2},\vec{e_3})$.\\

For a vector \(\vec{r} = \begin{pmatrix} x \\ y \\ z \end{pmatrix} \in \mathbb{R}^3\), we have
\begin{align*}
&\frac{x^2}{R_1^2} + \frac{y^2}{R_2^2} + \frac{z^2}{R_3^2} = \mu^2\\
\Leftrightarrow & \frac{x^2}{(\mu R_1)^2} + \frac{y^2}{(\mu R_2)^2} + \frac{z^2}{(\mu R_3)^2} = 1
\end{align*}
therefore, $\mu$ can be construed as the rescaling factor that has to be applied to the ellipsoid for $\vec{r}$ to be on its surface. Then, $\mu < 1$ if $\vec{r} \in \mathcal{A}\setminus\bar{\mathcal{A}}$, $\mu = 1$ if $\vec{r} \in \bar{\mathcal{A}}$, and $\mu > 1$ otherwise.\\

Since we have that $\forall \vec{r} \in E^3, \vec{r}^T\mathcal{C}\vec{r} = \mu^2 - 1$, $\mathcal{C}$ is the matrix we have been looking for, and we can define
\begin{equation}
\boxed{B(\vec{v},q,(R_i)_{i=1:3}) \equiv T_{-\vec{v}}^TQ_q\text{diag}(R_i^{-2},-1)_{i=1:3}Q_q^TT_{-\vec{v}}}
\label{belonging_matrix}
\end{equation}

One must notice that such a matrix is symmetric. We can also define the \textit{belonging function} of the ellipsoid $\mathcal{A}$
\begin{equation}
\boxed{\begin{aligned}\mathcal{F}_{\mathcal{A}} \colon &E^3 \to \mathbb{R}\\     &\phantomarrow{E^3}{\vec{r}} \vec{r}^TB(\vec{v},q,(R_i)_{i=1:3})\vec{r}\end{aligned}}
\label{belonging_function}
\end{equation}
such that $\forall \vec{r} \in E^3$, $\vec{r} \in \mathcal{A} \Leftrightarrow \mathcal{F}_{\mathcal{A}}(\vec{r}) \leq 0$ and
\begin{equation}
\boxed{\mathcal{F}_{\mathcal{A}}(\vec{r}) = \mu^2(\vec{r}) - 1}
\end{equation}
with $\mu^2(\vec{r})$ the rescaling factor that has to be applied to the ellipsoid for $\vec{r}$ to be on its surface.

\subsection{Reduced belonging matrix}

The equation \ref{belonging_matrix} is theoretically relevant, however it is not computationally efficient.\\

On the one side, resorting to homogeneous coordinates and expressing the translation with a translation matrix (equation \ref{translation_matrix}) leads to more operations than to express the rotation in $\mathbb{R}^3$ directly. On the other side, we have seen in part \ref{quaternions_interest} that the rotation of a vector was quicker performed when converting the quaternion associated to the rotation to a rotation matrix.\\

We then introduce, for an ellipsoid $\mathcal{A}$ whose centre is located in $\vec{v}$, whose orientation is described by the unit quaternion $q$ and whose semi-axes are $(R_i)_{i=1:3}$, the associate \textit{reduced belonging matrix}
\begin{equation}
\boxed{\bar{B}(q,(R_i)_{i=1:3}) \equiv \mathcal{Q}_q\text{diag}(R_i^{-2})_{i=1:3}\mathcal{Q}_q^T}
\label{reduced_belonging_matrix}
\end{equation}
with $\mathcal{Q}_q$ the $\mathbb{R}^3$ rotation matrix associated to $q$ (equation \ref{rotation_matrix_R3}). One must notice that the reduced belonging matrix is also symmetric. Furthermore, this reduced belonging matrix has the following properties
\begin{equation}
\forall \vec{r} \in \mathbb{R}^3, \begin{cases} (\vec{r} - \vec{v})^T \bar{B} (\vec{r} - \vec{v}) < 1 &\text{ if } \vec{r} \in \mathcal{A} \setminus \bar{\mathcal{A}} \\ (\vec{r} - \vec{v})^T \bar{B} (\vec{r} - \vec{v}) = 1 &\text{ if } \vec{r} \in \bar{\mathcal{A}} \\ (\vec{r} - \vec{v})^T \bar{B} (\vec{r} - \vec{v}) > 1 &\text{ if } \vec{r} \notin \mathcal{A} \end{cases}
\label{reduced_belonging_definition}
\end{equation}
according to equation \ref{belonging_definition}, which leads to the following expression for the belonging function:
\begin{equation}
\boxed{\begin{aligned}\mathcal{F}_{\mathcal{A}} \colon &\mathbb{R}^3 \to \mathbb{R}\\     &\phantomarrow{\mathbb{R}^3}{\vec{r}} (\vec{r} - \vec{v})^T\bar{B}(q,(R_i)_{i=1:3})(\vec{r}-\vec{v}) - 1\end{aligned}}
\label{belonging_function_reduced}
\end{equation}

\section{Unit normal vectors}

Since the surface of the ellipsoid -- the locus of points where $\mathcal{F}_{\mathcal{A}}(\vec{r}) = 0$ -- is an isosurface of the function $\mathcal{F}_{\mathcal{A}}$, we have that $\forall \vec{r_S} \in \bar{\mathcal{A}}, \vec{\nabla}\mathcal{F}_{\mathcal{A}}(\vec{r_S})$ is a non-unit outward-facing normal vector to the ellipsoid in $\vec{r_S}$.\\

Let us note $\forall \vec{r_S} \in \bar{\mathcal{A}}$, $\vec{n}(\vec{r_S})$ the unit outward-facing normal vector to the ellipsoid in $\vec{r_S}$. Then $\vec{n}(\vec{r_S}) // \vec{\nabla}\mathcal{F}_{\mathcal{A}}(\vec{r_S})$, we thus have to find the direction of this gradient.

\subsection{Expression with the belonging matrix}
\label{normal_vector_belonging}

We have that
\begin{align*}
\forall \vec{r} \in E^3, \mathcal{F}_{\mathcal{A}}(\vec{r} + \vec{dr}) &= (\vec{r} + \vec{dr})^TB(\vec{r} + \vec{dr})\\
&= \underbrace{\vec{r}^TB\vec{r}}_{\mathcal{F}_{\mathcal{A}}(\vec{r})} + \underbrace{\vec{r}^TB\vec{dr} + \vec{dr}^TB\vec{r}}_{2\vec{dr}^TB\vec{r}} + \vec{dr}^TB\vec{dr}
\end{align*}
therefore we can show that
\begin{align*}
\forall \vec{r} \in E^3, \forall i \in \llbracket1,3\rrbracket,~ &\mathcal{F}_{\mathcal{A}}(\vec{r} + dr_i\vec{e_i}) - \mathcal{F}_{\mathcal{A}}(\vec{r}) = 2dr_i\vec{e_i}^TB\vec{r} + dr_i^2\vec{e_i}B\vec{e_i}\\
\Rightarrow&\frac{\partial}{\partial e_i}\mathcal{F}_{\mathcal{A}}(\vec{r}) = \lim_{dr_i \to 0} \frac{\mathcal{F}_{\mathcal{A}}(\vec{r} + dr_i\vec{e_i}) - \mathcal{F}_{\mathcal{A}}(\vec{r})}{dr_i} = 2\vec{e_i}^TB\vec{r}\\
\Rightarrow&\vec{\nabla}\mathcal{F}_{\mathcal{A}}(\vec{r}) = \sum_{i=1}^3\left(\frac{\partial}{\partial e_i}\mathcal{F}_{\mathcal{A}}(\vec{r})\right)\vec{e_i} = 2\underbrace{\sum_{i=1}^3(\vec{e_i}^TB\vec{r})\vec{e_i}}_{B\vec{r}}
\end{align*}
\textit{i.e.} $\vec{\nabla}\mathcal{F}_{\mathcal{A}}(\vec{r}) = 2B\vec{r}$. Consequently, we finally have that
\begin{equation}
\boxed{\forall \vec{r_S}\in \bar{\mathcal{A}}, \vec{n}(\vec{r_S}) = \frac{B(\vec{v},q,(R_i)_{i=1:3})\vec{r_S}}{|B(\vec{v},q,(R_i)_{i=1:3})\vec{r_S}|}}
\end{equation}

\subsection{Expression with the reduced belonging matrix}

With the same notations while assuming that the centre of the ellipsoid is located in $\vec{v}$, we can notice that replacing $\vec{r}$ by $(\vec{r} - \vec{v})$ in the demonstration of part \ref{normal_vector_belonging} leads to the following expression
\begin{align*}
\forall \vec{r} \in E^3, \forall i \in \llbracket1,3\rrbracket,~ &\mathcal{F}_{\mathcal{A}}(\vec{r} + dr_i\vec{e_i}) - \mathcal{F}_{\mathcal{A}}(\vec{r}) = 2dr_i\vec{e_i}^T\bar{B}(\vec{r} - \vec{v}) + dr_i^2\vec{e_i}\bar{B}\vec{e_i}
\end{align*}
and thus, by analogy, the following result
\begin{equation}
\boxed{\forall \vec{r_S}\in \bar{\mathcal{A}}, \vec{n}(\vec{r_S}) = \frac{\bar{B}(q,(R_i)_{i=1:3})(\vec{r_S} - \vec{v})}{|\bar{B}(q,(R_i)_{i=1:3})(\vec{r_S} - \vec{v})|}}
\label{surface_vec_reduced}
\end{equation}

% \input{references/biblio}

\end{document}

% \end{cbunit}


%%%% 3D Skewness Example %%%%%%

\subsection{Dependence on Dimension}\label{ex:3dmap}
To further illustrate that relationship between skewness and accuracy holds as we move towards higher dimensions, we extend the numerical investigation to a three--dimensional parameter space.
Generally, we have fewer QoI than number of uncertain model parameters, so we assume that the potential QoI maps are defined by the $2\times 3$ matrices
\begin{equation}\label{eq:qmap3}
\qspace_S := \left \lbrace \qoi^{(s)} =  \mat{ccc}{1 & 0 & 0\\ \sqrt{s^2 - 1}& 1 & 0} \right \rbrace_{s\in S}.
\end{equation}
Here, as in the previous example, the index $s$ indicates the magnitude of skewness.
Furthermore, the results of Example~\ref{ex:rotation} justify the restriction of the maps to this form since any linear map of skewness $s$ is simply a rotation of maps of this form.

%Now, the generalized contours for inverses of maps from $\RR^3 \to \RR^2$ will be isomorphic to 2\--dimensional contour events in that the inverse sets will be columns in 3\-space orthogonal to the aforemntioned plane.
%
%IMAGE DEMONSTRATING THIS WOULD HELP.
%We define
%
%which is just the map from \eqref{eq:qmap2} appended with zeros in the third column.
%We make this choice solely for convience and are justified in doing so owing to Proposition~\ref{prop:rot_invariance} and the fact of generalized contours of maps from $\RR^3 \to \RR^2$ being parallel columns.
%The rotational invariance naturally extends to the third dimension.

%We note that $\bar{N}$ is much higher since we kept the convention of 200 grid cells per dimension in our reference.
%However, we kept the same number of random samples $N$, so we should expect higher errors due to the overresolved regular grid.
%Fortunately, we find that the results still generalize.
%We present the case where $M=1$:

\begin{figure}[h]
\begin{table}[H]
\begin{tabular}{ c | c | c | c }
\nsamps & $\qoiA$ & $\qoiB$ & $\qoiC$\\ \hline \hline
$200$ & $3.33E-01$ & $4.56E-01$ & $6.10E-01$\\ \hline

$400$ & $2.78E-01$ & $3.51E-01$ & $4.97E-01$\\ \hline

$800$ & $2.19E-01$ & $2.95E-01$ & $4.10E-01$\\ \hline

$1600$ & $1.72E-01$ & $2.37E-01$ & $3.35E-01$\\ \hline

$3200$ & $1.36E-01$ & $1.89E-01$ & $2.64E-01$\\ \hline

$6400$ & $1.09E-01$ & $1.47E-01$ & $2.09E-01$\\ \hline
\end{tabular}
\end{table}

\includegraphics[width=0.45\linewidth]{./images/Plot-reg_BigN_8000000_reg_M_1_rand_I_100000.png}

\caption{The results of $d^2_\text{TV}(\PP_{\pspace, \ndiscs, \nsamps}, \PP_{\pspace, \ndiscs, \bar{\nsamps}})$ for $\ndiscs = 1, \bar{\nsamps} = 8,000,000$, with $a, b, c = 1, 2, 4$ in three dimensions.}
\label{fig:M1_3d}
\end{figure}
\FloatBarrier
In Figure~\ref{fig:M1_3d}, it appears that the effect of skewness is even more pronounced in higher dimensions, and that the number of samples required to achieve similar levels of accuracy between two maps with a ratio of skewness 2 is now quadrupled.
The analysis of \cite{BGE+15} suggested a dependence of accuracy related to the skewness raised to a power related to the dimension of the data space.


\addcontentsline{toc}{section}{Exercise 2.1 Bayes rule for medical diagnosis}
\section*{Exercise 2.1 Bayes rule for medical diagnosis}
\import{sections/}{exercise 2-1.tex}
\newpage
\addcontentsline{toc}{section}{Exercise 2.2 Legal reasoning}
\section*{Exercise 2.2 Legal reasoning}
\import{sections/}{exercise 2-2.tex}
\newpage
\addcontentsline{toc}{section}{Exercise 2.3 Probabilities are sensitive to the form of the question that was used to generate the answer}
\section*{Exercise 2.3 Probabilities are sensitive to the form of the question that was used to generate the answer}
\import{sections/}{exercise 2-3.tex}
\newpage
\addcontentsline{toc}{section}{Exercise 2.4 Deriving the posterior predictive density for the healthy levels game}
\section*{Exercise 2.4 Deriving the posterior predictive density for the healthy levels game}
\import{sections/}{exercise 2-4.tex}