The previous chapter on motion planning and control introduced techniques for developing mathematical models to describe robot motion by analyzing its kinematics and dynamics. These models are typically expressed in the form of differential equations that are functions of a set of generalized coordinates/velocities and inputs to the system.
The next step is to discover how these models can be leveraged for robot motion planning and control. In particular this chapter and the next will focus on robot control, where the goal is to determine what inputs to apply to the system to achieve desirable behavior. To address the robot control problem a \textit{control law} must be developed, which is a set of rules or a mathematical function that determines what inputs should be applied to the system at any given time.

The ecosystem of techniques for robot control is vast, and control laws can generally be categorized in several ways. One of the most fundamental classifications for a control law is if it is \textit{open-loop} or \textit{closed-loop}. Open-loop control laws do not rely on observations to influence the choice of control input, while closed-loop control laws do. As a practical example, suppose you are standing in a room and wanted to walk to the other side and sit in a chair. For open-loop control you might look at where the chair is relative to your current position, think about how to walk there, and then \textit{with your eyes closed} walk to the chair and sit. Alternatively, for closed-loop control you might keep \textit{your eyes open} the whole time.

In practice, open-loop control laws suffer from robustness issues since they do not make corrections based on real-time observations. However, open-loop control is still an extremely important topic within the context of robotics.
In particular, suppose you are interested not just in getting your robot from one point to another, but doing so in the \textit{best} or \textit{optimal} way. This problem, known as \textit{trajectory optimization} or \textit{optimal control}\footnote{The terms trajectory optimization and optimal control will often be used interchangeably.}, can be solved to obtain an optimal trajectory for the robot along with the corresponding sequence of control inputs. In theory, applying this optimal control sequence as an open-loop control law would then make the robot follow the optimal trajectory.

This chapter will discuss several common techniques related to optimal control and trajectory optimization, including a brief review on dynamic/kinematic models, the formulation of the optimal control problem, approaches for solving optimal control problems, and some other topics useful in the context of robotics. The next chapter will then focus on the development of closed-loop control laws, including approaches that leverage the open-loop optimal control techniques discussed here.

\notessection{Open-Loop Motion Planning \& Control}
This chapter and the next will focus on two of the most fundamental classifications for a control law, namely whether it is \textit{open-loop} or \textit{closed-loop}. In particular, this chapter will focus on open-loop control laws that arise from the study of optimal control and trajectory optimization problems\cite{Kirk2004}\cite[\baselineskip]{Murray2009}. In general, open-loop control laws depend only on time and initial condition of the system.

\begin{definition} [Open-loop control] \label{def:openloop}
If the control law is determined as a function of time for a specified initial state value, i.e.,
\begin{equation}
    \bm{u}(t) = f ( \x ( t_0 ) , t ),
\end{equation}
then it is said to be in open-loop form.
\end{definition}

\subsection{Kinematic and Dynamic Models}
Chapter 1 discussed techniques for deriving kinematic and dynamic models of a robot in the form of ordinary differential equations (ODE). Such models are extremely useful in the context of robot motion planning and control, and are essential in the context of optimal control.
For the remainder of this chapter it will be assumed that such a model has already been identified and is expressed in the form
\begin{equation} \label{eq:dynamics}
    \dot{\x}(t) = a(\x(t),\bm{u}(t),t),
\end{equation}
where $\x \in \R^n$ may be comprised of generalized coordinates $\xi$ and velocities $\dot{\xi}$ and will be referred to as the robot's \textit{state}, $\bm{u} \in \R^m$ is the control input, and the function $a : \R^n \times \R^m \times \R \xrightarrow{} \R^n$ defines the model. While the set of ODEs \eqref{eq:dynamics} may have been derived by considering kinematics, dynamics, or a combination of the two, this model will be generally referred to as the robot's \textit{dynamics} model.

For clarity, note that \eqref{eq:dynamics} is a compact expression written in vector form for the system of $n$ first-order differential equations
\begin{align*}
    \dot{x}_1(t)&=a_1(x_1(t), x_2(t), \dots, x_n(t), u_1(t), u_2(t), \dots, u_m(t), t)\\
    \dot{x}_2(t)&=a_2(x_1(t), x_2(t), \dots, x_n(t), u_1(t), u_2(t), \dots, u_m(t), t)\\
&\vdots \\
    \dot{x}_n(t)&=a_1(x_1(t), x_2(t), \dots, x_n(t), u_1(t), u_2(t), \dots, u_m(t), t),
\end{align*}
where $x_i$ is the $i$-th component of the vector $\x$ and $u_j$ is the $j$-th component of the vector $\bm{u}$.

Solutions to the set of differential equations \eqref{eq:dynamics} are trajectories of the system. Given an initial condition $\x(t_0)$ and a control function $\bm{u}(t)$ defined for $t \geq t_0$, any technique for solving ODEs can be applied to compute the state trajectory $\x(t)$ for $t > t_0$. Common numerical integration approaches for solving the ODE system include the Runge-Kutta schemes, of which the most common are the forward or backward Euler schemes. The forward Euler scheme approximates $\dot{\x}(t) \approx \frac{\x_{i+1} - \x_i}{h_i}$ with $h_i = t_{i+1} - t_{i}$ and evaluates $a$ at time $t_i$. This leads to the recursive update
\begin{equation}
\x_{i+1} = \x_{i} + h_i a(\x_i,\bm{u}_i,t_i), \quad i = 0, 1,\dots
\end{equation}
where $\bm{u}_i = \bm{u}(t_i)$ and $\x_i = \x(t_i)$.

\subsection{Optimal Control Problem}
Perhaps the most common open-loop control laws used for motion planning and control in robotics are synthesized by formulating and solving optimal control problems. These problems are designed to answer the question: from the current state of the robot, $\x(t_0)$, what future control inputs $\bm{u}(t)$ would make the robot follow an optimal future trajectory? In general, generating optimal open-loop control laws require three major components:
\begin{enumerate}
    \item A model \eqref{eq:dynamics} that describes the robot's motion as a function of the input, developed by analyzing the robot's kinematics/dynamics.
    \item A metric that defines the quality of a particular trajectory, known as a \textit{cost function} or a \textit{reward function}\footnote{The term \textit{cost} is more commonly used in optimal control literature, while \textit{reward} is used in the reinforcement learning literature.}.
    \item An algorithm for searching the space of possible control inputs to find one that corresponds to an optimal trajectory\footnote[][\baselineskip]{For example, convex optimization solvers}.
\end{enumerate}

\subsubsection{Problem Formulation}
In this chapter the performance metric that defines the quality of a particular trajectory will be referred to as the \textit{cost function}. The standard form for defining the cost function in optimal control problems is
\begin{equation} \label{eq:cost}
J(\x(t), \bm{u}(t), t) = h(\x(t_f),t_f) + \int_{t_0}^{t_f} g(\x(t),\bm{u}(t),t) dt.
\end{equation}
where $h(\x(t_f),t_f)$ is referred to as a \textit{terminal cost} and where the integral can be viewed as a sum of \textit{stage costs} induced along the path from times $t_0$ to $t_f$.
In robotics, the function $J$ might quantify objectives such as ``get from point A to point B as quickly as possible” or “get from point A to point B while using as little effort as possible”.

Constraints can also be considered in the optimal control problem. In the field of robotics it is common to consider constraints on the state and control that are expressed compactly as
\begin{equation} \label{eq:constraints}
\x(t) \in \mathcal{X}, \quad \bm{u}(t) \in \mathcal{U},
\end{equation}
where $\mathcal{X}$ is the set of all \textit{admissible} states and $\mathcal{U}$ is the set of all \textit{admissible} control inputs. A common way to define the sets $\mathcal{X}$ and $\mathcal{U}$ is by a set of inequalities on $x$ and $u$, respectively. For example, let's assume the first element of $\x$ is constrained by $x_1 \geq 0$, then $\mathcal{X} = \{x \:|\: x_1 \geq 0\}$ such that any vector $\x$ with $x_1 \geq 0$ belongs to the set $\mathcal{X}$ (and is therefore \textit{admissible}). \marginnote{Constraints are commonly used in the context of robotics to account for actuator limits (e.g. how fast the wheels can turn, how much torque a motor can produce), or constraints on the trajectory itself (e.g. avoid collisions with surrounding objects).}

The optimal control problem is then expressed as an optimization problem over the state trajectory $\x(t)$ and control inputs $\bm{u}(t)$ with the goal of minimizing the cost function \eqref{eq:cost} while also satisfying the constraints \eqref{eq:constraints}.

\begin{definition}[Optimal Control Problem]
An optimal control problem seeks an \textit{admissible control} $\bm{u}(t)$ which causes the system (\ref{eq:dynamics}) to follow an \textit{admissible trajectory} $\bm{x}(t)$ that minimizes a performance metric $J(\x(t),\bm{u}(t),t)$. This problem can be expressed as an optimization problem:
\begin{equation} \label{eq:OCP}
\begin{split}
\underset{\bm{u},\x}{\text{minimize}} \:\: & h(\x(t_f),t_f) + \int_{t_0}^{t_f} g(\x(t),\bm{u}(t),t) dt,\\
\text{s.t.} \:\:& \dot{\x}(t) = a(\x(t),\bm{u}(t),t), \\
&\x(t) \in \mathcal{X}, \quad \bm{u}(t) \in \mathcal{U}, \\
&\x(t_0) = \x_0,
\end{split}
\end{equation}
where $t_0$ is the initial time, $t_f$ is either a fixed final time or an optimization variable, and $x_0$ is a known initial condition.
\end{definition}

The solution to the optimal control problem \eqref{eq:OCP} is an admissible and optimal trajectory defined over the interval $t \in [t_0, t_f]$, and is denoted by $\bm{u}^*(t)$ and $\x^*(t)$.

\subsubsection{Solving the Optimal Control Problem}
Once the optimal control problem \eqref{eq:OCP} has been formulated, the next step is to find a solution. However, this can be challenging since \eqref{eq:OCP} is an infinite-dimensional optimization problem (because the optimization is over an infinite-dimensional function and not a finite set of parameters). Unless an analytical solution to the problem can be found, this problem must be transformed into a finite dimensional problem so that it can be solved numerically on a computer.
In general, algorithms for numerically solving optimal control problems can be classified as either \textit{direct} or \textit{indirect} methods.

\paragraph{Direct Methods:}
Direct methods follow a ``first discretize, then optimize" approach. In the first step the problem \eqref{eq:OCP} is converted into a finite-dimensional problem by discretizing the functions $\x(t)$ and $\bm{u}(t)$. For example this might be accomplished by defining the new optimization variables to be $\x(t_i)$ and $\bm{u}(t_i)$ for a finite number of time points $t_i$. This finite-dimensional optimization problem is generally referred to as a \textit{nonlinear program} (NLP), which can be solved with existing numerical algorithms\footnote{Several solvers for solving general NLPs include IPOPT and SNOPT, and software packages for solving optimal control problems using the direct method include DIDO, PROPT, and GPOPS.}.

\paragraph{Indirect Methods:}
Indirect methods follow a ``first optimize, then discretize" approach. These methods first derive the necessary conditions of optimality, which are expressed as a two-point boundary value problem. This two-point boundary value problem is essentially a set of ODEs with boundary conditions at two points\footnote{This is in contrast to initial value problems, which have a single boundary condition and can easily be numerical integrated to find a solution.} that must be numerically solved.

\vspace{\baselineskip}
Indirect methods are less commonly used in robotics because the derivation of the necessary conditions of optimality must be done on a case by case basis, and can become quite challenging. They become particularly difficult to use when constraints are imposed in the problem. In contrast, direct methods offer much more flexibility and have been quite successful in practice.

\subsection{Differential Flatness}
Solving optimal control problems to compute optimal trajectories and optimal control inputs for a system can sometimes be computationally challenging. In fact, sometimes it is more desirable to have a computationally efficient way of generating ``good'' trajectories, rather than a challenging way of generating ``optimal'' ones.

For a special class of models, which are referred to as \textit{differentially flat}, computing ``good'' trajectories without having to formulate optimal control problems is quite easy. There are several models that are common in robotics that are differentially flat, including a simple car model and quadrotor models.

\begin{example}[Simple Car Model] \label{ex:carflatness}
\theoremstyle{definition}
\begin{marginfigure}
    \centering
    \includegraphics[width=0.95\linewidth]{tex/figs/ch02_figs/car.png}
    \caption{Simple model for an automobile. The state consists of the $(x,y)$ position of the center of the rear axle and the heading angle $\theta$. The control inputs are the steering angle $\phi$ and the forward velocity.}
    \label{fig:car-model}
\end{marginfigure}
Consider the car model corresponding to Figure \ref{fig:car-model}:
\begin{equation} \label{eq:car-dynamics}
\begin{split}
    \dot{x} &= v\cos\theta,\\
    \dot{y} &= v\sin\theta,\\
    \dot{\theta} &= \frac{v}{L}\tan\phi,
\end{split}
\end{equation}
where $(x, y)$ is the position and $\theta$ is the orientation of the vehicle, $v$ is the speed, $\phi$ is the steering angle, and $L$ is the length of the wheelbase. The state $\x$ is therefore defined as $\x = [x, \: y, \: \theta]^\top $ and the control is defined as $\bm{u} = [v, \:\phi]^\top $.

Suppose the motion planning task is to find a control sequence $\bm{u}(t)$ that will take the car from an initial state $\x_0$ to a final desired state $\x_{f}$. One option would be to formulate an optimal control problem with constraints $\x(t_0) = x_0$ and $\x(t_f) = \x_f$. However, it turns out that for this model there is a simpler approach. In fact, for this model it is sufficient to specify a differentiable trajectory for $x(t)$ and $y(t)$, and the remaining state variables and control inputs can be \textit{analytically} determined!

To see why this is, consider a differentiable trajectory for for $x(t)$ and $y(t)$ with derivatives $\dot{x}(t)$ and $\dot{y}(t)$. From the dynamics model \eqref{eq:car-dynamics} it can be seen that the first two equations can be leveraged to compute $\theta(t)$:
\begin{equation*}
\theta = \tan^{-1}(\dot{y}/\dot{x}).
\end{equation*}
Furthermore, once $\theta(t)$ has been computed the speed is defined:
\begin{equation*}
v = \dot{x}/\cos\theta, \quad \text{or}  \quad v = \dot{y}/\sin\theta.
\end{equation*}
Finally, given $\theta(t)$ and $v(t)$ it is possible to directly solve for the steering angle:
\begin{equation*}
\phi = \tan ^{-1}(\frac{L\dot{\theta}}{v}).
\end{equation*}

This property, that from the specification of a few variables and their derivatives the remaining state and control values are defined, is known as \textit{differential flatness}.
\end{example}

\begin{definition}[Differential Flatness]
A non-linear system
\begin{equation} \label{eq:diffflatsys}
\dot{\x}(t) = a(\x(t),\bm{u}(t)),
\end{equation}
is differentially flat with flat output $\z$ if there exists a function $\alpha$ such that
\begin{equation}
\z = \alpha (\x,\bm{u},\dot{\bm{u}},\dots,\bm{u}^{(p)}),
\end{equation}
and such that the solutions to the system $\x(t)$ and $\bm{u}(t)$ can be written as functions of the flat output $\z$ and a finite number of its derivatives:
\begin{equation} \label{eq:ztoxu}
\begin{split}
\x &= \beta (\z,\dot{\z},\dots,\z^{(q)}) \\
\bm{u} &= \gamma (\z,\dot{\z},\dots,\z^{(q)}).
\end{split}
\end{equation}
\end{definition}

For a differentially flat system, all of the feasible trajectories for the system can be written as functions of a flat output $\z(t)$ and its time derivatives. Additionally, note that the number of flat outputs is always equal to the number of system inputs. In the context of motion planning and control this is extremely useful for trajectory design because the flat outputs can be specified and then \textit{directly mapped} to the corresponding control inputs.

\subsubsection{Trajectory Design for Differentially Flat Systems}
As previously mentioned, trajectory design for differentially flat systems only requires specification of the trajectories of the flat outputs, which greatly simplifies motion planning and control.

Consider a nonlinear system model of the form \eqref{eq:diffflatsys} that is differentially flat with flat output $\z$ where the objective is to design a trajectory from $\x_0$ to $\x_f$ over a horizon of $T$ seconds. First, find the boundary conditions for the flat output $\z(0)$ and $\z(T)$ that satisfy the boundary conditions on $\x$ by noting that
\begin{equation} \label{eq:flatbc}
\begin{split}
\x_0 &= \beta (\z(0),\dot{\z}(0),\dots,\z^{(q)}(0)), \\
\x_f &= \beta (\z(T),\dot{\z}(T),\dots,\z^{(q)}(T)). \\
\end{split}
\end{equation}
Second, compute \textit{any} smooth trajectory for the flat outputs $\z(t)$ that satisfy these boundary conditions. Third, use \eqref{eq:ztoxu} to map the flat output trajectory $\z(t)$ to the state and control trajectories $\x(t)$ and $\bm{u}(t)$.

Since the flat outputs can be specified as any smooth trajectory, a common choice is to parameterize them using $N$ smooth basis functions:
\begin{equation} \label{eq:flat}
z_j(t) = \sum_{i=1}^{N} \alpha_i^{[j]} \psi_i(t),
\end{equation}
where $z_j$ is the $j$-th element of $\z$, $\alpha_i^{[j]} \in \mathbb{R}$ are variables that parameterize the trajectory and $\psi_i(t)$ are the smooth basis functions. One potential choice is to use polynomial basis functions $\psi_1(t) = 1$, $\psi_2(t) = t$, $\psi_3(t) = t^2$, and so on. Another advantage of choosing this parameterization of $z_j(t)$ is that it is linear in the variables $\alpha_i^{[j]}$. This makes it easy to map specifications on $\z$ into values for $\alpha_i$ that define the trajectory. Consider differentiating \eqref{eq:flat} $q$ times:
\begin{equation}
\begin{split}
\dot{z}_j(t) &= \sum_{i=1}^{N} \alpha_i^{[j]} \dot{\psi_i}(t), \\
&\vdots \\
z_j^{(q)}(t) &= \sum_{i=1}^{N} \alpha_i^{[j]} \psi_i^{(q)}(t). \\
\end{split}
\end{equation}
Now, from the initial and final conditions $z_j(0), \: \dot{z}_j(0), \: \dots , z_j^{(q)}(0)$ and $z_j(T), \: \dot{z}_j(T), \: \dots , z_j^{(q)}(T)$ the coefficients $\alpha_i^{[j]}$ can be computed by solving the following linear system (assuming the matrix is full rank):
\begin{equation} \label{eq:diffflatlinear}
\begin{bmatrix}
    \psi_1(0) & \psi_2(0) & \dots & \psi_N(0) \\
    \dot{\psi_1}(0) & \dot{\psi_2}(0) & \dots & \dot{\psi_N}(0) \\
    \vdots & \vdots & & \vdots \\
    \psi_1^{(q)}(0) & \psi_2^{(q)}(0) & \dots & \psi_N^{(q)}(0) \\
    \psi_1(T) & \psi_2(T) & \dots & \psi_N(T) \\
    \dot{\psi_1}(T) & \dot{\psi_2}(T) & \dots & \dot{\psi_N}(T) \\
    \vdots & \vdots & & \vdots \\
    \psi_1^{(q)}(T) & \psi_2^{(q)}(T) & \dots & \psi_N^{(q)}(T) \\
\end{bmatrix}
\begin{bmatrix}
    \alpha_1^{[j]} \\
    \alpha_2^{[j]} \\
    \vdots \\
    \alpha_N^{[j]} \\
\end{bmatrix} =
\begin{bmatrix}
    z_j(0) \\
    \dot{z}_j(0) \\
    \vdots \\
    z^{(q)}_j(0) \\
    z_j(T) \\
    \dot{z}_j(T) \\
    \vdots \\
    z_j^{(q)}(T)
\end{bmatrix}.
\end{equation}
Once the values for $\alpha_i^{[j]}$ are known, the entire trajectory $z_j(t)$ is therefore known!

Note that this approach is not strictly limited to specifying the initial and final conditions. It is also possible to specify other constraints on $z_j$ and its derivatives as long as they are \textit{equality} constraints. This is accomplished by simply adding equations corresponding to the desired constraints to the linear system of equations \eqref{eq:diffflatlinear}. However, if too many constraints are added the linear system \eqref{eq:diffflatlinear} may not have a solution (i.e. the system is over-determined). Assuming the constraints are not conflicting, this problem can typically be fixed by adding additional basis functions.

To summarize, for differentially flat nonlinear systems, the motion planning and control problem can be greatly simplified by planning in the flat output space. This is possible because of nonlinear functions that allow the flat output trajectory to be directly mapped to state and control trajectories that satisfy the system dynamics.

\subsubsection{Constraints and Time Scaling}
As previously shown, some constraints (e.g. boundary conditions) can be imposed on the trajectory by converting them into conditions on $\z$ and its derivatives, and then solving the linear system of equations \eqref{eq:diffflatlinear}. However, applying \textit{bound} constraints can be slightly more challenging since they are expressed as inequality constraints rather than equality constraints. Nonetheless, bound constraints are common in robotics and therefore it is important to be able to consider them in the trajectory generation process. For example, the simple car robot from Example \ref{ex:carflatness} could have an upper bound on its speed:
\begin{equation*}
|v(t)| \leq v_\text{max}.
\end{equation*}

One technique for handling these types of constraints is to use \textit{time scaling}. The general approach to satisfy bound constraints by time scaling is:
\begin{enumerate}
    \item Specify boundary conditions and solve the linear system of equations \eqref{eq:diffflatlinear} to get a candidate trajectory $\x(t)$ with control inputs $\bm{u}(t)$.
    \item If the candidate trajectory violates any bound constraints, generate a new trajectory by keeping the same geometric \textit{path} but decreasing the rate at which it moves along the path.
\end{enumerate}

\subsubsection{Geometric Path}
A geometric path is a sequence of states for the robot that is not associated with time. Given a candidate trajectory $\x(t)$, the geometric path can be defined by alternatively expressing the trajectory as $\x(t) = \x(s(t))$ where $s$ is a new ``path'' parameter and $s(t)$ is defined with $s(0) = s_0$, $s(T) = s_f$, and $\dot{s}(t) > 0$. A common choice for the path parameter $s$ is the arc length along the path. The geometric trajectory is then written as just $\x(s)$, such that the state is now a function of the position along the path and not time. Note that $\x(t): [0,T] \xrightarrow{} \R^n$ and $\x(s): [s_0, s_f] \xrightarrow{} \R^n$ are actually two different functions. In particular, the function $\x(t)$ can be derived from $\x(s)$ by the definition of the function $s(t): [0,T] \xrightarrow{} [s_0, s_f]$ and the composition $\x(s(t))$.

\subsubsection{Time Scaling}
For some systems, once the geometric path $\x(s)$ has been extracted from the candidate trajectory $\x(t)$, it is possible to arbitrarily redefine new trajectories with different time scales by simply redefining $s(t)$. In other words parts of the original candidate trajectory can be sped up or slowed down as desired.

To motivate why time scaling is important we can consider a simplified problem that does not involve a dynamics model. In particular, consider a scalar variable $x \in \R$ and a desired geometric path that connects $x_0$ and $x_f$ that is parameterized as $x(s) = x_0 + s(x_f - x_0)$ for $s \in [0,1]$ (note that $x(0) = x_0$ and $x(1) = x_f$). By choosing how $s$ varies in time (i.e. the function $s(t)$) this geometric path can be transformed into many different \textit{trajectories}, $x(t)$. As a simple choice, the function $s(t)$ can be parameterized as the cubic polynomial:
\begin{equation*}
    s(t) = \frac{3}{T^2}t^2 - \frac{2}{T^3}t^3.
\end{equation*}
This specific choice ensures that $s(0) = 0$, $s(T) = 1$, and $\dot{s}(0) = \dot{s}(T) = 0$ such that the trajectory will be defined over the time interval $t \in [0,T]$. Substituting this function into $x(s)$ then yields an expression for the trajectory $x(t)$:
\begin{equation*}
    x(t) = x_0 + \big(\frac{3}{T^2}t^2 - \frac{2}{T^3}t^3 \big)(x_f - x_0).
\end{equation*}
One easy way to scale the trajectory in this case is to simply change $T$, with larger values of $T$ meaning that it will take longer for $x$ to traverse the geometric path from $x_0$ to $x_f$. In fact, the maximum velocity can also be computed as:
\begin{equation*}
    \dot{x}_{\text{max}} = \frac{3}{2T}(x_f - x_0).
\end{equation*}
Therefore, not only does rescaling the trajectory by changing $T$ make the path traversal time change, but it can also be used to decrease quantities such as the maximum velocity!

\paragraph{Time Scaling with Differential Models:}
Some additional considerations need to be made when time-scaling trajectories that must also satisfy differential models.
First, note that the time derivative of the state can be rewritten by using the chain rule:
\begin{equation*}
\dot{\x}(t) = \frac{d\x(t)}{dt} = \frac{d\x(s)}{ds} \frac{ds(t)}{dt}.
\end{equation*}
Now consider a candidate trajectory $\x(t)$ and an associated geometric path $\x(s)$ for some $s(t)$ that is defined over the interval $t\in[0,T]$ with $s(0) = s_0$ and $s(T) = s_f$. Since $\x(t)$ is a trajectory of the dynamics \eqref{eq:diffflatsys}, the geometric path $\x(s)$ and time scaling law $s(t)$ satisfy
\begin{equation}
\frac{d\x(s)}{ds} \frac{ds(t)}{dt} = a(\x(s), \bm{u}(s)),
\end{equation}
for every point $s \in [s_0, s_f]$.

To design a new time scaling law $\tilde{s}(t)$ over some potentially new time interval $t \in [0,\tilde{T}]$ where $\tilde{s}(0) = s_0$ and $\tilde{s}(\tilde{T}) = s_f$, it is important to note that the dynamics equations must still be satisfied\footnote{The geometric path is still defined on the interval $[s_0, s_f]$ so this interval must remain the same for any new time scaling law, but the time interval can change.}. In other words, for every $\tilde{s} \in [s_0, s_f]$:
\begin{equation} \label{eq:stildedynamics}
\frac{d\x(\tilde{s})}{d\tilde{s}} \dot{\tilde{s}} = a(\x(\tilde{s}), \tilde{\bm{u}}(\tilde{s})).
\end{equation}
Since the geometric path is fixed, the terms $\frac{d\x(\tilde{s})}{d\tilde{s}}$ and $\x(\tilde{s})$ are fixed. Thus a new time scaling law $\tilde{s}(t)$ is only admissible if a new control $\tilde{\bm{u}}(\tilde{s})$ can also be found that guarantees that \eqref{eq:stildedynamics} holds.
Luckily, for some specific systems this is easy with the appropriate choice of path parameter $s$.

\begin{example}[Time Scaling for Simple Car Model]
\theoremstyle{definition} Consider again the simple car model \eqref{eq:car-dynamics} from Example \ref{ex:carflatness}. Suppose a candidate trajectory $\x_c(t)$ with control $\bm{u}_c(t)$ has been defined by leveraging the differential flatness of the model (i.e. setting up and solving \eqref{eq:diffflatlinear} and then mapping the flat outputs $\z_c(t)$ into the state and control). For this system a good choice for the path parameter is the arc-length, such that
\begin{equation*}
    s(t) = \int_0^\top  v(t') dt', \quad \dot{s}(t) = v(t).
\end{equation*}

With this choice of path parameter the geometric path function $\x_c(s)$, $s_0 = 0$, and $s_f = L_{\text{path}}$ are all fixed (where $L_{\text{path}}$ is the total length of the path). Rewriting the dynamics \eqref{eq:stildedynamics} based on the simple car model:
\begin{equation*}
\begin{split}
\frac{dx_c(\tilde{s})}{d\tilde{s}}\dot{\tilde{s}} &= v(\tilde{s})\cos\theta_c(\tilde{s}),\\
\frac{dy_c(\tilde{s})}{d\tilde{s}}\dot{\tilde{s}}&= v(\tilde{s})\sin\theta_c(\tilde{s}),\\
\frac{d\theta_c(\tilde{s})}{d\tilde{s}}\dot{\tilde{s}} &= \frac{v(\tilde{s})}{L}\tan\phi(\tilde{s}).
\end{split}
\end{equation*}
Any choice of the time scaling function $\tilde{s}(t)$ must be able to satisfy these equations, and note that the trivial choice of $\tilde{s}(t) = s(t)$ will automatically satisfy these equations with the candidate control inputs $\bm{u}_c(t)$.

Since the choice of the path parameter yields $\dot{\tilde{s}} = v(\tilde{s})$, these equations can be further simplified:
\begin{equation*}
\begin{split}
\frac{dx_c(\tilde{s})}{d\tilde{s}} &= \cos\theta_c(\tilde{s}),\\
\frac{dy_c(\tilde{s})}{d\tilde{s}}&= \sin\theta_c(\tilde{s}),\\
\frac{d\theta_c(\tilde{s})}{d\tilde{s}} &= \frac{1}{L}\tan\phi(\tilde{s}).
\end{split}
\end{equation*}
The first two equations are guaranteed to be satisfied for all $\tilde{s} \in [s_0, s_f]$ because the original candidate trajectory satisfies the dynamics. Additionally, the third equation is guaranteed to be satisfied by choosing $\phi(\tilde{s}) = \phi_c(\tilde{s})$ (i.e. using the same steering input as with the candidate trajectory).

This is interesting because it means that the equations are all satisfied \textit{independently} of the choice of $\dot{\tilde{s}}$. Therefore, since $\dot{\tilde{s}} = v(\tilde{s})$ this means that the speed input can be chosen arbitrarily while maintaining the same geometric path! This is extremely useful because it means that bound constraints on the speed $\lvert v(t)\rvert \leq v_\text{max}$ can be easily enforced.
\end{example}

\paragraph{Time Scaling with Kinematic Models:} Time-scaling trajectories is much more straightforward when kinematic models are used. Consider the case where the model of the system is derived from $k$ Pfaffian constraints $\bm{A}^\top (\x)\dot{\x} = 0$. In this case the kinematic model can be written in the form:
\begin{equation} \label{eq:kinmodel}
    \dot{\x} = G(\x) \bu,
\end{equation}
where the columns of the matrix $G(\x)$ span the null space of the matrix $A^\top (\x)$. Now again consider a path parameter $s$ that is used to reparameterize trajectories $\x(t)$ as $\x(s(t))$, and satisfies $s(0) = s_0$, $s(T) = s_f$, and $\dot{s}(t) > 0$\footnote{The condition $\dot{s}(t) > 0$ is critical to ensure that the function $s(t)$ is invertible. In other words, to guarantee that there is a one-to-one mapping between $t$ and $s$.}. Rewriting the time derivative of the state using the chain rule yields:
\begin{equation}
    \frac{d\x(s)}{ds} \dot{s} = G(\x) \bu(t).
\end{equation}
By making a substitution that $\bu(t) =  \bu_g(s)\dot{s}$ the dynamics can be further written as:
\begin{equation} \label{eq:kinematicgeometricmodel}
    \frac{d\x(s)}{ds} = G(\x)\bu_g(s).
\end{equation}
The terms $\bu_g(s)$ are referred to as \textit{geometric controls}, since they are defined only with respect to the path parameter $s$. Critically, \eqref{eq:kinematicgeometricmodel} says that once the geometric controls $\bu_g(s)$ are defined, the entire geometric path $\x(s)$ is also defined! The choice of the timing law $s(t)$ can then be chosen in any manner and it will not change the geometric path, but will change the time trajectory $\x(t)$. In particular, once the geometric control $\bu_g(s)$ and timing law are chosen, the actual controls are computed simply by the previous relationship $\bu(t) =  \bu_g(s)\dot{s}$.

Based on this analysis, the procedure for \textit{rescaling} a trajectory of a kinematic model can be made more concrete. First, consider a given trajectory $\x(t)$ with control $\bu(t)$ defined over $t \in [0,T]$ that satisfies the kinematic model \eqref{eq:kinmodel}. For simplicity, consider the path parameter $s$ to be arc-length of the trajectory such that $s(0) = 0$ and $s(T) = L_{\text{path}}$.
The following steps can then be used to define a new control input $\tilde{\bu}(t)$ that will make the kinematic model follow the same geometric path but with a different time scale:
\begin{enumerate}
\item Determine $s(t)$ based on the original trajectory $\x(t)$. In other words, figure out how far along the trajectory the system is at each time $t$. Then reparameterize the control $\bu(t)$ as a function of $s$, $\bu(s(t))$.
\item Compute the geometric controls $\bu_g(s) = \bu(s(t))/\dot{s}(t)$ for each point $s \in [s_0, s_f]$.
\item Define a new timing law $\tilde{s}(t)$ that satisfies $\tilde{s}(0) = 0$ and $\tilde{s}(\tilde{T}) = L_{\text{path}}$ with $\dot{\tilde{s}} > 0$ over the interval $[0, \tilde{T}]$.
\item Compute the new control $\tilde{\bu}(t) = \bu_g(\tilde{s}(t)) \dot{\tilde{s}}(t)$ for all $t \in [0, \tilde{T}]$.
\end{enumerate}

\begin{example}[Time Scaling for Unicycle Model] \label{ex:timescaleuni}
\theoremstyle{definition}
Consider the kinematic unicycle model:
\begin{equation} \label{eq:unicycle}
\begin{split}
    \dot{x} &= v\cos\theta,\\
    \dot{y} &= v\sin\theta,\\
    \dot{\theta} &= \omega,
\end{split}
\end{equation}
where $(x, y)$ is the position and $\theta$ is the orientation, $v$ is the speed, and $\omega$ is the rotation rate. The state $\x$ is defined as $\x = [x, \: y, \: \theta]^\top $ and the control is defined as $\bm{u} = [v, \:\omega]^\top $.

To time-scale trajectories of this system, consider the use of arc-length as path parameter:
\begin{equation*}
    s(t) = \int_0^t  v(\tau) d\tau, \quad \dot{s}(t) = v(t),
\end{equation*}
such that for a trajectory defined on the interval $t \in [0,T]$ with total length $L_{\text{path}}$, the path parameter is defined with $s(0) = 0$ and $s(T) = L_{\text{path}}$.
With this choice, the geometric controls are given by:
\begin{equation*}
\begin{split}
v_g(s) &= \frac{v(s)}{\dot{s}(t)} = 1, \\
\omega_g(s) &= \frac{\omega(s)}{\dot{s}(t)} = \frac{\omega(s)}{v(s)},
\end{split}
\end{equation*}
where $v(s(t))$ has been substituted in for $\dot{s}(t)$.
Therefore if a new timing law $\tilde{s}(t)$ is introduced this will automatically define a new velocity $\tilde{v}(\tilde{s})$ at each point $\tilde{s}$, which can then be used to solve for the new $\tilde{\omega}$ inputs by:
\begin{equation*}
\begin{split}
\tilde{\omega}(\tilde{s}) &= \omega_g(\tilde{s}) \dot{\tilde{s}}(t) = \frac{\omega(\tilde{s})}{v(\tilde{s})} \tilde{v}(\tilde{s}).
\end{split}
\end{equation*}
Alternatively, since it is easier to work with the velocity directly rather than $\tilde{s}(t)$, in this case it is possible to just specify $\tilde{v}(\tilde{s})$ for all $\tilde{s} \in [0, L_{\text{path}}]$ and then to compute $\tilde{\omega}(\tilde{s}) = \frac{\omega(\tilde{s})}{v(\tilde{s})} \tilde{v}(\tilde{s})$. Then, to determine the new controls as functions of time rather than $\tilde{s}$, it can be noted that
\begin{equation*}
    \tau(s) = \int_0^s \frac{ds'}{\tilde{v}(s')},
\end{equation*}
defines a function $\tau(s)$ that maps each point $s \in [0, L_{\text{path}}]$ to a new time.
\end{example}

\subsection{Exercises}
\subsubsection{Trajectory Generation via Differential Flatness}
Complete \textit{Problem 1:  Trajectory Generation via Differential Flatness} located in the online repository:

\vspace{\baselineskip}

\url{https://github.com/PrinciplesofRobotAutonomy/AA274A_HW1},

\vspace{\baselineskip}

where you will use an extended unicycle model to practice generating dynamically feasible trajectories by levering the system's differential flatness property. You will also have the chance to use time scaling techniques to design trajectories that satisfy control constraints.


\subsection{Introduction}

Used to sample from propability distribution function.

Useful when can't use direct sampling, because no closed form.

MORE GENERALLY FRAME THESE FIRST AS SAMPLING FROM PROBABILITY FUNCTION.

Generate pairs of \((x, y)\). If \(y<P(x)\) then keep \(x\).

Metropolis-Hastings and Gibb's sampling are extensions of this.


%!TEX root = ms.tex
\section{Conclusion and future work}
\label{sec:conclusion}
In this paper, the use of information criteria to compare regression models under general linear restrictions for both fixed and random predictors is discussed. It is shown that general versions for KL-based discrepancy (AICc and RAICc, respectively) and squared error-based discrepancy (C$_p$, FPE, RC$_p$ and S$_p$, respectively) can be formulated as effectively unbiased estimators of the test error (up to some terms that are free of the linear restrictions and hence are irrelevant when comparing criteria for different models). Model comparison based on the KL-based discrepancy measures is shown via simulations to be better-behaved than squared error-based discrepancies (including cross-validation) in selecting models with low predictive error and sparse subset.

The study of RAICc for variable selection in this paper focuses on OLS fits on pre-fixed predictors (e.g. nested predictors based on their physical orders in $X$). The discussion can be extended to other fitting procedures where the predictors in each subset are decided in a data-dependent way. For instance, \citet{tian2019use} discussed using AICc for least-squares based subset selection methods, and extending those results to the random-X scenario is a topic for future work.

Note also that only restrictions on the regression coefficients are considered here, corresponding to restrictions on the regression portion of the model. It is also possible that the data analyst could be interested in restrictions on the distributional parameters of the predictors (restricting the variances of some predictors to be equal to each other, for example, or restricting covariances to follow a specified pattern such as autoregressive of order $1$ or compound symmetry), and it would be interesting to try to generalize the criteria discussed here to that situation.


\chapter{Finite Markov Decision Process}

\section{Summary}

MDP's are a formal framework to model sequential decision making. It can handle immediate and delayed rewards, incorporating state.

\subsection{Agent-Environment Interface}

\begin{enumerate}
	\item agent: Leaner and decision maker.
	\item environment: The thing the agent interacts with.
	\item $P(s', r|s,a)$: dynamics of the system (as probability distribution)
\end{enumerate}

If the next state $S_{t+1}$ only depends on the current state $S_t$ and the input $A_t$, then the state is said to have the \textbf{Markov property} .

\begin{figure}[H]
	\centering
	\begin{tikzpicture}

	\node at (1, 0) (a) {Agent};
	\node at (4, 0) (b) {Enviroment};

	\draw [->, auto, bend left] (a) to node {$A_t$} (b);
	\draw [->, auto, bend left] (b) to node {$S_{t+1}$, $R_{t+1}$} (a);
	\end{tikzpicture}
	\caption{Agent Environment}
	\label{fig:agent-enviroment}
\end{figure}

\subsection{Goals and rewards}

The \textbf{reward hypothesis} say's that all goals/purposes can be expressed as maximizing reward.

Reward should only communicate \textbf{what} needs to be done, not \textbf{how}.

\subsection{Returns and episodes}
An \textbf{episodial task} has a finite number of steps, until it stops in the absorbing state. $G_t = R_{t+1} + R_{t+2} + ... + R_{T}$.

A \textbf{continuing task} never stops, so it keeps getting rewards. So an extra concept \textbf{discount factor}($\gamma$) is needed to define the expected reward.

\begin{equation}
\begin{split}
G_T & = R_{t+1} + \gamma R_{t+2} + ... \\
& = \sum_{k=0}^{\infty}\gamma R_{t+k+1} \\
& 0 \leq \gamma \leq 1
\end{split}
\label{eq:expected return continuing task}
\end{equation}

The discount factor is a geometric series, and equals to one.
\begin{equation}
\sum_{k=0}^{\infty} \gamma ^k = \frac{1}{1-\gamma}
\label{eq:discount factor geometry series}
\end{equation}

\subsection{Policies and value function}
The \textbf{state-value function} expressed how it is to be in a certain state under a certain policy($\pi$). It has a recursive definition that is derived in equation~\ref{eq:bellman equation value function derivation} and known as the \textbf{bellman equation}.

\begin{equation}
\begin{split}
V_{\pi}(s) & = \EX\left[G_t | S_t = s\right]\\
& = \EX\left[\sum_{k=0}^\infty \gamma^k R_{t + k + 1} | S_t = s\right]\\
& = \EX\left[R_{t+1} + \gamma G_{t+1} | S_t = s \right]\\
& = \sum_a \pi(a|s)\sum_{s',r} p(s', r|s,a)( r + \gamma \EX_\pi[G_{t+1}|S_{t+1}=s']) \\
& = \sum_a \pi(a|s)\sum_{s',r} p(s', r|s,a)( r + \gamma v_\pi(s'))
\end{split}
\label{eq:bellman equation value function derivation}
\end{equation}

The value of taking action $a$ under state $s$ is defined by the \textbf{action-value function} equation~\ref{eq:action-value function}.

\begin{equation}
\begin{split}
q_\pi(a, s)
& = \EX[G_t | S_t = s, A_t = a]\\
& = \EX\left[\sum_{k=0}^\infty \gamma^k R_{t + k + 1} | S_t = s, A_t = a\right]\\
& = \sum_{r,s'} p(s', r|s,a)( r + \gamma v_\pi(s'))
\label{eq:action-value function}
\end{split}
\end{equation}

\subsection{Optimal policies and optimal value function}
Solving a \textbf{reinforcement learning} problem is finding a policy that gets a lot of reward. The best possible policy is called the \textbf{optimal policy}, the value-state function of this policy is the \textbf{optimal state-value function} $v_*(s)=\max_{\pi}v_\pi(s)$. And the action-value function is the \textbf{optimal value-state function} $q_*(s, a)= \max_{\pi} q_\pi (s, a)$

\begin{equation}
\begin{split}
v_*
& = \max_a q_\pi (s, a) \\
& = \max_a \EX\left[ G_t | S_t=a, A_t=a \right] \\
& = \max_a \EX\left[ R_{t+1} + \gamma G_t | S_t=a, A_t=a \right] \\
& = \max_a \EX\left[ R_{t+1} + \gamma V_*(S_{t+1}) | S_t=a, A_t=a \right] \\
& = \max_a \sum_{s}p(s', r | a, s) [r+ \gamma V_*(s')] \\
\end{split}
\label{eq:bellman optimality equation state-value function derivation}
\end{equation}

The optimal state-value and action-value functions lead to \textbf{the bellman optimality equations} equation~\ref{eq:bellman optimality equation state-value function derivation} and equation~\ref{eq:bellman optimality equation action-value function derivation}

\begin{equation}
\begin{split}
q_*(s, a)
& = \EX\left[ R_{t+1} + \gamma \max_{a'} q(S_{t+1},a') | S_t=s, A_t=a \right] \\
& = p(s' r | s, a)[ r + \gamma \max_{a'} q(s', a)]
\end{split}
\label{eq:bellman optimality equation action-value function derivation}
\end{equation}

\section{Exercises}

\subsection{Exercise 3.1}
A Robot in a maze has a delayed reward, and needs to make a sequence of decisions. The position of the robot in the maze is the state, and the input is the decisions left/right/straight ahead. The reward is -1 until the absorbing state, which has a reward of zero.

A automatic poker player can be a mdp, the state is the current cards in the hand and the table.


\section{A Generative Model for Bayesian Sequence Combination}\label{sec:model}
%TODO do we state the obvious: that the annotator models effectively weight good annotators more heavily and discard spammers?
%The generative story for our approach,
To construct a generative model for \emph{Bayesian sequence combination (BSC)},
we first define a hidden Markov model (HMM)
with states $t_{n,\tau}$ and observations $x_{n,\tau}$
using categorical distributions:
\begin{flalign}
t_{n,\tau} & \sim \mathrm{Cat}(\bs T_{t_{n,\tau-1}}), \\
x_{n,\tau} & \sim \mathrm{Cat}(\bs\rho_{t_{n,\tau}}),
\end{flalign}
where $\bs T_j$ is a row of a transition matrix $\bs T$, and $\bs\rho_j$
is a vector of observation likelihoods for state $j$.
For text tagging, $n$ indicates a document and $\tau$ a token index, while
each state $t_{n,\tau}$ is a true sequence label
and $x_{n,\tau}$ is a token.
To provide a Bayesian treatment, we assume that
$\bs T_j$ and $\bs\rho_j$ have Dirichlet distribution priors as follows:
\begin{flalign}
T_j & \sim \mathrm{Dir}(\bs \gamma_j), \hspace{1.0cm}
\bs\rho_j \sim \mathrm{Dir}(\bs \kappa_j), &
\end{flalign}
where $\bs \gamma_j$ and $\bs \kappa_j$ are hyperparameters.

Next, we assume one of the annotator
models described in Section \ref{sec:annomodels} for each of $K$ annotators.
Selecting an annotator model is a design choice,
and all can be coupled with the Bayesian HMM above to form
a complete BSC model. In our
experiments in Section \ref{sec:expts_all},
we compare different choices of annotator model as components of BSC.
%We draw the parameters of the annotator models as follows:
%%The number of parameters depends on the choice of annotator model:
%%( t_{\tau}, c_{\tau}, c_{\tau-1})$.
%for \emph{acc}, only one parameter, $\pi^{(k)}$, is drawn per annotator $k$;
%for \emph{MACE}, we draw a single value $\pi^{(k)}$ and a vector $\xi^{(k)}$ of length $J$,
%while for \emph{CV} we draw $J$ independent values of $\pi_j^{(k)}$,
%and for \emph{CM}
%we draw a vector $\bs\pi^{(k)}_j$ of size $J$ for each true label value $j\in \{1,...,J\}$; in the case of \emph{seq},
%we draw vectors $\bs\pi^{(k)}_{j,\iota}$ for each true label value
%for each previous label value, $\iota$.\
All the parameters of these annotator models are probabilities,
so to provide a Bayesian treatment, we assume that they have Dirichlet priors.
For annotator $k$'s annotator model, we refer to the hyperparameters
of its Dirichlet prior as $\bs\alpha^{(k)}$.
%As shown in Section \ref{sec:annomodels},
The annotator model defines a categorical likelihood
over each annotation, $\bs c^{(k)}_{n,\tau}$:
%$A^{(k)}(t_{n,\tau}, \bs c^{(k)}_{n,\tau}, \bs c^{(k)}_{n,\tau-1})$, where $\bs c^{(k)}_{n,\tau}$ is the $\tau$th label of document $n$.
%%The argument $\bs c_{n,\tau-1}$ is only required if $A^{(k)}$ is an instance
%%of \emph{seq} and is ignored by the other annotator models.
%We draw annotator $k$'s label $c^{(k)}_{n,\tau}$
%for each token $\tau$ in each document $n$
%according to a categorical distribution:
\begin{flalign}
& c^{(k)}_{n,\tau} \sim \mathrm{Cat}( [A^{(k)}(t_{n,\tau}, 1, \bs c_{n,\tau-1}^{(k)}), ..., & \nonumber \\
& \hspace{3cm} A^{(k)}(t_{n,\tau}, J, \bs c_{n,\tau-1}^{(k)}) ]). &
\end{flalign}

The annotators are assumed to be conditionally independent of one another given the true labels,
$\bs t$, which means that their errors are assumed to be uncorrelated. This is a strong assumption
when considering that the annotators have to make their decisions based
on the same input data. However, in practice, dependencies do not usually cause the
most probable label to change~\citep{zhang2004optimality}, hence the performance of classifier combination methods
is only slightly degraded, while avoiding the complexity of modelling dependencies between annotators~\citep{kim2012bayesian}.
%
% \textbf{Black-box Sequence Taggers}:
% As an extension to our model, we can integrate $S$ automated methods as
% additional noisy annotators.
%  In comparison to human annotators,
% sequence taggers can quickly label large numbers of documents,
% providing a cheap source of additional annotations across the whole dataset.
% We model each sequence tagger, $s$,
% using an annotator model, $B^{(s)}$,
% of one of the types described in Section \ref{sec:annomodels} (analogous to $A^{(k)}$ for a human annotator),
% with hyperparameters $\bs \beta^{(s)}$.
%
% We extend the generative model for BSC with additional steps as follows.
% Each sequence tagger generates a sequence of labels, $\bs d_{n}^{(s)}$, for each document $n$
% (analogous to $\bs c_n^{(k)}$ produced by human annotators)
%  according to:
%  \begin{flalign}
%  & d_{n,\tau}^{(s)} \sim \mathrm{Cat}(
% [B^{(s)}(\bs t_{n,\tau}, 1, d_{n,\tau-1}^{(s)}), ..., && \nonumber \\
% & \hspace{3.0cm} B^{(s)}(\bs t_{n,\tau}, J, d_{n,\tau-1}^{(s)})]). &&
% \end{flalign}
%
% In the generative model, we draw a sequence of text tokens, $\bs x_n$,
% from a likelihood, $p \left(\bs x_n | \bs d_{n}^{(s)}, \bs\theta^{(s)} \right) $,
%  given internal parameters, $\bs\theta^{(s)}$, and
% label sequence, $\bs d_{n}^{(s)}$.
% This likelihood is defined by the black-box sequence tagger.
% If the sequence tagger is Bayesian, its parameters, $\bs\theta^{(s)}$, may also be drawn from
% an unknown prior distribution.
% However, since we are treating the tagger as a black box, we do not need to know these internal details.
% In the next section, we explain how we can avoid computing this likelihood explicitly during inference,
% and instead use only the sequence tagger's existing training and prediction functions to learn
% $\bs\theta^{(s)}$ in parallel with the parameters of the BSC model.
% Like the human annotators, each sequence tagger is assumed to produce labels that are conditionally independent
% of the other sequence taggers given $\bs t$.
% %Due to the fact that sequence taggers will typically use
% %the same features, i.e. the text of the documents, this independence assumption may be violated,
% %yet
% %%as with the human annotators,
% %this assumption in other models
% % has been shown not to hamper performance in
% %many practical situations~\citep{zhang2004optimality}.

\textbf{Joint distribution}: the complete model can be represented by the
joint distribution, given by:
\begin{flalign}
& p(\bs t, \bs A, \bs T, \bs\rho, \bs c, \bs x | \bs \alpha^{(1)},..., \bs \alpha^{(K)}, \bs\gamma,
\bs \kappa ) &  \\
%  & \approx q(\bs t, \bs A, \bs B, \bs\A^{(1)},...,\bs\A^{(K)},\bs\A^{(1)},...,\bs\A^{(S)}, \bs d^{(1)}, ...,\bs d^{(S)}) = q(\bs B) \prod_{n=1}^N q(\bs t_n) & \nonumber \\
& = \prod_{k=1}^K \left\{ p(A^{(k)} | \bs \alpha^{(k)}) \prod_{n=1}^N p(\bs c_n^{(k)} | A^{(k)}, \bs t)  \right\}
& \nonumber \\
&  \prod_{n=1}^N \prod_{\tau=1}^{L_n} p(t_{n,\tau} | \bs T_{t_{n,\tau-1}}) p(x_{n,\tau} | t_{n,\tau}, \bs\rho_{t_{n,\tau}}) & \nonumber \\
& \prod_{j=1}^J p(\bs T_j | \bs\gamma_j) p(\bs\rho_j | \bs\kappa_j)&
%\prod_{s=1}^S \bigg\{ p(\bs \theta^{(s)})  \nonumber \\
%& p(B^{(s)} | \bs\beta^{(s)}) \! \! \prod_{n=1}^N \!\! \left\{ p(\bs x | \bs d^{(s)}, \bs \theta^{(s)}) p(\bs d^{(s)} | B^{(s)}, \bs t)  \right\} \!\! \bigg\}, \nonumber
& \label{eq:joint}
\end{flalign}
where
$\bs c$ is the set of annotations for all documents from all annotators,
$\bs t$ is the set of all sequence labels for all documents,
$N$ is the number of documents,
$L_n$ is the length of the $n$th document,
$J$ is the number of classes,
 $\bs x$ is the set of all word sequences for all documents and
$\bs\rho$, $\bs\gamma$ and $\bs\kappa$ are the sets of parameters for all
label classes.
 %=\{\bs c^{(1)}, .., \bs c^{(K)} \}$,
%
%Terms distribution omit subscripts and superscripts are the sets of  parameters for all values of the omitted index.

%\begin{flalign}
%&
%=
%p\left(d_{n,\tau}^{(s)} | \bs t, d_{n,\tau-1}^{(s)}, A^{(s)} \right) & \nonumber\\
%& p \left(d_{n,\tau}^{(s)} | \bs\phi_n, \bs\theta^{(s)} \right) / p \left(d_{n,\tau}^{(s)} | \bs\theta^{(s)} \right),  &
%\end{flalign}
%% Imagine what happens for each of the different combinations of values of d. We could train a squillion different sequence taggers on these, then take a weighted sum.
%
%% multiply by p(phi | theta) to get joint, divide by p(d) to get likelihood of phi given d, then since phi is independent of t and A,  p(phi | theta) cancels out. This follows from the generative model, which should be described here. The inference section should
%% talk about learning theta, and computing the expectation of d.
%%We could avoid all this if the sequence tagger just learns to represent the left hand side. So t and A stay in the condition for the likelihood of the features. In which case, the first term on the right hand side is like a prior that needs to be
%% used when learning theta.
%where the first term on the right-hand side is defined by the annotator model
%with parameters $A^{(s)}$, and
%of the sequence tagger, $s$.
%integrating existing sequence taggers using the learning
%procedure described in the next section.


\subsection{Commutation of Lie groups}

We can measure commutation of Lie groups using:

\(ABA^{-1}B^{-1}\)

If the group commutes then:

\(ABA^{-1}B^{-1}=BA^{-1}B^{-1}=I\)

\subsubsection{Commutation of Lie algebra: COMPLETE THIS}

This corresponds to \([A,B]=AB-BA\) in the underlying lie algebra, if we expand.

\(A=e^{ta}\)

\(B=e^{tb}\)

\(ABA^{-1}B^{-1}=e\)


\documentclass{article}

\usepackage[english]{babel}
\usepackage{naproche}

\begin{document}
  \pagenumbering{gobble}

  \section*{Cantor's Theorem}

  Let us prove that every set is strictly smaller than ist powerset.
  This result is known as \textit{Cantor's Theorem}.

  \begin{forthel}
    \begin{theorem}[Cantor]
      Let $x$ be a set.
      There is no surjection from $x$ onto the powerset of $x$.
    \end{theorem}

    \begin{proof}
      Proof by case analysis.

      Case $x$ is empty. Obvious.

      Case $x$ is nonempty.
        Assume the contrary.
        Take a surjection $f$ from $x$ onto the powerset of $x$.

        Define $N = \{ u "in" x : u "is not an element of" f(u) \}$.

        Take an element $u$ of $x$ such that $N = f(u)$.

        Indeed we can show that $N$ is an element of the powerset of $x$.
          Every element of $N$ is an element of $x$.
          Hence $N$ is a subset of $x$.
          Thus $N$ is an element of the powerset of $x$.
        End.

        Then we have a contradiction.
      End.
    \end{proof}
  \end{forthel}
\end{document}


\documentclass[12pt]{article}

\usepackage{maria}

\title{Summations \\ \small{Red Group, MOP 2013}}

\author{Maria Monks}
\date{June 6, 2013}

\begin{document}

\maketitle{}

\section*{Some summation techniques}

\begin{itemize}
\item \textbf{Telescoping sums:}
Suppose we want to evaluate the sum $a_1+a_2+\cdots+a_n$, and we can write each $a_i$ as $b_i-b_{i+1}$ for some collection of numbers $b_i$.  Then all the $b_i$'s cancel except for $b_1$ and $b_{n+1}$, and we are left with $b_1-b_{n+1}$.

For instance, we can evaluate the sum $1+3+5+\cdots+(2k-1)$ by writing the $i$th odd number as $i^2-(i-1)^2$, obtaining the sum: $$(1^2-0^2)+(2^2-1^2)+(3^2-2^2)+\cdots+(k^2-(k-1)^2)=k^2-0^2=k^2.$$

%Telescoping products, such as $\frac{1}{2}\cdot \frac{2}{3} \cdot \frac{3}{4} \cdot\cdots\cdot \frac{n-1}{n}$, are useful in the same manner.

\item \textbf{Induction:} For finite sums, it's often possible to just guess the answer and then prove your guess by induction.

\item \textbf{Solve for the sum:} Let $S$ be the sum you wish to find, manipulate $S$ in order to simplify the sum, and finally solve for $S$.  For instance, to evaluate the infinite sum $$S=\sum_{j=0}^\infty\frac{j^2}{2^j},$$ we might consider multiplying by $2$ to shift the indices.  We have  $$2(S-1)=\sum_{j=1}^\infty \frac{j^2}{2^{j-1}}=\sum_{j=0}^\infty \frac{(j+1)^2}{2^j}$$  Subtracting $S$ from this, we find $2(S-1)-S=\sum_{j=0}^{\infty} \frac{2j+1}{2^{j}}$.  So far, we have reduced the $j^2$ to a $2j+1$ in the numerator.  Can you finish it from here?

\item \textbf{Generating functions:}  It can be useful to consider the sum as a coefficient in a generating function, especially in the following circumstances:
\begin{itemize}
\item \textbf{Partial sums:} Say we want to find a formula for $1^2+2^2+\cdots+n^2$.  First consider the generating function $S(x)=\sum_{i=0}^\infty n^2x^n$.  Now, if we divide by $1-x$, we get the partial sums as the coefficients of the new generating function: $$\frac{S(x)}{(1-x)}=\sum_{i=0}^\infty (1^2+2^2+\cdots+n^2)x^n.$$  So, we just need to find the coefficient of $x^n$ in $\frac{S(x)}{(1-x)}$, which equals $\frac{x^2+x^4}{(1-x)^4}$ (why?).  We expand $(1-x)^{-4}$ using the \textit{extended binomial theorem}:

\begin{eqnarray*}
\frac{1}{(1-x)^4} &=& \sum_{n=0}^\infty (-1)^n\binom{-4}{n}x^n \\
&=&\sum (-1)^n\frac{(-4)(-5)(-6)\cdots(-4-n+1)}{n!}x^n \\
     &=& \sum \frac{4\cdot 5\cdot \cdots\cdot (n+3)}{n!} x^n \\
     &=& \sum \frac{n(n+1)(n+2)}{6}x^n
\end{eqnarray*}

Multiplying this by $x+x^2$, the coefficient of $n$ in the result is $$\frac{(n-1)n(n+1)}{6}+\frac{n(n+1)(n+2)}{6}=\frac{n(n+1)(2n+1)}{6},$$ as desired.

\item \textbf{Convolutions:}  Sums of the form $$a_0b_n+a_1b_{n-1}+\cdots+a_nb_0$$ are called \textit{convolutions}, and they should alert your generating function radar.  Setting $F(x)=\sum a_nx^n$ and $G(x)=\sum b_nx^n$, this convolution is the coefficient of $x^n$ in $F(x)\cdot G(x)$.

Can you use this idea to compute $\sum_{i=0}^n i(n-i)$?

\end{itemize}

\end{itemize}

\section*{Problems}

\begin{enumerate}

\item Evaluate the sum $\sum_{i=0}^n i(n-i)$ using several different summation techniques.  Be creative!
%The answer turns out to be n(n^2-1)/6.  This is a good one.

\item Evaluate the sum $\sum_{k=1}^n \frac{k}{(k+1)!}$.
%Easily telescopes, or also by induction it turns out to be $1-\frac{1}{(n+1)!}$

\item Evaluate the sum $\sum_{k=1}^n k\cdot k!$.

\item (Brazilian Math Olympiad 2002.)  For a nonempty subset $A$ of $\{1,2,\ldots,n\}$ define $f(A)$ as the largest element of $A$ minus the smallest element of $A$.  Find $\sum f(A)$ where the sum is taken over all nonempty subsets $A$ of $\{1,2,\ldots,n\}$.
%Turns out to be (n-3)*2^n+n+3.  Do this by summing the values of all the min and max values separately: each i is the max for 2^{i-1} subsets, etc.

\item Evaluate the sum $\sum_{k=1}^n (-1)^{\frac{k(k+1)}{2}} k$.
% The answer turns out to be 0 when n is 3 mod 4, and the others are easily determined.  (You can prove the former by induction.)

\item (IMO Longlist 1978.)  Evaluate the sum $$1\cdot 2\cdot 3+2\cdot 3\cdot 4+\cdots+97\cdot 98\cdot 99.$$  (Bonus: What happens if we divide by $3!$ and think of the terms as binomial coefficients?)
%This turns out to be $97*98*99*100/4$, because of the hockey stick identity.  You can also do this by making each term $i(i-1)(i+1)$ and summing $i^3-i$, but this doesn't turn out as nicely in the formula in the end.

\item Find a closed formula for $\frac{1}{1\cdot 2 \cdot 3}+\frac{1}{2\cdot 3 \cdot 4}+\cdots+\frac{1}{n\cdot (n+1) \cdot (n+2)}.$
%%1/4 of n(n+3)/(n+1)(n+2), nicely enough.  Either: 1/1*(1/2-1/3)+1/2*(1/3-1/4)+... and it reduces to 2 products case, or: telescope by making it 1/(2n(n+1))-1/(2(n-1)n).

\item (Art and Craft of Problem Solving.)  A 2-inch elastic band is fastened to the wall at one end, and there's a bug at the other end.  Every minute (beginning at time 0), the band is instantaneously and uniformly stretched by 1 inch, and then the bug walks 1 inch toward the fastened end.  Will the bug ever reach the wall?
%Yep!  It's a harmonic series thing.  Just record the percent along the band that the bug is at at each step.  The ratio starts at 1 and decreases by 1/3, 1/4, 1/5, ... so eventually the ratio will be less than 0, at which point he has reached the wall.

\item In all of the following, let $F_n$ be the $n$th Fibonacci number, where $F_0=0,F_1=1$, and $F_{n+2}=F_n+F_{n+1}$ for all $n\ge 0$.
\begin{enumerate}
\item Evaluate $\sum_{n=0}^\infty \frac{F_n}{F_{n+1}\cdot F_{n+2}}$.
%This telescopes because each term is equal to $1/F_{n+1}-1/F_{n+2}$.  We are left with $1/F_1=1$.
\item Evaluate $\sum_{n=1}^\infty \frac{F_{n+1}}{F_n\cdot F_{n+2}}$.
%Also telescopes, but now we have a jumping thing so it's $1/1+1/1=2$.
\item Evaluate $\sum_{n=1}^\infty \frac{1}{F_n\cdot F_{n+2}}$.
%Telescope this guy by showing that the nth term equals $1/(F_n*F_{n+1})-1/(F_{n+1}*F_{n+2})$.
\item Show that $\sum_{k=0}^n F_k=F_{n+2}-1$.
%Induction works just fine here
\item Show that $\sum_{k=0}^n F_k^2=F_n\cdot F_{n+1}$.
%Also induction.
\end{enumerate}

\item (Canada 1989.)  Given the numbers $1,2,2^2,\ldots,2^{n-1}$, for a specific permutation $\sigma=x_1,x_2,\ldots,x_n$ of these numbers we define $$S_i(\sigma)=x_1+x_2+\cdots+x_i$$ for each $i=1,2,\ldots,n$.  Define $Q(\sigma)=S_1(\sigma)S_2(\sigma)\cdots S_n(\sigma)$.  Evaluate $\sum 1/Q(\sigma)$ where the sum is taken over all possible permutations.
%The answer is 1/2^{n choose 2}.  Basically, you can show by induction that you can group them according to the last factors in the denominators.  You can always factor out a 1/S_n=1/(1+2+...+2^{n-1}), and then to sum what is left, group it by the next largest term, namely the $S_{n-1}(\sigma)$'s.  Use strong induction to show that these sum corresponding to a fixed $S_{n-1}(\sigma)$ turn out to be one over the product of the terms in the sum $S_{n-1}(\sigma)$.  Then in the end we add up all these sums, weighted by 1/S_n, and we get S_n/((product of terms in S_n)*S_n)=1/(1*2*4*...*2^{n-1})=1/2^{n choose 2}.

\item (Ukraine.) Show that $$\frac{1}{\sqrt{1}+\sqrt{3}}+\frac{1}{\sqrt{5}+\sqrt{7}}+\cdots+\frac{1}{\sqrt{9997}+\sqrt{9999}}>24.$$
%Add in the terms $1/(sqrt(3)+sqrt(5))$ etc, which at most doubles the LHS, and then rationalize denominators.  The sum now telescopes, and you can check that the resulting term is greater than 48.

\item (IMO 1996.)  Show that $$\frac{1}{\sin 2x}+\frac{1}{\sin 4x}+\cdots+\frac{1}{\sin 2^nx}=\cot x-\cot 2^nx.$$
%It turns out that 1/sin(2x)=cot(x)-cot(2x) and so the sum telescopes.

\item (Math Olympiad Challenges.) Prove the identity $$\sum_{k=1}^n \tan^{-1}\frac{1}{2k^2}=\tan^{-1}\frac{n}{n+1}$$
%By inverse of tan subtraction formula, show that the kth term is $\tan^{-1}(2k+1)-\tan^{-1}(2k-1)$ and the sum now telescopes.  Then use tan subtraction again at the end.
\end{enumerate}

\end{document}


%!TEX root = TTK4215-Summary.tex
\section{Parameter estimation}

\subsection{SPR Lyapunov method}
Based on choosing an adaptive law so that a \emph{Lyapunov-like} function guarantees $\tilde{\theta} \rightarrow 0$. The parametric model $z = W(s) \theta^{*\T} \psi$ is rewritten $z = W(s) L(s) \theta^{*\T} \phi$, with $L(s)$ a proper stable t.f., and $W(s)L(s)$ a proper SPR t.f.

\begin{gather}
	z = W(s) L(s) \theta^{*\T} \phi \\
	\hat{z} = W(s)L(s) \theta\T \phi \\
	\epsilon = z - \hat{z} - W(s) L(s) \epsilon n_s^2 \\
	\dot{\theta} = \Gamma \epsilon \phi
\end{gather}

\subsection{Gradient method}
\begin{gather}
	z = \theta^{*\T} \phi \\
	\hat{z} = \theta\T \phi \\
	\epsilon = \frac{z - \hat{z}}{m^2}
\end{gather}

\subsubsection{Instantaneous cost}
\begin{gather}
	\dot{\theta} = \Gamma \epsilon \phi
\end{gather}

\subsubsection{Integral cost}
\begin{gather}
	\dot{\theta} = - \Gamma (R \theta + Q) \\
	\dot{R} = - \beta R + \frac{\phi \phi\T}{m^2} \\
	\dot{Q} = - \beta Q - \frac{z \phi}{m^2}
\end{gather}

\subsection{With projection}
\begin{gather}
	\dot{\theta} =
	\begin{cases}
		\Gamma \epsilon \phi & \mbox{if } \theta \in \mathcal{S}^0 \\
		\Gamma \epsilon \phi - \Gamma \frac{\nabla g \nabla g\T}{\nabla g\T \Gamma \nabla g} \Gamma \epsilon \phi & \mbox{otherwise}
	\end{cases}
\end{gather}

\subsection{Least squares}
\begin{gather}
	z = \theta^{*\T} \phi \\
	\hat{z} = \theta\T \phi \\
	\epsilon = \frac{z - \hat{z}}{m^2}
\end{gather}

\subsubsection{Pure least squares}
\begin{gather}
	\dot{\theta} = P \epsilon \phi \\
	\dot{P} = - P \frac{\phi \phi\T}{m^2} P
\end{gather}

\subsubsection{With covariance resetting}
\begin{gather}
	\dot{\theta} = P \epsilon \phi \\
	\dot{P} = - P \frac{\phi \phi\T}{m^2} P, \quad P(t_r^+) = P_0 = \rho_0 I
\end{gather}

\subsubsection{With forgetting}
\begin{gather}
	\dot{\theta} = P \epsilon \phi \\
	\dot{P} =
	\begin{cases}
		\beta P - P \frac{\phi \phi\T}{m^2} P & \mbox{if } ||P(t)|| \leq R_0 \\
		0                                     & \mbox{otherwise}
	\end{cases}
\end{gather}

TTK4215 System identification and adaptive control/sec-estimators.tex


\section{Surrogate Loss Function}
\frame{\tableofcontents[currentsection, hideothersubsections]}

\begin{frame}
\frametitle{Surrogate Loss Fn: Intro}

WHAT:\\
handle some \textbf{non}convex problems
by minimizing ``surrogate'' loss functions that are \text{convex}
\vspace{4mm}

WHY:\\
the natural loss function is not convex, e.g.  $0-1$ loss
\vspace{4mm}

HOW:\\
to upper bound the nonconvex loss function by a convex surrogate loss function
that
\begin{itemize}
    \item are convex
    \item upper bounds the original loss.
\end{itemize}
\end{frame}

\begin{frame}
\frametitle{Surrogate Loss Fn: Example}
In the context of learning halfspaces:\\
Hinge loss as a convex surrogate for the $0-1$ loss
\footnote{{\tiny https://scicomp.stackexchange.com/questions/5628/confusion-related-to-convexity-of-0-1-loss-function}}

\begin{figure}
    \centering
    \includegraphics[scale=0.25]{eq_hinge_loss}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[scale=0.25]{fig_surr_fn}
\end{figure}

\end{frame}


\subsection{EXAFS: $\chi(k)$}
\begin{frame} \frametitle{EXAFS: $\chi(k)$}

  XAFS is an {\RedEmph{interference effect}}, and depends on the
  wave-nature of the photo-electron.  We express the XAFS in terms of
  {\RedEmph{photo-electron wavenumber}}, $k$:

  \begin{center}$ k= \sqrt{\frac{2m(E-E_0)}{\hbar^2}} $\end{center}

  The XAFS dampens quickly with $k$, so $\chi(k)$ is often
  shown weighted by ${k^2}$ or ${k^3}$:

    \begin{center}
      \begin{tabular}{lr}
        \begin{minipage}{48mm}
          {\wgraph{48mm}{general/feo_chi}}
        \end{minipage}
        &
        \begin{minipage}{48mm}
          \wgraph{48mm}{general/feo_chik}
        \end{minipage}
      \end{tabular}
    \end{center}

%   We can model the EXAFS with the {\BlueEmph{EXAFS Equation}}:
%   \vmm

%   \begin{center}
%   \highlightbox{
%     \begin{minipage}{70mm}
%       \[
%       \chi(k) = \sum_j { {\frac{N_j f_j(k) e^{-2k^2\sigma_j^2}}
%           {kR_j^2}} \sin\bigl[ 2kR_j  +  \delta_j(k) \bigr] }
%       \]
%     \end{minipage}
%     }
%     \end{center}

\end{frame}


%!TEX root = ../notes.tex
\section{February 17, 2022}
\subsection{Special Integers}
\subsubsection{Fermat and Mersenne Primes}
We make the observation that many small primes are of the form $2^m\pm 1$ for some natural number m, for example
\[3, 5, 7, 17, 31\]
We deal with the $+1$ and $-1$ cases separately.
\begin{lemma}
	If $2^m + 1$ is prime, then $m = 2^n$ for some $n\geq 0$.
\end{lemma}
\begin{proof}
	We show the contrapositive. Suppose $m$ is not a power of $2$. We write $m=2^n\cdot q$ for some odd $q>1$.

	The polynomial
	\[f(t) = t^q + 1\]
	has $t=-1$ as a root, so
	\[f(t) = (t+1)g(t) \quad \text{where $\deg f = q > 1$}\]
	Thus
	\begin{align*}
		x^m+1 & = f(x^{2^n})                                   \\
		      & = (x^{2^n}+1)g(x^{2^n})\text{ where $m > 2^n$}
	\end{align*}
	Plugging in $x=2$ gives
	\[2^{2^n}+1\mid 2^m + 1, \text{ and }2^{2^n}+1 < 2^m+1\]
	so $2^m+1$ is not prime.
\end{proof}
\begin{definition}[Fermat Numbers]
	Numbers of the form $2^{2^n}+1$ are called \ul{Fermat numbers}.

	Fermat numbers that are prime are called \ul{Fermat primes}.
\end{definition}
The first few Fermat numbers happen to be prime: $3, 5, 17, 257, 65537$.

\begin{conjecture*}
	Fermat conjectured that Fermat numbers are prime.
\end{conjecture*}
This is very false! Euler found that
\[2^{2^5}+1 = 641\times 6700417\]

We now turn to Mersenne numbers.

\begin{lemma}
	If $m>1$ and $a^m - 1$ is prime, then $a=2$ and $m$ is prime.
\end{lemma}
\begin{proof}
	Suppose $m$ is composite and, so $m=nk$, $1 < k, n < m$. Then
	\begin{align*}
		a^m - 1 & = (a^k)^m - 1                        \\
		        & = (a^k - 1)(a^{k(n-1)} + \cdots + 1)
	\end{align*}

	This implies that $a^m - 1$ is composite. Hence $m$ had better be prime.

	Now $a^m - 1 = (a-1)(a^{m-1} + \cdots + 1)$, so we further have that $a = 2$.
\end{proof}
\begin{definition}[Mersenne Numbers]
	Integers of the form $2^p - 1$ where $p$ is a prime are called \ul{Mersenne numbers}.

	Mersenne numbers that are prime are called \ul{Mersenne primes}.
\end{definition}
There is a current ongoing search for more Mersenne primes on the internet. Currently, the largest known Mersenne prime (and largest known prime number) is
\[M(82,589,933)\]
That is,
\[2^{82,589,933} - 1\]

Mersenne primes are related to perfect numbers. There is a one-to-one correspondence with Mersenne primes and even perfect numbers.
\begin{definition}[Perfect Number]
	$n\in\ZZ_+$ is called \ul{perfect} if
	\[n = \sum_{\substack{d\mid n \\ d < n}} d\]
\end{definition}
\begin{example}
	We have
	\begin{align*}
		6  & = 1 + 2 + 3          \\
		28 & = 1 + 2 + 4 + 7 + 14
	\end{align*}
\end{example}
\begin{proposition}
	If $n^{2^{p-1}}(2^p - 1)$ where $p\in \ZZ_+$, and $p, 2^p - 1$ are prime, then $n$ is perfect.
\end{proposition}
\begin{proof}
	The function $\sigma(n) = \sum_{d\mid n} d$ is multiplicative. So if
	\[n = 2^{p-1}(2^p - 1)\]
	then
	\[\sigma(n) = \sigma(2^{p-1})\sigma(2^p - 1).\]
	since they are coprime. Now we also
	\begin{align*}
		\sigma(2^{p-1}) & = \frac{2^p - 1}{2 - 1} = 2^{p} - 1 \\
		\sigma(2^p - 1) & = 1 + (2^p - 1) = 2^p
	\end{align*}
	Hence $\sigma(n) = (2^{p-1})\cdot 2^p = 2n$.

	So $n$ is a perfect number.
\end{proof}
\begin{proposition}
	If $n\in \ZZ_+$ is even and perfect, then $n = 2^{p-1}(2^p - 1)$ where $p$ and $2^p - 1$ are both prime.
\end{proposition}
\begin{proof}
	\emph{This is a homework exercise!}
\end{proof}

It is currently conjectured that there are no odd perfect numbers.

\subsubsection{Pseudoprimes and Carmichael Numbers}
Homework 2 includes a problem for which a special case is Wilson's Theorem.
\begin{theorem}[Wilson's Theorem]
	If $p$ is a prime, then
	\[(p-1)! \equiv -1\pmod{p}\]
\end{theorem}
The converse is also true.
\begin{proposition}
	If $n\in\ZZ_+$ where $n \geq 2$ is such that
	\begin{equation}
		(n-1)!\equiv 1\pmod{n} \tag{$*$} \label{eqn:wilson-converse}
	\end{equation}
	then $n$ is prime.
\end{proposition}
We can think of \cref{eqn:wilson-converse} as a rudimentary `primality test'.

However, this is not a great primality test, because factorials are expensive to compute.

Recall Fermat's little theorem.
\begin{theorem}[Fermat's Little Theorem]
	If $p\in \ZZ_+$ is a prime and $a\in \ZZ$, then
	\[a^p\equiv a\mod{p}\]
\end{theorem}
Thus $n\in\ZZ_+$ and
\[a^n\equiv a\mod n\]
for some $a\in \ZZ_+$, then $n$ is composite.
\begin{example}
	If $a = 2$, then
	\[2^n\not\equiv 2\mod n\Rightarrow n=2 \text{ is composite}\]
\end{example}
\begin{ques*}
	We might wonder whether a converse to this holds. Disappointingly, no.
\end{ques*}

\begin{example}
	$2^{10} = 1024 = 1\pmod{341}$, so $2^{341} = (2^{10})^34\cdot 2 = 2\mod 341$.

	But $341 = 11\cdot 31$, so $341$ is composite.
\end{example}
\begin{definition}[Pseudoprime]
	We call $n$ a \ul{pseudoprime} to the \ul{base $a$} if $n$ is composite and happens to satisfy
	\[a^n\equiv a\mod n.\]
\end{definition}
\begin{example}
	$341$ is a pseudoprime to the base $2$.
\end{example}
We might hope that if this test failed for a particular $a$, there exists some other $a$ that can test whether $n$ is composite. However, this is not the case.\footnote{We learn in life to not be too hopeful.}

It is not true that given a composite $n$, there exists an $a\in \ZZ_+$ such that $n$ is not a pseudoprime to the base $a$.
\begin{definition}[Carmichael Numers]
	$n\in \ZZ_+$ is called a \ul{Carmichael number} if $n$ is composite and
	\[a^n\equiv a\mod n,\quad \forall a\in \ZZ\]
\end{definition}
\begin{example}
	The smallest Carmichael number is $561$.
\end{example}
\begin{ques*}
	There are variants on this question? Can you have pseudoprimes that satisfy all but one base?
\end{ques*}

\begin{proposition}
	If a composite number $n$ is \emph{not} a Carmichael number, then at least half of the congruence classes $a\in (\ZZ/n\ZZ)^\times$ are such that $n$ is \emph{not} a pseudoprime to the base $a$.
\end{proposition}
\begin{proof}
	Suppose $n$ is a pseudoprime to the base:
	\[a_1, a_2, \dots, a_r\in (\ZZ/n\ZZ)^\times\]
	and suppose we have some $a$ such that
	\[a^n \not\equiv a\mod n\]
	Then for all $i$,
	\begin{align*}
		(a\cdot a_i)^{n-1} & = a^{n-1}a_i^{n-1}   \\
		                   & \equiv a^{n-1}\mod n \\
		                   & \not\equiv 1\mod n
	\end{align*}
	Thus $n$ is not a pseudoprime to the bases $a\cdot a_1, a\cdot a_2, \dots, a\cdot a_r$.
\end{proof}
\begin{remark}
	The bases for pseudoprimes form a subgroup of the group of units.
\end{remark}
\todo{Someone check me on this.}


\chapter{Asymmetric-key cryptography}
\label{Asymmetric-key cryptography} % For referencing the chapter elsewhere, use \ref{Chapter1}

%----------------------------------------------------------------------------------------

\textit{Asymmetric-key} cryptography (also commonly referred to as \textit{public-key} cryptography) is a system of cryptography which uses \textit{pairs} of keys to
encrypt and decrypt data. The pairs of keys consist of public keys which can be distributed, and private keys which are only known to the owner. Essentially,
asymmetric-key cryptography allows us encrypt and decrypt data where the key for encryption is different than the key for decryption. Knowledge of either
of these keys does not allow us to find the other one \cite{classical_algebra}. The name comes from that fact that we use multiple keys in the process of encryption and decryption
and therefore have \textit{asymmetric} keys.

Suppose, John and Mary want to securely communicate using the scheme above. First, John would produce a encryption key $K_1$ and a decryption key $K_2$. John
would then send Mary the encryption key $K_1$. Mary can now encrypt data using the key $K_1$ and send the result to John who can decrypt her data using his decryption key
$K_2$. Let's now introduce Bob, who intercepts the transmission of the encryption key $K_1$. Bob can now encrypt messages using the key $K_1$ but is unable to decrypt
them---only John can decrypt messages as he is the only one who posses the decryption key $K_2$. If John also wants to send messages to Mary, Mary will have to create
an encryption key $x$ and a decryption key $y$ and then send the encryption key $x$ to John. Now, John can encrypt his messages using the encryption key $x$ (sent by Mary)
and then send the encrypted data to Mary where she will decrypt the message using the decryption key $y$. As you can see, in this scheme, we don't care if a third-party
(such as Bob) gets a hold of one of the encryption keys---hence they could just be \textit{public knowledge} \cite{classical_algebra}.

\section{The Rivest---Shamir---Adleman Scheme}

The \textit{Rivest---Shamir---Adleman} scheme (or \textit{RSA} for short) is one of the first practical asymmetric-key cryptosystems \cite{classical_algebra}.
The cryptosystem is named after it's discoverers \textit{Ron Rivest}, \textit{Adi Shamir}, and \textit{Len Adleman}, who gave a definition on how
asymmetric-key cryptography could be realized \cite{classical_algebra}.

The RSA scheme involves four distinct steps: \textit{key generation}, \textit{key distribution}, \textit{encryption}, and \textit{decryption}

\subsection{Key Generation}

A user who wishes to participate in a session of the scheme must first generate two keys: one public key for encryption and one private key for decryption. To do so,
the user will first select two large prime numbers $p$ and $q$, and then multiply them together to calculate the integer $n = pq$.
If $$\mathlarger{\phi(n) = (p-1)(q-1)}$$ then the user will also select an integer $e > 1$ such that $GCD(e,\phi(n))=1$.
The integer $e$ will be used for the calculation of the encryption key. The user now solves for the equation: $$\mathlarger{ed \equiv 1 \pmod{\phi(n)}}$$

According to the \textit{Linear Congurnece Theorem}, since $GCD(e,\phi(n))=1$, there is exactly one congruence class modulo $\phi(n)$ which satisfies the aforementioned
congruence---or in other words, there is only one integer between $0$ and $\phi(n)$ which satisfies the congruence, let $d$ represent this integer
\cite{classical_algebra}. Now, the pair of integers $(e,n)$ represent the user's public keys, and the pair of integers $(d,n)$ represent the user's private keys.
In other words, we now have the following keys:

\begin{itemize}
    \item $n$---the shared key.
    \item $e$---the public encryption key.
    \item $d$---the private decryption key.
\end{itemize}

\subsection{Key Distribution}

Suppose that John and Mary are communication using the RSA scheme. John must know Mary's public key $e$ to encrypt his message and Mary must use her private key $d$ to
decrypt his message. Therefore, Mary must transmit her $(e,n)$ key pair to John via a reliable (though not secret) route.

\subsection{Encryption and Decryption}

To encrypt the message $M$---first, the sender of the message would lookup the recipient's public key pair $(e,n)$ \cite{classical_algebra}.
Then the sender would compute the cipertext $C$ using the following equation:
$$\mathlarger{M^e \equiv C \pmod{n} \text{\hspace{1em}where\;\;}0 \leq C < n}$$

The sender will then take the ciphertext $C$ and send it to the recipient.

Decryption is the inverse of the encryption processes. The recipient of the message, will decrypt the message using the same equation for encryption but rather using it's
own private decryption key pair $(d,n)$ \cite{classical_algebra}. Decryption can be expressed as
$$\mathlarger{C^d \equiv R \pmod{n} \text{\hspace{1em}where\;\;}0 \leq R < n}$$

The decrypted message $R$ is the same as the original message $M$.

It is very important to note that our message $M$ must be smaller than our $n$. In the case that $M$ is greater than $n$, we must split $M$ into blocks which are smaller
than $n$. For example, if $n=12319$ and we want to encrypt the message $06212626$, we would split the message into two blocks---$0621$ and $2626$, encrypt each block,
and then finally combine the blocks into one block (making sure to maintain the order of the blocks when combining).


\title{Inference of Probabilistic Models}

\subsection{Inference of Probabilistic Models}

This tutorial asks the question: what does it mean to do inference of
probabilistic models? This sets the stage for understanding how to
design inference algorithms in Edward.

\subsubsection{The posterior}

How can we use a model $p(\mathbf{x}, \mathbf{z})$ to analyze some
data $\mathbf{x}$? In other words, what hidden structure $\mathbf{z}$
explains the data? We seek to infer this hidden structure using the
model.

One method of inference leverages Bayes' rule to define the
\emph{posterior}
\begin{align*}
  p(\mathbf{z} \mid \mathbf{x})
  &=
  \frac{p(\mathbf{x}, \mathbf{z})}{\int p(\mathbf{x}, \mathbf{z}) \text{d}\mathbf{z}}.
\end{align*}
The posterior is the distribution of the latent variables
$\mathbf{z}$, conditioned on some (observed) data $\mathbf{x}$.
Drawing analogy to representation learning, it is a probabilistic
description of the data's hidden representation.

From the perspective of inductivism, as practiced by classical
Bayesians (and implicitly by frequentists),
the posterior is our updated hypothesis about the latent variables.
From the perspective of hypothetico-deductivism, as practiced by
statisticians such as Box, Rubin, and Gelman, the posterior is simply
a fitted model to data, to be criticized and thus revised
\citep{box1982apology,gelman2013philosophy}.

\subsubsection{Inferring the posterior}

Now we know what the posterior represents. How do we calculate it? This is the
central computational challenge in inference.

The posterior is difficult to compute because of its normalizing
constant, which is the integral in the denominator.
This is often a high-dimensional integral that lacks an analytic (closed-form)
solution. Thus, calculating the posterior means \emph{approximating} the
posterior.

For details on how to specify inference in Edward, see the
\href{/api/inference}{inference API}. We describe several examples in
detail in the \href{/tutorials/}{tutorials}.

\subsubsection{References}\label{references}


\section{Mayer-Vietoris and Subdivision}
(Is it Meyer-Vietoris or Mayer-Vietoris?) Today is the lecture with a lot of formulae.
\begin{theorem}
Let $\sca$ be a cover of $X$, so that $X=\bigcup_{A\in \sca}\mathrm{Int}(A)$. Then the theorem we're going to prove is this. If $S^\sca_\ast(X)=\sum_{A\in\sca}S_\ast(A)\to S_\ast(X)$ induces an isomorphism in $ H_\ast$. (This is a ``quasi-isomorphism'' of chain complexes.)
\end{theorem}
Because this lecture's full of formulas, I'm going to stand around with a piece of paper in my hand.
\begin{example}
Let $\sca=\{A,B\}$ of X, so that:
\begin{equation*}
\xymatrix{A\cap B\ar[r]^{j_1}\ar[d]^{j_2} & A\ar[d]^{i_1}\\
B\ar[r]_{i_2} & X}
\end{equation*}
Then consider the following diagram:
\begin{equation*}
\xymatrix{0\ar[r] & S_\ast(A\cap B)\ar[r]^{\begin{pmatrix}j_{1\ast} \\ -j_{2\ast}\end{pmatrix}}\ar@{=}[d] & S_\ast(A)\oplus S_\ast(B)\ar[r]\ar[d] & S^\sca_\ast(X)\ar[r]\ar@{=}[d] & 0\\
0\ar[r] & S_\ast(A)\cap S_\ast(B)\ar[r] & S_\ast(A)\oplus S_\ast(B)\ar[dr]\ar[r]^{(i_{1\ast},\ i_{2\ast})} & S_\ast(A)+S_\ast(B)\ar[r]\ar@{^(->}[d] & 0\\
 & & & S_\ast(X) &}
\end{equation*}
The map $S_\ast(A)+S_\ast(B)\hookrightarrow S_\ast(X)$ is a quasi-isomorphism, this is what locality says. Take the homology of this to get a lexseq:
\begin{equation*}
\xymatrix{ \cdots\ar[rr] & & H_{n+1}(X)\ar[dll]_{\begin{pmatrix}j_{1\ast} \\ -j_{2\ast}\end{pmatrix}}\\
 H_n(A\cap B)\ar[r] & H_n(A)\oplus H_n(B)\ar[r]^{(i_{1\ast},\ i_{2\ast})} & H_n(X)\ar[dll]\\
 H_{n-1}(A\cap B)\ar[r] & H_{n-1}(A)\oplus H_{n-1}(B)\ar[r] & \cdots}
\end{equation*}
Voila, you have Mayer-Vietoris. (I have a different proof of this that I submitted in homework.)
\end{example}
\subsection{The cone construction}
Let $X\subseteq \mathbf{R}^N$ be a star-shaped region, and let $b\in\mathbf{R}^N$. Then we showed that the augmentation $S_\ast(X)\xrightarrow{\epsilon}\Z$ is a chain homotopy equivalence. There's another map going backwards $\Z\xrightarrow{\eta_b} S_\ast(X)$ sending $1\mapsto c^0_b$. Clearly the composition $\epsilon\circ\eta_b$ is the identity. We want to show that $\eta_b$ ad $\epsilon$ are chain homotopy inverses to each other. One direction is easy. The other map $S_\ast(X)\xrightarrow{\eta_b\epsilon}S_\ast(X)$ being homotopic to $1_{S_\ast(X)}$ is a little harder. This means that we want to construct a map $b\ast:S_n(X)\to S_{n+1}(X)$ such that $db\ast+b\ast d=1-\eta_b\epsilon$.

Consider some $\sigma:\Delta^1\to X$. Then because $X$ is star shaped, you can send $\sigma$ to $b$. This gives a $2$-simplex $b\ast \sigma$, called the \emph{join}. We'll define this $2$-simplex and label it so that the zero vertex is $b$ itself, the $1$ vertex is $d_1\sigma$, and the $2$ vertex is $d_0\sigma$. Define $b\ast\sigma$ as follows (where $(t_0,\cdots,t_{n+1})\in \Delta^{n+1}$):
\begin{equation*}
b\ast\sigma(t_0,\cdots,t_n,t_{n+1})=t_0b + (1-t_0)\sigma\left(\frac{t_1,\cdots,t_{n+1}}{1-t_0}\right)
\end{equation*}
When $t_0=0$, then you recover exactly $\sigma(t_1,\cdots,t_{n+1})$ and when $t_0=1$, this is exactly $b$. (Why can you divide by $1-t_0=0$?) This is a map $b\ast:\Sin_n(X)\to\Sin_{n+1}(X)$, so we can extend linearly to get $S_n(X)\to S_{n+1}(X)$, also denoted $b\ast$. What is $d_i(b\ast\sigma)$? This is exactly:
\begin{equation*}
d_i(b\ast\sigma)=\begin{cases}\sigma & i=0 \\
c^0_b & i=1,n=0\\
b\ast d_{i-1}\sigma & i>0,n>0\end{cases}
\end{equation*}
The latter thing seems true because in the case when $n=1$, $d_2(b\ast\sigma)$ is the cone on $d_1\sigma$. The middle thing is true because when $n=0$ you can't use the bottom thing (what is the boundary in that case?), and if you draw this out, noting our convention that when $t_0=1$ you have $d_1\sigma$ and when $t_0=1$ you have $b$, this automatically yields $d_1(b\ast\sigma)=b$ if $\sigma:\Delta^0\to X$. We can rewrite this as follows. Here $c\in S_n(X)$.
\begin{equation*}
d_i(b\ast c)=\begin{cases}
c & i=0\\
b\ast d_0c + \eta_b\epsilon c & i=1\\
b\ast d_{i-1}\sigma & i>1
\end{cases}
\end{equation*}
Because $d_0$ of a $0$-simplex is defined to be zero. This may seem confusing, but it's just a translation of what we wrote down above. We want to compute that this thing is actually a chain homotopy. Let's compute.
\begin{align*}
d(b\ast c)& = d_0(b\ast c) - d_1(b\ast c) + \sum_{i>1}(-1)^i d_i(b\ast c)\\
& = c-(b\ast d_0c + \eta_b\epsilon c) + \sum_{i=2}^n (-1)^ib\ast d_{i-1}c\\
& = c-\eta_b\epsilon c - \sum_{j=0}^{n-1}(-1)^jb\ast d_jc\\
& = c-\eta_b\epsilon c - b\ast dc
\end{align*}
Here $j=i-1$. The equality $\sum_{j=0}^{n-1}(-1)^jb\ast d_jc=b_\ast dc$ holds because $b\ast$ is linear (by definition on $S_n(X)$). This means that $b\ast$ is a chain homotopy, QED. This completes what we've claimed about the star shaped region. We want to use this cone construction to talk about subdivision.
\subsection{Subdivide the standard simplex}
Let's focus on the standard simplex. This is a nice thing about singular homology. For the $1$-simplex, you just cut in half. For the $2$-simplex, just look at the subdivision of each face, and look at the barycenter\footnote{The barycenter of the $n$-simplex is $b_n:=\frac{(1,\cdots,1)}{n+1}$.}, and join the barycenter to the $1$-simplex between each ``half'' $1$-simplex. We want to formalize this process. Define a natural transformation $\$:S_n(X)\to S_n(X)$ by defining on standard $n$-simplex, namely by specifying what $\$(\iota_n)$ is where $\iota_n:\Delta^n\xrightarrow{\mathrm{id}}\Delta^n$, and then extending by naturality (namely $\$(\sigma)=\sigma_\ast\$(\iota_n)$). Here's the definition. When $n=0$, define $\$=\mathrm{id}$, i.e., $\$(\iota_0)=\iota_0$. For $n>0$, define $\$\iota_n:=b_n\ast\$ d\iota_n$ where $b_n$ is the barycenter of $\Delta^n$. This makes a \emph{lot} of sense if you draw out a picture, and it's a very clever definition that captures the geometry we described. Let me tell you what we'll prove about this, most likely on Wednesday.
\begin{prop}
$\$$ is a chain map $S_\ast(X)\to S_\ast(X)$, i.e., $\$d=d\$$. Also, $\$\simeq 1$.
\end{prop}
Also, class is cancelled on Friday.


\subsection{Estimating HMMs with the Viterbi algorithm}

Assume we know transition matrix. and starting probls

Given we observe sequence of outputs, what were most likely actual paths?

Virbiti returns this


\section{Planes}

If $(x,y,z)$ is a point on the plane, then given $\textbf{P}_0=(x_0,y_0,z_0)$, $(x-x_0,y-y_0,z-z_0)$ is a vector on the plane perpendicular to $\textbf{n}$, the
normal vector. Thus, $(A,B,C)\cdot(x-x_0,y-y_0,z-z_0)=0$ where $A,B,C$ are vector coordinates of $\textbf{n}$. With expansion:

\begin{align*}
    A(x-x_0)+B(y-y_0)+C(z-z_0)&=0\\
    Ax+By+Cz=Ax_0+By_0+Cz_0&=0\\
    Ax+By+Cz=\textbf{n}\cdot \textbf{P}_0
\end{align*}

Note that $(A,B,C)$ form coordinates of $\textbf{n}$.

To find a plane containing 3 points $\textbf{v}_1,\textbf{v}_2,\textbf{v}_3$, compute, for example $\textbf{c}_1=\textbf{v}_3-\textbf{v}_1$ and $\textbf{c}_2=\textbf{v}_2-\textbf{v}_1$.
This finds 2 vectors in the plane. Then compute $\textbf{c}_1\times \textbf{c}_2=\textbf{n}$. \newline

\noindent
The trace of a plane is the intersection of a plane $\mathcal{P}$ with $xy$, $xz$, or $yz$ coordinate planes. Can be found by setting respective variable to 0.

\subsection{Cross-product rules and identities}

\begin{center}
    \includegraphics[]{figures/cross-product}
\end{center}

Overview
\begin{itemize}
    \item $||\textbf{a}\times \textbf{b}||=||\textbf{a}||||\textbf{b}||\sin\theta$
    \item $\textbf{a},\textbf{b}\perp \textbf{a}\times \textbf{b}$
\end{itemize}

Algebraic
\begin{itemize}
    \item $\textbf{a}\times \textbf{b}=\textbf{0}$
    \item $\textbf{a}\times \textbf{b}=-\textbf{b}\times \textbf{a}$
    \item Distributive properties hold -- preserve direction however
    \item $(\alpha \textbf{a})\times \textbf{b}=\alpha(\textbf{a}\times \textbf{b})$
\end{itemize}

\section{Graphs}

\subsection{Multivariable functions}

Function of $n$-variables is real-valued function with $f(x_1,\cdots,x_n)$ with domain $\mathcal{D}$
being a set of $n$-tuples $(x_1,\cdots,x_n)$ in $\R^n$, or where $f$ is defined.
Range of $f$ is all values $f(x_1,\cdots,x_n)$ for $(x_1,\cdots,x_n)$ in the domain.

\subsection{Graphing multivariable functions}

Traces are 2D curves obtained by intersection with planes parallel to coordinate plane.
\begin{itemize}
    \item Horizontal trace at height $c$ -- intersection of graph with plane $z=c$, so points $(x,y,c)$ such that $f(x,y)=c$
    \item Vertical trace in plane $x=a$ -- intersection of graph with vertical plane $x=a$ for all points $(a,y,f(a,y))$
    \item Vertical trace in plane $y=b$ -- intersection of graph with vertical plane $y=b$ for all points $(x,b,f(x,b))$
\end{itemize}

\begin{center}
    \includegraphics[scale=0.15]{figures/saddle-plot.png}
\end{center}

Saddle surface general form is $f(x,y)=x^2-y^2$. The horizontal traces are hyperbolas of the form $c=x^2-y^2$.
Vertical traces are parabolas, as either $x,y$ set to 0.\newline

\noindent
Linear functions in 2 variables are of the form $f(x,y)=mx+ny+r|m,n,r\in \R$.

\subsection{Contour maps and level curves}

\begin{center}
    \includegraphics[scale=0.15]{figures/contour-map.png}
\end{center}

Can specify a contour interval for each $z=c$ value. Is a 2D representation of
level curves of $f(x,y)$ at an interval. Going along level curve means change in altitude is
0. Altitude has change of $\pm m$ (contour interval) when going up/down contour levels. Average ROC is $\Delta \text{elevation}/\Delta \text{distance}$.
Path of steepest ascent follows the shortest possible segment from one contour line to another and always points in steepest direction.

\begin{center}
    \includegraphics[scale=0.3]{figures/steepest-ascent.png}
\end{center}

\section{Partial Derivatives}

\subsection{Definition}

If $f:\R^2\rightarrow \R$ is given by $f(x,y)=z$ abd $P_0=(a,b)$ is a point in the domain of $f$,
then the partial derivative are:
\begin{itemize}
    \item If $h:\R\rightarrow \R$ by $h(t)=f(t,b)$, then partial derivative with respect to $x$ at $P_0$
    is $h\,'(a)$ with following limit definition

    \[\left.\frac{\partial f}{\partial x}\right|_{(a, b)}=f_{x}(a, b)=\lim _{h \rightarrow 0} \frac{f(x+h, b)-f(x, b)}{h}=\lim _{x \rightarrow a} \frac{f(x, b)-f(a, b)}{x-a}\]

    \item If $g:\R\rightarrow \R$ by $g(t)=g(t,b)$ then partial derivative with respect to $y$ at $P_0$ is $g\,'(b)$ with following limit definition

    \[\left.\frac{\partial f}{\partial y}\right|_{(a, b)}=f_{y}(a, b)=\lim _{h \rightarrow 0} \frac{f(a, x+h)-f(a, b)}{h}=\lim _{y \rightarrow b} \frac{f(a, y)-f(a, b)}{y-b}\]

\end{itemize}

Can be thought of as the intersection of the plane shifted by $b$ with $f$, and the derivative of the resulting trace.

\subsection{Linear approximation with planes}

Let $z=f(x,y)$ be a scalar-valued function in $\R^2$ and $P_0=(a,b)$ be a point in domain of $f$. Can have 2 slope vectors representing partial derivatives:
$(1,0,f_x(a,b))$ and $(0,1,f_y(a,b))$. Can find a linear approximation by finding set of points in plane spanned by these vectors passing through
$(a,b,f(a,b))$.

\[\textbf{n}=(1,0,f_x(a,b))\times(0,1,f_y(a,b))=(-f_x(a,b),-f_y(a,b),1)\]

Building the plane:

\begin{align*}
    (x-a, y-b, z-f(a, b)) \cdot \textbf{n} &=0 \\
    (x-a, y-b, z-f(a, b)) \cdot\left(-f_{x}(a, b),-f_{y}(a, b), 1\right) &=0 \\
    -f_{x}(a, b)(x-a)-f_{y}(a, b)(y-b)+z-f(a, b) &=0
\end{align*}

Thus,

\[z=f(a, b)+f_{x}(a, b)(x-a)+f_{y}(a, b)(y-b)\]

\subsection{Higher-order derivatives}

Can be calculated using derivatives of $f_x$ and $f_y$. Notation:

\[f_{xx}=\frac{\partial}{\partial x}(\frac{\partial f}{\partial x}),\; f_{yy}=\frac{\partial}{\partial y}(\frac{\partial f}{\partial y})\]

Can also have mixed partials (read as with respect to $x$ or $y$):

\[f_{xy}=\frac{\partial}{\partial y}(\frac{\partial f}{\partial x}),\; f_{yx}=\frac{\partial}{\partial x}(\frac{\partial f}{\partial y})\]

By Clairaut's Theorem, if $f_{xy}$ and $f_{yx}$ are both continuous functions on a disk $D$, then $f_{xy}(a,b)=f_{yx}(a,b)\: \forall (a,b)\in D$. Means that $f_{xyxy}=f_{xxyy}=f_{yyxx}=f_{yxyx}$.

\section{Extrema}

\subsection{Definition and proofs}

A function $f$ has:
\begin{itemize}
    \item local maximum at $P_0$ in domain if $f(P_0)>f(x,y)\;\forall\;(x,y)$ sufficiently near $P_0$
    \item local minimum at $P_0$ in domain if $f(P_0)<f(x,y)\;\forall\;(x,y)$ sufficiently near $P_0$
\end{itemize}

Sufficiently near: positive radius $R$ used to build a circle centered at $P_0$ that traps points in domain with desired property (min or max).
Global extrema redefine sufficiently near as in the domain of $f$.

Critical point is defined as either of the following:
\begin{itemize}
    \item $f_x(P_0)=f_y(P_0)=0$
    \item either $f_x(P),f_y(P)$ does not exist
\end{itemize}

Method: find critical value $y$ from $f_y$ and $x$ from $f_x$ through cross-substitution.

Proof that if $f_x(P),f_y(P)$ both exist and there is a local max at $P$, then both partials are 0:

Define $g:\;\R\rightarrow \R$ to be single variable function from holding $y=b$ in $f$. Considering points sufficiently near:

\includegraphics[scale=0.5]{figures/Screen Shot 2021-03-21 at 7.55.34 PM.png}

Plugging into function $g$: $g(x) < f(x,b) \leq f(P_0)=g(a)$, so for $x$-values sufficiently
near $a$, $g(x)\leq g(a)$, so it is a local max of $g$. Invoking theorem of extrema, either $g'(a)=0$ or DNE. As $g'(a)=f_x(P_0)$, demonstrates that
$f_x=0$ or DNE for a local max. Same can be done for $f_y$.

Can minimize a function's distance to origin through distance formula. As a square root minimizes
at its argument, can just concentrate on minimizing the argument.

Reiteration: If $f: \mathbb{R}^{2} \rightarrow \mathbb{R}$ has continuous first and second order partial derivatives then $f_{x y}(x, y)=f_{y x}(x, y)$.
Continuous 2nd order partials: $C^2$; continous $n$ order partials: $C^\infty$.

\textbf{Fermat's theorem proof (same as above):} If $f(x,y)$ has a local min/max at $P=(a,b)$, then $P=(a,b)$ is a critical point of $f(x,y)$.

Assuming $f(x,y)$ has a local min at $P$, then this means that $f(x,y) \geq (a,b)$ for $(x,y)$ in the surrounding disk $D(r,P)$.
For some $y=b$, the distance between any 2 $x$-values must be contained in the disk: $|x-a|<r$. Shows that $g(x)=f(x,b)$ has a local min
at $x=a$, so $g'(a)=0$ or DNE. Because $g'(a)=f_x(a,b)$, $f_x(a,b)$ is either 0 or DNE. As the same can be said for $f_y$, $P$ is a critical point.

\subsection{Second-derivative test}

Given a function $f:\R^2 \rightarrow \R$ that has continuous 2nd partials near a critical point $P$, define
the discriminant of $f$ at $P$ to be:

\[D(P)=f_{x x}(P) f_{y y}(P)-\left(f_{x y}(P)\right)^{2}\]

Then:
\begin{itemize}
    \item If $D>0$, $P$ is a local extreme of $f$
    \begin{itemize}
        \item $f_{xx}(P)>0$ implies a local min
        \item $f_{xx}(P)<0$ implies a local max
    \end{itemize}
    \item If $D<0$ then there is neither a min or max at $P$, but an inflection point (saddle point)
    \item If $D=0$ then there is no information about $P$
    \item If $D>0$ and $f_{xx}(a,b)>0$, then local minimum
    \item If $D>0$ and $f_{xx}(a,b)<0$, then local maximum
\end{itemize}

Observe that $D(P)=\operatorname{det}\left(\begin{array}{ll}
                f_{x x}(P) & f_{x y}(P) \\
                f_{y x}(P) & f_{y y}(P)
            \end{array}\right)$

\subsection{Global extrema}

If $f:\R^2 \rightarrow \R$ is continuous on a closed and bounded subset of $\R^2$ then it has a global max/min
on the subset (at critical point or along boundary)

A closed subset of $\R^2$ is one that contains the boundary. Boundary points have the property that
any circle centered around them with positive radius will contain points in and out of the subset.
Bounded subset is where any distance between 2 points in the set never exceeds some fixed bound $M\in \R$.

\begin{center}
    \includegraphics[scale=0.5]{figures/Screen Shot 2021-03-22 at 3.01.05 PM.png}
\end{center}

The boundary curve is denoted as $\partial D$ where $D$ is a disk. By parameterizing $\partial D$ and using function
composition to take this curve $\textbf{c}(t)$ into $f$: $h(t)=f(\textbf{c}(t))$. A min/max for $f$ along boundary curve
is a min/max of $h$. Plug resulting coordinates into $f$ and determine global extrema.

\section{Lagrange Multipliers}

\subsection{Theory}

Let $f:\R^2\rightarrow \R$ and $\textbf{c}:\R\rightarrow \R^2$ such that there is a composite function $h(t)=f(\textbf{c}(t))$.
Then, $h\,'(t)=f_x(\textbf{c}(t))x\,'(t)+f_y(\textbf{c}(t))y\,'(t)=\left(f_{x}\left(\textbf{c}(t)), f_{y} \textbf{c}(t)\right) \cdot \textbf{c}^{\prime}(t)\right.$.
This brings definition of gradient vector, so that for $f:\R^2\rightarrow \R$, each point in domain is in domain of $f$ the vector $\left(f_{x}\left(P_{0}\right), f_{y}\left(P_{0}\right)\right)$:

\[\nabla f\left(P_{0}\right)=\left(f_{x}\left(P_{0}\right), f_{y}\left(P_{0}\right)\right)\]

Rewriting chain rule:

\[\frac{d}{dt}f(\textbf{c}(t_0))=\nabla f(\textbf{c}(t_0))\cdot \textbf{c}\,'(t_0)\]

Let $g(x,y)$ be boundary curve function and $f(x,y)$ be original.

Use a boundary curve which is level curve $c=0$ which represents the boundary of the subset where global extrema can be found.
If $f$ achieves maximum at point $P$ along curve, then either $g_x(P)=g_y(P)=0$ ($\nabla g(P)=\textbf{0}$) or there is a scalar $\lambda$ (Lagrange multiplier)
such that $f_x(P)=\lambda g_x(P)$ and $f_y(P)=\lambda g_y(P)$ ($\nabla f(P)=\lambda \nabla g(P)$). Can find locations of global extreme without parameterizing.

\subsection{Proof}

Suppose $f(P)\geq f(x,y)\;\forall\;(x,y)$ satisfying $g(x,y)=0$ where $g$ is the bounded constraint. Let $\textbf{c}$ be its parameterization where $\textbf{c}\,'(0)\neq 0$. Parameterizing $g(x,y)=0$ with $\textbf{c}(t)$
such that $\textbf{c}(0)=P$: Observe that $h(t)=f(\textbf{c}(t))$ is from $\R\rightarrow\R$ with local max at $t=0$.
Thus, $h'(t)=0$ or DNE. Taking derivative, $h'(t)=\nabla f(\textbf{c}(t))\cdot \textbf{c}\,'(t)$.
Then, $h^{\prime}(0)=\nabla f(\textbf{c}(0)) \cdot \textbf{c}^{\prime}(0)=0$ or DNE. This dot product is either 0 or DNE because
1 or both vectors could not exist, or if they are $\perp$. Thus, $\nabla f(P)\perp \textbf{c}\,'(0)$ and $\textbf{c}\,'(0)\neq \textbf{0}$ (because $\textbf{c}$ is assumed to be regular).

Showing $\nabla g(P)\perp \textbf{c}\,'(0)$. Define $j(t)=g(\textbf{c}(t))$. Thus, $j^{\prime}(t)=\nabla g(\textbf{c}(t)) \cdot \textbf{c}^{\prime}(t)$.
Because $\textbf{c}(t)$ is a parameterization of level curve $g(x,y)=0$, it always outputs 0, so $j(t)=g(\textbf{c}(t))=0$ always. Can the conclude that
$j^{\prime}(t)=\nabla g(\textbf{c}(t)) \cdot \textbf{c}^{\prime}(t)=0$. Plugging in $t=0$ gives $\nabla g(\textbf{c}(0)) \cdot \textbf{c}^{\prime}(0)=0$ so $\nabla g(P)\perp \textbf{c}\,'(0)$.

Because both vectors are perpendicular to $\textbf{c}\,'(0)$, they are parallel -- $\nabla f(P)=\lambda \nabla g(P)$. Or:

$$\begin{array}{l}
    f_{x}(P)=\lambda g_{x}(P) \\
    f_{y}(P)=\lambda g_{y}(P)
\end{array}$$