\documentclass[11 pt]{scrartcl}
\usepackage[header, margin, koma]{tyler}

\newcommand{\hwtitle}{Calculus and Probability Review}

\pagestyle{fancy}
\fancyhf{}
\fancyhead[l]{\hwtitle{}}
\fancyhead[r]{CS 70 Staff}
\cfoot{\thepage}

% TODO: add a joint pdf example to the double int. section

\begin{document}
\title{\Large \hwtitle{}}
\author{\large CS 70 Staff}
\date{\large\today}

\maketitle

This is a quick refresher of single and multi variable calculus in the context of probability. If any material needs refreshing, \cite{stewart} is the canonical calculus textbook, while Khan Academy has fantastic video explanations. Finally, this is a reminder that \textbf{calculus will not be over-emphasized on the final}; after all, this is a probability course, not a calculus course.
%\tableofcontents

\section{Derivatives}
Single variable calculus starts with the notion of a derivative, which represents the \emph{instantaneous} rate of change of a function $f(x)$. Formally,

\begin{definition}[Derivative]
    The \textbf{derivative} of a function $f(x)$ is defined to be
    \[ f'(x) = \lim_{h\to 0} \dfrac{f(x+h)-f(x)}{h}.\]
    We sometimes also denote it by $\frac{df}{dx}$ or $\frac{d}{dx} f(x)$ to emphasize that the derivative is with respect to $x$.
\end{definition}

The idea is that this value gives us the slope of the secant line passing through $(x, f(x))$ and $(x+h, f(x+h))$, and as $h$ becomes smaller and smaller, the secant line becomes the tangent line at the point $x$.

This definition is seldomly used once we know the basics of dealing with derivatives though. That comes in the form of the following common forms and properties. Assume $c\in \RR$ is a constant.

\begin{itemize}
    \ii $\frac{d}{dx} c = 0$.
    \ii $\frac{d}{dx} x^n = nx^{n-1}$.
    \ii $\frac{d}{dx} \sin x = \cos x$.
    \ii $\frac{d}{dx} \cos x = -\sin x$.
    \ii $\frac{d}{dx} e^x = e^x$ (the only such function!).
    \ii $\frac{d}{dx} c^x = c^x\ln c$.
    \ii $\frac{d}{dx} \ln x = \frac{1}{x}$.
\end{itemize}

Of course, derivatives add and subtract, and with the next two rules, you're ready to differentiate nearly any function you come across.

\begin{lemma}[Product Rule]
    For any two functions $f(x)$ and $g(x)$,
    \[ \dfrac{d}{dx} (f\cdot g)(x) = f'(x) g(x) + f(x)g'(x) \]
    or in other words,
    \[ (fg)' = f'g + fg'.\]
\end{lemma}

\begin{lemma}[Chain Rule]
    Suppose I have a function $f(x) = g(h(x))$. Then
    \[ f'(x) = g'(h(x))\cdot h'(x).\]
    Equivalently,
    \[ \dfrac{df}{dx} = \dfrac{df}{dh} \dfrac{dh}{dx}.\]
\end{lemma}

In other words, I keep unrolling my derivatives from outside in when I use the chain rule.

\begin{example}
    Compute the derivative of $f(x) = x^2e^{2x}$.
\end{example}
\begin{proof}[Solution]
    Applying the product rule followed by the chain rule gives
    \[ f'(x) = (x^2)'e^{2x} + x^2 (e^{2x})' = 2xe^{2x} + x^2 e^{2x} (2x)' = 2xe^{2x} + 2x^2e^{2x}.\]
\end{proof}

\begin{exercise}
    Find the derivatives of the following functions:
    \alphanum
        \ii $F(x) = 5(5-x)^4$
        \ii $F(x) = e^{-3x}$
        \ii $F(x) = \sqrt{x^3}$
    \enumend
\end{exercise}

\section{Integrals}
\subsection{Definitions}
Complementing the derivative is first the \emph{definite} integral, which represents the signed\footnote{Meaning positive if we're above the $x$-axis and negative otherwise.} area under the curve. The nice interpretations and motivations for this come from physics, so I'll move on and give an informal definition.

\begin{definition}[Definite Integral]
    The \textbf{definite integral} $\int_a^b f(x) dx$ is the signed area of the region of the $xy$-plane bounded by the graph of $f$, the $x$-axis, and the vertical lines $x = a$, $x = b$.
\end{definition}

\begin{figure}[!htb]
    \centering
    \includegraphics[scale=0.5]{def_integral.png}
    \caption{An example of such an integral.}
\end{figure}

It turns out that integration and differentiation are inverse operations. So we have the notion of an \emph{antiderivative}, a function $F$ whose derivative is a given function $f$, which is also sometimes called an \emph{indefinite} integral.

\begin{definition}[Indefinite Integral]
    An antiderivative $F(x)$ of $f(x)$ is called an \textbf{indefinite integral}, and written as
    \[ F(x) =  \int f(x) dx.\]
\end{definition}

One important point to note is that since constants vanish under differentiation, given an antiderivative $F(x)$, any other function $F(x) + c$ where $c$ is a constant is also an antiderivative. Never forget to plus $c$!

In the midst of all this back and forth, it's still not clear at all how we'd compute the area under the curve in order to actually find the definite integral. This is where the Fundamental Theorem of Calculus comes into play; it creates the connection between the definite and indefinite integrals (and therefore between definite integrals and derivatives).

\begin{theorem}[Fundamental Theorem of Calculus]
    If $f$ is a real-valued, integrable function on $[a,b]$, and $F$ is an antiderivative of $f$ on $[a,b]$ (so that $F' = f$), then
    \[ \int_a^b f(x) dx = F(b) - F(a).\]
\end{theorem}

This theorem is paramount for going back and forth between integrals and derivatives. If you think about it, this theorem is what lets us use CDFs to evaluate our pdfs. So under the surface, problems like Example~\ref{example:ft_rv} are really using the fundamental theorem somewhere.

Here are some common integrals you should know:
\begin{itemize}
    \ii $\int x^n\; dx = \frac{x^{n+1}}{n+1} + C$ for $n \not= -1$.
    \ii $\int \frac{1}{x}\; dx = \ln |x| + C$
    \ii $\int e^x\; dx = e^x + C$
    \ii $\int \sin x\; dx = - \cos x + C$
    \ii $\int \cos x\; dx = \sin x + C$
\end{itemize}

\begin{example}
    Compute $\int_0^4 x^2 dx$.
\end{example}
\begin{proof}[Solution]
    By inspection (or by looking at the list above) we find that an antiderivative is $F(x) = \frac{x^3}{3}$, and so by the fundamental theorem, our answer is just $F(4) - F(0)$. Written more concisely,
    \[ \int_0^4 x^2 dx = \dfrac{x^3}{3}\Big\rvert^4_0 = \dfrac{4^3}{3} - \dfrac{0}{3} = \dfrac{64}{3}.\]
\end{proof}

\subsection{Integrating techniques: $u$-sub, integration by parts, and others}
There's two common techniques you'll use when solving integrals. The first is pretty simple. It's utilizing a change of variables, otherwise known as $u$-substitution.

\begin{example}
    Compute $\int_0^5 e^{-6x}dx$.
\end{example}
\begin{proof}[Solution]
    This isn't an integral of one of the above forms since we have extra coefficients in the exponent, so we'll perform a change of variables to get it to look like it. Let $u = -6x$, so $du = -6 dx$ or $-\frac 16 du = dx$. Now if we change variables, we will get an antiderivative of
    \[\int e^{-6x} dx = \int -\frac 16 e^{u} du = -\frac 16 e^u + C = -\frac 16 e^{-6x} + C.\]

    Plugging this in gives
\[ \int_0^5 e^{-6x} dx = -\frac 16 e^{-6x} \Big\rvert^5_0 = -\frac 16 e^{-30} + \frac 16 .\]
\end{proof}

The second technique is \emph{integration by parts}, which is not as straightforward. The idea is that we want to be able to integrate products of functions that aren't as nice. This reminds us of the product rule for derivatives, which states that if I had two functions $u(x)$ and $v(x)$, then
\[ (uv)' = u'v + uv'.\]

Integrating both sides with respect to $x$ gives
\[ u(x) v(x) = \int (u(x) v(x))' dx  = \int u'(x) v(x) dx + \int u(x) v'(x) dx \]

If we use differentials and let $du = u'(x) dx$ and $dv = v'(x) dx$,\footnote{A differential is change in the function based on the dependent variable (how far up we move based on horizontal movement and the slope).} we can write this in a format thats more recognizable, namely
\[ u(x) v(x) = \int v(x) du + \int u(x) dv \implies \boxed{\int u\; dv = uv - \int v\; du}.\]

So now we can take an integral that's hard to compute, and flip around what we're integrating to make it potentially easier to simplify. It's easiest to understand through an example.

\begin{example}
    Compute $\int xe^{3x}\; dx$.
\end{example}
\begin{proof}[Solution]
    If I'm applying integration by parts, I need to find one expression to take the derivative of (which will be $u$) and another to take the integral of (which will be $dv$) to simplify. Here, I'll let $u = x$ and $dv = e^{3x} dx$, so that $du = dx$ and $v = \frac 13 e^{3x}$. Then applying integration by parts gives
    \[ \int xe^{3x} = \int u\; dv = uv - \int v\; du = \frac 13 xe^{3x} - \int \frac 13 e^{3x} dx = \frac 13 xe^{3x} - \frac 19 e^{3x} + C.\]
\end{proof}

It's not always obvious what I should be setting $u$ and $dv$ to typically. A good rule of thumb is to set $u$ to the first term you see in this list:
\begin{itemize}
    \ii Logarithm
    \ii Inverse trig function
    \ii Algebraic function (e.x. polynomials)
    \ii Trig function
    \ii Exponential function
\end{itemize}

Here's another example.
\begin{example}
    Compute $\int \ln x \; dx$.
\end{example}
\begin{proof}[Solution]
    You might think that we can't apply integration by parts here since we only have one term. But in fact, let $u = \ln x$ and $dv = dx$ so that $du = \frac{1}{x} dx$ and $v = x$. Then
    \[ \int \ln x\; dx = \int u \; dv = uv - \int v\; du = x\ln x - \int x\cdot \frac{1}{x} \; dx = x\ln x - x + C.\]
\end{proof}

You should also search for the tabular method if you want to speed things up when doing multiple integrations by parts (Khan academy is a good place to watch).

\begin{exercise}
    Compute $\int x^2e^{3x}\; dx$.
\end{exercise}

\section{Series}
A quick informal definition so everyone's on the same page here.

\begin{definition}[Series]
    Given some sequence of terms $(a_1, a_2, \dots, a_n)$, the \textbf{series} corresponding to this sequence is the sum
    \[ a_1 + a_2 + \dots + a_n.\]
    If the sequence is infinite, then the series will contain infinitely many summands.
\end{definition}

Series come up often in discrete probability when we ask what the probability of some range of events is. So it's useful to know how to sum the two main types of series and some tricks we can do with them.

There's two main series to know: the geometric series, and the Taylor series.

In a geometric series, we have an initial term $a$, and every term after that is some common ratio, $r$, times the previous terms. So a typical finite geometric series looks something like

\[ S = a + ar + ar^2 + \dots + ar^k\]

where $S$ is the final value of the series. To calculate this, notice that

\[ rS = ar + ar^2 + \dots + ar^k + ar^{k+1},\]

so if we subtract one from the other, all the middle terms cancel out, leaving us with

\[ S - rS = a - ar^{k+1} \implies \boxed{S = \dfrac{a(1-r^{k+1})}{1-r}}.\]

If we take $k \to \infty$, then we get an infinite geometric series, for which
\[ \sum_{i=0}^\infty ar^i = \dfrac{a}{1-r}\]

but only when $|r| < 1$ (otherwise the series won't converge). Here's an example that utilizes geometric series for a very fitting distribution.
\begin{example}
    Show that the sum of the probabilities of a random variable $X \sim \Geo(p)$ is 1.
\end{example}
\begin{proof}[Solution]
    Recall that $\PP(X = k) = (1-p)^{k-1} p$. Then
    \[ \sum_{k=1}^\infty \PP(X=k) = \sum_{k=1}^\infty (1-p)^{k-1} p.\]
    But this is a \emph{geometric} series with initial term $p$ and ratio $1-p$, so the sum is $\frac{p}{1-(1-p)} = 1$.
\end{proof}

The idea of a Taylor series is to approximate a function $f(x)$ by its derivatives. This looks like

\[ f(x) = f(0) + f'(0)x + \dfrac{1}{2!} f''(0) x^2 + \dfrac{1}{3!} f'''(0)x^3 + \dots = \sum_{k=0}^\infty \dfrac{1}{k!}f^{(k)}(0)x^k\]

where $f^{(k)}$ is the $k$th derivative of $f$. The only one you'll need for this class is $e^x$:

\[ e^x = 1 + x + \dfrac{x^2}{2!} + \dfrac{x^3}{3!} + \dots = \sum_{k=0}^\infty \dfrac{x^k}{k!}.\]

One last interesting note is that geometric series are \emph{also} Taylor series. If we let $f(r) = \frac{a}{1-r}$, then the derivatives are

\[ f(r) = \frac{a}{1-r}, \quad\quad f'(r) = \frac{a}{(1-r)^2}, \quad\quad f''(r) = 2!\cdot\frac{a}{(1-r)^3}, \quad\dots,\quad f^{(k)}(r) = k!\cdot \dfrac{a}{(1-r)^{k+1}} \]

so

\[ f(0) = a, \quad\quad f'(0) = 1!\cdot a, \quad\quad f''(0) = 2!\cdot a, \quad\dots,\quad f^{(k)}(0) = k!\cdot a.\]

Plugging this into the Taylor series expansion gives

\[ f(r) = \dfrac{a}{1-r} = a + ar + ar^2 + \dots\]

which is the geometric distribution as expected.

\begin{exercise}
    Show that the sum of the probabilities of a Poisson random variable $X\sim \text{Poisson}(\lambda)$ is 1.
\end{exercise}

\section{Intermission: Calculus and Series in Probability}
Now that we've covered some calculus, let's look at how the different results we've seen so far are used in probability.

Here's two problems that make use of the Taylor series for $e^x$.
\begin{example}[Dis 12 \#3]
    Let $X \sim \Geo(p)$ and $Y \sim \Poisson(\lambda)$ be independent random variables. Compute $\PP(X > Y)$.
\end{example}
\begin{proof}[Solution]
    Let's condition on $Y$ so that we can use $\PP(X > k) = (1-p)^k$. This gives
    \begin{align*}
        \PP(X > Y) &= \sum_{y=0}^\infty \PP(Y=y) \PP(X > y | Y = y) \\
                   &= \sum_{y=0}^\infty \dfrac{e^{-\la}\la^y}{y!}\cdot (1-p)^y \\
                   &= e^{-\la}\sum_{y=0}^\infty \dfrac{(\la(1-p))^y}{y!}
    \end{align*}
    But this sum is the Taylor series for $e^{\la(1-p)}$, so the probability is just $e^{-\la}e^{\la-\la p} = e^{-\la p}$.
\end{proof}

This somewhat surprising result was derived in Lecture 21.
\begin{example}
    Let $X_n$ be a random variable counting the number of fixed points of a permutation on $n$ variables. What is the distribution of $X_n$ as $n\to\infty$?
\end{example}
\begin{proof}[Solution]
    We wish to compute $\PP(X_n = j)$, so we count the number of permutations with $j$ fixed points. We need to choose $j$ points to be fixed and the rest to not have any fixed points, which means they form a derangment, so there are $\binom{n}{j}D_{n-j}$ such permutations. Hence
    \[ \PP(X_n = j) = \dfrac{\binom{n}{j}  D_{n-j}}{n!}.\]

    Recalling that $D_j = j!\sum_{k=0}^j \frac{(-1)^k}{k!}$, we find
    \begin{align*}
        \PP(X_n = j) &= \dfrac{1}{n!}\left[ \dfrac{n!}{j!(n-j)!} (n-j)! \sum_{k=0}^{n-j}\dfrac{(-1)^k}{k!}\right] \\
                     &= \dfrac{1}{j!} \sum_{k=0}^{n-j}\dfrac{(-1)^k}{k!}.
    \end{align*}
    But when $n\to\infty$, this is the Taylor Series for $e^{-1}$, so
    \[ \lim_{n\to\infty} \PP(X_n = j) = \frac{e^{-1} 1^j }{j!} \]
    which is the pdf of a $\Poisson(1)$ distribution.
\end{proof}
This result also agrees with the fact that $\EE[X_n] = 1$, which is true by linearity of expectation.

Here's an idea similar to Discussion 13, Problem 2 but in a different context. Under the hood, you should realize that we can do this cdf manipulation thanks to the Fundamental Theorem of Calculus.
\begin{example}
    Let $X, Y$ be a random variables where $Y = X^2$. Express the pdf of $Y$, $f_Y$, in terms of the pdf of $X$, $f_X$.
    \label{example:ft_rv}
\end{example}
\begin{proof}[Solution]
    The first step should usually be to compute the cdf of $Y$ in terms of $X$'s cdf. This gives
    \[ F_Y(x) = \PP(Y \leq x) = \PP(X^2 \leq x) = \PP(-\sqrt{x} \leq X \leq \sqrt{x}) = F_X(\sqrt{x}) - F_X(-\sqrt{x}).\]
    From here, we differentiate with respect to $X$ (being careful to use the chain rule) and get
    \[ f_Y(x) = \dfrac{dF_Y}{dx} = \dfrac{1}{2\sqrt{x}}F_X'(\sqrt{x}) - \left(-\dfrac{1}{2\sqrt{x}}\right)F_X'(-\sqrt{x})
    = \dfrac{1}{2\sqrt{x}} (f_X(\sqrt{x})+f_X(-\sqrt{x})) \]

\end{proof}

\section{Double Integrals}
Now we're finally talking about multivariable calculus. Before we start, we should ask the question: what even is a double integral?

We should first think about what a single integral represented. In summary, we defined it as
\[ \boxed{\text{Finding an \textcolor{blue}{area} under } f(x) \text{ over a \textcolor{blue}{one-dim. interval} } {\color{blue} [a,b]} \text{ on the } {\color{blue} x \text{-axis: }} A = \int_a^b f(x) \; dx.}\]

The double integral will be the natural generalization of this. Instead of integrating an area over an interval, we will be integrating a volume over a region in 2D space.
\[ \boxed{\text{Finding a \textcolor{red}{volume} under } f(x,y) \text{ over a \textcolor{red}{two-dim. region} } {\color{red} \mathcal{R}} \text{ on the } {\color{red} xy \text{-plane: }} V = \iint_\mathcal{R} f(x,y) \; dA.}\]

The hardest part of solving double integrals is often figuring out the region of integration and setting up the limits. After that, it's just single-variable calculus. Let's look at an example.

\begin{example}
    Integrate $f(x,y) = x^2 + y^2$ over the $4\times 4$ square centered at $(0,0)$.
\end{example}
\begin{proof}[Solution]
Before we begin, take a look at this visualization of our integral.

\begin{figure}[!htb]
    \centering
    \includegraphics[scale=0.5]{x2y2.jpeg}
    \caption{A visualization of the integral of $f(x,y) = x^2 + y^2$ over the square $[-2,2]\times [-2,2]$.}
\end{figure}

As you can tell, we're looking for the volume under the curve. Our region of integration is the region bounded by the lines $x = -2, x = 2, y = -2, y = 2$.

There's two ways in which we can integrate over this region: we can either integrate with respect to $x$ first, or with respect to $y$ first. Either way is the same in this case since everything is symmetric, so I'll set it up as the former.
    This means our integral is
\[ \int_{-2}^2 \int_{-2}^2 x^2+y^2 \; dx \; dy.\]

In order to integrate this, we treat the inside integral like a single integral and treat everything that's not $x$ like a constant. This gives

\[ \int_{-2}^2 \int_{-2}^2 x^2+y^2 \; dx \; dy = \int_{-2}^2 \left(\frac{x^3}{3}+xy^2\right)\Big\rvert^2_{x=-2} \; dy = \int_{-2}^2 \frac{16}{3} + 4y^2 \; dy\]

Now that we've gotten rid of our $x$ variable (as expected), all that's left is to finish the single integral, which gives us

\[ \int_{-2}^2 \frac{16}{3} + 4y^2 \; dy = \left(\frac{16}{3}y + \frac{4}{3} y^3\right)\Big\rvert^{2}_{-2} = \frac{128}{3}.\]
\end{proof}

Here's one where the region of integration is trickier.

\begin{example}
    Integrate $ye^{-x}$ over the \emph{triangle} enclosed by $y = x$, $y = 3$, and $x = 0$.
    \label{ex:ye-x}
\end{example}
\begin{proof}[Solution]
    \begin{figure}[!htb]
        \centering
        \begin{minipage}{0.45\linewidth}
            \includegraphics[scale=0.35]{boundary.png}
        \end{minipage}%
        \begin{minipage}{0.45\linewidth}
            \includegraphics[scale=0.45]{ye-x.png}
        \end{minipage}%
        \caption{The boundary and the integral of $f(x,y) = ye^{-x}$ over the region in Example~\ref{ex:ye-x}.}
    \end{figure}

    We already know the region is a triangle, but we need to figure out how to integrate over it. If we first integrate over $y$ then $x$, then the limits of our outer integral should go from $0$ to 3. In this case, we then have the variable $x$ to work with to set up our limits for $y$. Using the equations, we find that we should integrate from $x$ to $3$. Setting up our integral and simplifying gives

    \begin{align*}
    \int_0^3 \int_x^3 ye^{-x} \; dy \; dx &= \int_0^3 \frac 12 (y^2 e^{-x})\Big\rvert^3_{y = x} \; dx \\
                                          &= \frac 12 \int_0^3 9e^{-x} - x^2e^{-x}\; dx
    \end{align*}

    which would require the use of \emph{two} integrations by parts to evaluate; yuck! Let's see what happens if we set up our integral the other way. The outer limits are still from 0 to 3, but the inner ones are now from 0 to $y$ (convince yourself of this), so the integral is
    \begin{align*}
    \int_0^3 \int_0^y ye^{-x} \; dx \; dy &= \int_0^3  (-y e^{-x})\Big\rvert^y_{x = 0} \; dy  = \int_0^3 -y e^{-y} + y \; dy \\
                                          &= \int_0^3 y \; dy - \left(-ye^{-y}\Big\rvert^3_0 + \int_0^3 e^{-y}\; dy \right) \\
                                          &= \frac{9}{2} - (-3e^{-3} + 1 - e^{-3}) = \boxed{\frac{7}{2} + \frac{4}{e^3}}.
    \end{align*}
\end{proof}

We don't always have to integrate using $x$ and $y$ though. Sometimes it's more natural for us to use \emph{polar} coordinates where we label every point as $(r, \theta)$, $r$ being its distance from the origin and $\theta$ being its angle with the $x$-axis.

\begin{example}
    Integrate $f(x,y) = x^2 + y^2$ over the circle of radius $2$ centered at $(0,0)$.
\end{example}
\begin{proof}[Solution]
    The first step is to translate our function in terms of $r$ and $\theta$. We know that $x^2 + y^2 = r^2$, so $f(r, \theta) = r^2$. Our region is $0\leq r \leq 2$, so our integral is

\[ \int_{0}^{2\pi} \int_{0}^2 r^2 \; dr\; d\theta = \int_0^{2\pi} \frac{r^3}{3} \Big\rvert^2_{r=0} \; d\theta = \int_0^{2\pi} \frac{8}{3} \; d\theta = \dfrac{16}{3}\pi.\]
\end{proof}
We did something similar with Buffon's needle, where we integrated with respect to $y$ and $\theta$. All that matters is that we find the appropriate regions and limits to integrate over.

\begin{exercise}
    Integrate $e^{x+y}$ over the rectangular region determined by the lines $y = 0$, $x = 0$, $y = 3$, $x = 2$.
\end{exercise}
\begin{exercise}
    Integrate $-\frac{xe^x}{y^2}$ over the region defined by the lines $x = 5$, $y = -x$, $y = x$.
\end{exercise}
\begin{exercise}
    Suppose $X \sim \Exp(\la_1), Y \sim \Exp(\la_2)$ are independent exponential distributions so that $f(x,y) = \la_1\la_2 e^{-(\la_1+\la_2)x}$ for $x, y \geq 0$ is the density of their joint distribution.
    Show that the integral of this density is 1. % over its support.\footnote{The \emph{support} a function $f$ are all inputs where its nonzero. For example, the support of a uniform random variable on $[a,b]$ is just $[a,b]$.}
\end{exercise}

%\section{Joint Densities of Continuous Random Variables}

\begin{thebibliography}{12}
	\bibitem{stewart}
		James Stewart, \textit{Calculus 7 ed}. Cengage Learning, 2012.
\end{thebibliography}

\end{document}


\documentclass[11pt]{article}
\usepackage{amsmath}
\usepackage{amsfonts}
\begin{document}

\title{APPM 5510 HW2}
\author{Zane Jakobs}
\date{}
\maketitle

\subsection*{1(a)}
The Bayesian update for the mean of this linear process can be obtained from the Kalman gain matrix
\[
\begin{aligned}
K &= BH^T(HBH^T + \gamma^2)^{-1}\\
&= \dfrac{1}{\sigma_1^2 + \gamma^2} \begin{pmatrix}\sigma_1^2 \\ \sigma_1\sigma_2 r\end{pmatrix}, \\
\end{aligned}
\]
whose action we then apply to the observation error term,
\[
\begin{aligned}
\epsilon &= y - H\mathbf{\mu}\\
&= y - \mu_1.
\end{aligned}
\]
Thus, the Bayesian update is
\[
\begin{aligned}
\begin{pmatrix}\mu_1 \\ \mu_2 \end{pmatrix} \to \ & \begin{pmatrix}\mu_1 \\ \mu_2 \end{pmatrix} + K\epsilon\\
			&= \begin{pmatrix}\mu_1 \\ \mu_2 \end{pmatrix} + \dfrac{(y-\mu_1)}{\sigma_1^2 + \gamma^2} \begin{pmatrix}\sigma_1^2 \\ \sigma_1\sigma_2 r\end{pmatrix}.
\end{aligned}
\]
\subsection*{1(b)} If $r = 0$, then the posterior mean is
\[
\begin{pmatrix}\mu_1 \\ \mu_2 \end{pmatrix} + \dfrac{\sigma_1^2(y-\mu_1)}{\sigma_1^2 + \gamma^2} \begin{pmatrix}1 \\ 0\end{pmatrix},
\]
and the unobserved variable that is now uncorrelated with the observed variable is not updated at all.
\subsection*{1(c(1))} As the measurement variance $\gamma$ goes to infinity, the posterior mean approaches the prior mean, with their difference going to zero as $O(\frac{1}{\gamma^2})$. This makes sense; as the measurement variance gets larger, we should "trust" the observation less, and hence update our beliefs less.
\subsection*{1(c(2))} As the measurement variance goes to zero, the posterior mean becomes
\[
\begin{pmatrix}\mu_1 \\ \mu_2 \end{pmatrix} + \dfrac{(y-\mu_1)}{\sigma_1^2} \begin{pmatrix}\sigma_1^2 \\ \sigma_1\sigma_2 r\end{pmatrix},
\]
and thus the magnitude of the update to the mean depends on only on how uncertain we think the random variable is to begin with, as having zero observation error is essentially equivalent to directly sampling from the random variable.

\subsection*{1(c(3))} In this limit, the posterior mean is
\[
\begin{pmatrix}\mu_1 \\ \mu_2 \end{pmatrix} + \dfrac{(y-\mu_1)}{\sigma_1^2 + \gamma^2} \begin{pmatrix}1 \\ r\end{pmatrix},
\]
so the magnitude of the update goes to zero linearly as $\sigma_1^2$ and $\sigma_2^2$ go to infinity.

\subsection*{2} First, the restriction that $X_a$ has mean $x$ implies that $x = \alpha x + \beta x\implies \ \alpha = 1-\beta.$ We now want to minimize $\mathrm{Var}(X_a) = \alpha^2 \mathrm{Var}(X_b) + \beta^2 \mathrm{Var}(Y) = (1-\beta)^2\sigma_x^2 + \beta^2\sigma_y^2$ subject to that constraint. We differentiate w.r.t. $\beta$ and set the expression equal to zero to get
\[
0 = \dfrac{\partial\mathrm{Var}(X_a)}{\partial\beta} = -2(1-\beta)\sigma_x^2 + 2\beta\sigma_y^2,
\]
which reduces to
\[
\dfrac{\beta}{1-\beta} = \dfrac{\sigma_x^2}{\sigma_y^2} \implies \ \beta = \dfrac{\sigma_x^2/ \sigma_y^2}{1+\frac{\sigma_x^2}{\sigma_y^2}} = \dfrac{\sigma_x^2}{\sigma_y^2 + \sigma_x^2},
\]

and then $\alpha = 1 - \beta = 1 -  \dfrac{\sigma_x^2}{\sigma_y^2 + \sigma_x^2}$. Thus,

\[
X_a  \left( 1 - \dfrac{\sigma_x^2}{\sigma_y^2 + \sigma_x^2}\right) X_b +  \dfrac{\sigma_x^2}{\sigma_y^2 + \sigma_x^2} Y.
\]
The variance of $X_a$ can be obtained from the fact that
\[
\begin{aligned}
\mathrm{Var}(X_a) &= \alpha^2\mathrm{Var}(X_b) + \beta^2 \mathrm{Var}(Y)\\
&= \left( 1 - \dfrac{\sigma_x^2}{\sigma_y^2 + \sigma_x^2}\right)^2 \sigma_x^2 + \left(\dfrac{\sigma_x^2}{\sigma_y^2 + \sigma_x^2}\right)^2\sigma_y^2.
\end{aligned}
\]

\subsection*{3(a)} Bayes' theorem gives us that the posterior pdf $[X | y]$ is (a constant times) $[y | x][X]$, which are easily obtained quantities. Specifically, we have
\[
[y | x] = \exp(- |y - x - 1|)/2
\]
from the definition of $Y$, and the following prior on $X$:
\[
[X] = \dfrac{1}{c}\exp(-\frac{(x - \mu)^2}{\sigma_x^2}).
\]
This gives us the posterior distribution
\[
\begin{aligned}
[X | y] &= \dfrac{1}{c} \exp(- |y - x - 1|)\exp(-\frac{(x - \mu)^2}{\sigma_x^2})\\
&= \dfrac{1}{c}\exp\left(-\frac{(x - \mu)^2}{\sigma_x^2} - |y - x - 1|\right).
\end{aligned}
\]

\subsection*{3 (b)} We seek to maximize $\exp\left(-\frac{x^2}{2} - |0.2 - x |\right)$ over $x$, which is obtained when the absolute value of the quantity in the exponent, $-\frac{x^2}{2} - |0.2 - x |$ (which is clearly negative definite), is minimized. This happens at $x = 0.2$, with can be justified either by noting that, since $x^2$ grows faster than $x$ when $|x| >> 0$, and slower than $x$ when $|x|\approx 0$, the minimum is obtained at the zero of the absolute value, or by numerically (or even symbolically!) maximizing with Mathematica (or whatever your preferred symbolic optimization software is).

\end{document}


\documentclass{article}

% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{style}

\usepackage[final]{style}

\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{verbatim}
\usepackage{amsmath}
\usepackage{amssymb}

\title{Linear Algebra Primer Part 2}

% The \author macro works with any number of authors. There are two
% commands used to separate the names and addresses of multiple
% authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to
% break the lines. Using \AND forces a line break at that point. So,
% if LaTeX puts 3 of 4 authors names on the first line, and the last
% on the second line, try using \AND instead of \And before the third
% author name.

\author{
  Liangcheng Tao, Vivian Hoang-Dung Nguyen, Roma Dziembaj, Sona Allahverdiyeva \\
  Department of Computer Science\\
  Stanford University\\
  Stanford, CA 94305 \\
  \texttt{\{lctao13, vnguyen2, romad, sonakhan\}@cs.stanford.edu} \\
}

\begin{document}

\maketitle

\section{Vectors and Matrices Recap}
\subsection{Vector}
A column vector $\textbf{v} \in {\rm I\!R^{nx1}}$ where $$ \textbf{v} = \begin{bmatrix}
    v_1 \\
    v_2 \\
    \vdots \\
    v_n
  \end{bmatrix}
 $$
\newline
A row vector $\textbf{v}^T \in {\rm I\!R^{1xn}}$ where $$ \textbf{v}^T = \begin{bmatrix}
    v_1 & v_2 & \cdots & v_n
  \end{bmatrix}
 $$
\newline
$T$ denotes the transpose operation.
\newline
\newline
The $\textbf{norm}$ is $$||x||_2 = \sqrt{\sum_{i=1}^{n} x_i^2}$$
Formally, the norm can also be defined as any function $f:{\rm I\!R^{n}} \mapsto {\rm I\!R}$ that satisfies 4 properties:
\begin{itemize}
\item \textbf{Non-negativity:} For all $x \in {\rm I\!R^{n}}$, $f(x) \geq 0$
\item \textbf{Definiteness:} $f(x) = 0$ if and only if $x = 0$
\item \textbf{Homogeneity:} For all $x \in {\rm I\!R^{n}}$, $t \in {\rm I\!R}$, $f(tx) = |t|f(x)$
\item \textbf{Triangle inequality:} For all $x,y \in {\rm I\!R^{n}}$, $f(x+y) \leq f(x) + f(y)$
\end{itemize}
\subsubsection{Projection}
A \textbf{projection} is an inner product (dot product) of vectors. If $B$ is a unit vector, then $A \cdot B$ gives the length of $A$ which lies in the direction of $B$.
\subsection{Matrix}
A matrix $\textbf{A} \in {\rm I\!R^{mxn}}$ is an array of numbers with size $m$ by $n$.
$$ \textbf{A} = \begin{bmatrix}
    a_{11} & a_{12} & a_{13} & \cdots & a_{1n} \\
    a_{21} & a_{22} & a_{23} & \cdots & a_{2n} \\
    \vdots & & & & \vdots \\
    a_{m1} & a_{m2} & a_{m3} & \cdots & a_{mn}
  \end{bmatrix}
$$
\newline
If $m=n$, we say that $\textbf{A}$ is square.
\subsubsection{An Application of Matrices}
Grayscale images have one number per pixel and are stored as an $m\times n$ matrix. Color images have 3 numbers per pixel - red, green, and blue \section{Transformation Matrices}
Matrices can be	used to transform vectors in useful ways, through multiplication:
$x’= Ax$. The simplest application of that is through scaling, or multiplying a scaling matrix with scalars on its diagonal by the vector.

We can also use matrices to rotate vectors. When we multiply a matrix and a vector; the resulting $x$ coordinate is the original vector \textbf{dot} the first row.

In order to rotate a vector by an angle $\theta$, counter-clockwise:

$x'= cos\theta x -sin \theta y$ and

$y'= cos\theta y +sin \theta x$

therefore, we multiply it by the matrix

\[
M=
  \begin{bmatrix}
    cos\theta & -sin\theta \\
    sin\theta & cos\theta
  \end{bmatrix}
\]

which gives us that $P'= R P$.

We can also use multiple matrices to transform a point. For example, $p' = R_2 R_1 S p$. The transformations are applied one after another, from right to left. In our example, this would be $(R_2(R_1(S p)))$.

In order to translate vectors, we have to implement a somewhat hacky solution of adding a "1" at the very end of the vector. This way, using these "homogeneous coordinates", we can also translate vectors. The multiplication works out so the rightmost column of our matrix gets added to the respective coordinates. $A$ homogenous matrix will have [0 0 1] in the bottom row to ensure that the elements get added correctly, and the resulting vector has '1' at the bottom.

By convention, in homogeneous coordinates, we divide the result by its last coordinate after doing matrix multiplication.

\[
  \begin{bmatrix}
    x \\
    y \\
    7
  \end{bmatrix}
\]

\[
    \begin{bmatrix}
    x/7 \\
    y/7 \\
    1
  \end{bmatrix}
  \]

So to obtain the result of
$P(x,y) -> P' = (s_x x, s_y y)$

we have to first $P = (x, y) -> (x,y,1)$ and then $P' = (s_x x, s_y y) -> (s_x x, s_y y, 1)$ so we can then do the matrix multiplication S*P. Though, we have  to note that scaling and translating is not the same as translating and scaling. In other words, $T*S*P \neq S*T*P$

Any rotation matrix R belongs to the category of normal matrices, and it satisfies interesting properties. For example, $R \dot R^T = I$ and $det(R) = 1$

The rows of a rotation matrix are always mutually perpendicular (a.k.a. orthogonal) unit vectors; this is what allows for it to satisfy some of the few unique properties mentioned above.

\section{Matrix Inverse}
Given a matrix $A$, its inverse $A^{-1}$ is a matrix such that:
$$AA^{-1} = A^{-1}A = I$$
where $I$ is the identity matrix of the same size.

An example of a matrix inverse is:
$$
  \begin{bmatrix}
    2 & 0 \\
    0 & 3 \\
  \end{bmatrix}^{-1}
  =
  \begin{bmatrix}
    \frac{1}{2} & 0 \\
    0 & \frac{1}{3} \\
  \end{bmatrix}
$$

A matrix does not necessarily have an inverse. If $A^{-1}$ exists, $A$ is known as \textit{invertible} or \textit{non-singular}.

Some useful identities for matrices that are invertible are:
\begin{itemize}
\item $(A^{-1})^{-1} = A$
\item $(AB)^{-1} = B^{-1}A^{-1}$
\item $A^{-T} \triangleq (A^T)^{-1} = (A^{-1})^T$
\end{itemize}

\subsection{Pseudoinverse}

Frequently in linear algebra problems, you want to solve the equation $AX=B$ for $X$. You would like to compute $A^{-1}$ and multiply both sides to get $X=A^{-1}B$. In python, this command would be: np.linalg.inv(A)*B.

However, for large floating point matrices, calculating inverses can be very expensive and possibly inaccurate. An inverse could also not even exist for $A$. What should we do?

Luckily, we have what is known as a \textit{pseudoinverse}. This other matrix can be used to solve for $AX=B$. Python will try many methods, including using the pseudo-inverse, if you use the following command: np.linalg.solve(A,B). Additionally, using the pseudo-inverse, Python finds the closest solution if there exists no solution to $AX=B$.

\section{Matrix Rank}
\begin{itemize}
\item The rank of a transformation matrix  $A$ tells you how many dimensions it transforms a matrix to.

\item col-rank($A$) = maximum number of linearly independent column vectors of $A$

\item row-rank($A$) = maximum number of linearly independent row vectors of $A$.
Column rank always equals row rank.

\item For transformation matrices, the rank tells you the dimensions of the output.

\item For instance, if rank of $A$ is $1$, then the transformation $p' =Ap$ points onto a line.

\item Full rank matrix- if $mxm$ and rank is $m$

\item Singular matrix- if $mxm$ matrix rank is less than $m$, because at least one dimension is getting collapsed. (No way to tell what input was from result) --> inverse does not exist for non-square matrices.
\end{itemize}

\section{Eigenvalues and Eigenvectors (SVD)}

\subsection{Definitions}
An \textit{eigenvector} \textbf{x} of a linear transformation $A$ is a non-zero vector that, when $A$ is applied to it, does not change its direction.
Applying $A$ to the eigenvector scales the eigenvector by a scalar value $\lambda$, called an \textit{eigenvalue}.

The following equation describes the relationship between eigenvalues and eigenvectors:
$$A\mathbf{x}=\lambda \mathbf{x}, \quad \mathbf{x}\neq \mathbf{0}$$

\subsection{Finding eigenvectors and eigenvalues}
If we want to find the eigenvalues of $A$, we can manipulate the above definition as follows:

$$A\mathbf{x}=\lambda \mathbf{x}, \quad \mathbf{x}\neq \mathbf{0}$$
$$A\mathbf{x}=(\lambda I \mathbf{x}), \quad \mathbf{x}\neq \mathbf{0}$$
$$(\lambda I-A)\mathbf{x}=\mathbf{0}, \quad \mathbf{x}\neq \mathbf{0}$$

Since we are looking for non-zero \textbf{x}, we can equivalently write the above relation as:

$$|\lambda I-A|=\mathbf{0}$$

Solving this equation for $\lambda$ gives the eigenvalues of A, and these can be substituted back into the original equation to find the corresponding eigenvectors.

\subsection{Properties}
\begin{itemize}
\item The trace of A is equal to the sum of its eigenvalues:
$$tr(A)=\sum_{i=1}^{n} \lambda_i$$
\item The determinant of A is equal to the product of its eigenvalues:
$$|A|=\prod_{i=1}^{n} \lambda_i$$
\item The rank of A is equal to the number of non-zero eigenvalues of A.
\item The eigenvalues of a diagonal matrix $D = diag(d_1, \ldots, d_n)$ are just the diagonal entries $d_1, \ldots d_n$.
\end{itemize}

\subsection{Spectral Theory}
\subsubsection{Definitions}
\begin{itemize}
\item An \textit{eigenpair} is the pair of an eigenvalue and its associated eigenvector.
\item An \textit{eigenspace} of $A$ associated with $\lambda$ is the space of vectors where:
$$(A-\lambda I) = 0$$
\item The \textit{spectrum} of $A$ is the set of all its eigenvalues:
$$\sigma(A)=\lbrace\lambda \in \mathbb{C}: \lambda I - A \textrm{ is singular}\rbrace$$
Where $\mathbb{C}$ is the space of all eigenvalues of $A$
\item The \textit{spectral radius} of $A$ is the magnitude of its largest magnitude eigenvalue:
$$\rho(A)=max\lbrace |\lambda_1|, \ldots, |\lambda_n| \rbrace$$

\end{itemize}
\subsubsection{Theorem: Spectral radius bound} Spectral radius is bounded by the infinity norm of a matrix:
$$\rho(A)=\lim_{k\to\infty} ||A^k||^{1/k}$$
\textit{Proof}:
$$|\lambda|^k||\mathbf{v}||=||\lambda|^k\mathbf{v}||=||A^k\mathbf{v}||$$
By the Cauchy–Schwarz inequality ($||\mathbf{u}\mathbf{v}|| \leq ||\mathbf{u}||\cdot||\mathbf{v}||$):
$$|\lambda|^k||\mathbf{v}|| \leq ||A^k||\cdot||\mathbf{v}||$$
Since $\mathbf{v} \neq 0$:
$$|\lambda|^k \leq ||A^k||$$
And we thus arrive at:
$$\rho(A)=\lim_{k\to\infty} ||A^k||^{1/k}$$

\subsection{Diagonalization}
An $n\times n$ matrix A is diagonalizable if it has $n$ linearly independent eigenvectors.

Most square matrices are diagonalizable
\begin{itemize}
\item Normal matrices are diagonalizable

Note: Normal matrices are matrices that satisfy:
$$A^* A=AA^*$$
Where $A^*$ is the complex conjugate of $A$
\item Matrices with $n$ distinct eigenvalues are diagonalizable
\end{itemize}

\textbf{Lemma:} Eigenvectors associated with distinct eigenvalues are linearly independent.

To diagonalize the matrix $A$, consider its eigenvalues and eigenvectors. We can construct matrices $D$ and $V$, where $D$ is the diagonal matrix of the eigenvalues of $A$, and $V$ is the matrix of corresponding eigenvectors:
$$D=\begin{bmatrix}
    \lambda_1 & & \\
    & \ddots &    \\
    & & \lambda_n
\end{bmatrix}
$$

$$V=\begin{bmatrix}
    v_1 & v_2 & \dots & v_n
\end{bmatrix}
$$

Since we know that:
$$AV=VD$$
We can diagonalize $A$ by:
$$A=VDV^{-1}$$
If all eigenvalues are unique, then $V$ is orthogonal. Since the inverse of an orthogonal matrix is its transpose, we can write the diagonalization as:
$$A=VDV^T$$
\subsection{Symmetric Matrices}
If $A$ is symmetric, then all its eigenvalues are real, and its eigenvectors are orthonormal. Recalling the above diagonalization equation, we can diagonalize $A$ by:
$$A=VDV^T$$

Using the above relation, we can also write the following relationship:

Given $y=V^Tx$:
$$x^TAx=x^TVDV^Tx=y^TDy=\sum_{i=1}^{n} \lambda_i y_i^2$$

Thus, if we want to do the following maximization:
$$ max_{x\in\mathbb{R}^n} (x^TAx) \quad \textrm{ subject to   } ||x||_2^2=1$$

Then the maximizing $x$ can be found by finding the eigenvector corresponding to the largest eigenvalue of $A$.

\subsection{Applications}
Some applications of eigenvalues and eigenvectors include, but are not limited to:
\begin{itemize}
\item PageRank
\item Schroedinger's equation
\item Principle component analysis (PCA)
\end{itemize}
\section{Matrix Calculus}
\subsection{The Gradient}
If a function $f:{\rm I\!R^{mxn}} \mapsto {\rm I\!R}$ takes as input a matrix $A$ of size $(m\times n)$ and returns a real value, then the gradient of $f$ is
\newline
$$\nabla_A f(A) \in {\rm I\!R^{mxn}} = \begin{bmatrix}
    \frac{\partial f(A)}{\partial A_{11}} & \frac{\partial f(A)}{\partial A_{12}} & \cdots & \frac{\partial f(A)}{\partial A_{1n}} \\
    \frac{\partial f(A)}{\partial A_{21}} & \frac{\partial f(A)}{\partial A_{22}} & \cdots & \frac{\partial f(A)}{\partial A_{2n}} \\
    \vdots & \vdots & \ddots & \vdots \\
    \frac{\partial f(A)}{\partial A_{m1}} & \frac{\partial f(A)}{\partial A_{m2}} & \cdots & \frac{\partial f(A)}{\partial A_{mn}}
\end{bmatrix}
$$
\newline
Every entry in the matrix is: $$\nabla_A f(A)_{ij} = \frac{\partial f(A)}{\partial A_{ij}}$$
\newline
The size of $\nabla_A f(A)$ is always the same as the size of $A$. Thus, if $A$ is a vector $x$:
$$\nabla_x f(x) = \begin{bmatrix}
	\frac{\partial f(x)}{\partial x_{1}} \\
    \frac{\partial f(x)}{\partial x_{2}} \\
    \vdots \\
    \frac{\partial f(x)}{\partial x_{n}}
\end{bmatrix}
$$
\subsection{The Gradient: Properties}
\begin{itemize}
\item $\nabla_x (f(x) + g(x)) = \nabla_x f(x) + \nabla_x g(x)$
\item For $t \in {\rm I\!R}$, $\nabla_x (t f(x)) = t \nabla_x f(x)$
\end{itemize}
\subsection{The Hessian}
The Hessian matrix with respect to $x$ can be written as $\nabla_x^2 f(x)$ or as $H$. It is an $nxn$ matrix of partial derivatives
\newline
$$\nabla_x^2 f(x) \in {\rm I\!R^{nxn}} = \begin{bmatrix}
    \frac{\partial^2 f(x)}{\partial x_1^2} & \frac{\partial^2 f(x)}{\partial x_1 \partial x_2} & \cdots & \frac{\partial^2 f(x)}{\partial x_1 \partial x_n} \\
    \frac{\partial^2 f(x)}{\partial x_2 \partial x_1} & \frac{\partial^2 f(x)}{\partial x_2^2} & \cdots & \frac{\partial^2 f(x)}{\partial x_2 \partial x_n} \\
    \vdots & \vdots & \ddots & \vdots \\
    \frac{\partial^2 f(x)}{\partial x_n \partial x_1} & \frac{\partial^2 f(x)}{\partial x_n \partial x_2} & \cdots & \frac{\partial^2 f(x)}{\partial x_n^2} \\
\end{bmatrix}
$$
\newline
Every entry in the matrix is: $$\nabla_x^2 f(x)_{ij} = \frac{\partial^2 f(x)}{\partial x_i \partial x_j}$$
\newline
It's important to note that the Hessian is the gradient of \textbf{every entry} of the gradient of the vector. For instance, the first column of the Hessian is the gradient of $\frac{\partial f(x)}{\partial x_1}$.
\subsection{The Hessian: Properties}
Schwarz's theorem: The order of partial derivatives doesn't matter so long as the second derivative exists and is continuous.
\newline
\newline
Thus, the Hessian is always symmetric:
$$\frac{\partial^2 f(x)}{\partial x_i \partial x_j} = \frac{\partial^2 f(x)}{\partial x_j \partial x_i}$$
\subsection{Example Calculations}
\subsubsection{Example Gradient Calculation}
For $x \in {\rm I\!R^{n}}$, let $f(x) = b^Tx$ for some known vector $b \in {\rm I\!R^{n}}$
$$ f(x) = \begin{bmatrix}
    b_1 & b_2 & \cdots & b_n
  \end{bmatrix}^T
  \begin{bmatrix}
  	x_1 \\
    x_2 \\
    \vdots \\
    x_n
  \end{bmatrix}
 $$
\newline
Thus, $$f(x) = \sum_{i=1}^{n} b_i x_i$$
$$\frac{\partial f(x)}{\partial x_k} = \frac{\partial}{\partial x_k}\sum_{i=1}^{n} b_i x_i = b_k$$
Therefore, we can conclude that: $\nabla_x b^Tx = b$.
\subsubsection{Example Hessian Calculation}
Consider the quadratic function $f(x) = x^TAx$
$$f(x) = \sum_{i=1}^{n} \sum_{j=1}^{n} A_{ij} x_i x_j$$
$$\frac{\partial f(x)}{\partial x_k} = \frac{\partial}{\partial x_k}\sum_{i=1}^{n}\sum_{j=1}^{n} A_{ij} x_i x_j$$
$$= \frac{\partial}{\partial x_k}[\sum_{i \neq k} \sum_{j \neq k} A_{ij} x_i x_j + \sum_{i \neq k} A_{ik} x_i x_k + \sum_{j \neq k} A_{kj} x_k x_j + A_{kk}x_k^2]$$
$$= \sum_{i \neq k} A_{ik} x_i + \sum_{j \neq k} A_{kj} x_j + 2 A_{kk}x_k$$
$$= \sum_{i=1}^{n} A_{ik} x_i + \sum_{j=1}^{n} A_{kj} x_j = 2\sum_{i=1}^{n} A_{ki} x_i$$
$$\frac{\partial^2 f(x)}{\partial x_k \partial x_l} = \frac{\partial}{\partial x_k}[\frac{\partial f(x)}{\partial x_l}] = \frac{\partial}{\partial x_k}[\sum_{i=1}^{n} 2 A_{li} x_i]$$
$$= 2A_{lk} = 2A_{kl}$$
Thus, $$\nabla_x^2 f(x) = 2A$$
% References
% \small
% \bibliographystyle{plain}
% \bibliography{bibliography}
\end{document}


\section{Background Subtraction: autobk algorithm}

\begin{frame} \frametitle{Background Subtraction: $\mu_0(E)$}

\begin{cenpage}{130mm}

  \[ \chi(E) = \frac{\mu(E) - \mu_0(E)}{\Delta \mu_0(E_0)} \]

\vmm
We don't know ${\mu_0(E)}$, so use a {\BlueEmph{spline}}: a smooth, adjustable function.

\vmm

This could be dangerous -- a flexible enough spline would remove all the XAFS.

\vmm
\pause

The {\autobk} method in {\larch} chooses a background spline for $\mu_0(E)$ to

\begin{center}
  \begin{postitbox}{55mm}
    {\RedEmph{minimize the low-$R$ components of $\chi$}}
  \end{postitbox}
\end{center}

\begin{enumerate}
  \item Spline parameters for $\mu_0(E)$ are guessed.
  \item $\chi(k)$ is Fourier Transformed to $\chi(R)$.,
  \item Spline parameters optimized so that $\chi(R)$ below \feffc{R_{\rm bkg}} is minimized.
  \item Note that $\chi(R)$ above \feffc{R_{\rm bkg}} is completely ignored!
\end{enumerate}

\vmm  \hrule \vmm

Most important parameters:

\begin{enumerate}
\item \feffc{R_{\rm bkg}}: $R$ below which $\chi(R)$ is reduced.
\item {\Red{$k$-weight}}:  used for Fourier transform:   use 1, or 2.
\item \feffc{E_0}:         May need to adjust for initial guess (max of $d\mu/dE$).
\end{enumerate}

\end{cenpage}
\end{frame}

\begin{frame} \frametitle{Background Subtraction in {\xasviewer} }

  \begin{cenpage}{130mm}

Effect of \feffc{R_{\rm bkg}}  on XAFS $\chi(k)$ and $\chi(R)$:

\begin{tabular}{ll}
  \begin{minipage}{65mm}
    \includegraphics[width=60mm]{figs/experiment/bkg_ksp1}
  \end{minipage} &
  \begin{minipage}{65mm}
    \includegraphics[width=60mm]{figs/experiment/bkg_rsp1}
  \end{minipage} \\
  \begin{minipage}{55mm}
    $\chi(k)$ for FeO with
    $R_{\rm bkg}=0.1\, \rm \AA$ \\
    (stiff spline)  and $R_{\rm bkg}=1.0\, \rm \AA$.
  \end{minipage} &
  \begin{minipage}{55mm}
    $\chi(R)$ for FeO with
    $R_{\rm bkg}=0.1\, \rm \AA$ \\
    (stiff spline)   and  $R_{\rm bkg}=1.0\, \rm \AA$.
  \end{minipage} \\
\end{tabular}

\vmm \pause \vmm\vmm

Rules of thumb:

\begin{postitbox}{76mm}
  Use  $R_{\rm bkg}=1.0\, \rm \AA$ or half the near-neighbor distance.
\end{postitbox}

\begin{postitbox}{76mm}
  Don't spend too much time on background subtraction.
\end{postitbox}

\end{cenpage}

\end{frame}

\begin{frame} \frametitle{Background Subtraction in {\xasviewer}  }

  \begin{cenpage}{130mm}

Don't make \feffc{R_{\rm bkg}} too big!

\begin{tabular}{ll}
  \begin{minipage}{65mm}
    \includegraphics[width=60mm]{figs/experiment/bkg_ksp2}
  \end{minipage} &
  \begin{minipage}{55mm}
    \includegraphics[width=60mm]{figs/experiment/bkg_rsp2}
  \end{minipage} \\
  \begin{minipage}{55mm}
    $\chi(k)$ for FeO with $R_{\rm bkg}=2.0\, \rm \AA$
  \end{minipage} &
  \begin{minipage}{55mm}
    $\chi(R)$ for FeO with $R_{\rm bkg}=2.0\, \rm \AA$  Note: we have
    removed the first shell!!

  \end{minipage} \\
\end{tabular}

\vmm

    \vmm
    Having $R_{\rm bkg}$ too big is the most important thing to avoid.

    \vmm

    $R_{\rm bkg}$  that's a little bit small  and gives a small peak at
    very low $R$ is not that big of a problem.

    \end{cenpage}
\vfill

\end{frame}


%${LICENSE}$
\documentclass[twocolumn, noxcolor, maketitle]{rbt-mathnotes-hw}
\mathnotes{
  instructor  = Prof.~Ruth Charney ,
  name        = Rebecca Turner ,
  email       = rebeccaturner@brandeis.edu ,
  course      = \textsc{math} 104a (Intro to Topology) ,
  institution = Brandeis University ,
  semester    = Spring 2020 ,
}
\title{Homework 1}
\date{2020-01-18}
\def\T{\mathcal{T}}
\def\basis{\mathcal{B}}
\usepackage[
  letterpaper,
  margin = 1in,
]{geometry}
\raggedbottom
\begin{document}
\maketitle

\section{Topological Spaces}
\subsection{Open Sets and the Definition of a Topology}
\begin{problem}[1.7]
  Define a topology on $\R$ (by listing the open sets within it) that contains
  the open sets $(0,2)$ and $(1,3)$ that contains as few open sets as possible.
\end{problem}
$\T = \{ \emptyset, (0,2), (1,3), (1,2), (0,3), \R \}$.

\subsection{Basis for a Topology}
\begin{problem}[1.10]
  Show that $\mathcal{B} = \{[a,b) \subset \R : a < b\}$ is a basis for a
  topology on $\R$.
\end{problem}
\begin{enumerate}
  \item \textbf{$\emptyset \in \T$, $\R \in \T$.} $\emptyset \in \T$ (by the
    definition of the completion of a basis to a topology).

    Next, we show $\R \in \T$. For all $n \in \Z_{\ge 0}$, $[n-1, n) \in \basis$
    and $[-n + 1, -n) \in \basis$. We know that if $b_1, b_2 \in \basis$, $b_1
    \union b_2 \in \T$, so these short intervals can be gathered together (``unionized'')
    to produce $\R$:
    \[ \bigunion_{n=1}^{\infty} \left( [n-1, n) \union [-n + 1, -n] \right) = \R, \]
    so $\R \in \T$.

  \item \textbf{$\T$ contains all finite intersections of elements of $\T$.}
    Suppose we have two intervals $[a, b)$ and $[c, d)$. Then, we define
    \begin{alignat*}{1}
      a' &= \max(a, c) \\
      b' &= \min(b, d).
    \end{alignat*}
    If $a' > b'$, the intersection $[a, b) \inter [c, d) = \emptyset$, which is
    in $\T$. Otherwise, the intersection is $[a', b')$, which is an element of
    $\basis$. All elements of the basis are in $\T$, so the intersection of two
    elements is in the topology.

    Thankfully, the intersection is itself always a basis element, so we can
    use the same process to show that finite intersections are in $\T$ by
    induction.

  \item \textbf{Unions of elements of $\T$ are in $\T$.} By the definition of
    the completion of a basis to a topology, this is true (all unions of basis
    elements are included in $\T$).
\end{enumerate}

\begin{problem}[1.12]
  % See example 1.9 for defn of \R_l
  Determine which of the following are open sets in $\R_l$. In each case, prove
  your assertion.
  \[ A = [4,5)
    \quad B = \{3\}
    \quad C = [1,2]
    \quad D = (7,8) \]
\end{problem}
\begin{enumerate}
  \item $A$ is open in $\R_l$; $[4,5) \in \basis$.
  \item $B$ is not an open set in $\R_l$; there is no $[a, b) \subset \R$ where
    both $b > a$ and $|[a, b)| = 1$ (because $[0,1) \cong \R$, i.e.~all
    intervals contain infinitely many points).

    (Where $\cong$ means ``is isomorphic to.'')
  \item $C$ is not open in $\R_l$ because the upper bound of an open set in
    $\R_l$ is never inclusive. There is no set of intervals $[a_1, b_1), \dots$
    where the union or intersection of the intervals has an inclusive upper bound.
  \item $D$ is open because we can take
    \[ D = \lim_{n\to\infty} \left[7 + \frac{1}{n}, 8\right), \]
    where $[7 + 1/n, 8) \in \basis$ for any $n \in \R$ with $n \ne 0$.
\end{enumerate}

\pagebreak
\begin{problem}[1.15]
  An arithmetic progression in $\Z$ is a set
  \[ A_{a,b} = \{\dots, a - 2b, a - b, a, a + b, a + 2b, \dots\} \]
  with $a,b \in \Z$ and $b \ne 0$. Prove that the collection of arithmetic
  progressions
  \[ \mathcal{A} = \{ A_{a,b} : a, b \in \Z \text{ and } b \ne 0 \} \]
  is a basis for a topology on $\Z$. The resulting topology is called the
  arithmetic progression topology on $\Z$.
\end{problem}

\begin{proof}
  \def\Ar#1#2{A_{#1,#2}}
  \def\Ars{\mathcal{A}}
  Let us describe the \emph{minimal form} of an arithmetic progression $\Ar ab$
  to be the progression $\Ar{a'}{b'} = \Ar ab$ with $a', b' > 0$ and the
  smallest possible $a'$; in particular, that $a' < b'$.

  We can obtain the minimal form of the progression like so:
  \begin{alignat*}{1}
    a' &= a \bmod b \\
    b' &= |b|, \\
    \Ar{a'}{b'} &= \Ar ab.
  \end{alignat*}

  \begin{remark}
    Two arithmetic progressions have the same elements if their minimal forms are
    the same; this give an equivalence relation on $\Ars$.
  \end{remark}

  Now, suppose we have two arithmetic progressions $\Ar ab$ and $\Ar cd$. We
  assume that the progressions are in minimal form without loss of generality. We
  also assume that $b \le d$ (by swapping $(a,b)$ with $(c,d)$ if necessary),
  again without loss of generality.

  If $b \mid d$ and $a = c$, we have $\Ar ab \subset \Ar cd$. In particular, $\Ar
  ab \inter \Ar cd = \Ar cd$.

  If $b \mid d$ and $a \ne c$, we have $\Ar ab \inter \Ar cd = \emptyset$.

  If $b \nmid d$, we have a different progression. An intersection is generated by
  an index $(n_1, n_2)$, where
  \begin{alignat*}{1}
    a + b n_1 &= c + d n_2. \\
    \intertext{We can then solve for $n_1$:}
    t(n) &= c - a + dn \\
    n_1 &= \frac{t(n_2)}{b}. \\
    \intertext{Next, we have an infinite \emph{set} of possibilities for $n_2$:}
    n_2 &\in \left\{n \in \Z : t(n) \mid b \right\}.
    \intertext{Sorting the possible values of $n_2$ by absolute value, let us call
      the smallest two values $i_1$ and $i_2$. Then, the difference between
      adjacent elements in the intersection progression $\Ar ab \inter \Ar cd$
      is $i_2 - i_1$.
      \endgraf
      Let}
    a' &= a + bi_1 \\
    b' &= i_2 - i_1 \\
    \Ar ab \inter \Ar cd &= \Ar{a'}{b'}.
  \end{alignat*}
  This isn't super rigorous, admittedly (we're missing some inductive reasoning
  about the integers to prove that there are an infinite set of valid values of
  $n_2$, in particular), but I have some fairly convincing Haskell code. And the
  missing steps are mostly boilerplate, and it's late at night already\dots

  In all cases, the intersection of two arithmetic progressions is either empty or
  another arithmetic progression (i.e.~either the empty set or another basis
  element), so the same argument given above for $\R_l$ holds (namely that we can
  extend this to all finite intersections of elements of $\Ars$ inductively).

  Therefore, finite intersections are in the basis. Unions are in the completion
  of the basis (again by definition). The special element $\emptyset$ is in the
  completion (by definition), and $\Z = \Ar01$, so $\Z \in \basis$. Therefore,
  $\Ars$ forms the basis of a topology on $\Z$.
\end{proof}

\pagebreak
\subsection{Closed Sets}
\begin{problem}[1.27(a)]
  The infinite comb $C$ is the subset of the plane illustrated in Figure~1.17
  and defined by
  \begin{multline*}
    C = \{(x,0) : 0 \le x \le 1\} \;\union \\
    \bigg\{ \left( \frac{1}{2^n}, y \right) : n = 0,1,2, \dots \\
      \text{ and } 0 \le y \le 1 \bigg\}.
  \end{multline*}
  Prove that $C$ is not closed in the standard topology on $\R^2$.
\end{problem}
\begin{proof}
  Suppose $C$ is closed in the standard topology on $\R^2$. Then, its complement
  $C^c = \R^2 \setminus C$ must be an open set.

  The point $(0, 1)$ is not in $C$, so $(0, 1) \in C^c$. Every open ball in $\R^2$
  containing $(0, 1)$ also contains a smaller open ball centered about $(0, 1)$.
  (For example, the open ball about $(-1, 1)$ of radius $1.1$ contains the open
  ball centered about $(0, 1)$ of radius $0.1$.)

  However, every open ball centered about $(0, 1)$ contains infinitely many
  points of $C$; if the ball has radius $r$, all the comb's ``tines'' at $x =
  1/2^n$ for $n > - \log_2 r$ intersect with the ball.

  Therefore, every open ball containing $(0, 1)$ also contains points in $C$. As
  a result, $C^c$ is not open, which contradicts our assumption. Therefore, $C$
  is not closed.
\end{proof}

\begin{problem}[1.32]
  Prove that intervals of the form $[a, b)$ are closed in the lower limit
  topology on $\R$.
\end{problem}
\begin{proof}
  Take some interval $[a, b)$. Its complement is given by $(-\infty, a) \union
  [b, \infty)$. Given that
  \begin{alignat*}{1}
    (-\infty, a) &= \bigunion_{n=1}^\infty [a-n, a) \\
    [b, \infty) &= \bigunion_{n=1}^\infty [b, b+n),
  \end{alignat*}
  the complement of $[a, b)$ is the union of a number of lower-limit intervals
  in $\R$, i.e.~the basis elements. The basis elements and its unions are open
  sets, so the complement of $[a, b)$ is an open set. Then, by the definition of
  a closed set, $[a, b)$ is closed in $\R_l$.
\end{proof}

\end{document}


%!TEX program = xelatex

\documentclass[12pt,a4paper]{article}
\usepackage{xeCJK}
\usepackage{amsmath}
\setCJKmainfont{STSongti-SC-Regular}
\usepackage{setspace}
\usepackage{caption}
\usepackage{graphicx, subfig}
\usepackage{float}
\usepackage{listings}
\usepackage{booktabs}

\begin{document}
\title{homework3}
	\author{11611118 郭思源}

\section{Problem 1}
\begin{lstlisting}[language=matlab]
%指数分布
set(gcf,'Units','centimeters','Position',[0 0 30 30]);
num = [100,1000,5000];
mu = 3;
for i = 1:3
sample = exprnd(mu,num(i),1);% Simulated data
x=min(sample):1:max(sample);
subplot(3,3,3*i-2)
histogram(sample,x);
title(sprintf('Exponential distribution (n=%d)', num(i)));
fprintf("i=%d",i);
[muhat,muci] = expfit(sample, 0.05)
end

%泊松分布
num = [100,1000,5000];
lam=70;
for i = 1:3
sample=poissrnd(lam,num(i),1);% Simulated data
x=min(sample):1:max(sample);
subplot(3,3,3*i-1)
histogram(sample,x);
title(sprintf('Poisson distribution (n=%d)', num(i)));
fprintf("i=%d",i);
[Lambdahat, Lambdaci]=poissfit (sample, 0.05)
end

%正态分布
num = [100,1000,5000]*2;
for i = 1:3
sample = normrnd(0,1,num(i),1);
x=min(sample):0.1:max(sample);
subplot(3,3,3*i)
histogram(sample,x);
title(sprintf('Normal distribution (n=%d)', num(i)));
fprintf("i=%d",i);
[muhat,sigmahat,muci,sigmaci]=normfit(sample, 0.05)
end
\end{lstlisting}

\begin{figure}[htbp]
\centering
\includegraphics[bb=400 400 1300 1300,scale=.3]{figure/HW3_1.jpg}
\end{figure}

\begin{table}[htbp]
\caption{Exponential distribution confidence interval (95)\% \\ $\lambda=3$}
\centering
\begin{tabular}{lll}
	\toprule
	Number of samples &
	Estimates the mean $\lambda$ &
	Confidence intervals for the mean $\lambda$ \\
	\midrule
	100 & 2.7511 & [ 2.2826, 3.3813 ] \\
	1000 & 3.0171 & [ 2.8385, 3.2132 ] \\
	5000 & 2.9651 & [ 2.8846, 3.0490 ] \\
	\bottomrule
\end{tabular}
\end{table}

\newpage

\begin{table}[htbp]
\caption{Poisson distribution confidence interval (95)\% \\ $\lambda=70$}
\centering
\begin{tabular}{lll}
	\toprule
	Number of samples &
	Estimates the mean $\lambda$ &
	Confidence intervals for the mean $\lambda$ \\
	\midrule
	100 & 70.4200 & [ 68.7753, 72.0647 ] \\
	1000 & 70.2070 & [ 69.6877, 70.7263 ] \\
	5000 & 69.8896 & [ 69.6579, 70.1213 ] \\
	\bottomrule
\end{tabular}
\end{table}

\begin{table}[htbp]

\caption{Exponential distribution confidence interval (95)\% \\
		$\mu=0$ \\ $\sigma=1$}
\centering

\begin{tabular}{lll}

	\toprule
	Number of samples &
	Estimates the mean $\mu$ &
	Confidence intervals for the mean $\mu$ \\
	\midrule
	200 & 0.0279 & [ -0.1152, 0.1710 ] \\
	2000 & 0.0099 & [ -0.0339, 0.0536 ] \\
	10000 & -0.0122 & [ -0.0318, 0.0074 ] \\
	\bottomrule

\end{tabular}

\begin{tabular}{lll}

	\toprule
	Number of samples &
	Estimates the mean $\sigma$ &
	Confidence intervals for the mean $\sigma$ \\
	\midrule
	200 & 1.0263 & [ 0.9346, 1.1381 ] \\
	2000 & 0.9977 & [ 0.9677, 1.0296 ] \\
	10000 & 0.9990 & [ 0.9854, 1.0131 ] \\
	\bottomrule

\end{tabular}

\end{table}

\newpage

\section{Problem 2}

\subsection{0-norm}
As we know the error
$\left\|e\right\|_0$ = the number of ${y_i-ax_i-b\ne0}$\\
so that we choose the line y=ax+b such that the most point lie on the line.\\
so we just choose two point $(x_i,y_i)$ $(x_j,y_j)$\\

Then: \\
\begin{equation}
	\begin{aligned}
		a &= \frac{y_i-y_j}{x_i-x_j}\\
		b &= \frac{y_j x_i-x_j y_i}{x_i-x_j}\\
	\end{aligned}
\end{equation}

\subsection{1-norm}

As we know the error $\left\|e\right\|_1$=$\sum_{i=1}^n|y_i-ax_i-b|$\\

Error: \\
\begin{equation}
	\begin{aligned}
		\Bar{e} &= \frac{\sum_{i=1}^n|y_i-ax_i-b|}{n} \\
		e_i &= |y_i-ax_i-b|\\
	\end{aligned}
\end{equation}

So we define y$_i$$'$=$y_i-| e_i- \Bar{e} |$\\

\noindent So that we can know that those  y$_i$$'$ lie on two parallel lines. and the fit line is the middle line of those tow liens.

So is y$_i$$'$ satisfy:\\
\begin{equation}
	\begin{aligned}
		y &= ax+b \pm \Bar{e} \\
	\end{aligned}
\end{equation}

\newpage
\subsection{2-norm}
As we know the error $\left\|e\right\|_2$=$\sum_{i=1}^n(y_i-ax_i-b)^2$\\

So :
\begin{equation}
	\begin{aligned}
		\frac{\partial e}{\partial a} &= 0 \\
		\frac{\partial e}{\partial b} &= 0 \\\\
	\end{aligned}
\end{equation}

Then :
\begin{equation}
	\begin{aligned}
		a &= \frac{m \sum_{i=1}^n{x_i y_i}-\sum_{i=1}^n{x_i} \sum_{i=1}^n{y_i}}{m \sum_{i=1}^n{x_i^2}-(\sum_{i=1}^n{x_i})^2}\\\\
		b &= \frac{\sum_{i=1}^n{x_i^2} \sum_{i=1}^n{y_i}-\sum_{i=1}^n{x_i} m \sum_{i=1}^n{x_i y_i}}{m \sum_{i=1}^n{x_i^2}-(\sum_{i=1}^n{x_i})^2}
	\end{aligned}
\end{equation}

\subsection{$\infty$-norm}
As we know the error $\left\|e\right\|_\infty$=$\max{|y_i-ax_i-b|}$\\
Suppose when i=k,$|y_i-ax_i-b|$ is maximum\\
so that we just make the a,b to satisfy $ax_k-b=y_k$ \\
Then we will get the min $\left\|e\right\|_\infty$\\

\newpage

\section{Problem 3}
The lasso in Matlab is used for lasso or elastic net regularization for linear models. \\\\
\textbf{The syntax of the lasso function:}\\

\noindent \textbf{B = lasso(X,Y):}\\
Returns fitted least-squares regression coefficients for linear models of the predictor data X and the response y. Each column of B corresponds to a particular regularization coefficient in Lambda. By default, lasso performs lasso regularization using a geometric sequence of Lambda values.\\

\noindent \textbf{B = lasso(X,Y,Name,Value):}\\
Fits regularized regressions with additional options specified by one or more name-value pair arguments. For example, 'Alpha',0.5 sets elastic net as the regularization method, with the parameter Alpha equal to 0.5.\\

\noindent \textbf{$\textbf{[B,FitInfo]}$ = lasso($\_\_\_$):}\\
For any previous input syntax, also returns a structure containing information about the fits. It invokes a normalized regression analysis that is appropriate for additional options with specific names and values.\\

\noindent \textbf{Input: }\\\\
\textbf{X} is the numeric matrix. Each row represents one observation, and each column represents one predictor (variable).\\\\
\textbf{Y} is the numeric vector of length n, where n is the number of rows of \textbf{X}. \textbf{Y(i)} is the response to row i of \textbf{X}.\\

\noindent \textbf{ADMM algorithm:}\\\\
ADMM algorithm is used in an agrument called AbsTol which is used in the lasso to deal with tall array data.\\
ADMM is the algorithm that expands by ALM solves convex optimization problems by breaking them into smaller pieces, each of which are then easier to handle. It is aimed to make up the shortcomings of secondary punishment to make the algorithm more stable\\

\end{document}


\section{Approach and Secret Weapon}

\subsection{Serial Implementation}
We first implemented Baum-Welch in Python with numpy, using a mix of vectorized and non-vectorized code.  \footnote{We also used the range function instead of the xrange function in some places.  Our Python implementation also uses the default data type of numpy/Python; our later parallel implementation uses a mix of doubles and floats.  Our project desires to make a comparison based on scaling, so we don't consider these ``constant factor" sorts of differences to be an issue.}  Our initial
implementation ran into serious underflow issues from the calculations of
$\alpha$'s and $\beta$'s due to multiplying probabilities together. We solve
this by following the methods detailed by Zhai~\cite{normalizer}. We normalize by scaling $\alpha_i(t)$'s such that they sum up to $1$ over $i$'s---that is over all states. So

\begin{equation}
\sum_{i=1}^N \hat{\alpha}_t(i) = 1
\end{equation}

and we want to define the normalizers $\eta_k$ such that

\begin{equation}
    \hat{\alpha}_t(i) = \alpha_i(t) \prod_{k=1}^t \frac{1}{\eta_k}
\end{equation}

then
\begin{equation}
\prod_{k=1}^t \eta_k =\sum_{i=1}^N \alpha_t(i)
\end{equation}

We can define
\begin{equation}
    \hat{\beta}_{i}(t) = \beta_{i}(t) \prod_{k=t+1}^T \frac{1}{\eta_k}
\end{equation}

then the formulas for $\gamma$'s and $\xi$'s remain the same since the normalizers cancel out.

We verify the correctness of our implementation with the library HMM function in
MATLAB~\cite{matlab}, and our calculations of $\alpha$, $\beta$, and $\gamma$ seemed to match, though we found some disagreement in the final results, on both corner cases and non-corner cases in our final checks.  We believe that MATLAB may handle some corner cases differently, and in addition, may have a different method for computing some of the final sums necessary to update the model parameters.  Regardless, we don't think we missed anything in our implementation that would cause the performance to drastically differ, and that is what we are interested in here.

\subsection{Failed Parallel Attempt}
We first approached the challenge by representing the Hidden Markov model computation as one over a bipartite graph.  Consider the Markov chain that underlies the hidden Markov model.  We created two vertices for each state in this chain, one designated the ``odd vertex" and one designated the ``even vertex".  Call original vertex $u$'s two vertices $u_{odd}$ and $u_{even}$. For each original edge $(u, v)$ in the hidden Markov model, we created two edges in our bipartite graph, $(u_{odd}, v_{even})$, $(u_{even}, v_{odd})$.   Each vertex $v$, regardless of being an odd or even vertex, stores:

\begin{itemize}
	\item Two arrays of size $\Theta(T)$, where $T$ is the length of the observation sequence: one for each of the original state $v$'s forward and backward probabilities.  Odd vertices store forward-backward probabilities corresponding to odd time steps, and even vertices store probabilities corresponding to even time steps.  Note that it is possible to save space by a constant factor in this implementation by allocating an array of roughly $T/2$ in size.  In our initial implementation, we did not end up making this space optimization.

	\item The column of the observation matrix corresponding to its associated state in the original Markov chain, of size $\Theta(M)$.  Note that in this approach, both copies of the vertex---odd and even---receive the same column of the observation matrix, though this redundant space usage does not affect our asymptotic space usage.
\end{itemize}

Edges initially store the appropriate entry of the initial transition matrix $A$.  Like the observation matrix columns, the edge $a_{ij}$ values are duplicated, one for each version of an edge.

With this computation graph, one can now ``ping pong" the computations of the forward and backward probabilities across the graph's sides.  Recall the recursive case for the forward probabilities as in equation \ref{alpha}.

Suppose $t+1$ is an even number.  Then vertex $j_{even}$ can compute a sum based
on its incoming edges, one for each original state $i$ of the form $(i_{odd},
j_{even})$.  Vertex $i_{odd}$ stores the forward probability $\alpha_{i}(t)$,
since $t$ is an odd number.  Of course, the edge joining these vertices stores
the relevant transition probability $a_{ij}$.  The relevant emission probability
$b_j(O_{t + 1})$ is on the vertex $j_{even}$ too.  Thus, this sum can be computed using GraphLab's $\tt{triple\_apply}$ function, each relevant edge adding one term each to an accumulator variable on its right endpoint, $j_{even}$.  The backward probabilities can be computed in a similar way.

Recall the equation for the next phase of the Baum-Welch training algorithm, computing $\gamma_{it}$ values (as in equation \ref{gamma}).

This parameter is straightforward to compute in parallel, given our graph representation: every vertex can simply compute the $\gamma_{it}$ for odd or even $t$ pertaining to its original state $i$ in the Markov chain.  This computation can be done in parallel across all vertices.   Note that in our implementation, the normalization scheme is such that $\sum_{j = 1}^N \alpha_{j}(t)\beta_{j}(t) = 1$, so the division is unnecessary, which simplifies this portion of the algorithm.

Before we get to the good stuff---updating the $A$ matrix---we need to compute
one more parameter, $\xi_{i,j}(t)$ (see equation \ref{xi}).
Again, due to normalization, the denominator here turns out to be 1.  We also
noted a potential optimization that we applied in serial as well: the update of
the hidden markov model parameters only relies on the sum $\sum_{t = 1}^{T-1}
\xi_{i,j}(t) $ for fixed $(i, j)$, so in our implementation, we only store the sum for each $(i, j)$.

The numerator seemed like a perfect fit for another $\tt{triple\_apply}$
function call.  An edge joins the node storing the $\alpha_{i}(t)$ and the node
storing $\beta_{j}(t+1)$, since one of these $t$ values is odd and the other is
even.  This edge then provides the relevant transition probability $a_{ij}$ and
the observation matrix entry can just be grabbed off the version of node $j$,
odd or even.  We realized a problem. There are, as stated before, two copies of
the nodes, and two edges joining copies with different parities.  The sums of
the $\xi_{i,j}(t)$ values that are computed on each edge will only have access roughly half of the terms from the desired $(i, j)$ sum, because of the duplication of nodes into odd and even versions in this approach.  Communication would be required to combine the two values, and send them out to each edge for updates.  We considered performing join operations on the associated edge data table, but we worried about performance of this fix and ultimately opted for what we think is a simpler approach.

\subsection{Final Parallel Implementation}

Our final parallel implementation uses a much simpler graph representation: the graph is simply the hidden Markov chain in question! No duplicated edges or vertices.  The vertices and edges store the same data as the previous implementation, except that each vertex stores all of its forward and backward probabilities, not just half of them.  By design, this approach is compact, and also distributes space for the data evenly across each of the vertices.

Computation of the forward probabilities, backward probabilities, and
$\gamma_{it}$ probabilities can be computed similarly to before.\footnote{In our
    implementation, we perform the computation of the $\gamma_{it}$ using
vectorized arithmetic operations that GraphLab provides on the associated vertex
data table.}  Computation of the $\xi_{i, j}(t)$ values can be computed similar to before, except now each edge has full access to the relevant $\alpha$, $\beta$ values for all $t$, not just half of them.

\subsection{Secret Weapon}

We had a lot of secret weapons.

\begin{itemize}
    \item \textbf{GraphLab Create SDK (C++)}~\cite{graphlab-create}.  Released in December 2014, this SDK ``provide[s] 3rd party extensibility to GraphLab Create."  After difficulty with achieving strong performance in the GraphLab programming assignment, Surat Teerapittayanon suggested that we try this SDK, and we found it powerful.

    \item \textbf{Dato Forum}~\cite{datoforum}.  Dato, the company behind GraphLab, was extremely helpful with our reports of bugs and other issues when working with their SDK.  We even suggested an optimization for the core library of GraphLab, which they recommended we implement and submit in a pull request!

\item \textbf{MATLAB}~\cite{matlab}.  MATLAB comes with an HMM library in its Machine Learning Toolkit, and this library offers invaluable functions that allow one to see $\alpha_i(t)$, $\beta_i(t)$,  and $\gamma_{it}$ values for given training sequences, which was enormously invaluable in debugging.  MATLAB also reports which normalizing values are used, and used a trial-and-error debugging approach to figure out what was wrong with our code at times.  See the disclaimer earlier in the paper about matching the MATLAB implementation.
\end{itemize}


\section{Adaptive AR Model Based Time-Frequency Estimation}

\begin{enumerate}[label=\alph*), leftmargin=*]

%% a)
\item
%

In figure \ref{fig:4_2_a_1} the non-stationary frequency, $f(n)$, and the corresponding phase, $\phi(n)$, time-series are provided.
Clearly, the frequency is time-variant, hence a model capable of capturing non-stationarity is required.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/adaptive-AR-model-based-time-frequency-estimation/assets/a/frequency}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/adaptive-AR-model-based-time-frequency-estimation/assets/a/phase}
    \end{subfigure}
    \caption{FM: non-stationary frequency and phase time-series.}
    \label{fig:4_2_a_1}
\end{figure}

As expected, the static autoregressive order 1 model, AR(1), in figure \ref{fig:4_2_a_2}, fails to adapt to $f(n)$ over time, since a single peak is detected.
Increasing model order does not improve performance, since the static approach is incapable of dealing with non-stationary series, regardless the capacity of the model.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/adaptive-AR-model-based-time-frequency-estimation/assets/a/complete_aryule_1}
    \end{subfigure}
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/adaptive-AR-model-based-time-frequency-estimation/assets/a/complete_aryule_5}
    \end{subfigure}
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/adaptive-AR-model-based-time-frequency-estimation/assets/a/complete_aryule_10}
    \end{subfigure}
    \caption{FM: power spectrum and AR(p) models.}
    \label{fig:4_2_a_2}
\end{figure}

The frequency $f(n)$ is a branch function, a constant, a linear and a quadratic function, for different values of $n$. Splitting the signal into three series of segment length $N_{segment} = 500$,
we confirm that only the first segment (constant frequency $f = 100 Hz$ over time) can be adequately modelled, while the rest are failed due to their non-stationary nature.
Figure \ref{fig:4_2_a_2} illustrates the power spectral density estimates for the three segments.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/adaptive-AR-model-based-time-frequency-estimation/assets/a/minibatch_1_aryule_1}
    \end{subfigure}
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/adaptive-AR-model-based-time-frequency-estimation/assets/a/minibatch_2_aryule_1}
    \end{subfigure}
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{report/widely-linear-filtering-and-adaptive-spectrum-estimation/adaptive-AR-model-based-time-frequency-estimation/assets/a/minibatch_3_aryule_1}
    \end{subfigure}
    \caption{FM: power spectrum of segments.}
    \label{fig:4_2_a_3}
\end{figure}

%% b)
\item
%

Having considered the deficiencies of the static AR models, a dynamic approach is now taken, using the CLMS algorithm, since the signal is complex.
Comparing figure \ref{fig:4_2_b}, the time-frequency spectrum plots, with the time-series $f(n)$ in figure \ref{fig:4_2_a_1}, we verify that the dynamic CLMS AR(1) model
enables the modelling of non-stationary processes.

The step-size $\mu$ of the CLMS filter is also varied, where for small values (i.e $\mu = 0.001$) the filter does not converge in-time to the correct frequencies,
while large $\mu$ values (i.e $\mu = 0.1$) lead to oscillations around the target value. This reflect once again the trade-off between convergence rate and steady-state error.
The wider bounds in the time-frequency plots (i.e for $\mu = 0.1$) the less certain the estimate is.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{{report/widely-linear-filtering-and-adaptive-spectrum-estimation/adaptive-AR-model-based-time-frequency-estimation/assets/b/time_frequency-mu_0.001}.pdf}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{{report/widely-linear-filtering-and-adaptive-spectrum-estimation/adaptive-AR-model-based-time-frequency-estimation/assets/b/time_frequency-mu_0.010}.pdf}
    \end{subfigure}
    ~
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{{report/widely-linear-filtering-and-adaptive-spectrum-estimation/adaptive-AR-model-based-time-frequency-estimation/assets/b/time_frequency-mu_0.050}.pdf}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{{report/widely-linear-filtering-and-adaptive-spectrum-estimation/adaptive-AR-model-based-time-frequency-estimation/assets/b/time_frequency-mu_0.100}.pdf}
    \end{subfigure}
    \caption{FM: CLMS time-frequency plots.}
    \label{fig:4_2_b}
\end{figure}

%
\end{enumerate}


\chapter{Frame Camera Models}

Ames Stereo Pipeline supports a generic Pinhole camera model with
several lens distortion models which cover common calibration methods,
and also the somewhat more complicated panoramic (\textit{optical bar})
camera model.

\section{Pinhole Models}
\label{pinholemodels}

\section{Overview}

The generic Pinhole model uses the following parameters:

\begin{itemize}{}
\item  \textit{fu} = The focal length in horizontal pixel units.
\item  \textit{fv} = The focal length in vertical pixel units.
\item  \textit{cu} = The horizontal offset of the principal
point of the camera in the image plane in pixel units, from 0,0.
\item  \textit{cv} = The vertical offset of the principal
point of the camera in the image plane in pixel units, from 0,0.
\item  \textit{pitch} = The size of each pixel in the units used to specify
the four parameters listed above.  This will usually either be 1.0 if they
are specified in pixel units or alternately the size of a pixel in millimeters.
\end{itemize}

The focal length is sometimes known as the \textit{principal
distance}. The value $cu$ is usually approximately half the image width in
pixels times the pitch, while $cv$ is often the image height in pixels
times the pitch, though there are situations when these can be quite different.

A few sample Pinhole models are shown later in the text. The underlying mathematical model
is described in section \ref{pinholemath}.

Along with the basic Pinhole camera parameters, a lens distortion model
can be added.  Note that the units used in the distortion model must
match the units used for the parameters listed above.  For example, if the
camera calibration was performed using units of millimeters the focal lengths etc. must be given in
units of millimeters and the pitch must be equal to the size of each pixel in millimeters.
The following lens distortion models are currently supported:

\begin{itemize}{}
\item  \textbf{Null} = A placeholder model that applies no distortion.

\item  \textbf{Tsai} = A common distortion model similar to the one used by OpenCV and THEIA.
 This model uses the following parameters:
  \begin{description}
    \item \textit{K1, K2} = Radial distortion parameters.
    \item \textit{P1, P2} = Tangential distortion parameters.
  \end{description}
\hfill \\ The following equations describe the distortion, starting with the undistorted pixel $(Px, Py)$:

\[ (x, y) = \left(\frac{Px - cu}{fu}, \frac{Py-cv}{fv}\right) \]
\[ r^{2} = x^{2} + y^{2} \]
\[ x(distorted) = x\left(K_{1}r^{2} + K_{2}r^{4} + 2P_{1}y + P_{2}\left(\frac{r^{2}}{x} + 2x\right)\right)  \]
\[ y(distorted) = y\left(K_{1}r^{2} + K_{2}r^{4} + 2P_{2}x + P_{1}\left(\frac{r^{2}}{y} + 2y\right)\right)  \]
\hfill \\ References:
\begin{description}
   \item Roger Tsai, A Versatile Camera Calibration Technique for a High-Accuracy 3D
          Machine Vision Metrology Using Off-the-shelf TV Cameras and Lenses
\end{description}
\hfill \\ Note that this model uses normalized pixel units.

\item  \textbf{Adjustable Tsai} = A variant of the Tsai model where any number of \textit{K} terms and a skew term (alpha) can be used.  Can apply the AgiSoft Lens calibration parameters.

\item  \textbf{Brown-Conrady} = An older model based on a centering angle.

 This model uses the following parameters:
  \begin{description}
    \item \textit{K1, K2, K3} = Radial distortion parameters.
    \item \textit{P1, P2} = Tangential distortion parameters.
    \item \textit{xp, yp} = Principal point offset.
    \item \textit{phi} = Tangential distortion angle in radians.
  \end{description}
\hfill \\ The following equations describe the distortion:
\[ x = x(distorted) - xp \]
\[ y = y(distorted) - yp \]
\[ r^{2} = x^{2} + y^{2} \]
\[ dr = K_{1}r^{3} + K_{2}r^{5} + K_{3}r^{7} \]
\[ x(undistorted) = x + x\frac{dr}{r} - (P_{1}r^{2} +P_{2}r^{4})\sin(phi) \]
\[ y(undistorted) = y + y\frac{dr}{r} + (P_{1}r^{2} +P_{2}r^{4})\cos(phi) \]
\hfill \\ Note that this model uses non-normalized pixel units, so they
are in mm.

\hfill \\ References:
\begin{description}
   \item Decentering Distortion of Lenses - D.C. Brown,
          Photometric Engineering, pages 444-462, Vol. 32, No. 3, 1966
   \item Close-Range Camera Calibration - D.C. Brown,
          Photogrammetric Engineering, pages 855-866, Vol. 37, No. 8, 1971
\end{description}

\item  \textbf{Photometrix} = A model matching the conventions used by the Australis software from Photometrix.

  \begin{description}
    \item \textit{K1, K2, K3} = Radial distortion parameters.
    \item \textit{P1, P2} = Tangential distortion parameters.
    \item \textit{xp, yp} = Principal point offset.
    \item \textit{B1, B2} = Unused parameters.
  \end{description}
\hfill \\ The following equations describe the distortion:
\[ x = x(distorted) - xp \]
\[ y = y(distorted) - yp \]
\[ r^{2} = x^{2} + y^{2} \]
\[ dr = K_{1}r^{3} + K_{2}r^{5} + K_{3}r^{7} \]
\[ x(undistorted) = x + x\frac{dr}{r} + P_{1}(r^{2} +2x^{2}) + 2P_{2}xy \]
\[ y(undistorted) = y + y\frac{dr}{r} + P_{2}(r^{2} +2y^{2}) + 2P_{1}xy \]
\hfill \\ Note that this model uses non-normalized pixel units, so they
are in mm.

\item  \textbf{RPC} = A rational polynomial coefficient model.

In this model, one goes from distorted coordinates $(x, y)$ to undistorted
coordinates via the formula

\[ x(undistorted) = \frac{P_1(x, y)}{Q_1(x, y)} \]
\[ y(undistorted) = \frac{P_2(x, y)}{Q_2(x, y)} \]

The functions in the numerator and denominator are polynomials in $x$
and $y$ with certain coefficients. The degree of
polynomials can be any positive integer.

RPC distortion models can be generated as approximations to other pre-existing
models with the tool \texttt{convert\_pinhole\_model} (section \ref{convertpinholemodel}).

This tool also creates RPC to speed up the reverse operation, of going
from undistorted to distorted pixels, and those polynomial coefficients
are also saved as part of the model.

\end{itemize}

\hrule
\subsection{File Formats}
\label{file_format}
\bigskip

ASP Pinhole model files are written in an easy to work with plain text format
using the extension \texttt{.tsai}.
A sample file is shown below.

\begin{verbatim}
VERSION_4
PINHOLE
fu = 28.429
fv = 28.429
cu = 17.9712
cv = 11.9808
u_direction = 1  0  0
v_direction = 0  1  0
w_direction = 0  0  1
C = 266.943 -105.583 -2.14189
R = 0.0825447 0.996303 -0.0238243 -0.996008 0.0832884 0.0321213 0.0339869 0.0210777 0.9992
pitch = 0.0064
Photometrix
xp = 0.004
yp = -0.191
k1 = 1.31024e-04
k2 = -2.05354e-07
k3 = -5.28558e-011
p1 = 7.2359e-006
p2 = 2.2656e-006
b1 = 0.0
b2 = 0.0
\end{verbatim}

The first half of the file is the same for all Pinhole models:

\begin{itemize}{}
\item  \texttt{VERSION\_X} = A header line used to track the format of the file.
\item  \texttt{PINHOLE} = The type of camera model, so that other types can be stored with the .tsai extension.
\item  \texttt{fu, fv, cu, cv} = The first four intrinsic parameters described in the previous section.
\item  \texttt{u, v, and w\_direction} = These lines allow an additional permutation of the
axes of the camera coordinates.  By default, the positive column direction aligns with x, the
positive row direction aligns with y, and downward into the image aligns with z.
\item  \texttt{C} = The location of the camera center, usually in the geocentric coordinate system (GCC/ECEF).
\item  \texttt{R} = The rotation matrix describing the camera's absolute pose in the coordinate system (section \ref{pinholemath}).
\item  \texttt{pitch} = The pitch intrinsic parameter described in the previous section.
\end{itemize}

The second half of the file describes the lens distortion model being used.  The name of the
distortion model appears first, followed by a list of the parameters for that model.  The number
of parameters may be different for each distortion type.  Samples of each format are shown below:

\begin{itemize}{}
\item  \textbf{Null}
\begin{verbatim}
NULL
\end{verbatim}

\item  \textbf{Tsai}
\begin{verbatim}
TSAI
k1 = 1.31024e-04
k2 = -2.05354e-07
p1 = 0.5
p2 = 0.4
\end{verbatim}

\item  \textbf{Adjustable Tsai}
\begin{verbatim}
AdjustableTSAI
Radial Coeff: Vector3(1.31024e-04, 1.31024e-07, 1.31024e-08)
Tangential Coeff: Vector2(-2.05354e-07, 1.05354e-07)
Alpha: 0.4
\end{verbatim}

\item  \textbf{Brown-Conrady}
\begin{verbatim}
BrownConrady
xp = 0.5
yp = 0.4
k1 = 1.31024e-04
k2 = -2.05354e-07
k3 = 1.31024e-08
p1 = 0.5
p2 = 0.4
phi = 0.001
\end{verbatim}

\item  \textbf{Photometrix}
\begin{verbatim}
Photometrix
xp = 0.004
yp = -0.191
k1 = 1.31024e-04
k2 = -2.05354e-07
k3 = -5.28558e-011
p1 = 7.2359e-006
p2 = 2.2656e-006
b1 = 0.0
b2 = 0.0
\end{verbatim}

\item  \textbf{RPC}

\begin{verbatim}
RPC
rpc_degree = 1
image_size = 5760 3840
distortion_num_x   = 0 1 0
distortion_den_x   = 1 0 0
distortion_num_y   = 0 0 1
distortion_den_y   = 1 0 0
undistortion_num_x = 0 1 0
undistortion_den_x = 1 0 0
undistortion_num_y = 0 0 1
undistortion_den_y = 1 0 0
\end{verbatim}

This sample RPC lens distortion model represents the case of no distortion,
when the degree of the polynomials is 1, and both the distortion and undistortion
formula leave the pixels unchanged, that is, the distortion transform is
\begin{equation*}
(x, y) \to (x, y) = \left(\frac{ 0 + 1\cdot x + 0\cdot y}{1 + 0\cdot x + 0\cdot y},
 \frac{0 + 0\cdot x + 1\cdot y)}{1 + 0\cdot x + 0\cdot y}\right).
\end{equation*}

In general, if the degree of the polynomials is $n$, there are $2(n+1)(n+2)$ coefficients.
The zero-th degree coefficients in the denominator are always set to 1.

\end{itemize}{}

For several years Ames Stereo Pipeline generated Pinhole files in the binary \texttt{.pinhole} format.
That format is no longer supported.

Also in the past Ames Stereo Pipeline has generated a shorter version of the current file
format, also with the extension \texttt{.tsai}, which only supported the TSAI lens distortion model.
Existing files in that format can still be used by ASP.

Note that the \texttt{orbitviz} tool can be useful for checking the formatting of \texttt{.tsai}
files you create and to estimate the position and orientation. To inspect the orientation use
the optional \texttt{.dae} model file input option and observe the rotation of the 3D model.

\subsection{How the Pinhole model is applied}
\label{pinholemath}

As mentioned in section \ref{file_format}, the ASP Pinhole models store
the focal length as $fu$ and $fv$, the optical center $(cu, cv)$ (which
is the pixel location at which the ray coming from the center of the
camera is perpendicular to the image plane, in units of the pixel
pitch), the vector $C$ which is the camera center in world coordinates
system, and the matrix $R$ that is the transform from camera to world
coordinates.

To go in more detail, a point $Q$ in the camera coordinate system gets transformed
to a point $P$ in the world coordinate system via:
\begin{equation*}
 P = RQ + C
\end{equation*}

Hence, to go from world to camera coordinates one does:
\begin{equation*}
 Q = R^{-1}  P - R^{-1}  C
\end{equation*}

From here the pixel location is computed as:

\begin{equation*}
 \frac{1}{p} \left(fu \frac{Q_1}{Q_3} + cu, fv \frac{Q_2}{Q_3} + cv\right)
\end{equation*}

where $p$ is the pixel pitch.

\section{Panoramic Camera Model}
\label{panoramic}

ASP also supports a simple panoramic/optical bar camera model for use with images such as
the declassified Corona KH4 and Keyhole KH9 images. It implements the model
from \cite{schenk2003rigorous} with the motion compensation from \cite{sohn2004mathematical}.

Such a model looks as follows:
\begin{verbatim}
VERSION_4
OPTICAL_BAR
image_size = 110507 7904
image_center = 55253.5 3952
pitch = 7.0e-06
f = 0.61000001430511475
scan_time = 0.5
forward_tilt = -0.261799
iC = -1047140.9611702315 5508464.4323527571 3340425.4078937685
iR = -0.96635634448923746 -0.16918164442572045 0.1937343197650008 -0.23427205529446918 0.26804084264169648 -0.93448954557235941 0.10616976770014927 -0.94843643849513648 -0.29865750042675621
speed = 7700
mean_earth_radius = 6371000
mean_surface_elevation = 4000
motion_compensation_factor = 1.0
scan_dir = left
\end{verbatim}

Here, the image size and center are given in pixels, with the width followed by the
height. The pixel pitch and focal length \texttt{f} are in meters. The
scan time is seconds, the forward tilt is in radians,
the speed is in meters per second, and the Earth radius and mean
surface elevation are in meters. The initial camera center \texttt{iC} is
in meters, and the rotation matrix \texttt{iR} stores the absolute pose.
\texttt{scan\_dir} must be set to 'left' or 'right'.
\texttt{scan\_dir} and \texttt{use\_motion\_compensation}
control how the sensor model accounts accounts for the motion of the satellite
during the image scan.  Without the benefit of detailed historical documents it may
require experimentation to find the good initial values for these cameras.  When using
\texttt{bundle\_adjust}, the intrinsic parameters that are solved for are
\texttt{speed}, \texttt{motion\_compensation\_factor}, and \texttt{scan\_time}.


\section{The Classical Banach Spaces}
\subsection{The $L^p$ Spaces}
  \paragraph{1.}
  \begin{proof}
    Put $S=\|f\|_\infty$ and $T=\|g\|_\infty$. Then $|f(t)|\le S$ and $|g(t)|\le
    T$ a.e. Hence, $S+T\ge |f(t)|+|g(t)| \ge |f(t)+g(t)|$ a.e. Namely, $m\{t:
    \,|f(t)+g(t)|>S+T\}=0$. Thus, $S+T\ge \|f+g\|_\infty$ by the definition of
    $\esssup$.
  \end{proof}

  \paragraph{2.}
  \begin{proof}
    Put $S=\|f\|_\infty$. Since $S\ge |f|$ a.e.,
    \[
      \|f\|_p=
      \left\{\int_0^1|f|^p\right\}^{1/p} \le
      \left\{\int_0^1S^p\right\}^{1/p} = S.
    \]
    Therefore, $\uplim_{p\to\infty}\|f\|_p \le S$. For the converse part, let
    $\vep$ be any positive number. Then the measure $\delta$ of $E=\{t:\,|f(t)|>
    S-\vep\}$ is positive. Hence,
    \[
      \left\{\int_0^1|f|^p\right\}^{1/p} \ge
      \left\{\int_E|f|^p\right\}^{1/p} \ge
      \delta^{1/p}(S-\vep) \to
      S-\vep\quad\text{as $p\to\infty$.}
    \]
    Hence, $\lowlim_{p\to\infty}\ge S$, completing the proof.
  \end{proof}

  \paragraph{3.}
  \begin{proof}
    \[
      \|f+g\|_1 = \int|f+g| \le \int|f|+\int|g| = \|f\|_1+\|g\|_1.
    \]
  \end{proof}

  \paragraph{4.}
  \begin{proof}
    For every $M>\|g\|_\infty$, $|g|\le M$ a.e. Hence,
    \[
      \int|fg|\le M\int|f| = \|f\|_1 M.
    \]
    Since the choice of $M$ is arbitrary, $\int|fg|\le \|f\|_1\|g\|_\infty$.
  \end{proof}
% end

\subsection{The Minkowski and Hölder Inequalities}
  \paragraph{8}
  \begin{proof}
    $\,$\par
    (a) The logarithm function is concave, so
    \[
      \log(a^p/p+b^q/q) \ge \frac{1}{p}\log a^p + \frac{1}{q}\log b^q = \log ab.
    \]
    Taking $\exp$ on the both sides yields the inequality. The equality holds
    iff $a^p=b^q$. \par
    (b) The case where $p=\infty$ has been proved in Problem 4 and the case
    where $\|f\|_p=0$ or $\|g\|=0$ is straightforward. Hence, we assume that
    $1<p,q<\infty$ and $\|f\|_p\|g\|_q\ne 0$. \par
    Suppose $\alpha=\|f\|_p$ and $\beta=\|g\|_q$. By Young's inequality,
    \[
      \left|\frac{fg}{\alpha\beta}\right| \le
      \frac{1}{p}\left(\frac{|f|}{\alpha}\right)^p +
      \frac{1}{q}\left(\frac{|g|}{\beta}\right)^q
    \]
    for every $x$. Therefore,
    \begin{equation}
      \label{eq:6.2.8.1}
      \int|fg| = \alpha\beta\int\left|\frac{fg}{\alpha\beta}\right|
      \le \alpha\beta\int\left\{\frac{1}{p}\left(\frac{|f|}{\alpha}\right)^p +
      \frac{1}{q}\left(\frac{|g|}{\beta}\right)^q\right\}
      = \alpha\beta.
    \end{equation}
    The equality holds iff the equality in Young's inequality holds a.e. iff
    $\beta|f|^p = \alpha|g|^q$ a.e.\par
    (c) Let $p\hp=1/p$ and $q\hp=1-p\hp=-q/p$. Then for any nonnegative $c$ and
    $d$, by Young's inequality,
    \[
      cd \le \frac{c^{p\hp}}{p\hp}+\frac{d^{q\hp}}{q\hp}=
      pc^{1/p} - \frac{p}{q}d^{-q/p} \quad\Rightarrow\quad
      c^{1/p} \ge \frac{cd}{p} + \frac{d^{-q/p}}{q}.
    \]
    Putting $c=(ab)^p$ and $d=b^{-p}$ yields the desired inequality.\par
    (d) Just reverse the inequality in \eqref{eq:6.2.8.1}.
  \end{proof}
% end

\subsection{Convergence and Completeness}
  \paragraph{9.}
  \begin{proof}
    Suppose $\langle f_n\rangle\subset X$ converges to $f\in X$. Namely, for
    every $\vep>0$, there exists some $N$ such that for all $n>N$, $\|f_n-f\|<
    \vep$. Hence, for every $n,m>N$, by Minkowski inequality,
    \[
      \|f_n-f_m\| \le \|f_n-f\|+\|f-f_m\| < 2\vep.
    \]
    Hence, $\langle f_n\rangle$ is a Cauchy sequence.
  \end{proof}

  \paragraph{10.}
  \begin{proof}
    Suppose $f_n\to f$. Then $M_n=\|f_n-f\|_\infty=\esssup|f_n-f|\to 0$. Let
    $E_n=\{x:\,|f_n(x)-f(x)|>M_m\}$, each of which is with measure zero. And
    therefore $E=\bigcup_{n=1}^\infty E_n$ is with measure zero. Note that
    $\tilde{E}=\{x:\,|f_n(x)-f(x)|<M_n,\forall\,n\}$, which implies the uniform
    convergence of $f_n$ since $M_n\to 0$.\par
    For the converse part, the uniform convergence on $\tilde{E}$ implies that
    for every $\vep>0$, there exists some $N$ such that for every $n>N$ and $x
    \in\tilde{E}$, $|f_n(x)-f(x)|<\vep$. Since $mE=0$, this implies $\|f_n-f
    \|_\infty=\esssup|f_n(x)-f(x)|<\vep$. Hence, $f_n\to f$ in $L^\infty$.
  \end{proof}

  \paragraph{11.}
  \begin{proof}
    Let $\langle f_n\rangle\subset L^\infty$ be absolutely summable. Put $M_n=
    \|f_n\|_\infty$ and $A_n=\{t:\,|f_n(t)|>M_n\}$. By the definition of $\|
    \cdot\|_\infty$, $mA_n=0$. Hence, $A=\bigcup_{n=1}^\infty A_n$ is of measure
    zero. \par
    Note that $|f_n(x)|\le M_n$ for every $n$ and $x\in E\setminus A$. Thus, by
    the Weierstrass M-test, $\sum_{n=1}^\infty f_n$ converges uniformly. Hence,
    on $E\setminus A$, $\sup|\sum_{n=1}^\infty f_n - \sum_{n=1}^N f_n|\to 0$ as
    $N\to\infty$. Since $mA=0$, this implies the summability of $\langle f_n
    \rangle$.
  \end{proof}

  \paragraph{13.}
  \begin{proof}
    Suppose $\langle f_n\rangle\subset C$ be absolutely summable. Since for
    every $x$, $0\le|f_n(x)|\le\|f_n\|$, $\langle f_n\rangle$ is uniformly
    convergent on $[0,1]$. Put $s=\sum_{n=1}^\infty f_n$. Since each $f_n$ is
    continuous, so is $s$. Therefore, $s\in C$. \par
    For every $\vep>0$, there exists some $N$ such that for every $n>N$ and $x
    \in[0,1]$, $\left|s(x)-\sum_{k=1}^nf_k(x)\right|<\vep$. Hence, $\|s-
    \sum_{k=1}^nf_k\|<\vep$. Thus, $\langle f_n\rangle$ is summable and
    therefore $C$ is a Banach space.
  \end{proof}

  \paragraph{16.}
  \begin{proof}
    Since $\|f_n-f\| \ge |\|f_n\|-\|f\||$, $f_n\to f$ in $L^p$ implies $\|f_n\|
    \to \|f\|$. For the converse part, note that $2^p(|f_n|^p+|f|^p)-|f_n-f|^p
    \ge 0$ and for almost every $x$,
    \[
      2^p(|f_n|^p+|f|^p)-|f_n-f|^p \to 2^{p+1}|f|^p.
    \]
    By Fatou's Lemma,
    \begin{align*}
      2^{p+1}\|f\|^p = 2^{p+1}\int|f|^p
      &\le \lowlim\int\{2^p(|f_n|^p+|f|^p)-|f_n-f|^p\} \\
      &= 2^{p+1}\|f\|^p - \uplim\|f_n-f\|^p.
    \end{align*}
    Hence, $\uplim\|f_n-f\|^p \le 0$. Since clear that $\lowlim\|f_n-f\|^p\ge0$,
    $\lim\|f_n-f\|=0$, i.e., $f_n \to f$ in $L^p$.
  \end{proof}

  \paragraph{17.}
    I assume that $1/p+1/q=1$.
  \begin{proof}
    Since $g\in L^p$, $|g|^q$ is integrable on $E=[0,1]$ and therefore for every
    $\vep>0$, there exists some $\delta$ such that for every $A\subset E$ with
    $mA<\delta$, $\int_A|g|^q<\vep$. Meanwhile, since $f_n(x)\to f(x)$ for
    almost every $x$, by Egoroff's Theorem, there exists some $A\subset E$ with
    $mA<\delta$ such that $f_ng$ converges to $fg$ uniformly on $E\setminus A$.
    \par
    From the uniform convergence we conclude
    \begin{equation}
      \label{eq:6.17.1}
      \int_{E\setminus A} fg = \lim_{n\to\infty}\int_{E\setminus A}f_ng.
    \end{equation}
    Meanwhile, by Hölder inequality,
    \begin{align*}
      \left|\int_A (f-f_n)g\right| \le \int_A|(f-f_n)g|
      &\le\left\{\int_A|f_n-f|^p\right\}^{1/p}\left\{\int_A|g|^q\right\}^{1/q}
      \le M\vep^{1/q}.
    \end{align*}
    Hence, \eqref{eq:6.17.1} can be extended to $E$.\par
    For $p=1$, this is not true. $f_n=n\chi_{[0,1/n]}$ and $g=\chi_{[0,1]}$
    gives a counterexample.
  \end{proof}

  \paragraph{18.}
  \begin{proof}
    By Minkowski inequality,
    \[
      \|g_nf_n-gf\| = \|g_n(f_n-f)+(g_n-g)f\| \le \|g_n(f_n-f)\|+\|(g_n-g)f\|.
    \]
    Fix $\vep>0$. Since $f,g_n,g\in L^p$, $|g_n-g|^p|f|^p$ is integrable and
    therefore there exists some $\delta>0$ such that for all subsets with
    measure $<\delta$, the integral of over it $<\vep$. Meanwhile, since $g_n\to
    g$ a.e., by Egoroff's Theorem, there exists some $A\subset E=[0,1]$ with $mA
    <\delta$ such that $g_n\to g$ uniformly on $E\setminus A$ and therefore
    there exists some $N_1>0$ such that for all $n>N_1$, $|g_n(x)-g(x)|^p<\vep$
    for $x\in E\setminus A$. Thus, for every $n>N_1$,
    \begin{align*}
      \|(g_n-g)f\|
      &=\left\{\int_{E\setminus A}|g_n-g|^p|f|^p\right\}^{1/p} +
        \left\{\int_A|g_n-g|^p|f|^p\right\}^{1/p} \\
      & \le \sqrt[p]{\vep}\|f\| + \sqrt[p]{\vep} \le (\|f\|+1)\vep.
    \end{align*}
    Since $|g_n|\le M$, $\|g_n(f_n-f)\|\le M\|f_n-f\|$. And since $f_n\to f$ in
    $L^p$, there exists some $N_2>0$ such that for all $n>N_2$, $\|f_n-f\|<
    \vep$. Put $N=\max(N_1,N_2)$, then for every $n>N$,
    \[
      \|g_nf_n-gf\| \le (\|f\|+1+M)\vep.
    \]
    Hence, $g_nf_n\to gf$ in $L^p$.
  \end{proof}
% end

\subsection{Approximation in $L^p$}
  \paragraph{19.}
  \begin{proof}
    Since $\|T_\Delta f\|\le \|T_\Delta|f|\|$ and $\|f\|=\||f|\|$, we may assume
    without loss of generality that $f\ge 0$. For $p>1$, by Jensen's inequality,
    \begin{align*}
      \|T_\Delta f\|_p^p
      &= \sum_{k=1}^m\int_{\xi_{k-1}}^{\xi_k}
      \left(\frac{1}{\xi_k-\xi_{k-1}}\int_{\xi_{k=1}}^{\xi_k}f\right)^p \\
      &\le \sum_{k=1}^m\int_{\xi_{k-1}}^{\xi_k}
      \frac{1}{\xi_k-\xi_{k-1}}\int_{\xi_{k-1}}^{\xi_k}f^p \\
      &= \sum_{k=1}^m\int_{\xi_{k-1}}^{\xi_k}f^p \\
      &= \int_0^1f^p = \|f\|_p^p.
    \end{align*}
  \end{proof}
% end