\documentclass{article}

\include{stddefs}
\include{imodefs}

\begin{document}

\chapterno{8}
\chapter{The Hessian}

In Chapter \ref{chapter:convexfunctions} we exploited the second derivative
$f''(x)$ of a one variable real function $f:(a, b) \rightarrow \RR$ to analyze
convexity along with local minima and maxima.

In this chapter we introduce an analogue of the second derivative for real functions $f:\RR^n\rightarrow \RR$ of
several variables. This will be an $n\times n$ matrix. The important notion of a matrix
being positive (semi-) definite introduced in Section \ref{Sectionsymmat} will now
make its appearance.

\section{Introduction}

In Section \ref{section:Taylor} the Taylor expansion for a one variable differentiable
function $f:\RR\rightarrow \RR$ centered a $x_0$ with step size
$h  = x - x_0$ was introduced as
\begin{equation}\label{taylor}
f(x_0 + h) = f(x_0) + f'(x_0) h + \frac{1}{2} f''(x_0) h^2 + \cdots
\end{equation}

Recall that the second derivative $f''(x_0)$ contains a
wealth of information about the function. Especially if $f'(x_0) = 0$,
then we might glean from $f''(x_0)$ if $x_0$ is a local maximum or
minimum or none of these (see Theorem \ref{derconv} and review Exercise \ref{exderconv}).

We also noticed that gradient descent did not work so well only
descending along the gradient. We need to take the second
derivative into account to get a more detailed picture of the
function.

\section{Several variables}

Our main character is a differentiable function $F:\RR^n\rightarrow \RR$ in
several variables. We already know that
$$
F(x_0 + h) = F(x_0) + \nabla F(x_0) h + \epsilon(h) \abs{h},
$$
where $x_0$ and $h$ are vectors in $\RR^n$ (as opposed to the good old
numbers in \eqref{taylor}). Take a look back at Definition \ref{diffdef} for
the general definition of differentiability.

We wish to have an analogue of the Taylor expansion in \eqref{taylor} for
such a function of several variables. To this end we introduce the function
$g:\RR\rightarrow \RR$ given by
\begin{equation}\label{onedimg}
g(t) = F(x_0 + t h).
\end{equation}
Notice that
$$
g(t) = (F\circ A)(t),
$$
where $A: \RR\rightarrow \RR^n$ is the function given by $A(t) = x_0 + t h$. In particular
we get
\begin{equation}\label{chrule1}
g'(t) = F'(x_0 + t h) h = \nabla F(x_0 + t h) h
\end{equation}
by using the chain rule (see Theorem \ref{chainrule}).

\beginshex
Explain how the chain rule is applied to get \eqref{chrule1}.
\endshex

The derivative $g'(t)$ is also composed of several functions and again we may
compute $g''(t)$ by using the chain rule:
\begin{align}
  g''(t) &= (C\circ B \circ A)'(t)\\
        &= (C\circ B)'(A(t)) A'(t) \\
        &= C'(B(A(t))) B'(A(t)) A'(t),
\end{align}
where $B: \RR^n\rightarrow \RR^n$ is defined by
$$
B(v) = \nabla F(v)^T
$$
and $C:\RR^n \rightarrow \RR$ by
$$
C(v) = v^T h.
$$

\begin{definition}[emph]

The \emph{Hessian matrix} of $F$ at the point
$x\in \RR^n$ is defined by

\begin{equation*}
  \nabla^2 F(x) :=
  \begin{pmatrix}
    \dfrac{ \partial^2 F}{ \partial x_1 \partial x_1}(x) &
    \cdots & \dfrac{ \partial^2 F}{ \partial x_1 \partial
      x_n}(x)
    \\
    \vdots & \ddots & \vdots
    \\
    \dfrac{ \partial^2 F}{ \partial x_n \partial x_1}(x) &
    \cdots & \dfrac{\partial^2 F}{ \partial x_n\partial
      x_n}(x)
  \end{pmatrix}
  .

\end{equation*}
\end{definition}

A very important observation is that $\nabla^2 F(x)$ above is a
symmetric matrix. Again, you should review what this means by clicking
back to Section \ref{Sectionsymmat}.

\beginshex
Why is the Hessian matrix symmetric?
\endshex

\begin{example}\label{sagegradhess}
  Suppose that $f: \RR^2\rightarrow \RR$ is given by
  $$
  f(x, y) = \sin(x y) + x^2 y^2 + y.
  $$
  Then the gradient
  $$
  \nabla f = \left(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y} \right)
  $$
  and the Hessian

  $$
  \nabla^2 f =
  \begin{pmatrix}
    \dfrac{\partial^2 f}{\partial x^2} & \dfrac{\partial^2 f}{\partial x \partial y} \\
    \\
    \dfrac{\partial^2 f}{\partial y \partial x} & \dfrac{\partial^2 f}{\partial y^2}
  \end{pmatrix}
  $$
  of $f$ are computed in the Sage window below.

  \begin{sage}
x, y = var('x, y')
f = sin(x*y) + x^2*y^2 + y
print("gradient = ", f.gradient())
print("Hessian = ", f.hessian())
  \end{sage}

See the \url{further documentation}{http://doc.sagemath.org/html/en/reference/calculus/sage/calculus/functions.html} for Calculus functions in Sage.
\end{example}

\beginshex
Verify (just this once) by hand the computations done by Sage in Example \ref{sagegradhess}.

Also, experiment with a few other functions in the Sage window and compute their
Hessians.
\endshex

By applying Proposition \ref{proppd} it is not too hard to see that the Hessian
matrix fits nicely into the framework above, since
\begin{equation}\label{exhess}
B'(v) = \nabla^2 F(v).
\end{equation}

The full application of the chain rule then gives
\begin{equation}\label{chrule2}
g''(t) = h^T \nabla^2 F(x_0 + t h) h.
\end{equation}

\beginshex
Give a detailed explanation as to why \eqref{exhess} holds.
\endshex

\section{Newton's method for finding critical points}

We may use Newton's method for computing critical points for a function
$F:\RR^n\rightarrow \RR$ of several variables. Recall that a
critical point is a point $x_0\in \RR^n$ with $\nabla F(x_0) = 0$.
By \eqref{onestepmultnewt} and \eqref{exhess} the computation in Newton's method becomes
\begin{equation}\label{newthess}
x_1 = x_0 - \left(\nabla^2 F(x_0)\right)^{-1} \nabla F(x_0).
\end{equation}
In practice
the (inverse) Hessian appearing in \eqref{newthess} is often a heavy
computational burden. This leads to the socalled
\url{quasi-Newton methods}{https://en.wikipedia.org/wiki/Quasi-Newton_method}, where
the inverse Hessian in \eqref{newthess} is replaced by other matrices.

\begin{example}\label{Exbadlogist}
We will return to the logistic regression in Example \ref{challengerexample} about the
Challenger disaster. Here we sought to maximize the function
  \begin{equation}\label{logtwo}
  \ell(\alpha, \beta) = \sum_{i=1}^m E_i (\alpha + \beta x_i) - \log(1 + e^{\alpha + \beta x_i}).
  \end{equation}

In order to employ Newton's method we compute the gradient and the Hessian of \eqref{logtwo}
\begin{align}\label{hesslogist}
  \frac{\partial \ell}{\partial \alpha} &= \sum_{i=1}^m E_i - \sigma(\alpha + \beta x_i)\\
  \frac{\partial \ell}{\partial \beta} &= \sum_{i=1}^m E_i x_i - x_ i\sigma(\alpha + \beta x_i)\\
  \frac{\partial^2 \ell}{\partial \alpha^2} &= \sum_{i=1}^m - \sigma'(\alpha + \beta x_i)\\
  \frac{\partial^2 \ell}{\partial \beta \partial \alpha} &= \sum_{i=1}^m -\sigma'(\alpha + \beta x_i) x_i\\
  \frac{\partial^2 \ell}{\partial \beta^2} &= \sum_{i=1}^m - \sigma'(\alpha + \beta x_i) x_i^2,
\end{align}
where
$$
\sigma(t) = \frac{1}{1 + e^{-t}}
$$
is the sigmoid function.

Notice the potential problem in using Newton's method here: the formula for
the second order derivatives in \eqref{hesslogist} show that if the $\alpha +\beta x_i$ are just mildly big, say $\geq 50$, then
the Hessian is extremely close to the zero matrix and therefore Sage considers it non-invertible and
\eqref{newthess} fails.

In the code below we have nudged the initial vector so that it works, but you
can easily set other values and see its failure. Optimization is not just mathematics, it also calls
for some good (engineering) implementation skills (see for example details on the
\url{quasi Newton algorithms}{https://en.wikipedia.org/wiki/Quasi-Newton_method}).

In the instance below we do, however, get a gradient that is pratically $(0, 0)$.

\begin{hideinbutton}{Code for Newton's method}
\begin{sage}
x0 = [11,-0.2]
noofits = 10

x = [53.0, 56.0, 57.0, 63.0, 66.0, 67.0, 67.0, 67.0, 68.0, 69.0, 70.0, 70.0, 70.0, 70.0, 72.0, 73.0, 75.0, 75.0, 76.0, 76.0, 78.0, 79.0, 80.0, 81.0]
E = [1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

Es = sum(E)
Exs = sum(u*v for (u, v) in zip(x, E))

def sigmoid(a, b, x):
  s = 1/(1 + exp(-a - b*x))
  return s.n()

def sigmoidm(a, b, x):
  s = sigmoid(a, b, x)
  return s*(1-s)

def gradient(v):
  a = v[0]
  b = v[1]
  return vector((Es - sum(sigmoid(a, b, t) for t in x), Exs - sum(t*sigmoid(a, b, t) for t in x)))

def hessian(v):
  a = v[0]
  b = v[1]
  h11 = - sum(sigmoidm(a, b, t) for t in x)
  h12 = - sum(sigmoidm(a, b, t)*t for t in x)
  h22 = - sum(sigmoidm(a, b, t)*t*t for t in x)
  return matrix([[h11, h12], [h12, h22]])

def newtonstep(v0):
  d = gradient(v0)
  h = hessian(v0)
  return v0 - h.inverse()*d

v = vector(x0)
for k in range(noofits):
  print(v)
  v = newtonstep(v)

print("x0 = ", x0)
print("Number of Newton iterations =  ", noofits)
print("Predicted maximal point =  ", v)

alpha = v[0]
beta = v[1]

print("Gradient at predicted maximal point = ", gradient(v))
print("Predicted probability of failure at 31F = ", sigmoid(alpha, beta, 31))
\end{sage}
\end{hideinbutton}
\end{example}

\subsection{Transforming data for better numerical performance}

The numerical problems with Newton's method in Example \ref{Exbadlogist} can be prevented by transforming the input data.
It makes sense to transform data from large numbers to
smaller numbers around $0$. There is a rather standard way of doing this.

Suppose in logistic regression we have a set of data
\begin{equation}\label{datapts}
x_1, x_2, \dots, x_n
\end{equation}
associated with outcomes $E_1, \dots, E_n$. Then the function
  $$
  \ell(\alpha, \beta) = \sum_{i=1}^m E_i (\alpha + \beta x_i) - \log(1 + e^{\alpha + \beta x_i}).
  $$
  from Example \ref{exlogist} becomes much more manageable if we first
  transform the data according to
  $$
  x'_i = \frac{x_i - \overline{x}}{\sigma}
  $$
  and instead optimize the function
  $$
  \ell'(\alpha, \beta) = \sum_{i=1}^m E_i (\alpha + \beta x'_i) - \log(1 + e^{\alpha + \beta x'_i}).
  $$
  Here
  $$
  \overline{x} = \frac{x_1 + x_2 + \cdots + x_n}{n}
  $$
  is the mean value and
  $$
  \sigma^2 = \frac{(x_1 - \overline{x})^2 + (x_2 - \overline{x})^2 + \cdots + (x_n - \overline{x})^2}{n}
  $$
  the variance of the data in \eqref{datapts}.

  Now if $\alpha'$ and $\beta'$ is an optimum for $\ell'$, then
  \begin{align}\label{transformtrick}
    \alpha &= \alpha' - \frac{\overline{x}}{\sigma} \beta'\\
    \beta &= \frac{\beta'}{\sigma}
  \end{align}
  is an optimum for $\ell$, since
  $$
  \ell'(\alpha, \beta) = \ell\left(\alpha - \beta \frac{\overline{x}}{\sigma}, \frac{\beta}{\sigma}\right).
  $$

\beginshex
Why is the claim/trick alluded to in \eqref{transformtrick} true?

Below is a snippet of Sage code implementing the trick in \eqref{transformtrick}.
The function \texttt{test} takes as input \texttt{x0} (an initial vector like \texttt{[0,0]}) and
\texttt{noofits} (the number of iterations of Newton's method). You execute this in the Sage
window by adding for example
\begin{code}
  test([0,0], 10)
\end{code}
and then pressing \texttt{Compute}.

Experiment and compare with the official output from Example \ref{challengerexample}. Also, compute
the gradient of the output below for the original non-transformed problem.

\begin{hideinbutton}{Transformed code}
\begin{sage}
x = [53.0, 56.0, 57.0, 63.0, 66.0, 67.0, 67.0, 67.0, 68.0, 69.0, 70.0, 70.0, 70.0, 70.0, 72.0, 73.0, 75.0, 75.0, 76.0, 76.0, 78.0, 79.0, 80.0, 81.0]
E = [1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
mean = sum(x)/len(x)
std = sqrt(sum(map(lambda t: (t-mean)^2, x))/len(x))

x = list(map(lambda t: (t-mean)/std, x))

Es = sum(E)
Exs = sum(u*v for (u, v) in zip(x, E))

def sigmoid(a, b, x):
  s = 1/(1 + exp(-a - b*x))
  return s.n()

def sigmoidm(a, b, x):
  s = sigmoid(a, b, x)
  return s*(1-s)

def gradient(v):
  a = v[0]
  b = v[1]
  return vector((Es - sum(sigmoid(a, b, t) for t in x), Exs - sum(t*sigmoid(a, b, t) for t in x)))

def hessian(v):
  a = v[0]
  b = v[1]
  h11 = - sum(sigmoidm(a, b, t) for t in x)
  h12 = - sum(sigmoidm(a, b, t)*t for t in x)
  h22 = - sum(sigmoidm(a, b, t)*t*t for t in x)
  return matrix([[h11, h12], [h12, h22]])

def newtonstep(v0):
  d = gradient(v0)
  h = hessian(v0)
  return v0 - h.inverse()*d

def test(x0, noofits):
  v = vector(x0)
  for k in range(noofits):
    print(v)
    v = newtonstep(v)
  print("Number of Newton iterations =  ", noofits)
  print("Predicted maximal point in transformed problem =  ", v)
  tv =   matrix([[1, -mean/std], [0, 1/std]])*v
  print("Predicted maximal point in original problem =  ", tv)

\end{sage}
\end{hideinbutton}
\endshex

\section{The Taylor series in several variables}

Now we are in a position to state at least the first terms in the
Taylor expansion for a differentiable function $F:\RR^n\rightarrow \RR$.
The angle of the proof is to reduce to the one-dimensional case through
the function $g(t)$ defined in \eqref{onedimg}. Here
one may prove that

\begin{equation}\label{taylorg}
g(t) = g(0) + g'(0) t + \frac{1}{2} g''(0) t^2 + \epsilon(t)t^2,
\end{equation}
where $\epsilon(0) = 0$ with $\epsilon$ continuous at $0$, much like
in the definition of differentiability except that we also include
the second derivative.

Now \eqref{taylorg} translates into

\begin{equation}\label{critminmax}
  F(x_0 + t h) = F(x_0) + \left(\nabla F(x_0) h \right) t + \frac{1}{2} \left( h^T \nabla^2 F(x_0) h \right) t^2 + \epsilon(t) t^2
\end{equation}
by using \eqref{chrule1} and \eqref{chrule2}.

From \eqref{critminmax} one reads the following nice criterion, which may be viewed as a several variable generalization of Theorem \ref{derconv}.

\begin{theorem}[emph]\label{thmcritminmax}
  Let $x_0$ be a critical point for $F:\RR^n\rightarrow \RR$. Then
  \begin{enumerate}[(i)]
  \item
    $x_0$ is a local minimum if $\nabla^2 F(x_0)$ is positive definite.
  \item
    $x_0$ is a local maximum if $-\nabla^2 F(x_0)$ is positive definite (here we call
    $\nabla^2 F(x_0)$ negative definite).
  \item\label{thmcritminsaddle}
    $x_0$ is a \url{saddle point}{https://en.wikipedia.org/wiki/Saddle_point} if $\nabla^2 F(x_0)$ is indefinite.
  \end{enumerate}
\end{theorem}

\begin{remark}
  We need to clarify two things concerning the above theorem.

  \begin{enumerate}[(i)]
  \item
    An $n\times n$ indefinite matrix is a symmetric matrix $A$ with the property that there exists
    $u, v\in \RR^n$ with
    \begin{align*}
      u^T A u &>0\quad\text{and}\\
      v^T A v &<0
    \end{align*}
  \item
    A saddle point $x_0$ for $F$ is defined by the existence of two vectors $u, v\in \RR^n$, such that
    \begin{align*}
      &t=0\quad \text{is a local minimum for the function}\quad f(t) = F(x_0 + t u)\\
      &t=0\quad \text{is a local maximum for the function}\quad  g(t) = F(x_0 + t v)
    \end{align*}
    as illustrated in the graphics below.

    \includegraphics{saddle.png}
    \end{enumerate}
\end{remark}

\begin{example}\label{examplehesscrit}
Consider, with our new technology in Theorem \ref{thmcritminmax}, Exercise \ref{exsaddlept} once again.
Here we analyzed the point $v_0 = (0, 0)$ for the function
$$
f(x, y) = x^3 + x y + y^3
$$
and showed (by a trick) that $v_0$ is neither a local maximum nor a local minimum for $f$. The Hessian
matrix for $f(x, y)$ at $v_0$ is
$$
H = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}.
$$

Now
Theorem \ref{thmcritminmax}\ref{thmcritminsaddle} shows that $v_0$ is a saddle point, since
$$
\begin{pmatrix} x & y \end{pmatrix} H \begin{pmatrix} x \\ y \end{pmatrix} = 2 x y
$$
and
\begin{align*}
u^T H u &> 0\qquad\text{for } u = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\\
v H v &< 0\qquad\text{for } v =  \begin{pmatrix} 1 \\ -1 \end{pmatrix}.
\end{align*}

\begin{sage}
x, y = var('x, y')
a = 1
plot3d(x^3 + a*x*y+ y^3, (x, -0.4, 0.4), (y, -0.4, 0.4),
adaptive=True, color=rainbow(60, 'rgbtuple'))
\end{sage}
\end{example}

\beginshex
Try plotting the graph for \footnote{different values of \texttt{a}}{\texttt{a=4} shows the saddle point clearly.} in the Sage window in
Example \ref{examplehesscrit}. What do you observe for the point $v_0$ with
respect to the function? Does \texttt{a} have to be a number? Could it be a symbolic
expression in the variables \texttt{x} and \texttt{y} like \texttt{a = -10*cos(x)*sin(y)}?
\endshex

\beginshex
Check the computation of the Hessian matrix $H$ in Example \ref{examplehesscrit} by showing
that the Hessian matrix for $f$  at the point $(x, y)$ is
$$
\begin{pmatrix}
6 x & 1\\
1 & 6 y
\end{pmatrix}.
$$
\endshex

\beginshex
What about $u$ and $v$ in Example \ref{examplehesscrit}? How do they relate to the hint
given in Exercise \ref{exsaddlept}?
\endshex

\beginshex
  Give an example of a function $F:\RR^2\rightarrow \RR$ having a local minimum at
  $x_0$, where $\nabla^2 F(x_0)$ is not positive definite.
\endshex

\beginshex

The following exercise is a \texttt{sci2u} exercise from the Calculus book.

\begin{enumerate}[(i)]
\item
  The point $\left(0, \frac{\sqrt{3}}{3}\right)$ is a critical point for
  $$
  f(x, y) = x^3 + y^3 - y.
  $$
  What does Theorem \ref{thmcritminmax} say about this point?
\item
  The point $\left(\frac{1}{3}, \frac{1}{3}\right)$ is a critical point for
  $$
  f(x, y) = -x^3 -x^2 + x - y^3 + 2 y^2 - y.
  $$
  What does Theorem \ref{thmcritminmax} say about this point?
\item
  The point $(0, 1)$ is a critical point for
  $$
  f(x, y) = x^3 - x^2 + y^3 - y^2 - y.
  $$
  What does Theorem \ref{thmcritminmax} say about this point?
\end{enumerate}
\endshex

\beginshex
Consider the function
$$
f(x, y) = x^4 y^2 + x^2 y^4 - 3 x^2 y^2.
$$
Compute its critical points and decide on their types according to Theorem \ref{thmcritminmax}.
Try to convince yourself that
$$
f(x, y) \geq -1
$$
for every $x, y\in \RR$.
\endshex

\beginshex
Give an example of a function $f:\RR\rightarrow \RR$ that has a local maximum, but where
there exists $x\in \RR$ with $f(x) > M$ for any given (large) number $M$.
\endshex

\section{Convex functions of several variables}

Below is the generalization of Theorem \ref{fp} to several
variables. You have already seen this in an exercise in the
previous chapter, right?

\begin{theorem}[emph]\label{thmdiffsubdif}
  Let $f: U\rightarrow \RR$ be a differentiable function, where
  $U\subseteq \RR^n$ is an open convex subset. Then $f$ is convex if
  and only if
  \begin{equation}
    f(x) \geq f(x_0) + \nabla f(x_0) (x-x_0)\label{subgr}
  \end{equation}
  for every $x, x_0\in U$.
\end{theorem}
\begin{hideinbutton}{Proof}
  Suppose that \eqref{subgr} holds and let $x_t = (1-t)x_0 + t x$ with
  $0 \leq t \leq 1$, where $x_0, x\in U$.  To prove that $f$ is convex
  we must verify the inequality
  \begin{equation}
    f(x_t) \leq (1-t) f(x_0) + t f(x).\label{subdineq}
  \end{equation}
  Let $\xi = \nabla f(x_t)$. Then
  \begin{align*}
    f(x) &\geq f(x_t) + \xi (1-t) (x-x_0)\\
    f(x_0) &\geq f(x_t) - \xi t (x-x_0)
  \end{align*}
  by \eqref{subgr}.  If you multiply the first inequality by $t$, the
  second by $1-t$ and then add the two, you get \eqref{subdineq}.

  Suppose on the other hand that $f$ is a convex function. Let $x_0,
  x\in U$. Since $U$ is an open subset, it follows that $(1-t)x_0 + t
  x\in U$ for $t\in I=(-\delta, 1 + \delta)$, where $\delta>0$ is
  sufficiently small. Now define the function $g:I\rightarrow \RR$ by
  \begin{equation*}
    g(t) = f((1-t) x_0 + t x) = f(x_0 + t (x-x_0)).
  \end{equation*}
  Being the composition of two differentiable functions, $g$ is
  differentiable.  Suppose that $0\leq \alpha \leq 1$ and $t_1, t_2\in
  I$. Then
  \begin{align*}
   g((1- \alpha) t_1 + \alpha t_2) &= f(x_0 + ((1-\alpha)
    t_1 + \alpha t_2)(x-x_0))
    \\
    &=f((1-\alpha)(x_0 + t_1(x-x_0)) + \alpha (x_0+t_2(x-x_0)))
    \\
    &\leq(1-\alpha) f(x_0 + t_1 (x-x_0)) + \alpha f(x_0 + t_2(x-x_0))
    \\
    &=(1-\alpha) g(t_1) + \alpha g(t_2)
  \end{align*}
  showing that $g$ is a convex function.  By Theorem \ref{fp},
  \begin{equation*}
    g(1) \geq g(0) + g'(0),
  \end{equation*}
  which translates into
  \begin{equation*}
    f(x) \geq f(x_0) + \nabla f(x_0) (x-x_0)
  \end{equation*}
  by using the chain rule in computing $g'(0)$.
\end{hideinbutton}

\beginshex
  Prove that a bounded convex differentiable function $f:\RR^n\rightarrow \RR$ is
  constant.
\endshex

The following is the generalization of  Corollary \ref{corfdp}.

\newcommand{\Hess}[1]{\nabla^2 #1}

\begin{theorem}[emph]\label{thmhessconv}
  Let $f: U\rightarrow \RR$ be a differentiable function with continuous
  second order partial derivatives, where $U\subseteq \RR^n$ is a
  convex open subset. Then $f$ is convex if and only if the Hessian
  $\Hess{f}(x)$ is positive semidefinite for every $x\in U$. If
  $\Hess{f}(x)$ is positive definite for every $x\in U$, then $f$ is
  strictly convex.
\end{theorem}
\begin{hideinbutton}{Proof}
  We have done all the work for a convenient reduction to the one
  variable case. Suppose that $f$ is convex. Then the same reasoning
  as in the proof of Theorem \ref{thmdiffsubdif} shows that
  \begin{equation*}
    g(t) = f(x + t v)
  \end{equation*}
  is a convex function for every $x\in U$ and every $v\in \RR^n$ from
  an open interval $(-\delta, \delta)$ to $\RR$ for suitable
  $\delta>0$. Therefore $g''(0) = v^t \Hess{f}(x) v \geq 0$ by
  Theorem \ref{fdp}. This proves that the matrix $\Hess{f}(x)$ is
  positive semidefinite for every $x\in U$.  Suppose on the other hand
  that $\Hess{f}(x)$ is positive semidefinite for every $x\in U$.
  Then Theorem \ref{fdp} shows that $g(t) = f(x + t(y-x))$ is a convex
  function from $(-\delta, 1+\delta)$ to $\RR$ for $\delta>0$ small
  and $x, y\in U$, since
  \begin{equation*}
    g''(\alpha) = (y-x)^t \Hess{f}(x + \alpha(y-x)) (y-x) \geq 0
  \end{equation*}
  for $0\leq \alpha \leq 1$. Therefore $f$ is a convex function, since
  \begin{align*}
    f((1-t) x + t y) = &g((1-t)\cdot 0 + t\cdot 1) \\
    &\leq(1-t) g(0) + t g(1) = (1-t) f(x) + t f(y).
  \end{align*}
  The same argument (using the last part of Theorem \ref{fdp} on
  strict convexity), shows that $g$ is strictly convex if
  $\Hess{f}(x)$ is positive definite. It follows that $f$ is strictly
  convex if $\Hess{f}(x)$ is positive definite for every $x\in U$.
\end{hideinbutton}

\beginshex
  Prove that
  \begin{equation*}
    f(x, y) = x^2 + y^2
  \end{equation*}
  is a strictly convex function from $\RR^2$ to $\RR$. Also, prove that
  $$
  \{(x, y)\in \RR^2 \mid x^2 + y^2 \leq 1\}
  $$
  is a convex subset of $\RR^2$.
  \endshex

\beginshex
  Is $f(x, y) = \cos(x) + \sin(y)$ strictly convex on some non-empty
  open convex subset of the plane?
\endshex

\beginshex
 Show that $f:\RR^2 \rightarrow \RR$ given by
  \begin{equation*}
    f(x, y) = \log(e^x + e^y)
  \end{equation*}
  is a convex function. Is $f$ strictly convex?
\endshex

\beginshex
Let $f:\RR^2\rightarrow \RR$ be given by
  \begin{equation*}
    f(x, y) = a x^2 + b y^2 + c x y,
  \end{equation*}
  where $a, b, c\in \RR$.

  \begin{enumerate}[(i)]
  \item Show that $f$ is a strictly convex function if and only if $a > 0$
    and $4 a b - c^2 > 0$.
  \item Suppose now that $a > 0$ and $4 a b - c^2>0$.  Show that $g(x,
    y) = f(x, y) + x + y$ has a unique global minimum and give a
    formula for this minimum in terms of $a, b$ and $c$.
\end{enumerate}
\endshex

\section{How to decide the definiteness of a matrix}

In this section we will outline a straightforward method for
deciding if a matrix is positive definite, positive semidefinite,
negative definite or indefinite.

Before proceeding it is a must that you do the following exercise.

\beginshex
Show that a diagonal matrix
$$
\begin{pmatrix}
  \lambda_1 & 0 & \dots & 0\\
  0 &\lambda_2 & \dots &0\\
  \vdots & \vdots & \ddots & \vdots\\
  0 & 0 &\dots & \lambda_n
\end{pmatrix}
$$
is positive definite if and only if $\lambda_1 > 0, \dots, \lambda_n > 0$
and positive semidefinite
if and only if $\lambda_1 \geq  0, \dots, \lambda_n \geq 0$.

Also, show that if $A$ is a symmetric matrix, then $A$ is
positive definite if and only if
$$
B^t A B
$$
is positive definite, where $B$ is an invertible matrix.
\endshex

The crucial ingredient is the following result.

\begin{theorem}[emph]\label{thmsymred}
  Let $A$ be a real symmetric $n\times n$ matrix. Then there exists an
  invertible matrix $B$, such that $B^T A B$ is a diagonal matrix.
\end{theorem}

The proof contains an algorithm for building $B$ by different steps.
We will supply examples afterwards illustrating these. Incidentally,
how does this help you in deciding for example that a given matrix
is positive definite? If you cannot answer this question, please
do the exercise above.

\begin{hideinbutton}{Proof}
   Suppose that $A=(a_{ij})$. If $A$ has a non-zero entry in the upper
  left hand corner i.e., $a_{11}\neq 0$, then
  \begin{equation*}
    B_1^T A B_1 =
    \begin{pmatrix}
      a_{11} & 0 & \cdots & 0\\
      0 & c_{11} & \cdots & c_{1, n-1}\\
      \vdots & \vdots & \ddots & \vdots \\
      0 & c_{n-1, 1} & \cdots & c_{n-1, n-1},
    \end{pmatrix}
  \end{equation*}
  where $C = (c_{ij})$ is a real symmetric matrix and $B_1$ is the
  invertible $n\times n$ matrix
  \begin{equation*}
    % \def\frac#1#2{{#1\strut \over #2}\,}
    \begin{pmatrix}
      1 & -\frac{ a_{12}}{a_{11}} & \cdots & -\frac{
        a_{1n}}{a_{11}}\\
      0 & 1 & \cdots & 0\\
      \vdots & \vdots & \ddots & \vdots\\
      0 & 0 & \cdots & 1
    \end{pmatrix}
    .
  \end{equation*}
  By induction on $n$ we may find an invertible matrix $(n-1)\times
  (n-1)$ matrix $B_2$ such that
  \begin{equation*}
    B_2^t C B_2 =
    \begin{pmatrix}
      a_1 & 0 &\cdots & 0\\
      0 & a_2 & \cdots & 0\\
      \vdots & \vdots & \ddots &\vdots\\
      0 & 0 & \cdots & a_{n-1}
    \end{pmatrix}
    .
  \end{equation*}

  Putting
  \begin{equation*}
B = B_1
    \begin{pmatrix}
      1 & 0\\
      0 & B_2
    \end{pmatrix},
    \end{equation*}
    it follows that
\begin{equation*}
    B^t A B =
    \begin{pmatrix}
      a_{11} & 0 &\cdots & 0\\
      0 & a_1 & \cdots & 0\\
      \vdots & \vdots & \ddots &\vdots\\
      0 & 0 & \cdots & a_{n-1}
    \end{pmatrix}
    .
  \end{equation*}

We now treat the case of a zero entry in the upper left hand corner
  i.e., $a_{11}=\nobreak 0$.  Suppose first that $a_{jj} \neq 0$ for
  some $j > 1$. Let $P$ denote the identity matrix with the first and
  $j$-th rows interchanged.  The operation $A\mapsto A P$ amounts to
  interchanging the first and $j$-th columns in $A$.  Similarly
  $A\mapsto P^t A$ is interchanging that first and $j$-th rows in $A$.
  The matrix $P$ is invertible and $P^t A P$ is a symmetric matrix
  with $(P^t A P)_{11} = a_{jj}\neq 0$ and we have reduced to the case
  of a non-zero entry in the upper left hand corner.

  If $a_{ii} = 0$ for every $i = 1, \dots, n$ we may assume that $a_{1
    j}\neq 0$ for some $j>1$. Let $B$ denote the identity matrix where
  the entry in the first column and $j$-th row is $1$. The operation
  $A\mapsto A B$ amounts to adding the $j$-th column to the first
  column in $A$. Similarly $A\mapsto B^t A$ is adding the $j$-th row
  to the first row in $A$. All in all we get $(B^t A B)_{11} = 2 a_{1
    j} \neq 0$, where we have used that $a_{ii} = 0$ for $i=1, \dots,
  n$. Again we have reduced to the case of a non-zero entry in the
  upper left hand corner.
\end{hideinbutton}

\begin{example}\label{eksempel1}
  Consider the $3\times 3$ real symmetric matrix.
  \begin{equation*}
    A = (a_{ij}) =
    \begin{pmatrix}
      1 & 5 & 2\\
      5 & 3 & 0\\
      2 & 0 & 5
    \end{pmatrix}
    .
  \end{equation*}
  Here $a_{11} = 1 \neq 0$. Therefore the fundamental step in the
  proof of Theorem \ref{thmsymred} applies and
  \begin{equation*}
    \begin{pmatrix}
      1 & 0 & 0 \\
      -5 & 1 & 0\\
      -2 & 0 & 1
    \end{pmatrix}
    A
    \begin{pmatrix}
      1 & -5 & -2\\
      0 & 1 & 0\\
      0 & 0 & 1
    \end{pmatrix}
    =
    \begin{pmatrix}
      1 & 0 & 0\\
      0 & -22 & -10\\
      0 & -10 & 1
    \end{pmatrix}
  \end{equation*}
  and again
  \begin{equation*}
    \begin{pmatrix}
      1 & 0 & 0 \\
      0 & 1 & 0\\
      0 & -\frac{5}{11} & 1
    \end{pmatrix}
    \begin{pmatrix}
      1 & 0 & 0\\
      0 & -22 & -10\\
      0 & -10 & 1
    \end{pmatrix}
    \begin{pmatrix}
      1 & 0 & 0\\
      0 & 1 & -\frac{5}{11}\\
      0 & 0 & 1
    \end{pmatrix}
    =
    \begin{pmatrix}
      1 & 0 & 0\\
      0 & -22 & 0\\
      0 & 0 & \frac{61}{11}
    \end{pmatrix}
    .
  \end{equation*}

Summing up we get
  \begin{equation*}
    B =
    \begin{pmatrix}
      1 & -5 & -2\\
      0 & 1 & 0\\
      0 & 0 & 1
    \end{pmatrix}
    \begin{pmatrix}
      1 & 0 & 0\\
      0 & 1 & -\frac{5}{11}\\
      0 & 0 & 1
    \end{pmatrix}
    =
    \begin{pmatrix}
      1 & -5 & \frac{3}{11}\\
      0 & 1 & -\frac{5}{11}\\
      0 & 0 & 1
    \end{pmatrix}
    .
  \end{equation*}
  You are invited to check that
  \begin{equation*}
    B^t A B =
    \begin{pmatrix}
      1 & 0 & 0\\
      0 & -22 & 0\\
      0 & 0 & \frac{61}{11}
    \end{pmatrix}
    .
  \end{equation*}
\end{example}

\begin{example}\label{eksempel2}
  Let
  \begin{equation*}
    A =
    \begin{pmatrix}
      0 & 0 & 1 & 1\\
      0 & 0 & 2 & 3\\
      1 & 2 & 1 & 4\\
      1 & 3 & 4 & 0
    \end{pmatrix}.
  \end{equation*}
  Here $a_{11} = a_{22} = 0$, but the diagonal element $a_{33}\neq
  0$. So we are in the second step of the proof of
  Theorem \ref{thmsymred}.  Using the matrix
  \begin{equation*}
    P =
    \begin{pmatrix}
      0 & 0 & 1 & 0\\
      0 & 1 & 0 & 0\\
      1 & 0 & 0 & 0\\
      0 & 0 & 0 & 1
    \end{pmatrix}
  \end{equation*}
  we get
\begin{equation*}
  P^t A P =
    \begin{pmatrix}
      1 & 2 & 1 & 4\\
      2 & 0 & 0 & 3\\
      1 & 0 & 0 & 1\\
      4 & 3 & 1 & 0
    \end{pmatrix}
    .
  \end{equation*}

  As argued in the proof, this corresponds to interchanging the first
  and third columns and then interchanging the first and third
  rows. In total you move the non-zero $a_{33}$ to the upper left
  corner in the matrix.
\end{example}

\begin{example}\label{eksempel3}
  Consider the symmetric matrix
  \begin{equation*}
    A =
    \begin{pmatrix}
      0 & 1 & 1 & 1\\
      1 & 0 & 1 & 1\\
      1 & 1 & 0 & 1\\
      1 & 1 & 1 & 0
    \end{pmatrix}
    .
  \end{equation*}
  We have zero entries in the diagonal. As in the third step in the
  proof of Theorem \ref{thmsymred} we must find an invertible matrix
  $B_1$, such that the upper left corner in $B_1^t A B_1$ is
  non-zero. In the proof it is used that every diagonal element is
  zero: if we locate a non-zero element in the $j$-th column in the
  first row, we can add the $j$-th column to the first column and then
  the $j$-th row to the first row obtaining a non-zero element in the
  upper left corner. For $A$ above we choose $j=2$ and the matrix
  $B_1$ becomes
  \begin{equation*}
    B_1 =
    \begin{pmatrix}
      1 & 0 & 0 & 0\\
      1 & 1 & 0 & 0\\
      0 & 0 & 1 & 0\\
      0 & 0 & 0 & 1
    \end{pmatrix}
  \end{equation*}
  so that
\begin{equation*}
  B_1^t A B_1 =
    \begin{pmatrix}
      2 & 1 & 2 & 2\\
      1 & 0 & 1 & 1\\
      2 & 1 & 0 & 1\\
      2 & 1 & 1 & 0
    \end{pmatrix}
    .
  \end{equation*}

\end{example}

\beginshex
Let $A$ be any matrix. Show that
$$
A^T A
$$
is positive semidefinite.
\endshex

\beginshex
 Find inequalities defining the set
  \begin{equation*}
    \left\{(a, b)\in \RR^2 \middle\vert
      \begin{pmatrix}
        2 & 1 & a\\ 1 & 1 & 1 \\ a & 1 & b
      \end{pmatrix}
      \,\,\text{is positive definite}
    \right\}.
  \end{equation*}
  Same question with positive semidefinite. Sketch and compare the two
  subsets of the plane $\{(a, b) \mid a, b\in \RR\}$.
\endshex

\beginshex
Let $f:\RR^3\rightarrow \RR$ denote the function given by
$$
f(x, y, z) = x^2 + y^2 + z^2 +  a x y +  x z +  y z,
$$
where $a\in \RR$. Let $H$ denote the Hessian of
$f$ in a point $(x, y, z)\in \RR^3$.

\begin{enumerate}[(i)]
\item
  Compute $H$.
\item
  Show that $f(v) = v^T A v$ for $v=(x, y, z)\in \RR^3$ and $A = \frac{1}{2} H$.
\item
  Compute a non-zero vector $v\in \RR^3$, such that $H v = 0$ in
  the case, where $a=2$. Is $H$ invertible in this case?
\item
  Show that $f$ is strictly convex if $-1 < a < 2$.
\item
  Is $f$ strictly convex if $a=2$?

  \begin{hideinbutton}{Hint}
    Consider the line segment between $0$ and a suitable vector
    $u\neq 0$, where $f(u) = 0$.

  \end{hideinbutton}
\end{enumerate}
\endshex

\beginshex
Why is the subset given by the inequalities
  \begin{align*}
    x &\geq 0\\
    y &\geq 0\\
    x y - z^2 &\geq 0
  \end{align*}
  a convex subset of $\RR^3$?
\endshex

\end{document}


\documentclass[letterpaper]{article}

\usepackage{fullpage}
\usepackage{nopageno}
\usepackage{amsmath}
\usepackage{amssymb}
\allowdisplaybreaks

\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}

\begin{document}
\title{Notes}
\date{October 1, 2014}
\maketitle
if $G$ is a group, and $A\subseteq G$ and $B\subseteq G$ then $AB=\{ab|a\in A, b\in B\}\subseteq G$.

\section*{proposition}

let $G$ be a group, then $H,K$ subgroups of $G$. Assume that $h^{-1}kh\in K$ for all $h\in H$, $k\in K$ then $HK$ is a subgroup of $G$ that contain s both $H$ and $K$, in fact, $HK$ is the smallest subgroup of $G$ that contains both $H$ and $K$. Assumption only important if we are not dealing with abelian groups.

\subsubsection*{proof}
$a,b\in HK$. Write $a=h_1k_1,b=h_2k_2$ with $h_i\in H,k_i\in K$ then $a\cdot b=h_1k_1h_2k_2=h_1h_2(h_2^{-1}k_1h_2)k_2\in HK$

$a=hk, a^{-1}=(hk)^{-1}=k^{-1}h^{-1}=h^{-1}(hk^{-1}h^{-1})\in HK$

\subsection*{examples}
$S_3, H=\{(1),(12)\}, K=\{(1),(123),(132)\}, (12)(123)=(23)\in HK, (12)(132)=(13)\in HK$ so $HK=G$ and is therefore contained by G

$(\mathbb{Z},+)$, $H=a\mathbb{Z}, k=b\mathbb{Z}$, let $d=(a,b)$

claim: $a\mathbb{Z}+b\mathbb{Z}=d\mathbb{Z}$. clearly $a\mathbb{Z}\subseteq d\mathbb{Z}$, $b\mathbb{Z}\subseteq d\mathbb{Z}$.

$a\mathbb{Z}+b\mathbb{Z}$ is the smallest subgroup that contains both $a\mathbb{Z}$ and $b\mathbb{Z}$. so $a\mathbb{Z}+b\mathbb{Z}\subseteq d\mathbb{Z}$.

$d=\gcd(a,b)$ so we can write $d=ma+nb$. let $\alpha\in d\mathbb{Z}$ and write $\alpha=dt, t\in \mathbb{Z}$ then $\alpha=dt=mat+nbt\in a\mathbb{Z}+b\mathbb{Z}$. so $d\mathbb{Z}\subseteq a\mathbb{Z}+b\mathbb{Z}$
\section*{thm subgroup gen by a subset}
$G$ is a group, if $a\in G$ $<a>=\{a^i|i\in \mathbb{Z}\}$ is the smallest subgroupthat contains $a$.

\subsubsection*{proof}
let $S\subseteq G$, let $<S>=\{\underbrace{a_1a_2\dots a_k}_{\text{word}}|a_i\in S\text{ or }{a_i}^{-1}\in S, k\in \mathbb{N}\}$ then $<S>$ is a subgroup, $<S>=\cap \forall H$ where $S\subseteq H\subseteq G$, and $H$ is a subgroup of $G$, $<S>$ is the smallest subgroup of $G$that contains $S$.

so it is closed under multiplication, identity is in it, and the inverse of all words are in it.

show containment both ways, one is clear because we have words of length 1 that span $S$ and so $S$ is one of the elements of our $H$ intersection.

\section*{example}
$a,b\in G, S=\{a,b\}\subseteq G, <S>=\{a_1a_2\dots a_k|a_i\in\{a,a^{-1},b,b^{-1}\}\}$

if $ab=ba$ then $<S>=\{a^{i}b^{j}|i\in \mathbb{Z}, j\in \mathbb{Z}\}$

\section*{maps}
studied groups, subgroups. now we are going to talk about maps

if we have groups $G_1,G_2$ and $\varphi:G_1\to G_2$ is a group homomorphism provided $x\to\varphi(x), y\to\varphi$ means that $\varphi(x*y)=\varphi(x)*\varphi(y)$ for all $x,y\in G_1$.
\section*{examples}
identity: $x\to x$

$(\mathbb{R},+)=G_1, (\mathbb{R}^+,\cdot)=G_2$. $\varphi(x)=e^x$. ie $\varphi(x+y)=e^{x+y}=e^xe^y=\varphi(x)\varphi(y)$.

\section*{notation}
let $\varphi:G_1\to G_2$ be a group homomorphism, then $\ker\varphi=\{x\in G_1|\varphi(x)=e\}$

homomorphism always takes the identity in $G_1$ to $G_2$.

$\varphi(e_1)\varphi(e_1)^{-1}=\varphi(e_1e_1)\varphi(e_1)^{-1}=\varphi(e_1)\varphi(e_1)\varphi(e_1)^{-1}=e_2=\varphi(e_1)$

prove that $\ker \varphi$ is a subgroup

now we say that $\varphi$ is an isomorphism if $\varphi$ is a group homomorphism and $\varphi$ is bijective.

both of the previous examples are isomorphisms.

so from an algebraic point of view, there is no difference between addition on the reals and multiplication on the positive reals.

\section*{proposition}
let $\varphi$ be an isomorphism. the following are true
\begin{enumerate}
\item
$\varphi^{-1}$ which is the map from $G_2$ to $G_1$ is also an isomorphism.
\item
if $G_1$ is abelian, then $G_2$ is abelian.
\item
if $G_1$ is cyclic then so is $G_2$
\item
if $a\in G_1$ then $\text{ord}(a)=\text{ord}(\varphi(a))$
\end{enumerate}

\begin{enumerate}
\item
need to prove $\varphi^{-1}(\alpha\beta)=\varphi^{-1}(\alpha)\varphi^{-1}(\beta)$ for all $\beta\in G_2$, but $\varphi$ is injective so it is enough to prove that $\varphi(\varphi^{-1}(\alpha\beta))=\varphi(\varphi^{-1}(\alpha)\varphi^{-1}(\beta)=\varphi(\varphi^{-1}(\alpha))\varphi(\varphi^{-1}(\beta))=\alpha\beta$
\item
assume $G_1$ is abelian
\begin{align*}
  \alpha\beta=\varphi(\varphi^{-1}\left(\alpha\right)\varphi^{-1}(\beta))
\end{align*}
\item
hint: assume that $G_1=<a>$ for some $a\in G_1$ and then prove that $G_2=<\varphi(a)$
\item
no hint
\end{enumerate}
\section*{example}
\begin{align*}
  \mathbb{Z}_4\not\equiv\mathbb{Z}_2\times\mathbb{Z}_2
\end{align*}
by contradiction, assume that there exists an isomorphism $\varphi$ from z4 to z2+z2. $[1]\in \mathbb{Z}_4$ and $\text{ord}[1]=4$ so then $\text{ord}\varphi([1])=4$. But all elements of $\mathbb{Z}_2\times\mathbb{Z}_2$ has no elements of order 4, so there is no isomorphisms. however, if $\gcd(m,n)=1$ then $\mathbb{Z}_{mn}\equiv\mathbb{Z}_m\times\mathbb{Z}_n$
\end{document}


\chapter{Supplemental Algorithms}\label{chap:appendix-algo}

Find here concrete implementation details on the EFTs described
in Theorem~\ref{thm:eft}. They do not use branches, nor access to the
mantissa that can be time-consuming.

\begin{breakablealgorithm}
  \caption{\textit{EFT of the sum of two floating point numbers.}}

  \begin{algorithmic}
    \Function{\(\left[S, \sigma\right] = \mathtt{TwoSum}\)}{$a, b$}
      \State \(S = a \oplus b\)
      \State \(z = S \ominus a\)
      \State \(\sigma = (a \ominus (S \ominus z)) \oplus (b \ominus z)\)
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}

\noindent In order to avoid branching to check which among
\(\left|a\right|, \left|b\right|\) is largest, \texttt{TwoSum} uses 6 flops
rather than 3.

\begin{breakablealgorithm}
  \caption{\textit{Splitting of a floating point number into two parts.}}

  \begin{algorithmic}
    \Function{\(\left[h, \ell\right] = \mathtt{Split}\)}{$a$}
      \State \(z = a \otimes (2^r + 1)\)
      \State \(h = z \ominus (z \ominus a)\)
      \State \(\ell = a \ominus h\)
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}

\noindent For IEEE-754 double precision floating point number, \(r = 27\)
so \(2^r + 1\) will be known before \texttt{Split} is called. In all,
\texttt{Split} uses 4 flops.

\begin{breakablealgorithm}
  \caption{\textit{EFT of the product of two floating point numbers.}}

  \begin{algorithmic}
    \Function{\(\left[P, \pi\right] = \mathtt{TwoProd}\)}{$a, b$}
      \State \(P = a \otimes b\)
      \State \(\left[a_h, a_{\ell}\right] = \mathtt{Split}(a)\)
      \State \(\left[b_h, b_{\ell}\right] = \mathtt{Split}(b)\)
      \State \(\pi = a_{\ell} \otimes b_{\ell} \ominus (((P \ominus
          a_h \otimes b_h)
          \ominus a_{\ell} \otimes b_h) \ominus a_h \otimes b_{\ell})\)
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}

\noindent This implementation of \texttt{TwoProd} requires 17 flops.
For processors that provide a fused-multipy-add operator (\texttt{FMA}),
\texttt{TwoProd} can be rewritten to use only 2 flops:

\begin{breakablealgorithm}
  \caption{\textit{EFT of the sum of two floating point numbers with a FMA.}}
  \label{alg:two-prod-fma}

  \begin{algorithmic}
    \Function{\(\left[P, \pi\right] = \mathtt{TwoProdFMA}\)}{$a, b$}
      \State \(P = a \otimes b\)
      \State \(\pi = \mathtt{FMA}(a, b, -P)\)
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}

\noindent The following algorithms from \cite{Ogita2005} can be used as a
compensated method for computing a sum of numbers. The first is a vector
transformation that is used as a helper:

\begin{breakablealgorithm}
  \caption{\textit{Error-free vector transformation for summation.}}
  \label{alg:vec-sum}

  \begin{algorithmic}
    \Function{\(\mathtt{VecSum}\)}{$p$}
      \State \(n = \texttt{length}(p)\)
      \For{\(j = 2, \ldots, n\)}
        \State \(\left[p_j, p_{j - 1}\right] = \mathtt{TwoSum}\left(
            p_j, p_{j - 1}\right)\)
      \EndFor
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}

\noindent The second (\texttt{SumK}) computes a sum with results that are as
accurate as if computed in \(K\) times the working precision. It requires
\((6K - 5)(n - 1)\) floating point operations.

\begin{breakablealgorithm}
  \caption{\textit{Summation as in K-fold precision
      by \((K - 1)\)-fold error-free vector transformation.}}
  \label{alg:sum-k}

  \begin{algorithmic}
    \Function{\(\mathtt{result} = \mathtt{SumK}\)}{$p, K$}
      \For{\(j = 1, \ldots, K - 1\)}
        \State \(p = \mathtt{VecSum}(p)\)
      \EndFor
      \State \(\mathtt{result} = p_1 \oplus p_2 \oplus \cdots \oplus p_n\)
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}

\noindent Since the final error \(\cdb{K - 1}\) will not track the errors
during computation, we have a non-EFT version of
Algorithm~\ref{alg:local-error-eft}:

\begin{breakablealgorithm}
  \caption{\textit{Compute the local error (non-EFT).}}
  \label{alg:local-error}

  \begin{algorithmic}
    \Function{\(\widehat{\ell} =
        \mathtt{LocalError}\)}{$e, \rho, \delta b$}
      \State \(L = \texttt{length}(e)\)
      \\
      \State \(\widehat{\ell} = e_1 \oplus e_2\)
      \For{\(j = 3, \ldots, L\)}
        \State \(\widehat{\ell} = \widehat{\ell} \oplus e_j\)
      \EndFor
      \\
      \State \(\widehat{\ell} = \widehat{\ell} \oplus \left(
          \rho \otimes \delta b\right)\)
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}

In order to discuss varied implementations of Newton's method in
Chapter~\ref{chap:compensated-newton}, we define a generic algorithm
that takes an ``update function'' (\(\mathtt{update\_fn}\)), i.e.
a callable that will produce the next Newton update \(p(s) / p'(s)\)
computed in a problem-specific way.

\begin{breakablealgorithm}
  \caption{\textit{Generic Newton's method for scalar functions.}}
  \label{alg:generic-newton}

  \begin{algorithmic}
    \Function{\(x_{\ast} = \mathtt{NewtonGeneric}\)}
             {$\mathtt{update\_fn}, x_0, \mathtt{tol}, \mathtt{max\_iter}$}
      \State \(x = x_0\)
      \\
      \For{\(j = 1, \ldots, \mathtt{max\_iter}\)}
        \State \(\mathtt{update} = \mathtt{update\_fn}(x)\)
        \State \(x = x \ominus \mathtt{update}\)
        \If{\(\left|\mathtt{update}\right| < \mathtt{tol}\)}
          \State \textbf{break}
        \EndIf
      \EndFor
      \\
      \State \(x_{\ast} = x\)
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}

In \cite[Algorithm~3]{Jiang2010}, the \texttt{CompDeCasteljau} (
Algorithm~\ref{alg:comp-de-casteljau}) is modified for the computation
of the derivative \(p'(s)\). Since \(p'(s)\) has coefficients
\(n \Delta b_j\) where \(c_j = \Delta b_j = b_{j + 1} - b_j\),
we can begin the computation with nonzero
\(\widehat{\partial c}_j\) (as opposed to
\(\cdb{1}_j^{(n)} = 0\) in \texttt{CompDeCasteljau}).

\begin{breakablealgorithm}
  \caption{\textit{Compensated de Casteljau
      algorithm for polynomial first derivative evaluation.}}
  \label{alg:comp-de-casteljau-derivative}

  \begin{algorithmic}
    \Function{\(\mathtt{result} = \mathtt{CompDeCasteljauDer}\)}{$b, s$}
      \State \(n = \texttt{length}(b) - 1\)
      \State \(\left[\widehat{r}, \rho\right] = \mathtt{TwoSum}(1, -s)\)
      \\
      \For{\(j = 0, \ldots, n - 1\)}
        \State \(\left[\widehat{c}_j^{(n - 1)},
          \widehat{\partial c}_j^{(n - 1)}\right] =
          \mathtt{TwoSum}(b_{j + 1}, -b_j)\)
      \EndFor
      \\
      \For{\(k = n - 2, \ldots, 0\)}
        \For{\(j = 0, \ldots, k\)}
          \State \(\left[P_1, \pi_1\right] = \mathtt{TwoProd}\left(
              \widehat{r}, \widehat{c}_j^{(k + 1)}\right)\)
          \State \(\left[P_2, \pi_2\right] = \mathtt{TwoProd}\left(
              s, \widehat{c}_{j + 1}^{(k + 1)}\right)\)
          \State \(\left[\widehat{c}_j^{(k)}, \sigma_3\right] =
              \mathtt{TwoSum}(P_1, P_2)\)
          \State \(\widehat{\ell}_{1, j}^{(k)} = \pi_1 \oplus \pi_2 \oplus
              \sigma_3 \oplus \left(\rho \otimes
              \widehat{c}_j^{(k + 1)}\right)\)
          \State \(\widehat{\partial c}_j^{(k)} =
              \widehat{\ell}_{1, j}^{(k)} \oplus
              \left(s \otimes \widehat{\partial c}_{j + 1}^{(k + 1)}
              \right) \oplus
              \left(\widehat{r} \otimes
              \widehat{\partial c}_j^{(k + 1)}\right)\)
        \EndFor
      \EndFor
      \\
      \State \(\mathtt{result} = n \otimes \left[\widehat{c}_0^{(0)} \oplus
          \widehat{\partial c}_0^{(0)}\right]\)
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}

For an applied usage of \texttt{CompDeCasteljau}
(Algorithm~\ref{alg:comp-de-casteljau}) in
Chapter~\ref{chap:compensated-newton}, both the non-compensated
value \(\widehat{b}\) and the compensation term \(\widehat{\partial b}\) are
needed. So we define a partial EFT. We say \emph{partial} because
the compensation term is rounded.

\begin{breakablealgorithm}
  \caption{\textit{EFT for de Casteljau algorithm for polynomial evaluation.}}
  \label{alg:eft-de-casteljau}

  \begin{algorithmic}
    \Function{\(\left[\widehat{b}, \widehat{\partial b}\right] =
      \mathtt{DeCasteljauEFT}\)}{$b, s$}
      \State \(n = \texttt{length}(b) - 1\)
      \State \(\left[\widehat{r}, \rho\right] = \mathtt{TwoSum}(1, -s)\)
      \\
      \For{\(j = 0, \ldots, n\)}
        \State \(\widehat{b}_j^{(n)} = b_j\)
        \State \(\cdb{1}_j^{(n)} = 0\)
      \EndFor
      \\
      \For{\(k = n - 1, \ldots, 0\)}
        \For{\(j = 0, \ldots, k\)}
          \State \(\left[P_1, \pi_1\right] = \mathtt{TwoProd}\left(
              \widehat{r}, \widehat{b}_j^{(k + 1)}\right)\)
          \State \(\left[P_2, \pi_2\right] = \mathtt{TwoProd}\left(
              s, \widehat{b}_{j + 1}^{(k + 1)}\right)\)
          \State \(\left[\widehat{b}_j^{(k)}, \sigma_3\right] =
              \mathtt{TwoSum}(P_1, P_2)\)
          \State \(\widehat{\ell}_{1, j}^{(k)} = \pi_1 \oplus \pi_2 \oplus
              \sigma_3 \oplus \left(\rho \otimes
              \widehat{b}_j^{(k + 1)}\right)\)
          \State \(\cdb{1}_j^{(k)} =
              \widehat{\ell}_{1, j}^{(k)} \oplus
              \left(s \otimes \cdb{1}_{j + 1}^{(k + 1)}
              \right) \oplus
              \left(\widehat{r} \otimes
              \cdb{1}_j^{(k + 1)}\right)\)
        \EndFor
      \EndFor
      \\
      \State \(\widehat{b} = \widehat{b}_0^{(0)}\)
      \State \(\widehat{\partial b} = \cdb{1}_0^{(0)}\)
    \EndFunction
  \end{algorithmic}
\end{breakablealgorithm}


%\chapter{An Introduction of Machine Learning}
\section{What is a learning algorithm?}

        A \textbf{machine learning algorithm} is an algorithm that is able to learn from data. But what does "learning" mean? Mitchell (1997) provides a definition "A computer program is said to learn from experience $E$ with respect to some class of tasks	$T$	and performance measure	$P$, if its performance at tasks in	$T$, as measured by $P$, improves with experience $E$." Learning is not the task, but the means of attaining the ability to perform the task. The result of the learning is a model that is used to perform tasks.

        The machine learning algorithm grew out of work in \textbf{Artificial Intelligence (AI)} and take advantage of new capability of computing resources. \textbf{Deep learning} is a subset of machine learning.

    \subsection{Types of learning algorithm}

         Machine learning algorithms can be broadly categorized as \textbf{unsupervised} or \textbf{supervised} or \textbf{Reinforcement learning algorithms
         }	by what kind of experience they are allowed to have during the learning process.
    \begin{itemize}
        \item Unsupervised learning algorithms learn from a dataset containing many features, then learn useful properties of the structure of this dataset. E.g. K-means clustering
        \item Supervised learning algorithms learn from a dataset containing features, but each example is also associated with a label or target. E.g. classification , regression method to predict housing price
        \item Reinforcement learning algorithms interact with an environment, so there is feedback loop between the learning system and its experience. But it's out of the scope of this note.
    \end{itemize}
    Here we offer a simple example to illustrate the difference between the supervised and unsupervised learning algorithm.

     Suppose we have a set of points in $\mathbb{R}^2: \{(x_1,y_1),(x_2,y_2),\dots,(x_n,y_n)\}$.
     \begin{itemize}
         \item Ther's no more information about the points except their coordinates. We want to split them into several subsets (which is 'clustering') based on their coordinates. That's an unsupervised algorithm.
         \item The points are painted red,green or blue. We want to learn from the data about the reason why the point is red,green or blue. If given a new point, we can label its color according the points before. That's an example of classification, which is one kind of supervised learning algorithm.
     \end{itemize}
     \begin{figure}[htbp]
         \centering
         \includegraphics[width = 1.0 \textwidth]{Cluster.png}
         \caption{An example of clustering}
     \end{figure}
    \subsection{Maximum Likelihood Estimation}
    As the process of machine learning is to estimate a set of parameters �� based on observations of examples X, \textbf{maximum likelihood} is often considered the preferred estimator to use for machine learning.

    \begin{itemize}
        \item Consider a set of $m$ examples $\mathbb X=\{\bm x^{(1)},...,\bm x^{(m)}\}$ drawn independently from the true but unknown data generating distribution $p_{data}(\bm x)$.
        \item Let $p_{model}(\bm x;\bm \theta)$ be a parametric family of probability distributions over the same space indexed by $\bm \theta$.
        \item The maximum likelihood estimator for $\bm \theta$ is then defined as
            \begin{equation*}
                \begin{split}
                    \bm \theta_{ML} &= \arg \max_{\bm \theta}p_{model}(\mathbb X;\bm \theta) \\
                    &=\arg \max_{\bm \theta}\prod_{\bm x \in \mathbb X} p_{model}(\bm x;\bm \theta)\\
                    &=\arg \max_{\bm \theta}\sum_{\bm x \in \mathbb X} \log p_{model}(\bm x;\bm \theta)\\
                \end{split}
            \end{equation*}
    \end{itemize}

    \subsection{Input data}
    Previously observed data in the experience can be split to three categories:
    \begin{itemize}
        \item Training data: used to train the model
        \item Validation data: used to adjust the model
        \item Test data: used to evaluate the generalization of the model
    \end{itemize}
    Training data and validation data are usually splitted from a large observed data set. The ratio between training data and validation data is commenly 4:1 or 9:1.
    \subsection{Capacity and Performance}

    The capacity of the learning algorithms can be under fitting, appropriate capacity and overfitting. Underfitting occurs when the model is not able to obtain a sufficiently low error value on the training set. Overfitting occurs when the gap between the training error and test error is too large.

    \begin{figure}[htbp]
        \centering
        \includegraphics[width = 1.0 \textwidth]{fitCap.png}
        \caption{We fit three models to this example training set. The training data was
        generated synthetically, by randomly sampling $x$ values and choosing $y$ deterministically
        by evaluating a quadratic function. (Left)A linear function fit to the data suffers from underfitting-it cannot capture the curvature that is present in the data. (Center) A quadratic function fit to the data generalizes well to unseen points. It does not suffer from a significant amount of overfitting or underfitting. (Right)A polynomial of degree 9 fit to the data suffers from overfitting. Here we used the Moore-Penrose pseudoinverse to solve the underdetermined normal equations. The solution passes through all of the training points exactly, but we have not been lucky enough for it to extract the correct structure. It now has a deep valley in between two training points that does not appear in the true underlying function. It also increases sharply on the left side of the data, while the true function decreases in this area.}
    \end{figure}
    The factors determining the performance of the machine learning algorithms are:
    \begin{itemize}
        \item Test error (final error of the generated model) The error is calculated by a cost function based on the test data set
        \item The gap between training and validation error
    \end{itemize}
    The central challenge in machine learning is that we must perform well on new, previously unseen inputs, not just those on which our model was trained. So Machine learning algorithms indirectly minimize the test error. Regularization is the modification of machine learning algorithms to reduce validation error but not the training error in order to avoid overfitting.
    \begin{figure}[htbp]
        \centering
        \includegraphics[width = 1.0 \textwidth]{ErrorwithCap.png}
        \caption{Typical relationship between capacity and error. Training and test error behave differently. At the left end of the graph, training error and generalization error are both high. This is the underfitting regime. As we increase capacity, training error decreases, but the gap between training and generalization error increases. Eventually, the size of this gap outweighs the decrease in training error, and we enter the overfitting regime, where capacity is too large, above the optimal capacity.}
    \end{figure}
    \section{Regularization and generalization}
    \subsection{Regularization}
    Regularization is a process of introducing additional information in order to prevent \textbf{overfitting}. Empirical learning of models (learning from a finite data set) is always an underdetermined problem, because in general we are trying to infer a function of any $x$ given only some examples $ x_{1},x_{2},...x_{n}$.

    A \textbf{regularization term} (or \textbf{regularizer}) $R(f)$ is added to a loss function:

    \begin{equation}
        \min _{f}\sum _{i=1}^{n}L(f({\hat {x}}_{i}),{\hat {y}}_{i})+\lambda R(f)
    \end{equation}

    where $L$ is an underlying loss function that describes the cost of predicting $ f(x)$ when the label is $y$ (for classification problem); and $\lambda$  is a parameter which controls the importance of the regularization term. $ R(f)$ is typically chosen to impose a penalty on the complexity of $f$. Concrete notions of complexity used include restrictions for smoothness and bounds on the vector space norm.

    \subsection{Generalization error}
    In supervised learning applications in machine learning and statistical learning theory, \textbf{generalization error} (also known as the \textbf{out-of-sample error}) is a measure of how accurately an algorithm is able to predict outcome values for previously unseen data. Because learning algorithms are evaluated on finite samples, the evaluation of a learning algorithm may be sensitive to sampling error. As a result, measurements of prediction error on the current data may not provide much information about predictive ability on new data. Generalization error can be minimized by \textbf{avoiding overfitting} in the learning algorithm.

    The generalization error, $I[f_{n}]$ of a particular function $f_{n}$ over all possible values of $x$ and $y$ is:

    \begin{equation}
        I[f_{n}]=\int _{X\times Y}V(f_{n}(x),y)\rho (x,y)dxdy
    \end{equation}
    Since $I[f_{n}]$ cannot be computed for an unknown probability distribution, the generalization error cannot be computed explicitly. Instead, the aim of many problems in statistical learning theory is to bound or characterize the generalization error in probability:
    \begin{equation}
        P_{G}=P(I[f_{n}]-I_{S}[f_{n}]\leq \epsilon )\geq 1-\delta _{n}
    \end{equation}
    while
    \begin{equation}
        I_{S}[f_{n}]={\frac {1}{n}}\sum _{i=1}^{n}V(f_{n}(x_{i}),y_{i})
    \end{equation}
    is the empirical error computed from the sample points.

    \section{Types of problems}
    \subsection{Classification problem}
    In machine learning and statistics, classification is the problem of identifying to which of a set of categories (sub-populations) a new observation belongs, on the basis of a training set of data containing observations (or instances) whose category membership is known. An example would be assigning a given email into "spam" or "non-spam" classes or assigning a diagnosis to a given patient as described by observed characteristics of the patient (gender, blood pressure, presence or absence of certain symptoms, etc.).
     \begin{figure}[htbp]
         \centering
         \includegraphics[width = 1.0 \textwidth]{Classification.png}
         \caption{An example of classification}
     \end{figure}
    \subsection{Regression problem}
    In statistical modeling, regression is a statistical process for estimating the relationships among variables. It includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables (or 'predictors'). More specifically, regression analysis helps one understand how the typical value of the dependent variable (or 'criterion variable') changes when any one of the independent variables is varied, while the other independent variables are held fixed.  In all cases, the estimation target is a function of the independent variables called the \textbf{regression function}.
    \begin{figure}[htbp]
        \centering
        \includegraphics[width = 1.0 \textwidth]{Regression.png}
        \caption{An example of multivariate regression}
    \end{figure}

\input{6DL/PageRank.tex}


\section{Introduction}
Database systems are extremely important in the real world, in all use cases involving management of large quantities of data: of course, they need to provide some guarantees such as scalability, reliability and concurrency, which can be a challenge.

There are scenarios which can either be optimized, or generate some terrible runtime, such as a simple join: its complexity can vary from $O(n)$ (hash join) to $O(n^2)$ with a nested loop. Obviously, the latter case is ruled out since it forbids any kind of scalability.

This can get even more complex when one of the data sources fits in memory, and the other does not: in this case, an additional index structure is employed, to perform lookups in external memory.

When neither fits, sorting is an option, although already a difficult problem; another approach is partitioning, breaking data into sub-problems until they fit in memory.

In general, there are many corner cases to consider, and sometimes it is necessary to make assumptions about the data, making code complexity very high.

Historically, DMBS are designed such that I/O operations are random and expensive, and data is much larger than main memory. Conservative designs are scalable, yet in modern hardware systems main memory size is increasing, reducing I/O costs.

This has consequences for database implementation, since the old architecture becomes suboptimal. Ideally, DBMS should be adapted in order to also maintain durability and good performance.

\subsection{Set-oriented processing}
Set-oriented processing is a way to avoid using nested loops (accessing every element of one list and then each element of the other) and subsequent squared runtimes, as this operation is not optimal.

It is helpful to perform operations for large batches of data, handling input in sets. Notation takes advantage of relational algebra operations, since these are already set-oriented; even in the case of duplicates (bags), those can still be mapped to unique values.

Algebra needs some extensions for real-world scenarios, such as map, group by or dependent join (predicates which must be evaluated after others).

In short, processing whole batches of tuples can help sorting, hashing, re-organizing the data and creating index structures, ideally achieving worst-case logarithmic runtime.

Database Systems on Modern CPU Architectures/lectures/introduction.tex


\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}

\title{MATH 542 Homework 4}
\author{Saket Choudhary\\skchoudh@usc.edu}

\begin{document}
\maketitle
\section*{1a. Problem 1}

\begin{align*}
E[(X-a)(X-a)']&= E[(X-a)(X'-a')]\\
&= E[XX'-Xa'-aX'+aa']\\
&= E[XX']-E[Xa']-E[aX']+E[aa']\\
&= E[XX']-E[X]a'-aE[X']+E[aa']\\
&= (Var[X]+E[X]E[X]')-E[X]a'-aE[X']+aa'\\
&= Var[X]+ E[X]E[X]'-E[X]a'-aE[X]'+aa'\ \text{ since } E[X']=E[X]'\\
&= Var[X] + (E[X]-a)(E[X]'-a')\\
&= Var[X] + (E[X]-a)(E[X]-a)'
\end{align*}

$$Var[X] = \sum = (\sigma_{ij})$$

$$||X-a||_{1\times 1}^2 = (X-a)'_{1 \times r}(X-a)_{1 \times r}$$

And hence(replace $X$ with $X'$ and $a$ with $a'$):

\begin{align*}
E[||X-a||^2] &= E[(X-a)'(X-a)]\\
&= E[X'X]-E[X']a-a'E[X]+a'a\\
&= \sum_i E[X_i^2]-E[X']a-a'E[X]+a'a\\
&= \sum_i (Var[X_i]+E[X_i]^2)-E[X']a-a'E[X]+a'a\\
&= \sum_i Var[X_i]+ E[X']E[X]-E[X']a-a'E[X]+a'a\ \text{since } \sum_i E[X_i]^2 = E[X'X]\\
&=\sum_i Var[X_i]+ E[X]'E[X]-E[X]'a-a'E[X]+a'a\ \text{since } \sum_i E[X_i]^2 = E[X'X]\\
&= \sum_i Var[X_i] + (E[X]-a)'(E[X]-a)\\
&= \sum_i \sigma_i + ||E[X]-a||^2
\end{align*}

\section*{1a. Problem 2}
Fact: $X-a-E[X-a]=X-E[X]$
\begin{align*}
Cov[X-a,Y-b] &= E[(X-a-E[X-a])(Y-b-E[Y-b])'])\\
&= E[(X-E[X])(Y-E[Y])']\\
&= Cov[X,Y]
\end{align*}

\section*{1a. Problem 3}
$Y_i = X_i-X_{i-1}$

$Cov[Y_i,Y_j]=0$ for $i \neq j$

Consider the vector $(Y_1, Y_2, Y_3, \dots, Y_n)' = (X_1,X_2-X_1,X_3-X_2, \dots, X_n-X_{n-1})'$

We make use of $Var(AX) = AVar(X)A'$.

To find $A$, consider the vectors $(Y_1, Y_2, Y_3, \dots, Y_n)' = (X_1,X_2-X_1,X_3-X_2, \dots, X_n-X_{n-1})'$

\begin{align*}
\begin{pmatrix}
Y_1\\
Y_2\\
Y_3\\
\vdots\\
Y_n
\end{pmatrix} &= \begin{pmatrix}
X_1\\
X_2-X_1\\
X_3-X_2\\
\vdots\\
X_n-X_{n-1}
\end{pmatrix}\\
&= \begin{pmatrix}1 & 0 & 0 & \dots & 0 & 0\\
-1 & 1 & 0 & \dots & 0 & 0\\
0 & -1 & 1 & \dots & 0 & 0\\
\vdots & \vdots & \vdots & \vdots & \vdots & \vdots\\
0 & 0 & 0 & \dots & -1 & 1
\end{pmatrix} \times \begin{pmatrix}
X_1\\
X_2\\
X_3\\
\vdots\\
X_n
\end{pmatrix}
\end{align*}

Hence $A=\begin{pmatrix}1 & 0 & 0 & \dots & 0 & 0\\
-1 & 1 & 0 & \dots & 0 & 0\\
0 & -1 & 1 & \dots & 0 & 0\\
\vdots \\
0 & 0 & 0 & \dots & -1 & 1
\end{pmatrix}$

Now using $Var(Y) = AVar(X)A'$ we get

$$Var(X) = A^{-1}Var(Y)A'^{-1}$$

$Var(Y) = I_{n\times n}$
and hence $Var(X) = A^{-1}A'^{-1} = BB^T$ where $B=A^{-1}$

\newpage
\section*{Problem 4}

$X_{i+1} = \rho X_i$

Consider:

\begin{align*}
\begin{pmatrix}
X_1\\
X_2\\
X_3\\
\vdots\\
X_n
\end{pmatrix} &= \begin{pmatrix}
X_1\\
\rho X_1\\
\rho X_2\\
\vdots\\
\rho X_{n-1}
\end{pmatrix}\\
&= \begin{pmatrix}
1\\
\rho\\
\rho^2\\
\vdots\\
\rho^{n-1}
\end{pmatrix}X_1
\end{align*}

Let $A=\begin{pmatrix}1 & \rho & \rho^2 & \dots & \rho^{n-1}\end{pmatrix}'$
and hence variance $Var[X] = AVar(X_1)A' = \sigma^2AA'$

$Var[X] = \begin{pmatrix}
1 & \rho & \rho^2 & \dots & \rho^{n-1}\\
\rho & \rho^2 & \rho^3 & \dots & \rho^n\\
\rho^2 & \rho^3 & \rho^4 & \dots & \rho^{n+1}\\
\vdots & \vdots & \vdots & & \vdots\\
\rho^n & \rho^{n+1} & \rho^{n+2} & \dots & \rho^{2n-2}\\
\end{pmatrix}$

\section*{1b. Problem 1}

\begin{align*}
X_1^2+2X_1X_2-4X_2X_3+X_3^2 &= (X_1+X_2)X_1 + (X_1-2X_3)X_2 + (-2X_2+X_3)X_3\\
&= \begin{pmatrix}
X_1 & X_2 & X_2
\end{pmatrix}\begin{pmatrix}
1 & 1 & 0\\
1 & 0 & -2\\
0 & -2 & 1\\
\end{pmatrix}\begin{pmatrix}
X_1\\
X_2\\
X_3
\end{pmatrix}
\end{align*}

$$X=\begin{pmatrix}X_1 & X_2 & X_3\end{pmatrix}'$$

$$A = \begin{pmatrix}
1 & 1 & 0\\
1 & 0 & -2\\
0 & -2 & 1\\
\end{pmatrix}$$

$$A\sum = AVar[X] = \sigma^2\begin{pmatrix}
1 & 1 & \frac{1}{4}\\
1 & -\frac{1}{2} & -2\\
0 & -\frac{7}{4} & \frac{1}{2}\\
\end{pmatrix}$$

Thus, $$E[X'AX] = tr(A\sum) + \mu'A\mu = \sigma^2 + \mu'A\mu$$

\section*{1b. Problem 2}

\begin{align*}
\sum_i(X_i-\bar{X})^2 &= \sum_i (X_i^2-2X_i\bar{X}+\bar{X}^2)\\
&= \sum_i X_i^2 -2\sum_i X_i \bar{X} + \bar{X}^2\\
&= \sum_i X_i^2 -n\bar{X}^2
\end{align*}

Now $\sum_iX_i^2 = X'X$ and $\bar{X} = \frac{1}{n}\sum_i X_i = \frac{1}{n} \mathbf{1'}X = \frac{1}{n}X'\mathbf{1} $

Hence,

\begin{align*}
\sum_i(X_i-\bar{X})^2 &= \sum_i X_i^2-n\bar{X}^2\\
&= X'X - n\frac{1}{n^2}(X'\mathbf{11'}X)\\
&= X'X-\frac{1}{n}(X'\mathbf{11'}X)\\
&= X'(\mathbf{I}-\frac{1}{n}\mathbf{11'})X
\end{align*}

Let $A=(\mathbf{I}-\frac{1}{n}\mathbf{11'})$

Now using $E[X'AX]= tr(A\sum) + \mu'A\mu$ we have:

\begin{align*}A\sum &= \begin{pmatrix}1-1/n & -1/n & -1/n & \dots & -1/n\\
-1/n & 1-1/n & -1/n & \dots & -1/n\\
\vdots\\
-1/n & -1/n & -1/n & \dots & 1-1/n
\end{pmatrix}\times diag(\sigma_1^2, \sigma_2^2, \dots, \sigma_n^2)\\
&= (1-\frac{1}{n})\sum_i \sigma_i^2
\end{align*}

Also, $$\mu'A = \begin{pmatrix} \mu & \mu & \dots & \mu \end{pmatrix} \times \begin{pmatrix}1-1/n & -1/n & -1/n & \dots & -1/n\\
-1/n & 1-1/n & -1/n & \dots & -1/n\\
\vdots\\
-1/n & -1/n & -1/n & \dots & 1-1/n
\end{pmatrix} = 0$$

and hence $\mu'A\mu=0$

\begin{align*}
E[\sum_i(X_i-\bar{X})^2] &= (1-\frac{1}{n})\sum_i \sigma_i^2  \\
E[\frac{1}{n(n-1)}\sum_i(X_i-\bar{X})^2]  &= \frac{1}{n^2} \sum_i \sigma_i^2
\end{align*}

Finally,

\begin{align*}
var(\bar{X}) &= Var(\frac{1}{n}\sum_i X_i)\\
&= \frac{1}{n^2} \sum_i Var(X_i)\\ \text{ since $X_i$ are mutually independent}\\
&= \frac{1}{n^2}\sum_i \sigma_i^2\\
&= E[\frac{1}{n(n-1)}\sum_i(X_i-\bar{X})^2]
\end{align*}

\section*{1b. Problem 3}

Given: $\bar{X_w} = \sum_i w_i X_i $ and $\sum w_i =1 $

\begin{align*}
Var(\bar{X_w}) &= Var(\sum w_i X_i)\\
&= \sum w_i^2Var(X_i)\ \text{ since $X_i$ are mutually independent}\\
&= \sum w_i^2 \sigma_i^2
\end{align*}

Now consider,

$$
\text{minimize} \sum w_i^2\sigma_i^2 \text{ subject to } \sum_i w_i =1
$$

We consider the following lagrange formulation:

$$min_w f(\mathbf{w}) = \sum_i w_i^2\sigma_i^2 +\lambda(\sum_iw_i-1)$$

Now to find optimal $\lambda$, we solve $\frac{\partial f(\mathbf{w})}{\partial w_i} = 0$

\begin{align*}
\frac{\partial f(\mathbf)}{\partial w_i} &= 2w_i\sigma_i^2+\lambda =0\\
\implies w_i &= -\frac{\lambda}{2\sigma_i^2}
\end{align*}

Thus, $w_i = -\frac{\lambda}{2\sigma_i^2}$ or $w_i \propto \frac{1}{\sigma_i^2}$

Using $\sum_i w_i=1$  we get:
\begin{eqnarray*}
\sum_i w_i = 1\\
\sum_i \frac{\lambda}{-2\sigma_i^2} =1\\
\implies \lambda = \frac{-2}{\sum_i 1/\sigma_i^2 }\\
\implies w_i = \frac{1}{\sigma_i^2 \sum_i(1/\sigma_i^2)}\\
\implies f_{min}(w) = \sum_i (\frac{1}{\sigma_i^2 \sum_i(1/\sigma_i^2)}) \sigma_i^2\\
v_{min} = \frac{1}{\sum_i(1/\sigma_i^2)}
\end{eqnarray*}

\subsection*{Part b}

\begin{align*}
\sum_i w_i (X_i-\bar{X_w})^2 &= \sum_i w_i(X_i^2-2X_i\bar{X_w}+\bar{X_w}^2)\\
&= \sum_i w_iX_i^2-2\bar{X_w}\sum_iw_iX_i+\bar{X_w^2}\\
&= \sum_i w_iX_i^2 -2\bar{X_w}^2+\bar{X_w}^2\\
&= \sum_i w_iX_i^2 -\bar{X_w}^2
\end{align*}

Now, we rewrite $\sum_i w_iX_i^2 = X'\Lambda X$ where $\Lambda = diag(w_1, w_2, \dots, w_n)$

and $\bar{X_w} = \sum_i w_iX_i = X'w = w'X$

\begin{align*}
\bar{X_w}^2 &= (\sum_i w_iX_i)^2\\
&=X'ww'X
\end{align*}

and hence $\sum_i w_i (X_i-\bar{X_w})^2=X'(\Lambda-ww')X$

Define $A=\Lambda-ww' = \begin{pmatrix}
w_1-w_1^2 & w_1w_2 & w_1w_3 & \dots & w_1w_n\\
w_2w_1 & w_2-w_2^2 & w_2w_3 & \dots & w_2w_n\\
\vdots & \vdots & \vdots & \vdots \\
w_nw_1 & w_nw_2 & w_nw_3 & \dots w_n-w_n^2\\
\end{pmatrix}$

so that
$$E[\sum_i w_i (X_i-\bar{X_w})^2] = E[X'AX]= tr(A\sum) + \mu'A\mu$$

$$tr(A\sum) = \sum_i(w_i-w_i^2)\sigma_i^2 = \sum_i w_i\sigma_i^2-w_i(w_i\sigma_i^2) = \sum_i (a-aw_i) = na-a$$

and the $(1,1)$ element of the matrix $\mu'A$ is given by:

\begin{align*}
\mu'A_1 &= \mu (w_1-w_1^2 - w_1\sum_{i=2}w_i)\\ &= w_1-w_1^2-w_1(1-w_1)\\
&= 0\end{align*}

and hence it's essentially a zero matrix(other elements are zero similarly)

Also, $$v_{min} = \frac{1}{\sum_i 1/\sigma_i^2} = \frac{1}{\sum_i \frac{w_i}{a}} = a$$

Thus,

\begin{align*}
E[S_w^2] &= \frac{1}{n-1} tr(A\sum) + \mu'A\mu\\
&= \frac{1}{n-1} (na-a) + 0\\
&=a\\
&= v_{min}
\end{align*}

\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{tikz}
\setlength{\parindent}{0pt}

\newtheorem*{theorem}{Theorem}
\newtheorem*{definition}{Definition}
\newtheorem*{lemma}{Lemma}
\newtheorem*{corollary}{Corollary}
\newtheorem{example}{Example}
\newtheorem*{trick}{Trick}
\newtheorem*{question}{Question}

\title{Lecture 3: Matrices}
\author{}
\date{}

\begin{document}

\maketitle

\section{Matrices}

A motivation for bringing in matrices is better expressing linear relations
between variables.

\begin{example}
  The coordinates of a point in two different Cartesian coordinate systems have
  linear relations. Suppose in the first coordinate system, the point
  $P = (x_1, x_2, x_3)$, and in the second one, the point $P = (u_1, u_2, u_3)$.
  The relations between them can be described by linear equations, such as
  \[
    \left\{ \begin{array}{ll}
    u_1 = 2x_1 + x_2 + 5x_3 \\
    u_2 = x_1 + 3x_2 + 2x_3 \\
    u_3 = x_1 + 2x_2 + x_3
    \end{array} \right.
  \]
  The reason why their relation is linear is that each unit vector in the
  second coordinate system can be decomposed into three vectors along the
  directions of the unit vectors in the first coordinate system, i.e. each unit
  vector in the second coordinate system has a unique coordinate in the first
  system. Since $x_1$, $x_2$, $x_3$ are just scalars in three directions of the
  first coordinate system, the coordinates of the unit vectors of the second
  system can also be expressed with $x_1$, $x_2$, $x_3$, a linear combination
  of these three scalars. Then the coordinate of the point in the second
  coordinate system is also a linear combination of the unit vectors in the
  system, so the coordinate of the point in the second coordinate system is a
  linear combination of the coordinate of the point in the first one.

  Then we can actually use matrix product to express the linear system:
  \[
    \begin{bmatrix}
      u_1 \\
      u_2 \\
      u_3 \\
    \end{bmatrix}
    =
    \begin{bmatrix}
      2 & 1 & 5 \\
      1 & 3 & 2 \\
      1 & 2 & 1 \\
    \end{bmatrix}
    \begin{bmatrix}
      x_1 \\
      x_2 \\
      x_3 \\
    \end{bmatrix}
  \]
  \[
    U = AX
  \]
  Such a matrix $A = \begin{bmatrix}
                       2 & 1 & 5 \\
                       1 & 3 & 2 \\
                       1 & 2 & 1 \\
                     \end{bmatrix}$ is called a tranformation matrix, and such
  operation $AX$ is called matrix product.
\end{example}

\subsection{Definition of matrix product}

Entries in a matrix product $AB$ are the dot products of the corresponding rows
in the matrix $A$ and the corresponding columns in the matrix $B$ respectively.

Therefore, in order to apply the matrix product to the matrices $A$ and $B$ in
this way $AB$,
\[
  \textnormal{the number of columns in A} = \textnormal{the number of rows in B}
\]
In other words, the width of $A$ must equal the height of $B$.

\subsection{Property of matrix product}

\subsubsection{Associative property}

The matrix product satisfies the associative property.

\[
  (AB)X = A(BX)
\]

Therefore, when there are multiple tranformation matrices applied on a vector
such as $ABX$, it means the tranformation matrix $B$ is applied to $X$ first,
then the tranformation matrix $A$ is applied to their result.

\subsubsection{Commutative property}

The matrix product doesn't satisfy the commutative property.

\[
  AB \neq BA
\]

Not only in some cases $AB$ is valid while $BA$ doesn't make sense, but also
when both $AB$ and $BA$ are valid, their results can be different.

\subsection{Identity Matrix}

The identity matrix $I$ is defined as the matrix where
\[
  X = IX
\]

Deriving from the definition, we know that all identity matrices are square
matrices, and identity matrices always have the elements of the main diagonal
(the list of entries $A_{i,j}$ where $i = j$) as 1, and other elements outside
of the main diagonal as 0.

\[
  I_{4 \times 4} =
  \begin{bmatrix}
    1 & 0 & 0 & 0 \\
    0 & 1 & 0 & 0 \\
    0 & 0 & 1 & 0 \\
    0 & 0 & 0 & 1 \\
  \end{bmatrix}
\]

\begin{example}
  Find the transformation matrix to make a plane rotated by $90^{\circ}$
  counterclockwise.

  Since the transformation matrix represents a rotation of a plane, i.e.
  transforming each 2D point on the plane into another 2D point, the size of
  the transformation matrix is $2 \times 2$:
  \[
    R =
    \begin{bmatrix}
      r_1 & r_2 \\
      r_3 & r_4 \\
    \end{bmatrix}
  \]
  Then we can use some points and their transformed results to figure out the
  elements:
  \[
    A =
    \begin{bmatrix}
      1 \\
      0 \\
    \end{bmatrix}
    A' =
    \begin{bmatrix}
      0 \\
      1 \\
    \end{bmatrix}
  \]
  \[
    B =
    \begin{bmatrix}
      0 \\
      1 \\
    \end{bmatrix}
    B' =
    \begin{bmatrix}
      -1 \\
      0 \\
    \end{bmatrix}
  \]
  After calculation,
  \[
    R =
    \begin{bmatrix}
      0 & -1 \\
      1 & 0 \\
    \end{bmatrix}
  \]
  We can verify the result by multiply itself by 4 times, which in effect
  doesn't rotate the plane at all, hence the result should be the identity
  matrix.
  \[
    RRRR =
    \begin{bmatrix}
      0 & -1 \\
      1 & 0 \\
    \end{bmatrix}
    \begin{bmatrix}
      0 & -1 \\
      1 & 0 \\
    \end{bmatrix}
    \begin{bmatrix}
      0 & -1 \\
      1 & 0 \\
    \end{bmatrix}
    \begin{bmatrix}
      0 & -1 \\
      1 & 0 \\
    \end{bmatrix} =
    \begin{bmatrix}
      1 & 0 \\
      0 & 1 \\
    \end{bmatrix}
  \]
\end{example}

\section{Inverse Matrix}

\subsection{Definition of inverse matrix}

The inverse matrix of $A$, usually denoted as $A^{-1}$, is defined as
\begin{gather*}
  A^{-1}A = I \\
  AA^{-1} = I \\
\end{gather*}

Deriving from the definition, only square matrices have inverse matrix. The
matrices which have its corresponding inverse matrix are said to be invertible.

Invert matrices can help solve matrix equations:
\begin{gather*}
  AX = B \\
  A^{-1}AX = A^{-1}B \\
  X = A^{-1}B \\
\end{gather*}

\subsection{How to invert a matrix}

The formula of inverse matrices is:
\[
  A^{-1} = \frac{1}{det(A)}adj(A)
\]
$adj(A)$ is the adjugate matrix of $A$, or the classical adjoint matrix of $A$.

\bigskip

Steps to invert a matrix:

1. Calculate the minor $M$ of $A$. The entry on the position $(i,j)$ of the
minor, called the $(i,j)$-minor of $A$ and denoted $M_{i,j}$, is the determinant
of the matrices that results from deleting the row $i$ and column $j$ of $A$.

2. Calculate the cofactor matrix $C$ of $A$. The entry on the position $(i,j)$
of the cofactor matrix is the result of multiplying $M_{i,j}$ by $(-1)^{i+j}$.

3. Calculate the adjugate matrix $adj(A)$ of $A$ by transposing the cofactor
matrix of $A$. Transposing means switching rows and columns correspondingly.

4. Calculate the inverse matrix of $A$ by dividing the adjugate matrix of $A$ by the determinant of $A$.

\end{document}


% !Mode:: "TeX:UTF-8"
% !TEX program  = xelatex
In this section, we will give a proper and accurate definition of geometric progression, from which we will derive the summation formula for the geometric progression in both finite and infinite cases.

\subsection{Definition}
\begin{defnbox}{Geometric Progression\cite{Weisstein2019gp}}{gp}
    A geometric sequence is a sequence $\{a_k\}, k=0, 1, \ldots$, such that each term is given by a multiple $r$ of the previous one. Another equivalent definition is that a sequence is geometric iff\footnote{if and only if} it has a zero series bias. If the multiplier is $r$, then the $k$th term is given by
    \[
        a_k = ra_{k-1} = r^2a_{k-2} = \cdots = a_0r^k.
    \]

    Taking $a_0=1$ gives the simple special case
    \[
        a_k = r^k.
    \]
\end{defnbox}

Accordingly, we can define the arithmetic progression in advance to facilitate future introductions.
\begin{defnbox}{Arithmetic Progression\cite{Weisstein2019ap}}{ap}
    An arithmetic progression, also known as an arithmetic sequence, is a sequence of $n$ numbers $\{a_0+kd\}^{n-1}_{k=0}$ such that the differences between successive terms is a constant $d$. That is,
    \[
        a_k = a_{k-1}+d = a_{k-2}+d = \cdots = a_0+kd.
    \]

    Taking $a_0=0$ gives the simple special case
    \[
        a_k = kd.
    \]
\end{defnbox}

\subsection{Derivation}
\subsubsection{Summation Formula For the Geometric Progression}
According to the definition of the geometric progression, we have,
\begin{equation}\label{E:gp-1}
    S_n = \sum_{k=1}^{n} a_k = \sum_{k=1}^{n} a_1 r^{k-1}.
\end{equation}

Multiply both sides of the equation by $r$, then we have,
\begin{equation}\label{E:gp-2}
    rS_n = \sum_{k=1}^{n} a_1 r^{k} = \sum_{k=1}^{n} a_{k+1} = \sum_{k=2}^{n+1} a_k.
\end{equation}

Subtract equation~\eqref{E:gp-2} from equation~\eqref{E:gp-1}, we have,
\begin{equation}\label{E:gp-3}
    (1-r) S_n = a_1 - a_{n+1} = a_1(1-r^n).
\end{equation}

Therefore, we need a classification discussion, if the $r$ equals to 1, both sides of the equation cannot divide $1-r$ simultaneously, but we can derivate the formula from equation~\eqref{E:gp-1}, $S_n=na_1$.

Above all, we derivate the summation formula for the geometric progression,
\begin{equation}\label{E:gp-4}
    S_n = \begin{cases}
              na_1 & \text{if } r=1, \\
              \frac{a_1(1-r^n)}{1-r} & \text{if } r\neq 1.
          \end{cases}
\end{equation}

Moreover, if we use the form of the limit, we can simplify the formula,
\begin{equation}\label{E:gp-5}
    S_n = \lim_{q\to r} \frac{a_1(1-q^n)}{1-q}.
\end{equation}

Next we consider what happens when $n\to\infty$. $1-q$ is finite, the convergence of $S_n$ is equivalent to convergence of $\lim_{q\to r}1-q^n$. Therefore,  $S$ convergence if and only if $|r|<1$.
\begin{equation}\label{E:gp-6}
    S = \lim_{n\to\infty} S_n = \lim_{\substack{n\to\infty \\ q\to r}} \frac{a_1(1-q^n)}{1-q} = \lim_{n\to\infty}\frac{a_1(1-r^n)}{1-r} = \frac{a_1}{1-r}.
\end{equation}

In summary, we get the summation formula for the geometric progression,
\[
    S_n = \begin{cases}
              na_1 & \text{if } r=1, \\
              \frac{a_1(1-r^n)}{1-r} & \text{if } r\neq 1.
          \end{cases}
\]
and $S$ convergence if and only if $|r|<1$,
\[
    S = \lim_{n\to\infty} = \frac{a_1}{1-r}.
\]

\subsubsection{Summation Formula For the Arithmetic progression}
The summation formula for the arithmetic progression is slightly different from the geometric progression, and it relates back to a famous mathematician, Gauss. The main idea of the formula comes from
\[
    1+2+\cdots+n = \frac{n(n+1)}{2}.
\]

The proof part is left to readers as a practice question, we just provide clues below,
\begin{equation}\label{E:ap-1}
    \begin{aligned}
        S_{n+1} &= \sum_{k=0}^n a_k \\
            &= \sum_{k=0}^n (a_0+kd) \\
            &= \sum_{k=0}^n a_0 + d\sum_{k=0}^n k.
    \end{aligned}
\end{equation}


    \filetitle{!equations}{Block of equations or assignments}{sstatelang/equations}

	\paragraph{Syntax for a system of equations to be solved numerically}

 \begin{verbatim}
 !equations
     EQUATION;
     EQUATION;
     EQUATION;
     ...

     !solvefor
         LIST_OF_VARIABLES
 \end{verbatim}

 \paragraph{Syntax for a system of equations to be solved symbolically}

 \begin{verbatim}
 !equations
     EQUATION;
     EQUATION;
     EQUATION;
     ...

     !symbolic
     !solvefor
         LIST_OF_VARIABLES
 \end{verbatim}

 \paragraph{Syntax for block of assignments}

 \begin{verbatim}
 !equations
     VARIABLE_NAME = EXPRESSION;
     VARIABLE_NAME = EXPRESSION;
     VARIABLE_NAME = EXPRESSION;
     ...
 \end{verbatim}

 \paragraph{Description}

 \paragraph{Example}


\documentclass[12pt]{mmalatex}
\usepackage{examples}

\begin{document}

\section*{Displaying long expressions}

This example uses a simple (though contrived) example of a Taylor series expansion of $1/(1+x)$ to demonstrate the problems that can arise when displaying very long expressions.

\begin{minipage}[t]{0.47\textwidth}
\begin{mathematica}
   f[x_] = 1/(1+x)
   ans = f[x]                       (* mma (ans.511,ans) *)
   ans = Series[f[x],{x, 0, 10}]    (* mma (ans.512,ans) *)
   ans = Series[f[x],{x, 0, 20}]    (* mma (ans.513,ans) *)
   ans = Series[f[x],{x, 0, 23}]    (* mma (ans.514,ans) *)
\end{mathematica}
\end{minipage}
\hskip 0.5cm
\begin{minipage}[t]{0.53\textwidth}
\begin{latex}
   \begin{dgroup*}[spread={5pt}]
      \begin{dmath*} f(x) = \Mma*{ans.511} \end{dmath*}
      \begin{dmath*}    {}= \Mma*{ans.512} \end{dmath*}
      \begin{dmath*}    {}= \Mma*{ans.513} \end{dmath*}
      \begin{dmath*}    {}= \Mma*{ans.514} \end{dmath*}
      \begin{dmath*}    {}= \Mma*[\hskip 2cm]{ans.514} \end{dmath*}
   \end{dgroup*}
\end{latex}
\end{minipage}

\vspace{18pt}

The first four lines of the following output were set using {\tt\small\verb|\Mma*|}
while the final line used {\tt\small\verb|\Mma*[\hskip=2cm]|}. The last pair of lines displays
the output for the same tag {\tt\small ans.514} and clearly the formatting of the second
last line is not ideal as the text has overlapped the tag. This was corrected in the final
line by using the option argument {\tt\small\verb|[\hskip=2cm]|} in the call to {\tt\small\verb|\Mma*|}.

\begin{dgroup*}[spread={5pt}]
   \begin{dmath*} f(x) = \Mma*{ans.511} \end{dmath*}
   \begin{dmath*}    {}= \Mma*{ans.512} \end{dmath*}
   \begin{dmath*}    {}= \Mma*{ans.513} \end{dmath*}
   \begin{dmath*}    {}= \Mma*{ans.514} \end{dmath*}
   \begin{dmath*}    {}= \Mma*[\hskip 2cm]{ans.514} \end{dmath*}% LCB: do we need extra space for the tag?
\end{dgroup*}

\end{document}


\subsubsection{The Definition of Entropy}
Now that we know what micro and macro states are, we can define entropy using statistics! In this case, entropy is defined as
\begin{equation}
    \label{eqn:(41)}
    S=k_{b}\ln{\Omega}
\end{equation}

where $\Omega$ is the number of possible microstates within the current macrostate of the system. \\
With a closed system of constant volume, we can characterize macrostates using the temperature of the system, and microstates as the exact distribution of kinetic energies amongst the particles in the system. Using this we can start to see all the parallels between the two different ways of defining entropy.
\begin{enumerate}
    \item At $T=0\textrm{K}$ every particle must have no kinetic energy. So, there's only one possible microstate, and by equation \ref{eqn:(41)} above $S=0$. This is exactly what we'd expect from the third law of thermodynamics, at absolute zero temperature, we obtain that entropy has a constant value\footnote{Note that the third law of thermodynamics does not say a system reaches zero entropy as it approaches zero temperature, as there do exist certain systems where the minimum energy (i.e. low temperature) states are not unique. Therefore, there still can be more than one microstate, and hence nonzero (but still constant) entropy. You don't need to know about these in any kind of detail.}.
    \item If you have two systems (a and b), the amount of microstates in a system including both of them is $\Omega_{a}\Omega_{b}$. Therefore, the total entropy is...
    \begin{align*}
        S_{total}&=k_{b}\ln{\Omega_{a}\Omega_{b}} \\
        &=k_{b}\ln{\Omega_{a}}+k_{b}\ln{\Omega_{b}} \\
        &=S_{1}+S_{2}
    \end{align*}
    Since the total entropy is the sum of the entropy of both systems, entropy is an extensive quantity!
    \item In our example for microstates/macrostates above, the macrostate with the highest number of microstates was the one where each side had three particles (if you don't believe me count them). This analogy applies to two systems that can exchange energy, they have the maximum number of microstates (and therefore entropy) when they share the energy equally (i.e. have the same temperature).
\end{enumerate}
To close off our discussion of macrostates, microstates, and the statistical definition of entropy, let us consider another concrete example, in the form of the entropy of rolling two dice. Therein, we can consider the macrostate as the sum of the two dice rolls, and the microstates as the particular values we got on each dice (here, we imagine that the dice are unique, in that I can tell them apart from each other). For example, if I rolled the first dice to be 6, and the second dice to be 1, then the microstate could be described with $(6,1)$ and this microstate belongs to the macrostate of $7$. A complete description of all of the macrostates and microstates (as well as the entropy of the macrostate, as defined in this section) is given in the table below, though you may want to work this out for yourself first for practice.

\begin{center}
 \begin{tabular}{|c c c|}
 \hline
 \textbf{Macrostates} & \textbf{Microstates} & \textbf{Entropy}\\
 \hline\hline
 2 & (1,1) & $k_b\ln(1) = 0$\\
 \hline
 3 & (1,2),(2,1) & $k_b\ln(2)$ \\
 \hline
 4 & (1,3),(2,2),(3,1) & $k_b\ln(3)$ \\
 \hline
 5 & (1,4),(2,3),(3,2),(4,1) & $k_b\ln(4)$ \\
 \hline
 6 & (1,5),(2,4),(3,3),(4,2),(5,1) & $k_b\ln(5)$ \\
 \hline
 7 & (1,6),(2,5),(3,4),(4,3),(5,2),(6,1) & $k_b\ln(6)$\\
 \hline
 8 & (2,6),(3,5),(4,4),(5,3),(6,2) & $k_b\ln(5)$\\
 \hline
 9 & (3,6),(4,5),(5,4),(6,3) & $k_b\ln(4)$\\
 \hline
 10 & (4,6),(5,5),(6,4) & $k_b\ln(3)$\\
 \hline
 11 & (5,6),(6,5) & $k_b\ln(2)$\\
 \hline
 12 & (6,6) & $k_b\ln(1) = 0$\\
 \hline
\end{tabular}
\end{center}
This example also demonstrates an assumption I implicitly made up until this point: all possible microstates are equally likely to appear (in the last example, unless you had loaded dice, every dice roll would have been equally likely!). This is a very fundamental assumption, so much so that it is called the \textbf{Fundamental Assumption of Statistical Mechanics}. Just to restate it for a more general case (as not every system in the universe is a bunch of dice), it states that every microstate of a system is equally likely and the system on average spends the same amount of time in each of them. We will use this in the next section to conclude our discussion of entropy.


\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}

\title{MATH 542 Homework 9}
\author{Saket Choudhary\\skchoudh@usc.edu}

\begin{document}
\maketitle
\subsection*{Problem 1}
Since $\mathbf{X}$ is full rank, the least sqare estimate of $\beta$ for the linear model $\mathbf{Y}=\mathbf{y}\beta+\epsilon$ then $\hat{\beta} = \mathbf{(X'X)^{-1}X'y}$ also this is an unbiased estimator of $\beta$ that is $E[\hat{\beta}]=\beta$
\begin{align*}
a'\beta &= a'E[\hat{\beta}]\\
&= a'E[\mathbf{(X'X)^{-1}X'y}]\\
&= E[a'\mathbf{(X'X)^{-1}X'y}]\\
&= E[b'\mathbf{y}]
\end{align*}
Hence when $\mathbf{X}$ is full rank, we have can take $b'=a'\mathbf{(X'X)^{-1}X'}$ making every $a'\beta$ estimable

For each individual $\beta_i$ to be estimable, we set $a'=\begin{pmatrix}0 & 0 & \dots & 0 & 1 & 0 \dots 0 \end{pmatrix}$
setting $a_i=1$ remaining zero.

\subsection*{Problem 2}
Since each $a_i'\beta$ is estimable: $a_i'\beta = E[b_i'Y]$
Consider $\lambda_i \in \mathcal{R}$
\begin{align*}
\lambda_1a_1'\beta + \lambda_2a_2'\beta + \dots + \lambda_na_n'\beta &= \lambda_1 E[b_1'Y] +  \lambda_2 E[b_2'Y] + \dots \lambda_n E[b_n'Y]\\
&= \sum_i E[\lambda_ib_i'Y]\\
&= \sum E[c'Y] \text{ where }c'= \mathbf{\lambda I b'}
\end{align*}
where $\lambda =  \begin{pmatrix}\lambda_1 & \lambda_2 &  \dots & \lambda_n \end{pmatrix}$ and $b' = \begin{pmatrix} b_1 & b_2 & \dots b_n \end{pmatrix}$

Hence linear combination of $a_i'\beta$ is also estimable.

\subsection*{Problem 3}
\begin{align*}
X &= \begin{pmatrix}
1 & 1 & 3\\
1 & 1 & -2\\
1 & 1 & 0\\
1 & 0 & 0\\
\end{pmatrix}
\beta & = \begin{pmatrix}
\beta_1\\
\beta_2\\
\beta_3
\end{pmatrix}\\
\beta_1 & = \begin{pmatrix}1 & 1 & 0\end{pmatrix}\mathbf{\beta}\\
\beta_1-\beta_2 &= \begin{pmatrix}1 &-1 & 0\end{pmatrix}\mathbf{\beta}\\
5\beta_1+3\beta_2+9\beta_3 &= \begin{pmatrix}5 & 3& 9\end{pmatrix}\mathbf{\beta}\\
\end{align*}

X has full column rank and hence we can make use of the theorem we proved in Problem 1 to say that all three cases $(a,b,c)$ are indeed estimable, that is since $X$ is full rank, every $a'\beta$ is estimable (One particular case of the corollary proved in Problem 1 here is the case $a$ where $a'=\begin{pmatrix}1 & 0 & 0\end{pmatrix}$
\subsection*{Problem 4}
\subsubsection*{Problem 4.a}
Define $\mathbf{y}=\begin{pmatrix}Y_1 & Y_2 &  Y_3\end{pmatrix}'$, $\mathbf{\tau} = \begin{pmatrix} \tau_1 & \tau_2 & \tau_3 \end{pmatrix}$, $\mathbf{\epsilon} = \begin{pmatrix}\epsilon_1 & \epsilon_2 & \epsilon_3\end{pmatrix}$

Thus, $\mathbf{y} = \begin{pmatrix}1 & 1& 1\\
1 & 0 & 1\\
0 & 1 & 0
\end{pmatrix}\mathbf{\tau} + \mathbf{\epsilon}$

Rank of design matrix is 2

\subsubsection*{Problem 4.b}

$E[b'Y] = a'\beta = b'X\beta$ iff $a'=b'X$ or $a=Xb'$
%which is equivalent to saying that $a$ belongs to row space of $X$
$\tau_2$ is estimable:
\begin{align*}
\tau_2  &= a'\mathbf{\beta}= \begin{pmatrix}0 & 1 & 0\end{pmatrix}\mathbf{\tau}\\
&= EY_3\\
&= E[\begin{pmatrix}0 & 0 & 1\end{pmatrix}]
\end{align*}

$\tau_1 = \begin{pmatrix}1 & 0 & 0 \end{pmatrix}\mathbf{\tau}$ so we need to find $b$ such that $\begin{pmatrix}1 & 0 & 0 \end{pmatrix}' = X'b$ It is clear to see no b does not exist.

$\tau_2 =  \begin{pmatrix}0 & 1 & 0\end{pmatrix}\mathbf{\tau}$ is estimable because $b=\begin{pmatrix}0 & 1 & 0 \end{pmatrix}$

$\tau_3$ is not estimable because of symmetry with $\tau_1$

\subsubsection*{Problem 4.c}

$\tau_1-2\tau_2+\tau_3 = \begin{pmatrix}1 &-2 &1 \end{pmatrix}\mathbf{\tau} = \begin{pmatrix}1 &1 &0\\ 1 &0 &1\\ 0 & 1 & 0\end{pmatrix} \begin{pmatrix}0\\1\\-2\end{pmatrix}$ and hence it is estimable

\subsubsection*{Problem 4.d}

A possible unbiasd estimator of $\tau_1-2\tau_2+\tau_3 = E[Y_2-2Y_3]$ i.e $Y_2-2Y_3$ whihc is not necessarily BLUE.

For BLUE, we simply take the OLS estimate of $\beta$ as $\hat{\beta} = \mathbf{X'X}^{-1}\mathbf{X'y}$ and from Gauss-Markov model BLUE follows.

Generalized inverse(using $R$) $\mathbf{X'X}^{-1} = \begin{pmatrix} \frac{1}{6} & \frac{-1}{6} & \frac{1}{6}\\
\frac{-1}{6} & \frac{2}{3} & \frac{-1}{6}\\
\frac{1}{6} & \frac{-1}{6} & \frac{1}{6} \\
\end{pmatrix}$

so $\hat{\beta} =\mathbf{X'X}^{-1}\mathbf{X'y} = \begin{pmatrix} \frac{1}{6} & \frac{1}{3} & \frac{-1}{6}\\
\frac{1}{3} & \frac{-1}{3} & \frac{2}{3}\\
\frac{1}{6} & \frac{1}{3} & \frac{-1}{6}\\
\end{pmatrix}\mathbf{y}$

and hence $\tau_1-2\tau_2+\tau_3 = \begin{pmatrix} 1 &-2 &1\end{pmatrix}\begin{pmatrix} \frac{1}{6} & \frac{1}{3} & \frac{-1}{6}\\
\frac{1}{3} & \frac{-1}{3} & \frac{2}{3}\\
\frac{1}{6} & \frac{1}{3} & \frac{-1}{6}\\
\end{pmatrix}\mathbf{y} = \frac{-Y_1+4Y_2-5Y_3}{3}  $

Thus BLUE of  $\tau_1-2\tau_2+\tau_3$ : $ \frac{-Y_1+4Y_2-5Y_3}{3}$
\end{document}


\section{Numpy} % (fold)
\label{sec:numpy}
Generate matrices $A$, with random Gaussian entries, $B$, a Toeplitz matrix, where
$A \in \R^{n\times m}$ and $B \in \R^{m \times m}$, for n = 200, m = 500.

\begin{questions}

\titledquestion{Matrix operations}
\label{sub:mat_ops}

Calculate $A + A$, $A A^\top, A^\top A$ and $AB$.
Write a function that computes $A(B - \lambda I)$ for any $\lambda$.

\titledquestion{Solving a linear system}
\label{sub:linear_system}

Generate a vector $b$ with $m$ entries and solve $Bx = b$.

\titledquestion{Norms} % (fold)
\label{sub:norms}

Compute the Frobenius norm of $A$: $\|A\|_F$ and the infinity norm of $B$:
$\|B\|_{\infty}$. Also find the largest and smallest singular values of $B$.

% titledquestion norms (end)

\titledquestion{Power iteration}
\label{sub:power}

Generate a matrix $Z$, $n \times n$, with Gaussian entries, and use the power iteration to
find the largest eigenvalue and corresponding eigenvector of $Z$.
How many iterations are needed till convergence?

Optional: use the \texttt{time.clock()} method to compare computation time when varying $n$.

\titledquestion{Singular values} % (fold)
\label{sub:svd}

Generate an $n \times n$ matrix, denoted by $C$, where each entry is $1$ with probability $p$ and
$0$ otherwise.
Use the linear algebra library of Scipy to compute the singular values of $C$.
What can you say about the relationship between $n$, $p$ and the largest singular value?

% titledquestion svd (end)

\titledquestion{Nearest neighbor} % (fold)
\label{sub:nearest_neighbor}

Write a function that takes a value $z$ and an array $A$ and finds
the element in $A$ that is closest to $z$. The function should
return the closest value, not index.

Hint: Use the built-in functionality of Numpy rather than writing code to find this
value manually. In particular, use brackets and \texttt{argmin}.

% titledquestion nearest_neighbor (end)

\end{questions}
% section numpy (end)


\chapter{eq}

\begin{equation}
T_{motes}\ = N_{slots}*N_{motes}
\end{equation}

\begin{tabular}{lllp{10cm}}
where, &   &   &   \\
%\hline
  & $T_{motes}$  & = & the maximum number of motes that can be accommodated in the network \\
%\hline
  & $N_{slots}$ & = & the maximum number of transmission slots in a transmission period \\
%\hline
  & $N_{motes}$ & = & the maximum number of motes in each transmission slot of the \textit{\textbf{root}} mote \\
\end{tabular}

\begin{center}
$N_{motes}\ =\ \dfrac{internal\ message\ buffer\ size}{message\ length}$
\end{center}

$N_{slots}\ =\ \dfrac{transmission\ interval}{transmission\ slot\ duration}$

$B_{charged}\ =\ T_{sunlight}*I$

\begin{tabular}{lllp{10cm}}
where, &   &   &   \\
%\hline
  & $B_{charged}$  & = & the amount of battery charged $(milliamps)$ \\
%\hline
  & $T_{sunlight}$ & = &  the time for which sunlight is present $(hours)$\\
%\hline
  & $I$ & = & the amount of current used to charge the battery $(milliamps)$\\
\end{tabular}