\section{Stability Analysis}

With regards to the stability analysis, we aim to try and prove some results on
the "stability function" $R(z)$ as seen in Levecue's \cite{levecue} chapter five
through eight by following the arguments in Staff \cite{staff}.

Suppose we have the following ordinary differential equation:
\begin{equation} \label{eq:lode}
  \begin{cases}
    u' = \mu u, \quad \mu < 0, t > 0 \\
    u(0) = u_0
  \end{cases}
\end{equation}
We would like to analyze the stability of parareal on this system with a
coarse operator $\coarse$ and a fine operator $\fine$.

\subsection{Deriving a Stability Function}

With respect to Parareal, let's try and write our iteration in the form
$\lambda_n^k = H(n,k)\lambda_0$. Suppose that we have some integrators $\coarse$
and $\fine$, such that they're actually the same method, but with different time
scales. In addition, suppoes that they're iteration can be described by they're
own stability function $R(z)$. Then our parareal iteration goes to:
\begin{align*}
  \lambda_{n+1}^{k+1} & = \coarse(t^{n+1},t^n,\lambda_n^{k+1}) +
  \fine(t^{n+1},t^n,\lambda_n^k) -
  \coarse(t^{n+1},t^n,\lambda_n^k) \\
  & = R(\mu\Delta t)\lambda_n^{k+1} +
  R(\mu\delta t)^s \lambda_n^k -
  R(\mu\Delta t)\lambda_n^k \\
\end{align*}
where we say $s = \Delta t/ \delta t$, i.e. how many fine steps are needed to
make one coarse step. Combining like terms results in:
\[
  \lambda_{n+1}^{k+1} =
  R(\mu\Delta t)\lambda_n^{k+1} +
  \left( R(\mu\delta t)^s - R(\mu\Delta t)\right) \lambda_n^k
\]
Now, if we were to note the terms on $\lambda$, we note that we have something
very similar to the recurrence relation on combinations $\binom{n}{k} =
\binom{n}{k-1} + \binom{n-1}{k-1}$. Exploiting that relationship, we can unroll
our recursion into:
\[
  \lambda_{n+1}^{k+1} = \left( \sum_{i=0}^k \binom{n}{i} \left[R(\mu \delta t)^s -
    R(\mu \Delta t)\right]^i R(\mu \Delta t)^{n-i} \right) \lambda_0 =
    H(\mu, n,k,\delta t, \Delta t) \lambda_0
\]
We would like to see when it's true that $\abs{H} \leq 1$. As it turns out, this
function is very difficult to analyze by itself, none of my plots of $H$ by
itself generated useful stability regions. Instead we try to analyze this under
a few easy cases, and see exactly how parareal transforms the stability region
of known temporal integrators.

\subsection{Trivial case: $k = 0$}

First, we make a sanity check, suppose we make no Parareal iterations. We would
assume that in this case that the stability function of Parareal should devolve
to the stability function of $\coarse$, since that's the only thing computed.
Indeed:
\begin{align*}
  H & = \left( \sum_{i=0}^0 \binom{n}{i} \left[ R(\mu \delta t)^s -
    R(\mu \Delta t) \right]^i R(\mu \Delta t)^{n-i} \right) \lambda_0 \\
  & = \binom{n}{0} \left[ R(\mu \delta t)^s - R(\mu \Delta t) \right]^0 R(\mu
  \Delta t)^{n} \lambda_0 \\
  & = R(\mu \Delta t)^{n} \lambda_0
\end{align*}
which is precisely the stability function of $\coarse$.

\subsection{Trivial case $\coarse = \fine$.}

Suppose we were to choose $\coarse = \fine$, time scales and all. This implies
that $s = 1$. We would expect that in this case, our parareal algorithm should
result in the stability region of $\coarse$, since the corrector step should
result in a correction of $0$. Indeed we see that all parareal iterations with
$k > 0$ are canceled to zero by the term $[R(\mu \delta t)^s - R(\mu \Delta
t)]$. Otherwise we're just left with $\binom{n}{0} R(\mu \Delta t)^n \lambda_0 =
R(\mu \Delta t)^n \lambda_0$, which is exactly the stability region of
$\coarse$, as expected.

\subsection{Parareal Under Explicit Euler}

The most iconic temporal integrator is the Euler methods, and we would like to
see how Parareal changes they're stability regions.  Suppose now that we are
using explicit Euler for our coarse and fine integrators.  Recall, that this
implies that the stability function for our integrators $\coarse$ and $\fine$
are $(1+z)$ and $(1+z)^s$ respectively, and remember that original the
region of stability for explicit Euler on the given ODE is for $z$ in the unit
disk centered around $-1$.

Suppose $\fine = \coarse$ but with the time scale squared, i.e. $\delta t =
\Delta t^2$ . So for example, if $\coarse$ has a time step of $1/10$, then
$\fine$ has a time step of $1/100$.  Then, $s = \frac{\Delta t}{\delta t} =
\Delta t^{-1}$. As an example, suppose that $\Delta t = 1/2$. Then $s = 2$, and
(let $z = \Delta t$) our stability region goes to:
\begin{align*}
  H & = \sum_{i=0}^k \binom{n}{i} \left[ R(\mu \delta t)^s -
    R(\mu \Delta t) \right]^i R(\mu \Delta t)^{n-i} \\
  & = \sum_{i=0}^k \binom{n}{i} \left[ (1+\mu z^2)^2 -
    (1+\mu z) \right]^i (1+\mu z)^{n-i} \\
  & = \sum_{i=0}^k \binom{n}{i}
    \left[
      \mu^2 z^4 + 2\mu z^2 - \mu z
    \right]^i (1+\mu z)^{n-i} \\
  & = \sum_{i=0}^k \binom{n}{i}
    \left[
      \mu z(\mu z^3 + 2 z - 1)
    \right]^i (1+\mu z)^{n-i} \\
\end{align*}
At this point the roots of the cubic are ugly, so I don't take the general case
any further. Now we examine interesting cases under $k$. Recall, by arguments
made above, we already know that $k = 0 \implies$ that we just have stability
region $(1 + \mu z)$. For the following analysis, suppose we take $n = 2$ too.

\subsubsection{One Parareal Iteration: $k = 1$}

Suppose we only make one parareal iteration. Then the above goes to:
\[
  (1 + \mu z)^2 + 2\mu z(\mu z^3 + 2z - 1)(1+\mu z) = (1+\mu z)(2\mu^2 z^4 +
  4\mu z^2  - \mu z + 1)
\]

\subsubsection{Two Parareal Iteration: $k = 2$}

Suppose we make two parareal iteration. This is a little more painful to
compute:
\[
  (1 + \mu z)^2 + 2\mu z(\mu z^3 + 2z - 1)(1+\mu z) + \mu z(\mu z^3 + 2 z - 1) =
  (\mu z^2 + 1)^4
\]
So the stability region $(1 + \mu z)^2$ transformed to $(1 + \mu z^2)^4$ under
two parareal iteration, which is exactly the same region.

\begin{figure}[!htb]
  \centering
  \begin{subfigure}{\textwidth}
    \centering
    \includegraphics[width=.49\textwidth]
      {./resources/parareal_stability1}
    \includegraphics[width=.49\textwidth]
      {./resources/parareal_stability2}
  \end{subfigure}%
  \caption{On the left we have the stability region for $k = 1$, and
  on the right we have it for $k = 2$. It's a little bit difficult to see, but
  for $k = 1$, the stability region is a bit smaller in $\Delta t$, but larger
  in $\mu$.}\label{fig:parareal_stability}
\end{figure}


\include{config/config}

\begin{document}
% ========== Edit your name here
\title{MATH 2901 Basic Probability Lecture Notes 6}
\author{Instructor: Richard Kleeman}
\date{}
\maketitle

%\medskip

% ========== Contents begin here ==============
This lecture we study the expectation of the average of $n$ i.i.d. random variables $X_i$, $i=1,\dots, n$, \ie, the expectation of $Q_n = \frac{1}{n} (X_1 + \dots + X_n)$. As $n \to \infty$, we will introduce the central limit theorem and show that $Q_n$ converges to a normal distribution provided $\Var(X_i)$ exists.

\section{Sums of Discrete Random Variables}
Suppose $X$ and $Y$ are two independent discrete random variables with distribution functions $p_X(x)$ and $p_Y(y)$. Let $Z = X + Y$, we want to find the the distribution function of $Z$.

Suppose that $X = k$, where $k$ is some integer. Then $Z = z$ if and only if $Y = z-k$. So the event $Z = z$ is the union of the pairwise disjoint events
\begin{equation*}
    (X=k) \quad  \text { and } \quad  (Y=z-k)
\end{equation*}
where $k$ runs over the integers. Since these events are pairwise disjoint, we have
\begin{equation}
    \label{eq:6.1}
    \tag{6-1}
    \Prob(Z=z)=\sum_{k=-\infty}^{\infty} \Prob(X=k)  \Prob(Y=z-k)
\end{equation}
which is the \textbf{distribution function of the random variable $Z$}.

\begin{definition}
Let $X$ and $Y$ be two independent integer-valued random variables, with distribution functions $p_X(x)$ and $p_Y(y)$ respectively. Then the \textbf{convolution} of $p_X(x)$ and $p_Y(y)$ is the distribution function $p_Z = p_X * p_Y$ given by
\begin{equation*}
    p_Z(z) = \sum_{x} p_X(x) p_Y(z-x)
\end{equation*}
for $x\in \Z$. The function $p_Z(z)$ is the distribution function of the random variable $Z = X + Y$.
\end{definition}

\section{Sums of Continuous Random Variables}
\begin{definition}
Let $X$ and $Y$ be two continuous random variables with density functions $f_X(x)$ and $f_Y(y)$ respectively. Assume that both $f_X(x)$ and $f_Y(y)$ are defined for all real numbers. Then the \textbf{convolution} $f * g$ of $f$ and $g$ is the function given by
\begin{equation}
    \label{eq:6.2}
    \tag{6-2}
    (f_X * f_Y)(z) = \int_{-\infty}^{+\infty} f_X(z-y) f_Y(y) d y = \int_{-\infty}^{+\infty} f_Y(z-x) f_X(x) d x.
\end{equation}
\end{definition}

\begin{theorem}
Let $X$ and $Y$ be two independent random variables with density functions $f_X(x)$ and $f_Y(y)$ defined for all $x$ and $y$. Then the sum $Z = X + Y$ is a random variable with density function $f_Z(z)$, where $f_Z$ is the convolution of $f_X$ and $f_Y$.
\end{theorem}

\begin{theorem}
Let $\{X_i\}$, $i=1,\dots, n$ be a sequence of independent random variables with density functions $f_{X_1}(x), \dots, f_{X_n}(x)$ respectively, then we have
\begin{equation*}
    f_{X_1 + \cdots + X_n}(x) = f_{X_1} * \left( f_{X_2} * ( \cdots * f_{X_n}) \right)(x).
\end{equation*}
\end{theorem}

\begin{example}[Sum of two independent uniform random variables]
Let $X$ and $Y$ be random variables describing our choices and $Z = X + Y$ their sum. Then we have
\begin{equation*}
    f_{X}(x)=f_{Y}(x)=\left\{\begin{array}{ll}{1} & {\text { if } 0 \leq x \leq 1} \\ {0} & {\text { otherwise }}\end{array}\right.
\end{equation*}
and the density function for the sum is given by
\begin{equation*}
    f_{Z}(z)=\int_{-\infty}^{+\infty} f_{X}(z-y) f_{Y}(y) d y = \int_{0}^{1} f_{X}(z-y) d y.
\end{equation*}
Now the integrand is $0$ unless $0 \leq z-y \leq 1$ and then it is $1$. So if $0 \leq z \leq 1$, we have
\begin{equation*}
    f_{Z}(z)=\int_{0}^{z} d y=z,
\end{equation*}
while if $1 < z \leq 2$, we have
\begin{equation*}
    f_{Z}(z)=\int_{z-1}^{1} d y=2-z,
\end{equation*}
and if $z < 0$ or $z > 2$ we have $f_Z(z) = 0$. Hence
\begin{equation*}
    f_{Z}(z)=\left\{\begin{array}{ll}{z,} & {\text { if } 0 \leq z \leq 1} \\ {2-z,} & {\text { if } 1<z \leq 2} \\ {0,} & {\text { otherwise }}\end{array}\right.
\end{equation*}
\end{example}

\begin{example}[Sum of two independent exponential random variables]
Let $X, Y$, and $Z = X + Y$ denote the relevant random variables, and $f_X$ , $f_Y$ , and $f_Z$ their densities. Then
\begin{equation*}
    f_{X}(x)=f_{Y}(x)=\left\{\begin{array}{ll}{\lambda e^{-\lambda x},} & {\text { if } x \geq 0} \\ {0,} & {\text { otherwise }}\end{array}\right.
\end{equation*}
If $z > 0$,
\begin{equation*}
    f_{Z}(z)=\int_{-\infty}^{+\infty} f_{X}(z-y) f_{Y}(y) d y =\int_{0}^{z} \lambda e^{-\lambda(z-y)} \lambda e^{-\lambda y} d y =\int_{0}^{z} \lambda^{2} e^{-\lambda z} d y =\lambda^{2} z e^{-\lambda z},
\end{equation*}
while if $z < 0$, $f_Z(z) = 0$. Hence
\begin{equation*}
    f_{Z}(z)=\left\{\begin{array}{ll}{\lambda^{2} z e^{-\lambda z},} & {\text { if } z \geq 0} \\ {0,} & {\text { otherwise }}\end{array}\right.
\end{equation*}
\end{example}

\section{Generating functions}
A sequence $a = \{a_i \spacevert i = 0, 1 , 2, \dots \}$ of real numbers may contain a lot of information. One concise way of storing this information is to wrap up the numbers together in a ``generating function". For example, the (ordinary) \textbf{generating function} of the sequence $a$ is the function $G_a$ defined by
\begin{equation*}
    G_{a}(s)=\sum_{i=0}^{\infty} a_{i} s^{i} \quad \text { for } s \in \mathbb{R} \text { for which the sum converges. }
\end{equation*}
In many circumstances it is easier to work with the generating function $G_a$ than with the original sequence $a$.

\begin{theorem}[Abel's theorem]
If $a_i \geq 0$ for all $i$ and $G_a(s)$ is finite for $\abs{s} < 1$, then $\lim_{s \uparrow 1} G_a (s) = \sum_{i=1}^ \infty a_i$, whether the sum is finite or equals $+\infty$. This standard result is useful when the radius of convergence $R$ satisfies $R = 1$, since then one has no a priori right to take the limit as $s \uparrow 1$.
\end{theorem}

\subsection{Moment generating function}
\begin{definition}
The \textbf{moment generating function} of the random variable $X$ is the function $M: \R \mapsto [0, \infty)$ given by the Laplace transform of the corresponding p.d.f. $f_X(s)$:
\begin{equation*}
    M_X(t) = \Exp \left( e^{tX} \right) = \int_{-\infty}^\infty e^{tx} f_X(x) dx,
\end{equation*}
or corresponding p.m.f. $p_X(k)$:
\begin{equation*}
    M_X(t) = \sum_{k} e^{t k} \Prob(X=k)
    = \sum_k \sum_{n=0}^{\infty} \frac{(t k)^{n}}{n !} \Prob(X=k)
    = \sum_{n=0}^{\infty} \frac{t^{n}}{n !}\left(\sum_k k^{n} \Prob(X=k)\right)
    = \sum_{n=0}^{\infty} \frac{t^{n}}{n !} \Exp\left(X^{n}\right).
\end{equation*}
$M_X(-t)$ is so called \textbf{bilateral} Laplace transform of $f_X(x)$ or $p_X(k)$.
\end{definition}

Under the assumption that $M_X(t)$ is infinitely differentiable at $t=0$, the following statements are true.
\begin{enumerate}[(a)]
    \item $M^\prime(0) = \Exp(0) = \mu$.
    \item $M^{(n)}(0) = \Exp(X^n)$.
    \item Using Taylor's theorem, $M_X(t) = \sum_{k=0}^\infty \frac{t^k}{k!} \Exp(X^k)$.
\end{enumerate}

\begin{theorem}
If $X$ and $Y$ are independent, then
\begin{equation*}
    \begin{split}
        M_{X+Y}(t) &= \int_{-\infty}^\infty e^{tz} f_{x+y}(z) dz \\
        &= \int_{-\infty}^\infty e^{tz} \int_{-\infty}^\infty f_X(x) f_Y(z-x) dx dz \\
        &= \int_{-\infty}^\infty e^{t(x+y)} \int_{-\infty}^\infty f_X(x) f_Y(y) dx dy \\
        &= M_X(t) M_Y(t).
    \end{split}
\end{equation*}
\end{theorem}

\subsection{Characteristic functions}
Sometimes $\Exp(e^{tX})$ may blow up. So we consider some transformations in the complex domain, which usually perform better.

\begin{definition}
The \textbf{characteristic function} of $X$ is the function $\phi: \R \mapsto \C$ defined by
\begin{equation*}
    \phi(t)=\Exp\left(e^{i t X}\right) \quad \text { where } \quad i=\sqrt{-1}.
\end{equation*}
We often write $\phi_x$ for the characteristic function of the random variable $X$. Characteristic functions are related to Fourier transforms.
\end{definition}

\begin{theorem}
The characteristic function $\phi$ satisfies:
\begin{enumerate}[(a)]
    \item $\phi(0) = 1$, $\abs{\phi(t)} \leq 1$ for all $t$.
    \item $\phi$ is uniformly continuous on $\R$ w.r.t. $t$.
    \item If $X \sim \mathcal{N}(0,1)$, then $\phi_{X}(t) = e^{-t^2/2}$.
\end{enumerate}
\end{theorem}

\begin{proof}
We only prove the first statement.
\begin{equation*}
    \begin{split}
        \abs{\phi(t)} &= \abs{\int_{-\infty}^\infty e^{itx} f(x)dx } \\
        &\leq \int_{-\infty}^\infty \abs{e^{itx}} f(x) dx \quad \text{(triangle inequality)} \\
        &= \int_{-\infty}^\infty f(x) dx \quad (\abs{e^{itx}}=1) \\
        &= 1
    \end{split}
\end{equation*}
\end{proof}

\begin{example}[Cauchy distribution]
If $f(x) =
\frac{1}{\pi(1+x^2)}$, then the corresponding characteristic function is
\begin{equation*}
    \phi(t)=\frac{1}{\pi} \int_{-\infty}^{\infty} \frac{e^{i t x}}{1+x^{2}} d x = e^{-\abs{t}}.
\end{equation*}
\end{example}

\begin{theorem}
The following statements are true.
\begin{enumerate}[(a)]
    \item If $\phi^{(k)}(0)$ exists, then
    \begin{equation*}
        \left\{\begin{array}{ll}{\Exp\left|X^{k}\right|<\infty} & {\text { if $k$}  \text { is even }} \\ {\Exp\left|X^{k-1}\right|<\infty} & {\text { if $k$ } \text { is odd }}\end{array}\right.
    \end{equation*}

    \item If $\Exp(\abs{X^k}) < \infty$, then
    \begin{equation*}
        \phi(t)=\sum_{j=0}^{k} \frac{\mathbb{E}\left(X^{j}\right)}{j !}(i t)^{j}+\mathrm{o}\left(t^{k}\right)
    \end{equation*}
    and so $\phi^{(k)}(0)=i^{k} \mathbb{E}\left(X^{k}\right)$.
\end{enumerate}
\end{theorem}
\begin{theorem}
If $X$ and $Y$ are independent then
\begin{equation*}
    \phi_{X+Y}(t)=\phi_{X}(t) \phi_{Y}(t).
\end{equation*}
Similarly, if $X_1, \dots, X_n$ are independent, then
\begin{equation*}
    \phi_{X_1 + \cdots + X_n}(t)= \prod_{i=1}^n \phi_{X_i}(t).
\end{equation*}
\end{theorem}

\begin{theorem}
If $a, b \in \R$ and $Y = aX+b$, then $\phi_{Y}(t)=e^{i t b} \phi_{X}(a t)$.
\end{theorem}
\begin{proof}
\begin{equation*}
    \phi_{Y}(t)=\Exp\left(e^{i t(a X+b)}\right)=\Exp\left(e^{i t b} e^{i(a t) X}\right) = e^{i t b} \Exp\left(e^{i(a t) X}\right)=e^{i t b} \phi_{X}(a t).
\end{equation*}
\end{proof}

\begin{theorem}
Random variables $X$ and $Y$ are independent if and only if
\begin{equation*}
    \phi_{X, Y}(s, t)=\phi_{X}(s) \phi_{Y}(t) \quad \text { for all } s \text { and } t.
\end{equation*}
\end{theorem}

\begin{definition}
We say that the sequence $F_1 , F_2, \dots $ of distribution functions converges to the distribution function $F$, written $F_n \to F$, if $F(x) = \lim_{n\to\infty} F_n(x)$ at each point $x$ where $F$ is continuous.
\end{definition}

\begin{theorem}[Continnity theorem]
Suppose that $F_1 , F_2, \dots $ is a sequence of distribution functions
with corresponding characteristic functions $\phi_1 , \phi_2, \dots $.
\begin{enumerate}[(a)]
    \item If $F_n \to F$ for some distribution function $F$ with characteristic function $\phi$, then $\phi_n(t) \to \phi(t)$ for all $t$.
    \item Conversely, if $\phi(t) \lim_{n\to\infty} \phi_n(t)$ exists and is continuous at $t=0$, then $\phi$ is the characteristic function of some distribution function $F$, and $F_n \to F$.
\end{enumerate}
\end{theorem}

\section{Central limit theorem}
\begin{definition}
If $X, X_1 , X_2 , \dots$ is a sequence of random variables with respective distribution functions $F, F_1, F_2, \cdots$, we say that $X_n$ converges in distribution to $X$, written $X_{n} \stackrel{\mathrm{D}}{\rightarrow} X$, if $F_n \to F$ as $n \to\infty$.
\end{definition}

\begin{theorem}[Central limit theorem]
Let $X_1 , X_2, \dots$ be a sequence of independent identically distributed random variables with finite mean $\mu$ and finite nonzero variance $\sigma^2$, and let $S_n = X_1 + X_2 + \cdots + X_n$. Then
\begin{equation*}
    \frac{S_{n}-n \mu}{\sqrt{n \sigma^{2}}} \stackrel{\mathrm{D}}{\rightarrow} N(0,1) \quad \text { as } \quad n \rightarrow \infty.
\end{equation*}
\end{theorem}

\begin{proof}
First, write $Y_i = \frac{X_i - \mu}{\sigma}$, and let $\phi_Y$ be the characteristic function of the $Y_i$. We have that
\begin{equation*}
    \phi_{Y}(t)=1-\frac{1}{2} t^{2}+o\left(t^{2}\right).
\end{equation*}
Note that $Y_i$ are i.i.d. So the characteristic function of $\sum_{i=1}^n Y_i$ is
\begin{equation*}
    \phi_n = [\phi_{Y}(t)]^n = \left[ 1-\frac{1}{2} t^{2}+o\left(t^{2}\right) \right]^n.
\end{equation*}
Also, the characteristic function $\psi_n$ of
\begin{equation*}
    U_{n}=\frac{S_{n}-n \mu}{\sqrt{n \sigma^{2}}}=\frac{1}{\sqrt{n}} \sum_{i=1}^{n} Y_{i}
\end{equation*}
satisfies
\begin{equation*}
    \psi_{n}(t)=\left\{\phi_{Y}(t / \sqrt{n})\right\}^{n}=\left\{1-\frac{t^{2}}{2 n}+o\left(\frac{t^{2}}{n}\right)\right\}^{n} \rightarrow e^{-\frac{1}{2} t^{2}} \quad \text { as } \quad  n \rightarrow \infty,
\end{equation*}
where we used
\begin{equation*}
    \lim_{n\to\infty} \left( 1 + \frac{a}{n} \right)^n = e^a.
\end{equation*}
The last function is the characteristic function of the $\mathcal{N}(0, 1)$ distribution, and an application of the continuity theorem completes the proof.
\end{proof}

\begin{corollary}
$Q_n = \frac{1}{n}S_n \to \mathcal{N} \left(\mu, \frac{\sigma^2}{n} \right)$, $S_n \to \mathcal{N}(\mu, n \sigma^2)$. The sampling error is proportional to $\frac{1}{\sqrt{n}}$.
\end{corollary}
There is a generalization. If $X_i$ is not i.i.d., we can still use the central limit theorem.

\end{document}


% !TEX root = main.tex

%-------------------------------------------------
\section{Fisher information}\label{sec:fisher}

Let $X$ be a continuous random variable and let $\mathcal{M}=\{f(x;\theta):\theta\in\Theta\}$ be a statistical model for its distribution. How can we quantify the amount of information that $X$ carries about $\theta$?

We assume that our statistical model $\mathcal{M}$ satisfies Condition~\ref{cond:regularity1} along with the following additional requirements:
\begin{condition}\label{cond:regularity2}
\ben
\setcounter{enumi}{3}
\it The PDFs $f(x;\theta)$ are twice differentiable as functions of $\theta$.
\it The integral $\displaystyle\int f(x;\theta)\,dx$ is twice differentiable under the integral sign as a function of $\theta$.
\een
\end{condition}
We shalll not consider these in detail.

%----------------------------------------------------------------------
\subsection{The score function}

Let $X$ be a single observation.% and consider the log-likelihood function $\ell(\theta) = \ell(\theta;X) = \log f(X;\theta)$.

% defn: score function
\begin{definition}
The first derivative of the log-likelihood function is called the \emph{score function},
\[
u(\theta;x) = \frac{\partial}{\partial\theta}\log f(x;\theta)
\]
\end{definition}

\bit
\it Given a fixed observation $X=x$, we think of $u(\theta;x)$ as a function of $\theta$.
\it Given a fixed value of $\theta$, we think of $u(\theta;X)$ as a random variable. This is a transformation of $X$ which we denote by
	\[
		U = \frac{\partial}{\partial\theta}\log f(X;\theta)
	\]
\eit
% random variable (a transformation of $X$):
%\[
%U(X) = u(\theta,X) = = \frac{\partial}{\partial\theta}\log f(X;\theta)
%\]
%
%For a fixed realisation $X=x$, we think of $u(\theta;x)$ as a function of $\theta$. If $X$ is allowed to vary on the other hand, and $\theta$ is fixed, we think of $u(\theta;X)$ as a transformation of $X$:
%\[
%U(X) = u(\theta,X) = = \frac{\partial}{\partial\theta}\log f(X;\theta)
%\]

%
%\begin{remark}
%The score function $u(\theta;\mathbf{X})$ is the gradient of the log-likelihood function $\ell(\theta;\mathbf{X})$ at $\theta$, and thus quantifies the extent to which the log-likelihood depends on $\theta$.
%\end{remark}

%\begin{remark}
%\bit
%\it If $\theta$ is a vector of parameters, $u(\theta)$ is a vector of partial derivatives (one for each component).
%\it Given $\theta$, we can think of u(\theta;\boldX) as a random variable (a transformation of $\boldX$).
%%\it $V$ is the gradient of the log-likelihood function at $\theta$
%%\it Thus it quantifies the extent to which the log-likelihood depends on $\theta$.
%\eit

% lemma: expected score is zero
\begin{lemma}\label{lem:expe_score_function}
%Let $X$ be a random variable, and let $f(x;\theta)$ be its PMF/PDF, where $\theta$ is an unknown parameter. Under the regularity conditions stated above, the expected value of the score function $u(\theta;X)$ is zero:
%\[
%\expe\left(\frac{\partial}{\partial\theta}\log f(X;\theta)\right) = 0.
%\]
Under the regularity conditions stated above,
%\[
%%\expe\big[u(\theta,X)\big] = 0.
%\expe\big[U] = 0.
%\]
\[
\expe(U) = \expe\left(\frac{\partial}{\partial\theta}\log f(X;\theta)\right) = 0.
\]
\begin{proof}
We prove the lemma for the case where $X$ is a continuous random variable. For every value of $\theta$, the integral of $f(x;\theta)$ with respect to $x$ satisfies
\[
\int_{-\infty}^{\infty} f(x;\theta)\,dx = 1.
\]
Taking the derivative with respect to $\theta$ (and applying the regularity conditions), we see that
\[
\frac{\partial}{\partial\theta}\left(\int_{-\infty}^{\infty} f(x;\theta)\,dx\right)
	= \int_{-\infty}^{\infty} \frac{\partial f(x;\theta)}{\partial\theta} \,dx
	= 0.
\]
By the chain rule, the first partial derivative of $\log f(x;\theta)$ with respect to $\theta$ is given by
\[
\frac{\partial}{\partial\theta}\log f(x;\theta)
	= \frac{1}{f(x;\theta)}\frac{\partial}{\partial\theta} f(x;\theta).
\]
Hence
\begin{align*}
\expe\left(\frac{\partial}{\partial\theta}\log f(X;\theta)\right)
	= \int_{-\infty}^{\infty} \left(\frac{\partial}{\partial\theta}f(x;\theta)\right) f(x;\theta)\,dx
	= \int_{-\infty}^{\infty} \frac{\partial}{\partial\theta}f(x;\theta)\,dx
	= 0
\end{align*}
as required.
\end{proof}
\end{lemma}

% example: geometric
\begin{example}\label{ex:score_geometric}
Find the score function of a single observation from the $\text{Geometric}(\theta)$ distribution whose PMF is $\prob(X=k)=\theta(1-\theta)^{k-1}$ for $k=1,2,3,\ldots$ where $0<\theta<1$ is unknown.
\begin{solution}
\bit
\it The likelihood function is $L(\theta;x) = \theta(1-\theta)^{x-1}$.
\it The log-likelihood function is $\ell(\theta;x) = \log\theta + (x-1)\log(1-\theta)$.
\eit
The score function is therefore equal to
\begin{align*}
u(\theta;x) = \ell'(\theta,x)
	& = \frac{\partial}{\partial\theta} \big(\log\theta + (x-1)\log(1-\theta)\big) \\
	& = \frac{1}{\theta} - \frac{x-1}{1-\theta}
	= \frac{1-\theta x}{\theta(1-\theta)}
\end{align*}
so $U = \displaystyle\frac{1-\theta X}{\theta(1-\theta)}$. Note that because $\expe(X)=1/\theta$,
\[
\expe(U)
	= \expe\left(\frac{1-\theta X}{\theta(1-\theta)}\right)
	= \frac{1-\theta\expe(X)}{\theta(1-\theta)}
	= 0,
\]
which verifies Lemma~\ref{lem:expe_score_function}.
\end{solution}
\end{example}

%----------------------------------------------------------------------
\subsection{Fisher information}
%----------------------------------------------------------------------
% defn
\begin{definition}
The variance of the score function is called the \emph{Fisher information},
\[
I(\theta) = \var(U) = \var\left(\frac{\partial}{\partial\theta}\log f(X;\theta)\right).
\]
\end{definition}

The following lemma gives a conveient way of computing $I(\theta)$ in terms of the derivative of the score function.
% lemma
\begin{lemma}\label{lem:fisher}
Under the regularity conditions stated above, the Fisher information satisfies
\[
I(\theta) = -\expe(U')= -\expe\left(\frac{\partial^2}{\partial\theta^2}\log f(X;\theta)\right).
\]
\end{lemma}

\begin{proof}
By Lemma~\ref{lem:expe_score_function}, the expected value of the score function is zero:
\[
\expe(U) = \int_{-\infty}^{\infty} \left(\frac{\partial}{\partial\theta}\log f(x;\theta)\right) f(x;\theta) \,dx = 0.
\]
Taking the derivative of both sides with respect to $\theta$,
\[
\int_{-\infty}^{\infty} \left(\frac{\partial^2}{\partial\theta^2}\log f(x;\theta)\right)\,f(x;\theta) \,dx
+
\int_{-\infty}^{\infty} \left(\frac{\partial}{\partial\theta}\log f(x;\theta)\right)\,\frac{\partial f(x;\theta)}{\partial\theta} \,dx
= 0.
\]
By the chain rule,
\[
\frac{\partial}{\partial\theta}\log f(x;\theta)  = \frac{1}{f(x;\theta)}\frac{\partial}{\partial\theta}f(x;\theta).
%\frac{\partial f(x;\theta)}{\partial\theta} = \frac{\partial\log f(x;\theta)}{\partial\theta}\,f(x;\theta).
\]
Thus we have
\[
\int_{-\infty}^{\infty} \left(\frac{\partial^2}{\partial\theta^2}\log f(x;\theta)\right)\,f(x;\theta) \,dx
+
\int_{-\infty}^{\infty} \left(\frac{\partial}{\partial\theta}\log f(x;\theta)\right)^2 f(x;\theta)\,dx
= 0.
\]
This can be written as
\[
\expe\left(\frac{\partial^2}{\partial\theta^2}\log f(x;\theta)\right)
+
\expe\left[\left(\frac{\partial}{\partial\theta}\log f(x;\theta)\right)^2 \right]
= 0.
\]

Because the expected value of the score function is zero, the second term is the variance of the score function. Hence
\[
\expe\left(\frac{\partial^2}{\partial\theta^2}\log f(x;\theta)\right)
+
I(\theta) = 0
\]
and the result follows.
%
%\[
%I(\theta) + \int_{-\infty}^{\infty} \left(\frac{\partial\log f(x;\theta)}{\partial\theta}\right)^2\,f(x;\theta)\,dx = 0.
%\]
%Hence
%\[
%I(\theta)	= -\expe\left(\frac{\partial^2\log f(X;\theta)}{\partial\theta^2}\right),
%\]
%as required.
\end{proof}

% example: bernoulli
\begin{example}
Find the Fisher information of an observation from the $\text{Bernoulli}(\theta)$ distribution.
\begin{solution}
Let $X\sim\text{Bernoulli}(\theta)$. The PMF of $X$ can be written as
\[
f(x;\theta)=\theta^x(1-\theta)^{1-x} \text{\quad for $x\in\{0,1\}$, and zero otherwise.}
\]
To compute the Fisher information, we need the following:
\begin{align*}
\log f(x;\theta)
	& = \ x\log\theta + (1-x)\log(1-\theta), \\[2ex]
%\frac{\partial}{\partial\theta} \log f(x;\theta),
%	& = \ \frac{x}{\theta} - \frac{1-x}{1-\theta} \\[2ex]
%\frac{\partial^2}{\partial\theta^2} \log f(x;\theta),
%	& = \ -\frac{x}{\theta^2} - \frac{1-x}{(1-\theta)^2}.
\frac{\partial}{\partial\theta}\log f(x;\theta)
	& = \ \frac{x}{\theta} - \frac{1-x}{1-\theta}, \\[2ex]
\frac{\partial^2}{\partial\theta^2} \log f(x;\theta)
	& = \ -\frac{x}{\theta^2} - \frac{1-x}{(1-\theta)^2}.
\end{align*}
%\bit
%\it $\displaystyle\log f(x;\theta)
%	= x\log\theta + (1-x)\log(1-\theta)$.
%\it $\displaystyle\frac{\partial}{\partial\theta} \log f(x;\theta)
%	= \frac{x}{\theta} - \frac{1-x}{1-\theta}$.
%\it $\displaystyle\frac{\partial^2}{\partial\theta^2} \log f(x;\theta)
%	= -\frac{x}{\theta^2} - \frac{1-x}{(1-\theta)^2}$.
%\eit
Note that because $\expe(X)=\theta$, the expected value of the score is zero:
\[
\expe(U) = \expe\left(\frac{\partial}{\partial\theta}\log f(X;\theta)\right)
	= \expe\left(\frac{X}{\theta} - \frac{1-X}{1-\theta}\right)
	= \frac{\theta}{\theta} - \frac{1-\theta}{1-\theta}
	= 0.
\]
The Fisher information is
\begin{align*}
I(\theta)
	& = -\expe\left(-\frac{X}{\theta^2}-\frac{1-X}{(1-\theta)^2}\right) \\
	& = \frac{\theta}{\theta^2}+\frac{1-\theta}{(1-\theta)^2}
	= \frac{1}{\theta}+\frac{1}{1-\theta}
	= \frac{1}{\theta(1-\theta)}
\end{align*}
Recall that if $X\sim\text{Bernoulli}(\theta)$, then $\var(X)=\theta(1-\theta)$:
\bit
\it If the variance is small ($\theta\approx 0 \text{ or } \theta\approx 1$), $X$ carries a lot of information about $\theta$.
\it If the variance is large ($\theta\approx 1/2$), $X$ carries relatively little information about $\theta$.
\eit
\end{solution}
\end{example}

%%% example
%\begin{example}
%Find the Fisher information of an observation $X\sim\text{Geometric}(\theta)$.
%\begin{solution}
%From Example~\ref{ex:score_geometric}, the score function of $X$ is
%\[
%u(\theta;x) = \frac{1-\theta x}{\theta(1-\theta)}
%\]
%The derivative of $u(\theta;x)$ with respect to $\theta$ (which is also the second derivative of the log-likelihood function), is
%\begin{align*}
%\frac{\partial}{\partial\theta} u(\theta,X)
%	& = \frac{\theta(1-\theta)(-X) - (1-\theta X)(1-2\theta)}{\theta^2(1-\theta)^2} \\
%	& = -\frac{1 - 2\theta +\theta^2X}{\theta^2(1-\theta)^2}
%\end{align*}
%Because $\expe(X)=1/\theta$, the Fisher information is therefore
%\begin{align*}
%I(\theta)
%	& = -\expe\left(\frac{\partial}{\partial\theta} u(\theta,X)\right) \\
%	& = \frac{1 - 2\theta +\theta^2\expe(X)}{\theta^2(1-\theta)^2} \\
%	& = \frac{1 - \theta}{\theta^2(1-\theta)^2} \\
%	& = \frac{1}{\theta^2(1-\theta)}
%\end{align*}
%Note that $I(\theta)$ reaches a mainimum at $\theta=2/3$, which
%\end{solution}
%\end{example}
%

% example: normal
\begin{example}\label{example:fisher_information_normal}
Find the Fisher information of an observation from the $N(\theta,\sigma^2)$ distribution, whose mean $\theta$ is unknown but whose variance $\sigma^2$ is known.

\begin{solution}
Let $X\sim N(\theta,\sigma^2)$. Then
\begin{align*}
f(x;\theta)
	& = \ \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(x-\theta)^2}{2\sigma^2}\right), \\
\log f(x;\theta)
	& = \ -\frac{1}{2}\log(2\pi\sigma^2) - \frac{(x-\theta)^2}{2\sigma^2}, \\
\frac{\partial}{\partial\theta} \log f(x;\theta)
	& = \ \frac{x-\theta}{\sigma^2}, \\
\frac{\partial^2}{\partial\theta^2} \log f(x;\theta)
	& = \ -\frac{1}{\sigma^2}.
\end{align*}
Hence the Fisher information of $X\sim N(\theta,\sigma^2)$ is
\[
I(\theta)
	= -\expe\left(\frac{\partial^2}{\partial\theta^2}\log f(X;\theta)\right)
	= \frac{1}{\sigma^2}.
\]
\bit
\it If the variance is small, $X$ carries a lot of information about $\theta$.
\it If the variance is large, $X$ carries relatively little information about $\theta$.
\eit
\end{solution}
\end{example}

%----------------------------------------------------------------------
\subsection{Random samples}
%----------------------------------------------------------------------
The score function and Fisher information of a single observation extend naturally to random samples $X_1,X_2,\ldots,X_n$ from the distribution of $X$.

%Given a random sample $X_1,X_2,\ldots,X_n$, the log-likelihood function of $\theta$ is
%\[
%\ell(\theta;X_1,X_2,\ldots,X_n) = \prod_{i=1}^n f(X_i;\theta).
%\]

% defn: scores
\begin{definition}
The \emph{score function} of a random sample $X_1,X_2,\ldots,X_n$ is the first partial derivative of its log-likelihood function with respect to $\theta$. By independence,
\[
u(\theta;\mathbf{X})
	= \frac{\partial}{\partial\theta} \log L(\theta;\mathbf{X})
	= \sum_{i=1}^n \frac{\partial}{\partial\theta} \log L(\theta;X_i)
	= \sum_{i=1}^n u(\theta;X_i)
\]
\end{definition}

% fisher information for random samples
\begin{definition}
The \emph{Fisher information} of a random sample $X_1,X_2,\ldots,X_n$ is the variance of its score function:
\[
I_n(\theta)
	= \var\left(\frac{\partial}{\partial\theta}\log L(\theta;\mathbf{X})\right)
	= \sum_{i=1}^n\var\left(\frac{\partial}{\partial\theta}\log L(\theta;X_i)\right)
	= nI(\theta),
%	= -\expe\left(\frac{\partial^2}{\partial\theta^2}\log L(\theta;\mathbf{X})\right)
%	= -\sum_{i=1}^n \expe\left(\frac{\partial^2}{\partial\theta^2}\log L(\theta;X_i)\right)
\]
where $I(\theta)$ is the Fisher information of a single observation.
\end{definition}

%By independence, the Fisher information of a random sample is the total Fisher information of the individual observations.

%% lemma
%\begin{lemma}\label{lem:fisher_random_sample}
%The Fisher information of a random sample $X_1,X_2,\ldots,X_n$ is the total Fisher information of the individual observations,
%\[
%I_n(\theta) = nI(\theta)
%\]
%where $I(\theta)$ is the Fisher information of a single observation.
%\end{lemma}
%
%% proof
%\begin{proof}
%\begin{align*}
%I_n(\theta)
%	& = \var\left(\frac{\partial}{\partial\theta}\log f(X_1,X_2,\ldots,X_n;\theta)\right) \\
%	& = \var\left(\frac{\partial}{\partial\theta}\log \prod_{i=1}^n f(X_i;\theta)\right) \\
%	& = \var\left(\frac{\partial}{\partial\theta}\sum_{i=1}^n \log f(X_i;\theta)\right) \\
%	& = \var\left(\sum_{i=1}^n \frac{\partial}{\partial\theta}\log f(X_i;\theta)\right) \\
%	& = \sum_{i=1}^n \var\left(\frac{\partial}{\partial\theta}\log f(X_i;\theta)\right) \quad\text{(because the $X_i$ are independent)} \\
%	& = nI(\theta).
%\end{align*}
%\end{proof}

\begin{remark}
$I_n(\theta)$ increases with the sample size: the more data we have, the more information we have about $\theta$.
\end{remark}

%----------------------------------------------------------------------
\begin{exercise}
\begin{questions}
\question % <<< Poisson
Find the score function of an observation from the $\text{Poisson}(\theta)$ distribution, and verify that its expected value is zero.
\begin{answer}
The PMF of the $\text{Poisson}(\theta)$ distribution is
\[
f(x;\theta) = \displaystyle\frac{\theta^x\exp(-\theta)}{x!} \text{for $x=0,1,2,3,\ldots$ (zero otherwise)}
\]
Let $X\sim\text{Poisson}(\theta)$. The score function for a single observation $X=x$ is
\[
u(\theta;x)
	= \frac{\partial}{\partial\theta}\log f(x,\theta)
	= \frac{\partial}{\partial\theta}(x\log\theta - \theta - \log x!)
	= \frac{x-\theta}{\theta}.
\]
Since $\expe(X)=\theta$, by the linearity of expectation we have
\[
\expe\big[u(\theta;X)\big]
	= \expe\left(\frac{X-\theta}{\theta}\right)
	= \frac{\expe(X)-\theta}{\theta}
	= \frac{\theta-\theta}{\theta}
	= 0 \quad\text{as required.}
\]
\end{answer}
\question % <<< Geometric
Find the Fisher information $I(\theta)$ of an observation from the $\text{Geometric}(\theta)$ distribution, and find the value of $\theta$ for which $I(\theta)$ is minimum.
\begin{answer}
From Example~\ref{ex:score_geometric}, the score function of $X$ is
\[
u(\theta;X) = \frac{1-\theta X}{\theta(1-\theta)}
\]
The derivative of $u(\theta;X)$ with respect to $\theta$ is
\begin{align*}
\frac{\partial}{\partial\theta} u(\theta,X)
	= \frac{\theta(1-\theta)(-X) - (1-\theta X)(1-2\theta)}{\theta^2(1-\theta)^2}
	= -\frac{1 - 2\theta +\theta^2X}{\theta^2(1-\theta)^2}
\end{align*}
Because $\expe(X)=1/\theta$, the Fisher information is therefore
\begin{align*}
I(\theta)
	= -\expe\left(\frac{\partial}{\partial\theta} u(\theta,X)\right)
	& = \frac{1 - 2\theta +\theta^2\expe(X)}{\theta^2(1-\theta)^2} \\
	& = \frac{1 - \theta}{\theta^2(1-\theta)^2} \\
	& = \frac{1}{\theta^2(1-\theta)}
\end{align*}
To find the minimum, we differentiate $I(\theta)$ and set the resulting expression to zero:
\[
I'(\theta) = \frac{3\theta-2}{\theta^3(1-\theta)^2}	= 0.
\]
This shows that $I(\theta)$ has a turning point at $\theta=2/3$. The second derivative of $I(\theta)$ is
\[
I''(\theta) = \frac{9\theta^2 - 16\theta + 9}{\theta^4(1-\theta)^3}.
\]
The discriminant of the quadratic expression on the numerator is negative, which shows that $I''(\theta)>0$ for all $\theta$, and hence $I(\theta)$ has a minimum at $\theta=2/3$.
\par\smallskip
Note that for the $\text{Bernoulli}(\theta)$ distribution, $I(\theta)$ reaches its minimum at $\theta=1/2$:
\[
I(\theta) = \frac{1}{\theta(1-\theta)},
\qquad
I'(\theta) = \frac{2\theta -1}{\theta^2(1-\theta)^2},
\qquad
I''(\theta) = \frac{2\big[\theta(1-\theta)+(2\theta-1)^2\big]}{\theta^3(1-\theta)^3}.
\]
\end{answer}
\end{questions}
\end{exercise}
%----------------------------------------------------------------------


\documentclass[12pt]{article}

\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{float}
\usepackage{ulem}
\usepackage{bm}
\usepackage{anysize}

\marginsize{2cm}{2cm}{0.9cm}{1.8cm}

\title{EECE 5639 Computer Vision\\ [2ex] \begin{large} Homework \#1 \end{large} }
\author{Jiyu Tian}
\date{}

\begin{document}
\maketitle
%%---------------------------------------------------------------
%% Question 1
%%---------------------------------------------------------------
\section{Solution:}
(a) The equations of the image of the given line is:
\begin{equation*}
x = f\frac{X_0 + \lambda(X_1 - X_0)}{Z_0 + \lambda(Z_1 - Z_0)}
\end{equation*}
\begin{equation*}
y = f\frac{Y_0 + \lambda(Y_1 - Y_0)}{Z_0 + \lambda(Z_1 - Z_0)}
\end{equation*}
\begin{equation*}
z = f
\end{equation*}
\noindent(b) As $|\lambda| \to \infty$:
\begin{equation*}
\lim\limits_{\lambda\to\infty}x = f\frac{X_1 - X_0}{Z_1 - Z_0}
\end{equation*}
\begin{equation*}
\lim\limits_{\lambda\to\infty}y = f\frac{Y_1 - Y_0}{Z_1 - Z_0}
\end{equation*}
\begin{equation*}
\lim\limits_{\lambda\to\infty}z = f
\end{equation*}
\noindent If the 3D line is not parallel to the image plane, $Z_0 \neq Z_1$. The image of $P$ will converge to a point which depends on the relative position of $P_0$ and $P_1$.\\\\
(c) Let $\lambda = 1/2$, the image of the midpoint of the segment $P_0P_1$ is
\begin{equation*}
x_{midP_0P_1} = f\frac{X_0 + X_1}{Z_0 + Z_1}
\end{equation*}
\begin{equation*}
y_{midP_0P_1} = f\frac{Y_0 + Y_1}{Z_0 + Z_1}
\end{equation*}
\begin{equation*}
z_{midP_0P_1} = f
\end{equation*}
\noindent However the midpoint of the image segment $p_0p_1$ is
\begin{equation*}
x_{midp_0p_1} = \frac{f}{2}\left ( \frac{X_0}{Z_0} + \frac{X_1}{Z_1} \right )
\end{equation*}
\begin{equation*}
y_{midp_0p_1} = \frac{f}{2}\left ( \frac{Y_0}{Z_0} + \frac{Y_1}{Z_1} \right )
\end{equation*}
\begin{equation*}
z_{midp_0p_1} = f
\end{equation*}
\noindent If the 3D line is not parallel to the image plane, $Z_0 \neq Z_1$. Then $x_{midp_0p_1} \neq x_{midP_0P_1},\ y_{midp_0p_1} \neq y_{midP_0P_1}$,  the image of the midpoint of the segment $P_0P_1$ is not the midpoint of the image segment $p_0p_1$.

\vfill
\clearpage
%%---------------------------------------------------------------
%% Question 2
%%---------------------------------------------------------------
\section{Solution:}
From the picture we have
\begin{equation*}
\begin{aligned}
x_c &= - y_w + 2\\
y_c &= z_w - 2\\
z_c &= - x_w + 10
\end{aligned}
\end{equation*}
And thus
\begin{equation*}
M_{ext} = \left[ \begin{array}{cccc}
0 & -1 & 0 & 2\\
0 & 0 & 1 & -2\\
-1 & 0 & 0 & 10\\
0 & 0 & 0 & 1
\end{array} \right ]
\end{equation*}
Line equation for those sides of the floor tiles parallel to the camera optical axis can be given in world coordinate:
\begin{equation*}
P_w = \left[ \begin{array}{c}
-\lambda\\
n\\
0\\
1
\end{array} \right ]
\end{equation*}
where $n \in \{ 0,\ 0.5,\ 1,\ 1.5,\ 2,\ 2.5,\ 3,\ 3.5,\ 4 \}$ denotes each line of tiles, and $\lambda \in R$ denotes the points on line. World coordinate to camera coordinate is given by
\begin{equation*}
P_c = M_{ext} \cdot P_w = \left[ \begin{array}{cccc}
0 & -1 & 0 & 2\\
0 & 0 & 1 & -2\\
-1 & 0 & 0 & 10\\
0 & 0 & 0 & 1
\end{array} \right ]\left[ \begin{array}{c}
-\lambda\\
0.5n\\
0\\
1
\end{array} \right ] = \left[ \begin{array}{c}
-n+2\\
-2\\
\lambda + 10\\
1
\end{array} \right ]
\end{equation*}
And accordingly on the image plane,
\begin{equation*}
\left[ \begin{array}{c}
x'\\
y'\\
z'
\end{array} \right ] = \left[ \begin{array}{cccc}
f & 0 & 0 & 0\\
0 & f & 0 & 0\\
0 & 0 & 1 & 0
\end{array} \right ]  \left[ \begin{array}{c}
-n+2\\
-2\\
\lambda + 10\\
1
\end{array} \right ] = \left[ \begin{array}{c}
(-n+2)f\\
-2f\\
\lambda + 10
\end{array} \right ]
\end{equation*}
\begin{equation*}
P_i = \left[ \begin{array}{c}
x\\
y
\end{array} \right ] = \left[ \begin{array}{c}
x'/z'\\
y'/z'
\end{array} \right ] = \left[ \begin{array}{c}
\frac{-n+2}{\lambda + 10}f\\
-\frac{2}{\lambda + 10}f
\end{array} \right ]
\end{equation*}
As $|\lambda| \to \infty$,
\begin{equation*}
\lim\limits_{\lambda\to\infty}P_i = \lim\limits_{\lambda\to\infty}\left[ \begin{array}{c}
\frac{-n+2}{\lambda + 10}f\\
-\frac{2}{\lambda + 10}f
\end{array} \right ] = \left[ \begin{array}{c}
0\\
0
\end{array} \right ]
\end{equation*}
Therefore, the images of the sides of the floor tiles parallel to the camera optical axis intersect at a single point $[0,\ 0]'$, i.e. the origin.
\vfill
\clearpage
%%---------------------------------------------------------------
%% Question 3
%%---------------------------------------------------------------
\section{Solution:}
Suppose a point on the line is $P_0 = (x_0,\ y_0,\ z_0)$, and a vector parallel to the line is $v = (v_x\  v_y,\ v_z)$. Then the line equation in 3D under camera coordinate is
\begin{equation*}
P = P_0 + \lambda v = \left[ \begin{array}{c}
x_0 + \lambda v_x\\
y_0 + \lambda v_y\\
z_0 + \lambda v_z
\end{array} \right ]
\end{equation*}
The image of the 3D line is
\begin{equation*}
\left[ \begin{array}{c}
x'\\
y'\\
z'
\end{array} \right ] = \left[ \begin{array}{cccc}
f & 0 & 0 & 0\\
0 & f & 0 & 0\\
0 & 0 & 1 & 0
\end{array} \right ] \left[ \begin{array}{c}
x_0 + \lambda v_x\\
y_0 + \lambda v_y\\
z_0 + \lambda v_z\\
1
\end{array} \right ] = \left[ \begin{array}{c}
(x_0 + \lambda v_x)f\\
(y_0 + \lambda v_y)f\\
z_0 + \lambda v_z
\end{array} \right ]
\end{equation*}
\begin{equation*}
x = \frac{x'}{z'} = \frac{x_0 + \lambda v_x}{z_0 + \lambda v_z}f, \ y = \frac{y'}{z'} = \frac{y_0 + \lambda v_y}{z_0 + \lambda v_z}f
\end{equation*}
As $|\lambda| \to \infty$,

\begin{equation*}
\begin{aligned}
\lim\limits_{\lambda\to\infty}x &= \lim\limits_{\lambda\to\infty}\frac{x_0 + \lambda v_x}{z_0 + \lambda v_z}f = \frac{v_x}{v_z}f\\
\lim\limits_{\lambda\to\infty}y &= \lim\limits_{\lambda\to\infty}\frac{y_0 + \lambda v_y}{z_0 + \lambda v_z}f = \frac{v_y}{v_z}f
\end{aligned}
\end{equation*}
Since the vanishing point under camera coordinate is $(10, 0, f)$, we have
\begin{equation*}
\left[ \begin{array}{c}
\frac{v_x}{v_z}f=10\\
\frac{v_y}{v_z}f=0
\end{array} \right ]
\end{equation*}
Let $v_z = 1$, we have the orientation vector
\begin{equation*}
v = \left[ \begin{array}{c}
10/f\\
0\\
1
\end{array} \right ]
\end{equation*}
with respect to the camera coordinate system.
%%---------------------------------------------------------------
%% Question 4
%%---------------------------------------------------------------
\section{Solution:}
\noindent Suppose $Z_{face} - Z_{nose} = \Delta Z$. When taken frontally, from the perspective equation:
\begin{equation*}
x_{nose} = f\frac{X_{nose}}{Z_{nose}},\ x_{face} = f\frac{X_{face}}{Z_{face}}
\end{equation*}
\noindent The ratio nose appears to be is
\begin{equation*}
\frac{x_{nose}}{x_{face}} = \frac{X_{nose}Z_{face}}{Z_{nose}X_{face}} = \frac{X_{nose}}{X_{face}} \left(1 + \frac{\Delta Z}{Z_{nose}}\right )
\end{equation*}
\noindent where $X_{face},\ X_{nose}$ and $\Delta Z$ keep the same. When taken from a small distance, the nose is closer to the camera and $\Delta Z / Z_{nose}$ can be rather large. Therefore the nose appears to be much larger than the rest of the face.\\\\
This cannot be reduced by using another focal length, because $f$ has no effect in the ratio. However this can be reduced by changing object distance, i.e. to take picture in a larger distance.
\vfill
\clearpage
%%---------------------------------------------------------------
%% Question 5
%%---------------------------------------------------------------
\section{Solution:}
Red and magenta appear almost the same to person with a blue receptor deficiency.
%%---------------------------------------------------------------
%% Question 6
%%---------------------------------------------------------------
\section{Solution:}
(a) CMY system\\
\begin{equation*}
{\left[ \begin{array}{c}
C\\
M\\
Y
\end{array}
\right]} = {\left[ \begin{array}{c}
1\\
1\\
1
\end{array}
\right]} -
{\left[ \begin{array}{c}
R\\
G\\
B
\end{array}
\right]} =
{\left[ \begin{array}{c}
1\\
1\\
1
\end{array}
\right]} - \frac{1}{255}
{\left[ \begin{array}{c}
200\\
50\\
100
\end{array}
\right]}
={\left[ \begin{array}{c}
0.216\\
0.804\\
0.608
\end{array}
\right]}
\end{equation*}

\noindent(b) YIQ system
\begin{equation*}
{\left[ \begin{array}{c}
Y\\
I\\
Q
\end{array}
\right]} = {\left[ \begin{array}{ccc}
0.299 & 0.587 & 0.114\\
0.596 & -0.275 & -0.321\\
0.212 & -0.532 & 0.311
\end{array}
\right]}
{\left[ \begin{array}{c}
R\\
G\\
B
\end{array}
\right]} =
\frac{1}{255}
{\left[ \begin{array}{ccc}
0.299 & 0.587 & 0.114\\
0.596 & -0.275 & -0.321\\
0.212 & -0.532 & 0.311
\end{array}
\right]} {\left[ \begin{array}{c}
200\\
50\\
100
\end{array}
\right]}
={\left[ \begin{array}{c}
0.394\\
0.288\\
0.184
\end{array}
\right]}
\end{equation*}

\noindent(c) HSI system
\begin{equation*}
\theta = cos^{-1}\left(\frac{[(R-G)+(R-B)]/2}{[(R-G)^2+(R-B)(G-B)]^{1/2}}\right)=cos^{-1}\frac{125}{\sqrt{17500}}=19.107
\end{equation*}
\begin{equation*}
H = 360 - \theta = 340.893
\end{equation*}
\begin{equation*}
S = 1-\frac{3}{R+G+B}[min(R, G, B)] = 1- \frac{3\times50}{350} = 0.571
\end{equation*}

\begin{equation*}
I = 1-\frac{R+G+B}{3} = 1-\frac{350}{3\times255} = 0.673
\end{equation*}
\begin{equation*}
{\left[ \begin{array}{c}
H\\
S\\
I
\end{array}
\right]} = {\left[ \begin{array}{c}
340.893\\
0.571\\
0.673
\end{array}
\right]}
\end{equation*}

\end{document}


    \filetitle{\{...\}}{Lag or lead}{modellang/laglead}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
VariableName{-lag}
VariableName{lead}
VariableName{+lead}
\end{verbatim}

\paragraph{Description}\label{description}

To create a lag or a lead of a variable, use a pair of curly brackets.

\paragraph{Example}\label{example}

\begin{verbatim}
!transition_equations
    x = rho*x{-1} + epsilon_x;
    pi = 1/2*pie{-1} + 1/2*pie{1} + gamma*y + epsilon_pi;
\end{verbatim}


%!TEX root = /home/renaud/Documents/EPL/tfe/latex/tfe.tex
% \section{Methodology}
%-----------------------DESCRIPTION OF THE METHOD---------------------------------%
\section{Description of the method}
In order to apply a clustering algorithm on a physical advection-diffusion problem, we have to define how the problem can be considered as a graph. For the next, we consider a two-dimensional problem in the coordinate system $(y,z)$. Let us partition the domain into $\nby \times \nbz$ grid cells, and denote $N_{cell} = \nby\nbz$ the total number of grid cells. Figure~\ref{fig:box_scheme} represents an example of such a domain decomposition of the rectangular domain $[0,\,L]\times[0,\,H]$ with $\nby = 15$ and $\nbz = 10$. For any time $T$, the corresponding directed graph is build as follows: each node represents a grid cell, and the weight of the edge between nodes $i$ and $j$ is the probability $m_{i,j}(T)$ that a particle ends up in grid cell $j$ after a time $T$ if it was initially in grid cell $i$. If $m_{i,j}(T) = 0$, one can equivalently consider that there is no edge between nodes $i$ and $j$. We restrict ourselves to stationary velocity field and diffusivity tensor, hence $m_{i,j}(T)$ depends only on the elapsed time $T$, not on the initial time. Hence, the initial time can indifferently be considered as being zero. The adjacency matrix $\b M(T)$ of the graph is build from the weights $m_{i,j}(T)$: $[\b M(T)]_{i,j} = m_{i,j}(T)$. It is thus precisely the transition probability matrix introduced in chapter~\ref{chap:clustering} which is needed to apply the stability clustering method. Since we consider passive tracer's particles in a domain with no-through boundary condition, we have that for any time $T$, $\b M(T)\b 1 = \b 1$, where $\b 1$ is the $N_{cell}$-dimensional unit column vector. The latter has a straightforward physical interpretation: every particle remains in the domain and particles are neither created nor destructed.

\begin{figure}[!htp]
	\centering
	\input{fig/clusters/box_scheme}
	\caption{Illustration of the decomposition of a rectangular domain into grid cells with $\nby = 15$ and $\nbz = 10$.}
	\label{fig:box_scheme}
\end{figure}

To estimate the probabilities $m_{i,j}(T)$, a Lagrangian simulation is run for a time $T$ with each grid cell containing initially $P_0$ uniformly distributed particles. Let $P_{i \rightarrow j}(T)$ denote the number of particles in grid cell $j$ at time $T$, which were initially in grid cell $i$. Using that notation, $m_{i,j}(T)$ is then numerically estimated as
\begin{equation}
	m_{i,j}(T) = \frac{P_{i \rightarrow j}(T)}{P_0},
\end{equation}
i.e. the number of particles having started in grid cell $i$ and ending up in grid cell $j$ after time $T$, divided by $P_0$. This is exactly the \textit{box counting} method introduced in section~\ref{boxcounting_kernel} for the computation of the concentration, but instead of dividing the tracer's mass in a grid cell by the volume of that grid cell, we divide it by $P_0$. This yields an adimensional quantity that can be interpreted as a transition probability. Note that the box counting method is more adapted than a density kernel estimation for this problem because the volume average over grid cells is precisely what we want.

\section{Dealing with the time scales} \label{sec:timescales}
An important feature of the stability method for detecting community structures is that it is \textit{dynamic}: community structures are revealed as a function of the Markov time $t_M$. For the problems that we consider, this Markov time is intrinsically linked to the physical time: for a given time $T$, suppose that the stability method is applied on the adjacency matrix $\b M(T)$. In the discrete framework, a particle jumps from one node to another at every integer Markov time, and in the continuous framework, the expected time between jumps is $\langle \langle \b q \rangle/\b q \rangle = 1$ (cfr. page \pageref{expectedtimebetweenjumps}). Hence, a Markov time step of $1$ corresponds to a physical time step of $T$.

From the above discussion, two possibilities arise for dealing with the time scales: either we compute the adjacency matrix at one unique time $T$ and then compute the stability on the desired range of Markov times, or we compute the adjacency matrix at different times and then compute the stability on each adjacency matrix but for the Markov time $t_M = 1$ only. The advantage of the first method is that we do not have to fix \textit{a priori} the time scales at which we compute clusterings: such times scales arise naturally as plateaux in the community curve, with a low corresponding variation of information. Hence the relevant time scales are deduced from the stability curve as being the ones at which robust clusterings arise. At the contrary, the second method imposes that we choose the time scales beforehand; doing so, we lose one of the most appealing features of the stability approach. Furthermore, the first method is computationally lest costly. Even for a relatively coarse partitioning of the domain, say of about $300$ grid cells, if we release $10\,000$ particles in each grid cell there is a total of $3\e{6}$ trajectories to simulate. For long $T$, the simulation time might become restrictive, especially in the case where one does not have access to supercomputers to run the code in parallel. The same situation leads to a $300$ nodes network, which is a relatively small network size that can easily be handled by the stability software. The first method has however one important drawback: the errors in the adjacency matrix are spread and even amplified across the Markov times. If those errors become too important, the community structures found at large Markov times might become irrelevant. In other words, simulating the transition probability matrix for a time $T$ and taking the $n$th power of that matrix is not necessarily equivalent to simulating the transition probability matrix for a time $nT$. The ideal methodology is thus probably to use the first method to detect the interesting time scales and compute the corresponding community structures, and then to check that we get similar community structures at the same time scales using the second method.

%-------------------------USE OF THE TOOLBOX----------------------------%
\section{Use of the stability software}
% \newcommand\localFontSize@mlpr{10}
We present here briefly how the \textit{PartitionStability} software is used to compute the partitions. Every concept appearing here has been presented in chapter~\ref{chap:clustering}. The \mtlb{stability} function is simply called as follows : \vspace{-.2cm}
\begin{center}
	\mtlb{[S,N,VI,C] = stability(M,Markov_T,'directed','plot','teleport',tau);}
\end{center} %style = Matlab-bw for black and white
Here, \mtlb{M} is the matrix $\b M(T)$ at the desired time $T$; \mtlb{Markov_T} is the vector containing every Markov times at which the optimal stability partition has to be computed (ideally, the sampling should be exponential); the \mtlb{'directed'} option specifies that we consider a directed graph; \mtlb{'plot'} asks the program to plot the stability, number of communities and variation of information as a function of the Markov time; and \mtlb{'teleport',tau} allows to specify the value of the teleportation probability $\tau$ to \mtlb{tau}, the default value being $0.15$. In most cases, we will choose \mtlb{tau} $= 0$. This choice is motivated by the fact that if our approximation of the transition probability matrix is close enough the the exact one, then if the diffusivities are everywhere strictly positive the graph is ergodic (notice that there can be no dangling node whatever the precision of our approximation). Further in this work, one example where the graph is not ergodic will be encountered. In that case, the value of \mtlb{tau} must be chosen strictly positive in order to ensure ergodicity. A small value is then preferred, in order to minimize the impact of random teleportations on the dynamics of the graph. We will typically choose \mtlb{tau} $=10^{-3}$ in such a case.

Unfortunately, the software does not handle discrete-time stability. Instead, it allows to choose which type of laplacian should be used to calculate the (continuous-time) stability. However, the question does not arise here since both laplacians are equivalent in our case. Indeed, the total outgoing weight is the same at every node and is precisely equal to the number of particles $P_0$ released in each grid cell. Hence, $k_i = P_0$ for every node $i$ and $\langle \b k \rangle = P_0$, so that $\bs \lambda_{combi}(\b k) = \b k/ \langle \b k \rangle = \b 1 = \bs \lambda_{norm}(\b k)$. We let thus the program run with the default normalized Laplacian, since it does not make any difference in our case.

The output arguments \mtlb{S}, \mtlb{N}, \mtlb{VI} and \mtlb{C} contain respectively the stability, the number of communities, the variation of information, and the optimal partition for each Markov time contained in \mtlb{Markov_T}. If the latter is of size $n$, then \mtlb{S}, \mtlb{N} and \mtlb{VI} are $n$-dimensional vectors and \mtlb{C} is a $N_{cell} \times n$ matrix. At the $j$th Markov time, communities are labeled by consecutive integers between $0$ and \mtlb{N(j)}$-1$ such that \mtlb{C(i,j)} $= k$ means that node $i$ belongs to community $k$ at Markov time \mtlb{Markov_T(j)}.


\documentclass[fleqn]{article}

%% Language and font encodings
%\usepackage[english]{babel}
%\usepackage[utf8x]{inputenc}
%\usepackage[T1]{fontenc}

%% Sets page size and margins
\usepackage[margin=1in]{geometry}

%% Useful packages
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}
%\usepackage{mathrsfs}
\usepackage{amsfonts,amssymb}
%\usepackage[colorlinks=true, allcolors=blue]{hyperref}
\usepackage{parskip}
\usepackage{ragged2e}

\newcommand{\mat}[1]{\mathbf{#1}}
\newcommand{\tr}{^\intercal}
\newcommand{\real}{\mathbb{R}}
\newcommand{\vect}[1]{\begin{pmatrix}#1\end{pmatrix}}
\newcommand{\nablax}{\nabla_{\!\mat x}}

\DeclareMathOperator{\Lagr}{\mathcal{L}}

%\setlength{\parindent}{0pt}

\title{Homework 2}
\author{Group L\\
Yang Li, Jimmy Jusuf, Zhirong Zhang, Tiangang Zhang}

\begin{document}
\maketitle

\section{Problem 1}

\section{Problem 2}

Let
\begin{equation}
\mat A = \begin{pmatrix}
  13 & 12 & -2 \\
  12 & 17 & 6 \\
  -2 & 6 & 12 \\
  \end{pmatrix} = \mat A\tr,\quad
\mat a = \begin{pmatrix} -22 \\ -14.5 \\ 13 \\ \end{pmatrix},\quad
b = 1.
\end{equation}
\begin{equation}
\textnormal{objective function to be minimized }f:\real^3\rightarrow\real,
\enskip f(\mat x)=\frac12 \mat x\tr \mat A \mat x + \mat a\tr x + b,
\end{equation}
\begin{equation}\label{eq3}
\textnormal{subject to } {-1}\leq x_i\leq 1, i=1,2,3
\enskip\textnormal{ or } \begin{pmatrix} -1\\-1\\-1 \end{pmatrix}
  \leq \mat x\leq \begin{pmatrix} 1\\1\\1 \end{pmatrix},
  \mat x\in\real^3.
\end{equation}

Write inequality constraints \eqref{eq3} as
$h_1(\mat x)=-\mat x+\begin{pmatrix} -1\\-1\\-1 \end{pmatrix}\leq 0$
and
$h_2(\mat x)=\mat x+\begin{pmatrix} -1\\-1\\-1 \end{pmatrix}\leq 0$,
$h_1, h_2:\real^3\rightarrow\real$.
Note that
$\nablax f(\mat x)=\mat A\mat x + \mat a=
  \vect{21\\14.5\\-11} + \vect{-22\\14.5\\13} = \vect{-1\\0\\2}$,
$\nablax h_1(\mat x)=\vect{-1\\-1\\-1}$,
$\nablax h_2(\mat x)=\vect{1\\1\\1}$.

Let $
\Lambda_1=\begin{pmatrix}\lambda_1\\ \lambda_2\\ \lambda_3\end{pmatrix},\enskip
\Lambda_2=\begin{pmatrix}\lambda_4\\ \lambda_5\\ \lambda_6\end{pmatrix}$ be the Lagrange multipliers
for constraints $h_1(\mat x)$ and $h_2(\mat x)$ respectively.

Define Lagrangian
\begin{gather}
\Lagr(\mat x, \mat\Lambda)=f(\mat x)+ \Lambda_1\tr\cdot h_1(\mat x) +
  \Lambda_2\tr\cdot h_2(\mat x)
\end{gather}

%    Given $\mat x^* = \vect{1 \\ 0.5 \\ -1}$
%and $\lambda =
%\begin{pmatrix} \lambda_1 - \lambda_2 \\ \lambda_3 - \lambda_4 \\ \lambda_5 - \lambda_6 \\ \end{pmatrix}$

For $\mat x^*$ to be a local minimum, we want to show
the following KKT Necessary Conditions are met.
\begin{gather}
\nablax \Lagr(\mat x^*, \Lambda^*) =
  \nablax f(\mat x) + (\Lambda_1^*)\tr\cdot\nablax h_1(\mat x)
  + (\Lambda_2^*)\tr\cdot\nablax h_2(\mat x) = 0 \label{eq5}\\
%  &= (\mat A\mat x + \mat a) + (\Lambda_1^*)\tr\cdot\vect{-1\\-1\\-1}
%    + (\Lambda_2^*)\tr\cdot\vect{1\\1\\1}
(\Lambda_1^*)\tr \cdot h_1(\mat x^*) = 0 \label{eq6}\\
(\Lambda_2^*)\tr \cdot h_2(\mat x^*) = 0 \label{eq7}\\
\Lambda_1^* \geq 0 \label{eq8}\\
\Lambda_2^* \geq 0 \label{eq9}
% d^T \nabla_{xx}^2 L(x^*, \lambda^*) d > 0
\end{gather}

Plugging in $\mat x^* = \vect{1 \\ 0.5 \\ -1}$,\enskip
\begin{gather}
\nabla_{\!\mat x} \Lagr(\mat x^*, \Lambda^*) = \vect{-1\\0\\2}
  +\vect{-\lambda_1\\-\lambda_2\\-\lambda_3}
  +\vect{\lambda_4\\ \lambda_5\\ \lambda_6}=0 \tag{\ref{eq5}a}\label{eq5a}\\
\vect{\lambda_1&\lambda_2&\lambda_3} \vect{-2\\-1.5\\0} = 0
  \implies \lambda_1=\lambda_2=0 \tag{\ref{eq6}a}\label{eq6a}\\
\vect{\lambda_4&\lambda_5&\lambda_6} \vect{0\\-0.5\\-2} = 0
  \implies \lambda_5=\lambda_6=0 \tag{\ref{eq7}a}\label{eq7a}
%\Lambda_2^* \cdot h_2(\mat x^*) = 0 \\
%\Lambda_1^* \geq 0 \\
%\Lambda_2^* \geq 0 \\
\end{gather}
The \emph{complementary slackness condition} forces
$\lambda_1,\lambda_2,\lambda_5,\lambda_6$ to be zero as
$h_1(x_1), h_1(x_2), h_2(x_2), h_2(x_3) \leq 0$ in
equations \eqref{eq6a} and \eqref{eq7a}.
Back-substitution into \eqref{eq5a} yields $\lambda_4=1$ and
$\lambda_3=-2$. All the $\lambda$'s satisfy conditions \eqref{eq8}
and \eqref{eq9}.

We have shown the KKT Necessary Conditions are met and there is
a unique vector of Lagrange multipliers for all the constraints.

%From the constraints and the given solution, we have
%\begin{gather}
%\lambda_2 = \lambda_3 = \lambda_4 = \lambda_5 = 0 \\
%\lambda_1 = 1 \\
%\lambda_6 = 2
%\end{gather}

%As all the leading principal minor of \(A\) is positive, the second
%order condition is met.

\end{document}


The evaluation is divided into three parts: testing, performance, and expressive power. In all parts, Souffle\cite{SouffleHome} is the implementation evaluated against.

\subsection{Testing}
The Souffle pretty-printer (SPP) is used to output a Datalog program that may be executed by Souffle. The process of comparing Souffle output to that of the internal interpreter has been automated and a range of tests written. If the tests agree, then the SPP is said to be correct for the given test program. Assuming that the Souffle result is correct, the internal interpreter too is concluded to be correct for the given test program. The test cases are selected to cover mutual recursion, negation, meta-predicates, and the other various language extensions.

\subsection{Performance}
Souffle implements semi-naive evaluation\cite{Green:2013:DRQ:2688167.2688168} which is essentially the same as naive evaluation except that it utilizes the following key-insight. An instantiation of the terms of a rule may derive new tuple(s) if and only if at least one tuple that was derived in the previous iteration is used in the instantiation. Thus the number of tuples to consider can be greatly decreased.

The performance is evaluated against two examples shown in figures \ref{figure:nat} and \ref{figure:ancestor}.
\vspace*{-15pt}
\begin{figure}[!ht]
\begin{minipage}[b]{.5\textwidth}
\caption{Upper bounded Natural Numbers example.}
\begin{minted}{text}
Nat(0).
Nat(y) :- Nat(x), BIND(y, x + 1), y <= N.
\end{minted}
\label{figure:nat}
\end{minipage}
\begin{minipage}[b]{.5\textwidth}
\caption{Ancestor relation example.}
\begin{minted}{text}
r1: Ancestor(p, c) :- Parent(p, c).
r2: Ancestor(a, c) :- Parent(p, c), Ancestor(a, p).
\end{minted}
\label{figure:ancestor}
\end{minipage}
\end{figure}
\noindent
\vspace*{-15pt}
\subsubsection{Theoretical Prediction}

\paragraph{NAT Example}\NL
For the internal naive algorithm, at step $k$ in the iteration, the $Nat$-relation contains $k$ tuples. The internal implementation uses a tree-set to store the tuples and so each step takes $\mathcal{O}(k \cdot log(k))$ time (there is no join (Appendix A) between Nat and BIND). There are a total of $N$ steps in the algorithm, thus we get the following upper-bound for the worst-case running time:
\begin{align*}
\sum_{k = 1}^{N}k \cdot log (k) \leq N^2 log(N) = \mathcal{O}(N^2 log(N))
\end{align*}
For semi-naive evaluation, each iteration gives a single new element to consider. The corresponding time complexity thus reduces to the order of:
\begin{align*}
	\sum_{k = 1}^{N} log (k) \leq N log(N) = \mathcal{O}(N log(N))
\end{align*}

\paragraph{Ancestor Example}\NL
Assume the initial parent relation:
\begin{align*}
Parent(P_i, P_{i + 1}), i = 1\ldots N - 1
\end{align*}
\noindent
Then at the $k:th$ iteration of rule $r_2$, the \textit{Ancestor} relation contains $\sum_{l = 1}^{k} (N - l)$ elements. The parent relation is constant with $N$ elements. The dominating (non-indexed) \textit{join}-operation thus has accumulated time complexity:
\begin{align*}
\sum_{k = 1}^{N - 1} \Big (N \cdot \sum_{l = 1}^{k} (N - l)\Big) = \mathcal{O}(N^4)
\end{align*}
\noindent
Similarly, the semi-naive algorithm has expected time-complexity:
\begin{align*}
\sum_{k = 1}^{N - 1} N \cdot (N - k) = \mathcal{O}(N^3)
\end{align*}

\subsubsection{Experimental Results}

\paragraph{NAT Example}\NL
The NAT example was measured for inputs in range 100 to 10000 for the internal evaluation, and in range 4096 to 536870912 for Souffle. The theoretical results was confirmed as $O(N^2 log(N))$ for the internal evaluation. For Souffle, the experimental results show $\mathcal{O}(N)$ behavior. The $log(N)$ does not show in Souffle, most likely due to a more sophisticated relation representation\cite{Scholz:2016:FLP:2892208.2892226}.

%; Souffle better relation data structure as opposed to an ordered set.
%\begin{figure*}%[!ht]
%	\hspace*{-25pt}
%	\begin{minipage}[b]{.5\textwidth}
%		\includegraphics[scale=0.33]{img/internalloglog.png}
%	\end{minipage}%
%	\hspace*{-40pt}
%	\begin{minipage}[b]{.5\textwidth}
%		\includegraphics[scale=0.35]{img/souffleloglog.png}
%	\end{minipage}
%	\label{figure:natExperimental}
%	\caption{Nat Example, \textbf{Left: } Internal (Naive), \textbf{Right: } Souffle (Optimized Semi-Naive)}
%\end{figure*}
\paragraph{Ancestor Example}\NL
The ancestor example was measured for inputs in range 5 to 260 for the internal evaluation, and in range 50 to 16384 for Souffle. The theoretical results was again confirmed as $O(N^4)$ for the internal evaluation. Souffle performed better than expected at about $O(N^{2.33})$. Again, this is most likely due to better \textit{join}-performance which can be achieved by storing separate index data structures, thus removing the need to explicitly form the Cartesian product\cite{Scholz:2016:FLP:2892208.2892226}.
%\begin{figure*}%[!ht]
%	\hspace*{-70pt}
%	\begin{minipage}[b]{.5\textwidth}
%		\includegraphics[scale=0.345]{img/ancestorInternal.png}
%	\end{minipage}%
%\hspace*{-70pt}
%	\begin{minipage}[b]{.5\textwidth}
%		\includegraphics[scale=0.35]{img/ancestorSouffle.png}
%	\end{minipage}
%	\label{figure:ancestorExperimental}
%	\caption{Ancestor Example, \textbf{Left: } Internal (Naive), \textbf{Right: } Souffle (Optimized Semi-Naive)}
%\end{figure*}
\vspace*{-5pt}
\subsection{Expressive Power}
Souffle supports all the implemented language extensions except meta-predicates and type-inference\cite{SouffleHome}. The current set of meta-predicates offered by \datalogM does not extend the expressive power in any meaningful way, but rather permits more compact and (admittedly subjectively) more beautiful descriptions of Datalog programs.

Souffle contains a number of language extensions that are not currently supported by \datalogM. For example, aggregate functions (such as $COUNT$, $MIN$, $MAX$), union types (i.e. stating that a term has type $A$ OR type $B$), and inbuilt functions (e.g. string operations), and more\cite{SouffleHome}.


\chapter{\proj Edge Detection}
\label{ch_edge}
\index{edge detection}
% \chapterhead{Programs}
\markright{Edge detection}

\section{Introduction}
% An edge in an image  corresponds to a discontinuity in the intensity
% surface of the underlying scene.
An edge is defined as a local variation of image intensity. Edges can
be detected by the computation of a local derivative operator.

\begin{figure}[htb]
\centerline{
\vbox{
\psfig{figure=fig_inflex.ps,bbllx=3.5cm,bblly=4cm,bburx=19.5cm,bbury=12cm,width=12cm,height=6cm,clip=}
}}
\caption{First and second derivative of  $G_\sigma * f$. (a) Original signal,
(b) signal convolved by a Gaussian, (c) first derivative of (b), (d) second
derivative of (b).}
\label{fig_inflex}
\end{figure}

Figure~\ref{fig_inflex} shows how the inflection point of a signal can be
found from its first and second derivative.
Two methods
can be used for generating first order derivative edge gradients.

\section{First Order Derivative Edge Detection}

\subsection{Gradient}
The gradient of an image $f$ at location $(x,y)$, along the line
normal to the edge slope, is the vector \cite{ima:pratt91,ima:gonzalez93,ima:jain89}:
\begin{eqnarray}
\bigtriangledown f = &
\left[ \begin{array}{c}
f_x  \\
f_y
\end{array} \right] & =
\left[ \begin{array}{c}
 \partial f \over \partial x  \\
 \partial f \over \partial y
\end{array} \right]
\end{eqnarray}

% The gradient of an image $f$ along $r$ in a direction $\alpha$ is
%\begin{eqnarray}
%{\partial f \over \partial r} & = &{\partial f \over \partial x} \cos{\alpha} +
%{\partial f \over \partial y} \sin{\alpha} \\
%  & = & f_x \cos{\alpha} + f_y \sin{\alpha}
%\end{eqnarray}
%The maximum value of ${\partial f \over \partial r}$ is obtained when $

The spatial gradient amplitude is given by:
\begin{eqnarray}
G(x,y) = \sqrt{f_x^2 + f_y^2}
\end{eqnarray}
and the gradient direction with respect to the row axis is
\begin{eqnarray}
\Theta(x,y) = \arctan{\frac{f_y}{f_x}}
\end{eqnarray}

The first oder derivative edge detection can be carried out either by using
 two orthogonal directions in an image or by using a set of
directional derivatives.

\subsection{Gradient mask operators}
% details dans le Jain, p348 et dans le pratt p504

Gradient estimates can be obtained by using gradient operators of the form:
\begin{eqnarray}
f_x & = f & \otimes  \ \ H_x \nonumber \\
f_y & = f & \otimes \ \ H_y
\end{eqnarray}
where $\otimes$ denotes the convolution product, and $H_x$ and $H_y$ are $3
\times 3$
row and column operators, called gradient masks. Table \ref{tab_grad1} shows
the main gradient masks proposed in the literature. Pixel difference is
the simplest one, which consists just of making the difference of   pixels
along rows and columns of the image:
\begin{eqnarray}
f_x(x_m, y_n) & = & f(x_m , y_n) - f(x_m-1, y_n) \nonumber \\
f_y(x_m, y_n) & = & f(x_m , y_n )- f(x_m, y_n-1)
\end{eqnarray}

The Roberts gradient masks \cite{edge:roberts65} are more sensitive
to diagonal edges. Using these masks,
the orientation must be calculated by
\begin{eqnarray}
\Theta(x_m, y_n) = {\pi \over 4} + \arctan \left[
{f_y(x_m, y_n) \over f(x_m , y_n)} \right]
\end{eqnarray}

Prewitt \cite{edge:prewitt70}, Sobel, and Frei-Chen \cite{edge:frei77} produce
better results than the pixel difference, separated pixel difference and
Roberts
operator, because the mask is larger, and provides averaging of small luminance
fluctuations. The Prewitt operator is more sensitive to
horizontal and vertical edges  than diagonal edges, and the
reverse is true for the Sobel operator. The Frei-Chen  edge detector has the
same sensitivity for diagonal, vertical, and horizontal edges.

% table
\begin{table}[htb]
\begin{center}
\begin{tabular}{lllc}
Operator &    $H_x $     & $H_y$   & Scale factor\\
         &              &        \\
Pixel difference&
\( \left[ \begin{array}{ccc}
0 & 0 & 0\\
0 & 1 & -1\\
0 & 0 & 0
\end{array}\right]  \) &
\( \left[ \begin{array}{ccc}
0 & -1 & 0\\
0 & 1 & 0\\
0 & 0 & 0
\end{array}\right]  \) & 1\\

\begin{tabular}{l}
Separated \\
pixel difference \\
\end{tabular} &

\( \left[ \begin{array}{ccc}
0 & 0 & 0\\
1 & 0 & -1\\
0 & 0 & 0
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
0 & -1 & 0\\
0 & 0 & 0\\
0 & 1 & 0
\end{array}\right]  \) & 1\\
Roberts&
\( \left[ \begin{array}{ccc}
0 & 0 & -1\\
0 & 1 & 0\\
0 & 0 & 0
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 0
\end{array}\right]  \) & 1 \\
Prewitt&
\( \left[ \begin{array}{ccc}
1 & 0 & -1\\
1 & 0 & -1\\
1 & 0 & -1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-1 & -1 & -1\\
0 & 0 & 0\\
1 & 1 & 1
\end{array}\right]  \) & 1\\
Sobel&
 \( \left[ \begin{array}{ccc}
1 & 0 & -1\\
2 & 0 & -2\\
1 & 0 & -1
\end{array}\right]  \)&
 \( \left[ \begin{array}{ccc}
-1 & -2 & -1\\
0 & 0 & 0\\
1 & 2 & 1
\end{array}\right]  \) & $ \frac{1}{4} $ \\
Fei-Chen&
 \( \left[ \begin{array}{ccc}
1 & 0 & -1\\
\sqrt{2} & 0 & \sqrt{2}\\
1 & 0 & -1
\end{array}\right]  \)&
 \( \left[ \begin{array}{ccc}
-1 & -\sqrt{2} & -1\\
0 & 0 & 0\\
1 & \sqrt{2} & 1
\end{array}\right]  \) & $\frac{1}{2+\sqrt{2}}$ \\
 \end{tabular}\par
\caption{Gradient edge detector masks.}
\label{tab_grad1}
  \end{center}
\end{table}
\vspace{0.3cm}

\subsection{Compass operators}
Compass operators measure gradients in a selected number of directions.
The directions are $\Theta_k = k {\pi \over 4}$,  $k = 0,...,7$. The edge
template gradient is defined as:
\begin{eqnarray}
G(x_m, y_n) = \max_{k=0}^7 { \mid f(x_m, y_n) \otimes H_k(x_m, y_n) \mid }
\end{eqnarray}

Table~\ref{tab_grad2} shows the principal template gradient operators.

\begin{table}[htb]
\begin{center}

\begin{tabular}{lllll}
\begin{tabular}{c}
 Gradient \\
 direction \\
 \end{tabular} &
\begin{tabular}{c}
 Prewitt \\
 compass   \\
 gradient \\
\end{tabular} &
Kirsch&
\begin{tabular}{c}
Robinson \\
3-level \\
\end{tabular}&
\begin{tabular}{c}
Robinson \\
5-level\\
\end{tabular} \\
East&
\( \left[ \begin{array}{ccc}
1 & 1 & -1\\
1 & -2 & -1\\
1 & 1 & -1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
5 & -3 & -3\\
5 & 0 & -3\\
5 & -3 & -3
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
1 & 0 & -1\\
1 & 0 & -1\\
1 & 0 & -1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
1 & 0 & -1\\
2 & 0 & -2\\
1 & 0 & -1
\end{array}\right]  \)\\
 Northeast&
\( \left[ \begin{array}{ccc}
1 & -1 & -1\\
1 & -2 & -1\\
1 & 1 & 1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-3 & -3 & -3\\
5 & 0 & -3\\
5 & 5 & -3
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
0 & -1 & -1\\
1 & 0 & -1\\
1 & 1 & 0
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
0 & -1 & -2\\
1 & 0 & -1\\
2 & 1 & 0
\end{array}\right]  \)\\
 North&
\( \left[ \begin{array}{ccc}
-1 & -1 & -1\\
1 & -2 & 1\\
1 & 1 & 1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-3 & -3 & -3\\
-3 & 0 & -3\\
5 & 5 & 5
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-1 & -1 & -1\\
0 & 0 & 0\\
1 & 1 & 1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-1 & -2 & -1\\
0 & 0 & 0\\
1 & 2 & 1
\end{array}\right]  \)\\
 Northwest&
\( \left[ \begin{array}{ccc}
-1 & -1 & 1\\
-1 & -2 & 1\\
1 & 1 & 1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-3 & -3 & -3\\
-3 & 0 & 5\\
-3 & 5 & 5
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-1 & -1 & 0\\
-1 & 0 & 1\\
0 & 1 & 1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-2 & -1 & 0\\
-1 & 0 & 1\\
0 & 1 & 2
\end{array}\right]  \)\\
 West&
\( \left[ \begin{array}{ccc}
-1 & 1 & 1\\
-1 & -2 & 1\\
-1 & 1 & 1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-3 & -3 & 5\\
-3 & 0 & 5\\
-3 & -3 & 5
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-1 & 0 & 1\\
-1 & 0 & 1\\
-1 & 0 & 1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-1 & 0 & 1\\
-2 & 0 & 2\\
-1 & 0 & 1
\end{array}\right]  \)\\
 Southwest&
\( \left[ \begin{array}{ccc}
1 & 1 & 1\\
-1 & -2 & 1\\
-1 & -1 & 1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
-3 & 5 & 5\\
-3 & 0 & 5\\
-3 & -3 & -3
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
0 & 1 & 1\\
-1 & 0 & 1\\
-1 & -1 & 0
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
0 & 1 & 2\\
-1 & 0 & 1\\
-2 & -1 & 0
\end{array}\right]  \)\\
 South&
\( \left[ \begin{array}{ccc}
1 & 1 & 1\\
1 & -2 & 1\\
-1 & -1 & -1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
5 & 5 & 5\\
-3 & 0 & -3\\
-3 & -3 & -3
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
1 & 1 & 1\\
0 & 0 & 0\\
-1 & -1 & -1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
1 & 2 & 1\\
0 & 0 & 0\\
-1 & -2 & -1
\end{array}\right]  \)\\
 Southeast&
\( \left[ \begin{array}{ccc}
1 & 1 & 1\\
1 & -2 & -1\\
1 & -1 & -1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
5 & 5 & -3\\
5 & 0 & -3\\
-3 & -3 & -3
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
1 & 1 & 0\\
1 & 0 & -1\\
0 & -1 & -1
\end{array}\right]  \)&
\( \left[ \begin{array}{ccc}
2 & 1 & 0\\
1 & 0 & -1\\
0 & -1 & -2
\end{array}\right]  \)\\
 \begin{tabular}{c}
 Scale  \\
 factor    \\
 \end{tabular} &
 \begin{tabular}{c}
\( \frac{1}{5} \) \\
 \end{tabular}&
 \begin{tabular}{c}
\( \frac{1}{15} \) \\
 \end{tabular}&
 \begin{tabular}{c}
 \( \frac{1}{3} \) \\
  \end{tabular} &
  \begin{tabular}{c}
\( \frac{1}{4} \) \\
\end{tabular} \\
 \end{tabular}
\caption{Template gradients.}
\label{tab_grad2}
\end{center}
\end{table}

\clearpage

\subsection{Derivative of Gaussian}
The previous methods are relatively sensitive to the noise.  A solution
could be to extend the window size of the gradient mask operators.
Another approach is to use  the derivative of the
convolution of the image by a Gaussian.
The derivative of a Gaussian (DroG) operator is
\begin{eqnarray}
\bigtriangledown (g \otimes f) & =  & {\partial  (g \otimes f) \over \partial x} +
 {\partial  (g \otimes f) \over \partial y} \nonumber \\
  & =  & f_x + f_y
\end{eqnarray}
with $g = \exp{- {x^2+y^2 \over {2\sigma^2}}}$. Partial derivatives of the
Gaussian function are
\begin{eqnarray}
g_x(x,y) & = {\partial  g  \over \partial x} = & - {x \over \sigma^2} \exp{- {x^2+y^2 \over {2\sigma^2}}} \nonumber \\
g_y(x,y) & = {\partial  g  \over \partial y} = & - {y \over \sigma^2} \exp{- {x^2+y^2 \over {2\sigma^2}}}
\end{eqnarray}
The filters are separable so we have
\begin{eqnarray}
g_x(x,y) & = & g_x(x) * g(y) \nonumber \\
g_y(x,y) & = & g_y(y) * g(x)
\end{eqnarray}
Then
\begin{eqnarray}
f_x & = & g_x (x) \otimes g(y) \otimes f \nonumber \\
f_y & = & g_y (y) \otimes g(x) \otimes f
\end{eqnarray}

\subsection{Thinning the contour}
From the gradient map, we may want to consider only pixels which belong to the
contour. This can be done by looking for each pixel in the direction of
gradient. For each point P0 in the gradient map, we determine the two
adjacent pixels P1,P2
in the direction orthogonal to the gradient. If P0 is not a maximum in
this direction (i.e.\ P0 $<$ P1, or P0 $<$ P2), then we threshold P0 to
zero.

\section{Second Order Derivative Edge Detection}
Second derivative operators allow us to accentuate the
edges. The most frequently
used operator is the Laplacian one, defined by
\begin{eqnarray}
\bigtriangledown^2 f = {\partial^2  f  \over \partial x^2}
                     + {\partial^2  f  \over \partial y^2}
\end{eqnarray}

Table~\ref{tab_laplacian} gives three discrete approximation of this operators.

\begin{table}[htb]
\begin{center}
\begin{tabular}{ccc}
 Laplacian 1 &   Laplacian 2 &   Laplacian 3\\
             &           &    \\
\( {1 \over 4} \left[ \begin{array}{ccc}
0 & -1 & 0\\
-1 & 4 & -1\\
0 & -1 & 0
\end{array}\right]  \) &
\( {1 \over 8} \left[ \begin{array}{ccc}
-1 & -1 & -1\\
-1 & 8 & -1\\
-1 & -1 & -1
\end{array}\right]  \) &
\( {1 \over 8} \left[ \begin{array}{ccc}
-1 & -2 & -1\\
-2 & 4 & -2\\
-1 & -2 & -1
\end{array}\right]  \) \\
\end{tabular}
\caption{Laplacian operators.}
\label{tab_laplacian}
\end{center}
\end{table}

Marr and Hildreth \cite{edge:marr80} have proposed the Laplacian of Gaussian (LoG) edge detector operator.
It is defined as
\begin{eqnarray}
L(x,y) = {1 \over{\pi s^4}}
	  \left[ 1 - {{x^2+y^2} \over {2s^2}} \right]
\exp \left( - {{x^2+y^2}\over {2s^2}} \right)
\end{eqnarray}
where $\sigma$ controls the width of the Gaussian kernel.

Zero-crossings of a given image $f$ convolved with $L$
give its edge locations.

A simple algorithm for zero-crossings is:
\begin{enumerate}
\item For all pixels i,j do
\item ZeroCross(i,j) = 0
\item P0 = G(i,j); P1 =  G(i,j-1); P2 =  G(i-1,j); P3 =  G(i-1,j-1)
\item If (P0*P1 $<$  0) or  (P0*P2 $<$  0) or (P0*P3 $<$ 0) then ZeroCross(i,j) = 1
\end{enumerate}

\section{Edge Detection Program: im\_edge}
\index{im\_edge}
The program {\em im\_edge} detects the edges in an image by the
methods previously described.
{\bf
\begin{center}
     USAGE: im\_edge option file\_name\_in file\_name\_out
\end{center}}
where options are:
\begin{itemize}
\baselineskip=0.4truecm
\item {\bf [-M edge\_detection\_method}
{\small
\begin{enumerate}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item Derivative of a Gaussian (DroG)
\item Pixel difference
\item Separated pixel difference
\item Sobel
\item Prewitt
\item Roberts
\item Frei Chen
\item Laplacian: filter 1
\item Laplacian: filter 2
\item Laplacian: filter 3
\item Marr and Hildreth: Laplacian of Gaussian (LoG)
\item Prewitt compass gradient
\item Kirsch
\item Robinson 3-level
\item Robinson 5-level
\end{enumerate}}
Default is Sobel method
\item {\bf [-k]} \\
Select local maxima in gradient direction for first derivative methods,
and zero-crossings for second derivative methods. \\
Default is not to do these operations.
\item {\bf [-t ThresholdValue]} \\
Threshold all values in the edge map lower than {\em ThresholdValue}.
\item {\bf [-S Sigma]} \\
Scale parameter. Only used for DroG and LoG methods. \\
Default value is ${1 \over \sqrt{3}}$.
\end{itemize}

\clearpage
\newpage

\section{Wavelets and Edge Detection}
The LoG operator, also called the Mexican hat, is a well-known
wavelet function
(even if wavelets did not exist when the LoG operator was proposed!).
Furthermore, a scale parameter ($\sigma$) in this method leads to
a multiscale approach when $\sigma$ is varying. The advantage of using
the wavelet framework is the existence of very fast WT, such as the \`a trous
algorithm, which furnishes us with a  way to get the edges directly at all
dyadic scales, even for large scales.

\subsection{Multiscale first derivative}

Generalizing the concept of multiscale edge detection,
Mallat \cite{edge:mallat92a,edge:mallat92b,ima:mallat98} showed that
the DroG operators can be easily associated with a wavelet
transform.
If $\phi(x)$ is a smoothing function (i.e. $\int \phi(x) dx = 1$),
converges to zero at infinity, and is differentiable, we denote
\begin{eqnarray}
\psi^{1}(x,y) = {{d \phi(x,y)} \over {dx}} \quad \mbox{ and } \quad
\psi^{2}(x,y) = {{d \phi(x,y)} \over {dy}} \quad \mbox{     } \quad
 \end{eqnarray}
By definition, $\psi^x$ and $\psi^y$ are wavelets
(their integral is equal to zero). The local extrema of the wavelet coefficients
using $\psi^{x_1},\psi^{y_1}$ correspond to the inflection points of $f*\phi_s$
(with $\phi_s = {1 \over s} \phi({x\over s})$).

Using the directional \`a trous algorithm, sometimes also called dyadic wavelet
transform, we have at each scale $j$ and at pixel location $(x,y)$ two wavelet
coefficients $w_{j,x}, w_{j,y}$. The modulus of the gradient is then defined by
\begin{eqnarray}
G_j(x,y) = \sqrt{ w_{j,x}^2 + w_{j,y}^2}
\end{eqnarray}
and the directional angle $\theta_j$ is
\begin{eqnarray}
\theta_j(x,y) = \left\{ \begin{array}{cc}
 \arctan({w_{j,y} \over w_{j,x}})  & \mbox{ if } w_{j,x} \ge 0 \\
 \pi - \arctan({w_{j,y} \over w_{j,x}})  & \mbox{ if } w_{j,x} < 0
\end{array}\right.
\end{eqnarray}

Multiscale edge points, also called modulus maxima, are points where
the modulus is locally maximum with respect to its neighbors along the
direction $\theta_j$.

A similar approach can be developed for the
 LoG operator \cite{edge:mallat92a}.

\subsection{Image reconstruction from its multiscale edges}

\begin{figure}[htb]
\centerline{
\vbox{
\psfig{figure=fig_project.ps,bbllx=4cm,bblly=16cm,bburx=20.5cm,bbury=26cm,width=12cm,height=8cm,clip=}
\caption{Approximation of the wavelet transform of $f$.}
}}
\label{fig_project}
\end{figure}

An image can be reconstructed (approximately) from its multiscale edges
\cite{ima:mallat98} using an iterative algorithm. It does not
converge exactly towards the image, but in practice the error is very
small. The algorithm consists of searching for an image $h$
such that its wavelet transform has the same modulus maxima (i.e.\
same number of modulus maxima, same positions, and same amplitudes) as
those of the original image $f$.
Denoting as $w_j(X)$ the wavelet coefficients of an image $X$ at a scale $j$,
and $w^m_j(X)$ its modulus maxima,
we require that $w^m_j(h) = w^m_j(f)$.
The algorithm is the following:
\begin{enumerate}
\itemsep=0.1truecm
\item Set $w^m_j(h)$  to zero.
\item Loop:
\item Calculate the difference $w^m_j(f) - w^m_j(h)$, and
interpolate between the maxima: we get $w_j^d$.
\item Update $w_j(h)$: $w_j(h) = w_j(h) + w_j^d$.
\item Assign in $w_j(h)$ the correct values at maxima positions.
\item Reconstruct $h$  from  $w_j(h)$
\item Do a wavelet transform of $h$: $w_j(h)$ is then updated.
\item Goto 2
\item Assign in $w_j(h)$ the correct values at maxima positions.
\item Reconstruct $h$  from  $w_j(h)$.
\end{enumerate}
The algorithm converges quickly to the solution after a few iterations.
The interpolation \cite{edge:mallat92a} can be viewed
as a projection on an affine space
$\Gamma$, and the inverse and forward wavelet transform as a projection
on the space V of all possible wavelet transform signals. The two
projections are visualized in Figure~\ref{fig_project} ($Wh$ and $Wf$
are respectively the wavelet transform of $h$ and $f$).

\section{Multiscale Edge Detection Program}
\subsection{First derivative: mr\_edge}
\index{mr\_edge}
The program {\em mr\_edge} detects the edges in an image by the
methods previously described. The wavelet transform used is
the Mallat dyadic wavelet transform. The output file is a multiresolution
file (``.mr''). For each scale, two bands are stored, one
the maxima map and the second the gradient angle map.

{\bf
\begin{center}
     USAGE: mr\_edge option file\_name\_in file\_name\_out
\end{center}}
where options are:
\begin{itemize}
\item {\bf [-n number\_of\_scales]} \\
Number of scales used in the multiresolution transform.
Default is 4.
\end{itemize}
\subsubsection*{Examples:}
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item mr\_edge image.d med.mr \\
Build the multiscale edge detection file.
\item mr\_extract -B -s 1 med.mr mod1 \\
Create an image which contains the detected edge map (maxima)
of the first scale.
\item mr\_extract -B -s 2 med.mr ang1 \\
Create an image which contains the  gradient angle map
of the previous edge map.
\item mr\_extract -B -s 5 med.mr mod3 \\
\item mr\_extract -B -s 6 med.mr ang3 \\
Ditto for scale 3.
\end{itemize}

\subsection{Second derivative: mr\_at\_edge}
\index{mr\_at\_edge}
The program {\em mr\_at\_edge} detects the edges in an image by the
methods previously described. The wavelet transform used is
the \`a trous algorithm. Only zero crossings of each scale are
kept (i.e.\ not thresholded).
The output file is a multiresolution
file (``.mr'').
{\bf
\begin{center}
     USAGE: mr\_at\_edge option file\_name\_in file\_name\_out
\end{center}}
where options are:
\begin{itemize}
\item {\bf [-n number\_of\_scales]} \\
Number of scales used in the multiresolution transform.
Default is 4.
\end{itemize}
\subsubsection*{Examples:}
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item mr\_at\_edge image.d med.mr \\
Build the multiscale edge detection file.
\item mr\_extract -s 1 med.mr zero1 \\
Create an image which contains the detected edge map of the first scale.
\item mr\_extract -s 2 med.mr zero2 \\
Ditto for scale 2.
\end{itemize}

\subsection{Image reconstruction: mr\_rec\_edge}
\index{mr\_rec\_edge}
The program {\em mr\_rec\_edge} reconstructs an image from its multiscale
edges.
The multiscale edge file must have been obtained from the {\em mr\_edge}
program.
{\bf
\begin{center}
     USAGE: mr\_rec\_edge option file\_name\_in file\_name\_out
\end{center}}
where options are:
\begin{itemize}
\item {\bf [-i number\_of\_iterations]} \\
Number of iterations.
Default is 10.
\end{itemize}

\newpage

\section{Contrast Enhancement}
\label{sec_contrast}

\subsection{Introduction}
Because some features are hardly detectable by eyes in a image, we
often transform it before visualization. Histogram equalization is
certainly one the most well known method for contrast enhancement.
Such an approach is general useful for images
with a poor  intensity distribution. As edges play a fundamental
role in image understanding, a way to enhance the contrast is to
enhance the edges. For example, we can add to the original image its Laplacian
($I^{'}= I + \gamma \Delta I$, where $\gamma$ is a parameter). Only
features at the finest scale are enhanced (linearly). For a high
$\gamma$ value, only the high frequencies are visible.
Multiscale edge enhancement \cite{col:velde99} can be seen
as a generalization of this approach to all resolution levels.
Images with a high dynamic range are also
difficult to analyze. For example, astronomers generally visualize their
images using a logarithmic transformation. We see in the next
section that wavelet can also be used to compress the dynamic range
at all scales, and therefore allows us to clearly see some very faint
features.

\subsection{Multiscale Edge Enhancement}
\subsubsection{Gray Images}
Velde has proposed the following algorithm \cite{col:velde99}:
\begin{enumerate}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item The image  $L$ is mapped nonlinearly according to:
\begin{eqnarray}
   L(i) \rightarrow L(i)^{1-q} 100^q
\end{eqnarray}
\item The $L$ image is decomposed into a multiscale gradient
pyramid by using the dyadic wavelet transform (two directions per scale).
The gradient at the scale $j$, the pixel position $i$  is
calculated by: $G_j(i) = \sqrt{ (w_j^{(h)}(i))^2 + (w_j^{(v)}(i))^2}$ where
$w_j^{(h)}$ and $w_j^{(v)}$ are the wavelet coefficients in both
vertical and diagonal directions
at pixel position $i$.
\item The two wavelet coefficients at scale $j$ and at position $i$
are multiplied by  $y(G_j(i))$,
where $y$ is defined by:
\begin{eqnarray}
  y(x) & = & ({m \over c})^p \mbox{ if } \mid x \mid < c \nonumber \\
  y(x) & = & ({m \over \mid x \mid })^p  \mbox{ if } c \le \mid x \mid < m \nonumber \\
  y(x) & = & 1  \mbox{ if } \mid x \mid \ge m
\end{eqnarray}
\item The $\tilde L$image is reconstructed
from the modified wavelet coefficients.
\item the $\tilde L$ image is mapped nonlinearly according to:
\begin{eqnarray}
  \tilde L(i) \rightarrow \tilde L(i)^{1 \over 1-q} 100^{- {q \over {1-q}}}
\end{eqnarray}
\end{enumerate}

\begin{figure}[htb]
\vbox{
\centerline{
\hbox{
\psfig{figure=fig_velde.ps,bbllx=3cm,bblly=13cm,bburx=20cm,bbury=25.cm,width=11cm,height=8cm,clip=}
}}
}
\caption{Enhanced coefficients versus  original coefficients.
Parameters are m=30,c=3,p=0.5, and q=0.
}
\label{fig_velde1}
\end{figure}

Four parameters are needed $p$,$q$,$m$,$c$.
$p$ determines the degree of non-linearity in the nonlinear rescaling
of the luminance, and must be in $]0,1[$. $q$ must be in $[-0.5,0.5]$.
When $q > 0$, then darker parts are less enhanced than the lighter parts.
When $ q < 0$, then the dark parts are more enhanced than lighter parts.
Coefficients larger than $m$ are not modified by the algorithm.
The $c$ parameter corresponds to the noise level.

Figure~\ref{fig_velde1} shows the modified wavelet coefficients versus
the original wavelet coefficients for a given set parameters
(m=30,c=3,p=0.5, and q=0).

\subsection{The LOG-Wavelet Representation of Gray Images}

By using the \`a trous wavelet transform algorithm, an image $I$
\index{a trous wavelet transform}
\index{wavelet transform}
can be defined as the sum of its $J$ wavelet scales and the last smooth
array:
\begin{eqnarray}
I(x,y) = c_{J}(x,y) + \sum_{j=1}^{J} w_j(x,y)
\label{resid}
\end{eqnarray}
where the first term on the right is the last smoothed array,
and $w$ denotes a wavelet scale. See \cite{starck:book98} more
details about this algorithm.
The wavelet-log representations consists in replacing $w_j(x,y)$
by $\log(\mid w_j(x,y)\mid)$:
\begin{eqnarray}
I_w(x,y) = \log(c_{J}(x,y)) +  \sum_{j=1}^{J} \mathrm{sgn}(w_j(x,y)) \log(\mid w_j(x,y)\mid)
\end{eqnarray}

\begin{figure}[htb]
\vbox{
\centerline{
\hbox{
\psfig{figure=fig_comet.ps,bbllx=1.8cm,bblly=12.7cm,bburx=14.5cm,bbury=25.4cm,width=9cm,height=9cm,clip=}
}}
\centerline{
\hbox{
\psfig{figure=comet_wlog_m8.ps,bbllx=1.8cm,bblly=12.7cm,bburx=14.5cm,bbury=25.4cm,width=9cm,height=9cm,clip=}
}}}
\caption{Hale-Bopp Comet image and its wavelet log representation.}
\label{fig_halebopp_wavelet}
\end{figure}
Figure~\ref{fig_halebopp_wavelet} left shows  the  logarithm of
Hale-Bopp comet image and right its wavelet log representation. The jets
clearly appears in the last representation.

\subsection{Contrast Enhancement Program: mr\_contrast}
\index{mr\_contrast}
The program {\em mr\_contrast} enhances the contrast of a gray image.
Several methods are available.
{\bf
\begin{center}
 USAGE: mr\_contrast option in\_image out\_image
\end{center}}
where options are:
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item {\bf [-m contrast\_enhancement\_method]}
\begin{enumerate}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item Histogram Equalization
\item Wavelet Coefficients Enhancement
\item Wavelet-Log function: $f(w) = log(\mid w \mid +L) $
\item Wavelet-Log function: $f(w) = sgn(w).log(\mid w \mid+L)$
\item K-Sigma clipping.
\item Add to the image its Laplacian.
\end{enumerate}
Default is Wavelet Coefficients Enhancement.
\item {\bf [-n number\_of\_scales]} \\
Number of scales used in the wavelet transform.
Default is 4.
\item {\bf [-M M\_parameter]} \\
M Parameter. Only used if ``-e'' option is set.
Coefficients larger than M are not enhanced.
Default is 100.
\item {\bf [-P P\_parameter]} \\
P Parameter. Only used if ``-e'' option is set.
P must be in the interval $]0,1[$.
Default is $0.5$.
\item {\bf [-Q Q\_parameter]} \\
Q Parameter. Only used if ``-e'' option is set.
Q must be in the interval $[-0.5,0.5]$. When $Q > 0$,
darker part are less enhanced than lighter part.
When $Q < 0$, darker part are more enhanced than lighter part.
Default is $0$.
\item {\bf [-C C\_parameter]} \\
C Parameter. Only used if ``-e'' option is set.
Default is $0$.
\item {\bf [-K ClippingValue]} \\
Clipping value. Default is 3.
\item {\bf [-L Param]} \\
Parameter for for the Laplacian or the log method. Default is 0.1.
\end{itemize}
\subsubsection*{Examples:}
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item mr\_contrast image.fits image\_out.fits\\
Enhance the contrast by multiscale edge method.
\item mr\_contrast -n6 -m4 image.fits image\_out.fits\\
Enhance the contrast by wavelet log representation using six
resolution levels.
\end{itemize}


\documentclass[a4paper,12pt]{article}
\usepackage[utf8]{inputenc}
\usepackage{fontspec}
%\XeTeXlinebreaklocale "zh"
%\XeTeXlinebreakskip = 0pt plus 1pt
%\setmainfont[Mapping=tex-text]{DejaVu Serif} % rm
\usepackage{graphicx}
\usepackage[a4paper]{geometry}
%\setlength{\textwidth}{5in}
%\setlength{\hoffset}{0.6in}
%header
\usepackage{fancyhdr}
%math package
\usepackage[fleqn]{amsmath}
%\usepackage{amsmath,amssymb}
%code package
%\usepackage{minted}
\usepackage{listings}
\usepackage{minted}
\usepackage[lined,boxed]{algorithm2e}

\newcommand{\horrule}[1]{\rule{\linewidth}{#1}}
\title{
\normalfont \normalsize \vspace{-13em} % Your university, school and/or department name(s)
\horrule{1pt} \\[0.5cm] % Thick bottom horizontal rule
\huge Neural Network
\\ % The assignment title
\vspace{1em}
\large Assignment 1
\horrule{2pt} \\[0.5cm] % Thick bottom horizontal rule
}
\author{Zhao Shenjian 5110309748}
\date{}
\pagestyle{fancy}
\rhead{Two Spiral Problem}

\begin{document}
\maketitle
\thispagestyle{empty}
\setcounter{page}{1}
\section*{Part I: Back-propagation Algorithms}
  \subsection*{1.1 On-line learning}
  \subsubsection*{Case 1: Neuron j is an Output Node}
  \paragraph{} Because I don't have enough time to write the entire derivation, I only write the final result. It's definitely right, because I have check
  it using my program.\\
  \begin{align*}
    \delta_{j}(n) &= e_j(n)f^{\prime}_j(net_j) =  e_j(n)f(net_j)(1-f(net_j))\\
     \Delta u_{ji}(n) &= \eta \delta_j(n)x^{2}_i(n) \\
     \Delta v_{ji}(n) &= \eta \delta_j(n)x_i(n) \\
     \Delta b_j(n) &= \eta \delta_j(n)
  \end{align}

  $f$ is sigmoid activation function, $net_j = \Sigma  u_{kji}x^{2}_{k-1,i} + v_{kji}x_{k-1,i} + b_{kj}$

  \subsubsection*{Case 2: Neuron j is a Hidden Node}
  \begin{align*}
    \delta_{j}(n) &= f^{\prime}_j(net_j) \varSigma_{k}(\delta_{k}(n)(2u_{kj}(n)x_j + v_{kj}(n)))\\
     \Delta u_{ji}(n) &= \eta \delta_j(n)x^{2}_i(n) \\
     \Delta v_{ji}(n) &= \eta \delta_j(n)x_i(n) \\
     \Delta b_j(n) &= \eta \delta_j(n)
  \end{align}

  \subsection*{1.2 Batch learning}
  \subsubsection*{Case 1: Neuron j is an Output Node}
   \begin{align*}
    \delta_{j}(n) &= e_j(n)f^{\prime}_j(net_j) =  e_j(n)f(net_j)(1-f(net_j))\\
     \Delta u_{ji}(n) &= \cfrac{\eta}{N} \Sigma_n(\delta_j(n)x^{2}_i(n)) \\
     \Delta v_{ji}(n) &= \cfrac{\eta}{N} \Sigma_n(\delta_j(n)x_i(n)) \\
     \Delta b_j(n) &= \cfrac{\eta}{N}\Sigma_n\delta_j(n)
  \end{align}

   \subsubsection*{Case 2: Neuron j is a Hidden Node}
    \begin{align*}
    \delta_{j}(n) &= f^{\prime}_j(net_j) \varSigma_{k}(\delta_{k}(n)(2u_{kj}(n)x_j + v_{kj}(n)))\\
    \Delta u_{ji}(n) &= \cfrac{\eta}{N} \Sigma_n(\delta_j(n)x^{2}_i(n)) \\
     \Delta v_{ji}(n) &= \cfrac{\eta}{N} \Sigma_n(\delta_j(n)x_i(n)) \\
     \Delta b_j(n) &= \cfrac{\eta}{N}\Sigma_n\delta_j(n)
  \end{align}

 \section*{Part II: C++ implementation}
 \paragraph{}In order to master every detail of neural network and gain efficiency, I use c++ to solve this problem.
 The source code is in src folder. Please read the README to build and run the program.
 \paragraph{}However I use matlab to plot the result image, it doesn't matter.

 \section*{Part III: Test results}
 \subsection*{3.1 Correctness}
  \paragraph{}
 First to say, my result is correct. The misclassification is 0 for both online learning and batch learning. The following two pictures
 shows the results.

 \begin{figure}[h]
\centering
\subfigure{
\includegraphics[width=.35\textwidth]{batch_result}
}
\subfigure{
\includegraphics[width=.35\textwidth]{seq_result}
}

\caption{Batch result(left) and Online result(right)}
\label{fig:result}
\end{figure}

\subsection*{3.2 Efficiency}
  \paragraph{} Although the results are both correct, the online learning is more efficient according to my test result.
  The learning rate in batch mode should be much larger than online learning. I don't why. And I have tried many learning rate,
  the total epochs of batch mode is 10 times  more than online learning.

\begin{figure}[h!]
      \centering
      \includegraphics{running_time}
      \caption{Running time comparison}
      \end{figure}

\end{document}