\subsection{A neural algorithm of artistic style}
\label{sec:gatys}
\textit{A neural algorithm of artistic style} is a paper written by Gatys et al. \cite{Gatys:1}. This paper goes in depth about how to use a feature space from a convolutional neural network to generate images with a given style. To generate an image with a given style can be framed as an optimization problem, where we want to make the generated image look like the content image, but with the style of the style image. This optimization is done by using the gradient descent algorithm.
\subsubsection{The VGG-network}
\label{sec:vgg}
\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.20]{report/Background/images/vgg19.png}
\caption{The VGG-19 network}
\label{fig:rotasjon}
\end{center}
\end{figure}
The VGG-network is responsible for producing the feature space that gives filter responses for the content image and the style image. The filters get more complex with the position of the layer in the network, thus generating a more higher-dimensional representation of the image. Gatys et. al used 16 convolutional and 5 pooling layers of the 19 layer VGG-network. They replaced the max-pooling layers in the network with average-pooling layers since they gave more appealing results. \newline\newline
Let $\vec{p}$ be the content image (the image we want to style), and the $\vec{x}$ be the generated image. Then their feature representations in a layer $l$ can be described by the matrices $P^l$ and $F^l$, where $P^l, F^l\in\mathbb{R}^{N_l\times M_l}$. Here, $N_l$ are the distinct filters in layer $l$, each with feature maps of size $M_l$.
\subsubsection{Loss functions}
Since one of the goals of the optimization is to make $P^l$ similar to $F^l$, i.e. making the generated image like the original content image, Gatys et. al defined the following loss function, the content loss, between the two feature representations described above:
\begin{equation}
\label{eq:content_loss}
    \mathcal{L}_\text{content}(\vec{p},\vec{x},l)=\frac{1}{2}\sum_{i,j}{(F_{ij}^l-P_{ij}^l)^2},
\end{equation}
where $F_{i,j}$ and $P_{i,j}$ are the activations of the $i$th filter at position $j$ in layer $l$. This loss function is the squared-error loss function. Similarly, they defined the style loss as follows:
\begin{equation}
\label{eq:style_loss}
    \mathcal{L}_\text{style}(\vec{a},\vec{x})=\sum_{l=0}^L{w_lE_l},
\end{equation}
where
\begin{equation}
    E_l=\frac{1}{4N_l^2M_l^2}\sum_{i,j}{(G_{ij}^l-A_{ij}^l)^2}.
\end{equation}
Here $A^l$ is the is the style representation of the style image in layer $l$, and $G^l$ is the style representation of the the generated image. $\vec{a}$ is the style image, and $w_l$ are the weighting factors for how much each layer should contribute to the total loss. $G^l$ is also known as the Gram matrix. This matrix will be discussed in Section \ref{section:gram}. Now, the total loss can be written as:
\begin{equation}
\label{eq:total_loss}
    \mathcal{L}_\text{total}(\vec{p}, \vec{a}, \vec{x})=\alpha\mathcal{L}_\text{content}(\vec{p},\vec{x})+\beta\mathcal{L}_\text{style}(\vec{a},\vec{x})
\end{equation}
Here, $\alpha$ and $\beta$ are weights for the content image and the style image. The higher the weight, the more that loss is minimized, resulting in the stylized image to either be more equal to the content image, or the style image.
\subsubsection{Gram Matrix}
\label{section:gram}
\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.30]{report/Background/images/featuremap.png}
\caption{Feature map with three channels}
\label{fig:featuremap}
\end{center}
\end{figure}
The Gram matrix is a style representation built on top of the CNN. This matrix gives the feature correlations between the vectorised features maps $i$ and $j$ in layer $l$. This correlation is calculated by taking the inner product between the feature maps. Gatys et al. give the following formula for the Gram matrix:
\begin{equation}
    G_{ij}^l=\sum_k{F_{ik}^lF_{jk}^l}
\end{equation}
To understand the Gram matrix, we can reduce the feature maps to 2D vectors. If we have two vectors, say $\vec{a}$ and $\vec{b}$, the closer they are to each other, the more correlated they are. Then the angle between these two vectors will be small, resulting in a large cosine value, giving a large dot product. So the larger the dot product is, the more correlated $\vec{a}$ and $\vec{b}$ are.
This idea is similar for feature maps. Consider a feature map consisting of three channels, like the one in Figure \ref{fig:featuremap}. The red channel represents the feature of horizontal lines, the yellow represents the feature of vertical lines, and the green represents the features of the color green. If the red channel and the yellow channel have high activation values, then we can say that we have an image of for example a chess board. These two channels will have a higher correlation than that between the yellow channel and the green channel. In a sense, this correlation describes the style of the generated image.
\subsubsection{Gradient descent}
The derivative of the content loss is given by
\begin{equation}
    \frac{\partial\mathcal{L}_\text{content}}{\partial F_{ij}^l}=\begin{cases}
    (F^l-P^l)_{ij}\,&\text{ if }\,F_{ij}^l>0\\
    0\,&\text{ if }\,F_{ij}^l<0.
    \end{cases}
\end{equation}
The derivative of $E_l$ is given by
\begin{equation}
    \frac{\partial E_l}{\partial F_{ij}^l}=\begin{cases}
    \frac{1}{N_l^2M_l^2}\left((F^l)^T(G^l-A^l)\right)_{ji}\,&\text{ if }\,F_{ij}^l>0\\
    0\,&\text{ if }\,F_{ij}^l<0.
    \end{cases}
\end{equation}
The gradients can be calculated by using standard-error back-propagation, as described by Gatys et al.


\chapter{The \texorpdfstring{\Coq}{Coq} Proof Assistant}\label{chap:coq}

\Coq \citep{coq-refman-09} is based on the formal
language \emph{Calculus of Inductive Constructions}
\citep{coquand-huet-88,mohring-93}, which is essentially a typed
$\lambda$-calculus with inductive types. In this language, logical
propositions are represented as types and proofs of such propositions
are $\lambda$-terms, motivated by the Curry--Howard--de Bruijn
correspondence \citep{girard-89}. The core of the \Coq system is its type
checking algorithm.

We present a very short introduction to \Coq and refer to
\citet{bertot-casteran-04} and \citet{chlipala-09} for further
reading. Sections~\ref{sec:guardedness} and \ref{sec:positivity} discuss two
technicallities related to the \Coq development described in
Chapter~\ref{chap:implementation}.
%The seasoned \Coq user can skip to Section~\ref{sec:ordimp}.

\section{Types and Propositions}

Every term in \Coq has a type and every type is also a
term. The type of a type is called a \emph{sort} and the sorts
in \Coq are
\begin{compactitem}
\item \coqdockw{Prop}, the sort of logical propositions,
\item \coqdockw{Set}, the sort of program specifications and datatypes,
\item \coqdockw{Type}$_0$, the sort of \coqdockw{Prop} and
  \coqdockw{Set}, and
\item \coqdockw{Type}$_{i+1}$, the sort of
  \coqdockw{Type}$_i$.\footnote{The subscripts $i$ of the sorts
    \coqdockw{Type}$_i$ are invisible to the user and only used
    internally.}
\end{compactitem}
For example,
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}}
is the datatype of the natural numbers. It lives in \coqdockw{Set} and
is defined inductively.
\begin{singlespace}
\begin{coqdoccode}
\coqdocnoindent
\coqdockw{Inductive} \coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}}
:
\coqdockw{Set} :=\coqdoceol
\coqdocindent{1.0em}
\ensuremath{|} \coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{O}{\coqdocconstructor{O}}
:
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}}\coqdoceol
\coqdocindent{1.0em}
\ensuremath{|} \coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{S}{\coqdocconstructor{S}}
:
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}} $\rightarrow$
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}}.\coqdoceol
\end{coqdoccode}
\end{singlespace}
The logical proposition that for every natural number $n$, there
exists a natural number $m$ larger than $n$, can be stated as a term
of sort \coqdockw{Prop}.
\begin{singlespace}
\begin{coqdoccode}
\coqdocnoindent
(\ensuremath{\forall} \coqdocvar{n} :
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}},
\ensuremath{\exists} \coqdocvar{m} :
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}},
\coqdocvariable{n} < \coqdocvariable{m}) : \coqdockw{Prop}\coqdoceol
\end{coqdoccode}
\end{singlespace}
Using the vocabulary of types and terms, the universal quantifier
$\forall$ is called the \emph{product type constructor}. A product
type \begin{coqdoccode}$\forall$ \coqdocvar{x} : $T$,
  $U$\end{coqdoccode} is called \emph{dependent} if \coqdocvar{x} occurs free
in $U$, otherwise it is written $T \rightarrow U$. The type of the
constructor symbol \coqdocconstructor{S} defined above, for example,
is that of functions from
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}}
to
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}}
and is not dependent. The non-dependent function space notation $T \rightarrow
U$ is also used for logical implication, justified by the Curry--Howard--de
Bruijn correspondence.

\section{Terms and Proofs}

Proofs of logical propositions can be defined in two ways. First, we
can write a proof term directly. The only requirement is that this
term has as type the logical proposition that we want to
prove (again, justified by the Curry--Howard--de Bruijn
correspondence). Second, we can use \emph{tactics} to construct a proof term
interactively, in a way mimicking natural deduction.

As an example of the use of tactics, we prove the proposition from the
previous section. This is done by stating the proposition, after which
the system enters a goal-directed proof mode. In this mode, we are
presented with a goal, consisting of
\begin{inparaenum}[(i)]
\item a context of local variables that are available
\item a proposition denoting what remains to be proven.
\end{inparaenum}
Tactics can now be applied to progressively transform the goal into a
simpler goal. When the goal is simple enough to be solved directly by
applying a tactic, we are done proving the proposition.
\begin{singlespace}
\begin{coqdoccode}
\coqdocnoindent
\coqdockw{Lemma}
\coqdef{CoqIntro.ltserial}{lt\_serial}{\coqdoclemma{lt\_serial}} :
\ensuremath{\forall} \coqdocvar{n} :
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}},
\ensuremath{\exists} \coqdocvar{m} :
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}},
\coqdocvariable{n} < \coqdocvariable{m}.\coqdoceol
\coqdocnoindent
\coqdockw{Proof}.\coqdoceol
\coqdocindent{1.0em}
\coqdoctac{intro} \coqdocvar{n}. $\,$
\coqdoctac{exists}
(\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{S}{\coqdocconstructor{S}}
\coqdocvar{n}). $\,$
\coqdoctac{auto}.\coqdoceol
\coqdocnoindent
\coqdockw{Qed}.\coqdoceol
\end{coqdoccode}
\end{singlespace}
In this example, we use the tactic \coqdoctac{intro} to introduce the
variable \coqdocvariable{n} to the context. With \coqdoctac{exists},
we supply a witness for the existential quantification. At this point,
the goal is simple enough to be solved directly by the
\coqdoctac{auto} tactic.

Recursive functions are defined using the \coqdockw{Fixpoint}
keyword. The function must have an argument of an inductive type that is
structurally decreasing with each recursive call. Consider for example the
definition of
the factorial function, which also shows how a case analysis on values
of inductive types can be done with the \coqdockw{match} keyword.
\begin{singlespace}
\begin{coqdoccode}
\coqdocnoindent
\coqdockw{Fixpoint}
\coqdef{CoqIntro.factorial}{factorial}{\coqdocdefinition{factorial}}
(\coqdocvar{n} :
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}})
:
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}}
:=\coqdoceol
\coqdocindent{1.00em}
\coqdockw{match} \coqdocvariable{n} \coqdockw{with}\coqdoceol
\coqdocindent{1.00em}
\ensuremath{|}
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{O}{\coqdocconstructor{O}}
\ensuremath{\Rightarrow}
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{S}{\coqdocconstructor{S}}
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{O}{\coqdocconstructor{O}}\coqdoceol
\coqdocindent{1.00em}
\ensuremath{|}
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{S}{\coqdocconstructor{S}}
\coqdocvar{n} \ensuremath{\Rightarrow}
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{S}{\coqdocconstructor{S}}
\coqdocvariable{n} \ensuremath{\times}
(\coqref{CoqIntro.factorial}{\coqdocdefinition{factorial}}
\coqdocvariable{n})\coqdoceol
\coqdocindent{1.00em}
\coqdockw{end}.\coqdoceol
\end{coqdoccode}
\end{singlespace}

%\chapter{Two Syntactical Criteria in \texorpdfstring{\Coq}{Coq}}\label{chap:appendix}
%
%We include a short discussion of two \Coq technicalities.

\section{The Positivity Condition}\label{sec:positivity}

\Coq restricts inductive definitions to those that satisfy the
\emph{positivity condition}. The reason for this is that definitions
that fail this (syntactic) criterion may lead to an inconsistent
  system. For a precise definition of positivity, consult
  \citetalias[Section 4.5.3]{coq-refman-09}.

Consider again the definition of rewrite sequences from
Section~\ref{sec:seq}. A more natural way to define the type of the
\coqref{Rewriting.Lim}{\coqdocconstructor{Lim}} constructor might be
by using a $\Sigma$-type instead of a separate function for the target
terms of the branches.
\begin{singlespace}
\begin{coqdoccode}
\coqdocindent{1.00em}
\ensuremath{|} \coqdocconstructor{Lim} :
\ensuremath{\forall} \coqdocvar{s} \coqdocvar{t}
(\coqdocvar{f} :
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}}
\ensuremath{\rightarrow} \{ \coqdocvar{t'} : \coqref{Term.term}{\coqdocinductive{term}}
\& \coqdocvar{s}
\coqref{Rewriting.sequence}{$\rewrites_\mathcal{R}$}
\coqdocvariable{t'} \}),\coqdoceol
\coqdocindent{5.00em}
\coqref{Rewriting.converges}{\coqdocdefinition{converges}}
(\coqdockw{fun} \coqdocvar{n} \ensuremath{\Rightarrow}
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Specif}{projT1}{\coqdocdefinition{projT1}}
(\coqdocvariable{f} \coqdocvariable{n})) \coqdocvariable{t}
$\rightarrow$ (\coqdocvariable{s}
\coqref{Rewriting.sequence}{$\rewrites_\mathcal{R}$}
\coqdocvariable{t})\coqdoceol
\end{coqdoccode}
\end{singlespace}
However, this type definition does not satisfy the positivity
condition and therefore we cannot use it. We feel that the definition
from Section~\ref{sec:seq}, which does satisfy the condition, models
our intentions adequately.

\section{Guardedness in Corecursive Definitions}\label{sec:guardedness}

In \Coq, coinductive types can be defined using the
\coqdockw{CoInductive} keyword \citep{gimenez-casteran-07}. No
induction principles are defined for these types, because they are not
necessarily well-founded.\footnote{\Coq automatically derives induction
  principles for inductive definitions.}
Objects in a coinductive type may be infinite (i.e.\ contain an infinite
amount of constructors). However, in order to guarantee productivity,
definitions of such objects are required by \Coq to be in \emph{guarded}
form \citep{gimenez-94}. A corecursive definition in guarded form
satisfies two (syntactical) conditions. First, every corecursive call
must occur inside at least one constructor (of the same coinductive
type). Second, every corecursive call may only occur inside
abstractions or constructors (of the same coinductive
type).\footnote{To be more precise, the corecursive call is also
  allowed to occur inside \coqdockw{match} constructs and other
  corecursive definitions.}

In the \coqref{Term.term}{\coqdocinductive{term}} definition, we use a vector
type, parameterised by the type of its element and its size. Naturally, one
would implement a vector type in \Coq inductively, as for example has been
done in the standard library.
\begin{singlespace}
\begin{coqdoccode}
\coqdocnoindent
\coqdockw{Inductive} \coqdef{Bvector.vector}{vector}{\coqdocinductive{vector}}
(\coqdocvar{A} : \coqdockw{Type}) :
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}}
\ensuremath{\rightarrow} \coqdockw{Type} :=\coqdoceol
\coqdocindent{1.00em}
\ensuremath{|} \coqdef{Bvector.Vnil}{Vnil}{\coqdocconstructor{Vnil}}  :
\coqref{Bvector.vector}{\coqdocinductive{vector}} \coqdocvariable{A} 0\coqdoceol
\coqdocindent{1.00em}
\ensuremath{|} \coqdef{Bvector.Vcons}{Vcons}{\coqdocconstructor{Vcons}} :
\coqdocvariable{A} \ensuremath{\rightarrow} \ensuremath{\forall} \coqdocvar{n},
\coqref{Bvector.vector}{\coqdocinductive{vector}} \coqdocvariable{A}
\coqdocvariable{n} \ensuremath{\rightarrow}
\coqref{Bvector.vector}{\coqdocinductive{vector}} \coqdocvariable{A}
(\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{S}{\coqdocconstructor{S}}
\coqdocvariable{n}).\coqdoceol
\end{coqdoccode}
\end{singlespace}

Now consider the following trivial example of a basic operation on terms by
corecursive traversal.
\begin{singlespace}
\begin{coqdoccode}
\coqdocnoindent
\coqdockw{CoFixpoint} \coqdef{Term.id}{id}{\coqdocdefinition{id}}
(\coqdocvar{t} : \coqref{Term.term}{\coqdocinductive{term}}) :
\coqref{Term.term}{\coqdocinductive{term}} :=\coqdoceol
\coqdocindent{1.00em}
\coqdockw{match} \coqdocvariable{t} \coqdockw{with}\coqdoceol
\coqdocindent{1.00em}
\ensuremath{|} \coqref{Term.Var}{\coqdocconstructor{Var}} \coqdocvar{x}
\ensuremath{\Rightarrow} \coqref{Term.Var}{\coqdocconstructor{Var}}
\coqdocvariable{x}\coqdoceol
\coqdocindent{1.00em}
\ensuremath{|} \coqref{Term.Fun}{\coqdocconstructor{Fun}} \coqdocvar{f}
\coqdocvar{args} \ensuremath{\Rightarrow}
\coqref{Term.Fun}{\coqdocconstructor{Fun}} \coqdocvariable{f}
(\coqdocdefinition{vmap} \coqref{Term.id}{\coqdocdefinition{id}}
\coqdocvariable{args})\coqdoceol
\coqdocindent{1.00em}
\coqdockw{end}.\coqdoceol
\end{coqdoccode}
\end{singlespace}
This definition is ill-formed, since the corecursive call to
\coqref{Term.id}{\coqdocdefinition{id}} is not guarded.\footnote{The call to
  \coqref{Term.id}{\coqdocdefinition{id}} is hidden inside
  \coqdocdefinition{vmap}, which is defined by recursion on the vector
  \coqdocvariable{args}.}
We define a recursive type of vectors as an alternative to the
inductive type.
\begin{singlespace}
\begin{coqdoccode}
\coqdocnoindent
\coqdockw{Inductive} \coqdef{Vector.Fin}{Fin}{\coqdocinductive{Fin}} :
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}}
\ensuremath{\rightarrow} \coqdockw{Type} :=\coqdoceol
\coqdocindent{1.00em}
\ensuremath{|} \coqdef{Vector.First}{First}{\coqdocconstructor{First}} :
\ensuremath{\forall} \coqdocvar{n}, \coqref{Vector.Fin}{\coqdocinductive{Fin}}
(\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{S}{\coqdocconstructor{S}}
\coqdocvariable{n})\coqdoceol
\coqdocindent{1.00em}
\ensuremath{|} \coqdef{Vector.Next}{Next}{\coqdocconstructor{Next}}  :
\ensuremath{\forall} \coqdocvar{n}, \coqref{Vector.Fin}{\coqdocinductive{Fin}}
\coqdocvariable{n} \ensuremath{\rightarrow}
\coqref{Vector.Fin}{\coqdocinductive{Fin}}
(\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{S}{\coqdocconstructor{S}}
\coqdocvariable{n}).\coqdoceol
\coqdocemptyline
\coqdocnoindent
\coqdockw{Definition}
\coqdef{Vector.vector}{vector}{\coqdocdefinition{vector}} (\coqdocvar{A} :
\coqdockw{Type}) (\coqdocvar{n} :
\coqexternalref{http://coq.inria.fr/stdlib/Coq.Init.Datatypes}{nat}{\coqdocinductive{nat}})
:= \coqref{Vector.Fin}{\coqdocinductive{Fin}} \coqdocvariable{n}
\ensuremath{\rightarrow} \coqdocvariable{A}.\coqdoceol
\end{coqdoccode}
\end{singlespace}
% further explain this vector type?
This makes for a definition of \coqref{Vector.vmap}{\coqdocdefinition{vmap}}
that is just an abstraction, and therefore solves the guardedness problem in
\coqref{Term.id}{\coqdocdefinition{id}}.
\begin{singlespace}
\begin{coqdoccode}
\coqdocnoindent
\coqdockw{Definition} \coqdef{Vector.vmap}{vmap}{\coqdocdefinition{vmap}}
\coqdocvar{A} \coqdocvar{B} (\coqdocvar{f} :
\coqdocvariable{A} \ensuremath{\rightarrow} \coqdocvariable{B}) \coqdocvar{n}
: \coqref{Vector.vector}{\coqdocdefinition{vector}} \coqdocvariable{A}
\coqdocvariable{n} \ensuremath{\rightarrow}
\coqref{Vector.vector}{\coqdocdefinition{vector}} \coqdocvariable{B}
\coqdocvariable{n} :=\coqdoceol
\coqdocindent{1.00em}
\coqdockw{fun} \coqdocvar{v} \coqdocvar{i} \ensuremath{\Rightarrow}
\coqdocvariable{f} (\coqdocvariable{v} \coqdocvariable{i}).\coqdoceol
\end{coqdoccode}
\end{singlespace}


\chapter{\proj Multiresolution}
\label{ch_wave}
% \chapterhead{Programs}
\markright{Programs}

\section{Multiscale Transform}
\index{Multiscale}
\subsection{Introduction}
The Morlet-Grossmann definition \cite{wave:grossmann89}
of the continuous wavelet
transform for a 1-dimensional signal $f(x)\in L^2(R)$, the space of all
square integrable functions, is:
\index{wavelet transform, continuous}
\index{continuous wavelet transform}
\index{wavelet transform, Morlet-Grossman}
\begin{eqnarray}
W(a,b)=\frac{1}{\sqrt a}\int_{-\infty}^{+\infty}f(x) \psi^* \left(
\frac{x-b}{a} \right) dx
\label{eqn_wave}
\end{eqnarray}
where:
\begin{itemize}
\item $W(a,b)$ is the wavelet coefficient of the function $f(x)$
\item $\psi(x)$ is the analyzing wavelet
\item $a$ ($>0$) is the scale parameter
\item $b$ is the position parameter
\end{itemize}
Many discrete wavelet transform algorithms have been described
\cite{starck:book98}. The most widely-known one
is certainly the orthogonal one,
proposed by Mallat \cite{wave:mallat89} and its bi-orthogonal version.
Using the orthogonal wavelet transform (OWT), a signal $s$ can
be decomposed by:
\begin{eqnarray}
 s(l) = \sum_{k} c_{J,k} \phi_{J,l}(k)
       +  \sum_{k} \sum_{j=1}^J \psi_{j,l}(k) w_{j,k}
\end{eqnarray}
with $\phi_{j,l}(x) = 2^{-j} \phi(2^{-j}x-l)$ and  $\psi_{j,l}(x) = 2^{-j} \psi(2^{-j}x-l)$,
where $\phi$ and $\psi$ are respectively
the scaling function and the wavelet function.
$J$ is the number of resolutions used in the decomposition,
$w_{j}$ the wavelet (or details) coefficients at scale $j$, and
$c_{J}$ is a coarse or smooth version of the original signal $s$.
Thus, the algorithm outputs $J+1$ subband arrays.
The indexing is such that,
here, $j = 1$ corresponds to the finest scale (high frequencies).
Coefficients $c_{j,k}$ and $w_{j,k}$ are obtained by means of the filters
$h$ and $g$:
\begin{eqnarray}
c_{j+1,l} & = & \sum_k h(k-2l) c_{j,k} \nonumber \\
w_{j+1,l} & = & \sum_k g(k-2l) c_{j,k}
\end{eqnarray}
where $h$ and $g$ verify:
\begin{eqnarray}
\frac{1}{2} \phi(\frac{x}{2}) & = &  \sum_k h(k) \phi(x-k) \nonumber \\
\frac{1}{2} \psi(\frac{x}{2}) & = & \sum_k g(k) \phi(x-k)
\end{eqnarray}
and the reconstruction of the signal is performed with:
\begin{eqnarray}
c_{j,l} = 2 \sum_k [ \tilde h(k+2l) c_{j+1,k}  + \tilde g(k+2l) w_{j+1,k}  ]
\end{eqnarray}
where the filters $\tilde h$ and $\tilde g$ must verify the conditions of
dealiasing and exact reconstruction:
\begin{eqnarray}
\hat{h}(\nu+\frac{1}{2}) \hat{\tilde h}(\nu) + \hat{g}(\nu+\frac{1}{2}) \hat{\tilde g}(\nu)  & = & 0 \nonumber \\
\hat{h}(\nu) \hat{\tilde h} + \hat{g}(\nu) \hat{\tilde g}(\nu)  & = & 1
\end{eqnarray}
By this algorithm, the transformation of an image is an image.

\subsection{Methods}
The application of the OWT to image compression and filtering
has lead to impressive results compared to previous methods.
However some problems related to the OWT may impact on their
use in some applications, and include the following.
\begin{enumerate}
\item Edge representation: If the OWT performs better than
the FFT at representing edges in an image, it is still not optimal.
There is only a fixed number of directional elements independent of scale,
and there is no highly anisotropic element \cite{cur:candes99_1}.
For instance, the Haar 2D
wavelet transform is optimal for finding features with a
ratio $length/width = 2$, and a horizontal, vertical, or diagonal
orientation.
\item Isotropic feature representation:
the 2D OWT \cite{wave:mallat89} leads to a wavelet
transform with three wavelet functions (at each scale there are
three wavelet coefficient sub-images) which does not simplify the analysis and
the interpretation of the wavelet coefficients. An isotropic transform
seems more appropriate for images containing features or objects with
no favored orientation (such as astronomical or medical images).
\item Negative values:
By definition, the wavelet coefficient mean is zero. Every time we have
a positive structure at a scale, we have negative values surrounding it. These
negative values often create artifacts during the restoration process, or
complicate the analysis.
\item Point artifacts: For example, cosmic ray hits in optical astronomy
can ``pollute'' all
the scales of the wavelet transform. The wavelet transform is non-robust
relative to such real or detector faults.
\item Integer values: The OWT produce floating values which are not
easy to handle for lossless image compression.
\end{enumerate}
These problems have lead to the development of other multiscale
representations. Some of the other algorithms do not produce an image,
but a pyramid, or cube.
So we separate the different multiscale transform algorithms into classes,
depending on the output data type. Other multiscale methods, which are
nonlinear can also be categorized in this way. We distinguish between
fives classes of multiscale transform:
\bn
\item transforms which produce {\em cubes}
\item transforms which produce  {\em pyramids}
\item transforms which produce  {\em half pyramids}
\item transforms which produce {\em images} (non-redundant transforms)
\item transforms which produce directional {\em cubes}
\en
The first three, and the last,  are redundant
(i.e.\ there are more pixels in the transformation
than in the input data).

\subsection*{Cube transform}
For the first class, the input image $I$ can be expressed
as \cite{starck:bij94_1,starck:mur95_1,starck:mur95_3}:
\begin{eqnarray}
I(x,y) =  c_p (x,y) + \sum_{j=1}^{p} w_j(x,y)
\end{eqnarray}
$w_j$ (for $j=1 \dots p$) and $c_p$ represent the transformation of $I$.
$c_p$ is a very smoothed version of the image $I$, and $w_j$ is
the image which contains information at scale $j$. The transformation
is defined by $n=p+1$ images ($n = $ number of scales). The $p$
images have zero mean (or approximately zero, for nonlinear transforms).
Each of them corresponds to the information at a given scale, i.e.\
structure of a given size in the input image. Compact structures
(with size of one or two pixels) will be found at the first scale ($j=1$).
For this class of transformation, which is very redundant, the amount
of data is multiplied by the number of scales $n$. Therefore this
takes a lot of memory, but the multiresolution coefficients ($w_j(x,y)$)
are easy to interpret.

\subsection*{Pyramidal transform}

The second class of transformation is
less redundant. The first scale has the same size as the image, but
for the other scales, the number of pixels is reduced by four at
each resolution. Thus, if $N^2$ is the number of pixels of $I$, the
number of pixels of the transformation is $4/3N^2$.

\subsection*{Half-pyramidal transform}
This transform \cite{wave:bijaoui97} is relatively close to the previous one, but the two first
scales are not decimated (i.e.\ they have the same size as the input image).
See Appendix~D for more details.

\subsection*{Non-redundant transform}
\begin{figure}[htb]
\centerline{
\hbox{
\psfig{figure=fig_mallat.ps,bbllx=5.5cm,bblly=9.5cm,bburx=16cm,bbury=20cm,height=10cm,width=10cm,clip=}
}}
\caption{Mallat wavelet transform representation of an image.}
\index{Mallat's multiresolution}
\label{fig_mallat}
\end{figure}

The last class is completely non-redundant, and the number of pixels
 is the same as in the input image. This means that it is an image.
Figure~\ref{fig_mallat}
shows the representation of an image using the Mallat transform \cite{wave:mallat89,wave:antonini92,wave:daube88}. At a
given resolution, the image is shared between four parts. Three subimages
correspond to details of the image in the horizontal, vertical, and
diagonal directions, and the last part corresponds to the image at a lower
resolution. The process can then be repeated on the image at the
lower resolution. The Haar wavelet transform, lifting scheme transform,
and the G transform (which is a nonlinear transform based on the minimum and
the maximum) produce the same kind of representation.

\begin{figure}[htb]
\centerline{
\hbox{
\psfig{figure=fig_feauveau.ps,bbllx=5.5cm,bblly=9.5cm,bburx=16cm,bbury=20cm,height=12cm,width=12cm,clip=}
}}
\caption{Feauveau wavelet transform representation of an image.}
\label{fig_feauveau}
\end{figure}
The Feauveau transform \cite{wave:feauveau} (see Figure~\ref{fig_feauveau}) is not redundant,
but the representation is different.
There is no prioritized direction, and we have an intermediate
resolution (half resolution).

\subsection*{Directional cube transform}
In this case, just as for the orthogonal Mallat transform, the
analysis is directional, but there
is no decimation. This means that the number of output bands is equal to
the number of scales multiplied by the number of directions, and each band
has the same number
of pixels as the input image. The dyadic wavelet transform
uses two directions (vertical and horizontal), and the undecimated
wavelet transform uses three directions (vertical, horizontal, and diagonal).

\subsection*{Algorithms}
Since the output differs following the chosen algorithm, we will define
the term {\em band} as a subimage included in a given dyadic scale.
For the cube and pyramid transforms, the number of bands is equal to the
number of scales. For the Mallat algorithm and the undecimated
bi-orthogonal wavelet transform, we have three bands per scale.
For the Feauveau and the dyadic wavelet transform, we have two bands
per scale. Table~\ref{tab_algo} indicates for each \proj \ algorithm its
class, the number of bands per scale, and if it is redundant and linear.

\begin{table}
{\centering \begin{tabular}{|l|c|c|c|c|}
\hline
Multiscale algorithm & Class & Redund- &  Bands  & Linear\\
                     &       & ant         & per scale  &  \\
\hline
\hline
1: linear WT & Cube & Yes & 1 & Yes \\ \hline
2: B-spline WT & Cube & Yes & 1 & Yes \\ \hline
3: wavelet transform in Fourier space & Cube & Yes & 1 & Yes \\ \hline
4: morphological median transform (MT) & Cube & Yes & 1 & No \\ \hline
5: morphological minmax transform & Cube & Yes & 1 & No \\ \hline
6: pyramidal linear WT & Pyramid & Yes & 1 & Yes \\ \hline
7: pyramidal B-spline WT & Pyramid & Yes & 1 & Yes \\ \hline
8: pyramidal WT in Fourier space: alg.\ 1  & Pyramid & Yes & 1 & Yes \\ \hline
9: pyramidal WT in Fourier space: alg.\ 2   & Pyramid & Yes & 1 & Yes \\ \hline
10: pyramidal median transform  & Pyramid & Yes & 1 & No \\ \hline
11: pyramidal Laplacian  & Pyramid & Yes & 1 & No \\ \hline
12: morph.\ pyramidal minmax transform  & Pyramid & Yes & 1 & No \\ \hline
13: decomposition on scaling function & Pyramid & Yes & 1 & No \\ \hline
14: (bi-) orthogonal wavelet transform & Image & No & 3 & Yes \\
14-1: Antonini 7/9 filter             &        &     &   &    \\
14-2: Daubechies filter 4             &        &     &   &    \\
14-3: Biorthogonal 2/6 Haar filters   &        &     &   &    \\
14-4: Biorthogonal 2/10 Haar filters  &        &     &   &     \\
14-5: Odegard 7/9 filters             &        &     &   &     \\
14-6: User's filters                  &        &     &   &     \\ \hline
15: Feauveau WT & Image & No & 2 & Yes \\ \hline
16: Feauveau WT without undersampling & Cube & Yes & 2 & Yes \\ \hline
17: G transform (morph.\ min-max alg.)  & Image & No & 3 & No \\ \hline
18: Haar wavelet transform & Image & No & 3 & Yes \\ \hline
19: Half-pyramidal transform & Half-Pyramid & Yes & 1 & Yes \\ \hline
20: Mixed Half-pyramidal and MT  &  Half-Pyramid & Yes & 1 & No  \\ \hline
21: Dyadic wavelet transform   &  Dir. Cube & Yes & 2 & Yes  \\ \hline
22: Mixed WT and PMT method  & Pyramid & Yes & 1 & No \\ \hline
23: Undecimated Haar transform  & Dir. Cube & Yes & 1 & Yes \\ \hline
24: Undecimated bi-orthog. wavelet trans. & Dir. Cube & Yes & 3 & Yes \\
\ \ \ \ (filters 14-1 to 14-6 are available) &           &     &    &  \\ \hline
25: Wavelet transform via lifting scheme & Image & No & 3 &    \\
25-1: CDF WT                             &       &    &   &  Yes \\
25-2: Median prediction                  &       &    &   &  No \\
25-3: integer Haar WT                    &       &    &   &  No \\
25-4: integer CDF WT                     &       &    &   &  No \\
25-5: integer (4,2) interpolating transform &       &    &   &  No \\
25-6: Antonini 7/9 filter                &       &    &   &  Yes \\
25-7: integer  Antonini 7/9 filter       &       &    &   &  No \\ \hline
\hline
\end{tabular}\par}
\caption{Multiscale transform algorithms.}
\label{tab_algo}
\end{table}
\vspace{0.3cm}

\section{Multiresolution object}

\subsection{Multiresolution transform of image: mr\_transform}
\label{sect_trans}
\index{mr\_transform}
The program
{\em mr\_transform} computes the multiresolution transform of an image.
This transform can be a linear transform (wavelet transform) or
a morphological transform (based on the median, or the minimum
and the maximum).
The output file which contains the transformation has a
suffix, .mr. If the output file name
given by the user does not contain this suffix, it is automatically
added. The ``.mr'' file is a FITS format file, and can be manipulated by
any package dealing with FITS format, or using the
{\em mr\_extract} program.
{\bf
\begin{center}
 USAGE: mr\_transform option image\_in multiresolution\_transform\_out
\end{center}}
where options are:
\begin{itemize}
\baselineskip=0.4truecm
\item {\bf [-t type\_of\_multiresolution\_transform]}
{\small
\begin{enumerate}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item linear wavelet transform: \`a trous algorithm
\item B-spline wavelet transform: \`a trous algorithm
\item wavelet transform in Fourier space
\item morphological median transform
\item morphological minmax transform
\item pyramidal linear wavelet transform
\item pyramidal B-spline wavelet transform
\item pyramidal wavelet transform in Fourier space:
                     wavelet =  between two resolutions
\item  pyramidal wavelet transform in Fourier space:
                     wavelet = difference between the square of two resolutions
\item  pyramidal median transform
\item  pyramidal Laplacian
\item  morphological pyramidal minmax transform
\item  decomposition on scaling function
\item  (bi-) orthogonal wavelet transform. \\
Antonini 7/9 filters ~\cite{wave:antonini92} are used by default, with an
$L_1$ normalization. The filters can be changed using the ``-T'' option, and
an $L_2$ normalization is obtained by ``-L'' option.
\item  Feauveau wavelet transform
\item  Feauveau wavelet transform without undersampling
\item  G transform (non-redundant morphological min-max algorithm)
\item Haar wavelet transform (L2 normalization).
\item Half-pyramidal wavelet transform (HPWT)
\item Mixed HPWT and Median method
\item dyadic wavelet transform
\item Mixed WT and PMT method (WT-PMT)
\item Undecimated Haar transform: \`a trous algorithm
\item Undecimated (bi-) orthogonal wavelet transform. \\
Antonini 7/9 filters ~\cite{wave:antonini92} are used by default, with an
$L_1$ normalization. The filters can be changed using the ``-T'' option, and
an $L_2$ normalization is obtained by ``-L'' option.
\item Wavelet transform via lifting scheme
\end{enumerate}}
Default is 2.
\item {\bf [-T type\_of\_filters]}
{\small
\begin{enumerate}
\item Antonini 7/9 filters.
\item Daubechies filter 4.
\item Biorthogonal 2/6 Haar filters.
\item Biorthogonal 2/10 Haar filters.
\item Odegard 7/9 filters.
\item User's filters.
\end{enumerate}}
Default is Antonini 7/9 filters. \\
 This option is only available if the chosen transform method is
 the (bi-) orthogonal transform (-t 14 or -t 24).
\item {\bf [-L]} \\
Use an $L_2$ normalization. Default is $L_1$.
\item {\bf [-u]} \\
Number of undecimated scales used in the undecimated wavelet transform.
Default is all scales.
\item {\bf [-l type\_of\_lifting\_transform]}
{\small
\begin{enumerate}
\baselineskip=0.4truecm
\item Lifting scheme: CDF WT.
\item Lifting scheme: median prediction.
\item Lifting scheme: integer Haar WT.
\item Lifting scheme: integer CDF WT.
\item Lifting scheme: integer (4,2) interpolating transform.
\item Lifting scheme:  Antonini 7/9 filters.
\item Lifting scheme: integer Antonini 7/9 filters.
\end{enumerate}}
 Default is Lifting scheme: integer Haar WT. \\
 This option is only available if the chosen transform method is
 the lifing scheme (-t 24).
\item {\bf [-n number\_of\_scales]} \\
 Number of scales used in the multiresolution transform.
 Default is 4.
\item {\bf [-x]} \\
 Write all bands separately as images with prefix ``band\_j'', j=1,..,NbrBand
 (j being the band number). Note also that $j=1$ corresponds to
 the smallest scale (highest frequency band), and j increases for larger
 scales.
\item {\bf [-B]} \\
 Same as x option, but interpolate the bands by block in order to
 have the same size as the original image.
% \item {\bf [-i]} \\
% Same as x option, but interpolate the bands by a B$_3$-spline.
% This option is valid only if the chosen multiresolution
%  transform belongs to the pyramidal class.
\item {\bf [-c iter]} \\
An exact reconstruction cannot be achieved from all
pyramidal transformations, and a few iterations may be necessary to refine
the result.
Therefore, for transformations (6,7,8,9,10,12), this option
is valid, and {\em iter} sets the  number of iterations. Generally,
three is enough. Default is no iteration.
\item {\bf [-u number\_of\_undecimated\_scales]} \\
Number of undecimated scales used in the undecimated wavelet
transform (-t 24).
By default, all scales are undecimated.
\end{itemize}

The distribution of the transforms into classes is the following:
\bn
\item cube: transform 1 to 5, and 16, and 23.
\item pyramid: transforms 6 to 13, 22.
\item half-pyramid: transform 19,20.
\item image: 14, 15, 17, 18, 25.
\item directional cube: 21,24.
\en

Options ``-x'' and ``-B'' are mutually
exclusive. If one of these
options is set, the second parameter is used also as prefix for the
creation of the band-images.  \\
Option ``-u'' is only valid if the undecimated wavelet transform (-t 24)
is selected. \\
Wavelet transforms using the Fourier transform require the input
image to be a square image, with a number of pixels which is an integer
power of 2. \\
The result is stored in a file (suffix ``.mr"),
and images (or bands) of the transformation can be extracted
by using the {\em mr\_extract} program. \\

\subsubsection*{Examples:}
\begin{itemize}
\item mr\_transform -t 10 image.d pyr\_trans.mr \\
Apply the pyramidal median transform to an image, and store the
result in pyr\_trans.mr.
\item mr\_transform -t 10 -x image.d pyr\_trans.mr \\
Same as before, but write all scales separately
in files: \\
 (band\_1\_pyr\_trans.d, band\_2\_pyr\_trans.d, etc.).
\item mr\_transform -t 8 -B image.d pyr\_wave.mr \\
Apply a pyramidal wavelet transform, and each scale is interpolated
to the input image size before being saved in a file of name
``band\_j\_pyr\_wave.d'', where j is the scale number.
\item mr\_transform -t 14 -T 4 image.d bio\_wave.mr \\
Bi-orthogonal wavelet transform using Odegard 7/9 filters.
\item mr\_transform -t 25 -l 3 image.d bio\_wave.mr \\
Integer Haar wavelet transform.
\end{itemize}
Figure~\ref{fig_ngc} shows the galaxy NGC2997, and
Figure~\ref{pl_pave_gala} its
wavelet transform using the \`a trous algorithm with four scales (three
wavelet scales, and the last smooth array). Each scale has the same size
as the original image.

\begin{figure}[htb]
\centerline{
\hbox{
\psfig{figure=fig_ngc.ps,bbllx=1.9cm,bblly=12.6cm,bburx=14.6cm,bbury=25.4cm,width=11cm,height=11cm,clip=}
}}
\caption{Galaxy NGC2997.}
\label{fig_ngc}
\end{figure}

\begin{figure}[htb]
\centerline{
\hbox{
\psfig{figure=ch1_pave_gala.ps,bbllx=2cm,bblly=9.5cm,bburx=19cm,bbury=27cm,height=11cm,width=11cm,clip=}
}}
\caption{Wavelet transform of NGC 2997 by the \`a trous algorithm.}
\label{pl_pave_gala}
\end{figure}

\subsection{How to select a filter bank?}
The (bi-) orthogonal wavelet transform (OWT) and the
undecimated wavelet transform (UWT) allow the user to select between
five different filter banks (see Table~\ref{tab_algo}, from 14-1 to 14-5),
or to use one's own filter bank. In this case, a file must contain the
coefficient values of the filters $h$ and $\tilde h$.
The file format is the Bath Wavelet Warehouse file format
(see \\
http://dmsun4.bath.ac.uk/wavelets/warehouse.html \\
for more information).
The file is an ASCII file, with the ``.wvf'' extension,
and the general format for describing wavelet filter coefficients is
the following:
\begin{verbatim}
[Range low] [Range high]
[Analysis LP filter coefficients]
.
.
.
[Range low] [Range high]
[Synthesis LP filter coefficients]
.
.
.
\end{verbatim}
For example, the Daubechies filter 4 is (the filename is dau4.wvf):
\begin{verbatim}
0 3
0.4829629131445341
0.8365163037378079
0.2241438680420134
-0.1294095225512604
0 3
0.4829629131445341
0.8365163037378079
0.2241438680420134
-0.1294095225512604
\end{verbatim}
More than thirty different filters are available at the
Bath Wavelet Warehouse.
In the \proj programs, when the option ``-t 14'' (for the OWT) or ``-t 24''
(for the UWT) are chosen, the option ``-T 6'' or  ``-T 6,filename'' is used
for defining a user filter. If no file is given, the \proj programs first
check if the default filename {\bf ``mr1.wvf''} exits in the current
directory, and if it does not exist,
the environment variable {\bf CEA\_FILTER} \index{CEA\_FILTER} is tested.
Examples of command line syntax for the wavelet transform program are:
\begin{itemize}
\item mr\_transform -t 14 -T 6,dau4 image.d dwt\_trans.mr \\
Bi-orthogonal wavelet transform, using the filters defined in
the file ``dau4.wvf''.
\item mr\_transform -t 24 -T 6 image.d uwt\_trans.mr\\
Undecimated wavelet transform, using the file {\bf ``mr1.wvf''} if it
exists, otherwise using the file name contained in the environment
variable {\bf CEA\_FILTER}.
\end{itemize}

\clearpage

\subsection{Extraction of a scale: mr\_extract}
\label{sect_extr}
\index{mr\_extract}
The program {\em mr\_extract} allows the user to extract a scale or a band from
a multiresolution transform file (suffix ``.mr").
{\bf
\begin{center}
 USAGE: mr\_extract options multiresolution\_file  output\_image
\end{center}}
where options are:
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item {\bf  [-b band\_number] } \\
 Band number to extract. Default is 1.
\item {\bf  [-s scale\_number] } \\
 Scale number to extract. Default is 1.
\item {\bf  [-x]} \\
Extract all the bands. The second parameter is used
as a prefix. If this option is set, the second parameter {\bf must} only
contain the file name, and not the complete path of the file.
\item {\bf   [-B]} \\
Interpolate the scale by block, in order to have the same
size as the original image. For scale extraction,
this option is valid only if the
 multiresolution transform is pyramidal, and is always valid for
 band extraction.
%  \item {\bf   [-i]} \\
% Interpolate the scale with a B$_3$-spline, in order to have the same
% size as the original image.  This option is valid only if
%  the multiresolution transform is pyramidal.
\end{itemize}
\subsubsection*{Examples:}
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item mr\_extract -s 1 mr\_file.mr scale\_1.d \\
Extract the first scale of the wavelet transform, and write as a .d
image of name ``scale\_1.d''. If the transform is an orthogonal transform,
it will contain all of the transform.
\item mr\_extract -b 1 mr\_file.mr band\_1.d \\
Extract the first band of the wavelet transform, and write as a .d
image of name ``band\_1.d''. If the transform is an orthogonal transform,
it will contain only the first band (in one single direction)
of the transform.
\item mr\_extract -x  mr\_file.mr toto.d\\
Creates the files ``toto\_band\_1.d'', ..., ``toto\_band\_j.d''.
\end{itemize}

\subsection{Insertion of an image: mr\_insert}
\index{mr\_insert}
The program
{\em mr\_insert} replaces a scale or a band by some image, by inserting it
in the multiresolution transform file. The scale (or the band) and the image
must have the same size.

{\bf \begin{center}
 USAGE: mr\_insert options multiresolution\_file input\_image
\end{center}}
where options are:
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item  {\bf  [-b band\_number] } \\
 Band number to insert. Default is 1.
\item  {\bf  [-s scale\_number] } \\
 Scale number to insert. Default is 1.
\end{itemize}
 {\em multiresolution\_file} is the file (.mr) which contains
the multiresolution transformation. \\
 {\em input\_image} is the image
which must replace the band image. \\
 {\em band\_number} specifies
the band number to be replaced. Default is 1.
The multiresolution transform is updated. \\
\subsubsection*{Example:}
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item mr\_insert -b 3 mr\_file.mr image.d \\
Insert an image at the third scale of the multiresolution file.
\end{itemize}

\subsection{Reconstruction: mr\_recons}
\index{mr\_recons}
The program
{\em mr\_recons}  reconstructs an image from its multiresolution
transform.
{\bf
\begin{center}
 USAGE: mr\_recons  multiresolution\_file  image\_out
\end{center}}
{\em multiresolution\_file} is the file (.mr) which contains
the multiresolution transformation, {\em output\_image} is  the
output reconstructed image.

\section{Multiresolution tools}
\subsection{Visualization: mr\_visu}
\index{mr\_visu}
All scales of the multiresolution transform are normalized to 1, and inserted
in one image, which can be
visualized by a graphics tool  (xv, IDL, etc.).
The input
file can either be an image or a multiresolution file. In the second case
options ``-t, -n", which define the type of transformation and the number of
scales, are not used. By default the output image is represented in graylevel,
with each scale normalized to 255. The normalization can be avoided using
the ``-c"
option. Two other representations are also available, the perspective one
(all scales are represented in a 3-dimensional way), and the contour one.
In the contour representation, one contour is plotted per scale. The contour
level is determined from the standard deviation of the first scale and the {\em nsigma}
parameter (which can be modified by the ``-s" option). Non-isotropic
transforms cannot be used for perspective or contour representation.

%The output can
%be also saved in a Postscript file by the "-w" option.

\begin{figure}[htb]
\centerline{
\hbox{
\psfig{figure=ch1_wave_3D.ps,bbllx=1.7cm,bblly=5.3cm,bburx=19.3cm,bbury=23.3cm,height=18cm,width=15cm,clip=}
}}
\caption{3D perspective view of NGC 2997 wavelet scales.}
\index{wavelet transform}
\label{fig_wave_3D}
\end{figure}
Figure~\ref{fig_wave_3D} shows the wavelet transform of the galaxy NGC2997,
with five scales, and using a perspective representation.

{\bf
\begin{center}
 USAGE: mr\_visu image\_in\_or\_MRfile  image\_out
\end{center}}
where options are:
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item {\bf [-t type\_of\_multiresolution\_transform]} \\
  see \ref{sect_trans}.
\item {\bf [-T type\_of\_filters]}  \\
  see \ref{sect_trans}.
\item {\bf [-L]} \\
  see \ref{sect_trans}.
\item {\bf [-n number\_of\_scales]} \\
  see \ref{sect_trans}.
\item {\bf [-V Type\_Visu]}
\begin{enumerate}
\baselineskip=0.4truecm
\item Graylevel
\item Contour
\item Perspective
\end{enumerate}
 Default is 1.
\item {\bf [-b]} \\
 Save output image as bi-level.  Only used if {\em Type\_Visu} is
equal to 2 or 3.
 Default is not to save.
\item {\bf [-c]} \\
 Do not apply a normalization on the multiresolution coefficient.
Only used if {\em Type\_Visu} is equal to 1.
% \item {\bf [-w PS\_FileName]} \\
% Save also the result in a Postscript file.
 \item {\bf [-i Increment]} \\
 Number of lines of the image which will be used. \\
 If Increment = 3, only one line out of 3 is used. \\
 Only used if  {\em Type\_Visu} is equal to 3. \\
The default value is 1.
\item {\bf [-s nsigma]} \\
Plot contour at {\em nsigma}$*$Sigma if  {\em Type\_Visu} equal to 2. \\
Threshold value upper nsigma$*$Sigma if  {\em Type\_Visu} equal to 3. \\
Default is 3.
\end{itemize}

\subsection{Multiresolution support creation: mr\_support}
\label{sect_support}
\index{mr\_support}
\index{multiresolution support}
The multiresolution support of an image describes in a
logical or boolean way if an image $I$ contains information at a
given scale $j$ and at a given position $(x,y)$ \cite{starck:sta95_1,starck:mur95_2,starck:mur93}.
If $M^{(I)}(j,x,y) = 1$ (or {\it true}), then $I$ contains information at
scale $j$ and at the position $(x,y)$.
$M$ depends on several parameters:
\begin{itemize}
\item The input image.
\item The algorithm used for the multiresolution decomposition.
\item The noise.
\item All constraints we additionally want the support to satisfy.
\end{itemize}
Such a support results from the data, the treatment (noise
estimation, etc.), and from knowledge on our part of the objects contained
in the data (size of objects, linearity, etc.). In the most general case,
a priori information is not available to us.

The multiresolution support of an image is computed in several steps:
\begin{itemize}
\item Step 1 is to compute the wavelet transform of the image.
\item Binarization of each scale leads to the multiresolution support.
\item A priori knowledge can be introduced by modifying the support.
\end{itemize}

The last step depends on the knowledge we have of our images.
For instance, if we know there is no interesting object smaller or larger
than a
given size in our image, we can suppress, in the support, anything which is
due to that kind of object. This can often be done conveniently by  the use of
mathematical morphology. Negative detections can sometimes be suppressed too.
In the most general setting, we naturally have
no information to add to the multiresolution support.

A complete description of how to derive the multiresolution support from
the multiresolution coefficients is given in \cite{starck:sta95_1,starck:book98}. Several types
of noise are taken into account:
\begin{enumerate}
\item Gaussian noise (default case).
\item Poisson noise.
\item Poisson and Gaussian noise
\end{enumerate}
For the last two transforms, a variance stabilization transform is applied
to the input image, which allows us to obtain a transformed image which
contains  Gaussian noise with a standard deviation equal to one. After
transformation of the image (or the variance stabilized image), each
wavelet coefficient $w_j(x,y)$ is compared to the standard deviation at
the scale $j$, $\sigma_j$ multiplied by a coefficient $N_\sigma$.
This coefficient specifies
the confidence interval we want. If $\mid w_j(x,y) \mid \
> N_\sigma \sigma_j$, the
probability that $w_j(x,y)$ is due to noise is less than 1\%.

\begin{center}
 USAGE: mr\_support options image\_in multiresolution\_file\_out
\end{center}
 where options are:
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item {\bf [-t type\_of\_multiresolution\_transform]} \\
  see \ref{sect_trans}.
\item {\bf [-T type\_of\_filters]}  \\
  see \ref{sect_trans}.
\item {\bf [-L]} \\
  see \ref{sect_trans}.
\item {\bf [-u]} \\
  see \ref{sect_trans}.
\item {\bf [-g sigma]} \\
 The image contains Gaussian noise, and the standard deviation is
given by {\em sigma }. The option should be set only if the user
knows the standard deviation of the noise.
\item {\bf [-p]} \\
 If this option is set, the image is assumed to contain Poisson noise.
 The default is Gaussian noise, and its standard
 deviation is automatically estimated.
\item {\bf [-c gain,sigma,mean]} \\
The noise is composed of a Gaussian and Poisson component.  This is the
case for a CCD detector. \\
noise = Poisson noise and read-out noise \\
gain = inverse of the CCD gain (unit: $DN/e^{-}$) \\
sigma = standard deviation of the read-out noise (unit: DN) = CCD RON / CCD Gain\\
mean = mean of the read-out noise \\
If this option is set, \\
Noise = Poisson and Gaussian read-out noise \\
This is generally the case with a CCD. \\
Note: these parameters must be separated by a comma without spaces. \\
Example: -c 0.133,1.733,0. \\
If the mean, or sigma and the mean, are omitted, default values are 0. \\
The gain cannot be omitted if parameter -c is used.
\item {\bf [-n number\_of\_scales]} \\
Number of scales used in the multiresolution transform. Default is 4.
\item {\bf [-s NSigma]} \\
The detection level at each scale is determined by the product
of the standard deviation of the noise by the {\em NSigma}.
{\em NSigma} fixes the confidence interval we want. By default,
{\em NSigma} is equal to 3.
\item {\bf [-k]} \\
If this option is set, isolated pixels in the multiresolution support
are suppressed. If the PSF is large, then isolated pixels are
certainly residual noise, or spurious impulse noise objects,
or artifacts. Then
we can suppress these pixels in the support. This option
can only be used with transformations of class 1 and 2 (which
produce cubes and pyramids). Default is not to do this.
\item {\bf [-l]} \\
If this option is set, the morphological dilation operator is
applied on each scale of the multiresolution support. This option
can only be used with transformations of class 1 and 2 (which
produce cubes and pyramids). Default is not to do this.
\item {\bf [-w support\_file\_name]} \\
 If this option is set, a synthetic image is created from the
multiresolution support. Default is not to do this.
\end{itemize}
\subsubsection*{Examples:}
\begin{itemize}
\item mr\_support -w support.d -t 14 image.d  mr\_file.mr  \\
Creates the multiresolution support of an image, assuming the noise is
Gaussian (its standard deviation is automatically estimated). A .d image
is created from the support.
\item mr\_support -w support.d -p image.d  mr\_file.mr  \\
Creates the multiresolution support  of an image, assuming the noise follows
a Poisson distribution. The chosen multiresolution transform
is the default one
(\`a trous algorithm). A~.d image is created from the support. This is not
a boolean image as in the previous example, because it is obtained by
addition of all boolean scales of the multiresolution support.
\end{itemize}

\subsection{Statistical information: mr\_info}
\index{mr\_info}
Program
{\em mr\_info} gives statistical multiresolution information for an image
(min, max, sigma, mean, skewness and kurtosis at each scale).
If the transform belongs to
class 1 or 2
(output is a cube or a pyramid), then additional options allow the
significant coefficients to be analyzed, and furnish
\begin{itemize}
\item the percentage of significant coefficients at each scale,
\item the number of maxima at each scale,
\item the number of structures detected at each scale,
\item the size in pixels of the largest  detected structure at each scale.
\end{itemize}
If an output file name is given, a 2D array containing statistical information
is saved on the disk in the FITS format. The output file is a
fits file containing a two dimensional array $T[J-1,5]$ ($J$ being the
number of bands), with the following the syntax:
\begin{itemize}
\baselineskip=0.4truecm
\item $T[j,0] = $ standard deviation of the jth ridgelet band.
\item $T[j,1] = $ skewness of the jth ridgelet band.
\item $T[j,2] = $ kurtosis of the jth ridgelet band.
\item $T[j,3] = $ minimum of the jth ridgelet band.
\item $T[j,4] = $ maximum of the jth ridgelet band.
\end{itemize}

{\bf
\begin{center}
 USAGE: mr\_info option image\_in [out\_statfile]
\end{center}}
 where options are
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item {\bf [-t type\_of\_multiresolution\_transform]}
\item {\bf [-T type\_of\_filters]}
\item {\bf [-L]}
\item {\bf [-u]}
\item {\bf [-p]}
\item {\bf [-g sigma]}
\item {\bf [-c gain,sigma,mean]}
\item {\bf [-n number\_of\_scales]}
\item {\bf [-s NSigma]}
\item {\bf [-k]}
\item {\bf [-l]}
\item {\bf [-a]} \\
Significant structures analysis. Default is not to do this.
\end{itemize}
Options [t,T,L,u,p,g,c,n,s,k,l] are the same as those described in
section~\ref{sect_support}. \\
Options [a,p,g,c,s,k,l] are not
allowed for transforms producing images (class 4). \\
If any of
the options [a,p,g,c,s,k,l] is set, a significant structures
analysis is performed.
\subsubsection{Examples:}
\begin{itemize}
\item mr\_info image.d  \\
Writes on the standard output the minimum value, the maximum value,
the mean value, and the standard deviation of each scale.
\item mr\_info -a  image.d \\
Same as before, but computes also the percentage of significant
pixels per scale, and the number of maxima, the number of
structures, and the size (in pixels) of the bigger structures.
\end{itemize}

\subsection{Multiresolution segmentation: mr\_segment}
\index{mr\_segment}
Program
{\em mr\_segment} binarizes all the scales of the wavelet
transform of an image, and applies tp each one
a segmentation procedure.
Each scale of the output multiresolution file contains
an image in which values are ranged between 0 and $R_j$,
where $R_j$ is the number of regions found at this scale.
{\bf
\begin{center}
 USAGE: mr\_segment option image\_in mr\_file\_out
\end{center}}
where {\em mr\_file\_out} is the multiresolution file (``.mr")
which contains the segmented scales. Options are
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item {\bf [-t type\_of\_multiresolution\_transform]}
\item {\bf [-T type\_of\_filters]}
\item {\bf [-L]}
\item {\bf [-u]}
\item {\bf [-p]}
\item {\bf [-g sigma]}
\item {\bf [-c gain,sigma,mean]}
\item {\bf [-n number\_of\_scales]}
\item {\bf [-s NSigma]}
\item {\bf [-k]}
\item {\bf [-l]}
\end{itemize}
These options are the same as those described in
section \ref{sect_support}.
\subsubsection*{Example:}
\begin{itemize}
\item mr\_segment -g 1. image.d  mr\_file\\
Applies the default multiresolution transform (\`a trous algorithm)
to the input image, thresholds the scale assuming Gaussian noise
with a standard deviation equal to 1, and applies a segmentation
on each scale.
\end{itemize}

\subsection{Noise analysis: mr\_sigma}
\index{mr\_sigma}
Program
{\em mr\_sigma} estimates the standard deviation of  Gaussian
noise in an image \cite{starck:sta98_3}. Several methods can
be used. One of them
(Multiresolution support) is based on the multiresolution.
If the option ``-p" is set, the noise is considered to be
a combination of both Gaussian and Poisson noise. In this
case, the program seeks the standard deviation of the Gaussian
component of the noise using the generalized Anscombe transform.
{\bf
\begin{center}
 USAGE: mr\_sigma options image
\end{center}}
where options are:
 \begin{itemize}
 \baselineskip=0.4truecm
\itemsep=0.1truecm
\item {\bf [-m type\_of\_methods]}
{\small
\begin{enumerate}
 \item 3\_sigma\_clipping
 \item Median + 3\_sigma\_clipping
 \item Bspline + 3\_sigma\_clipping
 \item Multiresolution support
 \item Block method
 \item MAD method (Median of Absolute Deviation)
\end{enumerate}
}
Default is 2.
\item {\bf [-n number\_of\_scales]}
Number of scales used in the multiresolution transform.
Default is 4.
\item {\bf [-p gain]} \\
Computes the standard deviation of the Gaussian part of the
noise (RON), knowing the gain, in the case where the noise is
composed of Poisson and Gaussian noise.
\end{itemize}
\subsubsection*{Examples:}
\begin{itemize}
\item mr\_sigma image.d  \\
Computes the standard deviation of the Gaussian noise contained in the
input image, using the default method.
\item mr\_sigma -p 1 image.d  \\
Computes the standard deviation of the read-out noise, assuming a gain
equal to 1.
\end{itemize}

\subsection{Background subtraction: mr\_background}
\index{mr\_background}

\label{sect_bgr}
 Program
{\em mr\_background} subtracts the background from an image. The
background is considered as being the last scale of the pyramidal
median transform. The number of scales is automatically calculated in order
to have the last scale with a size lower than or equal to 16 $\times$
16 pixels.
This value of 16 can
be modified by the ``-n" option.
{\bf
\begin{center}
 USAGE: mr\_background option image\_in image\_out
\end{center}}
where options are
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item {\bf [-n number\_of\_pixels]} \\
Number of pixels used for background estimation. Default is 16.
\item {\bf [-w background\_file\_name]} \\
Creates the background image and writes it on disk.
\item {\bf [-W MedianWindowSize]} \\
Median window size using in the Pyramidal Median Transform.
\end{itemize}
\subsubsection*{Examples:}
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item mr\_background image.d  im\_bgr\_free.d\\
Computes the pyramidal median transform of the input image,
interpolates the last scale to the image, and subtracts it from
the input image.
\item mr\_background -n 32 -w bgr.d image.d  \\
The number of scales is calculated so that the last scale
will have less than 32 pixels in at least one dimension.
\end{itemize}

\subsection{Image comparison: mr\_compare}
\index{mr\_compare}
Program
{\em mr\_compare} compares a set of images to a reference image.
These comparisons are performed both in the direct space and in the
wavelet space. By default, only structures above the noise are considered
for the comparison. If the {\em NSigma} parameter is set to zero, then
all wavelet coefficients are taken into account. If the {\em number\_of\_scales}
parameter is set to 1, then the comparison is only done in the direct space.
For all images, the minimum, the maximum, the mean and standard deviation are
printed. For each image {\em ima\_i}, the wavelet transform of {\em ima\_i} and
{\em ima\_error (= ref\_image -- ima\_i)} is calculated. For each scale,
several results are given:
\begin{itemize}
\item the percentage of significant wavelet coefficients of {\em ima\_i}.
\item the correlation between the wavelet coefficients of {\em ima\_i}
and {\em ref\_image}.
\item the standard deviation of absolute values of the wavelet coefficients of {\em ima\_error}.
\item the root mean square (RMS) of the wavelet coefficients of {\em ima\_error}.
\item the minimum value of the wavelet coefficients of {\em ima\_error}.
\item the maximum value of the wavelet coefficients of {\em ima\_error}.
\item the signal-to-noise ratio (SNR) using the wavelet coefficients of the
reference image and those of the error image:
\[ SNR = \frac{\sigma^2(W_{ImaRef})}{\sigma^2(W_{Error})}  \]
\item the signal-to-noise ratio in dB (SNRb)
\[ SNRb = 10 log_{10}(SNR) \]
\end{itemize}
{\bf
\begin{center}
USAGE: mr\_compare option ref\_ima ima1 [ima2, [ima3, ...]]
\end{center}}
where options are
\begin{itemize}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item {\bf [-t type\_of\_multiresolution\_transform]} \\
See \ref{sect_trans}. But only transforms 1 to 13
(isotropic transforms) are available.
\item {\bf [-p]} \\
\item {\bf [-g sigma]} \\
\item {\bf [-c gain,sigma,mean]} \\
\item {\bf [-n number\_of\_scales]}
\item {\bf [-s NSigma]}
\end{itemize}
These options are the same as those described in
section \ref{sect_support}.
\subsubsection*{Examples:}
\begin{itemize}
\item mr\_compare -n 1 image\_ref.d ima\_1.d ima\_2.d \\
Computes the images {\em ima\_1} and {\em ima\_2} to {\em image\_ref}
\  in the direct space only.
\item mr\_compare -s 0. image\_ref.d ima\_1.d ima\_2.d  \\
Computes the images {\em ima\_1} and {\em ima\_2} to {\em image\_ref}
\ taking into account all wavelet coefficients.
\end{itemize}


\section{Risk-Neutral Policy Gradient Theorem}
In the previous sections, we saw that the core of a policy gradient algorithm consist in the approximation of the objective function gradient. In this section we present the \emph{policy gradient theorem} \cite{sutton1999policy}, which shows that the gradient can be rewritten in a form suitable for estimation from experience aided by an approximate action-value or advantage function. First, we will prove the theorem in the risk-neutral setting for which it was originally proposed. In particular, we will see how the GPOMDP algorithm can be easily derived by this result via a Monte Carlo approximation. Moreover, we will illustrate a class of actor-critic algorithms \cite{konda1999actor} that, in addition to a parametric approximation of the policy, also exploit an approximation of the action-value function or of an advantage function to reduce the variance of the gradient estimate. In particular, we review the powerful idea of compatible function approximation \cite{sutton1999policy} which assures the convergence to a local optimum of the objective function. Finally, we discuss the natural policy gradient idea \cite{kakade2001natural}, which forms the basis of many state-of-the-art algorithms. The extension to the risk-sensitive setting will be done in the next chapter.

\subsection{Theorem Statement and Proof}
\begin{theorem}[Risk-Neutral Policy Gradient]
\label{thm:risk_neutral_policy_gradient}
	Let $\pi_\theta$ be a differentiable policy. The policy gradient for the average reward formulation is given by
	\begin{equation}
		\nabla_\theta \rho(\theta) =
		\E[\substack{S \sim d^\theta\\A \sim \pi_\theta}]{\nabla_\theta\log
		\pi_\theta(S,A) Q_{\theta}(S, A)}
	\end{equation}
	where $d^\theta$ is the stationary distribution of the Markov chain induced by $\pi_\theta$. The policy gradient for the start value formulation is given by
	\begin{equation}
		\nabla_\theta J_{\text{start}}(\theta) =
		\E[\substack{S \sim d_\gamma^\theta(s_0, \cdot)\\A \sim \pi_\theta}]{\nabla_\theta\log
		\pi_\theta(S,A) Q_{\theta}(S, A)}
	\end{equation}
	where $d_\gamma^\theta(s_0, \cdot)$ is the $\gamma$-discounted visiting distribution over states starting from the initial state $s_0$ and following policy $\pi_\theta$
		\begin{equation}
			d_\gamma^\theta(s, x) = \sum_{k=0}^{\infty} \gamma^k \calP_\theta^{(k)}(s, x)
		\end{equation}
\end{theorem}
\begin{proof}
	We first prove the result for the average-reward formulation and then for the start state formulation. From the basic relation between state-value function and action-value function, we have
	\begin{equation*}
		\begin{split}
			\nabla_\theta V_\theta(s) &= \nabla_\theta \int_{\A} \pi_\theta(s,a) Q_\theta(s,a) da\\
				&= \int_{\A} \left[ \nabla_\theta \pi_\theta(s,a) Q_\theta(s,a) + \pi_\theta(s,a) \nabla_\theta Q_\theta(s,a)\right] da
		\end{split}
	\end{equation*}
	Using the Bellman expectation equation for $Q_\theta$
	\begin{equation*}
		\begin{split}
			\nabla_\theta Q_\theta(s,a) &= \nabla_\theta \left[ \calR(s,a) - \rho_\theta + \int_{\S} \calP(s,a,s') V_\theta(s') ds' \right]\\
			&= -\nabla_\theta \rho_\theta + \int_{\S} \calP(s,a,s') \nabla_\theta V_\theta(s') ds'
		\end{split}
	\end{equation*}
	Hence, plugging in the first equation
	\begin{equation*}
		\begin{split}
			\nabla_\theta V_\theta(s) &= \int_{\A} \nabla_\theta \pi_\theta(s,a) Q_\theta(s,a) da - \nabla_\theta \rho_\theta + \int_\A \pi_\theta(s,a) \int_{\S} \calP(s,a,s') \nabla_\theta V_\theta(s') ds'
		\end{split}
	\end{equation*}
	Integrating both sides with respect to the stationary distribution $d^\theta$ and noting that, because of stationarity,
	\begin{equation*}
		\int_{\S} d^\theta(s) \int_{\A} \pi(s,a) \int_{\S} \calP(s,a,s') \nabla_\theta V(s') ds' da ds = \int_{\S} d^\theta(s) \nabla_\theta V_\theta(s) ds
	\end{equation*}
	we obtain the result
	\begin{equation*}
		\begin{split}
		\nabla_\theta \rho_\theta &= \int_{\S} d^\theta(s) \int_{\A} \nabla_\theta \pi_\theta(s,a) Q_\theta(s,a) da ds\\
		&= \int_{\S} d^\theta(s) \int_{\A} \pi_\theta(s,a) \nabla_\theta \log\pi_\theta(s,a) Q_\theta(s,a) da ds\\
		&= \E[\substack{S \sim d^\theta\\A \sim \pi_\theta}]{\nabla_\theta\log
				\pi_\theta(S,A) Q_{\theta}(S, A)}
		\end{split}
	\end{equation*}
	Let us now prove the theorem for the start state formulation. The first step is exactly the same. Hence, using the Bellman expectation equation for $Q_\theta$
	\begin{equation*}
		\begin{split}
			\nabla_\theta Q_\theta(s,a) &= \nabla_\theta \left[ \calR(s,a) + \gamma \int_{\S} \calP(s,a,s') V_\theta(s') ds' \right]\\
			&= \gamma \int_{\S} \calP(s,a,s') \nabla_\theta V_\theta(s') ds'
		\end{split}
	\end{equation*}
	we obtain
	\begin{equation*}
		\begin{split}
			\nabla_\theta V_\theta(s) &= \int_{\A} \left[ \nabla_\theta \pi_\theta(s,a) Q_\theta(s,a) + \gamma \int_{\S} \calP(s,a,s') \nabla_\theta V_\theta(s') ds' \right] da\\
			&= \int_\S  \sum_{k=0}^{\infty} \gamma^k \calP_\theta^{(k)}(s, x) \int_{\A} \nabla_\theta \pi_\theta(x,a) Q_\theta(x,a) da dx
		\end{split}
	\end{equation*}
	after unrolling $\nabla_\theta V_\theta$ infinite times and denoting by $\calP_\theta^{(k)}(s, x)$ the probability of going from state $s$ to state $x$ in $k$ steps under policy $\pi_\theta$.
	Defining the $\gamma$-discounted visiting distribution of state $x$ starting from state $s$ as
	\begin{equation*}
		d_\gamma^\theta(s, x) = \sum_{k=0}^{\infty} \gamma^k \calP_\theta^{(k)}(s, x)
	\end{equation*}
	we have the result
	\begin{equation*}
		\begin{split}
			\nabla_\theta V_\theta(s) &= \int_\S d_\gamma^\theta(s, x) \int_{\A} \nabla_\theta \pi_\theta(x,a) Q_\theta(x,a) da dx\\
			&= \E[\substack{S \sim d_\gamma^\theta(s_0, \cdot)\\A \sim \pi_\theta}]{\nabla_\theta\log \pi_\theta(S,A) Q_{\pi_\theta}(S, A)}
		\end{split}
	\end{equation*}
\end{proof}
The action-value function is typically unknown and needs to be approximated.
As for the REINFORCE algorithm, we can subtract a state-dependent baseline from the action-value function without changing the value of the expectation. Indeed
\begin{equation*}
	\begin{split}
	\E[\substack{S \sim d^\theta\\A \sim \pi_\theta}]{\nabla_\theta\log
			\pi_\theta(S,A) B_\theta(S)}
	&= \int_\S d^\theta(s) \int_\A \pi_\theta(s,a) \nabla_\theta\log
				\pi_\theta(s,a) B_\theta(s) da ds\\
	&= \int_\S d^\theta(s)  B_\theta(s) \int_\A \nabla_\theta \pi_\theta(s,a) da ds\\
	&= \int_\S d^\theta(s)  B_\theta(s)  \nabla_\theta  \underbrace{\int_\A  \pi_\theta(s,a) da}_{= 1} ds = 0
	\end{split}
\end{equation*}
Hence, we can rewrite the policy gradient for the average reward formulation as
\begin{equation}
\label{eq:pg_theorem_baseline}
	\nabla_\theta \rho(\theta) =
	\E[\substack{S \sim d^\theta\\A \sim \pi_\theta}]{\nabla_\theta\log
	\pi_\theta(S,A) \left(Q_{\pi_\theta}(S, A) - B_\theta(S)\right)}
\end{equation}
and similarly for the start state formulation. This result can be used as the starting point to derive several policy gradient methods that use different approximation of the action-value function.

\subsection{GPOMDP}
For an episodic \gls{MDP}, the action-value function can be estimated with the total return obtained on a sample trajectory
\begin{equation*}
	Q_\theta(s_0,a_0) \approx \sum_{t=0}^{T^{(m)}} \gamma^t r_{t+1}^{(m)}
\end{equation*}
Combining this remark with a Monte Carlo approximation of Eq. (\ref{eq:pg_theorem_baseline}), we obtain the \gls{GPOMDP} gradient estimate (\ref{eq:GPOMDP}). Therefore, the \gls{GPOMDP} algorithm is a simple application of the policy gradient algorithm, which is much more general than the likelihood ratio technique used to derive the algorithm in the previous sections. In the next subsections we present other ways to exploit the policy gradient theorem to design efficient learning algorithms.

\subsection{Actor-Critic Policy Gradient}
A baseline should ideally measure the typical return obtained by an agent in a certain state when following a certain policy. Therefore, it becomes natural to use the state-value function as a benchmark for the action-value function \cite{grondman2012survey}. Eq. (\ref{eq:pg_theorem_baseline}) thus becomes
\begin{equation}
\label{eq:actor_critic_pg}
	\nabla_\theta \rho(\theta) =
		\E[\substack{S \sim d^\theta\\A \sim \pi_\theta}]{\nabla_\theta\log
		\pi_\theta(S,A) A_\theta(S,A)}
\end{equation}
where we introduced the advantage function
\begin{equation}
	A_\theta(s,a) = Q_\theta(s,a) - V_\theta(s)
\end{equation}
which measures how good is to take action $a$ in state $s$ compared to simply following the policy $\pi_\theta$. The advantage function is unknown and should be estimated from samples of the \gls{MDP}. \emph{Actor-Critic} algorithms consist in all those methods that employ an approximation of the advantage function or of the value function, also known as critic, to estimate the policy gradient. These methods thus maintain two sets of parameters: a \emph{critic} that updates the action-value function parameters $\psi$ and an \emph{actor} that updates the policy parameters $\theta$ in the direction suggested by the critic. The general structure for an online actor-critic algorithm is reported in Algorithm \ref{algo:actor_critic}.\\
\begin{algorithm}[t]
	\caption{Generic structure for an online actor-critic algorithm}
	\label{algo:actor_critic}
	\begin{algorithmic}[0]
		\Require{\\
			\begin{itemize}
				\item Initial actor parameters $\theta_0$,
				\item Initial critic parameters $\psi_0$,
				\item Learning rate $\{\alpha_k\}$
			\end{itemize}
		}
		\Ensure Approximation of the optimal policy $\pi_{\theta^*} \approx \pi_*$
		\begin{algorithmic}[1]
		\Repeat
			\State Observe tuple $<s_k, a_k, r_{k+1}, s_{k+1}>$ sampled from the MDP.
			\State Update critic parameters $\psi_{k+1}$ using a value-based method.
			\State Estimate policy gradient as $\widehat{g}_k^\text{AC} = \nabla_\theta \log \pi_{\theta_k}(s_k, a_k) \widehat{A}_{\psi_{k+1}}(s_k, a_k)$
			\State Update actor by gradient ascent $\theta_{k+1} = \theta_k + \alpha_k \widehat{g}_k^\text{AC}$.
			\State $k \leftarrow k + 1$
		\Until{converged}
		\end{algorithmic}
	\end{algorithmic}
\end{algorithm}
There are two possible approaches to estimate the advantage function: the first one is to estimate both the state-value function $\widehat{V}_\psi \approx V_\theta$ and the action-value function $\widehat{Q}_\xi \approx Q_\theta$ and derive an estimate for the advantage function $\widehat{A}_{\psi, \xi} = \widehat{Q}_\xi - \widehat{V}_\psi \approx A_\theta$. The second approach directly approximates the advantage function $\widehat{A}_\psi \approx A_\theta$ and is usually preferred, since it allows us to maintain only one critic. Actor-critic algorithms typically use a \gls{TD} algorithm to update an approximation of the value function $\widehat{V}_\psi \approx V_\theta$, from which we can derive an approximation for the advantage function. Assume that the true state-value function $V_\theta$ is given. Then the \gls{TD} error
\begin{equation}
	\delta_\theta = R - \rho_\theta + V_\theta(S') - V_\theta(S)
\end{equation}
is an unbiased estimate of the advantage function. Indeed
\begin{equation*}
	\begin{split}
		\E{\delta_\theta | S = s, A = a} &= \E{R - \rho_\theta + V_\theta(S') | S = s, A = a} - V_\theta(s)\\
		&= Q_\theta(s,a) - V_\theta(s)\\
		&= A_\theta(s,a)
	\end{split}
\end{equation*}
By a simple conditioning argument, the policy gradient can be rewritten as
\begin{equation}
	\nabla_\theta \rho(\theta) =
			\E[\substack{S \sim d^\theta\\A \sim \pi_\theta}]{\nabla_\theta\log
			\pi_\theta(S,A) \delta_\theta}
\end{equation}
In practice, we can use an approximate \gls{TD} error
\begin{equation}
\label{eq:td_error}
	\widehat{\delta}_k = r_{k+1} - \widehat{\rho}_k + \widehat{V}_{\psi_k}(s_{k+1}) - \widehat{V}_{\psi_k}(s_k)
\end{equation}
where $\widehat{\rho}_k$ is an estimate of the average reward. Hence, we can easily obtain an approximation of the policy gradient by
\begin{equation}
	\widehat{g}_\text{TD(0)}^k = \nabla_\theta\log \pi_\theta(s_k,a_k) \widehat{\delta}_k
\end{equation}
which can be used to update the critic parameter in the gradient ascent direction.
On the other hand, the critic parameters can be updated using a TD(0) temporal difference scheme
\begin{equation}
	\psi_{k+1} = \psi_k + \alpha_k \widehat{\delta}_k \nabla_\psi \widehat{V}_{\psi_k}(s_k)
\end{equation}
The discussion can be easily extended to more complex value-based methods for estimating the critic, such as the backward-view TD($\lambda$). This method employs eligibility traces to assign credit for the rewards obtained by the agent to all previous states and actions. More formally, the policy parameters update rule becomes
\begin{equation}
	\theta_{k+1} = \theta_k + \alpha_k \widehat{\delta}_k e_k
\end{equation}
where the eligibility trace $e_k$ is defined by the following recursive equation
\begin{equation}
	e_{k+1} = \lambda e_k + \nabla_\theta\log \pi_{\theta_k}(s_k,a_k)
\end{equation}
$0 \leq \lambda \leq 1$ is a parameter that manages the amount of bootstrap: for $\lambda = 0$ the method is the standard temporal difference scheme while for $\lambda = 1$ the method is equivalent to Monte Carlo. In the rest of the thesis we will refer to this algorithm as \gls{ARAC} and its pseudocode is reported in Algorithm \ref{algo:actor_critic_td}.
\begin{algorithm}[t!]
	\caption{Average-Reward Actor Critic}
	\label{algo:actor_critic_td}
	\begin{algorithmic}[0]
		\Require{\\
			\begin{itemize}
				\item Initial actor parameters $\theta_0$,
				\item Initial critic parameters $\psi_0$,
				\item Learning rate $\{\alpha_k\}$
			\end{itemize}
		}
		\Ensure Approximation of the optimal policy $\pi_{\theta^*} \approx \pi_*$
		\begin{algorithmic}[1]
		\State Initialize $k = 0$ and the eligibility trace $e_{-1} = 0$
		\Repeat
			\State Observe tuple $<s_k, a_k, r_{k+1}, s_{k+1}>$ sampled from the MDP.
			\State Compute TD error $\widehat{\delta}_k = r_{k+1} - \widehat{\rho}_k + \widehat{V}_{\psi_k}(s_{k+1}) - \widehat{V}_{\psi_k}(s_k) $
			\State Update critic parameters $	\psi_{k+1} = \psi_k + \alpha_k \widehat{\delta}_k \nabla_\psi \widehat{V}_{\psi_k}(s_k)$
			\State Update eligibility trace $e_{k+1} = \lambda e_k + \nabla_\theta\log \pi_{\theta_k}(s_k,a_k)$
			\State Update actor parameters $\theta_{k+1} = \theta_k + \alpha_k \widehat{\delta}_k e_k $.

			\State $k \leftarrow k + 1$
		\Until{converged}
		\end{algorithmic}
	\end{algorithmic}
\end{algorithm}

\subsection{Compatible Function Approximation}
In the previous sections we saw that a critic may reduce the variance of thew policy gradient estimate. However, this is achieved at the cost of introducing a bias in the approximation which, in certain cases, may endanger the convergence of the method to a good solution. Luckily, the \emph{compatible function approximation}  \cite{sutton1999policy} avoids introducing any bias by a careful choice of the critic. This technique consists in estimating the advantage function with a linear regression on a family of suitable basis functions consisting of the gradient of the likelihood score
\begin{equation}
	\widehat{A}_\theta (s,a) = \psi^T \nabla_\theta \log \pi_\theta(s,a)
\end{equation}
The power of these basis functions becomes apparent in the following theorem
\begin{theorem}[Compatible Function Approximation]
	If
	\begin{enumerate}[label={\roman*)}]
		\item the advantage function approximator is compatible to the policy
		\begin{equation}
			\nabla_\psi A_\psi(s,a) = \nabla_\theta \log \pi_\theta(s,a)
		\end{equation}
		\item the advantage function parameters $\psi$ minimize the mean square error
		\begin{equation}
			\theta = \argmin_\psi \E[\pi_\theta]{\left( A_{\pi_\theta}(S,A) - \widehat{A}_\psi(S,A)\right)^2}
		\end{equation}
	\end{enumerate}
	Then the policy gradient is exact
	\begin{equation}
		\nabla_\theta J(\theta) = \E[\pi_\theta]{\nabla_\theta \log \pi_\theta(S,A) \widehat{A}_\psi(S,A)}
	\end{equation}
\end{theorem}
Hence, the true advantage function in the policy gradient formula can be replaced by the compatible function approximation without changing the policy gradient value. A linear regression may seem a too simple representation of the advantage function. However, one must bear in mind that this approximation is only used to reduce the variance of the policy gradient estimate and should not be expected to provide an accurate representation of the true-value function. This technique has proved useful in many application.

\subsection{Natural Policy Gradient}
Despite all the advances in the variance reduction techniques, these methods still tend to perform surprisingly poorly. Even when applied to simple examples with rather few states, where the gradient can be determined very accurately, they turn out to be quite inefficient. Typically, one of the reasons of this behavior is the presence of large plateaus in the expected return landscape where the gradients are small and often do not point directly towards the optimal solution. In this context, the steepest ascent with respect to the Fisher information metric, called the \emph{natural policy gradient}, turns out to be significantly more efficient than normal gradients for such
plateaus. This technique was first proposed in the reinforcement learning setting in \cite{kakade2001natural} and later applied to actor-critic algorithms in in \cite{peters2008reinforcement}. The following properties of natural policy gradient make it one of the more reliable policy-based methods,
\begin{enumerate}[label={\roman*)}]
	\item Convergence to a local minimum is guaranteed.
	\item By choosing a more direct path to the optimal solution in parameter space, the natural gradient typically has faster convergence and avoids premature convergence of ``vanilla gradients''.
	\item  The natural policy gradient can be shown to be covariant, or equivalently independent of the coordinate frame chosen for expressing the policy parameters.
	\item As the natural gradient analytically averages out the influence of the stochastic policy (including the baseline of the function approximator), it requires fewer data points for a good gradient estimate than ``vanilla gradients’'.
\end{enumerate}

\begin{figure}[t]
    \centering
    \includegraphics[width=1\linewidth]{Images/4_0_euclidean_vs_natural}
	\caption[``Vanilla'' policy gradient vs. natural policy gradient.]{``Vanilla'' policy gradient (left) vs. natural policy gradient (right). The main difference is how the two approaches punish the change in parameters. This distance is 	indicated by the blue ellipses in the contour plot while the dashed lines show the expected return. Obtaining a gradient then corresponds to finding a vector pointing from the center of the ellipses to the location with maximum expected return on the ellipse. A vanilla policy gradient (a) considers a change in all parameters as equally distant, thus, it is a search for a maximum on a circle while the natural gradient (b) uses scales determined by the Fisher information which results in a reduction in exploration. The slower reduction in exploration results into a faster convergence to the optimal policy \cite{peters2008reinforcement}.}
\end{figure}

\subsubsection{Formalism of Natural Policy Gradients}
Policy gradient methods improve the policy $\pi_\theta$ by iteratively applying ``small'' changes $\Delta \theta$ to the policy parameters $\theta$. However, the meaning of ``small'' is ambiguous. For instance, when working with an Euclidean metric, the size of this update $\norm{\Delta \theta} = \sqrt{\Delta\theta^T \Delta\theta}$ and therefore the update size depends on the parameterization of the policy, which often results in unnaturally slow learning even if higher-order gradient methods were employed. This problem poses the question whether we can achieve a covariant gradient descent, meaning a gradient descent with respect to an invariant measure of the closeness between the current policy and the updated policy based upon the distribution of the paths generated by each of these. Standard measures of distance between probability distributions are the Kullbach-Leibler divergence $d_{\text{KL}}(p_\theta(h)\ ||\ p_{\theta + \Delta \theta}(h))$ and the Hellinger distance. These two distances can be approximation in first instance by a second-order Taylor expansion
\begin{equation*}
	d_{\text{KL}}(p_\theta(h)\ ||\ p_{\theta + \Delta \theta}(h)) \approx \frac{1}{2} \Delta\theta^T F_\theta \Delta\theta
\end{equation*}
where $F_\theta$ is the Fischer information matrix
\begin{equation}
	\begin{split}
	F_\theta &= \int_{\H} p_\theta(h) \nabla_\theta\log p_\theta(h) \nabla_\theta\log p_\theta(h)^T dh\\
		&= \E[H\sim p_\theta]{\nabla_\theta\log p_\theta(H) \nabla_\theta\log p_\theta(H)^T}
	\end{split}
\end{equation}
The goal is to find the optimal update $\Delta \theta$ for the policy parameters so as to maximize the objective function, under the constraint that the new policy must be in a radius $\epsilon$ from the previous policy with respect to the Kullbach-Leibler divergence
\begin{equation*}
	\begin{cases}
		\max_{\Delta\theta} J(\theta + \Delta \theta) \approx J(\theta) + \Delta \theta^T \nabla_\theta J(\theta)\\
		\text{s.t.}\ d_{\text{KL}}(p_\theta(h)\ ||\ p_{\theta + \Delta \theta}(h)) \approx \frac{1}{2} \Delta\theta^T F_\theta \Delta\theta < \epsilon\\
	\end{cases}
\end{equation*}
The optimal solution is given by
\begin{equation*}
	\Delta \theta = \alpha_n F_\theta^{-1} \nabla_\theta J(\theta)
\end{equation*}
with
\begin{equation*}
	\alpha_n = \sqrt{\frac{\epsilon}{\nabla_\theta J(\theta)^T F_\theta^{-1} \nabla_\theta J(\theta)} }
\end{equation*}
The direction $\widetilde{\nabla}_\theta J(\theta) = \Delta \theta / \alpha_n$ is called the natural gradient and learning algorithms that use this gradient instead of the standard one are called natural policy gradient algorithms. The strongest theoretical advantage of this approach is that its performance no longer depends on the parameterization of the policy and it is therefore safe to use for arbitrary policies. In practice, the learning process converges significantly faster in most practical cases and requires less manual parameter tuning of the learning algorithm. The only remaining point to discuss is how to compute the inverse of the Fischer information matrix. This is not an easy task and the matrix will usually need to be estimated from sample trajectories. However, we will see that this matrix can be computed analytically after reformulating the policy gradient theorem for the parameter-based search methods such as \gls{PGPE}. The formal derivation will be presented in Chapter \ref{ch:parameter_based_policy_gradient} and this result will lead to the \gls{NPGPE} algorithm \cite{miyamae2010natural}.