% s,fatrix2

\comment{
\newcommand{\A} {\blmath{A}}
\newcommand{\x} {\blmath{x}}
\newcommand{\y} {\blmath{y}}
\newcommand{\z} {\blmath{z}}
}

\subsection{The \fatrixx class}

The pronunciation rhymes with matrix.
Think ``fake matrix''
or ``fancy matrix.''

\subsubsection{Background}

Most iterative algorithms
for image reconstruction
are described conveniently using matrix notation,
but matrices are not necessarily
the most suitable data structure
for actually implementing an iterative algorithm
for large sized problems.
The \fatrixx class
provides a convenient bridge
between matrix notation
and practical system models
used for iterative image reconstruction.

Consider the simple iterative algorithm
for reconstructing \x from data \y
expressed mathematically:
\be
\x^{n+1} = \x^n + \alpha \A' (\y - \A\x^n)
,\ee{e,fatrixx,iter}
where \A is the \emph{system matrix}
associated with
the image reconstruction problem at hand.
%
If \A is small enough
to be stored as a matrix in \matlab
(sparse or full),
then this algorithm translates
very nicely into \matlab as follows.
%\begin{verbatim}
\be
\ty{x = x + alpha * A' * (y - A * x);}
\ee{e,mat2}
%\end{verbatim}
You really cannot get any closer connection
between the math and the program than this!
But often we work
with system models
that are too big to store
as matrices (even sparsely) in \matlab.
Instead,
the models
are implemented by subroutines
that compute the ``forward projection'' operation
$\A \x$
and the ``backprojection operation
$\A' \z$
for input vectors \x and \z respectively.
%
The conventional way
to use one of these system models
in \matlab (or C) would be to rewrite
the above program as follows.
\begin{verbatim}
Ax = forward_project(system_arguments, x)
residual = y - Ax;
correction = back_project(system_arguments, residual)
x = x + alpha * correction
\end{verbatim}
Yuch!
This is displeasing for two reasons.
First,
the code looks a \emph{lot} less like the mathematics.
Second,
usually you end up with a different version of the code
for every different system model
(forward/back-projector subroutine pair)
that you develop.
Having multiple versions
of a simple algorithm
creates a software maintenance headache.

The elegant solution
is to develop \matlab objects
that know how to perform
the following operations:
\blist
\item
\makebox[4em][l]{%
\ty{A * x}
}
(matrix vector multiplication,
operation \ty{mtimes})
\item
\makebox[4em][l]{%
\ty{A'}
}
(\ty{transpose}), and
\item
\makebox[4em][l]{%
\ty{A' * z}
}
(\ty{mtimes} again,
with a transposed object).
\elist
Once such an object is
defined,
one can use \emph{exactly}
the same iterative algorithm
that one would have used
with an ordinary matrix,
\eg,
\eref{e,mat2}.
%
The \fatrixx class
provides a convenient mechanism
for implementing
such linear operators.

\subsubsection{Creating a \fatrixx object}

Suppose \x is of length 1000
and \y is of length 2000.
One can create a corresponding \fatrixx object
using the following call:
\begin{verbatimtab}
A = fatrix2('imask', [1000 1], 'omask', [2000 1], ...
	'arg', system_arguments, ...
	'forw', @forward_project, 'back', @back_project);
\end{verbatimtab}
The resulting \fatrixx object \ty{A}
acts just like a matrix
in most important respects.
In particular,
we can use exactly the same iterative algorithm
\eref{e,fatrixx,iter} as before,
because
\ty{Ax = A * x}
is handled internally
by calling
\[
\ty{
Ax = forward_project(system_arguments, x)
}
\]
and similarly for
\ty{A' * z}.

Basic operations
like \ty{A(:,7)}
are also implemented,
but nonlinear operations
like \ty{exp(A)}
are not
because those cannot be computed readily
using \ty{forward_project}.

For many examples, see
the \ty{systems} subdirectory of \irt.

\subsubsection
{
Operations on a \fatrixx object
that return another \fatrixx object
}

\blist
\item
\ty{B = A'} or \ty{B = ctranspose{A}}
\\
\fatrixx object Hermitian transpose
\item
\ty{C = A * B}
\\
\fatrixx object multiplication
(requires compatible sizes)
\item
\ty{B = 7 * A}
\\
multiplying a scalar times a \fatrixx object
\item
\ty{A = [A1; A2; A3]}
\\
vertical concatenation (\ty{vertcat} is also supported)
(requires compatible sizes)
\item
\ty{A = [A1, A2, A3]}
\\
horizontal concatenation (\ty{horzcat} is also supported) % todo?
(requires compatible sizes)
\item
\ty{A(:,[3 7])} or \ty{A([3 7],:)} or \ty{A(:,2:end)} etc.
\\
these return a smaller sized \fatrixx
%\item
%\ty{A(:,newmask)} % todo
\item
\ty{C = B + A}
\\
\fatrixx object addition
(requires compatible sizes)
\elist

\subsubsection
{
Operations on a \fatrixx object
that return a vector.
}

\blist
\item
\ty{A(:,7)} or \ty{A(7,:)}
\elist

\subsubsection
{
Operations on a \fatrixx object
that return a matrix.
}

These are practical only if \ty{A} has sufficiently small size!

\blist
\item
\ty{A(:,:)} or \ty{full(A)}
\item
\ty{svd(A)}
\item
\ty{eig(A)}
% todo: eigs inv
\elist

\subsubsection
{
Other operations on a \fatrixx object.
}

\blist
\item
\ty{A(7,9)}
\\
This returns a scalar value. % todo
\item
\ty{sparse(A)}
\\
By default, internally this will compute each column of \A
using \ty{A(:,j)}
and then create a sparse matrix
from the nonzero elements of those columns.
For most large cases,
this will be very slow.
However,
a few \fatrixx objects
such as \ty{Gdiag}
have an internal \ty{sparse} method
(provided by the \ty{'sparse'} option
to the \ty{fatrix2} call)
that is very efficient.

\elist

\subsubsection
{
Support for arrays instead of columns
}

For an image reconstruction problem
with the mask illustrated in \fref{fig,reg,mask},
the size of \ty{A} should be
$\nd \times 8$
where \nd is the number of rows of \A.

So if we have a $6 \times 5$ image \ty{x}
and we would like to compute
$\A \x$,
conventionally
one would need to do the following:
\\
\cent{
\ty{y = A * x(mask)}
}

The statement \ty{x(mask)}
extracts the relevant $\np = 8$ values
out of the $6 \times 5$ array \ty{x}.

Objects in the \fatrixx class
often can spare the user the need to use \ty{x(mask)}
if the object is defined properly.

Suppose that in a tomographic image reconstruction problem
corresponding to \fref{fig,reg,mask},
the sinogram size is $9 \times 8$
so $\nd = 72$.

Then if \ty{A} is one of the predefined tomographic system models
in \irt such as \ty{Gtomo2_wtmex},
then the statement
\\
\cent{
\ty{yc = A * x(mask)}
}
\\
will produce \emph{column vector} \ty{yc}
with $\nd = 72$ elements.

On the other hand,
the convenient syntax
\\
\cent{
\ty{ya = A * x}
}
\\
will produce a $9 \times 8$ sinogram \emph{array} output \ty{ya}.
The two different outputs
are related by
\\
\cent{
\ty{yc = ya(:)}
}

In other words,
if the input is a column vector (of length \np),
then the output will also be a column vector,
just like one would expect for an ordinary matrix.
But if the input is an \emph{array},
of the appropriate dimensions,
then the output will also be an array.

When defining a new \fatrixx object,
there are several options
that one can use to specify
the appropriate dimensions.

\blist
\item
\ty{imask}
is the usual input mask.
Default is \ty{true(idim)}
\\
This can also be called just \ty{mask}
for backwards compatibility with \fatrix objects.

\item
\ty{idim}
describes the input array dimensions.
Default is \ty{sum(imask(:))},
\ie, an ordinary column vector.

\item
\ty{omask}
is a rarely used option.
Default is \ty{true(odim)}

\item
\ty{odim}
describes the output array dimensions.
Default is \ty{sum(omask(:))},
\ie, an ordinary column vector.
\elist

For a typical \fatrixx object,
one will use only the \ty{imask} and \ty{odim} options.

For the tomography example above,
we would add the name-value pairs
\ty{'odim', [9 8]}
to the \ty{fatrix2} call.

\subsubsection{Defining \fatrixx methods}

The key methods for a \fatrixx object
are the \ty{forw} and \ty{back} operations.
For the obsolete \fatrix objects,
these methods had to support columns and arrays
and multiples thereof
which made them fairly complicated.
For the new \fatrixx objects,
this complexity is handled by the object itself,
and the user-defined methods are much simpler.

\blist
\item
The \ty{forw} routine
accepts a single input array
of size \ty{idim}
and returns a single output array
of size \ty{odim}.

\item
Caution:
nonzero values in the input array
outside of \ty{imask}
may cause unpredictable results.

\item
Conversely,
the \ty{back} routine
accepts a single input array
of size \ty{odim}
and returns a single output array
of size \ty{idim}.

\item
Caution:
the output array of the \ty{back} routine
must be zero for any pixels
outsize of the \ty{omask}.
%
Sometimes this will happen automatically
because a computationally efficient
\ty{back} routine
will only evaluate the output
within the \ty{omask},
and will set the other values to zero.
%
But for some objects,
like \ty{Gdft},
the \ty{back} routine
first evaluates the entire output
(using \ty{ifftn})
and then performs a \ty{.*} with \ty{omask}
to comply with the requirement.

\elist

\subsubsection{The 1D dilemma}

Unfortunately,
\matlab does not really support 1D arrays.
(\matlab's \ty{size} command always returns
at least a two-element vectors.)
So for any application where
\ty{odim} is 1D,
such as MRI with irregular non-Cartesian k-space samples,
\ty{A' * y}
will produce a 1D array
not a 2D array
even if \ty{imask} is 2D,
because \ty{y} will be ``1D'' in such cases.


\section{Application to color normalization}

Color normalization is the process of imposing the same color palette on a group of images. This color palette is always somehow related to the color palettes of the original images. For instance, if the goal is to cancel the illumination of a scene (avoid color cast), then the imposed histogram should be the histogram of the same scene illuminated with white light. Of course, in many occasions this information is not available. Following Papadakis et al.~\cite{Papadakis_ip11}, we define an in-between histogram, which is chosen here as the regularized OT barycenter.

%The advantage with respect to Papadakis et al.'s method is that the influence of the input histograms on the barycenter can be easily tuned by a change of parameters, in fact it has as a special case any of the original image histograms.

%Some other examples for which color normalization is useful are the color balancing of videos,  or as a preprocessing to register/compare several images taken with different cameras (see \cite{Papadakis_ip11} for more examples).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Algorithm}

Given a set of input images $(X^{0[r]})_{r \in R}$, the goal is to impose on all the images the same histogram $\mu_X$ associated to the barycenter $X$. As for the colorization problem tackled in Section~\ref{sec-appli-color}, the first step is to subsample the original cloud of points $X^{0[r]}$ to make the problem tractable. Thus, for every $X^{0[r]}$ we compute a smaller associated point set $X^{[r]}$ using K-means clustering. Then, we obtain the barycenter $X$ of all the point clouds $(X^{[r]})_{r \in R}$ with the algorithms presented in Section~\ref{algobar}. Figure~\ref{im:baryillu} first row, shows an example on two synthetic cloud of points, $X^{[1]}$ in blue and $X^{[2]}$ in red. The cloud of points in green corresponds to the barycenter $X$, which can change its position depending on the parameter $\rho=(\rho_1,\rho_2)$ in~\eqref{eqbar} from $X^{[1]}$ for $\rho=(1,0)$ to $X^{[2]}$ when $\rho=(0,1)$.  This data set $X$ represents the 3-D histogram we want to impose on all the input images.

Once we have $X$, we compute the regularized and relaxed OT transport maps $T^{[r]}$ between each $X^{[r]}$ and the barycenter $X$, by solving~\eqref{eq-symm-reg-energy}. The line segments in Figure~\ref{im:baryillu} represent the transport between point clouds, i.e. if $\Sig^{[1]}_{i,j}>0$, $X^{[1]}_i$ is linked to $X_j$, and similarly for $\Sig^{[2]}$.

We apply $T^{[r]}$ to $X^{[r]}$, obtaining  $\tilde X^{[r]}$, for all $r \in R$, that is to say, we obtain a set of point clouds $\tilde X^{[r]}$ with a color distribution close to $X$.  Finally, to recover a set of high resolution images, we compute each $\tilde X^{0[r]}$ from $X^{0[r]}$ by up-sampling. A detailed description of the method is given in Algorithm~\ref{alg-norm}.

\begin{algorithm}[ht!]
\caption{Regularized OT Color Normalization}
\label{alg-norm}
% \begin{algorithmic}[1]
\Require Images $\left( X^{0[r]} \right)_{r \in R} \in \RR^{N_0 \times d}$, $\la \in \RR^+ $, $\rho \in [0,1]^{|R|}$ and $k \in \RR^+$.

\Ensure Images $\left( \tilde X^{0[r]} \right)_{r \in R} \in \RR^{N_0 \times d}$.
% \Statex
\begin{enumerate}
	\algostep{Histogram down-sample} Compute $X^{[r]}$ from $X^{0[r]}$ using K-means clustering.
	\algostep{Compute barycenter} Compute with either~\eqref{eq-bar-l2} or~\eqref{eq-bar-l1} a barycenter $\mu_X$ where $X$ is a local minimum of~\eqref{eqbar} using the block coordinate descent described in Section~\ref{algobar}, see Algorithm~\ref{algo-block-barycenters}.
	\algostep{Compute transport mappings} For all $r \in R$ compute $T^{[r]}$ between \\
		$X$ and $X^{[r]}$ by solving~\eqref{eq-symm-reg-energy}, such that $T^{[r]}(X^{[r]}_i) = Z^{[r]}_i$, where $Z^{[r]}= \diag(\Sig^{[r]} \U)^{-1} \Sig^{[r]} X$.
	\algostep{Transport up-sample} For every $T^{[r]}$ compute $\tilde T^{0[r]}$ following~\eqref{eq-upsample}.
	\algostep{Obtain high resolution results} Compute $\foralls r, \tilde X^{0[r]} = \tilde T^{0[r]}(X^{0[r]})$.
\end{enumerate}
% \end{algorithmic}
\end{algorithm}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Results}

We now show some example of color normalization using Algorithm~\ref{alg-norm}.

\begin{figure*}[!h]
\centering
\setlength{\arrayrulewidth}{2pt}
%%% OT %%%%
\begin{tabular}{@{}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{}}
	$\rho=(1,0)$ & $\rho=(0.7,0.3)$ & $\rho=(0.4,0.6)$ & $\rho=(0,1)$ \\
\includegraphics[width=.23\linewidth]{./syntheticbary/Barycenter_DiagsyntheticINVrho-0-ksum1lambda0nnx4QP1png} &
\includegraphics[width=.23\linewidth]{./syntheticbary/Barycenter_DiagsyntheticINVrho-03-ksum1lambda0nnx4QP1png} &
\includegraphics[width=.23\linewidth]{./syntheticbary/Barycenter_DiagsyntheticINVrho-06-ksum1lambda0nnx4QP1png} &
\includegraphics[width=.23\linewidth]{./syntheticbary/Barycenter_DiagsyntheticINVrho-1-ksum1lambda0nnx4QP1png} \\
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag2-syntheticrho-0-ksum1lambda0nnx4QP1} &
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag2-syntheticrho-03-ksum1lambda0nnx4QP1} &
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag2-syntheticrho-06-ksum1lambda0nnx4QP1} &
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag2-syntheticrho-1-ksum1lambda0nnx4QP1}\\
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag1-syntheticrho-0-ksum1lambda0nnx4QP1} &
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag1-syntheticrho-03-ksum1lambda0nnx4QP1} &
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag1-syntheticrho-06-ksum1lambda0nnx4QP1} &
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag1-syntheticrho-1-ksum1lambda0nnx4QP1} \\\hline
%%% Regularized
\includegraphics[width=.23\linewidth]{./syntheticbary/Barycenter_DiagsyntheticINVrho-0-ksum20lambda00005nnx4QP1} &
\includegraphics[width=.23\linewidth]{./syntheticbary/Barycenter_DiagsyntheticINVrho-03-ksum20lambda00005nnx4QP1} &
\includegraphics[width=.23\linewidth]{./syntheticbary/Barycenter_DiagsyntheticINVrho-06-ksum20lambda00005nnx4QP1} &
\includegraphics[width=.23\linewidth]{./syntheticbary/Barycenter_DiagsyntheticINVrho-1-ksum20lambda00005nnx4QP1} \\
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag2-syntheticrho-01-ksum20lambda00005nnx4QP1} &
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag2-syntheticrho-0307-ksum20lambda00005nnx4QP1}&
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag2-syntheticrho-0604-ksum20lambda00005nnx4QP1} &
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag2-syntheticrho-10-ksum20lambda00005nnx4QP1}\\
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag1-syntheticrho-01-ksum20lambda00005nnx4QP1} &
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag1-syntheticrho-0307-ksum20lambda00005nnx4QP1} &
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag1-syntheticrho-0604-ksum20lambda00005nnx4QP1} &
\includegraphics[width=.23\linewidth,height=.2\linewidth]{./syntheticbary/Diag1-syntheticrho-10-ksum20lambda00005nnx4QP1} \\
\end{tabular}
 \caption{Comparison of classical OT (top 3 first rows) and relaxed/regularized OT (bottom 3 last rows). The original input images $X^{0,[1]}$ and $X^{0,[2]}$ are shown in Figure~\ref{im:synth}~(a).
 		Rows \#1 and \#4 shows the 2-D projections of $X^{[1]}$ (blue) and $X^{[2]}$ (red), and in green the barycenter distribution for different values of $\rho$. We display a line between $X^{[r]}_i$ and $X_j$ if $\Sig^{[r]}_{i,j} > 0.1$.
		Rows \#2 and \#5 (resp. \#3 and \#6) show the resulting normalized images $\tilde X^{0[1]}$ (resp. $\tilde X^{0[2]}$),
		for each value of $\rho$.
 	\textbf{Top 3 first rows:} classical OT corresponding to setting $k=1$ and $\la=0$.
	\textbf{Bottom 3 last rows:}  regularized and relaxed OT, with parameters $k=20$ and $\lambda=0.0005$. See main text for comments. \vspace{0.5cm}
}
\label{im:baryillu}
\end{figure*}

%%%%%%%%%%%%%%%%%%
\paragraph{Synthetic example}

Figure~\ref{im:baryillu} shows a comparison of normalization of two synthetic images using classical OT and our proposed relaxed/regularized OT. The results obtained using Algorithm~\ref{alg-norm} (setting $p=q=2$), using the set of two images ($|R|=2$) already used in Figure~\ref{im:synth}~(a), denoting here $X^{0[1]}=X^0$ and $X^{0[2]}=Y^0$. Each column shows the same experiment but with different values of $\rho$, which allows to visualize the interpolation between the color palettes (the colors in the images  evolve from the colors in $X^{[1]}$ towards the colors of $X^{[2]}$).

With classical OT, the structure of the original data sets in not preserved as we change $\rho$, and the consequence on the final images (second and third row), is that the geometry of the original images changes in the barycenters. In contrast to classical OT, for all values of $\rho$ the relaxed/regularized barycenters $X$ have the same number of clusters of the original sets. Note that the consequence of having a transport that maintains the clusters of the original images, is that the geometry is preserved, while the histograms change.

\newcommand{\sidecapY}[1]{ \begin{sideways}\parbox{.19\linewidth}{\centering #1}\end{sideways} }
\newcommand{\myimgY}[1]{\includegraphics[width=.26\linewidth,height=.22\linewidth]{#1}}

\begin{figure*}[!h]
\centering
\begin{tabular}{@{}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{} }
\sidecapY{ Original $X^{0[1]}$ }  &
\myimgY{star/fleur_1} &
\myimgY{star/wheat_1} &
\myimgY{star/parrot_1} \\
\sidecapY{$\rho=(1,0)$ } &
\myimgY{barycenter/DiagYfleurrho-1-ksum11lambda00009nnx4QP1} &
\myimgY{barycenter/wheat/Diag1-wheatrho-1-ksum13lambda001nnx4QP1} &
\myimgY{barycenter/parrot/Diag1-parrotrho-1-ksum1lambda0001nnx4QP1} \\
\sidecapY{$\rho=(0.7,0.3)$} &
\myimgY{barycenter/DiagYfleurrho-06-ksum11lambda00009nnx4QP1} &
\myimgY{barycenter/wheat/Diag1-wheatrho-06-ksum13lambda001nnx4QP1}  &
\myimgY{barycenter/parrot/Diag1-parrotrho-06-ksum1lambda0001nnx4QP1} \\
\sidecapY{ $\rho=(0.4,0.6)$ } &
\myimgY{barycenter/DiagYfleurrho-03-ksum11lambda00009nnx4QP1} &
\myimgY{barycenter/wheat/Diag1-wheatrho-03-ksum13lambda001nnx4QP1} &
\myimgY{barycenter/parrot/Diag1-parrotrho-03-ksum1lambda0001nnx4QP1} \\
\sidecapY{ $\rho=(0,1)$ } &
\myimgY{barycenter/DiagYfleurrho-0-ksum11lambda00009nnx4QP1} &
\myimgY{barycenter/wheat/Diag1-wheatrho-0-ksum13lambda001nnx4QP1} &
\myimgY{barycenter/parrot/Diag1-parrotrho-0-ksum1lambda0001nnx4QP1} \\
\sidecapY{ Original $X^{0[2]}$ } &
\myimgY{star/fleur_2} &
\myimgY{star/wheat_2} &
\myimgY{star/parrot_2} \\
& (a) &  (b) & (c)
\end{tabular}
\caption{Results for the barycenter algorithm on different images computed with the method proposed in Section~\ref{algobarysobolev}. The parameters were set to \textbf{(a)} $k=1.1,\la=0.0009$, \textbf{(b)} $k=1.3,\la=0.01$, and  \textbf{(b)} $k=1,\la=0.001$. Note how as $\rho$ approaches $(0,1)$, the histogram of the barycenter image becomes similar to the histogram of $X^{0[2]}$.}
\label{im:bar}
\end{figure*}

\paragraph{Example on natural images} Fig.~\ref{im:bar} shows the results of the same experiment as in Fig.~\ref{im:baryillu}, but on the natural images labeled as $X^{0[1]}$ and $X^{0[2]}$ in rows $\#1$ and $\#6$. In this case, we only show the transport from $X$ to $X^{0[1]}$, that is to say, we maintain the geometry of $X^{0[1]}$ (row $\#1$) and match its histogram to the barycenter distribution. As in the previous experiment, note how the colors change smoothly from $(1,0)$ to $(0,1)$ without generating artifacts and match the color and contrast of image $X^{0[2]}$ for $\rho=(0,1)$. The change in contrast is specially visible for the (b) wheat image.

%%%%%%%
\paragraph{Color Normalization}

Computing the barycenter distribution of the histograms of a set of images is useful for color normalization. We show in Figures~\ref{barflower}, and~\ref{barclock} the results obtained with Algorithm~\ref{alg-norm}, and compare them with the standard OT and the method proposed by Papadakis et al.~\cite{Papadakis_ip11}. The improvement of the relaxation and regularization is specially noticeable in Figures~\ref{barflower} where OT creates artifacts such as coloring the leaves on violet for Figure~\ref{barflower}~(a), or introducing new colors on the background in Figure~\ref{barflower}~(c). In Figure~\ref{barclock}, OT and Papadakis et al.'s method introduce artifacts mostly on the sky of Figure~\ref{barclock}~(a) and  Figure~\ref{barclock}~(b), while the relaxed and regularized version displays a smoother result for Figure~\ref{barclock}~(a) and~(c) and a more meaningful color transformation (all the clouds have the same color in the fourth row) for Figure~\ref{barclock}~(b).

\newcommand{\myimgZ}[1]{\includegraphics[width=.28\linewidth,height=.26\linewidth]{#1}}
\newcommand{\myTriTab}[1]{ \begin{tabular}{@{}c@{\hspace{2mm}}c@{\hspace{2mm}}c@{} } #1	\end{tabular} }

\begin{figure*}[ht]
\centering
\myTriTab{
\myimgZ{./barycenter/flowers/flowers-1} &
\myimgZ{./barycenter/flowers/flowers-2} &
\myimgZ{./barycenter/flowers/flowers-3} \\
\myimgZ{./barycenter/flowers/Diag1-OT} &
\myimgZ{./barycenter/flowers/Diag2-OT} &
\myimgZ{./barycenter/flowers/Diag3-OT} \\
\myimgZ{./barycenter/nicoflowers1} &
\myimgZ{./barycenter/nicoflowers2} &
\myimgZ{./barycenter/nicoflowers3} \\
\myimgZ{./barycenter/flowers/Diag1-flowersrho-033333-ksum2lambda0005nnx4QP1} &
\myimgZ{./barycenter/flowers/Diag2-flowersrho-033333-ksum2lambda0005nnx4QP1} &
\myimgZ{./barycenter/flowers/Diag3-flowersrho-033333-ksum2lambda0005nnx4QP1} \\
(a) & (b) & (c)
}
\caption{In the first row, we show the original images. In the following rows, we show the result of computing the barycenter histogram and imposing it on each of the original images, with different algorithms. In the second row, we use OT. In the third row, the results were obtained with the method proposed by Papadakis et al.~\cite{Papadakis_ip11}. On the last row, we show the results obtained with the relaxed and regularized OT barycenter with $k=2,\la=0.005$. Note how the proposed algorithm is the only one that does not produce artifacts on the final images such as (a) color artifacts on the leaves and (c) different colors on the background.}
\label{barflower}
\end{figure*}

\begin{figure*}[ht]
\centering
\myTriTab{
\myimgZ{./barycenter/clockmontague-1} &
\myimgZ{./barycenter/clockmontague-2} &
\myimgZ{./barycenter/clockmontague-3} \\
\myimgZ{./barycenter/Diag1-clockmontaguerho-033333-ksum1lambda0nnx4QP1} & %OT
\myimgZ{./barycenter/Diag2-clockmontaguerho-033333-ksum1lambda0nnx4QP1} &
\myimgZ{./barycenter/Diag3-clockmontaguerho-033333-ksum1lambda0nnx4QP1} \\
\myimgZ{./barycenter/nicoclock1} &
\myimgZ{./barycenter/nicoclock2} &
\myimgZ{./barycenter/nicoclock3} \\
\myimgZ{./barycenter/Diag1-clockmontaguerho-033333-ksum13lambda00005nnx4QP1} &
\myimgZ{./barycenter/Diag2-clockmontaguerho-033333-ksum13lambda00005nnx4QP1} &
\myimgZ{./barycenter/Diag3-clockmontaguerho-033333-ksum13lambda00005nnx4QP1} \\
(a) & (b) & (c)
}
\caption{Experiment as in Figure~\ref{barflower} applied on the images of the first row. Our results, presented in the final row, were obtained with $k=1.3$ and $\la=0.0005$. Contrary to OT (second row) or the method proposed by Papadakis et al.~\cite{Papadakis_ip11} (third row), the proposed method does not create artifacts on the sky and the clock for images (a) and (c).}
\label{barclock}
\end{figure*}

\begin{figure*}[ht]
\centering
\myTriTab{
\myimgZ{./barycenter/clockHD-1} &
\myimgZ{./barycenter/clockHD-2} &
\myimgZ{./barycenter/clockHD-3} \\
\myimgZ{./barycenter/Diag1-clockHDrho-033333-ksum11lambda00005nnx4QP1} &
\myimgZ{./barycenter/Diag2-clockHDrho-033333-ksum11lambda00005nnx4QP1} &
\myimgZ{./barycenter/Diag3-clockHDrho-033333-ksum11lambda00005nnx4QP1} \\
}
\caption{The proposed method can be applied as a preprocessing step in a pipeline for objects detection or image registration, where canceling illumination is important. On the first row, we show a set of pictures of the same object taken at different hours of the day or night, and on the second row, the result of our algorithm setting  $(p,q)=(2,2)$, $k=1$ and $\la=0.0005$. Note how the algorithm is able to normalize the illumination conditions of all the images.}
\label{im:colornorm}
\end{figure*}

As a final example, we would like to show in Figure~\ref{im:colornorm} how this method can be applied as a preprocessing before comparing/registering images of the same object obtained under different illumination conditions.


\section*{Math}

$\vec{A}\cdot\vec{B} = |\vec{A}||\vec{B}|\cos\phi_{\text{AB}} = A_xB_x + A_yB_y +
A_xB_z$
\begin{align*}
  \vec{A}\times\vec{B} &= |\vec{A}||\vec{B}|\sin\phi_{\text{AB}}\hat{n}\\
  &= \quad(A_yB_z-A_zB_y)\hat{x}\\
  &\quad -(A_xB_z-A_zB_x)\hat{y}\\
  &\quad +(A_xB_y-A_yB_x)\hat{z}
\end{align*}
The cross product can be represented as a determinant
\begin{equation*}
  \vec{A}\times\vec{B} = %
  \begin{vmatrix}
    \hat{x} & \hat{y} & \hat{z}\\
    A_x & A_y & A_z \\
    B_x & B_y & B_z \\
  \end{vmatrix}
\end{equation*}

If $ax^2 + bx +c = 0$, $x=\frac{-b \pm \sqrt{b^2 = 4ac}}{2a}$

% Derivatives
$\frac{dx^m}{dx} = mx^{m-1}$\\

% Integrals
$\int x^m\;dx = \frac{x^{m+1}}{m+1} + C$


\newpage
\subsection{Goal}\label{subsec:goal}

The use of first- and second-order methods (Gradient Descent, Conjugate Gradient Descent, Newton's method and Levenberg-Marquardt algorithm) in the tasks of unconstrained nonlinear optimization.

\subsection{Formulation of the problem}\label{subsec:formulation-of-the-problem}

Generate random numbers $\alpha \in (0, 1)$ and $\beta \in (0, 1)$.
Furthermore, generate the noisy data $\{x_k, y_k\}$, where $k = 0, \dots, 100$, according to the following rule:

\begin{equation}
    y_k = \alpha x_k + \beta + \delta_k, x_k = \frac{k}{100},
\end{equation}

where $\delta_k \sim N(0, 1)$ are values of a random variable with standard normal distribution.
Approximate the data by the following linear and rational functions:

\begin{enumerate}
    \item $F(x, a, b) = ax + b$ (linear approximant),
    \item $F(x, a, b) = \frac{a}{1 + bx}$ (rational approximant),
\end{enumerate}

by means of least squares through the numerical minimization (with precision $\varepsilon = 0.001$ of the following function:

\begin{equation}
    D(a, b) = \sum^{100}_{k=0}(F(x_k, a, b) - y_k)^2.
\end{equation}

To solve the minimization problem, use the methods of Gradient Descent, Conjugate Gradient Descent, Newton's method and Levenberg-Marquardt algorithm.
If necessary, set the initial approximations and other parameters of the methods.
Visualize the data and the approximants obtained separately for each type of approximant.
Analyze the results obtained (in terms of number of iterations, precision, number of function evaluations, etc.) and compare them with those from Task 2 for the same dataset.

\subsection{Brief theoretical part}\label{subsec:brief-theoretical-part}

Optimization methods are numerical methods for finding optimal (in some sense) values of objective functions, for example, in the framework of mathematical models of certain processes.

\textit{First-and second-order optimization methods} use to minimize the objective function $f$ on the set $Q$ its value $f(x)$ and the values of its first and second derivatives (gradient, Hessian), respectively.

\paragraph{Gradient descent}

\textit{Gradient descent} is an optimization algorithm that's used when training a machine learning model.
It's based on a convex function and tweaks its parameters iteratively to minimize a given function to its local minimum.

\paragraph{Conjugate gradient method}

\textit{Conjugate gradient methods} represent a kind of steepest descent approach.
With steepest descent, minimization of a function begins $f$ starting at $x_0$ by traveling in the direction of the negative gradient - $f'(x_0)$.
In subsequent steps, the movement continues in the direction of the negative gradient evaluated at each successive point until convergence.

\paragraph{Newton's method}

\textit{Newton's method} is an iterative method for finding the roots of a differentiable function $f$, which are solutions to the equation $f(x) = 0$.
In optimization, Newton's method is applied to the derivative $f'$ of a twice-differentiable function $f$ to find the roots of the derivative (solutions to $f'(x) = 0$), also known as the stationary points of $f$.
These solutions may be minima, maxima, or saddle points.

\paragraph{Levenberg-Marquardt algorithm}

The \textit{Levenberg-Marquardt algorithm} (\textit{LMA}) is a popular trust region algorithm that is used to find a minimum of a function (either linear or nonlinear) over a space of parameters.
Essentially, a trusted region of the objective function is internally modeled with some function such as a quadratic.
When an adequate fit is found, the trust region is expanded.
As with many numerical techniques, the Levenberg-Marquardt method can be sensitive to the initial starting parameters.

\subsection{Results}\label{subsec:results}

It is known that optimization by \textit{linear approximation} has a unique solution, so these methods give similar optimal values for $a$ and $b$, regardless of the choice of initial approximations.
In the case of a \textit{rational approximation}, significant nonlinearities occur, so the result depends on the initial approximations (Figure~\ref{ris:plot}).

\begin{figure}[H]
    \center
    \includegraphics[width=\textwidth]{img/plot.png}
    \caption{Direct methods of optimization.}
    \label{ris:plot}
\end{figure}

From the tables below (Tables~\ref{tbl:direct},~\ref{tbl:order}), we can conclude that first-and second-order optimization methods require orders of magnitude less computational operations, in contrast to direct methods.

\begin{table}[ht]
\caption{Direct methods}
\begin{tabular}{l|l|l|l|}
\cline{2-4}
                                                & \textbf{method}             & \textbf{iterations} & \textbf{function eval} \\ \hline
\multicolumn{1}{|l|}{\multirow{3}{*}{\textbf{linear}}}   & exhaustive\_search & 5448000  & 5448000     \\ \cline{2-4}
\multicolumn{1}{|l|}{}                          & gauss\_method      & 77376     & 77376        \\ \cline{2-4}
\multicolumn{1}{|l|}{}                          & nelder\_mead       & 18         & 3             \\ \hline
\multicolumn{1}{|l|}{\multirow{3}{*}{\textbf{rational}}} & exhaustive\_search & 5448000  & 5448000     \\ \cline{2-4}
\multicolumn{1}{|l|}{}                          & gauss\_method      & 6448      & 6448         \\ \cline{2-4}
\multicolumn{1}{|l|}{}                          & nelder\_mead       & 10         & 2             \\ \hline
\end{tabular}
\label{tbl:direct}
\end{table}

\begin{table}[ht]
\caption{First- and second-order methods}
\begin{tabular}{l|l|l|l|}
\cline{2-4}
                                                         & \textbf{method}              & \textbf{iterations} & \textbf{function eval} \\ \hline
\multicolumn{1}{|l|}{\multirow{4}{*}{\textbf{linear}}}   & gradient\_descent            & 150                 & 150                    \\ \cline{2-4}
\multicolumn{1}{|l|}{}                                   & conjugate\_gradient\_descent & 80                  & 80                     \\ \cline{2-4}
\multicolumn{1}{|l|}{}                                   & newton                       & 250                 & 250                    \\ \cline{2-4}
\multicolumn{1}{|l|}{}                                   & levenberg\_marquardt         & 34                  & 34                     \\ \hline
\multicolumn{1}{|l|}{\multirow{4}{*}{\textbf{rational}}} & gradient\_descent            & 150                 & 150                    \\ \cline{2-4}
\multicolumn{1}{|l|}{}                                   & conjugate\_gradient\_descent & 104                 & 104                    \\ \cline{2-4}
\multicolumn{1}{|l|}{}                                   & newton                       & 208                 & 208                    \\ \cline{2-4}
\multicolumn{1}{|l|}{}                                   & levenberg\_marquardt         & 56                  & 56                     \\ \hline
\end{tabular}
\label{tbl:order}
\end{table}

\paragraph{Newton's method gives a different result}

\begin{theorem}
    \label{theorem:conv_newton}
    For doubly continuously differentiable functions (i.e.\ for $f \in C^2(D)$) with a non-degenerate matrix $\nabla^2 f(y^*)$, there exists a $\varepsilon$-neighborhood of the stationary point $y^*$ of the function $f(y)$ such that for any initial point $y^0$ from this neighborhood, the Newton method will converge superlinearly, and if the Lipschitz condition is met in this neighborhood for Hesse matrices, it will converge quadratically.
\end{theorem}

If we consider the Newton method in more detail, then we can conclude from the theorem above [1]: if the Hesse matrix is not degenerate, but not sign-positive, then the Newton method can converge to a stationary point that is not a minimum point, but a maximum point or a saddle point.
Accordingly, a more careful choice of initial approximations is necessary.

\subsection{Conclusion}\label{subsec:conclusion}

In the course of the laboratory work,  first- and second-order methods were implemented and analyzed within the problem of the unconstrained optimization problem.
This paper also provides a comparative analysis of direct optimization methods and first-and second-order methods.

\subsection{References}\label{subsec:references}
\begin{enumerate}[label={[\arabic*]}]
    \item С.Ю. Городецкий, Лабораторный практикум по методам локальной оптимизации в программной системе LocOpt, 2007.
\end{enumerate}

\subsection{Appendix}\label{subsec:appendix}

The source code is located \href{https://github.com/vanSultan/anal_dev_algo/tree/lab_03}{here}: \url{https://github.com/vanSultan/anal_dev_algo/tree/lab_03}.


\documentclass[12pt]{article}

%\usepackage{fullpage}
\usepackage{amsmath,amssymb}

\begin{document}

\title{Two Dimensional Euler Simulation}

\author{
S.J. Montgomery-Smith\\
Department of Mathematics\\
University of Missouri\\
Columbia, MO 65211, U.S.A.\\
stephen@math.missouri.edu\\
http://www.math.missouri.edu/\~{}stephen}

\date{September 10, 2000}

\maketitle

This document describes a program I wrote to simulate the
two dimensional Euler Equation --- a program that is part
of the {\tt xlock} screensaver as the {\tt euler2d}
mode.  A similar explanation may also be found in the
book by Chorin \cite{C}.

\section{The Euler Equation}

The Euler Equation describes the motion of an incompressible
fluid that has no viscosity.  If the fluid is contained
in a domain $\Omega$ with boundary $\partial \Omega$, then
the equation is in the vector field $u$ (the velocity)
and the
scalar field $p$ (the pressure):
\begin{eqnarray*}
\frac{\partial}{\partial t} u &=& -u \cdot \nabla u + \nabla p \\
\nabla \cdot u &=& 0 \\
u \cdot n &=& 0 \quad \text{on $\partial \Omega$}
\end{eqnarray*}
where $n$ is the unit normal to $\partial \Omega$.

\section{Vorticity}

It turns out that it can be easier write these equations
in terms of the vorticity.  In two dimensions the vorticity
is the scalar $w = \partial u_2/\partial x - \partial u_1/\partial y$.
The equation for vorticity becomes
\[ \frac{\partial}{\partial t} w = -u \cdot \nabla w .\]
A solution to this equation can be written as follows.  The velocity
$u$ causes a flow, that is, a function $\varphi(t,x)$ that tells where
the particle initially at $x$ ends up at time $t$, that is
\[
\frac\partial{\partial t} \varphi(t,x)
= u(t,\varphi(t,x)) .\]
Then the equation
for $w$ tells us that the vorticity is ``pushed'' around by the flow,
that is, $w(t,\varphi(t,x)) = w(0,x)$.

\section{The Biot-Savart Kernel}

Now, once we have the vorticity, we can recover the velocity $u$ by
solving the equation
\begin{eqnarray*}
\partial u_2/\partial x - \partial u_1/\partial y &=& w \\
\nabla \cdot u &=& 0 \\
u \cdot n &=& 0 \quad \text{on $\partial \Omega$}.
\end{eqnarray*}
This equation is solved by using a Biot-Savart kernel $K(x,y)$:
$$ u(x) = \int_\Omega K(x,y) w(y) \, dy .$$
The function $K$ depends upon the choice of domain.  First let us consider
the case when $\Omega$ is the whole plane (in which case the boundary
condition $u \cdot n = 0$ is replaced by saying that $u$ decays at infinity).
Then
\begin{equation*}
K(x,y) = K_1(x,y) = c \frac{(x-y)^\perp}{|x-y|^2} .
\end{equation*}
Here $x^\perp = (-x_2,x_1)$, and $c$ is a constant, probably something
like $1/2\pi$.  In any case we will set it to be one, which in effect
is rescaling the time variable, so we don't need to worry about it.

We can use this as a basis to find $K$ on the unit disk
$\Omega = \Delta = \{x:|x|<1\}$.  It turns out to be
\begin{equation*}
K_2(x,y) = K_1(x,y) - K_1(x,y^*) ,
\end{equation*}
where $y^* = y/|y|^2$ is called the reflection of $y$ about the
boundary of the unit disk.

Another example is if we have a bijective analytic function
$p:\Delta \to {\mathbb C}$, and we let $\Omega = p(\Delta)$.
(Here we think of $\Delta$ as a subset of $\mathbb C$, that is,
we are identifying the plane with the set of complex numbers.)
In that case we get
\[ K_p(p(x),p(y)) = K_2(x,y)/|p'(x)|^2 .\]
Our simulation considers the last case.  Examples of such
analytic functions include series
$p(x) = x + \sum_{n=2}^\infty c_n x^n$, where
$\sum_{n=2}^\infty n |c_n| \le 1$.
(Thanks to David Ullrich for pointing this out to me.)

\section{The Simulation}

Now let's get to decribing the simulation.  We assume a rather
unusual initial distribution for the vorticity --- that the
vorticity is a finite sum of dirac delta masses.
\[ w(0,x) = \sum_{k=1}^N w_k \delta(x-x_k(0)) .\]
Here $x_k(0)$ is the initial place where the points
of vorticity are concentrated, with values $w_k$.
Then at time $t$, the vorticity becomes
\[ w(t,x) = \sum_{k=1}^N w_k \delta(x-x_k(t)) .\]
The points of fluid $x_k(t)$ are pushed by the
flow, that is, $x_k(t) = \varphi(t,x_k(0))$, or
\[ \frac{\partial}{\partial t} x_k(t) = u(t,x_k(t)) .\]
Putting this all together, we finally obtain the equations
\[ \frac{\partial}{\partial t} x_k = \alpha_k \]
where
\[ \alpha_k   = \sum_{l=1}^N w_l K(x_k,x_l) .\]
This is the equation that our simulation solves.

In fact, in our case, where the domain is $p(\Delta)$,
the points are described by points
$\tilde x_k$, where $x_k = p(\tilde x_k)$.  Then
the equations become
\begin{eqnarray}
\label{tildex-p1}
\frac{\partial}{\partial t} \tilde x_k &=& \tilde\alpha_k \\
\label{tildex-p2}
\tilde\alpha_k &=& \frac1{|p'(\tilde x_k)|^2}
     \sum_{l=1}^N w_l K_2(\tilde x_k,\tilde x_l) .
\end{eqnarray}

We solve this $2N$ system of equations using standard
numerical methods, in our case, using the second order midpoint method
for the first step, and thereafter using the second order Adams-Bashforth
method.  (See for example the book
by Burden and Faires \cite{BF}).

\section{The Program - Data Structures}

The computer program solves equation (\ref{tildex-p1}), and displays
the results on the screen, with a boundary.  All the information
for solving the equation and displaying the output is countained
in the structure {\tt euler2dstruct}.  Let us describe some of
the fields in {\tt euler2dstruct}.
The points $\tilde x_k$ are contained
in {\tt double *x}: with the coordinates of
$\tilde x_k$ being the two numbers
{\tt x[2*k+0]}, {\tt x[2*k+1]}.  The values $w_k$ are contained
in {\tt double *w}.  The total number of points is
{\tt int N}.  (But only the first {\tt int Nvortex} points
have $w_k \ne 0$.)  The coefficients of the analytic function
(in our case a polynomial) $p$
are contained in {\tt double p\_coef[2*(deg\_p-1)]} --- here
{\tt deg\_p} is the degree of $p$, and the real and imaginary
parts of the coefficient
$c_n$ is contained in {\tt p\_coef[2*(n-2)+0]} and {\tt p\_coef[2*(n-2)+1]}.

\section{Data Initialization}

The program starts in the function {\tt init\_euler2d}.  After allocating
the memory for the data, and initialising some of the temporary variables
required for the numerical solving program, it randomly assigns the
coefficients of $p$, making sure that $\sum_{n=2}^{\tt deg\_p} n |c_n| = 1$.
Then the program figures out how to draw the boundary, and what rescaling
of the data is required to draw it on the screen.  (This uses the
function {\tt calc\_p} which calculates $p(x)$.)

Next, it randomly assigns the initial values of $\tilde x_k$.  We want
to do this in such a way so that the points are uniformly spread over the
domain.  Let us first consider the case when the domain is the unit circle
$\Delta$.  In that case the proportion of points that we would expect
inside the circle of radius $r$ would be proportional to $r^2$.  So
we do it as follows:
\[ r = \sqrt{R_{0,1}},\quad \theta = R_{-\pi,\pi}, \quad
   \tilde x_k = r (\cos \theta, \sin \theta) .\]
Here, and in the rest of this discussion, $R_{a,b}$ is a function
that returns a random variable uniformly distributed over the interval
$[a,b]$.

This works fine for $\Delta$, but for $p(\Delta)$, the points
$p(\tilde x_k)$ are not uniformly distributed over $p(\Delta)$,
but are distributed with a density proportional to
$1/|p'(\tilde x_k)|^2$.  So to restore the uniform density we need
to reject this value of $\tilde x_k$ with probability proportional
to $|p'(\tilde x_k)|^2$.  Noticing that the condition
$\sum_{n=2}^{\tt deg\_p} n |c_n| = 1$ implies that
$|p'(\tilde x_k)| \le 2$, we
do this by rejecting if $|p'(\tilde x_k)|^2 < R_{0,4}$.
(This makes use of the function {\tt calc\_mod\_dp2} which calculates
$|p'(x)|^2$.)

\section{Solving the Equation}

The main loop of the program is in the function {\tt draw\_euler2d}.
Most of the drawing operations are contained in this function, and
the numerical aspects are sent to the function {\tt ode\_solve}.
But there is an aspect of this that I would like
to discuss in the next section, and so we will look at a simple method for
numerically solving differential equations.

The Euler Method
(nothing to do with the Euler Equation), is as
follows.  Pick a small number $h$ --- the time step (in
the program call {\tt delta\_t}).  Then we approximate
the solution of the equation:
\begin{equation}
\label{method-simple}
\tilde x_k(t+h) = \tilde x_k(t) + h \tilde\alpha_k(t) .
\end{equation}
The more sophisticated methods we use are variations of
the Euler Method, and so the discussion in the following section
still applies.

In the program, the quantities $\tilde\alpha_k$, given by
equations (\ref{tildex-p2}) are calculated by the function
{\tt derivs}
(which in turns calls {\tt calc\_all\_mod\_dp2} to
calculate $|p'(\tilde x_k)|^2$ at all the points).

\section{Subtle Perturbation}

Added later: the scheme described here seems to not be that effective,
so now it is not used.

One problem using a numerical scheme such as the Euler Method occurs
when the points $\tilde x_k$ get close to the boundary
of $\Delta$.  In that case, it is possible that the new
points will be pushed outside of the boundary.  Even if they
are not pushed out of the boundary, they may be much closer
or farther from the boundary than they should be.
Our system of equations is very sensitive to how close points
are to the boundary --- points with non-zero vorticity
(``vortex points'') that are close to the boundary travel
at great speed alongside the boundary, with speed that is
inversely proportional to the distance from the boundary.

A way to try to mitigate this problem is something that I call
``subtle perturbation.''
We map the points in
the unit disk to points in the plane using the map
\begin{equation*}
F(x) = f(|x|) \frac x{|x|} ,
\end{equation*}
where $f:[0,1]\to[0,\infty]$ is an increasing continuous
bijection.  It turns out that a good choice is
\begin{equation*}
f(t) = -\log(1-t) .
\end{equation*}
(The reason for this is that points close to each other
that are a distance
about $r$ from the boundary will be pushed around so that
their distance from each other is about multiplied by the
derivative of $\log r$, that is, $1/r$.)
Note that the inverse of this function is given by
\begin{equation*}
F^{-1}(x) = f^{-1}(|x|) \frac x{|x|} ,
\end{equation*}
where
\begin{equation*}
f^{-1}(t) = 1-e^{-t} .
\end{equation*}

So what we could do is the following: instead of working with
the points $\tilde x_k$, we could work instead with the points
$y_k = F(\tilde x_k)$.  In effect this is what we do.
Instead of performing the computation (\ref{method-simple}),
we do the calculation
\begin{equation*}
y_k = F(\tilde x_k(t)) + h {\cal A}(\tilde x_k) \tilde\alpha_k(t)
\end{equation*}
where
${\cal A}(x)$ is the matrix of partial derivatives of $F$:
\begin{equation*}
{\cal A}(x) =
\frac{f(|x|)}{|x|}
\left[
\begin{matrix}
1 & 0\\
0 & 1
\end{matrix}
\right]
+ \frac1{|x|}
  \left(\frac{f'(|x|)}{|x|} - \frac{f(|x|)}{|x|^2}\right)
\left[
\begin{matrix}
x_{1}^2   & x_{1} x_{2}\\
x_{1} x_{2} & x_{2}^2
\end{matrix}
\right],
\end{equation*}
and then compute
\begin{equation*}
\tilde x_k(t+h) = F^{-1}(y_k).
\end{equation*}
These calculations are done in the function {\tt perturb}, if
the quantity {\tt SUBTLE\_PERTURB} is set.

\section{Drawing the Points}

As we stated earlier, most of the drawing functions are contained
in the function {\tt draw\_euler2d}.  If the variable
{\tt hide\_vortex} is set (and the function {\tt init\_euler2d}
will set this with probability $3/4$), then we only display
the points $\tilde x_k$ for ${\tt Nvortex} < k \le N$.  If
{\tt hide\_vortex} is not set, then the ``vortex points''
$\tilde x_k$ ($1 \le k \le {\tt Nvortex}$) are displayed in white.
In fact the points $p(\tilde x_k)$ are what are put onto the screen,
and for this we make use of the function {\tt calc\_all\_p}.

\section{Addition to Program: Changing the Power Law}

A later addition to the program adds an option {\tt eulerpower},
which allows one to change the power law that describes how
the vortex points influence other points.  In effect, if this
option is set with the value $m$, then the Biot-Savart Kernel
is replace by
$$ K_1(x,y) = \frac{(x-y)^\perp}{|x-y|^{m+1}}, $$
and
$$ K_2(x,y) = K_1(x,y) - |y|^{1-m} K_1(x,y) .$$
So for example, setting $m=2$ corresponds to the
quasi-geostrophic equation.  (I haven't yet figured out
what $K_p$ should be, so if $m \ne 1$ we use the unit circle
as the boundary.)

\begin{thebibliography}{9}

\bibitem{BF} Richard L. Burden, J. Douglas Faires, Numerical Analysis,
sixth edition, Brooks/Cole, 1996.

\bibitem{C} Alexandre J. Chorin, Vorticity and Turbulence,
Applied Mathematical Sciences, Vol 103, Springer Verlag, 1994.

\end{thebibliography}

\end{document}


\chapter{Derivation of Pipe-flow Process Model}
\label{AppendixB}
Since the derivation of the model is based on physical relationships and conservation principles it would be convenient to study at least once the consecutive steps performed in order to obtain it. This may provide reader with better understanding of the process as well as present assumptions and approximations that are utilized. The model is derived starting from two equations: equation of continuity and equation of motion. The derivation provided below is based on %\cite{billmann_isermann,keerthi_phd}.

\section{Equation of Continuity}

We start from definition of the mass:

\begin{equation}
\label{eq:mass}
m = \rho A \Delta z
\end{equation}

Considering mass change in time, i. e. mass-flow

\begin{equation}
\label{eq:massflow}
q = \frac{\partial m}{\partial t} = \rho A  w
\end{equation}

and then, introducing principle of conservation for elementary pipe segment we obtain

\begin{equation}
\label{eq:massflow2}
\frac{\partial m}{\partial t} = \rho A w - A\left(w + \frac{\partial w}{\partial z}\Delta z\right) \left( \rho + \frac{\partial \rho}{\partial z}\Delta z\right)
\end{equation}

Substituting (\ref{eq:mass}) to  (\ref{eq:massflow2}) we obtain

\begin{equation}
\label{eq:massflow3}
\frac{\partial \left( \rho A \Delta z \right)}{\partial t} = \rho A  w - A\left( \rho w + \Delta z \left( \rho \frac{\partial w}{\partial z} +w \frac{\partial \rho}{\partial z} \right)+\frac{\partial w}{\partial z} \Delta z  \frac{\partial \rho}{\partial z} \Delta z  \right)
\end{equation}

where the last term $\frac{\partial w}{\partial z} \Delta z  \frac{\partial \rho}{\partial z} \Delta z$ is small enough to be neglected. Then, substituting

\begin{equation}
\label{eq:ms4}
\rho \frac{\partial w}{\partial z} +w \frac{\partial \rho}{\partial z} = \frac{\partial}{\partial z} \left( \rho w \right)
\end{equation}

as

\begin{equation}
\label{eq:ms5}
\frac{\partial}{\partial z} \left( \rho w \right) + \frac{\partial \rho}{\partial t} = 0
\end{equation}

Assuming, that the process is isothermal we can introduce

\begin{equation}
\label{eq:ms6}
\nu = \sqrt{\frac{p}{\rho}}
\end{equation}

thus, substituting (\ref{eq:massflow}) and (\ref{eq:ms6}) to (\ref{eq:ms5}) we obtain first equation of our model:

\begin{equation}
\label{eq:cont_fin3}
\frac{A}{\nu^2} \frac{\partial p}{\partial t} + \frac{\partial q}{\partial z} = 0
\end{equation}

\section{Equation of Motion}

Here we start from definition of momentum:

\begin{equation}
\label{eq:mm1}
M = \rho A \Delta z w
\end{equation}

then,using principle of conservation:

\begin{equation}
\label{eq:mm2}
\frac{\partial \left( \rho A w \Delta z \right)}{\partial t} = A \left(p + \frac{\rho w^2}{2} \right) - A\left(p + \frac{\partial p}{\partial z} \Delta z + \frac{\rho w^2}{2} + \frac{\partial}{\partial z} \left( \frac{\rho w^2}{2} \right) \Delta z \right) - F - Y
\end{equation}

By rearranging equation presented above, we obtain:

\begin{equation}
\label{eq:mm3}
\frac{\partial \left( \rho A w \Delta z \right)}{\partial t} = A \left(p + \frac{\rho w^2}{2} \right) - A\left(\left(p + \frac{\rho w^2}{2} \right) + \Delta z \left( \frac{\partial p}{\partial z} +  \frac{\partial}{\partial z} \left( \frac{\rho w^2}{2} \right) \right)\right) - F - Y
\end{equation}

Substituting

\begin{equation}
\label{eq:mm4}
 \frac{\partial}{\partial z} \left(p +  \frac{\rho w^2}{2} \right) = \frac{\partial p}{\partial z} +  \frac{\partial}{\partial z} \left( \frac{\rho w^2}{2} \right)
\end{equation}

into (\ref{eq:mm3}) we will obtain
\begin{equation}
\label{eq:mm5}
\frac{\partial \left( \rho  w  \right)}{\partial t} = -  \frac{\partial}{\partial z} \left(p +  \frac{\rho w^2}{2} \right) - F- Y
\end{equation}

Again, assuming that the process is isothermal we obtain:

\begin{equation}
\label{eq:mm_fin}
\frac{1}{A} \frac{\partial q}{\partial t} + \left( 1 - \frac{q^2 \nu^2}{2 A^2 p}\right) \frac{\partial p}{\partial z} + \frac{q \nu^2}{A^2 p} \frac{\partial q}{\partial z}= - \frac{\lambda \nu^2}{2DA^2} \frac{q|q|}{p} - \frac{g sin \alpha}{\nu^2} p
\end{equation}

Taking into account, that if $w^2 \ll \nu^2$, i. e. velocity of the fluid is significantly lower than velocity of the sound in this fluid, we may neglect term $ \frac{q^2 \nu^2}{2 A^2 p}$. Moreover, if the pipeline is long, then we may neglect term $ \frac{q \nu^2}{A^2 p}$. Thus, we obtain second equation that is included in physical, continuous time model:

\begin{equation}
\label{eq:momen_fin3}
\frac{1}{A} \frac{\partial q}{\partial t} + \frac{\partial p}{\partial z} = - \frac{\lambda \nu^2}{2DA^2} \frac{q|q|}{p} - \frac{g sin \alpha}{\nu^2} p
\end{equation}


\documentclass[12pt]{article}

\title{Tic-Tac-Toe with SAT}
\author{William John Holden}
\date{\today}

\usepackage{amsmath,amsthm}
\usepackage{mathtools}
%\usepackage{parskip}
\usepackage{amssymb}
\usepackage[margin=1.0in]{geometry}
\usepackage[hyphens]{url}
\usepackage[hidelinks]{hyperref}
\usepackage{listings}
\lstset{basicstyle=\ttfamily,columns=flexible,frame=single,breaklines=true,linewidth=1\textwidth}

\newcommand{\tictactoe}[9]{\begin{center}\begin{tabular}{ c | c | c }{$#1$} & {$#2$} & {$#3$} \\ \hline {$#4$} & {$#5$} & {$#6$} \\ \hline {$#7$} & {$#8$} & {$#9$} \end{tabular}\end{center}}

\begin{document}

\maketitle

\section{Introduction}

In this exercise we will construct a series of boolean satisfiability clauses to find a winning move for the game Tic-Tac-Toe.
Throughout this exercise, assume that we are playing as $x$ and that it is our turn to play for any game shown.
We treat the SAT solver as a ``black box'' and will not worry about how it works.
A SAT solver accepts a \textit{formula} of \textit{clauses} containing boolean \textit{variables} in \textit{conjunctive normal form} (CNF).
An example of a formula in CNF is

\begin{equation}(A \vee \neg B)(C)(C \vee D \vee E)(B \vee \neg C).\end{equation}

This formula is \textit{satisfiable} with the literals $A=T$, $B=T$, $C=T$, $D=F$, and $E=F$.
This is not the only satisfying assignment.
$D$ and $F$ may each take the values $T$ or $F$, but we are constrained in $C=T$.
If $C=F$ then the clause $(C)$ cannot be satisfied, and any unsatisfied clause falsifies the entire formula.

Some formulas cannot be satisfied under any assignment of literals to variables. An obvious example is $(A)(\neg A)$.
A less obvious example is

\begin{equation}(A \vee \neg B)(B \vee \neg C)(C)(\neg A \vee \neg B \vee \neg C).\end{equation}

In this case, the $(C)$ clause implies $C=T$, which means $B=T$, which then implies $A=T$, but now the final clause is $(F \vee F \vee F)$ which falsifies the formula.

A \textit{SAT solver} is a computer program for finding a set of satisfying literals or proving if no such satisfying assignment exists.
Though the boolean satisfiability problem is NP-complete, state of the art SAT solvers are very efficient in practice.
Martin Horenovsky has a very nice article at \url{https://codingnest.com/modern-sat-solvers-fast-neat-underused-part-1-of-n/} on this subject.
I originally learned of the satisfiability problem and SAT solvers attending a course on NP-Complete Problems from Alexander Kulikov (\url{https://www.edx.org/course/np-complete-problems-uc-san-diegox-algs203x}).

Others have successfully solved Sudoku puzzles using SAT solvers.
In this exercise, we will do the same for tic-tac-toe.
Of course, tic-tac-toe is much less computationally demanding than Sudoku.
Tic-tac-toe has a smaller board ($3 \times 3$ instead of $9 \times 9$) and it has fewer constraints.
If we consider the size of the tic-tac-toe puzzle to be $n$ for dimensions $\sqrt{n} \textrm{ rows} \times \sqrt{n} \textrm{ columns} = x \times y$ then we can iterate over every row and column to find a winning move in

\begin{align}O(x \times y) + O(y \times x ) + O(2 \textrm{ diagonals} \times \sqrt{n}) &= \\
O(xy)+O(xy)+O(2\sqrt{n})
&=O(n).
\end{align}

Using an exponential time tool for a linear time problem may seem crazy, but the goal here is to learn to \textit{reduce} problems to SAT.

\section{Reduction to SAT}

\subsection{Variables}

We need to express the positions of our tic-tac-toe board as boolean variables. This is pretty easy. We assign a variable letter to each position in reading order:

\tictactoe{a}{b}{c}{d}{e}{f}{g}{h}{i}

There are actually \textit{three} possible states for each position: X, O, and unassigned. However, in this problem we do not need to concern ourselves with the third state.
We will consider a variable to be ``true'' if it is marked wtih X and false otherwise.

\subsection{Contraint 1: Winning Combinations}

There are eight total winning combinations in tic-tac-toe: satisfy any of the three rows, any of the columns, or either of the diagonals.
I should note that the word \textit{or} in the previous sentence is an inclusive or; winning combinations allow for both a row and a column, both a row and a diagonal, and both a column and a diagonal. For example, the assignment

\tictactoe{o}{o}{x}{o}{o}{x}{x}{x}{}

can be won by setting $i=x$.  We can express the eight winning combinations as

\begin{equation}(a \wedge b \wedge c) \vee (d \wedge e \wedge f) \vee (g \wedge h \wedge i)
\vee
(a \wedge d \wedge g) \vee (b \wedge e \wedge h) \vee (c \wedge f \wedge i)
\vee
(a \wedge e \wedge i) \vee (c \wedge e \wedge g).\end{equation}

This long formula is in \textit{disjunctive normal form} (DNF) and not the CNF we want.
We could use rules of discrete mathematics, such as DeMorgan's Law and Distributivity, but in this paper on computational tools let us instead use \textit{Mathematica's} \texttt{BooleanConvert} function (\url{https://reference.wolfram.com/language/ref/BooleanConvert.html}).
\texttt{BooleanConvert} accepts an expression and converts it to the desired form.
(\textit{Mathematica} functionality is available at \url{https://www.wolframalpha.com}).

\begin{lstlisting}
In[1] := BooleanConvert[((a && b && c) || (d && e && f) || (g && h && i) || (a && d && g) || (b && e && h) || (c && f && i) || (a && e && i) || (c && e && f)),"CNF"]
Out[1] := (a || b || c || d || g) && (a || b || f || g) && (a || b || f ||  i) && (a || c || d || h) && (a || c || e || h) && (a || e || f || g) && (a || e || i) && (a || f || h) && (b || c || d || i) && (b || d || e || f || h) && (b || d || e || i) && (b || d || f ||  i) && (b || e || f || g) && (b || e || g || i) && (b || f || g || i) && (c || d || e || h) && (c || d || e || i) && (c || d || h || i) && (c || e || g) && (c || f || g || h || i)
\end{lstlisting}

Now we have our first set of constraints that we can provide to a SAT solver.
An assignment of literals to the variables $a$ through $i$ is a winning move if all three variables in any row, column, or diagonal are $x$.

\subsection{Contraint 2: Board Configuration}

The game will be shown with some configuration of $x$ and $o$ settings on the board.
For each position with an $x$ we will create a clause of unit length where that variable must be true.
All positions marked $o$ induce a clause where that variable must be false.
For example, given the game

\tictactoe{x}{o}{x}{o}{x}{o}{o}{}{}

we will set $a=T$, $b=F$, $c=T$, $d=F$, $e=T$, $f=F$, $g=F$ with the clauses

\begin{equation}(a)(\neg b)(c)(\neg d)(e)(\neg f)(\neg g).\end{equation}

These clauses must be generated based on game state.

\subsection{Constraint 3: One Move}

We assume that it is our turn to play, and we only get to make one move.
One might be tempted to construct an ``exactly-one-of'' clause using the \textit{exclusive or} operation.
However, this is not correct.
Consider $A \oplus B \oplus C$ for $A = B = C = T$. Then $A \oplus B = F$, but $(A \oplus B) \oplus C = (F) \oplus T = T$.
We need a different ``gadget'' to constrain our SAT solver to mark only one square of our tic-tac-toe board with $x$.

\end{document}


\subsection{Fermat's method}

Identify the integer as the difference of two squares, and use this.

\(x=a.b\)

We use the midpoint of the two as \(c=\dfrac{a+b}{2}\)

This only works for odd numbers. If we have

The we have:

\begin{itemize}
\item \(a=c+d\)
\item \(b=c-d\)
\item \(x=(c+d)(c-d)\)
\item \(x=c^2-d^2\)
\end{itemize}

We can test this by trying \(a\) to get \(a^2-x\), and seeing if this is a square number.


\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}

\title{MATH 542 Homework 10}
\author{Saket Choudhary\\skchoudh@usc.edu}

\begin{document}
\maketitle
\subsection*{Problem 3c.1}

\subsubsection*{Problem 3c.1.a}
\begin{align*}
var[S^2]  &= Var[\frac{Y'(I_n-P)Y}{n-p}]\\
&= \frac{1}{(n-p)^2}Var[Y'(I_n-P)Y]\\
&= \frac{1}{(n-p)^2}\times (2\sigma^4(n-p))\\
&= \frac{2\sigma^4}{n-p}
\end{align*}

\subsubsection*{Problem 3c.1.b}
\begin{align*}
A_1 &= \frac{1}{n-p+2}[I_n-X(X'X)^{-1}X']\\
&= \frac{R}{n-p+2}\\
E[(Y'A_1Y-\sigma^2)^2] &= Var(Y'A_1Y-\sigma^2) + (E[Y'A_1Y-\sigma^2])^2\\
&= Var(Y'A_1Y) + (E[Y'A_1Y]-\sigma^2)^2\\
&= \frac{Var(Y'RY)}{(n-p+2)^2} + ( \frac{E[Y'RY]}{(n-p+2)}-\sigma^2)^2\\
&= \frac{2\sigma^4(n-p)}{(n-p+2)^2} + (\frac{\sigma^2(n-p)}{n-p+2}-\sigma^2)^2 \text{ using 3.12 from textbook}\\
&= \frac{2\sigma^4(n-p)}{(n-p+2)^2} + \frac{4\sigma^4}{(n-p+2)^2}\\
&= \frac{2\sigma^4}{n-p+2}
\end{align*}

\subsubsection*{Problem 3c.1.c}
\begin{align*}
E[Y'A_1Y] &= \frac{E[Y'RY]}{n-p+2}\\
&= \frac{\sigma^2(n-p)}{n-p+2}\text{ using 3.12 from textbook}\\
MSE[Y'A_1Y] &= E[(Y'A_1Y-\sigma^2)^2]\\
&= \frac{2\sigma^4}{n-p+2}\\
MSE[S^2] &= E[S^2-(E[S^2])^2]\\
&= Var(S^2)\\
&= \frac{2\sigma^4}{n-p}\\
&<\frac{2\sigma^4}{n-p+2}\\
&\leq MSE[Y'A_1Y]
\end{align*}

\section*{Problem 3d.1}
\subsubsection*{Problem 3d.1.a}
Given $Y_i \sim N(\theta, \sigma^2)$ or $Y_i = \theta + \epsilon_i$ where $\epsilon_i \sim N(0, \sigma^2)$ \\
$\mathbf{Y} = \mathbf{1_n}\theta + \mathbf{\epsilon}$ thus $\hat{\theta} = (\mathbf{1_n}'\mathbf{1_n})^{-1}\mathbf{1_n'}Y = \frac{1}{n}\mathbf{1_n'}Y  = \bar{Y}$

Thus,  using theorem  $3.5(ii)$ $\bar{Y}$ and $S^2=\sum_i(Y_i-\bar{Y})^2$ are independent

\subsubsection*{Problem 3d.1.b}
Borrowing from part (a)  we have: $RSS=Q=\sum_i(Y_i-\bar{Y})^2$ $\implies$ using  theorem $3.5(iii)$:

$RSS/\sigma^2 \sim \chi^2_{n-1}$

\subsection*{Problem 3d.2}
\begin{align*}
RSS &= Y'(I_n-P)Y\\
&= Y'(I_n-P)Y-\beta'X'(I-P)(Y-X\beta)+Y'(I-P)(-X\beta) \text{ both terms are zero using PX=P and P=P'} \\
&= (Y-X\beta)'(I_n-P)(Y-X\beta)\\
&= \epsilon'(I_n-P)\epsilon
\end{align*}

\begin{align*}
(\hat{\beta}-\beta)'X'X(\hat{\beta}-\beta) &= Z'Z\\
Z &= X(\hat{\beta}-\beta)\\
&= X((X'X)^{-1}X'Y-(X'X)^{-1}X'X\beta)\\
&= P(Y-X\beta)\\
&= P\epsilon\\
(\hat{\beta}-\beta)'X'X(\hat{\beta}-\beta) &= \epsilon'P'P\epsilon
\end{align*}

\begin{align*}
Cov[RSS, (\hat{\beta}-\beta)'X'X(\hat{\beta}-\beta)] &= Cov[\epsilon'(I_n-P)\epsilon, \epsilon'P'P\epsilon]\\
&=  Cov[\epsilon'(I_n-P)\epsilon, \epsilon'PP\epsilon] \text{ using } P'=P\\
&=  Cov[\epsilon'(I_n-P)\epsilon, \epsilon' P\epsilon] \text{ using } PP = P\\
&= \sigma^2(I-P)P \\
&= 0
\end{align*}
Thus, $RSS$ and $(\hat{\beta}-\beta)'X'X(\hat{\beta}-\beta)$ are indepedent

\subsection*{Problem 3.12}

\begin{align*}
Y &= X\beta + \epsilon\\
\bar{Y} &= \frac{1}{n}\mathbf{1_n}Y\\
\sum_i(Y_i-\hat{Y_i})^2  &= (Y-X\hat{\beta})'(Y-X\hat{\beta})\\
&= (Y-X(X'X)^{-1}X'Y)'(Y-X(X'X)^{-1}X'Y)\\
&= (Y-PY)'(Y-PY)\\
&= Y'(I-P)'(I-P)Y\\
&= Y'(I-P)Y\text{ using idempotency of } I-P\\
Cov[\frac{1}{n}\mathbf{1_n}Y, (I-P)Y]\\
&= \frac{1}{n}\mathbf{1_n}Cov[Y](I-P)'\\
&= \sigma^2(n-p)\frac{1}{n}\mathbf{1_n}(I-P)'
\end{align*}
Since the first column of the design matrix is all 1, $1_n$ belongs to the column space of $X$  and is orthogonal to $(I-P)'$ (P being the projection matrix) $\implies Cov[\frac{1}{n}\mathbf{1_n}Y, (I-P)Y] = 0$
\end{document}


\lab{Poisson's equation}{Poisson's equation}
\label{lab:poisson2d}

Suppose that we want to describe the distribution of heat throughout a region $\Omega$.
Let $h(x)$ represent the temperature on the boundary of $\Omega$ ($\partial \Omega$), and let $g(x)$ represent the initial heat distribution at time $t = 0$.
If we let $f(x,t)$ represent any heat sources/sinks in $\Omega$, then the flow of heat can be described by the boundary value problem (BVP)
\begin{align}
	\begin{split}
		& { } u_t = \triangle u + f(x,t), \quad x \in \Omega, \quad t >0,\\
		& { }u(x,t) = h(x), \quad x \in \partial \Omega, \\
		& { }u(x,0) = g(x).
	\end{split}
\end{align}
When the source term $f$ does not depend on time, there is often a steady-state heat distribution $u_{\infty}$ that is approached as $t \to \infty$.
This steady state $u_{\infty}$ is a solution of the BVP
\begin{align}
	\begin{split}
		& { }  \triangle u + f(x) = 0, \quad x \in \Omega,\\
		& { }u(x,t) = h(x), \quad x \in \partial \Omega.
	\end{split}
\end{align}

This last partial differential equation, $\triangle u = -f$, is called Poisson's equation.
This equation is satisfied by the steady-state solutions of many other evolutionary processes.
Poisson's equation is often used in electrostatics, image processing, surface reconstruction, computational fluid dynamics, and other areas.

\section*{Poisson's equation in two dimensions}

 Consider Poisson's equation together with Dirichlet boundary conditions on a square domain $R = [a,b]\times [c,d]$:
 \begin{align}
	\begin{split}
 	u_{xx} + u_{yy} &= f,\quad x \text{ in } R \subset \mathbb{R}^2,\\
 	u &= g, \quad x \text{ on } \partial R.
	\end{split}\label{eqn:2d_poisson}
\end{align}
Let $a = x_{-1}, x_0, \ldots, x_{N-1} = b$ be a partition of $[a,b]$, and let $c = y_{-1}, y_0, \ldots, y_{N-1} = d$ be a partition of $[c,d]$.
Suppose that there are $N+1$ evenly spaced points, so that $N$ is the number of subintervals in each dimension, and $x_i, y_j$ are given by
\begin{align*}
	x_i &= a + (i+1)h, \\
	y_j &= c + (j+1)h,
\end{align*}
for $i,j = 0, \ldots, N-2$, where $h = x_i-x_{i-1} = y_i-y_{i-1}$.
We look for an approximation $U_{i,\,j}$ on the grid $\{(x_i,y_j)\}_{i,j=-1}^{N-1}$.

Recall that
 \begin{align*}
 \triangle u &= u_{xx}(x,y) + u_{yy}(x,y) \\
&= \frac{u(x+h,y) - 2u(x,y)+ u(x-h,y)}{h^2} \\
 & \qquad{}+
 \frac{u(x,y+h) - 2u(x,y)+ u(x,y-h)}{h^2} + \mathcal{O}(h^2).
 \end{align*}
 We replace $\triangle $ with the finite difference operator $\triangle_h$, defined by
 \begin{align*}
 \triangle_h U_{ij} &= \frac{U_{i+1,\,j} - 2U_{i,\,j} + U_{i-1,\,j}}{h^2} + \frac{U_{i,\,j+1} - 2U_{i,\,j}+ U_{i,\,j-1}}{h^2},\\
&= \frac{1}{h^2}(U_{i-1,\,j} + U_{i+1,\,j} + U_{i,\,j-1} + U_{i,\,j+1}-4U_{i,\,j}).
 \end{align*}

 Then the set of equations
\[
\triangle_h U_{ij} = f_{ij}, \quad i,j = 0,\ldots,N-2,
\]% $i,j = 1,\ldots,m$
can be written in matrix form as
 \[AU + p +  q  = f.\]

$A$ is a block tridiagonal matrix, given by
\begin{align}
	\frac{1}{h^2}
\begin{bmatrix}
T & I & &  &\\
I &T & I & &\\
&\ddots  & \ddots & \ddots & \\
&  & I & T & I \\
&  &  & I & T\end{bmatrix}\label{poisson2d:matrixA}
\end{align}

where $I$ is the $N-1\times N-1$ % $m\times m$
identity matrix, and $T$ is the tridiagonal matrix
\[\begin{bmatrix}
-4 & 1 & &  &\\
1 &-4 & 1 & &\\
&\ddots  & \ddots & \ddots & \\
&  & 1 & -4 & 1 \\
&  &  & 1 & -4 \end{bmatrix}.\]

The vector $U$ is given by
\[U = \begin{bmatrix} U^0 \\ U^1 \\ \\ U^{N-2} \end{bmatrix} \text{ where } U^j =
\begin{bmatrix} U_{0,\,j} \\ U_{1,\,j} \\ \\ U_{N-2,\,j} \end{bmatrix} \text{ for each } j, \text{ }0\leq j \leq N-2.\]

% \[U = \begin{bmatrix} U^1 \\ U^2 \\ \\ U^m \end{bmatrix} \text{ where } U^j =
% \begin{bmatrix} U_{1,\,j} \\ U_{2,\,j} \\ \\ U_{m,\,j} \end{bmatrix} \text{ for each } j, 1\leq j \leq m\]
% So $U^j$ represents the $j$th row of interior points in our grid, where $y_j = jh$

The vectors $p$ and $q$ come from the boundary conditions of \eqref{eqn:2d_poisson}, and are given by
\[p = \begin{bmatrix} p^0 \\ \ldots \\ \\ p^{N-2} \end{bmatrix}, \quad  q = \begin{bmatrix} q^0 \\ \ldots \\ \\ q^{N-2} \end{bmatrix},\]
where
\[p^j = \frac{1}{h^2} \begin{bmatrix} g_{-1,\,j} \\ 0 \\ \vdots \\0\\ g_{N-1,\,j} \end{bmatrix} ,\,\,\, 0 \leq j \leq N-2,\]
and
\[q^0 = \frac{1}{h^2}\begin{bmatrix} g_{0,-1}  \\ g_{1,-1} \\ \vdots \\ g_{N-3,-1}\\ g_{N-2,-1} \end{bmatrix}, \quad q^{N-2} = \frac{1}{h^2}\begin{bmatrix} g_{0,N-1} \\ g_{1,N-1} \\ \vdots \\ g_{N-3,N-1}\\ g_{N-2,N-1} \end{bmatrix}, \quad q^{j} = \begin{bmatrix} 0 \\ 0 \\ \vdots \\ 0 \\ 0 \end{bmatrix} ,\,\,\, 1 \leq j \leq N-3.\]

% The vector $q$ is given by $u = [q^0 \ldots q^{N-2}]^T$, %   $u = [q^1 \ldots q^m]^T$,
% where
% \[q^j = \frac{1}{h^2} \begin{bmatrix} g_{0,\,j} \\ 0 \\ \vdots \\0\\ g_{m+1,\,j} \end{bmatrix} , \,\,\, 2 \leq j \leq m-1\]
% and
% \[q^1 = \frac{1}{h^2}\begin{bmatrix} g_{1,0} + g_{0,1} \\ g_{2,0} \\ \vdots \\ g_{m-1,0}\\ g_{m,0} + g_{m+1,1}\end{bmatrix}, \quad q^m = \frac{1}{h^2}\begin{bmatrix} g_{1,m+1} + g_{0,m}\\ g_{2,m+1} \\ \vdots \\ g_{m-1,m+1}\\ g_{m,m+1} + g_{m+1,m}\end{bmatrix}\]

% \begin{problem}
% Find the solution $u$ of the 2D Poisson equation with the given Dirichlet boundary conditions:
% \begin{align*}
% 	\Delta u &= -\pi^2 \sin(\pi x)\sin(\pi y), \quad (x,y) \in [0,1]\times [0,1], \\
% 	u(x,0) &= 1-x, \\
% 	u(x,1) &= 1-2x, \\
% 	u(0,y) &= 1, \\
% 	u(1,y) &= -y.
% \end{align*}
%
% Graph your solution, and demonstrate convergence of the numerical approximation by
% creating a log-log plot of the error $E(h).$
% \end{problem}

% The matrix $A$ is sparse, and so we can use several functions from the package \texttt{scipy.sparse.linalg}.
% In particular, we use the functions \texttt{spdiags} and \texttt{spsolve}.
%
%  \begin{verbatim}
% D1,D2,D3 = -4*np.ones((1,m**2)), np.ones((1,m**2)), np.ones((1,m**2))
% Dm1, Dm2 = np.ones((1,m**2)), np.ones((1,m**2))
% for j in range(0,D2.shape[1]):
% 	if (j%m)==m-1:
% 		D2[0,j]=0
% 	if (j%m)==0:
% 		D3[0,j]=0
% diags = np.array([0,-1,1,-m,m])
% data = np.concatenate((D1,D2,D3,Dm1,Dm2),axis=0) # This stacks up rows
% A = 1./h**2.*spdiags(data, diags, m**2,m**2).asformat('csr') # This ap
%  \end{verbatim}

\begin{comment}
\subsection{2D Heat Equation}
Recall that the collection of finite difference equations
\[\nabla^2_h U_{ij} = 0, \quad 1 \leq i,j\leq m\]
can be written in matrix form as
\[AU + q  = 0\]

The Crank-Nicolson method for the 2D heat equation is given by
\[U_{i,\,j}^{n+1}- U_{i,\,j}^{n} = \frac{\Delta t}{2}(\nabla_h^2 U_{i,\,j}^{n} + \nabla_h^2 U_{i,\,j}^{n+1}) \text{ for each } 1 \leq i,j \leq m\]
is a second order accurate in both space and time. Basically we're using a midpoint scheme in time,
and a trapezoidal scheme in space. The resulting method is implicit, and can be written in matrix form as
\begin{align*}
	IU^{n+1} &= IU^n + \frac{\Delta t}{2}(AU^n + q + AU^{n+1} + q)\\
	(I - \frac{\Delta t}{2}A)U^{n+1}&= (I + \frac{\Delta t}{2}A)U^n + \Delta t q
\end{align*}

% TODO: What size must the time step be to ensure stability?

We will need to take many time steps, where many equations must be solved with the matrix $(I - \frac{\Delta t}{2}A)$.
The function \texttt{factorized} from \texttt{scipy.sparse.linalg} computes the LU decomposition of the matrix.
This decomposition reduces the time required for solving consecutive time steps.
\end{comment}

The following code implements the greater portion of the finite difference method; it leaves the construction of the matrix $A$ in \eqref{poisson2d:matrixA} to problem \ref{poisson2d:laplace}.

\begin{lstlisting}
from __future__ import division
from scipy.sparse import spdiags
from scipy.sparse.linalg import spsolve

def poisson_square(a1,b1,c1,d1,n,bcs, source):
	# n = number of subintervals
	# We discretize in the x dimension by
	# a1 = x_0 < x_1< ... < x_n=b1, and
	# We discretize in the y dimension by
	# c1 = y_0 < y_1< ... < y_n=d1.
	# This means that we have interior points
	# {x_1, ..., x_{n-1}}\times {y_1, ..., y_{n-1}}
	# or {x_1, ..., x_m}\times {y_1, ..., y_m} where m = n-1.
	# In Python, this is indexed as
	# {x_0, ..., x_{m-1}}\times {y_0, ..., y_{m-1}}
	# We will have m**2 pairs of interior points, and
	# m**2 corresponding equations.
	# We will organize these equations by their
	# y coordinates: all equations centered
	# at (x_i, y_0) will be listed first,
	# then (x_i, y_1), and so on till (x_i, y_{m-1})
	delta_x, delta_y, h, m = (b1-a1)/n, (d1-c1)/n, (b1-a1)/n, n-1

	####    Construct the matrix A    ####

	####    Here we construct the vector b    ####
	b, Array = np.zeros(m**2), np.linspace(0.,1.,m+2)[1:-1]
	# In the next line, source represents
	# the inhomogenous part of Poisson's equation
	for j in xrange(m):
		b[j*m:(j+1)*m] = source(a1+(b1-a1)*Array, c1+(j+1)*h*np.ones(m) )

    # In the next four lines, bcs represents the
	# Dirichlet conditions on the boundary
	# y = c1+h, d1-h
	b[0:m] -= h**(-2.)*bcs(a1+(b1-a1)*Array,c1*np.ones(m))
	b[(m-1)*m:m**2] -= h**(-2.)*bcs(a1+(b1-a1)*Array,d1*np.ones(m))
	# x = a1+h, b1-h
	b[0::m] -= h**(-2.)*bcs(a1*np.ones(m),c1+(d1-c1)*Array)
	b[(m-1)::m] -= h**(-2.)*bcs(b1*np.ones(m),c1+(d1-c1)*Array)

    ####    Here we solve the system A*soln = b    ####
	soln = spsolve(A,b)

	# We return the solution, and the boundary values,
	# in the array z.
	z = np.zeros((m+2,m+2) )
	for j in xrange(m):
		z[1:-1,j+1] = soln[j*m:(j+1)*m]

	x, y = np.linspace(a1,b1,m+2), np.linspace(c1,d1,m+2)
	z[:,0], z[:,m+1]  = bcs(x,c1*np.ones(len(x)) ), bcs(x,d1*np.ones(len(x)) )
	z[0,:], z[m+1,:] = bcs(a1*np.ones(len(x)),y), bcs(b1*np.ones(len(x)),y)
	return z

\end{lstlisting}

\begin{problem}
Construct the matrix $A$ in \eqref{poisson2d:matrixA}. Make sure your matrix is sparse. Then use the code
above to solve the boundary value problem
\begin{align}
	\begin{split}
	\Delta u = 0, &{}\quad x \in [0,1]\times [0,1],\\
	u(x,y) = x^3, &{}\quad (x,y) \in \partial ([0,1]\times [0,1]).
	\end{split}
	\label{poisson2d:laplace}
\end{align}
Plot the 3D solution with $n=100$.
\end{problem}

\begin{figure}
\includegraphics[width=\textwidth]{Laplace.png}
\caption{The solution of \eqref{poisson2d:laplace}.}
\end{figure}

\section*{Poisson's equation and conservative forces}
In physics Poisson's equation is used to describe the scalar potential of a conservative force.
In general
\[ \Delta V = - f\]
where $V$ is the scalar potential of the force, or the potential energy a particle would have at that point, and $f$ is a source term.
Examples of conservative forces include Newton's Law of Gravity (where matter become the source term) and Coulomb's Law, which gives the force between two charge particles (where charge is the source term).

In electrostatics the electric potential is also known as the voltage, and is denoted by $V.$
From Maxwell's equations it can be shown that that the voltage obeys Poisson's equation with the electric charge density (like a continuous cloud of electrons) being the source term:
\[
 \Delta V = -\frac{\rho}{\epsilon_0},
\]
where $\rho$ is the charge density and $\epsilon_0$ is the permissivity of
free space, which is a constant that we'll leave as $1$.

Usually a non zero $V$ at a point will cause a charged particle to move to a lower potential, changing $\rho$ and the solution to $V$.
However, in this analysis we'll assume that the charges are fixed in place.

Suppose we have 3 nested pipes.
The outer pipe is attached to "ground," which usually we define to be $V=0$, and the inner two have opposite relative charges.
Physically the two inner pipes would function like a capacitor.

The following code will plot the charge distribution of this setup.
\begin{lstlisting}
import matplotlib.colors as mcolors

def source(X,Y):
    """
    Takes arbitrary arrays of coordinates X and Y and returns an array of the same shape
    representing the charge density of nested charged squares
    """
    src = np.zeros(X.shape)
    src[ np.logical_or(
        np.logical_and( np.logical_or(abs(X-1.5) < .1,abs(X+1.5) < .1) ,abs(Y) < 1.6),
        np.logical_and( np.logical_or(abs(Y-1.5) < .1,abs(Y+1.5) < .1) ,abs(X) < 1.6))] = 1
    src[ np.logical_or(
        np.logical_and( np.logical_or(abs(X-0.9) < .1,abs(X+0.9) < .1) ,abs(Y) < 1.0),
        np.logical_and( np.logical_or(abs(Y-0.9) < .1,abs(Y+0.9) < .1) ,abs(X) < 1.0))] = -1
    return src

#Generate a color dictionary for use with LinearSegmentedColormap
#that places red and blue at the min and max values of data
#and white when data is zero

def genDict(data):
    zero = 1/(1 - np.max(data)/np.min(data))
    cdict = {'red':   [(0.0,  1.0, 1.0),
                   (zero,  1.0, 1.0),
                   (1.0,  0.0, 0.0)],
         'green': [(0.0,  0.0, 0.0),
                   (zero,  1.0, 1.0),
                   (1.0,  0.0, 0.0)],
         'blue':  [(0.0,  0.0, 0.0),
                   (zero,  1.0, 1.0),
                   (1.0,  1.0, 1.0)]}
    return cdict

a1 = -2.
b1 = 2.
c1 = -2.
d1 = 2.
n =100
X = np.linspace(a1,b1,n)
Y = np.linspace(c1,d1,n)
X,Y = np.meshgrid(X,Y)

plt.imshow(source(X,Y),cmap =  mcolors.LinearSegmentedColormap('cmap', genDict(source(X,Y))))
plt.colorbar(label="Relative Charge")
plt.show()
\end{lstlisting}

\begin{figure}
\includegraphics[width=\textwidth]{pipesRho.pdf}
\caption{The charge density of the 3 nested pipes.}
\end{figure}

The function \li{genDict} scales the color values to be white when the charge density is zero.
This is mostly to help visualize where there are neutrally charged zones by forcing them to be white.
You may find it useful to also apply it when you solve for the electric  potential.

With this definition of the charge density, we can solve Poisson's equation for the potential field.

\begin{problem}
Solve
% \[\Delta V = -\rho(x,y)\]
\begin{align}
	\begin{split}
	\Delta V = -\rho(x,y), &{}\quad x \in [-2,2]\times [-2,2],\\
	u(x,y) = 0, &{}\quad (x,y) \in \partial ([-2,2]\times [-2,2]).
	\end{split}
	\label{poisson2d:source}
\end{align}
%
for the electric potential $V.$
Use the source function ($-\rho$) defined above. % and grid sizes defined above.
% Use $V=0$ for the boundary conditions on all sides.
% \textit{(Due to the size of $A$, this is best done using sparse matrices)}
Plot the 2D solution using $n=100$.
\end{problem}

\begin{figure}
\includegraphics[width=\textwidth]{pipesV.png}
\caption{The electric potential of the 3 nested pipes.}
\end{figure}


\section{Application of our previous calculation of $ H_\ast(S^n)$ and the ``locality principle''}
\begin{theorem}
Let $n\geq 1$. There is a surjective monoid homomorphism $[S^n,S^n]\to \Z_\times$, where $\Z_\times$ is the multiplicative monoid of $\Z$. $[S^n,S^n]$ is a monoid under composition. (This is basically the degree...)
\end{theorem}
\begin{proof}
Given $f:S^n\to S^n$, take the homology, which is just a homomorphism $\Z\to \Z$, all of which are simply multiplication by an integer. The integer by which you're multiplying to get this homomorphism is the integer associated to $f$.

Construction. If $n=1$, this is just the winding number. Suppose I've constructed this in dimension $n-1$. We have:
	\begin{equation*}
	\xymatrix{ H_{n-1}(S^{n-1})\ar[d]^n & \ar[l] H_n(D^n,S^{n-1})\ar[r]\ar[d] & H_n(S^n)\ar[d]\\
	 H_{n-1}(S^{n-1}) & \ar[l] H_n(D^n,S^{n-1})\ar[r] & H_n(S^n)}
	\end{equation*}
So we're basically suspending $f:S^{n-1}\to S^{n-1}$. More explicitly, if you have $f:S^{n-1}\to S^{n-1}$. We can extend to $\overline{f}:D^n\to D^n$ by sending $tx\mapsto tf(x)$ where $tx$ denotes the ray connecting $x\in S^{n-1}$ to the origin, and we can then quotient out by $S^{n-1}$ to get the map $S^n\to S^n$ as required.
\end{proof}
\subsection{Addendum to the ES axioms}
There's a further axiom, which isn't due to ES, but rather due to Milnor. It's this.
\begin{itemize}
\item Suppose $I$ a set. For each $\alpha\in I$ there's have a space $X_\alpha\in\mathbf{Top}$. I can consider $\coprod_\alpha X_\alpha$. There are inclusion maps $X_\alpha\to\coprod_\alpha X_\alpha$. Then $\bigoplus_\alpha h_n(X_\alpha)\cong h_n\left(\coprod_\alpha X_\alpha\right)$.
\end{itemize}
This is known for ordinary singular homology.
\subsection{Homological algebra}
\begin{enumerate}
\item Suppose $A,B\subseteq C$ are abelian groups. Then $A+C\subseteq C$. You have a sexseq $0\to A\cap B\to A\oplus B\to A+C\to 0$ where the map $c\mapsto (c,-c)$ is how the map $A\cap B\to A\oplus B$ is defined.
\item ``Fundamental isomorphism for abelian groups'' says the following. We have two sexseqs.
	\begin{equation*}
	\xymatrix{0\ar[r] & A\cap B\ar[r]\ar[d] & B\ar[r]\ar[d] & B/A\cap B\ar[r]\ar@{-->}[d]^\cong & 0\\
	0\ar[r] & A\ar[r] & A+B\ar[r] & (A+B)/A\ar[r] & 0}
	\end{equation*}
	I'm not going to write out the diagram chase that we did.
\item ``Snake lemma''. Suppose I have\footnote{``It's my Turn'', Jill Clayburgh}:
	\begin{equation*}
	\xymatrix{ & \ker f^\prime\ar[r]\ar[d] & \ker f\ar[r]\ar[d] & \ker f^{\prime\prime}\ar[d]\\
	0\ar[r] & A^\prime\ar[r]\ar[d]^{f^\prime} & A\ar[r]\ar[d]^f & A^{\prime\prime}\ar[r]\ar[d]^{f^{\prime\prime}} & 0\\
	0\ar[r] & B^\prime\ar[r]\ar[d] & B\ar[r]\ar[d] & B^{\prime\prime}\ar[r]\ar[d] & 0\\
	 & \coker f^\prime\ar[r] & \coker f\ar[r] & \coker f^{\prime\prime}}
	\end{equation*}
	Claim is that there's a map $\ker f^{\prime\prime}\to\coker f$ so that $0\to \ker f^\prime\to \ker f\to \ker f^{\prime\prime}\to\coker f^\prime\to\coker f\to \coker f^{\prime\prime}\to 0$. This is basically the lexseq in homology associated to the sexseqs of the following three chain complexes: $0\to A^\prime\to B^\prime\to 0$, $0\to A\to B\to 0$, and $0\to A^{\prime\prime}\to B^{\prime\prime}\to 0$. Work this out yourself.
\end{enumerate}
\subsection{Locality}
\begin{definition}
The (not necessarily open) cover of a topological space. Won't write this.
\end{definition}
\begin{definition}
Let ${\mathscr{A}}$ be a cover of $X$. An $n$-simplex $\sigma$ is ${\mathscr{A}}$-small if there is $A\in \mathscr{A}$ such that the image of $\sigma$ is entirely in $A$.
\end{definition}
Notice that if $\sigma:\Delta^n\to X$ is ${\mathscr{A}}$-small, then so is $d^i\sigma$. Let's denote by $\Sin^{\mathscr{A}}_n(X)$ the set of ${\mathscr{A}}$-small $n$-simplices. This means that we get a map $\Sin^{\mathscr{A}}_n(X)\to \Sin^{\mathscr{A}}_{n-1}(X)$. Let $S^{\mathscr{A}}_n(X)=\Z[\Sin^{\mathscr{A}}_n(X)]$. Then there's a subchain complex $S^{\mathscr{A}}_\ast(X)$.
\begin{theorem}
The inclusion $S^\mathscr{A}_\ast(X)\subseteq S_\ast(X)$ is a chain homotopy equivalence.
\end{theorem}
\begin{corollary}
If $ H^\mathscr{A}_\ast(X):= H(S^\mathscr{A}_\ast(X))$, then $ H^\mathscr{A}_\ast(X)\cong H_\ast(X)$.
\end{corollary}
We'll do this on Monday.
\begin{example}
If $\mathscr{A}=\{A,B\}$, then $\overline{X-B}=X-\mathrm{Int}(B)\subseteq\mathrm{Int}(A)$. Let $X-B=U$. Then $U\subseteq \overline{U}\subseteq \mathrm{Int}(A)\subseteq A\subseteq X$. This is an excision! So $U\subseteq A\subseteq X$ is an excision. But now, $(X-U,A-U)\to (X,A)$ is an excision, but $(X-U,A-U)=(B,A\cap B)$, so we have $(B,A\cap B)\to (X,A)$ is an excision. Now, also, $S^\mathscr{A}_\ast(X)=S_\ast(A)+S_\ast(B)$. Says he got off track, let me just write things out and explain in a moment.
\begin{equation*}
	\xymatrix{0\ar[r] & S_n(A)\cap S_n(B)=S_n(A\cap B)\ar[r]\ar[d] & S_n(B)\ar[r]\ar[d] & S_n(B,A\cap B)\ar[r]\ar@{-->}[d]^\cong & 0\\
	0\ar[r] & S_n(A)\ar[r] & S_n(A)+S_n(B)=S^\mathscr{A}_n(X)\ar[r] & S^\mathscr{A}_n(X)/S_n(A)\ar[r] & 0}
	\end{equation*}
But we can consider $S_\ast(X)/S_\ast(A)=S_\ast(X,A)$. By the lexseq + 5 lemma, this thing is isomorphic to $S^\mathscr{A}_n(X)/S_n(A)$, so $S_\ast(B,A\cap B)\cong S_\ast(X,A)$ in homology. This is precisely the excision theorem. QED.
\end{example}