% This LaTeX was auto-generated from an M-file by MATLAB.
% To make changes, update the M-file and republish this document.

%%% \documentclass{article}
%%% \usepackage{graphicx}
%%% \usepackage{color}

%%% \sloppy
%%% \definecolor{lightgray}{gray}{0.5}
\setlength{\parindent}{0pt}

%%% \begin{document}

\subsection*{Example of the QWTB use}

\begin{par}
Data are simulated, QWTB is used with different algorithms.
\end{par} \vspace{1em}

\subsubsection*{Contents}

\begin{itemize}
\setlength{\itemsep}{-1ex}
   \item Generate ideal data
   \item Apply three algorithms
   \item Compare results for ideal signal
   \item Noisy signal
   \item Compare results for noisy signal
   \item Non-coherent signal
   \item Compare results for non-coherent signal
   \item Harmonically distorted signal.
   \item Compare results for harmonically distorted signal.
   \item Harmonically distorted, noisy, non-coherent signal.
   \item Compare results for harmonically distorted, noisy, non-coherent signal.
\end{itemize}

\subsubsection*{Generate ideal data}

\begin{par}
Sample data are generated, representing 1 second of sine waveform of nominal frequency \lstinline{fnom} 1000 Hz, nominal amplitude \lstinline{Anom} 1 V and nominal phase \lstinline{phnom} 1 rad. Data are sampled at sampling frequency \lstinline{fsnom} 10 kHz, perfectly synchronized, no noise.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
Anom = 1; fnom = 1000; phnom = 1; fsnom = 10e4;
timestamps = [0:1/fsnom:0.1-1/fsnom];
ideal_wave = Anom*sin(2*pi*fnom*timestamps + phnom);
\end{lstlisting}
\begin{par}
To use QWTB, data are put into two quantities: \lstinline{t} and \lstinline{y}. Both quantities are put into data in structure \lstinline{DI}.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
DI = [];
DI.t.v = timestamps;
DI.y.v = ideal_wave;
\end{lstlisting}

\subsubsection*{Apply three algorithms}

\begin{par}
QWTB will be used to apply three algorithms to determine frequency and amplitude: \lstinline{SP-FFT}, \lstinline{PSFE} and \lstinline{FPNLSF}. Results are in data out structure \lstinline{DOxxx}. Algorithm \lstinline{FPNLSF} requires an estimate, select it to 0.1\% different from nominal frequency. \lstinline{SP-FFT} requires sampling frequency.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
DI.fest.v = fnom.*1.001;
DI.fs.v = fsnom;
DOspfft = qwtb('SP-FFT', DI);
DOpsfe = qwtb('PSFE', DI);
DOfpnlsf = qwtb('FPNLSF', DI);
\end{lstlisting}

        \begin{lstlisting}[style=output]
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: PSFE wrapper: sampling time was calculated from sampling frequency
QWTB: no uncertainty calculation
Fitting started

Local minimum found.

Optimization completed because the size of the gradient is less than
the default value of the function tolerance.

Fitting finished
\end{lstlisting} \color{black}

\subsubsection*{Compare results for ideal signal}

\begin{par}
Calculate relative errors in ppm for all algorithm to know which one is best. \lstinline{SP-FFT} returns whole spectrum, so only the largest amplitude peak is interesting. One can see for the ideal case all errors are very small.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
disp('SP-FFT errors (ppm):')
[tmp, ind] = max(DOspfft.A.v);
ferr  = (DOspfft.f.v(ind) - fnom)/fnom .* 1e6
Aerr  = (DOspfft.A.v(ind) - Anom)/Anom .* 1e6
pherr = (DOspfft.ph.v(ind) - phnom)/phnom .* 1e6

disp('PSFE errors (ppm):')
ferr  = (DOpsfe.f.v - fnom)/fnom .* 1e6
Aerr  = (DOpsfe.A.v - Anom)/Anom .* 1e6
pherr = (DOpsfe.ph.v - phnom)/phnom .* 1e6

disp('FPNLSF errors (ppm):')
ferr  = (DOfpnlsf.f.v - fnom)/fnom .* 1e6
Aerr  = (DOfpnlsf.A.v - Anom)/Anom .* 1e6
pherr = (DOfpnlsf.ph.v - phnom)/phnom .* 1e6
\end{lstlisting}

        \begin{lstlisting}[style=output]
SP-FFT errors (ppm):

ferr =

     0

Aerr =

     0

pherr =

  -4.2920e+05

PSFE errors (ppm):

ferr =

  -2.2737e-10

Aerr =

   4.8850e-09

pherr =

   2.3093e-08

FPNLSF errors (ppm):

ferr =

  -3.4106e-10

Aerr =

  -4.0512e-07

pherr =

   1.8208e-08

\end{lstlisting} \color{black}

\subsubsection*{Noisy signal}

\begin{par}
To simulate real measurement, noise is added with normal distribution and standard deviation \lstinline{sigma} of 100 microvolt. Algorithms are again applied.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
sigma = 100e-6;
DI.y.v = ideal_wave + 100e-6.*randn(size(ideal_wave));
DOspfft = qwtb('SP-FFT', DI);
DOpsfe = qwtb('PSFE', DI);
DOfpnlsf = qwtb('FPNLSF', DI);
\end{lstlisting}

        \begin{lstlisting}[style=output]
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: PSFE wrapper: sampling time was calculated from sampling frequency
QWTB: no uncertainty calculation
Fitting started

Local minimum found.

Optimization completed because the size of the gradient is less than
the default value of the function tolerance.

Fitting finished
\end{lstlisting} \color{black}

\subsubsection*{Compare results for noisy signal}

\begin{par}
Again relative errors are compared. One can see amplitude and phase errors increased to several ppm, however frequency is still determined quite good by all three algorithms. FFT is not affected by noise at all.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
disp('SP-FFT errors (ppm):')
[tmp, ind] = max(DOspfft.A.v);
ferr  = (DOspfft.f.v(ind) - fnom)/fnom .* 1e6
Aerr  = (DOspfft.A.v(ind) - Anom)/Anom .* 1e6
pherr = (DOspfft.ph.v(ind) - phnom)/phnom .* 1e6

disp('PSFE errors:')
ferr  = (DOpsfe.f.v - fnom)/fnom .* 1e6
Aerr  = (DOpsfe.A.v - Anom)/Anom .* 1e6
pherr = (DOpsfe.ph.v - phnom)/phnom .* 1e6

disp('FPNLSF errors:')
ferr  = (DOfpnlsf.f.v - fnom)/fnom .* 1e6
Aerr  = (DOfpnlsf.A.v - Anom)/Anom .* 1e6
pherr = (DOfpnlsf.ph.v - phnom)/phnom .* 1e6
\end{lstlisting}

        \begin{lstlisting}[style=output]
SP-FFT errors (ppm):

ferr =

     0

Aerr =

   -0.6603

pherr =

  -4.2920e+05

PSFE errors:

ferr =

    0.0010

Aerr =

   -0.6318

pherr =

   -1.0933

FPNLSF errors:

ferr =

   -0.0011

Aerr =

   -0.6601

pherr =

   -0.3809

\end{lstlisting} \color{black}

\subsubsection*{Non-coherent signal}

\begin{par}
In real measurement coherent measurement does not exist. So in next test the frequency of the signal differs by 20 ppm:
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
fnc = fnom*(1 + 20e-6);
noncoh_wave = Anom*sin(2*pi*fnc*timestamps + phnom);
DI.y.v = noncoh_wave;
DOspfft = qwtb('SP-FFT', DI);
DOpsfe = qwtb('PSFE', DI);
DOfpnlsf = qwtb('FPNLSF', DI);
\end{lstlisting}

        \begin{lstlisting}[style=output]
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: PSFE wrapper: sampling time was calculated from sampling frequency
QWTB: no uncertainty calculation
Fitting started

Local minimum found.

Optimization completed because the size of the gradient is less than
the default value of the function tolerance.

Fitting finished
\end{lstlisting} \color{black}

\subsubsection*{Compare results for non-coherent signal}

\begin{par}
Comparison of relative errors. Results of \lstinline{PSFE} or \lstinline{FPNLSF} are correct, however FFT is affected by non-coherent signal considerably.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
disp('SP-FFT errors (ppm):')
[tmp, ind] = max(DOspfft.A.v);
ferr  = (DOspfft.f.v(ind) - fnc)/fnc .* 1e6
Aerr  = (DOspfft.A.v(ind) - Anom)/Anom .* 1e6
pherr = (DOspfft.ph.v(ind) - phnom)/phnom .* 1e6

disp('PSFE errors:')
ferr  = (DOpsfe.f.v - fnc)/fnc .* 1e6
Aerr  = (DOpsfe.A.v - Anom)/Anom .* 1e6
pherr = (DOpsfe.ph.v - phnom)/phnom .* 1e6

disp('FPNLSF errors:')
ferr  = (DOfpnlsf.f.v - fnc)/fnc .* 1e6
Aerr  = (DOfpnlsf.A.v - Anom)/Anom .* 1e6
pherr = (DOfpnlsf.ph.v - phnom)/phnom .* 1e6
\end{lstlisting}

        \begin{lstlisting}[style=output]
SP-FFT errors (ppm):

ferr =

  -19.9996

Aerr =

   -2.8780

pherr =

  -4.3550e+05

PSFE errors:

ferr =

  -1.1368e-10

Aerr =

   3.8924e-07

pherr =

   3.3073e-04

FPNLSF errors:

ferr =

  -1.1368e-10

Aerr =

  -3.2940e-07

pherr =

   2.6867e-08

\end{lstlisting} \color{black}

\subsubsection*{Harmonically distorted signal.}

\begin{par}
In other cases a harmonic distortion can appear. Suppose a signal with second order harmonic of 10\% amplitude as the main signal.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
hadist_wave = Anom*sin(2*pi*fnom*timestamps + phnom) + 0.1*Anom*sin(2*pi*fnom*2*timestamps + 2);
DI.y.v = hadist_wave;
DOspfft = qwtb('SP-FFT', DI);
DOpsfe = qwtb('PSFE', DI);
DOfpnlsf = qwtb('FPNLSF', DI);
\end{lstlisting}

        \begin{lstlisting}[style=output]
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: PSFE wrapper: sampling time was calculated from sampling frequency
QWTB: no uncertainty calculation
Fitting started

Local minimum found.

Optimization completed because the size of the gradient is less than
the default value of the function tolerance.

Fitting finished
\end{lstlisting} \color{black}

\subsubsection*{Compare results for harmonically distorted signal.}

\begin{par}
Comparison of relative errors. \lstinline{SP-FFT} or \lstinline{PSFE} are not affected by harmonic distortion, however \lstinline{FPNLSF} is thus is not suitable for such signal.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
disp('SP-FFT errors (ppm):')
[tmp, ind] = max(DOspfft.A.v);
ferr  = (DOspfft.f.v(ind) - fnom)/fnom .* 1e6
Aerr  = (DOspfft.A.v(ind) - Anom)/Anom .* 1e6
pherr = (DOspfft.ph.v(ind) - phnom)/phnom .* 1e6

disp('PSFE errors:')
ferr  = (DOpsfe.f.v - fnom)/fnom .* 1e6
Aerr  = (DOpsfe.A.v - Anom)/Anom .* 1e6
pherr = (DOpsfe.ph.v - phnom)/phnom .* 1e6

disp('FPNLSF errors:')
ferr  = (DOfpnlsf.f.v - fnom)/fnom .* 1e6
Aerr  = (DOfpnlsf.A.v - Anom)/Anom .* 1e6
pherr = (DOfpnlsf.ph.v - phnom)/phnom .* 1e6
\end{lstlisting}

        \begin{lstlisting}[style=output]
SP-FFT errors (ppm):

ferr =

     0

Aerr =

     0

pherr =

  -4.2920e+05

PSFE errors:

ferr =

  -2.2737e-10

Aerr =

   6.7212e-04

pherr =

    0.5311

FPNLSF errors:

ferr =

   -0.7356

Aerr =

    0.1407

pherr =

  231.4553

\end{lstlisting} \color{black}

\subsubsection*{Harmonically distorted, noisy, non-coherent signal.}

\begin{par}
In final test all distortions are put in a waveform and results are compared.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
err_wave = Anom*sin(2*pi*fnc*timestamps + phnom) + 0.1*Anom*sin(2*pi*fnc*2*timestamps + 2) + 100e-6.*randn(size(ideal_wave));
DI.y.v = err_wave;
DOspfft = qwtb('SP-FFT', DI);
DOpsfe = qwtb('PSFE', DI);
DOfpnlsf = qwtb('FPNLSF', DI);
\end{lstlisting}

        \begin{lstlisting}[style=output]
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: PSFE wrapper: sampling time was calculated from sampling frequency
QWTB: no uncertainty calculation
Fitting started

Local minimum found.

Optimization completed because the size of the gradient is less than
the default value of the function tolerance.

Fitting finished
\end{lstlisting} \color{black}

\subsubsection*{Compare results for harmonically distorted, noisy, non-coherent signal.}

\begin{lstlisting}[style=mcode]
disp('SP-FFT errors (ppm):')
[tmp, ind] = max(DOspfft.A.v);
ferr  = (DOspfft.f.v(ind) - fnc)/fnc .* 1e6
Aerr  = (DOspfft.A.v(ind) - Anom)/Anom .* 1e6
pherr = (DOspfft.ph.v(ind) - phnom)/phnom .* 1e6

disp('PSFE errors:')
ferr  = (DOpsfe.f.v - fnc)/fnc .* 1e6
Aerr  = (DOpsfe.A.v - Anom)/Anom .* 1e6
pherr = (DOpsfe.ph.v - phnom)/phnom .* 1e6

disp('FPNLSF errors:')
ferr  = (DOfpnlsf.f.v - fnc)/fnc .* 1e6
Aerr  = (DOfpnlsf.A.v - Anom)/Anom .* 1e6
pherr = (DOfpnlsf.ph.v - phnom)/phnom .* 1e6
\end{lstlisting}

        \begin{lstlisting}[style=output]
SP-FFT errors (ppm):

ferr =

  -19.9996

Aerr =

    1.1501

pherr =

  -4.3550e+05

PSFE errors:

ferr =

   -0.0072

Aerr =

    4.1189

pherr =

    4.6464

FPNLSF errors:

ferr =

   -0.7241

Aerr =

    3.6943

pherr =

  229.3720

\end{lstlisting} \color{black}

%%% \end{document}


\chapter{Dimension Reduction}\label{chap:dimred}

Dimension reduction is a statistical process, which concentrates the
amount of information in multivariate data into a fewer number of
variables (or dimensions). An interesting review of the domain has been done by Fodor~\cite{Fodor2002dimensionred}.

Though there are plenty of non-linear methods in the litterature, OTB
provides only linear dimension reduction techniques applied to images for now.

Usually, linear dimension-reduction algorithms try to find a set of
linear combinations of the input image bands that maximise a given
criterion, often chosen so that image information concentrates on the
first components. Algorithms differs by the criterion to optimise and
also by their handling of the signal or image noise.

In remote-sensing images processing, dimension reduction algorithms
are of great interest for denoising, or as a preliminary processing
for classification of feature images or unmixing of hyperspectral
images. In addition to the denoising effect, the advantage of
dimension reduction in the two latter is that it lowers the size of
the data to be analysed, and as such, speeds up the processing time
without too much loss of accuracy.

\section{Principal Component Analysis}

\input{PCAExample}

\section{Noise-Adjusted Principal Components Analysis}

\input{NAPCAExample}

\section{Maximum Noise Fraction}

\input{MNFExample}

\section{Fast Independent Component Analysis}

\input{ICAExample}

\section{Maximum Autocorrelation Factor}

\input{MaximumAutocorrelationFactor}


\subsection{Test 3 Answers}

\begin{enumerate}[label=\arabic*.]
	\item
		Let $f(t)$ be a function defined for $t \geq 0$.
		\begin{equation*}
			\Laplace{f} = \int_{0}^{\infty}{e^{-st}f(t) \mathrm{d}t}
		\end{equation*}
	\item
		\begin{enumerate}[label = (\alph*)]
			\item
				\begin{align*}
					\Laplace{e^{x}} &= \int_{0}^{\infty}{e^{-st}e^{t} \mathrm{d}t} \\
					&= \int_{0}^{\infty}{e^{t(1-s)} \mathrm{d}t} \\
					&= \frac{1}{1-s}e^{t(1-s)} \bigg\rvert_{0}^{\infty} \\
					&= \frac{1}{1-s}\left(\lim_{a \to \infty}{e^{a(1-s)}} - e^{0(1-s)} \right) \\
					&= \frac{1}{1-s}\left(0 - 1\right) \text{, }  s > 1\\
					&= \frac{1}{s-1} \text{, } s > 1
				\end{align*}
			\item
				\begin{equation*}
					\Laplace{\sin{t} + 2\cos{3t}} = \Laplace{\sin{t}} + 2\Laplace{\cos{3t}}
				\end{equation*}
				by the linearity of the Laplace transform.
				\begin{equation*}
					 = \Laplace{\sin{t}} + \frac{2}{3}\mathcal{L}\left\{\cos{t}\right\}{(\frac{s}{3})}
				\end{equation*}
				as specified in the table of Laplace transforms for $\sin$ and $\cos$.
				\begin{align*}
					&= \frac{1}{s^2 + 1^2} + \frac{2}{3}\left(\frac{s/3}{(s/3)^2 + 1^2}\right) \\
					&= \frac{1}{s^2 + 1^2} + \frac{2}{3}\frac{3s}{s^2 + 3^2} \\
					&= \frac{1}{s^2+1^2} + \frac{2s}{s^2 + 3^2} \\
					&= \frac{2s^3 + s^2 + 2s + 9}{\left(s^2 + 1\right)\left(s^2+9\right)}
				\end{align*}
			\item
				\begin{equation*}
					\Laplace{e^{3t}\left(t^2+3t+2\right)} = \Laplace{e^{3t}t^2} + 3\Laplace{e^{3t}x} + 2\Laplace{e^{3t}}
				\end{equation*}
				by the linearity of the Laplace transform.
				\begin{equation*}
					 = \frac{2!}{(s-3)^{2+1}} + 3\frac{1!}{(s-3)^{1+1}} + 2\frac{0!}{(s-3)^{0+1}}
				\end{equation*}
				as specified in the table of Laplace transforms for an exponential and power of $t$.
				\begin{align*}
					 &= \frac{2}{(s-3)^3} + \frac{3}{(s-3)^2} + \frac{2}{(s-3)} \\
					 &= \frac{2(s-3)^2 + 3(s-3) + 2}{(s-3)^3} \\
					 &= \frac{2s^2 - 9s + 11}{(s-3)^3}
				\end{align*}
			\item
				\begin{equation*}
					\Laplace{\dd{}{t}{(e^{3t+1} + e^{1-t})}} = s\Laplace{e^{3t+1} + e^{1-t}} - \left(e^{3\cdot 0 + 1} + e^{1-0}\right)
				\end{equation*}
				by the Laplace transform of a derivative.
				\begin{equation*}
					= s\left(e\Laplace{e^{3t}} + e\Laplace{e^{-t}}\right) - 2e
				\end{equation*}
				by the linearity of the Laplace transform.
				\begin{equation*}
					 = s\left(e\frac{1}{s-3} + e\frac{1}{s+1}\right) - 2e
				\end{equation*}
				as specified in the table of Laplace trasforms for the Laplace transform of an exponential.
				\begin{align*}
					&= \frac{se}{s-3} + \frac{se}{s+1} - 2e \\
					&= \frac{se(s+1) + se(s-3) -2e(s+1)(s-3)}{(s+1)(s-3)} \\
					&= \frac{2e(s+3)}{(s+1)(s-3)}
				\end{align*}
		\end{enumerate}
	\item
		\begin{enumerate}[label=(\alph*)]
			\item
				\begin{equation*}
					\inverseLaplace{\frac{1}{1+s}} = e{-t}
				\end{equation*}
			\item
				\begin{align*}
					\inverseLaplace{\frac{s^2+2s+1}{s^3-4s^2+5s-2}} &= \inverseLaplace{\frac{-8}{s-1}}-\inverseLaplace{\frac{4}{(s-1)^2}}+\inverseLaplace{\frac{9}{s-2}} \\
					&= -8e^{t} - 4e^{t}t + 9e^{2t}
				\end{align*}
			\item
				\begin{align*}
					\inverseLaplace{\frac{s-4}{s^2-8s+32}} &= \inverseLaplace{\frac{s-4}{(s-4)^2 + 4^2}} \\
					&= e^{4t}\cos{(4t)}
				\end{align*}
			\item
				\begin{align*}
					\inverseLaplace{\frac{768}{(2s+3)^5}} &= \inverseLaplace{\frac{4!}{(s+\frac{3}{2})^{4+1}}} \\
					&= e^{-\frac{3}{2}t}t^4
				\end{align*}
		\end{enumerate}
	\item
		We'll use method of undetermined coefficients.
		Extracting and solving the axillary equation,
		\begin{equation*}
			2r^2 - 3r + 1 = 0 \implies r = \frac{1}{2} \text{, } 1.
		\end{equation*}
		So, our homogeneous solution is
		\begin{equation*}
			y_h = C_1e^{\frac{x}{2}} + C_2e^{x}.
		\end{equation*}
		We'll guess that the particular solution has the form
		\begin{align*}
			y_p &= A\sin{x} + B\cos{x} \\
			2y_p'' - 3y_p' + y_p &= 10\sin{x} \implies A = -1 \text{, } B = 3.
		\end{align*}
		So,
		\begin{equation*}
			y_p = -\sin{x} + 3\cos{x},
		\end{equation*}
		and our general solution is
		\begin{equation*}
			y = C_1e^{\frac{x}{2}} + C_2e^{x} -\sin{x} + 3\cos{x}.
		\end{equation*}
		Now we'll use Laplace transform. Taking the Laplace transform of both sides,
		\begin{align*}
			\Laplace{2y'' - 3y' + y} &= \Laplace{10\sin{x}} \\
			\Laplace{y} \cdot (2s^2 - 3s + 1) &= \frac{10}{s^2+1} + 2y'(0) + 3y(0) -2sy(0) \\
			\Laplace{y} \cdot (2s^2 - 3s + 1) &= \frac{10 + 3y(0)s^2 + 3y(0) - 2s^3y(0) - 2sy(0) + 2s^2y'(0) + 2y'(0)}{s^2+1} \\
			\Laplace{y} &= \frac{-2y(0)s^3 + (3y(0)+2y'(0))s^2 -2sy(0) + 3y(0) + 2y'(0) + 10}{\left(s^2+1\right)\left(2s^2-3s+1\right)} \\
			&= \frac{-2y(0)s^3 + (3y(0)+2y'(0))s^2 -2sy(0) + 3y(0) + 2y'(0) + 10}{\left(s^2+1\right)\left(2s-1\right)\left(s-1\right)} \\
			&= \frac{-4(y(0)+y'(0)+4)}{2s-1} + \frac{y(0)+2y'(0)+5}{s-1} + \frac{3s-1}{s^2+1}.
		\end{align*}
		Let $C_1 = -2(y(0)+y'(0)+4)$ and $C_2 = y(0) + 2y'(0) + 5$.
		\begin{equation*}
			\Laplace{y} = \frac{2C_1}{2s-1} + \frac{C_2}{s-1} - \frac{1}{s^2+1} + 3\frac{s}{s^2+1}.
		\end{equation*}
		Taking the inverse Laplace transform of both sides,
		\begin{equation*}
			y = C_1e^{\frac{x}{2}} + C_2e^x -\sin{x} + 3\cos{x}.
		\end{equation*}
		We can see that we have the same solution for $y$ in both methods.
	\item
		Taking the Laplace transform of both sides and solving for the Laplace transform of $y$,
		\begin{align*}
			\Laplace{2y'' + 4y' - 6y} &= \Laplace{te^{-3t}} \\
			\Laplace{y} \cdot (2s^2 + 4s - 6) - 2sy(0) - 2y'(0) - 4y(0) &= \frac{1}{(s+3)^2} \\
			\Laplace{y} \cdot (2s^2 + 4s - 6) - 2s - 4 &= \frac{1}{(s+3)^2} \\
			\Laplace{y} \cdot (2s^2 + 4s - 6) &= \frac{1}{(s+3)^2} + 2s + 4 \\
			&= \frac{2s^3 + 16s^2 + 42s + 37}{(s+3)^2} \\
			\Laplace{y} &= \frac{2s^3 + 16s^2 + 42s + 37}{2(s+3)^3(s-1)}.
		\end{align*}
		Converting the right side to partial fractions,
		\begin{equation*}
			\Laplace{y} = \frac{31}{128(s+3)} - \frac{1}{32(s+3)^2} - \frac{1}{8(s+3)^3} + \frac{97}{128(s-1)}.
		\end{equation*}
		Taking the inverse Laplace transform of both sides to solve for $y$,
		\begin{equation*}
			y = \frac{31}{128}e^{-3t} - \frac{1}{32}te^{-3t} - \frac{1}{16}t^2e^{-3t} + \frac{97}{128}e^{t}.
		\end{equation*}
\end{enumerate}


\documentclass{report}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{multicol}
\usepackage{graphicx}
\setlength{\oddsidemargin}{0in}
\setlength{\evensidemargin}{0in}
\setlength{\topmargin}{-.5in}
\setlength{\textwidth}{6.5in}
\setlength{\textheight}{9.5in}

\begin{document}
\section{Orbits}
$r^2 \dot{\theta} =h$.
$r = {\frac {h^2}{k(1+e \cdot cos( \theta ))}}$.
$r_{min}= {\frac {h^2}{k^3}} {\frac {1} {1+e}}$,
$r_{max}= {\frac {h^2}{k^3}} {\frac {1} {1-e}}$.
$\dot{\theta} = {\frac {k^2}{h^3}} (1+ e \cdot cos(\theta ))^2$.
$\int {\frac {h^3}{k^2}} {\frac {1} {(1+e \cdot cos( \theta ))^2}}= t$.
$S$ is the satellite and $O$ is the observer.
\\
\\
1. Compute $\theta(t), r(t)$ from integral above in orbital plane.  At $t_0$,
$S$ is at ascending node position.
\\
2. Rotate orbital parameters by the inclination, $i$, in the line formed by the
ascending node and descending node.
\\
3. Rotate around $z$ by the angle of the ascending node to get to ecliptic coordinates.
$S$ is at ascending node when $t=0$.
\\
4. Get $\delta$, $RA$ of $S$.
\\
5. Get $\lambda$, $L$ of $O$.
\\
6. Find $\delta '$ and $RA '$, from $O$'s point of view.
\\
\\
$S$ is at at $( \delta , RA )$, $O$ is at $( \lambda , L)$. $\overline{x}$ means
$90-x$.
$cos ( \overline{ \delta ' }) =
cos( RA+L ) sin(\overline{\delta}) sin(\overline{\lambda}) +
cos(\overline{\delta}) cos(\overline{\lambda})$.
${\frac {\overline{\delta}} {sin(RA')}} =
{\frac {\overline{\delta'}} {sin(RA+L)}} $.  $RA$ is measured from the arc $ON$ to the arc
$OS$.
\end{document}


\section{Result}
\subsection{Distance Between Two Lasar Device}

\begin{table}[H]
  \centering
  \begin{tabular}{|l|c |c |c |}
    \hline
    \multicolumn{4}{|c|}{distance $S$ [mm] $\pm$ 1 [mm]} \\
    \hline
    S &  Ruler Start Point & Ruler End Point & Calculated Length \\
    \hline
    $S_1$ & 30.0 & 177.0 & 147.0 \\ \hline
    $S_2$ & 60.0 & 207.0 & 147.0 \\ \hline
    $S_3$ & 40.0 & 186.5 & 146.5 \\ \hline
  \end{tabular}
  \caption{Distance measurement data.}
\end{table}

Then we can find
$$  \bar{S} = \frac{1}{3} \sum_{k=1}^{3} S_k = 146.8 mm \pm 1 mm   $$
$$  u_{S,r} = 0.68 \%  $$

\subsection{Time Measurement}

\begin{table}[H]
  \centering
  \begin{tabular}{|l|c|}
    \hline
    \multicolumn{2}{|c|}{time $t$ [s] $\pm$ 0.01 [s] } \\
    \hline
    $t_1$ & 6.75 \\ \hline
    $t_2$ & 6.82 \\ \hline
    $t_3$ & 6.91 \\ \hline
    $t_4$ & 6.88 \\ \hline
    $t_5$ & 6.91 \\ \hline
    $t_6$ & 6.84 \\ \hline
  \end{tabular}
  \caption{Time measurement data.}
\end{table}

Then we can find
$$  \bar{t} = \frac{1}{6} \sum_{k=1}^{6} t_k =6.85 s \pm 0.01 s   $$
$$  u_{t,r} = 0.15 \%  $$

\subsection{The Diameters of The Balls}

The initial reading of the meter is 0.38 mm.
Thus, the raw data of measurement should firstly minus 0.38 mm, and is presented
as following,

\begin{table}[H]
  \centering
  \begin{tabular}{|p{2cm}|p{3cm}||p{2cm} |p{3cm} |}
    \hline
    \multicolumn{4}{|c|}{diameter $d$ [mm] $\pm$ 0.005 [mm]  } \\
    \hline
    $d_1$ & 1.995 & $d_6$ & 1.995 \\ \hline
    $d_2$ & 1.995 & $d_7$ & 2.000 \\ \hline
    $d_3$ & 2.000 & $d_8$ & 1.800 \\ \hline
    $d_4$ & 1.995 & $d_9$ & 1.995 \\ \hline
    $d_5$ & 2.000 & $d_{10}$ & 1.995 \\ \hline
  \end{tabular}
  \caption{Ball diameter measurement data.}
\end{table}

Then we can find
$$  \bar{d} = \frac{1}{10} \sum_{k=1}^{10} d_k = 1.977  mm \pm  0.005 mm   $$
$$  u_{t,r} =  0.25 \%  $$

\subsection{The Inner Diameter of The Flask}

\begin{table}[H]
  \centering
  \begin{tabular}{|p{1cm}|c|}
    \hline
    \multicolumn{2}{|c|}{diameter $D$ [mm] $\pm$ 0.02 [mm]  } \\
    \hline
    $D_1$ & 61.40 \\ \hline
    $D_2$ & 61.46 \\ \hline
    $D_3$ & 61.20 \\ \hline
    $D_4$ & 61.36 \\ \hline
    $D_5$ & 61.20 \\ \hline
    $D_6$ & 61.50 \\ \hline
  \end{tabular}
  \caption{Flask diameter measurement data.}
\end{table}

Then we can find
$$  \bar{D} = \frac{1}{6} \sum_{k=1}^{6} D_k =  61.3533 mm \pm  0.02 mm   $$
$$  u_{D,r} =   0.03\%  $$

\subsection{Other Physical Quantities}

\begin{table}[H]
  \centering
  \begin{tabular}{|c|c|}
    \hline
    density of the castor oil $ \rho_1 [g/cm^3] \pm 0.001 [g/cm^3] $ & 0.955  \\ \hline
    mass of 40 metal balls $ m  [g] \pm 0.001 [g] $ & 1.357 \\ \hline
    temperature in the lab $ T  [\circ C] \pm 2 [\circ C] $ & 25 \\ \hline
    acceleration due to gravity in the lab $ g [m/s^2] $ & 9.794 \\ \hline
  \end{tabular}
  \caption{Other Physical Quantities Measurement}
\end{table}

\subsection{Calculation of Density of One Ball}

The mass of one  metal ball can be calculated as
$$  m_0 = \frac{m}{40} = \frac{1.357 \times 10^{-3} }{40} = 3.3925 \times 10^-5 kg
\pm (2.500 \times 10^-8) kg $$
We can furtherly get the density,
$$ \rho_2 = \frac{m_0}{\frac{1}{6} \pi d^3} = \frac{3.3925 \times 10^-5
}{\frac{1}{6} \times 3.151593 \times (1.977 \times 10^-3)^3 } = 8.385 \times
10^3 kg/m^3 \pm (5.756 \times 10  ) kg/m^3  $$
$$  u_{\rho_2,r} =   0.6875\%  $$

\subsection{Calculation For The Viscosity Coefficient}
From the last equation in introduction part
\begin{multline*}
\mu = \frac{2}{9} g R^2 \frac{( \rho_2 - \rho_1 ) t  }{s} (1 + 2.4
\frac{R}{R_c})  =  \frac{2 \times (1.977 \times 10^-3 \times \frac{1}{2})^2
  (8.385\times 10^3 - 0.595 \times 10^3 ) \times 9.974 \times 6.85 }{9 \times
  146.8\times 10^-3 \times (1 + 2.4 \times \frac{1.977}{61.3533})} \\
 = 0.7307 Pa \times s \pm (6.8329 \times 10^-3 ) Pa \times s
 \end{multline*}
$$  u_{\mu,r} =  0.93512 \%  $$


\section{Bisection and Newton-Raphson method for finding roots}

\subsection{Bisection method}

We will use the bisection method to find when $y_p(\phi)=0$ for a given tolerance $\epsilon$.
The requirement for the bisection method is that the function in question is continuous on the closed interval $[a,b]$\footnote{\href{https://mathworld.wolfram.com/Bisection.html}{Bisection method - Wolfram MathWorld}}, so the roots of $y_p(\phi)$ can be found. The roots of $w\,'(\phi)$ will be found using the Newton's method later.

\subsection{Newton-Raphson method}

In Eq. 40 from the presentation, $w_p\,'(\phi)$ was not expressed in terms of $\beta$,
so the following expression will be used:

\begin{equation}
    w_p\,'(\phi)=\frac{2(\sin^2(2\phi+\pi/2)-3)\sin(2\phi+\pi/2)}{(3\pi+8)\beta^2\sin^3(2\phi+\pi/2)}
    -\frac{\cos(2\phi+\pi/2)\left(1-\frac{4}{(3\pi+8)\beta^2}(3\phi+2)\right)}{r_0\sin^3(2\phi+\pi/2)}
\end{equation}

This method cannot be used to solve for the root of $y_p(\phi)$ because although the function exists in the domain $\phi\in[0,\pi/4)$,
$\lim_{\phi\to 0^+}y_p\,'(\phi)$ and $\lim_{\phi\to \pi/4^-}y_p\,'(\phi)$ do not exist, so the function is not differentiable on the entire interval\footnote{\href{http://amsi.org.au/ESA_Senior_Years/SeniorTopic3/3j/3j_2content_2.html}{Newton's method}}.

From observing the graph of $y_p(\phi)$ on Page 6, this is confirmed visually. Since the endpoints of $y_p(\phi)$ on this interval are its two roots,
they cannot be found through this method. The bisection method works, however, since the only requirement is continuity of $y_p(\phi)$ on the interval.

\begin{figure}[H]
    \centering
    \includegraphics[]{plots/phi-functions.pdf}
    \caption{A plot of $\phi_\mathrm{min}$ and $\phi_0$ vs $\delta$}\label{phifig}
\end{figure}

The code used to implement both algorithms is located \href{https://gist.github.com/sidnb13/d715682a9915ec4cf49d66ceb2e54855}{here}. Figure \ref{phifig} demonstrates the bisection method for finding $\phi_0$ and Newton's method for $\phi_\mathrm{min}$.

\section{Curve fitting}

\subsection{Objective}

The least-squared method of curve fitting\footnote{\href{https://www.dam.brown.edu/people/alcyew/handouts/leastsq.pdf}{Curve fitting: least squares methods}} was used to generate the closed-form equations $\phi_0(\delta)$ and $\phi_\mathrm{min}(\delta)$.

The approach is to define an error function $E(\phi)=\sqrt{\frac{1}{n}\sum_{i=1}^n(\phi_i-\phi(\delta_i))^2}$ that represents the RMS error between the fitted curve and original data. $n$ is the number of data points in the set $\{(\delta_1,\phi_1),\ldots,(\delta_n,\phi_n)\}$ and $\phi(\delta)$ represents the fitted curve.
Because a polynomial curve is desired, the coefficients of the terms in $\delta(\phi)$ belong to $\vec c\in \mathbb{R}^{k+1}$, so there are $k+1$ coefficients in the fitted polynomial. Since $\phi(\delta;\vec{c})$, $E:\mathbb{R}^{k+1}\to \mathbb{R};\vec{c}$. The objective is then to find a $\vec{c}$ which minimizes $E(\vec{c})$. Similarly, each $\phi(\delta_i)$ becomes a function of $\vec{c}$.

Minimizing $E(\vec{c})=\sqrt{\frac{1}{n}\sum_{i=1}^n(\phi_i-\phi(\delta_i))^2}$ is the same as minimizing $E_1(\vec{c})=\sum_{i=1}^n(\phi_i-\phi(\delta_i))^2$, so the next step is to solve

\begin{gather}
    \frac{\partial E_1}{\partial c_j}=2\sum_{i=1}^n(\phi_i-\phi(\delta_i)) \frac{\partial \phi(\delta_i)}{\partial c_j}=0\\
    \sum_{i=1}^n\frac{\partial \phi(\delta_i)}{\partial c_j}\phi_i=\sum_{i=1}^n\frac{\partial \phi(\delta_i)}{\partial c_j}\phi(\delta_i)\\
    \sum_{i=1}^n \delta_i^{j-1}\phi_i=\sum_{i=1}^n c_j\delta_i^{2j-1}+\ldots+c_1\delta_i^{j-1}=\sum_{\ell=1}^j\sum_{i=1}^n c_{j-\ell+1}\delta_i^{2j-\ell-1}
\end{gather}

where $j=1,\ldots,k+1$ and the polynomial is of degree $k$.

A vectorized implementation yields

\begin{equation}
    \begin{bmatrix}\sum_{i=1}^n \delta_i^k\phi_i\\\vdots\\\sum_{i=1}^n \delta_i^1\phi_i\end{bmatrix}
    =\begin{bmatrix}\sum_{i=1}^n\delta_i^{2k}&\cdots&\sum_{i=1}^n\delta_i^{k}\\\vdots&\ddots&\vdots\\\sum_{i=1}^n \delta_i^{k}&\cdots&\sum_{i=1}^n\end{bmatrix}
    \begin{bmatrix}c_{k+1}\\\vdots\\c_1\end{bmatrix}
    =\begin{bmatrix}\sum_{i=1}^nc_{k+1}\delta_i^{2k}&\cdots&\sum_{i=1}^nc_1\delta_i^{k}\\\vdots&\ddots&\vdots\\\sum_{i=1}^n c_{k+1}\delta_i^{k}&\cdots&\sum_{i=1}^nc_1\end{bmatrix}\\
\end{equation}

\begin{equation}
    \vec{b}=A\vec{c}
\end{equation}

The matrix-vector equation $A\vec{c}=\vec{b}$ can then be solved for $\vec{c}$ using an optimized linear algebra library, and the resulting coefficients used to generate a function $\phi(\delta)$.

\begin{figure}[H]
    \centering
    \includegraphics[]{plots/phi-curve-fit-poly.pdf}
    \caption{Using degree 5 polynomials to approximate $\phi_0(\delta)$ and $\phi_\mathrm{min}(\delta)$}\label{polyerror}
\end{figure}

A better approximation is desired since even with degree 5, a polynomial approximation cannot fit well to the smooth curves of $\phi_0$ and $\phi_\mathrm{min}$. This is evidenced by the Runge effect visible in the error plot of Figure \ref{polyerror}.

\subsection{Attempted approximation using Chebyshev polynomial interpolation}

The Chebyshev polynomial approximations resulted in a more pronounced Runge effect, as depicted below.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.5]{plots/cheby-approx.pdf}
    \caption{Chebyshev polynomial approximations}
\end{figure}

\subsection{Attempted approximation using exponential function}

By far the most accurate fit before the original polynomial approximation, an exponential function of the form $\phi=a\exp(b(\delta-c))+d$ was used.
Significant Runge effect is still evident in the plot below.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.5]{plots/exponential-approx.pdf}
    \caption{Exponential approximations}
\end{figure}

\subsection{Attempted elliptical integral of first kind approximation}

Through parameterizing the first-order elliptic integral $K(k;n)$ for some $n\in \mathbb{R}$ and $k\in[0,1]$ for both translation and dilation control, curve fitting was attempted:

\begin{equation}
    K(k;n)=-\frac{\pi}{2n}+\int_0^1 \frac{dx}{n\sqrt{(1-x^2)(1-k^2x^2)}}
\end{equation}

The optimization routine returned a value of $n=1$ for both $\phi_0$ and $\phi_\mathrm{min}$, indicating that the first-order approximation is not effective. The plot is shown below.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.5]{plots/elliptical-approx.pdf}
    \caption{Elliptical approximations}
\end{figure}

Further parameterization attempts of $K(k)$ did not yield better results, as the curvature of the elliptic integral function cannot be manipulated easily to fit the data.

\subsection{Univariate spline approximation}

This method yielded the best results. A univariate spline of degree $k=3$ with smoothness factor $s=0.001$ was used to approximate $\phi_0$ and $\phi_\mathrm{min}$.
Figure \ref{splineplot} shows that the method was successful, with minimized Runge effect and typical error magnitudes lower than that of the polynomial approximation.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.8]{plots/spline.pdf}
    \caption{Spline approximation plot}\label{splineplot}
\end{figure}

The computer output below details the properties of each spline:

{\small\begin{verbatim}
    ---------- phi_0 spline approx ----------
LSQ Error =         0.001000051241244917
Knots =         [0.001 0.501 0.751 0.876 1.   ]
Coeffs =        [-4.40529375e-04  3.92784349e-02  1.04654145e-01  2.56034241e-01
  4.15005836e-01  5.76096213e-01  7.73738936e-01]

---------- phi_min spline approx ----------
LSQ Error =         0.0009999540547108812
Knots =         [0.001 0.501 0.626 0.751 0.814 0.876 0.907 0.938 0.969 0.977 0.985 0.993
 0.997 1.   ]
Coeffs =        [-4.58365176e-05  1.99513345e-02  4.68542634e-02  1.06810792e-01
  1.41001801e-01  1.78919227e-01  2.10975405e-01  2.46198594e-01
  2.81539962e-01  3.21468750e-01  3.64172354e-01  3.99187610e-01
  4.27132615e-01  5.02562701e-01  4.92718655e-01  7.78749733e-01]
\end{verbatim}}


\subsection{A Worked Example}
We have already constructed Riemann surfaces associated to the multi-valued functions $\log$ and $\sqrt[k]{\cdot}$.
Here we will treat another function in this way, namely $w=\sqrt{z^3-z}$.
One possible approach to do it is to give a conformal structure on the graph $\{(w,z)\in\mathbb C^2:w^2=z^3-z\}$.
This approach will not be discussed here since we are going to do it on example sheet.\\
We will use our old idea of ``gluing'' again.
As with $\log$, we need to consider some function elements.
Start with $f(z)=z^3-z=z(z+1)(z-1)$.
We know that $\sqrt{\cdot}$ has branches locally near any point but $0$ which corresponds to the roots $0,\pm 1$ of $f$.\\
Let $D=\mathbb C\setminus([-1,0]\cup[1,\infty))$.
We want to construct function elements $g$ on $D$ such that $g(z)^2=f(z)$.
Fix any $z_0\in D$ and let $g(z_0)=w_0$ for a chosen $w_0^2=f(z_0)$.
Now set
$$g(x)=g(z_0)\exp\left( \frac{1}{2}\int_\gamma\frac{f^\prime(\zeta)}{f(\zeta)}\,\mathrm d\zeta \right)$$
for any choice $\gamma$ of path from $z_0$ to $z$.
To see the choice of the path $\gamma$ does not intervene the value of $g$, consider a loop $\gamma$.
By the argument principle,
$$\frac{1}{2\pi i}\int_\gamma\frac{f^\prime(\zeta)}{f(\zeta)}\,\mathrm d\zeta=\sum_{z_i\text{ zeros of }f}n(\gamma,z_i)-\sum_{p_j\text{ poles of }f}n(\gamma,p_j)$$
where $n$ denote the winding number.
In this particular case, this evaluates to $n(\gamma,0)+n(\gamma,-1)+n(\gamma,1)$.
In example sheet, we have shown that $n(\gamma,1)=0$ and $n(\gamma,-1)=n(\gamma,0)$, hence this integral is an even integer, therefore
$$\exp\left( \frac{1}{2}\int_\gamma\frac{f^\prime(\zeta)}{f(\zeta)}\,\mathrm d\zeta \right)=1$$
This means $g$ is well-defined.
It is fairly standard to show that $g$ is continuous since $f^\prime/f$ is continuous wherever relevant.
It is also pretty clear that $g$ is analytic, since we can choose a local branch such that $\sqrt{\cdot}$ is analytic and $g(z)=\sqrt{f(z)}$.
Now the choice of $w_0$ gives two branches $g_+,g_-$ of $g$, so we get the function elements $(g_+,D_+),(g_-,D_-)$ where $D_+$ and $D_-$ are both copies of $D$.
Obviously, on the Riemann sphere $D$ is just the sphere removed two closed segments.
This is homeomorphic to the cylinder $S^1\times\mathbb R$.
So we want to figure out a way to glue this two cylinders corresonding to $D_+,D_-$ together along the branch cuts to make it the surface we wanted.
For $z_0\in (-1,0)\cup(1,\infty)$, we have
$$\lim_{z\to z_0^-}g_+(z)=\lim_{z\to z_0^+}g_-(z),\lim_{z\to z_0^+}g_+(z)=\lim_{z\to z_0^-}g_-(z)$$
where $z\to z_0^+$ denotes the limit of approaching $z_0$ from the upper half-plane and $z\to z_0^-$ is approaching from the lower half-plane.
This immediately tell us that we want to glue them together at the corresponding boundaries where the upper half-plane from one side is glued to the lower half-plane from the other side and leave $0,\pm 1,\infty$ alone.
This gives our Riemann surface $R$ (with the obvious conformal structure) which is a torus with $4$ points removed.
We see this via our geometrical intuition.
We will later see a computational technique to identify these surfaces.\\
Like in previous examples, we get some analytic functions with nice properties.
$g_\pm$ defines an analytic map $g:R\to\mathbb C$ and the inclusion $D_\pm\hookrightarrow\mathbb C\setminus{0,\pm 1}$ gives a covering map (also analytic) $\pi:R\to\mathbb C$.
Also, locally (hence globally) we have $g(p)^2=f\circ\pi(p)=\pi(p)^3-\pi(p)$.


\section{Genetic Algorithms}

\paragraph{Chromosome} A chromosome is a representation in which:
\begin{itemize}
  \item There is a list of lements called genes
  \item The chromosome determines the overall fitness manifested by
    some mechanism that uses the chromosome's genes as a sort of
    blueprint
\end{itemize}
With constructors that:
\begin{itemize}
  \item Create a chromosome, given a list of elements; this
    constructor might be called the \textit{genesis constructor}
  \item Create a chromosome by crossing a pair of existing
    chromosomes
\end{itemize}
With writers that:
\begin{itemize}
  \item Mutate an existing chromosome by changing one of the genes
\end{itemize}
With readers that:
\begin{itemize}
  \item Produce a specified gene, given a chromosome
\end{itemize}

\subsection{Fitness}

The \textbf{standard method} for fitness computation is:
\begin{math}
  f_i = \frac{q_i}{\sum_{j}{q_j}}
\end{math}

The \textbf{rank method} for fitness computation is:
\begin{itemize}
  \item Sort the $n$ individuals by quality
  \item Let the probability of selecting the $i$th candidate, given
    that the first $i-1$ candidates have not been selected, be $p$,
    except for the final candidate, which is selected if no
    previous candidate has been selected
  \item Select a candidate using the computed probabilities
\end{itemize}

The \textbf{rank-space method} for fitness computation is:
\begin{itemize}
  \item Sort the $n$ individuals by quality
  \item Sort the $n$ individuals by the sum of their inverse
    squared distances to already selected candidates (the lower the
    sum, the better the rank)
  \item Use the rank method, but sort on the sum of the quality
    rank and the diversity rank, rather than on quality rank only
\end{itemize}

\subsection{Crossover}

\begin{itemize}
  \item Crossover enables to search high-dimensional spaces efficiently;
it reduces the dimensionality of the optimum search space
  \item Crossover enables genetic algorithms to travers obstructing
    moats
\end{itemize}

\subsection{Natural selection}

To mimic natural selection in general:
\begin{itemize}
  \item Create an initial population of one chromosome
  \item Mutate one or more genes in one or more of the current
    chromosomes, producitng one new offspring for each chromosome
    mutated
  \item Mate one or more pairs of chromosomes
  \item Add the mutated na doffspring chromosomes to the current
    population
  \item Create a new generation by keeping the best of the current
    population's chromosomes, along with other chromosomes selected
    randomly from the current population. Bias the random selection
    according to assessed fitness
\end{itemize}


%
% Chapter 2.4
%

\section*{2.4 Derivatives of Trigonometric Functions}

\[ \lim_{\theta \to 0} \frac{\sin(\theta)}{\theta}=1 \]
\[ \lim_{\theta \to 0} \frac{\cos(\theta)-1}{\theta}=0 \]
\[ \frac{d}{dx}(\sin(x))=\cos(x) \]
\[ \frac{d}{dx}(\cos(x))=-\sin(x) \]
\[ \frac{d}{dx}(\tan(x))=\sec^2(x) \]
\[ \frac{d}{dx}(\csc(x))=-\csc(x)\cot(x) \]
\[ \frac{d}{dx}(\sec(x))=\sec(x)\tan(x) \]
\[ \frac{d}{dx}(\cot(x))=-\sec^2(x) \]


% Fill out with more info

\chapter{Density Estimation}
Probability density functions are the baseline for many mathematical models, with the Gaussian distribution being the most frequently used distribution. Probability densities have a wide range of uses and can be used for classification, clustering of data, dimensional reduction, and field-specific applications such as noise-filtering or pricing financial derivatives. The central problem is that probability density functions in most cases are unknown. This is where density estimation comes in: if we are given a set of independent observed variables $x_i$, $i=1,\dots,n$, then we wish to estimate the unknown underlying probability distribution.

\section{Normalizing Flows}
Normalizing Flows are a family of generative models with tractable probability distributions, where both sampling and evaluation of probabilities can be exact, unlike Generative Adversarial Networks (GANs) or Variational Auto-Encoders (VAEs).
Normalizing Flows were initially introduced by \citeauthor*{tabak-2010-normalizing-flows-dual-ascent-log-likelihood} \parencite*{tabak-2010-normalizing-flows-dual-ascent-log-likelihood, tabak-2013-normalizing-flows-nonparametric-density-estimation-algorithms}, but first gained traction within variational inference when \ccite{rezende-2015-normalizing-flows} demonstrate the ability to create extremely rich posterior approximations that is able to contain the true posterior distribution in the asymptotic regime, and within density estimation when \ccite{dinh-2014-nice} achieves results competitive with the then state-of-the-art while having tractable evaluation and sampling.

A normalizing flow is a transformation of a simple baseline probability distribution into a more complex distribution by letting it "flow" through a sequence of differentiable and invertible mappings. The density of an observed sample can then be evaluated by transforming it back to the latent simple baseline distribution and computing the product of the inverse-transformed sample under the latent distribution and the changes in volume that originate from the sequence of inverse transformations. This method also allows us to sample from the latent simple distribution and then applying the transformations to obtain a sample from the complex distribution.

\subsection*{Change of Variables}
Normalizing Flows utilizes the Change of Variables Theorem, which describes how volumes get distorted by differentiable and invertible functions.

If we have a differentiable map $\Phi : U \to \mathbb{R}^n$ where $U\subset\mathbb{R}^n$, then the Jacobian matrix is given by
\begin{equation}
    \nabla\Phi(\bm{x}) = \begin{bmatrix}
        \frac{\partial\Phi_1}{\partial x_1} & \cdots & \frac{\partial\Phi_1}{\partial x_n} \\
        \vdots & \ddots & \vdots \\
        \frac{\partial\Phi_n}{\partial x_1} & \cdots & \frac{\partial\Phi_n}{\partial x_n}
    \end{bmatrix}
\end{equation}
The determinant of the Jacobian matrix is called the Jacobian and will be denoted as $J_{\Phi}(\bm{x})=\det\left|\nabla\Phi(\bm{x})\right|$.
If we constrain $J_{\Phi}(\bm{x})\neq0$ for all $\bm{x}\in U$ then it is given, through the Inverse Function Theorem, that $\Phi(\bm{x})$ is invertible, that $\Phi^{-1}$ is continuously differentiable, and $\nabla\Phi^{-1}(\Phi(\bm{x}))=(\nabla\Phi(\bm{x}))^{-1}$, which means we have a $C^1$-diffeomorphism between $U$ and its image $\Phi(U)$.

If we have a $C^1$-diffeomorphism $\Phi:U\to V$, where $V \subset \mathbb{R}^n$, then for any continuous function $f$ in $V$, we get the Change of Variables Theorem
\begin{equation}
    \int_V f(\bm{x}) d\bm{x} = \int_U f(\Phi(\bm{y})) |J_\Phi(\bm{y})| d\bm{y}
\end{equation}

\subsection*{Change of Random Variable}
Now let $\bm Z$ and $\bm X$ be random variables that are related by a mapping $\Phi:\mathbb{R}^n\to \mathbb{R}^n$ such that $\bm X = \Phi(\bm Z)$ and $\bm Z = \Phi^{-1}(\bm X)$. If we then compute the expectation of $f(\bm X)$ where $f$ is some continuous function, then
\begin{align}
    \mathbb{E}_{p_Z(\cdot)}[f(\bm Z)] &= \int_{\text{supp}(\bm Z)} f(\bm z) p_Z(\bm z) d\bm z\\
    &= \int_{\text{supp}(\bm X)} f(\Phi^{-1}(\bm x)) p_Z(\Phi^{-1}(\bm x)) |J_{\Phi^{-1}}(\bm x)| d\bm x \\
    &= \int_{\text{supp}(\bm X)} f(\Phi^{-1}(\bm x)) p_X(\bm x) d\bm x \\
    &= \mathbb{E}_{p_X(\cdot)}[f(\Phi^{-1}(\bm x))]
\end{align}

As we have the equality
\begin{align}
    \int_{\text{supp}(\bm X)} f(\Phi^{-1}(\bm x)) p_Z(\Phi^{-1}(\bm x)) |J_{\Phi^{-1}}(\bm x)| d\bm x
    &= \int_{\text{supp}(\bm X)} f(\Phi^{-1}(\bm x)) p_X(\bm x) d\bm x
\end{align}
it follows that
\begin{align}
    p_X(\bm x) &= p_Z(\Phi^{-1}(\bm x)) |J_{\Phi^{-1}}(\bm x)| \\
    &= p_Z(\bm z) |J_{\Phi}(\bm z)^{-1}|
    \label{eq:pX-mapping}
\end{align}
which means that we can describe the random variable $\bm X$ through the use of a $C^1$-diffeomorphism and a latent variable $\bm Z$.

\subsection*{Constructing arbitrarily complex density}
An arbitrarily complex density can be constructed by composing several simple maps and applying \cref{eq:pX-mapping} successively. If a random variable $\bm{z}_0$ comes from a distribution $q_0$, then we can obtain $q_K(\bm z)$ through a chain of $K$ transformations $\Phi_k$:
\begin{align}
    \bm{z}_K &= \Phi_K\circ\cdots\circ \Phi_1(\bm{z}_0)\\
    \ln q_K(\bm{z}_K) &= \ln q_0(\bm{z}_0) - \sum_{k=1}^{K} \ln \left|J_{\Phi_{k}}(\bm{z}_{k-1})\right|
    \label{eq:lnqK}
\end{align}
where \cref{eq:lnqK} is obtained by repeatedly applying the log of \cref{eq:pX-mapping}. The path traversed by $\bm{z}_k=\Phi_k(\bm{z}_{k-1})$ is called the flow and the path formed by the successive distributions $q_k$ is a Normalizing Flow (\ccite{rezende-2015-normalizing-flows}). The big problem with this form is finding mappings such that the calculation of the Jacobian isn't $\mathcal{O}(D^3)$, where $D$ is the dimensionality.

\section{Continuous Normalizing Flows}
\ccite{chen-2018-cnf} builds upon the concepts from \ccite{rezende-2015-normalizing-flows} by introducing Instantaneous Change of Variables. If $\bm{z}(t)$ is a finite continuous random variable with probability $p(\bm{z}(t))$ dependent on time, $\frac{d\bm z}{dt}=f(\bm{z}(t), t)$ is a differential equation describing a continuous-in-time transformation of $\bm{z}(t)$, and $f$ is uniformly Lipschitz continuous in $\bm z$ and continuous in $t$, then the change in log probability is given as
\begin{equation}
    \frac{\partial \ln p(\bm{z}(t))}{\partial t} = - \text{tr}\left(\frac{df}{d\bm{z}(t)}\right)
\end{equation}
This means that the change in log probability can be calculated using a trace operation instead of calculating the Jacobian, which requires calculating a determinant. The total change in log probability can then be found by integrating over time
\begin{equation}
    \ln p(\bm{z}(t_1)) = \ln p(\bm z(t_0)) - \int_{t_0}^{t_1} \text{tr}\left(\frac{df}{d\bm{z}(t)}\right) dt
    \label{eq:cnf}
\end{equation}
This method reduces the primary bottleneck to $\mathcal{O}(D^2)$, but comes at the cost of introducing a numerical ODE solver.

\section{Free-form Jacobian of Reversible Dynamics}
Free-form Jacobian of Reversible Dynamics (FFJORD) was introduced by \ccite{grathwohl-2018-ffjord} and builds upon Continuous Normalizing Flows. FFJORD utilizes vector-Jacobian products with reverse-mode automatic differentiation, which allows calculating $\bm v^T \frac{\partial f}{\partial \bm z}$ at roughly the same cost as evaluating $f$, and Hutchinson's trace estimator (\ccite{hutchinson-1989-stochastic-trace}), which says that for any $D\times D$-matrix $\bm A$ the trace can be estimated as
\begin{equation}
    \text{tr}(\bm A) = \mathbb{E}_{p(\bm\epsilon)}[\bm\epsilon^T\bm A\bm\epsilon]
\end{equation}
where $p(\bm\epsilon)$ is a distribution over $D$-dimensional vectors with $\mathbb{E}[\bm\epsilon]=0$ and $\text{Cov}(\bm\epsilon)=\bm I$.

This means that \cref{eq:cnf} ends up becoming
\begin{align}
    \ln p(\bm{z}(t_1)) &= \ln p(\bm z(t_0)) - \int_{t_0}^{t_1} \text{tr}\left(\frac{df}{d\bm{z}(t)}\right) dt\\
                       &= \ln p(\bm z(t_0)) - \int_{t_0}^{t_1} \mathbb{E}_{p(\bm\epsilon)}\left[\bm\epsilon^T\frac{\partial f}{\partial \bm z(t)}\bm\epsilon\right] dt\\
                       &= \ln p(\bm z(t_0)) - \mathbb{E}_{p(\bm\epsilon)}\left[\int_{t_0}^{t_1}\bm\epsilon^T\frac{\partial f}{\partial \bm z(t)}\bm\epsilon dt\right]
\end{align}
where the bottleneck of calculating the trace estimate is $\mathcal{O}(D)$ and allows for an unconstrained Jacobian due to utilizing vector-Jacobian products.


\documentclass[12pt,a4paper]{article}

\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{makeidx}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{caption}
\usepackage{float}
\usepackage[left=2cm,right=2cm,top=2cm,bottom=2cm]{geometry}
\usepackage{tikz}
\usepackage{pdfpages}
\pagestyle{plain}
\usepackage{bm}
\usepackage{ulem}
\usepackage{units}
\usepackage{makecell}

%Kopf und Fußzeile:
\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\renewcommand{\footrulewidth}{0.4pt}
\usepackage{array}   % for \newcolumntype macro
\newcolumntype{C}{>{$}c<{$}} % math-mode version of "c" column type

\fancyhead[L]{Computational photonics}
%\fancyhead[C]{Dr. Bj\"orn Leder}
\fancyhead[R]{Excercise 1}
\fancyfoot[L]{\today}
\fancyfoot[R]{page \thepage}
\fancyfoot[C]{Julien Kluge}

% Code-Listings
\usepackage{listings}
\lstset{
language=c,
showstringspaces=false,
numbers=left,
xleftmargin=2em}

% Ganze Dateien als Verbatim einbinden
%\usepackage{verbatimfiles} % downloaded from ctan.org

\title{Computational photonics}
\author{Julien Kluge}
\date{\today}

%============================================================
% Dokument
%============================================================
\begin{document}

\lstset{numbers=left}

\begin{center}
\large{\textbf{Computational photonics -- Excercise 1}} \\
~\\
\small{-- Julien Kluge (564513) --}\\
~\\
Date: \today
\end{center}
\hrule

\section*{Task 1}
	\begin{table}[H]
		\caption{Program file index - Task 1}
		\begin{tabular}{r|c|l}
			\textit{A1/A1.m} & Matlab & program with whole task in sections
		\end{tabular}
	\end{table}
	\subsection*{a)}
		\begin{lstlisting}
gauss1D = @(x) exp(-(x * x));
x = -xMax:dx:xMax;
y = arrayfun(gauss1D, x);
plot(x, y)
		\end{lstlisting}
		\begin{figure}[H]
			\centering
			\includegraphics[width=0.7\textwidth]{A1/A1_a.png}
			\caption{Plot of the function \(f(x)=\exp\left(-x^2\right)\) in matlab.}
		\end{figure}
	\subsection*{b)}
		\begin{lstlisting}
gauss2D = @(x, y) exp(-(x * x + y * y));
g = -gMax:dg:gMax;
[x, y] = meshgrid(g, g);
z = arrayfun(gauss2D, x, y);
mesh(x, y, z)
surface(x, y, z)
pcolor(x, y, z)
contour(x, y, z)
		\end{lstlisting}
		\begin{figure}[H]
			\centering
			\includegraphics[width=0.9\textwidth]{A1/A1_b.png}
			\caption[]{Plot of the function
			\(f(x)=\exp\left(-\left(x^2+y^2\right)\right)\) in matlab for different commands.}
		\end{figure}
	\subsection*{c)}
		Quiver code omitted for clarity in this document.
		\begin{figure}[H]
			\centering
			\includegraphics[width=\textwidth]{A1/A1_c.png}
			\caption[]{Vectorfields \(\mathbf{E_1}=(-y, x)\) and \(\mathbf{E_2}=-(y, x)\) plotted with quiver.}
		\end{figure}
	\subsection*{d)}
		\begin{lstlisting}
	h = @(x) exp(1 - 1 ./ x) ./ x;
	int1 = integral(h, eps, 1);
		\end{lstlisting}
		\begin{figure}[H]
			\centering
			\includegraphics[width=0.7\textwidth]{A1/A1_d.png}
			\caption[]{Function \(h(x)\) plotted in matlab.}
		\end{figure}
		Even though matlab properly integrates this function and acquires
		the right value of about
		\begin{align}
			\int_0^1\text{d}x\,h(x)=e\cdot\Gamma\left(0,1\right)\approx 0.5963
		\end{align}
		it could still pose a problem for similar functions since the function
		inhibits a pole on \(x=0\). It can easily through variable substitution be shown,
		that
		\begin{align}
			\lim_{x\rightarrow 0}h(x)=0
		\end{align}
		but programs cannot do the symbolic limit implicitly and therefore acquire an error
		due to the division by zero. This can be cirumvented with two possibilities:
		\begin{enumerate}
			\item Since we know \(h(x\rightarrow0)\rightarrow0\) we can explicitly spare the
			integral-function of the pole by adjusting integration limit to \(\left[\epsilon,1\right]\)
			where \(\epsilon\) is the machine-epsilon of the according floating point type (\textit{eps} in matlab).
			\item We can do a variable substition \(x=1/y\) under the integral
			\begin{align}
				\int_0^1\text{d}x\,\frac{\exp\left(1-\frac{1}{x}\right)}{x}=\int_1^\infty\text{d}y\,\frac{\exp\left(1-y\right)}{y}
			\end{align}
			This seems to pose the exchanged problem of integrating to infinity. However, after quick
			calculations in can be shown that \(h(y)<\epsilon\) for \(y\geq 34.31\) with a
			strong monotonic descrease (\(h(y)\propto1/\left(y\cdot\exp y\right)\)). So we can comfortably
			set the integration limits to \([1,34.31]\) (or adapting to a specific precision-limit).
		\end{enumerate}
		\newpage

\section*{Task 2}
	\begin{table}[H]
		\caption{Program file index - Task 2}
		\begin{tabular}{r|c|p{10cm}}
			\textit{A2/A2.jl} & Julia & program with tasks a, b and part of c \\
			\textit{A2/naiveDFT.jl} & Julia & implementation of the naive dft in matrix and\newline iterative form \\
			\textit{A2/simpleDFT.jl} & Julia & implementation of the recursive dft \\
			\textit{A2/benchmarkFunction.jl} & Julia & benchmark function for fft-functions \\

			\textit{A2/A2c\_MatlabFFT.m} & Matlab & programm to acquire matlab fft benchmarks \\
			\textit{A2/A2d.nb} & Mathematica & notebook to calculate aliasing graphs \\
			\textit{A2/data/Visualize.nb} & Mathematica & notebook to calculate graphs for c
		\end{tabular}
	\end{table}
	\subsection*{a) and b)}
		According to the program file index the ffts where implemented in naive and recursive
		form and compared to the FFTW package. The comparison was made with an artificial created
		dataset of three sine waves with different amplitude, frequency and phase.
		\begin{figure}[H]
			\centering
			\includegraphics[width=\textwidth]{A2/data/fftPlot.png}
			\caption[]{Comparison of real, imaginary and absolute values of FFTW, the naive and recursive implementation}
		\end{figure}
		The summed differences where all calculated to be below \(10^{-10}\).

	\subsection*{c)}
		The Julia functions and a Matlab benchmark where made with different problem sizes
		all with a power of two. The stopping criterion was set, when an implementation
		reached calculation times over a second. Each test was repeated five times and
		was averaged out.\\
		Following results where acquired:
		\begin{figure}[H]
			\centering
			\includegraphics[width=0.7\textwidth]{A2/data/Comparison.pdf}
			\caption[]{Comparisons between different dft implementations. The simple, recursive
			Julia implementation is notably only 5 times slower than FFTW.}
		\end{figure}
		The fastest implementation seems to be matlabs fft. However, it is known
		that matlab utilizes FFTW-plan implementations. The exact same speed could
		be therefore in Julia be acquired with a single FFTW-plan allocation.\\
		The naive implementations are fitted with a runtime slightly above
		\(\mathcal{O}\left(N^2\right)\). All other implemntations are approximately
		\(\approx\mathcal{O}\left(N^{1.1}\right)\) for the simple, recursive implementation
		and \(\mathcal{O}\left(N\log N\right)\) in the given uncertainty intervals.

	\subsection*{d)}
		\begin{align}
			\hat{\mathcal{F}}_t^\omega\left\{\exp\left(-t^2\right)\right\}=\sqrt{\pi}\exp\left(\frac{-\omega^2}{4}\right)
		\end{align}
		\begin{figure}[H]
			\centering
			\begin{subfigure}[b]{0.45\textwidth}
				\includegraphics[width=0.9\columnwidth]{A2/data/DT_Comparison.pdf}
				\caption[]{Comparison between different sampling distances \(\Delta t\)}
			\end{subfigure}
			\begin{subfigure}[b]{0.45\textwidth}
				\includegraphics[width=0.9\columnwidth]{A2/data/TSampling_Comparison.pdf}
				\caption[]{Comparison between different sampling intervals \(t_{smpl}\in\left[-t, t\right]\)}
			\end{subfigure}
			\caption[]{Comparison of aliasing in different sample intervals and sampling distances.}
		\end{figure}
		\noindent It can quickly be seen how the different intervals and distances distort the resulting
		fourier points. The sampling interval seems to have a stronger effect.\\
		To confirm that, a density plot of the absolute differences to the analytical solution
		can be made.
		\begin{figure}[H]
			\centering
			\includegraphics[width=0.7\textwidth]{A2/data/DT_TSampling_Contour.png}
			\caption[]{Absolute differences of fft to analytical solution, scaled
			by the squareroot (for better visibility) in respect to the sampling
			interval and distances.}
		\end{figure}
		\noindent The multiplication by \(N\,\Delta t\, (-1)^n\) is necessary to
		normalize the fourier coefficients and the last term is for shifting the resulting
		coefficients to the zero-frequency.
		\newpage

\section*{Task 3}
	\begin{align}
		\frac{\partial \mathbf{H}(\mathbf{r,t})}{\partial t}&=-\frac{1}{\mu_0\,\mu(\mathbf{r})}\nabla\times\mathbf{E}(\mathbf{r},t)\\
		\frac{\partial \mathbf{E}(\mathbf{r,t})}{\partial t}&=\frac{1}{\epsilon_0\,\epsilon(\mathbf{r})}\nabla\times\mathbf{H}(\mathbf{r},t)
	\end{align}
	First we introduce to arbitrary field strengths \(E_0, H_0\) such that
	\begin{align}
		\frac{\mathbf{E}}{E_0}&=\widetilde{\mathbf{E}} \\
		\frac{\mathbf{H}}{H_0}&=\widetilde{\mathbf{H}}
	\end{align}
	Which then renders the upper equations to
	\begin{align}
		\frac{\partial \widetilde{\mathbf{H}}(\mathbf{r,t})}{\partial t}&=-\frac{E_0}{H_0\,\mu_0}\frac{1}{\mu(\mathbf{r})}\nabla\times\widetilde{\mathbf{E}}(\mathbf{r},t)\\
		\frac{\partial \widetilde{\mathbf{E}}(\mathbf{r,t})}{\partial t}&=\frac{H_0}{E_0\,\epsilon_0}\frac{1}{\epsilon(\mathbf{r})}\nabla\times\widetilde{\mathbf{H}}(\mathbf{r},t)
	\end{align}
	We can equate the terms with \(E_0,H_0,\epsilon_0,\mu_0\) to a constant.
	Since the field strength modifier are arbitrary, the constants are also arbitrary.
	Therefore we set it equal to one:
	\begin{align}
		1&=\frac{E_0}{H_0\,\mu_0} \\
		1&=\frac{H_0}{E_0\,\epsilon_0}
	\end{align}
	This system is underdetermined. We can therefore only solve for one solution.
	Multiplication and division yields following equations respectively
	\begin{align}
		1&=\frac{1}{\epsilon_0\,\mu_0}=c^2 \\
		1&=\frac{E_0^2\,\epsilon_0}{H_0^2\,\mu_0}
	\end{align}
	The first gives the natural units where \(c=1\). This enables further manipulation
	for time or space such that \(\mathbf{r}/r_0=\widetilde{\mathbf{r}}\) and
	\(t\cdot c/r_0=\widetilde{t}\). The second equation gives a solution for either variable.
	For the arbitrary decision to solve \(H_0\) we get
	\begin{align}
		H_0=E_0\sqrt{\frac{\epsilon_0}{\mu_0}}
	\end{align}
	Thus the final result in natural units (\(c=1\)) can be archived:
	\begin{align}
		\frac{\partial \widetilde{\mathbf{H}}(r_0\,\widetilde{\mathbf{r}},r_0\,\widetilde{t})}{\partial \widetilde{t}}&=-\frac{1}{\mu(r_0\,\widetilde{\mathbf{r}})}\nabla_{\widetilde{r}}\times\widetilde{\mathbf{E}}(r_0\,\widetilde{\mathbf{r}},r_0\,\widetilde{t})\\
		\frac{\partial \widetilde{\mathbf{E}}(r_0\,\widetilde{\mathbf{r}},r_0\,\widetilde{t})}{\partial \widetilde{t}}&=\frac{1}{\epsilon(r_0\,\widetilde{\mathbf{r}})}\nabla_{\widetilde{r}}\times\widetilde{\mathbf{H}}(r_0\,\widetilde{\mathbf{r}},r_0\,\widetilde{t})
	\end{align}
	Which also gives us the spatial discretization as \(r_0\).
\end{document}


\chapter{Generalized linear models and the exponential family}
\label{chap:GLM}

\section{The exponential family}
\label{sec:exponential-family}

Before defining the exponential family, we mention several reasons why it is important:
\begin{itemize}
\item{It can be shown that, under certain regularity conditions, the exponential family is the only family of distributions with finite-sized sufficient statistics, meaning that we can compress the data into a fixed-sized summary without loss of information. This is particularly useful for online learning, as we will see later.}
\item{The exponential family is the only family of distributions for which conjugate priors exist, which simplifies the computation of the posterior (see Section \ref{sec:Bayes-for-the-exponential-family}).}
\item{The exponential family can be shown to be the family of distributions that makes the least set of assumptions subject to some user-chosen constraints (see Section \ref{sec:Maximum-entropy-derivation-of-the-exponential-family}).}
\item{The exponential family is at the core of generalized linear models, as discussed in Section \ref{sec:GLMs}.}
\item{The exponential family is at the core of variational inference, as discussed in Section TODO.}
\end{itemize}

\subsection{Definition}
A pdf or pmf $p(\vec{x}|\vec{\theta})$,for $\vec{x} \in \mathbb{R}^m$ and $\vec{\theta} \in \mathbb{R}^D$, is said to be in the \textbf{exponential family} if it is of the form
\begin{align}
p(\vec{x}|\vec{\theta}) & =\dfrac{1}{Z(\vec{\theta})}h(\vec{x})\exp[\vec{\theta}^T\phi(\vec{x})] \\
    & = h(\vec{x})\exp[\vec{\theta}^T\phi(\vec{x})-A(\vec{\theta})] \label{eqn:exponential-family}
\end{align}
where
\begin{align}
Z(\vec{\theta}) & =\int h(\vec{x})\exp[\vec{\theta}^T\phi(\vec{x})]\mathrm{d}\vec{x} \\
A(\vec{\theta}) & =\log Z(\vec{\theta})
\end{align}

Here $\vec{\theta}$ are called the \textbf{natural parameters} or \textbf{canonical parameters}, $\phi(\vec{x}) \in \mathbb{R}^D$ is called a vector of \textbf{sufficient statistics}, $Z(\vec{\theta})$ is called the \textbf{partition function}, $A(\vec{\theta})$ is called the \textbf{log partition function} or \textbf{cumulant function}, and $h(\vec{x})$ is the a scaling constant, often 1. If $\phi(\vec{x})=\vec{x}$, we say it is a \textbf{natural exponential family}.

Equation \ref{eqn:exponential-family} can be generalized by writing
\begin{equation}
p(\vec{x}|\vec{\theta}) = h(\vec{x})\exp[\eta(\vec{\theta})^T\phi(\vec{x})-A(\eta(\vec{\theta}))]
\end{equation}
where $\eta$ is a function that maps the parameters $\vec{\theta}$ to the canonical parameters $\vec{\eta}=\eta(\vec{\theta})$.If $\mathrm{dim}(\vec{\theta})<\mathrm{dim}(\eta(\vec{\theta}))$, it is called a \textbf{curved exponential family}, which means we have more sufficient statistics than parameters. If $\eta(\vec{\theta})=\vec{\theta}$, the model is said to be in \textbf{canonical form}. We will assume models are in canonical form unless we state otherwise.

\subsection{Examples}

\subsubsection{Bernoulli}
The Bernoulli for $x \in \{0,1\}$ can be written in exponential family form as follows:
\begin{equation}\begin{split}
\mathrm{Ber}(x|\mu)& =\mu^x(1-\mu)^{1-x} \\
   & =\exp[x\log\mu+(1-x)\log(1-\mu)]
\end{split}\end{equation}
where $\phi(x)=(\mathbb{I}(x=0),\mathbb{I}(x=1))$ and $\vec{\theta}=(\log\mu,\log(1-\mu))$.

However, this representation is \textbf{over-complete} since $\vec{1}^T\phi(x)=\mathbb{I}(x=0)+\mathbb{I}(x=1)=1$. Consequently $\vec{\theta}$ is not uniquely identifiable. It is common to require that the representation be \textbf{minimal}, which means there is a unique $\theta$ associated with the distribution. In this case, we can just define
\begin{align}
\mathrm{Ber}(x|\mu) & =(1-\mu)\exp\left(x\log\dfrac{\mu}{1-\mu}\right) \\
\text{where } \phi(x) & =x, \theta=\log\dfrac{\mu}{1-\mu}, Z=\dfrac{1}{1-\mu}  \nonumber
\end{align}

We can recover the mean parameter $\mu$ from the canonical parameter using
\begin{equation}
\mu=\mathrm{sigm}(\theta)=\dfrac{1}{1+e^{-\theta}}
\end{equation}

\subsubsection{Multinoulli}
We can represent the multinoulli as a minimal exponential family as follows:
\begin{equation*}\begin{split}
& \mathrm{Cat}(\vec{x}|\vec{\mu}) = \prod\limits_{k=1}^K = \exp\left(\sum\limits_{k=1}^K x_k\log\mu_k\right) \\
    & = \exp\left[\sum\limits_{k=1}^{K-1} x_k\log\mu_k+  (1-\sum\limits_{k=1}^{K-1} x_k)\log(1-\sum\limits_{k=1}^{K-1} \mu_k)\right] \\
	& = \exp\left[\sum\limits_{k=1}^{K-1} x_k\log\dfrac{\mu_k}{1-\sum_{k=1}^{K-1} \mu_k} + \log(1-\sum\limits_{k=1}^{K-1} \mu_k) \right] \\
	& = \exp\left[\sum\limits_{k=1}^{K-1} x_k\log\dfrac{\mu_k}{\mu_K}+\log\mu_K\right] \text{, where } \mu_K \triangleq 1-\sum\limits_{k=1}^{K-1} \mu_k
\end{split}\end{equation*}

We can write this in exponential family form as follows:
\begin{align}
\mathrm{Cat}(\vec{x}|\vec{\mu}) & = \exp[\vec{\theta}^T\phi(\vec{x})-A(\vec{\theta})] \\
\vec{\theta} & \triangleq (\log\dfrac{\mu_1}{\mu_K},\cdots,\log\dfrac{\mu_{K-1}}{\mu_K}) \\
\phi(\vec{x}) & \triangleq (x_1,\cdots,x_{K-1})
\end{align}

We can recover the mean parameters from the canonical parameters using
\begin{align}
\mu_k & = \dfrac{e^{\theta_k}}{1+\sum_{j=1}^{K-1} e^{\theta_j}} \\
\mu_K & = 1- \dfrac{\sum_{j=1}^{K-1} e^{\theta_j}}{1+\sum_{j=1}^{K-1} e^{\theta_j}}=\dfrac{1}{1+\sum_{j=1}^{K-1} e^{\theta_j}}
\end{align}
and hence
\begin{equation}
A(\vec{\theta]} = -\log\mu_K=\log(1+\sum\limits_{j=1}^{K-1} e^{\theta_j})
\end{equation}

\subsubsection{Univariate Gaussian}
The univariate Gaussian can be written in exponential family form as follows:
\begin{align}
\mathcal{N}(x|\mu,\sigma^2) & =\dfrac{1}{\sqrt{2\pi}\sigma}\exp\left[-\dfrac{1}{2\sigma^2}(x-\mu)^2\right] \nonumber \\
    & = \dfrac{1}{\sqrt{2\pi}\sigma}\exp\left[-\dfrac{1}{2\sigma^2}x^2+\dfrac{\mu}{\sigma^2}x-\dfrac{1}{2\sigma^2}\mu^2\right] \nonumber \\
	& = \dfrac{1}{Z(\vec{\theta})}\exp[\vec{\theta}^T\phi(x)]
\end{align}
where
\begin{align}
\vec{\theta} & = (\dfrac{\mu}{\sigma^2}, -\dfrac{1}{2\sigma^2}) \\
\phi(x) & =(x,x^2) \\
Z(\vec{\theta}) & =\sqrt{2\pi}\sigma\exp(\dfrac{\mu^2}{2\sigma^2})
\end{align}

\subsubsection{Non-examples}
Not all distributions of interest belong to the exponential family. For example, the uniform distribution,$X \sim U(a,b)$, does not, since the support of the distribution depends on the parameters. Also, the Student T distribution (Section TODO) does not belong, since it does not have the required form.

\subsection{Log partition function}
An important property of the exponential family is that derivatives of the log partition function can be used to generate \textbf{cumulants} of the sufficient statistics.\footnote{The first and second cumulants of a distribution are its mean $\mathbb{E}[X]$ and variance $\mathrm{var}[X]$, whereas the first and second moments are its mean $\mathbb{E}[X]$ and $\mathbb{E}[X^2]$.} For this reason, $A(\vec{\theta})$ is sometimes called a \textbf{cumulant function}. We will prove this for a 1-parameter distribution; this can be generalized to a $K$-parameter distribution in a straightforward way. For the first derivative we have

For the second derivative we have
\begin{align}
\dfrac{\mathrm{d} A}{\mathrm{d} \theta} & = \dfrac{\mathrm{d}}{\mathrm{d} \theta}\left\{\log\int\exp\left[\theta\phi(x)\right]h(x)\mathrm{d}x\right\} \nonumber \\
    & = \dfrac{\frac{\mathrm{d}}{\mathrm{d} \theta}\int\exp\left[\theta\phi(x)\right]h(x)\mathrm{d}x}{\int\exp\left[\theta\phi(x)\right]h(x)\mathrm{d}x} \nonumber \\
	& = \dfrac{\int\phi(x)exp\left[\theta\phi(x)\right]h(x)\mathrm{d}x}{\exp(A(\theta))} \nonumber \\
	& = \int \phi(x)\exp\left[\theta\phi(x)-A(\theta)\right]h(x)\mathrm{d}x \nonumber \\
	& = \int \phi(x)p(x)\mathrm{d}x=\mathbb{E}[\phi(x)]
\end{align}

For the second derivative we have
\begin{align}
\dfrac{\mathrm{d}^2 A}{\mathrm{d} \theta^2} & = \int \phi(x)\exp\left[\theta\phi(x)-A(\theta)\right]h(x)\left[\phi(x)-A'(\theta)\right]\mathrm{d}x \nonumber \\
    & = \int \phi(x)p(x)\left[\phi(x)-A'(\theta)\right]\mathrm{d}x \nonumber \\
	& = \int \phi^2(x)p(x)\mathrm{d}x-A'(\theta)\int \phi(x)p(x)\mathrm{d}x \nonumber \\
	& = \mathbb{E}[\phi^2(x)]-\mathbb{E}[\phi(x)]^2=\mathrm{var}[\phi(x)]
\end{align}

In the multivariate case, we have that
\begin{equation}
\dfrac{\partial^2 A}{\partial \theta_i \partial \theta_j}=\mathbb{E}[\phi_i(x)\phi_j(x)]-\mathbb{E}[\phi_i(x)]\mathbb{E}[\phi_j(x)]
\end{equation}
and hence
\begin{equation}
\nabla^2A(\vec{\theta}) = \mathrm{cov}[\phi(\vec{x})]
\end{equation}

Since the covariance is positive definite, we see that $A(\vec{\theta})$ is a convex function (see Section \ref{sec:Convexity}).

\subsection{MLE for the exponential family}
The likelihood of an exponential family model has the form
\begin{equation}
p(\mathcal{D}|\vec{\theta})=\left[\prod\limits_{i=1}^N h(\vec{x}_i)\right]g(\vec{\theta})^N\exp\left[\vec{\theta}^T\left(\sum\limits_{i=1}^N \phi(\vec{x}_i)\right)\right]
\end{equation}

We see that the sufficient statistics are $N$ and
\begin{equation}
\phi(\mathcal{D})=\sum\limits_{i=1}^N \phi(\vec{x}_i)=(\sum\limits_{i=1}^N \phi_1(\vec{x}_i),\cdots,\sum\limits_{i=1}^N \phi_K(\vec{x}_i))
\end{equation}

The \textbf{Pitman-Koopman-Darmois theorem} states that, under certain regularity conditions, the exponential family is the only family of distributions with finite sufficient statistics. (Here, finite means of a size independent of the size of the data set.)

One of the conditions required in this theorem is that the support of the distribution not be dependent on the parameter.

\subsection{Bayes for the exponential family}
\label{sec:Bayes-for-the-exponential-family}
TODO

\subsubsection{Likelihood}

\subsection{Maximum entropy derivation of the exponential family *}
\label{sec:Maximum-entropy-derivation-of-the-exponential-family}

\section{Generalized linear models (GLMs)}
\label{sec:GLMs}
Linear and logistic regression are examples of \textbf{generalized linear models}, or \textbf{GLM}s (McCullagh and Nelder 1989). These are models in which the output density is in the exponential family (Section \ref{sec:exponential-family}), and in which the mean parameters are a linear combination of the inputs, passed through a possibly nonlinear function, such as the logistic function. We describe GLMs in more detail below. We focus on scalar outputs for notational simplicity. (This excludes multinomial logistic regression, but this is just to simplify the presentation.)

\subsection{Basics}

\section{Probit regression}

\section{Multi-task learning}


\chapter{isdet} \label{isdet}

\section{Introduction}

The \texttt{isdet} command checks whether the specified LPE is deterministic.
The command may yield false negatives.

\section{Formal background}

\subsection{Summand determinism}

Consider two summands, $s_\alpha$ and $s_\beta$, and reference their elements conform \ref{summandelements}.

Summands $s_\alpha$ and $s_\beta$ are said to be \emph{deterministic} if one of these conditions holds:

\begin{itemize}
\item $s_\alpha$ and $s_\beta$ are the same summand; that is, $s_\alpha = s_\beta$.

\item $s_\alpha$ and $s_\beta$ communicate over different channels:
\begin{align*}
C_\alpha \neq C_\beta \land (C_\alpha \in \{\istep{}, \cistep{}\} \not\leftrightarrow C_\beta \in \{\istep{}, \cistep{}\})
\end{align*}

\item $s_\alpha$ and $s_\beta$ communicate with different numbers of communication variables; that is, $m_\alpha \neq m_\beta$.

\item $s_\alpha$ and $s_\beta$ are never simultaneously enabled, or $s_\alpha$ and $s_\beta$ always lead to the same next state.
In order to determine this, check if
\begin{align*}
g_\alpha[X_\beta] \land g_\beta \rightarrow \bigwedge\limits_{j=1}^{k} v_\alpha(p_j)[X_\beta] = v_\beta(p_j)
\end{align*}

is a tautology, where $X_\beta$ is defined as
\begin{align*}
X_{\beta} &= [x_\alpha(j) \rightarrow x_\beta(j) \;|\; 1 \leq j \leq \text{min}(m_\alpha, m_\beta)]
\end{align*}
\end{itemize}

Note that this approach is \emph{not} guaranteed to correctly recognize that two summands are deterministic (false negatives are tolerated)!

\section{Algorithm}

The algorithm invoked by the \texttt{isdet} command checks for all pairs of different summands whether the first summand is deterministic with the second summand (see the previous section).
The algorithm yields \textbf{true} if and only if this is the case for all summand pairs.

\section{Example}

Consider the following LPE:

\begin{lstlisting}
//Process definition:
PROCDEF example[A :: Int](x, y :: Int)
  = A ? i [[x==0 /\ i>=0 /\ i<=5]] >-> example[A](1, 0)
  + A ? i [[y==0 /\ i>=5 /\ i<=9]] >-> example[A](0, 1)
  ;

//Initialization:
example[A](0, 1);
\end{lstlisting}

Consider the two summands, calling the first $s_1$ and the second $s_2$.

The first three conditions for detecting determinism are false.

For the fourth condition, the antecedent is
\begin{align*}
g_1[X_2] \land g_2 &\Leftrightarrow (\texttt{x} = 0 \land \texttt{i} \geq 0 \land \texttt{i} \leq 5)[\texttt{i} \rightarrow \texttt{i}] \land (\texttt{y} = 0 \land \texttt{i} \geq 5 \land \texttt{i} \leq 9) \\
&\Leftrightarrow \texttt{x} = 0 \land \texttt{y} = 0 \land \texttt{i} = 5
\end{align*}

Given the antecedent, the conclusion must hold.
This is not the case:
\begin{align*}
v_1(\texttt{x})[X_2] = 1[\texttt{i} \rightarrow \texttt{i}] = 1 \neq 0 = v_2(\texttt{x}) \\
v_1(\texttt{y})[X_2] = 0[\texttt{i} \rightarrow \texttt{i}] = 0 \neq 1 = v_2(\texttt{y}) \\
\end{align*}

$s_1$ is therefore \emph{not} deterministic with $s_2$.

Changing the LPE to

\begin{lstlisting}
//Process definition:
PROCDEF example[A :: Int](x, y :: Int)
  = A ? i [[x==0 /\ i>=0 /\ i<=4]] >-> example[A](1, 0)
  + A ? i [[y==0 /\ i>=5 /\ i<=9]] >-> example[A](0, 1)
  ;

//Initialization:
example[A](0, 1);
\end{lstlisting}

will result in an antecedent that is false; therefore, $s_1$ is deterministic with $s_2$.