\chapter{Affine schemes: the Zariski topology}
\label{ch:spec_zariski}
Now that we understand sheaves well,
we can define an affine scheme.
It will be a ringed space, so we need to define
\begin{itemize}
	\ii The set of points,
	\ii The topology on it, and
	\ii The structure sheaf on it.
\end{itemize}
In this chapter, we handle the first two parts;
\Cref{ch:spec_sheaf} does the last one.

Quick note: \Cref{ch:spec_examples}
contains a long list of examples of affine schemes.
So if something written in this chapter is not making sense,
one thing worth trying is skimming through the next chapter
to see if any of the examples there are more helpful.

\section{Some more advertising}
Let me describe what the construction of $\Spec A$ is going to do.

In the case of $\Aff^n$, we used $\CC^n$ as the set of points
and $\CC[x_1, \dots, x_n]$ as the ring of functions
but then remarked that the set of points
of $\CC^n$ corresponded to the maximal ideals of $\CC[x_1, \dots, x_n]$.
In an \emph{affine scheme}, we will take an \emph{arbitrary} ring $A$,
and generate the entire structure from just $A$ itself.
The final result is called $\Spec A$, the \vocab{spectrum} of $A$.
The affine varieties $\VV(I)$ we met earlier will just be
$\CC[x_1, \dots, x_n] / I$, but now we will be able to take
\emph{any} ideal $I$, thus finally completing the table at the end
of the ``affine variety'' chapter.

The construction of the affine scheme in this way
will have three big generalizations:
\begin{enumerate}
	\ii We no longer have to work over an algebraically
	closed field $\CC$, or even a field at all.
	This will be the most painless generalization:
	you won't have to adjust your current picture much for this to work.

	\ii We allow non-radical ideals:
	$\Spec \CC[x] / (x^2)$ will be the double point
	we sought for so long.
	This will let us formalize the notion of a ``fat'' or ``fuzzy'' point.

	\ii Our affine schemes will have so-called \emph{non-closed points}:
	points which you can visualize as floating around,
	somewhere in the space but nowhere in particular.
	(They'll correspond to prime non-maximal ideals.)
	These will take the longest to get used to,
	but as we progress we will begin to see that these non-closed points
	actually make life \emph{easier},
	once you get a sense of what they look like.
\end{enumerate}

\section{The set of points}
\prototype{$\Spec \CC[x_1, \dots, x_n] / I$.}

First surprise, for a ring $A$:
\begin{definition}
	The set $\Spec A$ is defined as the set of prime ideals of $A$.
\end{definition}

This might be a little surprising, since we might have guessed
that $\Spec A$ should just have the maximal ideals.
What do the remaining ideals correspond to?
The answer is that they will be so-called \emph{non-closed points}
or \emph{generic points} which are ``somewhere'' in the space,
but nowhere in particular.
(The name ``non-closed'' is explained next chapter.)

\begin{remark}
	As usual $A$ itself is not a prime ideal, but $(0)$
	is prime if and only if $A$ is an integral domain.
\end{remark}

\begin{example}
	[Examples of spectrums]
	\listhack
	\begin{enumerate}[(a)]
		\ii $\Spec \CC[x]$ consists of a point $(x-a)$ for every $a \in \CC$,
		which correspond to what we geometrically think of as $\Aff^1$.
		In additionally consists of a point $(0)$,
		which we think of as a ``non-closed point'', nowhere in particular.

		\ii $\Spec \CC[x,y]$ consists of points $(x-a,y-b)$
		(which are the maximal ideals) as well as $(0)$ again,
		a non-closed point that is thought of as ``somewhere in $\CC^2$,
		but nowhere in particular''.
		It also consists of non-closed points corresponding to irreducible
		polynomials $f(x,y)$, for example $(y-x^2)$,
		which is a ``generic point on the parabola''.

		\ii If $k$ is a field, $\Spec k$ is a single point,
		since the only maximal ideal of $k$ is $(0)$.
	\end{enumerate}
\end{example}

\begin{example}
	[Complex affine varieties]
	Let $I \subseteq \CC[x_1, \dots, x_n]$ be an ideal.
	By \Cref{prop:prime_quotient},
	the set \[ \Spec \CC[x_1, \dots, x_n] /I \]
	consists of those prime ideals of $\CC[x_1, \dots, x_n]$
	which contain $I$: in other words, it has a
	point for every closed irreducible subvariety of $\VV(I)$.
	So in addition to the ``geometric points''
	(corresponding to the maximal ideals $(x_1-a_1, \dots, x_n-a_n)$
	we have non-closed points along each of the varieties).
\end{example}

The non-closed points are the ones you are not used to:
there is one for each non-maximal prime ideal
(visualized as ``irreducible subvariety'').
I like to visualize them in my head like a fly:
you can hear it, so you know it is floating \emph{somewhere} in the room,
but as it always moving, you never know exactly where.
So the generic point of $\Spec \CC[x,y]$ corresponding to the prime
ideal $(0)$ is floating everywhere in the plane,
the one for the ideal $(y-x^2)$ floats along the parabola, etc.
\begin{center}
	\includegraphics[scale=0.4]{media/calvin-hobbes-fly.png} \\
	\footnotesize Image from \cite{img:calvin_hobbes_fly}.
\end{center}

\begin{example}
	[More examples of spectrums]
	\listhack
	\begin{enumerate}[(a)]
		\ii $\Spec \ZZ$ consists of a point for every prime $p$,
		plus a generic point that is somewhere, but no where in particular.

		\ii $\Spec \CC[x] / (x^2)$ has only $(x)$ as a prime ideal.
		The ideal $(0)$ is not prime since $0 = x \cdot x$.
		Thus as a \emph{topological space},
		$\Spec \CC[x] / (x^2)$ is a single point.

		\ii $\Spec \Zc{60}$ consists of three points.
		What are they?
	\end{enumerate}
\end{example}

\section{The Zariski topology on the spectrum}
\prototype{Still $\Spec \CC[x_1, \dots, x_n] / I$.}

Now, we endow a topology on $\Spec A$.
Since the points on $\Spec A$ are the prime ideals, we continue
the analogy by thinking of the points $f$ as functions on $\Spec A$. That is:
\begin{definition}
	Let $f \in A$ and $\kp \in \Spec A$.
	Then the \vocab{value} of $f$ at $\kp$ is defined to be $f \pmod{\kp}$,
	an element of $A/\kp$.
	We denote it $f(\kp)$.
\end{definition}
\begin{example}
	[Vanishing locii in $\Aff^n$]
	Suppose $A = \CC[x_1, \dots, x_n]$,
	and $\km = (x_1-a_1, x_2-a_2, \dots, x_n-a_n)$ is a maximal ideal of $A$.
	Then for a polynomial $f \in \CC$,
	\[ f \pmod \km = f(a_1, \dots, a_n) \]
	with the identification that $\CC/\km \cong \CC$.
\end{example}
\begin{example}
	[Functions on $\Spec \ZZ$]
	Consider $A = \Spec \ZZ$.
	Then $2019$ is a function on $A$.
	Its value at the point $(5)$ is $4 \pmod 5$;
	its value at the point $(7)$ is $3 \pmod 7$.
\end{example}

Indeed if you replace $A$ with $\CC[x_1, \dots, x_n]$
and $\Spec A$ with $\Aff^n$ in everything that follows,
then everything will become quite familiar.

\begin{definition}
	Let $f \in A$. We define the \vocab{vanishing locus} of $f$ to be
	\[ \VV(f) = \left\{ \kp \in \Spec A \mid f(\kp) = 0 \right\}
		= \left\{ \kp \in \Spec A \mid f \in \kp \right\}. \]
	More generally, just as in the affine case,
	we define the vanishing locus for an ideal $I$ as
	\begin{align*}
		\VV(I) &= \left\{ \kp \in \Spec A \mid f(\kp)=0 \; \forall f \in I \right\} \\
		&= \left\{ \kp \in \Spec A \mid f \in \kp \; \forall f \in I \right\} \\
		&= \left\{ \kp \in \Spec A \mid I \subseteq \kp \right\}.
	\end{align*}
	Finally, we define the \vocab{Zariski topology} on $\Spec A$
	by declaring that the sets of the form $\VV(I)$ are closed.
\end{definition}

We now define a few useful topological notions:
\begin{definition}
	Let $X$ be a topological space.
	A point $p \in X$ is a \vocab{closed point}
	if the set $\{p\}$ is closed.
\end{definition}
\begin{ques}
	[Mandatory]
	Show that a point (i.e.\ prime ideal)
	$\km \in \Spec A$ is a closed point
	if and only if $\km$ is a maximal ideal.
\end{ques}
Recall also in \Cref{def:closure} we denote by $\ol S$
the closure of a set $S$ (i.e.\ the smallest closed set containing $S$);
so you can think of a closed point $p$ also
as one whose closure is just $\{p\}$,
while with a generic point
Therefore the Zariski topology lets us refer back to the old ``geometric''
as just the closed points.

\begin{example}
	[Non-closed points, continued]
	Let $A = \CC[x,y]$ and let $\kp = (y-x^2) \in \Spec A$;
	this is the ``generic point'' on a parabola.
	It is not closed, but we can compute its closure:
	\[
		\ol{\{\kp\}}
		= \VV(\kp) = \left\{ \kq \in \Spec A \mid \kq \supseteq \kp \right\}.
	\]
	This closure contains the point $\kp$ as well
	as several maximal ideals $\kq$, such as $(x-2,y-4)$ and $(x-3,y-9)$.
	In other words, the closure of the ``generic point'' of the parabola
	is literally the set of all points that are actually on the parabola
	(including generic points).

	That means the way to picture $\kp$ is a point that
	is floating ``somewhere on the parabola'', but nowhere in particular.
	It makes sense then that if we take the closure,
	we get the entire parabola,
	since $\kp$ ``could have been'' any of those points.
\end{example}

\begin{center}
\begin{asy}
	graph.xaxis();
	graph.yaxis();
	real f(real x) { return x*x; }
	graph.xaxis("$x$");
	graph.yaxis("$y$");
	draw(graph(f,-2,2,operator ..), red+dotted, Arrows(TeXHead));
	dot("$(y-x^2)$", (1.4, f(1.4)), dir(-45), red);
	dot("$(x+1,y-1)$", (-1,1), dir(225), blue);
\end{asy}
\end{center}

\begin{example}
	[The generic point of the $y$-axis isn't on the $x$-axis]
	Let $A = \CC[x,y]$ again.
	Consider $\VV(y)$, which is the $x$-axis of $\Spec A$.
	Then consider $\kp = (x)$, which is the generic point on the $y$-axis.
	Observe that
	\[ \kp \notin \VV(y). \]
	The geometric way of saying this is that a \emph{generic point}
	on the $y$-axis does not lie on the $x$-axis.
\end{example}

We now also introduce one more word:
\begin{definition}
	A topological space $X$ is \vocab{irreducible}
	if either of the following two conditions hold:
	\begin{itemize}
		\ii The space $X$ cannot be written as the
		union of two proper closed subsets.
		\ii Any two nonempty open sets of $X$ intersect.
	\end{itemize}
	A subset $Z$ of $X$ (usually closed) is irreducible
	if it is irreducible as a subspace.
\end{definition}
\begin{exercise}
	Show that the two conditions above are indeed equivalent.
	Also, show that the closure of a point is always irreducible.
\end{exercise}

This is the analog of the ``irreducible''
we defined for affine varieties,
but it is now a topological definition,
although in practice this definition is only
useful for spaces with the Zariski topology.
Indeed, if any two nonempty open sets intersect
(and there is more than one point),
the space is certainly not Hausdorff!
As with our old affine varieties,
the intuition is that $\VV(xy)$ (the union of two lines)
should not be irreducible.

\begin{example}
	[Reducible and irreducible spaces]
	\listhack
	\begin{enumerate}[(a)]
		\ii The closed set $\VV(xy) = \VV(x) \cup \VV(y)$ is reducible.
		\ii The entire plane $\Spec \CC[x,y]$ is irreducible.
		There is actually a simple (but counter-intuitive,
		since you are just getting used to generic points)
		reason why this is true:
		the generic point $(0)$ is in \emph{every} open set,
		ergo, any two open sets intersect.
	\end{enumerate}
\end{example}

So actually, the generic points
kind of let us cheat our way through the following bit:
\begin{proposition}
	[Spectrums of integral domains are irreducible]
	If $A$ is an integral domain,
	then $\Spec A$ is irreducible.
\end{proposition}
\begin{proof}
	Just note $(0)$ is a prime ideal,
	and in every open set.
\end{proof}
You should compare this with our old classical result that
$\CC[x_1, \dots, x_n]/I$
was irreducible as an affine variety exactly when $I$ was prime.
This time, the generic point actually takes care
of the work for us:
the fact that it is \emph{allowed} to float
anywhere in the plane lets us capture the idea that
$\Aff^2$ should be irreducible
without having to expend any additional effort.
\begin{remark}
	Surprisingly, the converse of this proposition is false:
	we have seen $\Spec \CC[x]/(x^2)$ has only one point,
	so is certainly irreducible.
	But $A = \CC[x]/(x^2)$ is not an integral domain.
	So this is one weird-ness introduced by allowing ``non-radical'' behavior.
\end{remark}

At this point you might notice something:
\begin{theorem}
	[Points are in bijection with irreducible closed sets]
	Consider $X = \Spec A$.
	For every irreducible closed set $Z$,
	there is exactly one point $\kp$ such that $Z = \ol{\{\kp\}}$.
	(In particular points of $X$ are in bijection
	with closed subsets of $X$.)
\end{theorem}
\begin{proof}
	[Idea of proof]
	The point $\kp$ corresponds to the closed set $\VV(\kp)$,
	which one can show is irreducible.
	% Maybe I really should prove this here,
	% but I don't really want to draw to much attention to radicals yet;
	% there's too much going on already.
\end{proof}
This gives you a better way to draw non-closed points:
they are the generic points lying along any irreducible closed set
(consisting of more than just one point).

At this point\footnote{Pun not intended},
I may as well give you the real definition of generic point.
\begin{definition}
	Given a topological space $X$,
	a \vocab{generic point} $\eta$
	is a point whose closure is the entire space $X$.
\end{definition}
So for us, when $A$ is an integral domain,
$\Spec A$ has generic point $(0)$.
\begin{abuse}
	Very careful readers might note I am being a little careless
	with referring to $(y-x^2)$ as
	``the generic point along the parabola''
	in $\Spec \CC[x,y]$.
	What's happening is that $\VV(y-x^2)$ is a closed set,
	and as a topological subspace, it has generic point $(y-x^2)$.
\end{abuse}

\section{On radicals}
Back when we studied classical algebraic geometry in $\CC^n$,
we saw Hilbert's Nullstellensatz (\Cref{thm:hilbert_null}) show up
to give bijections between radical ideals and affine varieties;
we omitted the proof, because it was nontrivial.

However, for a \emph{scheme}, where the points \emph{are} prime ideals
(rather than tuples in $\CC^n$),
the corresponding results will actually be \emph{easy}:
even in the case where $A = \CC[x_1, \dots, x_n]$,
the addition of prime ideals (instead of just maximal ideals)
will actually \emph{simplify} the proof,
because radicals play well with prime ideals.

We still have the following result.
\begin{proposition}
	[$\VV(\sqrt I) = \VV(I)$]
	For any ideal $I$ of a ring $A$
	we have $\VV(\sqrt I) = \VV(I)$.
\end{proposition}
\begin{proof}
	We have $\sqrt I \supseteq I$.
	Hence automatically $\VV(\sqrt I) \subseteq \VV(I)$.

	Conversely, if $\kp \in \VV(I)$, then $I \subseteq \kp$,
	so $\sqrt I \subseteq \sqrt{\kp} = \kp$
	(by \Cref{prop:radical}).
\end{proof}

We hinted the key result in an earlier remark,
and we now prove it.
\begin{theorem}
	[Radical is intersection of primes]
	\label{thm:radical_intersect_prime}
	Let $I$ be an ideal of a ring $A$.
	Then \[ \sqrt I = \bigcap_{\kp \supseteq I} \kp. \]
\end{theorem}
\begin{proof}
	This is a famous statement from commutative algebra,
	and we prove it here only for completeness.
	It is ``doing most of the work''.

	Note that if $I \subseteq \kp$,
	then $\sqrt I \subseteq \sqrt{\kp} = \kp$;
	thus $\sqrt I \subseteq \bigcap_{\kp \supseteq I} \kp$.

	Conversely, suppose $x \notin \sqrt I$,
	meaning $1, x, x^2, x^3, \dots \notin I$.
	Then, consider the localization $A[1/x]$.
	Like any ring, it has some maximal ideal (Krull's theorem).
	This means our usual bijection between prime ideals of $A[1/x]$
	gives some prime ideal $\kp$ of $A$ containing $I$ but not containing $x$.
	Thus $x \notin \bigcap_{\kp \supseteq I} \kp$,
	as desired.
\end{proof}
\begin{remark}
	[A variant of Krull's theorem]
	The longer direction of this proof is essentially
	saying that for any $x \in A$,
	there is a maximal ideal of $A$ not containing $x$.
	The ``short'' proof is to use Krull's theorem on $A[1/x]$ as above,
	but one can also still prove it directly using Zorn's lemma
	(by copying the proof of the original Krull's theorem).
\end{remark}

\begin{example}
	[$\sqrt{(2016)} = (42)$ in $\ZZ$]
	In the ring $\ZZ$, we see that $\sqrt{(2016)} = (42)$,
	since the distinct primes containing $(2016)$
	are $(2)$, $(3)$, $(7)$.
\end{example}

Geometrically, this gives us a good way to describe $\sqrt I$:
it is the \emph{set of all functions vanishing on all of $\VV(I)$}.
Indeed, we may write
\[ \sqrt I = \bigcap_{\kp \supset I} \kp
	= \bigcap_{\kp \in \VV(I)} \kp
	= \bigcap_{\kp \in \VV(I)} \left\{ f \in A \mid f(\kp) = 0 \right\}. \]

We can now state:
\begin{theorem}
	[Radical ideals correspond to closed sets]
	Let $I$ and $J$ be ideals of $A$,
	and considering the space $\Spec A$.
	Then
	\[ \VV(I) = \VV(J) \iff \sqrt I = \sqrt J. \]
	In particular, radical ideals exactly
	correspond to closed subsets of $\Spec A$.
\end{theorem}
\begin{proof}
	If $\VV(I) = \VV(J)$,
	then $\sqrt I = \bigcap_{\kp \in \VV(I)} \kp =
	\bigcap_{\kp \in \VV(J)} \kp = \sqrt J$ as needed.

	Conversely, suppose $\sqrt I = \sqrt J$.
	Then $\VV(I) = \VV(\sqrt I) = \VV(\sqrt J) = \VV(J)$.
\end{proof}

Compare this to the theorem we had earlier
that the \emph{irreducible} closed subsets correspond to \emph{prime} ideals!

\section{\problemhead}
As \Cref{ch:spec_examples} contains many
examples of affine schemes to train your intuition,
it's possibly worth reading even before attempting these problems,
even though there will be some parts that won't make sense yet.

\begin{problem}
	[{$\Spec \QQ[x]$}]
	Describe the points and topology of $\Spec \QQ[x]$.
	\begin{hint}
		Galois conjugates.
	\end{hint}
\end{problem}

\begin{problem}
	[Product rings]
	Describe the points and topology of $\Spec A \times B$
	in terms of $\Spec A$ and $\Spec B$.
\end{problem}

\begin{problem}
	Show that if $A$ is \emph{not} an integral domain,
	then $\Spec A$ is not irreducible.
\end{problem}

\endinput

\begin{problem}
	[From Andrew Critch]
	\gim
	Let $A$ be a Noetherian ring.
	Show that $A$ is an integral domain if and only if it has no idempotents,
	and $A_\kp$ is an integral domain for every prime $\kp$.
	\begin{hint}
		Show that if $\Spec A$ is connected and its stalks are irreducible,
		then $\Spec A$ is itself irreducible.
		Consider nilradical $N = \sqrt{(0)}$.
	\end{hint}
	\begin{sol}
		This is the proposition on the second page of
		\url{http://www.acritch.com/media/math/Stalk-local_detection_of_irreducibility.pdf}
	\end{sol}
\end{problem}


\documentclass{article}
\usepackage[T1,T2A]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage{amsmath}
\usepackage{siunitx}
\begin{document}
\title{Flight Model}
\author{Vasili V}
\renewcommand{\today}{November 22, 2018}
\maketitle
\section{Lift}
The buoyant force on a balloon of diameter $d$ with density of content $\rho_{in}$ and environmental density $\rho_{out}$:
\begin{equation}
\begin{aligned}
F_b &= (\rho_{out} - \rho_{in})g\frac{\pi{d}^3}{6},
\end{aligned}
\end{equation}
where content is Helium with density (at STP):
\begin{equation}
\begin{aligned}
\rho_{in} &= 0.1786 && [\si{kg.m^{-3}}].
\end{aligned}
\end{equation}
The environment is standard atmosphere taken from \cite{NASA76} at zero altitude:
\begin{equation}
\begin{aligned}
\rho_{out} &= 1.225 && [\si{kg.m^{-3}}].
\end{aligned}
\end{equation}
under following conditions:
\begin{equation}
\begin{aligned}
p &= \num{1.01325e5} && [\si{kg.m^{-3}}]. \\
T &= \num{288.15} && [\si{\kelvin}].
\end{aligned}
\end{equation}
Under the conditions Helium density is:
\begin{equation}
\begin{aligned}
\rho_{in} &= 0.1786\cdot\frac{\num{1.01325e5}}{\num{1e5}}\cdot\frac{\num{273.15}}{\num{288.15}} = 0.1715 && [\si{kg.m^{-3}}].
\end{aligned}
\end{equation}
Let's simplify equation (1):
\begin{equation}
\begin{aligned}
F_b &= k_0d^3,
\end{aligned}
\end{equation}
where $k_0$ is:
\begin{equation}
\begin{aligned}
k_0 &= \frac{\pi}{6}\cdot\num{9.81}\cdot(\num{1.225}-\num{0.1715}) = 5.411. && [\si{\newton.m^{-3}}]
\end{aligned}
\end{equation}
For example balloons with diameters $\num{0.36}$, $\num{0.70}$ and $\num{1.15}$ $\si{m}$ get following lifts:
\begin{equation}
\begin{aligned}
F_s &= \num{5.411}\cdot\num{0.36}^3 = 0.2525 [\si{\newton}] = 25.74 [\si{gf}]; \\
F_m &= \num{5.411}\cdot\num{0.70}^3 = 1.856 [\si{\newton}] = 189.3 [\si{gf}]; \\
F_b &= \num{5.411}\cdot\num{1.15}^3 = 8.230 [\si{\newton}] = 839.2 [\si{gf}].
\end{aligned}
\end{equation}
These balloons have following volumes:
\begin{equation}
\begin{aligned}
V_s &= 24.43 && [\si{L}]; \\
V_m &= 179.6 && [\si{L}]; \\
V_b &= 796.3 && [\si{L}],
\end{aligned}
\end{equation}
which is approximately $\frac{1}{57}$, $\frac{1}{7}$ and $\frac{2}{3}$ of 10 L tank.
\begin{thebibliography}{9}
\bibitem{NASA76}
  NOAA, NASA, USAF,
  US. Standard Atmosphere,
  1976
\end{thebibliography}
\end{document}


\subsection{Proof of Theorem \ref{thm:finite}}
First, for any $\epsilon > 0$,
$$\begin{aligned}
\Pr[U_S(F) > \epsilon] =& \Pr[\exists f\in F, A_S(f) - A_D(f) > \epsilon] \\
\leq & \sum_{f\in F} \Pr[A_S(f) - A_D(f) > \epsilon] \\
=& \sum_{f\in F} \Pr[\frac{1}{k} \sum_{i=1}^k f(s_i) - A_D(f) > \epsilon]. \\
\end{aligned}$$
By Hoeffding's inequality \cite{BBL04} we have
$$\Pr[\frac{1}{k} \sum_{i=1}^k f(s_i) - A_D(f) > \epsilon] \leq \exp\left(-\frac{2k\epsilon^2}{m^2}\right).$$
Hence
$$\Pr[U_S(F) > \epsilon] \leq |F| \exp\left(-\frac{2k\epsilon^2}{m^2}\right).$$
Similarly one can show that
$$\Pr[\sup_{f\in F} [ A_D(f) - A_S(f) ] > \epsilon] \leq |F| \exp\left(-\frac{2k\epsilon^2}{m^2}\right).$$
Thus putting the two cases together,
$$\Pr[\sup_{f\in F} | A_S(f) - A_D(f) | > \epsilon] \leq 2|F| \exp\left(-\frac{2k\epsilon^2}{m^2}\right).$$
Equivalently for any $\delta>0$, with probability at least $1-\delta$,
$$\sup_{f\in F} | A_S(f) - A_D(f) | \leq m\sqrt{\frac{\log(2|F|) + \log(1/\delta)}{2k}}.$$
We have obtained the desired upper bound in Theorem \ref{thm:finite}.

\subsection{Proof of Theorem \ref{thm:main}}
In this section, we show our main result (Theorem \ref{thm:main}).
We start from the definition of self bounding function \cite{Oneto13}.
\begin{definition}
Let $s_1$, $s_2$, $\cdots$, $s_k$ be independent random variables taking values from a set $D$. A function $f: D^k \to [0, +\infty]$ is a self bounding function if there exists a constant $c$ and a function $g: D^{k-1}\to \mathbb{R}$ such that for any $s_1$, $\cdots$, $s_{j-1}$, $s_{j+1}$, $\cdots$, $s_k \in D$, the following conditions hold:
$$0 \leq f(s_1, \cdots, s_k) - g(s_1,\cdots, s_{j-1}, s_{j+1}, \cdots, s_k) \leq c,$$
$$\sum_{j=1}^k [f(s_1, \cdots, s_k) - g(s_1,\cdots, s_{j-1}, s_{j+1}, \cdots, s_k)] \leq f(s_1, \cdots, s_k).$$
\end{definition}

The following concentration inequality can be achieved for self bounding functions \cite{BLM99}.

\begin{lemma}
\cite{BLM99} If a function $Z = f(s_1,\cdots,s_k)$ is a self bounding function with constant $c$, then for $t \leq \E Z$,
$$\Pr[\E Z - Z \geq t] \leq \exp\left(-\frac{t^2}{2c\E Z}\right).$$
For $t > \E Z$, the left probability is zero trivially.
Here we take randomness over $s_1$, $s_2$, $\cdots$, $s_k$.
\end{lemma}

By using the above lemma, we can show a similar inequality for Rademacher average.

\begin{lemma}
\label{lem1}
$$\Pr[\E R_S(F) \geq R_S(F) + t] \leq \exp\left(-\frac{kt^2}{4m\E R_S(F)}\right),$$
where $\E$ takes randomness over the samplings $s_1$, $s_2$, $\cdots$, $s_k$.
\end{lemma}
\begin{proof}
It suffices to show that $R_S(F)$ is a self bounding function with constant $c=2m/k$.
Define
$$Z = R_S(F) = \E_\sigma \sup_{f\in F} \left[\frac{2}{k}\sum_{i=1}^k \sigma_i f(s_i) \right],$$
$$G_j = \E_\sigma \sup_{f\in F} \left[\frac{2}{k}\sum_{i\not= j} \sigma_i f(s_i) \right].$$
It is clear that $Z$ is non-negative:
$$Z \geq \sup_{f\in F} \left[ \E_\sigma \frac{2}{k}\sum_{i=1}^k \sigma_i f(s_i) \right]=0.$$
Also it is clear that $Z \geq G_j$ for each $j$: suppose $\tilde{f}$ achieves the supreme of $G_j$. Then
$$\begin{aligned}
G_j &= \E_\sigma \left[\frac{2}{k}\sum_{i=1}^k \sigma_i \tilde{f}(s_i) - \frac{2}{k} \sigma_j \tilde{f}(s_j)\right] \\
&= \E_\sigma \left[\frac{2}{k}\sum_{i=1}^k \sigma_i \tilde{f}(s_i)\right] - \E_\sigma \left[ \frac{2}{k} \sigma_j \tilde{f}(s_j)\right] \\
&= \E_\sigma \left[\frac{2}{k}\sum_{i=1}^k \sigma_i \tilde{f}(s_i) \right]  \leq Z.
\end{aligned}$$
Next we show $Z- G_j \leq 2m/k =c$:
$$\begin{aligned}
G_j &= \E_\sigma \sup_{f\in F} \left[ \frac{2}{k} \sum_{i=1}^k \sigma_i f(s_i) - \frac{2}{k} \sigma_j f(s_j) \right] \\
&\geq \E_\sigma \sup_{f\in F} \left[ \frac{2}{k} \sum_{i=1}^k \sigma_i f(s_i) \right] - \E_\sigma \sup_{f\in F} \left[ \frac{2}{k} \sigma_j f(s_j) \right] \\
&\geq \E_\sigma \sup_{f\in F} \left[ \frac{2}{k} \sum_{i=1}^k \sigma_i f(s_i) \right] - \frac{2m}{k}.
\end{aligned}$$
Finally we need to verify $\sum_{j=1}^k Z-G_j \leq Z$:
$$\begin{aligned}
\sum_{j=1}^k G_j &= \E_\sigma \sum_{j=1}^k \sup_{f\in F} \left[ \frac{2}{k}\sum_{i\not= j} \sigma_i f(s_i) \right]\\
&\geq \E_\sigma \sup_{f\in F}\left[ \frac{2}{k} \sum_{j=1}^k \sum_{i\not= j} \sigma_i f(s_i) \right]\\
&=\frac{2(k-1)}{k} \E_\sigma \sup_{f\in F}\left[\sum_{j=1}^k \sigma_i f(s_i) \right] = (k-1)Z.
\end{aligned}$$
\end{proof}

We still need the following lemma on the relation between uniform deviation and Rademacher average.

\begin{lemma}
\label{lem2}
$$\E \sup_{f\in F} [ A_S(f) - A_D(f) ] \leq \E R_S(F),$$
$$\E \sup_{f\in F} [ A_D(f) - A_F(f) ] \leq \E R_S(F).$$
Here we take randomness over the $k$ samplings.
\end{lemma}
\begin{proof}
The proof idea is based on ghost samplings, i.e., independently draw another $k$ samples: $s_1'$, $\cdots$, $s_k'$, and then we have
$$A_D(f) = \frac{1}{m}\sum_{i=1}^m f(i) = \E \frac{1}{k} \sum_{j=1}^k f(s_j'),$$
where $\E$ takes randomness over the $k$ ghost samples.
Thus
$$\begin{aligned}
\E \sup_{f\in F} [ A_S(f) - A_D(f) ] &= \E \sup_{f\in F} \left[\frac{1}{k} \sum_{i=1}^k f(s_i) - \E \frac{1}{k} \sum_{j=1}^k f(s_j') \right]\\
&\leq \E \sup_{f\in F} \left[\frac{1}{k} \sum_{i=1}^k f(s_i) - \frac{1}{k} \sum_{j=1}^k f(s_j') \right].\\
\end{aligned}$$
Since all the samples $s$, $s'$ are independently identically distributed, flipping the sign of $f(s_i) - f(s_i')$ will not change the expected supreme, i.e.,
$$\E \sup_{f\in F} \left[\frac{1}{k} \sum_{i=1}^k f(s_i) - \frac{1}{k} \sum_{j=1}^k f(s_j') \right] = \E \sup_{f\in F} \frac{1}{k} \sum_{i=1}^k \left[\sigma_i (f(s_i) - f(s_i')) \right],$$
where $\sigma_i$ is uniformly distributed over $\{-1, 1\}$.
Since
$$\E \sup_{f\in F} \frac{1}{k} \sum_{i=1}^k \left[\sigma_i (f(s_i) - f(s_i')) \right] \leq 2 \E \sup_{f\in F} \frac{1}{k} \sum_{i=1}^k \sigma_i f(s_i)  = \E R_S(F),$$
we have shown the first inequality. The second inequality is analogous.
\end{proof}

We also need McDiarmid's inequality \cite{M89}.
\begin{lemma}
\cite{M89} Let $s_1$, $\cdots$, $s_k$ be independent random variables taking values from a set $D$. Suppose a function $h: D^k \to \mathbb{R}$ satisfies
$$\sup_{x_1,\cdots,x_k,x_i'\in D} |h(x_1,\cdots,x_k) - h(x_1,\cdots,x_{i-1},x_i',x_{i+1},\cdots,x_k)| \leq c_i$$
for some constants $c_i$ and every $1\leq i \leq k$. Then for any $t>0$, we have
$$\Pr[h(s_1,\cdots,s_k) - \E h(s_1,\cdots,s_k) \geq t] \leq \exp\left(-\frac{2t^2}{\sum_{i=1}^k c_i^2}\right).$$
\end{lemma}

By the above three lemmas, we can bound the difference between true average and sampled average as follows.
\begin{lemma}
$$\begin{aligned}
&\Pr\left[\sup_{f\in F} |A_D(f) - A_S(f)| \geq R_S(F) + t\right] \\
\leq& 4\exp\left(-\frac{2kt^2}{(m+\sqrt{8m\E R_S(F)})^2} \right).\end{aligned}$$
\end{lemma}
\begin{proof}
First by Lemma \ref{lem2},
$$\begin{aligned}
&\Pr\left[\sup_{f\in F} [A_D(f) - A_S(f)] \geq R_S(F) + t\right] \\
\leq& \Pr\left[\sup_{f\in F} [A_D(f) - A_S(f)] \geq \E\sup_{f\in F} [A_D(f) - A_S(f)] + at\right] \\
&+ \Pr\left[\E R_S(F) \geq R_S(F)+(1-a)t\right] \\
\end{aligned}$$
for any $a\in [0,1]$. Let
$$h(s_1,\cdots,s_k) = A_D(f) - A_S(f) = A_D(f) - \frac{1}{k}\sum_{i=1}^k f(s_i).$$
It is clear that
$$\begin{aligned}
&\sup_{x_1,\cdots,x_k,x_i'\in D} |h(x_1,\cdots,x_k) - h(x_1,\cdots,x_{i-1},x_i',x_{i+1},\cdots,x_k)| \\
=&\sup_{x_1,\cdots,x_k,x_i'\in D} \left|\frac{1}{k}\sum_{j=1, j\not=i}^k f(x_j)+\frac{1}{k}f(x_i') - \frac{1}{k}\sum_{i=1}^k f(x_i)\right| \\
=&\sup_{x_1,\cdots,x_k,x_i'\in D} \left|\frac{1}{k}f(x_i') - \frac{1}{k} f(x_i)\right| \leq \frac{m}{k}.
\end{aligned}$$
By McDiarmid's inequality,
$$\begin{aligned}
&\Pr\left[\sup_{f\in F} [A_D(f) - A_S(f)] \geq \E\sup_{f\in F} [A_D(f) - A_S(f)] + at\right] \\
\leq& \exp\left(-\frac{2a^2t^2}{\sum_{i=1}^k m^2/k^2}\right)
=\exp\left(-\frac{2ka^2t^2}{m^2}\right).
\end{aligned}$$

By Lemma \ref{lem1},
$$\Pr\left[\E R_S(F) \geq R_S(F)+(1-a)t\right] \leq \exp\left(-\frac{k(1-a)^2t^2}{4m\E R_S(F)}\right).$$
Let $a = 1/(1+\sqrt{8\E R_S(F) / m})$. Then putting everything together, we have
$$\begin{aligned}
&\Pr\left[\sup_{f\in F} [A_D(f) - A_S(f)] \geq R_S(F) + t\right] \\
\leq& 2\exp\left(-\frac{2kt^2}{(m+\sqrt{8m\E R_S(F)})^2} \right).\end{aligned}$$
Similarly one can show
$$\begin{aligned}
&\Pr\left[\sup_{f\in F} [A_S(f) - A_D(f)] \geq R_S(F) + t\right] \\
\leq& 2\exp\left(-\frac{2kt^2}{(m+\sqrt{8m\E R_S(F)})^2} \right).\end{aligned}$$
Thus we have the inequality as desired.
\end{proof}
By the above lemma we have the following important corollary.
\begin{corollary}
With probability at least $1-\delta$, we have
$$\sup_{f\in F}|A_S(f) - A_D(f)| \leq R_S(F) + (m+\sqrt{8m \E R_S(F)})\sqrt{\frac{\log \frac{4}{\delta}}{2k}}.$$
\end{corollary}

We still need to upper bound $\E R_S(F)$. By Lemma \ref{lem1}, with probability at least $1-\delta$,
$$\E R_S(F) \leq R_S(F) + \sqrt{4m\E R_S(F)\frac{\log \frac{1}{\delta}}{k}}.$$
Or equivalently,
$$\sqrt{\E R_S(F)} \leq \sqrt{\frac{m}{k}\log \frac{1}{\delta}} + \sqrt{\frac{m}{k}\log \frac{1}{\delta} + R_S(F)}.$$
Hence with probability at least $1-2\delta$, we have
$$\sup_{f\in F}|A_S(f) - A_D(f)| \leq R_S(F) + \left(m+m\sqrt{\frac{8}{k}\log \frac{1}{\delta}} + m\sqrt{\frac{8}{k}\log \frac{1}{\delta} + \frac{8R_S(F)}{m}}\right)\sqrt{\frac{\log \frac{4}{\delta}}{2k}}.$$
We have shown Theorem \ref{thm:main}.

\subsection{Proof of Theorem \ref{thm2}}
For any $s>0$, by Jensen's inequality,
$$\begin{aligned}
\exp(skR_S(F)) & = \exp\left(2s\E_\sigma \sup_{f\in F} \sum_{i=1}^k \sigma_i f(s_i)\right) \\
&\leq \E_\sigma \exp\left(2s \sup_{f\in F} \sum_{i=1}^k \sigma_i f(s_i)\right) \\
&\leq \E_\sigma \sum_{f\in F} \exp\left(2s\sum_{i=1}^k \sigma_i f(s_i)\right).
\end{aligned}$$
By Hoeffding's Lemma \cite{H63},
$$\begin{aligned}
&\E_\sigma \sum_{f\in F} \exp\left(2s\sum_{i=1}^k \sigma_i f(s_i)\right) \\
\leq & \sum_{f\in F}\prod_{i=1}^k \exp\left(2s^2f(s_i)^2\right) \\
=& \sum_{f\in F} \exp\left(2s^2\sum_{i=1}^k f(s_i)^2\right).
\end{aligned}$$
Let $\ell^2 = \sup_{f\in F}\sum_{i=1}^k f(s_i)^2$, and then
$$\sum_{f\in F} \exp\left(2s^2\sum_{i=1}^k f(s_i)^2\right) \leq |F| \exp\left(2s^2\ell^2\right).$$
Thus
$$R_S(F) \leq \frac{1}{sk}(\log|F| + 2s^2\ell^2),$$
for any $s>0$. It turns out that to minimize the right hand side of the above equation, we have
$$s = \sqrt{\frac{\log|F|}{2\ell^2}}.$$
Then
$$R_S(F) \leq \frac{\ell}{k}\sqrt{8\log |F|}.$$

\subsection{Discussions of Theorem \ref{thm:new}}
Theorem \ref{thm:new} can be treated as a special case of Theorem \ref{thm:finite} and Theorem \ref{thm:main} when $m=c$. All the reasoning will not be affected and thus the desired bound follows.


\subsection{Loss functions}
Here we are going to discuss some of the loss functions that have been tested in this field so far, and that are relevant to this article. We will look away from the content loss as this part has stayed the same for virtually all papers on the matter.
\newline\newline
Style loss was initially discovered to be separable from content loss in a CNN (namely VGG16) by gatys et al \cite{Gatys:1}. They calculated what is known as the gram matrix which represents the style of an image as correlation between different features. The style loss is the squared error between the gram matrix of the features of the reference style image and the image for which we are minimizing the loss function. This function has been the basis for several other style transfer algorithms, and was for instance approximated with a neural net by Johnson et al \cite{Johnson:1} trained on a certain style.
\newline\newline
Another key finding was that the distribution of the features was a more complete representation of style. This lead to Huang et al. \cite{Huang:1} testing out minimizing the wasserstein distance between the features instead. This new representation of style produced results of higher artistic quality. Other techniques based the idea of the distance between two distributions as loss has also been tested out with MMD (Maximum mean discrepancy) as the distance measure in a paper by Li et al. \cite{Li:2} where they also combined it with the gram matrix. Chuan Li and Michael Wand used markov random fields to calculate an alternative to the gram matrix entirely \cite{Li:3}. Other style loss functions that noteworthy but not too relevant to delve into here are histogram loss \cite{Risser:1} and coral loss \cite{sun:1}
\newline\newline
Temporal loss has not been the main focus on this article we will just briefly go through the ones used. To go from picture style transfer to video style transfer we must introduce a temporal loss to avoid flickering, meaning that we should punish frames for being to different. The easiest way to do this is to just look at the difference between the current frame and the previous. This is known as short-term temporal loss. Long-term temporal loss is doing the same but for multiple previous frames. this will create more stable results given a greater context of the current frame.
\newline\newline
After applying style transfer on an image with Gatys et al \cite{Gatys:1}, we can end up with an image with many high frequency artifacts. To get rid of these variations in the output image, we want to penalize local variation. We can define this local variation loss as
\begin{equation}
    \sum{|y_{i+1}-y_i|^\beta}
\end{equation}
where $\beta\in\{1,2\}.$ Here $y$ are the pixel-values in one row of the output image \cite{Zhaoyou:1}. Now we can extend Equation (\ref{eq:total_loss}) with the total variation loss
\begin{equation}
    \mathcal{L}_\text{total}(\vec{p}, \vec{a}, \vec{x})=\alpha\mathcal{L}_\text{content}(\vec{p},\vec{x})+\beta\mathcal{L}_\text{style}(\vec{a},\vec{x})+\lambda\mathcal{L}_\text{TV}(\vec{x})
\end{equation}
where $\lambda$ is the weight for the total variation loss.
\newpage


\documentclass[12pt]{cdblatex}
\usepackage{examples}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% ============================================================================================
\bgroup
\CdbSetup{action=hide}
\begin{cadabra}
   import cdblib
   checkpoint_file = 'tests/semantic/output/example-02.json'
   cdblib.create (checkpoint_file)
   checkpoint = []
\end{cadabra}
\egroup

\clearpage

% ============================================================================================
\section*{Example 2 Covariant derivatives}

\begin{cadabra}
   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u#}::Indices.

   \nabla{#}::Derivative.
   \partial{#}::PartialDerivative.

   # rule for covariant derivative of v^{a}

   deriv := \nabla_{a}{v^{b}} -> \partial_{a}{v^{b}} + \Gamma^{b}_{c a} v^{c}.

   # create an expression

   foo := \nabla_{a}{v^{b}}.                     # cdb (ex-02.101,foo)

   # apply the rule, then simplify

   substitute   (foo,deriv)                      # cdb (ex-02.102,foo)
   canonicalise (foo)                            # cdb (ex-02.103,foo)

   checkpoint.append (foo)
\end{cadabra}

\begin{align}
   \cdb{ex-02.101} &= \Cdb{ex-02.102}\\
                   &= \Cdb{ex-02.103}
\end{align}

\clearpage

% ============================================================================================
\section*{Example 2 Covariant derivatives using ``position=independent''}

\begin{cadabra}
   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u#}::Indices(position=independent).

   \nabla{#}::Derivative.
   \partial{#}::PartialDerivative.

   # rule for covariant derivative of v^{a}

   deriv := \nabla_{a}{v^{b}} -> \partial_{a}{v^{b}} + \Gamma^{b}_{c a} v^{c}.

   # create an expression

   foo := \nabla_{a}{v^{b}}.                     # cdb (ex-02.201,foo)

   # apply the rule, then simplify

   substitute   (foo,deriv)                      # cdb (ex-02.202,foo)
   canonicalise (foo)                            # cdb (ex-02.203,foo)

   checkpoint.append (foo)
\end{cadabra}

\begin{align}
   \cdb{ex-02.201} &= \Cdb{ex-02.202}\\
                   &= \Cdb{ex-02.203}
\end{align}

\clearpage

% ============================================================================================
\section*{Example 2 Covariant derivatives using generic rule for deriv}

\begin{cadabra}
   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u#}::Indices(position=independent).

   \nabla{#}::Derivative.
   \partial{#}::PartialDerivative.

   # template for covariant derivative of a vector

   deriv := \nabla_{a}{A?^{b}} -> \partial_{a}{A?^{b}} + \Gamma^{b}_{c a} A?^{c}.

   # create an expression

   foo := \nabla_{a}{u^{b}} + \nabla_{a}{v^{b}}. # cdb (ex-02.301,foo)

   # apply the rule, then simplify

   substitute   (foo,deriv)                      # cdb (ex-02.302,foo)
   canonicalise (foo)                            # cdb (ex-02.303,foo)

   checkpoint.append (foo)
\end{cadabra}

\begin{align}
   \cdb{ex-02.301} &= \Cdb{ex-02.302}\\
                   &= \Cdb{ex-02.303}
\end{align}

\clearpage

% ============================================================================================
% export to json format

\bgroup
\CdbSetup{action=hide}
\begin{cadabra}
   for i in range( len(checkpoint) ):
      cdblib.put ('check{:03d}'.format(i),checkpoint[i],checkpoint_file)
\end{cadabra}
\egroup

\end{document}


% !TEX root = BusSim.tex
\section*{Appendix A: The BusSim model \label{appendix:BusSim}}

Figure 2 illustrates the workflow for BusSim-truth. At each current time step $t$, each Bus agent checks whether the next time step would be larger than the vehicle's scheduled dispatch time $\delta_j$. If $t>\delta_j$, we then check whether the bus is on the road (Status equals $MOVING$), or at a stop for passenger dwelling (Status equals $DWELLING$), or has finished its service (Status equals $FINISHED$), otherwise the bus remains $IDLE$.

If the status is $MOVING$, we first check whether the bus is at a bus stop, by comparing the $GeoFence$ area of each bus stop agent with the bus' location. If the bus is not approaching a bus stop, its current speed $v_j$ will be compared with the surrounding traffic speed $V$. If $v_j<V$, we assume that the bus will speed up with an acceleration rate $a_j$, thus we have:
\begin{equation}
v_j^{t} = v_j^{t-dt} + a_j \cdot dt
\end{equation}

Therefore for the next time step, the bus will cover a distance of:
\begin{equation}
S_j^t = S_j^{t-dt} + v_j^t \cdot dt
\end{equation}

If the speed already matches the traffic speed $V$, the bus will maintain the same speed. Or else if the bus is approaching a bus stop, the system will first check if the stop is the last stop. If it is the last stop, then the bus' status will be changed to $FINISHED$ and bus speed is changed to zero. If it is not the last stop, the system will change the status of agent Bus $j$ to $DWELLING$ and its speed to zero. The number of boarding and alighting passengers from the bus $j$, and the time that it will leave the stop are estimated as follows.

The number of boarding passenger is proportional to the time gap between the current time (when Bus $j$ approaches the bus stop $m$) and the last time any bus visits the bus stop $m$:
\begin{equation}
B_{j,m} = \nint{Po(Arr_m \cdot (t^a_{j+1,m}-t^a_{j,m}) } \quad | \quad B_{j,m}\in\mathbb{N}
\label{eq:Boarding_est}
\end{equation}

Equation \ref{eq:Boarding_est} shows that the number of boarding passengers is estimated using a stochastic Poisson process. A Poisson process is widely adopted in literature to estimate the count of passengers waiting at a public transport stop \citep{toledo2010mesoscopic,cats2010mesoscopic}. Extensions of this stochastic process have been introduced, such as non-homogeneous Poisson process \citep{kieu2018stochastic}, where the arrival rate is time-dependent, but for simplicity we adopt a homogeneous Poisson process for this paper. Equation \ref{eq:Boarding_est} makes the BusSim-truth model stochastic, because there is randomness in the way the Poisson process generates a number. For more details on the number generation process using stochastic Poisson process (e.g. thinning algorithm), interested readers may refer to \citep{lewis1979simulation}. The number of boarding passengers is also limited by the available capacity of the bus:
\begin{equation}
B_{j,m} = \text{max} \big( B_{j,m}, C - Occ_m )   \big)
\label{eq:Boarding_limit}
\end{equation}

The number of alighting passengers is proportional to the number of passenger on board (bus occupancy) and the departure rate at the stop $m$.  For simplicity, we assume that $A_{j,m}$ is the product between the departure rate from bus stop $m$ and the current bus occupancy (the number of passenger on board leaving the last stop):
\begin{equation}
A_{j,m} = \nint{Dep_m \cdot Occ_{j,m-1}} \quad | \quad A_{j,m}\in\mathbb{N}
\end{equation}

To estimate the amount of time that bus will have to stay at the bus stop $m$ for passenger boarding and alighting, a.k.a. \textit{dwell time} $D_{j,m}$, we adopt the approach in \citep{bertini2004modeling} and the Transit Capacity and Quality of Service Manual (TCQSM) \citep{kfh2013transit}:
\begin{equation}
D_{j,m} = \theta_1 + \theta_2 \times B_{j,m} + \theta_3 \times A_{j,m}
\label{eq:dwell_time}
\end{equation}
The parameter set [$\theta_1,\theta_2,\theta_3$] represents the time spent for passenger boarding, alighting, and a fixed value for vehicle stopping and starting, respectively. Equation \ref{eq:dwell_time} is the formulation for a single-door bus system, where boarding and alighting occurs sequentially.

The departure time of bus $j$ from stop $m$ is calculated from the arrival time $t^a_{j,m}$ plus the time spent at stops for passenger boarding and alighting, or in other words the dwell time $D_m$:
\begin{equation}
t^d_{j,m} = t^a_{j,m} + D_{j,m}
\end{equation}
In BusSim, the bus $j$ is only allowed to leave the bus $m$ at time $t^d_{j,m}$, so this is also called the $Leave\_stop\_time$, as can be seen in the Figure 2.

If the status of bus $j$ is $DWELLING$, it is at a stop for passenger boarding and alighting. We then check if the next time step would be larger or equal to the leave stop time $t^d_{j,m}$. If it would, then the bus would start accelerate to leave the stop, otherwise it would stay for at least another time interval. Finally, if the status of the bus is $FINISHED$, then we would do nothing. The modelling process then moves to the next Bus agent until the last Bus, then the whole model moves to the next time step until the last time step.

BusSim-truth also assumes that parameters dynamically change over time by introducing an additional parameter $\xi$ to represent the change in passenger demand or surrounding traffic speed. For simplicity, we assume that a single, deterministic parameter $\xi$ can model these dynamic changes. In practice, it is possible, and more desirable, to use a time-dependent value of $\xi$ such that dynamic change is better captured, and multiple $\xi$ to model different changes. $\xi>0$ represents an increase in passenger demand and traffic speed, and $\xi<0$ represents otherwise. In this paper, the change in passenger demand or traffic speed is modelled as:
\begin{align}
V = V \cdot \big( 1 - \frac{t}{T} \cdot \frac{100}{\xi} \big) \\
Arr_m = Arr_m \cdot (1 - \frac{t}{T} \cdot \frac{100}{\xi}
\label{eq:dynamic_bussim}
\end{align}
A positive value of $\xi$ in Equation \ref{eq:dynamic_bussim} gradually reduces the surrounding traffic speed $V$ and increases the arrival rate $Arr_m$, which would lead to more bus delays and congestion.


% !TEX root = Main.tex
\section{Non-Negative Matrix Factorization}
\textbf{Context Model:} $p(w | d) = \sum_{z=1}^K p(w | z) p(z | d)$\\
\textbf{Conditional independence assumption ($*$):}\\
$p(w|d) = \sum_z p(w,z|d) = \sum_z p(w|d,z)p(z|d) \stackrel{*}{=} \sum_z p(w|z)p(z|d)$\\
\textbf{Symmetric parameterization:}\\
$p(w, d) = \sum_z p(z)p(w | z) p(d | z)$

\subsection*{EM for pLSA}
Log-Likelihood: $L(\mathbf{U}, \mathbf{V}) = \sum_{i,j} x_{i,j}\log p(w_j|d_i) \\
= \sum_{(i,j) \in X} \log \sum_{z=1}^K p(w_j|z)p(z|d_i)$ \\
$ p(w_j|z) = v_{zj}$, $p(z|d_i) = u_{zi}$, $\sum_j^N v_{zj} = \sum_z^K u_{zi} = 1$\\
E-Step (optimal q):\\
$q_{zij} = \frac{p(w_j|z)p(z|d_i)}{\sum_{k=1}^K p(w_j|k)p(k|d_i)} := \frac{v_{zj}u_{zi}}{\sum_{k=1}^K v_{kj}u_{ki}}$\\
M-Steps:\\
$p(z|d_i) = \frac{\sum_j x_{ij}q_{zij}}{\sum_j x_{ij}}, p(w_j|z) = \frac{\sum_i x_{ij}q_{zij}}{\sum_{i,l}x_{il}q_{zil}}$\\

\subsection*{Latent Dirichlet Allocation}
To sample a new document, we need to extend $X$ and $U^T$ with a new row, s.t. $X=U^T V$. pLSA fixes both dimensions.\\
Dirichlet distribution: $p(u_i|\alpha) = \prod_{z=1}^K u_{zi}^{\alpha_k-1}$\\
LDA model: $p(x|V,u) = \frac{l!}{\prod_j x_j!}\prod_j \pi_j^{x_j}$\\
where $\pi_j=\sum_z v_{zj} u_z$, $l=\sum_j x_j$

\subsection*{NMF Algorithm for quadratic cost function}
$\mathbf{X} \in \mathbb{Z}^{N \times M}_{\geq 0}$, NMF: $\mathbf{X} \approx \mathbf{U^\top V}, x_{ij}$

$\min_{\mathbf{U}, \mathbf{V}} J(\mathbf{U}, \mathbf{V}) = \frac{1}{2} \|\mathbf{X} - \mathbf{U}^\top\mathbf{V}\|_F^2$\\
s.t. $\forall i,j,z:u_{zi},v_{zj} \geq 0 $

1. init: $\mathbf{U}, \mathbf{V} = rand()$ 2. repeat for $\mathit{maxIters}$:\\
3. upd. $(\mathbf{VV}^\top)\mathbf{U} = \mathbf{VX}^\top$, proj. $u_{zi} = \max \{ 0, u_{zi} \}$\\
4. update $(\mathbf{UU}^\top)\mathbf{V} = \mathbf{UX}$, proj. $v_{zj} = \max \{ 0, v_{zj} \}$


\documentclass[11pt]{article}

\input{preamble}

\title{Week 2: Gradients}
\author{\url{http://mlvu.github.io}}

\begin{document}

\maketitle

\noindent In this session we will practice searching for a good or optimal model using the \emph{gradient}. We will define the \emph{loss function} of a model, with respect to some target data, and we will will minimize the loss function with respect to the parameters.

In the following explanation, some parts are replaced by green dots. Fill these in.

\section{Partial derivatives}

We'll begin by reviewing the idea of \emph{partial derivatives}: derivatives of functions with multiple inputs. If you haven't worked with derivatives for a while (or ever), start by having a look at the following resources:
\begin{itemize}
\item \url{https://www.youtube.com/watch?v=ANyVpMS3HL4}
\item \url{https://www.youtube.com/watch?v=hCLfogkqzEk}
\item \url{http://betterexplained.com/articles/derivatives-product-power-chain/}
\end{itemize}

\noindent A partial derivative is the derivative with respect to one of the parameters, treating all the others as constants. For a simple example, consider the function
\[
 f(\rc{a},\bc{b})= 3\rc{a}^2 + \bc{b}^2 - \rc{a}\bc{b} + \rc{a}.
\]
\noindent
We first take the derivative with respect to \rc{a}:
\[
	\frac{\kp f(\rc{a},\bc{b})}{\kp \rc{a}} = \frac{\kp(3\rc{a}^2 + \bc{b}^2 - \rc{a}\bc{b} + \rc{a})}{\kp \rc{a}} = 6\rc{a} -\bc{b} + 1 \p
\]
Note that the second term of the function ($\bc{b}^2$) does not contain $\rc{a}$, i.e. it is constant with respect to $\rc{a}$, so it disappears from the derivative.

\noindent Then, we take the derivative wrt. to \bc{b}:
\[
\frac{\kp f(\rc{a},\bc{b})}{\kp \bc{b}} = \frac{\kp(3\rc{a}^2 + \bc{b}^2 - \rc{a}\bc{b} + \rc{a})}{\kp \bc{b}} = 2\bc{b} - \rc{a} \p
\]
The \emph{gradient} $\nabla f(\rc{a},\bc{b})$ is simply all the partial derivatives of a function arranged in a (row) vector:
\[
\nabla f(\rc{a},\bc{b}) = \left ( \frac{\kp f(\rc{a},\bc{b})}{\kp \rc{a}},\;\;\frac{\kp f(\rc{a},\bc{b})}{\kp \bc{b}}\right) = \left(6\rc{a} - \bc{b} + 1, \;\;2\bc{b} - \rc{a}\right) \p
\]

\noindent If we imagine the function f as a ``landscape'' over the plane defined by the $\rc{a}$ and $\bc{b}$-axes, the gradient at each point in that plane, is an arrow pointing in the direction in which the ascent is the steepest.\footnotemark~A marble placed at some point, and released, would roll in the opposite direction to the gradient.

\footnotetext{We can interpret a vector $\x \in R^n$ as a point in $R^n$, but also as a \emph{direction}. The direction is that of the arrow between the origin and the point $\x$. The gradient only makes sense as a direction.}

To find the point at which the function has reached its minimum, we must find out where all partial derivatives are equal to zero.\footnotemark
\footnotetext{To be precise, if all partial derivatives are zero, the function may be at a minimum, a maximum, or a plateau. It may even be a saddle-point, a place where the function is a minimum in one dimension and a maximum in another. For the purposes of this exercise, you may assume that if the gradient is zero, you have found a minimum.}

\qu To find the \rc{a} and \bc{b} for which f(\rc{a}, \bc{b}) is minimal, we set the partial derivatives to zero. Fill in the blanks (indicated by ``\ldots'') in this derivation:
\begin{align*}
6\rc{a} - \bc{b} + 1 &= 0 & 2\bc{b}-\rc{a} &= 0 \\
\rc{a} &= (\bc{b}-1)/6 & \bc{b} &= \rc{a}/2 \\
 & & \bc{b} &= \ans{\frac{\bc{b}-1}{6}\frac{1}{2}}{\dots}\\
 & & \ans{\bc{b} - \frac{\bc{b}}{12}}{\ldots}&= \ans{- \frac{1}{12}}{\ldots}\\
  & & \bc{b} &= -\frac{1}{11}\\
  \rc{a} &= \ans{\frac{-\frac{1}{11} - 1}{6}}{\ldots}=-\frac{2}{11} & & \\
\end{align*}

\noindent A quick check on \href{http://wolfr.am/9DrZFKcR}{Wolfram Alpha} shows that this solution is correct.

\subsection{Rules}

While it's important to understand intuitively what differentation means (see the \emph{better explained} link above), actually finding a specific derivative is usually a very mechanical process of matching existing rules and rewriting a function to a useful form. The following rules are usually sufficient:
Let $c$ be a constant, independent of $x$, and let $f(x)$ and $g(x)$ be arbitrary functions of $x$. Then:

\begin{align*}
	\frac{\kp c}{\kp x}     &=0        & \text{ the \emph{constant} rule} \\
	\frac{\kp x^\rc{n}}{\kp x}   &=\rc{n}x^{\rc{n}-1} & \text{the \emph{exponent} rule} \\
	\frac{\kp x}{\kp x}     &= 1       & \text{(follows from the exponent rule)} \\
	\frac{\kp \rc{c}f(x)}{\kp x} &= \rc{c}\frac{\kp f(x)}{\kp x} & \text{the \emph{constant factor} rule} \\
	\frac{\kp \kc{\left ( \bc{f(x)} + \gc{g(x)}\right )}}{\kp x} &= \frac{\kp \bc{f(x)}}{\kp x} + \frac{\kp \gc{g(x)}}{\kp x} & \text{the \emph{sum} rule} \\
	\frac{\kp \bc{f(\gc{g(x)})} }{\kp x} &= \frac{\kp \bc{f(\gc{g(x)})}}{\kp \gc{g(x)}}\frac{\kp \gc{g(x)}}{\kp x} & \text{the \emph{chain} rule} \\
\end{align*}

\noindent To find the derivative of $f(x, y, z)$ with respect to $y$, you write down
\[
\frac{\kp f(x, y, z)}{\kp y} \text{,}\]
fill in the definition of $f$, and match the result (whole, or part of it) with the left-hand side of one of these rules. You replace it by the right-hand side and keep going until all the $\kp$'s are gone.

\qu To practice, let's take the 	derivative of $(x + y + z)^2$ with respect to $y$. Fill in the blanks.

\begin{align*}
\frac{\kp \bc{(\gc{x + y + z})^2}}{\kp y} &= \frac{\kp \bc{(\gc{x + y + z})^2}}{\kp \kc{(\gc{x + y + z})}}\frac{\kp \kc{(\gc{x + y + z})}}{\kp y} & \text{using the \ans{chain}{\ldots} rule} \\
 &= 2(x + y + z)\frac{\kp \kc{(x + y + z)}}{\kp \kc{y}} & \text{using the \ans{exponent}{\ldots} rule} \\
 &= \kc{2(x + y + z)}\left (\frac{\kp x}{\kp y} + \frac{\kp y}{\kp y} + \frac{\kp z}{\kp y} \right) & \text{using the \ans{sum}{\ldots} rule} \\
  &= \kc{2(x + y + z)}(0 + 1 + 0) & \text{using the \ans{constant}{\ldots} and \ans{exponent}{\ldots} rules} \\
  &=  2(x + y + z) & \\
\end{align*}

\section{Linear regression}

As discussed in the lecture, the optimal model for standard linear regression can be computed directly. In this assignment we will derive this function. Let’s say we are trying to predict the size to which a particular newborn baby will grow in the coming months. We have measured the child in its first few months and found the following data:

\begin{table}[H]
\centering
\begin{tabular}{l | l}
	age($a$, months) & height ($h$, cm) \\
	\hline
	$a_1 = 0$ & $h_1 = 30$ \\
	$a_2 = 2$ & $h_2 = 40$ \\
	$a_3 = 4$ & $h_3 = 50$
\end{tabular}
\end{table}

\noindent Let $n$ be the number of examples we have (3 in this case, but we want to derive a solution for arbitrary $n$). Our model, $f$, is a linear function, described by two parameters: the slope $s$ and the intercept $b$:
\[
f_{s, b}(a) = sa + b
\]

\noindent Where $f(a_i)$ should be as close to $h_i$ as possible. We will use the sum of squared errors as our loss function. That is, we will compute the residual $f(a_i) - h_i$ for each data point, square them, and sum these squared values. In other words, our loss function is
\[
\text{loss}(s, b) = \frac{1}{2}\sum_i\left ( \rc{f_{s, b}(a_i)} - h_i\right )^2 = \frac{1}{2}\sum_i\left ( \rc{sa_i + b} - h_i\right )^2
\]

\noindent The $\frac{1}{2}$ multiplier is there to simplify the derivatives later. It doesn't affect the minimum of the loss function, which is what we're after.\footnotemark

\footnotetext{In the slides, we used the \emph{mean} sum of squared errors (i.e. we multiplied by $\frac{1}{n}$ as well). Since $n$ is a constant, the two functions have the same minimum and we can use either one.}

\qu For $s=1$ and $b=0$, and the data given above, what is the loss?
\ans{
\begin{align*}
	\text{loss}(1, 0) &= \frac{1}{2}\sum_i\left ( \rc{1 \times a_i + 0} - h_i\right )^2 \\
	&= \frac{1}{2}\left ((0 - 30)^2 + (2 - 40)^2 + (4 - 50)^2 \right) \\
	&= \frac{1}{2}\left (900 + 1444 + 2116 \right) = 2230\\
\end{align*}
}{}

\qu The term $f(a_i) - h_i$ represents the difference between our model's prediction and the observed value (the \emph{residual}). Per example, this is a good measure of the error. Why do we not just sum these, and check how far it is from 0? Why square it first, and \emph{then} sum?

\ans{
If we summed them, one big positive error could cancel out against one big negative error, giving the false impression that the model is highly accurate. We need to square the errors before summing them. The choice for square instead of another approach (like taking the absolute value, or raising to some other power) is more subtle. Squaring emphasizes the loss of large errors compared to the absolute value.

It turns out that if we assume that the data is linear, but with added Gaussian noise, optimizing for the sum-of-squared errors is equivalent to optimizing likelihood (we will discuss this later in the probability lectures.)
}{}

\qu Let's find the derivative of the loss function. One with respect to $s$ and one with respect to $b$. Fill in the blanks.

\begin{align*}
\frac{\kp\text{loss}(s, b)}{\kp s} & = \frac{\kp \frac{1}{2} \sum_i\left (s a_i + b - h_i \right) ^2}{\kp s} \\
&= \frac{1}{2} \sum_i \frac{\kp \left (sa_i + b -h\right )^ 2}{\kp (sa_i + b -h_i)}\frac{\kp (sa_i + b -h_i)}{\kp s}\\
&= \ans{\frac{1}{2}\sum_i 2(a_is + b - h_i)a_i}{\ldots} \\
&= \sum_i (a_is + b - h_i)a_i \\
&= \sum_i ({a_i}^2s + a_ib - a_ih_i) \\
&= s \sum_i {a_i}^2 + b \sum_ia_i - \sum_i a_i h_i \\
\end{align*}

\begin{align*}
\frac{\kp\text{loss}(s, b)}{\kp b} & = \ans{\frac{\kp \frac{1}{2} \sum_i\left (s a_i + b - h_i \right) ^2}{\kp b}}{\dots} \\
&= \ans{\frac{1}{2} \sum_i \frac{\kp \left (sa_i + b -h_i\right )^ 2}{\kp (sa_i + b -h_i)}\frac{\kp (sa_i + b -h_i)}{\kp b}}{\ldots}\\
&= \ans{\frac{1}{2}\sum_i 2(a_is + b - h_i)}{\ldots} \\
&= \sum_i (a_is + b - h_i) \\
&= s \sum_i {a_i} + bn - \sum_i h_i \\
\end{align*}

\qu For many loss functions, setting the gradients equal to zero results in a system of equations that cannot be solved analytically. In that case we will have to \emph{search}. We can still use the gradient though, in an algorithm called \emph{gradient descent}. Starting with the parameter values $s=1$ and $b=0$, describe one step of the gradient descent algorithm (with learning rate 0.01).
\ans{
For these parameters, the gradient is
\begin{align*}
&\left (\rc{s \sum_i {a_i}^2 + b \sum_i a_i - \sum_i a_i h_i }, \;\; \bc{s \sum_i a_i + bn - \sum_i h_i} \right) \\
&\;= \left(\rc{\sum_i {a_i}^2 - \sum_i a_i h_i},\;\; \bc{\sum_i {a_i} - \sum_i h_i} \right) \\
&\;= \left( \rc{4 + 16 - (2\cdot 40 + 4\cdot 50)},\;\; \bc{6 - 120}\right) \\
&\;= \left( \rc{-260},\;\; \bc{-114}\right) \\
\end{align*}
For gradient descent we pick the opposite direction (since the gradient points up), multiply by the learning rate, and add the result to the current parameters. Thus, the new parameters are:
\begin{align*}
\begin{bmatrix}s^\text{new} \\b^\text{new}\end{bmatrix} &= \begin{bmatrix}s \\b\end{bmatrix} - \eta \nabla \text{loss}(s, b)\\
&= \begin{bmatrix}1 \\0\end{bmatrix} - 0.01 \begin{bmatrix}\rc{-260} \\\bc{-114}\end{bmatrix} = \begin{bmatrix}3.6 \\ 1.14\end{bmatrix}\p
\end{align*}
}{}

\qu Set the two derivatives found above equal to zero, and solve to obtain expressions for the optimal model. Make sure that your expression for $s$ does not depend on $b$, so that the solution can actually be computed.
To simplify notation, it can be helpful to use the following conventions for the data means and other statistics:
\begin{align*}
\overline{a} &= \frac{1}{n}\sum_i a_i	\\
\overline{h} &= \frac{1}{n}\sum_i h_i	\\
\overline{a^2} &= \frac{1}{n}\sum_i {a_i}^2	\\
\overline{h^2} &= \frac{1}{n}\sum_i {h_i}^2	\\
\overline{ah} &= \frac{1}{n}\sum_i a_ih_i	\\
\end{align*}

\noindent Fill in the blanks:

\begin{align}
	s \sum_i {a_i}^2 + b \sum_ia_i - \sum_i a_i h_i &= 0 \notag\\
	s \sum_i {a_i}^2 &= \ans{- b \sum_ia_i + \sum_i a_i h_i}{\dots} \notag\\
	s &= - b \frac{\sum_i a_i}{\sum_i{a_i}^2} + \frac{\sum_i a_i h_i}{\sum_i{a_i}^2} \notag \\
	s &= \ans{- b \frac{\frac{1}{n}\sum_i a_i}{\frac{1}{n}\sum_i{a_i}^2} + \frac{\frac{1}{n}\sum_i a_i h_i}{\frac{1}{n}\sum_i{a_i}^2}}{\ldots} \notag\\
	s &= -b\frac{\overline{a}}{\overline{a^2}} + \frac{\overline{ah}}{\overline{a^2}} \label{line:s}
\end{align}

\begin{align}
	s \sum_i {a_i} + bn - \sum_i h_i &=0 \notag \\
	bn &= \ans{\sum_ih_i - s\sum_i a_i}{\ldots} \notag\\
	b &= \ans{\frac{1}{n}\sum_ih_i - s\frac{1}{n}\sum_i a_i}{\ldots} \notag\\
	b &= \overline{h}-s\overline{a} \label{line:b}
\end{align}
Fill equation (\ref{line:b}) into equation (\ref{line:s}):
\begin{align*}
s &= -\left(\ans{\overline{h}-s\overline{a}}{\ldots}\right)\frac{\overline{a}}{\overline{a^2}} + \frac{\overline{ah}}{\overline{a^2}} \\
s &= - \frac{\overline{h}\overline{a}}{\overline{a^2}} + s \frac{\overline{a}^2}{\overline{a^2}} + \ans{\frac{\overline{ah}}{\overline{a^2}}}{\ldots} \\
 s \left(1-\frac{\overline{a}^2}{\overline{a^2}}\right)&= \ans{- \frac{\overline{h}\overline{a}}{\overline{a^2}} +\frac{\overline{ah}}{\overline{a^2}}}{\ldots} \\
 s &= \frac{\overline{ah} - \overline{a}\overline{h}}{\overline{a^2} - \overline{a}^2} \\
\end{align*}
\noindent Note that we have now expressed $s$ and $b$ purely in terms of statistics that are easily computed from our data, like the mean height $\overline{h}$ and the mean age $\overline{a}$.

\qu Fill in the values from the data set, and compute the optimal parameters for this data. Can you explain what the parameters $s$ and $b$ mean? That is, what can they tell us about the baby we've measured?
\ans{Filling in the data gives us $\overline{ah} = \frac{280}{3}$, $\overline{a} = 2$, $\overline{h} = 40$, $\overline{a^2} = \frac{20}{3}$. This gives us
\begin{align}
s = \frac{\frac{280}{3} - 2\cdot 40}{\frac{20}{3} - 4} = \frac{\frac{40}{3}}{\frac{8}{3}} = 5 \\
b = 40 - 5 \cdot 2 = 30
\end{align}
This tells us that this baby grows about 5 cm per month, and its size at birth was 30 cm. (Note these values were made up, and are actually a little small for a newborn baby.)}{}
See if \href{https://goo.gl/dEcRBG}{Wolfram Alpha} agrees with your answer.

\end{document}


\chapter{Transient and Steady-State Analysis of the LMS Adaptive Filter}
\label{app:lms_analysis}
In this appendix, we give a more detailed transient and steady-state analysis of the LMS adaptive filter than found in Lecture~\ref{ch:SD_LMS}. Specifically, we derive the condition for mean-square convergence and the expressions for the learning curve, the excess mean-square error (EMSE), the mean-square deviation (MSD), and the misadjustment. We do this for two reasons. Firstly, it is instructive to see how the analysis is carried out, and, secondly, it highlights how difficult the analysis is - even of the simple LMS adaptive filter.

In the derivation below, we use the analysis model previously discussed in Sec.~\ref{ssec:analysis_model}. We also need the following important result.

\section{A Special Fourth Order Moment of a Gaussian Random Vector}
Let $\vect{u}$ be a real-valued $M\times 1$ dimensional Gaussian random vector with zero-mean and correlation matrix $\vect{R}_u$. Moreover, let $\vect{W}$ be an $M\times M$ symmetric matrix. We then have that \cite[p.~44]{Sayed2003}
\bmath
  E[\vect{uu}^T\vect{Wuu}^T] = \vect{R}_u\tr{\vect{WR}_u}+2\vect{R}_u\vect{WR}_u\ .
\emath
If $\vect{R}_u = \vect{X\Lambda X}^T$ is the eigenvalue decomposition of the correlation matrix and if we define $\vect{F}=\vect{X}^T\vect{WX}$, we may write
\bmath
  E[\vect{uu}^T\vect{Wuu}^T] = \vect{X}[\vect{\Lambda}\tr{\vect{F\Lambda}}+2\vect{\Lambda F\Lambda}]\vect{X}^T\ .
\emath
With this result in mind, we can now evaluate the following expectation
\begin{align}
  E[(\vect{I}-\mu\vect{u}\vect{u}^T)\vect{W}(\vect{I}-\mu\vect{u}\vect{u}^T)]&= \vect{W}+\mu^2E[\vect{uu}^T\vect{Wuu}^T]-\mu\vect{R}_u\vect{W}-\mu\vect{W}\vect{R}_u\\
  &= \vect{W} + \mu^2\vect{R}_u\tr{\vect{WR}_u}+2\mu^2\vect{R}_u\vect{WR}_u -\mu\vect{R}_u\vect{W}-\mu\vect{W}\vect{R}_u\\
  &= \vect{X}[\vect{F} - \mu(\vect{\Lambda F}+\vect{F\Lambda}) + \mu^2(\vect{\Lambda}\tr{\vect{F\Lambda}} + 2\vect{\Lambda F\Lambda})]\vect{X}^T\ .
\end{align}
If $\vect{F}$ is diagonal, then $\vect{F\Lambda}=\vect{\Lambda F}$ and we have that
\bmath
	E[(\vect{I}-\mu\vect{u}\vect{u}^T)\vect{W}(\vect{I}-\mu\vect{u}\vect{u}^T)] = \vect{X}[\vect{F} - 2\mu\vect{\Lambda F} + \mu^2(\vect{\Lambda}\tr{\vect{F\Lambda}} + 2\vect{\Lambda}^2 \vect{F})]\vect{X}^T\ .
	  \label{eq:app_lms_4mom}
\emath
This result is very important for the derivation below.

\section{The Analysis Model}
The LMS algorithm is given by
\bmath
  \vect{w}(n+1) = \vect{w}(n) + \mu\vect{u}(n)e(n)
\emath
where $\vect{w}(n)$, $\vect{u}(n)$, and $e(n)$ are the filter vector, the input vector and the error, respectively, at time index $n$. The positive scalar $\mu$ is the step-size, and the error is given by
\bmath
  e(n) = d(n) - \vect{u}^T(n)\vect{w}(n)
\emath
where the desired signal $d(n)$ in the analysis model (see also Sec.~\ref{ssec:analysis_model}) is given by
\bmath
  d(n) = v(n) + \vect{u}^T(n)\vect{w}_o\ .
\emath
We assume that $v(n)$ is white Gaussian noise with variance $\sigma_v^2$ and uncorrelated with the input signal $u(n)$. Finally, we assume that $\vect{u}(n)$ is a white Gaussian process with correlation matrix $\vect{R}_u$. This implies that $\vect{u}(n)$ and $\vect{w}(n)$ are uncorrelated.

The difference between the optimal filter vector $\vect{w}_o$ and the filter vector $\vect{w}(n)$ is called the weight error, and it is given by
\begin{align}
  \vect{\Delta}\vect{w}(n) &= \vect{w}_o-\vect{w}(n) = \vect{w}_o - \vect{w}(n-1) - \mu\vect{u}(n-1)e(n-1)\\
  &= \vect{\Delta}\vect{w}(n-1) - \mu \vect{u}(n-1)[v(n)+\vect{u}^T(n-1)\vect{w}_o-\vect{u}^T(n-1)\vect{w}(n-1)]\\
  &= [\vect{I}-\mu\vect{u}(n-1)\vect{u}^T(n-1)]\vect{\Delta}\vect{w}(n-1)-\mu\vect{u}(n-1)v(n)\ .
 \label{app:lms_weight_error}
\end{align}
In terms of the weight error, the error is given by
\bmath
  e(n) = d(n) - \vect{u}^T(n)\vect{w}(n) = v(n)+\vect{u}^T(n)\vect{\Delta}\vect{w}(n)\ ,
\emath
and the minimum mean-squared error (MMSE) is achieved at the optimum where $\vect{\Delta}\vect{w}(n)=\vect{0}$ and thus given by
\bmath
  J_\textup{min} = J_1(\vect{w}_o) = E[v^2(n)] = \sigma_v^2\ .
\emath

\section{Transient Analysis}
In the transient analysis, we first consider how we should select the step-size $\mu$ in order to ensure the mean-square stability of the LMS algorithm. Second, we derive an expression for the learning curve.

\subsection{Mean-Square Convergence}
As already discussed in Lecture~\ref{ch:SD_LMS}, we say that the LMS algorithm converges in the mean-square if
\bmath
  \lim_{n\to\infty} E[\|\vect{\Delta w}(n)\|^2] = c <\infty\ .
\emath
In this section, we show that the LMS algorithm converges in the mean square if the step-size $\mu$ is selected such that it satisfies
\bmath
  f(\mu) = \frac{\mu}{2}\sum_{m=1}^M \frac{\lambda_m}{1-\mu\lambda_m} < 1
\emath
where $\lambda_m$ is the $m$'th eigenvalue of the correlation matrix $\vect{R}_u$.

In order to do this, we investigate $E[\|\vect{\Delta w}(n)\|^2]$ as $n$ grows. For convenience of notation, it turns out to be easier to work with $E[\|\vect{c}(n)\|^2]$ where $\vect{c}(n)= \vect{X}^T\vect{\Delta}\vect{w}(n)$ and $\vect{X}$ contains the $M$ eigenvectors of $\vect{R}_u$. Since $\vect{X}$ is orthogonal, it follows that
\bmath
  E[\|\vect{c}(n)\|^2] = E[\|\vect{X}^T\vect{\Delta w}(n)\|^2] = E[\|\vect{\Delta w}(n)\|^2]
\emath
Thus, $E[\|\vect{c}(n)\|^2]$ converges in the same way as $E[\|\vect{\Delta w}(n)\|^2]$ does. In the derivation below, we express $E[\|\vect{c}(n)\|^2]$ as a function of $\vect{c}(0)$, $\mu$, $\vect{\Lambda}$, and $J_\textup{min}$. We do this in a recursive manner starting by expressing $E[\|\vect{c}(n)\|^2]$ as a function of $\vect{c}(n-1)$, $\mu$, $\vect{\Lambda}$, and $J_\textup{min}$. Once we have obtained an expression for this recursion, we express $E[\|\vect{c}(n)\|^2]$ as a function of $\vect{c}(n-2)$, $\mu$, $\vect{\Lambda}$, and $J_\textup{min}$ in the next recursion and so on.

\subsubsection{The First Recursion}
Multiplying \eq{app:lms_weight_error} from the left with $\vect{X}^T$ and inserting it into $E[\|\vect{c}(n)\|^2]$ yields
\begin{align}
  E[\|\vect{c}(n)\|^2] ={}& E[\|\vect{X}^T[\vect{I}-\mu\vect{u}(n-1)\vect{u}^T(n-1)]\vect{X}\vect{c}(n-1)-\mu\vect{X}^T\vect{u}(n-1)v(n)\|^2]\\
   ={}& E[\vect{c}^T(n-1)\vect{X}^T[\vect{I}-\mu \vect{u}(n-1)\vect{u}^T(n-1)]^2\vect{X}\vect{c}(n-1)]\notag\\
   &{}+\mu^2E[v^2(n)\vect{u}^T(n-1)\vect{u}(n-1)]
   \label{eq:app_lms_first_recur_start}
\end{align}
where the last equality follows from the assumption that $v(n)$ and $\vect{u}(n)$ are uncorrelated.
The expectation of the last term is given by
\begin{align}
 E[v^2(n)\vect{u}^T(n-1)\vect{u}(n-1)] &= E[v^2(n)]E[\vect{u}^T(n-1)\vect{u}(n-1)]\\
 &= J_\textup{min} E[\tr{\vect{u}^T(n-1)\vect{u}(n-1)}]\\
 &= J_\textup{min} E[\tr{\vect{u}(n-1)\vect{u}^T(n-1)}]\\
 &= J_\textup{min} \tr{E[\vect{u}(n-1)\vect{u}^T(n-1)]}\\
 &= J_\textup{min} \tr{\vect{R}_u}\\
 &= J_\textup{min} \tr{\vect{\Lambda}}\ .
\end{align}
The second equality follows since the trace of a scalar equals the value of the scaler. The third equality follows from a property of the trace operator given by
\bmath
  \tr{\vect{AB}} = \tr{\vect{BA}}
\emath
for any pair of matrices $\vect{A}$ and $\vect{B}$ with $\vect{A}$ and $\vect{B}^T$ being of the same dimension. The fourth equality follows since both the expectation and the trace operators are linear operators whose order can therefore be interchanged. Finally, the last equality follows since
\bmath
  \tr{\vect{R}_u} = \tr{\vect{X\Lambda X}^T} = \tr{\vect{\Lambda X}^T\vect{X}} = \tr{\vect{\Lambda}}\ .
\emath
This was previously stated in the beginning of Lecture~\ref{ch:SD_LMS}.

In order to evaluate the expectation of the first term in \eq{eq:app_lms_first_recur_start}, we use the law of iterated expectations given by\footnote{It follows since
\begin{align*}
  E[E[X|Y]] &= \int E[X|Y]f_Y(y)dy = \int \left[\int x f_{X|Y}(x|y)dx\right]f_Y(y)dy = \int\int x f_{X,Y}(x,y)dxdy\\
  &= \int x \left[\int f_{X,Y}(x,y)dy\right]dx = \int x f_X(x)dx = E[X]\ .
\end{align*}}
\bmath
  E[X] = E[E[X|Y]]\ ,
\emath
and the assumption that $\vect{u}(n)$ and $\vect{w}(n)$ are uncorrelated. From this, we obtain that
\begin{align}
   E[\vect{c}^T(n-1)\vect{X}^T[\vect{I}-\mu &\vect{u}(n-1)\vect{u}^T(n-1)]^2\vect{X}\vect{c}(n-1)] = \notag\\
   &E[\vect{c}^T(n-1)\vect{X}^TE[(\vect{I}-\mu \vect{u}(n-1)\vect{u}^T(n-1))^2]\vect{X}\vect{c}(n-1)]\ .
\end{align}
The inner expectation is on the same form as \eq{eq:app_lms_4mom} with $\vect{W}=\vect{F}=\vect{I}$. Thus, we have that
\begin{align}
  \vect{S}(1) &= \vect{X}^TE[(\vect{I}-\mu \vect{u}(n-1)\vect{u}^T(n-1))^2]\vect{X}\\
  &= \vect{I}-2\mu\vect{\Lambda}+\mu^2[\vect{\Lambda}\tr{\vect{\Lambda}}+2\vect{\Lambda}^2]\ .
\end{align}
which is a diagonal matrix. Thus, we can write \eq{eq:app_lms_first_recur_start} as
\bmath
  E[\|\vect{c}(n)\|^2] = E[\vect{c}^T(n-1)\vect{S}(1)\vect{c}(n-1)]+\mu^2J_\textup{min}\tr{\vect{\Lambda}}\ .
  \label{eq:app_lms_first_recur}
\emath

\subsubsection{The Second Recursion}
Since the second term of \eq{eq:app_lms_first_recur} does not depend on $\vect{c}(n)$, we consider the first term of \eq{eq:app_lms_first_recur}. Multiplying \eq{app:lms_weight_error} for time index $n-1$ from the left with $\vect{X}^T$ and inserting it into the first term yields
\begin{align}
  E[&\vect{c}^T(n-1)\vect{S}(1)\vect{c}(n-1)] = \notag\\
  &E[\vect{c}^T(n-2)\vect{X}^T[\vect{I}-\mu\vect{u}(n-2)\vect{u}^T(n-2)]\vect{X}\vect{S}(1)\vect{X}^T[\vect{I}-\mu\vect{u}(n-2)\vect{u}^T(n-2)]\vect{X}\vect{c}(n-2)]\notag\\
  &{}+\mu^2 E[v^2(n)\vect{u}^T(n-2)\vect{X}\vect{S}(1)\vect{X}^T\vect{u}(n-2)]\ .
  \label{eq:app_lms_second_recur_start}
\end{align}
The expectation of the last term is given by
\begin{align}
 E[v^2(n)\vect{u}^T(n-2)\vect{X}\vect{S}(1)\vect{X}^T\vect{u}(n-2)] &= E[v^2(n)]E[\vect{u}^T(n-2)\vect{X}\vect{S}(1)\vect{X}^T\vect{u}(n-2)]\notag\\
 &= J_\textup{min} E[\tr{\vect{u}^T(n-2)\vect{X}\vect{S}(1)\vect{X}^T\vect{u}(n-2)}]\notag\\
 &= J_\textup{min} \tr{\vect{X}\vect{S}(1)\vect{X}^TE[\vect{u}(n-2)\vect{u}^T(n-2)]}\notag\\
 &= J_\textup{min}\tr{\vect{S}(1)\vect{\Lambda}}\ .
\end{align}
The equalities in the derivation above follows from the same arguments as in the derivation for the equivalent expression in the first recursion. Using the same arguments as in the first recursion, the first term of \eq{eq:app_lms_second_recur_start} may be written as an outer and an inner expectation. The inner expectation can be evaluated using \eq{eq:app_lms_4mom} with $\vect{W}=\vect{XS}(1)\vect{X}^T$ or $\vect{F}=\vect{S}(1)$. Thus, we have that $\vect{S}(2)$ is given by
\begin{align}
  \vect{S}(2) &= \vect{X}^TE\left[[\vect{I}-\mu\vect{u}(n-2)\vect{u}^T(n-2)]\vect{X}\vect{S}(1)\vect{X}^T[\vect{I}-\mu\vect{u}(n-2)\vect{u}^T(n-2)]\right]\vect{X}\notag\\
  &= \vect{S}(1) - 2\mu\vect{\Lambda}\vect{S}(1)+\mu^2[\vect{\Lambda}\tr{\vect{S}(1)\vect{\Lambda}}+2\vect{\Lambda}\vect{S}(1)\vect{\Lambda}]
\end{align}
which is diagonal since $\vect{S}(1)$ is diagonal. Thus, in total we have that
\begin{align}
    E[\|\vect{c}(n)\|^2] &= E[\vect{c}^T(n-1)\vect{S}(1)\vect{c}(n-1)] +\mu^2J_\textup{min}\tr{\vect{\Lambda}}\\
    &= E[\vect{c}^T(n-2)\vect{S}(2)\vect{c}(n-2)]+\mu^2J_\textup{min}\tr{\vect{S}(1)\vect{\Lambda}}+\mu^2J_\textup{min}\tr{\vect{\Lambda}}\\
    &= E[\vect{c}^T(n-2)\vect{S}(2)\vect{c}(n-2)]+\mu^2J_\textup{min}\tr{[\vect{S}(1)+\vect{I}]\vect{\Lambda}}\ .
    \label{eq:app_lms_second_recur}
\end{align}

\subsubsection{The Third Recursion}
Since the second term of \eq{eq:app_lms_second_recur} does not depend on $\vect{c}(n)$, we consider the first term of \eq{eq:app_lms_second_recur}. This has the same form as the first term of \eq{eq:app_lms_first_recur} so the derivation of the third recursion is the same as for the second recursion, except that all the time indices should be decreased by one. We therefore obtain that
\begin{align}
    E[\|\vect{c}(n)\|^2] &= E[\vect{c}^T(n-1)\vect{S}(1)\vect{c}(n-1)] +\mu^2J_\textup{min}\tr{\vect{\Lambda}}\\
    &= E[\vect{c}^T(n-2)\vect{S}(2)\vect{c}(n-2)]+\mu^2J_\textup{min}\tr{[\vect{S}(1)+\vect{I}]\vect{\Lambda}}\\
    &= E[\vect{c}^T(n-3)\vect{S}(3)\vect{c}(n-3)]+\mu^2J_\textup{min}\tr{[\vect{S}(2)+\vect{S}(1)+\vect{I}]\vect{\Lambda}}\ .
    \label{eq:app_lms_third_recur}
\end{align}

\subsubsection{The $n$'th Recursion}
From the first, second, and third recursion, it is not hard to see a pattern for the recursions. Therefore, for the $n$'th recursion, we have that
\begin{align}
    E[\|\vect{c}(n)\|^2] &= \vect{c}^T(0)\vect{S}(n)\vect{c}(0)+\mu^2J_\textup{min}\sum_{i=0}^{n-1}\tr{\vect{S}(i)\vect{\Lambda}}
    \label{eq:app_lms_nth_recur}
\end{align}
where
\bmath
  \vect{S}(n) = \vect{S}(n-1) - 2\mu\vect{\Lambda}\vect{S}(n-1)+\mu^2[\vect{\Lambda}\tr{\vect{S}(n-1)\vect{\Lambda}}+2\vect{\Lambda}\vect{S}(n-1)\vect{\Lambda}]
  \label{eq:app_lms_Sn_recur}
\emath
with $\vect{S}(0) = \vect{I}$. From the recursion in \eq{eq:app_lms_nth_recur}, we see that we must require that $\vect{c}^T(0)\vect{S}(n)\vect{c}(0)$ remains bounded for $n\to\infty$, regardless of the initial conditions $\vect{c}(0)$. Since $\vect{c}^T(0)\vect{S}(n)\vect{c}(0)$ remains bounded if and only if all the eigenvalues of $\vect{S}(n)$ are in the interval $[-1,1]$, we therefore perform an eigenvalue analysis of $\vect{S}(n)$.

\subsubsection{Eigenvalue Analysis of $\vect{S}(n)$}
Define the vector of ones
\bmath
  \vect{1} = \bbmtx 1 & 1 & \cdots & 1\ebmtx^T\ .
\emath
Using this vector, we may write the trace of a diagonal matrix $\vect{M}$ as
\bmath
  \tr{\vect{M}} = \vect{1}^T\vect{M1}\ .
\emath
Since $\vect{S}(n)$ and $\vect{\Lambda}$ are diagonal matrices, the product $\vect{S}(n)\vect{\Lambda}$ is also diagonal, and we have that
\begin{align}
  \vect{S}(n-1)\vect{\Lambda} &= \vect{\Lambda}\vect{S}(n-1)\\
  \tr{\vect{S}(n)\vect{\Lambda}} &= \vect{1}^T\vect{S}(n)\vect{\Lambda 1} = \vect{1}^T\vect{\Lambda}\vect{S}(n)\vect{1}\ .
  \label{eq:app_lms_tr_diag}
\end{align}
Multiplying both sides of \eq{eq:app_lms_Sn_recur} from the right with $\vect{1}$, we obtain
\begin{align}
  \vect{S}(n)\vect{1} &= \vect{S}(n-1)\vect{1} - 2\mu\vect{\Lambda}\vect{S}(n-1)\vect{1}+\mu^2[\vect{\Lambda}\vect{1}\vect{1}^T\vect{\Lambda}\vect{S}(n-1)\vect{1}+2\vect{\Lambda}^2\vect{S}(n-1)\vect{1}]\\
  &= [\vect{I}-2\mu\vect{\Lambda}+\mu^2(\vect{\Lambda}\vect{1}\vect{1}^T\vect{\Lambda}+2\vect{\Lambda}^2)]\vect{S}(n-1)\vect{1}\ .
\end{align}
Now, we define
\bmath
  \vect{D} = \vect{I}-2\mu\vect{\Lambda}+\mu^2(\vect{\Lambda}\vect{1}\vect{1}^T\vect{\Lambda}+2\vect{\Lambda}^2)
  \label{eq:app_lms_D}
\emath
so that we have
\begin{align}
  \vect{S}(n)\vect{1} &= \vect{D}\vect{S}(n-1)\vect{1}\\
  &\underset{\vdots}{=} \vect{D}^2\vect{S}(n-2)\vect{1}\\
  & = \vect{D}^n\vect{S}(0)\vect{1}
  \label{eq:app_lms_S_D_recur}
\end{align}
This means that the elements of the diagonal matrix $\vect{S}(n)$ are bounded if and only if all the eigenvalues of $\vect{D}$ are in the interval $[-1,1]$.

\subsubsection{Eigenvalue Analysis of $\vect{D}$}
We can write $\vect{D}$ as
\bmath
  \vect{D} = 2(\vect{I}-\mu\vect{\Lambda})^2+2\mu\vect{\Lambda}+\mu^2\vect{\Lambda}\vect{1}\vect{1}^T\vect{\Lambda}
\emath
which is clearly positive semidefinite since $\mu>0$ and $\lambda_m > 0$. Thus, all eigenvalues of $\vect{D}$ are non-negative, and the only requirement to the eigenvalues is therefore that they are smaller than one. This is equivalent to requiring that $\vect{I}-\vect{D}$ is positive definite. From Sylvester's criterion, we know that an $M\times M$ matrix $\vect{A}$ is positive definite if and only if the determinant of all the upper $m\times m$ matrices $\vect{A}_m$ of $\vect{A}$ are positive for $m = 1,\ldots,M$. If $\vect{1}_m$ is a vector consisting of $m$ ones, $\vect{I}_m$ is the $m\times m$ identity matrix, and $\vect{\Lambda}_m$ and $\vect{D}_m$ are the upper $m\times m$ matrices of $\vect{\Lambda}$ and $\vect{D}$, respectively, we have that
\bmath
  \vect{D}_m = \vect{I}_m - 2\mu\vect{\Lambda}_m+\mu^2\vect{\Lambda}_m(\vect{1}_m\vect{1}_m^T+2\vect{I}_m)\vect{\Lambda}_m\ ,\qquad \text{for }m=1,\ldots,M\ ,
\emath
and we require that
\bmath
  |\vect{I}_m - \vect{D}_m| > 0\ ,\qquad \text{for }m=1,\ldots,M
\emath
for $\vect{I}-\vect{D}$ to be positive definite. The determinant of $\vect{I}_m - \vect{D}_m$ is
\begin{align}
  |\vect{I}_m-\vect{D}_m| &= |2\mu\vect{\Lambda}_m-\mu^2\vect{\Lambda}_m(\vect{1}_m\vect{1}_m^T+2\vect{I}_m)\vect{\Lambda}_m|\\
  &= |\mu^2\vect{\Lambda}_m||2\mu^{-1}\vect{I}_m-2\vect{\Lambda}_m - \vect{1}_m\vect{1}_m^T\vect{\Lambda}_m|
\end{align}
Since $\mu>0$ and $\lambda_m > 0$, the determinant of $\mu^2\vect{\Lambda}_m$ is positive for all $m=1,\ldots,M$. In order to evaluate the second determinant, we use the matrix determinant lemma which states that
\bmath
  |\vect{A}+\vect{U}\vect{V}^T| = |\vect{A}||\vect{I}+\vect{V}^T\vect{A}^{-1}\vect{U}|
\emath
where $\vect{A}$ is an $N\times N$ matrix, and $\vect{U}$ and $\vect{V}$ are $N\times M$ matrices. From the matrix determinant lemma, we have that
\begin{align}
  |2\mu^{-1}\vect{I}_m-2\vect{\Lambda}_m - \vect{1}_m\vect{1}_m^T\vect{\Lambda}_m| &= |2\mu^{-1}\vect{I}_m-2\vect{\Lambda}_m||1-\frac{1}{2}\vect{1}_m^T\vect{\Lambda}_m(\mu^{-1}\vect{I}_m-\vect{\Lambda}_m)^{-1}\vect{1}_m|
\end{align}
The argument of the first determinant is a diagonal matrix. Thus, it leads to
\begin{alignat}{2}
  && 0 &< 2\mu^{-1}-2\lambda_m\ ,\qquad \text{for }m=1,\ldots,M\\
  \ArrowBetweenLines
  && \mu &< \frac{1}{\lambda_\textup{max}}\ .
  \label{eq:app_lms_u1}
\end{alignat}
The argument of the second determinant is a scalar. It leads to
\begin{alignat}{2}
  && 0 &< 1-\frac{1}{2}\vect{1}_m^T\vect{\Lambda}_m(\mu^{-1}\vect{I}_m-\vect{\Lambda}_m)^{-1}\vect{1}_m\ ,\qquad \text{for }m=1,\ldots,M\\
  \ArrowBetweenLines
  && 1 &> \frac{1}{2}\vect{1}_m^T\vect{\Lambda}_m(\mu^{-1}\vect{I}_m-\vect{\Lambda}_m)^{-1}\vect{1}_m\ ,\qquad \text{for }m=1,\ldots,M\\
  && &= \frac{1}{2}\tr{\vect{\Lambda}_m(\mu^{-1}\vect{I}_m-\vect{\Lambda}_m)^{-1}}\ ,\qquad \text{for }m=1,\ldots,M\\
  && &= \frac{1}{2}\sum_{i=1}^m\frac{\lambda_i}{\mu^{-1}-\lambda_i}\ ,\qquad \text{for }m=1,\ldots,M\\
  && &= \frac{\mu}{2}\sum_{i=1}^m\frac{\lambda_i}{1-\mu\lambda_i} = f_m(\mu)\ ,\qquad \text{for }m=1,\ldots,M\ .
\end{alignat}
The first bound on the step-size in \eq{eq:app_lms_u1} ensures that $\lambda_i/(1-\mu\lambda_i)$ is always positive. Thus,
\bmath
  f_1(\mu) < f_2(\mu) < \cdots < f_M(\mu)\ ,\qquad\text{for }\mu\in[0,\lambda_\textup{max}^{-1}]\ .
\emath
Therefore, if the step-size satisfies
\bmath
  f(\mu) = f_M(\mu) < 1\ ,
\emath
then all of the functions $f_m(\mu)$ are also smaller than one. Moreover, $f(0) = 0$, and $f(\mu)$ is an increasing function as long as the first bound on the step-size in \eq{eq:app_lms_u1} is satisfied. The latter follows since the derivative of $f(\mu)$ satisfies
\bmath
  \frac{df}{d\mu} = \frac{1}{2}\sum_{m=1}^M\frac{\lambda_m}{(1-\mu\lambda_m)^2} > 0\ ,\qquad\text{for }\mu\in[0,\lambda_\textup{max}^{-1}]\ .
\emath
These observations lead to the conclusion that $\vect{I} - \vect{D}$ is positive definite, provided that the step-size satisfies the bound
\bmath
  \boxed{f(\mu) = \frac{\mu}{2}\sum_{m=1}^M\frac{\lambda_m}{1-\mu\lambda_m} < 1}\ .
  \label{eq:app_lms_u2}
\emath
In matrix notation, we can write this bound as
\bmath
  \boxed{f(\mu) = \frac{1}{2}\vect{1}^T\vect{\Lambda}(\mu^{-1}\vect{I}-\vect{\Lambda})^{-1}\vect{1} =  \frac{1}{2}\tr{\vect{\Lambda}(\mu^{-1}\vect{I}-\vect{\Lambda})^{-1}}< 1}
  \label{eq:app_lms_u2_matrix}
\emath
where the last equality follows since $\vect{\Lambda}(\mu^{-1}\vect{I}-\vect{\Lambda})^{-1}$ is diagonal. Moreover, since
\bmath
  \lim_{\mu\to \lambda_\textup{max}^{-1}} f(\mu) = \infty > 1\ ,
\emath
the first bound in \eq{eq:app_lms_u1} is always satisfied if the second bound in \eq{eq:app_lms_u2} is satisfied. Thus, we have shown that the LMS algorithm converges in the mean-square if and only if the step-size satisfies \eq{eq:app_lms_u2} or, equivalently, \eq{eq:app_lms_u2_matrix}.

\subsection{Learning Curve}
The value of the cost function at time $n$ is given by
\begin{align}
  J_1(\vect{w}(n)) &= E[e^2(n)] = E[(v(n)+\vect{u}^T(n)\vect{\Delta}\vect{w}(n))^2]\\
  &= E[\vect{\Delta}\vect{w}^T(n)\vect{u}(n)\vect{u}^T(n)\vect{\Delta}\vect{w}(n)]+J_\textup{min}
\end{align}
where the last equality follows from the fact that $v(n)$ and $\vect{u}(n)$ are uncorrelated. If we also use the law of iterated expectations and that $\vect{u}(n)$ and $\vect{w}(n)$ are uncorrelated, we obtain that
\begin{align}
  J_1(\vect{w}(n)) &= E[\vect{\Delta}\vect{w}^T(n)E[\vect{u}(n)\vect{u}^T(n)|\vect{w}(n)]\vect{\Delta}\vect{w}(n)]+J_\textup{min}\\
  &= E[\vect{\Delta}\vect{w}^T(n)E[\vect{u}(n)\vect{u}^T(n)]\vect{\Delta}\vect{w}(n)]+J_\textup{min}\\
  &= E[\vect{\Delta}\vect{w}^T(n)\vect{R}_u\vect{\Delta}\vect{w}(n)]+J_\textup{min}\ .
\end{align}
Finally, if we replace the correlation matrix $\vect{R}_u$ of $\vect{u}(n)$ with its eigenvalue decomposition $\vect{R}_u=\vect{X\Lambda X}^T$, we have that
\bmath
  J_1(\vect{w}(n)) = E[\vect{c}^T(n)\vect{\Lambda}\vect{c}(n)]+J_\textup{min}
\emath
where we again have defined that $\vect{c}(n)= \vect{X}^T\vect{\Delta}\vect{w}(n)$. Now, for $\vect{S}(0)=\vect{\Lambda}$, we obtain from \eq{eq:app_lms_nth_recur} that
\bmath
    \boxed{J_1(\vect{w}(n)) = \vect{c}^T(0)\vect{S}(n)\vect{c}(0)+\mu^2J_\textup{min}\sum_{i=0}^{n-1}\tr{\vect{S}(i)\vect{\Lambda}}+J_\textup{min}}
    \label{eq:app_lms_learning_curve}
\emath
where
\begin{align}
  \vect{S}(n) &= \vect{S}(n-1) - 2\mu\vect{\Lambda}\vect{S}(n-1)+\mu^2[\vect{\Lambda}\tr{\vect{S}(n-1)\vect{\Lambda}}+2\vect{\Lambda}\vect{S}(n-1)\vect{\Lambda}]\\
  &= \diag{\vect{S}(n)\vect{1}} = \diag{\vect{D}^n\vect{\Lambda}\vect{1}}\ .
\end{align}
Here, $\diag{\cdot}$ creates a diagonal matrix from a vector.

\section{Steady-State Analysis}
In the steady-state analysis, we derive expressions for the mean-square deviation (MSD), the excess mean-square error (EMSE), and the misadjustment of the LMS algorithm. These expressions can be derived in two different ways. One way is presented in \cite[pp.~462--465]{Sayed2003}. Here, we give another derivation which we believe is more intuitive.

\subsection{Mean-Square Deviation}
The MSD is given by the limit
\bmath
  \text{MSD:}\qquad  \lim_{n\to\infty} E[\|\vect{\Delta w}(n)\|^2]\ ,
\emath
and we denote this limit by $E[\|\vect{\Delta w}(\infty)\|^2]$. We make direct use of the definition in the derivation of the MSD. Again, we define $\vect{c}(n)= \vect{X}^T\vect{\Delta}\vect{w}(n)$. Thus, from \eq{eq:app_lms_nth_recur}, we have that
\begin{align}
  E[\|\vect{\Delta w}(\infty)\|^2] &= \lim_{n\to\infty} E[\|\vect{\Delta w}(n)\|^2] = \lim_{n\to\infty} E[\|\vect{c}(n)\|^2]\\
  &= \lim_{n\to\infty} \left\{\vect{c}^T(0)\vect{S}(n)\vect{c}(0)+\mu^2J_\textup{min}\sum_{i=0}^{n-1}\tr{\vect{S}(i)\vect{\Lambda}}\right\}\\
  &= \lim_{n\to\infty} \vect{c}^T(0)\vect{S}(n)\vect{c}(0) + \lim_{n\to\infty} \mu^2J_\textup{min}\sum_{i=0}^{n-1}\tr{\vect{S}(i)\vect{\Lambda}}\ .
\end{align}
with $\vect{S}(0)=\vect{I}$. If we select the step-size $\mu$ such that the LMS algorithm converges in the mean-square, the first term equals zero. Thus, we have that
\begin{align}
  E[\|\vect{\Delta w}(\infty)\|^2] &= \lim_{n\to\infty} \mu^2J_\textup{min}\sum_{i=0}^{n-1}\tr{\vect{S}(i)\vect{\Lambda}} = \mu^2J_\textup{min}\sum_{i=0}^{\infty}\tr{\vect{S}(i)\vect{\Lambda}}\ .
\end{align}
The matrices $\vect{S}(i)$ and $\vect{\Lambda}$ are both diagonal, and we therefore use \eq{eq:app_lms_tr_diag} to obtain
\begin{align}
  E[\|\vect{\Delta w}(\infty)\|^2] &= \mu^2J_\textup{min}\sum_{i=0}^{\infty}\vect{1}^T\vect{\Lambda}\vect{S}(i)\vect{1}\\
  &= \mu^2J_\textup{min}\sum_{i=0}^{\infty}\vect{1}^T\vect{\Lambda}\vect{D}^i\vect{1}\\
  &= \mu^2J_\textup{min}\vect{1}^T\vect{\Lambda}\sum_{i=0}^{\infty}\left[\vect{D}^i\right]\vect{1}
\end{align}
where the second equality follows from \eq{eq:app_lms_S_D_recur} with  $\vect{S}(0)=\vect{I}$ and $\vect{D}$ defined in \eq{eq:app_lms_D}. Since all the eigenvalues of $\vect{D}$ have a magnitude smaller than 1, we have from the geometric series of matrices that \cite[p.~58]{Petersen2008}
\bmath
  \sum_{i=0}^{\infty}\vect{D}^i = (\vect{I}-\vect{D})^{-1}\ .
\emath
Thus, we obtain that
\begin{align}
  E[\|\vect{\Delta w}(\infty)\|^2] &= \mu^2J_\textup{min}\vect{1}^T\vect{\Lambda}(\vect{I}-\vect{D})^{-1}\vect{1}\\
  &= \mu^2J_\textup{min}\vect{1}^T\vect{\Lambda}(2\mu\vect{\Lambda}-\mu^2(\vect{\Lambda}\vect{1}\vect{1}^T\vect{\Lambda}+2\vect{\Lambda}^2))^{-1}\vect{1}\\
  &= \mu^2J_\textup{min}\vect{1}^T\vect{\Lambda}[(2\vect{\mu}^{-1}\vect{I}-2\vect{\Lambda}-\vect{\Lambda}\vect{1}\vect{1}^T)\mu^2\vect{\Lambda}]^{-1}\vect{1}\\
  &= J_\textup{min}\vect{1}^T(2\vect{\mu}^{-1}\vect{I}-2\vect{\Lambda}-\vect{\Lambda}\vect{1}\vect{1}^T)^{-1}\vect{1}\ .
\end{align}
Now, by defining
\bmath
  \vect{G} = \vect{\mu}^{-1}\vect{I}-\vect{\Lambda}
\emath
and using the matrix inversion lemma from \eq{eq:woodbury}, we obtain
\begin{align}
  E[\|\vect{\Delta w}(\infty)\|^2] &= J_\textup{min}\vect{1}^T(2\vect{G}-\vect{\Lambda}\vect{1}\vect{1}^T)^{-1}\vect{1}\\
  &= J_\textup{min}\vect{1}^T\left[\frac{1}{2}\vect{G}^{-1}+\frac{1}{2}\vect{G}^{-1}\vect{\Lambda}\vect{1}\left(1-\frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{\Lambda}\vect{1}\right)^{-1}\vect{1}^T\vect{G}^{-1}\frac{1}{2}\right]\vect{1}\\
  &= J_\textup{min}\frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{1}\left[1+\left(1-\frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{\Lambda}\vect{1}\right)^{-1}\frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{\Lambda}\vect{1}\right]\\
  &= J_\textup{min}\frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{1}\left(1-\frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{\Lambda}\vect{1}\right)^{-1}\ .
\end{align}
Finally, from \eq{eq:app_lms_u2_matrix}, we have that
\bmath
  f(\mu) = \frac{1}{2}\vect{1}^T\vect{\Lambda}\vect{G}^{-1}\vect{1} = \frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{\Lambda}\vect{1}
\emath
which leads to
\bmath
  \boxed{ E[\|\vect{\Delta w}(\infty)\|^2] = \frac{J_\textup{min}}{1-f(\mu)}\frac{\mu}{2}\sum_{m=1}^M\frac{1}{1-\mu\lambda_m}}\ .
\emath

\subsection{Excess Mean-Square Error}
The derivation of the EMSE is done in the same way as the MSE was derived. The EMSE is given by the limit
\bmath
  \text{EMSE:}\qquad  \lim_{n\to\infty} J_1(\vect{w}(n))-J_\textup{min}\ ,
\emath
and we denote it by $J_\textup{ex}$. Inserting the expression for the learning curve from \eq{eq:app_lms_learning_curve} in this limit yields
\begin{align}
  J_\textup{ex} &= \lim_{n\to\infty} \left\{\vect{c}^T(0)\vect{S}(n)\vect{c}(0)+\mu^2J_\textup{min}\sum_{i=0}^{n-1}\tr{\vect{S}(i)\vect{\Lambda}}\right\}\\
  &= \lim_{n\to\infty} \vect{c}^T(0)\vect{S}(n)\vect{c}(0)+\lim_{n\to\infty}\mu^2J_\textup{min}\sum_{i=0}^{n-1}\tr{\vect{S}(i)\vect{\Lambda}}
\end{align}
where $\vect{S}(0)=\vect{\Lambda}$ and $\vect{c}(n)= \vect{X}^T\vect{\Delta}\vect{w}(n)$. If we select the step-size $\mu$ such that the LMS algorithm converges in the mean-square, the first term equals zero. Thus, we have that
\begin{align}
  J_\textup{ex} &= \lim_{n\to\infty} \mu^2J_\textup{min}\sum_{i=0}^{n-1}\tr{\vect{S}(i)\vect{\Lambda}} = \mu^2J_\textup{min}\sum_{i=0}^{\infty}\tr{\vect{S}(i)\vect{\Lambda}}\ .
\end{align}
Note that the expression for the EMSE is the same as for the MSD, except for the value of $\vect{S}(0)$. The matrices $\vect{S}(i)$ and $\vect{\Lambda}$ are both diagonal, and we therefore use \eq{eq:app_lms_tr_diag} to obtain
\begin{align}
  J_\textup{ex} &= \mu^2J_\textup{min}\sum_{i=0}^{\infty}\vect{1}^T\vect{\Lambda}\vect{S}(i)\vect{1}\\
  &= \mu^2J_\textup{min}\sum_{i=0}^{\infty}\vect{1}^T\vect{\Lambda}\vect{D}^i\vect{\Lambda}\vect{1}\\
  &= \mu^2J_\textup{min}\vect{1}^T\vect{\Lambda}\sum_{i=0}^{\infty}\left[\vect{D}^i\right]\vect{\Lambda}\vect{1}
\end{align}
where the second equality follows from \eq{eq:app_lms_S_D_recur} with $\vect{S}(0)=\vect{\Lambda}$ and $\vect{D}$ defined in \eq{eq:app_lms_D}. Since all the eigenvalues of $\vect{D}$ have a magnitude smaller than 1, we have from the geometric series of matrices that \cite[p.~58]{Petersen2008}
\bmath
  \sum_{i=0}^{\infty}\vect{D}^i = (\vect{I}-\vect{D})^{-1}\ .
\emath
Thus, we obtain that
\begin{align}
  J_\textup{ex} &= \mu^2J_\textup{min}\vect{1}^T\vect{\Lambda}(\vect{I}-\vect{D})^{-1}\vect{\Lambda}\vect{1}\\
  &= \mu^2J_\textup{min}\vect{1}^T\vect{\Lambda}(2\mu\vect{\Lambda}-\mu^2(\vect{\Lambda}\vect{1}\vect{1}^T\vect{\Lambda}+2\vect{\Lambda}^2))^{-1}\vect{\Lambda}\vect{1}\\
  &= \mu^2J_\textup{min}\vect{1}^T\vect{\Lambda}[(2\vect{\mu}^{-1}\vect{I}-2\vect{\Lambda}-\vect{\Lambda}\vect{1}\vect{1}^T)\mu^2\vect{\Lambda}]^{-1}\vect{\Lambda}\vect{1}\\
  &= J_\textup{min}\vect{1}^T(2\vect{\mu}^{-1}\vect{I}-2\vect{\Lambda}-\vect{\Lambda}\vect{1}\vect{1}^T)^{-1}\vect{\Lambda}\vect{1}\ .
\end{align}
Now, by defining
\bmath
  \vect{G} = \vect{\mu}^{-1}\vect{I}-\vect{\Lambda}
\emath
and using the matrix inversion lemma from \eq{eq:woodbury}, we obtain
\begin{align}
  J_\textup{ex} &= J_\textup{min}\vect{1}^T(2\vect{G}-\vect{\Lambda}\vect{1}\vect{1}^T)^{-1}\vect{\Lambda}\vect{1}\\
  &= J_\textup{min}\vect{1}^T\left[\frac{1}{2}\vect{G}^{-1}+\frac{1}{2}\vect{G}^{-1}\vect{\Lambda}\vect{1}\left(1-\frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{\Lambda}\vect{1}\right)^{-1}\vect{1}^T\vect{G}^{-1}\frac{1}{2}\right]\vect{\Lambda}\vect{1}\\
  &= J_\textup{min}\frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{\Lambda}\vect{1}\left[1+\left(1-\frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{\Lambda}\vect{1}\right)^{-1}\frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{\Lambda}\vect{1}\right]\\
  &= J_\textup{min}\frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{\Lambda}\vect{1}\left(1-\frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{\Lambda}\vect{1}\right)^{-1}\ .
\end{align}
Finally, from \eq{eq:app_lms_u2_matrix}, we have that
\bmath
  f(\mu) = \frac{1}{2}\vect{1}^T\vect{\Lambda}\vect{G}^{-1}\vect{1} = \frac{1}{2}\vect{1}^T\vect{G}^{-1}\vect{\Lambda}\vect{1}
\emath
which leads to
\bmath
  \boxed{ J_\textup{ex} = J_\textup{min}\frac{f(\mu)}{1-f(\mu)}}\ .
\emath

\subsection{Misadjustment}
The expression for the misadjustment is
\bmath
  \boxed{\mathcal{M} = \frac{J_\textup{ex}}{J_\textup{min}} = \frac{f(\mu)}{1-f(\mu)}}\ .
\emath


\section{Examples of Conformal Structures; Analytic Functions}
\subsection{Covering Maps and Analyticity}
\begin{lemma}\label{covering_conformal}
    If $\pi:\tilde{R}\to R$ is a covering map where $R$ is a Riemann surface, then there is a unique conformal structure on $\tilde{R}$ such that $\pi$ is analytic.
\end{lemma}
\begin{proof}
    We construct an atlas on $\tilde{R}$ as follows:
    For $p\in\tilde{R}$, as $\pi$ is a covering map, there is an open neighbourhood $\tilde{N}_p$ of $p$ such that $\pi|_{\tilde{N}_p}:\tilde{N}_p\to N_p$ is a homeomorphism for some neighbourhood $N(p)$ of $\pi(p)$.
    Now there is a conformal structure on $R$, so we can pick a chart $(\phi_p,U_p)$ on $R$ such that $p\in U_p$.
    Our desired chart is then $(\tilde{\phi}_p,\tilde{U}_p)$ where $\tilde{U}_p=(\pi|_{\tilde{N}_p})^{-1}(N_p\cap U_p)$ and $\tilde{\phi}_p=\phi_p|_{N_p\cap U_p}\circ\pi|_{\tilde{U}_p}$.
    It is easy to see it is a chart.
    The collection of all charts produced in this form is an atlas.
    To see this, just observe that the transition function is
    $$\tilde{\phi}_p\circ\tilde{\phi}_q^{-1}=\phi_p\circ\pi\circ\pi^{-1}\circ\phi_q^{-1}=\phi_p\circ\phi_q^{-1}$$
    (with proper restrictions to local open sets) which is analytic as we got $\phi_p,\phi_q$ from another atlas.
    So it extends to a conformal structure $\tilde{A}$ on $\tilde{R}$.\\
    We shall show that $\pi$ is analytic in this choice of conformal structure.
    Let $p\in\tilde{R}$ and choose chart $(\phi_p,U_p)$ in $R$ around $\pi(p)$ and chart $(\tilde{\phi}_p,\tilde{U}_p)$ in $\tilde{R}$ constructed from $(\phi_p,U_p)$ in the above way.
    Then with proper restrictions,
    $$\phi_p\circ\pi\circ\tilde{\phi}_p^{-1}=\phi_p\circ\pi\circ\pi^{-1}\circ\phi_p^{-1}=\operatorname{id}_{\mathbb C}$$
    which is analytic.
    We are then done by Lemma \ref{analytic_local}.\\
    To see the uniqueness of this conformal structure, let $\tilde{B}$ be any conformal structure on $\tilde{R}$ such that $\pi$ is analytic.
    Let $p\in\tilde{R}$, $(\psi,V)\in\tilde{B}$ a chart around $p$ and $(\phi_p,U_p)$ a chart around $f(p)$.
    But then locally the transition function $\tilde\phi_p\circ\psi^{-1}=\phi_p\circ\pi\circ\psi^{-1}$ is analytic.
    So by maximality $\tilde{B}=\tilde{A}$.
\end{proof}
\begin{example}
    Consider the Riemann surface $R$ associated with $\log$ we considered earlier.
    Let $f,\pi$ be the functions as usual.
    As $\pi$ is a covering map, there is a unique way to make $R$ a Riemann surface by the preceding lemma where $\pi:R\to C_\star$ is analytic.
    Furthermore, locally $f|_{U_{I(n)}}=f_{I(n)}\circ\pi$, so $f$ is analytic as well.\\
    Furthermore, we know that there is a homeomorphism $f_{I(n)}:U_{I(n)}\to\tilde{V}_{I(n)}$ (where $\tilde{V}_I=\mathbb R+iI$) having $\exp|_{\tilde{V}_{I(U)}}$ as inverse.
    Then $f_{I(n)}^{-1}$ agree wherever their domains intersect, so we can piece them together to give a conformal equivalence between $R$ and $\mathbb C$.
\end{example}
\begin{example}
    Similar case happened with $R_k$ and $\sqrt[k]{\cdot}$.
    Again $\pi$ induces a unique conformal structure on $R_k$ that makes it and $g$ analytic.
    By the same argument as above, we get $g$ is a conformal equivalence.\\
    Actually, we can even do better with this example.
    Note that the singularities in $0$ and $\infty$ are removable by identifying $\hat{p}_k(0)=\hat{\pi}(0)=\hat{g}(0)=0$ and $\hat{p}_k(\infty)=\hat{\pi}(\infty)=\hat{g}(\infty)=\infty$ (where $\hat{g},\hat{p}_k,\hat{\pi}$ are our notation for $g,p_k,\pi$ with this extended domain and codomain).
    This gives
    \[
        \begin{tikzcd}
            \hat{R}_k=R_k\cup\{0,\infty\}\arrow{r}{\hat{g}}\arrow[swap]{dr}{\hat{\pi}}&\mathbb C_\infty\arrow{d}{\hat{p}_k}\\
            &\mathbb C_\infty
        \end{tikzcd}
    \]
    which is pretty nice except now $\hat{\pi}$ is not a covering map anymore.
\end{example}