\chapter{Inference for Statistical Experiments}\label{S:StatExps}
\section{Introduction}\label{S:ExpsIntro}
We formalize the notion of a staistical experiment.  Let us first motivate the need for a statistical experiment.  Recall that statistical inference or learning is the process of using observations or data to infer the distribution that generated it.  A generic question is:
\[
\text{Given realizations from $X_1, X_2, \ldots, X_n \sim$ some unknown DF $F$, how do we infer $F$} ?
\]
However, to make this question tractable or even sensible it is best to restrict ourselves to a particular  class or family of DFs that may be assumed to contain the unknown DF $F$.

\begin{definition}[Experiment]
A statistical experiment $\EE{E}$ is a set of probability distributions (DFs, PDFs or PMFs)
$\Pz := \{\P_{\theta} : \theta \in \BB{\Theta} \}$ associated with a RV $X$ and indexed by the set $\BB{\Theta}$.
We refer to $\BB{\Theta}$ as the parameter space or the index set and $d:\BB{\Theta} \rightarrow \Pz$ that associates to each $\theta \in \BB{\Theta}$ a probability $\P_{\theta} \in \Pz$ as the index map:
\[ \BB{\Theta} \ni \theta \mapsto \P_{\theta} \in \Pz \enspace .\]
\end{definition}

\section{Some Common Experiments}
Next, let's formally consider some experiments we have already encountered.
\begin{Exp}[The Fundamental Experiment]\label{Exp:Uniform01}
The `uniformly pick a number in the interval $[0,1]$' experiment is the following singleton family of DFs  :
\[
\Pz = \{ \,  F(x) = x \BB{1}_{[0,1]}(x)  \, \}
\]
where, the only distribution $F(x)$ in the family $\Pz$ is a re-expression of~\eqref{E:Uniform01DF} using the indicator function $\BB{1}_{[0,1]}(x)$.  The parameter space of the fundamental experiment is a singleton whose DF is its own inverse, ie.~$F(x) = F^{[-1]}(x)$.
Recall from Exercise~\ref{underMPSA} that this is equivalent to infinitely many independent and identical $\bernoulli(1/2)$ trials, i.e., independently tossing a fair coint infinitely many times.
%The two dimensional parameter space or index set for this experiment is $\BB{\Theta} = \{ -\infty < a < b < \infty \} = \{ (a,b) \in \Rz \times \Rz :  a < b \}$, a half-plane.
\end{Exp}

\begin{Exp}[Bernoulli]\label{Exp:Bernoulli}
The `toss 1 times' experiment is the following family of densities (PMFs) :
\[
\Pz = \{ \,  f(x; \theta) :  \theta \in [0,1] \, \}
\]
where, $f(x; \theta)$ is given in~\eqref{E:Bernoullipdf}.  The one dimensional parameter space or index set for this experiment is  $\BB{\Theta} = [0,1] \subset \Rz$.
\end{Exp}

\begin{Exp}[Point~Mass]\label{Exp:PointMass}
The `deterministically choose a specific real number' experiment is the following family of DFs :
\[
\Pz = \{ \,  F(x; a) :  a \in \Rz \, \}
\]
where, $F(x; a)$ is given in~\eqref{E:PointMasscdf}.  The one dimensional parameter space or index set for this experiment is $\BB{\Theta} = \Rz$, the entire real line.
\end{Exp}
Note that we can use the PDF's or the DF's to specify the family $\Pz$ of an experiment.  When an experiment can be parametrized by finitely many parameters it is said to a {\bf parametric} experiment.  \hyperref[Exp:Bernoulli]{Experiment~\ref*{Exp:Bernoulli}} involving discrete RVs as well as \hyperref[Exp:PointMass]{Experiment \ref*{Exp:PointMass}} are {\bf parametric} since they both have only one parameter (the parameter space is one dimensional for Experiments \ref*{Exp:Bernoulli} and \ref*{Exp:PointMass}).   The \hyperref[Exp:Uniform01]{Fundamental Experiment \ref*{Exp:Uniform01}} involving the continuous RV of \hyperref[M:Uniform01]{Model \ref*{M:Uniform01}} is also parametric since its parameter space, being a point, is zero-dimensional.  The next example is also parametric and involves $(k-1)$-dimensional families of discrete RVs.

\begin{Exp}[{de~Moivre[k]}]\label{Exp:GenDiscrete}
The `pick a number from the set $[k] := \{1,2,\ldots,k\}$ somehow' experiment is the following family of densities (PMFs) :
\[
\Pz = \{ \,  f(x; \theta_1,\theta_2,\ldots,\theta_k) :   (\theta_1,\theta_2,\ldots,\theta_k) \in \bigtriangleup_k \, \}
\]
where, $f(x; \theta_1,\theta_2,\ldots,\theta_k)$ is any PMF such that
\[
f(x; \theta_1,\theta_2,\ldots,\theta_k) = \theta_x, \qquad x \in \{1,2,\ldots,k\} \ .
\]
The $k-1$ dimensional parameter space $\BB{\Theta}$ is the $k$-Simplex $\bigtriangleup_k$.  This as an `exhaustive' experiment since all possible densities over the finite set $[k] := \{1,2,\ldots,k\}$ are being considered that can be thought of as ``the outcome of rolling a convex polyhedral die with $k$ faces and an arbirtary center of mass specified by the $\theta_i$'s.''
\begin{figure}
\caption{Geometry of the $\BB{\Theta}$'s for $\demoivre[k]$ Experiments with $k \in \{1, 2, 3, 4\}$.}
\vspace{5cm}
\end{figure}
\end{Exp}
An experiment with infinite dimensional parameter space $\BB{\Theta}$ is said to be {\bf nonparametric} .  Next we consider two nonparametric experiments.
\begin{Exp}[All DFs]\label{Exp:AllDFs}
The `pick a number from the Real line in an arbitrary way' experiment is the following family of distribution functions (DFs) :
\[
\Pz = \{ \,  F(x; F) :  F~is~a~DF \, \} = \BB{\Theta}
\]
where, the DF $F(x; F)$ is indexed or parameterized by itself. Thus, the parameter space
\[
\BB{\Theta}=\Pz=\{ \text{all DFs}\}
\]
is the infinite dimensional space of {\bf All DFs} ''.
\end{Exp}
Next we consider a {\bf nonparametric} experiment involving continuous RVs.
\begin{Exp}[Sobolev Densities]\label{Exp:Sob}
The `pick a number from the Real line in some reasonable way' experiment is the following family of densities (pdfs) :
\[
\Pz = \left\{ \,  f(x; f) :  \int(f''(x))^2 < \infty \, \right\} = \BB{\Theta}
\]
where, the density $f(x; f)$ is indexed by itself.  Thus, the parameter space $\BB{\Theta}=\Pz$ is the infinite dimensional {\bf Sobolev space} of ``not too wiggly functions''.
\end{Exp}

\section{Typical Decision Problems with Experiments}
Some of the concrete problems involving experiments include:
\begin{itemize}
\item {\bf Simulation:} Often it is necessary to simulate a RV with some specific distribution to gain insight into its features or simulate whole systems such as the air-traffic queues at `London Heathrow' to make better management decisions.
\item {\bf Estimation:}
\begin{enumerate}
\item {\bf Parametric Estimation:} Using samples from some unknown DF $F$ parameterized by some unknown $\theta$, we can estimate $\theta$ from a statistic $T_n$ called the estimator of $\theta$ using one of several methods (maximum likelihood, moment estimation, or parametric bootstrap).
\item {\bf Nonparametric Estimation of the DF:}  Based on $n$ IID observations from an unknown DF $F$, we can estimate it under the general assumption that $F \in \{ \text{all DFs} \}$.
\item {\bf Confidence Sets:}  We can obtain a $1-\alpha$ confidence set for the point estimates, of the unknown parameter $\theta \in \BB{\Theta}$ or the unknown  DF $F \in \{ \text{all DFs} \}$
\end{enumerate}
\item {\bf Hypothesis Testing:}  Based on observations from some DF $F$ that is hypothesized to belong to a subset $\BB{\Theta}_0$ of $\BB{\Theta}$ called the space of null hypotheses, we will learn to test (attempt to reject) the falsifiable null hypothesis that $F \in \BB{\Theta}_0 \subset \BB{\Theta}$.
\item $\ldots $
\end{itemize}

\section{Decision Problems and Procedures for Actions}\label{S:Decisions}

Write down the Table from lectures 1 \& 2 giving examples of decision problems, procedures and action spaces for typical estimation, hypothesis testing and prediction problems with associated principles (Maximum Likelihood, Empirical Risk Minimisation where risk is expectation of specific loss functions, etc.) and algorithms (including optimisation (Stochastic)Newton/gradient-descent, etc.).

\vspace{10cm}~\\


\section{Algorithm Analysis}

%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Algorithm analysis}
  \begin{itemize}
	\item Given a problem $P$
	\item Design algorithms $A, A', \ldots$
	\item Input space $\mathcal{X}_{n}$: inputs of size $n$
  \end{itemize}

  \pause
  \[
	T_{A}(n) \triangleq \max_{X \in \mathcal{X}_{n}} T_A(X)
  \]

  \pause
  \[
	T_{P}(n) \triangleq \min_{A \text{ solves } P}T_{A}(n) \pause = \min_{A \text{ solves } P} \max_{X \in \mathcal{X}_{n}} T_A(X)
  \]
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Algorithm analysis}
  \begin{columns}
	\column{0.40\textwidth}
	  Sorting:
	  \begin{align*}
		\onslide<2->{& n! \\}
		\onslide<3->{& \implies n^2 \\}
		\onslide<4->{& \implies n \log n \\}
		\onslide<6->{n \log n & \Longleftarrow \\}
		\onslide<5->{& n \\}
	  \end{align*}
	\column{0.40\textwidth}
	  Selection (median):
	  \begin{align*}
		\onslide<7->{& n^2 \\}
		\onslide<8->{& \implies n \log n \\}
		\onslide<9->{& \implies 16n \\}
		\onslide<10->{& \implies 2.95n \\}
		\onslide<13->{2n & \Longleftarrow \\}
		\onslide<12->{\frac{3n}{2} - \frac{3}{2} \log n & \Longleftarrow \\}
		\onslide<11->{& n \\}
	  \end{align*}
  \end{columns}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%


\documentclass[twoside]{MATH77}
\usepackage{multicol}
\usepackage[fleqn,reqno,centertags]{amsmath}
\begin{document}
\begmath 2.20  Binomial Coefficients

\silentfootnote{$^\copyright$1997 Calif. Inst. of Technology,
  \thisyear \ Math \`a la Carte, Inc.}

\subsection{Purpose}

Compute the binomial coefficient $\displaystyle \binom{n}{k} =
\frac{n!}{(n-k)!\, k!}$ which gives the number of ways one can select
$k$ items from a set containing $n$ items.  Binomial coefficients also
appear in a number of other contexts.

\subsection{Usage}

\subsubsection{Program Prototype, Single Precision}
\begin{description}
\item[REAL] \ {\bf SBINOM, ANS}
\item[INTEGER] \ {\bf N, K}
\end{description}
Assign values to N and K.
$
\fbox{\bf ANS = SBINOM(N, K)}
$

\subsubsection{Argument Definitions}

\begin{description}
\item[N, K] \ [in] The input integers $n$ and $k$ as described in
"Purpose" above.  One must have $0 \leq \text{K} \leq \text{N}.$
\end{description}

\subsubsection{Modifications for Double Precision}

For double-precision usage change the name SBINOM to DBINOM, and
change the REAL declaration to DOUBLE PRECISION.

\subsection{Examples and Remarks}

The program DRSBINOM compares the true values with the results of
computing the binomial coefficients for two large values of N where SBINOM
uses a different algorithm internally.  It then finds the first case where
results using the Pascal triangle give different results from SBINOM (the
true value for the case printed is 67863915, which is slightly closer to
the value given by SBINOM).  If one is computing binomial coefficients for
strictly increasing values of N, one may want to use the Pascal triangle
code in DRSBINOM instead of SBINOM for computing the binomial
coefficients.

\subsection{Functional Description}

For N $\leq$ 150, the routine maintains and uses a table of saved
factors of factorials, $\phi_j$.  If the input N is larger than any
seen so far, this table is extended.  When there is no problem with
overflow, $\phi_j = j \phi_{j-1}$.  To deal with overflow a second
table $i_\nu$ is maintained.  Initially $\nu=0$.  When adding entry
$j$ to the table would cause an overflow, $\nu$ is incremented,
$i_\nu$ is set to $j-1$, and $\phi_j$ to $j$.

To compute the binomial coefficients, the code uses
\begin{equation*}
\binom{n}{k} = \frac{n!}{(n-k)!\,k!}, \quad
j!  = \Big (\prod_{\ell = 1}^{\nu-1} \phi(i_{\ell}) \Big ) \phi (j),
\end{equation*}
with $\nu $ such that $ i_{\nu-1} < j \leq i_\nu$.  When a factorial is so
large it is represented by more than one piece, the pieces are combined in
such a way as to avoid overflow.  Finally, for large values, the result is
refined by setting the result equal to $p \times \lfloor r / p + .5
\rfloor $, where $r$ is the originally computed result, $p$ is either one
prime or the product of two primes in the interval $\big (\max(n-k,\ k),\
n \big ]$ and $ \lfloor \cdot \rfloor$ denotes the integer part.

In the case of IEEE single precision arithmetic, only one piece of the
factorial is needed for $\text{N} \leq 33$, and for IEEE double precision
$150!$ does not overflow.  When only one piece is needed for the factorial
and it is already computed, the binomial coefficient is computed with one
multiply, one divide, and a few tests.

If N $>$ 150, the log gamma function from Chapter~2.3 is used.  In this
case
\begin{equation*}
\binom{n}{k} = e^{\log \Gamma (n+1) - \log \Gamma (n-k+1) -
\log \Gamma (k+1)}
\end{equation*}
It is very easy to shorten the code by replacing the call to the log
gamma function for large N with an error message reporting too large an N.
It is only slightly more difficult to replace the code for keeping pieces
of the factorial with code that calls log gamma whenever N!\ would
overflow.  This would lose some precision as illustrated by the difference
in the errors for N = 150 and N = 151 in the results.

\subsection{Error Procedures and Restrictions}

An error exit is taken if the condition $0 \leq \text{K} \leq \text{N}$
is not satisfied, or if the result would overflow.  Errors are
processed using the routines in Chapter 19.2.  If one references one
of these routines to change the error action, then instead of stopping on
an error, the routine will return with a result of $-1$.

\subsection{Supporting Information}

The source language is ANSI Fortran~77.

\begin{tabular}{@{\bf}l@{\hspace{5pt}}l}
\bf Entry & \hspace{.35in} {\bf Required Files}\vspace{2pt} \\
DBINOM & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, DBINOM, DERM1, DERV1, DLGAMA, DGAMMA, ERFIN, ERMSG, IERM1,
IERV1\rule[-5pt]{0pt}{8pt}}\\
SBINOM & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, ERFIN, ERMSG, IERM1, IERV1, SBINOM, SERM1, SERV1, SLGAMA,
SGAMMA\rule[-5pt]{0pt}{8pt}}\\ \end{tabular}

Algorithm and code due to F. T. Krogh, JPL, December~1995.

\begcodenp

\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRSBINOM}\vspace{10pt}
\lstinputlisting{\codeloc{sbinom}}

\vspace{20pt}\centerline{\bf \large ODSBINOM}\vspace{0pt}
\lstset{language={}}
\lstinputlisting{\outputloc{sbinom}}
\end{document}


%!TEX root = ../main.tex
%-------------------------------------------------------------------------------
\subsection{Calibration procedure}
%-------------------------------------------------------------------------------
EKW models are calibrated to data on observed individual decisions and experiences under the hypothesis that the individual behaves according to the model. The goal is to back out information on utility functions, preference parameters, and transition probabilities. This requires the full parameterization $\theta$ of the model.

Economists have access to information for $i = 1, \hdots, N$ individuals in each time period $t = 1, \dots, T_i$. For every observation $(i, t)$ in the data, we observe the action $a_{it}$, some components $\bar{u}_{it}$ of the utility, and a subset $\bar{s}_{it}$ of the state $s_{it}$. Therefore, from an economist's point of view, we need to distinguish between two types of state variables $s_{it} = (\bar{s}_{it}, \epsilon_{it})$. At time $t$, the economist and individual both observe $\bar{s}_{it}$ while $\epsilon_{it}$ is only observed by the individual. In summary, the data $\mathcal{D}$ has the following structure:
%
\begin{align*}
  \mathcal{D} = \{a_{it}, \bar{s}_{it}, \bar{u}_{it}: i = 1, \hdots, N; t = 1, \hdots, T_i\},
\end{align*}
where $T_i$ is the number of observations for which we observe individual $i$.

Numerous calibration procedures for different settings exist \citep{Davidson.2003, Gourieroux.1996}. We briefly outline likelihood-based and simulation-based calibration. Independent of the calibration criterion, it is necessary to solve for the optimal policy $\pi^*$ at each candidate parameterization of the model.

Likelihood-based calibration seeks to find the parameterization $\hat{\theta}$ that maximizes the likelihood function $\mathcal{L}(\theta\mid\mathcal{D})$, i.e. the probability of observing the given data as a function of $\theta$. As we only observe a subset $\bar{s}_t$ of the state, we can determine the probability $p_{it}(a_{it}, \bar{u}_{it} \mid \bar{s}_{it}, \theta)$ of individual $i$ at time $t$ in $\bar{s}_{it}$ choosing $a_{it}$ and receiving $u_{it}$ given parametric assumptions about the distribution of $\epsilon_{it}$. The objective function takes the following form:
%
\begin{align*}
  \hat{\theta} \equiv \argmax_{\theta \in \Theta}  \underbrace{\prod^N_{i= 1} \prod^{T_i}_{t= 1}\, p_{it}(a_{it}, \bar{u}_{it} \mid \bar{s}_{it}, \theta)}_{\mathcal{L}(\theta\mid\mathcal{D})}.
\end{align*}

\noindent In simulation-based calibration, our goal is to find the parameterization $\hat{\theta}$ that yields a simulated data set from the model that closest resembles the observed data. More precisely, the goal is often to minimize the weighted squared distance between a set of moments $M_D$ computed on the observed data and the same set of moments computed on the simulated data $M_S(\theta)$. The objective function takes the following form:
%
\begin{align*}
    \hat{\theta} \equiv \argmin_{\theta \in \Theta} (M_D - M_S(\theta))' W (M_D - M_S(\theta)).
\end{align*}


\chapter{Trees}
\label{chapter:trees}
Let us consider the following problem. Given a network of several computers,
in this network if a computer $A$ receives some message from a computer $B$,
it broadcasts it to all the connected computers except $B$.
However, in such setting there is an issue known as broadcast radiation.
Assume we have three computers $A$, $B$, and $C$ such that they form a cycle.
If $A$ sends something to $B$ and $C$ both of them send received information to
$C$ and $B$, respectively; after that $B$ and $C$ send this information to $A$
and $A$ start sending this information again, which leads to an infinite
cycle.\footnote{%
  This problem is a simplified version of a problem that is solved by STP
  protocols in the modern networks.
}

Therefore to avoid such problem we need to disable some connecion so that the
graph of this network does not have cycles.
In this chapter we are going to study properties of the graphs without cycles.
\begin{definition}
  We say that a connected graph $G$ is a \emph{tree} iff $G$ does not have cycles.
\end{definition}

\section{Minimally Connected Graphs}
First we may make the following observation.
\begin{theorem}
  Let $G = (V, E)$ be a connected graph. Then the following statements are
  equivalent.
  \begin{itemize}
    \item $G$ is a tree.
    \item $G$ is minimally connected, that is, $G - e$ is not connected for any
      $e \in E$.
  \end{itemize}
\end{theorem}
\begin{proof}
  Assume that $G$ is minimally connected but $G$ has a cycle
  $v_1$, \dots, $v_k$. Consider $G' = G - (v_1, v_k)$, we claim that $G'$
  is still connected. Indeed, let $x$ and $y$ be some vertices of $G'$.
  Since $G$ is connected, there is a path $p$ from $x$ to $y$. If $p$ does not
  contain the edge $(v_1, v_k)$, then $x$ and $y$ are connected in $G'$. If
  $p$ contains $(v_1, v_k)$, then we replace this edge by the path $v_1$, \dots,
  $v_k$, so $x$ and $y$ are connected in $G'$. Therefore $G$ is not a minimally
  connected graph, which is a contradiction.

  Let us now assume that $G$ is not minimally connected, we wish to prove that
  it implies that $G$ is not a tree. Since $G$ is not minimally connected, there
  is an edge $(x, y) \in E$ such that $G - e$ is connected. Since $G - (x, y)$
  is connected, there is a path $x = v_1, \dots, v_k = y$ in $G - (x, y)$.
  Therefore $v_1, \dots, v_k$ is a cycle in $G$, which is a contradiction.
 \end{proof}

Therefore in order to get a tree from a graph, we just need to delete edges
in an arbitrary way until the moment when we cannot delete them anymore.
\begin{corollary}
  For any connected graph $G = (V, E)$, there is a tree $T = (V, E')$ such
  that $T$ is a subgraph of $G$. Such a tree is called a spanning tree of $G$.
\end{corollary}

Another question we may ask is how many edges we need to delete in this process.
Apparenly, the answer is always $m - n + 1$, where $m$ is the number of edges in
the initial graph and $n$ is the number of vertices.
\begin{theorem}
\label{theorem:tree-is-minimally-connected}
  Let $G$ be a connected graph on $n$ vertices.
  If $G$ is a tree, then it has $n - 1$ edges. Moreover, if $G$ has $n - 1$
  edge, then it is a tree.
\end{theorem}

Before we prove the theorem, let us prove the following lemma.
\begin{lemma}
  If a tree $T$ has at least $2$ vertices, then it has at least two
  vertices whose degree is $1$.
\end{lemma}
\begin{proof}
  Let us choose a vertex $v$ of $T$ such that its degree is not $1$ (if such a
  vertex does not exist, then we found at least $2$ vertices whose degree is
  $1$). Let us start walking from $v$ to its neighbour, then to a new neighbor
  of this neighbor, and so on, never revisiting a vertex. As $T$ has finite
  number of vertices, we will eventually have to stop at a vertex $u$. We claim
  that the only reason for us to stop at $u$ could be that $u$ is of degree $1$.
  Indeed, the only possible other reason would be that $u$ has neighbors other
  than the neighbor $u'$ we reached $u$ from, but they have all been visited
  already. However, that would mean that there are at least two paths from $v$
  to $u$, and that cannot happen in a tree. So $u$ is of degree $1$. To get
  another vertex of degree $1$, remember that $v$ is of degree more than $1$.
  So take another neighbor of $v$, and repeat this argument. This will result in
  another vertex $w$ of degree $1$, and $u \neq w$ as that would again yield two
  paths from $v$ to $u$.
\end{proof}
The vertices of a tree that have degree $1$ are called \emph{leaves}.

\begin{proof}[Proof of Theorem~\ref{theorem:tree-is-minimally-connected}]
  We prove the statement using induction by $n$. If $n = 1$, the statement is
  clearly true.
  Assume that the statement is true for trees on $n$ vertices. Consider a tree
  $T$ on $n + 1$ vertices. Consider a leaf $\ell$ of $T$. Note that $T - \ell$
  is a tree as well, therefore by the induction hypothesis, it has $n - 2$
  edges. Hence, $T$ has $n - 1$ edges.

  Let us now prove that if a graph $G$ has $n - 1$ edges and is connected,
  then $G$ is a tree. Assume that it is not a tree, we start deleting edges as
  long as the graph is connected, we call the resulting graph $T$. Note that
  $T$ is minimally connected, so $T$ is a tree. Note that $T$ has $n$ vertices.
  Therefore, it has $n - 1$ eges, which implies that we removed $0$ edges and
  $T = G$. As a result, $G$ is a tree.
\end{proof}

\begin{exercise}
  A graph such that every connected component of this graph is a tree is called
  a \emph{forest}.
  Show that a forest with $k$ connected components has $n - k$ edges.
\end{exercise}

\section{Minimum-weight Spanning Trees}
In the initial example about the network, we missed an important detail: not
all the connections are equally fast. Let us label each connection
(edge in our graph) with the weight (the number that represents how slow is this
connection). So now we need to choose a spanning tree of
the graph of the network so that it has the minimal possible sum of weights.
\begin{definition}
  Let $G = (V, E)$ be a connected graph, and $w : E \to \R$ be weights of edges.
  Then we say that a spanning tree $T = (V, E')$ of $G$ is a
  \emph{minimum-weight spanning tree} of $G$ if
  $\sum_{e \in E'} w(e) \le \sum_{e \in E''} w(e)$ for any spanning tree
  $T' = (V, E'')$ of $G$.

  The number $\sum_{e \in E'} w(e)$ is called the \emph{weight} of $T$.
\end{definition}

It is obvious that such a tree exists. The question is
``how to find efficiently the minimum-weight spanning tree''.
\begin{exercise}
  Let $G = (V, E)$ be some graph and $w : E \to \R$ be a weight function such
  that $w(e) = 1$. How to to find efficiently the minimum-weight spanning tree
  of $G$?
\end{exercise}

Surprisingly, one may find such a minimum-weight spanning tree using a simple
greedy algorithm (Algorithm~\ref{algorithm:kruskal}).
\begin{algorithm}
  \begin{algorithmic}[1]
    \Function{MinimumSpanningTree}{$n$, $E$, $w$}
      \State Let $e_1$, \dots, $e_m$ be the edges from $E$ sorted in the
        ascending order with respect to $w$.
      \State $i \gets 1$
      \State Set $T$ to be an empty graph on $[n]$.
      \label{line:kruskal-while}
      \While{$i \le n$}
        \If{$T + e_i$ does not have cycles}
          \State $T \gets T + e_i$
        \EndIf
        \State Increase $i$ by $1$.
      \EndWhile
      \State \Return{$T$}
    \EndFunction
  \end{algorithmic}
  \caption{Kruskal's algorithm, the algorithm that returns a minimum-weight
  spanning tree of the graph on $[n]$ with the set of edges $E$.}
  \label{algorithm:kruskal}
\end{algorithm}

\begin{theorem}
\label{theorem:kruskal}
  If the graph $([n], E)$ is connected, then Algorithm~\ref{algorithm:kruskal}
  returns a minimum-weight spanning tree of the graph $([n], E)$.
\end{theorem}

To prove this statement we need a technical lemma.
\begin{lemma}
  Let $F_1$ and $F_2$ be forests on the same vertex set $V$. If $F_1$ has less
  edges than $F_2$, then $F_2$ has an edge $e$ not in $F_1$ so
  that the graph $F_1 + e$ is still a forest.
\end{lemma}
\begin{proof}
  Let $E_i$ be the set of edges of $F_i$.
  Assume that there such edge does not exist; i.e., $F_1 + e$ has a cycle for
  any edge $e \in E_2 \setminus E_1$.

  Therefore any edge of $F_2$ is between two vertices in the same component of
  $F_1$. Hence, $F_2$ has at least as many connected components as $F_1$.
  Indeed, consider two connected components $U_1$ and $U_2$ of $F_1$ we claim
  that they any $x \in U_1$ and $y \in U_2$ are not connected in $F_2$ since
  there are no edges going outside of $U_1$ and $U_2$ in $F_2$.

  However, $F_i$ has $n - |E_i|$ connected components, which contradicts to the
  fact that $|E_1| < |E_2|$.
\end{proof}

\begin{proof}[Proof of Theorem~\ref{theorem:kruskal}]
  Let $T_1$, \dots, $T_m$ be the states of $T$ after iterations of
  line~\ref{line:kruskal-while} of Algorithm~\ref{algorithm:kruskal}.
  Note that $T_i$ does not have cycles for $i \in \range{m}$. Therefore $T_i$ is a
  forest for $i \in \range{m}$.

  First we need to prove that Algorithm~\ref{algorithm:kruskal} returns a
  spanning tree; i.e. that $T_m$ is connected.
  Assume the opposite. Consider two vertices $x, y \in \range{n}$ such that
  they are not connected in $T$. Since $G = ([n], E)$ is connected there is a
  path $x = v_1, \dots, v_k = y$ in $G$. Consider the minimal $i \in [k - 1]$
  such that $(v_i, v_{i + 1})$ is not an edge of $T_m$. Let
  $e_j = (v_i, v_{i + 1})$. It is easy to see that $T_m + (v_i, v_{i + 1})$ does
  not have cycles so $T_{j - 1} + e_j$ does not have cycles as well and $T_j =
  T_{j - 1} + e_j$ which implies that $T_m$ has the edge $(v_i, v_{i + 1})$
  which is a contradiction.

  Before we start the second part of the proof note that if $w(e) < w(e_i)$,
  then $T_{i - 1} + e_i$ has a cycle.

  Now we need to prove that $T_m$ is a minimum-weight spanning tree. Assume
  that there is a spanning tree $H$ such that the weight of $H$ is less than
  the weight of $T_m$. Consider the edges $t_1$, \dots, $t_{n - 1}$ of $T_m$
  and the edges $h_1$, \dots, $h_{n - 1}$ of $H$ such that
  $w(t_1) \le w(t_2) \le \dots \le w(t_{n - 1})$ and
  $w(h_1) \le w(h_2) \le \dots \le w(h_{n - 1})$.
  Let us consider the first step when $H$ is better than $T_m$; i.e., the
  minimal $i$ so that $\sum_{j = 1}^i w(h_j) < \sum_{j = 1}^i w(t_j)$
  (obviously $i > 1$).

  It is easy to see that $h_i < t_i$. Let $e_j = t_i$ and
  $H_i = H[h_1, \dots, h_i]$. Since $H_i$ has more edges than $T_j$ there is an
  edge $h_{i'}$ ($i' < i$) such that $T_{j - 1} + h_{i'}$ does not have cycles.
  Wich is a contradiction since $h_{i'} < h_i < t_i$.
\end{proof}

\begin{chapterendexercises}
  \exercise[recommended] Let $G$ be a graph with $k$ connected components and
    $n - k$ edges. Show that $G$ is a forest.
  \exercise Prove that if $G$ is a simple graph on $[n]$, then at least one of
    $G$ and its complement is connected. Show an example when they are both
    connected. The complement $\bar{G}$ of $G$ has the same vertex set as $G$
    and $(x, y)$ is an edge in $\bar{G}$ if and only if it is not an edge in
    $G$.
  \exercise[recommended] Let $H$ be a simple graph on $n$ vertices that has $m$
    edges. Prove that $H$ contains at least $m - n - 1$ cycles.
    \begin{solution}
      First of all, let us consider the case when $G$ is connected. We prove
      this statement using induction by $d = m - n + 1$. Note that if $d = 0$,
      then the statement is obviously true (since number of cycles should be at
      least $0$).

      Let us prove now the induction step: let $G$ be a graph having $m > n - 1$
      edges. It is not a tree, hence there is a cycle. Let $e$ be any edge of this
      cycle. Note that $G - e$ has $m - 1$ edges, hence, there are at least $m -
      1 - n + 1$ cycles, additionally, note that number of cycles in $G$ is at
      least number of cycles in $g - e$ plus one. As a result, there are at
      least $m - 1 - n + 1$ cycles in $G$.

      If $G$ is not connected, we denote connected component of $G$ by $G_1$,
      \dots, $G_\ell$ with number of vertices $n_1$, \dots, $n_\ell$ and number
      of edges $m_1$, \dots, $m_\ell$. We prove that there are
      $\sum_{i = 1}^\ell m_i - n_i + 1 = m - n + \ell \ge m - n + 1$ cycles in
      $G$.
    \end{solution}
\end{chapterendexercises}


\subsection{End Behaviour and Comparative Growth Rates}

Let us now look at the last two subsections and go deeper. In the last two
subsections we looked at horizontal and slant asymptotes. Both are special
cases of the end behaviour of functions, and both concern situations where
the graph of a function approaches a straight line as $x\to \infty$
or $-\infty$. But not all functions have this kind of end behviour. For
example, $f(x)=x^2$ and $f(x)=x^3$ do not
approach a straight line as $x\to \infty$ or $-\infty$. The best
we can say with the notion of limit developed at this stage are that%
\begin{eqnarray*}
	\lim_{x\to \infty}x^2 &=&\infty\text{, }\lim_{x\to -\infty}x^2=\infty, \\
	\lim_{x\to \infty}x^3 &=&\infty\text{, }\lim_{x\to -\infty}x^3=-\infty.
\end{eqnarray*}%
Similarly, we can describe the end behaviour of transcendental functions
such as $f(x)=e^x$ using limits, and in this case, the graph
approaches a line as $x\to -\infty$ but not as $x\to \infty$.
\begin{equation*}
\lim_{x\to -\infty}e^x=0\text{, }\lim_{x\to \infty}e^x=\infty.
\end{equation*}

People have found it useful to make a finer distinction between these end
behaviours all thus far captured by the symbols $\infty$ and $-\infty$.
Specifically, we will see that the above functions have different growth
rates at infinity. Some increases to infinty faster than others.
Specifically,

\begin{definition}{Comparative Growth Rates}{CompareGrowthRates}
Suppose that $f$ and $g$ are two functions such that $\lim\limits_{x%
	\to \infty }f\left( x\right) =\infty $ and $\lim\limits_{x%
	\to \infty }g\left( x\right) =\infty .$ We say that $f\left(
x\right) $ grows faster than $g\left( x\right) $ as $x\to \infty $
if the following holds:%
\begin{equation*}
\lim_{x\to \infty }\frac{f\left( x\right) }{g\left( x\right) }%
=\infty ,
\end{equation*}%
or equivalently,%
\begin{equation*}
\lim_{x\to \infty }\frac{g\left( x\right) }{f\left( x\right) }=0.
\end{equation*}
\end{definition}

Here are a few obvious examples:

\begin{example}{}{ComparingMonomials}
Show that if $m>n$ are two positive integers, then $f(x)=x^m$ grows faster
than $g(x)=x^n$ as $x\to \infty$.
\end{example}
\begin{solution}
Since $m>n,$ $m-n$ is a positive integer. Therefore,%
\begin{equation*}
\lim_{x\to \infty}\frac{f(x)}{g(x)}=\lim_{x\to \infty}\frac{x^m}{x^n}=\lim_{x\to \infty}x^{m-n}=\infty.
\end{equation*}
\end{solution}

\begin{example}{}{CompareMonicPoly}
Show that if $m>n$ are two positive integers, then any monic polynomial
$P_{m}(x)$ of degree $m$ grows faster than any monic polynomial
$P_{n}(x)$ of degree $n$ as $x\to \infty$. [Recall that
a polynomial is monic if its leading coefficient is 1.]
\end{example}
\begin{solution}
By assumption, $P_{m}(x)=x^m+$ terms of degrees less than
$m=x^{m}+a_{m-1}x^{m-1}+\ldots$, and $P_{n}(x)=x^n+$ terms of
degrees less than $n=x^{n}+b_{n-1}x^{n-1}+\ldots$. Dividing the numerator and
denominator by $x^n$, we get%
\begin{equation*}
\lim_{x\to \infty}\frac{f(x)}{g(x)}=\lim_{x\to \infty}\frac{x^{m-n}+a_{m-1}x^{m-n-1}+\ldots}{1+\frac{%
		b_{n-1}}{x}+\ldots}=\lim_{x\to \infty}x^{m-n}(\frac{1+\frac{%
		a_{m-1}}{x}+\ldots}{1+\frac{b_{n-1}}{x}+\ldots}) =\infty,
\end{equation*}%
since the limit of the bracketed fraction is 1 and the limit of $x^{m-n}$ is
$\infty$, as we showed in Example~\ref{exa:ComparingMonomials}.
\end{solution}

\begin{example}{}{HighestDegreeTermGrowth}
Show that a polynomial grows exactly as fast as its highest degree term
as $x\to \infty$ or $-\infty$. That is, if $P(x)$ is
any polynomial and $Q(x)$ is its highest degree term, then both limits
\begin{equation*}
\lim_{x\to \infty}\frac{P(x)}{Q(x)}\text{ and }\lim_{x\to -\infty}\frac{P(x)}{Q(x)}
\end{equation*}%
are finite and nonzero.
\end{example}
\begin{solution}
Suppose that $P(x)=a_{n}x^{n}+a_{n-1}x^{n-1}+\ldots+a_{1}x+a_{0},$ where $a_{n}\neq 0.$ Then the
highest degree term is $Q\left( x\right) =a_{n}x^{n}.$ So,%
\begin{equation*}
\lim_{x\to \infty }\frac{P\left( x\right) }{Q\left( x\right) }%
=\lim_{x\to \infty }\left( a_{n}+\frac{a_{n-1}}{x}+...+\frac{a_{1}}{%
	x^{n-1}}+\frac{a_{0}}{x^{n}}\right) =a_{n}\neq 0.
\end{equation*}
\end{solution}

Let's state a theorem we mentioned when we discussed the last example in the
last subsection:

\begin{theorem}{}{}
Let $n$ be any positive integer and let $a>1$. Then $f(x)=a^x$
grows faster than $g(x)=x^n$ as $x\to \infty$:
\begin{equation*}
\lim_{x\to \infty}\frac{a^x}{x^n}=\infty\text{, }\lim_{x\to \infty}\frac{x^n}{a^x}=0.
\end{equation*}%
In particular,%
\begin{equation*}
\lim_{x\to \infty}\frac{e^x}{x^n}=\infty\text{, }\lim_{x\to \infty}\frac{x^n}{e^x}=0.
\end{equation*}
\end{theorem}

The easiest way to prove this is to use the L'H\^{o}pital's Rule, which we will
introduce in a later chapter. For now, one can plot and compare the graphs
of an exponential function and a power function. Here is a comparison
between $f(x)=x^2$ and $g(x)=2^x$:

\begin{center}
\begin{tikzpicture}[]
\begin{axis}[
	axis lines=middle,
	ymin=0,
	ymax=130,
	xmin=0,
	xmax=7,
	xlabel={$x$},
	]
	\addplot [mark=none,samples=200,red] {x^2};
	\addplot [mark=none,samples=200,blue] {2^x};
\end{axis}
\end{tikzpicture}
\end{center}

Notice also that as $x\to -\infty$, $x^n$ grows in size but
$e^x$ does not. More specifically, $x^n\to \infty$ or $-\infty$
according as $n$ is even or odd, while $e^x\to 0$. So, it is
meaningless to compare their ``growth''
rates, although we can still calculate the limit
\begin{equation*}
\lim_{x\to -\infty}\frac{e^x}{x^n}=0.
\end{equation*}

Let's see an application of our theorem.

\begin{example}{}{}
Find the horizontal asymptote(s) of $f(x)=\dfrac{x^3+2e^x}{e^x-4x^2}$.
\end{example}
\begin{solution}
To find horizontal asymptotes, we calculate the limits of $f(x)$
as $x\to \infty$ and $x\to -\infty$. For $x\to \infty$,
we divide the numerator and the denominator by $e^x$,
and then we take limit to get%
\begin{equation*}
\lim_{x\to \infty}\dfrac{x^3+2e^x}{e^x-4x^2}=%
\lim_{x\to \infty}\dfrac{\frac{x^3}{e^x}+2}{1-4\frac{x^2}{e^x}}%
=\frac{0+2}{1-4(0)}=2.
\end{equation*}%
For $x\to -\infty$, we divide the numerator and the denominator by $x^2$ to get
\begin{equation*}
\lim_{x\to -\infty}\dfrac{x^3+2e^x}{e^x-4x^2}%
=\lim_{x\to -\infty}\dfrac{x+2\frac{e^x}{x^2}}{\frac{e^x}{x^2}-4}.
\end{equation*}%
The denominator now approaches $0-4=-4$. The numerator has limit $-\infty$.
So, the quotient has limit $\infty$:
\begin{equation*}
\lim_{x\to -\infty}\dfrac{x+2\frac{e^x}{x^2}}{\frac{e^x}{x^2}-4}=\infty.
\end{equation*}%
So, $y=2$ is a horizontal asymptote. The function $y=f(x)$
approaches the line $y=2$ as $x\to \infty$. And this is the only
horizontal asymptote, since the function $y=f(x)$ does not
approach any horizontal line as $x\to -\infty$.
\end{solution}

Since the growth rate of a polynomial is the same as that of its leading
term, the following is obvious:

\begin{example}{}{}
If $P(x)$ is any polynomial, then%
\begin{equation*}
\lim_{x\to \infty}\frac{P(x)}{e^x}=0.
\end{equation*}
\end{example}

Also, if $r$ is any real number, then we can place it between two
consecutive integers $n$ and $n+1.$ For example, $\sqrt{3}$ is between 1 and
2, $e$ is between 2 and 3, and $\pi$ is between 3 and 4. Then the following
is totally within our expectation:

\begin{example}{}{}
Prove that if $a>1$ is any basis and $r>0$ is any exponent,
then $f(x)=a^x$ grows faster than $g(x)=x^r$
as $x\to \infty$.
\end{example}
\begin{solution}
Let $r$ be between consecutive integers $n$ and $n+1$. Then for
all $x>1$, $x^{n}\leq x^{r}\leq x^{n+1}$. Dividing by $a^{x}$, we get%
\begin{equation*}
\frac{x^n}{a^x}\leq \frac{x^r}{a^x}\leq \frac{x^{n+1}}{a^x}.
\end{equation*}%
Since
\begin{equation*}
\lim_{x\to \infty}\frac{x^n}{a^x}=0.
\end{equation*}
\end{solution}

What about exponential functions with different bases? We recall from the
graphs of the exponential functions that for any base $a>1$,%
\begin{equation*}
\lim_{x\to \infty}a^x=\infty.
\end{equation*}

So, the exponential functions with bases greater than 1 all grow to infinity
as $x\to \infty$. How do their growth rates compare?

\begin{theorem}{}{}
If $1<a<b$, then $f(x)=b^x$ grows faster than
$g(x)=a^x$ as $x\to \infty$.
\end{theorem}
\begin{proof}
Proof. Since $a<b$, we have $\frac{b}{a}>1$. So,%
\begin{equation*}
\lim_{x\to \infty}\frac{b^x}{a^x}=\lim_{x\to \infty}\left(\frac{b}{a}\right)^x=\infty.
\end{equation*}
\end{proof}

Another function that grows to infinity as $x\to \infty$ is
$g(x)=\ln x$. Recall that the natural logarithmic function is
the inverse of the exponential function $y=e^x$. Since $e^x$ grows very
fast as $x$ increases, we should expect $\ln x$ to grow very slowly as $x$
increases. The same applies to logarithmic functions with any basis $a>1$.
This is the content of the next theorem.

\begin{theorem}{}{}
Let $r$ be any positive real number and $a>1$. Then

\begin{enumerate}[(a)]
\item	$f(x)=x^r$ grows faster than $g(x)=\ln x$ as $x\to \infty$.
\item	$f(x)=x^r$ grows faster than $g(x)=\log_{a}x$ as $x\to \infty$.
\end{enumerate}
\end{theorem}
\begin{proof}
\begin{enumerate}
\item	We use a change of variable. Letting $t=\ln x$, then $x=e^t$.
So, $x\to \infty$ if and only if $t\to \infty$, and
\begin{equation*}
\lim_{x\to \infty}\frac{\ln x}{x^r}=\lim_{t\to \infty}\frac{t}{(e^t)^r}%
=\lim_{t\to \infty}\frac{t}{(e^r)^t}.
\end{equation*}%
Now, since $r>0$, $a=e^r>1$. So, $a^t$ grows as $t$ increases, and it
grows faster than $t$ as $t\to \infty$. Therefore,%
\begin{equation*}
\lim_{x\to \infty}\frac{\ln x}{x^{r}}%
=\lim_{t\to \infty}\frac{t}{(e^r)^t}=\lim_{t\to \infty}\frac{t}{a^t}=0.
\end{equation*}
\item	The change of base identity $\log_{a}x=\dfrac{\ln x}{\ln a}$ implies
that $\log_{a}x$ is simply a constant multple of $\ln x$. The result now
follows from (a).
\end{enumerate}
\end{proof}


\chapter{A Brief Survey of Approaches for Unconstrained Optimization Problems}

\section{Basic Conceptions}
\subsection{Problem Description}
The goal of unconstrained optimization problem is to minimize the objective function f

\begin{eqnarray*}
			\min\limit_{x\in\Rn} \quad f(x),
\end{eqnarray*}
where $f: \Rn \longmapsto \R$.

The objective function f can be convex or nonconvex, differentiable or non-differentiable. The acquirable information include function value, derivative and so on.

On the other hand, the constrained optimization contains some constraints
\begin{eqnarray*}
	    	\min\limits_{x\in\Rn} \quad f(x), \quad \st \quad x\in C.
\end{eqnarray*}
It is equivalent to the unconstrained optimization problem in the way
\begin{eqnarray*}
	    	\min\limits_{X\in\Rn} \,\, f(x) + \delta_C(x), where~  \delta_C(x):=\ifelse{0, & \mbox{if\,}x\in C;\\ 1, & \mbox{otherwise.}}
\end{eqnarray*}
Meanwhile we can have exact penalty functions such as $\ell_1$ penalty, augmented Lagrangian, etc.

%******************************************************
\subsection{Optimality Conditions}
There are several cases as to the optimality conditions.

	\subsubsection{First-order optimality conditions}
When it's the first-order optimality conditions, $f$ is differentiable: $\nabla f(x)=0$. Moreover, $f$ is nondifferentiable but convex
		$$0\in \partial f(x):=\{g\mid f(y)\geq f(x)+ g\zz (y-x),\,\forall\, y \}.$$
	\subsubsection{Second-order necessary (sufficient) optimality conditions}
On the other hand, when the optimality condition is the second-order necessary (sufficient) optimality condition, $f$ is second-order differentiable:
\begin{eqnarray*}
	    	\nabla^2 f(x)\succeq (\succ) 0.
\end{eqnarray*}

	\subsubsection{Optimization condition (differentiability is assumed)}
When only differentiability is assumed, there are many cases.
It is simple when $f$ is convex: $x^*$ is a global minimizer $\Leftrightarrow\,\nabla f(x^*)=0$.

When $f$ is nonconvex:
		\begin{itemize}
			\item \clr{$x^*$ is a first-order stationary point $\Leftrightarrow\,\nabla f(x^*)=0$\qquad
			$\bigstar$}\\[1mm]
			\item \clr{$x^*$ is a local minimizer $\Rightarrow\,\nabla f(x^*)=0$}\\[1mm]
			\item \clg{$x^*$ is a second-order stationary point
				$\Leftrightarrow\,{\color{red}\bigstar}\mbox{\,\,and}\,\,\nabla^2 f(x^*)\succeq 0$}\\[1mm]
			\item \clg{$x^*$ is a local minimizer $\Rightarrow\,\nabla^2 f(x^*)\succeq 0$}\\[1mm]
			\item \clb{$x^*$ is a local minimizer $\Leftarrow\,{\color{red}\bigstar}\mbox{\,\,and}\,\,\nabla^2 f(x^*)\succ 0$}
		\end{itemize}

	\subsubsection{Finding a minimizer (nonconvexness is assumed)}
When the non-convexness is assumed, we have some conclusions
	\begin{itemize}
		\item finding global minimizer is \clb{numerically impossible}
		\item finding global minimizer for quartic polynomial is already \clb{NP-hard}
		\item finding local minimizer is \clb{not easier}
	\end{itemize}

With all the discussions above, the task of numerical optimization methods becomes clear.
As for the first-order methods, it is to find first-order stationary point.
We want to find second-order stationary point for second-order methods. What's more, finding global minimizer or local minimizer becomes possible only when $f$ is structured.

%******************************************************
\subsection{Iterative Methods}
We can use the iterative methods to find the desired stationary points.

	\subsubsection{Framework}
The framework of the iterative methods is as follows.
	\begin{itemize}
		\item[(1)] Input: initial guess $x^0$, tolerance $\epsilon$, $k:=0$;
		\item[(2)] Main iteration: $x^{k+1} = h(x^k)$;
		\item[(3)] Check stopping criterion, if satisfied, then terminate and return $x^{k+1}$;
		otherwise, set $k:=k+1$ and goto step (2).
	\end{itemize}
The stopping criterions is different according to the orders.
For first-order criterion, it is $||\nabla f(x)||<\epsilon$.
It is $\lambda_{\min}(\nabla^2 f(x))>-\epsilon$ for second-order criterion.

	\subsubsection{Choosing $h$}
A question naturally arises that how do we choose an appropriate iteration function h.
For line search methods such as gradient methods and Newton methods, we let $x^{k+1} = x^k + \alpha^k d^k$ with$\alpha^k$ as the stepsize.
The $h$ can be so different as for trust region methods, block coordinate descent methods and so on.

	\subsubsection{Fixed-point convergence -- contraction}
Moreover we can find the desired points with fixed-point convergence if $||\mathcal{J}_h(x)||<1$ holds for a given norm $||\cdot||$ and any $x\in\Rn$, where $\mathcal{J}_h$ stands for the Jacobian of $h$.
But we should note that the following example shows that $\rho\left(\mathcal{J}_h(x)\right)<1$ is not sufficient for nonstationary iteration
\begin{eqnarray*}
	\small \mathcal{J}_h(x^{2k-1}) &=\mat{cc}{0.5 & 10\\ 0 & 0.5},
\\	\mathcal{J}_h(x^{2k}) &=\mat{cc}{0.5 & 0\\ 10 & 0.5},  \forall\, k=1, ...
\end{eqnarray*}

%******************************************************
%\begin{frame}
%	\frametitle{Iterative Methods (Cont'd)}
	\subsubsection{Global convergence -- to stationarity}
However, it is not so easy to find the global stationary points. Some conditions of $f$ need to be met as following
	\begin{itemize}
		\item objective is bounded below: \clg{$f(x)>-\infty$}.
		\item sufficient function value reduction:
		\clr{
		$$f(x^k)-f(x^{k+1})\geq c||\nabla f(x^k)||_2^2.$$
		}
		\vspace{-6mm}

		\item convergence to first-order
		stationarity: \clb{$\lim\limits_{k\rightarrow +\infty} \nabla f(x^k)=0$}
		\end{itemize}
And we should note
		\begin{itemize}
		\item if iterate sequence is bounded, \clb{subsequence convergence to a stationary point}
	\end{itemize}

	\subsubsection{Local convergence}
Suppose we have
	\begin{eqnarray*}
		\lim\limits_{k\rightarrow +\infty} x^k=x^*,\qquad q^k = \frac{||x^{k+1}-x^*||}{||x^{k}-x^*||^p}.
	\end{eqnarray*}
Then the local convergence rate is defined accordingly as following
	\begin{itemize}
		\item \clm{$p=1$}, \clg{\small $\lim\limits_{k\rightarrow +\infty}q^k=q=1$}:
		local Q-sublinear convergence
		\item \clm{$p=1$}, \clg{\small $\lim\limits_{k\rightarrow +\infty}q^k=q\in(0,1)$}:
		local Q-linear convergence
		\item \clm{$p=1$}, \clg{\small $\lim\limits_{k\rightarrow +\infty}q^k=q=0$}:
		local Q-superlinear convergence
		\item \clm{$p>1$}, \clg{\small $\lim\limits_{k\rightarrow +\infty}q^k=q$}:
		local convergence with order $p$
		\begin{itemize}
			\item $p=2$, quadratic
			\item $p=3$, cubic
		\end{itemize}
	\end{itemize}
	\vspace{-5mm}

	\begin{eqnarray*}
		\lim\limits_{k\rightarrow +\infty} x^k=x^*,\qquad
		||x^k-x^*|| \leq cq^k.
	\end{eqnarray*}
	\begin{itemize}
		\item \clg{$q\in(0,1)$}, local R-linear convergence rate
	\end{itemize}

	\subsubsection{Worst case complexity/Global convergence rate}
The  complexity for global convergence rate varies a lot.\\
We get $\epsilon-$solution after $O\left(\log \frac{1}{\epsilon}\right)$ iterations as for global linear convergence. \\
But in the worst case, we get $\epsilon-$solution after $O\left(\frac{1}{\epsilon^{1/q}}\right)$ iterations for global sublinear convergence:
\begin{eqnarray*}
			\lim\limits_{k\rightarrow +\infty} f(x^k)=f^*,\qquad
			f(x^k)-f^*<\frac{c}{k^q},\quad q>0.
\end{eqnarray*}

	\subsubsection{Global convergence -- iterate convergence}
The global convergence can be achieved by the considering iterate convergence in several ways
	\begin{itemize}
		\item {\color{blue} Sufficient reduction:}
		\vspace{-1mm}
		\clm{\begin{eqnarray*}
				f(x^{k})-f(x^{k+1})\geq c_1 ||x^k-x^{k+1}||_2^2.
		\end{eqnarray*}}
		\vspace{-8mm}
		\item {\color{blue} Asmptotic small stepsize safe-guard:}
		\vspace{-1mm}
		\clm{\begin{eqnarray*}
				||x^k-x^{k+1}||_2\geq c_2 ||g^k||_2,\qquad g^k\in\partial f(x^k).
		\end{eqnarray*}}
		\vspace{-8mm}
		\item {\color{blue}\L{}ojasiewicz property:} {$\exists\, \theta\in[0,1)$ such that}
		\vspace{-1mm}
		\clm{
			\begin{eqnarray*}
				|f(x)- f(x^*)|^\theta\leq  c_3||g||_2,\qquad \forall x\in {\cal B}(x^*,\epsilon),
				\quad \forall g\in \partial f(x).
		\end{eqnarray*}}
		\vspace{-6mm}
		\item \clg{iterate convergence:}  \clr{$\sum\limits_{k=1}^\infty ||x^k-x^{k+1}||_2 < +\infty.$}
		\item \clg{local convergence rate}
		\begin{itemize}
			\item if $\theta = 0$, the sequence $\{x^k\}_{k\in\N}$ \clb{finite termination;}\\[1mm]
			\item if $\theta \in \left(0,\frac{1}{2}\right]$, there exist $c>0$ and $Q\in[0,1)$
			such that
			\clb{$||x^k-x^*||_2\leq {c}\cdot{q^k}$;}\\[1mm]
			\item if $\theta \in \left(\frac{1}{2},1\right)$, there exist $c>0$ such that
			\clb{$||x^k-x^*||_2\leq {c}\cdot{k^{-\frac{1-\theta}{2\theta-1}}}$.}
		\end{itemize}
	\end{itemize}

%********************* section ********************
\section{Classical Optimization Methods}
\iffalse
%\begin{frame}
	\begin{center}
		\huge\color{blue}\bf
		Section 2. Classical Optimization Methods
	\end{center}
%\end{frame}
\fi
\subsection{Gradient Methods}
%\begin{frame}
%	\frametitle{Gradient Methods}
	\subsubsection{Line search}
	If we got the iterative direction $d^k$, we need to find the step size $\alpha^k$ in the $k$-th step
	\clr{\begin{eqnarray*}
		x^{k+1} = x^k + \alpha^k d^k.
	\end{eqnarray*}}
	There are many line search methods. We can classify them into 2 classes. \\
	One is exact line search which solve the $\alpha$ exactly, namely $\alpha$ satisfies\\
	 \clm{$\alpha^k=\argmin\limits_{\alpha\in\R} f( x^k + \alpha d^k)$}.\\
	Another one is inexact line search, an example of the inexact line search is Armijo line search:
	\begin{itemize}
		\item Armijo line search (back tracking):
		\begin{itemize}
			\item set $c_1\in(0,1)$, $\tau\in(0,1)$, $\alpha_0>0$, and $j:=0$;\\[1mm]
			\item if \clg{$f(x^k)-f(x^k+\alpha_j d^k)\geq -\alpha_j c_1\nabla f(x^k)\zz d^k$,} return
			\clm{$\alpha^k := \alpha_j$;}\\[1mm]
			\item otherwise, set $j:=j+1$ and \clb{$\alpha_{j}=\tau \alpha_{j-1}$.}
		\end{itemize}
		\item Wolfe condition: {\small additional curvature condition with $c_2\in (c_1,1)$,}
		\vspace{-1mm}
		\clg{
		\begin{eqnarray*}
			-\nabla f(x^k + \alpha_j d^k)\zz d^k \leq - c_2 \nabla f(x^k)\zz d^k.
		\end{eqnarray*}}
	\end{itemize}
%\end{frame}

%******************************************************
%\begin{frame}
%	\frametitle{Gradient Methods (Cont'd)}
	\subsubsection{Gradient methods}
	In gradient methods, the iterative direction is the negative direction of gradient:
	\clm{\begin{eqnarray*}
			d^k = - \nabla f(x^k).
	\end{eqnarray*}}
	Then we have many strategies to find the stepsize:
	\begin{itemize}
		\item steepest descent: exact line search
		\item gradient descent with inexact line search:
		global convergence and local linear rate related
		to $\kappa(\nabla^2 f(x^*))$.\\[1mm]
		\item Barzilai-Borwein (BB) stepsize:
		\clb{\begin{eqnarray*}
			\alpha^k =\frac{{s^k}\zz y^k}{{y^k}\zz y^k},\quad \mbox{or}
			\quad \alpha^k =\frac{{s^k}\zz s^k}{{s^k}\zz y^k}.
		\end{eqnarray*}}
		\vspace{-3mm}

		where \clg{$s^k=x^k-x^{k-1} \clm{=\alpha^{k-1}d^{k-1}}$, $y^k=\nabla f(x^k)-\nabla f(x^{k-1})$,}\\[1mm]

		 global convergence and local linear convergence only
		for $f(x)=\frac{1}{2}x\zz Ax +b\zz x$ with $A\succ 0$; local superlinear convergence
		in the case $n=2$; global convergence if combined with nonmonotone line search.
	\end{itemize}
%\end{frame}

%******************************************************
%\begin{frame}
%	\frametitle{Gradient Methods (Cont'd)}
	\subsubsection{Conjugate gradient methods}
	The conjugate gradient methods are  originally proposed for solving linear system. We use the combination of gradient and the last step's direction to generate this step's direction.
	\clm{\begin{eqnarray*}
			d^k = - \nabla f(x^k) + \beta^k d^{k-1}.
	\end{eqnarray*}}
	The strategies of choosing parameters:
	\begin{itemize}

		\item $\alpha^k$: exact line search
		\item updating rules for $\beta^k$
		\begin{itemize}
			\item Fletcher-Reeves: \clb{$\beta^k = \nabla f(x^k)\zz \nabla f(x^k)\left/\,\nabla f(x^{k-1})\zz \nabla f(x^{k-1}) \right.$;}\\[1mm]
			\item Polak-Ribi\`{e}re: \clb{$\beta^k = \nabla f(x^k)\zz y^k\left/\,\nabla f(x^{k-1})\zz \nabla f(x^{k-1}) \right.$;}\\[1mm]
			\item Hestenes-Stiefel: \clb{$\beta^k = \nabla f(x^k)\zz y^k\left/\,{d^{k-1}}\zz y^k \right.$;}\\[1mm]
			\item Dai-Yuan: \clb{$\beta^k = \nabla f(x^k)\zz \nabla f(x^k)\left/\,{d^{k-1}}\zz y^k \right.$.}
		\end{itemize}
		\item subspace strategy:
		\clm{
		\begin{eqnarray*}
			x^{k+1} :=\argmin_{x-x^k \,\in\, \mathrm{span}\{\nabla f(x^k),\, d^{k-1}\}}\quad
			f(x).
		\end{eqnarray*}
		}

		\vspace{-2mm}
		\clr{\footnotesize global convergence if combined with line search, local linear convergence rate not related to $\kappa(\nabla^2 f(x^*))$.}
	\end{itemize}
%\end{frame}

%********************* section ********************
\subsection{Newton Methods}
%\begin{frame}
%	\frametitle{Newton Methods}
	The gradient methods only use 1-order derivative information of $f$. The Newton methods need to use 2-order derivative information.
	\subsubsection{Newton methods}
	The step direction of Newton methods is:
	\clm{\begin{eqnarray*}
			d^k = - {\nabla^2 f(x^k)}\inv \nabla f(x^k).
	\end{eqnarray*}}
	The strategies of choosing parameters:
	\begin{itemize}
		\item original ones: \clb{$\alpha^k = 1$ or exact line search}\\[1mm]
		\clr{\footnotesize local quadratic convergence.}
		\item hybrid Newton method: \clg{$d^k = -\beta \nabla f(x^k) - {\nabla^2 f(x^k)}\inv \nabla f(x^k)$}
		\item negative curvature descent: set \clg{$d^k=d$} if $d\zz \nabla^2 f(x^k) d<0$.
		\item damped Newton method: \clb{$\alpha^k= 1\left/\left(1+ \sqrt{\nabla f(x^k)\zz {\nabla^2 f(x^k)}\inv \nabla f(x^k)}\right) \right.$}\\[1mm]
		\clr{\footnotesize global convergence.}
	\end{itemize}
%\end{frame}

%******************************************************
%\begin{frame}
%	\frametitle{Newton Methods (Cont'd)}
	\subsubsection{Motivation of quasi-Newton methods}
	Because we need to spend lots of time on calculating the $\nabla^2 f(x)$ every step, the quasi-Newton methods use the approximation of $\nabla^2 f(x^k)$ written as $B^k$ which is much easier to calculate to get the direction:
	\clm{\begin{eqnarray*}
			d^k = - {B^k}\inv \nabla f(x^k).
	\end{eqnarray*}}
	\vspace{-7mm}
	\begin{itemize}
		%\item $\clb{B^k}$ is an approximation of $\nabla^2 f(x^k)$
		\item easy to calculate, possess the essential characteristics of Hessian, descent direction (positive definiteness of $B^k$)
		\item solution: \clg{the secant equation}
		\clr{\begin{eqnarray*}
				B^k s^k = y^k.
		\end{eqnarray*}}
		\vspace{-7mm}
		\item SR$-1$ (symmetric rank$-1$ update) can not guarantee the positive definiteness
		\item rank$-2$ update is more favorable
		\begin{itemize}
			\item start from \clb{$B^0$}, (e.g. $\alpha I_n$.)\\[1mm]
			\item in each iteration, add rank$-2$ update
			\clb{$B^{k+1}=B^k+\alpha uu\zz +vv\zz$;}\\[1mm]
			\item choose \clb{$u=y^k$, $v=B^ks^k$,} we arrive at -- \clr{BFGS.}
		\end{itemize}
	\end{itemize}
%\end{frame}

%******************************************************
%\begin{frame}
%	\frametitle{Newton Methods (Cont'd)}
	\subsubsection{BFGS (Broyden-Fletcher-Goldfarb-Shanno)}
	\clm{\footnotesize $$B^{k+1}_{\tiny\mbox{BFGS}} = B^k + \frac{{y^k}\zz {y^k}}{{y^k}\zz s^k} -\frac{B^ks^k{s^k}\zz B^k}
		{{s^k}\zz B^ks^k}.$$}
	\vspace{-5mm}
	\begin{itemize}
		\item consider the update for inverse $H^k={B^k}\inv$
		\clr{\footnotesize  $$H^{k+1}_{\tiny\mbox{BFGS}} = \left(I-\frac{{s^k}{y^k}\zz}{{s^k}\zz y^k}\right)H^k\left(I-\frac{{s^k}{y^k}\zz}{{s^k}\zz y^k}\right) + \frac{{s^k} {s^k}\zz}{{s^k}\zz y^k}.$$}
		\vspace{-5mm}
		\item minimum change property:
		\clb{\small  $$H^{k+1}=\min\limits_{H\in\Sn} ||H-H^k||_G,\quad \st \quad Hy^k = s^k$$}
		\vspace{-5mm}
		{\footnotesize where $||A||_G = ||G^{\frac{1}{2}}AG^{\frac{1}{2}}||\ff$, $G\in\{G\mid Gs^k=y^k\}$,
		e.g. $G=\int_0^1 \nabla^2 f(x^k+\tau \alpha^k d^k)d\tau$}\\[7mm]
	\end{itemize}

	\clr{\footnotesize global convergence if combined with line search; local linear convergence
		if $f$ is strict convex; local superlinear convergence if $f$ is strongly convex.}
%\end{frame}

%******************************************************
%\begin{frame}
%	\frametitle{Newton Methods (Cont'd)}
	\subsubsection{DFP (Davidon-Fletcher-Powell)}
	\clm{\footnotesize $$B^{k+1}_{\tiny\mbox{DFP}} = \left(I-\frac{{s^k}{y^k}\zz}{{s^k}\zz y^k}\right)B^k\left(I-\frac{{s^k}{y^k}\zz}{{s^k}\zz y^k}\right) + \frac{{y^k} {y^k}\zz}{{s^k}\zz y^k}.$$}
	\vspace{-5mm}
	\begin{itemize}
		\item consider the update for inverse $H^k={B^k}\inv$
		\clg{\footnotesize  $$H^{k+1}_{\tiny\mbox{DFP}} = H^k + \frac{{s^k}\zz {s^k}}{{y^k}\zz s^k} -\frac{H^ky^k{y^k}\zz H^k}
			{{y^k}\zz H^ky^k}.$$}
	\end{itemize}

	\clr{\footnotesize global convergence if combined with line search and local linear convergence
		if $f$ is strict convex; local superlinear convergence if $f$ is strongly convex.}
	\bigskip

%	\structure{\bf The Broyden family}
	\begin{eqnarray*}
		B^{k+1} = (1-\phi^k) B^{k+1}_{\tiny\mbox{BFGS}} + \phi^k B^{k+1}_{\tiny\mbox{DFP}},\qquad \phi^k\in[0,1].
	\end{eqnarray*}
	\clr{\footnotesize $\phi^k\in[0,1)$ same convergence property with BFGS.}
%\end{frame}

%******************************************************
%\begin{frame}
%	\frametitle{Newton Methods (Cont'd)}
	\subsubsection{Limited memory quasi-Newton method}
	\begin{itemize}
		\item if the storage of \clr{$B^k$} (\clr{$H^k$}) is not affordable\footnote{
		The difference between using $B^k$ or $H^k$ appears at the computational cost, and the storage
	    is a whole other story.}
		\item rank$-2$ update provides a limited memory strategy
		\begin{itemize}
			\item store \clb{$\mathcal{L}:=\{s^k,s^{k-1},...,s^{\max\{k-l+1,0\}},y^k,y^{k-1}),...,y^{\max\{k-l+1,0\}}\}$;}\\[1mm]
			\item $H^k$ is built up from $H^0$ by a \clm{rank$-2\max\{l,k\}$ update}\\[1mm]
			\item reduce the storage from $O(n^2)$ to \clg{$O(mn)$} at a cost of $O(mn)$ arithmetic operation\\[1mm]
			\item reduce the computational cost from $O(n^2)$ to \clg{$O(mn)$}, if there is no structure
		\end{itemize}
		\item numerically successful
		\begin{itemize}
			\item BFGS update\\[1mm]
			\item $m=10$\\[2mm]
		\end{itemize}
	\end{itemize}

	\clr{\footnotesize global convergence if combined with line search and local linear convergence.}
%\end{frame}

%******************************************************
%\begin{frame}
%	\frametitle{Newton Methods (Cont'd)}
	\subsubsection{The explanation of BB stepsize}
	\clb{\begin{eqnarray*}
			x^{k+1} = x^k -\alpha^k \nabla f(x^k),\quad \mbox{with} \quad
			\clr{\alpha^k =\frac{{s^k}\zz y^k}{{y^k}\zz y^k}},\quad \mbox{or}
			\quad \clm{\alpha^k =\frac{{s^k}\zz s^k}{{s^k}\zz y^k}}.
	\end{eqnarray*}}
    \vspace{-3mm}
    \begin{itemize}
    	\item Using $\frac{1}{\alpha}\cdot I$ to approximate $\nabla^2 f(x^k)$
    	\clm{$$\alpha^k = 1 \left/ \argmin\limits_{\beta\in\R} \left\|\beta s^k -y_k\right\|_2^2\right.. $$}
    	\vspace{-5mm}
    	\item Using $\alpha\cdot I$ to approximate ${\nabla^2 f(x^k)}\inv$
    	\clr{$$\alpha^k = \argmin\limits_{\alpha\in\R} ||\alpha y^k -s_k||_2^2. $$}
    \end{itemize}
%\end{frame}

%********************* section ********************
\subsection{Trust Region Methods}
%\begin{frame}
%	\frametitle{Trust Region Methods}
	Region Methods are another kinds of optimize methods. The motivation of them are finding the minimize of the quadratic approximation of $f$ in a limited region. If the quadratic approximation can approximate $f$ well, expand the region, else reduce the region. In mathematics,
	\clr{\begin{eqnarray*}
			x^{k+1} &=& x^k + s^k,\\
			s^k &=& \argmin\limits_{s\in \R} \quad m^k(s),\quad \st\quad ||s||_2\leq \Delta^k.
		\end{eqnarray*}
	}
	\vspace{-5mm}
	\begin{itemize}
		\item $m^k(s)$ quadratic approximation of $f(x^k+s)$ at $x^k$
		\clb{\begin{eqnarray*}
				m^k(s) := \nabla f(x^k)\zz s + \frac{1}{2}s\zz B^k s.
		\end{eqnarray*}}
		\vspace{-5mm}
		\item solving subproblem
		\begin{itemize}
			\item exactly solver: \clg{Mor\'{e}-Sorensen}
			\item approximate: Chauchy point, dog-leg
			\item inexact solver: \clg{truncated CG, $2-$D subspace minimization}
		\end{itemize}
		\item the choice of \clb{$B^k$}
		\begin{itemize}
			\item $\nabla^2 f(x^k)$\\[1mm]
			\item quasi-Newton update\\[1mm]
			\item other approximation of $\nabla^2 f(x^k)$
		\end{itemize}
	\end{itemize}
%\end{frame}

%******************************************************
%\begin{frame}
%	\frametitle{Trust Region Methods (Cont'd)}
	\begin{itemize}
		\item approximation ratio
		\clm{\small $$\eta^k = \frac{\mbox{red}_{\mbox{\tiny real}}}{\mbox{red}_{\mbox{\tiny pred}}}
		= \frac{f(x^k)-f(x^k+s^k)}{m(0)-m(s^k)}.$$}
		\vspace{-5mm}
		\item accept trial step of not:
		\clb{\small
		$$x^{k+1} = \ifelse{
		x^k + s^k, & \mbox{if\,} \eta^k>0;\\
		x^k, & \mbox{otherwise.}
	    }$$
		}
		\vspace{-5mm}
		\item updating trust region radius \clg{$\Delta^k$}
		\clb{\small
			$$\Delta^{k+1} = \ifelse{
				b_2 \Delta^k, & \mbox{if\,} \eta^k > c_2;\\
				\Delta^k, & \mbox{if\,} c_2\geq \eta^k>c_1;\\
				b_1 \Delta^k, & \mbox{otherwise.}
			}$$
		}
		\vspace{-5mm}
		{\small where  $0<c_1<c_2<1$, $0<b_1<1<b_2$}.\\[7mm]
	\end{itemize}

	\clr{\footnotesize global convergence only requires subproblem inexactly solved; convergence to second-order stationary point if $B^k=\nabla^2 f(x^k)$ and subproblem exactly solved.}
%\end{frame}

%********************* section ********************
\subsection{Methods for Nonlinear Least Squares}
%\begin{frame}
%	\frametitle{Methods for Nonlinear Least Squares}
	\subsubsection{Nonlinear least squares}
	For nonlinear least squares, the objective function $f$ reads:
	\clm{
	$$f(x)= ||F(x)||_2^2 = \sum\limits_{i=1}^m f^2_i(x)~.$$
	}
	\vspace{-5mm}
	\begin{itemize}
		\item $F(x):=\left(f_1(x),...,f_m(x)\right)\zz$, each $f_i(x):\Rn\mapsto \R$ ($i=1,...,m$)
		\item Jacobian matrix: \clb{$\mathcal{J}_F(x)=\left(\nabla f_1(x),...,\nabla f_m(x)\right)\zz$}
		\item gradient: \clr{$\nabla f(x) = \mathcal{J}_F(x)\zz F(x)$}
		\item Hessian: \clr{$\nabla^2 f(x) = \mathcal{J}_F(x)\zz \mathcal{J}_F(x) + \sum\limits_{i=1}^m
			f_i(x)\nabla^2 f_i(x)$}
		\item linear approximation:\clr{ $F(x) \approx F(x^{(k)})+\mathcal{J}_F(x^{(k)})(x-x^{(k)}) $}
		\item new approximation of Hessian: \clg{$\mathcal{J}_F(x)\zz \mathcal{J}_F(x)$}
		\begin{itemize}
			\item approximation quality depends on residuals $f_i(x)$  ($i=1,...,m$)\\[1mm]
			\item obtain partial Hessian information by collecting derivatives\\[1mm]
			\item positive definiteness
		\end{itemize}
	\end{itemize}
%\end{frame}

%******************************************************
%\begin{frame}
%	\frametitle{Methods for Nonlinear Least Squares (Cont'd)}
	\subsubsection{Gauss Newton method}
	The step direction of Gauss Newton methods is:
	\clm{
		$$d^k = - \left(
		\mathcal{J}_F(x^k)\zz \mathcal{J}_F(x^k)	\right)\inv \nabla f(x^k)$$
	}
	\vspace{-6mm}
	\begin{itemize}
		\item similar performance as Newton method if small residual
		\item similar performance as gradient method if large residual
		\item numerically unstable if \clr{$\mathcal{J}_F(x^k)$} is singular or close to singular
 	\end{itemize}
    \medskip

%    \structure{\bf Levenberg-Marquardt method}
    \clg{
    	$$s^k = - \left(
    	\mathcal{J}_F(x^k)\zz \mathcal{J}_F(x^k) + \clb{\mu^k} \cdot I\right)\inv \nabla f(x^k)$$
    }
    \vspace{-6mm}
    \begin{itemize}
    	\item regularization parameter $\clb{\mu^k}$ can be tuned
    	\begin{itemize}
    		\item in the same manner as trust region radius
    		\item \clb{$||F(x^k)||_2^t$ ($t=[1,2]$)}\\[3mm]
    	\end{itemize}
    \end{itemize}

	\clr{\footnotesize global convergence; quadratic local convergence rate if $\mu^k\rightarrow 0$
		and zero residual at solution}
%\end{frame}

%********************* section ********************
\subsection{Block Coordinate Descent}
%\begin{frame}
%	\frametitle{Block Coordinate Descent}
	\clr{
	\begin{eqnarray*}
		\left\{
		\begin{array}{l}
			x^{k+1}_1 = \argmin\limits_{x_1\in\R^{n_1}} f(x_1,x_2^k,...,x_p^k);\\
			x^{k+1}_2 = \argmin\limits_{x_2\in\R^{n_2}} f(x_1^{k+1},x_2,x_3^k...,x_p^k);\\
			\cdots \cdots\\
			x^{k+1}_p = \argmin\limits_{x_p\in\R^{n_p}} f(x_1^{k+1},...,x_{p-1}^{k+1},x_p).
		\end{array}
		\right.
	\end{eqnarray*}
	}
	\begin{itemize}
		\item $x=(x_1\zz,x_2\zz,...,x_p\zz)\zz$, $x_i\in\R^{n_i}$ ($i=1,...,p$)
		\item convergence under strongly convex
		\item essentially \clb{Gauss-Seidel} iteration: \clg{\small $f=\frac{1}{2}x\zz Ax -b\zz x$ with $A\succ 0$}
		\footnote{This condition can be relaxed to $A\succeq 0$, $A_{ii}\succeq 0$ ($i=1,...,p$).}
		\item \clm{question: does Jacobi iteration work?} linear proximal variant:
		\clr{$$
		x^{k+1}_i = \argmin\limits_{x_i\in\R^{n_i}} \nabla_{x_i} f(x^k)\zz x_i
		+ \frac{\beta^k}{2}||x_i-x_i^k||_2^2,\quad i=1,...,p.
		$$}
	\end{itemize}
%\end{frame}

%********************* section ********************
\section{Global Optimization Strategies}
\iffalse
\begin{frame}
	\begin{center}
		\huge\color{blue}\bf
		Section 3. Global Optimization Strategies
	\end{center}
\end{frame}
\fi
Global optimization for non-convex function is a very difficult problem. We can only use it on some special problems. This section discuss a few strategies which try to solve this problem:
\subsection{Overview}
%\begin{frame}
%	\frametitle{Overview}
	\subsubsection{A few strategies}
	\begin{itemize}
		\item deterministic methods\footnote{Combinatorial optimization
			can be modeled as binary variable programming. Since $x\in\{0,1\}\,\Leftrightarrow\, x^2=x$,
			it can be viewed as a special nonlinear programming.}
		\begin{itemize}
			\item branch and bound
			\item cutting plane

		\end{itemize}
		\item undeterministic methods
		\begin{itemize}
			\item homotopy
			\item randomly multi-start
			\item simulated annealing
			\item {genetic algorithm}
			\item {ant colony algorithm}
		\end{itemize}
		\item approximation methods
		\begin{itemize}
			\item \clb{SDP relaxation:} $x\zz Ax=\langle A,xx\zz \rangle$,\quad $xx\zz \Rightarrow X\succeq 0$
		\end{itemize}
		\item problems have nice properties
		\begin{itemize}
			\item \clg{special quartic objective: phase retrieval, matrix completion, ...}
			\item \clm{problem input obeys a certain distribution}
			\item \clm{no nonglobal local minimizer: stationary $\Leftrightarrow$ global or saddle}
		\end{itemize}
	\end{itemize}
%\end{frame}

%**************************************************
\subsection{Undeterministic Methods}
%\begin{frame}
%	\frametitle{Undeterministic Methods}
	\subsubsection{Homotopy (Global continuation)}
	\begin{itemize}
		\item let \clb{$g(x)$} be a convex relaxation\footnote{Usually, it means that the epigraph of $g(x)$,
			$\{(x,v)\mid v\geq f(x)  \}$, completely contains the epigraph of $f(x)$.} of \clr{$f(x)$}
		\item define the homotopy function: \clg{$F(x,t) :\Rn\times [0,1]\mapsto \R$}
		\begin{itemize}
			\item \clr{$F(x,0)=f(x)$;}\\[1mm]
			\item \clb{$F(x,1)=g(x)$;}\\[1mm]
			\item e.g. \clg{$F(x,t)= (1-t)\cdot f(x) + t\cdot g(x)$.}
		\end{itemize}
		\item main idea -- solving
		\clm{$$\min\limits_{x\in\Rn} \quad F(x,t),$$}
		with \clg{$t$} varying from \clb{$1$} to \clr{$0$}.
		\item particularly useful for problems
		\begin{itemize}
			\item one main valley
			\item surrounded by side valleys
			\item side valleys occur by oscillation
		\end{itemize}
	\end{itemize}
%\end{frame}

%**************************************************
%\begin{frame}
%	\frametitle{Undeterministic Methods (Cont'd)}
	\subsubsection{Randomly multi-start}
	\begin{itemize}
		\item different with multi-start from grids or other patterns
		\item main procedure
		\begin{itemize}
			\item[1.] input: $\mbox{MaxL}\in\N$, $\mbox{MaxW}\in\N$.
			\item[2.] set $\mbox{CL} :=0$, $\mbox{CW} := 0$, \clm{$x^{\tiny\mbox{rec}}:=0$,}
			\clg{$f^{\tiny\mbox{rec}}=+\infty$.}
			\item[3.] certain \clb{random sampling procedure:} obtain $x^{\tiny\mbox{sp}}$.
			\item[4.] certain \clr{local search procedure:} obtain $x^{\tiny\mbox{loc}}$,
			 $\mbox{CL}:=\mbox{CL}+1$.
			\item[5.] if $f(x^{\tiny\mbox{loc}})< f^{\tiny\mbox{rec}}$, set
				\clm{$x^{\tiny\mbox{rec}} := x^{\tiny\mbox{loc}}$,} \clg{$f^{\tiny\mbox{rec}}
				= f(x^{\tiny\mbox{loc}})$,} $\mbox{CW} := 0$, goto 3.
			\item[6.] otherwise, $\mbox{CW}:=\mbox{CW}+1$.
			\item[7.] if $\mbox{CL}=\mbox{MaxL}$ or $\mbox{CW}=\mbox{MaxW}$, terminate
			and return \clm{$x^{\tiny\mbox{rec}}$.}
			\item[8.] otherwise, goto 3.
		\end{itemize}
		\item trade off between \clb{sampling phase} and \clr{local search phase}
		\item convergence
		\begin{itemize}
			\item finding global minimizer in a compact domain
			\item locally Lipschitz
			\item \clg{when $\mbox{MaxL}\rightarrow +\infty$,} \clm{probability approaches $1$}
		\end{itemize}
	\end{itemize}
%\end{frame}

%**************************************************
%\begin{frame}
%	\frametitle{Undeterministic Methods (Cont'd)}
	\subsubsection{Simulated annealing}
	This method's inspiration comes from annealing in metallurgy.
	\begin{itemize}
		\item main framework
		\begin{itemize}
			\item[1.] input: initial temperature \clg{$T\gg 1$,} initial point \clr{$x$,} $L\in\N$,
			$\mbox{MaxW}\in\N$; set \clm{$\mbox{CW} := 0$,} $i:=0$.\\[1mm]
			\item[2.] if $i=L$, goto Step 7; otherwise, goto Step 3.\\[1mm]
			\item[3.] \clb{find a new point $x'$} by certain simple procedure.\\[1mm]
			\item[4.] evaluate the incremental $\Delta':=f(x')-f(x)$.\\[1mm]
			\item[5.] if $\Delta'\leq 0$, \clr{$x:=x'$,} \clm{$\mbox{CW} = 0$;} else if, set
			\clr{$x:=x'$,} \clm{$\mbox{CW} = 0$} in probability \clb{$\exp(-\Delta'/(kT))$}\footnote{$k$ takes Boltzmann constant.}; otherwise,
			\clm{$\mbox{CW}:=\mbox{CW}+1$.}\\[1mm]
			\item[6.] if \clm{$\mbox{CW}\geq\mbox{MaxW}$} and \clg{$T=0$,} terminate; otherwise,
			set $i:=i+1$ and goto Step 2.\\[1mm]
			\item[7.] \clg{decrease temperature $T$ slowly,} set $i:=0$ and goto Step 2.
		\end{itemize}
	\end{itemize}
%\end{frame}


% !TEX root = ../main_lecture_notes.tex
\chapter{Efficiency of blockchain systems}\label{chap:efficiency}

\section{A queueing model with bulk service}\label{sec:queue}
Blockchain users send transactions to the network of validators according to some rate $\lambda$. These transactions enter a queue of pending transactions. The validators select a subset of $b$ transactions to be recorded in the next block. The block is built by a leader elected via a consensus protocol. The block is then communicated to the other validators and the $b$ transactions exit the queue. We assume that building a block takes some exponentially distributed time with mean $\mu$. What we just described is exactly a single server with bulk service queueing system, described for instance in \citet{Bailey1954} and \citet{Chaudhry1981} with exponential arrival times, that processes $k$ items at a time, with an exponential service time. This a $M/M^b/1$ queue in Kendall's notation summarized in \cref{fig:blockchain_queue}.
\begin{figure}[!ht]
\begin{center}
\begin{tikzpicture}[-, >=stealth', auto, semithick, node distance=1cm]

\tikzstyle{phantom block}=[rectangle, fill=white,draw=white, thick,text=black,scale=2]
\tikzstyle{block}=[rectangle, fill=white,draw=black,thick,text=black,scale=4]
\tikzstyle{Intensity}=[circle, fill=white,draw=blue,very thick, text=black,scale=1.2]
\tikzstyle{transaction pending}=[circle, fill=white,draw=blue,very thick, text=black,scale=1]
\tikzstyle{transaction considered}=[circle, fill=blue, text=black, scale=1]
\node[Intensity]    (1){$\lambda$};
\node[phantom block]  (2)[right of=1] {};
\node[transaction pending] (3)[right of=2] {};
\node[transaction pending] (4)[above of=3] {};
\node[transaction pending] (5)[above of=4] {};
\node[transaction pending] (6)[above of=5] {};
\node[transaction pending] (7)[below of=3] {};
\node[transaction pending] (8)[below of=7] {};
\path
(1) edge[->,bend left]     node{} (4)
    edge[->, bend right]     node{}          (7);

\node[Intensity]  (14)[above of=6]{$\mu$};
% \path
% (1) edge[->,bend left]     node{$\text{Exp}(\lambda)$}        (4)
%     edge[->, bend right]     node{}          (7);

\node[transaction considered] (3)[right of=2] {};
\node[transaction considered] (4)[above of=3] {};
\node[transaction considered] (5)[above of=4] {};
\node[transaction considered] (6)[above of=5] {};

\node[phantom block]  (10)[right of=3] {};
\node[block]  (11)[right of=10] {};
\path
(6) edge[->]   node{} (11)
(5) edge[->]   node{} (11)
(4) edge[->]   node{} (11)
(3) edge[->]   node{} (11)
;
\node[Intensity]  (12)[above of=11] {$\mu$};
\node[phantom block]  (13)[above of=12] {};

\path
(11) edge[-]     node{}        (12);
\path
(12) edge[->]     node{}        (13);

\end{tikzpicture}
\end{center}
\caption{Blockchain queue}
\label{fig:blockchain_queue}
\end{figure}
One specificity of this queue is that the server is always busy. Our goal is to assess the efficiency which is characterized by
\begin{itemize}
  \item Throughputs: Number of transaction being processed per time unit
  \item Latency: Average transaction confirmation time
\end{itemize}
This can be done by studying the distribution of the number of pending transaction in the queue over the long run. A stationary state can only be reached if
\begin{equation}\label{eq:stationarity_cond}
\mu \cdot b > \lambda.
\end{equation}
Denote by $N^q$ the length of the queue upon stationarity, The following result holds.
\begin{theo}
Assume that \eqref{eq:stationarity_cond} holds then $N^q$ is geometrically distributed
$$
\mathbb{P}(N^q = n) = (1-p)\cdot p^n,\text{ } n\geq0
$$
where $p = 1/z^\ast$ and $z^\ast$ is the only root of
$$
-\frac{\lambda}{\mu}z^{b+1}+z^b\left(\frac{\lambda}{\mu}+1\right) - 1,
$$
such that $|z^\ast$|>1.
\end{theo}
\begin{proof}
Let $N^q_t$ be the number of transactions in the queue at time $t\geq0$ and $X_t$ the time elapsed since the last block was found. Further define
\[
P_{n}(x,t)\text{d}x  =\mathbb{P}[N_t^q = n, X_t \in(x, x + \text{dx})]
\]
If $\lambda < \mu\cdot b$ holds then the process admits a limiting distribution given by
\[
\underset{t\rightarrow\infty}{\lim}P_{n}(x,t) = P_{n}(x).
\]
Adding the variable $X_t$ is a known trick going back to \citet{Cox1955}, it allows us to make the process $(N_t^q)_{t\geq0}$ Markovian. We aim at finding the distribution of the queue length upon stationarity
\begin{equation}\label{eq:alpha_n}
\mathbb{P}(N^q=n):=\alpha_n =\int_{0}^\infty P_{n}(x)\text{d}x, \text{ }n\geq0.
\end{equation}
Consider the possible transitions over a small time lapse \text{h} during which no block is being generated. Over this time interval, either
\begin{itemize}
  \item no transactions arrives
  \item one transaction arrives
\end{itemize}
We have for $n\geq1$
\[
P_{n}(x+h) = e^{-\mu h}\left[e^{-\lambda h}P_{n}(x)+\lambda h e^{-\lambda h}P_{n-1}(x)\right].
\]
Differentiating with respect to $h$ and letting $h\rightarrow0$ leads to
\begin{equation}\label{eq:diff_eq_n_geq_1}
P_{n}'(x) = -(\lambda+\mu)P_{n}(x)+\lambda P_{n-1}(x),\text{ }n \geq1.
\end{equation}
Similarly for $n = 0$, we have
\begin{equation}\label{eq:diff_eq_n_eq_0}
P_{0}'(x) = -(\lambda+\mu)P_{0}(x).
\end{equation}
We denote by
$$
\xi(x)\text{d}x =\mathbb{P}(x\leq X< x+\text{d}x|X\geq x)= \mu\text{d}x,
$$
the hazard function of the block arrival time (constant as it is exponentially distributed). The system of differential equations \eqref{eq:diff_eq_n_geq_1}, \eqref{eq:diff_eq_n_eq_0} admits boundary conditions at $x = 0$ with
\begin{equation}\label{eq:boundary_cond_1}
\begin{cases}
P_{n}(0) = \int_0^{+\infty} P_{n+b}(x)\xi(x)\text{d}x = \mu\alpha_{n+b},&n \geq1,\\
P_{0}(0) = \mu\sum_{n=0}^{b}\alpha_n,&n = 0,\ldots,b\\
\end{cases}
\end{equation}
Define the probability generating function of $N^q$ at some elapsed service time $x\geq 0$ as
$$
G(z;x) = \sum_{n=0}^\infty P_{n}(x)z^n.
$$
By differentiating with respect to $x$, we get (using \eqref{eq:diff_eq_n_geq_1} and \eqref{eq:diff_eq_n_eq_0})
$$
\frac{\partial}{\partial x}G(z;x) = -\left[\lambda(1-z)+\mu\right]G(z;x),
$$
and therefore
$$
G(z;x) = G(z;0)\exp\left\{-\left[\lambda(1-z)+\mu\right]x\right\}.
$$
We get the probability generating function of $N^q$ by integrating over $x$ as
\begin{equation}\label{eq:G_z_solve_ODE}
G(z) = \frac{G(z;0)}{\lambda(1-z)+\mu}.
\end{equation}
Using the boundary conditions \eqref{eq:boundary_cond_1}, we write
\begin{eqnarray}
G(z;0) &= &\sum_{n = 0}^\infty P_{n}(0)z^n \nonumber\\
&= &P_{0}(0)+\sum_{n=1}^{+\infty}P_{n}(0)z^n\nonumber\\
&=& \mu\sum_{n = 0}^{b}\alpha_n  + \mu\sum_{n=1}^{+\infty}\alpha_{n+b} z^n\nonumber\\
&=& \mu\sum_{n = 0}^{b}\alpha_n + \mu z^{-b}\left[G(z)-\sum_{n = 0}^{b}\alpha_n z^n\right]\label{eq:G_z_0}
\end{eqnarray}
Replacing the left hand side of \eqref{eq:G_z_0} by \eqref{eq:G_z_solve_ODE}, multiplying on both side by $z^b$ and rearranging yields
\begin{equation}\label{eq:G_z_as_rational_function}
\frac{G(z)}{M(z)}[z^b - M(z)] =\sum_{n=0}^{b-1}\alpha_n(z^b - z^n),
\end{equation}
where $M(z) = \mu/(\lambda(1-z)+\mu)$. Using Rouche's theorem, we find that both side of the equation shares $b$ zeros inside the circle $\mathcal{C} = \{z\in\mathbb{C}\text{ ; }|z| <1+\epsilon\}$ for some epsilon.
\begin{lemma}
Let $\mathcal{C}\in \mathbb{C}$ and $f$ and $g$ two holomorphic functions on $\mathcal{C}$. Let $\partial\mathcal{C}$ be the contour of $\partial\mathcal{C}$. If
$$
|f(z)-g(z)|<|g(z)|\text{, }\forall z\in\partial\mathcal{C}
$$
then $Z_f-P_f = Z_g-P_g$, where $Z_f$, $P_f$, $Z_g$, and $P_g$ are the number of zeros and poles of $f$ and $g$ respectively.
\end{lemma}
We have $\partial\mathcal{C} =\{z\in\mathbb{C}\text{ ; }|z| =1+\epsilon\}$. The left hand side can be rewritten as
$$
G(z)\left[-\frac{\lambda}{\mu}z^{b+1} + \left(1 + \frac{\lambda}{\mu}\right)z^b -1\right].
$$
Define $f(z) = -\frac{\lambda}{\mu}z^{b+1} + \left(1 + \frac{\lambda}{\mu}\right)z^b -1$ and $g(z)=\left(1 + \frac{\lambda}{\mu}\right)z^b$. We have
 $$
|f(z) - g(z)| = |-\frac{\lambda}{\mu}z^{b+1}-1|\leq \frac{\lambda}{\mu}(1+\epsilon)^{b+1}+1< \left(1 + \frac{\lambda}{\mu}\right)(1+\epsilon)^b= |g(z)| ,\text{ with }\epsilon \rightarrow 0.
$$
Regarding the right hand side, define $f(z) = \sum_{n=0}^{b-1}\alpha_n(z^b - z^n)$ and $g(z) =\sum_{n=0}^{b-1}\alpha_nz^b $. We have
$$
|f(z) - g(z)| < |\sum_{n=0}^{b-1}\alpha_n z^n| \leq \sum_{n=0}^{b-1}\alpha_n (1+\epsilon)^n<(1+\epsilon)^b\sum_{n=0}^{b-1}\alpha_n = |g(z)|.
$$
We deduce from Rouche's theorem that both sides have $b$ share roots inside $\mathcal{C}$. Note that one of them is $1$, and we denote by $z_k$, $k = 1,\ldots, b-1$ the remaining $b-1$ roots. Given the polynomial form of the right hand side of \eqref{eq:G_z_as_rational_function}, the fundamental theorem of algebra indicates that the number of zero is $b$. Given the left hand side
$$
G(z)\left[-\frac{\lambda}{\mu}z^{b+1} + \left(1 + \frac{\lambda}{\mu}\right)z^b -1\right].
$$
we deduce that there is one zeros outside $\mathcal{C}$, we can further show that it is a real number $z^\ast$. Multiplying both side of \eqref{eq:G_z_as_rational_function} by $(z-1)\prod_{k =1}^{b-1}(z-z_k)$, and using $G(1)=1$ yields
$$
G(z) = \frac{1-z^\ast}{z-z^{\ast}}.
$$
$N^q$ is then a geometric random variable with parameter $p = \frac{1}{z^\ast}.$
\end{proof}
The result above can be found in \citet{Bailey1954}. The application to blockchain under more general assumptions over the block discovery time is given in \citet{Kawase2017}.
\section{Latency and throughputs computation}\label{sec:latency_throughputs}
The practical computation of latency and throughputs then follow from a standard result in queueing, known as Little's law, see \citet{Little1961}.
\begin{theo}
Consider a stationary queueing system and denote by
\begin{itemize}
  \item $1/\lambda$ the mean of the unit inter-arrival times
  \item $L$ be the mean number of units in the system
  \item $W$ be the mean time spent by units in the system
\end{itemize}
We have
$$
L = \lambda \cdot W
$$
\end{theo}
\begin{itemize}
  \item Latency is the confirmation time of a transaction
    $$
    \text{Latency} = W + \frac{1}{\mu} = \frac{\mathbb{E}(N^q)}{\lambda} + \frac{1}{\mu} =  \frac{p}{(1-p)\lambda} + \frac{1}{\mu}.
    $$
  \item Throughput is the number of transaction confirmed per time unit
  $$
    \text{Throughput} = \mu\mathbb{E}(N^q\mathbb{I}_{N^q\leq b}+b\mathbb{I}_{N^q> b}) = \mu\sum_{n = 0}^bn(1-p)p^n + bp^{b+1}.
  $$
\end{itemize}
Avenue for future research includes
\begin{itemize}
	\item the inclusion of priority consideration, accounting for the transaction fee, see \citet{Kawase2020}
	\item Refine the hypothesis of the queueing system to better adapt to the different consensus protocol, see \citet{Li2018} and \citet{Li2019}.
\end{itemize}
\newpage