\chapter{Measures of distance}
\label{dist}

We take a little time out here to consider some ideas regarding multivariate distance and introduce some properties of multivariate distance matrices.   These concepts are most obviously relevant when considering multivariate technique such as cluster analysis and scaling methods, where we wish to examine the difference between individuals.   In doing this, we need to find some definition of the concept of ``difference between individuals'', and will therefore consider a range of proximity measures.   We also provide some discussion of difference between variables.   These are currently important concepts in bio-informatic applications but earlier work in multivariate statistics involved consideration of variables which may be carrying similar information where cluster analysis of variables could be used as a preliminary data analytical exercise.   We start by considering one particular measure, the Mahalanobis distance.

\section{Mahalanobis Distance}
\label{standarddist}

The Mahalanobis distance has an important role in multivariate theory, albeit this is often an implied consideration rather than an explicit one.   For example, development of forms of discriminant analysis considered in chapter \ref{discriminant} involve this measure.   There are however a number of important distributional properties of the Mahalanobis distance which could be more used in determining multivariate normality.    It should be noted that use of standard distance requires a parametric view of the world, and in particular it is most applicable for symmetric distributions.   We follow \cite{Flury:1997} in providing the following exposition of the standard distance.

Firstly, if we consider the \emph{univariate} standard distance we see that this is a measure of the absolute distance between two observations in units of their standard deviation.
\marginnote{The standardisation is important; this measure it is invariant under non-degenerate linear transformations.   A univariate example would be given by considering $Y = \alpha  + \beta X$, where $\beta \neq 0$ and $\alpha$ are fixed constants.   Consider transforming $x_{1}$ and $x_{2}$ to $y_{i} = \alpha + \beta x_{i}; i = 1,2$. Then considering the standard distance between these two transformed variables we find:

\begin{eqnarray*}
d(y_{1},y_{2}) &=& \frac{|y_{1} - y_{2}|}{\sqrt{var(Y)}}\\
 &=&  \frac{|\beta(x_{1} - x_{2}|)}{\sqrt{\beta^{2}\sigma^{2}}}\\
 &=& d(x_{1},x_{2})
\end{eqnarray*}
}
Given $X$, a random variable with mean $\mu$ and variance $\sigma^{2} > 0$, the \emph{standard distance}, between two numbers $x_{1}$ and $x_{2}$ is defined as follows:

\begin{displaymath}
d(x_{1}, x_{2}) = \frac{|x_{1} - x_{2}|}{\sigma}
\end{displaymath}

\marginnote{Where $\sigma = 1$, this standard distance is the same as the Euclidean distance given later in section \ref{euclidean}}.

The univariate standard distance has a straightforward generalisation to a multivariate setting.   Considering now two vectors  $\boldsymbol{x}_{1}$ and $\boldsymbol{x}_{2}$, with a common covariance matrix $\boldsymbol{\Sigma}$ the multivariate standard distance is given by:

\begin{displaymath}
d(\boldsymbol{x}_{1},\boldsymbol{x}_{2}) = \sqrt{(\boldsymbol{x}_{1} - \boldsymbol{x}_{2})^{T}\boldsymbol{\Sigma}^{-1}(\boldsymbol{x}_{1},\boldsymbol{x}_{2}) }
\end{displaymath}

Depending on whichever textbook is consulted, this multivariate standard distance may be referred to as the \emph{statistical distance}, the \emph{elliptical distance} or the \emph{Mahalanobis distance}.   \cite{Flury:1997} notes that the squared Mahalanobis distance $d(\boldsymbol{x}_{1},\boldsymbol{x}_{2})^{2}$ is sometimes simply referred to as the Mahalanobis distance, although it is not a valid distance measure.   We refer here to the multivariate standard distance,  $d(\boldsymbol{x}_{1},\boldsymbol{x}_{2})$ as the Mahalanobis distance, and where necessary, to  $d(\boldsymbol{x}_{1},\boldsymbol{x}_{2})^{2}$ as the \emph{squared} Mahalanobis distance.

It is worth noting that this measure was originally proposed by
\cite{Mahalanobis:1930} as a measure of distance between two populations:

\begin{displaymath}
\Delta(\boldsymbol{\mu}_{1},\boldsymbol{\mu}_{2}) = \sqrt{(\boldsymbol{\mu}_{1} - \boldsymbol{\mu}_{2})^{T}\boldsymbol{\Sigma}^{-1}(\boldsymbol{\mu}_{1},\boldsymbol{\mu}_{2}) }
\end{displaymath}
which has an obvious sample analogue as the distance between two mean vectors:

\begin{displaymath}
\Delta(\boldsymbol{\bar{x}}_{1},\boldsymbol{\bar{x}}_{2}) = \sqrt{(\boldsymbol{\bar{x}}_{1} - \boldsymbol{\bar{x}}_{2})^{T}\boldsymbol{S}^{-1}(\boldsymbol{\bar{x}}_{1},\boldsymbol{\bar{x}}_{2}) }
\end{displaymath}
where $\boldsymbol{S}$ is the pooled estimate of $\boldsymbol{\Sigma}$ given by $\boldsymbol{S} = \left[ (n_{1}-1) \boldsymbol{S}_{1} +  (n_{2}-1) \boldsymbol{S}_{2} \right] / (n_{1} + n_{2} - 2)$.

Here, we are going to consider the distance between $\boldsymbol{x}$, a vector of random variables with mean $\boldsymbol{\mu}$ and covariance matrix $\boldsymbol{\Sigma}$ and its mean:

\begin{displaymath}
\Delta(\boldsymbol{x},\boldsymbol{\mu}) = \sqrt{(\boldsymbol{x} - \boldsymbol{\mu})^{T}\boldsymbol{\Sigma}^{-1}(\boldsymbol{x},\boldsymbol{\mu}) }
\end{displaymath}
and clearly we can find a sample analogue by estimating $\boldsymbol{\mu}$ by $\hat{\boldsymbol{x}}$ and $\boldsymbol{\Sigma}$ by $\boldsymbol{S} = \frac{1}{n-1} \boldsymbol{X}^{T}\boldsymbol{X}$.  We note that in \textbf{R}, the \verb+mahalanobis()+ function is intended to returns the \emph{squared} multivariate distance between a matrix $\boldsymbol{X}$ and a mean vector $\boldsymbol{\mu}$, given a user-supplied covariance matrix $\boldsymbol{\Sigma}$, i.e. we wish to calculate:

\begin{displaymath}
d(\boldsymbol{x}_{i},\hat{\boldsymbol{\mu}})^{2}
= (\boldsymbol{x}_{i} - \hat{\boldsymbol{\mu}})^{T}
\hat{\boldsymbol{\Sigma}}^{-1}
(\boldsymbol{x}_{i} - \hat{\boldsymbol{\mu}})
\end{displaymath}

We could also consider the Mahalanobis angle $\theta$ between two vectors at the origin:

\begin{displaymath}
\cos \theta = \frac{\boldsymbol{x}_{1}^{T} \boldsymbol{S}^{-1} \boldsymbol{x}_{2}}
{d(\boldsymbol{x}_{1},\boldsymbol{0})d(\boldsymbol{x}_{2},\boldsymbol{0})}
\end{displaymath}

This can be extracted from within $\boldsymbol{R}$ using the following:

\begin{Schunk}
\begin{Sinput}
> mahangle <- function(x1, x2, covmat){
+   zero <- vector("numeric", length(x1) )
+   num <- t(x1) %*% solve(covmat) %*% x2
+   denom <- sqrt(mahalanobis(x1, zero, covmat)) *
+      sqrt(mahalanobis(x2, zero, covmat))
+   angle <- acos(num / denom)
+   return(angle)
+ }
\end{Sinput}
\end{Schunk}

\subsection{Distributional properties of the Mahalanobis distance}

Remembering that where $z_{1}, \ldots, z_{p} \sim N(0,1)$, if we form $y = \sum_{j=1}^{p} z_{j}^{2}$ then $y \sim \chi_{p}^{2}$ \citep{Bilodeau+Brenner:1999}; for multivariate normal data, with $p$ variables, the squared Mahalanobis distance can be considered against a $\chi_{p}^{2}$ distribution:

\begin{equation}
(\boldsymbol{x}_{i} - \hat{\boldsymbol{\mu}})^{T}
\hat{\boldsymbol{\Sigma}}^{-1}
(\boldsymbol{x}_{i} - \hat{\boldsymbol{\mu}}) = \boldsymbol{z}^{T}\boldsymbol{z} \sim \chi^{2}_{p}
\end{equation}

This immediately affords one method for assessing multivariate normality, quantiles of the Mahalanobis distance of $\boldsymbol{x}_{i}$, $i = 1, \ldots, n$ with respect to $\boldsymbol{\mu}$ can be plotted against quantiles of the $\chi^{2}_{p}$ distribution as an assessment of multivariate normality.

\begin{figure}
\begin{center}
\includegraphics[width = 0.5\textwidth]{images/DistChiPlot}
\caption{QQ plot of squared Mahalahobis distance plotted against $\chi^{2}$ distribution}
\label{qqchimahlanobis}
\end{center}
\end{figure}

We can also define contours as a set of points of equal probility in terms of equal Mahalanobis distance:

\begin{equation}
(\boldsymbol{x}_{i} - \hat{\boldsymbol{\mu}})^{T}
\hat{\boldsymbol{\Sigma}}^{-1}
(\boldsymbol{x}_{i} - \hat{\boldsymbol{\mu}}) = \boldsymbol{z}^{T}\boldsymbol{z} = c^{2}
\end{equation}

for any constant $c > 0$.  We will also find later in section \ref{mahalpca} that the squared Mahalanobis distance is equivalent to the sum of squared prinicipal component scores.   However, this chapter on distance is implicitly geared towards presentations in the chapter \ref{clustan} on cluster analysis as well as  chapter \ref{mds} on scaling methods.   In that context it is worth noting that Mahalanobis distance is rarely used in cluster analysis, certainly \cite{Kendall:1975} points out its limitations in this context.  Sporadic reports in the literature include \cite{Maronna+Jacovkis:1974} who report use of a particular clustering algorithm, $k$-means, with the Mahalanobis distance whereas \cite{Gnanadesikan+etal:1993} use it with hierarchical cluster analysis.   This latter work may illustrate one of the difficulties in using the Mahalanobis distance in the requirement to assume a common covariance.     However, whilst not proposing it's use in an automatic clustering algorithm, \cite{Atkinson+etal:2004} report use of Mahalanobis distance within the forward search to reliably identify subgroups within the data.   They propose a small modification to the Mahalanobis distance for use in cluster analysis as follows.   The Mahalanobis distance is multiplied by $(|\hat{\boldsymbol{\Sigma}}^{-1}_{k}|^{1/2})^{r}$ for group $k$.   Where $r=0$ we have the usual distance, when $r=1$ we have what they call the \emph{standardised} Mahalanobis distance which eliminates the different variance between groups.

Having provided an overview of one distributionally important distance measure, before considering further measures we consider a few definitions.   \cite{Flury:1997} notes that the squared Mahalanobis distance does not satisfy the axioms of distance.

\section{Definitions}
\label{distdefinitions}

We now formalise our idea of a proximity measure.  This term encapsulates both similarity and disssimilarity measures which have the obvious interpretation (measuring similarity and dissimilarity between entities), and can be found from each other by means of an appropriate monotonic transformation.   We usually assume that these measures are symmetic.

A \emph{distance} can be defined as a function $d(\cdot)$ that satisfies the following properties:

\begin{itemize}
\item[(1)] Non-negative, that is $d(\boldsymbol{x},\boldsymbol{y}) \geq 0$ for all $\boldsymbol{x},\boldsymbol{y} \in \mathbb{R}^{p}$ and
\item[(2)] Identified, that is $d(\boldsymbol{x},\boldsymbol{x}) = 0$ for all $\boldsymbol{x} \in \mathbb{R}^{p}$;
\item[(3)] Symmetric, that is $d(\boldsymbol{x},\boldsymbol{y}) = d(\boldsymbol{y},\boldsymbol{x})$ for all $\boldsymbol{x},\boldsymbol{y} \in \mathbb{R}$;
\end{itemize}

In addition to satisfying these three properties, a \emph{metric} also satisfies the following two properties:
\begin{itemize}
\item[(4)] Definite, that is $d(\boldsymbol{x},\boldsymbol{y}) = 0$ if and only if $\boldsymbol{x} = \boldsymbol{y}$ for all $\boldsymbol{x},\boldsymbol{y} \in \mathbb{R}^{p}$;
\item[(5)] Triange inequality $d(\boldsymbol{x},\boldsymbol{y}) + d(\boldsymbol{y},\boldsymbol{z}) \geq d(\boldsymbol{x},\boldsymbol{z})$ for all $\boldsymbol{x},\boldsymbol{y},\boldsymbol{z} \in \mathbb{R}$
\end{itemize}

It is worth noting that it is possible to compute a similarity measure, often denoted $s$, where $0 \leq S \leq 1$.   A \emph{similarity} function $s(\cdot,\cdot)$ satisfies (1) non-negativity $s(\boldsymbol{x}, \boldsymbol{y}) \geq 0$, (2) symmetry $S(\boldsymbol{x},\boldsymbol{y}) = s(\boldsymbol{y},\boldsymbol{x})$ as well as:

\begin{itemize}
\item[(3)] $s(\boldsymbol{x}, \boldsymbol{y})$ increases in a monotone fashion as $\boldsymbol{x}$ and $\boldsymbol{y}$ become more similar.   A \emph{dissimilarity} function satisfies the first two but clearly 3 is reversed, i.e. it decreases as  $\boldsymbol{x}$ and $\boldsymbol{y}$ become more similar.
\end{itemize}

Dissimilarity is the opposite of similarity, therefore any monotonically decreasing transformation of $s$ can provide a dissimilarity measure.   The most obvious tranform would be to take $d = 1 - s$ but we will consider a few alternatives later.

\section{Distance between points}
\label{pointdistance}

Two \textbf{R} packages are needed to provide most of the distance functions considered here.   In addition to the default \verb+stats+ library, which provides the \verb+dist()+ function, we require the \verb+cluster+ package for the \verb+daisy()+ function.   Some further correlation based measures can be found in the \verb+Dist()+ function in the \verb+amap+ package as well as \verb+BioBase+ from Bioconductor.

We now consider a range of ways in which a multivariate distance can be measured.  In conducting an analysis, some decision needs to be made as to whether to scale variables, or whether to remove highly correlated variables from the analysis.   For example, \cite{Gnanadesikan:1997} gives an artificial example which illustrates how rescaling variables can subsequently alter impression of groupings.

\subsection{Quantitative variables - Interval scaled}
\label{distancequant}

It is reasonably straightforward to suggest a number of dissmilarity measures $d_{ij}$ which measure the distance between individual $i$ and $j$.

%\begin{itemize}
%\item
\subsection{Euclidean distance}.
\label{euclidean}

The Euclidean distance, or the $l_{2}$ norm, is perhaps the most commonly used distance measure.   As mentioned in section \ref{standarddist}, this distance could be considered simply as the Mahalanobis distance where $\sigma$ = 1.   Especially in the context of cluster analysis, where we hope to identify distinct sub-groups within the data, it is not clear how we might determine the covariance matrix hence the Mahalanobis distance has seen little use.   The Euclidean distance, which is quite simply the square root of the squared distance between any two vectors, which can be quite simply interpreted as the physical distance between two $p$-dimensional points is also a convenient measure to understand.  Formally, we can express this measure as:

\begin{displaymath}
\label{euclideanF}
d_{ij} =  \left( \Sigma_{k=1}^{p} (x_{ik} - x_{jk})^2 \right)^{\frac{1}{2}}
\end{displaymath}

where we are trying to measure the distance between observations in row $i$ and row $j$, in other words $x_{ik}$ is the $k$th observation in row $i$, and $x_{jk}$ is the corresponding $k$th observation in row $j$.   Euclidean distance can be readily calculated in \textbf{R} using the \verb+dist()+ function with the default \verb+method = "euclidean"+, as well as by \verb+daisy()+ with the default \verb+metric = "euclidean"+, although in \verb+daisy()+ it is possible to standardise the data within the calculations by adding \verb+stand = TRUE+ to the function call.

\subsection{Scaled Euclidean distance}
\label{scaledeuclidean}

It is possible to introduce a suitable weight $w_{k}$ such as the inverse of the standard deviation of the $k$th variable, i.e. $w_{k} = s_{k}^{-1}$, or even the inverse of the range of the data.

\begin{displaymath}
\label{scaledeuclideanF}
d_{ij} = \sqrt{\left( \Sigma_{k=1}^{p} w_{k}^{2}(x_{ik} - x_{jk})^2 \right)}
\end{displaymath}

No explicit routines are available to compute this measure, but clearly if the co-ordinates are rescaled by $\sqrt{w_{k}}$ this can be calculated implicitly.

\subsection{City Block metric}
\label{cityblock}

The City Block metric, formally referred to as an $l_{1}$ norm, measures the absolute difference between two vectors.   It is so-named because it measures the distance between two points in terms of movements parallel to the axis and therefore resembles the distance between two points in a city.   \cite{Krause:1975} (who had obviously never been in a London taxi) called this distance the \emph{taxicab} distance, \cite{Brandeau+Chiu:1988} used the term \emph{rectilinear}, but perhaps the most common alternative name is \emph{Manhattan}, suggested by \cite{Larson+Sadiq:1983} reflecting the famous city block layout in Manhattan.   Formally, we can express this distance as:

\begin{displaymath}
\label{cityblockF}
d_{ij} =  \left( \Sigma_{k=1}^{p} |x_{ik} - x_{jk}| \right)
\end{displaymath}

It can be calculated in R using the \verb+dist()+ function with \verb+method = "manhattan"+

\subsection{Minkowski metric}
\label{minkowski}

The Minkowski metric, or the $l_{r}$ norm, is a generalisation of the Manhattan and Euclidean distances.

\begin{displaymath}
\label{minkowskiF}
d_{ij} =  \left( \Sigma_{k=1}^{p} |x_{ik} - x_{jk}|^\lambda \right)^{1/\lambda}
\end{displaymath}

Where $\lambda = 1$ we have the Manhattan metric, where $\lambda = 2$ we have the Euclidean distance.   It can be noted that increasing $\lambda$ exaggerates dissimilar units relative to similar ones.   This metric can be calculated in R using the \verb+dist()+ function with \verb+method = "minkowski"+ but additionally requires an argument to \verb+p+ to set $\lambda$, the power of this distance.   Therefore, for example \verb+dist(x, method = "minkowski", p=2)+ gives the Euclidean distance for matrix \verb+x+.

\subsection{Canberra metric}
\label{canberra}

The Canberra metric \citep{Lance+Williams:1966} can be regarded as a generalisation of binary dissimilarity measures, and is very sensitive to small changes close to $x_{ik} = x_{jk} = 0$.   It can be scaled by division by $p$, the number of variables to ensure it lies in the range (0,1).   Terms with zero numerator and denominator are omitted from the sum and treated as if the values were missing.

\begin{displaymath}
\label{canberraF}
d_{ij} = \left\{ \begin{array}{ll} 0 & for\ x_{ik} = x_{jk} = 0\\
  \Sigma\left( \frac{|x_{ik} - x_{jk}|}{ |x_{ik} + x_{jk}|} \right) & for\  x_{ik} \neq 0\ or\  x_{jk} \neq 0 \end{array} \right.
\end{displaymath}

This metric can be calculated in \textbf{R} using the \verb+dist()+ function with \verb+method = "canberra"+

\subsection{Czekanowski Coefficient}
\label{czekanowski}

Finally, we mention the Czekanowski Coefficient, which for continuous variables can be given as:

\begin{displaymath}
\label{czekanowskiF}
d_{ij} = 1 - \frac{2 \sum_{k=1}^{p} min(x_{ik},x_{jk})}{\sum_{k=1}^{p}(x_{ik} + x_{jk})}
\end{displaymath}

%\end{itemize}

\subsection{Distance between variables}
\label{corrdist}

We next consider a number of correlation based distance measures.   Note that when used conventionally for calculating the correlation between two variables we work with standardised columns.  In order to measure the similarity between two individuals we must therefore work with standardised rows, this may not be a sensible procedure.   For example, if variables are measured on different scales the idea of a row mean may not be clear.   There is further material in the literature questioning the use of these measures \citep{Jardine+Sibson:1971,Fleiss+Zubin:1969} and \cite{Everitt+etal:2001} note that correlation measures cannot distinguish the size of two different observations, giving the example $\boldsymbol{x}^{T}_{1} = c(1,2,3)$ and $\boldsymbol{x}^{T}_{2} = c(1,2,3)$ have correlation $\rho_{12} = 1$ yet $\boldsymbol{x}^{T}_{2}$ is three times the size of $\boldsymbol{x}^{T}_{1}$.   Nevertheless, correlation based measures have become particular popular in a bio-informatics setting where some of the noted limitations do not apply (all variables are measured on a comparable scale) and in fact it is not always clear what a row and a column mean in that application area.

We therefore consider four four distances that can be obtained a correlation measure.  Some thought needs to be given to determining the transformation from a correlation coefficient to a distance measure.   The Pearson correlation coefficient is defined in the range $-1 \leq \rho_{ij} \leq 1$.   \cite{Everitt+etal:2001} suggest using $d_{ij} = \frac{1 - \rho_{ij}}{2}$.  \cite{Gentleman+etal:2005} suggest that it may be appropriate under some circumstances to use the absolute value of the correlation, that is $d_{ij} = 1 - |\rho_{ij}|$ which means that there will be little distance between rows having strong positive and strong negative correlation.   In terms of measuring the dissimilarity between variables, \cite{Krzanowski:2000} suggests a further alternative using  $d_{ij} = 1 - (\rho_{ij})^{2}$.   Examining pre-Bioinformatics data, \cite{Lance+Williams:1979} who compared a number of transformations and expressed a strong preference for the first transformation, and a strong disdain for the third.

It should be noted that these measures are quite badly affected by outliers.   As a result, non-parametric versions may often be preferred.   Conversely, these measures are invariant to change of location or scale transformation which is rather useful.   It should be noted in bio-informatics practice that they tend to group genes whose expression patterns are linearly related, there is some empirical support from that application for their use in a particular context.

\subsection{Pearson correlation distance}
\label{pearsondist}

\begin{displaymath}
\label{pearsondistF}
d(x_{ij},x_{ik}) = 1 - \rho_{ij} = 1 - \frac{\sum_{i=1}^{p} (x_{ij} - \bar{x}_{\cdot j}) (x_{ik} - \bar{x}_{\cdot k}}{\sqrt{\sum_{i=1}^{p} (x_{ij} - \bar{x}_{\cdot j})^{2} \sum_{i=1}^{p} (x_{ik} - \bar{x}_{\cdot k})^{2}}}
\end{displaymath}

Where data are scaled, i.e. mean centred and standardised by the variance so that $\boldsymbol{x}_{\cdot j}$ and $\boldsymbol{x}_{\cdot k}$ are $p$ variable vecotres with zero mean and unit variance the relationship between the Euclidean distance and the Pearson correlation is given by:

\begin{displaymath}
d_{ij}^{(Euclidean)} = \sqrt{2p(1-\rho_{ij})}
\end{displaymath}

The pearson based distance measure can apparently be obtained from \verb+Dist()+ in the \verb+amap+ package, where it is referred to as the ``Centred Pearson'' by specifying \verb+method = "correlation"+ in the function call.

\subsection{Cosine correlation coefficient}
\label{cosinedist}

This is similar to the Pearson Correlation coefficient based distance measure but without the mean standardisation

\begin{displaymath}
\label{cosinedistF}
d(x_{ij},x_{ik}) = 1 - \frac{\boldsymbol{x}_{\cdot j}^{T} \boldsymbol{x}_{\cdot k}}{||\boldsymbol{x}_{\cdot j}|| ||\boldsymbol{x}_{\cdot k}||} = 1 - \frac{|\sum_{i=1}^{p} (x_{ij}) x_{ik} }{\sqrt{\sum_{i=1}^{p} x_{ij}^{2} \sum_{i=1}^{p} x_{ik}^{2}}}
\end{displaymath}

The cosine correlation based distance measure, referred to as the ``Not-centred Pearson  can be obtained from \verb+Dist()+ in the \verb+amap+ package by specifying \verb+method = "pearson"+ in the function call.    It is not clear from the help file how the correlation measure is transformed into a distance measure.

\subsection{Spearman sample correlation distance}
\label{spearmandist}

\begin{displaymath}
\label{spearmandistF}
d(x_{ij},x_{ik}) = 1 - \rho_{ij} =
1 - \frac{\sum_{i=1}^{p} (rank(x)_{ij} - rank(\bar{x})_{\cdot j}) (rank(x)_{ik} - rank(\bar{x})_{\cdot k})}
{\sqrt{\sum_{i=1}^{p} (rank(x)_{ij} - rank(\bar{x})_{\cdot j})^{2} \sum_{i=1}^{p} ( rank(x)_{ik} - rank(\bar{x})_{\cdot k})^{2}}}
\end{displaymath}

This requires \verb+spearman.dist()+ in package \verb+bioDist+, and can be computed via \verb+Dist()+ in the \verb+amap+ package with a call containing \verb+method="spearman"+.

\subsection{Kendall's $\tau$ sample correlation distance}
\label{kendalltaudist}

\begin{displaymath}
\label{kendalltaudistF}
d(x_{ij},x_{ik}) = 1 - \tau_{ij} = 1 - \frac{\sum_{i=1}^{p} sign(x_{ij} - \bar{x}_{\cdot j}) sign(x_{ik} - \bar{x}_{\cdot k})}{p(p-1)}
\end{displaymath}

This requires \verb+tau.dist()+ in package \verb+bioDist+

\subsection{Quantitative variables: Ratio Scaled}

\cite{Kaufman+Rousseeuw:1989} briefly discuss ratio scaled variables, and give examples including micro-organism growth which follows and exponential power law.   Clearly, we could just consider these as interval scale variables and use any of the previous measures.   They discuss the possibility of taking a logarithmic transformation of such data where it may be appropriate, obviously having the exponential growth application in mind.  The logarithmic transformation can be dealt with in \verb+daisy()+ by using the \verb+type="logratio"+ command.   Alternatively, it would be possible to treat such variables as being continuous ordinal data and use rank-based non-parametric procedures.   As discussed further in section \ref{qualitative}, using the  \verb+type="ordratio"+ command within \verb+daisy()+ generates standardised variates from the ranks which are subsequently analysed with a scaled City Block metric, alternativly, the two non-parameteric correlation derived measures described in section \ref{spearmandist} and \ref{kendalltaudist} may also be useful.

\subsection{Dichotomous data}
\label{dichotomousdist}

Where $x_{ik}$ can only take one of two values, these are coded as $0$ and $1$:

\begin{tabular}{lr|cc}
 & & Object & Two \\
 & & 1 & 0\\
\hline
 & 1 & a & b\\
Object 2 & & & \\
 & 0 & c & d\\
\end{tabular}
where $p = a + b + c + d$, some common dissimilarity measures are:

In this table, $a$ denotes an agreement (both objects have a zero in the same position), $d$ shows an agreement where both objects have a one, $c$ and $d$ denote the two possible disagreements.    We should firstly comment in more detail on the nature of dichotomous data.   \cite{Gower:1971} distinguishes two types of binary variables, symmetric and assymetric.   Binary variables such as gender (male and female) or handedness (left or right) are clearly symmetric and the distance measure should not change depending on the way we code these two levels as 0 and 1.   In other words, $a$ and $d$ should act the same way in the table.

We can therefore consider the following symmetric measures.

\subsection{(Based on the) simple matching coefficient}

The \emph{simple matching coefficient}, also known as the \emph{M-coefficient} or the \emph{affinity index}, is quite simply the proportion of variables in agreement in two objects.   The distance measure is found by subtracting this value from 1 (or calculating the proportion of disagreements):

\begin{equation}
\label{simplematch}
d_{ij} = 1 - \frac{a + d}{a + b + c + d} = \frac{b + c}{a + b + c + d}
\end{equation}

This measure can be calculated in \verb+daisy()+ by providing a list indicating those variables to be regarded as symmetric, i.e. \verb+(list("symm", "symm", "symm")+.   It may be noted in passing that if we force a calculation of Manhattan distance we get estimate $b+c$ we omit standardisation and simply calculate the sum of disagreements.   Also, the Euclidean distance is the square root of the dissimilarity derived from the simple matching coefficient.   Two further symmetric measures include \cite{Rogers+Tanimoto:1960} which doubles the weight of the disagreements:

\begin{displaymath}
d_{ij} = 1 - \frac{a + d}{(a + d) + 2(b + c)} = \frac{2(b + c)}{(a + d) + 2(b + c)}
\end{displaymath}

and the \cite{Sokal+Sneath:1963} measure which doubles the weight of the agreements:

\begin{displaymath}
d_{ij} = 1 - \frac{2(a + d)}{2(a + d) + (b + c)} = \frac{b + c}{2(a + d) + (b + c)}
\end{displaymath}

All three measures are monotonically related and there seems little imperative to use anything other than the simple matching coefficient based dissimilarity measure.   Life does however get rather more interesting if we want to work with assymetric binary variables.   Some care is needed in analysis in determining whether binary variables are symmetric or assymetric.   A classical example would concern variables measuring presence or absence.   The thought is that if two individuals share the presence of some attribute we can consider them similar, but if they share the absence of an attribute we do not know whether they can be considered similar.   For example, if we collect data on individuals who travelled to a particular location, we can consider them similar if they both drove by car, but if neither drove by car it is clear there are a range of reasons, which could include not owning a car, preferring another form of transport, living within walking distance and so on.

\subsection{Jaccard coefficient}

Perhaps the most common assymetric measure of distance is the Jaccard Coefficient \cite{Sneath:1957}, which measures the proportion of agreements on the variable coded 1 among all such agreements and disagreements (i.e. ignoring all possible agreements on variable coded 0).  Formally, this can be set out as:

\begin{displaymath}
d_{ij} = 1 - \frac{a}{a + b + c} =  \frac{b + c}{a + b + c}
\end{displaymath}

This seems to be the value calcuated by \textbf{R}, when \verb+method="binary"+ is used in the call to \verb+dist()+, it is also available in \verb+daisy()+ when the a list is supplied which indicates those variables to be considered as binary assymetric variables, i.e. \verb+list("asym", "asym")+

As with symmetric measures, there are a few alternatives which alter the weightings.

\subsection{Czekanowski coefficient}

The Czekanowski coefficient \citep{Dice:1945} increases the weight of the agreements

\begin{displaymath}
d_{ij} = 1 - \frac{2a}{2a + b + c} = \frac{b + c}{2a + b + c}
\end{displaymath}

whearas the \cite{Sokal+Sneath:1963} coefficient increases the weight of the disagreements:

\begin{displaymath}
d_{ij} = 1 - \frac{a}{a +2(b + c)} = \frac{2(b + c)}{a + 2(b + c)}
\end{displaymath}

We extract a small part of an example given by \cite{Kaufman+Rousseeuw:1989} to illustrate the non-monotonicity of the symmetric and asymmetric measures.

\begin{tabular}{l|rrrrrrrrrr}
Name & $x_{1}$ & $x_{2}$ & $x_{3}$ & $x_{4}$ & $x_{5}$ & $x_{6}$ & $x_{7}$ & $x_{8}$ & $x_{9}$ & $x_{10}$  \\
\hline
Ilan       & 1 & 0 & 1 & 1 & 0 & 0 & 1 & 0 & 0 & 0\\
Jacqueline & 0 & 1 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0\\
Lieve      & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 0\\
Peter      & 1 & 1 & 0 & 0 & 1 & 0 & 1 & 1 & 0 & 0\\
\end{tabular}

where $x_{1}$ = Sex(Male = 1, Female = 0), $x_{2}$ = Married(Yes = 1, No = 0), $x_{3}$ = Hair(Fair = 1, Dark = 1), $x_{4}$ = Eyes (Blue = 1, Brown = 0), $x_{5}$ = Wears Glasses(Yes = 1, No = 1), $x_{6}$ = Face (Round = 1, Oval = 0), $x_{7}$ = Outlook(Pessimist = 1, Optimist = 0), $x_{8}$ = Type(Evening = 1, Morning = 0) $x_{9}$ = Only Child (1 = Yes, 0 = No) $x_{10}$ = Handedness (1 = Left, 0 = Right).

Using the symmetric, simple matching coefficient based distance measure they note that:
\begin{displaymath}
d(Jacqueline, Lieve) = 0.300\ d(Ila, Peter) = 0.500
\end{displaymath}

whereas for the asymmetric, Jaccard coefficient we have:

\begin{displaymath}
d(Jacqueline, Lieve) = 0.750\ d(Ila, Peter) = 0.714
\end{displaymath}

 Although \cite{Kaufman+Rousseeuw:1989} state that the Jaccard coefficient is inappropriate, it could be argued that some of these variables are assymetric (there are a variety of reasons why someone might record that they were not-married).   Nevertheless, the point of their illustration was to highight the non-monotonicity.   Whilst we expect the measures to be different, note that for the symmetric coefficient $d(Jacqueline, Lieve) < d(Ila, Peter)$, whereas for the assymetric coefficient $d(Jacqueline, Lieve) > d(Ila, Peter)$.

\subsection{Similarities between variables}

\begin{table}
\begin{center}
\begin{tabular}{cccc}
 && \multicolumn{2}{c}{Variable 1} \\
&& + & - \\
Variable & + & a & b\\
 & - & c & d\\
\end{tabular}
\end{center}
\end{table}

\begin{displaymath}
\chi^{2} = \frac{(ad-bc)^{2} (a + b + c + d)}
{(a + b)(a + c) (c + d) (b + d)}
\end{displaymath}

which may require some standardisation:

\begin{displaymath}
d_{kl} = 1 - \sqrt{\frac{\chi^{2}}{a  + b + c + d}}
\end{displaymath}

\subsection{Qualitative variables}
\label{qualitative}

Following \cite{Kaufman+Rousseeuw:1989} we consider a variable where we have $m = 1, \ldots, M$ states.   It would be possible to create a set of $M$ binary variables, with 0 indicating absence of a particular category within a variable and 1 indicating presence.   Alternatively, a nominal variable could be collapsed in some suitable manner.   However, \cite{Sokal+Michener:1958} suggest a simple matching coefficient, a corresponding distance can be found by substracting this from 1.   Denoting the number of variables on which objects $i$ and $i$ agree by u, and the total number of variables by $p$ this can be expressed as:

\begin{displaymath}
d(x_{ij},x_{ik}) = 1 - \frac{u}{p} = \frac{p-u}{p}
\end{displaymath}
This measure is invariant to the codings used or the order of the variables, and can be extended in the same way as that suggested for binary variables by \cite{Rogers+Tanomoto:1960} and \cite{Sokal+Sneath:1963} by doubling the weight of disagreements and agreements respectively.   \cite{Kaufman+Rousseeuw:1989} review proposals to weight the measure depending on the size of $M$.

The simple matching coefficient is available in \textbf{R} by using \verb+daisy()+ having specified that the variable concerned is a factor, by ensuring the elements of \verb+x+ supplied to the function have class \verb+factor+.

It's also obvious that such variables can be ordered, and also that ordered variables may be derived from continuous data.   We can either obtain the ranks and treat the ranks as continuous variables applying any of the quantitative distance measures discussed above.   A possible derivation, having first scaled the ranks is given by:

\begin{displaymath}
\label{daisyrank}
z_{ij} = \frac{r_{ij} - 1}{M_{j} - 1}
\end{displaymath}

When using \verb+daisy()+, if a discrete variable has the class set to \verb+"ordered"+, or if a continuous variable is supplied with the argument \verb+type = "ordratio"+ $z_{ij}$ will be computed as in figure \ref{daisyrank} and treated as a continuous variable.   Distance will subsequently be computed by means of the City Block distance, which will be scaled by the number of such variables analysed.

Alternative, one of the non-parametric correlation measures in section \ref{spearmandist} or section \ref{kendalltaudist} could be used, especially where we are measuring distance between variables rather than between individuals.

\subsection{Different variable types}
\label{gowers}

Finally, we consider the possibility that a particular data set contains a variety of variable types.   It might be possible to treat all variables as interval scaled continuous variables, or somehow recode them all as binary or ordinal variables.  It may be better to find some way of combining distance that has been measured in the most appropriate way for each type of variable.  As a result, possibly the most popular method for measuring dissimilarity in this situation has been derived from Gower's coefficient of similarity \cite{Gower:1971}.   In its original incarnation, this measure could combine interval, nominal and binary data.   Consider the following, where we basically sum the individual similarities however calculated and divide them by the total number of applicable comparisons:

\begin{equation}
d(x_{ij},x_{jk}) = 1 - \frac
{\sum_{k=1}^{p} \delta_{ijk} s_{ijk}}
{\sum_{k=1}^{p} \delta_{ijk}}
\end{equation}

The indicator  $\delta_{ijk}$ is set to 1 when both measurements for $x_{ij}$ and $x_{ik}$ are non-missing, it is zero otherwise.   It is also zero for binary variables where there is a $0-0$ match, i.e. the original measure assumed assymetric dichotomous variables.   We briefly consider how similarities for each of the three variable types is calculated:

\begin{itemize}
\item Quantitative (interval scaled) variables.

The similarity measure is given by:
\begin{equation}
\label{gowercont}
s_{ijk} = 1 - \frac{|x_{ik} - x_{jk}|}{range\ of\ variable\ k}
\end{equation}
This is essentially the City Block distance with the extra assumption that all variables had first been standardised by dividing by their range.   If there are mixed variables within the data frame or matrix \verb+x+ supplied to \verb+daisy()+, this standardisation is applied by default (regardless of any arguments supplied to \verb+stand+.

\item Qualitative (nominal) variables.
These are derived from the simple matching coefficient, the similarity is therefore the proportion of matches among all possible matches:
\begin{displaymath}
s_{ijk} = \left\{ \begin{array}{r} 1\ \mbox{if i and i agree on variable k} \\ 0\ \mbox{otherwise} \end{array} \right.
\end{displaymath}

\item Dichotomous variables
The original incantation assumed asymmetric variables, hence the Jaccard coeffcient is used.   If we consider $k = 1, \ldots, 4$ variables for individuals $i$ and $j$, we can see the possible outcomes:
\begin{tabular}{rcccc}
$i$ & 1 & 1 & 0 & 0\\
$j$ & 1 & 0 & 1 & 0\\
\hline
$s_{ijk}$ & 1 & 0 & 0 & 0\\
$\delta_{ijk}$ & 1 & 1 & 1 & 0\\
\end{tabular}
\end{itemize}

The original measure has been extended by \cite{Kaufman+Rousseeuw:1989} (who set out the calculations as distances rather than similarities) to incorporate symmetric binary and ordinal and ratio variables.   Ordinal variables are ranked, and the ranks used in \ref{gowercont}, ratio variables are either ranked or logged and then \ref{gowercont} is used.   As has been noted earlier, this is achieved by setting the class of the variables to be "numeric", "factor" or "ordered", or providing the arguments \verb+type =  "asymm", "symm", "ordratio", "logratio"+ to estimate appropriate measures for assymetric binary variables (Jaccard), symmetric binary, ordinal ratio variables or log transformed ratio variables respectively.

It should be noted that \cite{Gower:1971} shows, provided there are no missing values the $n \times n$ similarity matrix obtained from an $n \times p$ data matrix $\boldsymbol{X}$ is positive semi-definite.   If we obtain a dissimilarity matrix from this measure using $d(x_{ik},x_{jk}) = \sqrt{(1 - s(x_{ik},x_{jk})}$ the resultant matrix:

\begin{displaymath}
\Delta = \left( \begin{array}{rrrr}
0 & d(\boldsymbol{x}_{1},\boldsymbol{x}_{2}) & \cdots & d(\boldsymbol{x}_{1},\boldsymbol{x}_{p})\\
d(\boldsymbol{x}_{2},\boldsymbol{x}_{1}) & 0  & \cdots & d(\boldsymbol{x}_{2},\boldsymbol{x}_{p})\\
\vdots & \vdots & \cdots & \vdots \\
d(\boldsymbol{x}_{p},\boldsymbol{x}_{1})  & d(\boldsymbol{x}_{p},\boldsymbol{x}_{2}) & \cdots  & 0 \end{array} \right)
\end{displaymath}
is Euclidean.   We will next consider this important property of proximity matrices, it will particularly inform later developments in terms of metric scaling.

\section{Properties of proximity matrices}

A few words are placed here concerning proximity matrices, the $n \times n$ matrix comparing every individual with each other.   It is possible that the only information available in a particular study is such a matrix, as happens with sensory experiments or the rather famous study comparing matched judgements on morse code characters \citep{Rothkopf:1957}.  Some properties of these matrices will be important in later developments, particularly scaling as discussed in chapter \ref{mds}.   Earlier, in section \ref{gowers}, we rather glibly stated that a dissimilarity matrix obtained from Gower's coefficient of similarity can be Euclidean.   As might be anticipated, a matrix where the elements have been derived from the Euclidean distance (section \ref{euclidean}) is also Euclidean, but the concept requires further examination.   \cite{Gower:1966} demonstrated that Euclidean properties could be met by transforming for the elements $s(x_{i},x_{j})$ of a similarity matrix $\boldsymbol{S}$ to the elements $d(x_{i},x_{j})$ of a dissimilarity matrix $\boldsymbol{D}$:

\begin{displaymath}
d(x_{i},x_{j}) = \sqrt{1 - s(x_{i},x_{j})}
\end{displaymath}

Denoting the distance between two points by $d(x_{i},x_{j})$ and the proximity between two individuals by $\delta(x_{i},x_{j})$, we are ultimately interested in the proximity matrix $\boldsymbol{\Delta}$

When forming a  matrix $\boldsymbol{\Delta}$ from a dissimilarity matrix $\boldsymbol{D}$, \cite{Lingoes:1971} suggested tranforming the elements by $\delta(x_{i},x_{j}) = \sqrt{d(x_{i},x_{j})^{2} + c_{1}}$, \cite{Cailliez:1983} suggested  $\delta(x_{i},x_{j}) = d(x_{i},x_{j}) + c_{1}$, both providing methods for finding the constants $c_{1}$ and $c_{2}$.

In general, when considering the $n \times n$ dissimilarity matrix, $\Delta$, containing elements $d(\boldsymbol{x}_{i\cdot}, \boldsymbol{x}_{j\cdot})$.   This matrix can be considered Euclidean if the $n$ individuals can be represented as points in space such that the Euclidean distance between points $i$ and $j$ is $d(\boldsymbol{x}_{i\cdot}, \boldsymbol{x}_{j\cdot})$.   In general, $\boldsymbol{\Delta}$ is Euclidean if and only if the following matrix is positive semi-definite:
\begin{displaymath}
(\boldsymbol{I} - \boldsymbol{1}\boldsymbol{s}^{T}) \boldsymbol{\Gamma} (\boldsymbol{I} - \boldsymbol{1}\boldsymbol{s}^{T})
\end{displaymath}
where  $\boldsymbol{\Gamma}$ has elements $\frac{1}{2}d(\boldsymbol{x}_{i\cdot}, \boldsymbol{x}_{j\cdot})^{2}$, $\boldsymbol{I}$ is the identity matrix, $\boldsymbol{1}$ is a vector of $n$ ones and $\boldsymbol{s}$ is an $n$-element vector such that $\boldsymbol{s}^{T}\boldsymbol{1} = 1$.

Important special cases are where $\boldsymbol{s} = \frac{1}{n}\boldsymbol{1}$ which centres at the origin (proof is given in \cite{Mardia+etal:1979} regarding the Euclidian property) and where $\boldsymbol{s}$ which has 1 in its $i$th position and 0 elsewhere (proof is given by \cite{Gower:1984} of the Euclidean property).   The significance of the Euclidean property will be discussed in chapter \ref{mds} when we consider scaling methods.

In addition to the Euclidean property, a matrix can be considered \emph{metric}, if the metric inequality holds for all triplets $i$, $j$, $k$ within $\boldsymbol{\Delta}$

\begin{equation}
\label{triangleinequality}
d(\boldsymbol{x}_{i\cdot}, \boldsymbol{x}_{j\cdot}) + d(\boldsymbol{x}_{i\cdot}, \boldsymbol{x}_{k\cdot}) \geq d(\boldsymbol{x}_{j\cdot}, \boldsymbol{x}_{k\cdot})d(\boldsymbol{x}_{i\cdot}, \boldsymbol{x}_{j\cdot})
\end{equation}

If $\boldsymbol{Delta}$ is metric, then so are matrices with elements
$d(\boldsymbol{x}_{i\cdot}, \boldsymbol{x}_{j\cdot}) + c^{2}$ as well as $d(\boldsymbol{x}_{i\cdot}, \boldsymbol{x}_{j\cdot}) /( d(\boldsymbol{x}_{i\cdot}, \boldsymbol{x}_{j\cdot}) + c^{2})$ where $c$ is any real constant and $i\neq j$, as is  $d(\boldsymbol{x}_{i\cdot}, \boldsymbol{x}_{j\cdot})^{1/r}$ for $r \geq 1$ with again $i \neq j$.    In the case that $\boldsymbol{\Delta}$ is non-metric, \cite{Krzanowski+Marriott:1994I} indicate that where $c \geq max_{i,j,k} |\delta(x_{i},x_{j}) + \delta(x_{i},x_{k}) - \delta(x_{j},x_{k})|$, the matrix with elements $\delta(x_{i},x_{j}) + c$ is metric.

%If $\boldsymbol{\Delta}$ is metric then so are matrices with elements $\delta(x_{i},x_{j}) + c^{2}$,  $\delta(x_{i},x_{j})^{1/r}$ for $r geq 1$ and  $\delta(x_{i},x_{j}) / ( \delta(x_{i},x_{j}) + c^{2})$   krz and marriot vol 1

Again, the significance of the metric property will be discussed in chapter \ref{mds} when we consider scaling methods.

%%% Local Variables: ***
%%% mode:poly-noweb+r-mode ***
%%% TeX-master: "../book.tex"  ***
%%% End: ***


\chapter{Project Plan}

\section{Milestone $\#$ 1 – Formulation of model and Solving it}
\vspace{\baselineskip}
\subsubsection*{Goal}
\addcontentsline{toc}{subsubsection}{Goal}

Formulate and implement a \textit{mixed-integer linear programming} model with the aim of computing exact solutions for the dataset supplied by Royal Mail. For this milestone, the schedules generated  have complete freedom to schedule the meal-relief breaks anywhere within the atomic time blocks (representing the HGV's routes). At the same time, legality needs to be maintained by respecting the EU rules for the Driving and Working time directives.

\vspace{\baselineskip}
\subsubsection*{Progress}
\addcontentsline{toc}{subsubsection}{Progress}

Initial model formulations have been made and currently working towards testing with synthetic data. Testing of the current model formulation with inputs from the real dataset with the goal of discovering proposed improvements to the model, is the immediate next step. This will  determine whether an exact solution is possible.

\vspace{\baselineskip}
\subsubsection*{Fallback position}
\addcontentsline{toc}{subsubsection}{Fallback position}

Given that this milestone will in principle comprise the bulk of the project a fallback position, in the event that an exact solution is not possible, is to explore the use of approximation algorithms such as the \textit{randomised rounding }approach to reach approximate solutions for the problem.

\vspace{\baselineskip}
\subsubsection*{Timetable }
\addcontentsline{toc}{subsubsection}{Timetable }

Conducting critical analyses of the improvements to the model that arise from testing is to be undertaken until mid-February with the goal of determining by then whether an exact solution is possible. Depending on that outcome, by the end of February the focus will shift either to getting an approximate solution through approximation algorithm techniques or focusing on pursuing Milestones $\#$ 2, $\#$ 3.

\vspace{\baselineskip}
\section{Milestone $\#$ 2 – Incorporate Company Policy Requirements}
\vspace{\baselineskip}
\subsubsection*{Goal}
\addcontentsline{toc}{subsubsection}{Goal}

Incorporate in the model the Royal Mail company policy that breaks should occur only after a certain subset of activities out of the set of possible activities undertaken by the drivers. This is to be modelled with \textit{precedence constraints}. Given the complexity that this task would introduce, approximation algorithms will almost certainly need to be explored.

\vspace{\baselineskip}
\subsubsection*{Timetable}
\addcontentsline{toc}{subsubsection}{Timetable}

Provided that Milestone $\#$ 1 has been achieved by the end of February, the period throughout March – April will be spent on this second milestone. If Milestone $\#$ 1 has not been achieved and implementation of the fallback position is necessary, an extra month will approximately need to be spent on Milestone $\#$ 1.

\vspace{\baselineskip}
\section{Milestone $\#$ 3 – Additional Constraints}
\vspace{\baselineskip}
\subsubsection*{Goal}
\addcontentsline{toc}{subsubsection}{Goal}

Incorporate jobs that have a time-window limitation assigned to them, modelled by fixed atomic blocks.

\vspace{\baselineskip}
\subsubsection*{Timetable}
\addcontentsline{toc}{subsubsection}{Timetable}

Whether or not this milestone can be pursued will depend on the other two milestones.

\vspace{\baselineskip}
\subsection*{Final Report Preparation}

\subsubsection*{Timetable}
\addcontentsline{toc}{subsubsection}{Timetable}

Expected to commence after the $``$Project health check-up$"$  meeting during the week 11/05 - 15/05 and is expected to last for one-month, hence from the 17/05 until \textbf{17/06} which is the \textbf{report submission deadline}.


\documentclass[]{article}
\usepackage{amsmath, amssymb, graphicx, float, dsfont}

%opening
\title{Notes on Least Squares Optimization}
\author{Delbert Yip}

\newcommand{\norm}[1]{\lvert #1 \rvert}

\begin{document}

\maketitle

\begin{abstract}
	These notes were taken from Chapter 8 from Strang's "Computational Science and Engineering," \cite{strang_computational_2012} and cover ordinary, weighted, and regularized least squares (Sections 8.1-8.3 of \cite{strang_computational_2012}). Some background material has been taken from Lay's "Linear algebra and its applications" \cite{lay_linear_2012}.
\end{abstract}

\section{Least Squares}
Consider a matrix A of size $m x n$ with $n$ independent columns, and $m > n$.
$A^T A$ is symmetric positive definite, as Rank(A) = $n$.
We wish to solve $Au = b$ for some vector $b$ with $m$ components. The solution $\hat{u}$ has $n$ components.

This gives rise to the \textbf{least squares problem}:
\begin{equation}
\min \parallel Au - b \parallel ^2
\end{equation}

Below are the normal equations, which give the solution $\hat{u}$:
\begin{align}
Au &= b \nonumber \\
A^T A \hat{u} &= A^T b
\end{align}

The residual of the cost function is $e = b - Au$, and has dimensions $m \times 1$. Thus,
\begin{equation}
A^T e = A^T b - A^T A u
\end{equation}
Therefore, $A^T e = 0$ when $u = \hat{u}$.

$e$ is in the nullspace of $A^T$, and is therefore perpendicular to the columns (column space) of $A$ (equivalently, $e$ is perpendicular to the rows/rowspace of $A^T$). Recall that orthogonality means that the dot product of two vectors is 0.

\subsection{Refresher on Orthogonality}
Before we start on the next section, let's review some concepts from linear algebra. Specifically, the orthogonality of vectors and vector spaces.

 Orthogonal Vectors
There is an analogue between geometry in Euclidean space and in $\mathds{R}^n$.

First, *two vectors $\mathbf{u}$ and $\mathbf{v}$ are perpendicular i.f.f. the distance between $\mathbf{u}$ and $\mathbf{-v}$ is the same as the distance between $\mathbf{u}$ and $\mathbf{+v}$.*

First, the distance between $\mathbf{u}$ and $\mathbf{-v}$:
\begin{align}
[\textrm{dist}(\mathbf{u}, \mathbf{-v})]^2 &= \parallel \mathbf{u}-(-\mathbf{v}) \parallel ^2 = \parallel \mathbf{u} +\mathbf{v} \parallel ^2 \\
&= (\mathbf{u} + \mathbf{v}) \cdot (\mathbf{u} + \mathbf{v}) \\
&= \parallel \mathbf{u} \parallel^2 + \parallel \mathbf{v} \parallel^2 + 2\mathbf{u} \cdot \mathbf{v}
\end{align}

Next, the distance between $\mathbf{u}$ and $\mathbf{+v}$:
\begin{align}
[\textrm{dist}(\mathbf{u}, \mathbf{v})]^2 &= \parallel \mathbf{u} -\mathbf{v} \parallel ^2 \\
&= (\mathbf{u} - \mathbf{v}) \cdot (\mathbf{u} - \mathbf{v}) \\
&= \parallel \mathbf{u} \parallel^2 + \parallel \mathbf{v} \parallel^2 - 2\mathbf{u} \cdot \mathbf{v}
\end{align}

Therefore,

\begin{align}
	[\textrm{dist}(\mathbf{u}, \mathbf{-v})]^2 = [\textrm{dist}(\mathbf{u}, \mathbf{v})]^2 &\iff - 2\mathbf{u} \cdot \mathbf{v} = 2\mathbf{u} \cdot \mathbf{v} \\
	&\implies \quad \mathbf{u} \cdot \mathbf{v} = 0 \\
	&\implies \parallel \mathbf{u} + \mathbf{v} \parallel ^2 = \parallel \mathbf{u} \parallel^2 + \parallel \mathbf{v} \parallel^2
\end{align}

\subsection{Orthogonal Complements in $\mathds{R}^n$}
If a vector $\mathbf{z}$ is orthogonal to every vector in a subspace $W$ of $\mathds{R}^n$, then $\mathbf{z}$ is orthogonal to $W$. The set of all such vectors $\mathbf{z}$ is the \textbf{orthogonal complement} of $W$ and denoted by $W^\bot$. Let's go through the relation between nullspace, orthogonal complements, and the columns and rows of a $m \times n$ matrix $A$.

We have the folllowing theorem:
\begin{equation}
(\textrm{Row}(A))^\bot = \textrm{Nul}(A) \quad \textrm{and} \quad (\textrm{Col}(A))^\bot = \textrm{Nul}(A)^\bot
\end{equation}

If $Ax = 0$ for some $n \times 1$ vector $x$, then $x \in \textrm{Nul}(A)$, which means $x$ is orthogonal to the rows of $A$. The rows of $A$ span its row space, so $x$ is orthogonal to Row($A$). This is a consequence of how matrix multiplication works, as shown below:
\begin{align}
	A &= \begin{pmatrix} a & b & c \\ d & e & f \end{pmatrix}, \quad u = \begin{pmatrix} x \\ y \\ z \end{pmatrix}. \\
	Au &= \begin{pmatrix}	a \\ d	\end{pmatrix} x + \begin{pmatrix}	b \\ e	\end{pmatrix} y + \begin{pmatrix}	c \\ f	\end{pmatrix} z, \quad \textrm{ linear combination of the columns}\\
		&= \begin{pmatrix} ax + by + cz \\ dx + ey + fz \end{pmatrix}, \quad \textrm{ dot product of the rows}\\
		&= \begin{pmatrix} 0 \\ 0 \end{pmatrix}
\end{align}

The elements of the product $Au$ are dot products between $u$ and the row vectors of $A$; the product $Au$ can also be thought of as a linear combination of the columns of $A$. Since $Au = 0$, $u$ is in the nullspace of A, and from the above, we see that $u$ is orthogonal to the rows of $A$.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.7\linewidth]{figs/Lay_fundamental_subspaces}
	\caption{Fundamental subspaces of $A$. From Lay.}
	\label{fig:layfundamentalsubspaces}
\end{figure}

\subsection{Primal and Dual Problems}
In the 'primal' problem, we find the solution $\hat{u}$ by projecting $b$ onto the column space of $A$, i.e. $A\hat{u}$ is the closest point to $b$ in the column space of $A$. In general, the product $Au$ yields a linear combination of the columns of $A$. The column space of $A$ contains all such products.

\begin{equation}
Au = y \quad \implies \quad y \in \textrm{Col}(A), \quad \forall u
\end{equation}
Where, Col($A$) refers to the column space of $A$.

The 'dual' problem solves $A^T e = 0$ for $e$, which is the projection of $b$ onto the nullspace of $A^T$. Recall from above:
$$ \textrm{Nul}(A^T) = (\textrm{Row}(A^T))^\bot = (\textrm{Col}(A))^\bot $$
Thus, $e$ is contained in (Col($A$))$^\bot$, which contains \textit{all vectors perpendicular to the columns of $A$}. The dimension of (Col($A$))$^\bot$ is given by the Rank-Nullity Theorem:

\begin{align}
	&\textrm{Rank}(A) + \textrm{Nullity}(A) = n, \quad \textrm{ where } n \textrm{ is the number of columns in } A. \\
	&\textrm{Rank}(A^T) + \textrm{Nullity}(A^T) = m \\
	&\textrm{Rank}(A) = \textrm{Rank}(A^T) \quad \implies \quad \textrm{Nullity}(A^T) = m - n \\
	&\textrm{Nullity}(A^T) = \textrm{dim(Row(}A^T))^\bot = \textrm{dim(Col(}A)^\bot) = m - n
\end{align}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.7\linewidth]{figs/Strang_projection_lsq}
	\caption{Ordinary least squares. From Strang.}
	\label{fig:strangprojectionlsq}
\end{figure}

$e$ and $\hat{u}$ solve the two linear equations shown by the figure above:
\begin{align}
e + A \hat{u} &= b \rightarrow m \textrm{ equations} \\
A^T e &= 0 \rightarrow n \textrm{ equations}
\end{align}

These are called the "Primal-Dual", "Saddle Point", and "Kuhn-Tucker (KKT)" equations.

\subsubsection{"Saddle Point" Equations}
The Saddle Point, or KKT, matrix is a block matrix that represents the Primal-Dual equations:
$$ S = \begin{bmatrix} I & A \\ A^T & 0 \end{bmatrix} \quad \implies \quad S \begin{bmatrix} e \\ \hat{u} \end{bmatrix} = \begin{bmatrix} Ie + A\hat{u} \\ A^T e + 0 \end{bmatrix} $$
Where, $I$ is an $m \times m$ identity matrix. Thus, $I$ constitutes the first $m$ pivot columns of $S$. Thus, to reduce $S$, we need to make the lower left hand corner of $S$ a $n \times n$ matrix of zeroes. To do this, let's review elimination on block matrices:

\textbf{Elimination of block matrices}:

\begin{align}
\begin{bmatrix} I & 0 \\ -CA^{-1} & I \end{bmatrix} \begin{bmatrix} A & B \\ C & D \end{bmatrix} &= \begin{bmatrix} A + 0 & B + 0 \\ -CA^{-1}A + C & -CA^{-1}B + D \end{bmatrix} \\
&= \begin{bmatrix} A & B \\ 0 & D - CA^{-1}B \end{bmatrix}
\end{align}

The first matrix, $\begin{bmatrix} I & 0 \\ -CA^{-1} & I \end{bmatrix}$, is also known as the Gaussian elimination matrix, often represented using the symbol $E$. If $A$ was not an identity matrix, then we can also reduce $A$ into a matrix of pivots:

\begin{align}
\begin{bmatrix} A^{-1} & 0 \\ -CA^{-1} & I \end{bmatrix} \begin{bmatrix} A & B \\ C & D \end{bmatrix} &= \begin{bmatrix} AA^{-1} + 0 & BA^{-1} + 0 \\ -CA^{-1}A + C & -CA^{-1}B + D \end{bmatrix} \\
&= \begin{bmatrix} I & BA^{-1} \\ 0 & D - CA^{-1}B \end{bmatrix}
\end{align}

We now reduce our Primal-Dual matrix $S$ with $E = \begin{bmatrix} I & 0 \\ -A^T & I \end{bmatrix}$:

\begin{align}
ES &= \begin{bmatrix} I & 0 \\ -A^T & I \end{bmatrix} \begin{bmatrix} I & A \\ A^T & 0 \end{bmatrix} \\
	&= \begin{bmatrix} I + 0 & A + 0 \\ -A^T + A^T & -AA^T + 0 \end{bmatrix} \\
	&= \begin{bmatrix} I & A \\ 0 & -AA^T \end{bmatrix}
\end{align}

By convention, the lower right-hand block remaining after elimination is known as the \textbf{Schur complement}. More importantly, the first $m$ pivots are positive (=1), but the final $n$ pivots, determined by $-AA^T$, are negative. When a matrix has pivots of both signs, it is \textbf{indefinite}. For such matrices, \textit{there is no maximum or minimum, positive or negative definite}. Instead, there is a \textbf{saddle point, $(\hat{u}, e)$}.

\subsubsection{"Primal-Dual" Equations}
The \textit{primal} problem is to minimize $\frac{1}{2} \parallel Au - b \parallel^2$. The \textit{dual} problem is to minimize $\frac{1}{2} \parallel e - b \parallel^2$ with the constraint $A^T e = 0$. In the dual problem, $u$ acts a Lagrange multiplier to enforce the constraint $A^T e = 0$. Solutions to the primal and dual problems are found simultaneously, and add to $b$.

Let's do an example!

\subsubsection{An Example}
\begin{equation}
A = \begin{bmatrix}2 \\ 1 \end{bmatrix}, \quad b = \begin{bmatrix}3 \\ 4\end{bmatrix}, \quad A^T A = 5, \quad A^T b = 10
\end{equation}
\begin{align}
	&\textrm{Projection: } A\hat{u} = \begin{bmatrix}4 \\ 2\end{bmatrix} \\
	&\textrm{Error: } e = b - A\hat{u} = \begin{bmatrix}3 \\ 4 \end{bmatrix} - \begin{bmatrix}4 \\ 2\end{bmatrix} = \begin{bmatrix}-1 \\ 2\end{bmatrix}
\end{align}

Let's verify that $A\hat{u}$ and $e$ are perpendicular. First, $A\hat{u}$ and $e$ sum to $b$:

\begin{equation}
A\hat{u} + e = A\hat{u} + (b - A\hat{u}) = b
\end{equation}

\begin{align}
\parallel b \parallel^2 &= \parallel A\hat{u} \parallel^2 + \parallel e \parallel^2 \\
&= 20 + 5 = 25 \\
A \hat{u} \cdot e &= \parallel A \hat{u} \parallel \parallel e \parallel \cos \theta \\
0 &= \sqrt{20} \sqrt{5} \cos \theta \\
\therefore \cos \theta &= 0 \quad \implies \quad \theta = \frac{\pi}{2}
\end{align}

The normal equation is: $A^T A \hat{u} = A^T b$, so $\hat{u} = 2$. This solves the primal problem. The dual minimizes $\frac{1}{2}\parallel e-b \parallel^2$ under the constraint $A^T e = 0$. The solution to the dual problem gives $e$ and $\hat{u}$:

The \textbf{Lagrange} function is:

\begin{align}
L &= \frac{1}{2}\parallel e-b \parallel^2 + u(A^T e) \\
L(e_1, e_2, u) &= \frac{1}{2}(e_1 - b_1 )^2 + \frac{1}{2}(e_2 - b_2 )^2 + u(A^{T}_{1} e_1 + A^{T}_{2} e_2) \\
\frac{\partial L}{\partial (e_1, e_2, u)} &= \begin{bmatrix}
	e_1 - b_1 + 2\hat{u} = 0 \\
	e_2 - b_2 + \hat{u} = 0 \\
	2e_1 + e_2 = 0
\end{bmatrix} = \begin{bmatrix}
	e_1 & e_2 & \hat{u} \\
	1 & 0 & 2 \\
	0 & 1 & 1 \\
	2 & 1 & 0
\end{bmatrix} \begin{bmatrix} e_1 \\ e_2 \\ \hat{u} \end{bmatrix} = \begin{bmatrix} b_1 \\ b_2 \\ 0 \end{bmatrix}
\end{align}

The matrix $
\begin{bmatrix}
1 & 0 & 2 \\
0 & 1 & 1 \\
2 & 1 & 0
\end{bmatrix} $
is the saddle point matrix $S$, whose columns correspond to $e_1$, $e_2$, and $u$, respectively.

\section{Weighted Least Squares} \label{Weighted Least Squares}
The weighted least squares equation is:
\begin{equation}
\min \parallel WAu - Wb \parallel^2
\end{equation}

with normal equations:
\begin{align}
(WA)^T (WA) \hat{u}_W &= (WA)^T (Wb) \\
A^T W^T W A \hat{u}_W &= A^T W^T W b \\
A^T C A \hat{u}_W &= A^T C b, \quad C = W^T W \\
\implies 0 &= A^T C b - A^T C A \hat{u}_W \\
	&= A^T C (b - A \hat{u}_W ) = A^T C e, \quad e = (b - A \hat{u}_W )
\end{align}

Where, $\hat{u}_W$ is the solution, and changes with the weight matrix $W$. Let's consider the geometric effects of introducing $W$.
\begin{itemize}
	\item First, $A^T e = 0$ becomes $A^T C e = 0$. So, $e$ is perpendicular to $A^T C$ now, not $A^T$ by itself. So, $e$ is in the nullspace of $A^T C$, not that of $A^T$.
	\item With $W$, $e$ is now in the orthogonal complement to the rows of $A^T C$, or equivalently, the columns of $C^T A$. Thus, $e$ is not orthogonal to the column space of $A$. We say that $e$ is "$C$-orthogonal" to the columns of $A$.
	\item Consequently, solving weighted least squares involves finding a projection of $b$ that is orthogonal to Col($C^T A)$, but not Col($A$).
	\item We still split $b$ into two components: $A\hat{u}_W \in \textrm{Col}(A)$, and $e \in \textrm{Nul}(A^T C)$.
\end{itemize}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.7\linewidth]{figs/Strang_projection_wlsq}
	\caption{Weighted least squares.}
	\label{fig:strangprojectionwlsq}
\end{figure}

So, we have the following primal-dual equations to solve:
\begin{align}
e + A \hat{u}_W &= b \\
A^T C e &= 0
\end{align}

The saddle point matrix is: \begin{equation}
\begin{bmatrix}
	I & A \\
	A^T C & 0
\end{bmatrix},
\end{equation} where the columns represent coefficients of $e$ and $\hat{u}_W$, respectively. Ideally, we'd like that the equations be symmetric. However, we have a factor of $C$ in the bottom-left that is not in the top-right, so our current matrix is not symmetric. To make it symmtric, introduce $w = Ce$ and $e = C^{-1}w$, and let $\hat{u}_W = u$:

\begin{align}
C^{-1}w + Au &= b \label{wlsq1} \\
A^T w &= 0 \label{wlsq2}
\end{align}

This gives rise to a symmetric matrix: \begin{equation}
S = \begin{bmatrix}
	C^{-1} & A \\ A^T & 0
\end{bmatrix},
\end{equation} where columns represent coefficients for $w$ and $u$, respectively. Since $C = W^T W$, which is the vector analogue to squaring, the elements of $C$, and thus $C^{-1}$, are positive. $W$ is a $m \times n$ matrix, so $C^{-1}$ contributes $m$ \textit{positive} pivot columns. The remaining pivots are found by elimination on $S$:

\begin{align}
E &= \begin{bmatrix} I & 0 \\ -A^T C & I  \end{bmatrix} \\
ES &= \begin{bmatrix} I & 0 \\ -A^T C & I  \end{bmatrix} \begin{bmatrix} C^{-1} & A \\ A^T & 0 \end{bmatrix} \\
	&= \begin{bmatrix} C^{-1} + 0 & A + 0 \\ -A^T C C^{-1} + A & -A^T C A + 0 \end{bmatrix} \\
	&= \begin{bmatrix} C^{-1} & A \\ -A^T + A & -A^T C A \end{bmatrix} = \begin{bmatrix} C^{-1} & A \\ 0 & -A^T C A \end{bmatrix}
\end{align}

The Schur complement is $-A^T C A$, which is negative like before. So, we have $n$ negative pivots from this block and $m$ positive pivots from $C^{-1}$, making $S$ indefinite. The equations to solve are:
\begin{align}
\begin{bmatrix} C^{-1} & A \\ A^T & 0 \end{bmatrix} \begin{bmatrix} w \\ u \end{bmatrix} &= \begin{bmatrix} b \\ 0 \end{bmatrix} \\
\begin{bmatrix} C^{-1} & A \\ 0 & -A^T C A \end{bmatrix} \begin{bmatrix} w \\ u \end{bmatrix} &= \begin{bmatrix} b \\ -A^T C b \end{bmatrix} \quad \textrm{after elimination.}
\end{align}

Due to the constraint that $A^T w = 0$, $Au$ and $w$ are perpendicular:
\begin{equation}
(Au)^T w = u^T (A^T w) = 0
\end{equation}

\subsubsection{Duality and Weak Duality}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.9\linewidth]{figs/Strang_wsq_weakduality}
	\caption{Geometric interpretation of weak duality. From Strang.}
	\label{fig:strangwsqweakduality}
\end{figure}

Imagine we have a rectangle, where the perpendicular sides are $Au$ and $w$. The remaining corners of this rectangle are $0$ and $Au + w$. Lines from $b$ to the four corners of this rectangle are vectors $b - X + a$ where $X$ is a corner of the rectangle and $a$ is some offset (since the lines don't all start from 0).

We can ignore the offset by comparing the vectors' magnitudes. In particular, the diagonal distance starting from either the lower-left or lower-right corner of the rectangle (right side of Fig. \ref{fig:strangwsqweakduality}) is the same, but involve different vectors. The magnitude of each diagonal is the sum of two vectors that meet at $b$. One diagonal connects points $(0, b, Au + w)$, and has magnitude $\parallel b \parallel^2 + \parallel b - (Au + w) \parallel^2$. The other diagonal connects points $(Au, b, w)$, and has magnitude $\parallel b - Au \parallel^2 + \parallel b - w \parallel^2$. Equating these yields the following relationship:

\begin{equation}
\parallel b - Au \parallel^2 + \parallel b-w \parallel^2 = \parallel b \parallel^2 + \parallel b - Au - w \parallel^2
\end{equation}

At the minimum, we have that $\parallel b \parallel^2 = \parallel b - Au \parallel^2 + \parallel b-w \parallel^2$. This is summarized by the image on the right of Fig. \ref{fig:strangwsqweakduality}, which clearly shows how $Au$ and $w$ are projections of $b$ under this condition:
\begin{align}
\parallel b - Au - w \parallel^2 &= 0 \\
b - Au - w &= 0 \\
Au - w &= b
\end{align}

The solution $u = \hat{u}$ and $w = e$ gives \textbf{duality}: $\parallel b - A\hat{u} \parallel^2 + \parallel b-e \parallel^2 = \parallel b \parallel^2$. \textbf{Weak duality} is when $\parallel b - A\hat{u} \parallel^2 + \parallel b-e \parallel^2 > \parallel b \parallel^2$. In this scenario, we have a "duality gap" $\parallel b - Au - w \parallel^2$. Duality is achieved when the gap is zero. When this is true, $b = Au - w$, and $b$ becomes a corner of the rectangle, as shown in the right side of Fig. \ref{fig:strangwsqweakduality}. Finally, by Pythagoras' rule:
\begin{equation}
\parallel b-w \parallel^2 + \parallel b-Au \parallel^2 = \parallel b \parallel^2 = \parallel Au + w \parallel^2,
\end{equation}
which is exactly what happens when we have perfect duality!

To summarize this section, weighted least squares involves solving two problems:
\begin{enumerate}
	\item Project $b$ onto the column space of $A\hat{u}$:
	\begin{align*}
	\centering
	\min \parallel b-Au \parallel^2
	\end{align*}
	\item Project $b$ across to $e$: \begin{align*}
		\min &\quad \parallel b-w \parallel^2 \\
		\textrm{s.t.} &\quad A^T w = 0
	\end{align*}
\end{enumerate}

The solutions to these problems converge at optimality, as we showed above. At optimality, $\hat{w} = C(b - A\hat{u})$. $C$ is the bridge between the two dual problems and identiies the solutions $\hat{u}$ and $\hat{w}$. Alternatively, we can see the problem as a maximization problem, simply by attaching a negative sign:
\begin{align}
	\parallel b - e \parallel^2 + \parallel b - A \hat{u} \parallel^2 &= \parallel b \parallel^2 \\
	\parallel b - e \parallel^2 - \parallel b \parallel^2 &= -\parallel b - A \hat{u} \parallel^2 \\
	\therefore \quad \min (\parallel b-e \parallel^2 - \parallel b \parallel^2) \quad &\equiv \quad \max (-\parallel b - A \hat{u} \parallel^2)
\end{align}

\subsection{Minimizing with Constraints using Lagrange's Method}
Imagine we have a line of two springs and one mass and that we wish to minimize the energy in the springs. The constraint is $A^T w = f$, where we have forces $w_1$ and $w_2$, and a mass that exerts external force $f$. The springs are vertically arranged with the mass in between, which causes spring 1 (top) to stretch, and spring 2 (bottom) to compress (Fig. \ref{fig:strangconstrainedsprings}).

\begin{figure}[H]
	\centering
	\includegraphics[width=0.8\linewidth]{figs/Strang_constrained_springs}
	\caption{System with two linear springs and one mass.}
	\label{fig:strangconstrainedsprings}
\end{figure}

To solve the force balance equation $w_1 + w_2 = f$, we won't use calculus (setting derivatives to zero), because while we have an energy function $E(w_1 , w_2)$, it is unclear which derivatives we should adjust to achieve force balance. We could make the substitution: $w_2 = w_1 - f$, so that we only minimize over one variable, $w_1$. Here, we will use Lagrange multipliers to \textit{build constraints into the function} by adding an unknown $u$ that enforces our constraint.

With $n$ constraints on $m$ unknowns, this method has $m + n$ unknowns. We will add a Lagrange multiplier for each constraint. The Lagrange function is:
\begin{equation}
L(w_1, w_2, u) = E_1 (w_1) + E_2 (w_2) - u(w_1 - w_2 - f) = E_1 + E_2 - u (w_1 - w_2 - f)
\end{equation}

Where, $u$ takes a negative sign by convention. $w_2$ takes a negative sign as well, to distinguish the compression of spring 2 from the stretching of spring 1. Note that constraints, the term multiplied by $u$, must equal zero. The Lagrange multiplier $u$ has a special meaning: here, the displacement of the mass. In economics, the selling price for maximal profit. In all problems, $u$ measures the answer's sensitivity (here, the minimum energy $E_{\textrm{min}}$) to changes in constraints.

Hooke's law tells us that the force in a linear spring is proportional to elongation $e$, so $w = ce$. Work = force $\times$ displacement, and the integral of work is energy. Thus, Work = $ce$, and energy $E$ = $\int$ Work = $\frac{1}{2} ce^2$. Substituting $e = \frac{w}{c}$, we get $E = \frac{w^2}{2c}$. The resulting minimization problem is then:
\begin{align}
\min &\quad E(w) = \frac{w_{1}^2}{2c} + \frac{w_{2}^2}{2c} \\
\textrm{s.t. } &\quad w_1 - w_2 = f \nonumber
\end{align}

\subsubsection{Geometry of the method}
\begin{figure}[H]
	\centering
	\includegraphics[width=0.9\linewidth]{figs/Strang_constrained_geo}
	\caption{Geometric depiction of $E(w_1, w_2)$ and constraint $w_1 - w_2 = f$.}
	\label{fig:strangconstrainedgeo}
\end{figure}

Notice that $E(w)$ is the equation of an ellipse. We can think of the constraint geometrically in the plane with axes $w_1$ and $w_2$: $E(w)$ is some ellipse in this plane, and $w_1 - w_2 = f$ is a line with positive slope and negative y-intercept (Fig. \ref{fig:strangconstrainedgeo}). The solution is achieved when the line meets the ellipse at a right-angle. At this point $(w_1 , w_2)$, there are two vectors of interest that describe the rate of change of the ellipse and line with respect to $w_1$ and $w_2$. One is perpendicular to the line: \begin{equation*}
\begin{bmatrix}
\partial l / \partial w_1 \\ \partial l / \partial w_1
\end{bmatrix} = \begin{bmatrix}
1 \\ -1
\end{bmatrix}, \quad l = w_1 - w_2 - f = 0
\end{equation*}

The second is perpendicular to the ellipse, and arises from the gradient of $E(w)$:
\begin{align*}
L(w_1, w_2, u) &= E_1 + E_2 - u (w_1 - w_2 - f) \\
\frac{\partial L}{\partial w_1} &= \frac{\partial E}{\partial w_1} - u = 0 \\
\frac{\partial L}{\partial w_2} &= \frac{\partial E}{\partial w_2} + u = 0 \\
\begin{bmatrix}
\partial E / \partial w_1 \\ \partial E / \partial w_2
\end{bmatrix} &= \begin{bmatrix}
u \\ -u
\end{bmatrix}
\end{align*}

The above gives us $w_1$ and $w_2$ as follows:
\begin{align}
\frac{\partial E}{\partial w_1} &= \frac{\partial}{\partial w_1} \frac{w^{2}_{1}}{2c_{1}} = \frac{w_{1}}{c_{1}} \\
\therefore \begin{bmatrix}
\partial E / \partial w_1 \\ \partial E / \partial w_2
\end{bmatrix} &= \begin{bmatrix}
w_1 / c_1 \\ w_2 / c_2
\end{bmatrix} = \begin{bmatrix}
u \\ -u
\end{bmatrix} \\
w_1 = c_1 u, &\quad w_2 = -c_2 u \\
w_1 - w_2 &= f = c_1 u + c_2 u = (c_1 + c_2)u
\end{align}

$c_1 + c_2$ is the stiffness matrix $A^T C A$, which we discussed in Section \ref{Weighted Least Squares}. Since the problem is small, $K = A^T C A$ is just $1 \times 1$:
\begin{equation*}
A^T = \begin{bmatrix}
1 \\ -1
\end{bmatrix}, \quad K = A^T C A = \begin{bmatrix}
1 \\ -1
\end{bmatrix}\begin{bmatrix}
c_1 & \\ & c_2
\end{bmatrix} \begin{bmatrix}
1 \\ -1
\end{bmatrix} = \begin{bmatrix}
c_1 + c_2
\end{bmatrix}
\end{equation*}

We now find the forces $w_1$ and $w_2$:
\begin{align*}
Ku &= f \quad \implies \quad u = \frac{f}{c_{1} + c_{2}} \\
w_1 &= c_1 u = \frac{c_1 f}{c_1 + c_2}, \quad w_2 = -c_2 u = -\frac{c_2 f}{c_1 + c_2} \\
E_{\textrm{min}} &= \frac{w_{1}^2}{2c} + \frac{w_{2}^2}{2c} = \frac{c_1 f^2}{2 (c_1 + c_2)^2} + \frac{c_2 f^2}{2 (c_1 + c_2)^2} \\
 &= \frac{f^2}{2 (c_1 + c_2)}
\end{align*}

Above, we mentioned that $u$ measures the sensitivity of $E_{\textrm{min}}$ to changes in the constraint, $f$. We can show this by computing the derivative $\frac{dE_{\textrm{min}}}{df}$:
\begin{align}
\frac{dE_{\textrm{min}}}{df} &= \frac{d}{df} \left(\frac{f^2}{2 (c_1 + c_2)}\right) \nonumber \\
&= \frac{f}{c_1 + c_2} = u
\end{align}

\subsection{The Fundamental Problem}
This is what Strang describes as the \textit{fundamental problem of scientific computing}: the full linear case with $m$ springs. Each spring experiences a force $w_i$, which we can represent as $w = (w_1, \dotsm, w_m)$. We have $n$ constraints for force balance, $A^{T}w = (f_1, \dotsm, f_n)$, enforced by $n$ Lagrange multipliers $u = (u_1, \dotsm, u_n)$.

Let $C$ be a matrix with masses $c_1, \dotsm, c_m$ on the diagonal and zeroes everywhere else. Then, the total energy function for $m$ springs is given by: $E(w) = \frac{1}{2}w^T C^{-1} w$. Our Lagrange function is thus:
\begin{equation}
L(w, u) = \frac{1}{2}w^T C^{-1} w - u^{T}(A^{T}w - f)
\end{equation}

The minimizer $w$ is found by setting the $m+n$ partial derivatives of $L$ to zero:
\begin{align}
\frac{\partial L}{\partial w} &= C^{-1}w - Au = 0 \label{fundamental_pLpw}\\
\frac{\partial L}{\partial u} &= -A^{T}w + f = 0 \label{fundamental_pLpu}
\end{align}

From Eq. \ref{fundamental_pLpw}, we have that $w = Ce = CAu$, which implies that $e = Au$. Eq. \ref{fundamental_pLpu} indicates that $f = A^{T}w$. Together, we have that:
\begin{equation}
A^T C A u = A^T C e = A^T w = f
\end{equation}

Least squares problems have $e = b - Au$, which appears in $E$ as:
\begin{equation*}
E(b, w) = \frac{1}{2}w^T C^{-1} w - b^T w
\end{equation*}

The derivatives of $L$ (with $u$ positive) are:
\begin{align}
\frac{\partial L}{\partial w} &= C^{-1}w + Au - b = 0 \\
\frac{\partial L}{\partial u} &= -A^{T}w - f = 0 \\
\begin{bmatrix}
C^{-1} & A \\ A^{T} & 0
\end{bmatrix} &\begin{bmatrix}
w \\ u
\end{bmatrix} = \begin{bmatrix}
b \\ f
\end{bmatrix}
\end{align}

The vector $(w, u)^{T}$ has size $m + n$, since $w$ has $m$ components and $u$ has $n$. If $C$ is positive definite, and $A$ has full column rank (all columns are independent, Rank($A$) = $n$), then $A^{T}CA$ and $S$ are invertible. We can factor $S$ into three parts as follows:
\begin{equation*}
S = \begin{bmatrix}
C^{-1} & A \\ A^{T} & 0
\end{bmatrix} = \begin{bmatrix}
I & 0 \\ A^{T}C & I
\end{bmatrix} \begin{bmatrix}
C^{-1} & 0 \\ 0 & -A^{T}CA
\end{bmatrix}\begin{bmatrix}
I & CA \\ 0 & I
\end{bmatrix}
\end{equation*}

$S$ is invertible when both block columns have full rank. Otherwise, it can be singular.

\section{Regularized Least Squares}

Regularized least squares is a special case of weighted least squares:
\begin{align*}
\textrm{\textbf{Ordinary LSQ}} \quad & \min \parallel Au - b \parallel^2 \\
\textrm{Solve: } & A^TA\hat{u} = A^T b \\
\textrm{\textbf{Weighted LSQ}} \quad & \min (b-Au)^{T} C (b-Au) \\
\textrm{Solve: } & A^T C A\hat{u} = A^T C b \\
\textrm{\textbf{Regularized LSQ}} \quad & \min \parallel Au - b \parallel^2 + \alpha \parallel Bu-d \parallel^2 \\
\textrm{Solve: } & (A^T A + \alpha B^{T}B)\hat{u} = A^{T}b + \alpha B^{T}d
\end{align*}

We can rewrite the regularized LSQ equations to resemble those for weighted LSQ by replacing $A$ in the latter by $[A \ B]^T$:
\begin{equation}
\begin{bmatrix}
A^T & B^T
\end{bmatrix} \begin{bmatrix}
I & 0 \\ 0 & \alpha I
\end{bmatrix} \hat{u} = \begin{bmatrix}
A^T & B^T
\end{bmatrix}  \begin{bmatrix}
I & 0 \\ 0 & \alpha I
\end{bmatrix} \begin{bmatrix}
b \\ d
\end{bmatrix}
\end{equation}

Where, the weighting matrix $C =  \begin{bmatrix}
I & 0 \\ 0 & \alpha I
\end{bmatrix}$. There are two important applications that lead to this sum of two squares $(A^T A, B^{T}B)$:
\begin{enumerate}
	\item If $A^{T}A$ is ill-conditioned, e.g. very large ratio of largest to smallest eigenvalues, $m < n$, $A^{T}A$ is singular. \\
	Regularization acts to 'reduce the noise.' Normally, as $\alpha$ is increased, $\parallel \hat{u} \parallel$ decreases and $\parallel A\hat{u} - b \parallel$ increases. The \textbf{Discrepancy Principle} selects $\alpha$ so that $\parallel \hat{u} \parallel$ decreases and $\parallel A\hat{u} - b \parallel \approx $ expected noise (uncertainty in $b$).
	\item As $\alpha \rightarrow \infty$, $Bu \rightarrow d$. The limiting $\hat{u}_{\infty}$ solves
	\begin{align}
	\min &\quad \parallel Au-b \parallel^2 \\
	\textrm{s.t.} &\quad Bu=d \nonumber
	\end{align}
\end{enumerate}

\subsection{Example: Large Penalty Enforced $Bu=d$}
\begin{align*}
\textrm{Problem:} \quad \min &\quad \parallel Au \parallel^2 = u_{1}^{2}+u_{2}^{2} \\
\textrm{s.t.} &\quad Bu = u_1 - u_2 = 8
\end{align*}
Let $A = \begin{bmatrix}
1 & 0 \\ 0 & 1
\end{bmatrix}, \quad b = \begin{bmatrix}
0 \\ 0
\end{bmatrix}, \quad B = \begin{bmatrix}
1 & -1
\end{bmatrix}, \quad d = \begin{bmatrix}
8
\end{bmatrix}$.

\begin{align*}
A^T A = I, \quad B^T B = \begin{bmatrix}
1 & -1 \\ -1 & 1
\end{bmatrix}, \quad &\begin{bmatrix}
1+\alpha & -\alpha \\ -\alpha & 1+\alpha
\end{bmatrix}\begin{bmatrix}
u_1 \\ u_2
\end{bmatrix} = \begin{bmatrix}
8\alpha \\ -8\alpha
\end{bmatrix} = \alpha B^T d \\
u_1 + u_2 = 0, \quad &(1 + 2\alpha)u_1 = 8\alpha \\
&u_1 = \frac{8\alpha}{1+2\alpha} = \frac{4}{1 + \frac{\alpha}{2}} = 4 - \frac{4}{2\alpha} + \dotsm \rightarrow u_1 = 4
\end{align*}

The order is of order $1/\alpha$, so we need large $\alpha$.

\subsection{The Pseudoinverse}
Suppose $A$ is $m \times n$ and $b$ has $m$ components. The normal equations $A^{T}A\hat{u} = A^{T}b$ only give $\hat{u}$ when $A^{T}A$ is invertible. Otherwise, we can use the pseudoinverse to find the best solution $u^{+}$. This works even when $A$ is not full rank and $A^{T}A$ is singular

\subsubsection{What is $u^{+}$?}
$u^{+} = A^{+}b$ provides the \textbf{shortest} vector that solves $A^{T}Au^{+} = A^{T}b$.

Other solutions, longer than $u^{+}$, have components in the nullspace of $A$. $u^{+}$ is \textbf{the particular solution with no nullspace component}. The \textbf{pseudoinverse} of $A$ is a matrix $A^{+}$, which produces $u^{+} = A^{+}b$. If $A$ is square and invertible, then $u^{+} = u = A^{-1}b$, and $A^{+} = A^{-1}$.

If $A$ has independent columns, but $m < n$, then $\hat{u} = (A^{T}A)^{-1}A^{T}b$ is the only solution, where $A^{+} = (A^{T}A)^{-1}A^{T}$. If $A$ has dependent columns and thus a nonzero nullspace: any $Au$ is a linear combination of the columns of $A$, but if the columns are dependent, then there exists a nonzero vector $x$ such that $Ax = 0$, and so $x \in \textrm{Nul}(A)$ and Nul($A$) $\neq \{0\}$.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.8\linewidth]{figs/Strang_pseudoinverse}
	\caption{$Au$ takes vectors $u = u_{\textrm{row}} + u_{\textrm{null}}$ to the column space of $A$. Since $u_{\textrm{row}}$ and $u_{\textrm{null}}$ are orthogonal, a non-zero $u_{\textrm{null}}$ increases $\norm{u}^{2}$. Thus, $u^{+} = u_{\textrm{row}}$}.
	\label{fig:strangpseudoinverse}
\end{figure}

$u^{+}$ solves $Au^{+} = p$, where $p$ is the projection of $b$ onto the column space of $A$. The error $\norm{e} = \norm{b-p} = \norm{b-Au^{+}}$ is minimized by $u^{+}$. In summary, $u^{+}$ is in the row space (to be as short as possible), and $Au^{+} = p$ (to be as close to $b$ as possible); $u^{+}$ minimizes $e$ and solves $A^T A u^{+} = A^T b$.

\subsubsection{Singular Value Decomposition (SVD)}
One method to compute $u^{+}$ is by SVD. Let's review diagonalization and SVD.

Recall the \textbf{Diagonalization Theorem}:
\begin{itemize}
	\item An $n \times n$ matrix $A$ is diagonalization if and only if $A$ has $n$ linearly independent eigenvectors.
	\item $A = PDP^{-1}$ with $D$ being a diagonal matrix, if and only if the columns of $P$ are $n$ linearly independent eigenvectors of $A$. Then, the entries of $D$ are the corresponding eigenvalues of $A$ for the eigenvectors in $P$.
\end{itemize}

\paragraph{Diagonalization of Symmetric Matrices \\}
Furthermore, if $A$ is symmetric, then:
\begin{itemize}
	\item $A^{T} = A$
	\item All eigenvectors from different eigenspaces (i.e. corresponding to different eigenvalues) are \textbf{orthogonal} to each other.
	\item $A = PDP^{-1}$, but $P$ is square and has orthonormal columns $\implies$ $P$ is an \textit{orthogonal matrix}, and $P^{-1} = P^{T}$.
	\item $A$ is orthogonally diagonalizable if and only if $A$ is symmetric.
\end{itemize}

We can prove that, for symmetric $A$, two given eigenvectors $v_1$ and $v_2$ are orthogonal:
\begin{align*}
\lambda_1 v_1 \cdot v_2 &= (\lambda_1 v_1)^{T} v_2 = (Av_1)^{T}v_2 \quad \leftarrow \lambda_1 v_1 = A v_1 \\
&= (v_{1}^{T}A^{T})v_2 = v_{1}^{T}(Av_2) = v^{T}_{1}(Av_{2}) \quad \leftarrow A^T = A \\
&= v_{1}^{T} \lambda_{2}v_{2} = \lambda_{2}v_{1}^{T}v_{2} = \lambda_{2}v_{1} \cdot v_{2} \\
(\lambda_{1} - \lambda_{2})v_{1}\cdot v_{2} &= 0 \\
\lambda_{1} - \lambda_{2} &\neq 0 \quad \implies \quad v_{1} \cdot v_{2} = 0
\end{align*}

An $n \times n$ matrix $A$ is \textit{orthogonally diagonalizable} if there are an orthogonal matrix $P$ with $P^{-1} = P^{T}$, and a diagonal matrix $D$ such that
\begin{equation}
A = PDP^{T} = PDP^{-1}
\end{equation}

This requires $n$ linearly independent and orthonormal eigenvectors. This is possible when $A$ is symmetric:
\begin{equation*}
A^{T} = (PDP^{T})^{T} = P^{TT}D^{T}P^{T} = PDP^{T} = A
\end{equation*}

\paragraph{Spectral decomposition of a Symmetric Matrix \\}
\textbf{Spectral decomposition} is possible when $A$ is symmetric:
\begin{align*}
A = PDP^{T} &= \begin{bmatrix}
u_1 \dotsm u_n
\end{bmatrix} \textrm{diag}(\lambda_1 , \dotsm, \lambda_n) \begin{bmatrix}
u_{1}^{T} \\ \vdots \\ u_{n}^{T}
\end{bmatrix} \\
&= \begin{bmatrix}
\lambda_{1}u_{1} \dotsm \lambda_{n}u_{n}
\end{bmatrix}\begin{bmatrix}
u_{1}^{T} \\ \vdots \\ u_{n}^{T}
\end{bmatrix} \\
&= \lambda u_{1} u_{1}^{T} + \dotsm + \lambda_{n}u_{n}u_{n}^{T}
\end{align*}

Here, each matrix $u_{i}u_{i}^{T}$ is $1 \times 1$ and is a \textbf{projection matrix} in the sense that, $u_{i}u_{i}^{T}x$, for all $x \in \mathds{R}^{n}$, is the orthogonal projection of $x$ onto the subspace spanned by $u_{i}$.

\paragraph{SVD of an $m \times n$ matrix $A$ \\}

Let's finally talk about SVD. The key insight is that, even if we can't factorize an $m \times n$ matrix $A$ as $A = PDP^{-1}$, \textbf{we can always perform SVD to get $\mathbf{A = QDP^{-1}}$. }

Let $A$ be an $m \times n$ matrix. Then, $A^{T}A$ is symmetric, and can be orthogonally diagonalized as $A^{T}A = PDP^{T}$. Let $\{v_1, \dotsm v_{n} \}$ be an orthonormal basis for $\mathds{R}^{n}$ consisting of eigenvectors of $A^{T}A$. Let $\lambda_{1}, \dotsm, \lambda_{n}$ be the corresponding eigenvalues. Then,
\begin{align*}
\norm{Av_i}^{2} &= (Av_i)^{T}(Av_i) = v_{i}^{T}A^{T}Av_{i} \\
&= v_{i}^{T} (\lambda_{i}v_{i}) \quad \leftarrow Av_{i} = \lambda_{i}v_{i} \\
&= \lambda_{i} v_{i}^{T}v_{i} = \lambda_{i} \quad \leftarrow v_{i}^{T}v_{i} = 1 \\
\implies \quad \lambda_{i} \geq 0
\end{align*}

The \textbf{singular values} of $A$ are the square roots of the eigenvalues of $A^{T}A$: $\sigma_{i} = \sqrt{\lambda_{i}} = \sqrt{\norm{Av_i}^{2}} = \norm{Av_i}$. Therefore, we can also see that $\sigma_{i}$ is the length of the vector $Av_{i}$.

Since $v_i$ and $\lambda_{j}v_j$ are orthogonal while $i \neq j$,
\begin{equation*}
(Av_i)^{T}(Av_j) = v_{i}^{T}A^{T}Av_{j} = v_{i}^{T}(\lambda_j v_{j}) = 0
\end{equation*}

This shows that $\{Av_{1}, \dotsm, Av_{n} \}$ is an \textbf{orthogonal set}. The lengths of $Av_{i}$ is given by the corresponding singular values of $A$. Typically, we arrange the singular values in order of decreasing magnitude, so that:
\begin{align*}
&\lambda_{1} \geq \lambda_{2} \geq \dotsm \lambda_{n} \geq 0 \\
&\sigma_{1} \geq \sigma_{2} \geq \dotsm \sigma_{n} \geq 0
\end{align*}

If $A$ has $r$ non-zero singular values, then Rank($A$) = $r$, and $\{Av_{1}, Av_{2}, \dotsm, Av_{r} \}$ is an \textbf{orthogonal basis} for Col($A$). To show that the basis is $r-$dimensional, rather than $n-$dimensional, we will show that a vector $y$ in Col($A$) can be expressed as a linear combination of $\{Av_{1}, \dotsm, Av_{r} \}$:
\begin{align*}
x &= c^{T}v = c_{1}v_{1} + \dotsm + c_{n}v_{n} \\
y &= Ax = c_{1}Av_{1} + \dotsm c_{r}Av_{r} + c_{r+1}Av_{r+1} + \dotsm + c_{n}Av_{n} \\
&= c_{1}Av_{1} + \dotsm + c_{r}Av_{r} + 0 + \dotsm + 0
\end{align*}

Thus, $y$ is in Span$\{Av_{1}, \dotsm, Av_{r} \}$. Hence, Rank($A$) = dim(Col($A$)) = $r$.

\paragraph{SVD \\}
Let $A$ be an $m \times n$ matrix of rank $r$. Then, there is an $m \times n$ matrix $\Sigma$,
\begin{equation*}
\Sigma = \begin{bmatrix}
D & 0 \\ 0 & 0
\end{bmatrix}
\end{equation*}
for which the diagonal entries in $D$ are the first $r$ (i.e. the non-zero) singular values of $A$, where
\begin{equation*}
\sigma_{1} \geq \sigma_{2} \geq \dotsm \geq \sigma_{r} > 0
\end{equation*}
Finally, there is an $m \times m$ orthogonal matrix $U$ and an $n \times n$ matrix $V$ such that:
\begin{equation}
A = U \Sigma V^{T}
\end{equation}

$U$ and $V$ are not uniquely determined by $A$, and are referred to as the left- and right-singular vectors of $A$, respectively. We prove SVD below, starting with the assumption that $A$ is a $m \times n$ matrix with eigenvalues $\lambda_{i}$, corresponding eigenvectors $v_{i}$, and $r$ non-zero singular values $\sigma_{i}$. As shown above, $\{Av_{1}, \dotsm, Av_{r} \}$ will be an orthogonal basis for Col($A$).

Normalize each $Av_{i}$ to obtain an orthonormal basis $U$:
\begin{align*}
U &= \{u_{1}, \dotsm, u_{r} \}, \textrm{ where} \\
 u_{i} &= \frac{Av_{i}}{\norm{Av_{i}}} = \frac{Av_{i}}{\sigma_{i}} \\
 Av_{i} &= \sigma_{i}u_{i}, \quad 1 \leq i \leq r
\end{align*}

Extend $U$ to a basis of $\mathds{R}^{m}$:
\begin{equation*}
U = \begin{bmatrix}
u_{1} & u_{2} & \dotsm & u_{m}
\end{bmatrix}
\end{equation*}
Let $V = \begin{bmatrix}
	v_{1} & v_{2} & \dotsm & v_{n}
\end{bmatrix}$. Then, $U$ and $V$ are both orthogonal matrices. We also have that
\begin{align*}
AV &= \begin{bmatrix}
Av_{1} & \dotsm & Av_{r} & 0 & \dotsm & 0
\end{bmatrix} = \begin{bmatrix}
\sigma_{1}u_{1} & \dotsm & \sigma_{r}u_{r} & 0 & \dotsm & 0
\end{bmatrix}
\end{align*}

Let $D$ be a diagonal matrix with diagonal entries $\sigma_{1}, \dotsm, \sigma_{r}$. Then,
\begin{align*}
U\Sigma &= \begin{bmatrix}
u_{1} & u_{2} & \dotsm & u_{m}
\end{bmatrix} \begin{bmatrix}
D & \textrm{$n - r$ columns of $0$} \\
\textrm{$m-r$ rows of $0$} & 0 \\
\end{bmatrix} \\
&= \begin{bmatrix}
\sigma_{1}u_{1} & \sigma_{2}u_{2} & \dotsm & \sigma_{r}u_{r} & 0 & \dotsm & 0
\end{bmatrix} \\
&= AV \\
V^{-1} = V^{T} \quad \implies &\quad U\Sigma V^{T} = AVV^{T} = A
\end{align*}

\subsubsection{Computing $u^{+}$ with SVD}

\begin{equation}
\textrm{\textbf{SVD:}} \quad A = U\Sigma V^T = \begin{bmatrix}
U_{\textrm{col}} & U_{\textrm{null}}
\end{bmatrix} \begin{bmatrix}
\Sigma_{\textrm{pos}} & 0 \\ 0 & 0
\end{bmatrix} \begin{bmatrix}
V_{\textrm{row}} & V_{\textrm{null}}
\end{bmatrix}^{T}
\end{equation}

$U$ and $V$ are square matrices that have orthonormal columns: $U^{T}U = I$ and $V^{T}V = I$. $U_{\textrm{col}}$ and $V_{\textrm{row}}$ are bases for the column and row spaces of $A$, respectively, and have dimensions $r = \textrm{Rank}(A)$. The other 'null' columns are in the nullspaces of $A^T$ ($U_{\textrm{null}}$) and $A$ ($V_{\textrm{null}}$).

The pseudoinverse of $A$ is computed as:
\begin{equation}
A^{+} = V_{row}(\Sigma_{pos})^{-1}U_{col}^{T}
\end{equation}

Where, $\Sigma_{pos}$ is a diagonal matrix containing the \textit{positive} singular values of $A$. $A^{+}b = u^{+}$ puts $u^{+}$ in the row space of $A^{+}$.

\subsubsection{Computing $u^{+}$ with Tychonov Regularization}
SVD may not always be the most practical approach for solving the normal equations. When $A^{T}A$ is singular (the problem here), we can add a small multiple of $\alpha I$, producing what is known as \textbf{Tychonov Regularization}:
\begin{align}
\min &\quad \norm{Au-b}^{2} + \alpha \norm{u}^{2} \\
\textrm{s.t.} &\quad (A^{T}A + \alpha I)\hat{u}_{\alpha} = A^{T}b \nonumber
\end{align}

This is similar to our original formulation of regularized LSQ, with $B = I$, $d=0$, and $\alpha \norm{u}^{2}$ as the regularizing term. When $\alpha$ is small, we prioritize the minimization of $\norm{Au-b}^{2}$. In the limit that $\alpha \rightarrow 0$, we have that
\begin{align*}
\lim\limits_{\alpha \rightarrow 0} \hat{u}_{\alpha} &= \lim\limits_{\alpha \rightarrow 0} (A^{T}A + \alpha I)^{-1} A^{T}b \\
&= (A^{T}A)^{-1}A^{T}b = A^{+}b \\
&= u^{+}
\end{align*}

We don't get $u^{+}$ exactly, with small $\alpha > 0$. There is also uncertainty in $b$. The presence of a small $\alpha$ in $(A^{T}A + \alpha I)^{-1}A^{T}$ introduces small, non-zero singular values along the diagonal of our singular values matrix $\Sigma$ in SVD. Importantly, $\frac{\sigma}{\sigma^{2} + \alpha} \rightarrow \frac{1}{\sigma}$ while $\alpha > 0$, but stays zero if $\sigma = 0$.

\subsection{Tychonov Regularization}
This section discusses how to choose $\alpha$. The key ideas are as follows:
\begin{enumerate}
	\item There is noise in measurement of $b$, represented by an error $e$.
	\item $e$ is amplified when $A$ is ill-conditioned (e.g. singular $A^{T}A$) and $\alpha = 0$ (no regularization). This is why we use regularization.
	\item Choosing an appropriate $\alpha$, we can stabilize the problem by adding $\alpha I$. However, if $\alpha$ is too small, then $e$ will still grow with $A^{-1}$. If $\alpha$ is too large, we lose accuracy on $\hat{u}$.
\end{enumerate}

We will refer to multiple versions of the solution using the notation $\hat{u}_{i}^{j}$, where $i$ represents our choice of $\alpha$, and $j$ the expected size of uncertainty in $b$. For instance, $\hat{u}_{0}^{0}$ refers to the solution without regularization or measurement error. On the other hand, $\hat{u}_{\alpha}^{e}$ is the solution attained from data with measurement error $e$, using regularization parameter $\alpha$.

\subsubsection{Error bounds in $\hat{u}$}
\paragraph{Error bounds when $A$ is a scalar. \\ }
Let's first see how different we expect the solution $\hat{u}$ to be in the two extreme cases: $\hat{u}_{0}^{0}$ (no error, no regularization) and $\hat{u}_{\alpha}^{e}$ (error and regularization).
\begin{align}
\hat{u}_{0}^{0} - \hat{u}_{\alpha}^{e} &= \hat{u}_{0}^{0} - \hat{u}_{0}^{0} + \hat{u}_{0}^{0} - \hat{u}_{\alpha}^{e} \\
\norm{\hat{u}_{0}^{0} - \hat{u}_{0}^{\alpha}} \leq C\alpha\norm{b} &\qquad \norm{\hat{u}_{\alpha}^{0} - \hat{u}_{\alpha}^{e}} \leq \frac{\norm{e}}{2\sqrt{\alpha}} \label{errorbounds}
\end{align}

Where, $C$ is a constant that we will define below. Let's first prove the error bounds in Eq. \ref{errorbounds} when $A$ is a scalar $s$. Then, $A^{T}A = s^{2}$. When noise is absent from measurements:
\begin{align}
\textrm{Without regularization:} &\quad s^2 \hat{u}_{0}^{0} = sb \\
\textrm{With regularization:} &\quad (s^{2}+\alpha)\hat{u}_{\alpha}^{0} = sb \\
\hat{u}_{0}^{0} - \hat{u}_{\alpha}^{0} &= \frac{b}{s} - \frac{sb}{s^{2}+\alpha} \nonumber \\
&= \frac{b(s^{2}+\alpha)-s^{2}b}{s(s^{2}+\alpha)} \nonumber \\
&= \frac{\alpha b}{s(s^{2}+\alpha)} \leq C \alpha b, \quad C = \frac{1}{s^{3} + s\alpha} \label{errorbounds_scalar_nonoise}
\end{align}

Eq. \ref{errorbounds_scalar_nonoise} shows that, in the absence of measurement noise, the error associated with regularization is proportional to $\alpha$, i.e. $O(\alpha)$. $C$ scales steeply with $1/s^{3}$. We perform the same procedure to compare the effect of measurement noise:

\begin{align}
\textrm{Without noise:} &\quad (s^{2} + \alpha)\hat{u}_{\alpha}^{0} = sb \\
\textrm{With noise:} &\quad (s^{2} + \alpha)\hat{u}_{\alpha}^{e} = s(b-e) \\
\hat{u}_{\alpha}^{0} - \hat{u}_{\alpha}^{e} &= \frac{sb}{s^{2}+\alpha} - \frac{s(b-e)}{s^{2} + \alpha} \nonumber \\
&= \frac{s}{s^{2}+\alpha}e
\end{align}
{The maximum value of $s/(s^{2}+\alpha)$ will determine the bound. We can find this value by setting its derivative to zero.
\begin{align}
0 &= \frac{d}{ds}\frac{s}{s^{2}+\alpha} = (s^{2}+\alpha)^{-1} - s(s^{2}+\alpha)^{-2}(2s) \nonumber \\
&= \frac{1}{s^{2}+\alpha} - \frac{2s^{2}}{(s^{2}+\alpha)^{2}} \nonumber \\
\frac{2s^{2}}{(s^{2}+\alpha)^{2}} &= \frac{1}{s^{2}+\alpha} \implies 2s^{2} = s^{2} + \alpha, \quad \therefore s = \sqrt{\alpha} \nonumber \\
\therefore \frac{s}{s^{2}+\alpha} &\leq  \frac{1}{2\sqrt{\alpha}} \label{errorbound_scalar_withnoise}
\end{align}

\paragraph{Error bounds when $A$ is a matrix. \\}
SVD of $A$ produces orthonormal bases $\{u_{1}, u_{2}, \dotsm \}$ and $\{v_{1}, v_{2}, \dotsm \}$, with $Av_{j} = \sigma_{j}u_{j}$ and $A^{T}u_{j} = \sigma_{j}v_{j}$. Note that, unlike before, we are not using $v_i$ to refer to the eigenvectors of $A$; instead, they are the right-singular vectors of $A$.

We can apply SVD to rewrite the regularized LSQ problem in terms of $u_{j}$ and $v_{j}$:
\begin{align}
(A^{T}A + \alpha I)\hat{u}^{0}_{\alpha} &= A^{T}b \nonumber \\
b = B_{1}u_{1} + B_{2}u_{2} + \dotsm, &\quad \hat{u}^{0}_{\alpha} = U_{1}v_{1} + U_{2}v_{2} + \dotsm \nonumber \\
(A^{T}A + \alpha I)U_{j}v_{j} &= A^{T}B_{j}u_{j}  \\
U_{j} A^{T}A v_{j} + U_{j} \alpha I v_{j} &= \sigma_{j}B_{j}v_{j} \nonumber \\
U_{j} \sigma_{j} A^{T} u_{j} + U_{j} \alpha v_{j} &=  \nonumber \\
U_{j} \sigma_{j} (\sigma_{j}v_{j}) + U_{j} \alpha v_{j} &= \nonumber \\
U_{j}v_{j}(\sigma_{j}^{2}+\alpha) &= \nonumber \\
U_{j}(\sigma_{j}^{2}+\alpha) &= \sigma_{j}B_{j} \\
\implies U_{j} &= \frac{\sigma_{j}}{\sigma_{j}(\sigma_{j}^{2}+\alpha)}B_{j} = \frac{\sigma_{j}}{\sigma_{j}^{3}+\sigma_{j}\alpha}B_{j} \label{errorbound_matrix_nonoise}
\end{align}

Eq. \ref{errorbound_matrix_nonoise} is identical to what we saw for the scalar case (Eq. \ref{errorbounds_scalar_nonoise}) if $s = \sigma_{j}$. Likewise, the results in the presence of noise are also similar to the scalar case. To get the total difference between either solution group, we need to sum each of the $U_{j}$ coefficients:
\begin{align}
\hat{u}_{0}^{0} - \hat{u}_{\alpha}^{0} &= \sum_{j=1}^{\infty} \frac{\alpha B_{j}}{\sigma_{j}(\sigma_{j}^{2}+\alpha)} v_{j} \\
\hat{u}_{\alpha}^{0} - \hat{u}_{\alpha}^{e} &= \sum_{j=1}^{\infty} \frac{s_{j} E_{j}}{s_{j}^{2} + \alpha} v_{j}
\end{align}

Where, in the presence of noise, we replace $b$ with $e = E_{1}v_{1} + E_{2}v_{2} \dotsm$. The norm of either sum is a sum of squares wherein the $v_{j}$ terms disappear because they are orthonormal: $\sum_{j=1}^{\infty} v_{j}^{2} = 1$. Hence,

\begin{align}
\norm{\hat{u}_{0}^{0} - \hat{u}_{\alpha}^{0}}^{2} &= \sum_{j=1}^{\infty} \left(\frac{\alpha B_{j}}{\sigma_{j}(\sigma_{j}^{2}+\alpha)}\right)^{2} \\
\norm{\hat{u}_{\alpha}^{0} - \hat{u}_{\alpha}^{e}}^{2} &= \sum_{j=1}^{\infty} \left(\frac{s_{j} E_{j}}{s_{j}^{2} + \alpha}\right)^{2}
\end{align}

The $u_{j}$ terms are also orthonormal, so
\begin{align*}
\norm{b^{2}} = B_{1}^{2}u_{1}^{2} + B_{2}^{2}u_{2}^{2} + \dotsm = \sum_{j=1} \norm{B_{j}}^{2} \\
\norm{e^{2}} = E_{1}^{2}u_{1}^{2} + E_{2}^{2}u_{2}^{2} + \dotsm = \sum_{j=1} \norm{E_{j}}^{2}
\end{align*}

This gives
\begin{align}
\norm{\hat{u}_{0}^{0} - \hat{u}_{\alpha}^{0}} &\leq \left( \sum_{j=1}^{\infty} \frac{1}{\sigma_{j}(\sigma_{j}^{2}+\alpha)}\right) \alpha \norm{b} \approx \frac{\alpha}{\sum \sigma_{j}^{3}} \norm{b} = \frac{\alpha}{s_{min}^{3}} \norm{b} \\
\norm{\hat{u}_{\alpha}^{0} - \hat{u}_{\alpha}^{e}} &\leq \left(\sum_{j=1}^{\infty}\frac{s_{j}}{s_{j}^{2} + \alpha}\right)^{2} \norm{e} = \frac{\norm{e}}{2\sqrt{\alpha}} \label{errorbound_matrix_noise_final}
\end{align}

In Eq. \ref{errorbound_matrix_noise_final}, we made the substitution $s_{j} = \sqrt{\alpha}$, by analogy with the scalar case (see Eq. \ref{errorbound_scalar_withnoise}).

\paragraph{What is a good $\alpha$? \\}
We can find a possible $\alpha$ by finding the value that equalizes the two parts of overall error (Eq. \ref{errorbounds}):
\begin{align}
C\alpha \norm{b} &= \frac{\norm{e}}{2\sqrt{\alpha}} \\
2\alpha^{3/2} = \frac{\norm{e}}{C\norm{b}} &\implies \alpha = \left(\frac{\norm{e}}{2C\norm{b}}\right)^{2/3}
\end{align}

Then, the overall error is found by plugging this value into the two parts of error, and summing them:
\begin{align*}
\norm{\hat{u}_{0}^{0} - \hat{u}_{\alpha}^{e}} &\leq C\alpha \norm{b} + \frac{\norm{e}}{2\sqrt{\alpha}} \\
&\leq \frac{\norm{e} + 2C \alpha^{3/2} \norm{b} }{2\sqrt{\alpha}} \\
&\leq \frac{\norm{e} + C\norm{b}\left(\frac{\norm{e}}{C\norm{b}}\right) }{2 \left(\frac{\norm{e}}{2C\norm{b}}\right)^{1/3}} \\
&\leq \frac{2\norm{e} \left(2C\norm{b}\right)^{1/3} }{2\norm{e}^{1/3}} = \frac{\norm{e} \left(2C\norm{b}\right)^{1/3} }{\norm{e}^{1/3}} \\
&\leq \norm{e}^{2/3} \left(2C\norm{b}\right)^{1/3} = \left[\norm{e}^{2} \norm{b}C\right]^{1/3}
\end{align*}

Strang writes that this rule suggests that "we know more than we really do." I'm not sure what to make of this, but my guess is that this is due to the exponent of $\norm{e}$ being 2/3, rather than $\geq 1$. Perhaps, this means that this choice of $\alpha$ underestimates how much error there really is in our measurements.

\medskip
\bibliographystyle{unsrt}
\bibliography{./bib/strang_and_lay_linalg}

\end{document}