\chapter{Robust linear regression}
\label{chap:robreg}

This chapter is devoted to the estimation of the parameters of linear
regression models. Let us first precise some notations.

\section{The linear regression model}

In a linear regression model, we try to explain a variable $y$---the
\emph{dependent} variable---as a linear function of some \emph{explanatory}
variables (or \emph{predictors}) $x_1, \dots, x_p$: we assume that
%
\begin{equation}
    \label{eq:linear_regr_model}
    y = \beta_0 + \beta_1x_1 + \dots + \beta_px_p + \varepsilon
\end{equation}
%
where $\beta_0, \beta_1, \dots, \beta_p$ are unknown regression
coefficients---$\beta_0$ is called the \emph{intercept} and $\beta_1,
\dots, \beta_p$ are the \emph{slopes}---that have to be estimated and
$\varepsilon$ is a random error term (the error of the statistical model, due
to omitted factors, errors of measurement, random effects, etc.).

To estimate the regression coefficients, we need a random sample of
realizations of $(y_i, x_{i1}, \ldots, x_{ip})$, $i=1, \dots, n$, where $n$
is the sample size. We have
%
\begin{equation}
    \label{eq:linear_regr_model_sample}
    y_i = \beta_0 + \beta_1x_{i1} + \dots + \beta_px_{ip} + \varepsilon_i,
    \qquad i = 1, \dots, n
\end{equation}
%
where $\varepsilon_i$'s are generally assumed to be i.i.d.\ random variables.
That is,
\[
    \varepsilon_i \stackrel{\text{i.i.d.}}{\sim} F_{0, \sigma}
\]
where distribution $F_{0, \sigma}$ has a location (centrality) parameter equal
to zero and a scale parameter equal to $\sigma$.

Denoting by $\stvec{x}_i$ and $\boldsymbol\beta$ the $(p+1)$ dimensional
column vectors with coordinates $(1, x_{i1}, \dots, x_{ip})$ and $(\beta_0,
\beta_1, \dots, \beta_p)$, respectively, equation
(\ref{eq:linear_regr_model_sample}) can be more compactly written as
%
\begin{equation}
    \label{eq:linear_regr_model_sample_bis}
    y_i = \stvec{x}_i^t\boldsymbol\beta + \varepsilon_i,
    \qquad i = 1, \dots, n.
\end{equation}
%
Furthermore, letting $\stvec{y} = (y_1, \dots, y_n)^t$,
$\boldsymbol\varepsilon = (\varepsilon_1, \dots, \varepsilon_n)^t$ and
\[
    \stmat{X} =
    \begin{pmatrix}
        1       & x_{11} & \dots & x_{1p} \\
        1       & x_{21} & \dots & x_{2p} \\
        \vdots  & \vdots & \dots & \vdots \\
        1       & x_{n1} & \dots & x_{np}
    \end{pmatrix}
    =
    \begin{pmatrix}
        \stvec{x}_1^t \\
        \stvec{x}_2^t \\
        \vdots         \\
        \stvec{x}_n^t
    \end{pmatrix}
\]
equations (\ref{eq:linear_regr_model_sample_bis}) takes the matrix-notation form
\[
    \stvec{y} = \stmat{X}\boldsymbol\beta + \boldsymbol\varepsilon.
\]

Let us note here that, \emph{conditionally to the predictors}, the linear
regression model (\ref{eq:linear_regr_model_sample_bis}) may be considered as a
location-scale model. Indeed, under the assumption of homoscedasticity---an
identical scale parameter $\sigma$ for each error term
$\varepsilon_i$---regression model (\ref{eq:linear_regr_model_sample_bis})
may be formulated as follows:
%
\begin{equation}
    \label{eq:location_scale_regr_model}
    y_i = \stvec{x}_i^t\boldsymbol\beta + \sigma\nu_i,
    \qquad i = 1, \dots, n
\end{equation}
%
where the $\nu_i$'s are i.i.d.\ with distribution function $F_{0,1}$ (a
distribution with a location parameter equal to zero and a scale parameter
equal to one). In this case, the conditional distribution of $y_i$ given
$\stvec{x}_i$ is of the form:
%
\begin{equation}
    \label{eq:distr_function_yi}
    F_{y_i | \stvec{x}_i}(y)
    = \Pr(y_i \leq y | \stvec{x}_i)
    = \Pr\left(\nu_i \leq \frac{y - \stvec{x}_i^t\boldsymbol\beta}{\sigma} \bigg| \stvec{x}_i\right)
    = F_{0,1}\left(\frac{y - \stvec{x}_i^t\boldsymbol\beta}{\sigma}\right)
\end{equation}
%
Furthermore, if $f_{0,1}$ denotes the density function of the error terms
$\nu_i$, that is,
\[
    f_{0,1}(u) = \frac{d F_{0,1}(u)}{d u} = F_{0,1}'(u)
\]
then
%
\begin{equation}
    \label{eq:dens_function_yi}
    f_{y_i | \stvec{x}_i}(y)
    = \frac{1}{\sigma} f_{0,1}\left(\frac{y - \stvec{x}_i^t\boldsymbol\beta}{\sigma}\right).
\end{equation}
%
Hence, $\stvec{x}_i^t\boldsymbol\beta$ corresponds to the unknown location
parameter of the distribution of $y_i$ and $\sigma$ is the scale parameter of
the distribution of $y_i$. For simplicity, we will consider that the
distribution $F_{0,1}$ is continuous and symmetric around zero (and exception
is Section \ref{sec:inference}).

\begin{stremark}
The classic \emph{location-scale model} may be seen as a particular case of
regression model (\ref{eq:location_scale_regr_model}). It suffices to set
$\beta_1 = \dots = \beta_p = 0$ and to assume that the $\nu_i$'s are
i.i.d.\ with distribution $F_{0,1}$ (such that $E(\nu_i)=0$). In this case,
the observations
%
\begin{equation}
    \label{eq:location_scale_model}
    y_i = \beta_0 + \sigma\nu_i,
    \qquad i=1, \dots, n,
\end{equation}
%
are i.i.d.\ with a common distribution $F$ characterized by mean $\mu =
\beta_0$ and scale parameter $\sigma$.
\end{stremark}

\begin{stremark}
Most textbook presentations of the linear regression model assume the
explanatory variables to be fixed (and measured without error). That is, the
explanatory variables are not assumed to be random variables. In the context of
a \emph{designed experiment}, this assumption is reasonable since the values of
the experimental factors are determined \emph{a priori} by the researchers. In
other contexts such as, for example, when using social-science survey data, the
assumption makes no sense.
    \todo{Say here what the consequence is: The fact that the $X$'s are random
    doesn't really change anything. (unlike measurement error which attenuates
    the estimates)}

Nonetheless, since we focus on the problem of outlying values, we will ignore
the issue in this chapter and consider $x_{ij}$, $i = 1, \dots, n$, $j = 1,
\dots, p$, as predetermined. That is, results will always be conditional on the
particular \emph{values} taken by the explanatory variables.
    \todo{I'm not sure whether this remark makes sense. First, LS results are
    valid also if the $X$'s are random. Second, if we talk about $X$ outliers
    it makes not much sense to assume $X$ fixed.}
\end{stremark}

\section{Different types of outliers}

Model (\ref{eq:linear_regr_model}) assume that \emph{all} units of the
population and, \emph{de facto}, all units of the sample are consistent with
the supposed linear model. If a unit has a behavior that does not respect the
underlying theoretical model, we define it as an \emph{outlying} unit with
respect to the model.

Of course, in the case of \emph{simple} linear regression model ($p=1$), a
visual inspection of the scatterplot is generally sufficient to detect the
outliers. But, when the number of explanatory variables is greater than two,
it becomes impossible to visualize all the data set and the use of robust
methods to estimate the regression parameters is then essential. More
precisely, we aim at developing procedures that provide a good fit to the bulk
of the data without being perturbed by a small proportion of outliers, and
that do not require deciding previously which observations are outliers.
Moreover, the comparison between the estimations provided by the classical
least squares estimator and those obtained using a robust estimation procedure
will allow to bring to the fore the outlyingness of some data.

In cross-sectional regression analysis, three types of outliers may influence
the estimations. \citet{rousseeuw:leroy:1987} define them as \emph{vertical
outliers}, \emph{good leverage points} and \emph{bad leverage points}. To
illustrate this terminology, consider a simple linear regression as shown in
figure~\ref{fig:outlier_types} (the generalization to higher dimensions is
straightforward). \emph{Vertical outliers} are those observations that have
outlying values for the corresponding error term (that is, in the
$y$-dimension) but are not outlying in the space of explanatory variables (in
the $x$-dimension). \emph{Good leverage points} are observations that are
outlying in the space of explanatory variables but that are located close to
the regression hyperplane. Finally, \emph{bad leverage points} are observations
that are both outlying in the space of explanatory variables and located far
from the true regression hyperplane.

\begin{figure}[h!]
    \centering
    \epsfig{file=eps/4/1}
    \caption{Vertical outlier, good leverage point and bad leverage point}
    \label{fig:outlier_types}
\end{figure}

All these types of outliers risk to affect the estimation of the regression
hyperplane but their effect changes according to the estimator we will
consider and the type of outlyingness. For the classical least squares
estimation method, for instance, the bad leverage points are considered as the
most dangerous outliers because their presence can change the sign of the
slope of the regression line (in simple regression); the good leverage points
have little influence on the estimation of the regression coefficients but
they have an impact on the variances and covariances of the regression
coefficients' estimators and, consequently, risk to influence the inferential
procedures (tests and confidence intervals).

The most popular estimation method in linear regression is certainly the
\emph{least squares} (\stsc{LS}) method introduced in 1805 by Legendre.          \todo{Please provide citation details!}
One of its principal advantage is the simplicity of the
computation of the \stsc{LS}~estimates. Its popularity has also be reinforced
by the fact that, under the normality of the error terms, \stsc{LS}~estimates
of the regression coefficients coincide with the maximum likelihood estimates.
We will first briefly review the logic behind least squares (\stsc{LS})
estimation and recall why the \stsc{LS}~estimator is particularly affected by
the presence of atypical individuals. We will thereafter introduce some
alternative estimation methods that have been proposed to try to cope with
outliers.

\section{LS estimation}

Let us denote by $\sthat{y}_i(\boldsymbol\beta)$ the value fitted by the
regression model for the $i$th statistical unit of the sample when taking
$\boldsymbol\beta$ as value for the vector of regression coefficients:
\[
    \sthat{y}_i(\boldsymbol\beta)  = \stvec{x}_i^t\boldsymbol\beta,
    \qquad i = 1, \dots, n
\]
The difference between the observed value $y_i$ and the fitted value
$\sthat{y}_i(\boldsymbol\beta)$ is the residual $r_i(\boldsymbol\beta)$:
\[
    r_i(\boldsymbol\beta) = y_i - \sthat{y}_i(\boldsymbol\beta),
    \qquad i = 1, \dots, n.
\]

Although $\boldsymbol\beta$ can be estimated in several ways, the underlying
idea is often to take an estimate $\sthat{\boldsymbol\beta}$ in such a way that
the fitted values $\sthat{y}_i(\sthat{\boldsymbol\beta})$ for the dependent
variable are as close as possible to the observed values $y_i$ ($i = 1,
\dots, n$), i.e., in such a way that we minimize globally the magnitude of the
residuals $r_i(\sthat{\boldsymbol\beta})$. This idea leads to try to find the
estimate $\sthat{\boldsymbol\beta}$ that minimizes a specific aggregate
prediction error.

In the case of the well-known ordinary least squares (\stsc{LS}), this
aggregate prediction error is defined as the sum of squared residuals:
%
\begin{equation}\label{eq:LS_min}
    \sthat{\boldsymbol\beta}_\stsc{LS} = \argmin_{\boldsymbol\beta}
    \sum_{i=1}^{n} r_i(\boldsymbol\beta)^2
\end{equation}
%
where “$\argmin$” stands for “the value minimizing”. In other terms,
$\sthat{\boldsymbol\beta}_\stsc{LS}$ is solution of the so called
\emph{normal equations} system---we will also call it the \emph{estimating
equations} system---obtained by differentiating the function $\sum_{i=1}^{n}
r_i(\boldsymbol\beta)^2$ to minimize with respect to each component of
$\boldsymbol\beta$, that is, $\sthat{\boldsymbol\beta}_\stsc{LS}$ is the
solution of
%
\begin{equation}\label{eq:LS_equations}
    \sum_{i=1}^{n}  r_i(\boldsymbol\beta)\stvec{x}_i = \stvec{0}
\end{equation}
%
which is equivalent to the linear equations system
\[
    \stmat{X}^t\stmat{X}\boldsymbol\beta = \stmat{X}^t\stvec{y}.
\]
If $\stmat{X}$ has full rank\footnote{The matrix of predictors $\stmat{X}$ is
said to have \emph{full rank} if its columns are linearly independent (absence
of multicollinearity), that is, if $\stmat{X}\stvec{a}\neq\stvec{0}$ for all
$\stvec{a}\neq\stvec{0}$. This is equivalent to the nonsingularity of
$\stmat{X}'\stmat{X}$.}, then the solution of (\ref{eq:LS_equations}) is unique
and is given by
%
\begin{equation}\label{eq:LS_estimator}
    \sthat{\boldsymbol\beta}_\stsc{LS}
    = \sthat{\boldsymbol\beta}_\stsc{LS}(\stmat{X}, \stvec{y})
    = (\stmat{X}^t\stmat{X})^{-1} \stmat{X}^t\stvec{y}.
\end{equation}
%
This estimate can be computed in Stata using the \stcmd{regress} command (see
\rref{regress}).

Note here that, if the model contains a constant term $\beta_0$, that is, if
the first component of the vectors $\stvec{x}_i$, $i = 1, \dots, n$, is equal
to one, it follows from (\ref{eq:LS_equations}) that the residuals
$r_i(\sthat{\boldsymbol\beta}_\stsc{LS})$, $i = 1, \dots, n$, have zero
average.

It is easy to verify that the \stsc{LS}~estimator satisfies
(see \citealp[92]{maronna:etal:2006})
%
\begin{align}
    \label{eq:regression_equivariance}
    \sthat{\boldsymbol\beta}_\stsc{LS}(\stmat{X},\stvec{y}+\mathbf{X}\boldsymbol\gamma)
        &= \sthat{\boldsymbol\beta}_\stsc{LS}(\stmat{X},\stvec{y})+\boldsymbol\gamma
        \qquad\text{for all $\boldsymbol\gamma\in\mathbb{R}^{p+1}$} \\
    \label{eq:scale_equivariance}
    \sthat{\boldsymbol\beta}_\stsc{LS}(\stmat{X},\lambda\stvec{y})
        &= \lambda\sthat{\boldsymbol\beta}_\stsc{LS}(\stmat{X}, \stvec{y})
        \qquad\text{for all $\lambda\in\mathbb{R}$}
\end{align}
%
and, for any nonsingular $(p+1) \times (p+1)$ matrix $\stmat{A}$,
%
\begin{equation}\label{eq:affine_equivariance}
    \sthat{\boldsymbol\beta}_\stsc{LS}(\stmat{X}\stmat{A},\stvec{y})
    = \stmat{A}^{-1}\sthat{\boldsymbol\beta}_\stsc{LS}(\stmat{X},\stvec{y}).
\end{equation}
%
The properties (\ref{eq:regression_equivariance}),
(\ref{eq:scale_equivariance}) and (\ref{eq:affine_equivariance}) are called
\emph{regression}, \emph{scale} and \emph{affine equivariance} of
$\sthat{\boldsymbol\beta}_\stsc{LS}$, respectively. In the sequence, it will
be desirable that every other estimator of $\boldsymbol\beta$ also satisfies
these natural properties.

It is also well known that the \stsc{LS}~estimator of $\boldsymbol\beta$ coincides
with the maximum likelihood estimator in case of normally distributed error
terms in (\ref{eq:linear_regr_model_sample}). Hence, $\widehat
{\boldsymbol\beta}_\stsc{LS}$ is the most efficient estimator of
$\boldsymbol\beta$ in the Gaussian regression model.

However, an important drawback of \stsc{LS} is that, by considering squared
residuals, it tends to award an excessive importance to observations with large
residuals and, consequently, distort parameters estimation when outliers exist.

\section{M estimation}

\subsection{Least Absolute Deviation (LAD) estimation}

\citet{edgeworth:1887} realized that due to the squaring of the residuals,
\stsc{LS} becomes extremely vulnerable to the presence of outliers. To cope
with this, he proposed a method consisting in minimizing the sum of the
absolute values of the residuals rather than the sum of their squares. More
precisely, his method defines the $L_1$ or \emph{least absolute deviation}
(\stsc{LAD}) estimate as
%
\begin{equation}\label{eq:L1_min}
    \sthat{\boldsymbol\beta}_{\stsc{LAD}}
        = \argmin_{\boldsymbol\beta} \sum_{i=1}^{n} |r_i(\boldsymbol\beta)|.
\end{equation}
%
This estimate is solution of the estimating equations system obtained by
differentiating the sum of the absolute values of the residuals with respect
to each component of $\boldsymbol\beta$:
%
\begin{equation}\label{eq:L1_equations}
    \sum_{i=1}^{n}\sign(r_i(\sthat{\boldsymbol\beta}_{\stsc{LAD}}))
    \stvec{x}_i = \stvec{0}
\end{equation}
%
If the model contains an intercept term, (\ref{eq:L1_equations}) implies that
the residuals $r_i(\sthat{\boldsymbol\beta}_{\stsc{LAD}})$, $i = 1, \dots,
n$, have a median equal to zero; this motivates the fact that the \stsc{LAD}
regression estimator is also sometimes called the \emph{median regression
estimator}.

Unlike for $\sthat{\boldsymbol\beta}_\stsc{LS}$, there is no explicit
expression for $\sthat{\boldsymbol\beta}_{\stsc{LAD}}$.\footnote{Note also that
the \stsc{LAD}~estimate of $\boldsymbol\beta$ may not be unique and has the
property that at least $(p+1)$ residuals are equal to zero.} However, there
exist very fast algorithms to compute it and
$\sthat{\boldsymbol{\beta}}_{\stsc{LAD}}$ is available in Stata via the
\stcmd{qreg} command as a standard function (see \rref{qreg}).

Finally, it can easily be seen from (\ref{eq:L1_min}) and
(\ref{eq:L1_equations}) that this estimator does protect against vertical
outliers (but not against bad leverage points). However, this gain in
robustness with respect to the \stsc{LS}~estimator comes with an important loss
of efficiency: the asymptotic relative efficiency of
$\sthat{\boldsymbol{\beta}}_{\stsc{LAD}}$ with respect to
$\sthat{\boldsymbol\beta}_\stsc{LS}$ is equal to $2/\pi = 63.7\%$ at a
Gaussian error distribution (see \citealp{huber:1981}).

\subsection{The principle of M estimation}

\citet{huber64} hence generalized median regression to a wider class of
estimators, called \stsc{M}~estimators, by considering other functions than
the absolute value in (\ref{eq:L1_min}) in order to find a reasonable balance
between robustness and Gaussian efficiency.

An \stsc{M}~estimate $\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ of
$\boldsymbol\beta$ is defined by
%
\begin{equation}
    \label{eq:M_min}
    \sthat{\boldsymbol\beta}_{\stsc{M};\rho}
        = \argmin_{\boldsymbol\beta}\sum_{i=1}^{n}
            \rho\left(\frac{y_i - \stvec{x}_i^t\boldsymbol\beta}{\sthat{\sigma}}\right)
        = \argmin_{\boldsymbol\beta}\sum_{i=1}^{n}
            \rho\left(\frac{r_i(\boldsymbol\beta)}{\sthat{\sigma}}\right)
\end{equation}
%
where $\rho(u)$ is a loss function that is positive, even such that $\rho(0) =
0$, and non decreasing for positive values $u$, and $\sthat{\sigma}$ is an
auxiliary estimate of the scale parameter $\sigma$ required to standardize the
residuals and to make $\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ scale
equivariant; see (\ref{eq:scale_equivariance}). In most situations,
$\sthat{\sigma}$ is computed in advance, but it can also be computed
simultaneously through a scale \stsc{M} estimating equation. This problem will
be discussed in more details later.

\begin{stremark}
The \stsc{LS}~estimate and the \stsc{LAD}~estimate correspond respectively to
$\rho(u) = u^2$ and $\rho(u) = |u|$. In these two cases, $\sthat{\sigma}$
becomes a constant factor outside the summation sign in (\ref{eq:M_min}) and
\[
    \argmin_{\boldsymbol\beta} \sum_{i=1}^{n} \rho\left(\frac{r_i(\boldsymbol\beta)}{\sthat{\sigma}}\right)
    = \argmin_{\boldsymbol\beta}\sum_{i=1}^{n}\rho(r_i(\boldsymbol\beta)).
\]
Thus neither the \stsc{LS} nor the \stsc{LAD}~estimate require an auxiliary
scale estimate.
\end{stremark}

Of course, if we want a \stsc{M}~estimator more robust against vertical
outliers than the \stsc{LS}~estimator, we have to take a loss function $\rho$
that is less rapidly increasing than the square function in order to give less
weight to big (in absolute value) residuals in the minimization problem. In
order to combine robustness and efficiency under a Gaussian error distribution,
\citet{huber64} has suggested to use for $\rho$ a function of the form (see
figure~\ref{fig:rho_psi_Huber}):
\[
    \rho_{\kappa}^{\stsc{H}}(u) =
    \begin{cases}
        u^2                    & \text{if $|u| \leq \kappa$}\\
        2\kappa |u| - \kappa^2 & \text{if $|u| > \kappa$}
    \end{cases}
\]
where $\kappa$ is a constant determining the trade-off between robustness and
efficiency. These functions of Huber are convex on the whole real line and may
be seen as intermediate functions between the quadratic function (leading to
the non robust but efficient \stsc{LS}~estimate) and the absolute value function
(associated with the robust but poorly efficient \stsc{LAD}~estimate).

Another class of loss functions $\rho$ widely used in the literature is the
class of the Tukey-Biweight                                                     \todo{Please provide citation!}
functions (see figure~\ref{fig:rho_psi_Biweight})
%
\begin{equation}
    \label{eq:Tukey_Biweight_function}
    \rho_\kappa^\stsc{B}(u) =
        \begin{cases}
            \frac{\kappa^2}{6} \left[1 - \left(1 - \left(\frac{u}{\kappa}\right)^{2}\right)^{\!3}\right]
                & \text{if $|u| \leq \kappa$}\\
            \frac{\kappa^2}{6}
                & \text{if $|u| > \kappa$}
        \end{cases}
\end{equation}
%
These functions are bounded. Once again, the constant $\kappa$ allows the
trade-off between robustness and Gaussian efficiency. We will show the
advantage and disadvantage to use a bounded function $\rho$ hereafter.

We may also characterize $\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ as a
solution of the estimating equations system obtained by differentiating the
function to minimize in (\ref{eq:M_min}) with respect to each component of
$\boldsymbol\beta$, that is, as a solution of the equations system
%
\begin{equation}\label{eq:M_equations}
    \sum_{i=1}^{n} \psi\left(\frac{r_i(\boldsymbol\beta)}{\sthat{\sigma}}\right) \stvec{x}_i = \stvec{0}
\end{equation}
%
where $\psi(u) = d\rho(u) / du = \rho'(u)$. For instance, taking
$\rho(u)=\rho_{\kappa}^{\stsc{H}}(u)$, we have
\[
    \psi_{\kappa}^{\stsc{H}}(u) =
    \begin{cases}
        -2\kappa    & \text{if $u<-\kappa$}\\
        2u          & \text{if $-\kappa\leq u\leq\kappa$}\\
        2\kappa     & \text{if $u>\kappa$}
    \end{cases}
\]
for $\rho(u) = \rho_{\kappa}^{\stsc{B}}(u)$, we obtain
\[
    \psi_{\kappa}^{\stsc{B}}(u) =
    \begin{cases}
        u\left(1 - \left(\frac{u}{\kappa}\right)^2\right)^{\!2} & \text{if $|u| \leq \kappa$}\\
        0                                                       & \text{if $|u| > \kappa$}
    \end{cases}
\]
(see figures \ref{fig:rho_psi_Huber} and \ref{fig:rho_psi_Biweight}). If the
loss function $\rho$ is convex on $\mathbb{R}$---this is the case for
$\rho_{\kappa}^{\stsc{H}}$---the score function $\psi$ is monotone (non
decreasing) on $\mathbb{R}$ and $\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ is
called a \emph{monotone} regression \stsc{M}~estimator; if $\rho$ is
bounded---this is the case for $\rho_{\kappa}^{\stsc{B}}$---the score function
$\psi$ vanishes out of a certain interval of $\mathbb{R}$ and
$\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ is then called a \emph{redescending}
regression \stsc{M}~estimator.

\begin{figure}[h!]
    \centering
    \epsfig{file=eps/4/2}
    \caption{Huber loss function $\rho_{\kappa}^{\stsc{H}}$ and score function $\psi_{\kappa}^{\stsc{H}}$}
    \label{fig:rho_psi_Huber}
\end{figure}

\begin{figure}[h!]
    \centering
    \epsfig{file=eps/4/3}
    \caption{Tukey-Biweight loss function $\rho_{\kappa}^{\stsc{B}}$ and score function $\psi_{\kappa}^{\stsc{B}}$}
    \label{fig:rho_psi_Biweight}
\end{figure}

The main advantage of monotone score functions $\psi$ is that all solutions of
(\ref{eq:M_equations}) are solutions of (\ref{eq:M_min}). In the case of
redescending score functions $\psi$, the estimating equations
(\ref{eq:M_equations}) may have multiple solutions corresponding to multiple
local minima of $\sum_{i=1}^{n}\rho(r_i(\boldsymbol\beta)/\sthat{\sigma})$,
and generally only one of them (the “good” solution) corresponds to the global
minimizer $\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ defined by
(\ref{eq:M_min}), which makes the computation of the \stsc{M}~estimate
considerably more complex.

\begin{stremark}
Applying the \stsc{M}~estimation procedure in the particular case of the
location-scale model (\ref{eq:location_scale_model}) leads to the
\stsc{M}~\emph{estimators of location and scale}---we just mentioned the
existence of these estimators in the previous chapter. The interested reader
will find some results relative to these specific estimators in appendix
\ref{sec:robreg:appendix1} at the end of this chapter.
\end{stremark}

\subsection{M estimation as a generalization of maximum likelihood (ML) estimation}

The \stsc{M}~estimation as defined above may be seen, as already explained, as
a generalization of the \stsc{LS} or \stsc{LAD}~estimation, but also as a
generalization of the maximum-likelihood (\stsc{ML}) estimation (see, for
instance, \citealp{maronna:etal:2006}). Indeed, assuming model
(\ref{eq:location_scale_regr_model}) with fixed $\stvec{x}_i$ and with
$\nu_i$, $i = 1, \dots, n$, i.i.d.\ of density $f_{0,1}$, the likelihood of
the sample $\{y_1, \dots, y_n\}$ is given by                                \todo{Couldn't we also just use unspecified $f$ instead of $f_{0,1}$?}
\[
    \frac{1}{\sigma^{n}}\prod\limits_{i=1}^{n}
    f_{0,1}\left(\frac{y_i-\stvec{x}_i^t\boldsymbol\beta}{\sigma}\right).
\]
Hence, maximum likelihood estimation of the parameters $\boldsymbol\beta$
and $\sigma$ consists in looking for
%
\begin{align}
    \left(\sthat{\boldsymbol\beta}_{\stsc{ML}}^t,\sthat{\sigma}_{\stsc{ML}}\right)^{\!t}
    &= \arg\max_{\boldsymbol\beta,\sigma} \frac{1}{\sigma^{n}} \prod\limits_{i=1}^{n}
        f_{0,1}\left(\frac{y_i-\stvec{x}_i^t\boldsymbol\beta}{\sigma}\right)
    \nonumber\\
    &= \arg\max_{\boldsymbol\beta,\sigma} \left[\sum_{i=1}^{n}
        \ln f_{0,1}\left(\frac{y_i-\stvec{x}_i^t\boldsymbol\beta}{\sigma}\right) - n\ln\sigma\right]
    \nonumber\\
    & = \argmin_{\boldsymbol\beta,\sigma} \left[\sum_{i=1}^{n}
        \rho_{\stsc{ML}}\left(\frac{r_i(\boldsymbol\beta)}{\sigma}\right) + n\ln\sigma\right]
    \label{eq:ML_beta_sigma_min}
\end{align}
%
where $\rho_{\stsc{ML}}(u) = -\ln f_{0,1}(u)$. If $\sigma$ is known, the
minimization problem simply becomes
%
\begin{equation}\label{eq:ML_beta_min}
    \sthat{\boldsymbol\beta}_{\stsc{ML}}
        = \argmin_{\boldsymbol\beta}\sum_{i=1}^{n}
            \rho_{\stsc{ML}}\left(\frac{r_i(\boldsymbol\beta)}{\sigma}\right)
\end{equation}
%
and $\sthat{\boldsymbol\beta}_{\stsc{ML}}$ is solution of the estimating
equations system
\[
    \sum_{i=1}^{n} \psi_{\stsc{ML}}\left(\frac{r_i(\boldsymbol\beta)}{\sigma}\right) \stvec{x}_i = \stvec{0}
\]
where $\psi_{\stsc{ML}}(u) = \rho_{\stsc{ML}}'(u) = - (1/f_{0,1}(u))
f_{0,1}'(u)$. If $f_{0,1}$ is the standard normal density function,
$\sthat{\boldsymbol\beta}_{\stsc{ML}}$ coincides with
$\sthat{\boldsymbol\beta}_\stsc{LS}$. If $f_{0,1}$ is the density function of
the Laplace distribution, that is, if $f_{0,1}(u) =
\frac{1}{\sqrt{2}}\exp(-\sqrt{2}|u|)$, $u\in\mathbb{R}$, then
$\sthat{\boldsymbol\beta}_{\stsc{ML}}$ is equal to
$\sthat{\boldsymbol\beta}_{\stsc{LAD}}$.

If $\sigma$ is not known but is estimated beforehand and fixed in
(\ref{eq:ML_beta_min}), the estimating equations system becomes
\[
    \sum_{i=1}^{n} \psi_{\stsc{ML}}\left(\frac{r_i(\boldsymbol\beta)}{\sthat{\sigma}}\right) \stvec{x}_i = \stvec{0}
\]
Note that, if $\boldsymbol\beta$ and $\sigma$ are estimated simultaneously,
the estimating equations system related to (\ref{eq:ML_beta_sigma_min}) is
%
\begin{equation}\label{eq:ML_beta_sigma_equations}
    \begin{aligned}
        \sum\limits_{i=1}^{n} \psi_{\stsc{ML}}\left(\frac{r_i(\boldsymbol\beta)}{\sigma}\right) \stvec{x}_i
        & = \stvec{0}
        \\
        \frac{1}{n} \sum\limits_{i=1}^{n} \rho_{\stsc{ML};\mathrm{scale}}\left(\frac{r_i(\boldsymbol\beta)}{\sigma}\right)
        & = \delta
    \end{aligned}
\end{equation}
%
where $\rho_{\stsc{ML};\mathrm{scale}}(u) = u\psi_{\stsc{ML}}(u)$ and
$\delta=1$.

\subsection{Practical implementation of M estimates}
\label{subsec:practical_implementation_Mestimate}

Let us first assume, for simplicity, that the scale parameter $\sigma$ is
known. In that case, the regression M-estimate
$\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ is solution of the estimating
equations system (\ref{eq:M_equations}) where $\sthat{\sigma}$ is replaced by
$\sigma$. Defining the weight function $w$ by
\[
    w(u) =
    \begin{cases}
        \frac{\psi(u)}{u} & \text{if $u\neq0$}\\
        \psi'(0)          & \text{if $u=0$}
    \end{cases}
\]
the estimating system (\ref{eq:M_equations}) can be rewritten as
%
\begin{equation}\label{eq:weighted_LS_equations}
    \sum_{i=1}^{n} w_i r_i(\boldsymbol\beta) \stvec{x}_i = \stvec{0}
\end{equation}
%
where $w_i = w(r_i(\boldsymbol\beta)/\sigma)$. Hence, the equations to solve in
the \stsc{M}~estimation procedure appear as \emph{weighted} versions of the
normal equations (\ref{eq:LS_equations}) related to \stsc{LS}~estimation, and
if the $w_i$'s were known, the equations (\ref{eq:weighted_LS_equations}) could
be solved by applying \stsc{LS} to $\sqrt{w_i}y_i$ and $\sqrt{w_i}\stvec{x}_i$.
But the weights $w_i$ are functions of $\boldsymbol\beta$ and depend upon the
data, and hence are not known. So we have to use an iterative procedure. Using
an initial estimate $\sthat{\boldsymbol\beta}_0$ for $\boldsymbol\beta$ (for
instance, the \stsc{LAD}~estimate of $\boldsymbol\beta$), the weights can be
computed and serve as the start of an \emph{iteratively reweighted least
squares algorithm} (\stsc{IRWLS}). Note however that the latter is guaranteed
to converge to the global minimum of (\ref{eq:M_min}) only if the loss function
$\rho$ is convex on the whole real line $\mathbb{R}$ (which is the case for the
$\rho_{c}^{\stsc{H}}$ functions introduced by Huber).\footnote{In the case of a
convex loss function $\rho$, the convergence of the algorithm to the global
minimum of (\ref{eq:M_min}) is guaranteed whatever the starting point
$\sthat{\boldsymbol\beta}_0$.}

If $\sigma$ is not known, it can be estimated (in a robust way) beforehand
using the residuals $r_i(\sthat{\boldsymbol\beta}_0)$, $i = 1, \dots, n$,
and then fixed in the iterative procedure described above. It is of course
also possible to estimate simultaneously $\boldsymbol\beta$ and $\sigma$ in
this procedure, by updating $\sthat{\sigma}$ at each iteration (see
\citealp{maronna:etal:2006} for more details).

\subsubsection{Regression M estimate with preliminary scale estimation}

In practice, we may take the \stsc{LAD}~estimate as initial estimate
$\sthat{\boldsymbol\beta}_0$ for $\boldsymbol\beta$ (recall that the
\stsc{LAD}~estimate does not require estimating a scale). Then we may estimate
$\sigma$ using normalized \stsc{MAD} of the residuals
$r_i(\sthat{\boldsymbol\beta}_0)$. More precisely, we may take
\[
    \sthat{\sigma} = 1.4826 \cdot
    \med_i\left(\left|r_i(\sthat{\boldsymbol\beta}_0)\right|; r_i(\sthat{\boldsymbol\beta}_0) \neq 0\right).
\]
The reason for using only \emph{non null} residuals is that, since at least
$(p+1)$ residuals $r_i(\sthat{\boldsymbol\beta}_0) =
r_i(\sthat{\boldsymbol\beta}_{\stsc{LAD}})$ are equal to zero, determining
the \stsc{MAD} of the $n$ residuals could lead to underestimating $\sigma$ when
$p$ is large.

Since $\sthat{\boldsymbol\beta}_{\stsc{LAD}}$ is regression, scale and
affine equivariant, it is easy to show that
%
\begin{align*}
    \sthat{\sigma}(\stmat{X}, \stvec{y} + \stmat{X}\boldsymbol\gamma)
        &= \sthat{\sigma}(\stmat{X}, \stvec{y})
        &&\text{for all $\boldsymbol\gamma \in \mathbb{R}^{p+1}$}
    \\
    \sthat{\sigma}(\stmat{X}\stmat{A}, \stvec{y})
        &= \sthat{\sigma}(\stmat{X}, \stvec{y})
        &&\text{for any nonsingular $\stmat{A} \in \mathbb{R}^{(p+1)\times(p+1)}$}
    \\
\intertext{and}
    \sthat{\sigma}(\stmat{X}, \lambda\stvec{y})
        &= |\lambda| \sthat{\sigma} (\stmat{X}, \stvec{y})
        &&\text{for all $\lambda\in\mathbb{R}$.}
\end{align*}
Hence, $\sthat{\sigma}$ is regression and affine invariant, as well as scale
equivariant, which ensures the regression, scale end affine equivariance of
the \stsc{M}~estimator $\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$.

\subsection{Regression quantiles as regression M estimates}

Let
\[
    \rho_{\alpha}(u)=
    \begin{cases}
        \alpha u     & \text{if $u\geq0$}\\
        -(1-\alpha)u & \text{if $u<0$}
    \end{cases}
\]
for $\alpha \in (0,1)$. \citet{Koenker:1978} defined the \emph{regression $\alpha$-quantile}
$\sthat{\boldsymbol\beta}_{\alpha}$ as follows:
%
\begin{equation}\label{eq:quantile_regr_min}
    \sthat{\boldsymbol\beta}_{\alpha}
    = \argmin_{\boldsymbol\beta} \sum_{i=1}^{n} \rho_{\alpha}(y_i-\stvec{x}_i^t\boldsymbol\beta)
\end{equation}

The case $\alpha=0.5$ corresponds to the \stsc{LAD}~estimate. Assume the model
\[
    y_i = \stvec{x}_i^t \boldsymbol\beta_{\alpha} + \epsilon_i,
    \qquad i = 1, \dots, n
\]
where the $\stvec{x}_i$'s are fixed and the $\alpha$-quantile of
$\epsilon_i$ is zero; this is equivalent to assuming that the
$\alpha$-quantile of $y_i$ is, conditionally to $\stvec{x}_i$, equal to
$\stvec{x}_i^t\boldsymbol\beta_{\alpha}$. Then $\widehat
{\boldsymbol\beta}_{\alpha}$ defined by (\ref{eq:quantile_regr_min}) is an
estimate of $\boldsymbol\beta_{\alpha}$. It may be seen as a generalization of
the \stsc{LAD}~estimate as well as a specific case of \stsc{M}~estimate.

Regression quantiles are especially useful with heteroskedastic data. There is
a very large literature on regression quantiles; see, for instance,
\citet{Koenker:2005}.

\subsection{Monotone vs. redescending M estimators}

As already mentioned, taking a loss function $\rho(u)$ in the minimization
problem (\ref{eq:M_min}) that is less rapidly increasing than the square
function provides a certain robustness of the regression \stsc{M}~estimate with
respect to the vertical points. But what about the robustness of
$\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ with respect to leverage points? To
answer to this question, let us recall that
$\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ is solution of the estimating
equations system (\ref{eq:M_equations}).

It is easy to see that \emph{monotone} \stsc{M}~estimates break down in
presence of a single bad leverage point. Indeed, if $\psi(u)$ is a monotone
function, an $\stvec{x}$-outlier will dominate the solution of
(\ref{eq:M_equations}) in the following sense: if for some $i$, $\stvec{x}_i$
is “much larger than the rest”, then in order to make the sum in the left part
of (\ref{eq:M_equations}) to zero, the residual
$r_i(\sthat{\boldsymbol\beta}) = y_i -
\stvec{x}_i^t\sthat{\boldsymbol\beta}$ must be near zero, that is, the
regression hyperplane has to fit the point $(\stvec{x}_i, y_i)$ as well as
possible, and hence $\sthat{\boldsymbol\beta}$ is essentially determined by
this leverage point $(\stvec{x}_i, y_i)$.

This does not happen with the \emph{redescending} \stsc{M}~estimate since the
use of a function $\psi$ that vanishes for “outlying” residuals allows to find
a solution $\widehat{\boldsymbol\beta}_{\stsc{M};\rho}$ of
(\ref{eq:M_equations}) which is not affected by the presence of a bad leverage
point $(\stvec{x}_i, y_i)$ in the data set. Hence, from the robustness
point of view, the redescending regression \stsc{M}~estimators are more interesting
than the monotone \stsc{M}~estimators. Unfortunately, as already explained, the
practical implementation of \stsc{M}~estimators is less easy for the
redescending than for the monotone ones.

\subsection{GM estimation}

Other approaches have been considered to limit the influence of leverage points
on the estimation of the regression coefficients. For instance, defining
$\underline{\stvec{x}}_i = (x_{i1}, \ldots, x_{ip})'$ such that
$\stvec{x}_i = (1,\underline{\stvec{x}}_i')'$, a simple way to robustify a
monotone \stsc{M}~estimate is to downweight the influential
$\underline{\stvec{x}}_i$'s to prevent them from dominating the estimating
equations. Hence we may define an estimate as solution of
\begin{equation}\label{eq:GM_equations_1}
    \sum_{i=1}^{n} \psi\left(\frac{r_i(\boldsymbol\beta)}{\sthat{\sigma}}\right)
    \widetilde{w}(d(\underline{\stvec{x}}_i)) \stvec{x}_i = \stvec{0}
\end{equation}
where $\widetilde{w}$ is a weight function and $d(\underline{\stvec{x}}_i)$
is some measure of the “largeness” of $\underline{\stvec{x}}_i$. Here $\psi$
is monotone and $\sthat{\sigma}$ is simultaneously estimated by an
\stsc{M}~estimating equation of the form
\[
    \frac{1}{n}\sum_{i=1}^{n} \rho_{\mathrm{scale}}\left(\frac{r_i(\boldsymbol\beta)}{\sigma}\right)  = \delta.
\]
In order to bound the effect of influential points, $\widetilde{w}$ must be
such that $\widetilde{w}(t)t$ is bounded.

More generally, we may let the weights depend on the residuals as well as on
the predictor variables, and use a \emph{generalized} \stsc{M}~estimate
(\stsc{GM}~estimate) $\sthat{\boldsymbol\beta}_{\stsc{GM}}$ defined as solution
of
%
\begin{equation}\label{eq:GM_equations_2}
    \sum_{i=1}^{n}\eta\left(d(\underline{\stvec{x}}_i), \frac{r_i(\boldsymbol\beta)}{\sthat{\sigma}}\right)
    \stvec{x}_i = \stvec{0}
\end{equation}
%
where for each $s$, $\eta(s,u)$ is a nondecreasing and bounded $\psi$-function
of $u$. The estimating equations system (\ref{eq:GM_equations_1}) may be seen
as a particular case of (\ref{eq:GM_equations_2}) when choosing $\eta(s,u) =
\widetilde{w}(s) \psi(u)$. This particular choice corresponds to the class of
\emph{Mallows estimates} (see \citealp{Mallows:1975}) which has been
extensively studied in the literature.

The most usual way to measure the “largeness” of $\underline{\stvec{x}}_i$,
$i=1, \dots, n$, is to take the \emph{leverage} of $\underline{\stvec{x}}_i$,
that is, to consider
%
\begin{equation}
    \label{eq:leverage}
    d(\underline{\stvec{x}}_i) =
    \sqrt{\left(\underline{\stvec{x}}_i - \sthat{\boldsymbol\mu}_{\underline{\stvec{x}}}\right)^t
    \sthat{\boldsymbol\Sigma}_{\underline{\stvec{x}}}^{-1}
    \left(\underline{\stvec{x}}_i - \sthat{\boldsymbol\mu}_{\underline{\stvec{x}}}\right)}
\end{equation}
%
where $\sthat{\boldsymbol\mu}_{\underline{\stvec{x}}}$ and
$\sthat{\boldsymbol\Sigma}_{\underline{\stvec{x}}}$ are a robust location
vector and robust dispersion matrix of the $\underline{\stvec{x}}_i$'s,
respectively (see chapter~\ref{chap:mv}). If
$\sthat{\boldsymbol\mu}_{\underline{\stvec{x}}}$ and
$\sthat{\boldsymbol\Sigma}_{\underline{\stvec{x}}}$ are the sample mean and
covariance matrix, $d(\cdot)$ is known as the Mahalanobis distance.

As stated in \citet{rousseeuw:leroy:1987}, the \stsc{GM}~estimators were
constructed in the hope of bounding the influence of a single outlying
observation. Relying on this, optimal choices of $\psi$ and $\widetilde{w}$
were made (see, among others, \citealp{Ronchetti:Rousseeuw:1985} for a survey).
However, \citet{Maronna:1979} have proven that the breakdown point of all
\stsc{GM}~estimators is non-zero but decreases as a function of $p$ (i.e., the
breakdown point is less or equal to $1/(p+1)$) pointing out that a
\stsc{GM}~estimator is interesting to be used only when the number of
explanatory variables is very small. Furthermore, \citet{maronna:etal:2006}
show that, to obtain affine equivariance of
$\sthat{\boldsymbol\beta}_{\stsc{GM}}$, it is necessary that
$\sthat{\boldsymbol\mu}_{\underline{\stvec{x}}}$ and
$\sthat{\boldsymbol\Sigma}_{\underline{\stvec{x}}}$ used in (\ref{eq:leverage})
are affine equivariant, which presents the same computational difficulties as
for redescending \stsc{M}~estimates and reduce substantially the appeal of this
estimator.

\section{Robust regression with a high breakdown point}

As explained previously, \stsc{LS}~regression is now being criticized more and
more for its dramatic lack of robustness. Indeed, one single outlier can have
an arbitrarily large effect on the estimate: the breakdown point
$\varepsilon^*$ of $\sthat{\beta}_\stsc{LS}$ is clearly equal to zero.
Although \stsc{LAD}~regression protects against outlying $y_i$, it cannot
cope with grossly aberrant values of $\stvec{x}_i$: \stsc{LAD}~regression
yields the same value $\varepsilon^*=0$ as \stsc{LS}. \stsc{M}~estimation
provides a certain robustness with respect to vertical points, but not with
respect to bad leverage points when the loss function $\rho$ is unbounded: the
breakdown point $\varepsilon^*$ associated with a monotone \stsc{M}~estimator
is then still equal to zero.

Because of this vulnerability to bad leverage points, generalized
\stsc{M}~estimators (\stsc{GM}~estimators) were introduced, with the basic purpose of
bounding the influence of outlying $\stvec{x}_i$. It turns out, however, that
the GM-estimators now in use have a breakdown point of at most $1/(p+1)$, where
$(p+1)$ is the dimension of $\stvec{x}_i$. Various other estimators have been
proposed by \citet{Theil:1950}, \citet{Brown:1951}, \citet{Sen:1968},
\citet{Jaeckel:1972}, and \citet{Andrews:1974}, but none of them achieves
$\varepsilon^*=30\%$ in the case of simple regression ($p=1$).

All of this raises the question whether robust regression with a high breakdown
point is at all possible. The affirmative answer was given by
\citet{Siegel:1982}, who proposed an estimator (the \emph{repeated median})
with a 50\% breakdown point. Note that 50\% is the best that can be expected:
for larger amounts of contamination, it becomes impossible to distinguish
between the “good” and the “bad” parts of the sample. Siegel's estimator can be
calculated explicitly but is not equivariant for linear transformations of the
$\stvec{x}_i$ (it is not affine equivariant). This explains why we do not
study this estimator in more details and prefer to present other estimators
introduced by Rousseeuw and Yohai, and all based on a robust scale measure.

\subsection{LTS and LMS estimation}

Robustness can be achieved by tackling the estimation of the regression
parameters vector $\boldsymbol\beta$ from a different perspective. We know that
\stsc{LS}~estimation is based on the minimization of the variance of the
residuals. However, since the variance is highly sensitive to outliers,
L\stsc{S}~estimate will be sensitive to them as well. An interesting idea would then
consist in minimizing a measure of the residual dispersion
$s(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta))$ that is less
sensitive to extreme residuals.

Relying on this idea, \citet{Rousseeuw:1983} introduced the \emph{Least Trimmed
Sum of Squares} (\stsc{LTS}) estimator which is based on the minimization of a
trimmed variance of the residuals:
\[
    \sthat{\boldsymbol\beta}_{\stsc{LTS}}
     = \argmin_{\boldsymbol\beta}s_{\stsc{LTS}}(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta))
\]
with
\[
    s_{\stsc{LTS}}(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta))
    = \sqrt{\frac{1}{\lceil\alpha n\rceil}
      \sum_{i=1}^{\lceil\alpha n\rceil}r_{(i)}(\boldsymbol\beta)^2}
\]
where $1/2 \leq \alpha \leq 1$ and $r_{(1)}(\boldsymbol\beta)^2 \leq \dots
\leq r_{(n)}(\boldsymbol\beta)^2$ are the ordered squared residuals. The
constant $\alpha$ determines the trade-off between the robustness and the
efficiency of the estimator. Indeed, if $\alpha$ tends to one, the
\stsc{LTS}~estimator tends to the \stsc{LS}~estimator. In contrast, if $\alpha = 1/2$, the
\stsc{LTS}~estimator will resist up to 50\% of outlying data and, consequently,
will have a breakdown point equal to 50\%. Unfortunately, even if
$\sthat{\boldsymbol\beta}_{\stsc{LTS}}$ converges to $\boldsymbol\beta$ at a
rate of $1/\sqrt{n}$, its efficiency is low (under Gaussian conditions, the
asymptotic relative efficiency of $\sthat{\boldsymbol\beta}_{\stsc{LTS}}$ with
respect to $\sthat{\boldsymbol\beta}_\stsc{LS}$ reaches only 7\% when 50\% of
the data are trimmed).

Despite its relatively low efficiency, the \stsc{LTS}~estimator is quite popular
because it can be quickly computed using the \emph{Fast-lts algorithm}
developed by \citet{rousseeuw&vdriessen99}; this estimator is available in
Stata through the command \stcmd{robreg lts}.

Following the same idea, \citet{rousseeuw:1984} introduced the \emph{Least Median
Squares} (\stsc{LMS}) estimator based on the minimization of the median of
the squared residuals:\footnote{The variance of the residuals corresponds to
the arithmetic mean of the squared residuals; why not replace the mean by the
more robust median?}
\[
    \sthat{\boldsymbol\beta}_{\stsc{LMS}}
    = \argmin_{\boldsymbol\beta} s_{\stsc{LMS}}(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta))
\]
with
\[
    s_{\stsc{LMS}}(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta))
    = \sqrt{\mathrm{med}_i\;r_i(\boldsymbol\beta)^2}.
\]
\stsc{LMS} satisfies $\varepsilon^* = 50\%$ but has unfortunately a very low
efficiency because of its $1/\sqrt[3]{n}$ convergence rate. The
\stsc{LMS}~estimator is available in Stata through the command \stcmd{robreg lms}.        \todo{Add paragraph on LQS, as this is also supported by robreg.}

\subsection{S estimation}

Following always the same principle, \citet{rousseeuw:yohai:1984} have
introduced a more general class of estimators: the regression
\stsc{S}~estimators.

In order to well understand the basic intuition behind the \stsc{S}~estimation,
let us consider once again the \stsc{LS}~estimation. For \stsc{LS}~estimation,
we actually are looking for the value of the regression coefficients vector
$\boldsymbol{\beta}$ that minimizes the variance (or standard deviation) of the
residuals $r_i(\boldsymbol\beta)$, $i = 1, \dots, n$. More formally, we have
\[
    \sthat{\boldsymbol\beta}_\stsc{LS}
    = \argmin_{\boldsymbol\beta} s_\stsc{LS}(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta))
\]
with
\[
    s_\stsc{LS}(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta))
    = \sqrt{\frac{1}{n}\sum_{i=1}^{n} r_i(\boldsymbol\beta)^2}.
\]
The dispersion measure $s_\stsc{LS}$ may be characterized as follows: given
the realizations $e_1, \dots, e_n$ of $n$ i.i.d.\ random variables whose
distribution is characterized by a mean equal to zero and a scale parameter
$\sigma$, the dispersion measure $s_\stsc{LS}(e_1, \dots, e_n)$ of the
sample is an estimate of $\sigma$ satisfying the equality
\[
    \frac{1}{n}\sum_{i=1}^{n} \left(\frac{e_i}{s_\stsc{LS}(e_1, \dots, e_n)}\right)^{\!\!2} = 1
\]
or, taking $\rho(u) = u^2$,
\[
    \frac{1}{n}\sum_{i=1}^{n} \rho\left(\frac{e_i}{s_\stsc{LS}(e_1, \dots, e_n)}\right) = 1.
\]
Moreover, if $u \sim \mathcal{N}(0,1)$, then $E(\rho(u)) = E(u^2) = 1$.

The \stsc{S}~estimation procedure proposed by \citet{rousseeuw:yohai:1984}
relies on the same philosophy as the one underlying the \stsc{LS}~estimation,
but introduces robustness by using specific robust residual dispersion measures
which correspond to \stsc{M}~estimators of the scale parameter $\sigma$. More
formally, given the realizations $e_1, \dots, e_n$ of $n$ i.i.d.\ random
variables with scale parameter $\sigma$ (and a location parameter equal to
zero), the \stsc{M}~estimate $\sthat{\sigma}_{\rho}$ of $\sigma$ is the
measure of dispersion $s_{\rho}(e_1, \dots, e_n)$ defined as the solution
of the equation
%
\begin{equation}\label{eq:M_scale_equation}
    \frac{1}{n}\sum_{i=1}^{n} \rho\left(\frac{e_i}{s_{\rho}(e_1, \dots, e_n)}\right) = \delta
\end{equation}
%
where
\begin{itemize}
    \item the function $\rho(\cdot)$ is positive, even (such that
    $\rho(0) = 0$), non decreasing for positive values and bounded;

    \item the constant $\delta$ is defined such that $\sthat{\sigma}_{\rho} =
    s_{\rho}(e_1, \dots, e_n)$ is a consistent estimate of $\sigma$ for the
    Gaussian regression model (generally $\delta$ is defined by $\delta =
    E(\rho(u))$ for $u \sim \mathcal{N}(0,1)$; the consistency parameter
    $\delta$ would therefore be nothing else than the population counterpart of
    the lefthand side of equation (\ref{eq:M_scale_equation})).
\end{itemize}

Then \citet{rousseeuw:yohai:1984} defined an \stsc{S}~estimate of
$\boldsymbol\beta$ by
\[
    \sthat{\boldsymbol\beta}_{\stsc{S};\rho}
    = \argmin_{\boldsymbol\beta}s_{\rho} (r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol{\beta}))
\]
where $s_{\rho}$ is a measure of dispersion defining a scale \stsc{M}~estimator, that is,
satisfying
%
\begin{equation}\label{eq:M_scale_equation_res}
    \frac{1}{n}\sum_{i=1}^{n} \rho\left(\frac{r_i(\boldsymbol{\beta})}%
        {s_{\rho}(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta))}\right) = \delta
    \qquad\text{for all $\boldsymbol\beta \in \mathbb{R}^{p+1}$.}
\end{equation}

One important fact is that an \stsc{S}~estimate of $\boldsymbol\beta$ is also an
\stsc{M}~estimate. More precisely, $\sthat{\boldsymbol\beta}_{\stsc{S};\rho}$
is an \stsc{M}~estimate (in the sense of \ref{eq:M_min}) in that
%
\begin{equation}\label{eq:S_Minequality_1}
    \sum_{i=1}^{n}\rho\left(\frac{r_i(\sthat{\boldsymbol\beta}_{\stsc{S};\rho})}{\sthat{\sigma}_{\rho}}\right)
    \leq \sum_{i=1}^{n}\rho\left(\frac{r_i(\widetilde{\boldsymbol\beta})}{\sthat{\sigma}_{\rho}}\right)
    \qquad
    \text{for all $\widetilde{\boldsymbol\beta} \in \mathbb{R}^{p+1}$}
\end{equation}
%
where the residuals are standardized by the same scale \stsc{M}~estimate
$\sthat{\sigma}_{\rho} =
s_{\rho}(r_1(\sthat{\boldsymbol\beta}_{\stsc{S};\rho}),\allowbreak
\dots,\allowbreak r_n(\sthat{\boldsymbol\beta}_{\stsc{S};\rho}))$ of $\sigma$
on both sides of the inequality (\ref{eq:S_Minequality_1}). Indeed,
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho}$ minimizes the residual dispersion
measure $s_{\rho}(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta))$
which satisfies (\ref{eq:M_scale_equation_res}). This means that, if we denote
$\sthat{\sigma}_{\rho} =
s_{\rho}(r_1(\sthat{\boldsymbol\beta}_{\stsc{S};\rho}), \dots,
r_n(\sthat{\boldsymbol\beta}_{\stsc{S};\rho}))$ and
$\widetilde{\sigma}_{\rho} = s_{\rho}(r_1(\widetilde{\boldsymbol\beta}),
\dots, r_n(\widetilde{\boldsymbol\beta}))$ for $\widetilde{\boldsymbol\beta}
\in \mathbb{R}^{p+1}$, we have $\sthat{\sigma}_{\rho} \leq
\widetilde{\sigma}_{\rho}$ and
\[
    \sum_{i=1}^{n} \rho\left(\frac{r_i(\sthat{\boldsymbol\beta}_{\stsc{S};\rho})}{\sthat{\sigma}_{\rho}}\right)
    = n\delta
    = \sum_{i=1}^{n}\rho\left(\frac{r_i(\widetilde{\boldsymbol\beta})}{\widetilde{\sigma}_{\rho}}\right)
\]
Then, since $\rho$ is monotone and $\sthat{\sigma}_{\rho} \leq
\widetilde{\sigma}_{\rho}$, we necessarily have
\[
    \sum_{i=1}^{n} \rho\left(\frac{r_i(\sthat{\boldsymbol\beta}_{\stsc{S};\rho})}{\sthat{\sigma}_{\rho}}\right)
    = \sum_{i=1}^{n} \rho\left(\frac{r_i(\widetilde{\boldsymbol\beta})}{\widetilde{\sigma}_{\rho}}\right)
    \leq \sum_{i=1}^{n}\rho\left(\frac{r_i(\widetilde{\boldsymbol\beta})}{\sthat{\sigma}_{\rho}}\right)
\]
which proves (\ref{eq:S_Minequality_1}).

If $\rho$ has a derivative $\psi$, it follows that
$\sthat{\boldsymbol{\beta}}_{\stsc{S};\rho}$ is also an \stsc{M}~estimate in
the sense of (\ref{eq:M_equations}), but with the condition that the scale
parameter $\sigma$ is estimated simultaneously with $\boldsymbol\beta$. More
formally, $\boldsymbol\beta$ is estimated by
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho}$ and $\sigma$ by
$\sthat{\sigma}_{\rho}=s_{\rho}(r_1(\sthat{\boldsymbol\beta}_{\stsc{S};\rho}),
 \dots, r_n(\sthat{\boldsymbol\beta}_{\stsc{S};\rho}))$, with
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho}$ and $\sthat{\sigma}_{\rho}$ such that
\[
    \begin{aligned}
        \sum_{i=1}^{n}
        \psi\left(\frac{r_i(\sthat{\boldsymbol\beta}_{\stsc{S};\rho})}{\sthat{\sigma}_{\rho}}\right) \stvec{x}_i
        & = \stvec{0}
        \\
        \frac{1}{n} \sum_{i=1}^{n}
        \rho\left(\frac{r_i(\sthat{\boldsymbol\beta}_{\stsc{S};\rho})}{\sthat{\sigma}_{\rho}}\right)
        & = \delta
    \end{aligned}
\]
Note that, taking $\rho(u) = u^2$ and $\delta=1$, we retrieve
the standard \stsc{LS}~minimization problem.

The choice of $\rho(\cdot)$ is crucial to have good robustness
properties\footnote{Note that the function $\rho$ defining the
\stsc{S}~estimator needs to be \emph{bounded} to get a positive breakdown point for the
regression estimator $\sthat{\boldsymbol\beta}_{\stsc{S};\rho}$.} and a high
Gaussian efficiency. The Tukey-Biweight function defined in
(\ref{eq:Tukey_Biweight_function}), with $\kappa = 1.547$, is a common choice.
This \stsc{S}~estimator resists to a contamination of up to 50\% of outliers
and, hence, has a breakdown point of 50\%. Unfortunately, this
\stsc{S}~estimator has a Gaussian efficiency of only 28.7\%. If $\kappa = 5.182$, the
Gaussian efficiency raises to 96.6\% but the breakdown point drops to 10\%.
Actually an \stsc{S}~estimator cannot simultaneously have a high breakdown
point and a high efficiency. In particular, \citet{Hossjer:1992} has shown that the
maximum Gaussian asymptotic efficiency of an \stsc{S}~estimator with a
breakdown point of 50\% is 33\%.

\subsection{MM estimation}
\label{subsec:MM_estimation}

We have just seen that \stsc{S}~estimation does not allow to reach jointly a
high breakdown point and a high Gaussian efficiency. How should we then
estimate the parameters of the regression model if we aim to combine high
efficiency under normal errors with a high breakdown point? Several proposals
have been made: the \stsc{MM}~estimators of \citet{yohai:1987}, the $\tau$
estimators of \citet{Yohai:1988}, the constrained \stsc{M} (\stsc{CM})
estimators of \citet{Mendes:1996}. All these estimators can have a Gaussian
asymptotic efficiency as close to 1 as desired, and simultaneously a breakdown
point of 50\%. Furthermore, \citet{Gervini:2002} proposed one estimator that
has a breakdown point of 50\% and an efficiency equal to 1.

Let us here focus our attention on the regression \stsc{MM}~estimators since
they are based on the \stsc{M} and \stsc{S}~estimation procedures studied in
the previous sections. An \stsc{MM}~estimator is defined in two successive
steps:

\begin{enumerate}
    \item Take an \stsc{S}~estimate $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$
    with high breakdown point (but possibly low Gaussian efficiency) where the scale
    measure $s_{\rho_0}$ is defined by
    \[
        \frac{1}{n} \sum_{i=1}^{n}\rho_0\left(\frac{r_i(\boldsymbol\beta)}
            {s_{\rho_0}(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta))}\right)
            = \delta
        \qquad\text{for all $\boldsymbol\beta\in\mathbb{R}^{p+1}$}
    \]
    ($s_{\rho_0}$ is associated with the function $\rho_0(\cdot)$ and the
    constant $\delta$). Let $\sthat{\sigma}_{\rho_0} =
    s_{\rho_0}(r_1(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}),\allowbreak
    \dots,\allowbreak r_n(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}))$.

    \item Take any other function $\rho(\cdot) \leq \rho_0(\cdot)$ and find
    the \stsc{MM}~estimate $\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$
    as a local minimum of
    %
    \begin{equation}\label{eq:MM_min}
        \sum_{i=1}^{n} \rho\left(\frac{r_i(\boldsymbol\beta)}{\sthat{\sigma}_{\rho_0}}\right)
    \end{equation}
    %
    such that
    %
    \begin{equation}\label{eq:MM_inequality}
        \sum_{i=1}^{n} \rho\left(\frac{r_i(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})}
            {\sthat{\sigma}_{\rho_0}}\right)
     \leq \sum_{i=1}^{n}\rho\left(\frac{r_i(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})}
         {\sthat{\sigma}_{\rho_0}}\right).
    %
    \end{equation}
    %
\end{enumerate}

The key result is given in \citet{yohai:1987}. Recall that all local minima of
(\ref{eq:MM_min}) are solutions of the estimating equations
(\ref{eq:M_equations}) with $\psi(u) = \rho'(u)$ and $\sthat{\sigma} =
\sthat{\sigma}_{\rho_0}$:
%
\begin{equation}\label{eq:MM_equations}
    \sum_{i=1}^{n} \psi\left(\frac{r_i(\boldsymbol\beta)}{\sthat{\sigma}_{\rho_0}}\right)
    \stvec{x}_i = \stvec{0}
\end{equation}
%
\citeauthor{yohai:1987} shows that if $\rho(u) \leq \rho_0(u)$ for all $u \in
\mathbb{R}$ and if (\ref{eq:MM_inequality}) is satisfied, then
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ is consistent. Moreover,
it can be shown that the \stsc{MM}~estimator
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ has the same breakdown
point than the \stsc{S}~estimator
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ of the first step, determined by
the function $\rho_0(\cdot)$. If, furthermore,
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ is any solution of
(\ref{eq:MM_equations}), then it has the same efficiency---this efficiency is
determined by the choice of the function $\rho(\cdot)$---as the global minimum
of (\ref{eq:MM_min}). In conclusion, it is not necessary to find the absolute
minimum of (\ref{eq:MM_min}) to ensure consistency, a high breakdown point and
a high efficiency.

It is common to use a Tukey-Biweight $\rho_{\kappa}^{\stsc{B}}(\cdot)$ function
for both the preliminary \stsc{S}~estimator and the final \stsc{MM}~estimator.
The tuning constant $\kappa$ can be set to 1.547 for the preliminary
\stsc{S}~estimator to guarantee a 50\% breakdown point, and it can be set to 4.685 for
the second step \stsc{MM}~estimator to guarantee a 95\% asymptotic Gaussian
efficiency of this final estimator. Note, however, that though not
breaking-down, an \stsc{MM}~estimator with a very high efficiency may have a
high bias \emph{under moderate contamination}: the larger the efficiency, the
larger the bias. It is therefore important to choose the efficiency so as to
maintain reasonable bias control. Results in Section 5.9 of
\citet{maronna:etal:2006} show that an efficiency of 0.95 yields too high a
bias, and hence it is safer to choose an efficiency of 0.85 which gives a
smaller bias while retaining a sufficiently high efficiency. We will raise once
again this problem of bias in section~\ref{subsec:Hausman}.

\subsubsection{Numerical computation of the S and MM estimate}

The numerical computation of the estimate
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ at the second step of the
procedure follows the approach described in
section~\ref{subsec:practical_implementation_Mestimate}: starting with
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$, we use the iteratively
reweighted least squares (\stsc{IRWLS}) algorithm to attain a solution of the
equation (\ref{eq:MM_equations}). It may be shown (see
\citet{maronna:etal:2006} that (\ref{eq:MM_min}) decreases at each iteration,
which insures (\ref{eq:MM_inequality}). Hence, once the initial
\stsc{S}~estimate is computed, $\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$
comes at almost no additional computational cost.

We programmed an \stsc{S} and an \stsc{MM}~estimator in Stata (with
Tukey-Biweight loss function) using the fast algorithm of
\citet{salibian:yohai:2006} for computing the \stsc{S}~estimator. Explicit
formulas for the estimators are not available and it is necessary to call on
numerical optimization to compute them. We present just below a sketch of the
fast algorithm for regression \stsc{S}~estimates we implemented in Stata.

Consider an estimate $\sthat{\boldsymbol\beta}_{\stsc{S};\rho}$ defined as
%
\begin{equation}\label{eq:S_min}
    \argmin_{\boldsymbol\beta\in\mathbb{R}^{p+1}}
    s_{\rho}(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta)).
\end{equation}
%
An approximate solution of (\ref{eq:S_min}) can be obtained by finding
$\sthat{\boldsymbol\beta}$ equal to
\[
    \argmin_{\boldsymbol\beta\in\mathcal{D}_{N}}
    s_{\rho}(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta))
\]
where
\[
    \mathcal{D}_{N} = \{\sthat{\boldsymbol\beta}_1, \dots, \sthat{\boldsymbol\beta}_{N}\}
\]
is a finite set of well selected candidates for
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho}$. One way to select these candidates
is by subsampling elementary sets among the sample $(\stvec{x}_1, y_1),
\dots, (\stvec{x}_n,y_n)$ (see \citealp{rousseeuw:1984}). More formally,
take a first random subsample of $(p+1)$ observations\footnote{Recall that
$(p+1)$ is the number of regression parameters to estimate, that is the
dimension of the regression coefficients vector $\boldsymbol\beta$ to estimate.}
\[
    (\stvec{x}_{i_1}, y_{i_1}), \dots, (\stvec{x}_{i_{(p+1)}}, y_{i_{(p+1)}});
\]
then the candidate $\sthat{\boldsymbol\beta}_1$ is obtained by fitting a
hyperplane containing these $(p+1)$ points:
\[
    \stvec{x}_{i_{j}}^t \sthat{\boldsymbol\beta}_1 = y_{i_j},\quad j = 1, \dots, p+1.
\]
Taking $N$ subsamples we obtain the $N$ candidates. Note that if a subsample
is collinear, it is replaced by another.

How large should $N$ be? We have to guarantee that $\mathcal{D}_{N}$ includes
at least one “good” candidate with high probability, say $(1 - \alpha)$ (with,
for example, $\alpha = 0.01$). A necessary condition to have a “good” candidate
is that it comes from a clean subsample, i.e., a subsample without outliers.

The probability of getting a clean subsample depends on the fraction of
outliers in the sample and on $p$. When the fraction of outliers in the sample
increases, the probability of getting a clean subsample decreases. Suppose the
sample contains a proportion $\xi$ of outliers. Then the probability of an
outlier-free subsample is $\gamma = (1-\xi)^{p+1}$, and the probability of at
least one clean subsample among the $N$ selected subsamples is equal to
$1-(1-\gamma)^N$. If we want this probability to be larger than $(1-\alpha)$,
we must have
\[
    \log\alpha \geq N\log(1-\gamma) \approx -N\gamma
\]
and hence
%
\begin{equation}\label{eq:N}
    N \geq \frac{|\log\alpha|}{|\log(1 - (1-\xi)^{p+1})|}
    \approx \frac{|\log\alpha|}{(1-\xi)^{p+1}}
\end{equation}
%
for $p$ not too small (see \citealp{Salibian-Barrera:2004}). Therefore $N$
must grow exponentially with $p$.

The following observation allows to save much computing time. Suppose we have
examined $(M-1)$ subsamples and
\[
    \sthat{\sigma}_{\rho;M-1} =
    s_{\rho} \left(r_1(\widehat{\boldsymbol\beta}_{M-1}), \dots,
    r_n(\widehat{\boldsymbol\beta}_{M-1})\right)
\]
is the current minimum of the residual dispersion measure $s_{\rho}$. Now we
draw the $M$-th subsample which yields the candidate
$\widehat{\boldsymbol\beta}_{M}$. Let us consider $\sthat{\sigma}_{\rho;M} =
s_{\rho}(r_1(\sthat{\boldsymbol\beta}_{M}),\allowbreak \dots,\allowbreak
r_n(\sthat{\boldsymbol\beta}_{M}))$. Since $\rho$ is a monotone function, the
inequality $\sthat{\sigma}_{\rho;M} < \sthat{\sigma}_{\rho;M-1}$ implies that
%
\begin{equation}\label{eq:S_algorithm}
    n\delta = \sum_{i=1}^{n}
    \rho\left(\frac{r_i(\widehat{\boldsymbol\beta}_{M})}{\sthat{\sigma}_{\rho;M}}\right)
    \geq \sum_{i=1}^{n}
    \rho\left(\frac{r_i(\sthat{\boldsymbol\beta}_{M})}{\sthat{\sigma}_{\rho;M-1}}\right).
\end{equation}
%
Consequently, if we observe that $\sum_{i=1}^{n}
\rho(r_i(\sthat{\boldsymbol\beta}_{M})/\sthat{\sigma}_{\rho;M-1}) > n\delta$,
this necessarily means that $\sthat{\sigma}_{\rho;M} \geq
\sthat{\sigma}_{\rho;M-1}$ and we may spare the effort of computing the scale
estimate $\sthat{\sigma}_{\rho;M}$ and discard
$\widehat{\boldsymbol\beta}_{M}$. Therefore $\sthat{\sigma}_{\rho}$ has to be
computed only for those subsamples that verify the inequality
(\ref{eq:S_algorithm}).

Although the $N$ given by (\ref{eq:N}) ensures that the approximation
$\sthat{\boldsymbol\beta}$ of $\sthat{\boldsymbol\beta}_{\stsc{S};\rho}$
has the desired breakdown point, it does not imply that it is a good
approximation to the exact \stsc{S}~estimate. To solve this problem,
\citet{salibian:yohai:2006} have proposed a procedure based on a “local
improvement” step of the resampling initial candidates. This
allows for a substantial reduction of the number of candidates required to
obtain a good approximation to the optimal solution.

This algorithm can be called in Stata either directly using the \stcmd{robreg
s} function\footnote{The default values that are used in Stata for the
implementation of the fast \stsc{S}~algorithm are $\xi = 0.2$ and $\alpha =
0.01$.} or indirectly using the \stcmd{robreg mm} function developed to compute
\stsc{MM}~estimate, and invoking the \stcmd{initial} option. Once the
\stsc{S}~estimate is obtained, the \stsc{MM}~estimate directly follows by applying the
iteratively reweighted least squares algorithm up to convergence. As far as
inference is concerned, standard errors robust to heteroskedasticity (and
asymmetric errors) are computed according to the formulas available in the
literature (see Section \ref{sec:inference}).

\subsection{MS estimation}

Explicit formulas for $\sthat{\boldsymbol\beta}_{\stsc{S};\rho}$ are generally
not available and, as explained in the previous section, empirical
implementation of \stsc{S}~estimation requires numerical optimization based on
a subsampling algorithm. But this method presents an Achille's heel: it becomes
inapplicable in practice when several \emph{dummy} explanatory variables are
involved in the regression model (\ref{eq:linear_regr_model}). Indeed, when
several of the explanatory variables are binary, there is a high probability
that random selection of subsamples yields collinear subsamples.

To cope with this, \citet{maronna:yohai:2000} have introduced the \stsc{MS}~estimator.
The intuition behind this estimator is simple. For the sake of clarity, let us
separate continuous and dichotomous variables in (\ref{eq:linear_regr_model})
and rewrite the regression model equation as follows:
%
\begin{equation}\label{eq:linear_regr_model_MS}
    y = (\beta_0 + \beta_1x_1 + \dots + \beta_{p_1}x_{p_1})
    + (\beta_1^*x_1^* + \dots + \beta_{p_2}^*x_{p_2}^*) + \varepsilon
\end{equation}
%
where $x_1, \dots, x_{p_1}$ are $p_1$ continuous explanatory variables and
$x_1^*, \dots, x_{p_2}^*$ are $p_2$ dichotomous explanatory variables ($p = p_1
+ p_2$). If $\boldsymbol\beta = (\beta_0, \beta_1, \dots, \beta_{p_1})^t$ was
known in equation (\ref{eq:linear_regr_model_MS}), then $\boldsymbol\beta^*% =
(\beta_1^*, \dots, \beta_{p_2}^*)^t$ would be robustly estimated using a
monotone \stsc{M}~estimator (since $x_1^*, \dots, x_{p_2}^*$ are all dummy
variables, the data set can only contain, at worst, vertical outliers). On the
other hand, if $\boldsymbol\beta^*$ was known, then $\boldsymbol\beta$ should
be estimated using an \stsc{S}~estimator\footnote{Since $x_1, \dots, x_{p_1}$
are continuous explanatory variables, we cannot assume that there are no
leverage points.} and the subsampling algorithm should not generate collinear
subsamples since all explanatory variables are continuous. The idea is then to
alternate these two estimators till convergence.

Technically speaking, an \stsc{MS}~regression estimate is obtained iteratively;
at the $k$-th step, we define $\sthat{\boldsymbol\beta}_{\stsc{MS}}^{(k)}$ and
$\sthat{\boldsymbol\beta}_{\stsc{MS}}^{*(k)}$ as follows. Let $s_\rho$ be a
measure of dispersion satisfying (\ref{eq:M_scale_equation_res}), $\stvec{x}_i
= (1, x_{i1}, \dots, x_{ip_1})^t$ and $\stvec{x}_i^* = (x_{i1}^*, \dots,
x_{ip_2}^*)^t$:
\[
    \begin{aligned}
        \sthat{\boldsymbol\beta}_{\stsc{MS}}^{(k)}
        & = \argmin_{\boldsymbol\beta \in \mathbb{R}^{p_1 + 1}}
        s_\rho\left(\left[y_i - (\stvec{x}_i^*)^t \sthat{\boldsymbol\beta}_{\stsc{MS}}^{*(k-1)}\right]
        - \stvec{x}_i^t \boldsymbol\beta; i = 1, \dots, n\right)
        \\
        \sthat{\boldsymbol\beta}_{\stsc{MS}}^{*(k)}
        & = \argmin_{\boldsymbol\beta^* \in \mathbb{R}^{p_2}}
        \sum_{i=1}^{n} \rho\left(\frac{\left[y_i - \stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{MS}}^{(k-1)}\right]
        - \left(\stvec{x}_i^*\right)^t \boldsymbol\beta^*}{\sthat{\sigma}^{(k-1)}}\right)
    \end{aligned}
\]
where
\[
    \sthat{\sigma}^{(k-1)} = s_\rho
    \left(y_i - \stvec{x}_i^t \sthat{\boldsymbol\beta}_{\stsc{MS}}^{(k-1)}
    - (\stvec{x}_i^*)^t \sthat{\boldsymbol\beta}_{\stsc{MS}}^{*(k-1)}
    ; i = 1, \dots, n\right).
\]
Note that \stcmd{robreg s} and \stcmd{robreg mm} automatically recognize the
presence of dummy variables among the explanatory variables and, if
appropriate, automatically apply the \stsc{MS}~procedure.

Unfortunately, as stated above, the price to pay for robustness is efficiency.
However this \stsc{MS}~estimator can be particularly helpful in the fixed
effects panel data models, as suggested by \citet{bramati:croux:2007}.

\section{Robust inference for M, S and MM estimators}
\label{sec:inference}

Consistency and asymptotic normality of \stsc{M}~estimators under the
assumption of i.i.d.\ error terms have been studied by \citet{Yohai:1979} and
for \stsc{MM}~estimators by \citet{yohai:1987}. Under fairly general
conditions, allowing also for heteroskedasticity, asymptotic normality for
\stsc{S} and \stsc{MM}~estimators has been shown by
\citet{Salibian-Barrera:2004} in the location case. Some of these results are
summarized in \citet{maronna:etal:2006} with a distinction made between the
case of \emph{fixed} predictors and the case of \emph{random} predictors.

\citet{Croux:2003} have established the asymptotic normality of \stsc{M},
\stsc{S} and \stsc{MM}~estimators in the regression case under quite general
conditions: they only assume that the observations $(\stvec{x}_1, y_1), \dots,
(\stvec{x}_n, y_n)$ are generated by a \emph{stationary} and \emph{ergodic}
process $H$.\footnote{A \emph{stationary} process is a stochastic process whose
joint probability distribution does not change when shifted in time or space.
Consequently, parameters such as the mean and the variance, if they exist, also
do not change over time or position. Hence, the mean and the variance of the
process do not follow trends. Furthermore, a stochastic process is said to be
\emph{ergodic} if its statistical properties (such as its mean and variance)
can be estimated consistently from a single, sufficiently long sample
(realization) of the process.} Under this assumption, the observations do not
need to be independent, we may have heteroskedasticity (the processes
$\stvec{x}_i$ and $\varepsilon_i$ are not necessarily independent) and the
distribution of the error terms is not necessarily symmetric. In this context,
the authors of \citet{Croux:2003} have showed that the \stsc{M}, \stsc{S} and
\stsc{MM}~estimators of the regression parameters $\boldsymbol\beta$ and of the
scale parameter $\sigma$ are first-order equivalent with exactly-identified \stsc{GMM}
(Generalized Method of Moments) estimators and have then deduced the asymptotic
variance matrix of the \stsc{M}, \stsc{S} and \stsc{MM}~estimators of
$\boldsymbol\beta$ from results established for \stsc{GMM} (see
\citealp{Hansen:1982}). The interest of the results of \citet{Croux:2003} is
multiple. They propose explicit formulas for the asymptotic variance matrices
of the robust regression estimators, so recourse to bootstrap techniques is not
necessary. Moreover, these variances are valid in the presence of
autocorrelation and heteroskedasticity; as we will show it, if we impose the
independence between the observations, the absence of heteroskedasticity or the
symmetry of the distribution of the error terms, the expressions of the
variances become much simpler and coincide with the results previously proved
by other authors. The robustness with respect to outliers of the estimates of
the variance matrices is also taken into account. Finally, the results of
\citet{Croux:2003} may be used to develop robust confidence intervals and
robust tests for the regression parameters; they are also on the basis of the
extension of the Hausman test presented at the end of this section, which
allows to check for the presence of outliers---by comparing the regression
coefficients estimated by least squares and by a robust
\stsc{S}~procedure---and to fix the maximal efficiency that may have an
\stsc{MM}~estimator without suffering of significant bias in the presence of
contamination of the data set by (moderately) bad leverage points---by
comparing an \stsc{S}~estimate of $\boldsymbol\beta$ with several
\stsc{MM}~estimates of different efficiencies.

\subsection{Asymptotic distribution of M, S and MM estimators}
\label{subsec:asymptotic_distr_M_S_MM_estimators}

Let us here present some of the fundamental results established by
\citet{Croux:2003} for the asymptotic distribution of \stsc{M}, \stsc{S} and
\stsc{MM}~estimators. The interested reader will find some details about the
main steps of the approach used to demonstrate these results in Appendix~2
(Section~\ref{sec:robreg:appendix2}) at the end of this chapter.

Let $y$ be the scalar dependent variable and $\stvec{x} = (1, x_1, \dots,
x_p)^t$ be the $(p+1)$-vector of covariates. Consider once again the regression
model (\ref{eq:location_scale_regr_model}). Here, the observations
$(\stvec{x}_1, y_1), \dots, (\stvec{x}_n, y_n)$ are assumed to be generated by
a \emph{stationary} and \emph{ergodic} process. To avoid too much
technicalities, we also assume that the observations $(\stvec{x}_i, y_i)$, $i =
1, \dots, n$, are \emph{independent}.\footnote{The interested reader can find
very general results, valid in presence of \emph{autocorrelation}, in
\citet{Croux:2003}.}

Let us denote by $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ the
\stsc{S}~estimator of $\boldsymbol\beta$ associated with the loss function
$\rho_0$:
\[
    \sthat{\boldsymbol\beta}_{\stsc{S};\rho_0} =
    \argmin_{\boldsymbol{\beta}} s_{\rho_0}(r_1(\boldsymbol\beta), \dots,
    r_n(\boldsymbol\beta))
\]
where $s_{\rho_0}$ is a measure of dispersion satisfying
\[
    \frac{1}{n} \sum_{i=1}^{n}
    \rho_0\left(\frac{r_i(\boldsymbol\beta)}{s_{\rho_0}(r_1(\boldsymbol\beta),
        \dots, r_n(\boldsymbol\beta))}\right)
    = \delta\quad\text{for all $\boldsymbol\beta \in \mathbb{R}^{p+1}$.}
\]
This leads to the scale \stsc{M}~estimator
\[
    \sthat{\sigma}_{\rho_0} =
    s_{\rho_0} \left(r_1(\widehat{\boldsymbol\beta}_{\stsc{S};\rho_0}), \dots,
    r_n(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})\right).
\]

Let $\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ be the
\stsc{MM}~estimator of $\boldsymbol\beta$ associated with the loss function $\rho_0$ for
the first step of the estimation procedure (\stsc{S}~estimation) and with the
loss function $\rho$ for the second step (\stsc{M}~estimation):
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ is a (local) minimum of
\[
    \sum_{i=1}^{n} \rho\left(\frac{r_i(\boldsymbol\beta)}{\sthat{\sigma}_{\rho_0}}\right).
\]

To avoid any ambiguity in the formulation of the results, we will denote the
vector of regression parameters by $\boldsymbol\beta$ when it is estimated by
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ and by $\boldsymbol\beta_0$
when it is estimated by $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$. Moreover,
we will use the generic notations $u_0 =
(y-\stvec{x}^t\boldsymbol\beta_0)/\sigma$ and $u =
(y-\stvec{x}^t\boldsymbol\beta)/\sigma$, and we will simply replace $\psi(u) =
\rho'(u)$ by $\psi$, and $\rho_0(u_0)$ by $\rho_0$.

Using these notations, we may formulate the results shown by \citet{Croux:2003}
as follows.

\begin{stproposition}
If the observations $(\stvec{x}_i, y_i)$, $i = 1, \dots, n$, are generated
by a stationary and ergodic process, and are independent (Assumption A), then
\[
    \sqrt{n} \left(
    \begin{bmatrix}
        \sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}\\
        \sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}      \\
        \sthat{\sigma}_{\rho_0}
    \end{bmatrix}
    -
    \begin{bmatrix}
        \boldsymbol\beta\\
        \boldsymbol\beta_0\\
        \sigma
    \end{bmatrix}
    \right) \rightarrow^d \mathcal{N}(\stvec{0},\stmat{V}_\stsc{MM})
\]
where
%
\begin{equation}\label{eq:V_MM}
    \stmat{V}_\stsc{MM} = \stmat{G}_\stsc{MM}^{-1}
    \boldsymbol\Omega_\stsc{MM} \left(\stmat{G}_\stsc{MM}^t\right)^{-1}
%
\end{equation}
%
with the matrices $\stmat{G}_\stsc{MM}$ and $\boldsymbol\Omega_\stsc{MM}$ given
by:
%
\begin{equation}\label{eq:G_MM}
    \stmat{G}_\stsc{MM} = - \frac{1}{\sigma}
    E\begin{pmatrix}
        \psi'\stvec{x}\stvec{x}^t & \stvec{0}                    & \psi'u\stvec{x}\\
        \stvec{0}                 & \rho_0''\stvec{x}\stvec{x}^t & \rho_0''u_0\stvec{x}\\
        \stvec{0}                 & \stvec{0}                    & \rho_0'u_0
    \end{pmatrix}
\end{equation}
%
and
%
\begin{equation}\label{eq:Omega_MM}
    \boldsymbol\Omega_\stsc{MM} =
    E\begin{pmatrix}
        \psi^2\stvec{x}\stvec{x}^t      & \psi\rho_0'\stvec{x}\stvec{x}^t & \psi\rho_0\stvec{x} \\
        \psi\rho_0'\stvec{x}\stvec{x}^t & (\rho_0')^2\stvec{x}\stvec{x}^t & \rho_0\rho_0'\stvec{x}\\
        \psi\rho_0\stvec{x}^t           & \rho_0\rho_0'\stvec{x}^t        & \rho_0^2-\delta^2
    \end{pmatrix}.
\end{equation}
\end{stproposition}

In particular, this result establishes the consistency of the regression
\stsc{MM}~estimator $\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ and
\stsc{S}~estimator $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$, and of the
scale \stsc{M}~estimator $\sthat{\sigma}_{\rho_0}$.

Moreover, it allows to derive explicit formulas for the asymptotic variances
of $\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ and
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$---denoted hereafter by
$\mathrm{Avar}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$ and
$\mathrm{Avar}(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})$, respectively---, and for the asymptotic
covariance of $\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ and
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$---denoted by
$\mathrm{Acov}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho},
\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})$:
%
\begin{align}
    \label{eq:Avar_betahat_MM}
    \mathrm{Avar}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})
    & = \frac{1}{n} \big[ \stmat{A} E(\psi^2\stvec{x}\stvec{x}^t)\stmat{A}
        - \stmat{a} E(\psi\rho_0\stvec{x}^t) \stmat{A}
    \\\nonumber & \qquad
        - \stmat{A} E(\psi\rho_0\stvec{x}) \stmat{a}^t
        + E(\rho_0^2 - \delta^2)  \stmat{a}\stmat{a}^t \big]
    \\
    \label{eq:Avar_betahat_S}
    \mathrm{Avar}(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})
    & = \frac{1}{n} \big[ \stmat{A}_\stsc{S} E((\rho_0')^2 \stvec{x}\stvec{x}^t) \stmat{A}_\stsc{S}
        - \stmat{a}_\stsc{S} E(\rho_0\rho_0'\stvec{x}^t) \stmat{A}_\stsc{S}
    \\\nonumber & \qquad
        - \stmat{A}_\stsc{S} E(\rho_0\rho_0'\stvec{x}) \stmat{a}_\stsc{S}^t
        + E(\rho_0^2 - \delta^2) \stmat{a}_\stsc{S}\stmat{a}_\stsc{S}^t \big]
    \\
    \label{eq:Acov_betahat_MM_S}
    \mathrm{Acov}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho},
        \sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})
    & = \frac{1}{n} \big[ \stmat{A} E(\psi\rho_0'\stvec{x}\stvec{x}^t) \stmat{A}_\stsc{S}
        - \stmat{a} E(\rho_0\rho_0'\stvec{x}^t) \stmat{A}_\stsc{S}
    \\\nonumber & \qquad
        - \stmat{A} E(\psi\rho_0\stvec{x}) \stmat{a}_\stsc{S}^t
        + E(\rho_0^2 - \delta^2) \stmat{a}\stmat{a}_\stsc{S}^t \big]
\end{align}
%
with
%
\begin{align}
    \label{eq:A}
    \stmat{A} &  = \sigma \left[ E(\psi'\stvec{x}\stvec{x}^t)\right]^{-1}
    \\
    \label{eq:a}
    \stmat{a} &  = \stmat{A} \frac{E(\psi'u\stvec{x})}{E(\rho_0'u_0)}
    \\
    \label{eq:A_S}
    \stmat{A}_\stsc{S} & = \sigma\left[E(\rho_0''\stvec{x}\stvec{x}^t)\right]^{-1}
    \\
    \label{eq:a_S}
    \stmat{a}_\stsc{S} & = \stmat{A}_\stsc{S}\frac{E(\rho_0''u_0\stvec{x})}{E(\rho_0'u_0)}.
\end{align}

\begin{stremark}
Note that \citet{Croux:2003} have also considered the case where we estimate
the parameters $\boldsymbol\beta$ and $\sigma$ simultaneously by an
\stsc{M}~estimation procedure. Some results about the asymptotic distribution of
$(\sthat{\boldsymbol\beta}_{\stsc{M};\rho}^t, \sthat{\sigma}_{\rho_0})^t$ are
presented in Appendix~2 (Section~\ref{sec:robreg:appendix2}) at the end of this
chapter.
\end{stremark}

The authors have also shown that the asymptotic variances and covariances can
be estimated consistently by taking their empirical counterpart. More
precisely, the estimates are obtained by applying the following two rules:
\begin{enumerate}
    \item Replace, in $u$ and $u_0$, the parameters $\boldsymbol\beta$,
    $\boldsymbol\beta_0$ and $\sigma$ by the estimates
    $\widehat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$,
    $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ and $\sthat{\sigma}_{\rho_0}$.

    \item Replace $E(\cdot)$ by $\frac{1}{n}\sum_{i=1}^{n}(\cdot)$.
\end{enumerate}
For example, the first term of
$\sthat{\mathrm{Avar}}(\widehat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$ is
given by
\[
    \frac{1}{n} \left(\sthat{\stmat{A}}\left[
    \frac{1}{n} \sum_{i=1}^{n} \psi\left(
    \frac{y_i - \stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}}
    {\sthat{\sigma}_{\rho_0}}\right)^{\!\!2} \stvec{x}_i\stvec{x}_i^t\right]
    \widehat{\stmat{A}}\right)
\]
with
\[
    \sthat{\stmat{A}} = \sthat{\sigma}_{\rho_0}
    \left[\frac{1}{n} \sum_{i=1}^{n}
    \psi'\left(\frac{y_i-\stvec{x}_i^t\widehat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}}
    {\sthat{\sigma}_{\rho_0}}\right) \stvec{x}_i\stvec{x}_i^t\right]^{-1}.
\]
It is interesting to note that the estimate $\sthat{\mathrm{Avar}}
(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$ of the asymptotic variance
$\mathrm{Avar}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$ is robust
with respect to bad leverage points and vertical outliers. Indeed, if there are
observations yielding large residuals with respect to the robust \stsc{MM}~fit,
then
$\psi((y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})/\sthat{\sigma}_{\rho_0})$
has a small value when $\psi$ is a redescending function.\footnote{Recall that,
if $\psi$ is redescending, it has the property to be equal to zero for large
arguments.} Hence, if there are bad leverage points in the sample, then their
$\stvec{x}_i$-value is large, but at the same time
$\psi((y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})/\sthat{\sigma}_{\rho_0})$
will be zero. This explains intuitively why vertical outliers and bad leverage
points have only a limited influence on the estimate
$\sthat{\mathrm{Avar}}(\widehat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$.

\begin{stremark}
As previously explained, the \stsc{LS}~estimator
$\sthat{\boldsymbol\beta}_\stsc{LS}$ may be seen as a particular
\stsc{S}~estimator of $\boldsymbol{\beta}$ associated with the loss function
$\rho_0(u_0) = u_0^2$ (such that $\rho_0'(u_0) = 2 u_0$ and $\rho_0''(u_0) =
2$) and with the constant $\delta = 1$. The expression of the asymptotic
variance matrix of $\widehat{\boldsymbol\beta}_\stsc{LS}$ may then be simply
derived from the one obtained for
$\mathrm{Avar}(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})$:
%
\begin{align}
    \label{eq:Avar_betahat_LS}
    \mathrm{Avar}(\sthat{\boldsymbol\beta}_\stsc{LS})
    & = \frac{1}{n} \big[ \stmat{A}_\stsc{LS} E(4 u_0^2\stvec{x}\stvec{x}^t) \stmat{A}_\stsc{LS}
        -\stmat{a}_\stsc{LS} E(2 u_0^3 \stvec{x}^t) \stmat{A}_\stsc{LS}
    \\\nonumber & \qquad
        - \stmat{A}_\stsc{LS} E(2 u_0^3 \stvec{x}) \stmat{a}_\stsc{LS}^t
        + E(u_0^4 - 1) \stmat{a}_\stsc{LS}\stmat{a}_\stsc{LS}^t \big]
\end{align}
%
with
%
\begin{equation}\label{eq:A_a_LS}
    \stmat{A}_\stsc{LS} = \frac{\sigma}{2} E(\stvec{x}\stvec{x}^t)^{-1}
    \qquad\text{and}\qquad
    \stmat{a}_\stsc{LS} = \stmat{A}_\stsc{LS} \frac{E(u_0\stvec{x})}{E(u_0^2)}.
\end{equation}
%
If, in addition, there is homoskedasticity\footnote{There is
\emph{homoskedasticity} when the processes $\stvec{x}_i$ and $(u_i, u_{0i})$
are independent.} and if the distribution $F_{0,1}$ of the error terms is
symmetric (around 0), we retrieve the well-known asymptotic variance matrix
\[
    \frac{\sigma^2}{n} E(\stvec{x}\stvec{x}^t)^{-1}
\]
that we may estimate by
\[
    \frac{\sthat{\sigma}_{\rho_0}^2}{n}
    \left(\frac{1}{n} \sum_{i=1}^{n} \stvec{x}_i\stvec{x}_i^t\right)^{\!\!-1}.
\]
Note that this latter estimator is absolutely not robust with respect to
leverage points.

Finally, since the \stsc{LS}~estimation can be considered as the special case of the
\stsc{MM}~estimation associated with $\rho(u) = u^2$, it can be shown that:
%
\begin{equation}
    \label{eq:Acov_betahat_LS_S}
    \begin{split}
    \mathrm{Acov}(\sthat{\boldsymbol\beta}_\stsc{LS},\widehat{\boldsymbol\beta}_{\stsc{S};\rho_0})
    & = \frac{1}{n} \big[ \stmat{A} E(2 u \rho_0'\stvec{x}\stvec{x}^t) \stmat{A}_\stsc{S}
        - \stmat{a} E(\rho_0\rho_0'\stvec{x}^t) \stmat{A}_\stsc{S}
    \\
    & \qquad
        - \stmat{A} E(2 u \rho_0\stvec{x}) \stmat{a}_\stsc{S}^t
        + E(\rho_0^2 - \delta^2) \stmat{a}\stmat{a}_\stsc{S}^t \big]
    \end{split}
\end{equation}
%
with%
\[
    \stmat{A} = \stmat{A}_\stsc{LS} = \frac{\sigma}{2} E(\stvec{x}\stvec{x}^t)^{-1}
    \qquad\text{and}\qquad
    \stmat{a} = \stmat{A}_\stsc{LS} \frac{E(2 u \stvec{x})}{E(\rho_0'u_0)}
\]
while $\stmat{A}_\stsc{S}$ and $\stmat{a}_\stsc{S}$ remain unchanged with
respect to (\ref{eq:A_S}) and (\ref{eq:a_S}).
\end{stremark}

Of course, in absence of heteroskedasticity or if the distribution $F_{0,1}$ of
the error terms is symmetric (around 0), the expressions of the asymptotic
variances and covariances simplify quite considerably, as shown in Appendix~2
(Section~\ref{sec:robreg:appendix2}). Unfortunately, their estimates---their
empirical counterparts---are not robust anymore with respect to (good and bad)
leverage points. Hence, \citet{Croux:2003} do advise against the use of these
simplified variances and covariances, even when the assumptions of absence of
heteroskedasticity and symmetry hold.

\subsection{Robust confidence intervals and tests with robust regression estimators}

As just explained, we may consider that, under the model
(\ref{eq:location_scale_regr_model}) and Assumption A,\footnote{Recall that
Assumption A specifies that the observations $(\stvec{x}_i, y_i)$, $i = 1,
\dots, n$, are generated by a stationary and ergodic process, and are mutually
independent.} a robust \stsc{M}, \stsc{S} or \stsc{MM}~estimator
$\sthat{\boldsymbol\beta}$ is, for large $n$, approximately normally
distributed with mean $\boldsymbol\beta$ and variance
$\widehat{\mathrm{Avar}}(\sthat{\boldsymbol\beta})$, where
$\sthat{\mathrm{Avar}}(\sthat{\boldsymbol\beta})$ corresponds to the empirical
counterpart of the asymptotic matrix $\mathrm{Avar}(\sthat{\boldsymbol\beta})$
specified in the previous subsection. This result underlies the inference
procedures developed for linear combinations of the regression parameters.

\subsubsection{Inference for a single linear combination of the regression parameters}

Let $\gamma$ be a linear combination of the regression coefficients:
\[
    \gamma = \stvec{b}^t \boldsymbol\beta
\]
with $\stvec{b}$ a constant (non random) vector. Then the natural estimate of
$\gamma$ is $\sthat{\gamma} = \stvec{b}^t\sthat{\boldsymbol\beta}$, which is,
under Assumption A and for large $n$, approximately $\mathcal{N}(\gamma,
\sthat{\sigma}_\gamma^2)$ where
\[
    \sthat{\sigma}_\gamma^2 =
    \stvec{b}^t \sthat{\mathrm{Avar}}(\sthat{\boldsymbol\beta}) \stvec{b}.
\]
Hence an approximate two-sided confidence interval for $\gamma$ with
confidence level $(1- \alpha)$ is given by
\[
    \left[\sthat{\gamma} \pm z_{1-\alpha/2} \sthat{\sigma}_\gamma\right]
\]
where $z_{1-\alpha/2}$ is the $(1-\alpha/2)$-quantile of the standard normal
distribution.

Similarly, the test of level $\alpha$ for the null hypothesis $\mathcal{H}_0:
\gamma = \gamma_0$ against the two-sided alternative $\mathcal{H}_1: \gamma
\neq \gamma_0$ has the rejection region
\[
    |\sthat{\gamma}-\gamma_0| > z_{1-\alpha/2}\sthat{\sigma}_\gamma
\]
or equivalently, since the approximate normal distribution of
$\widehat{\gamma}$ implies that
$((\sthat{\gamma}-\gamma)/\widehat{\sigma}_\gamma)^2 \approx \chi_1^2$, rejects
$\mathcal{H}_0$ when
\[
    T > \chi_{1; 1-\alpha}^2
\]
where
\[
    T = \left(\frac{\sthat{\gamma}-\gamma_0}{\sthat{\sigma}_\gamma}\right)^{\!2}
\]
and $\chi_{1;1-\alpha}^2$ is the $(1-\alpha)$-quantile of the chi-square
distribution with one degree of freedom.

In particular, if $\mathbf{b} = (0, \dots, 0, 1, 0, \dots, 0)^t$, that is, if
all the components of $\mathbf{b}$ are equal to zero except the $j$th component
equal to 1, we have $\gamma = \beta_j$ and $\sthat{\sigma}_\gamma^2 =
[\sthat{\mathrm{Avar}}(\sthat{\boldsymbol\beta})]_{jj}$. Then, the two-sided
confidence interval for $\beta_j$ with confidence level $(1-\alpha)$ is given by
\[
    \left[\sthat{\beta}_j \pm
    z_{1-\alpha/2} \sqrt{[\sthat{\mathrm{Avar}}(\sthat{\boldsymbol\beta})]_{jj}}\right]
\]
and the test of level $\alpha$ for the null hypothesis $\mathcal{H}_0: \beta_j =
0$ against the alternative $\mathcal{H}_1: \beta_j \neq 0$ has the rejection
region
\[
    \frac{\sthat{\beta}_j^2}{[\sthat{\mathrm{Avar}}(\sthat{\boldsymbol\beta})]_{jj}}
    > \chi_{1;1-\alpha}^2.
\]

\subsubsection{Inference for several linear combinations of the regression parameters}

Let us now consider several linear combinations of the $\beta_{j}$'s
represented by the vector $\boldsymbol{\gamma} = \stmat{B}\boldsymbol\beta$
where $\stmat{B}$ is a $q \times (p+1)$ matrix of rank $q$. Then
$\widehat{\boldsymbol{\gamma}} = \stmat{B}\sthat{\boldsymbol\beta}$ is, under
Assumption A and for large $n$, approximately
$\mathcal{N}_q(\boldsymbol{\gamma},
\sthat{\boldsymbol\Sigma}_{\boldsymbol\gamma})$ with
\[
    \sthat{\boldsymbol\Sigma}_{\boldsymbol\gamma}
    = \stmat{B}\widehat{\mathrm{Avar}}(\sthat{\boldsymbol\beta})\stmat{B}^t.
\]
This implies that
\[
    (\sthat{\boldsymbol\gamma} - \boldsymbol\gamma)^t
    \sthat{\boldsymbol\Sigma}_{\boldsymbol\gamma}^{-1}
    (\widehat{\boldsymbol\gamma} - \boldsymbol\gamma)  \approx \chi_q^2
\]
where $\chi_q^2$ is the chi-square distribution with $q$ degrees of freedom.
Hence, to test the linear hypothesis $\mathcal{H}_0: \boldsymbol\gamma =
\boldsymbol\gamma_0$ for a given $\boldsymbol\gamma_0$, with level $\alpha$, we
may use the test that rejects $\mathcal{H}_0$ if
\[
    T > \chi_{q;1-\alpha}^2
\]
where
\[
    T = (\sthat{\boldsymbol\gamma} - \boldsymbol\gamma_0)^t
    \sthat{\boldsymbol\Sigma}_{\boldsymbol\gamma}^{-1}
    (\sthat{\boldsymbol\gamma} - \boldsymbol\gamma_0)
\]
and $\chi_{q;1-\alpha}^2$ is the $(1-\alpha)$-quantile of the $\chi_q^2$
distribution. The most common application of this test is when $\mathcal{H}_0$
is the hypothesis that some of the coefficients $\beta_j$ are equal to zero.
If, for example, the null hypothesis is
\[
    \mathcal{H}_0: \beta_1 = \beta_2 = \dots = \beta_q = 0
\]
then $\boldsymbol\gamma = \stmat{B}\boldsymbol\beta$ with $\stmat{B} =
(\stmat{I}_{q\times q}, \stmat{0}_{q\times(p+1-q)})$, where $\stmat{I}_{q\times
q}$ is the $(q\times q)$ identity matrix, and $\mathcal{H}_0$ takes the form
$\mathcal{H}_0: \stmat{B}\boldsymbol\beta = 0$.

\subsection{Robust R-squared}

The coefficient of determination or $R^2$ is a very simple tool---probably
the one most used by practitioners---to assess the quality of fit in a multiple
linear regression. It provides an indication of the suitability of the chosen
explanatory variables in predicting the response. In the classical setting,
$R^2$ is usually presented as the quantity that estimates the percentage of
variance of the response variable explained by its (linear) relationship with
the explanatory variables. It is defined as the ratio
%
\begin{equation}
    \label{eq:R2_classic_SS}
    R^2 = \frac{\stsc{ESS}}{\stsc{TSS}} = 1-\frac{\stsc{RSS}}{\stsc{TSS}}
          = 1 - \frac{\sum_{i=1}^{n} (y_i - \sthat{y}_i)^2}
                {\sum_{i=1}^{n} (y_i - \overline{y})^2}
\end{equation}
%
where $stsc{ESS}$, $stsc{TSS}$ and $stsc{RSS}$ are the explained, total, and
residual sum of squares, respectively. Note that $y_i - \sthat{y}_i =
r_i(\sthat{\boldsymbol\beta}_\stsc{LS})$ are the \stsc{LS}~residuals. Moreover,
$\overline{y}$ is the \stsc{LS}~estimate of $\mu = E(y)$, that is the
\stsc{LS}~estimate of the intercept $\beta_0$ in the linear regression model
(\ref{eq:linear_regr_model}) in which $\beta_1 = \dots = \beta_p = 0$.

When there is an intercept term in the linear model, this coefficient of
determination $R^2$ is actually equal to the square of the correlation
coefficient between the observed $y_i$'s and the predicted $\sthat{y}_i$'s
(see, e.g., \citealp{Greene:1997}), that is,
%
\begin{equation}
    \label{eq:R2_classic_corr}
    R^2 = \left(\frac{\sum_{i=1}^{n} (y_i - \overline{y})(\sthat{y}_i - \overline{\sthat{y}})}
          {\sqrt{\sum_{i=1}^{n} (y_i - \overline{y})^2}
           \sqrt{\sum_{i=1}^{n} (\sthat{y}_i - \overline{\sthat{y}})^2}}\right)^{\!\!2}
\end{equation}
%
with $\overline{\sthat{y}}$ as the arithmetic mean of the predicted responses.
Equation (\ref{eq:R2_classic_corr}) has a nice interpretation in that $R^2$
measures the goodness of fit of the regression model by its ability to predict
the response variable, ability measured by the correlation. Note that $R^2$ is
a consistent estimator of the population parameter
%
\begin{equation}
    \label{eq:phi2}
    \phi^2 = \max_{\boldsymbol\beta} \mathrm{Corr}(y, \stvec{x}^t\boldsymbol\beta)^2
\end{equation}
%
that is, of the squared correlation between $y$ and the best linear
combination of the $\stvec{x}$ (cf.\ \citealp{Anderson:1984}). In finite
samples, $R^2$ is biased upward and is generally adjusted as in
%
\begin{equation}
    \label{eq:R2_adj}
    R_{\mathrm{adj}}^2 = 1 - \left(1-R^2\right) \left(\frac{n-1}{n-(p+1)}\right).
\end{equation}                                                              \todo{maybe also give other formula with residual variance}

It is rather obvious that the $R^2$ given by (\ref{eq:R2_classic_SS}) can be
driven by extreme observations, not only through the \stsc{LS}~estimator
$\sthat{\boldsymbol\beta}_\stsc{LS}$ used to compute the predicted responses
$\sthat{y}_i$, but also through the average response $\overline{y}$ and the
possible large residuals $y_i - \sthat{y}_i$ or deviations $y_i -
\overline{y}$. Several robust $R^2$ variants have then been proposed in the
literature (see \citealp{Renaud:VictoriaFeser:2010}). A \emph{robust} $R^2$ should give an
indication of the fit for the \emph{majority} of the data, possibly leaving
aside a few outlying observations. In other words, the (robust) goodness-of-fit
criterion is used to choose a good model for the majority of the data rather
than an “average” model for all the data. Let us focus our attention here on
the two robust coefficients of determination available in Stata: $R_\rho^2$ and
$R_w^2$.

If instead of the \underbar{LS} estimate we use an \underbar{M} estimate (associated with the loss
function $\rho$) with general scale, defined as in (\ref{eq:M_min}), a robust
coefficient of determination can be defined by
%
\begin{equation}\label{eq:R2_rho}
    R_\rho^2 = 1 - \frac{\sum_{i=1}^{n}
    \rho\left(\frac{y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{M};\rho}}{\sthat{\sigma}}\right)}
    {\sum_{i=1}^{n} \rho\left(\frac{y_i-\sthat{\mu}_{\stsc{M};\rho}}{\sthat{\sigma}}\right)}
\end{equation}
where $\sthat{\mu}_{\stsc{M};\rho}$ is the \stsc{M}~estimate of the location
parameter $\mu = E(y)$, solution of
\[
    \argmin_\mu \sum_{i=1}^{n} \rho\left(\frac{y_i-\mu}{\sthat{\sigma}}\right)
\]
and $\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ and $\sthat{\sigma}$ are robust
estimates of $\boldsymbol\beta$ and $\sigma$ for the full model (see
\citealp{maronna:etal:2006}).

Note that, independently, \citet{croux:dehon:2003} have proposed a class of
robust $R^2$ which generalizes (\ref{eq:R2_classic_SS}) given by
%
\begin{equation}\label{eq:R2_S}
    R_{\stsc{S}}^2 = 1 - \frac{s(y_i-\stvec{x}_i^t\widehat{\boldsymbol\beta}; i = 1, \dots, n)}
    {s(y_i-\sthat{\mu}; i = 1, \dots, n)}
\end{equation}
%
where $s(\cdot)$ is a robust dispersion measure.

Although (\ref{eq:R2_rho}) and (\ref{eq:R2_S}) are direct generalizations of
(\ref{eq:R2_classic_SS}) to the robust framework, they suffer from an important
drawback: in practice, they are often biased. One possible reason why this
phenomenon happens is that the computation of $R_\rho^2$ or $R_\stsc{S}^2$
requires and uses the estimation of two models: the full regression model and a
location model. The associate residuals $y_i -
\stvec{x}_i^t\sthat{\boldsymbol\beta}$ and $y_i - \sthat{\mu}$ are not
influenced by model deviation (as presence of outliers, for instance) in the
same way, so that bounding these quantities directly and separately is not
necessarily appropriate in the regression model framework.

To remedy this problem, \cite{Renaud:VictoriaFeser:2010} have proposed to
“robustify” the expression (\ref{eq:R2_classic_corr}) of the coefficient of
determination. Suppose $\boldsymbol\beta$ has been estimated by an \stsc{M},
\stsc{S} or \stsc{MM}~estimate $\sthat{\boldsymbol\beta}$ using a loss function
$\rho(\cdot)$, and let $\sthat{\sigma}$ be the final robust estimate of the
scale parameter $\sigma$. Let, as usual, $\psi(u) = \rho'(u)$ for
$u\in\mathbb{R}$. Define, as in section
\ref{subsec:practical_implementation_Mestimate}, the weight function $W$ by
\[
    W(u) =
    \begin{cases}
        \frac{\psi(u)}{u} & \text{if $u \neq 0$} \\
        \psi'(0)          & \text{if $u = 0$}
    \end{cases}
\]
and the weights
\[
    w_i = W \left(r_i(\sthat{\boldsymbol\beta})/\sthat{\sigma}\right),
    \quad i = 1, \dots, n.
\]
Note that these weights $w_i$ coincide with those used in the last iteration
of the \emph{iteratively reweighted least squares algorithm} used to implement
the \stsc{M}~estimation procedure. In particular, if $\rho(\cdot)$ is the
Tukey-Biweight function $\rho_{\kappa}^{B}(\cdot)$ given by
(\ref{eq:Tukey_Biweight_function}), we have
\[
    w_i =
    \begin{cases}
        \left(1 - \left(\frac{r_i(\sthat{\boldsymbol\beta})}{\kappa\sthat{\sigma}}\right)^{\!2}\right)^{\!\!2}
        & \text{if $|r_i(\sthat{\boldsymbol\beta})/\sthat{\sigma}| \leq \kappa$}\\
        0
        & \text{if $|r_i(\sthat{\boldsymbol\beta})/\sthat{\sigma}| > \kappa$}.
    \end{cases}
\]
Then a robust version of (\ref{eq:R2_classic_corr}) is given by
%
\begin{equation}
    \label{eq:R2_w}
    R_w^2 = \left(\frac{\sum_{i=1}^{n} w_i(y_i-\overline{y}_w) (\sthat{y}_i-\overline{\sthat{y}}_w)}
    {\sqrt{\sum_{i=1}^{n} w_i(y_i-\overline{y}_w)^2}
     \sqrt{\sum_{i=1}^{n} w_i(\sthat{y}_i-\overline{\sthat{y}}_w)^2}}\right)^{\!\!2}
\end{equation}
%
where $\sthat{y}_i = y_i - \stvec{x}_i^t\sthat{\boldsymbol\beta}$,
$\overline{y}_w = (1/\sum w_i)\sum w_i y_i$ and $\overline{\sthat{y}}_w =
(1/\sum w_i)\sum w_i\sthat{y}_i$.

With the same weights and predictions, another robust coefficient of
determination can be defined from (\ref{eq:R2_classic_SS}):
%
\begin{equation}
    \label{eq:Rtilde2_w}
    \widetilde{R}_w^2 = 1 - \frac{\sum_{i=1}^{n} w_i(y_i - \sthat{y}_i)^2}
    {\sum_{i=1}^{n} w_i(y_i - \overline{y}_w)^2}
\end{equation}
%
It is shown in \citet{Renaud:VictoriaFeser:2010} that
\[
    R_w^2 = \widetilde{R}_w^2.
\]

\citet{Renaud:VictoriaFeser:2010} have also proposed the following more
general formulation for a robust coefficient of determination in order to take
into account consistency considerations:
%
\begin{equation}
    \label{eq:Rtilde2_w_a}
    \widetilde{R}_{w,a}^2 = \frac{\sum_{i=1}^{n} w_i (\sthat{y}_i - \overline{\sthat{y}}_w)^2}
    {\sum_{i=1}^{n} w_i (\sthat{y}_i - \overline{\sthat{y}}_w)^2 +
    a\sum_{i=1}^{n} w_i (y_i - \sthat{y}_i)^2}
\end{equation}
%
where $a$ is a constant factor. It has been shown that $R_w^2$ and
$\widetilde{R}_w^2$ are both equal to $\widetilde{R}_{w,a}^2$ with
$a=1$. Moreover, with no assumption on the distribution of the explanatory
variables, but under the assumption of normality of the errors and for a
consistent estimator $\sthat{\sigma}$ of the residual scale, $\widetilde{R}_{w,a}^2$
is a consistent estimator of the population coefficient of
determination (\ref{eq:phi2}) if we take
\[
    a = \frac{E[\psi(u)/u]}{E[\psi(u)]},
    \quad\text{with $u \sim \mathcal{N}(0,1)$}.
\]
For example, choosing $\psi(u) = \psi_{\kappa}^B(u) = \rho_{\kappa}^{B}{'}(u)$,
where $\rho_{\kappa}^B$ is the Tukey-Biweight loss function with
$\kappa=4.685$, leads to $a=1.2076$.

As shown by a simulation study in \citet{Renaud:VictoriaFeser:2010}, for
small samples and a relatively large number of covariates, using the same
rationale than for the classical $R^2$, the robust coefficient might benefit
of being adjusted, hence leading to the adjusted coefficient
%
\begin{equation}
    \label{eq:Rtilde2_w_a_adj}
    \widetilde{R}_{w,a;\mathrm{adj}}^2 = 1 - \left(1 - \widetilde{R}_{w,a}^2\right)
    \left(\frac{n-1}{n - (p+1)}\right).
\end{equation}

\subsection{Extension of the Hausman test to check for the presence of outliers}
\label{subsec:Hausman}

In practice, it is usual to ask oneself if it is necessary to use a robust
regression estimator or if it is preferable to use a classical estimator that
is more efficient under the model and more easy to compute. When the data are
not contaminated by outliers, classical and robust estimations of the
regression coefficients are quite similar, while a moderate contamination of
the sample may imply a possible clear difference between classical and robust
estimations. Hence, no significant difference between the classical and robust
estimations of $\boldsymbol\beta$ may lead us to conclude that the data do
not contain outliers or that the influence of the outliers is rather limited:
in such a case, we will prefer to retain the classical estimator given its
higher efficiency (its higher statistical precision). On the contrary, a
significant difference between the classical and robust estimations of
$\boldsymbol\beta$ indicates that the data are contaminated by outliers in
such a way that it biases the classical estimator: a robust estimator should
then be preferred.

But which tool may we use to compare adequately two regression estimators and
to judge if their values are significantly different or not?

To solve this question, \citet{Dehon:2009,Dehon:2012} have proposed a
statistical test, based on the methodology developed by \citet{Hausman:1978}.
Their testing procedure allows to compare a robust \stsc{S}~estimate and the
classical \stsc{LS}~estimate (in order to detect the presence of outliers). But
it also allows to compare an \stsc{S}~estimator with an \stsc{MM}~estimator
with a given efficiency level; repeating this test by considering different
efficiency levels for the \stsc{MM}~estimator may be seen as a procedure
allowing, in the presence of moderate contamination of the sample, to find in
an appropriate way the maximum efficiency level that may have this
\stsc{MM}~estimator without suffering from too large bias.

In all the cases, the problem of test may be formalized as follows. Consider
the regression model (\ref{eq:linear_regr_model}). The null hypothesis
$\mathcal{H}_0$ is that this model is valid for the entire population. Thus,
at the sample level, under the null, no outliers are present. The alternative
hypothesis $\mathcal{H}_1$ is that the model is misspecified for a
minority of the population, implying a potentially moderate contamination of
the sample. Note that we will also systematically consider that, under the
null hypothesis $\mathcal{H}_0$, Assumption~A1 is satisfied.

Before to describe the test statistics and the decision rules, let us precise a
last point: since \citet{Gervini:2002} showed that, \emph{in the presence of
outliers}, only the $p$ slopes $\beta_1, \dots, \beta_p$ of the regression
model can be satisfactorily estimated when the error distribution is
asymmetric, the test will be based on the comparison of the slopes estimations
and the estimations of the intercept $\beta_0$ will be disregarded. Hence, in
the sequel of this section, we will use the following notations to take this
characteristic into account: $\underline{\boldsymbol\beta} = (\beta_1, \dots,
\beta_p)^t$ and $\underline{\sthat{\boldsymbol\beta}} = (\sthat{\beta}_1,
\dots, \sthat{\beta}_p)^t$, such that $\boldsymbol\beta = (\beta_0,
\underline{\boldsymbol\beta}^t)^t$ and $\sthat{\boldsymbol\beta} = (\beta_0,
\underline{\sthat{\boldsymbol\beta}}^t)^t$.

\subsubsection{Some preliminary results}

The development of the tests proposed in \citet{Dehon:2012} relies on the
results presented in Subsection~\ref{subsec:asymptotic_distr_M_S_MM_estimators}
providing the asymptotic distribution of $\sthat{\boldsymbol\beta}_\stsc{LS}$,
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ and
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ under $\mathcal{H}_0$ and
Assumption~A. In Subsection~\ref{subsec:asymptotic_distr_M_S_MM_estimators}, to
avoid any ambiguity, the regression parameters vector was denoted by
$\boldsymbol\beta_0$ if it was estimated by the \stsc{S}~estimator
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$, and by $\boldsymbol\beta$ if it
was estimated by the \stsc{MM}~estimator
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$. From now on, we will
exclusively denote the regression parameters vector in model
(\ref{eq:location_scale_regr_model}) by $\boldsymbol\beta$ as soon as there is
no risk of confusion anymore.

We have seen that, under $\mathcal{H}_0$ and Assumption~A, for large $n$,
%
\begin{align*}
    \sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho} & \approx
        \mathcal{N}_{p+1}\left(\boldsymbol\beta,
        \mathrm{Avar}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})\right)
    \\
    \sthat{\boldsymbol\beta}_{\stsc{S};\rho_0} &  \approx
    \mathcal{N}_{p+1}\left(\boldsymbol\beta,
        \mathrm{Avar}(\widehat{\boldsymbol\beta}_{\stsc{S};\rho_0})\right)
    \\
    \sthat{\boldsymbol\beta}_\stsc{LS} & \approx
        \mathcal{N}_{p+1}\left(\boldsymbol\beta,
        \mathrm{Avar}(\sthat{\boldsymbol\beta}_\stsc{LS})\right)
\end{align*}
%
where the matrices
$\mathrm{Avar}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$,
$\mathrm{Avar}(\widehat{\boldsymbol\beta}_{\stsc{S};\rho_0})$ and
$\mathrm{Avar}(\sthat{\boldsymbol\beta}_\stsc{LS})$ are given by
(\ref{eq:Avar_betahat_MM}), (\ref{eq:Avar_betahat_S}) and
(\ref{eq:Avar_betahat_LS}), respectively. Moreover,
%
\begin{align*}
    \sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
        - \widehat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}
     & \approx \mathcal{N}_{p+1}\left(\stvec{0},
         \mathrm{Avar}(\widehat{\boldsymbol\beta}_{\stsc{S};\rho_0})
       + \mathrm{Avar}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})
     - 2 \mathrm{Acov}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho},
             \sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})\right)
\end{align*}
%
where $\mathrm{Acov}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho},
\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})$ is given by
(\ref{eq:Acov_betahat_MM_S}). Since
$\mathrm{Avar}(\widehat{\boldsymbol\beta}_{\stsc{S};\rho_0})$,
$\mathrm{Avar}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$ and
$\mathrm{Acov}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho},
\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})$ may be consistently estimated by
their empirical counterparts
$\sthat{\mathrm{Avar}}(\widehat{\boldsymbol\beta}_{\stsc{S};\rho_0})$,
$\sthat{\mathrm{Avar}}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$ and
$\sthat{\mathrm{Acov}}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho},
\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})$,\footnote{As explained in
Subsection~\ref{subsec:asymptotic_distr_M_S_MM_estimators}, these empirical
counterparts are simply obtained by replacing, in $u$ and $u_0$, the parameters
$\boldsymbol\beta$, $\boldsymbol\beta_0$ and $\sigma$ by the estimates
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$,
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ and $\sthat{\sigma}_{\rho_0}$, and
$E(\cdot)$ by $\frac{1}{n} \sum_{i=1}^{n} (\cdot)$.} we have, under
$\mathcal{H}_0$ and Assumption~A, for large $n$:
\[
    \sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
        - \sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}
    \approx \mathcal{N}_{p+1}\left(\stvec{0},
    \sthat{\boldsymbol\Sigma}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
        - \sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})}
    \right)
\]
where
%
\begin{align}
    \label{eq:Hausman_sigmahat_S-MM}
    \sthat{\boldsymbol\Sigma}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
        - \sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})}
    &  = \sthat{\mathrm{Avar}}(\widehat{\boldsymbol\beta}_{\stsc{S};\rho_0})
       + \sthat{\mathrm{Avar}}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})
     - 2 \sthat{\mathrm{Acov}}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho},
             \sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})
\end{align}
%
If we only consider the slopes estimates, we simply have, under $\mathcal{H}_0$
and Assumption~A, for large $n$:
%
\begin{equation}
    \label{eq:Hausman_normality_S-MM}
    \sthat{\underline{\boldsymbol\beta}}_{\stsc{S};\rho_0}
        - \sthat{\underline{\boldsymbol\beta}}_{\stsc{MM};\rho_0,\rho}
    \approx \mathcal{N}_p\left(\underline{\stvec{0}},
    \sthat{\underline{\boldsymbol\Sigma}}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
        - \sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})}
    \right)
\end{equation}
%
where
$\sthat{\underline{\boldsymbol\Sigma}}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
- \sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})}$ is the matrix
$\sthat{\boldsymbol\Sigma}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0} -
\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})}$ without its first line and
its first column.

Following a similar approach, we have, under $\mathcal{H}_0$ and Assumption~A,
for large $n$:
\[
    \sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
        - \sthat{\boldsymbol\beta}_\stsc{LS}
    \approx\mathcal{N}_{p+1} \left(\stvec{0},
        \sthat{\boldsymbol\Sigma}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
            - \sthat{\boldsymbol\beta}_\stsc{LS})}
    \right)
\]
with
%
\begin{align}
    \label{eq:Hausman_sigmahat_S-LS}
    \sthat{\boldsymbol\Sigma}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
        - \sthat{\boldsymbol\beta}_\stsc{LS})}
    & = \sthat{\mathrm{Avar}}(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})
      + \sthat{\mathrm{Avar}}(\sthat{\boldsymbol{\beta}}_\stsc{LS})
    - 2 \sthat{\mathrm{Acov}}(\sthat{\boldsymbol\beta}_\stsc{LS},
        \sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})
\end{align}
%
where
$\sthat{\mathrm{Avar}}(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})$,
$\sthat{\mathrm{Avar}}(\sthat{\boldsymbol{\beta}}_\stsc{LS})$ and
$\sthat{\mathrm{Acov}}(\sthat{\boldsymbol\beta}_\stsc{LS},
\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})$ are the
empirical counterparts of the matrices
$\mathrm{Avar}(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})$,
$\mathrm{Avar}(\sthat{\boldsymbol{\beta}}_\stsc{LS})$ and
$\mathrm{Acov}(\sthat{\boldsymbol\beta}_\stsc{LS},
\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})$
given by (\ref{eq:Avar_betahat_S}), (\ref{eq:Avar_betahat_LS}) and
(\ref{eq:Acov_betahat_LS_S}), respectively. As a consequence, under
$\mathcal{H}_0$ and Assumption~A, for large $n$:
%
\begin{equation}
    \label{eq:Hausman_normality_S-LS}
    \sthat{\underline{\boldsymbol\beta}}_{\stsc{S};\rho_0}
        - \sthat{\underline{\boldsymbol\beta}}_\stsc{LS}
    \approx\mathcal{N}_p \left(\underline{\stvec{0}},
        \sthat{\underline{\boldsymbol\Sigma}}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
            - \sthat{\boldsymbol\beta}_\stsc{LS})}
    \right)
\end{equation}
%
where
$\sthat{\underline{\boldsymbol\Sigma}}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
- \sthat{\boldsymbol\beta}_\stsc{LS})}$
is the matrix
$\sthat{\boldsymbol\Sigma}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
- \sthat{\boldsymbol\beta}_\stsc{LS})}$
without its first line and its first column.

\subsubsection{Comparison of LS and S}

Let us consider the classical \stsc{LS}~estimator
$\sthat{\boldsymbol\beta}_\stsc{LS}$ and the \stsc{S}~estimator
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ associated with the loss function
$\rho_0(\cdot)$. As already mentioned, the choice of $\rho_0$ is crucial to
guarantee robustness. The function $\rho_0$ usually used in the present context
is the Tukey-Biweight function (\ref{eq:Tukey_Biweight_function}): if the
tuning constant $\kappa$ is set at 1.547,
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ has a breakdown point equal to
50\% (but a rather low Gaussian efficiency of only 28\%). Under the null
hypothesis (and Assumption~A), $\sthat{\boldsymbol\beta}_\stsc{LS}$ and
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ are both consistent estimators of
$\boldsymbol\beta$, but $\sthat{\boldsymbol\beta}_\stsc{LS}$ has a higher
Gaussian efficiency. Under the alternative hypothesis of a moderate
contamination, $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ still converges to
$\boldsymbol\beta$ (see \citealp{Omelka:2010}) but it is not the case for
$\sthat{\boldsymbol\beta}_\stsc{LS}$ anymore (the outliers distort the LS
estimate and introduce a bias, in such a way that
$\sthat{\boldsymbol\beta}_\stsc{LS}$ possesses another limit in probability
than $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$).

The test statistics proposed by \citet{Dehon:2012} to check whether the
\stsc{LS} and \stsc{S}~estimates of the regression coefficients are
statistically different is defined as
\begin{equation}
    \label{Hausman_SvsLS}
    H = \left(\sthat{\underline{\boldsymbol\beta}}_{\stsc{S};\rho_0}
            - \sthat{\underline{\boldsymbol\beta}}_\stsc{LS}\right)^{\!t}
        \sthat{\underline{\boldsymbol\Sigma}}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
            - \sthat{\boldsymbol\beta}_\stsc{LS})}^{-1}
        \left(\sthat{\underline{\boldsymbol\beta}}_{\stsc{S};\rho_0}
            - \sthat{\underline{\boldsymbol\beta}}_\stsc{LS}\right)
%
\end{equation}
with
$\sthat{\underline{\boldsymbol\Sigma}}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
- \sthat{\boldsymbol\beta}_\stsc{LS})}$ computed from
(\ref{eq:Hausman_sigmahat_S-LS}). It follows from
(\ref{eq:Hausman_normality_S-LS}) that $H$ is, under the null hypothesis
$\mathcal{H}_0$ (and Assumption~A), asymptotically distributed as a
$\chi_p^2$ (a chi-square distribution with $p$ degrees of freedom).
Consequently, we may consider that the classical estimate and the
\stsc{S}~estimate of the regression slopes are significantly different, and hence decide
to reject the null hypothesis $\mathcal{H}_0$, if
\[
    H > \chi_{p;1-\alpha}^2,
\]
where $\alpha$ is the given significance level and $\chi_{p;1-\alpha}^2$ is the
$(1-\alpha)$-quantile of the $\chi_p^2$ distribution.

\subsubsection{Comparison of S and MM}

Suppose now that the previous test has rejected the null hypothesis
$\mathcal{H}_0$: the significant difference between
$\sthat{\underline{\boldsymbol\beta}}_\stsc{LS}$ and
$\sthat{\underline{\boldsymbol\beta}}_{\stsc{S};\rho_0}$ indicates the presence
of influential outliers in the sample and a robust regression estimator should
then be preferred. In this case, it might be a good strategy to replace the
\stsc{S}~estimator $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ by an
\stsc{MM}~estimator $\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$, since a good
choice of the loss function $\rho(\cdot)$ allows this \stsc{MM}~estimator to
reach a much higher efficiency than the initial \stsc{S}~estimator
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$. \footnote{Recall here that
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ possesses the same breakdown
point as $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$.}. For instance, if we
take for $\rho$ the Tukey-Biweight function (\ref{eq:Tukey_Biweight_function})
with the tuning constant $\kappa$ equal to 4.685, the Gaussian efficiency of
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ attains 95\%, and for
$\kappa=6.256$, the Gaussian efficiency of
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ is equal to 99\%. However,
as already mentioned when we have studied the \stsc{MM}~estimation procedure,
it is not advised to consider too highly efficient \stsc{MM}~estimators:
indeed, a moderate contamination of the sample induces a bias for
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ and, for a fixed sample,
this bias grows when the efficiency of the estimator raises (see
\citealp{maronna:etal:2006} and \citealp{Omelka:2010}). As a consequence, it is
of the utmost importance to find the highest efficiency we may fix for the
\stsc{MM}~estimator without paying the price of an excessive bias.

The statistical comparison of $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ and
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$
(with a fixed value of the tuning constant $\kappa$ for the loss function
$\rho$, hence a fixed Gaussian efficiency for the \stsc{MM}~estimator) can be made
using the statistics
%
\begin{equation}
    \label{Hausman_MMvsS}
    H = \left(\sthat{\underline{\boldsymbol\beta}}_{\stsc{S};\rho_0}
            - \sthat{\underline{\boldsymbol\beta}}_{\stsc{MM};\rho_0,\rho}\right)^{\!t}
        \sthat{\underline{\boldsymbol\Sigma}}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}
            - \sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})}^{-1}
        \left(\sthat{\underline{\boldsymbol\beta}}_{\stsc{S};\rho_0}
            - \sthat{\underline{\boldsymbol\beta}}_{\stsc{MM};\rho_0,\rho}\right)
\end{equation}
%
with $\sthat{\boldsymbol\Sigma}_{(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0} -
\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})}$ given by
(\ref{eq:Hausman_sigmahat_S-MM}). Under the null hypothesis $\mathcal{H}_0$,
$\sthat{\underline{\boldsymbol\beta}}_{\stsc{S};\rho_0}$ and
$\sthat{\underline{\boldsymbol\beta}}_{\stsc{MM};\rho_0,\rho}$ are both
consistent estimators of \underline{$\boldsymbol\beta$} and $H \approx
\chi_p^2$. Under the alternative hypothesis $\mathcal{H}_1$, that is, under a
moderate contamination of the sample, the bias of
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ risks to be large (the
magnitude of the bias depends of the fixed efficiency of the
\stsc{MM}~estimator) and a potentially significant difference may appear between the
\stsc{S}~estimate and the \stsc{MM}~estimate of the regression slopes. As a
consequence, we will decide to reject $\mathcal{H}_0$---that is, in practice,
we will conclude that the contamination of the sample by outliers significantly
biases $\sthat{\underline{\boldsymbol\beta}}_{\stsc{MM};\rho_0,\rho}$ and hence
distorts the \stsc{MM}~estimation with respect to the \stsc{S}~estimation---if
\[
    H > \chi_{p;1-\alpha}^2,
\]
where $\alpha$ is the chosen significance level. \citet{Dehon:2012}
propose to repeat this test by considering successively different values for
the constant $\kappa$ in function $\rho$ (that is, different levels for the
efficiency of $\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$) and
to retain ultimately the \stsc{MM}~estimator that, while not being significantly
different from $\sthat{\underline{\boldsymbol\beta}}_{\stsc{S};\rho_0}$
and hence not rejecting the null, has the highest efficiency. This way of
proceeding allows to find heuristically the highest efficiency that may have
the \stsc{MM}~estimator without suffering from an excessive bias in presence of
moderate contamination of the sample by outliers.

\section{Examples}

\subsection{Comparing estimators}

In the first example, we will use a dataset made available by
\citet{rousseeuw:leroy:1987}. The dataset contains 47 stars in the direction of
Cygnus. The explanatory variable is the logarithm of the effective temperature
at the surface of the star ($T_e$), and the dependent variable is the logarithm
of its light intensity ($L/L_0$). In the scatterplot of
figure~\ref{fig:stars_scatterplot}, it is evident that some stars (represented
by hollow circles) have a very different behavior than the bulk of the data. To
illustrate graphically the influence that these stars have on the estimation of
the regression line, we superpose to the scatterplot two lines estimated by (1)
ordinary least squares (solid line) and (2) a robust regression estimation
method (more precisely, \stsc{S}~estimation; dashed line).                  \todo{Graph-Ex-1.do was missing; I recreated the graph from the description.}

\begin{figure}[h!]
    \centering
    \epsfig{file=eps/4/4}
    \caption{Effective temperatures and light intensities in star cluster CYG OB1}
    \label{fig:stars_scatterplot}
\end{figure}

To obtain the \stsc{LS}~estimates for the intercept and slope of the regression
of log intensity on log temperature in Stata, without making any difference
between stars, we can type:

\begin{stlog}
\input{dof/4/5.log.tex}
\end{stlog}

The results of the \stsc{LS}~estimation (solid line in
figure~\ref{fig:stars_scatterplot}) indicate that, against intuition, if the
temperature of a star increases, its light intensity decreases in average
(although the effect is not significantly different from zero). If instead of
using a classical estimator we use a robust estimator, the result changes
drastically. For illustrative purposes we will estimate the above model using
\stsc{LTS}, \stsc{LMS}, \stsc{M}, \stsc{GM}, \stsc{S} and \stsc{MM} (with an
efficiency fixed at 95\%) estimators. However, since it has now been widely
accepted that \stsc{S} and \stsc{MM}~estimators are preferable to \stsc{LTS}
and \stsc{LMS} because of their higher efficiency and to \stsc{M} and
\stsc{GM}~estimator because of their higher robustness with respect to outliers (see
above), in the subsequent examples we will focus exclusively on \stsc{S} and
\stsc{MM}~estimates.

\subsubsection{LTS}

A robust \stsc{LTS}~estimator can be easily fit using the \stcmd{robreg} package
running the following command:

\begin{stlog}
\input{dof/4/6.log.tex}
\end{stlog}

The results from the \stsc{LTS}~estimator are very different from those
obtained for the \stsc{LS}~estimation. Indeed what we observe here is that if
the log of the temperature of a star increases, its luminosity will increase as
well. In term of size of effect, the \stsc{LTS}~estimator suggests that an
increase of 100\% of the temperature is associated to an increase of the
luminosity of approximately 473\%.

\subsubsection{LMS}

If instead of the \stsc{LTS}~estimator we wish to use the \stsc{LMS}~estimator,
we can type:

\begin{stlog}
\input{dof/4/7.log.tex}
\end{stlog}

Even if the size of effect seems to be slightly smaller than with \stsc{LTS},
the sign of the relation is the same pointing towards a positive association
between log-temperature and lightness of stars.

\subsubsection{M estimator}

If we use the \stsc{M}~estimator (with a Huber
loss function), we do not expect the estimation to resist to outliers. Indeed,
in the theoretical section it has been shown how this
estimator resists to vertical outliers but not to bad leverage points (i.e.
points outlying in the space of the explanatory variables). As expected, the
\stsc{M}~estimation provides results very similar to those of \stsc{LS} and we can conclude
that the estimator breaks down.

The command to run the \stsc{M}~estimator is:

\begin{stlog}
\input{dof/4/8.log.tex}
\end{stlog}

\subsubsection{GM estimator}

The Generalized \stsc{M}~estimate is slightly more complicated to compute than
the \stsc{M}~estimate. We first need to estimate the outlyingness of each
individual in the x-dimension, and then downweight leverage points while
estimating the model using an \stsc{M}~estimator. In this example, given that
there is a single explanatory variable, the outlyingness in the horizontal
dimension can be measured by centering the data around a robustly estimated
location parameter (e.g.\ the Hodges-Lehman estimate or the median) and reducing
it using a robustly estimated measure of dispersion (e.g.\ the Croux and
Rousseeuw $Q_n$ estimate). In the case of multiple explanatory variables, the
outlyingness in the space of the explanatory variables will have to be measured
using robust multivariate estimates of location and scatter described in
chapter \alert{XXX}. As far as the down-weighting scheme for outliers is
concerned, several alternatives have been proposed in the literature. In this
example we award a weight equal to zero to any star associated to a leverage
larger than 2.5 and equal to one otherwise. Given that there is one single
explanatory variable, the \stsc{GM}~estimator should behave satisfactory.

The commands used for \stsc{GM}~estimation are:                                 \todo{To be updated}

\begin{stlog}
\input{dof/4/9.log.tex}
\end{stlog}

\subsubsection{S estimator}

If we estimate the model using an \stsc{S}~estimator, we do not expect to have
large differences with respect to \stsc{LTS}, \stsc{LMS} and \stsc{GM} in terms
of point estimates. However, its higher efficiency makes it theoretically more
appealing. The command to run the \stsc{S}~estimator is:

\begin{stlog}
\input{dof/4/10.log.tex}
\end{stlog}

The results indicate that if the temperature of a star doubles, its light
intensity increases by approximately 329\%. As stated in the theoretical
section, the gaussian efficiency of the \stsc{S}~estimator with a 50\%
breakdown point (and a Tukey biweight loss function) is only 28\%. In order to
increase the efficiency while keeping the breakdown point at 50\%, we can use
\stsc{MM}~estimators.

\subsubsection{MM estimator}

It is well-known that even if an \stsc{MM}~estimator has a breakdown point of 50\%,
it can be associated to a relatively large bias if its efficiency is set too
high. As explained in Subsection \ref{subsec:Hausman}, a general procedure is
therefore to compare the \stsc{MM}~estimate with a given level of efficiency to the
\stsc{S}~estimate, and see if there is a significant difference. If the difference
is small, this means that the bias should not be too big.

We compute here an \stsc{MM}~estimator with an efficiency set at 95\%:

\begin{stlog}
\input{dof/4/11.log.tex}
\end{stlog}

We see that the \stsc{MM}~estimation leads to results comparable to the
\stsc{S}~estimation in terms of point estimates but is associated to a much
higher efficiency. As explained above, a formal test could have been used but
we leave this for another example. The \stsc{MM}~estimated model suggests that
an increase of 100\% of the temperature of a star is associated with an
increase of its luminosity by approximately 225\%. In terms of the quality of
the fit, if we rely on the robust $R^2_w$ described previously, we see that
the model is pretty good in predicting the luminosity of stars for the vast
majority of the observations. Indeed close to 42\% of the variations in terms
of light intensity for the vast majority of the observations can be explained
by the differences in temperatures.

\subsection{Identifying outliers}

In this second example where the objective is
to unmask outliers, we use a dataset made available by Jeffrey D. Sachs and
Andrew M. Warner in their article “Natural Resource Abundance
and Economic Growth” (\citeyear{sachs:warner:1997}). In this paper, the authors show
that economies with a high ratio of natural resource exports to GDP in 1970
(the base year) tended to grow slowly during the subsequent 20 year period
1970--1990. In the article the authors acknowledge the existence of outliers and
try to deal with them working with differences in fits. More precisely, they
look at how the predicted value for each observation varies when this specific
observation is removed from the sample when fitting the model and compare the
results with the model estimated using all of the observations. They expect to
see big differences in fits for outlying observations. However, if there are
clusters of outliers, atypical observations will mask one another and will
most probably not be detected with this approach. The outliers they identify
are Chad, Gabon, Guyana, and Malaysia.

We propose here to use another procedure to identify the outliers. This
procedure is simply based on the examination of the standardized residuals
related to a regression \stsc{S}~estimator. That is, to identify the outliers,
we first estimate the regression model by running the following command:

\begin{stlog}
\input{dof/4/12.log.tex}
\end{stlog}

To compute the standardized residuals, we then obtain the predicted values from
the model and divide the difference between observed values and predictions by
the scale parameter from the regression \stsc{S}~estimate:

\begin{stlog}
\input{dof/4/13.log.tex}
\end{stlog}

After that we can plot the standardized residuals and identify those that are
larger or smaller than two given cut-off points corresponding to two specific
quantiles of the normal distribution. We use here the percentiles 2.5 and 97.5
which are respectively equal to $-1.96$ and $1.96$.

\begin{stlog}
\input{dof/4/14.log.tex}
\end{stlog}

\begin{figure}[h!]
    \centering
    \epsfig{file=eps/4/14}
    \caption{Outliers in the Sachs and Warner data}
    \label{fig:countries_S_standardized_res}
\end{figure}

We see in Figure \ref{fig:countries_S_standardized_res} that, among the four
countries identified as outliers by Sachs and Warner, only Malaysia is still
emerging as outlier when using the \stsc{S}~estimation procedure. On the other
hand, other countries such as Hong Kong, Ecuador or Iran seem to be atypical
countries in the \stsc{S}~regression but were not detected by the original
authors.

\subsection{Testing for the presence of outliers and setting the efficiency for MM estimation}

For this example, we again use the dataset relating the logarithm of the
effective temperature at the surface of the star (explanatory variable $T_{e}$)
and the logarithm of its light intensity (dependent variable $L/L_0$). The
first question one might raise is: is there a significant difference between
the classic estimate and the robust one? To answer this question we simply
compute an \stsc{S}~estimate using the \stcmd{robreg s} command and apply the
\stcmd{hausman} option. This implies that the testing procedure comparing the
\stsc{S}~estimate with the \stsc{LS}~estimate (see Subsection
\ref{subsec:Hausman}) is carried out.

\begin{stlog}
\input{dof/4/15.log.tex}
\end{stlog}

The results of the Hausman test indicate that the difference between the
\underbar{S} estimate and the \stsc{LS}~estimate is significant (p-value $<
.05$) and thus that outliers distort the \stsc{LS}~estimation. We should
therefore use a robust estimator.

As stated previously \stsc{S}~estimators are very robust against outlier
contamination but are relatively inefficient. \stsc{MM}~estimators on the other
hand are more efficient than \stsc{S}~estimators but might be associated with a
large bias if efficiency is set too high. To choose the level of efficiency to
use in practice we have to apply the testing procedure described in Subsection
\ref{subsec:Hausman} that compares the \stsc{MM}~estimates related to some
given levels of efficiency with respect to an \stsc{S}~estimate. We can then
finally set the efficiency of the \stsc{MM}~estimator at the highest efficiency
level that does not lead to a rejection of the equality between the
\stsc{MM}~estimate and the \stsc{S}~estimate. Doing this in practice is very simple as
the testing procedure is implemented in the \stcmd{robreg mm} command. For
example, if the \stcmd{robreg mm} command is run with the efficiency set at
75\% , the \stcmd{hausman} option compares the \stsc{MM}~estimate with 75\%
efficiency to the \stsc{S}~estimate obtained at the first step of the
\stsc{MM}~estimation procedure. Similarly, if the efficiency is set at 85\%, the
\texttt{hausman} option compares the \stsc{MM}~estimate with 85\% efficiency to
the \stsc{S}~estimate, and so on. In this example we check whether we can set
the efficiency at 75\%, 85\%, 95\%, and 99\%. We obtain the following results:

\subsubsection{MM estimation with 75\% efficiency}

\begin{stlog}
\input{dof/4/16.log.tex}
\end{stlog}

The results of the Hausman test indicate that there is no significant
difference between the \stsc{MM}~estimate and the \stsc{S}~estimate. It would
therefore be preferable to work with the \stsc{MM}~estimator with an efficiency
equal to 75\% as it provides results comparable to the \stsc{S}~estimator in
terms of bias but has a much higher efficiency.

\subsubsection{MM estimation with 85\% efficiency}

\begin{stlog}
\input{dof/4/17.log.tex}
\end{stlog}

Here again the Hausman test statistics takes a low value which tells us that
there is no significant difference between the \stsc{MM}~estimate and the
\stsc{S}~estimate.

\subsubsection{MM estimation with 95\% efficiency}

\begin{stlog}
\input{dof/4/18.log.tex}
\end{stlog}

If we set the efficiency of the \stsc{MM}~estimator to 95\%, we still do not
observe any significant difference between the \stsc{MM}~estimate and the
\stsc{S}~estimate.

\subsubsection{MM estimation with 99\% efficiency}

\begin{stlog}
\input{dof/4/19.log.tex}
\end{stlog}

In the case of an efficiency of the \stsc{MM}~estimator equal to 99\%, the
Hausman test rejects the null hypothesis of equality between the
\stsc{MM}~estimate and the \stsc{S}~estimate, which means that for this very high level
of efficiency, the \stsc{MM}~estimator suffers from a too large bias.

To summarize, it is clear that a classic estimator cannot be used in the
example because the \stsc{LS}~estimates are clearly distorted. A robust
estimator should be preferred. We may use an \stsc{MM}~estimator with an
efficiency equal to 95\% instead of the less efficient \stsc{S}~estimator since
despite the bias from which the \stsc{MM}~estimator potentially suffers, the
\stsc{MM}~estimates of the regression parameters appear no significantly
different from the \stsc{S}~estimates. It is not recommended to consider a
higher level of efficiency for the \stsc{MM}~estimator (99\%, for instance),
since the statistical test indicates that the bias becomes too big in that case.

\subsection{Recognizing the type of outliers}

For this example, we will use the famous auto dataset available from Stata.
This dataset contains the price of a set of cars as well as a series of
characteristics. To see if outliers are present in the dataset, we regress the
price on all the available characteristics and compute the robust standardized
residuals. Obviously this will not allow to recognize the types of outliers. To
do so, we will use the graphical tool of \citet{rousseeuw:zomeren:1990}. The
idea here is to use a scatter plot considering on the vertical dimension the
standardized residuals and on the horizontal dimension the leverage of the
observations measured using the robust Mahalanobis distance (as described in
(\ref{eq:leverage})). For gaussian data it is well known that the standardized
residuals are normally distributed while the robust distances are distributed
as a $\chi_p^2$ where $p$ is the number of continuous explanatory variables. It
is then natural to compare the standardized residuals and the leverages to some
specific quantiles of the $\mathcal{N}(0,1)$ or $\chi_p^2$ distributions in
order to detect if an individual has to be considered as an outlier and, if it
is the case, to which type of outlier it corresponds. We decide to choose here
the 2.5th and 97.5th percentiles of the $\mathcal{N}(0,1)$ distribution, and
the 95th percentile of the $\chi_p^2$ distribution. Those individuals leading
to small robust standardized residuals in absolute value and small leverages
are considered as standard individuals; those giving large standardized
residuals in absolute value and large leverages are defined as bad leverage
points; those that coincide with large standardized residuals in absolute value
but small leverages are considered as vertical outliers and, finally, those
that give small standardized residuals in absolute value but large leverages
are good leverage points.

\alert{\todo{Example will be fixed later.}
\begin{stlog}
\input{dof/4/20.log.tex}
\end{stlog}
}

Figure \textbf{???} %\ref{fig:autos_res_leverages}
clearly shows, for example, that the Cadillac Seville is a bad leverage point.
This auto is indeed associated with a very large positive robust standardized
residual and has a big leverage effect which means that its characteristics in
the space of the explanatory variables are very different from the bulk of the
data. On the other hand, the Cadillac Eldorado, the Lincoln Versaille and some
other cars have a small leverage effect---their characteristics do not appear
as different from the vast majority of the observations---but are highly
overpriced given their large positive residuals; these cars are identified as
vertical outliers. Finally some other cars such as the Plymouth Arrow or the
Volkswagen (VW Diesel) \emph{inter alia} are not outliers in terms of prices
but have characteristics very different from the others. They are thus good
leverage points. Note that even if these good leverage points do not have major
effect on the estimation of the slope parameter and the constant, they might
affect inference and shrink standard errors. It is hence important for
researchers to identify them.

\subsection{Dealing with dummies}

For this example, we use the \stcmd{fertil1.dta} data set provided provided by
\cite{Wooldridge:2001} which is a pooled cross section on more than a thousand
U.S.\ women for the even years between 1972 and 1984. These data are used to
study the relationship between women's education and fertility. We estimate a
model relating the number of children ever born to a woman (kids) to the years
of education, age, age squared, regional dummies, race dummies, the type of
environment in which the women have been reared and year dummies, using an
\stsc{MS}~estimator. Given the large number of dummy variables, it is very
likely that the subsampling algorithm described in Subsection
\ref{subsec:MM_estimation} leads to perfectly collinear subsamples. Using an
\stsc{MS}~estimator should tackle the problem.

\alert{\todo{Example will be fixed later.}
\begin{stlog}
\input{dof/4/21.log.tex}
\end{stlog}
}

The results clearly point towards a robust and statistically significant
negative relationship between education and fertility. Indeed, each additional
year of schooling is associated to an average reduction of fertility (i.e.\
number of children) equal to 0.19. To identify the outliers and recognize their
type, we again call on the graphical tool proposed by
\citet{rousseeuw:zomeren:1990}. The only difference with the previous example
is that dummy explanatory variables cannot create any leverage effect and
should therefore be treated differently from the other explanatory variables.
To estimate robust distances, we rely on the Stahel and Donoho multivariate
estimator of location and scatter (this estimator will be described in details
in Chapter \textbf{???}). The latter is a projection based estimator that allows
the pratialling out of dummy variables to calculate leverage effects. As before
we can choose a quantile above which individuals can be seen as potentially
outlying. We use here the 0.5th and 99.5th percentiles of the
$\mathcal{N}(0,1)$ distribution as cut-off points for the robust standardized
residuals, and the 99th percentile of the chi-square distribution with $p_1$
degrees of freedom, where $p_1$ is the number of continuous explanatory
variables, as cut-off point for the robust distances.
In Figure \textbf{???}%\ref{fig:fertil1_res_leverages}
, we highlight the women for which the robust standardized residuals and (or)
the robust distances exceed the cut-off points.

It is evident that individuals such as 565 have more children that one would
expect given their characteristics (which are not quite different
from the bulk of the data). On the other hand individuals such as 706, 767 or
1063 have characteristics that are very different from the vast majority of
the individuals; however their number of children is in accordance with her
characteristics. Finally individual such as 519, or 490 or 967 have
characteristics that are very different from the others. The first one has a
number of children that is much smaller than one would expect according to the
estimated model while the two others have more children than expected.

\section{Appendix 1: M estimators of location and scale}
\label{sec:robreg:appendix1}

The application of the \stsc{M}~estimation approach in the particular case of the
location-scale model (\ref{eq:location_scale_model}) leads to the \stsc{M}~estimators
of location and scale.

\subsection{M estimator of location}

An \stsc{M}~estimate $\sthat{\mu}_{\stsc{M};\rho}$ of $\mu$ is defined by
\[
    \sthat{\mu}_{\stsc{M};\rho}
    = \argmin_\mu \sum_{i=1}^{n}\rho\left(\frac{y_i-\mu}{\sthat{\sigma}}\right)
\]
where $\rho(\cdot)$ is a loss function that is positive, even (such that
$\rho(0) = 0$) and not decreasing for positive values $u$, and $\sthat{\sigma}$
is a preliminary robust estimate of $\sigma$ if this scale parameter is unknown
(the \stsc{MADN}, for example). We may also characterize
$\sthat{\mu}_{\stsc{M};\rho}$ as a solution of the following estimating
equation:
%
\begin{equation}
    \label{eq:M_location_equation}
    \sum_{i=1}^{n} \psi\left(\frac{y_i-\mu}{\sthat{\sigma}}\right) = 0
\end{equation}
%
where $\psi(u) = \rho'(u)$.

Taking $\rho(u) = u^2$, we obtain $\psi(u) = 2u$ and hence
\[
    \sum_{i=1}^{n} (y_i-\sthat{\mu}_{\stsc{M};\rho}) = 0
\]
implying that $\sthat{\mu}_{\stsc{M};\rho} = \frac{1}{n}\sum_{i=1}^{n} y_i =
\sthat{\mu}_\stsc{LS}$. Taking $\rho(u) = |u|$, we have $\psi(u) = \sign(u)$
and $\sum_{i=1}^{n} \sign(y_i-\sthat{\mu}_{\stsc{M};\rho}) = 0$; this leads to
$\sthat{\mu}_{\stsc{M};\rho} = \med\{ y_i\} = \sthat{\mu}_\stsc{LAD}$.

In general, if $\psi$ is not redescending, the equation
(\ref{eq:M_location_equation}) may be solved using the Newton-Raphson algorithm
with a robust estimate of $\mu$---the empirical median $\med\{ y_i\}$, for
instance---as initial value for $\mu$.

The influence function of the functional $T$ associated to the location
\stsc{M}~estimator $\sthat{\mu}_{\stsc{M};\rho}$ under the distribution $F_{0,1}$
of the error term $\nu$ in the location-scale model---recall here that
$F_{0,1}$ is assumed to be symmetric around zero---takes the form:
\[
    \stsc{IF}(u;T,F_{0,1}) = \frac{\psi(u)}{E_{F_{0,1}}[\psi'(\nu)]}.
\]
Consequently, the choice of the function $\rho$, and hence of the function
$\psi$, completely conditions the form of the influence function.

Moreover, it has been proven that an univariate location \stsc{M}~estimator has
an asymptotic breakdown point equal to 50\% whenever the function $\psi$ is
\emph{non decreasing}, bounded and symmetric, and the preliminary estimator of
the scale parameter $\sigma$ is the \stsc{MADN} (see
\citealp[54]{Huber:2009}).\footnote{The breakdown point of
$\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ is actually equal to the breakdown
point of the preliminary estimator of the scale parameter $\sigma$.} The
asymptotic breakdown point is null if $\psi$ is unbounded. If $\psi$ is equal
to the function $\psi_{\kappa}^{\stsc{B}}$ and hence is redescending, the
breakdown point of $\sthat{\mu}_{\stsc{M};\rho}$ is strictly smaller than 50\%
and depends upon the breakdown point of the preliminary scale estimator, upon
the constant $\kappa$, but also upon the configuration of the sample (see
\citealp[78]{maronna:etal:2006}).\footnote{Note however that it is possible to
prove that, using the \stsc{MADN} as initial scale estimator, the breakdown
point of $\sthat{\mu}_{\stsc{M};\rho_{\kappa}^{\stsc{B}}}$ is strictly greater
than 0.49 in the Gaussian case.}

\subsection{M estimator of scale}

A \stsc{M}~estimate $\sthat{\sigma}_{\stsc{M};\rho}$ of the scale parameter
$\sigma$ is defined as the solution of the equation
%
\begin{equation}
    \label{eq:M_scale_loc_equation}
    \frac{1}{n} \sum_{i=1}^{n} \rho\left(\frac{y_i-\sthat{\mu}}{\sigma}\right) = \delta
\end{equation}
%
where $\rho(\cdot)$ is a loss function that is positive, even, not decreasing
for positive values and bounded, and $\sthat{\mu}$ is a preliminary robust
estimate of $\mu$ if this location parameter is unknown (the median, for
instance). To ensure the consistency of $\sthat{\sigma}_{\stsc{M};\rho}$ for
$\sigma$, we have to take $\delta = E_{F_{0,1}}[\rho(\nu)]$. An usual choice
for the loss function $\rho$ is the Tukey-Biweight function $\rho_{\kappa}^{B}$
defined by (\ref{eq:Tukey_Biweight_function}).

The \stsc{M}~estimators of scale are translation invariant and scale
equivariant. The influence function of the functional $S$ associated to the
scale \stsc{M}~estimator $\sthat{\sigma}_{\stsc{M};\rho}$ under the
distribution $F_{0,1}$ of the error term $\nu$ of the location-scale model is
given by
\[
    \stsc{IF}(u;S,F_{0,1}) = \frac{\rho(u) -\delta}{E_{F_{0,1}}[\rho'(\nu)\nu]}.
\]
Hence, the choice of a bounded function $\rho$ implies that the influence
function is also bounded. The asymptotic breakdown point of the scale
\stsc{M}~estimator is:
\[
    \varepsilon^*(S,F_{0,1}) = \min\left(\frac{\delta}{\rho(\infty)},
        1-\frac{\delta}{\rho(\infty)}\right)
\]
which is strictly positive but not always equal to 50\%, even if $\rho$ is
bounded.

\begin{stremark}
We may try to jointly estimate $\mu$ and $\sigma$ by solving simultaneously two
equations of the type (\ref{eq:M_location_equation}) and
(\ref{eq:M_scale_loc_equation}) (see, for example, \citealp{Huber:2009},
chapter 6). This makes computations more complicated. Moreover, as explained in
\cite{maronna:etal:2006}, it generally provides for
$\sthat{\mu}_{\stsc{M};\rho}$ an asymptotic breakdown point smaller than
50\%---hence, smaller than the breakdown point attainable by using the
\stsc{MADN} as preliminary estimator of $\sigma$. Consequently, the joint
estimation of $\mu$ and $\sigma$ is not recommended, especially when the scale
parameter $\sigma$ is considered as a nuisance parameter in the location-scale
model.
\end{stremark}

\section{Appendix 2: Generalized Method of Moments (GMM) and asymptotic distributions of regression M, S and MM estimators}
\label{sec:robreg:appendix2}

\subsection{GMM estimation principle}

For simplicity, let us consider immediately the context of the regression model
(\ref{eq:linear_regr_model}). Let $y$ be the scalar dependent variable and
$\stvec{x} = (1, x_1, \dots, x_p)^t$ be the $(p+1)$-vector of covariates. We
assume here that the observations $(\stvec{x}_1, y_1), \dots, (\stvec{x}_n,
y_n)$ are generated by a \emph{stationary} and \emph{ergodic} process
$H$.\footnote{A \emph{stationary} process is a stochastic process whose joint
probability distribution does not change when shifted in time or space.
Consequently, parameters such as the mean and the variance, if they exist, also
do not change over time or position. Hence, the mean and the variance of the
process do not follow trends. Furthermore, a stochastic process is said to be
\emph{ergodic} if its statistical properties (such as its mean and variance)
can be estimated consistently from a single, sufficiently long sample
(realization) of the process.\alert{THIS FOOTNOTE IS A REPETITION}} We also
assume, to avoid too much technicalities, that there is \emph{no
autocorrelation}, that is, that the observations $(\stvec{x}_i, y_i)$, $i = 1,
\dots, n$, are \emph{independent}.\footnote{The interested reader can find very
general results, valid in presence of autocorrelation, in \cite{Croux:2003}.}

Suppose that our objective is to estimate the functional $\boldsymbol\theta =
\boldsymbol\theta(H)$ that is implicitly defined by the equation
%
\begin{equation}
    \label{Eq:GMM_moments_conditions}
    E_H[\stvec{m}(y, \stvec{x}, \boldsymbol\theta)] = \stvec{0}
\end{equation}
%
where $\stvec{m}$ is a known $k$-valued function, and $E_H[\cdot]$ denotes the
mathematical expectation with respect to $H$. If $k$ equals the dimension of
the parameter $\boldsymbol\theta$ to estimate, that is, if the number of
moments conditions specified by (\ref{Eq:GMM_moments_conditions}) coincides
with the dimension of $\boldsymbol\theta$, then the \stsc{GMM}~estimation
problem is said to be \emph{exactly-identified}. Note that it is the case in
the setting studied hereafter. The \stsc{GMM}~estimator
$\sthat{\boldsymbol\theta}_\stsc{GMM}$ of $\boldsymbol\theta$ is then simply
obtained by solving the sample analogue of (\ref{Eq:GMM_moments_conditions}),
that is,
%
\begin{equation}
    \label{Eq:GMM_equations}
    \frac{1}{n} \sum_{i=1}^{n} \stvec{m}(y_i, \stvec{x}_i,
    \sthat{\boldsymbol\theta}_\stsc{GMM}) = \stvec{0}.
\end{equation}

Under regularity conditions detailed in \citet{Hansen:1982}, the
\stsc{GMM}~estimator $\sthat{\boldsymbol\theta}_\stsc{GMM}$ defined by
(\ref{Eq:GMM_equations}) has a limiting normal distribution:
%
\begin{equation}
    \label{Eq:GMM_estimator_normality}
    \sqrt{n}(\sthat{\boldsymbol\theta}_\stsc{GMM} - \boldsymbol\theta)
    \rightarrow^d \mathcal{N}(\stvec{0},\stmat{V})
\end{equation}
%
where, in the exactly-identified case,
%
\begin{equation}
    \label{Eq:GMM_estimator_V}
    \stmat{V} = \stmat{G}^{-1} \boldsymbol\Omega (\stmat{G}^t)^{-1}
\end{equation}
%
with\footnote{Here and later, we simply write $E[\cdot]$ for $E_H[\cdot]$.}
%
\begin{equation}
    \label{Eq:GMM_estimator_G_Omega}
    \stmat{G} = E\left(\frac{\partial\stvec{m}(y, \stvec{x}, \boldsymbol\theta)}
                            {\partial\boldsymbol\theta^t}\right)
    \quad\text{and}\quad
    \boldsymbol\Omega = E\left[\stvec{m}(y,\stvec{x},\boldsymbol\theta)
                        \stvec{m}(y,\stvec{x},\boldsymbol\theta)^t\right]
\end{equation}

\subsection{M, S and MM estimators as GMM estimators}

Let us first consider the case where we estimate the parameters
$\boldsymbol\beta$ and $\sigma$ simultaneously by an \stsc{M}~estimation
procedure. Let us denote by $\rho(\cdot)$ and $\rho_0(\cdot)$ the loss
functions used for the \stsc{M}~estimation of $\boldsymbol\beta$ and $\sigma$,
respectively. Then the \stsc{M}~regression estimator
$\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ and the \stsc{M}~scale estimator
$\sthat{\sigma}_{\rho_0}$ are such that
%
\begin{equation}
    \label{Eq:GMM_equations_M}
    \begin{aligned}
        \frac{1}{n} \sum_{i=1}^{n}
            \psi\left(\frac{y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{M};\rho}}
            {\sthat{\sigma}_{\rho_0}}\right) \stvec{x}_i
        & = \stvec{0}
        \\
        \frac{1}{n}\sum_{i=1}^{n}
            \rho_0\left(\frac{y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{M};\rho}}
            {\sthat{\sigma}_{\rho_0}}\right) - \delta
        & = 0
    \end{aligned}
\end{equation}
%
where $\psi(u) = \rho'(u)$, $\delta$ is a selected constant and, using similar
notations as in the previous sections, $\sthat{\sigma}_{\rho_0} =
s_{\rho_0}(r_1(\sthat{\boldsymbol\beta}_{\stsc{M};\rho}), \dots,
r_n(\sthat{\boldsymbol\beta}_{\stsc{M};\rho}))$. This shows that the
\stsc{M}~estimator $(\sthat{\boldsymbol\beta}_{\stsc{M};\rho}^t,
\sthat{\sigma}_{\rho_0})^t$ is an exactly-identified \stsc{GMM}~estimator for
$\boldsymbol\theta = (\boldsymbol\beta^t, \sigma)^t$, with
%
\begin{equation}
    \label{Eq:GMM_moment_function_M}
    \stvec{m}(y, \stvec{x}, \boldsymbol\theta) =
    \begin{pmatrix}
        \psi\left(\frac{y-\stvec{x}^t\boldsymbol\beta}{\sigma}\right) \stvec{x}
        \\[1ex]
        \rho_0\left(\frac{y-\stvec{x}^t\boldsymbol\beta}{\sigma}\right) - \delta
    \end{pmatrix}
\end{equation}

\stsc{S}~estimators of regression and scale depend only on a chosen loss function
$\rho_0$ and on a constant $\delta$. We have defined the \stsc{S}~regression
estimator $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ as follows:
%
\begin{equation}
    \label{eq:S_min_rho0}
    \sthat{\boldsymbol\beta}_{\stsc{S};\rho_0} = \argmin_{\boldsymbol\beta}
    s_{\rho_0}(r_1(\boldsymbol\beta), \dots, r_n(\boldsymbol\beta))
\end{equation}
%
where $s_{\rho_0}$ is a measure of dispersion satisfying
\[
    \frac{1}{n} \sum_{i=1}^{n}
    \rho_0\left(\frac{r_i(\boldsymbol\beta)}{s_{\rho_0}(r_1(\boldsymbol\beta),
        \dots, r_n(\boldsymbol\beta))}\right)
    - \delta =0
    \quad\text{for all $\boldsymbol\beta \in \mathbb{R}^{p+1}$}.
\]
The scale estimator is then simply given by
\[
    \sthat{\sigma}_{\rho_0} =
    s_{\rho_0}\left(r_1(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}), \dots,
    r_n(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})\right).
\]
As previously explained, $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$
and $\sthat{\sigma}_{\rho_0}$ satisfy the first order conditions
%
\begin{equation}
    \label{Eq:GMM_equations_S}
    \begin{aligned}
        \frac{1}{n} \sum_{i=1}^{n}
            \rho_0'\left(\frac{y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}}
            {\sthat{\sigma}_{\rho_0}}\right) \stvec{x}_i
        & = \stvec{0}
        \\
        \frac{1}{n} \sum_{i=1}^{n}
            \rho_0\left(\frac{y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}}
            {\sthat{\sigma}_{\rho_0}}\right) - \delta
        & = 0.
    \end{aligned}
\end{equation}
%
Note that the equations (\ref{Eq:GMM_equations_S}) are of the same form as
(\ref{Eq:GMM_equations_M}). Hence an \stsc{S}~estimator is first-order
equivalent with an M-estimator where $\rho(\cdot) =\rho_0(\cdot)$, and has the
same asymptotic distribution (see \citealp{rousseeuw:yohai:1984}). Note however
that the function $\rho_0$ defining the \stsc{S}~estimator needs to be bounded
to get a positive breakdown point for the regression estimator. But if $\rho_0$
is bounded, $\rho_0'$ is redescending and the first set of equations in
(\ref{Eq:GMM_equations_S})---the set of equations involving $\rho_0'$---may
have multiple solutions. Therefore one usually uses (\ref{eq:S_min_rho0}) to
compute the \stsc{S}~estimate but (\ref{Eq:GMM_equations_S}) to determine its
asymptotic distribution. Actually, (\ref{Eq:GMM_equations_S}) implies that
$(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}^t,\sthat{\sigma}_{\rho_0})^t$
 is first-order equivalent with the \stsc{GMM}~estimator for $\boldsymbol\theta
= (\boldsymbol\beta^t,\sigma)^t$,
with
\[
    \stvec{m}(y,\stvec{x},\boldsymbol\theta) =
    \begin{pmatrix}
        \rho_0'\left(\frac{y-\stvec{x}^t\boldsymbol\beta}{\sigma}\right) \stvec{x}
        \\[1ex]
        \rho_0\left(\frac{y-\stvec{x}^t\boldsymbol\beta}{\sigma}\right) - \delta
    \end{pmatrix}
\]

Let us now focus on \stsc{MM}~estimators of regression. First one needs to
compute \stsc{S}~estimators
$(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}^t,\sthat{\sigma}_{\rho_0})^t$ for
a given function $\rho_0$ and a constant $\delta$. Secondly, for a given
function $\psi=\rho'$, the \stsc{MM}~estimator of regression solves
\[
    \frac{1}{n} \sum_{i=1}^{n}
    \psi\left(\frac{y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}}
        {\sthat{\sigma}_{\rho_0}}\right) \stvec{x}_i = \stvec{0}.
\]
Note that $\rho$ needs to be different from $\rho_0$, otherwise the
\stsc{MM}~estimator would be equivalent with an \stsc{S}~estimator and share
the low efficiency of the latter. In this \stsc{MM}~estimation procedure,
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$,
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ and $\sthat{\sigma}_{\rho_0}$ are
such that
%
\begin{equation}
    \label{Eq:GMM_equations_MM}
    \begin{aligned}
        \frac{1}{n} \sum_{i=1}^{n}
            \psi\left(\frac{y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}}
                {\sthat{\sigma}_{\rho_0}}\right) \stvec{x}_i
        & = \stvec{0}
        \\
        \frac{1}{n} \sum_{i=1}^{n}
            \rho_0'\left(\frac{y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}}
            {\sthat{\sigma}_{\rho_0}}\right) \stvec{x}_i
        & = \stvec{0}
        \\
        \dfrac{1}{n} \sum_{i=1}^{n}
            \rho_0\left(\frac{y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}}
                {\sthat{\sigma}_{\rho_0}}\right) - \delta
        & = 0.
    \end{aligned}
\end{equation}
%
Defining $\boldsymbol\theta =
(\boldsymbol\beta^t,\boldsymbol\beta_0^t,\sigma)^t$, where the first parameter
$\boldsymbol\beta$ will be estimated by
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ and the latter two by
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ and $\sthat{\sigma}_{\rho_0}$,
equations (\ref{Eq:GMM_equations_MM}) show that
$(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}^t,
\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}^t,\sthat{\sigma}_{\rho_0})^t$ is
first-order equivalent with the \stsc{GMM}~estimator for $\boldsymbol\theta$,
with
\[
    \stvec{m}(y,\stvec{x},\boldsymbol\theta) =
    \begin{pmatrix}
        \psi\left(\frac{y-\stvec{x}^t\boldsymbol\beta}{\sigma}\right) \stvec{x}
        \\[1ex]
        \rho_0'\left(\frac{y-\stvec{x}^t\boldsymbol\beta_0}{\sigma}\right) \stvec{x}
        \\[1ex]
        \rho_0\left(\frac{y-\stvec{x}^t\boldsymbol\beta_0}{\sigma}\right) - \delta
    \end{pmatrix}
\]
Using the generic notations $u_0 = (y-\stvec{x}^t\boldsymbol{\beta}_0)/\sigma$
and $u = (y-\stvec{x}^t\boldsymbol\beta)/\sigma$, the moment function
$\stvec{m}(y,\stvec{x},\boldsymbol\theta)$ takes the simpler form
\[
    \stvec{m}(y,\stvec{x},\boldsymbol\theta) =
    \begin{pmatrix}
        \psi(u) \stvec{x}       \\
        \rho_0'(u_0) \stvec{x}  \\
        \rho_0(u_0) - \delta
    \end{pmatrix}
\]
or still more shortly,
%
\begin{equation}
    \label{Eq:GMM_moment_function_MM}
    \stvec{m}(y,\stvec{x},\boldsymbol\theta) =
    \begin{pmatrix}
        \psi\stvec{x}       \\
        \rho_0'\stvec{x}    \\
        \rho_0-\delta
    \end{pmatrix}
\end{equation}
%
if we simply replace $\psi(u)$ by $\psi$, $\rho_0(u_0)$ by $\rho_0$, and
$\rho_0'(u_0)$ by $\rho_0'$. This compact notation for the moment function will
be more practice to use in the sequel.

\subsection{Asymptotic variance matrix of an MM estimator}

\subsubsection{Assumption A1: The observations are generated by a stationary and ergodic process and are independent}

The first-order equivalence of
$(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}^t,
\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}^t, \sthat{\sigma}_{\rho_0})^t$ with
a \stsc{GMM}~estimator for $\boldsymbol\theta = (\boldsymbol\beta^t,
\boldsymbol\beta_0^t, \sigma)^t$ allows us to conclude that, if the
observations $(\stvec{x}_1, y_1), \dots, (\stvec{x}_n, y_n)$ are generated by a
\emph{stationary} and \emph{ergodic} process, and are \emph{independent}
(Assumption~A1),\footnote{This Assumption~A1 coincides with Assumption~A in
Section~\ref{subsec:asymptotic_distr_M_S_MM_estimators}. We add here an number
to the letter “A” in order to clearly distinguish the various assumptions we
will consider in the sequel of this appendix.}
\[
    \sqrt{n}\left(
    \begin{bmatrix}
        \sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}\\
        \sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}\\
        \sthat{\sigma}_{\rho_0}
    \end{bmatrix}
    -
    \begin{bmatrix}
        \boldsymbol\beta\\
        \boldsymbol\beta_0\\
        \sigma
    \end{bmatrix}
    \right)  \rightarrow^{d} \mathcal{N}(\stvec{0}, \stmat{V}_\stsc{MM})
\]
where
\[
    \stmat{V}_\stsc{MM} = \stmat{G}_\stsc{MM}^{-1} \boldsymbol\Omega_\stsc{MM}
        (\stmat{G}_\stsc{MM}^t)^{-1}
\]
with the matrices $\stmat{G}_\stsc{MM}$ and $\boldsymbol\Omega_\stsc{MM}$
obtained by applying relations (\ref{Eq:GMM_estimator_G_Omega}) to the moment
function (\ref{Eq:GMM_moment_function_MM}):
\[
    \stmat{G}_\stsc{MM} = -\frac{1}{\sigma} E
    \begin{pmatrix}
        \psi'\stvec{x}\stvec{x}^t & \stvec{0}                    & \psi'u\stvec{x}      \\
        \stvec{0}                 & \rho_0''\stvec{x}\stvec{x}^t & \rho_0''u_0\stvec{x} \\
        \stvec{0}                 & \stvec{0}                    & \rho_0'u_0
    \end{pmatrix}
\]
and
\[
    \boldsymbol\Omega_\stsc{MM} = E
    \begin{pmatrix}
        \psi^2\stvec{x}\stvec{x}^t      & \psi\rho_0'\stvec{x}\stvec{x}^t & \psi\rho_0\stvec{x}   \\
        \psi\rho_0'\stvec{x}\stvec{x}^t & (\rho_0')^2\stvec{x}\stvec{x}^t & \rho_0\rho_0'\stvec{x}\\
        \psi\rho_0\stvec{x}^t           & \rho_0\rho_0'\stvec{x}^t        & \rho_0^2-\delta^2
    \end{pmatrix}
\]
In particular, this result establishes the consistency of the \stsc{MM}~regression
estimator $\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$.
Moreover, using the upper left $(p+1) \times (p+1)$ submatrix of
$\stmat{V}_\stsc{MM}$, we obtain that the asymptotic variance of
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ is equal to
%
\[
    \mathrm{Avar}_1 (\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})
        = \frac{1}{n} \left[\stmat{A} E(\psi^2\stvec{x}\stvec{x}^t) \stmat{A}
        - \stmat{a} E(\psi\rho_0\stvec{x}^t) \stmat{A}
        - \stmat{A} E(\psi\rho_0\stvec{x}) \stmat{a}^t
        + E(\rho_0^2-\delta^2) \stmat{a}\stmat{a}^t \right]
\]
%
where
\[
    \stmat{A} = \sigma E(\psi'\stvec{x}\stvec{x}^t)^{-1}
    \quad\text{and}\quad
    \stmat{a} = \stmat{A}\frac{E(\psi'u\stvec{x})}{E(\rho_0'u_0)}.
\]

This expression of
$\mathrm{Avar}_1(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$ is then
estimated by its empirical counterpart
$\sthat{\mathrm{Avar}}_1(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$, by
applying the following two rules:
\begin{enumerate}
    \item Replace, in $u$ and $u_0$, the parameters $\boldsymbol\beta$,
    $\boldsymbol\beta_0$ and $\sigma$ by the estimates
    $\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$,
    $\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ and $\sthat{\sigma}_{\rho_0}$.

    \item Replace $E(\cdot)$ by $\frac{1}{n}\sum_{i=1}^{n}(\cdot)$.
\end{enumerate}

For example, the first term of
$\sthat{\mathrm{Avar}}_1(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$ is
given by
\[
    \frac{1}{n} \left(\sthat{\stmat{A}}\left[
        \frac{1}{n}\sum_{i=1}^{n}\psi\left(
            \frac{y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}}
            {\sthat{\sigma}_{\rho_0}}
            \right)^{\!\!2} \stvec{x}_i\stvec{x}_i^t
    \right] \sthat{\stmat{A}}\right)
\]
with
\[
    \sthat{\stmat{A}} = \sthat{\sigma}_{\rho_0}\left[
        \frac{1}{n}\sum_{i=1}^{n}\psi'\left(
            \frac{y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}}
            {\sthat{\sigma}_{\rho_0}}
    \right) \stvec{x}_i\stvec{x}_i^t\right]^{-1}.
\]

Using standard asymptotic arguments, it can be shown that
$\sthat{\mathrm{Avar}}_1(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$ is
a consistent estimate of
$\mathrm{Avar}_1(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$. From
$\sthat{\mathrm{Avar}}_1(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$,
standard errors for the regression coefficients are obtained in the usual way:
for $j = 0, 1, \dots, p$,
\[
    \mathrm{SE}\left(\big[\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}\big]_j\right)
    = \sqrt{\big[\sthat{\mathrm{Avar}}_1(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})\big]_{jj}}.
\]

Moreover, the estimate
$\sthat{\mathrm{Avar}}_1(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$ of
the asymptotic variance
$\mathrm{Avar}_1(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$ is robust
with respect to bad leverage points and vertical outliers. Indeed, if there are
observations yielding large residuals with respect to the robust \stsc{MM}~fit,
then
$\psi[(y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})/\sthat{\sigma}_{\rho_0}]$
has a small value when $\psi$ is a redescending function.\footnote{Recall that,
if $\psi$ is redescending, it has the property to be equal to zero for large
arguments.} Hence, if there are bad leverage points in the sample, then their
$\stvec{x}_i$-value is large, but at the same time
$\psi[(y_i-\stvec{x}_i^t\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})/\sthat{\sigma}_{\rho_0}]$
will be zero. This explains intuitively why vertical outliers and bad leverage
points have only a limited influence on the estimate
$\sthat{\mathrm{Avar}}_1(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$.

\subsubsection{Assumption A2: Absence of heteroskedasticity}

A simplification of the asymptotic variance of
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ occurs when, in addition to
Assumption~A1, we assume that there is \emph{no heteroskedasticity}, that is,
we assume that the processes $\stvec{x}_i$ and $(u_i,u_{0i})$ are independent
(Assumption~A2). In that case, the asymptotic variance of
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ becomes
\[
\begin{split}
    \mathrm{Avar}_{12}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})
    & = \frac{1}{n} \big[ E(\psi^2)\stmat{A}_2 E(\stvec{x}\stvec{x}^t) \stmat{A}_2
      - E(\psi\rho_0) \stmat{a}_2 E(\stvec{x}^t) \stmat{A}_2
    \\
    & \qquad
      - E(\psi\rho_0) \stmat{A}_2 E(\stvec{x}) \stmat{a}_2^t
      + E(\rho_0^2-\delta^2)  \stmat{a}_2\stmat{a}_2^t \big]
\end{split}
\]
where
\[
    \stmat{A}_2 = \sigma \frac{E(\stvec{x}\stvec{x}^t)^{-1}}{E(\psi')}
    \quad\text{and}\quad
    \stmat{a}_2 = \stmat{A}_2 \frac{E(\psi'u) E(\stvec{x})}{E(\rho_0'u_0)}.
\]
Taking the empirical counterpart yields
$\sthat{\mathrm{Avar}}_{12}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$.
However, \citet{Croux:2003} do advise against the use of this variance matrix
estimator in practice, even when assumptions A1 and A2 hold. The reason is that
this estimator will not be robust with respect to (good and bad) leverage
points. Indeed, $\sthat{\stmat{A}}_2$, for example, is proportional to the
inverse of an empirical second moment matrix of the observations $\stvec{x}_i$.
Leverage points are outlying in the covariates' space, and will then have a
strong influence on $\sthat{\stmat{A}}_2$. This can even lead
$\sthat{\mathrm{Avar}}_{12}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$
to break down, where breakdown of a variance matrix estimator means that the
latter has a determinant close to zero or enormously large.

\subsubsection{Assumption A3: The distribution of the error terms is symmetric around zero}

A condition often imposed in the literature is that the distribution of $u_i =
(y_i-\stvec{x}_i^t\boldsymbol\beta)/\sigma$, given $\stvec{x}_i$, is symmetric
(Assumption~A3). If this condition is met, the regression parameter estimator
and the estimator of residual scale are asymptotically independent, and the
different expressions simplify considerably, due to the fact that $\stmat{a} =
\stvec{0}$.

Under Assumptions A1 and A3, the asymptotic variance of
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ becomes
\[
    \mathrm{Avar}_{13}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})
    = \frac{1}{n} \stmat{A} E(\psi^2\stvec{x}\stvec{x}^t) \stmat{A}
    = \frac{\sigma^2}{n} E(\psi'\stvec{x}\stvec{x}^t)^{-1}
        E(\psi^2\stvec{x}\stvec{x}^t)
        E(\psi'\stvec{x}\stvec{x}^t)^{-1}.
\]

The empirical counterpart of the latter expression,
$\sthat{\mathrm{Avar}}_{13}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$,
is an estimate of the asymptotic variance of
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ that is robust against
vertical outliers and bad leverage points. But it relies on symmetry of the
errors distribution, a quite strong assumption. A simulation study in
\cite{Croux:2003} shows that, even when symmetry is present, there is no gain
in using $\sthat{\mathrm{Avar}}_{13}$ compared to $\sthat{\mathrm{Avar}}_1$:
the authors of \cite{Croux:2003} then recommend to use
$\sthat{\mathrm{Avar}}_1$ in any case.

When all of Assumptions A1, A2 and A3 hold, then
$\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho}$ has asymptotic variance
\[
    \mathrm{Avar}_{123}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})
    = \frac{\sigma^2}{n} \frac{E(\psi^2)}{E(\psi')^2}
      E(\stvec{x}\stvec{x}^t)^{-1}.
\]
This corresponds to the expression for the variance of the \stsc{MM} regression
estimator that was derived in \cite{yohai:1987}. The empirical counterpart
$\sthat{\mathrm{Avar}}_{123}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$
is an estimate of this asymptotic variance that, as
$\sthat{\mathrm{Avar}}_{12}(\sthat{\boldsymbol\beta}_{\stsc{MM};\rho_0,\rho})$,
lacks robustness with respect to leverage points.

\subsection{Asymptotic variance matrix of an S estimator}

If Assumption~A1 holds, the asymptotic variance matrix of
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ is simply derived from the central
$(p+1) \times (p+1)$ submatrix of $\stmat{V}_\stsc{MM}$ (cf.
(\ref{eq:V_MM}), (\ref{eq:G_MM}) and (\ref{eq:Omega_MM})):
\[
\begin{split}
    \mathrm{Avar}_1(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})
    & = \frac{1}{n} \big[
        \stmat{A}_{\stsc{S}} E[(\rho_0')^2\stvec{x}\stvec{x}^t] \stmat{A}_{\stsc{S}}
        - \stmat{a}_{\stsc{S}} E(\rho_0\rho_0'\stvec{x}^t) \stmat{A}_{\stsc{S}}
    \\
    & \qquad
        - \stmat{A}_{\stsc{S}} E(\rho_0\rho_0'\stvec{x}) \stmat{a}_{\stsc{S}}^t
        + E(\rho_0^2-\delta^2) \stmat{a}_{\stsc{S}}\stmat{a}_{\stsc{S}}^t
    \big]
\end{split}
\]
where
\[
    \stmat{A}_{\stsc{S}} = \sigma E(\rho_0''\stvec{x}\stvec{x}^t)^{-1}
    \quad\text{and}\quad
    \stmat{a}_{\stsc{S}} = \stmat{A}_{\stsc{S}}
        \frac{E(\rho_0''u_0\stvec{x})}{E(\rho_0'u_0)}.
\]
If, in addition, Assumption~A2 holds, then the asymptotic variance matrix of
$\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0}$ takes the form
\[
\begin{split}
    \mathrm{Avar}_{12}(\sthat{\boldsymbol\beta}_{\stsc{S};\rho_0})
    & = \frac{\sigma^2}{n}\frac{E\left[(\rho_0')^2\right]}
            {E(\rho_0'')^2} E(\stvec{x}\stvec{x}^t)^{-1}
        + \frac{\sigma^2}{n} \frac{E(\rho_0''u_0)}{E(\rho_0'')^2
              E(\rho_0'u_0)}
    \\
    & \qquad
    \times \left(\frac{E(\rho_0''u_0) E(\rho_0^2-\delta^2)}{E(\rho_0'u_0)}
        - 2 E(\rho_0\rho_0')\right)
    \\
    & \qquad
    \times E(\stvec{x}\stvec{x}^t)^{-1} E(\stvec{x})
        E(\stvec{x}^t) E(\stvec{x}\stvec{x}^t)^{-1}.
\end{split}
\]
Under Assumption~A3, the expressions are the same as those for the
\stsc{MM}~estimator, with $\psi$ replaced by $\rho_0'$.

\subsection{Asymptotic variance matrix of an M estimator}

Here the expressions are less explicit. Under Assumption A1, the asymptotic
variance of $\sthat{\boldsymbol\beta}_{\stsc{M};\rho}$ is derived from the
upper left $(p+1) \times (p+1)$ block of $\stmat{G}_\stsc{M}^{-1}
\boldsymbol\Omega_\stsc{M} (\stmat{G}_\stsc{M}^t)^{-1}$ where
\[
    \stmat{G}_\stsc{M} =
    E\left(\frac{\partial\stvec{m}(y,\stvec{x},\boldsymbol\theta)}
                {\partial\boldsymbol\theta^t}\right)
    \quad\text{and}\quad
    \boldsymbol\Omega_\stsc{M} =
    E\left[\stvec{m}(y,\stvec{x},\boldsymbol\theta)
           \stvec{m}(y,\stvec{x},\boldsymbol\theta)^t\right]
\]
with $\stvec{m}(y,\stvec{x},\boldsymbol\theta)$ given by
(\ref{Eq:GMM_moment_function_M}). Defining $u =
(y-\stvec{x}^t\boldsymbol\beta)/\sigma$, and denoting $\rho'(u) =
\psi(u)$ by $\psi$ and $\rho_0(u)$ by $\rho_0$, we have
\[
    \stmat{G}_\stsc{M} = - \frac{1}{\sigma} E
    \begin{pmatrix}
        \psi'\stvec{x}\stvec{x}^t & \psi'u\stvec{x}\\
        \rho_0'\stvec{x}^t        & \rho_0'u
    \end{pmatrix}
    \quad\text{and}\quad
    \boldsymbol\Omega_\stsc{M} = E
    \begin{pmatrix}
        \psi^2\stvec{x}\stvec{x}^t & \psi\rho_0\stvec{x}\\
        \psi\rho_0\stvec{x}^t      & \rho_0^2-\delta^2
    \end{pmatrix}.
\]

If in addition Assumption~A2 holds, then
\[
    \stmat{G}_\stsc{M} = -\frac{1}{\sigma}
    \begin{pmatrix}
        E(\psi') E(\stvec{x}\stvec{x}^t) & E(\psi'u) E(\stvec{x}) \\
        E(\rho_0') E(\stvec{x}^t)        & E(\rho_0'u)
    \end{pmatrix}
\]
and
\[
    \boldsymbol\Omega_\stsc{M} =
    \begin{pmatrix}
        E(\psi^2) E(\stvec{x}\stvec{x}^t) & E(\psi\rho_0) E(\stvec{x}) \\
        E(\psi\rho_0) E(\stvec{x}^t)      & E(\rho_0^2) - \delta^2
    \end{pmatrix}
\]
Under Assumption~A3, the expressions of
$\mathrm{Avar}_{13}(\sthat{\boldsymbol\beta}_{\stsc{M};\rho})$ and
$\mathrm{Avar}_{123}(\sthat{\boldsymbol\beta}_{\stsc{M};\rho})$ are
exactly similar to those for the \stsc{MM}~estimator.

\endinput