%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Contents: Math typesetting with LaTeX
% $Id: math.tex 534 2015-04-09 13:03:16Z oetiker $
%
% Changes by Stefan M. Moser: 2008/10/22
%
% -Section 2: "Single Equations": added comment about preference of
%  equation* over \[
% -Replaced (almost) all examples with \[ by equation*
% -New section 4: "Single Equations that are Too Long: multline"
% -New section 5: "Multiple Equations"
% -Section 6: "Arrays and Matrices": made a full section and added
%  some material
% -Section 9: "Theorems, Lemmas, ...": added a subsection about proofs
%  with new material
%
% Other Changes:
% -in lshort.sty:
%    *example environment adapted: changed in three places
%     \textwidth by \linewidth. This is necessary for
%     example-environment within a itemize-list.
%    *added \RequirePackage[retainorgcmds]{IEEEtrantools}
%
% THINGS TO DO:
% -adapt typesetting of new sections to rest of lshort, including all
%  the usual commands used so far. In particular, I guess we have to
%  get rid of the \verb-commands everywhere
% -include index-commands
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Typesetting Mathematical Formulae}

\begin{intro}
  Now you are ready! In this chapter, we will attack the main strength
  of \TeX{}: mathematical typesetting. But be warned, this chapter
  only scratches the surface. While the things explained here are
  sufficient for many people, don't despair if you can't find a
  solution to your mathematical typesetting needs here. It is highly likely
  that your problem is addressed in \AmS-\LaTeX{}.
\end{intro}

\section{The \texorpdfstring{\AmS}{AMS}-\LaTeX{} bundle}

If you want to typeset (advanced) \wi{mathematics}, you should
use \AmS-\LaTeX{}. The \AmS-\LaTeX{} bundle is a collection of packages and classes for
mathematical typesetting. We will mostly deal with the \pai{amsmath} package
which is a part of the bundle. \AmS-\LaTeX{} is produced by The \emph{\wi{American Mathematical Society}}
and it is used extensively for mathematical typesetting. \LaTeX{} itself does provide
some basic features and environments for mathematics, but they are limited (or
maybe it's the other way around: \AmS-\LaTeX{} is \emph{unlimited}!) and
in some cases inconsistent.

\AmS-\LaTeX{} is a part of the required distribution and is provided
with all recent \LaTeX{} distributions.\footnote{If yours is missing it, go to
  \CTAN|pkg/amslatex|.} In this chapter, we assume
  \pai{amsmath} is loaded in the preamble; \verb|\usepackage{amsmath}|.

\section{Single Equations}

A mathematical formula can be typeset in-line within a paragraph (\emph{\wi{text style}}), or the paragraph can be broken and the formula typeset separately
(\emph{\wi{display style}}). Mathematical \wi{equation}s
\emph{within} a paragraph are entered \index{$@\texttt{\$}} %$
between \texttt{\$} and \texttt{\$}:
\begin{example}
Add $a$ squared and $b$ squared
to get $c$ squared. Or, using
a more mathematical approach:
$a^2 + b^2 = c^2$
\end{example}
\begin{example}
\TeX{} is pronounced as
$\tau\epsilon\chi$\\[5pt]
100~m$^{3}$ of water\\[5pt]
This comes from my $\heartsuit$
\end{example}

If you want your larger equations to be set apart
from the rest of the paragraph, it is preferable to \emph{display} them
rather than to break the paragraph apart.
To do this, you enclose them between \verb|\begin{|\ei{equation}\verb|}| and
\verb|\end{equation}|.\footnote{This is an \textsf{amsmath} command. If you don't
have access to the package for some obscure reason, you can use \LaTeX's own
\ei{displaymath} environment instead.} You can then \ci{label} an equation number and refer to
it somewhere else in the text by using the \ci{eqref} command. If you want to
name the equation something specific, you \ci{tag} it instead.
\begin{example}
Add $a$ squared and $b$ squared
to get $c$ squared. Or, using
a more mathematical approach
 \begin{equation}
   a^2 + b^2 = c^2
 \end{equation}
Einstein says
 \begin{equation}
   E = mc^2 \label{clever}
 \end{equation}
He didn't say
 \begin{equation}
  1 + 1 = 3 \tag{dumb}
 \end{equation}
This is a reference to
\eqref{clever}.
\end{example}

If you don't want \LaTeX{} to number the equations, use the starred
version of \texttt{equation} using an asterisk, \ei{equation*}, or even easier, enclose the
equation in \ci{[} and \ci{]}:\footnote{\index{equation!\textsf{amsmath}}
  \index{equation!\LaTeX{}}This is again from \textsf{amsmath}. Standard \LaTeX{}'s has only the \texttt{equation} environment without the star.}
\begin{example}
Add $a$ squared and $b$ squared
to get $c$ squared. Or, using
a more mathematical approach
 \begin{equation*}
   a^2 + b^2 = c^2
 \end{equation*}
or you can type less for the
same effect:
 \[ a^2 + b^2 = c^2 \]
\end{example}
While \ci{[} is short and sweet, it does not allow switching between numbered and not numbered style as easily as
\ei{equation} and \ei{equation*}.

Note the difference in typesetting style between \wi{text style} and \wi{display style}
equations:
\begin{example}
This is text style:
$\lim_{n \to \infty}
 \sum_{k=1}^n \frac{1}{k^2}
 = \frac{\pi^2}{6}$.
And this is display style:
 \begin{equation}
  \lim_{n \to \infty}
  \sum_{k=1}^n \frac{1}{k^2}
  = \frac{\pi^2}{6}
 \end{equation}
\end{example}

In text style, enclose tall or deep math expressions or sub
expressions in \ci{smash}. This makes \LaTeX{} ignore the height of
these expressions. This keeps the line spacing even.

\begin{example}
A $d_{e_{e_p}}$ mathematical
expression  followed by a
$h^{i^{g^h}}$ expression. As
opposed to a smashed
\smash{$d_{e_{e_p}}$} expression
followed by a
\smash{$h^{i^{g^h}}$} expression.
\end{example}

\subsection{Math Mode}

There are also differences between \emph{\wi{math mode}} and \emph{text mode}. For
example, in \emph{math mode}:

\begin{enumerate}

\item \index{spacing!math mode} Most spaces and line breaks do not have any significance, as all spaces
are either derived logically from the mathematical expressions, or
have to be specified with special commands such as \ci{,}, \ci{quad} or
\ci{qquad} (we'll get back to that later, see section~\ref{sec:math-spacing}).

\item Empty lines are not allowed. Only one paragraph per formula.

\item Each letter is considered to be the name of a variable and will be
typeset as such. If you want to typeset normal text within a formula
(normal upright font and normal spacing) then you have to enter the
text using the \verb|\text{...}| command (see also section \ref{sec:fontsz} on
page \pageref{sec:fontsz}).

\end{enumerate}
\begin{example}
$\forall x \in \mathbf{R}:
 \qquad x^{2} \geq 0$
\end{example}
\begin{example}
$x^{2} \geq 0\qquad
 \text{for all }x\in\mathbf{R}$
\end{example}

Mathematicians can be very fussy about which symbols are used:
it would be conventional here to use the `\wi{blackboard bold}' font,
\index{bold symbols} which is obtained using \ci{mathbb} from the
package \pai{amssymb}.\footnote{\pai{amssymb} is not a part
  of the \AmS-\LaTeX{} bundle, but it is perhaps still a part of your \LaTeX{}
  distribution. Check your distribution
  or go to \texttt{CTAN:/fonts/amsfonts/latex/} to obtain it.}
\ifx\mathbb\undefined\else
The last example becomes
\begin{example}
$x^{2} \geq 0\qquad
 \text{for all } x
 \in \mathbb{R}$
\end{example}
\fi
See Table~\ref{mathalpha} on page~\pageref{mathalpha} and
Table~\ref{mathfonts} on page~\pageref{mathfonts} for more math fonts.

\section{Building Blocks of a Mathematical Formula}

In this section, we describe the most important commands used in mathematical
typesetting. Most of the commands in this section will not require
\textsf{amsmath} (if they do, it will be stated clearly), but load it anyway.

\textbf{Lowercase \wi{Greek letters}} are entered as \verb|\alpha|,
 \verb|\beta|, \verb|\gamma|, \ldots, uppercase letters
are entered as \verb|\Gamma|, \verb|\Delta|, \ldots\footnote{There is no
  uppercase Alpha, Beta etc. defined in \LaTeXe{} because it looks the same as a
  normal roman A, B\ldots{}}

Take a look at Table~\ref{greekletters} on page~\pageref{greekletters} for a
list of Greek letters.
\begin{example}
$\lambda,\xi,\pi,\theta,
 \mu,\Phi,\Omega,\Delta$
\end{example}

\textbf{Exponents, Superscripts and Subscripts} can be specified using\index{exponent}\index{subscript}\index{superscript}
the \verb|^|\index{^@\verb"|^"|} and the \verb|_|\index{_@\verb"|_"|} characters.
Most math mode commands act only on the next character, so if you
want a command to affect several characters, you have to group them
together using curly braces: \verb|{...}|.

Table~\ref{binaryrel} on page \pageref{binaryrel} lists a lot of binary
relations like $\subseteq$ and $\perp$.

\begin{example}
$p^3_{ij} \qquad
 m_\text{Knuth}\qquad
\sum_{k=1}^3 k \\[5pt]
 a^x+y \neq a^{x+y}\qquad
 e^{x^2} \neq {e^x}^2$
\end{example}

The \textbf{\wi{square root}} is entered as \ci{sqrt}; the
$n^\text{th}$ root is generated with \verb|\sqrt[|$n$\verb|]|. The size of
the root sign is determined automatically by \LaTeX. If just the sign
is needed, use \verb|\surd|.

See various kinds of arrows like $\hookrightarrow$ and $\rightleftharpoons$ on
Table~\ref{tab:arrows} on page \pageref{tab:arrows}.
\begin{example}
$\sqrt{x} \Leftrightarrow x^{1/2}
 \quad \sqrt[3]{2}
 \quad \sqrt{x^{2} + \sqrt{y}}
 \quad \surd[x^2 + y^2]$
\end{example}

\index{dots!three}
\index{vertical!dots}
\index{horizontal!dots}
While the \textbf{\wi{dot}} sign to indicate
the multiplication operation is normally left out, it is sometimes written
to help the eye in grouping a formula.
Use \ci{cdot} to typeset a single centered dot. \ci{cdots} is
three centered \textbf{\wi{dots}} while \ci{ldots} sets the dots low (on the
baseline). Besides that, there are \ci{vdots} for
vertical and \ci{ddots} for \wi{diagonal dots}. There are more examples in
section~\ref{sec:arraymat}.
\begin{example}
$\Psi = v_1 \cdot v_2
 \cdot \ldots \qquad
 n! = 1 \cdot 2
 \cdots (n-1) \cdot n$
\end{example}

The commands \ci{overline} and \ci{underline} create
\textbf{horizontal lines} directly over or under an expression:
\index{horizontal!line} \index{line!horizontal}
\begin{example}
$0.\overline{3} =
 \underline{\underline{1/3}}$
\end{example}

The commands \ci{overbrace} and \ci{underbrace} create
long \textbf{horizontal braces} over or under an expression:
\index{horizontal!brace} \index{brace!horizontal}
\begin{example}
$\underbrace{\overbrace{a+b+c}^6
 \cdot \overbrace{d+e+f}^7}
 _\text{meaning of life} = 42$
\end{example}

\index{mathematical!accents} To add mathematical accents such as \textbf{small
arrows} or \textbf{\wi{tilde}} signs to variables, the commands
given in Table~\ref{mathacc} on page~\pageref{mathacc} might be useful.  Wide hats and
tildes covering several characters are generated with \ci{widetilde}
and \ci{widehat}. Notice the difference between \ci{hat} and \ci{widehat} and the placement of
\ci{bar} for a variable with subscript. The \wi{apostrophe} mark
\verb|'|\index{'@\verb"|'"|} gives a \wi{prime}:
% a dash is --
\begin{example}
$f(x) = x^2 \qquad f'(x)
 = 2x \qquad f''(x) = 2\\[5pt]
 \hat{XY} \quad \widehat{XY}
 \quad \bar{x_0} \quad \bar{x}_0$
\end{example}

\textbf{Vectors}\index{vectors} are often specified by adding small
\wi{arrow symbols} on the tops of variables. This is done with the
\ci{vec} command. The two commands \ci{overrightarrow} and
\ci{overleftarrow} are useful to denote the vector from $A$ to $B$:
\begin{example}
$\vec{a} \qquad
 \vec{AB} \qquad
 \overrightarrow{AB}$
\end{example}

Names of functions are often typeset in an upright
font, and not in italics as variables are, so \LaTeX{} supplies the
following commands to typeset the most common function names:
\index{mathematical!functions}

\begin{tabular}{llllll}
\ci{arccos} &  \ci{cos}  &  \ci{csc} &  \ci{exp} &  \ci{ker}    & \ci{limsup} \\
\ci{arcsin} &  \ci{cosh} &  \ci{deg} &  \ci{gcd} &  \ci{lg}     & \ci{ln}     \\
\ci{arctan} &  \ci{cot}  &  \ci{det} &  \ci{hom} &  \ci{lim}    & \ci{log}    \\
\ci{arg}    &  \ci{coth} &  \ci{dim} &  \ci{inf} &  \ci{liminf} & \ci{max}    \\
\ci{sinh}   & \ci{sup}   &  \ci{tan}  & \ci{tanh}&  \ci{min}    & \ci{Pr}     \\
\ci{sec}    & \ci{sin} \\
\end{tabular}

\begin{example}
\begin{equation*}
  \lim_{x \rightarrow 0}
  \frac{\sin x}{x}=1
\end{equation*}
\end{example}

For functions missing from the list, use the \ci{DeclareMathOperator}
command. There is even a starred version for functions with limits.
This command works only in the preamble so the commented lines in the
example below must be put into the preamble.

\begin{example}
%\DeclareMathOperator{\argh}{argh}
%\DeclareMathOperator*{\nut}{Nut}
\begin{equation*}
  3\argh = 2\nut_{x=1}
\end{equation*}
\end{example}

For the \wi{modulo function}, there are two commands: \ci{bmod} for the
binary operator ``$a \bmod b$'' and \ci{pmod}
for expressions
such as ``$x\equiv a \pmod{b}$:''
\begin{example}
$a\bmod b \\
 x\equiv a \pmod{b}$
\end{example}

A built-up \textbf{\wi{fraction}} is typeset with the
\ci{frac}\verb|{...}{...}| command. In in-line equations, the fraction is shrunk to
fit the line. This style is obtainable in display style with \ci{tfrac}. The
reverse, i.e.\ display style fraction in text, is made with \ci{dfrac}.
Often the slashed form $1/2$ is preferable, because it looks better
for small amounts of `fraction material:'
\begin{example}
In display style:
\begin{equation*}
  3/8 \qquad \frac{3}{8}
  \qquad \tfrac{3}{8}
\end{equation*}
\end{example}

\begin{example}
In text style:
$1\frac{1}{2}$~hours \qquad
$1\dfrac{1}{2}$~hours
\end{example}

Here the \ci{partial} command for \wi{partial derivative}s is used:
\begin{example}
\begin{equation*}
  \sqrt{\frac{x^2}{k+1}}\qquad
  x^\frac{2}{k+1}\qquad
  \frac{\partial^2f}
  {\partial x^2}
\end{equation*}
\end{example}

To typeset \wi{binomial coefficient}s or similar structures, use
the command \ci{binom} from \pai{amsmath}:
\begin{example}
Pascal's rule is
\begin{equation*}
 \binom{n}{k} =\binom{n-1}{k}
 + \binom{n-1}{k-1}
\end{equation*}
\end{example}

For \wi{binary relations} it may be useful to stack symbols over each other.
\ci{stackrel}\verb|{#1}{#2}| puts the symbol given
in \verb|#1| in superscript-like size over \verb|#2| which
is set in its usual position.
\begin{example}
\begin{equation*}
 f_n(x) \stackrel{*}{\approx} 1
\end{equation*}
\end{example}

The \textbf{\wi{integral operator}} is generated with \ci{int}, the
\textbf{\wi{sum operator}} with \ci{sum}, and the \textbf{\wi{product operator}}
with \ci{prod}. The upper and lower limits are specified with~\verb|^|
and~\verb|_| like subscripts and superscripts:
\begin{example}
\begin{equation*}
\sum_{i=1}^n \qquad
\int_0^{\frac{\pi}{2}} \qquad
\prod_\epsilon
\end{equation*}
\end{example}

To get more control over the placement of indices in complex
expressions, \pai{amsmath} provides the \ci{substack} command:
\begin{example}
\begin{equation*}
\sum^n_{\substack{0<i<n \\
        j\subseteq i}}
   P(i,j) = Q(i,j)
\end{equation*}
\end{example}

\LaTeX{} provides all sorts of symbols for \textbf{\wi{bracketing}} and other
\textbf{\wi{delimiters}} (e.g.~$[\;\langle\;\|\;\updownarrow$).
Round and square brackets can be entered with the corresponding keys and
curly braces with \verb|\{|, but all other delimiters are generated with
special commands (e.g.~\verb|\updownarrow|).
\begin{example}
\begin{equation*}
{a,b,c} \neq \{a,b,c\}
\end{equation*}
\end{example}

If you put \ci{left} in front of an opening delimiter and
\ci{right} in front of a closing delimiter, \LaTeX{} will automatically
determine the correct size of the delimiter. Note that you must close
every \ci{left} with a corresponding \ci{right}. If you
don't want anything on the right, use the invisible ``\ci{right.}'':
\begin{example}
\begin{equation*}
1 + \left(\frac{1}{1-x^{2}}
    \right)^3 \qquad
\left. \ddagger \frac{~}{~}\right)
\end{equation*}
\end{example}

In some cases it is necessary to specify the correct size of a
mathematical delimiter\index{mathematical!delimiter} by hand,
which can be done using the commands \ci{big}, \ci{Big}, \ci{bigg} and
\ci{Bigg} as prefixes to most delimiter commands:
\begin{example}
$\Big((x+1)(x-1)\Big)^{2}$\\
$\big( \Big( \bigg( \Bigg( \quad
\big\} \Big\} \bigg\} \Bigg\} \quad
\big\| \Big\| \bigg\| \Bigg\| \quad
\big\Downarrow \Big\Downarrow
\bigg\Downarrow \Bigg\Downarrow$
\end{example}
 For a list of all delimiters available, see Table~\ref{tab:delimiters} on page
\pageref{tab:delimiters}.

\section{Single Equations that are Too Long: multline}
\index{long equations}
\label{sec:multline}

If an equation is too long, we have to wrap it somehow. Unfortunately,
wrapped equations are usually less easy to read than not wrapped
ones. To improve the readability, there are certain rules on how to do
the wrapping:
\begin{enumerate}
\item In general one should always wrap an equation \textbf{before} an
  equality sign or an operator.
\item A wrap before an equality sign is preferable to a wrap before
  any operator.
\item A wrap before a plus- or minus-operator is preferable to a wrap
  before a multiplication-operator.
\item Any other type of wrap should be avoided if at all possible.
\end{enumerate}
The easiest way to achieve such a wrapping is the use of the
\ei{multline} en\-vi\-ron\-ment:\footnote{The
  \texttt{multline}-environment is from \texttt{amsmath}.}
\begin{example}
\begin{multline}
  a + b + c + d + e + f
  + g + h + i
  \\
  = j + k + l + m + n
\end{multline}
\end{example}
\noindent
The difference from the \ei{equation} environment is that an arbitrary
line-break (or also multiple line-breaks) can be introduced. This is
done by putting a \verb+\\+ on those places where the equation needs
to be wrapped. Similarly to \ei{equation*} there also exists a
\ei{multline*} version for preventing an equation number.

Often the
\ei{IEEEeqnarray} environment (see section~\ref{sec:IEEEeqnarray})
will yield better results.  Consider the following
situation:
\begin{example}
\begin{equation}
  a = b + c + d + e + f
  + g + h + i + j
  + k + l + m + n + o + p
  \label{eq:equation_too_long}
\end{equation}
\end{example}
\noindent
Here it is actually the RHS that is too long to fit on one line. The
\ei{multline} environment creates the following output:
\begin{example}
\begin{multline}
  a = b + c + d + e + f
  + g + h + i + j \\
  + k + l + m + n + o + p
\end{multline}
\end{example}

This is better than \eqref{eq:equation_too_long}, but
it has the disadvantage that the equality sign loses its natural
greater importance with respect to the plus operator in front of
$k$. The better solution is provided by the
\ei{IEEEeqnarray} environment that will be discussed in detail in
Section~\ref{sec:IEEEeqnarray}.

\section{Multiple Equations}
\index{equation!multiple}
\label{sec:IEEEeqnarray}

In the most general situation we have a sequence of several
equalities that do not fit onto one line. Here we need to work with
vertical alignment in order to keep the array of equations in a nice
and readable structure.

Before we offer our suggestions on how to do this, we start with a few
bad examples that show the biggest drawbacks of some common solutions.

\subsection{Problems with Traditional Commands}
\label{sec:problems_traditional}

To group multiple equations the
\ei{align} environment\footnote{The \texttt{align}-environment can
  also be used to group several blocks of equations beside each other.
  Another excellent use case for the
  \ei{IEEEeqnarray} environment. Try an argument like
  \texttt{\{rCl+rCl\}}.} could be used:
\begin{example}
\begin{align}
  a & = b + c \\
  & = d + e
\end{align}
\end{example}

this approach fails once a single line is too long:
\begin{example}
\begin{align}
  a & = b + c \\
  & = d + e + f + g + h + i
  + j + k + l \nonumber \\
  & + m + n + o \\
  & = p + q + r + s
\end{align}
\end{example}
\noindent
Here $+\:m$ should be below $d$ and not below the equality sign. Of
course, one could add some space (\verb+\hspace{...}+),
but this will never yield a precise arrangement (and is bad
style\ldots).

A better solution is offered by the \ei{eqnarray} environment:
\begin{example}
\begin{eqnarray}
  a & = & b + c \\
  & = & d + e + f + g + h + i
  + j + k + l \nonumber \\
  && +\: m + n + o \\
  & = & p + q + r + s
\end{eqnarray}
\end{example}

This is still not optimal. The spaces around the equality signs are too big.
Particularly, they are \textbf{not} the same as in the
\ei{multline} and \ei{equation} environments:
\begin{example}
\begin{eqnarray}
  a & = & a = a
\end{eqnarray}
\end{example}

\noindent \ldots and the expression sometimes overlaps with the equation number even
  though there would be enough room on the left:
\begin{example}
\begin{eqnarray}
  a & = & b + c
  \\
  & = & d + e + f + g + h^2
  + i^2 + j
  \label{eq:faultyeqnarray}
\end{eqnarray}
\end{example}

\noindent While the environment offers a command \ci{lefteqn} that can
  be used when the LHS is too long:
\begin{example}
\begin{eqnarray}
  \lefteqn{a + b + c + d
    + e + f + g + h}\nonumber\\
  & = & i + j + k + l + m
  \\
  & = & n + o + p + q + r + s
\end{eqnarray}
\end{example}
\noindent This is not optimal either as the RHS is too short and the array is
not properly centered:
\begin{example}
\begin{eqnarray}
  \lefteqn{a + b + c + d
    + e + f + g + h}
  \nonumber \\
  & = & i + j
\end{eqnarray}
\end{example}

\noindent Having badmouthed the competition sufficiently, I can now steer you gently towards the glorious \ldots

\subsection{IEEEeqnarray Environment}
\label{sec:IEEEeqnarray_intro}

The \ei{IEEEeqnarray} environment is a very powerful command with
many options. Here, we will only introduce its basic
functionalities. For more information please refer to the
manual.\footnote{The official manual is called
  \CTAN|macros/latex/contrib/IEEEtran/IEEEtran_HOWTO.pdf|. The part about \texttt{IEEEeqnarray}
  can be found in Appendix~F.}

First of all, in order to be able to use the
\ei{IEEEeqnarray} environment one needs to load the
package\footnote{The \pai{IEEEtrantools} package may not be included in your setup, it can be found on CTAN.}
\pai{IEEEtrantools}. Include the following line in the header of
your document: \small
\begin{verbatim}
\usepackage[retainorgcmds]{IEEEtrantools}
\end{verbatim}
\normalsize

The strength of \ei{IEEEeqnarray} is the ability to specify
the number of \emph{columns} in the equation array. Usually, this
specification will be \verb+{rCl}+, \emph{i.e.}, three columns, the
first column right-justified, the middle one centered with a little
more space around it (therefore we specify capital \texttt{C} instead of
lower-case \texttt{c}) and the third column left-justified:
\begin{example}
\begin{IEEEeqnarray}{rCl}
  a & = & b + c
  \\
  & = & d + e + f + g + h
  + i + j + k \nonumber\\
  && \negmedspace {} + l + m + n + o
  \\
  & = & p + q + r + s
\end{IEEEeqnarray}
\end{example}
Any number of columns can be specified:
\verb+{c}+ will give only one column with all entries centered, or
\verb+{rCll}+ would add a fourth, left-justified column to use
for comments. Moreover, beside \texttt{l}, \texttt{c}, \texttt{r}, \texttt{L},
\texttt{C}, \texttt{R} for math mode entries there are also \texttt{s},
\texttt{t}, \texttt{u} for left, centered, and right text mode entries.
Additional space can be added with \texttt{.} and
\texttt{/} and \texttt{?} in increasing order.\footnote{For more spacing
  types refer to Section~\ref{sec:putting-qed-right}.}
Note the spaces around the equality signs in contrast to the space produced
by the \texttt{eqnarray} environment.

\subsection{Common Usage}
\label{sec:common-usage}

In the following we will describe how we use \texttt{IEEEeqnarray} to
solve the most common problems.

If a line overlaps with the equation number as in
  \eqref{eq:faultyeqnarray}, the command
\small
\begin{verbatim}
\IEEEeqnarraynumspace
\end{verbatim}
\normalsize
  can be used: it has to be added in the corresponding line and makes
  sure that the whole equation array is shifted by the size of the
  equation numbers (the shift depends on the size of the number!):
  instead of
\begin{example}
\begin{IEEEeqnarray}{rCl}
  a & = & b + c
  \\
  & = & d + e + f + g + h
  + i + j + k
  \\
  & = & l + m + n
\end{IEEEeqnarray}
\end{example}
  we get
\begin{example}
\begin{IEEEeqnarray}{rCl}
  a & = & b + c
  \\
  & = & d + e + f + g + h
  + i + j + k
  \IEEEeqnarraynumspace\\
  & = & l + m + n.
\end{IEEEeqnarray}
\end{example}

If the LHS is too long, as a replacement for the faulty
  \ci{lefteqn} command, \texttt{IEEEeqnarray} offers the
  \ci{IEEEeqnarraymulticol} command which works in all situations:
\begin{example}
\begin{IEEEeqnarray}{rCl}
  \IEEEeqnarraymulticol{3}{l}{
    a + b + c + d + e + f
    + g + h
  }\nonumber\\ \quad
  & = & i + j
  \\
  & = & k + l + m
\end{IEEEeqnarray}
\end{example}
The usage is identical to the \ci{multicolumns} command in the
\texttt{tabular}-en\-vi\-ron\-ment. The first argument \verb+{3}+
specifies that three columns shall be combined into one which will be
left-justified \verb+{l}+.

Note that by inserting \ci{quad} commands one can easily adapt
the depth of the equation signs,\footnote{I think that one quad is the
  distance that looks good for most cases.} \emph{e.g.},
\begin{example}
\begin{IEEEeqnarray}{rCl}
  \IEEEeqnarraymulticol{3}{l}{
    a + b + c + d + e + f
    + g + h
  }\nonumber\\ \qquad\qquad
  & = & i + j
  \\
  & = & k + l + m
\end{IEEEeqnarray}
\end{example}

If an equation is split into two or more lines, \LaTeX\
  interprets the first $+$ or $-$ as a sign instead of operator.
  Therefore, it is necessary to add an additional space \ci{:}
  between the operator and the term: instead of
\begin{example}
\begin{IEEEeqnarray}{rCl}
  a & = & b + c
  \\
  & = & d + e + f + g + h
  + i + j + k \nonumber\\
  && + l + m + n + o
  \\
  & = & p + q + r + s
\end{IEEEeqnarray}
\end{example}
  we should write
\begin{example}
\begin{IEEEeqnarray}{rCl}
  a & = & b + c
  \\
  & = & d + e + f + g + h
  + i + j + k \nonumber\\
  && \negmedspace {} + l + m + n + o
  \\
  & = & p + q + r + s
\end{IEEEeqnarray}
\end{example}
\noindent Note the space difference between $+$ and $l$!
The construction \verb|{} + l| forces the \verb|+|-sign to be a binary operator rather
than just a sign, and the unwanted ensuing space between
\verb|{}| and \verb|+| is compensated by a negative medium space
\ci{negmedspace}.

If a particular line should not have an equation number, the
  number can be suppressed using \ci{nonumber} (or
  \ci{IEEEnonumber}). If on such a line a label
  \verb+\label{eq:...}+ is defined, then this label is passed on
  to the next equation number that is not suppressed. Place the labels right before the line-break
  \verb+\\+ or the next to the equation it belongs to. Apart from
  improving the readability of the source code this prevents a
  compilation error when a \ci{IEEEmulticol} command
  follows the label-definition.

There also exists a *-version where all equation numbers are
  suppressed. In this case an equation number can be made to appear
  using the command \ci{IEEEyesnumber}:
\begin{example}
\begin{IEEEeqnarray*}{rCl}
  a & = & b + c \\
  & = & d + e \IEEEyesnumber\\
  & = & f + g
\end{IEEEeqnarray*}
\end{example}

Sub-numbers are also easily possible using
  \ci{IEEEyessubnumber}:
\begin{example}
\begin{IEEEeqnarray}{rCl}
  a & = & b + c
  \IEEEyessubnumber\\
  & = & d + e
  \nonumber\\
  & = & f + g
  \IEEEyessubnumber
\end{IEEEeqnarray}
\end{example}

\section{Arrays and Matrices} \label{sec:arraymat}

To typeset \textbf{arrays}, use the \ei{array} environment. It works
in a similar way to the \texttt{tabular} environment. The \verb|\\| command is
used to break the lines:
\begin{example}
  \begin{equation*}
    \mathbf{X} = \left(
      \begin{array}{ccc}
        x_1 & x_2 & \ldots \\
        x_3 & x_4 & \ldots \\
        \vdots & \vdots & \ddots
      \end{array} \right)
  \end{equation*}
\end{example}

The \ei{array} environment can also be used to typeset \wi{piecewise function}s by
using a ``\verb|.|'' as an invisible \ci{right} delimiter:
\begin{example}
\begin{equation*}
  |x| = \left\{
    \begin{array}{rl}
      -x & \text{if } x < 0,\\
      0 & \text{if } x = 0,\\
      x & \text{if } x > 0.
    \end{array} \right.
\end{equation*}
\end{example}
The \ei{cases} environment from \textsf{amsmath} simplifies
the syntax, so it is worth a look:
\begin{example}
  \begin{equation*}
    |x| =
    \begin{cases}
      -x & \text{if } x < 0,\\
      0 & \text{if } x = 0,\\
      x & \text{if } x > 0.
    \end{cases}
\end{equation*}
\end{example}

Matrices\index{matrix} can be typeset by \ei{array}, but
\pai{amsmath} provides a better solution using the different \ei{matrix}
environments. There are six versions with different delimiters: \ei{matrix}
(none), \ei{pmatrix} $($, \ei{bmatrix} $[$, \ei{Bmatrix} $\{$, \ei{vmatrix} $\vert$ and
\ei{Vmatrix} $\Vert$. You don't have to specify the number of columns as with
\ei{array}. The maximum number is 10, but it is customisable (though it is not
very often you need 10 columns!):
\begin{example}
\begin{equation*}
  \begin{matrix}
    1 & 2 \\
    3 & 4
  \end{matrix} \qquad
  \begin{bmatrix}
    p_{11} & p_{12} & \ldots
    & p_{1n} \\
    p_{21} & p_{22} & \ldots
    & p_{2n} \\
    \vdots & \vdots & \ddots
    & \vdots \\
    p_{m1} & p_{m2} & \ldots
    & p_{mn}
  \end{bmatrix}
\end{equation*}
\end{example}

\section{Spacing in Math Mode} \label{sec:math-spacing}

\index{math spacing} If the spacing within formulae chosen by \LaTeX{}
is not satisfactory, it can be adjusted by inserting special spacing
commands: \ci{,} for $\frac{3}{18}\:\textrm{quad}$
(\demowidth{0.166em}), \ci{:} for $\frac{4}{18}\: \textrm{quad}$
(\demowidth{0.222em}) and \ci{;} for $\frac{5}{18}\: \textrm{quad}$
(\demowidth{0.277em}).  The escaped space character \verb*|\ |
generates a medium sized space comparable to the interword spacing and
\ci{quad} (\demowidth{1em}) and \ci{qquad} (\demowidth{2em}) produce
large spaces. The size of a \ci{quad} corresponds to the width of the
character `M' of the current font. \verb|\!|\cih{"!} produces a
negative space of $-\frac{3}{18}\:\textrm{quad}$
($-$\demowidth{0.166em}).

\begin{example}
\begin{equation*}
  \int_1^2 \ln x \mathrm{d}x
  \qquad
  \int_1^2 \ln x \,\mathrm{d}x
\end{equation*}
\end{example}

Note that `d' in the differential is conventionally set in roman.
In the next example, we define a new command \ci{ud} (upright d) which produces
``$\,\mathrm{d}$'' (notice the spacing \demowidth{0.166em} before the
$\text{d}$), so we don't have to write it every time. The \ci{newcommand} is
placed in the preamble. %  More on
% \ci{newcommand} in section~\ref{} on page \pageref{}. To Do: Add label and
% reference to "Customising LaTeX" -> "New Commands, Environments and Packages"
% -> "New Commands".
\begin{example}
\newcommand{\ud}{\,\mathrm{d}}

\begin{equation*}
 \int_a^b f(x)\ud x
\end{equation*}
\end{example}

If you want to typeset multiple integrals, you'll discover that the spacing
between the integrals is too wide. You can correct it using \ci{!}, but
\pai{amsmath} provides an easier way for fine-tuning
the spacing, namely the \ci{iint}, \ci{iiint}, \ci{iiiint}, and \ci{idotsint}
commands.

\begin{example}
\newcommand{\ud}{\,\mathrm{d}}

\begin{IEEEeqnarray*}{c}
  \int\int f(x)g(y)
                  \ud x \ud y \\
  \int\!\!\!\int
         f(x)g(y) \ud x \ud y \\
  \iint f(x)g(y)  \ud x \ud y
\end{IEEEeqnarray*}
\end{example}

See the electronic document \texttt{testmath.tex} (distributed with
\AmS-\LaTeX) or Chapter 8 of \companion{} for further details.

\subsection{Phantoms}

When vertically aligning text using \verb|^| and \verb|_| \LaTeX{} is sometimes
just a little too helpful. Using the \ci{phantom} command you can
reserve space for characters that do not show up in the final output.
The easiest way to understand this is to look at an example:
\begin{example}
\begin{equation*}
{}^{14}_{6}\text{C}
\qquad \text{versus} \qquad
{}^{14}_{\phantom{1}6}\text{C}
\end{equation*}
\end{example}
If you want to typeset a lot of isotopes as in the example, the \pai{mhchem}
package is very useful for typesetting isotopes and chemical formulae too.

\section{Fiddling with the Math Fonts}\label{sec:fontsz}
Different math fonts are listed on Table~\ref{mathalpha} on page
\pageref{mathalpha}.
\begin{example}
 $\Re \qquad
  \mathcal{R} \qquad
  \mathfrak{R} \qquad
  \mathbb{R} \qquad $
\end{example}
The last two require \pai{amssymb} or \pai{amsfonts}.

Sometimes you need to tell \LaTeX{} the correct font
size. In math mode, this is set with the following four commands:
\begin{flushleft}
\ci{displaystyle}~($\displaystyle 123$),
 \ci{textstyle}~($\textstyle 123$),
\ci{scriptstyle}~($\scriptstyle 123$) and
\ci{scriptscriptstyle}~($\scriptscriptstyle 123$).
\end{flushleft}

If $\sum$ is placed in a fraction, it'll be typeset in text style unless you tell
\LaTeX{} otherwise:
\begin{example}
\begin{equation*}
 P = \frac{\displaystyle{
   \sum_{i=1}^n (x_i- x)
   (y_i- y)}}
   {\displaystyle{\left[
   \sum_{i=1}^n(x_i-x)^2
   \sum_{i=1}^n(y_i- y)^2
   \right]^{1/2}}}
\end{equation*}
\end{example}
Changing styles generally affects the way big operators and limits are displayed.

% This is not a math accent, and no maths book would be set this way.
% mathop gets the spacing right.

\subsection{Bold Symbols}
\index{bold symbols}

It is quite difficult to get bold symbols in \LaTeX{}; this is
probably intentional as amateur typesetters tend to overuse them.  The
font change command \verb|\mathbf| gives bold letters, but these are
roman (upright) whereas mathematical symbols are normally italic, and
furthermore it doesn't work on lower case Greek letters.
There is a \ci{boldmath} command, but \emph{this can only be used
outside math mode}. It works for symbols too, though:
\begin{example}
$\mu, M \qquad
\mathbf{\mu}, \mathbf{M}$
\qquad \boldmath{$\mu, M$}
\end{example}

The package \pai{amsbsy} (included by \pai{amsmath}) as well as the
package \pai{bm} from the \texttt{tools} bundle make this much easier as they include
a \ci{boldsymbol} command:

\begin{example}
$\mu, M \qquad
\boldsymbol{\mu}, \boldsymbol{M}$
\end{example}

\section{Theorems, Lemmas, \ldots}

When writing mathematical documents, you probably need a way to
typeset ``Lemmas'', ``Definitions'', ``Axioms'' and similar
structures.
\begin{lscommand}
\ci{newtheorem}\verb|{|\emph{name}\verb|}[|\emph{counter}\verb|]{|%
         \emph{text}\verb|}[|\emph{section}\verb|]|
\end{lscommand}
The \emph{name} argument is a short keyword used to identify the
``theorem''. With the \emph{text} argument you define the actual name
of the ``theorem'', which will be printed in the final document.

The arguments in square brackets are optional. They are both used to
specify the numbering used on the ``theorem''. Use  the \emph{counter}
argument to specify the \emph{name} of a previously declared
``theorem''. The new ``theorem'' will then be numbered in the same
sequence.  The \emph{section} argument allows you to specify the
sectional unit within which the ``theorem'' should get its numbers.

After executing the \ci{newtheorem} command in the preamble of your
document, you can use the following command within the document.
\begin{code}
\verb|\begin{|\emph{name}\verb|}[|\emph{text}\verb|]|\\
This is my interesting theorem\\
\verb|\end{|\emph{name}\verb|}|
\end{code}

The \pai{amsthm} package (part of \AmS-\LaTeX) provides the
\ci{theoremstyle}\verb|{|\emph{style}\verb|}|
command which lets you define what the theorem is all about by picking
from three predefined styles: \texttt{definition} (fat title, roman body),
\texttt{plain} (fat title, italic body) or \texttt{remark} (italic
title, roman body).

This should be enough theory. The following examples should
remove any remaining doubt, and make it clear that the
\verb|\newtheorem| environment is way too complex to understand.

% actually define things
\theoremstyle{definition} \newtheorem{law}{Law}
\theoremstyle{plain}      \newtheorem{jury}[law]{Jury}
\theoremstyle{remark}     \newtheorem*{marg}{Margaret}

First define the theorems:

\begin{verbatim}
\theoremstyle{definition} \newtheorem{law}{Law}
\theoremstyle{plain}      \newtheorem{jury}[law]{Jury}
\theoremstyle{remark}     \newtheorem*{marg}{Margaret}
\end{verbatim}

\begin{example}
\begin{law} \label{law:box}
Don't hide in the witness box
\end{law}
\begin{jury}[The Twelve]
It could be you! So beware and
see law~\ref{law:box}.\end{jury}
\begin{jury}
You will disregard the last
statement.\end{jury}
\begin{marg}No, No, No\end{marg}
\begin{marg}Denis!\end{marg}
\end{example}

The ``Jury'' theorem uses the same counter as the ``Law''
theorem, so it gets a number that is in sequence with
the other ``Laws''. The argument in square brackets is used to specify
a title or something similar for the theorem.
\begin{example}
\newtheorem{mur}{Murphy}[section]

\begin{mur} If there are two or
more ways to do something, and
one of those ways can result in
a catastrophe, then someone
will do it.\end{mur}
\end{example}

The ``Murphy'' theorem gets a number that is linked to the number of
the current section. You could also use another unit, for example chapter or
subsection.

If you want to customize your theorems down to the last dot, the
\pai{ntheorem} package offers a plethora of options.

\subsection{Proofs and End-of-Proof Symbol}
\label{sec:putting-qed-right}

The \pai{amsthm} package also provides the \ei{proof} environment.

\begin{example}
\begin{proof}
 Trivial, use
 \begin{equation*}
   E=mc^2.
 \end{equation*}
\end{proof}
\end{example}

With the command \ci{qedhere} you can move the `end of proof' symbol
around for situations where it would end up alone on a line.

\begin{example}
\begin{proof}
 Trivial, use
 \begin{equation*}
   E=mc^2. \qedhere
 \end{equation*}
\end{proof}
\end{example}

Unfortunately, this correction does not work for \texttt{IEEEeqnarray}:
\begin{example}
\begin{proof}
  This is a proof that ends
  with an equation array:
  \begin{IEEEeqnarray*}{rCl}
    a & = & b + c \\
    & = & d + e. \qedhere
  \end{IEEEeqnarray*}
\end{proof}
\end{example}
\noindent
The reason for this is the internal structure of \texttt{IEEEeqnarray}:
it always puts two invisible columns at both sides of the array that
only contain a stretchable space. By this \texttt{IEEEeqnarray} ensures
that the equation array is horizontally centered. The
\ci{qedhere} command should actually be put \emph{outside} this
stretchable space, but this does not happen as these columns are
invisible to the user.

There is a very simple remedy. Define the stretching
explicitly!
\begin{example}
\begin{proof}
  This is a proof that ends
  with an equation array:
  \begin{IEEEeqnarray*}{+rCl+x*}
    a & = & b + c \\
    & = & d + e. & \qedhere
  \end{IEEEeqnarray*}
\end{proof}
\end{example}
\noindent
Note that the \verb=+= in \verb={+rCl+x*}= denotes stretchable spaces, one
on the left of the equations (which, if not specified, will be done
automatically by \texttt{IEEEeqnarray}!) and one on the right of the
equations. But now on the right, \emph{after} the stretching column,
we add an empty column \verb=x=. This column will only be needed on
the last line if the \ci{qedhere} command is put
there. Finally, we specify a \verb=*=. This is a null-space that
prevents \texttt{IEEEeqnarray} from adding another unwanted \verb=+=-space!

In the case of equation numbering, there is a similar problem. Comparing
\begin{example}
\begin{proof}
  This is a proof that ends
  with a numbered equation:
  \begin{equation}
    a = b + c.
  \end{equation}
\end{proof}
\end{example}
\noindent
with
\begin{example}
\begin{proof}
  This is a proof that ends
  with a numbered equation:
  \begin{equation}
    a = b + c. \qedhere
  \end{equation}
\end{proof}
\end{example}
\noindent
you notice that in the (correct) second version the $\Box$ is much
closer to the equation than in the first version.

Similarly, the correct way of putting the QED-symbol at the end of an
equation array is as follows:
\begin{example}
\begin{proof}
  This is a proof that ends
  with an equation array:
  \begin{IEEEeqnarray}{+rCl+x*}
    a & = & b + c \\
    & = & d + e. \\
    &&& \qedhere\nonumber
  \end{IEEEeqnarray}
\end{proof}
\end{example}
\noindent
which contrasts with
\begin{example}
\begin{proof}
  This is a proof that ends
  with an equation array:
  \begin{IEEEeqnarray}{rCl}
    a & = & b + c \\
    & = & d + e.
  \end{IEEEeqnarray}
\end{proof}
\end{example}

%

% Local Variables:
% TeX-master: "lshort"
% mode: latex
% mode: flyspell
% End:

synctex test files/less basic/2017/lshort-5.05/src/math.tex


\section{The Adjugate Matrix}
Given a square matrix $A\in M_n(F)$ with columns $A^{(i)}$.
If we swap two neighbouring columns (or rows), then $\det A$ changes sign.
\begin{remark}
    We can prove properties of determinant using the decomposition of $A$ into elementary matrices.
\end{remark}
A column expansion is a strategy to compute the determinant of a matrix by using its linkage to some of its submatrices.
\begin{definition}
    Let $A\in M_n(F)$ and pick $i,j\in\{1,\ldots,n\}$.
    We define $A_{\widehat{ij}}\in A_{n-1}(F)$ to be the matrix obtained by deleting the $i^{th}$ row and $j^{th}$ column from $A$.
\end{definition}
\begin{example}
    Take
    $$A=\begin{pmatrix}
        1&2&-7\\
        2&1&0\\
        -3&6&1
    \end{pmatrix}$$
    Then
    $$A_{\widehat{32}}=\begin{pmatrix}
        1&-7\\
        2&0
    \end{pmatrix}$$
\end{example}
\begin{lemma}[Expansion of Determinant]
    Let $A\in M_n(F)$.\\
    1. We have the expansion along the $j^{th}$ column
    $$\det A=\sum_{i=1}^n(-1)^{i+j}a_{ij}\det A_{\widehat{ij}}$$
    2. We have the expansion along the $i^{th}$ row
    $$\det A=\sum_{j=1}^n(-1)^{i+j}a_{ij}\det A_{\widehat{ij}}$$
\end{lemma}
\begin{example}
    Take
    $$A=\begin{pmatrix}
        1&2&-1\\
        3&-1&1\\
        4&2&-7
    \end{pmatrix}$$
    So expanding along the second column gives
    $$\det A=-(2)\det\begin{pmatrix}
        3&1\\
        4&-7
    \end{pmatrix}+(-1)\det\begin{pmatrix}
        1&-1\\
        4&-7
    \end{pmatrix}-2\det\begin{pmatrix}
        1&-1\\
        3&1
    \end{pmatrix}$$
\end{example}
\begin{proof}
    Suffices to show the first part.
    Pick $1\le j\le n$ and write $A=(a_{ij}),A^{(j)}=\sum_ia_{ij}e_i$.
    Then
    \begin{align*}
        \det A&=\det\left( A^{(1)},\ldots,\sum_{i=1}^na_{ij}e_i ,\ldots, A^{(n)}\right)\\
        &=\sum_{i=1}^na_{ij}\det(A^{(1)},\ldots,e_i,\ldots,A^{(n)})\\
        &=\sum_{i=1}^na_{ij}(-1)^{j-1}\det(e_i,A^{(1)},\ldots,A^{(n)})\\
        &=\sum_{i=1}^na_{ij}(-1)^{j-1}(-1)^{i-1}\det
        \begin{pmatrix}
            1&\ast&\cdots&\ast\\
            0&&&\\
            \vdots&&A_{\widehat{ij}}&\\
            0&&&
        \end{pmatrix}\\
        &=\sum_{i=1}^n(-1)^{i+j}a_{ij}\det A_{\widehat{ij}}
    \end{align*}
    as desired.
\end{proof}
\begin{definition}
    Let $A\in M_n(F)$, then the adjugate matrix $\operatorname{adj}A$ of $A$ is the $n\times n$ matrix with entries
    $$(\operatorname{adj}A)_{ji}=(-1)^{i+j}\det A_{\widehat{ij}}=\det(A^{(1)},\ldots,A^{(j-1)},e_i,A^{(j+1)},\ldots,A^{(n)})$$
\end{definition}
\begin{theorem}
    Let $A\in M_n(F)$, then $\operatorname{adj}(A)A=(\det A)I$ where $I$ is the identity matrix.
\end{theorem}
In particular, if $A$ is invertible, then as $\det A\neq 0$,
$$A^{-1}=\frac{1}{\det A}\operatorname{adj}A$$
\begin{proof}
    For any $j$, the preceding lemma translates to
    $$\det A=\sum_{i=1}^n(\operatorname{adj}A)_{ij}a_{ij}=(\operatorname{adj}(A)A)_{jj}$$
    Now for $j<k$, let $A'$ be the matrix obtained by putting $A^{(k)}$ in the place of $A^{(j)}$, then
    \begin{align*}
        0&=\det(A')\\
        &=\det(A^{(1)},\ldots,A^{(k)},\ldots,A^{(k)},\ldots,A^{(n)})\\
        &=\det\left(A^{(1)},\ldots,\sum_{i=1}^na_{ik}e_i,\ldots,A^{(k)},\ldots,A^{(n)}\right)\\
        &=\sum_{i=1}^na_{ik}\det(A^{(1)},\ldots,e_i,\ldots,A^{(k)},\ldots,A^{(n)})\\
        &=\sum_{i=1}^na_{ik}(\operatorname{adj}A)_{ji}\\
        &=(\operatorname{adj}(A)A)_{jk}
    \end{align*}
    which implies the result.
\end{proof}
\begin{proposition}
    Let $A\in M_n(F)$ be invertible, and let $b\in F^n$, then the unique solution to $Ax=b$ is given by
    $$x_i=\frac{1}{\det A}\det A_{\hat{i}b}$$
    where $A_{\hat{i}b}$ is the matrix obtained by replacing the $i^{th}$ column of $A$ by $b$.
\end{proposition}
\begin{proof}
    As $A$ is invertible, such an $x$ exists and is unique.
    Let $x$ be a solution, then note that
    \begin{align*}
        \det(A_{\hat{i}b})&=\det(A^{(1)},\ldots,A^{(i-1)},b,A^{(i+1)},\ldots,A^{(n)})\\
        &=\det(A^{(1)},\ldots,A^{(i-1)},Ax,A^{(i+1)},\ldots,A^{(n)})\\
        &=\det\left(A^{(1)},\ldots,A^{(i-1)},\sum_{j=1}^nx_jA^{(j)},A^{(i+1)},\ldots,A^{(n)}\right)\\
        &=\sum_{j=1}^nx_j\det(A^{(1)},\ldots,A^{(i-1)},A^{(j)},A^{(i+1)},\ldots,A^{(n)})\\
        &=\sum_{j=1}^nx_j\delta_{ij}\det A\\
        &=x_i\det A
    \end{align*}
    just as we wanted.
\end{proof}


\subsection{Majority voting}

\subsubsection{Combining classifiers}

If we generate different classifiers then each can give different predictions for the same input.

If we have different predictions how should we proceed?

\subsubsection{Using one model or multiple models}

It is not obvious that we should want to use more than one model. If one model was superior to another then there may be no benefit to using the information from the additional model.

However if the errors in different models are varied, then combining multiple models can lead to better performance as each individual model can have unique information.

\subsubsection{Majority voting}

One approach to using different predictions is to use majority voting.

If we have a collection of hard classifiers then we choose the classification with the most votes.

\subsubsection{Condorcet's Jury Theorem}

Consider a collection of classifiers. For each classifier there is a chance \(p_i\) of the classification being correct.

If the voters are independent

If voters are independent, and the chance of one vote being right is greater than \(0.5\), then the more voters, the better.

A weak model can still be useful, if it is independent.


%%!TEX TS-program = latex
\documentclass[11pt]{article} %DIF >
\usepackage{etex}
\usepackage[utf8]{inputenc}
\input ../AuxFiles/PreambleNotes.tex

\begin{document}
\onehalfspace

\vspace*{\fill}
\begingroup
\centering

\Large {\scshape Introduction to Statistics}\\

(Lectures 7-8: Estimation)

\endgroup
\vspace*{\fill}

\newpage

\section{Estimation}

{\scshape Overview:} \noindent Lectures 7-8 will focus on \emph{estimation}. As we have mentioned before, a strategy in the estimation problem is an \emph{estimator}. The loss function used for this problem is the squared distance between the estimated value and the true parameter. This loss gives rise to the popular \emph{mean squared error} performance criterion.

We will analyze two commonly used estimation strategies---Maximum Likelihood (ML) and Posterior Mean estimators---in the context of the homoskedastic Normal regression model with (known variance); i.e.,
\begin{equation} \label{equation:Regression-model}
Y \sim \mathcal{N}(X \beta, \sigma^2 \mathbb{I}_n ),
\end{equation}
 where $Y$ is the $n \times 1$ vector of outcome variables, $X$ is the $n \times k$ matrix of non-stochastic regressors, and $\beta \in \mathbb{R}^{k}$ is the parameter of interest.

 We will show that if $k \leq n$ and $X$ has full-column rank, the Maximum Likelihood estimator for the model in (\ref{equation:Regression-model}) is the OLS estimator
\[ \widehat{\beta}_{\textrm{OLS}} \equiv (X’X)^{-1} X’Y,  \]

\noindent (which we introduced in the last problem set). We will show this estimator is ``best'' among all \emph{unbiased} estimators and we will connect this result to the \emph{Cram\'er-Rao} bound for the variance of estimators in parametric models. We will also argue that ``best’’ among the class of unbiased estimators need not imply admissibility with respect to all estimators. In particular, we show that if $k \geq 3$, the OLS estimator is dominated and we present an ``empirical Bayes’’ estimator that dominates it.

The posterior mean estimator for the regression model assuming $\beta \sim \mathcal{N}_{k}(0 , (\sigma^2/ \lambda) \mathbb{I}_k )$ is the popular \emph{Ridge estimator}
\[ \widehat{\beta}_{\textrm{Ridge}} \equiv (X’X + \lambda \mathbb{I}_k)^{-1} X’Y. \]
The Ridge estimator is well-defined regardless of the number of covariates and it is admissible by construction. The Ridge estimator provides a simple example of ``shrinkage’’ or ``regularization’’. A practical concern is that the Ridge estimator is biased, and the bias depends on the choice of the prior hyperparameters and the true parameter at which we evaluate the bias.

One result that we will not cover in the notes, but that is important to keep in mind is that despite the inadmissibility of the OLS estimator for the \emph{full} vector of coefficients $\beta$, OLS is admissible for the problem in which we are interested in estimating only one regression coefficient but controlling for other variables.

\subsection{Quadratic Loss}
Let $\mathcal{A} =\Theta = \mathbb{R}^{k}$. The loss function we will work with is the so-called quadratic loss
\[ \mathcal{L}(a, \beta) = || a- \beta ||^2 = (a-\beta)’(a-\beta) =  \sum_{j=1}^{k} (a_j  - \theta_j)^2,  \]
which measures the squared distance between the estimator $\widehat{\beta}$ and the parameter $\beta$.
The risk of any estimator $\widehat{\beta}$ (which is a map from data $(Y,X)$ to $\Theta$) is given by
\begin{equation} \label{equation:MSE}
\mathbb{E}_{\mathbb{P}_{\beta}}[ \mathcal{L}(\widehat{\beta}, \beta) ] = \mathbb{E}_{\mathbb{P}_{\beta}}[ (\widehat{\beta}- \beta)’(\widehat{\beta}-\beta) ].
\end{equation}
Equation (\ref{equation:MSE}) is referred to as \emph{the mean-squared estimation error} at $\beta$. We now present a simple algebraic decomposition of the mean squared error in terms of the ``bias’’ and ``variance’’ of the estimator $\widehat{\beta}$. Assuming that $\overline{\beta}  \equiv \mathbb{E}_{\beta}[ \widehat{\beta} ]$ is finite, we define the \underline{bias} of $\widehat{\beta}$ at $\beta$ as
$$ B_\beta(\widehat{\beta}) = \overline{\beta} - \beta.$$
If the covariance matrix of $\widehat{\beta}$---denoted $\textrm{V}_{\beta} (\widehat{\beta})$---is also finite, the mean squared-error can be written as
\begin{eqnarray*}
\mathbb{E}_{\mathbb{P}_{\beta}}[ (\widehat{\beta}- \beta)’(\widehat{\beta}-\beta) ] &=& \mathbb{E}_{\mathbb{P}_{\beta}}[ (\widehat{\beta}-\overline{\beta} +\overline{\beta} - \beta)’(\widehat{\beta}-\overline{\beta} +\overline{\beta} -\beta) ],  \\
&=& (\overline{\beta}-\beta)’(\overline{\beta}-\beta) +  \mathbb{E}_{\mathbb{P}_{\beta}}[ (\widehat{\beta}-\overline{\beta} )’(\widehat{\beta}-\overline{\beta}) ]  \\
&=& ||B_\beta(\widehat{\beta})  ||^2 + \textrm{tr} \left( \textrm{V}_{\beta} (\widehat{\beta}) \right),
\end{eqnarray*}
where $\textrm{tr}(\cdot)$ is the trace operator.  The decomposition is fairly straightforward, but it highlights the fact that the bias and variance of the estimator fully determine its risk whenever the loss is quadratic. Also, the correlation between any of the components of $\widehat{\beta}$ is not relevant for the risk calculation.

\subsection{Maximum Likelihood Estimation}

According to model (\ref{equation:Regression-model}) the vector of outcome variables is a multivariate normal with parameters $X\beta$ and $\sigma^2 \mathbb{I}_n$ and thus has a p.d.f given by
\begin{equation}\label{equation:Regression-pdf}
f( Y | \beta , X ) = \frac{1}{(2 \pi \sigma^2)^{n/2}} \exp \left( -\frac{1}{2 \sigma^2} (Y - X \beta)’ (Y-X\beta) \right).
\end{equation}
This is true, regardless of whether we have very many covariates or not. For a fixed realization of the data, define the likelihood function, L($\beta$; (Y,X)), as the value attained by the p.d.f. in (\ref{equation:Regression-pdf}) at different values of the data ($Y, X$); that is
\[ L(\beta, (Y,X)) \equiv f(Y | \beta, X ). \]
The maximum likelihood estimator at data $(Y,X)$ is then defined as the value of $\beta$ that maximizes the likelihood; that is
\[ \widehat{\beta}_{\textrm{ML}} \equiv \textrm{argmax}_{\beta \in \mathbb{R}^{k}} L(\beta, (Y,X)).   \]

\noindent The likelihood function implied by (\ref{equation:Regression-pdf}) is decreasing in $(Y-XB)’(Y-XB)$, a term which is usually referred to as the sum of square residuals. Therefore, maximizing the likelihood is equivalent to solving the \emph{least-squares} problem
\[ \textrm{min}_{\beta \in \mathbb{R}^{k}} (Y-X\beta)’(Y-X\beta).   \]
The first-order conditions for the program above, which are necessary and sufficient, yield
\[ X’(Y-X\widehat{\beta}_{\textrm{ML}}) = \textbf{0}_{k \times 1} \iff X’Y = (X’X) \widehat{\beta}_{\textrm{ML}}. \]

\noindent If $k > n$, there are infinitely many solutions that maximize the likelihood.  If $k \leq n$, and $X$ has full-column rank there is a unique solution to this problem given by
\begin{equation} \label{equation:OLS}
\widehat{\beta}_{\textrm{ML}} = \widehat{\beta}_{\textrm{OLS}} = (X’X)^{-1} X’Y.
\end{equation}

\noindent The bias of the maximum likelihood estimator:
\begin{equation} \label{equation:bias}
\textrm{B}_{\beta}(\widehat{\beta}_{\textrm{ML}}) = \mathbb{E}_{\mathbb{P}_{\beta}} [ (X’X)^{-1} X’Y ] - \beta = \mathbf{0}_{k \times 1}.
\end{equation}
for any $\beta$. Thus, we say that the ML estimator of $\beta$ is \underline{unbiased}.\footnote{An estimator $\widehat{\beta}$ is unbiased if $\mathbb{E}_{\beta} [\widehat{\beta}] = \beta$ for all $\beta \in \Theta$.}

The variance of the ML estimator is
\begin{equation} \label{equation:variance}
\mathbb{V}_{\beta} (\widehat{\beta}_{\textrm{ML}}) = \mathbb{E}_{\mathbb{P}_{\beta}} [ (\widehat{\beta}_{\textrm{ML}}-\beta) (\widehat{\beta}_{\textrm{ML}} -\beta )^{\prime}  ]  = (X’X)^{-1} X’ \mathbb{E}_{\mathbb{P}_{\beta}} [ (Y-X\beta) (Y-X\beta)’] X (X’X)^{-1} = \sigma^2 (X’X)^{-1}.
\end{equation}
This means that the mean squared error at $\beta$---denoted $\textrm{MSE}(\beta ; \widehat{\beta}_{\textrm{ML}})$---equals
\[ \sigma^2 \textrm{tr} \left ( (X’X)^{-1} \right). \]

\noindent for any $\beta$. Consequently, an interesting property of the ML/OLS estimator is that its risk function is constant over the parameter space.

\noindent

\subsection{Posterior Mean under a Normal Prior}
In this subsection we analyze the Bayes estimator of the parameter $\beta$. We have already showed that any Bayes rule can be obtained by minimizing posterior loss at each data realization. Let $\pi$ denote a prior over the parameter $\beta$. In our set-up the posterior loss of an action $a$ is
\[ \mathbb{E}_{\pi} [ \: || a - \beta ||^2 \: | \: (Y,X) \: ]. \]
Using the same argument that we used to decompose the mean squared-error in terms of bias and variance we can show that
\begin{eqnarray*}
\mathbb{E}_{\pi} [ \: || a - \beta ||^2 \: | \: (Y,X) \: ] &=& \mathbb{E}[ \: || a - \mathbb{E}_{\pi} [ \: \beta \: | \:  (Y,X) \:  ] + \mathbb{E}_{\pi} [ \beta \: | \: (Y,X) ] -  \beta ||^2 \: | \: (Y,X) \: ],  \\
&=& || a - \mathbb{E}_{\pi} [ \beta \: | \: (Y,X) ] ||^2  + \textrm{tr} \left( \mathbb{V}_{\pi} ( \beta \: | \: (Y,X) ) \right).
\end{eqnarray*}
This shows that, regardless of the specific prior we pick, the Bayes estimator is the posterior mean of $\beta$
\[ \widehat{\beta}_{\textrm{Bayes}} =   \mathbb{E}_{\pi} [ \: \beta \: | \: (Y,X) \: ].  \]

Before committing to a specific prior $\pi$, we will show that under quadratic loss the Bayes estimator is the posterior mean.\\

{\scshape Posterior Mean under a Normal Prior:}  Consider then the following prior on $\beta$:

\begin{equation}
\beta  \sim \pi (\beta) \equiv \mathcal{N}_{k}( \beta_0 \: , \: \sigma^2 V^{-1}).
\end{equation}

\noindent The prior assumes that all the coefficients are approximately normal with values close to the vector $\beta_0$ and covariance matrix given by $\sigma^2 V^{-1}$. There is typically no magical recipe to select a prior. More often than not, the selection of a prior trades-off interpretation and convenience in its implementation.

We derive the posterior distribution of $\beta$. One way of deriving this posterior distribution is by an application of Bayes Theorem
\begin{equation*}
\pi(\beta \: | \: \sigma^2, y, X ) = \frac{f(Y  \: | \: \beta, X) \pi(\beta ) }{\int_{\Theta} f(Y, \: | \: \beta, X) \pi (\beta ) d \beta}.
\end{equation*}

\noindent The posterior is thus proportional to the likelihood times the prior, both of which are Gaussian.

\noindent Consequently, $f(Y | X, \beta, \sigma^2 )\:  \pi(\beta | \sigma^2 )$ is, up to a constant that does not depend on $\beta$, proportional to
\begin{equation} \label{equation:posterior}
\exp \left( -\frac{1}{2\sigma^2} (Y-X\beta)^{\prime} (Y-X \beta) \right)  \exp \left(-\frac{1}{2 \sigma^2} (\beta-\beta_0)^{\prime} V (\beta-\beta_0) \right).
\end{equation}

\noindent The expression above equals:
\[ \exp \left( -\frac{1}{2\sigma^2} Y^{\prime} Y \right) \exp \left(  - \frac{1}{2 \sigma^2 } \beta \left(V + X’X \right) \beta + \frac{1}{\sigma^2}(Y^{\prime} X+V\beta_0) \beta \right). \]
Completing the square and ignoring all the terms that do not have $\beta$ on them, gives the posterior distribution as a constant times the exponential of:
\begin{equation*}
-\frac{1}{2 \sigma^2} \left( \beta -  \left( V + X^{\prime} X \right)^{-1} (X^{\prime} y+ V \beta_0) \right) \left(V+ X’X \right) \left(\beta -  ( V^{-1} + X^{\prime} X )^{-1} (X^{\prime} Y+ V \beta_0) \right).
\end{equation*}
This implies that:
\begin{equation}
\beta | Y, X \sim \mathcal{N}_{k} \left(  \left( V + X’X \right)^{-1} (X^{\prime} Y+V\beta_0) \: , \:   \sigma^2 \left(V + X^{\prime} X \right)^{-1}   \right).
\end{equation}
This means that the Bayesian Estimator of $\beta$ given the Gaussian prior $\pi(\beta)$ is:

\[ \widehat{\beta}_{\textrm{Bayes}} \equiv \left( V + X^{\prime} X \right)^{-1} \left( X^{\prime} Y + V \beta_0 \right). \]

\noindent The posterior mean estimator is well defined regardless the number of
covariates.\footnote{If $V$ is positive definite, then the matrix $V+ X’X$ is
  positive definite as well, and thus invertible.} The posterior mean estimator
for $V= \lambda \mathbb{I}_{k}$ and $\beta_0 = 0$ is called the Ridge estimator

\[  \widehat{\beta}_{\textrm{Ridge}} \equiv (X’X + \lambda \mathbb{I}_{k})^{-1} X’Y, \]

\noindent which, by construction, is admissible. The Ridge estimator is biased:
\begin{eqnarray*}
\mathbb{E}_{\mathbb{P}_{\beta}} [ \widehat{\beta}_{\textrm{Ridge}} ] - \beta &=& (X’X + \lambda \mathbb{I}_{k})^{-1} X’X\beta - \beta,  \\
&=& (X’X + \lambda \mathbb{I}_{k})^{-1} (X’X\beta - (X’X + \lambda \mathbb{I}_{k}) \beta), \\
&=& -\lambda (X’X + \lambda \mathbb{I}_k)^{-1} \beta.
\end{eqnarray*}
\noindent and the magnitude of the bias depends on $\beta$. The variance of the Ridge estimator is
\[ \mathbb{V}_{\beta} (\widehat{\beta}_{\textrm{Ridge}}) = \sigma^2( X’X + \lambda \mathbb{I}_k)^{-1} X’X ( X’X + \lambda \mathbb{I}_k)^{-1}  \]

\noindent When $k<n$, the trace of this variance has to be smaller than that of the ML/OLS estimator.

{\scshape Posterior Mean as a Penalized/Regularized OLS estimator:} \noindent Equation (\ref{equation:posterior}) is decreasing as a function of the \emph{penalized} sum of squared residuals
\[ (y-X\beta)^{\prime} (y-X \beta) + (\beta-\beta_0)^{\prime} V (\beta-\beta_0). \]

\noindent Under the Gaussian posterior the posterior mean and posterior mode are the same. Therefore, another way of deriving the posterior mean estimator is by finding a solution to the problem:
\[ \min_{\beta} (y-X\beta)^{\prime} (y-X \beta) + (\beta-\beta_0)^{\prime} V (\beta-\beta_0) \]
The F.O.C are
\[ X’(Y-X\beta) + V(\beta - \beta_0) = \textbf{0}_{n \times k} \iff  X’Y + V \beta_0 = (X’X + V) \beta. \]
\noindent Solving for $\beta$ gives the estimator $\widehat{\beta}_{\textrm{Bayes}}$.

\subsection{Optimality of the OLS estimator among unbiased estimators}

In this section we will show that the OLS estimator minimizes risk among the class of all unbiased estimators, provided some regularity conditions are met.

\begin{proposition}(Optimality of OLS) Consider the Normal regression model in (\ref{equation:Regression-model}). Let $\widehat{\beta}$ be an estimator of $\beta$ for which
\[ \frac{\partial}{\partial \beta} \int_{\mathbb{R}^{k}} \widehat{\beta} f(Y | \beta; X ) dY = \int_{\mathbb{R}^{k}} \widehat{\beta} \Big ( \frac{\partial}{\partial \beta} f(Y | \beta , X) \Big)’ dY,   \]
at any $\beta$ in the parameter space. If $\widehat{\beta}$ is unbiased, then
\[ \mathbb{V}_{\beta} ( \widehat{\beta} ) - \mathbb{V}_{\beta} (\widehat{\beta}_{\textrm{OLS}})  \]
is positive semi-definite, implying that the mean squared error of $\widehat{\beta}$ is larger than that of the OLS everywhere on the parameter space.
\end{proposition}

\begin{proof}
We would like show  that for any $c \in \mathbb{R}^{k}$, $c \neq 0$
\[ c’ \left( \mathbb{V}_{\beta} ( \widehat{\beta} ) - \mathbb{V}_{\beta} (\widehat{\beta}_{\textrm{OLS}}) \right) c \geq 0. \]

To do this, we start by computing the variance of $c’\widehat{\beta} - c’ \widehat{\beta}_{\textrm{OLS}}$,
which by definition of variance, has to be nonegative. Since $\widehat{\beta}$ and $\widehat{\beta}_{\textrm{OLS}}$ are both unbiased
\begin{eqnarray*}
\mathbb{V}_{\beta} \left( c’\widehat{\beta} - c’ \widehat{\beta}_{\textrm{OLS}} \right)&=&  \mathbb{V}_{\beta} (c’\widehat{\beta}) + \mathbb{V}_{\beta} (c’\widehat{\beta}_{\textrm{OLS}}) - 2 \textrm{cov}_{\beta} ( c’\widehat{\beta} , c’ \widehat{\beta}_{\textrm{OLS}}  ),  \\
&&  \textrm{(since $\textrm{Var}(X-Y) = \textrm{Var}(X) + \textrm{Var}(Y)- 2 \textrm{cov}(X,Y)$)}\\
&=& c’ \mathbb{V}_{\beta}(\widehat{\beta}) c + c’ \mathbb{V}_{\beta}(\widehat{\beta}_{\textrm{OLS}}) c - 2 \mathbb{E}_{\mathbb{P}_{\beta}} \left[  c’ (\widehat{\beta}-\beta) (\widehat{\beta}_{\textrm{OLS}}-\beta)’ c \right] \\
&& \textrm{(using the fact that both estimators are unbiased)}. \\
&=& c’ \mathbb{V}_{\beta}(\widehat{\beta}) c + c’ \mathbb{V}_{\beta}(\widehat{\beta}_{\textrm{OLS}}) c - 2 c’ \mathbb{E}_{\mathbb{P}_{\beta}} \left[  \widehat{\beta} (\widehat{\beta}_{\textrm{OLS}}-\beta)’  \right] c.
\end{eqnarray*}
Where the last term is the covariance between $\widehat{\beta}$ and the OLS residual. Using the definition of OLS
\[ \widehat{\beta}_{\textrm{OLS}}- \beta = (X’X)^{-1} X’(Y-X\beta),  \]
and using the definition of the the Gaussian p.d.f. $f(Y | \beta,X)$ we can see verify that
\[\widehat{\beta}_{\textrm{OLS}}- \beta =  \sigma^2  (X’X)^{-1} \frac{\partial}{\partial \beta} \ln f(Y | \beta, X ).  \]
Consequently
\begin{eqnarray*}
\mathbb{E}_{\mathbb{P}_{\beta}} \left[  \widehat{\beta} (\widehat{\beta}_{\textrm{OLS}}-\beta)’  \right] &=& \sigma^2 \mathbb{E}_{\mathbb{P}_{\beta}} \left[  \widehat{\beta} \left( \frac{\partial}{\partial \beta} \ln f(Y | \beta, X )\right)’ \right]  (X’X)^{-1}, \\
&=& \sigma^2 \left( \int_{\mathbb{R}^{k}} \widehat{\beta}  \left( \frac{\partial}{\partial \beta} \ln f(Y | \beta, X )\right)’  f(Y | \beta , X) dY \right) (X’X)^{-1} \\
&=& \sigma^2 \left( \int_{\mathbb{R}^{k}} \widehat{\beta}  \left( \frac{\partial}{\partial \beta} f(Y | \beta, X )\right)’ dY \right) (X’X)^{-1}\\
&& \textrm{(where we have used the chain rule and $\partial \ln x/ \partial x = 1/x$)}\\
&=& \sigma^2 \frac{\partial }{\partial \beta }\left( \int_{\mathbb{R}^{k}} \widehat{\beta}   f(Y | \beta, X )\right)  (X’X)^{-1}\\
&& (\textrm{by assumption}) \\
&=& \sigma^2 \left( \frac{\partial }{\partial \beta } \mathbb{E}_{\mathbb{P}_{\beta}}[\widehat{\beta}]  \right) (X’X)^{-1}\\
&=& \sigma^2 (X’X)^{-1}\\
&& \textrm{(since $\widehat{\beta}$ is unbiased).} \\
&=& \mathbb{V}_{\beta}(\widehat{\beta}_{\textrm{OLS}}).
\end{eqnarray*}
Hence,
\[ 0 \leq \mathbb{V}_{\beta} \left( c’\widehat{\beta} - c’ \widehat{\beta}_{\textrm{OLS}} \right) = c’ \mathbb{V}_{\beta}(\widehat{\beta}) c - c’ \mathbb{V}_{\beta}(\widehat{\beta}_{\textrm{OLS}}) c,  \]
for every $\beta$. The result then follows.
\end{proof}

\subsection{Suboptimality of the OLS estimator}

If $k \geq 3$, the OLS estimator is dominated. In this section we present an estimator that dominates OLS.

Assume that $X’X = \mathbb{I}_n$. Start with a Bayesian estimator for $\beta$ under the normal prior $\beta \sim \mathcal{N}_{k}(0,  v \mathbb{I}_k ) = \mathcal{N}_{k}(0,  \sigma^2 (v/\sigma^2) \mathbb{I}_k ) $. We have shown that such Bayes estimator is
\begin{eqnarray*}
\widehat{\beta}_{\textrm{Bayes}} &=& \left(  \mathbb{I}_k + \frac{\sigma^2}{v} \mathbb{I}_k  \right)^{-1} \widehat{\beta} _{\textrm{OLS}}, \\
&=& \left( \frac{v}{v+\sigma^2} \right) \widehat{\beta}_{\textrm{OLS}}, \\
&=& \left(1 -  \frac{\sigma^2}{v+\sigma^2} \right) \widehat{\beta}_{\textrm{OLS}}
\end{eqnarray*}
The hyperparameter $v$ ``shrinks’’ the OLS estimator towards the prior mean. Instead of picking $v$ \emph{a priori}, it is possible to use data to estimate it. Such an approach is usually called ``empirical Bayes’’.

The distribution of the data conditional on the parameter is
\[ \widehat{\beta}_{\textrm{OLS}} \sim \mathcal{N}_{k}(\beta, \sigma^2 \mathbb{I}_k).  \]
This conditional distribution, along with the prior, specify a full joint distribution over $(\widehat{\beta}_{\textrm{OLS}}, \beta)$. The marginal distribution of the data is
\[\widehat{\beta}_{\textrm{OLS}} \sim \mathcal{N}_{k}\left ( 0 , (\sigma^2  + v)  \mathbb{I}_k  \right). \]
We can use this statistical model to estimate the ``shrinkage’’ factor that appears in the Bayes estimator. Note that
\[ ||\widehat{\beta}_{\textrm{OLS}} ||^2 \sim (\sigma^2 + v) \chi^2_{k}, \]
Therefore, standard results for the mean of the inverse of a chi-square distribution yield
\[\mathbb{E} \left[ \frac{k-2 }{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right] = \frac{1}{v+\sigma^2}, \]
provided $k \geq 3$. An unbiased estimator for the ``shrinkage’’ factor that appears in the formula of $\widehat{\beta}_{\textrm{Bayes}}$ is thus:
\[\left(1 -  \frac{\sigma^2 (k-2)}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right). \]
We now show that the ``empirical Bayes’’ estimator
\[\widehat{\beta}_{JS} \equiv \left(1 -  \frac{\sigma^2 (k-2)}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right) \widehat{\beta}_{\textrm{OLS}}, \]
dominates OLS. This estimator was first proposed by Willard James and Charles Stein in 1961. The estimator is typically referred to as the James-Stein estimator.

\begin{proposition}
Suppose $X’X = \mathbb{I}_{k}$ and $\sigma^2=1$. If $k \geq 3$, the James-Stein estimator dominates the ML/OLS estimator.
\end{proposition}
\begin{proof}
\[ || \widehat{\beta}_{\textrm{JS}} - \widehat{\beta}_{\textrm{OLS}} ||^2  = || \widehat{\beta}_{\textrm{JS}} - \beta ||^2 + || \beta -\widehat{\beta}_{\textrm{OLS}} ||^2 + 2 ( \widehat{\beta}_{\textrm{JS}} - \beta  )’ (\beta -\widehat{\beta}_{\textrm{OLS}}) \]
implies
\[|| \widehat{\beta}_{\textrm{JS}} - \beta ||^2 = || \widehat{\beta}_{\textrm{JS}} - \widehat{\beta}_{\textrm{OLS}} ||^2 - || \beta -\widehat{\beta}_{\textrm{OLS}} ||^2 + 2 ( \widehat{\beta}_{\textrm{JS}} - \beta  )’ (\widehat{\beta}_{\textrm{OLS}}- \beta). \]
Taking expectation on both sides yields
\begin{eqnarray*}
\mathbb{E}_{\mathbb{P}_{\beta}} \left[ || \widehat{\beta}_{\textrm{JS}} - \beta ||^2 \right] & = & \mathbb{E}_{P_{\beta}} \left[  || \widehat{\beta}_{\textrm{JS}} - \widehat{\beta}_{\textrm{OLS}}||^2 \right] \\
&-& \mathbb{E}_{\mathbb{P}_{\beta}} \left[ || \widehat{\beta}_{\textrm{OLS}}- \beta ||^2 \right] \\
&+& 2 \mathbb{E}_{\mathbb{P}_{\beta}} \left[ (\widehat{\beta}_{\textrm{JS}} -\beta)’( \widehat{\beta}_{\textrm{OLS}}- \beta)  \right].
\end{eqnarray*}
Algebra shows that
\[\widehat{\beta}_{JS} - \widehat{\beta}_{\textrm{OLS}} \equiv \left(  \frac{(k-2)}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right) \widehat{\beta}_{\textrm{OLS}}. \]
Therefore,
\[  \mathbb{E}_{\mathbb{P}_{\beta}} \left[ ||\widehat{\beta}_{JS} -\beta ||^2 \right] =  \mathbb{E}_{\mathbb{P}_{\beta}}  \left[ \frac{(k-2)^2}{|| \widehat{\beta}_{\textrm{OLS}} ||^2} \right] - k + 2 \mathbb{E}_{\mathbb{P}_{\beta}} \left[ (\widehat{\beta}_{\textrm{JS}} -\beta)’( \widehat{\beta}_{\textrm{OLS}}- \beta)  \right]. . \]
Also
\begin{eqnarray*}
\mathbb{E}_{\mathbb{P}_{\beta}} \left[ (\widehat{\beta}_{\textrm{JS}} - \beta)’( \widehat{\beta}_{\textrm{OLS}}- \beta)  \right] &=& \sum_{j=1}^{k} \textrm{Cov}( \widehat{\beta}^j_{\textrm{JS}},\widehat{\beta}^j_{OLS} ).
\end{eqnarray*}
and the last term can be shown to equal
\begin{eqnarray*}
\sum_{j=1}^{k} \mathbb{E}_{\mathbb{P}_{\beta}} \left[ \frac{\partial \widehat{\beta}^j_{JS} }{\partial \widehat{\beta}^j_{\textrm{OLS}}} \right] &=& K - \mathbb{E}_{\mathbb{P}_{\beta}} \left[ \frac{k  (k-2)}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right] + \mathbb{E}_{\mathbb{P}_{\beta}} \left[ \frac{2(k-2)}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right]’ \\
&=&  K - \mathbb{E}_{\mathbb{P}_{\beta}} \left[ \frac{(k-2)^2}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right]
\end{eqnarray*}
Therefore,
\[  \mathbb{E}_{\mathbb{P}_{\beta}} \left[ ||\widehat{\beta}_{JS} -\beta ||^2 \right] = k - \mathbb{E}_{\mathbb{P}_{\beta}} \left[ \frac{(k-2)^2}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right] \leq k =\mathbb{E}_{\mathbb{P}_{\beta}} \left[ ||\widehat{\beta}_{OLS} -\beta ||^2 \right].    \]
\end{proof}

\newpage

\bibliographystyle{../AuxFiles/ecta}
\bibliography{../AuxFiles/BibMaster}

\end{document}


\lab{Application}{Stochastic Cake-Eating Problem}{Stochastic Cake-Eating Problem}
\newcommand\ve{\varepsilon}

\objective{In this section we study the stochastic cake-eating problem with both normal and AR(1) shocks.}

\section*{Infinite Horizon, Stochastic, i.i.d.}\label{SecRecProbInfinHorStochiid}

In practice, dynamic programming problems often involve some level of uncertainty.  For example as time progresses prices may change, resources may vary, or preferences themselves may change.  In this lab, we reexamine the cake eating problem, this time allowing for uncertainty.

We consider again the problem of opimizing a sequence of decisions over an infinite time horizon.  We assume that the individuals preferences fluctuate each period according to some "shock" $\ve$, meaning $\ve$ is a random variable.  We assume that the $\ve$ are identically and independently distributed (i.i.d).  In effect, this means the probabilities associated with the $\ve$ are the same for any time $t$ and do not depend on each other.  We assume for now that the $\ve$ are distributed normally with mean $\mu$ and variance $\sigma^2$.  The Bellman equation can be easily rewritten in the following way to incorporate the uncertainty,
\begin{equation*}\label{stoch_Bellman}
   V\left(W,\ve\right) = \max_{W'\in[0,W]}\: \ve u\left(W - W'\right) + \beta E_{\ve'}\left[V\left(W',\ve'\right)\right] \quad\text{where}\quad \ve \sim \text{N}(\mu,\sigma^2)
\end{equation*}
where $E$ is the unconditional expectations operator over $\ve$.  Note that now the value function is a function of two variables.  It represents the value of entering the period with $W$, the amount of cake, and a preference shock of $\ve$.  For example, in a period where the realization of $\ve$ is higher, we will get more value from the cake eaten in the current period.  Because we do not know the value of the shock in the next period $\ve'$, we consider only the expected value for future time.

It turns out, we can solve this problem in a manner similar to the infinite horizon deterministic cake-eating problem considered in the Value Function Iteration lab.  It is worth noting that in this case, computationally the value and policy functions will be two dimensional as they will depend on both $W$ and $\ve$.

In order to deal with $\ve$ computationally, we would like to represent it as a vector of possible values it could take along with the corresponding probabilities that it takes each of those values.  However, $N(\mu,\sigma^2)$ is a continuous distribution so we cannot represent every value $\ve$ could take.  We need a discrete distribution that approximates $N(\mu,\sigma^2)$.

To do this we can think of breaking the distribution $N(\mu,\sigma^2)$ up into bins with endpoints $v_1,v_2,\ldots, v_K$.

\begin{figure}[h!]
\label{stoch1_fig1}
\begin{center}
\includegraphics[scale = 0.5]{discnorm.pdf}
\end{center}
\caption{Discretization of $N(\mu,\sigma^2)$.  We approximate $P(\ve = \ve_k)$ by the area of the shaded region.}
\end{figure}

We can then associate $\ve_k$ with the area under the curve from $v_k$ to $v_{k+1}$.  In python we can find the area using the function norm.cdf in the stats package.  The cdf function gives the area under the curve from minus infinity to a specified value.  For example, in the following code, eps is the area under the curve from 0 to 1.

\begin{lstlisting}[style=python]
from scipy import stats as st

mu = 0
sigma = 1
eps = st.norm.cdf(1,mu,sigma) - st.norm.cdf(0,mu,sigma)
\end{lstlisting}

In general, it is sufficient to take our points $\ve_k$ ranging from $\mu - 3\sigma$ to $\mu + 3\sigma$.

\begin{problem}
Write a function called discretenorm that accepts an integer representing the number of discrete points desired, a mean, and a standard deviation and returns a 1 by N vector of values ranging from $\mu - 3\sigma$ to $\mu + 3\sigma$  and a 1 by N vector of the associated probabilities.  Plot the approximation of $N(0,1)$ using 7 points.
\end{problem}

Now that we have a discrete distribution for $\ve$, we can solve for the value and policy functions determined by \eqref{stoch_Bellman}.

\begin{problem}
Complete the following steps to solve the problem described above.
\begin{enumerate}
   \item First we establish our approximation of $\ve$ using the discretenorm function created in Problem 1. Let $\sigma^2 = 0.25$ and $\mu=4\sigma$ (This way the shocks, $\ve$, are always positive.) Use $K=7$ equally spaced points to approximate $N(\mu,\sigma^2)$ so that $\ve$ is a $K$-length row vector.  Using the function discretenorm from problem 1, generate the probability distribution $\Gamma$ for $\ve$. This should be a $K$-length row vector whose entries indicate $P(\ve = \ve_k)$

   \item As in the Value Function Iteration lab, assume that the vector of possible cake sizes is $W$ with $N=100$ equally spaced values between 0 and 1.  Represent the value function as a matrix with each row corresponding to different values of $W'$ and each column corresponding to different values of $\ve'$. Initialize the value function to an $N$ by $K$ matrix of zeros.  Assume that the period utility function is $u(c)=\sqrt{c}$, and that the discount factor is $\beta = 0.9$.

   \item In order to evaluate the value function equation we need to compute $\ve u(W-W')$ for all values of $\ve,W,W'$.  Thus $\ve u(W-W')$ will be represented by a three-dimensional array of size $N\times N\times K$.  This can be achieved using the tile function.  For example, if $u(W-W')$ is represented by the $N\times N$ matrix util\_grid, and eps is the vector of $\ve$ values, then we could create the $\ve u(W-W')$ matrix by the following:

       \begin{lstlisting}[style = python]
       util3 = sp.tile(util_grid[:,:,sp.newaxis],(1,1,K))
       eps_grid = eps[sp.newaxis,sp.newaxis,:]
       eps_util = eps_grid*util3
       \end{lstlisting}

       The input (1,1,K) tells sp.tile to repeat util\_grid K times in the third dimension.  Note we have to put util\_grid[:,:,sp.newaxis] in order to have a three-dimensional input since util\_grid is only two dimensional.

       As in the Value Function Iteration Lab, we replace negative entries in $W-W'$ by 0.

   \item We also need to compute $E_{\ve'}\Bigl[V_k\left(W',\ve'\right)\Bigr]$.  The expected value is simply
       \begin{equation*}
       E_{\ve'}\Bigl[V_k\left(W',\ve'\right)\Bigr] = \sum_k \Gamma(k)V_k(W',\ve_k')
       \end{equation*}  The result is an $N\times 1$ vector that gives the value for each $W'$.  However we need an $N\times N\times K$ array to add to $\ve u(W-W')$.  Since the expected value does not depend on $W,\ve$, we tile the vector to create a 3D array that is constant along the $W$ and $\ve$ dimensions.

   \item We can now compute the value function contraction
     \begin{equation}\label{EqContractStochiid}
      V_{k+1}\left(W,\ve\right) \equiv C\Bigl(V_k\left(W,\ve\right)\Bigr) \equiv \max_{W'\in[0,W]}\: \ve u\left(W-W'\right) + \beta E_{\ve'}\Bigl[V_k\left(W',\ve'\right)\Bigr].
      \end{equation}
        Before doing so, replace the entries of $\ve u\left(W-W'\right) + \beta E_{\ve'}\Bigl[V_k\left(W',\ve'\right)\Bigr]$ that corrsepond to $W-W' < 0$ with a large negative number (e.g. $-10^{10}$) so they will not be chosen in the maximization.

   \item As we iterate on the value function equation, we need a norm
    \begin{equation*}
        \delta_k = \norm{V_k\left(W,\ve\right) - V_{k-1}\left(W,\ve\right)}
    \end{equation*}
    that measures the distance between the two value functions to determine convergence.  Compute the norm using the scipy function scipy.linalg.norm.  Iterate on the contraction until $\delta_k < 10^{-9}$.

   \item Make a 3-D surface plot of the policy function for the converged problem $W' = \psi\left(W,\ve\right)$ which gives the value of the cake tomorrow as a function of the cake today  and the taste shock today.  Do the same for the value function.

\end{enumerate}
\end{problem}

\section*{Infinite Horizon, Stochastic, AR(1)}\label{SecRecProbInfinHorStochAR1}

In the previous example we assumed that the shocks at time $t$ were independent of what happened in previous periods.  Often a shock may depend on recent events.  We will assume now that the shocks are persistent, meaning preferences in the current period are more likely to be close to what they were in the previous period.  We can characterize the persistence by what is called an autoregressive process of order one, denoted AR(1).  Such a process is defined as follows.
\begin{equation}\label{EqAR1shock}
   \ve' = (1-\rho)\mu + \rho\ve + \nu' \quad\text{where}\quad \rho\in(0,1) \quad\text{and}\quad \nu\sim N(0,\sigma^2)
\end{equation}

Essentially, instead of allowing the shocks to have a mean which is independent of the past, the mean is now a weighted average (weighted by $\rho$) of some $\mu$ and the previous realization of the shock, $\ve$.  As it turns out, we can approximate this process by thinking of it like a Markov Chain. This means we need to determine a discrete set of points representing possible values of $\ve$ and a Markov transition matrix that gives the probabilities of moving from one value of $\ve$ to another.  There are methods for determining the discrete approximation of $\ve$ with a Markov transition matrix.  The methods are beyond the scope of this section, but you can use the file tauchenhussey.py to implement them in the next problem.

The Bellman equation becomes the following, in which the only change from the i.i.d. shock case is that the expectations operator is now conditional on the current shock $\ve$.

\begin{equation*}
   V\left(W,\ve\right) = \max_{W'\in[0,W]}\: \ve u\left(W - W'\right) + \beta E_{\ve'|\ve}\left[V\left(W',\ve'\right)\right]
\end{equation*}
where $\ve'$ is distributed according to \eqref{EqAR1shock}. Let $\Gamma(\ve'|\ve)=\text{Pr}\left(\ve_j'|\ve_i\right)$ where $\ve_j'$ is the value of the shock in the next period and $\ve_i$ is the value of the shock in the current period.  In other words, $\Gamma(\ve' | \ve)$ is the Markov transition matrix.

The solution to this problem is of the same type as that in the i.i.d. case since the only difference is the probability distributions of the $\ve$.

\begin{problem}
\begin{enumerate}
   \item Use the file tauchenhussey.py to approximate the AR(1) process for $\ve$ from \eqref{EqAR1shock} as a Markov chain. The provided Python function tauchenhussey.py will produce a vector of length $M$ for the support of $\ve$ and an $M\times M$ transition matrix $\Gamma(\ve'|\ve)=\text{Pr}\left(\ve_j'|\ve_i\right)$ where each element in row $i$ and column $j$ represents the probability of the shock $\ve_j'$ next period given the current shock is $\ve_i$. As inputs to tauchenhussey.py, let $M=7$, the mean of the process $\mu=4\sigma$, $\rho = 1/2$, $\sigma=\sqrt{\sigma^2}=1/2$, and $basesigma=(0.5+\frac{\rho}{4})\sigma + (0.5 - \frac{\rho}{4})*\frac{\sigma}{\sqrt{1-\rho^2}}$.

  \item Solving this version of the problem should be very similar to the i.i.d. case completed in problem 2.  The most significant difference is that now we need to compute the conditional expectation
      \begin{equation}
      E_{\ve'|\ve}\left[V\left(W',\ve'\right)\right].
      \end{equation}
      In this case, the expectation is two-dimensional since it depends on both $W'$ and on $\ve$.  The expectation can be computed by matrix multiplying $V(W',\ve')\Gamma(\ve' | \ve)'$ where the last "$'$" represents transposing $\Gamma$.  Again we will need an $N\times N\times K$ array, so we tile the result $K$ times representing that the expectation is constant along $W$.

  \item Solve for the optimal policy by Value Function Iteration.  Plot the value function and policy function surfaces.

\end{enumerate}
\end{problem}


\documentclass[11pt]{article}
\usepackage{
  amsmath,
  amssymb,
  tikz,
  microtype,
  amsthm,
  thmtools,
  stmaryrd,
  hyperref,
  algorithm2e
}

\title{Special topics course: Numerical Polynomial Algebra}
\author{Anand Deopurkar and Markus Hegland}
\date{ANU winter semester 2018}

\renewcommand{\algorithmautorefname}{Algorithm}
\newcommand{\C}{\mathbb C}
\newcommand{\Z}{\mathbb Z}
\newcommand{\R}{\mathbb R}
\DeclareMathOperator{\LT}{LT}
\DeclareMathOperator{\wt}{wt}

\declaretheorem{theorem}
\declaretheorem[sibling=theorem, style=theorem]{proposition}
\declaretheorem[sibling=theorem, style=definition]{definition}
\declaretheorem[sibling=theorem, style=definition]{example}
\declaretheorem[sibling=theorem, style=definition]{problem}
\declaretheorem[sibling=theorem, style=remark]{remark}

\begin{document}

\maketitle

\part*{Algebraic introduction}
This is a quick introduction to the commutative algebra relevant to systems of polynomial equations.
Our main references are Stetter \cite{stetter04} and Cox, Little, O'Shea \cite{cox15}.

\tableofcontents

\section{Polynomials and ideals}

Let $k$ be a field.
Denote by $k[x_1,\dots,x_s]$ the set of polynomials in variables $x_1, \dots, x_s$ with coefficients in $k$.
Stetter denotes $k[x_1,\dots,x_s]$ by $P^s_k$.
For numerical analysis, we will take the base-field $k = \R$, but a lot of the algebraic geometry works best with $k = \C$.
The case of $k = \Z/p\Z$ is used frequently for exact symbolic computation.
We will try to be base-field agnostic; but for some results, the base-field will play an important role.

Let $f_1, \dots, f_k$ be elements of $k[x_1,\dots, x_s]$.
We are interested in solving the system of polynomial equations
\begin{align}\label{eqn:system}
  \begin{split}
  f_1(x_1, \dots, x_s) &= 0,\\
  &\vdots\\
  f_n(x_1, \dots, x_s) &= 0.
\end{split}
\end{align}
Our experience with systems of linear equation suggests that we should look at not only the $n$ equations listed above, but also their linear combinations.
This idea suggests the following.
\begin{definition}[Ideal]
  A subset $I$ of $k[x_1, \dots, x_s]$ is called an \emph{ideal} if the following two conditions are satisfied.
  \begin{enumerate}
  \item If $f, g \in I$, then $f+g \in I$.
  \item If $f \in I$ and $c \in k[x_1, \dots, x_s]$, then $cf \in I$.
  \end{enumerate}
\end{definition}
The definition of an ideal should be reminiscent of the definition of a subspace of a vector space.
The only difference is that the multipliers in the second condition are not necessarily scalars (elements of $k$) but polynomials.
By putting the two conditions together, we see that an ideal is a subset of $k[x_1, \dots, x_s]$ that is closed under polynomial linear combinations.
That is, if $f_1, \dots, f_n \in I$ and $c_1, \dots, c_n \in k[x_1, \dots, x_s]$, then $c_1f_1 + \dots + c_nf_n \in I$.

\begin{example}[The unit ideal]
  The set $I = k[x_1,\dots,x_s]$ is obviously an ideal; it is called the unit ideal.
  Observe that it is the only ideal that contains $1$.
\end{example}

\begin{example}[Ideal generated by a set of polynomials]
  If $f_1, \dots, f_n$ are elements of $k[x_1, \dots, x_s]$, then the set consisting of all polynomial linear combinations of $f_1, \dots, f_s$, namely
  \[ I = \{c_1f_1 + \dots + c_nf_n \mid c_1, \dots, c_n \in k[x_1,\dots,x_s]\}\]
  is an ideal.
  It is the smallest ideal containing $f_1, \dots, f_n$.
  We say that $I$ is \emph{generated by} $f_1, \dots, f_n$, and write
  \[ I = \langle  f_1, \dots, f_n \rangle.\]
\end{example}

A fundamental theorem in commutative algebra is that every ideal has a finite generating set.
\begin{theorem}[Hilbert basis theorem]
  Let $I \subset k[x_1, \dots, x_s]$ be an ideal.
  Then there exists polynomials $f_1, \dots, f_n$ such that $I = \langle  f_1, \dots, f_n \rangle$.
\end{theorem}
There are multiple generating sets for a given ideal, just as there are multiple spanning sets for a subspace of a vector space.
Depending on what we want to do, some generating sets are more useful than others.
A common theme in the course will be to learn how to construct and use good generating sets.

\subsection{Zero loci of ideals}
Let $I \subset k[x_1, \dots, x_s]$ be an ideal.
We can associate to $I$ a subset $V(I)$ of $k^s$ called the \emph{zero locus} or the \emph{vanishing locus} of $I$.
It is defined as follows:
\[
  V(I) = \{x \in k^s \mid f(x) = 0 \text{ for all $f \in I$}\}.
\]
Observe that if $I = \langle  f_1, \dots, f_n \rangle$, then $V(I)$ is simply the set of solutions of the system of equations \eqref{eqn:system}.
Thus, solving \eqref{eqn:system} is equivalent to finding $V(I)$.

A subset of $k^s$ of the form $V(I)$ for some ideal $I$ is called an \emph{algebraic set}.
Some authors also call an algebraic set an \emph{algebraic variety}, but some other authors reserve the term \emph{variety} only for special types of algebraic sets (namely, the ``irreducible'' ones).

We can turn the construction $I \leadsto V(I)$ around.
Starting with a subset $Z \subset k^s$, we can construct an ideal $I(Z) \subset k[x_1, \dots, x_s]$ as follows:
\[ I(Z) = \{f \in k[x_1,\dots,x_s] \mid f(x) = 0 \text{ for all $x \in Z$}\}.\]
It is easy to check that $I(Z)$ is an ideal.
Furthermore, it is easy to verify that $I(Z)$ is a \emph{radical} ideal.
That is, it satisfies the property that if $f^n \in I(Z)$ for some $n \geq 1$, then $f \in I(Z)$.
(This property is not shared by all ideals.
For example, the ideal $\langle  x^2 \rangle \subset k[x]$ is not radical---it contains $x^2$, but not $x$.)

\begin{example}[Ideal of $3$ points in $k^2$]
  \label{ex:3pts}
  Let $Z = \{(0,0), (0,1), (1,0)\}$.
  Then we have
  \[ I(Z) = \langle x(x-1), xy, y(y-1)) \rangle.\]
\end{example}
\begin{example}[Ideal of the ``twisted cubic'']
    \label{ex:twistedcubic}
Let $Z = \{(t, t^2, t^3) \mid t \in \C\} \subset \C^3$.
  Then
  \[ I(Z) = \langle  x^2-y, y^2-xz, x^3-z \rangle.\]
\end{example}
\begin{example}[Ideal of $4$ points in $k^2$]
  \label{ex:4pts}
  Let $Z = \{(0,0), (1,0), (0,1), (1,1)\}$.
  Then
  \[ I(Z) = \langle  x(x-1), y(y-1) \rangle.\]
\end{example}
\begin{remark}
  The ideal in \autoref{ex:4pts} is different in a crucial way from the ideals in \autoref{ex:3pts} and \autoref{ex:twistedcubic}.
  It is a \emph{complete intersection} ideal.
  A complete intersection ideal is an ideal that is generated by as many polynomials as the codimension of its vanishing set.
  In \autoref{ex:4pts}, the codimension of the vanishing set is 2, and indeed, the ideal is generated by 2 polynomials.
  One can prove that if the base-field is algebraically closed, then an ideal needs at least as many generators as the codimension of its vanishing locus.
  A complete intersection ideal is one where the bound is achieved.

  A complete intersection ideal is stable under perturbation of its generators.
  We will not make this precise, but illustrate it in the examples above.
  If we perturb one (or both) of the generators of the ideal in \autoref{ex:4pts}, then the perturbed ideal remains the ideal of $4$ points.
  In contrast, if we perturb the generators of the ideals in \autoref{ex:3pts} or \autoref{ex:twistedcubic}, then the ideal changes dramatically in shape.
  In \autoref{ex:3pts}, a perturbation of the generators will make the ideal become the unit ideal.
  In \autoref{ex:twistedcubic}, a perturbation of the generators will make the ideal become the ideal of a finite set of points.
  You are encouraged to verify these statements about perturbations either by hand or using a computer algebra system.
\end{remark}

\subsection{The Nullstellensatz}
The following important theorem links the constuctions $I \mapsto V(I)$ and $Z \mapsto I(Z)$.
\begin{theorem}[Nullstellensatz]
  Suppose $k$ is an algebraically closed field (for example, $k = \C$).
  Then we have a one-to-one correspondence
  \[
    \{\text{Radical ideals of $k[x_1,\dots,x_s]$}\}
    \leftrightarrow
    \{\text{Algebraic subsets of $k^s$}\},
  \]
  where the $\rightarrow$  direction is given by $I \mapsto V(I)$ and the $\leftarrow$ direction is given by $I(Z) \mapsfrom Z$.
\end{theorem}

\begin{example}[Failure of the Nullstellensatz for non-closed fields]
  In the Nullstellensatz, it is essential for the base-field to be algebraically closed.
  To see what can go wrong otherwise, take $k = \R$ and $I = \langle  x^2 + 1 \rangle \subset k[x]$.
  Then $I$ is a radical ideal, and we have $V(I) = \emptyset$.
  But we also have $V(\langle  1 \rangle) = \emptyset$.
  So the correspondence is not one-to-one.

  This example shows that when $k$ is ``too small'' (not algebraically closed), then the geometric (right) side may fail to faithfully capture the algebraic (left) side.
  But this happens only because the points on the right hand side that \emph{should} be there are not there because the base-field is too small.
\end{example}

A consequence of the Nullstellensatz is the following dichotomy for systems of polynomial equations.
\begin{theorem}\label{thm:null}
  Consider the system of polynomial equations $f_1 = 0, \dots, f_n = 0$ over an algebraically closed field $k$.
  Then either there exists a solution or there exist polynomials $c_1, \dots, c_n$ such that
  \[ 1 = c_1 f_1 + \dots + c_nf_n.\]
  Equivalently, given an ideal $I$, either $V(I)$ is non-empty, or $I = \langle  1 \rangle$.
\end{theorem}
\begin{proof}
  Suppose $V(I)$ is empty.
  We must prove that $1 \in I$.
  If $I$ is radical, then we have
  \[ V(I) = V(\langle 1 \rangle) = \emptyset.\]
  So by the Nullstellensatz, we have $I = \langle  1 \rangle$, and we are done.

  If $I$ is not radical, then we replace it by its radical $\sqrt I$, defined by
  \[ \sqrt I = \{f \mid f^n \in I \text{ for some $n \geq 1$}\}.\]
  It is not too hard to check (do it!) that $\sqrt I$ is an ideal, is radical, and $V(I) = V(\sqrt I)$.
  From the argument above, we get that $\sqrt I = \langle  1 \rangle$, that is $1 \in \sqrt I$.
  But by the definition of $\sqrt I$, this means that $1^n \in I$ for some $n \geq 1$, so $I = \langle  1 \rangle$.
\end{proof}

\begin{remark}
  Over the real numbers, although the Nullstellensatz fails, an analogous theorem called the Positivestellensatz holds.
  Although it is tempting, we will not discuss this further.
  This topic belongs to an interesting field of mathematics that lies on the interface of algebraic geometry, combinatorics, applied mathematics, and logic, called \emph{real algebraic geometry}.
\end{remark}

\section{Ideal membership and the division algorithm}
Let us return to the system of polynomial equations
\begin{align}\label{eqn:system1}
  \begin{split}
  f_1(x_1, \dots, x_s) &= 0,\\
  &\vdots\\
  f_n(x_1, \dots, x_s) &= 0.
\end{split}
\end{align}
As a first step towards solving this system, we would like to determine whether the system has any solutions at all.
By \autoref{thm:null}, this is equivalent to determining whether $1$ lies in the ideal $\langle  f_1, \dots, f_n \rangle$.
In this section, we will consider a general version of this problem.
\begin{problem}[Ideal membership]
  Given polynomials $f_1, \dots, f_n$ and a polynomial $g$, determine whether $g$ lies in the ideal $\langle  f_1, \dots, f_n \rangle$.
\end{problem}

\subsection{The one-variable case}
Suppose $s = 1$ and $n = 1$.
So, we are given a (non-zero) polynomial $f(x)$ and another polynomial $g(x)$.
We want to determine whether $g(x)$ is lies in the ideal $\langle  f(x) \rangle$, that is, whether $g(x)$ is a multiple of $f(x)$.
Here we know what to do: simply run the division algorithm (\autoref{alg:divalg1}).

\begin{algorithm}
  \While{
    $\deg (g) \geq \deg(f)$
  }
  {
    Replace $g$ by $g - \frac{\LT(f)}{\LT(g)} f$, where $\LT$ denotes the leading (highest degree) term.
  }
  Return $g$ as the remainder.
  \bigskip

  \caption{The division algorithm for $g(x) \pmod {f(x)}$}
  \label{alg:divalg1}
\end{algorithm}

In the one-variable case, we have the following key fact.
\begin{proposition}
  The polynomial $g(x)$ lies in $\langle  f(x) \rangle$ if and only if the remainder obtained in the division algorithm for $g(x) \pmod {f(x)}$ is zero.
\end{proposition}

\subsection{Term orderings}
To extend the division algorithm to handle more than one variable and ideals generated by more than one polynomial, we must first extend the notion of the leading term.
It turns out that there are multiple ways to do this.
It gives rise to the idea of a term ordering.

Let $T^s$ be the set of monomials in the variables $\{x_1, \dots, x_s\}$.
We use multi-indices to denote the elements in $T^s$.
For a multi-index $i = (i_1, \dots, i_s)$, the notation $x^i$ means the monomial $x_1^{i_1} \cdots x_s^{i_s}$.
\begin{definition}[Term ordering]
  A \emph{term ordering} is a total ordering $\prec$ on the elements of $T^s$ satisfying the following two properties:
  \begin{enumerate}
  \item if $x^i \prec x^j$ then $x^i \cdot x^k \prec x^j \cdot x^k$ for all multi-indices $k$.
  \item $1 \prec x^i$ for all multi-indices $i$.
  \end{enumerate}
\end{definition}

For $s = 1$, there is only one possible term ordering, namely \[1 \prec x \prec x^2 \prec \cdots.\]
For $s > 1$, however, there are many possibilities.
\begin{example}[Lexicographical ordering $\prec_{\rm lex}$]
  Fix an ordering on the variables, say $x_s \prec x_{s-1} \prec \cdots \prec x_1$.
  Extend this ordering to monomials using the lexicographical rule.
  That is, set $x^i \prec_{\rm lex} x^j$ if the first non-zero entry of $j-i$ is positive.
\end{example}

\begin{example}[Graded lexicographical ordering $\prec_{\rm grlex}$]
  Fix an ordering on the variables, say $x_s \prec x_{s-1} \prec \cdots \prec x_1$.
  Extend this ordering to monomials by first comparing the total degree and then using the lexicographical rule.
  That is, set $x^i \prec_{\rm grlex} x^j$ if $\deg(x^i) < \deg(x^j)$ or if $\deg(x^i) = \deg(x^j)$ and $x^i \prec_{\rm lex} x^j$.
\end{example}

\begin{example}[Weight ordering]
  Assign a weight $\alpha_i \in \R_{> 0}$ to each variable $x_i$.
  Extend the weight assignment to monomials by taking a weighted sum
  \[ \wt(x^i) = \sum_{\ell = 1}^s i_\ell \alpha_\ell.\]
  Use this to define a term ordering by setting $x^i \prec_\alpha x^j$ if $\wt(x^i) < \wt(x^j)$.

  We have to take care that there are no ties in the weights of the monomials (or devise a tie-breaker).
  For generic (almost all) choices of weights, there will be no ties.
\end{example}

Fix a term ordering $\prec$.
Using $\prec$, we can define the leading term of a polynomial.
For $f \in k[x_1,\dots,x_s]$, let $\LT(f)$, or more precisely $\LT_\prec(f)$, be the term of $f$ with the highest monomial (according to $\prec$).
For example, we have
\begin{align*}
  \LT_{\prec_{\rm lex}} (5x^3y^3-3x^4)  &= -3x^4 \\
  \LT_{\prec_{\rm grlex}} (5x^3y^3-3x^4)  &= 5x^3y^3.
\end{align*}
We can now generalize the division algorithm as follows.

\begin{algorithm}
  \While{
    there exists an $i$ such that $\LT(f_i)$ divides $\LT(g)$.
  }
  {
    Replace $g$ by $g - \frac{\LT(f_i)}{\LT(g)} f_i$.
  }
  Return $g$ as the remainder.
  \bigskip

  \caption{The division algorithm for $g \pmod {f_1, \dots, f_n}$}
\end{algorithm}

It is easy to see that if the division algorithm for $g \pmod {f_1, \dots, f_n}$ returns $0$ as the remainder, then $g$ is in the ideal $\langle  f_1, \dots, f_n \rangle$.
However, the converse is not true, as the next examples show.
This severely diminishes the utility of the division algorithm.

The division algorithm suffers from another drawback.
Note that there may be choices involved in the running of the algorithm.
Namely, if there are multiple $i$'s such that $\LT(f_i)$ divides $\LT(g)$, then we may choose any one and proceed.
The final remainder obtained may change depending on the choices made in the algorithm.

Nevertheless, we will see in the next section, that we can overcome both o these drawbacks by making a judicious choice of the generating set for our ideal.

\begin{example}[Failure of the division algorithm]
  Let us work with $\prec_{\rm grlex}$.
  Take $f_1  = x^3-2xy$ and $f_2 = x^2y-2y^2+x$; and $g = x^2$.
  Then the division algorithm stops without any computation, and returns $x^2$ as the remainder.
  However, the following computation shows that $g$ is actually in the ideal $\langle  f_1, f_2 \rangle$:
  \[ g = x \cdot f_2 - f_1.\]
\end{example}

\begin{example}[Failure of the division algorithm]
  We can work in any term ordering.
  Take $f_1 = xy-1$, $f_2 = x^3-y^3$, and $f_3 = x^2+y^2$.
  Check that $1 \in \langle  f_1, f_2, f_3 \rangle$ (either directly or using that $V(\langle  f_1, f_2, f_3 \rangle) = \emptyset$).
  However, the division algorithm stops without any computation, and returns $1$ as the remainder.
\end{example}

\subsection{Gr\"obner bases}

\bibliography{ch2}
\bibliographystyle{amsalpha}

\end{document}