\chapter{Background \& Related Work}
\label{chap:background}

In this section, constraint optimization and the distributed, as well as dynamic variants are briefly explained and brought into context of the related work. Also, the meeting scheduling problem will be described and different algorithm designs and their advantages and disadvantages are going to be briefly discussed.

\section{Dynamic Distributed Constraint Optimization}

% -----------------Constraint Optimization --------------------

A constraint optimization problem (COP) contains a set of variables \(V=\{V_{1},V_{2}, ...,  V_{n}\}\). These variables are assigned to a value or state \(s_{j} \in S_{j}\), which is contained in a set of possible values defined by a finite problem domain \(D=\{D_{1},D_{2}, ...,  D_{n}\}\). A constraint \(C = <V_{c}, R_{c}>\) contains one (unary), two (binary) or multiple (k-ary) variables and their relationship. The constraint defines a rule for the variables that needs to be fulfilled. One of those rules could be that none of the variables should take the same value. This would for example be the case for a meeting scheduling problem where none of the meetings should take place at the same time. \newline
A utility function for the constraint \({c}_{k}\) on variable state \(s\) in the form of \(u_{{c}_{k}}(s_{{c}_{k}})\) needs to be formulated that defines a certain cost respectively reward for a given configuration of the involved states. The global utility function \(u_{g}\) would then be the summation of all utility functions of all constraints.

\[u_{g}(s) = u_{c_{1}}(s_{c_{1}}) \oplus \cdots \oplus u_{c_{k}}(s_{c_{k}}) \oplus \cdots \oplus u_{c_{l}}(s_{c_{l}}) \]

Constraints can be attributed with varying levels of importance through weighting. One can, instead of so-called soft constraints, define hard constraints by multiplying their utility instead of using addition in the global utility function. By defining the utility of a violated hard constraint as 0, the global utility would also go to 0 if this hard constraint is not satisfied \cite{Chapman2011, Petcu2003}. A problem only containing hard constraints would represent a constraint satisfaction problem (CSP). A formal definition of such a combined utility function including soft constraints (SC) and hard constraints (HC) would look like the following formula, where the product of all hard constraints is multiplied by the sum of all utilities of a state \(s\) in the soft constraint utility functions:

\[ u_{g}(s) = \prod_{\substack{hc_{k} \in HC}} u_{SC_{g}}(s) \bigg( \sum_{sc_{k} \in SC} u_{SC_{g}}(s) \bigg)\]

% ----------------- Distributed Constraint Optimization --------------------
The definition of a distributed constraint optimization problem (DCOP) extends the basic constraint optimization by distributing sets of variables to autonomous agents. These agents all have the goal to maximize the utility of their variables in a private utility function and thereby also contribute to a global utility function. Agent's whose variables are linked to at least one common constraint are called neighbours \cite{Chapman2011, Farinelli, Petcu2003}.
\newline\newline
% ----------------- Dynamic Distributed Constraint Optimization --------------------
The problem definition in dynamic distribute constraint optimization (DynDCO) is, as a further extension to DCOPs, moved from a static to a dynamic attribute. Constraints can change and  therefore change neighbourhoods and the outcome of private and global utility functions. A change of constraints inherently changes the area of satisfying solutions if hard constraints have been included in a problem definition. \cite{Nguyen2012} state that changing the constraints might lead to the discovery of a better global optima. \cite{Maillera} define a dynamic distributed constraint satisfaction problem (DCSP) as a sequence of DCSPs \(\{P_{0}, P_{1}, ..., P_{n}\}\) where every DCSP is a static problem definition. \(P_{i}\) is therefore a result of the previous DSCP in the sequence and also a result of the added and removed constraints: \(P_{i} = P_{i-1} + c_{i^{a}} - c_{i^{r}}\). This definition should also hold for DynDCOPs. Utility functions could also be dynamically changed. Modifying this property could especially have an impact on real-world problems like meeting scheduling, where it could move the global optima from one disconnected solution space to another \cite{Nguyen2012}. Furthermore, variables could be added or removed in a dynamic setting and the problem domain \(D\) also could be changed during the course of the problem solving process.

\section{Meeting Scheduling Problem}

%--------------------- Introduction with examples
Scheduling is the problem of allocating tasks to a given set of ressources in an optimal order. The meeting scheduling problem is an exemplary type of this family of problems and is supposedly well-know to all of us. Participants of a meeting have private schedules with preferences when a meeting should be held according to their calendar. The challenge is to identify a time for a meeting that maximizes the preferences of all participants while being valid in then sense that every person is able to attend \cite{Farinelli}. \cite{Scheduling} have formally defined a meeting scheduling problem as:

\begin{itemize}
\item \(P = {p1, p, ... pn}\) is the set of people where every person has a calendar that holds r slots, \(S = {s1,s2,s3}\)
\item \(M = {m1, m2, ..., m3}\) is a set of k meetings
\item \(At = {at1, at2, ..., atk}\) defines all attendee's of a meeting
\end{itemize}

The c parameter has been neglected as it is not relevant to this thesis. From the definition of a valid solution, one can derive two important criteria to the problem solving process:
\theoremstyle{definition}
\newtheorem{hardconstraint1}{Validity Criterium}
\begin{hardconstraint1}
All participants need to agree on the same time for the meeting.
\end{hardconstraint1}
\begin{hardconstraint1}
Meetings need to be scheduled in a way that there are no overlaps of meeting times in the schedules of the participants.
\end{hardconstraint1}

There is further an inherent privacy aspect to the problem. Meeting participants are often not willing to share their schedules with others except to find a time for the specific meeting. It will later be shown that some of the algorithms can guarantee this privacy to a certain degree \cite{Farinelli} \cite{Scheduling}. The meeting scheduling problem will be mapped as a distributed constraint optimization problem in the design chapter.

    %--------------------- Explanation of the components with formal definition
\section{Algorithm Design Approaches}

\begin{figure}[h]
\includegraphics[width=400px]{graphics/overview_algos}
\caption{Categorization of DCO algorithms \cite{Chapman2011}}
\label{fig:categorization}
\end{figure}

\cite{Chapman2011} categorize distributed constraint optimization algorithms into local-iterative and distributed complete algorithms. They further divide local-iterative into message-passing algorithms and approximate best-response algorithms (Figure \ref{fig:categorization}). The following subsections are going to explain the differences between the three categories and introduce the specific algorithms, which have been chosen from these three different approaches for benchmarking. Advantages, as well as disadvantages will be described and which behaviour one can expect of these algorithms under certain parameter configurations.

%Other Approaches: Bee Hive optimization, Genetic Algorithms, .. DynDCOAA, SBDO, Bee Colony algorithm, Ant colony algorithm, adopt, dsa-a, dsa-b stochastic ...\cite{Likhachev}

\subsection{Distributed Complete}

%-----------------  Basic Concept and Advantages Disadvantes ----------------------
Distributed complete algorithms always discover a configuration of value assignments for a set of variables that maximizes the global utility function. This completeness guarantee increases the complexity of computation and leads to exponentially growing message numbers or calculations when increasing the amount of variables in a problem. Messages between agents often contain complex structures and constraint problems usually need to be transformed to an extensive graph structure \cite{Chapman2011}. These types of algorithms therefore are not expected to scale well and quickly find qualitative solutions, but they fit well if one wants to find the maximal utility of a problem.
    %----------------- Chosen algorithm ----------------------
\begin{figure}[H]
\centering
\includegraphics[width=250px]{graphics/pseudotree}
\caption{Pseudotree in DPOP \cite{Petcu2003}}
\label{fig:pseudotree}
\end{figure}

For this thesis, it was decided to implement the Dynamic Programming OPptimization algorithm (DPOP) proposed by \cite{Petcu2003} as a comparison to the local-iterative approaches. In this algorithm, constraint optimization problems need to be converted to a pseudotree (Fig. \ref{fig:pseudotree}), which is a modification of a DFS Tree. The original DCOP graph is transformed in a way that previous neighbours are placed in the same branches of a binary tree. They are connected trough ordinary tree edges and additionally, so-called back-edges between unconnected previous neighbours are established. The leaf nodes propose UTIL messages containing their utility values for each value assignment upwards the tree and the root node sends a VALUE message downwards, containing the best value to choose as a variable state. Nodes in the middle of the tree propagate UTIL and VALUE messages. The message structure is fairly complex as it involves all the utilities of the pseudoparents connected by the back-edges and their context in the graph, which increases the message size exponentially. The number of messages on the other hand is linear \cite{Petcu2003}.

%----------- Dynamics -----------------------
\subsection{Local-Iterative - Best Response}

%-----------------  Basic Concept and Advantages Disadvantes ----------------------
In a local-iterative best-response algorithm, agents only communicate their current state, e.g. their value assignment and react to these value messages in the best possible way from their perspective. The agents are only connected to their neighbours with whom they share constraints and there exists no complex graph structure controlling the message flow \cite{Chapman2011}. Through this local property, the types of algorithms should be inherently scalable as the messages and computations do not increase exponentially. Further, this approach is optimal from a privacy perspective as the neighbours only share their current preference and no other details of their schedule \cite{Chapman2010}. %\cite{Maheswaran} % Achtung muss das richtig paper finden
\newline\newline
%----------------- Chosen algorithm: Structure, Messages ---------------------
For this thesis, it was decided to implement the Maximum-Gain Messaging algorithm (MGM). In this algorithm, agents  calculate the maximal gain in utility they can achieve when assigning to another value and send this value as a message. If they have the highest gain compared to all received gain messages from their neighbours, the local value is changed. Otherwise the local value stays the same. This algorithm fullfills the anytime property, i.e it can provide a solution at every timepoint during calculation and also reaches good solutions quickly \cite{Chapman2010}. As the decision of an agent depends on a complete set of message of all of its neighbours, this algorithm will supposedly not perform well in asynchronous running mode. This type of algorithm does further not always converge and the deliverance of an optimal solution to a problem is not guaranteed.

\subsection{Local-Iterative - Message Passing}
%-----------------  Basic Concept and Advantages Disadvantes ----------------------
The difference of message-passing to best-response algorithms lays in the fact that the agents send and receive messages containing a specific data structure, which contains the utilities respectively costs that various assignments hold for a local variable. Received messages are used to calculate the next message, which is sent to the connected neighbours. These types of algorithms are - like best-response algorithms - able to provide an acceptable solution in a short period of time, but also share the charasteric to sometimes not converge or not providing an optimal solution \cite{Chapman2011}.
%-----------------  Choosen algorithm ----------------------
\begin{figure}[H]
\centering
\includegraphics[width=170px]{graphics/factorgraph}
\caption{Conversion of a general DCOP to a factor graph \cite{Zivan2012}}
\label{fig:factorgraph}
\end{figure}

For this thesis, it was decided to implement the MaxSum algorithm introduced by \cite{Farinelli2008}. The algorithm has currently gotten a lot of attention from researchers. For this work, the algorithm is especially interesting because of its proposed abilities in dynamic environments.  \cite{Farinelli2008} wrote in their paper:
\begin{quote}
[...] we note that if messages are continuously propagated,
and the states of the agents are continuously updated, then the algorithm may be applied to dynamic problems where the interactions between agents, or the utilities resulting from these interactions, may change at any time.
\end{quote}

In MaxSum, the original DCOP graph is transformed to a factor graph, which is a form of a bipartite graph and of cyclic nature (Fig. \ref{fig:factorgraph}). After the transformation, an agent is made up of a variable and a function node, whereas variables are connected to all corresponding function nodes of  their previous neighbours. The function nodes are vice versa connected to all previous neighbours of its variable node.
% ------------- Messages sent --------------------
The messages sent from variable nodes differ from the function nodes. A message from variable to function contains for every value \(d \in D_{x}\) the sum of utilities regarding this value, which the node has received from all connected function nodes. It is important to note that this sum does not include the values provided by the message target. The values are normalized at this point to avoid an infinite increase of the sum of the utilities. A message from a function node to a variable node holds for every value  \(d \in D_{x}\)  the summation of all costs received from all connected variable nodes except the message receiver and the original cost respectively utility of the constraint represented by the function node \cite{Zivan2012}.


\documentclass[11pt,letterpaper]{article}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{graphicx}

\usepackage[charter]{mathdesign}
\usepackage{fullpage}
\pagestyle{empty}

\input{../fncextra}

\begin{document}

\begin{center}
  \bf
  Project: To be continued
\end{center}

Recall the case of a circular elastic membrane pinned at $z=1$ and subjected to an electrostatic force applied on the plane $z=0$. If we exploit the circular symmetry, we can reduce the problem to a BVP in the disk radius $r$:
\begin{equation}
  \label{eq:mems}
  u''(r) + \frac{1}{r} u'(r) = \frac{\lambda}{u^2},  \qquad u'(0)=0, \quad u(1)=1.
\end{equation}
This has been used as a model for an actuator device in a microelectromechanical system (MEMS). Because we would have some headaches with division by zero in this ODE, we multiply through by $r$ to get
\begin{equation}
  \label{eq:rmems}
  r u''(r) + u'(r) = \frac{r\lambda}{u^2}.
\end{equation}

The behavior of this system as a function of the field strength $\lambda$ is quite interesting. Clearly $u\equiv 1$, $\lambda=0$ is a trivial solution. As $\lambda$ increases from zero, the membrane begins to deflect downward toward zero, with the greatest amount of deflection (i.e., smallest value of $u$) at the center point $r=0$. The amount of deflection continues to decrease until, at some critical value $\lambda^*$, no solution exists. However, for some values $\lambda < \lambda^*$ there are at least two valid solutions. It's not at all clear how to figure out good initial guesses that will drive a Newton-type iteration to find the others.

When the BVP is discretized, we use $n+1$ unknown values of $u$ and $n+1$ equations to specify them. But if we also think of $\lambda$ as a parameter, we have $n+1$ equations in $n+2$ variables. This defines a curve or path in $(n+2)$-dimensional space---just like $x^2+y^2=1$ defines a curve in the plane. The situation can be visualized by this graph:
\begin{center}
  \includegraphics{strip_path}
\end{center}
(This picture is \emph{not} quantitatively accurate for your results!)

What we really want to do is follow along the path, a technique known as \emph{continuation}. There are many ways this can be done, but this particular problem gives us a fairly easy option. The key is to observe in the figure above that $\lambda$ is a poor choice for parameterizing the path, because it fails the ``vertical line test.'' We are fortunate that in this problem, the value $u(0)$ is monotonic along the path, so it will serve as the parameter. To be specific, we solve a discretized form of~\eqref{eq:rmems} for the $n+2$ unknowns $\lambda,u_0,u_1,\ldots,u_n$. As has been our practice, the ODE is discretized at the $n-1$ interior nodes only. To this are added the \emph{three} supplemental conditions
\begin{equation}
  \label{eq:threebc}
  u'(0)=0, \qquad u(1)=1, \qquad u(0)=1-s,
\end{equation}
where $0\le s < 1$ is a value under your control. You start at $s=0$ with the trivial solution mentioned above, then gradually increase $s$ to get a sequence $s_1,s_2,\ldots$, corresponding to solutions $(\lambda_1,\bfu_1),(\lambda_2,\bfu_2)\ldots$, which are points along the path. When solving the nonlinear equations for $s=s_{k+1}$, the starting guess should be $(\lambda_k,\bfu_k)$, which is hopefully near the next point.

\subsection*{Project assignment}
\label{sec:project-assignment}

The heart of the assignment is a function
\begin{verbatim}
[u,lambda] = mems_solve(u0,lam0,s)
\end{verbatim}
that solves the ODE with the three side conditions in~\eqref{eq:threebc}. Here \texttt{lam0} is an initial guess for $\lambda$, and the $(n+1)$-vector \texttt{u0} is an initial guess for $\bfu$. You should model this function after \texttt{bvp}, but your function does not need to solve a general problem, just the specific one at hand. Like \texttt{bvp}, your function will call \texttt{levenberg} on a subfunction named \texttt{residual}, but your version of \texttt{residual} evaluates $n+2$ quantities for values of the $n+2$ variables in a given vector $\bfz=
\begin{bmatrix}
  \lambda \\ \bfu
\end{bmatrix}$.
Once \verb!mems_solve! is working, you can call it within a loop to implement the steady increase of $s$.

Your objectives are:
\begin{enumerate}
\item Find $\lambda^*$ to at least two significant digits. This means taking small enough steps along the curve as well as giving evidence that the answer has converged as a function of the discretization size $n$.
\item Find a second turning point, $\lambda^\dagger<\lambda^*$, to two digits. Past this turning point, $\lambda$ begins to increase again as a function of $s$.
\item Make a plot like the one above for $0\le s \le 0.99$.
\item On one graph, plot $u(r)$ against $r$ for $s=0.1,0.3,0.5,0.7,0.9,0.95$.
\item How accurately can you find $\lambda^*$? Think about how to get the most out the discretization accuracy and how to use something more sophisticated than just taking super-small steps in $s$. Style points matter here!
\end{enumerate}

Submit a PDF report describing your results and providing any necessary supporting information. Keep it concise and to the point---for instance, don't describe multiple attempts at the last item above, just your best one. Don't forget to label your plots! Also include your \verb!mems_solve.m! and the code to reproduce the results for items~1--3 above.

\end{document}


\section{Adding to Front}

\frame{\tableofcontents[currentsection]}

\begin{frame}
    \frametitle{Problem Statement}
    \begin{center}\ttfamily
        prepend([1, 2, 3, 4, 5], 0) \\[4mm]
        $\downarrow$ \\[4mm]
        [0, 1, 2, 3, 4, 5]
    \end{center}
\end{frame}

\begin{frame}
    \frametitle{Add to Front of Array}
    \begin{center}
        \begin{tikzpicture}
            \draw[thick] (0,0) grid ++(6,1);
            \draw[thick] (0,-2) grid ++(7,1);

            \foreach[evaluate={int(\i+1)} as \j] \i in {0,...,5} {
                \node at (\i + 0.5,0.5) {\j};
                \node at (\i + 1.5,-1.5) {\j};
                \draw[-latex] (\i + 0.5,0.25) -- (\i + 1.5,-1.25) node[midway,sloped,font=\tiny,above] {copy};
            }

            \node at (0.5,-1.5) {0};
        \end{tikzpicture}
    \end{center}
    \vskip4mm
    \structure{Algorithm}
    \begin{itemize}
        \item Create new array with larger size
        \item Copy all elements
        \item $O(n)$
    \end{itemize}
\end{frame}

\begin{frame}
    \frametitle{Add to Front of Linked List}
    \begin{center}
        \begin{tikzpicture}[link/.style={thick,-latex}]
            \foreach[evaluate={(\i - 1) * 1.5} as \x] \i in {1,...,5} {
                \coordinate (p\i) at (\x,0);
                \llnode[position={p\i},size=0.5cm,value=\i]
            }

            \foreach[evaluate={(\i - 1) * 1.5} as \x] \i in {1,...,4} {
                \draw[-latex] ($ (p\i) + (0.75,0.25) $) -- ++(0.75,0);
            }

            \coordinate (p0) at ($ (p1) + (0,-1.5) $);
            \llnode[position=p0,size=0.5cm,value=0]
            \draw[-latex] ($ (p0) + (0.75,0.25) $) -- ($ (p1) + (0.5,0) $);
        \end{tikzpicture}
    \end{center}
    \vskip4mm
    \structure{Algorithm}
    \begin{itemize}
        \item Create new node
        \item Have it point to the (originally) first node
        \item $O(1)$
    \end{itemize}
\end{frame}


\chapter{Introduction to linear regression}
\label{linRegrForTwoVar}

\index{linear regression|textbf}

Linear regression is a very powerful statistical technique. Many people have some familiarity with regression just from reading the news, where graphs with straight lines are overlaid on scatterplots. Linear models can be used for prediction or to evaluate whether there is a linear relationship between two numerical variables.

Figure~\ref{perfLinearModel} shows two variables whose relationship can be modeled perfectly with a straight line. The equation for the line is
\begin{eqnarray*}
y = 5 + 57.49x
\end{eqnarray*}
Imagine what a perfect linear relationship would mean: you would know the exact value of $y$ just by knowing the value of $x$. This is unrealistic in almost any natural process. For example, if we took family income $x$, this value would provide some useful information about how much financial support $y$ a college may offer a prospective student. However, there would still be variability in financial support, even when comparing students whose families have similar financial backgrounds.

\begin{figure}
   \centering
   \includegraphics[width=0.6\textwidth]{05/figures/perfLinearModel/perfLinearModel}
   \caption{Requests from twelve separate buyers were simultaneously placed with a trading company to purchase Target Corporation stock (ticker \texttt{TGT}, April 26th, 2012), and the total cost of the shares were reported. Because the cost is computed using a linear formula, the linear fit is perfect.}
   \label{perfLinearModel}
\end{figure}

Linear regression assumes that the relationship between two variables, $x$ and $y$, can be modeled by a straight line:
\begin{eqnarray}
y = \beta_0 + \beta_1x
\label{genLinModelWNoErrorTerm}
\end{eqnarray}
\marginpar[\raggedright\vspace{-10mm}

$\beta_0, \beta_1$\vspace{0.7mm}\\\footnotesize Linear model\\ parameters]{\raggedright\vspace{-10mm}

$\beta_0, \beta_1$\vspace{0.7mm}\\\footnotesize Linear model\\ parameters}where $\beta_0$ and $\beta_1$ represent two model parameters\index{parameter} ($\beta$ is the Greek letter \emph{beta}\index{Greek!beta@beta ($\beta$)}). These parameters are estimated using data, and we write their point estimates as $b_0$ and $b_1$. When we use $x$ to predict $y$, we usually call $x$ the explanatory or \textbf{predictor} variable, and we call $y$ the response.

It is rare for all of the data to fall on a straight line, as seen in the three scatterplots in Figure~\ref{imperfLinearModel}. In each case, the data fall around a straight line, even if none of the observations fall exactly on the line. The first plot shows a relatively strong downward linear trend, where the remaining variability in the data around the line is minor relative to the strength of the relationship between $x$ and $y$. The second plot shows an upward trend that, while evident, is not as strong as the first. The last plot shows a very weak downward trend in the data, so slight we can hardly notice it. In each of these examples, we will have some uncertainty regarding our estimates of the model parameters, $\beta_0$ and $\beta_1$. For instance, we might wonder, should we move the line up or down a little, or should we tilt it more or less? As we move forward in this chapter, we will learn different criteria for line-fitting, and we will also learn about the uncertainty associated with estimates of model parameters.

\begin{figure}
   \centering
   \includegraphics[width=\textwidth]{05/figures/imperfLinearModel/imperfLinearModel}
   \caption{Three data sets where a linear model may be useful even though the data do not all fall exactly on the line.}
   \label{imperfLinearModel}
\end{figure}

We will also see examples in this chapter where fitting a straight line to the data, even if there is a clear relationship between the variables, is not helpful. One such case is shown in Figure~\ref{notGoodAtAllForALinearModel} where there is a very strong relationship between the variables even though the trend is not linear. We will discuss nonlinear trends in this chapter\MultipleRegressionChapter{ and the next}{}, but the details of fitting nonlinear models are saved for a later course.

\begin{figure}
   \centering
   \includegraphics[width=0.87\textwidth]{05/figures/notGoodAtAllForALinearModel/notGoodAtAllForALinearModel}
   \caption{A linear model is not useful in this nonlinear case. These data are from an introductory physics experiment.}
   \label{notGoodAtAllForALinearModel}
\end{figure}

\textA{\newpage}

%__________________
\section{Line fitting, residuals, and correlation}
\label{lineFittingResidualsCorrelation}

It is helpful to think deeply about the line fitting process. In this section, we examine criteria for identifying a linear model and introduce a new statistic, \emph{correlation}.

\subsection{Beginning with straight lines}

\index{data!possum|(}

Scatterplots were introduced in Chapter~\ref{introductionToData} as a graphical technique to present two numerical variables simultaneously. Such plots permit the relationship between the variables to be examined with ease. Figure~\ref{scattHeadLTotalL} shows a scatterplot for the head length and total length of 104 brushtail possums from Australia. Each point represents a single possum from the data.

\begin{figure}
   \centering
   \includegraphics[width=0.85\textwidth]{05/figures/scattHeadLTotalL/scattHeadLTotalL}
   \caption{A scatterplot showing head length against total length for 104 brushtail possums. A point representing a possum with head length 94.1mm and total length 89cm is highlighted.}
   \label{scattHeadLTotalL}
\end{figure}

\begin{figure}
   \centering
   \includegraphics[width=0.8\textwidth]{05/figures/possumPic/possumPic}
   \caption{The common brushtail possum of Australia. Photo by wollombi on Flickr}
   \label{possumPic}
\end{figure}

The head and total length variables are associated. Possums with an above average total length also tend to have above average head lengths. While the relationship is not perfectly linear, it could be helpful to partially explain the connection between these variables with a straight line.

\begin{figure}
   \centering
   \includegraphics[width=0.9\textwidth]{05/figures/scattHeadLTotalLTube/scattHeadLTotalLTube}
   \caption{The figure on the left shows head length versus total length, and reveals that many of the points could be captured by a straight band. On the right, we see that a curved band is more appropriate in the scatterplot for \textbf{weight} and \textbf{mpgCity} from the \textbf{cars} data set.}
   \label{scattHeadLTotalLTube}
\end{figure}

Straight lines should only be used when the data appear to have a linear relationship, such as the case shown in the left panel of Figure~\ref{scattHeadLTotalLTube}. The right panel of Figure~\ref{scattHeadLTotalLTube} shows a case where a curved line would be more useful in understanding the relationship between the two variables.

\begin{caution}
{Watch out for curved trends}
{We only consider models based on straight lines in this chapter. If data show a nonlinear trend, like that in the right panel of Figure~\ref{scattHeadLTotalLTube}, more advanced techniques should be used.\vspace{0.7mm}}
\end{caution}

\textA{\pagebreak}

\subsection{Fitting a line by eye}

We want to describe the relationship between the head length and total length variables in the possum data set using a line. In this example, we will use the total length as the predictor variable, $x$, to predict a possum's head length, $y$. We could fit the linear relationship by eye, as in Figure~\ref{scattHeadLTotalLLine}. The equation for this line is
\begin{eqnarray}
\hat{y} = 41 + 0.59x
\label{headLLinModTotalL}
\end{eqnarray}
We can use this line to discuss properties of possums. For instance, the equation predicts a possum with a total length of 80 cm will have a head length of
\begin{align*}
\hat{y} &= 41 + 0.59\times 80 \\
	&= 88.2 % mm
\end{align*}
A ``hat'' on $y$ is used to signify that this is an estimate. This estimate may be viewed as an average: the equation predicts that possums with a total length of 80 cm will have an average head length of 88.2 mm. Absent further information about an 80 cm possum, the prediction for head length that uses the average is a reasonable estimate.

\begin{figure}
   \centering
   \includegraphics[width=0.8\textwidth]{05/figures/scattHeadLTotalLLine/scattHeadLTotalLLine}
   \caption{A reasonable linear model was fit to represent the relationship between head length and total length.}
   \label{scattHeadLTotalLLine}
\end{figure}

\subsection{Residuals}

\index{residual|(}

\termsub{Residuals}{residual} are the leftover variation in the data after accounting for the model fit:
\begin{align*}
\text{Data} = \text{Fit} + \text{Residual}
\end{align*}
Each observation will have a residual. If an observation is above the regression line, then its residual, the vertical distance from the observation to the line, is positive. Observations below the line have negative residuals. One goal in picking the right linear model is for these residuals to be as small as possible.

Three observations are noted specially in Figure~\ref{scattHeadLTotalLLine}. The observation marked by an ``$\times$'' has a small, negative residual of about -1; the observation marked by ``$+$'' has a large residual of about +7; and the observation marked by ``$\triangle$'' has a moderate residual of about -4. The size of a residual is usually discussed in terms of its absolute value. For example, the residual for ``$\triangle$'' is larger than that of ``$\times$'' because $|-4|$ is larger than $|-1|$.

\begin{termBox}{\tBoxTitle{Residual: difference between observed and expected}
The residual of the $i^{th}$ observation $(x_i, y_i)$ is the difference of the observed response ($y_i$) and the response we would predict based on the model fit ($\hat{y}_i$):
\begin{eqnarray*}
e_i = y_i - \hat{y}_i
\end{eqnarray*}
We typically identify $\hat{y}_i$ by plugging $x_i$ into the model.}
\end{termBox}

\begin{example}{The linear fit shown in Figure~\ref{scattHeadLTotalLLine} is given as $\hat{y} = 41 + 0.59x$. Based on this line, formally compute the residual of the observation $(77.0, 85.3)$. This observation is denoted by ``$\times$'' on the plot. Check it against the earlier visual estimate,~-1.}
We first compute the predicted value of point ``$\times$'' based on the model:
\begin{eqnarray*}
\hat{y}_{\times} = 41+0.59x_{\times} = 41+0.59\times 77.0 = 86.4
\end{eqnarray*}
Next we compute the difference of the actual head length and the predicted head length:
\begin{eqnarray*}
e_{\times} = y_{\times} - \hat{y}_{\times} = 85.3 -  86.4 = -1.1
\end{eqnarray*}
This is very close to the visual estimate of -1.
\end{example}

\begin{exercise}
If a model underestimates an observation, will the residual be positive or negative? What about if it overestimates the observation?\footnote{If a model underestimates an observation, then the model estimate is below the actual. The residual, which is the actual observation value minus the model estimate, must then be positive. The opposite is true when the model overestimates the observation: the residual is negative.}
\end{exercise}

\begin{exercise}
Compute the residuals for the observations $(85.0, 98.6)$ (``$+$'' in the figure) and $(95.5, 94.0)$ (``$\triangle$'') using the linear relationship $\hat{y} = 41 + 0.59x$. \footnote{($+$) First compute the predicted value based on the model: $$\hat{y}_{+} = 41+0.59x_{+} = 41+0.59\times 85.0 = 91.15$$ Then the residual is given by $$e_{+} = y_{+} - \hat{y}_{+} = 98.6-91.15=7.45$$This was close to the earlier estimate of 7.

($\triangle$) $\hat{y}_{\triangle} = 41+0.59x_{\triangle} = 97.3$. $e_{\triangle} = y_{\triangle} - \hat{y}_{\triangle} = -3.3$, close to the estimate of -4.}
\end{exercise}

Residuals are helpful in evaluating how well a linear model fits a data set. We often display them in a \textbf{residual plot} such as the one shown in Figure~\ref{scattHeadLTotalLResidualPlot} for the regression line in Figure~\ref{scattHeadLTotalLLine}. The residuals are plotted at their original horizontal locations but with the vertical coordinate as the residual. For instance, the point $(85.0,98.6)_{+}$ had a residual of 7.45, so in the residual plot it is placed at $(85.0, 7.45)$. Creating a residual plot is sort of like tipping the scatterplot over so the regression line is horizontal.
\index{data!possum|)}

\begin{figure}
   \centering
   \includegraphics[width=0.73\textwidth]{05/figures/scattHeadLTotalLResidualPlot/scattHeadLTotalLResidualPlot}
   \caption{Residual plot for the model in Figure~\ref{scattHeadLTotalLLine}.}
   \label{scattHeadLTotalLResidualPlot}
\end{figure}

\begin{example}{One purpose of residual plots is to identify characteristics or patterns still apparent in data after fitting a model. Figure~\ref{sampleLinesAndResPlots} shows three scatterplots with linear models in the first row and residual plots in the second row. Can you identify any patterns remaining in the residuals?}

\begin{figure}
   \centering
   \includegraphics[width=\textwidth]{05/figures/sampleLinesAndResPlots/sampleLinesAndResPlots}
   \caption{Sample data with their best fitting lines (top row) and their corresponding residual plots (bottom row).}
   \label{sampleLinesAndResPlots}
\end{figure}
In the first data set (first column), the residuals show no obvious patterns. The residuals appear to be scattered randomly around the dashed line that represents 0.

The second data set shows a pattern in the residuals. There is some curvature in the scatterplot, which is more obvious in the residual plot. We should not use a straight line to model these data. Instead, a more advanced technique should be used.

The last plot shows very little upwards trend, and the residuals also show no obvious patterns. It is reasonable to try to fit a linear model to the data. However, it is unclear whether there is statistically significant evidence that the slope parameter is different from zero. The point estimate of the slope parameter, labeled $b_1$, is not zero, but we might wonder if this could just be due to chance. We will address this sort of scenario in Section~\ref{inferenceForLinearRegression}.
\index{residual|)}
\end{example}

\subsection{Describing linear relationships with correlation}

\index{correlation|(}

\begin{termBox}{\tBoxTitle{Correlation: strength of a linear relationship}
\termsub{Correlation}{correlation}, which always takes values between -1 and 1, describes the strength of the linear relationship between two variables. We denote the correlation by $R$.}
\end{termBox}\marginpar[\raggedright\vspace{-11.5mm}

$R$\\\footnotesize correlation]{\raggedright\vspace{-11.5mm}

$R$\\\footnotesize correlation}

We can compute the correlation using a formula, just as we did with the sample mean and standard deviation. However, this formula is rather complex,\footnote{Formally, we can compute the correlation for observations $(x_1, y_1)$, $(x_2, y_2)$, ..., $(x_n, y_n)$ using the formula
\begin{eqnarray*}
R = \frac{1}{n-1}\sum_{i=1}^{n} \frac{x_i-\bar{x}}{s_x}\frac{y_i-\bar{y}}{s_y}
\end{eqnarray*}
where $\bar{x}$, $\bar{y}$, $s_x$, and $s_y$ are the sample means and standard deviations for each variable.} so we generally perform the calculations on a computer or calculator. Figure~\ref{posNegCorPlots} shows eight plots and their corresponding correlations. Only when the relationship is perfectly linear is the correlation either -1 or 1. If the relationship is strong and positive, the correlation will be near +1. If it is strong and negative, it will be near -1. If there is no apparent linear relationship between the variables, then the correlation will be near zero.

\begin{figure}
   \centering
   \includegraphics[width=0.9\textwidth]{05/figures/posNegCorPlots/posNegCorPlots}
   \caption{Sample scatterplots and their correlations. The first row shows variables with a positive relationship, represented by the trend up and to the right. The second row shows variables with a negative trend, where a large value in one variable is associated with a low value in the other.}
   \label{posNegCorPlots}
\end{figure}

The correlation is intended to quantify the strength of a linear trend. Nonlinear trends, even when strong, sometimes produce correlations that do not reflect the strength of the relationship; see three such examples in Figure~\ref{corForNonLinearPlots}.

\begin{figure}
   \centering
   \includegraphics[width=0.96\textwidth]{05/figures/posNegCorPlots/corForNonLinearPlots}
   \caption{Sample scatterplots and their correlations. In each case, there is a strong relationship between the variables. However, the correlation is not very strong, and the relationship is not linear.}
   \label{corForNonLinearPlots}
\end{figure}

\begin{exercise}
It appears no straight line would fit any of the datasets represented in Figure~\ref{corForNonLinearPlots}. Instead, try drawing nonlinear curves on each plot. Once you create a curve for each, describe what is important in your fit.\footnote{We'll leave it to you to draw the lines. In general, the lines you draw should be close to most points and reflect overall trends in the data.}
\index{correlation|)}
\end{exercise}

%__________________
\section{Fitting a line by least squares regression}
\label{fittingALineByLSR}

\index{least squares regression|(}

Fitting linear models by eye is open to criticism since it is based on an individual preference. In this section, we use \emph{least squares regression} as a more rigorous approach.

This section considers family income and gift aid data from a random sample of fifty students in the 2011 freshman class of Elmhurst College in Illinois.\footnote{These data were sampled from a table of data for all freshman from the 2011 class at Elmhurst College that accompanied an article titled \emph{What Students Really Pay to Go to College} published online by \emph{The~Chronicle of Higher Education}: \href{http://chronicle.com/article/What-Students-Really-Pay-to-Go/131435}{chronicle.com/article/What-Students-Really-Pay-to-Go/131435}} Gift aid is financial aid that does not need to be paid back, as opposed to a loan. A scatterplot of the data is shown in Figure~\ref{elmhurstScatterW2Lines} along with two linear fits. The lines follow a negative trend in the data; students who have higher family incomes tended to have lower gift aid from the university.

\begin{figure}
\centering
\includegraphics[width=0.75\textwidth]{05/figures/elmhurstPlots/elmhurstScatterW2Lines}
\caption{Gift aid and family income for a random sample of 50 freshman students from Elmhurst College. Two lines are fit to the data, the solid line being the \emph{least squares line}.}
\label{elmhurstScatterW2Lines}
\end{figure}

\begin{exercise}
Is the correlation positive or negative in Figure~\ref{elmhurstScatterW2Lines}?\footnote{Larger family incomes are associated with lower amounts of aid, so the correlation will be negative. Using a computer, the correlation can be computed: -0.499.}
\end{exercise}

\subsection{An objective measure for finding the best line}

We begin by thinking about what we mean by ``best''. Mathematically, we want a line that has small residuals. Perhaps our criterion could minimize the sum of the residual magnitudes:\textA{\vspace{-4mm}}
\begin{eqnarray}
|e_1| + |e_2| + \dots + |e_n|
\label{sumOfAbsoluteValueOfResiduals}
\end{eqnarray}
which we could accomplish with a computer program. The resulting dashed line shown in Figure~\ref{elmhurstScatterW2Lines} demonstrates this fit can be quite reasonable. However, a more common practice is to choose the line that minimizes the sum of the squared residuals:\textA{\vspace{-1mm}}
\begin{eqnarray}
e_{1}^2 + e_{2}^2 + \dots + e_{n}^2
\label{sumOfSquaresForResiduals}
\end{eqnarray}
The line that minimizes this \textbf{least squares criterion} is represented as the solid line in Figure~\ref{elmhurstScatterW2Lines}. This is commonly called the \textbf{least squares line}. The following are three possible reasons to choose Criterion~\eqref{sumOfSquaresForResiduals} over Criterion~\eqref{sumOfAbsoluteValueOfResiduals}:
\begin{enumerate}
\item It is the most commonly used method.
\item Computing the line based on Criterion~\eqref{sumOfSquaresForResiduals} is much easier by hand and in most statistical software.
\item In many applications, a residual twice as large as another residual is more than twice as bad. For example, being off by 4 is usually more than twice as bad as being off by 2. Squaring the residuals accounts for this discrepancy.
\end{enumerate}
The first two reasons are largely for tradition and convenience; the last reason explains why Criterion~\eqref{sumOfSquaresForResiduals} is typically most helpful.\footnote{There are applications where Criterion~\eqref{sumOfAbsoluteValueOfResiduals} may be more useful, and there are plenty of other criteria we might consider. However, this book only applies the least squares criterion.}

\subsection{Finding the least squares line}
\label{findingTheLeastSquaresLineSection}

For the Elmhurst data, we could write the equation of the least squares regression line as
\begin{eqnarray*}
\widehat{aid} = \beta_0 + \beta_{1}\times family\_\hspace{0.3mm}income
\end{eqnarray*}
Here the equation is set up to predict gift aid based on a student's family income, which would be useful to students considering Elmhurst. These two values, $\beta_0$ and $\beta_1$, are the \emph{parameters} of the regression line.

As in Chapters~4-6, the parameters are estimated using observed data. In practice, this estimation is done using a computer in the same way that other estimates, like a sample mean, can be estimated using a computer or calculator. However, we can also find the parameter estimates by applying two properties of the least squares line:
\begin{itemize}
\setlength{\itemsep}{0mm}
\item The slope of the least squares line can be estimated by
\begin{eqnarray}
b_1 = \frac{s_y}{s_x} R
\label{slopeOfLSRLine}
\end{eqnarray}
where $R$ is the correlation between the two variables, and $s_x$ and $s_y$ are the sample standard deviations of the explanatory variable %(variable on the horizontal axis)
and response% (variable on the vertical axis)
, respectively.
\item If $\bar{x}$ is the mean of the horizontal variable (from the data) and $\bar{y}$ is the mean of the vertical variable, then the point $(\bar{x}, \bar{y})$ is on the least squares line.
\end{itemize}
\marginpar[\raggedright\vspace{0.5mm}

$b_0, b_1$\vspace{0.5mm}\\\footnotesize Sample\\estimates\\ of $\beta_0$, $\beta_1$]{\raggedright\vspace{0.5mm}

$b_0, b_1$\vspace{0.5mm}\\\footnotesize Sample\\estimates\\ of $\beta_0$, $\beta_1$}We use $b_0$ and $b_1$ to represent the point estimates of the parameters $\beta_0$ and $\beta_1$.

\begin{exercise}
Table~\ref{summaryStatsOfSATGPAData} shows the sample means for the family income and gift aid as \$101,800 and \$19,940, respectively. Plot the point $(101.8, 19.94)$ on Figure~\vref{elmhurstScatterW2Lines} to verify it falls on the least squares line (the solid line).\footnote{If you need help finding this location, draw a straight line up from the x-value of 100 (or thereabout). Then draw a horizontal line at 20 (or thereabout). These lines should intersect on the least squares line.}
\end{exercise}

\begin{table}[ht]
\centering
\begin{tabular}{l rr}
\hline
\vspace{-4mm} & & \\
\vspace{0.4mm}	&	\ \ family income, in \$1000s (``$x$'')	& \ \ gift aid, in \$1000s (``$y$'') \\
\hline
  \vspace{-3.9mm} & & \\
mean	& $\bar{x} = 101.8$		& $\bar{y} = 19.94$ \\
sd		& $s_x = 63.2$		& $s_y = 5.46$\vspace{0.4mm} \\
\hline
\vspace{-4mm}\ &\\
	& \multicolumn{2}{r}{$R=-0.499$} \\
\hline
\end{tabular}
\caption{Summary statistics for family income and gift aid.}
\label{summaryStatsOfSATGPAData}
\end{table}

\begin{exercise} \label{findingTheSlopeOfTheLSRLineForIncomeAndAid}
Using the summary statistics in Table~\ref{summaryStatsOfSATGPAData}, compute the slope for the regression line of gift aid against family income.\footnote{Apply Equation~\eqref{slopeOfLSRLine} with the summary statistics from Table~\ref{summaryStatsOfSATGPAData} to compute the slope:
\begin{eqnarray*}
b_1 = \frac{s_y}{s_x} R = \frac{5.46}{63.2}(-0.499) = -0.0431
\end{eqnarray*}\vspace{-3mm}}
\end{exercise}

You might recall the \textbf{point-slope} form of a line from math class (another common form is \emph{slope-intercept}). Given the slope of a line and a point on the line, $(x_0, y_0)$, the equation for the line can be written as
\begin{eqnarray}
y - y_0 = slope\times (x - x_0)
\label{pointSlopeFormForALine}
\end{eqnarray}
A common exercise to become more familiar with foundations of least squares regression is to use basic summary statistics and point-slope form to produce the least squares line.

\begin{tipBox}{\tipBoxTitle{Identifying the least squares line from summary statistics}
To identify the least squares line from summary statistics:\vspace{-1mm}
\begin{itemize}
\setlength{\itemsep}{0mm}
\item Estimate the slope parameter, $\beta_1$, by calculating $b_1$ using Equation~\eqref{slopeOfLSRLine}.
\item Noting that the point $(\bar{x}, \bar{y})$ is on the least squares line, use $x_0=\bar{x}$ and $y_0=\bar{y}$ along with the slope $b_1$ in the point-slope equation:
$$y - \bar{y} = b_1 (x - \bar{x}) $$
\item Simplify the equation.\vspace{0.5mm}
\end{itemize}}
\end{tipBox}

\textA{\pagebreak}

\begin{example}{Using the point $(101.8, 19.94)$ from the sample means and the slope estimate $b_1 = -0.0431$ from Guided Practice~\ref{findingTheSlopeOfTheLSRLineForIncomeAndAid}, find the least-squares line for predicting aid based on family income.} \label{exampleToFindLSRLineOfElmhurstData}
Apply the point-slope equation using $(101.8, 19.94)$ and the slope $b_1 = -0.0431$:
\begin{align*}
y - y_0     &= b_1 (x - x_0) \\
y - 19.94  &= -0.0431(x - 101.8)
\end{align*}
Expanding the right side and then adding 19.94 to each side, the equation simplifies:
$$ \widehat{aid} = 24.3 - 0.0431 \times family\_\hspace{0.3mm}income $$
Here we have replaced $y$ with $\widehat{aid}$ and $x$ with $family\_\hspace{0.3mm}income$ to put the equation in context.
\end{example}

We mentioned earlier that a computer is usually used to compute the least squares line. A summary table based on computer output is shown in Table~\ref{rOutputForIncomeAidLSRLine} for the Elmhurst data. The first column of numbers provides estimates for ${b}_0$ and ${b}_1$, respectively. Compare these to the result from Example~\ref{exampleToFindLSRLineOfElmhurstData}.

\begin{table}[ht]
\centering
\begin{tabular}{l rrrr}
  \hline
  \vspace{-3.7mm} & & & & \\
 & Estimate & Std. Error & t value & Pr($>$$|$t$|$) \\
  \hline
  \vspace{-3.6mm} & & & & \\
(Intercept) & 24.3193 & 1.2915 & 18.83 & 0.0000 \\
family\_\hspace{0.3mm}income & -0.0431 & 0.0108 & -3.98 & 0.0002 \\
  \hline
\end{tabular}
\caption{Summary of least squares fit for the Elmhurst data. Compare the parameter estimates in the first column to the results of Example~\ref{exampleToFindLSRLineOfElmhurstData}.}
\label{rOutputForIncomeAidLSRLine}
\end{table}

\begin{example}{Examine the second, third, and fourth columns in Table~\ref{rOutputForIncomeAidLSRLine}. Can you guess what they represent?}
We'll describe the meaning of the columns using the second row, which corresponds to~$\beta_1$. The first column provides the point estimate for $\beta_1$, as we calculated in an earlier example: -0.0431. The second column is a standard error for this point estimate: 0.0108. The third column is a $t$ test statistic for the null hypothesis that $\beta_1 = 0$: $T=-3.98$. The last column is the p-value for the $t$ test statistic for the null hypothesis $\beta_1=0$ and a two-sided alternative hypothesis: 0.0002. We will get into more of these details in Section~\ref{inferenceForLinearRegression}.
\end{example}

\begin{example}{Suppose a high school senior is considering Elmhurst College. Can she simply use the linear equation that we have estimated to calculate her financial aid from the university?}
She may use it as an estimate, though some qualifiers on this approach are important. First, the data all come from one freshman class, and the way aid is determined by the university may change from year to year. Second, the equation will provide an imperfect estimate. While the linear equation is good at capturing the trend in the data, no individual student's aid will be perfectly predicted.
\index{least squares regression|)}
\end{example}

\subsection{Interpreting regression line parameter estimates}

\index{least squares regression!interpreting parameters|(}

Interpreting parameters in a regression model is often one of the most important steps in the analysis.

\begin{example}{The slope and intercept estimates for the Elmhurst data are -0.0431 and 24.3. What do these numbers really mean?}
Interpreting the slope parameter is helpful in almost any application. For each additional \$1,000 of family income, we would expect a student to receive a net difference of $\$\text{1,000}\times (-0.0431) = -\$43.10$ in aid on average, i.e. \$43.10 \emph{less}. Note that a higher family income corresponds to less aid because the coefficient of family income is negative in the model. We must be cautious in this interpretation: while there is a real association, we cannot interpret a causal connection between the variables because these data are observational. That is, increasing a student's family income may not cause the student's aid to drop. (It would be reasonable to contact the college and ask if the relationship is causal, i.e. if Elmhurst College's aid decisions are partially based on students' family income.)

The estimated intercept $b_0=24.3$ (in \$1000s) describes the average aid if a student's family had no income. The meaning of the intercept is relevant to this application since the family income for some students at Elmhurst is \$0. In other applications, the intercept may have little or no practical value if there are no observations where $x$ is near zero.
\end{example}

\begin{termBox}{\tBoxTitle{Interpreting parameters estimated by least squares}
The slope describes the estimated difference in the $y$ variable if the explanatory variable $x$ for a case happened to be one unit larger. The intercept describes the average outcome of $y$ if $x=0$ \emph{and} the linear model is valid all the way to $x=0$, which in many applications is not the case.}
\index{least squares regression!interpreting parameters|)}
\end{termBox}

\subsection{Extrapolation is treacherous}

\index{least squares regression!extrapolation|(}

{\em\small When those blizzards hit the East Coast this winter, it proved to my satisfaction that global warming was a fraud. That snow was freezing cold. But in an alarming trend, temperatures this spring have risen. Consider this: On February $6^{th}$ it was 10 degrees. Today it hit almost 80. At this rate, by August it will be 220 degrees. So clearly folks the climate debate rages on.\vspace{0.5mm}}

\noindent\hspace{\textwidth}\hspace{-40mm}Stephen Colbert

\noindent\hspace{\textwidth}\hspace{-40mm}April 6th, 2010 \footnote{\urlwofont{http://www.colbertnation.com/the-colbert-report-videos/269929/}} \\

Linear models can be used to approximate the relationship between two variables. However, these models have real limitations. Linear regression is simply a modeling framework. The truth is almost always much more complex than our simple line. For example, we do not know how the data outside of our limited window will behave.

\textA{\pagebreak}

\begin{example}{Use the model $\widehat{aid} = 24.3 - 0.0431\times family\_\hspace{0.3mm}income$ to estimate the aid of another freshman student whose family had income of \$1 million.}
Recall that the units of family income are in \$1000s, so we want to calculate the aid for $family\_\hspace{0.3mm}income = 1000$:
\begin{align*}
24.3 - 0.0431\times family\_\hspace{0.3mm}income  = 24.3 - 0.0431\times 1000 = -18.8
\end{align*}
The model predicts this student will have -\$18,800 in aid (!). Elmhurst College cannot (or at least does not) require any students to pay extra on top of tuition to attend.
\end{example}

Applying a model estimate to values outside of the realm of the original data is called \textbf{extrapolation}. Generally, a linear model is only an approximation of the real relationship between two variables. If we extrapolate, we are making an unreliable bet that the approximate linear relationship will be valid in places where it has not been explored.

\index{least squares regression!extrapolation|)}

\subsection{Using $R^2$ to describe the strength of a fit}

\index{least squares regression!R-squared ($R^2$)|(}

We evaluated the strength of the linear relationship between two variables earlier using the correlation, $R$. However, it is more common to explain the strength of a linear fit using $R^2$, called \termsub{R-squared}{least squares regression!R-squared ($R^2$)}. If provided with a linear model, we might like to describe how closely the data cluster around the linear fit.

\begin{figure}
\centering
\includegraphics[width=0.75\textwidth]{05/figures/elmhurstPlots/elmhurstScatterWLSROnly}
\caption{Gift aid and family income for a random sample of 50 freshman students from Elmhurst College, shown with the least squares regression line.}
\label{elmhurstScatterWLSROnly}
\end{figure}

The $R^2$ of a linear model describes the amount of variation in the response that is explained by the least squares line. For example, consider the Elmhurst data, shown in Figure~\ref{elmhurstScatterWLSROnly}. The variance of the response variable, aid received, is $s_{aid}^2=29.8$. However, if we apply our least squares line, then this model reduces our uncertainty in predicting aid using a student's family income. The variability in the residuals describes how much variation remains after using the model: $s_{_{RES}}^2 = 22.4$. In short, there was a reduction of
$$\frac{s_{aid}^2 - s_{_{RES}}^2}{s_{_{aid}}^2}
	= \frac{29.8 - 22.4}{29.8} = \frac{7.5}{29.8}
	= 0.25$$
or about  25\% in the data's variation by using information about family income for predicting aid using a linear model. This corresponds exactly to the R-squared value:
\begin{align*}
R &= -0.499 &R^2 &= 0.25
\end{align*}

\begin{exercise}
If a linear model has a very strong negative relationship with a correlation of -0.97, how much of the variation in the response is explained by the explanatory variable?\footnote{About $R^2 = (-0.97)^2 = 0.94$ or 94\% of the variation is explained by the linear model.}
\index{least squares regression!R-squared ($R^2$)|)}
\end{exercise}

\subsection{Categorical predictors with two levels}
\label{categoricalPredictorsWithTwoLevels}

Categorical variables are also useful in predicting outcomes. Here we consider a categorical predictor with two levels (recall that a \emph{level} is the same as a \emph{category}). We'll consider Ebay auctions for a video game, \emph{Mario Kart} for the Nintendo Wii, where both the total price of the auction and the condition of the game were recorded.\footnote{These data were collected in Fall 2009 and may be found at \href{http://www.openintro.org}{openintro.org}.} Here we want to predict total price based on game condition, which takes values \resp{used} and \resp{new}. A plot of the auction data is shown in Figure~\ref{marioKartNewUsed}.

\begin{figure}
\centering
\includegraphics[width=0.65\textwidth]{05/figures/marioKartNewUsed/marioKartNewUsed}
\caption{Total auction prices for the video game \emph{Mario Kart}, divided into used ($x=0$) and new ($x=1$) condition games. The least squares regression line is also shown.}
\label{marioKartNewUsed}
\end{figure}

To incorporate the game condition variable into a regression equation, we must convert the categories into a numerical form. We will do so using an \textbf{indicator variable} called \textbf{cond\_\hspace{0.3mm}new}, which takes value 1 when the game is new and 0 when the game is used. Using this indicator variable, the linear model may be written as
\begin{align*}
\widehat{price} = \beta_0 + \beta_1 \times \text{\textbf{cond\_\hspace{0.3mm}new}}
\end{align*}
The fitted model is summarized in Table~\ref{marioKartNewUsedRegrSummary}, and the model with its parameter estimates is given as
\begin{align*}
\widehat{price} = 42.87 + 10.90 \times \text{\textbf{cond\_\hspace{0.3mm}new}}
\end{align*}

\begin{table}
\centering
\begin{tabular}{rrrrr}
  \hline
  \vspace{-3.7mm} & & & & \\
 & Estimate & Std. Error & t value & Pr($>$$|$t$|$) \\
  \hline
  \vspace{-3.6mm} & & & & \\
(Intercept) & 42.87 & 0.81 & 52.67 & 0.0000 \\
  cond\_\hspace{0.3mm}new & 10.90 & 1.26 & 8.66 & 0.0000 \\
   \hline
\end{tabular}
\caption{Least squares regression summary for the final auction price against the condition of the game.}
\label{marioKartNewUsedRegrSummary}
\end{table}

\begin{example}{Interpret the two parameters estimated in the model for the price of \emph{Mario Kart} in eBay auctions.}
The intercept is the estimated price when \textbf{cond\_\hspace{0.3mm}new} takes value 0, i.e. when the game is in used condition. That is, the average selling price of a used version of the game is \$42.87.

The slope indicates that, on average, new games sell for about \$10.90 more than used games.
\end{example}

\begin{tipBox}{\tipBoxTitle{Interpreting model estimates for categorical predictors.}
The estimated intercept is the value of the response variable for the first category (i.e. the category corresponding to an indicator value of 0). The estimated slope is the average change in the response variable between the two categories.}
\end{tipBox}

\MultipleRegression{We'll elaborate further on this Ebay auction data in Chapter~\ref{multipleRegressionAndANOVA}, where we examine the influence of many predictor variables simultaneously using multiple regression\index{multiple regression}. In multiple regression, we will consider the association of auction price with regard to each variable while controlling for the influence of other variables. This is especially important since some of the predictors are associated. For example, auctions with games in new condition also often came with more accessories.}{}

%__________________
\section{Types of outliers in linear regression}
\label{typesOfOutliersInLinearRegression}

In this section, we identify criteria for determining which outliers are important and influential.

Outliers in regression are observations that fall far from the ``cloud'' of points. These points are especially important because they can have a strong influence on the least squares line.

\textA{\pagebreak}

\begin{example}{There are six plots shown in Figure~\ref{outlierPlots} along with the least squares line and residual plots. For each scatterplot and residual plot pair, identify any obvious outliers and note how they influence the least squares line. Recall that an outlier is any point that doesn't appear to belong with the vast majority of the other points.}\label{outlierPlotsExample}
\begin{itemize}
\item[(1)] There is one outlier far from the other points, though it only appears to slightly influence the line.
\item[(2)] There is one outlier on the right, though it is quite close to the least squares line, which suggests it wasn't very influential.
\item[(3)] There is one point far away from the cloud, and this outlier appears to pull the least squares line up on the right; examine how the line around the primary cloud doesn't appear to fit very well.
\item[(4)] There is a primary cloud and then a small secondary cloud of four outliers. The secondary cloud appears to be influencing the line somewhat strongly, making the least square line fit poorly almost everywhere. There might be an interesting explanation for the dual clouds, which is something that could be investigated.
\item[(5)] There is no obvious trend in the main cloud of points and the outlier on the right appears to largely control the slope of the least squares line.
\item[(6)] There is one outlier far from the cloud, however, it falls quite close to the least squares line and does not appear to be very influential.
\end{itemize}
\end{example}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{05/figures/outlierPlots/outlierPlots}
\caption{Six plots, each with a least squares line and residual plot. All data sets have at least one outlier.}
\label{outlierPlots}
\end{figure}

Examine the residual plots in Figure~\ref{outlierPlots}. You will probably find that there is some trend in the main clouds of (3) and (4). In these cases, the outliers influenced the slope of the least squares lines. In (5), data with no clear trend were assigned a line with a large trend simply due to one outlier (!).

 \begin{termBox}{\tBoxTitle{Leverage}
Points that fall horizontally away from the center of the cloud tend to pull harder on the line, so we call them points with \textbf{high leverage}.}
\end{termBox}

Points that fall horizontally far from the line are points of high leverage; these points can strongly influence the slope of the least squares line. If one of these high leverage points does appear to actually invoke its influence on the slope of the line -- as in cases (3), (4), and (5) of Example~\ref{outlierPlotsExample} -- then we call it an \textbf{influential point}. Usually we can say a point is influential if, had we fitted the line without it, the influential point would have been unusually far from the least squares line.

It is tempting to remove outliers. Don't do this without a very good reason. Models that ignore exceptional (and interesting) cases often perform poorly. For instance, if a financial firm ignored the largest market swings -- the ``outliers'' --  they would soon go bankrupt by making poorly thought-out investments.

\begin{caution}{Don't ignore outliers when fitting a final model}
{If there are outliers in the data, they should not be removed or ignored without a~good reason. Whatever final model is fit to the data would not be very helpful if it ignores the most exceptional cases.}
\end{caution}

\begin{caution}{Outliers for a categorical predictor with two levels}{
Be cautious about using a categorical predictor when one of the levels has very few observations. When this happens, those few observations become influential points.}
\end{caution}

%__________________
\section{Inference for linear regression}
\label{inferenceForLinearRegression}

In this section we discuss uncertainty in the estimates of the slope and y-intercept for a regression line. Just as we identified standard errors for point estimates in previous chapters, we first discuss standard errors for these new estimates. However, in the case of regression, we will identify standard errors using statistical software.

\subsection{Conditions for the least squares line}

When performing inference on a least squares line, we generally require the following:
\begin{description}
\setlength{\itemsep}{0mm}
\item[Linearity.] The data should show a linear trend. If there is a nonlinear trend (e.g. left panel of Figure~\ref{whatCanGoWrongWithLinearModel}), an advanced regression method from another book or later course should be applied.
\item[Nearly normal residuals.] Generally the residuals must be nearly normal.
When this condition is found to be unreasonable, it is usually because of outliers or concerns about influential points, which we  will discuss in greater depth in Section~\ref{typesOfOutliersInLinearRegression}. An example of non-normal residuals is shown in the second panel of Figure~\ref{whatCanGoWrongWithLinearModel}.
\item[Constant variability.] The variability of points around the least squares line remains roughly constant. An example of non-constant variability is shown in the third panel of Figure~\ref{whatCanGoWrongWithLinearModel}.
\item[Independent observations.] Be cautious about applying regression to data collected sequentially in what is called a \textbf{time series}. Such data may have an underlying structure that should be considered in a model and analysis. An example of a time series where independence is violated is shown in the fourth panel of Figure~\ref{whatCanGoWrongWithLinearModel}.
%The observations in the data set must be independent. A common example of data that are not independent are data collected sequentially, called \textbf{time series data}, where consecutive observations tend to be close to one another. Plotting the data in the order it was collected is often helpful for evaluating this condition.
\end{description}
For additional information on checking regression conditions, see Section~\ref{multipleRegressionModelAssumptions}.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{05/figures/whatCanGoWrongWithLinearModel/whatCanGoWrongWithLinearModel}
\caption{Four examples showing when the methods in this chapter are insufficient to apply to the data. In the left panel, a straight line does not fit the data. In the second panel, there are outliers; two points on the left are relatively distant from the rest of the data, and one of these points is very far away from the line. In the third panel, the variability of the data around the line increases with larger values of $x$. In the last panel, a time series data set is shown, where successive observations are highly correlated.}
\label{whatCanGoWrongWithLinearModel}
\end{figure}

\begin{example}{Should we have concerns about applying inference to the Elmhurst data in Figure~\ref{elmhurstScatterLSRLine_conditions}?}
The trend appears to be linear, the data fall around the line with no obvious outliers, the variance is roughly constant. These are also not time series observations. It would be reasonable to analyze the model using inference.
\end{example}

\begin{figure}
\centering
\includegraphics[width=0.75\textwidth]{05/figures/elmhurstPlots/elmhurstScatterLSRLine_conditions}
\caption{Gift aid and family income for a random sample of 50 freshman students from Elmhurst College. Two lines are fit to the data, the solid line being the \emph{least squares line}.}
\label{elmhurstScatterLSRLine_conditions}
\end{figure}

\subsection{Midterm elections and unemployment}

\index{data!midterm elections|(}

Elections for members of the United States House of Representatives occur every two years, coinciding every four years with the U.S. Presidential election. The set of House elections occurring during the middle of a Presidential term are called \indexthis{midterm elections}{midterm election}. In America's two-party system, one political theory suggests the higher the unemployment rate, the worse the President's party will do in the midterm elections.

To assess the validity of this claim, we can compile historical data and look for a connection. We consider every midterm election from 1898 to 2010, with the exception of those elections during the Great Depression. Figure~\ref{unemploymentAndChangeInHouse} shows these data and the least-squares regression line: \vspace{-2mm}
\begin{align*}
&\text{\% change in House seats for President's party}  \\
&\qquad\qquad= -6.71 - 1.00\times \text{(unemployment rate)}
\end{align*}
We consider the percent change in the number of seats of the President's party (e.g. percent change in the number of seats for Democrats in 2010) against the unemployment rate.

Examining the data, there are no clear deviations from linearity, the constant variance condition, or in the normality of residuals (though we don't examine a normal probability plot here). While the data are collected sequentially, a separate analysis was used to check for any apparent correlation between successive observations; no such correlation was found.

\begin{figure}
\centering
\includegraphics[width=0.95\textwidth]{05/figures/unemploymentAndChangeInHouse/unemploymentAndChangeInHouse}
\caption{The percent change in House seats for the President's party in each election from 1898 to 2010 plotted against the unemployment rate. The two points for the Great Depression have been removed, and a least squares regression line has been fit to the data.}
\label{unemploymentAndChangeInHouse}
\end{figure}

\begin{exercise}
The data for the Great Depression (1934 and 1938) were removed because the unemployment rate was 21\% and 18\%, respectively. Do you agree that they should be removed for this investigation? Why or why not?\footnote{We will provide two considerations. Each of these points would have very high leverage on any least-squares regression line, and years with such high unemployment may not help us understand what would happen in other years where the unemployment is only modestly high. On the other hand, these are exceptional cases, and we would be discarding important information if we exclude them from a final analysis.}
\end{exercise}

There is a negative slope in the line shown in Figure~\ref{unemploymentAndChangeInHouse}. However, this slope (and the y-intercept) are only estimates of the parameter values. We might wonder, is this convincing evidence that the ``true'' linear model has a negative slope? That is, do the data provide strong evidence that the political theory is accurate? We can frame this investigation into a two-sided statistical hypothesis test. We use a two-sided test since a statistically significant result in either direction would be interesting.
\begin{itemize}
\item[$H_0$:] $\beta_1 = 0$. The true linear model has slope zero.
\item[$H_A$:] $\beta_1 \neq 0$. The true linear model has a slope different than zero. The higher the unemployment, the greater the loss for the President's party in the House of Representatives, or vice-versa.
\end{itemize}
We would reject $H_0$ in favor of $H_A$ if the data provide strong evidence that the true slope parameter is less than zero. To assess the hypotheses, we identify a standard error for the estimate, compute an appropriate test statistic, and identify the p-value.

\subsection{Understanding regression output from software}
\label{testStatisticForTheSlope}

Just like other point estimates we have seen before, we can compute a standard error and test statistic for $b_1$. We will generally label the test statistic using a $T$, since it follows the $t$ distribution.

We will rely on statistical software to compute the standard error and leave the explanation of how this standard error is determined to a second or third statistics course. Table~\ref{midtermElectionUnemploymentRRegressionOutput} shows software output for the least squares regression line in Figure~\ref{unemploymentAndChangeInHouse}. The row labeled \emph{unemp} represents the information for the slope, which is the coefficient of the unemployment variable.

\begin{table}[ht]
\centering
\begin{tabular}{rrrrr}
  \hline
  \vspace{-3.7mm} & & & & \\
 & Estimate & Std. Error & t value & Pr($>$$|$t$|$) \\
  \hline
  \vspace{-3.6mm} & & & & \\
(Intercept) & -6.7142 & 5.4567 & -1.23 & 0.2300 \\
  unemp & -1.0010 & 0.8717 & -1.15 & 0.2617 \\
   \hline
   \multicolumn{5}{r}{$df=25$} \\
\end{tabular}
\caption{Output from statistical software for the regression line modeling the midterm election gains and losses for the President's party as a response to unemployment.}
\label{midtermElectionUnemploymentRRegressionOutput}
\end{table}

\begin{example}{What do the first and second columns of Table~\ref{midtermElectionUnemploymentRRegressionOutput} represent?}
The entries in the first column represent the least squares estimates, $b_0$ and $b_1$, and the values in the second column correspond to the standard errors of each estimate.
\end{example}

We previously used a $t$ test statistic for hypothesis testing in the context of numerical data. Regression is very similar. In the hypotheses we consider, the null value for the slope is 0, so we can compute the test statistic using the T (or Z) score formula:
\begin{align*}
T = \frac{\text{estimate} - \text{null value}}{\text{SE}} = \frac{-1.0010 - 0}{0.8717} = -1.15
\end{align*}
We can look for the two-tailed p-value -- shown in Figure~\ref{oneSidedTailForMidtermUnemploymentHT} -- using the probability table for the $t$ distribution in Appendix~\ref{tDistributionTable} on page~\pageref{tDistributionTable}.

\begin{figure}
\centering
\includegraphics[width=0.82\textwidth]{05/figures/pValueMidtermUnemploymentHT/pValueMidtermUnemploymentHT}
\caption{The distribution shown here is the sampling distribution for $b_1$, if the null hypothesis was true. The shaded tail represents the p-value for the hypothesis test evaluating whether there is convincing evidence that higher unemployment corresponds to a greater loss of House seats for the President's party during a midterm election.}
\label{oneSidedTailForMidtermUnemploymentHT}
\end{figure}

\textA{\pagebreak}

\begin{example}{Table~\ref{midtermElectionUnemploymentRRegressionOutput} offers the degrees of freedom for the test statistic $T$: $df=25$. Identify the p-value for the hypothesis test.}
Looking in the 25 degrees of freedom row in Appendix~\ref{tDistributionTable}, we see that the absolute value of the test statistic is smaller than any value listed, which means the tail area and therefore also the p-value is larger than 0.200 (two tails!). Because the p-value is so large, we fail to reject the null hypothesis. That is, the data do not provide convincing evidence that unemployment is a good predictor of how well a president's party will do in the midterm elections for the House of Representatives.
\index{data!midterm elections|)}
\end{example}

We could have identified the $t$ test statistic from the software output in Table~\ref{midtermElectionUnemploymentRRegressionOutput}, shown in the second row (unemp) and third column (t value). The entry in the second row and last column in Table~\ref{midtermElectionUnemploymentRRegressionOutput} represents the p-value for the two-sided hypothesis test where the null value is zero.

\begin{termBox}{\tBoxTitle{Inference for regression}
We usually rely on statistical software to identify point estimates and standard errors for parameters of a regression line. After verifying conditions hold for fitting a line, we can use the methods learned in Section~\ref{oneSampleMeansWithTDistribution} for the $t$ distribution to create confidence intervals for regression parameters or to evaluate hypothesis tests.}
\end{termBox}

\begin{caution}{Don't carelessly use the p-value from regression output}{The last column in regression output often lists p-values for one particular hypothesis: a two-sided test where the null value is zero. If a hypothesis test should be one-sided or a comparison is being made to a value other than zero, be cautious about using the software output to obtain the p-value.}
\end{caution}

\textA{\pagebreak}

\begin{example}{Examine Figure~\vref{elmhurstScatterWLSROnly}, which relates the Elmhurst College aid and student family income. How sure are you that the slope is statistically significantly different from zero? That is, do you think a formal hypothesis test would reject the claim that the true slope of the line should be zero?} \label{overallAidIncomeInformalAssessmentOfRegressionLineSlope}
While the relationship between the variables is not perfect, there is an evident decreasing trend in the data. This suggests the hypothesis test will reject the null claim that the slope is zero.
\end{example}

\begin{exercise}
Table~\ref{rOutputForIncomeAidLSRLineInInferenceSection} shows statistical software output from fitting the least squares regression line shown in Figure~\ref{elmhurstScatterWLSROnly}. Use this output to formally evaluate the following hypotheses. $H_0$: The true coefficient for family income is zero. $H_A$: The true coefficient for family income is not zero.\footnote{We look in the second row corresponding to the family income variable. We see the point estimate of the slope of the line is -0.0431, the standard error of this estimate is 0.0108, and the $t$ test statistic is -3.98. The p-value corresponds exactly to the two-sided test we are interested in: 0.0002. The p-value is so small that we reject the null hypothesis and conclude that family income and financial aid at Elmhurst College for freshman entering in the year 2011 are negatively correlated and the true slope parameter is indeed less than 0, just as we believed in Example~\ref{overallAidIncomeInformalAssessmentOfRegressionLineSlope}.}
\end{exercise}

\begin{table}[ht]
\centering
\begin{tabular}{rrrrr}
  \hline
  \vspace{-3.7mm} & & & & \\
 & Estimate & Std. Error & t value & Pr($>$$|$t$|$) \\
  \hline
  \vspace{-3.6mm} & & & & \\
(Intercept) & 24.3193 & 1.2915 & 18.83 & 0.0000 \\
family\_\hspace{0.3mm}income & -0.0431 & 0.0108 & -3.98 & 0.0002 \\
   \hline
   \multicolumn{5}{r}{$df=48$} \\
\end{tabular}
\caption{Summary of least squares fit for the Elmhurst College data.\textA{\vspace{-5mm}}}
\label{rOutputForIncomeAidLSRLineInInferenceSection}
\end{table}

\begin{tipBox}{\tipBoxTitle{Always check assumptions}
If conditions for fitting the regression line do not hold, then the methods presented here should not be applied. The standard error or distribution assumption of the point estimate -- assumed to be normal when applying the $t$ test statistic -- may not be valid.}
\end{tipBox}

\ANOVASection{\subsection{An alternative test statistic}

We considered the $t$ test statistic as a way to evaluate the strength of evidence for a hypothesis test in Section~\ref{testStatisticForTheSlope}. However, we could focus on $R^2$. Recall that $R^2$ described the proportion of variability in the response variable ($y$) explained by the explanatory variable ($x$). If this proportion is large, then this suggests a linear relationship exists between the variables. If this proportion is small, then the evidence provided by the data may not be convincing.

This concept -- considering the amount of variability in the response variable explained by the explanatory variable -- is a key component in some statistical techniques. The \emph{analysis of variance (ANOVA)}\index{analysis of variance (ANOVA)} technique introduced in Section~\ref{anovaAndRegrWithCategoricalVariables} uses this general principle. The method states that if enough variability is explained away by the categories, then we would conclude the mean varied between the categories. On the other hand, we might not be convinced if only a little variability is explained. ANOVA can be further employed in advanced regression modeling to evaluate the inclusion of explanatory variables, though we leave these details to a later course.}{}