\documentclass{article}
\usepackage{algpseudocode}
\usepackage{algorithm}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{fancyhdr}
\usepackage{listings}
\usepackage{graphicx}
\usepackage[section]{placeins}
\usepackage{titlesec}
\usepackage{amsmath}
\usepackage[margin=1in]{geometry}
\pagestyle{fancy}
\fancyhf{}
\rhead{Rahul Shah}
\lhead{Machine Learning: Nanodegree}
\rfoot{\thepage}
\title{Supervised Learning: Linear Regression}
\begin{document}
\maketitle
\section{Motivation}
% Talk about the housing problem and trying to predict the next house in
Imagine you are trying to predict the prices of houses. The only data you have available to you are the prices of a bunch of houses, and the square footage of all the houses. If given another house, alongside the square footage of said house, can you predict the price of the house?

If we assume that increasing square footage is approximately proportional to the price by a constant (the data is approximately linear), we can plot each point $P_i = (x, y) = (sq ft, \$\$)$ in a scatter plot and find a line of best fit to the data.

\section{Fitting a Line}
% If we know the thing is linear, then we can use a linear model to predict it
For any set of data \emph{assuming the data is approximately linear}, we can use a linear model to predict the output of any new data that comes in.
% make sure you mention the linear model we are using and what the coefficients mean
As a reminder, the linear model we are working with (for now) is:\newline $\hat{y} = \beta_1x + \beta_2$\newline where \newline $\hat{y} :=$ Predicted output of the linear model.\newline Note that if we increase $\beta_1$, we rotate the line CCW, if we decrease $\beta_1$, we rotate the line CW, and increasing/decreasing $\beta_2$ pulls up/down the line.
\section{Moving a Line Towards a Point}
% This is how we can approximate a line movement
Let's assume the line we have to work with is: $y(x) = \beta_1x + \beta_2$ ($y(x)$ will be treated as y unless explicitly used as a function), where $\beta_1, \beta_2$ are arbitrary constants, like a really bad guess as to what the actual line would be. Place a point reasonably far away from the line, $P = (p, q)$. How do we, using the data that we have, move the line such that it gets closer to the point?
\subsection{Absolute Trick}
The Absolute Trick is as such: Every iteration, we change $\beta_1$ by the value $p$ (the x coordinate), and change the value $\beta_2$ by the value 1 (we will get to cases where we increase/decrease in a second). So our model transforms to:\\
$y = \beta_1x+\beta_2 \rightarrow y = (\beta_1 \pm p)x+(\beta_2 \pm 1)$, where each $\pm$ is independent of one another\\
If we continue under the assumption that the point is far away, we might get a good approximation. In all likelihood, however, we are likely to overshoot. So instead we iterate over this multiple times, taking steps towards the right solution. We can do this by multiplying $p$ and 1 by $\alpha$, the learning rate. So our formula becomes:\\
$y = \beta_1x+\beta_2 \rightarrow y = (\beta_1 \pm p\alpha)x+(\beta_2 \pm \alpha)$\\
To determine whether to use + or -, we need to look at how we want the line to move. For $\beta_1$, we see which direction (CCW or CW) we should rotate the line to reach the point (smallest angle to sweep across so the line meets the point). For $\beta_2$, do the same thing, but this time your distance is up/down.
\subsection{Square Trick}
The above trick works to get near to the point, but we never factor in the y distance from the line (other than to determine the sign). What if we adjust $\beta_1$ and $\beta_2$ the same way as above, but also account for the vertical distance from the point to the line? We can do this by multiplying our $p\alpha$ and $\alpha$ term by $q-\hat{q}$, where $\hat{q} = y(p)$. So our formula for the square trick becomes:\\
$y = \beta_1x+\beta_2 \rightarrow y = (\beta_1 + p\alpha(q-\hat{q}))x+(\beta_2 + \alpha(q-\hat{q}))$\\
Note that we don't need to use $\pm$ anymore because multiplying by $q-\hat{q}$ means that if our point is below our line, this difference will yield a negative number,  and if the point is above our line, the difference will yield a positive number.
\subsection{Why not just do $\frac{q}{p}x$ as our line?}
Obviously these tricks are pretty bad standalone operations, but their power is in the fact that they aren't exact. When we do linear regression, we're modifying our line of best fit so it coincides with all the data, and, unless all of our data points are co-linear (HIGHLY unlikely!), we want our line of best fit to be close to all of our data points. Think of each point pulling towards a line, the farther away the point from the line, the harder it pulls the line towards it.

\section{What is the Line of Best Fit}
% Note that if we define error as the difference between our predicted y from our actual point, our best fit line is essentially the placement of the line such that the sum of all errors is at its smallest
Now that we have an iterative approach to pulling a line closer to a point, we need to define what a line of best fit is. If we define error (roughly) as the difference between our predicted y for each x value in our data, $\hat{y} = y(x)$, and the actual y value from each data point in our value, then our line of best fit is such that $\sum f(|y-\hat{y}|)$, such that $f$ is a monotonic function, is at a minimum.

\section{Linear Least Squares}
% Note on Linear Least Squares (matrix solution), and how, though it does solve the problem completely, it also involves inverting a matrix, which can be computationally taxing.

Turns out, finding the exact line of best fit for a set of points $(x_i, y_i)$ is an already solved problem, with a nice, closed form solution. Let our linear model be $\beta_1 x + \beta_0 = y$. If we consider the linear system:\\
$X\beta = y$\\
Where:\\
$X :=$ the overdetermined matrix
$\begin{bmatrix}
	1 & x_1\\
	\vdots & \vdots\\
	1 & x_n
\end{bmatrix}$\\
$y:=$ the vector of y values associated with each x in the data\\
$\beta :=$ the vector of beta constants in the linear system (from this model, it's $\beta_1$ and $\beta_0$)\\
We have an overdetermined system that does not have exact solutions for $\beta_1, \beta_0$. Since we need a square matrix were $X$ is, so we can solve it, we can multiply both sides by $X^T$ to get:\\
$X^TX\beta=X^Ty$\\
Solving for $\beta$, we get:\\
$\beta=(X^TX)^{-1}y$, where $\beta$ is guaranteed to be such that our model $\beta_1 x + \beta_0 = y$ minimizes the error.

%NEED TO CHECK THIS THERE IS A WAY BUT I DONT WANT TO DO THIS
%This can even be generalized, if we have a model $y = \beta_0 + \sum_{1}^{n} beta_if_i(\vec{x})$, where each $f$ is a polynomial, we can make the entries of each row of our matrix the functions $f_i(\vec{x})$, with every row in the matrix being one point from the input data $\vec{x} = x_1, ..., x_n$\\

Unfortunately, this linear least squares approach is rarely used. This is because the formula $\beta=(X^TX)^{-1}y$ means that we need to calculate the inverse of a matrix, which is an $O(n^3)$ operation.

% So, we take an iterative approach, (mention the mountain thing, and how, to get to the bottom, you just go where it falls off the most)

Since we can't solve this exactly, we need to get an approximation of our answer via iterative methods.

\section{Error Functions}
Since our method must be iterative and will not give us the exact answer, it would be useful to have an idea of what our error for any given model is.
\subsection{Mean Approximation Error (MAE)}
MAE $=\frac{1}{m}\sum |y_i - \hat{y}(x_i)$\\
We use absolute value so we do not have negative differences accidentally canceling positive ones. It's also a metric.
\subsection{Mean Squared Error (MSE)}
MSE $=\frac{1}{2m}\sum (y_i - \hat{y}(x_i))^2$\\
The added $\frac{1}{2} is a constant added for convienience.$

\section{Gradient Descent}
% Gradient is basically just the slope of the moutain at any point around you
Since our goal is to minimize error, why not just change $\beta_0, \beta_1$ in the direction of the greatest decrease of the error function, multiplied by a small number (so we don't overshoot)? That is, every iteration,\\
$\beta_i = \beta_i - \alpha\frac{\partial}{\partial \beta_i} Error = \beta_i - \alpha\frac{\partial Error}{\partial y_i}\frac{\partial y_i}{\partial x_i}$\\
If we apply this to our MAE and MSE square functions, we essentially get the Absolute and Square Trick, respectively

\subsection{Types of Gradient Descent}
% Batch, Stochastic, and Mini-Batch
% First two are basically terrible
% give pseudocode for mini-batch
Batch - compute the average change in $\beta$ using all data points, update $\beta$, and iterate over that
Stochastic - compute the change in $\beta$ using one data point, updating $\beta$, then do another, etc.
Mini-Batch - Split your data into small, even groups, and compute the change, and iterate over that.

Batch and Stochastic are pretty slow, so we do mini batch typically
% Can we make b just part of the matrix
\begin{algorithm}
	\caption{Gradient Descent- Mini Batch with MSE}
	\begin{algorithmic}
		\Procedure{Step}{$X$, $\beta$, $y$, $b$, $\alpha$}\\
			 \Comment $X$ the matrix containing loaded input data, $\beta$ the current values of our model's coefficients, $b$ current value of the regression intercept ($\beta_0$), $\alpha$ the step size.

			 $\hat{y} = XW + b$
			 $error = y - \hat{y}$

			 $W += \alpha\times error\times X$ \Comment this is just the derivative (negative is removed)
			 $b += \alpha * \sum error_i$
		\EndProcedure
	\end{algorithmic}
\end{algorithm}

\section{Generalizing to Higher Dimensions}
% Absolutely give matrix translations here or else nothing makes any goddamn sense

For an input vector $\vec{x} = x_1, ..., x_n$, we get an $n-1$ dimensional hyperplane.

\section{Warnings}
% Just copy whats on the Udacity site basically

\section{Polynomial Regression}
Basically, you can fit your model to any polynomial, so long as you treat each power of each input variable as another term on your x vector. Meaning, to model a quadratic,, you need rows of length 3 (const, power 1, power 2)

\section{Regularization}
Naturally we can fit multiple models to the same set of data. We can also minimize error by just doing linear least squares to a polynomial of an arbitrarily high power, but this makes an extremely computationally complex problem to solve. We also end up with a model that does not generalize to new data. To account for that, we regularize the model by summing up the elements in $\beta$ and tacking that onto our total error. Models that do not generalize well will have a much larger regularization parameter.

Sometimes, we want our model to be more accurate (less actual error) and are okay if the model is complex, so we multiply this regularization parameter by $\lambda$ to lower the error "punishment". If we can tolerate higher errors & need simplicity (perhaps operating on a lot of data), we use a higher $\lambda$ on our regularization parameter. \\

2 kinds of regularization:\\
L1: take the L1 distance of the coefficients\\
L2:    "     L2             "
\section{Feature Scaling}
When we implement regularization or are using a distance based metric, we need to scale the range of our input vector values so that the penalty for error is uniform, and you reach convergence faster. Essentially what you want to do is to make each feature (each $x_i$ for each piece of data) and make it so its mean is 0 and standard deviation is 1. You can also just normalize the data from 0 to 1 or standardize the values such that each value is a z-score
\section{External Resources}
%no later i hate this
\end{document}


% !TEX root = ../master/master.tex

%% Dexter Barrows, 2016
%% dbarrows.github.io

	Particle filters are similar to MCMC-based methods in that they use likelihoods to evaluate the validity of proposed parameter sets given observed data $D$, but differ in that they are largely trying to produce point estimates of the parameters instead of samples from the posterior distribution.

	Instead of constructing a Markov chain and approximating its stationary distribution, a cohort of ``particles'' are used to move through the data in an on-line (sequential) fashion with the cohort being culled of poorly-performing particles at each iteration via importance sampling. If the culled particles are not replenished, this will be a Sequential Importance Sampling (SIS) particle filter. If the culled particles are replenished from surviving particles, in a sense setting up a process analogous to Darwinian selection, then this will be a Sequential Importance Resampling (SIR) particle filter \cite{Arulampalam2002}.

\section{Formulation}

	Particle filters, also called Sequential Monte-Carlo (SMC) filters, feature similar core functionality as the venerable Kalman Filter. As the algorithm moves through the data (sequence of observations), a prediction-update cycle is used to simulate the evolution of the model $M$ with different particular parameter selections, track how closely these predictions approximate the new observed value, and update the current cohort appropriately \cite{Arulampalam2002}.

	Two separate functions are used to simulate the evolution and observation processes. The ``true'' state evolution is specified by

	\begin{equation}
		X_{t+1} \sim f_1 (X_t, \theta),
	\end{equation}

	And the observation process by

	\begin{equation}
		Y_t \sim f_2 (X_t, \theta).
	\end{equation}

	Components of $\theta$ can contribute to both functions, but a typical formulation is to have some components contribute to $f_1 (\cdot, \theta)$ and others to $f_2 (\cdot,\theta)$.

	The prediction part of the cycle uses $f_1 (\cdot, \theta)$ to update each particle's current state estimate to the next time step, while $f_2 (\cdot, \theta)$ is used to evaluate a weighting $w$ for each particle which will be used to determine how closely that particle is estimating the true underlying state of the system. Note that $f_2 (\cdot, \theta)$ could be thought of as a probability of observing a piece of data $y_t$ given the particle's current state estimate and parameter set, $P(y_t | X_t, \theta)$. Then, the new cohort of particles is drawn from the old cohort proportional to the weights. This process is repeated until the set of observations $D$ is exhausted.

\section{Algorithm}

    Now we will formalize the particle filter.

    We will denote each particle $p^{(j)}$ as the $j^{\text{th}}$ particle consisting of a state estimate at time $t$, $X_t^{(j)}$, a parameter set $\theta^{(j)}$, and a weight $w^{(j)}$. Note that the state estimates will evolve with the system as the cohort traverses the data.

    The algorithm for a Sequential Importance Resampling particle is shown in Algorithm [\ref{pfsir}].

    \begin{algorithm}

        \BlankLine

        \SetKwInOut{Input}{Input}
        \SetKwInOut{Output}{Output}
        \DontPrintSemicolon

        \tcc{Select a starting point}
        \Input{Observations $D = y_1, y_2, ..., y_T$, initial particle distribution $P_0$ of size $J$}

        \BlankLine

        \tcc{Setup}
        Initialize particle cohort by sampling $(p^{(1)}, p^{(2)}, ..., p^{(J)})$ from $P_0$

        \BlankLine

        \For{$t = 1:T$}{

            \BlankLine

            \tcc{Evolve}
            \For{j = 1:J}{
            	$X_t^{(j)} \gets f_1 (X_{t-1}^{(j)}, \theta^{(j)})$
            }

            \BlankLine

            \tcc{Weight}
            \For{j = 1:J}{
            	$w^{(j)} \gets P(y_t | X_t^{(j)}, \theta^{(j)}) = f_2 (X_t^{(j)}, \theta^{(j)})$
            }

            \BlankLine

            \tcc{Normalize}
            \For{j = 1:J}{
            	$w^{(j)} \gets w^{(j)} / \sum_{1}^{J} w^{(j)}$
            }

            \BlankLine

            \tcc{Resample}
            $p^{(1:J)} \gets \text{sample}(p^{(1:J)}, \text{prob} = w, \text{replace} = true)$
        }

        \BlankLine

        \tcc{Samples from approximated posterior distribution}
        \Output{Cohort of posterior samples $(\theta^{(1)},\theta^{(2)},...,\theta^{(J)})$}

        \BlankLine

        \caption{SIR particle filter \label{pfsir}}

    \end{algorithm}

\section{Particle Collapse}

	Often, a situation may arise in which a single particle is assigned a normalized weight very close to 1 and all the other particles are assigned weights very close to 0. When this occurs, the next generation of the cohort will overwhelmingly consist of descendants of the heavily-weighted particle, termed particle collapse or degeneracy \cite{Bengtsson2008}\cite{Arulampalam2002}.

	Since the basic SIR particle filter does not perturb either the particle system states or system parameter values, the cohort will quickly consist solely of identical particles, effectively halting further exploration of the parameter space as new data is introduced.

	A similar situation occurs when a small number of particles (but not necessarily a single particle) split almost all of the normalized weight between them, then jointly dominate the resampling process for the remainder of the iterations. This again halts the exploration of the parameter space with new data.

	In either case, the hallmark feature used to detect collapse is the same -- at some point the cohort will consist of particles with very similar or identical parameter sets which will consequently result in their assigned weights being extremely close together.

	Mathematically, we are interested in the number of effective particles, $N_{\small\text{eff}}$, which represents the number of particles that are acceptably dissimilar. This is estimated by evaluating

	\begin{equation}
		N_{\small\text{eff}} = \frac{1}{\sum_1^J (w^{(j)})^2}.
	\end{equation}

	This can be used to diagnose not only when collapse has occurred, but can also indicate when it is near \cite{Arulampalam2002}.

\section{Iterated Filtering and Data Cloning}

	A particle filter hinges on the idea that as it progresses through the data set $D$, its estimate of the posterior carried in the cohort of particles approaches maximum likelihood. However, this convergence may not be fast enough so that the estimate it produces is of quality before the data runs out. One way around this problem is to ``clone'' the data and make multiple passes through it as if it were a continuation of the original time series. Note that the system state contained in each particle will have to be reset with each pass.

	Rigorous proofs have been developed \cite{Ionides2006}\cite{Ionides2015} that show that by treating the parameters as stochastic processes instead of fixed values, the multiple passes through the data will indeed force convergence of the process mean toward maximum likelihood, and the process variance toward 0.

\section{Iterated Filtering 2 (IF2)}

	The successor to Iterated Filtering 1 \cite{Ionides2006}, Iterated Filtering 2 \cite{Ionides2015} is simpler, faster, and demonstrated better convergence toward maximum likelihood. The core concept involves a two-pronged approach. First, a data cloning-like procedure is used to allow more time for the parameter stochastic process means to converge to maximum likelihood, and frequent cooled perturbation of the particle parameters allow better exploration of the parameter space while still allowing convergence to good point estimates.

	IF2 is not designed to estimate the full posterior distribution, instead to produce a Maximum Likelihood (ML) point estimate. Further, IF2 thwarts the problem of particle collapse by keeping at least some perturbation in the system at all times. It is important to note that while true particle collapse will not occur, there is still risk of a pseudo-collapse in which all particles will be extremely close to one another so as to be virtually indistinguishable. However this will only occur with the use of overly-aggressive cooling strategies or by specifying an excessive number of passes through the data.

	An important new quantity is the particle perturbation density denoted $h(\theta|,\sigma)$. Typically this is multivariate Normal with $\sigma$ being a vector of variances proportional to the expected values of $\theta$. In practice the proportionality can be derived from current means or specified ahead of time. Further, these intensities must decrease over time. This can be done via exponential or geometric cooling, a decreasing step function, a combination of these, or through some other similar scheme.

	The algorithm for IF2 can be seen in Algorithm [\ref{if2}].\\

    \begin{algorithm}

        \BlankLine

        \SetKwInOut{Input}{Input}
        \SetKwInOut{Output}{Output}
        \DontPrintSemicolon

        \tcc{Select a starting point}
        \Input{Observations $D = y_1, y_2, ..., y_T$, initial particle distribution $P_0$ of size $J$, decreasing sequence of perturbation intensity vectors $\sigma_1, \sigma_2, ..., \sigma_M$}

        \BlankLine

        \tcc{Setup}
        Initialize particle cohort by sampling $(p^{(1)}, p^{(2)}, ..., p^{(J)})$ from $P_0$

        \BlankLine

        \tcc{Particle seeding distribution}
        $\Theta \gets P_0$

        \BlankLine

        \For{$m = 1:M$}{

        	\BlankLine

        	\tcc{Pass perturbation}
            \For{j = 1:J}{
            	$p^{(j)} \sim h(\Theta^{(j)}, \sigma_m)$
            }

            \BlankLine

	        \For{$t = 1:T$}{

	        	\BlankLine

		        \For{j = 1:J}{

		        	\BlankLine
		        	\tcc{Iteration perturbation}
	            	$p^{(j)} \sim h(p^{(j)}, \sigma_m)$

	            	\BlankLine
	            	\tcc{Evolve}
	            	$X_t^{(j)} \gets f_1 (X_{t-1}^{(j)}, \theta^{(j)})$

	            	\BlankLine
	            	\tcc{Weight}
	            	$w^{(j)} \gets P(y_t | X_t^{(j)}, \theta^{(j)}) = f_2 (X_t^{(j)}, \theta^{(j)})$

	            }

	            \BlankLine

	            \tcc{Normalize}
	            \For{j = 1:J}{
	            	$w^{(j)} \gets w^{(j)} / \sum_{1}^{J} w^{(j)}$
	            }

	            \BlankLine

	            \tcc{Resample}
	            $p^{(1:J)} \gets \text{sample}(p^{(1:J)}, \text{prob} = w, \text{replace} = true)$

	        }

	        \BlankLine

	        \tcc{Collect particles for next pass}
	        \For{$j = 1:J$}{
	        	$\Theta^{(j)} \gets p^{(j)}$
	        }

	    }

        \BlankLine

        \tcc{Samples from approximated posterior distribution}
        \Output{Cohort of posterior samples $(\theta^{(1)},\theta^{(2)},...,\theta^{(J)})$}

        \BlankLine

        \caption{IF2 \label{if2}}

    \end{algorithm}

\section{IF2 Fitting}

    Here we will examine a test case in which IF2 will be used to fit a Susceptible-Infected-Removed (SIR) epidemic model to mock infectious count data.

    As in the previous section, the model in Equation [\ref{sirode}] was use to produce synthetic data. The same parameters and initial conditions were used, namely: parameter values were set to $\mathcal{R}_0 = 3.0, \gamma = 0.1, N = 500$, initial conditions were set to 5 infectious individuals, 495 people susceptible to infection, and no one had yet recovered from infection and been removed, and observation error was taken to be $\varepsilon_{obs} \sim \mathcal{N}(0,\sigma)$, where individual values were drawn for each synthetic data point.

    Figure [\ref{mcmcdataplot}] in the previous section shows the true SIR ODE system solution and data.

    The IF2 algorithm was implemented in C++ for speed, and integrated into the R workflow using the Rcpp package.

    There are three primary reasons we implemented our own version of IF2 instead of using POMP. First, POMP does not provide final particle state distributions, making it difficult to calibrate the algorithm parameters against the parameters used in RStan (this procedure is described in the next chapter). Second, it is prudent to cross-check the validity of an algorithm using another implementation. Third, this code can then serve as a jumping-off point for further development using Graphics Processing Unit acceleration (outlined in Chapter 8). We must acknowledge the disadvantages as well: POMP has been extensively vetted with real-world usage, and using it would require far less work as we would only need to specify the model. That being said, we believe the advantages outweigh the disadvantages in this case, and so have proceeded to develop our own implementation of IF2.

	Figure [\ref{if2kernelplot}] shows the final kernel estimates for four of the key parameters. As with HMC, the distributions are not perfect, but are promising. Unlike with HMC, these distributions are not meant to consist of samples from the true posterior distribution, but rather serve a diagnostic role.

    \begin{figure}
        \centering
        \captionsetup{width=0.8\linewidth}
        \includegraphics[width=0.8\textwidth]{./images/if2kernels.pdf}
        \caption{Kernel estimates for four essential system parameters. True values are indicated by dashed lines. \label{if2kernelplot}}
    \end{figure}


\section{Washburn's Algorithm}

We have attempted to break down Washburn's algorithm and explain it in
somewhat more detail than he does in his paper. Readers should be
warned that we use different notation and our $i,j$ indices are
transposed from his.

Washburn notes that the problem can be conceived as that of finding
the appropriate links to draw between the two parts of a bipartite
graph with Area nodes on one side and Resource nodes on the other. For
example, in Figure \ref{Fig:wbnodes} there are 4 areas (index $i$) and
3 resources (index $j$).
\begin{figure}[htbp]
  \begin{center}
    \includegraphics{wbnodes.eps}
    \caption{Illustration of the nodes in a Washburn graph. Any link between A and R is an allocation of resource $j$ to area $i$.}
    \label{fig:wbnodes}
  \end{center}
\end{figure}
In his paper, Washburn proves that an optimal allocation will involve
no cycles in the graph. That proof is what makes his algorithm
possible.

\subsection{Initialization Step}
\label{sec:init}
To start the algorithm, choose an initial set of links and initialize
some variables. Let $G$ be the corresponding graph.
\begin{enumerate}
\item Choose an initial set of links (corresponding to combinations
  which would have $t_{ij} > 0)$ if we used this as our allocation) such that:
  \begin{enumerate}
  \item Each resource is assigned. $\forall T_j > 0, \exists i :
    t_{ij} > 0$
  \item If $t_{ij} > 0$, $w_{ij} > 0$. Otherwise the algorithm breaks.
  \item $G$ is a forest. (It contains no cycles.)
  \end{enumerate}
\item Let $\Delta_1 = 0$  \fbox{Necessary?}
\item Let $p = 1$. \fbox{Necessary?}
\end{enumerate}

We have made an arbitrary resource allocation. A fine choice is to
allocate all resources to area 1. Washburn initially allocates all
resources to some highly improbable fake area. It does not seem to be
important.

The overview is that with each iteration we update our choices by
deleting some links and adding others. In Washburn's paper, he
calculates the resulting $t_ij$ at every step. However, that appears
to be unnecessary. Note that we do not calculate or specify them for
this initialization step.

\subsection{Step 1}
\label{sec:step1}

\subsection{Step 2: Construct $\lambda$ and $\mu$}
\label{sec:step2}
Construct vectors $\lambda$ and $\mu$ such that:
\begin{enumerate}
\item $\lambda_j = w_{ij}\mu_i$ for every edge $(ij)$ in $G$
\item $\lambda_j > 0, j = 1..R$
\item $\mu_i > 0, i = 1..A$
\end{enumerate}
You are free to set your initial $\mu$ to 1. The $\mu$s correspond to
areas, and the $\lambda$s correspond to resources. If there are any
joins in the graph (several resources assigned to the same area, or
several areas served by the same resource), there will be some
constraints on your choice of $\mu$. However, because there will be no
cycles, you can just walk the tree of nearest neighbors from your
first node setting things as you go. We have sample MATLAB code that
does this.

\subsection{Section 3: Find the $\alpha$s}
\label{sec:alpha}

The graph $G$ may not be connected.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: master
%%% End:


\chapter{Chapter 3. Elementary Set Theory}

\section*{3.9 Self-Study Questions}

	\begin{enumerate}

		\item[3.9.1] (a) is the definition of $X\subseteq Y$. Note that (d) is equivalent to (a): if there exists no $x\in X$ such that $x\notin Y$, then every $x\in X$ must also be in $Y$. Why? Well, can you find a counterexample? (h) is perhaps the most difficult to see to be correct. Also here it helps to think whether you can find a counterexample if (h) holds. Suppose that $X\nsubseteq Y$ and that (h) holds. That $X\nsubseteq Y$ means that there exists an $x\in X$ and $x\notin Y$. And (h) says that every $x\notin Y$ is also such that $x\notin X$. This would mean that some $x$ would need to be both such that $x\in Y$ and $x\notin X$, which is impossible. Hence $X\nsubseteq Y$ cannot be and we have to have $X\subseteq Y$, instead.

		\item[3.9.2] (e) is the only correct answer.

		\item[3.9.3] (a) is correct by the axiom of extensionality. (b) is not correct, since we might have, for example, \emph{one} element which is in both sets (and hence in the one iff in the other), but \emph{another} which is only in one and not the other (and hence the sets are different). (c) is more or less obviously not enough. (d) can be seen to be correct by the reasoning from 3.9.1.(d). (e) is not correct since it's not enough that we can ``pair'' the elements, they need to be the same. And (f) is too weak, it only implies $X\subseteq Y$.

		\item[3.9.4] Remember that two sets are distinct as soon as they have different members.

		\item[3.9.5--3.9.8] The correct answers follow immediately from the definitions of $\cap$ and $\cup$. Note that in logic and mathematics, we read ``or'' inclusively and therefore (e) is also correct in 3.9.5. Note further that to say that it's not the case that one thing or the other is the case is to say that both are not the case. Similarly, to say that it's not the case that two things are the case is to say that at least one of them is not the case. This, hopefully, helps with 3.9.6 and 3.9.8.

		\item[3.9.9] The correct answers follow directly from the conditions on what a function needs to do.

		\item[3.9.10] Remember from 3.6.11 that $\{f(x):x\in X'\}$ for $f:X\to Y$ and $X'\subseteq X$ is defined as the set $\{y:\text{there exists a }x\in X'\text{, such that }y=f(x)\}$. This means that an element $m$ is not in the set $\{n^2:n\in\mathbb{N}\text{ and }0\leq n\leq 10\}$ just in case it's not the case that there exists an $n\in\mathbb{N}$ such that $0\leq n\leq 10$ and $m=n^2$. But that's just the same as (b). Note that (a) is not correct since $4\in \{n^2:n\in\mathbb{N}\text{ and }0\leq n\leq 10\}$ since $0\leq 2\leq 4$ and $2^2=4$ but there exists nine numbers $0\leq n\leq 10$ with $n^2\neq 4$: $0^2=0, 1^2=1, 3^2=9, \mathellipsis, 10^2=100$.

	\end{enumerate}

\section*{3.10 Exercises}

	\begin{itemize}

	\item[3.10.1]

	\begin{enumerate}[(a)]
		    \item $\{1,3\}$
 		   \item $\{1,2,3,5\}$
 		   \item $X\setminus Y = \{2\},\ Y\setminus X = \{5\}$
   		 \item $\wp(X) = \{\emptyset,\{1\},\{2\},\{3\}, \{1,2\},\{1,3\},\{2,3\},\{1,2,3\}\}$\\$\wp(Y)=\{\emptyset,\{1\},\{3\},\{5\}, \{1,3\},\{1,5\},\{3,5\},\{1,3,5\}\} $
   		 \item $X \times Y = \{(1,1),(1,3),(1,5),(2,1),(2,3),(2,5),(3,1),(3,3),(3,5)\}$\\$Y \times X = \{(1,1),(1,2),(1,3),(3,1),(3,2),(3,3),(5,1),(5,2),(5,3)\}$
		\end{enumerate}

      \item[3.10.2]

        \begin{enumerate}[(a)]

          \item To prove $X\subseteq Y$ iff $X\cup Y = Y$, we need to prove both directions.\\$\Rightarrow\\ to\ prove:\\$If $ X\subseteq Y,\ $then $ X\cup Y = Y \\$ Suppose $X \subseteq Y$. We'll show that $X \cup Y = Y$. To show this, we must prove two things:

                \begin{enumerate}[(i)]
                  \item $Y \subseteq X \cup Y$: from the definition of union follows that $X\cup Y$ consists of all elements in $X$ and all elements in $Y$, hence all elements in $Y$ are in $X\cup Y$, which is what we needed to show.
                  \item $X\cup Y \subseteq Y$ Let $x$ be an arbitrary element of $X \cup Y$.
                        By the definition of union, we know that $x \in Y$ or $X \in X$.
                        We now make a distinction by cases (see \S2.3.9).
                        When $x \in Y$, we are fine, as we want to show that $x \in Y$.
                        When $x \in X$, we know by our assumption that $X\subseteq Y$, that $x \in Y$.
                        Therefore all $x \in X\cup Y$ are in $Y$, therefore $X\cup Y \subseteq Y$.
                \end{enumerate}
            Using the axiom of Extensionality, we can now conclude that $X \cup Y = Y$. \\\\
            $\Leftarrow\\to\ prove:$\\If $X\cup Y=Y$, then $X\subseteq Y$. We'll prove the contrapositive. We assume that $X \not \subseteq Y$. Hence there must be some element $x\in X$ such that $x\not \in Y$. As $X \cup Y$ contains all elements of $X$, it must also contain $x$. As there's one element that is in $X\cup Y$ and not in $Y$, $X \cup Y \neq Y$, which is what we needed to prove.\\\\ As we've now proved both directions, we've proved $X\subseteq Y$ iff $X\cup Y = Y \square$

          \item  We aim to show that $X \subseteq Y$ iff $X \cap Y = X$. To prove this we need to prove both sides of the biconditional:
            \begin{itemize}
              \item $\Rightarrow$. We need to show that if $X \subseteq Y$ then  $X \cap Y = X$. So, assume $X \subseteq Y$.
                We want to show that $X \cap Y = X$.
                By extionsionality, we must show two things:
                \begin{itemize}
                  \item First, we need to show that $X \cap Y \subseteq X$.
                    So, let an element $x \in X \cap Y$.
                    That means, by definition of $\cap$, that $x \in X$ and $x \in Y$.
                    But then certainly, $x \in X$, as desired.
                  \item Second, we need to show that $X \subseteq X \cap Y$.
                    So, let an element $x \in X$.
                    We have assumed that $X \subseteq Y$,
                    from which it follows that $x \in Y$.
                    As such we have $x \in X$ and $x \in Y$, which means, by definition of $\cap$, that $x \in X \cap Y$.
                \end{itemize}

                By the axiom of extensionality we can conclude $X \cap Y = X$.

              \item $\Leftarrow$. We need to show that if $X \cap Y = X$, then $X \subseteq Y$.
                We do a proof by contraposition.
                So, assume $X \not \subseteq Y$.
                That means there is an element $x \in X$ for which $x \not \in Y$.
                If $x \not \in Y$, then $x \not \in X \cap Y$.
                So $X \not \subseteq X \cap Y$ (because there is an $x \in X$ for which $x \not \in X \cap Y$) so $X \not = X \cap Y$.

            \end{itemize}

We have now proved both directions, so $X \subseteq Y$ iff $X \cap Y = X$.

\end{enumerate}
\item[3.10.3] \
\begin{enumerate}
    \item $	\begin{matrix}
		(1,1) && \overset{f}{\mapsto} && 1 \\
		(1,2) && \overset{f}{\mapsto} && 1 \\
		(1,3) && \overset{f}{\mapsto} && 1 \\
	\end{matrix}
	\qquad
	\begin{matrix}
		(2,1) && \overset{f}{\mapsto} && 1 \\
		(2,2) && \overset{f}{\mapsto} && 2\\
		(2,3) && \overset{f}{\mapsto} && 2\\
	\end{matrix}
	\qquad
		\begin{matrix}
 		(3,1)&& \overset{f}{\mapsto} && 1\\
		(3,2)&& \overset{f}{\mapsto} && 2\\
		(3,3) && \overset{f}{\mapsto} && 3\\
	\end{matrix}$
	\item $\begin{array}{c|ccc}
		f & 1 & 2 & 3 \\ \hline
		1 & 1& 1& 1 \\
		2 & 1 & 2 &2 \\
		3 & 1 & 2 & 3
	\end{array}$
	\item 	$
	f((x, y)) =
	\begin{cases}
	x \text{ if } x < y,\\
	y \text{ otherwise}
	\end{cases}
	$
\end{enumerate}{}

      \item[3.10.5] We want to prove that $f(n,m) = n + m$ for all $n, m \in \mathbb{N}$.
        We use the principal of mathematical induction.
        \emph{Base case}:
        We need to show that $f(n,0) = n + 0$.
        By the definition $f(n,0) = n$.
        It is trivial that $n = n + 0$.
        \emph{Induction step}:
        We need to show that for all $n, m \in \mathbb{N}$,
        if $f(n,m)=n+m$, then $f(n,m+1)=(n+m)+1$.
        So, let $n, m \in \mathbb{N}$.
        Assume the induction hypothesis that $f(n,m) = n + m$ (IH).
        Consider $f(n, m + 1)$.
        By the definition $f(n,m+1) = f(n,m) + 1$.
        By the IH and get $f(n,m+1) = n + m + 1$, which is what we had to show.
        By the principle of mathematical induction,
        we conclude that for all $n, m \in \mathbb{N}$ it is true that $f(n,m) = n + m$.

\item[3.10.7]
\begin{enumerate}[(a)]
    \item
    $l: Gargle \rightarrow \mathbb{N}$:\begin{enumerate}[(i)]
        \item \begin{enumerate}[(a)]
            \item $l(\clubsuit) = 1$
        \item $l(\spadesuit) = 1$
        \end{enumerate}
        \item \begin{enumerate}[(a)]
            \item $l(\Diamond x\Diamond)= l(x) + 2$
            \item $l(x\heartsuit y)= l(x)+l(y)+1 $
        \end{enumerate}
    \end{enumerate}

    \item     $\textbf{1}_\heartsuit: Gargle \rightarrow \{0,1\}$:\begin{enumerate}[(i)]
        \item    $\textbf{1}_\heartsuit(\clubsuit) =\textbf{1}_\heartsuit(\spadesuit) = 0$
        \item \begin{enumerate}[(a)]
            \item $\textbf{1}_\heartsuit(\Diamond x\Diamond)= \textbf{1}_\heartsuit(x)$
            \item $\textbf{1}_\heartsuit (x\heartsuit y)= 1$
        \end{enumerate}
    \end{enumerate}
\end{enumerate}
\item[3.10.8] We will prove by induction that the amount of $\Diamond$'s in a gargle is always even.
\begin{enumerate}[(i)]
    \item Base case 1: $\spadesuit$ has an even number of $\Diamond$'s, as 0 is even.
Base case 2: $\clubsuit$ has an even number of $\Diamond$'s, as 0 is even.
\item \begin{enumerate}[(a)]
    \item Assume $x$ has an even number of $\Diamond$'s.Then $\Diamond x \Diamond$ must also have an even number of $\Diamond$'s, as the number of $\Diamond$'s of $\Diamond x \Diamond$ is the number of $\Diamond$'s of $x$+2. We know that an even number +2 results in another even number.
    \item Assume $x,\ y$ have an even number of $\Diamond$'s. Then the number of $\Diamond$'s of $x\heartsuit y$ will be the number of $\Diamond$'s of $x$ + the number of $\Diamond$'s of $y$, which will be an even number, as an even number added to an even number results in an even number, which is what we needed to show.\\
\end{enumerate}
\end{enumerate}
    By means of induction we've now proved that the number of $\Diamond$'s in a Gargle is always even.
\item[3.10.9]
We will prove that $\spadesuit\Diamond\heartsuit\spadesuit \not \in Gargle$. We have just proven that every gargle has an even amount of $\Diamond$, but this one has exactly 1, and $1 = 0 \cdot 2 + 1$. Therefore 1 is odd, and not even. So we must conclude it's not a Gargle.
\end{itemize}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../../logic.tex"
%%% End:


\section{Polar Coordinates}\label{sec:PolarCoordinates}
Coordinate systems are tools that let us use algebraic methods to
understand geometry. While the \dfont{rectangular} (also called
\dfont{Cartesian}) coordinates that we
have been using are the most common, some problems are easier to
analyze in alternate coordinate systems. \index{Cartesian coordinates} \index{rectangular coordinates}

A coordinate system is a scheme that allows us to identify any point
in the plane or in three-dimensional space by a set of numbers. In
rectangular coordinates these numbers are interpreted, roughly
speaking, as the lengths of the sides of a rectangle.
In \dfont{polar coordinates}\index{polar coordinates} a
point in the plane is identified by a pair of numbers $(r,\theta)$.
The number $\theta$ measures the angle between the positive
$x$-axis and a ray that goes through the point,
as shown in Figure~\ref{fig:polar coordinates}; the number
$r$ measures the distance from the origin to the
point (the radius). Figure~\ref{fig:polar coordinates} shows the point with
rectangular coordinates $\ds (1,\sqrt3)$ and polar coordinates
$(2,\pi/3)$, 2 units from the origin and $\pi/3$ radians from the
positive $x$-axis.

\figure[H]
\centerline{\vbox{\beginpicture
\normalgraphs
%%\ninepoint
\setcoordinatesystem units <15truemm,15truemm>
\setplotarea x from 0 to 1.25, y from 0 to 1.9
\axis left ticks length <2pt> withvalues
  {1.728} / at 1.728 / /
\axis bottom ticks length <2pt> withvalues {$1$} / at 1 / /
\arrow <4pt> [0.35, 1] from 0 0 to 1.2 2.078
\put {$(2,\pi/3)$} [l] <3pt,0pt> at 1 1.73
\put {$\bullet$} at 1 1.728
\circulararc 60 degrees from 0.2 0 center at 0 0
\endpicture}}
\caption{Polar coordinates of the point $\ds (1,\sqrt3)$. \label{fig:polar coordinates}}
\endfigure

Just as we describe curves in the plane using equations involving $x$
and $y$, so can we describe curves using equations involving $r$ and
$\theta$. Most common are equations of the form $r=f(\theta)$.

\begin{example}{Circle in Polar Coordinates}{circleinpolar}
 Graph the curve given by $r=2$.
\end{example}

\begin{solution}
All points with $r=2$ are at
distance 2 from the origin, so $r=2$ describes the circle of radius 2
with center at the origin.
\end{solution}

\begin{example}{Cardioid}{cardiodexample}
 Graph the curve given by $r=1+\cos\theta$.
\end{example}

\begin{solution}
We first consider
$y=1+\cos x$, as in figure~\ref{fig:cardioid}. As $\theta$ goes
through the values in $[0,2\pi]$, the value of $r$ tracks the value of
$y$, forming the ``cardioid''
shape of  figure~\ref{fig:cardioid}.
For example, when $\theta=\pi/2$, $r=1+\cos(\pi/2)=1$, so we graph the
point at distance 1 from the origin along the positive $y$-axis, which
is at an angle of $\pi/2$ from the positive $x$-axis. When
$\theta=7\pi/4$, $\ds r=1+\cos(7\pi/4)=1+\sqrt2/2\approx 1.71$, and the
corresponding point appears in the fourth quadrant. If we look at the curve, we should explain the indentation which occurs at $\theta=\pi$. It should be obvious that $\cos x<0$ on $(\pi/2,3\pi/2)$, thus $r=1+\cos\theta<1$. Specifically, when $\theta=\pi$, $\cos\pi=-1$, so $r=0$, and we get the resulting indentation. This illustrates
one of the potential benefits of using polar coordinates: the equation
for this curve in rectangular coordinates would be \emph{quite} complicated.
\end{solution}

\figure[H]
\centerline{\vbox{\beginpicture
\normalgraphs
%%\ninepoint
\setcoordinatesystem units <12truemm,12truemm>
\setplotarea x from 0 to 6.2832, y from 0 to 2.2
\axis left ticks length <2pt> numbered from 0 to 2 by 1 /
\axis bottom ticks length <2pt> withvalues
  {$\pi/2$} {$\pi$} {$3\pi/2$} {$2\pi$} / at 1.57 3.14 4.7 6.28 / /
\multiput {$\bullet$} at 1.57 1 5.5 1.71 /
\setquadratic
\plot
0.000 2.000 0.105 1.995 0.209 1.978 0.314 1.951 0.419 1.914
0.524 1.866 0.628 1.809 0.733 1.743 0.838 1.669 0.942 1.588
1.047 1.500 1.152 1.407 1.257 1.309 1.361 1.208 1.466 1.105
1.571 1.000 1.676 0.895 1.780 0.792 1.885 0.691 1.990 0.593
2.094 0.500 2.199 0.412 2.304 0.331 2.409 0.257 2.513 0.191
2.618 0.134 2.723 0.086 2.827 0.049 2.932 0.022 3.037 0.005
3.142 0.000 3.246 0.005 3.351 0.022 3.456 0.049 3.560 0.086
3.665 0.134 3.770 0.191 3.875 0.257 3.979 0.331 4.084 0.412
4.189 0.500 4.294 0.593 4.398 0.691 4.503 0.792 4.608 0.895
4.712 1.000 4.817 1.105 4.922 1.208 5.027 1.309 5.131 1.407
5.236 1.500 5.341 1.588 5.445 1.669 5.550 1.743 5.655 1.809
5.760 1.866 5.864 1.914 5.969 1.951 6.074 1.978 6.178 1.995
6.283 2.000 /
\setcoordinatesystem units <12truemm,12truemm> point at -8 -1
\setplotarea x from -1 to 2.5, y from -1.5 to 1.5
\axis left shiftedto x=0 /
\axis bottom shiftedto y=0 /
\put {$\bullet$} at 0 1
\put {$\bullet$} at 1.207 -1.207
\setquadratic
\plot 2.000 0.000 1.984 0.208 1.935 0.411 1.856 0.603 1.748 0.778
1.616 0.933 1.464 1.063 1.295 1.166 1.117 1.240 0.933 1.285
0.750 1.299 0.572 1.285 0.405 1.245 0.251 1.182 0.115 1.098
-0.000 1.000 -0.094 0.891 -0.165 0.775 -0.214 0.657 -0.241 0.542
-0.250 0.433 -0.242 0.333 -0.221 0.246 -0.191 0.172 -0.155 0.112
-0.116 0.067 -0.079 0.035 -0.047 0.015 -0.021 0.005 -0.005 0.001
0.000 0.000 -0.005 -0.001 -0.021 -0.005 -0.047 -0.015 -0.079 -0.035
-0.116 -0.067 -0.155 -0.112 -0.191 -0.172 -0.221 -0.246 -0.242 -0.333
-0.250 -0.433 -0.241 -0.542 -0.214 -0.657 -0.165 -0.775 -0.094 -0.891
0.000 -1.000 0.115 -1.098 0.251 -1.182 0.405 -1.245 0.572 -1.285
0.750 -1.299 0.933 -1.285 1.117 -1.240 1.295 -1.166 1.464 -1.063
1.616 -0.933 1.748 -0.778 1.856 -0.603 1.935 -0.411 1.984 -0.208
2.000 0.000 /
\setdashes
\arrow <4pt> [0.35, 1] from 0 0 to 1.207 -1.207
\endpicture}}
\caption{A cardioid: $y=1+\cos x$ on the left, $r=1+\cos\theta$ on the right.\label{fig:cardioid}}
\endfigure

Each point in the plane is associated with exactly one pair of numbers
in the rectangular coordinate system; each point is associated with an
infinite number of pairs in polar coordinates. In the cardioid example,
we considered only the range $0\le \theta\le2\pi$, and
already there was a duplicate: $(2,0)$ and $(2,2\pi)$ are the same
point. Indeed, every value of $\theta$ outside the interval $[0,2\pi)$
duplicates a point on the curve $r=1+\cos\theta$ when
$0\le\theta<2\pi$. We can even make sense of polar coordinates like
$(-2,\pi/4)$: Go to the direction $\pi/4$ and then move a distance 2
in the opposite direction; see figure~\ref{fig:negative r
coordinate}. As usual, a negative angle $\theta$ means an angle
measured clockwise from the positive $x$-axis. The point in
figure~\ref{fig:negative r coordinate} also has coordinates
$(2,5\pi/4)$ and $(2,-3\pi/4)$.

\figure[H]
\centerline{\vbox{\beginpicture
\normalgraphs
%\eightpoint
\setcoordinatesystem units <10truemm,10truemm>
\setplotarea x from -2 to 2, y from -2 to 2
\axis left shiftedto x=0 ticks length <2pt>
  withvalues $-2$ $-1$ {} $1$ $2$ / quantity 5 /
\axis bottom shiftedto y=0 ticks length <2pt>
  withvalues $-2$ $-1$ {} $1$ $2$ / quantity 5 /
\put {$\pi/4$} [bl] <3pt,-4pt> at 0.4 0.3
\put {$\bullet$} at -1.4142 -1.4142
\circulararc 45 degrees from 0.4 0 center at 0 0
\setdashes\arrow <4pt> [0.35, 1] from 1 1 to -1.4142 -1.4142
\endpicture}}
\caption{The point $(-2,\pi/4)=(2,5\pi/4)=(2,-3\pi/4)$ in polar coordinates.\label{fig:negative r coordinate}}
\endfigure

The relationship between rectangular and polar coordinates is quite easy to
understand. The point with polar coordinates $(r,\theta)$ has
rectangular coordinates $x=r\cos\theta$ and $y=r\sin\theta$; this
follows immediately from the definition of the sine and cosine
functions. Using figure~\ref{fig:negative r coordinate} as an example, the point shown has rectangular coordinates
$\ds x=(-2)\cos(\pi/4)=-\sqrt2\approx 1.4142$ and
$\ds y=(-2)\sin(\pi/4)=-\sqrt2$.  This makes it very easy to convert
equations from rectangular to polar coordinates. \index{polar coordinates!convert to rectangular}

\begin{example}{Straight Line in Polar Coordinates}{StraightLineinPolar}
 Find the equation of the line $y=3x+2$ in polar
coordinates.
\end{example}

\begin{solution}
We merely substitute: $r\sin\theta=3r\cos\theta+2$, or
$\ds r= {2\over \sin\theta-3\cos\theta}$.
\end{solution}

\begin{example}{Equation of a Circle}{equationofcircleinpolar}
 Find the equation of the circle $\ds (x-1/2)^2+y^2=1/4$ in polar
coordinates.
\end{example}

\begin{solution}
Again substituting:
$\ds (r\cos\theta-1/2)^2+r^2\sin^2\theta=1/4$. A bit of algebra turns this
into $r=\cos(t)$. You should try plotting a few $(r,\theta)$ values to
convince yourself that this makes sense.
\end{solution}

\begin{example}{Spiral of Archimedes}{spiralofArchimedes}
 Graph the polar equation $r=\theta$.
\end{example}

\begin{solution}
Here the distance from
the origin exactly matches the angle, so a bit of thought makes it
clear that when $\theta\ge0$ we get the spiral of
Archimedes in figure~\ref{fig:spiral of Archimedes}. When $\theta<0$, $r$ is also
negative, and so the full graph is the right hand picture in the figure.
\end{solution}

\figure[H]
\centerline{\vbox{\beginpicture
\normalgraphs
%\eightpoint
\setcoordinatesystem units <5truemm,5truemm>
\setplotarea x from -3.4 to 6.5, y from -4.8 to 2
\axis left shiftedto x=0 /
\axis bottom shiftedto y=0 /
\multiput {$\bullet$} at 0.54 0.84 0 1.57 -3.14 0 6.28 0 /
\put {$(2\pi,2\pi)$} [b] <0pt,3pt> at 6.28 0
\put {$(\pi,\pi)$} [br] <-3pt,0pt> at -3.14 0
\put {$(\pi/2,\pi/2)$} [br] <-2pt,4pt> at 0 1.57
\put {$(1,1)$} [bl] <3pt,0pt> at 0.54 0.84
\plot 0.000 0.000 0.104 0.011 0.205 0.044 0.299 0.097 0.383 0.170
0.453 0.262 0.508 0.369 0.545 0.490 0.561 0.623 0.554 0.762
0.524 0.907 0.469 1.052 0.388 1.195 0.283 1.332 0.153 1.458
0.000 1.571 -0.175 1.666 -0.370 1.741 -0.582 1.793 -0.809 1.818
-1.047 1.814 -1.293 1.779 -1.542 1.712 -1.790 1.612 -2.033 1.477
-2.267 1.309 -2.487 1.107 -2.689 0.874 -2.868 0.610 -3.020 0.317
-3.142 0.000 -3.229 -0.339 -3.278 -0.697 -3.287 -1.068 -3.253 -1.448
-3.174 -1.833 -3.050 -2.216 -2.879 -2.593 -2.663 -2.957 -2.401 -3.304
-2.094 -3.628 -1.746 -3.922 -1.359 -4.183 -0.936 -4.405 -0.482 -4.582
0.000 -4.712 0.504 -4.791 1.023 -4.814 1.553 -4.781 2.087 -4.688
2.618 -4.534 3.139 -4.321 3.644 -4.047 4.125 -3.714 4.575 -3.324
4.988 -2.880 5.357 -2.385 5.677 -1.845 5.941 -1.263 6.145 -0.646
6.283 0.000 /
\setcoordinatesystem units <5truemm,5truemm> point at -18 0
\setplotarea x from -6.5 to 6.5, y from -4.8 to 2
\axis left shiftedto x=0 /
\axis bottom shiftedto y=0 /
\multiput {$\bullet$} at -0.54 0.84 0 1.57 3.14 0 -6.28 0 /
\put {$(-2\pi,-2\pi)$} [b] <0pt,3pt> at -6.28 0
\put {$(-\pi,-\pi)$} [bl] <3pt,2pt> at 3.14 0
\put {$(-\pi/2,-\pi/2)$} [br] <-2pt,4pt> at 0 1.57
\put {$(-1,-1)$} [t] <0pt,0pt> at -1.5 -1
\plot -6.283 0.000 -6.145 -0.646 -5.941 -1.263 -5.677 -1.845 -5.357 -2.385
-4.988 -2.880 -4.575 -3.324 -4.125 -3.714 -3.644 -4.047 -3.139 -4.321
-2.618 -4.534 -2.087 -4.688 -1.553 -4.781 -1.023 -4.814 -0.504 -4.791
0.000 -4.712 0.482 -4.582 0.936 -4.405 1.359 -4.183 1.746 -3.922
2.094 -3.628 2.401 -3.304 2.663 -2.957 2.879 -2.593 3.050 -2.216
3.174 -1.833 3.253 -1.448 3.287 -1.068 3.278 -0.697 3.229 -0.339
3.142 0.000 3.020 0.317 2.868 0.610 2.689 0.874 2.487 1.107
2.267 1.309 2.033 1.477 1.790 1.612 1.542 1.712 1.293 1.779
1.047 1.814 0.809 1.818 0.582 1.793 0.370 1.741 0.175 1.666
0.000 1.571 -0.153 1.458 -0.283 1.332 -0.388 1.195 -0.469 1.052
-0.524 0.907 -0.554 0.762 -0.561 0.623 -0.545 0.490 -0.508 0.369
-0.453 0.262 -0.383 0.170 -0.299 0.097 -0.205 0.044 -0.104 0.011
0.000 0.000 0.104 0.011 0.205 0.044 0.299 0.097 0.383 0.170
0.453 0.262 0.508 0.369 0.545 0.490 0.561 0.623 0.554 0.762
0.524 0.907 0.469 1.052 0.388 1.195 0.283 1.332 0.153 1.458
0.000 1.571 -0.175 1.666 -0.370 1.741 -0.582 1.793 -0.809 1.818
-1.047 1.814 -1.293 1.779 -1.542 1.712 -1.790 1.612 -2.033 1.477
-2.267 1.309 -2.487 1.107 -2.689 0.874 -2.868 0.610 -3.020 0.317
-3.142 0.000 -3.229 -0.339 -3.278 -0.697 -3.287 -1.068 -3.253 -1.448
-3.174 -1.833 -3.050 -2.216 -2.879 -2.593 -2.663 -2.957 -2.401 -3.304
-2.094 -3.628 -1.746 -3.922 -1.359 -4.183 -0.936 -4.405 -0.482 -4.582
0.000 -4.712 0.504 -4.791 1.023 -4.814 1.553 -4.781 2.087 -4.688
2.618 -4.534 3.139 -4.321 3.644 -4.047 4.125 -3.714 4.575 -3.324
4.988 -2.880 5.357 -2.385 5.677 -1.845 5.941 -1.263 6.145 -0.646
6.283 0.000 /
\setdashes <2pt>
\setlinear
\plot -2 -0.9 -0.54 0.84 /
%\altarrow <4pt,5pt> [0.35, 1] from -2 -0.9 to -0.54 0.84
\endpicture}}
\caption{The spiral of Archimedes and the full graph of $r=\theta$.\label{fig:spiral of Archimedes}}
\endfigure

Converting polar equations to rectangular equations can be somewhat
trickier, and graphing polar equations directly is also not always easy.

\begin{example}{Graphing Polar Equations}{graphingpolarequations}
 Graph $r=2\sin\theta$.
\end{example}

\begin{solution}
Given that the sine is periodic, we know
that we will get the entire curve for values of $\theta$ in
$[0,2\pi)$. As $\theta$ runs from 0 to $\pi/2$, $r$ increases from 0
to 2. Then as $\theta$ continues to $\pi$, $r$ decreases again to
0. When $\theta$ runs from $\pi$ to $2\pi$, $r$ is negative, and it
is not hard to see that the first part of the curve is simply traced
out again, so in fact we get the whole curve for values of $\theta$
in $[0,\pi)$. Thus, the curve looks something like
figure~\ref{fig:circle from polar equation}. Now, this suggests
that the curve could possibly be a circle, and if it is, it would
have to be the circle $\ds x^2+(y-1)^2=1$. Having made this guess, we
can easily check it. First we substitute for $x$ and $y$ to get
$\ds (r\cos\theta)^2+(r\sin\theta-1)^2=1$; expanding and simplifying
does indeed turn this into $r=2\sin\theta$.
\end{solution}

\figure[H]
\centerline{\vbox{\beginpicture
\normalgraphs
%%\ninepoint
\setcoordinatesystem units <15truemm,15truemm>
\setplotarea x from -1 to 1, y from 0 to 2.2
\axis left shiftedto x=0 ticks length <2pt> numbered from 1 to 1 by 1 /
\axis bottom ticks length <2pt> numbered from -1 to 1 by 1 /
\circulararc 360 degrees from 1 1 center at 0 1
\endpicture}}
\caption{Graph of $r=2\sin\theta$.\label{fig:circle from polar equation}}
\endfigure

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:PolarCoordinates}}

\begin{enumialphparenastyle}

%%%%%%%%%%
\begin{ex}
Plot these polar coordinate points on one graph:
$(2,\pi/3)$, $(-3,\pi/2)$, $(-2,-\pi/4)$, $(1/2,\pi)$, $(1,4\pi/3)$,
$(0,3\pi/2)$.
\end{ex}

%%%%%%%%%%
\begin{ex}
Find an equation in polar coordinates that has the same
graph as the given equation in rectangular coordinates.
\begin{multicols}{3}
\begin{enumerate}
	\item	$\ds y=3x$
	\item	$\ds y=-4$
	\item	$\ds xy^2=1$
	\item	$\ds x^2+y^2=5$
	\item	$\ds y=x^3$
	\item	$\ds y=\sin x$
	\item	$\ds y=5x+2$
	\item	$\ds x=2$
	\item	$\ds y=x^2+1$
\end{enumerate}
\end{multicols}
\begin{sol}
\begin{multicols}{3}
\begin{enumerate}
	\item	$\ds \theta=\arctan(3)$
	\item	$\ds r=-4\csc\theta$
	\item	$\ds r=\sec\theta\csc^2\theta$
	\item	$\ds r=\sqrt{5}$
	\item	$\ds r^2=\sin\theta\sec^3\theta$
	\item	$\ds r\sin\theta=\sin(r\cos\theta)$
	\item	$\ds r=2/(\sin\theta-5\cos\theta)$
	\item	$\ds r=2\sec\theta$
	\item	$\ds 0=r^2\cos^2\theta-r\sin\theta+1$
\end{enumerate}
\end{multicols}
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Sketch the following curves:
\begin{multicols}{2}
\begin{enumerate}
	\item	$\ds r=\cos\theta$
	\item	$\ds r=1+\theta^1/\pi^2$
	\item	$\ds r=\sin(\theta+\pi/4)$
	\item	$\ds r=-\sec\theta$
	\item	$\ds r=\theta/2$, $\theta\ge0$
	\item	$\ds r=\cot\theta\csc\theta$
	\item	$\ds r={1\over\sin\theta+\cos\theta}$
	\item	$\ds r^2=-2\sec\theta\csc\theta$
\end{enumerate}
\end{multicols}
\end{ex}

%%%%%%%%%%
\begin{ex}
Find an equation in rectangular coordinates that has the same
graph as the given equation in polar coordinates.
\begin{multicols}{2}
\begin{enumerate}
	\item	$\ds r=\sin(3\theta)$
	\item	$\ds r=\sin^2\theta$
	\item	$\ds r=\sec\theta\csc\theta$
	\item	$\ds r=\tan\theta$
\end{enumerate}
\end{multicols}
\begin{sol}
\begin{multicols}{2}
\begin{enumerate}
	\item	$\ds (x^2+y^2)^2=4x^2y-(x^2+y^2)y$
	\item	$\ds (x^2+y^2)^{3/2}=y^2$
	\item	$\ds x^2+y^2=x^2y^2$
	\item	$\ds x^4+x^2y^2=y^2$
\end{enumerate}
\end{multicols}
\end{sol}
\end{ex}

\end{enumialphparenastyle}


\subsection{Heteroskedasticity and Autocorrelation (HAC) adjusted standard errors}


% $Id$
%
% Author: David Fournier
% Copyright (c) 2008 Regents of the University of California
%

\section{The Kalman filter}

The Kalman filter is a device for estimating parameters in
a class of ``time-series''-like models that are put into state-space
form. We have used the
notation from \cite{harvey1990}, Chapter~3.
\X{state-space form}
\X{Kalman filter}
The general state-space form is an  multivariate time series
$$y_t=Z_t\alpha_t+d_t+\epsilon_t$$
where $Z_t$ is an $N\times m$ matrix, $d_t$ is an $N$-dimensional
vector, $y_t$ is an $N$-dimensional vector, and $\epsilon_t$ is
a set of serially uncorrelated $N$-dimensional random vectors
with mean $0$ and correlation $H_t$.
The elements of $\alpha_t$ are not observable, but are
assumed to be generated by a first-order Markov process
$$\alpha_t=T_t\alpha_{t-1}+c_t+R_t\eta_t$$
where $T_t$ is an $m\times m$ matrix, $c_t$ is an $m\times 1$
vector, $R_t$ is an $m\times g$ matrix and $\eta_t$
is a $g\times 1$ vector of serially uncorrelated
random vectors with mean $0$ and covariance matrix $H_t$.
The specification of the state-space system is completed by
two further assumptions:
\begin{enumerate}
  \item The initial state vector $\alpha_0$ has a mean of $a_0$ and a
variance of~$P_0$.
  \item The random vectors $\epsilon_t$ and $\eta_t$ are uncorrelated
with each other, and uncorrelated with the initial state.
\end{enumerate}

In applications of the model, many of the
parameters $Z_t$, $d_t$, $H_t$, $T_t$, $c_t$, $R_t$, and $Q_t$
may be independent of $t$, in which case we will write them
without the subscript. Also, $R$ may be the identity matrix,
in which case we will omit it.

As a simple example of such a model, consider the (2-dimensional) random walk observed with error: %fixed.
 \begin{align}
  \nonumber \alpha_t =&\alpha_{t-1}+\eta_t\\
   y_t =&\alpha_t+\epsilon_t
 \end{align}
For this model, the following parameters are fixed
$$ T=\begin{pmatrix}1 & 0\\
                    0 & 1 \end{pmatrix}
  \quad
   Z=\begin{pmatrix}1 & 0\cr
                    0 & 1 \end{pmatrix}
  \quad
   R=\begin{pmatrix}1 & 0\cr
                    0 & 1\end{pmatrix}
  \quad
  d=(0,0)
  \quad
  c=(0,0)
$$
while the covariance matrices $Q$ and $H$ are estimated. The true values of $Q$ and $H$ used in the simulation were %fixed
$$
   Q=\begin{pmatrix}1 & 0.8\cr
                   0.8 & 1 \end{pmatrix}
  \quad
   H=\begin{pmatrix}3 & -2.5\cr
                   -2.5 & 3 \end{pmatrix}
$$
and the initial value of $a$ is $(0,0)$.%fixed

\section{Equations for the Kalman filter}

For a moment, go back to the general state-space model.
Given $a_0$ and $P_0$, we recursively calculate the a number of quantities
via the relationships
 \begin{align}
   \nonumber  \aone &= T_t\, a_{t-1}+c_t\\[3pt]
    \nonumber   \Pone &=T_t\, P_{t-1}\, T^\prime_t +R_t\, Q_t\, R^\prime_t\\[3pt]
   \nonumber    v_t &=y_t-Z_t\, \aone-d_t\\[3pt]
    \nonumber   F_t &=Z_t\, \Pone\, Z_t^\prime+H_t\\[3pt]
    \nonumber   a_t &=\aone-\Pone\, Z_t^\prime\, F_t^{-1}\, v_t\\[3pt]
      P_t &= \Pone-\Pone\, Z_t^\prime\, F_t^{-1}\, Z_t\, \Pone
 \end{align}
The log-likelihood function for the models parameters is
given by:
$$\log L = -\tfrac{NT}{ 2}\log 2\pi -0.5 \sum_{t=1}^T \log|F_t|
        -0.5 \sum_{t=1}^T v_t F_t^{-1} v_t
$$

The \textsc{tpl} file for the random walk Kalman filter code
follows:
\begin{lstlisting}
DATA_SECTION
  init_int nt
  init_int N
  init_int m
  int m1
  init_matrix Y(1,nt,1,N)
  matrix P0(1,m,1,m)
 !! P0.initialize();
 !! m1=m*(m+1)/2;
PARAMETER_SECTION
  init_bounded_vector Qcoff(1,m1,-10.,10.1)
  init_bounded_vector Hcoff(1,m1,-10.,10.1)
  init_vector a0(1,m)
  matrix T(1,m,1,m)
  matrix TT(1,m,1,m)
  vector d(1,N)
  vector c(1,m)
  matrix chQ(1,m,1,m)
  sdreport_matrix Q(1,m,1,m)
  matrix chH(1,N,1,N)
  sdreport_matrix H(1,N,1,N)
  matrix Z(1,N,1,m)
  matrix TZ(1,m,1,N)
  objective_function_value f
 LOCAL_CALCS
   d.initialize();
   c.initialize();
   Z.initialize();
   Z(1,1)=1; Z(2,2)=1;
   T.initialize();
   T(1,1)=1; T(2,2)=1;
   TZ=trans(Z);
   TT=trans(T);
PROCEDURE_SECTION
  setup_Q();
  setup_H();
  f+=kalman_filter();
  cout << " f = " << f << endl;

FUNCTION setup_Q
  chQ.initialize();
  int ii=1;
  for (int i=1;i<=m;i++)
    for (int j=1;j<=i;j++)
      chQ(i,j)=Qcoff(ii++);
  Q=chQ*trans(chQ);
FUNCTION setup_H
  chH.initialize();
  int ii=1;
  for (int i=1;i<=N;i++)
    for (int j=1;j<=i;j++)
      chH(i,j)=Hcoff(ii++);
  H=chH*trans(chH);

FUNCTION dvariable kalman_filter(void)
  dvar3_array P(0,nt,1,m,1,m);
  dvar3_array P1(1,nt,1,m,1,m);
  dvar3_array F(1,nt,1,N,1,N);
  dvar3_array Finv(1,nt,1,N,1,N);
  dvar_matrix Ptemp(1,m,1,m);
  dvar_matrix a(0,nt,1,m);
  dvar_matrix a1(1,nt,1,m);
  dvar_matrix v(1,nt,1,N);
  a(0)=a0;
  P(0)=P0;
  // This is the Kalman filter recursion. The objects tmp1
  // and tmp2 hold common calculations to optimize a bit
  int t;
  for (t=1;t<=nt;t++)
  {
    a1(t)=T*a(t-1)+c;
    P1(t)=T*P(t-1)*TT+Q;
    dvar_vector pred_y=Z*a1(t)+d;
    v(t)=Y(t)-pred_y;
    dvar_matrix tmp1=P1(t)*TZ;
    F(t)=Z*tmp1+H;
    Finv(t)=inv(F(t));
    dvar_matrix tmp2= tmp1*Finv(t);
    P(t)=P1(t)-tmp2*Z*P1(t);
    a(t)=a1(t)+tmp2*v(t);
  }
  int sgn=0;
  dvariable f=0.0;
  for (t=1;t<=nt;t++)
    f+=ln_det(F(t),sgn)+v(t)*Finv(t)*v(t);
  return f;
TOP_OF_MAIN_SECTION
  arrmblsize=20000000;
  gradient_structure::set_CMPDIF_BUFFER_SIZE(3000000);
  gradient_structure::set_GRADSTACK_BUFFER_SIZE(1000000);
\end{lstlisting}
This example was deliberately not optimized as much as it could be,
in order to retain the flavor of the more general state-space
problem. For example, since $T$ is the identity matrix and $c$ is the
zero vector, the line of code
\begin{lstlisting}
  a1(t)=T*a(t-1)+c;
\end{lstlisting}
reduces to
\begin{lstlisting}
  a1(t)=a(t-1);
\end{lstlisting}
The parameters being estimated are  $a_0$, $Q$, and~$H$.

To parameterize the covariance matrices, the Choleski decomposition
parameterization was used. This ensures that the covariance matrices are
positive (semi-) definite. The technique can be seen in the function
\texttt{setup\_Q}.
The lower triangular matrix \texttt{ch\_Q} is filled with parameters
from a bounded vector:
\begin{lstlisting}
FUNCTION setup_Q
  chQ.initialize();
  int ii=1;
  for (int i=1;i<=m;i++)
    for (int j=1;j<=i;j++)
      chQ(i,j)=Qcoff(ii++);
  Q=chQ*trans(chQ);   // chQ is the choleski decomposition of Q
\end{lstlisting}
Notice that the bounded vector \texttt{Qcoff} has slightly asymmetric
bounds. This is a simple way to ensure that its initial value is not
identically zero, which would lead to a singular covariance matrix.
\begin{lstlisting}
  init_bounded_vector Qcoff(1,m1,-10.,10.1)
\end{lstlisting}

The model parameters, standard deviations, and
correlations are reproduced from the
standard \textsc{admb} report.

\begin{smallcode}
 index   name    value      std.dev    7      8      9     10     11     12     13     14     15      16
    7   a0    -1.1682e+00 9.0191e-01  1.000
    8   a0     1.2218e+00 8.6442e-01  0.352  1.000
    9   Q      9.9468e-01 1.0862e-01  0.059 -0.006  1.000
   10   Q      7.8808e-01 7.8737e-02  0.038  0.028  0.683  1.000
   11   Q      7.8808e-01 7.8737e-02  0.038  0.028  0.683  1.000  1.000
   12   Q      8.7279e-01 9.6118e-02 -0.018  0.069  0.185  0.721  0.721  1.000
   13   H      3.1352e+00 1.8123e-01 -0.015 -0.007 -0.305 -0.136 -0.136 -0.018  1.000
   14   H     -2.7119e+00 1.4922e-01 -0.021  0.001 -0.102 -0.238 -0.238 -0.139 -0.692  1.000
   15   H     -2.7119e+00 1.4922e-01 -0.021  0.001 -0.102 -0.238 -0.238 -0.139 -0.692  1.000  1.000
   16   H      3.2264e+00 1.7936e-01  0.015 -0.029 -0.031 -0.121 -0.121 -0.249  0.370 -0.698 -0.698  1.000
\end{smallcode}

\section{Parameterizing the covariance\br matrix parameterizations}

The Choleski decomposition parameterization merely ensures that the
matrix is positive semi-definite. By adding a small positive number to the
diagonal elements, one can ensure that the covariance matrix is
positive definite, and can speed up and improve the stability of the
estimation. Of course, what is meant by ``small'' will depend on the
particular problem being considered.
A modified form of the routine \texttt{setup\_Q}
follows:
\begin{lstlisting}
FUNCTION setup_Q
  int i;
  chQ.initialize();
  int ii=1;
  for (i=1;i<=m;i++)
    for (int j=1;j<=i;j++)
      chQ(i,j)=Qcoff(ii++);
  Q=chQ*trans(chQ);   // chQ is the choleski decomposition of Q
  for (i=1;i<=m;i++)
    Q(i,i)+=0.1;   // make Q positive definite
\end{lstlisting}
Performing this modification for the present model for both $Q$ and
$H$ causes the program to converge about twice as fast.


\subsection{Time Series Decomposition}
\label{decomp}

Concerning the time table in Figure \ref{f:timetable}, a seasonal demand
    pattern is inherent to both horizontal and vertical time series.
First, the weekday influences if people eat out or order in with our partner
    receiving more orders on Thursday through Saturday than the other four
    days.
This pattern is part of both types of time series.
Second, on any given day, demand peaks occur around lunch and dinner times.
This only regards vertical series.
Statistical analyses show that horizontally sliced time series indeed exhibit
    a periodicity of $k=7$, and vertically sliced series only yield a seasonal
    component with a regular pattern if the periodicity is set to the product
    of the number of weekdays and the daily time steps indicating a distinct
    intra-day pattern per weekday.

Figure \ref{f:stl} shows three exemplary STL decompositions for a
    $1~\text{km}^2$ pixel and a vertical time series with 60-minute time steps
    (on the x-axis) covering four weeks:
With the noisy raw data $y_t$ on the left, the seasonal and trend components,
    $s_t$ and $t_t$, are depicted in light and dark gray for increasing $ns$
    parameters.
The plots include (seasonal) na\"{i}ve forecasts for the subsequent test day
    as dotted lines.
The remainder components $r_t$ are not shown for conciseness.
The periodicity is set to $k = 7 * 12 = 84$ as our industry partner has $12$
    opening hours per day.

\begin{center}
\captionof{figure}{STL decompositions for a medium-demand pixel with hourly
                   time steps and periodicity $k=84$}
\label{f:stl}
\includegraphics[width=.95\linewidth]{static/stl_gray.png}
\end{center}

As described in Sub-section \ref{stl}, with $k$ being implied by the
    application, at the very least, the length of the seasonal smoothing
    window, represented by the $ns$ parameter, must be calibrated by the
    forecaster:
It controls how many past observations go into each smoothened $s_t$.
Many practitioners, however, skip this step and set $ns$ to a big number, for
    example, $999$, then referred to as "periodic."
For the other parameters, it is common to use the default values as
    specified in \cite{cleveland1990}.
The goal is to find a decomposition with a regular pattern in $s_t$.
In Figure \ref{f:stl}, this is not true for $ns=7$ where, for
    example, the four largest bars corresponding to the same time of day a
    week apart cannot be connected by an approximately straight line.
On the contrary, a regular pattern in the most extreme way exists for
    $ns=999$, where the same four largest bars are of the same height.
This observation holds for each time step of the day.
For $ns=11$, $s_t$ exhibits a regular pattern whose bars adapt over time:
The pattern is regular as bars corresponding to the same time of day can be
    connected by approximately straight lines, and it is adaptive as these
    lines are not horizontal.
The trade-off between small and large values for $ns$ can thus be interpreted
    as allowing the average demand during peak times to change over time:
If demand is intermittent at non-peak times, it is reasonable to expect the
    bars to change over time as only the relative differences between peak and
    non-peak times impact the bars' heights with the seasonal component being
    centered around $0$.
To confirm the goodness of a decomposition statistically, one way is to verify
    that $r_t$ can be modeled as a typical error process like white noise
    $\epsilon_t$.

However, we suggest an alternative way of calibrating the STL method in an
    automated fashion based on our unified CV approach.
As hinted at in Figure \ref{f:stl}, we interpret an STL decomposition as a
    forecasting method on its own by just adding the (seasonal) na\"{i}ve
    forecasts for $s_t$ and $t_t$ and predicting $0$ for $r_t$.
Then, the $ns$ parameter is tuned just like a parameter for an ML model.
To the best of our knowledge, this has not yet been proposed before.
Conceptually, forecasting with the STL method can be viewed as a na\"{i}ve
    method with built-in smoothing, and it outperformed all other
    benchmark methods in all cases.