%{
\begin{octave}
%}
sheffieldmlToolboxes;
%{
\end{octave}

\title{SheffieldML - MATLAB Software}

\begin{document}
\section{Examples}

\section{Gaussian Process Models}

\subsection{Functions from Gaussians}

This example shows how points which look like they come from a
function to be sampled from a Gaussian distribution. The sample is 25
dimensional and is from a Gaussian with a particular covariance.

\begin{octave}
%}
demGpSample
%{
\end{octave}

\begin{center}
\inputdiagram{gpSample}\hfill\inputdiagram{gpCovariance}

\emph{Left} A single, 25 dimensional, sample from a Gaussian
distribution. \emph{Right} the covariance matrix of the Gaussian
distribution.

\end{center}

\subsection{Joint Distribution over two Variables}

Gaussian processes are about conditioning a Gaussian distribution
on the training data to make the test predictions. To illustrate this
process, we can look at the joint distribution over two variables.

\begin{octave}
%}
 demGpCov2D([1 2])
%{
\end{octave}

Gives the joint distribution for $f_1$ and
$f_2$. The plots show the joint distributions as well
as the conditional for $f_2$ given
$f_1$.

\begin{center}
\inputdiagram{demGpCov2D1_2_3} 0.5\textwidth \inputdiagram{demGpCov2D1_5_3} 0.5\textwidth

\emph{Left} Blue line is
contour of joint distribution over the variables $f_1$
and $f_2$. Green line indicates an observation of
$f_1$. Red line is conditional distribution of
$f_2$ given $f_1$. \emph{Right} Similar
for $f_1$ and $f_5$.
\end{center}

\subsection{Different Samples from Gaussian Processes}

A script is provided which samples from a Gaussian process with the
provided covariance function.

\begin{octave}
%}
 gpSample('rbf', 10, [1 1], [-3 3], 1e5)
%{
\end{octave}

will give 10 samples from an RBF covariance function with a
parameter vector given by [1 1] (inverse width 1, variance 1) across
the range -3 to 3 on the $x$-axis. The random seed will be set to
1e5.

\begin{octave}
%}
 gpSample('rbf', 10, [16 1], [-3 3], 1e5)
%{
\end{octave}

is similar, but the inverse width is now set to 16 (length scale 0.25).

\begin{center}\inputdiagram{gpSampleRbfSamples10Seed100000InverseWidth1Variance1} 0.5\textwidth \inputdiagram{gpSampleRbfSamples10Seed100000InverseWidth16Variance1} 0.5\textwidth \emph{Left} samples from an RBF style covariance function
with length scale 1. \emph{Right} samples from an RBF style covariance
function with length scale 0.25.
\end{center}

Other covariance functions can be sampled, an interesting one is
the MLP covariance which is non stationary and can produce point
symmetric functions,

\begin{octave}
%}
 gpSample('mlp', 10, [100 100 1], [-1 1], 1e5)
%{
\end{octave}

gives 10 samples from the MLP covariance function where the `bias
variance' is 100 (basis functions are centered around the origin
with standard deviation of 10) and the `weight variance' is
100.

\begin{octave}
%}
 gpSample('mlp', 10, [100 1e-16 1], [-1 1], 1e5)
%{
\end{octave}

gives 10 samples from the MLP covariance function where the `bias
variance' is approximately zero (basis functions are placed on
the origin) and the `weight variance' is 100.

\begin{center}\inputdiagram{gpSampleMlpSamples10Seed100000WeightVariance100BiasVariance100Variance1} 0.5\textwidth \inputdiagram{gpSampleMlpSamples10Seed100000WeightVariance100BiasVariance1e-16Variance1}
0.5\textwidth \emph{Left} samples from an MLP style covariance
function with bias and weight variances set to 100. \emph{Right}
samples from an MLP style covariance function with weight variance 100
and bias variance approximately zero.
\end{center}

\subsection{Posterior Samples}

Gaussian processes are non-parametric models. They are specified by their covariance function and a mean function. When combined with data observations a posterior Gaussian process is induced. The demos below show samples from that posterior.

\begin{octave}
%}
  gpPosteriorSample('rbf', 5, [1 1], [-3 3], 1e5)
%{
\end{octave}

and

\begin{octave}
%}
  gpPosteriorSample('rbf', 5, [16 1], [-3 3], 1e5)
%{
\end{octave}

\begin{center}\inputdiagram{gpPosteriorSampleRbfSamples5Seed100000InverseWidth1Variance1bw} 0.5\textwidth \inputdiagram{gpPosteriorSampleRbfSamples5Seed100000InverseWidth16Variance1bw} 0.5\textwidth

\emph{Left} samples from the posterior induced by an RBF style covariance function
with length scale 1 and 5 `training' data points taken from a sine wave. \emph{Right} Similar but for a length scale of 0.25.
\end{center}

\subsection{Simple Interpolation Demo}

This simple demonstration plots, consecutively, an increasing
number of data points, followed by an interpolated fit through the
data points using a Gaussian process. This is a noiseless system, and
the data is sampled from a GP with a known covariance function. The
curve is then recovered with minimal uncertainty after only nine data
points are included. The code is run with

\begin{octave}
%}
 demInterpolationGp
%{
\end{octave}

\begin{center}

\inputdiagram{demInterpolation3} 0.5\textwidth \inputdiagram{demInterpolation4} 0.5\textwidth

Gaussian process prediction \emph{left} after two points with a new
data point sampled \emph{right} after the new data point is included
in the prediction.

\inputdiagram{demInterpolation7} 0.5\textwidth \inputdiagram{demInterpolation8} 0.5\textwidth

Gaussian process prediction \emph{left} after five points with a four
new data point sampled \emph{right} after all nine data points are
included.
\end{center}

\subsection{Simple Regression Demo}

The regression demo very much follows the format of the
interpolation demo. Here the difference is that the data is sampled
with noise. Fitting a model with noise means that the regression will
not necessarily pass right through each data point.

The code is run with

\begin{octave}
%}
 demRegressionGp
%{
\end{octave}

\begin{center}
\inputdiagram{demRegression3} 0.5\textwidth \inputdiagram{demRegression4} 0.5\textwidth

Gaussian process prediction \emph{left} after two points with a new
data point sampled \emph{right} after the new data point is included
in the prediction.

\inputdiagram{demRegression7} 0.5\textwidth \inputdiagram{demRegression8} width 0.5\textwidth

Gaussian process prediction \emph{left} after five points with a four
new data point sampled \emph{right} after all nine data points are
included.
\end{center}

\subsection{Optimizing Hyper-parameters}

One of the advantages of Gaussian processes over pure kernel
interpretations of regression is the ability to select the hyper
parameters of the kernel automatically. The demo

\begin{octave}
%}
 demOptimiseGp
%{
\end{octave}

shows a series of plots of a Gaussian process with different length
scales fitted to six data points. For each plot there is a
corresponding plot of the log likelihood. The log likelihood peaks for
a length scale equal to 1. This was the length scale used to generate
the data.

\begin{center}\inputdiagram{demOptimiseGp1} 0.33\textwidth \inputdiagram{demOptimiseGp3} 0.33\textwidth \inputdiagram{demOptimiseGp5}
0.33\textwidth \inputdiagram{demOptimiseGp7} 0.33\textwidth \inputdiagram{demOptimiseGp9} 0.33\textwidth \inputdiagram{demOptimiseGp11} 0.33\textwidth \inputdiagram{demOptimiseGp13} 0.33\textwidth \inputdiagram{demOptimiseGp15} 0.33\textwidth \inputdiagram{demOptimiseGp17}
0.33\textwidth

From top left to bottom right, Gaussian process
regression applied to the data with an increasing length scale. The
length scales used were 0.05, 0.1, 0.25, 0.5, 1, 2, 4, 8 and
16.

\inputdiagram{demOptimiseGp18} 0.5\textwidth

Log-log plot of
the log likelihood of the data against the length scales. The log
likelihood is shown as a solid line. The log likelihood is made up of
a data fit term (the quadratic form) shown by a dashed line and a
complexity term (the log determinant) shown by a dotted line. The data
fit is larger for short length scales, the complexity is larger for
long length scales. The combination leads to a maximum around the true
length scale value of 1.

\end{center}

\subsection{Regression over Motion Capture Markers}

As a simple example of regression for real data we consider a motion capture data set. The data is from Ohio State University. In the example script we perform Gaussian process regression with time as the input and the x,y,z position of the marker attached to the left ankle. To demonstrate the behavior of the model when the marker is lost, we remove data from This code can be run with

\begin{verbatim}
demStickGp1
\end{verbatim}

\begin{octave}
%}
load('demStickGp1');
% Plot results
fillColor = [0.7 0.7 0.7];
tTest = linspace(0, 2, 200)';
[mu, varSigma] = gpPosteriorMeanVar(model, tTest);

for i = 1:length(outputIndex);
  figure
  fill([tTest; tTest(end:-1:1)], ...
       [mu(:, i); mu(end:-1:1, i)] ...
       + 2*[sqrt(varSigma(:, i)); -sqrt(varSigma(end:-1:1, i))], ...
       fillColor,'EdgeColor',fillColor)
  hold on;
  plot(t(testIndex), y(testIndex, outputIndex(i)), 'ko');
  b=plot(tTrain, yTrain(:, i), 'k.');
  a = plot(tTest, mu(:, i), 'k-');
  set(gca, 'xlim', [0 2])
  set(a, 'linewidth', 2);
  fileName = ['dem' capName 'Gp' num2str(experimentNo) 'Out' num2str(i)];
  printLatexPlot(fileName, '../tex/diagrams', 0.3*textWidth, options);
end
%{
\end{octave}

The code will optimize hyper parameters and show plots of the posterior process through the training data and the missing test points.

The result of the script is given in the plot below.

\begin{center}
\inputdiagram{demStickGp1Out1} 0.3\textwidth \inputdiagram{demStickGp1Out2} 0.3\textwidth"> <\inputdiagram{demStickGp1Out3} width
="0.3\textwidth"> Gaussian process regression through the x (left), y (middle) and z (right) position of the left ankle. Training data is shown as black spots, test points removed to simulate a lost marker are shown as circles, posterior mean
prediction is shown as a black line and two standard deviations are
given as grey shading.\end{center}

Notice how the error bars are tight except in the region where the training data is missing and in the region where the training data disappears.

\subsection{Sparse Pseudo-input Gaussian Processes}

The sparse approximation used in this toolbox is based on the
Sparse Pseudo-input Gaussian Process model described by Snelson
and Ghahramani. Also provided are the extensions suggested by Quiñonero-Candela
and Rasmussen. They provide a unifying terminology for describing
these approximations which we shall use in what follows.

There are three demos provided for Gaussian process regression in
1-D. They each use a different form of likelihood approximation. The
first demonstration uses the `projected latent variable'
approach first described by Csato
and Opper and later used by Seeger
et al.. In the terminology of Quiñonero-Candela and
Rasmussen (QR-terminology) this is known as the `deterministic
training conditional' (DTC) approximation.

To use this approximation the following script can be run.

\begin{octave}
%}
demSpgp1dGp1
%{
\end{octave}

The result of the script is
given in the plot below.

\begin{center}<\inputdiagram{demSpgp1dGp1} width
="0.5\textwidth"> Gaussian process using the DTC approximation with nine
inducing variables. Data is shown as black spots, posterior mean
prediction is shown as a black line and two standard deviations are
given as grey shading.\end{center}

The improved approximation suggested by Snelson and Ghahramani, in
QR-terminology this is known as the fully independent training
conditional (FITC). To try this approximation run the following script

\begin{octave}
%}
demSpgp1dGp2
%{
\end{octave}

The result of the script is given on the left of the plot below.

\begin{center}\inputdiagram{demSpgp1dGp2} width="0.49\textwidth">

\emph{Left}: Gaussian process using the FITC approximation with nine
inducing variables. Data is shown as black spots, posterior mean
prediction is shown as a black line and two standard deviations are
given as grey shading. \emph{Right}: Similar but for the PITC
approximation, again with nine inducing variables.\end{center}

At the Sheffield
Gaussian Process Round Table Lehel Csato pointed out that the
Bayesian Committee Machine of Schwaighofer
and Tresp can also be viewed within the same framework. This idea
is formalised in Quiñonero-Candela
and Rasmussen's review. This approximation is known as the
`partially independent training conditional' (PITC) in
QR-terminology. To try this approximation run the following script

\begin{octave}
%}
 demSpgp1dGp3
%{
\end{octave}

The result of the script is given on the right of the plot above.

Finally we can compare these results to the result from the full
Gaussian process on the data with the correct hyper-parameters. To do
this the following script can be run.

\begin{octave}
%}
 demSpgp1dGp4
%{
\end{octave}

The result of the script is given in the plot below.

\begin{center}\inputdiagram{demSpgp1dGp4} width="0.5\textwidth"> Full Gaussian
process on the toy data with the correct hyper-parameters. Data is
shown as black spots, posterior mean prediction is shown as a black
line and two standard deviations are given as grey shaded
area.\end{center}

\section{Gaussian Process Latent Variable Model Examples}

The three approximations outlined above can be used to speed up learning in the GP-LVM. They have the advantage over the IVM approach taken in the original GP-LVM toolbox that the algorithm is fully convergent and the final mapping from latent space to data space takes into account all of the data (not just the points in the active set).

As well as the new sparse approximation the new toolbox allows the GP-LVM to be run with dynamics as suggested by Wang et al..

Finally, the new toolbox allows the incorporation of `back constraints' in learning. Back constraints force the latent points to be a smooth function of the data points. This means that points that are close in data space are constrained to be close in latent space. For the standard GP-LVM points close in latent space are constrained to be close in data space, but the converse is not true.

Various combinations of back constraints and different approximations are used in the exmaples below.

\subsection{Oil Data}

The `oil data' is commonly used as a bench mark for visualisation algorithms. For more details on the data see this page.

The C++ implementation of the GP-LVM has details on training the full GP-LVM with this data set. Here we will consider the three different approximations outlined above.

\subsubsection{FITC Approximation}

In all the examples we give there will be 100 points in the active set. We first considered the FITC approximation. The script \texttt{demOilFgplvm1.m} runs the FITC approximation giving the result on the left of the figure shown below.

\begin{center}\inputdiagram{demOilFgplvm1} width="0.49\textwidth">\inputdiagram{demOilFgplvm2} width="49%">
\emph{Left}: GP-LVM on the oil data using the FITC approximation without back constraints. The phases of flow are shown as green circles, red crosses and blue plusses.  One hundred inducing variables are used. \emph{Right}: Similar but for a back-constrained GP-LVM, the back constraint is provided by a multi-layer perceptron with 15 hidden nodes.\end{center}

Back constraints can be added to each of these approximations. In the example on the right we used a back constraint given by a multi-layer perceptron with 15 hidden nodes. This example can be recreated with \texttt{demOilFgplvm2.m}.

\subsubsection{DTC Approximation}

The other approximations can also be used, in the figures below we give results from the DTC approximation. The can be recreated using \texttt{demOil3.m} and \texttt{demOil4.m}.

\begin{center}\inputdiagram{demOilFgplvm3} width="49%">\inputdiagram{demOilFgplvm4} width="49%">
\emph{Left}: GP-LVM on the oil data using the DTC approximation without back constraints. The phases of flow are shown as green circles, red crosses and blue plusses.  One hundred inducing variables are used. \emph{Right}: Similar but for a back-constrained GP-LVM, the back constraint is provided by a multi-layer perceptron with 15 hidden nodes.\end{center}

\subsubsection{PITC Approximation}

We also show results using the PITC approximation, these results can be recreated using the scripts \texttt{demOilFgplvm5.m} and \texttt{demOilFgplvm6.m}.

\begin{center}\inputdiagram{demOilFgplvm5} width="49%">\inputdiagram{demOilFgplvm6} width="49%">
\emph{Left}: GP-LVM on the oil data using the PITC approximation without back constraints. The phases of flow are shown as green circles, red crosses and blue plusses.  One hundred inducing variables are used. \emph{Right}: Similar but for a back-constrained GP-LVM, the back constraint is provided by a multi-layer perceptron with 15 hidden nodes.\end{center}

\subsubsection{Variational DTC Approximation}

Finally we also show results using the variational DTC approximation of Titsias, these results can be recreated using the scripts \texttt{demOilFgplvm7.m} and \texttt{demOilFgplvm8.m}.

\begin{center}\inputdiagram{demOilFgplvm7} width="49%">\inputdiagram{demOilFgplvm8} width="49%">
\emph{Left}: GP-LVM on the oil data using the variational DTC approximation without back constraints. The phases of flow are shown as green circles, red crosses and blue plusses.  One hundred inducing variables are used. \emph{Right}: Similar but for a back-constrained GP-LVM, the back constraint is provided by a multi-layer perceptron with 15 hidden nodes.\end{center}

\subsection{Back Constraints and Dynamics}

First we will demonstrate the dynamics functionality of the toolbox. We raw x-y-z values from a motion capture data set, the \texttt{Figure Run 1} example available from Ohio State University. To run without dynamics use the script:
\texttt{
 demStickFgplvm1
\end{octave}

The results are given on the left of the figure below.

\begin{center}\inputdiagram{demStickFgplvm1} width="49%">
GP-LVM on the motion capture data without dynamics in the latent space. \end{center}

Notice that the sequence (which is a few strides of a man running) is split into several sub-sequences. These sub-sequences are aligned to the strides of the man. By introducing a dynamics prior, we can force the sequence to link up. Samples from the dynamics prior used are shown in the plot below.

\begin{center}\inputdiagram{dynamicsSamp1} width="49%">\inputdiagram{dynamicsSamp2} width="49%">
\inputdiagram{dynamicsSamp3} width="49%">\inputdiagram{dynamicsSamp4} width="49%">
Samples from the dynamics prior which is placed over the latent space. This prior has \emph{Left}: GP-LVM on the motion capture data without dynamics in the latent space. \emph{Right}: GP-LVM with dynamics. Samples from the dynamics prior used are given in the figure above.\end{center}

This prior is used in the model to obtain the results below,

\begin{octave}
 demStickFgplvm2
\end{octave}

\begin{center}\inputdiagram{demStickFgplvm2} width="49%">
GP-LVM with dynamics. Samples from the dynamics prior used are given in the figure above.\end{center}

Note now the circular form of the latent space.

Back constraints can also be used to achieve a similar effect,

\begin{octave}
 demStickFgplvm3
\end{octave}

\begin{center}\inputdiagram{demStickFgplvm3} width="49%">
GP-LVM with back constraints. A RBF kernel mapping was used to form the back constraints with the inverse width set to 1e-4 (\emph{i.e.}length scale set to 100).\end{center}

\subsection{Loop Closure in Robotics}

In on-going work with Dieter Fox and Brian Ferris at the University of Washington we are interested in loop closure for robotic navigation, included as an example is a data set of a robot completing a loop while reading wireless access point signal strengths. To produce a neat track and close the loop it turns out it is necessary to use dynamics and back constraints as seen in the images below. These results can be recreated with \texttt{demRobotWireless1.m} through \texttt{demRobotWireless4.m}.
}

\begin{center}\inputdiagram{demRobotWireless1} width="49%">\inputdiagram{demRobotWireless2} width="49%">
\inputdiagram{demRobotWireless3} width="49%">\inputdiagram{demRobotWireless4} width="49%">
Use of back constraints and dynamics to obtain loop closure in a robot navigation example. \emph{Top Left}: GP-LVM without back constraints or dynamics, \emph{Top right}: GP-LVM with back constraints, no dynamics, \emph{Bottom Left}: GP-LVM with dynamics, no back constraints, \emph{Bottom right}: GP-LVM with back constraints and dynamics. \end{center}

\subsection{Vocal Joystick and Vowel Data}

Another ongoing piece of work with Jeff Bilmes and Jon Malkin involves embedding vowel sounds in a two dimensional space as part of vocal joystick system. Jon has provided a simple data set of 2,700 examples of different vowels. These are embedded in a two dimensional latent space with and without back constraints.

\begin{center}\inputdiagram{demVowels2} width="49%">\inputdiagram{demVowels3} width="49%">
\emph{Left}: embedding of the vowel data without back constraints, \emph{Right}: embedding of the vowel data with back constraints. $/a/$ - red cross, $/ae/$ - green circle, $/ao/$ - blue plus, $/e/$ - cyan asterix, $/i/$ - magenta square, $/ibar/$ - yellow diamond, $/o/$ - red down triangle, $/schwa/$ - green up triangle, $/u/$ - blue left triangle.
\end{center}

\subsection{Larger Human Motion Data Sets}

For an AISTATS paper we recreated an experiment from Taylor et al.'s NIPS paper. They created a data set from a motion capture data in the CMU data base of running and walking. The data set can now be recreated using the DATASETS toolbox. We repeated missing data experiments by Taylor et al.. The model learning for these experiments can be recreated with:

\begin{octave}
 demCmu35gplvm1
\end{octave}

for the four dimensional latent space, \texttt{demCmu35gplvm2} for the three dimensional latent space and \texttt{demCmu35gplvm3} for the five dimensional latent space. The test data reconstruction can then be performed for all models with \texttt{demCmu35gplvmReconstruct}. Taylor et al.'s nearest neighbour results can be recreated using \texttt{demCmu35TaylorNearestNeighbour}.

Data was pre-processed by mapping angles to be between -180 and 180 and scaling the data such that the variance of each dimension was one.
The quality of the trained model was evaluated using a missing data problem with a test sequence of data. The model was required to fill in either upper body angles or right leg angles. Results for the GP-LVM and nearest neighbour in both scaled space and original angle space are given in the table below.
\begin{center}

	Leg	Leg	Body	Body

	Cumulative	RMS	Cumulative	RMS

	Scaled	Angles	Scaled	Angles

GP-LVM ($q$=3)	11.4	3.40	16.9	2.49

GP-LVM ($q$=4)	9.7	3.38	20.7	2.72

GP-LVM ($q$=5)	13.4	4.25	23.42.78

Scaled NN	13.5	4.44	20.82.62

Nearest Neighbour	14.0	4.11	30.93.20

\end{center}
The cumulative scaled error is a recreation of the error reported in Taylor et al. which was the average (across angles) cumulative sum (across time) of the squared errors in the down-scaled (\emph{i.e.} variance one) space of angles. We also present the root mean squared angle error for each joint which we find to be a little easier to interpret.

Taylor et al. used a slightly different representation of
the data set which included the absolute $x$ and $z$
position of the root node and rotation around the $y$-axis. For
this data set, this information does help, principally because the
subject seems to start in roughly the same position at the beginning
of each sequence. However, in general absolute position will not help,
so we discarded it in favour of a representation of these values in
terms of differences between frames. Finally Taylor et al.
concatenated two frames to form each data point for the model. We
chose not to do this as we wanted to test the ability of the Gaussian
process dynamics to fully recreate the data set. There results are given in their paper and summarised below.

\begin{center}

	Leg	Body

	Cumulative	Cumulative

	Scaled	Scaled

Binary Latent Variable Model	11.7	8.8

Scaled NN	22.2	20.5

\end{center}
Finally we show a plot of reconstructions of two of the angles in the data.

\begin{center}\inputdiagram{demCmu35gplvmLegReconstruct1_8} width="0.5\textwidth">\inputdiagram{demCmu35gplvmLegReconstruct1_9} width="0.5\textwidth">\end{center}
Prediction for first two angles of the right hip joint (see plots in Taylor et al. for comparison). Dotted line is nearest neighour in scaled space, dashed line is GP-LVM with 4-D latent space.\end{center}

\section{Informative Vector Machines}

\section{\texttt{demClassificationOneIvm1}}

The first example given is \texttt{demClassificationOneIvm1} which is a simple classification data set, where only one direction of the input is relevant in determining the decision boundary. An ARD MLP kernel is used in combination with a linear kernel. The ARD parameters in the linear and MLP kernel are constrained to be the same by the line:

\begin{octave}

% Constrain the ARD parameters in the MLP and linear kernels to be the same.

model.kern = cmpndTieParameters(model.kern, {[4, 7], [5, 8]});

\end{octave}

The resulting classification is shown below.

\begin{center}\inputdiagram{demClassificationOneIvm1}

Decision boundary from the \texttt{demClassificationOneIvm1.m} example. Postive class is red circles, negative class green crosses and active points are yellow dots. Decision boundary shown in red, contours at 0.25 and 0.75 probability shown in blue.\end{center}

\section{\texttt{demClassificationTwoIvm1}}

The second example attempts to learn a Gaussian process give data that is sampled from a Gaussian process. The code is \texttt{demClassificationTwoIvm1}. The underlying Gaussian process is based on an RBF kernel with variance inverse width 10. The IVM learns an inverse width of 15 and gives the classification is shown below.

\begin{center}\inputdiagram{demClassificationTwoIvm1}

Decision boundary from the \texttt{demClassificationTwoIvm1.m} example. Postive class is red circles, negative class green crosses and active points are yellow dots. Decision boundary shown in red, contours at 0.25 and 0.75 probability shown in blue.\end{center}

\section{\texttt{demOrderedOneIvm1}}

In this example the ordered categorical noise model is used (ordinal regression). The data is a simple data set for which a linear one dimensional model suffices. The IVM is given a combination of an RBF and linear kernel with ARD.For the ordered categorical case there are several parameters associated with the noise model (in particular the category widths), these are learnt too. The model learns that the system is linear and only one direction is important. The resulting classification is given below.

\begin{center}\inputdiagram{demOrderedOneIvm1}

Decision boundary from the \texttt{demOrderedOneIvm1.m} example. Class 0 - red cross, Class 1 - green circles, Class 2 - blue crosses, Class 3 - cyan asterisks, Class 4 - pink squares, Class 5 - yellow diamonds. Class 6 - red triangles. Active points are yellow dots, note that because the kernel is linear by now the most informative points tend to be at the extrema. Decision boundaries shown in red, contours at 0.25 and 0.75 probability shown in blue.\end{center}

\section{\texttt{demOrderedTwoIvm1}}

Another example with the ordered categorical noise model, here the data is radial, the categories being along the radius of a circle. The IVM is given a combination of an RBF and linear kernel with ARD. Again there are several parameters associated with the noise model, and these are learnt using \texttt{ivmOptimiseNoise}. The resulting classification is given below.

\begin{center}\inputdiagram{demOrderedTwoIvm1}

Decision boundary from the \texttt{demOrderedTwoIvm1.m} example. Class 0 - red cross, Class 1 - green circles, Class 2 - blue crosses, Class 3 - cyan asterisks, Class 4 - pink squares, Class 5 - yellow diamonds. Class 6 - red triangles. Active points are yellow dots, note that because the kernel is linear by now the most informative points tend to be at the extrema. Decision boundaries shown in red, contours at 0.25 and 0.75 probability shown in blue.\end{center}

\section{\texttt{demRegressionOneIvm1}}

In this example the Gaussian noise model is used (standard regression). The data is sampled from a Gaussian process, only one input dimension is important. The IVM is given a combination of an RBF and linear kernel with ARD. The resulting regression is given below.

\begin{center}\inputdiagram{demRegressionOneIvm1}

Regression from the example \texttt{demRegressionOneIvm1.m}. Targets are red dots and active points are yellow dots.\end{center}

\section{\texttt{demRegressionTwoIvm1}}

A second example with Gaussian noise, sampled from a Gaussian process, but this time with differing length scales.

\begin{center}\inputdiagram{demRegressionTwoIvm1}

Regression from the example \texttt{demRegressionTwoIvm1.m}. Targets are red dots and active points are yellow dots.\end{center}

\section{Benchmark Data Sets}

The function \texttt{ivmGunnarData} allows you to test the IVM on Gunnar Raetsch's benchmark data sets. Download the data sets, from here and expand the ringnorm data set into '\$DATASETSDIRECTORY/gunnar/ringnorm'. Then run the following script.

\begin{octave}
%}
ivmGunnarData('ringnorm', 1, {'rbf', 'bias', 'white'}, 1, 100)
%{
\end{octave}
\begin{verbatim}

Final model:

IVM Model:

 Noise Model:

  Probit bias on process 1: 0.0439

  Probit Sigma2: 0.0000

 Kernel:

  Compound kernel:

    RBF inverse width: 0.0866 (length scale 3.3984)

    RBF variance: 1.2350

    Bias Variance: 8.2589

    White Noise Variance: 0.0000

Test Error 0.0183

Model likelihood -56.7120

\end{verbatim}

You can try any of the data sets by replacing ringnorm with the relevant data set (note that they don't all work with only 100 active points inas in the example above, for example the 'banana' data set needs 200 active points to get a reasonable result,

\begin{octave}

 ivmGunnarData('banana', 1, {'rbf', 'bias', 'white'}, 1, 200)
\end{octave}

\begin{verbatim}
Final model:

IVM Model:

 Noise Model:

  Probit bias on process 1: 0.1067

  Probit Sigma2: 0.0000

 Kernel:

  Compound kernel:

    RBF inverse width: 1.6411 (length scale 0.7806)

    RBF variance: 0.2438

    Bias Variance: 0.0000

    White Noise Variance: 0.0148

Test Error 0.1129

Model likelihood 175.3588

\end{verbatim}

\begin{center}
\inputdiagram{demBanana1}

Decision boundary from the banana example. Postive class is red circles, negative class green crosses and active points are yellow dots. Decision boundary shown in red, contours at 0.25 and 0.75 probability shown in blue.
\end{center}

\section{Null Category Noise Model}

\section{Examples}

The toy data example in the papers can be recreated using:

\begin{octave}

 demUnlabelledOneIvm1

\end{octave}

and leads to the decision boundary given below. A standard IVM based classifier can be run on the data using

\begin{octave}

 demUnlabelledOneIvm2

\end{octave}

\begin{center}\inputdiagram{demUnlabelledOneIvm1}\inputdiagram{demUnlabelledOneIvm2}

The null category noise model run on toy data. \emph{Top}: using the null category, the true nature of the decision boundary is recovered. \emph{Bottom}: the standard IVM, does not recover the true decision boundary.\end{center}

The other USPS digit classification example given in the NIPS paper can be re-run with:

\begin{octave}

 demThreeFiveIvm

\end{octave}

Be aware that this code can take some time to run. The results, in the form of averaged area under ROC curve against probability of missing label, can be plotted using

\begin{octave}
%}
 demThreeFiveResults
%{
\end{octave}

\begin{center}

\inputdiagram{demThreeFive}

Plot of average area under ROC curve against probability of label being present. The red line is the standard IVM based classifier, the blue dotted line is the null category noise model based classifier, the green dash-dot line is the a normal SVM and the mauve dashed line is the transductive SVM.

\end{center}

\end{document}
%}


\subsection{Monte-Carlo Tree Search}

\subsubsection{Introduction}

In search tree, each node has wins/total.

So start with just root in 0/0.

Algo:

\begin{itemize}
\item Start at root
\item Take n choices to arrive at node which has not been explored (or until w/l state)
\item play randomly from there
\item Back prob up (eg if win, then 1/1 for path back to root, or 0/1 if loss)
\end{itemize}

Wins/simulation count

So deterministic when choosing paths, up until play randomly.


%!TEX root = ../notes.tex
\section{February 8, 2022}

\subsection{Dirichlet Convolutions}

\begin{definition}[Dirichlet Convolution]
    Let $f, g$ be arithmetic functions. Then the \ul{Dirichlet convolution/product} of $g$ and $g$ it
    \begin{align*}
        (f * g)(n) : & = \sum_{d_1d_2=n}f(d_1)g(d_2) \\
                     & = \sum_{d\mid n}f(d) g(n/d)
    \end{align*}
\end{definition}

We do check that this has properties that we want it to have, like associativity:
\begin{align*}
    ((f * g) * h)(n) & = (f * (g * h))(n)                      \\
                     & = \sum_{d_1d_2d_3=n} f(d_1)g(d_2)h(d_3)
\end{align*}
It is also clearly commutative.

We also have that this product has a multiplicative identity.

\begin{definition}
    Let $I : \ZZ_+\to \{0, 1\}$ be given by
    \[I(n) = \begin{cases}
            1 & \text{if } n = 1 \\
            0 & \text{otherwise}
        \end{cases}\]
    Then $I$ is an identity for $*$, in the sense that $f * I = I * f = f$.
\end{definition}

\begin{lemma}
    If $f$ is an arithmetic function such that $f(1)\neq 0$, then there exists an arithmetic function $g$ such that $f * g = I$.

    It is given recursively by
    \begin{align*}
        g(1) & = \frac{1}{f(1)}                                         \\
        g(n) & = -\frac{1}{f(1)}\cdot \sum_{d\mid n, d < n} g(d) f(n/d)
    \end{align*}
\end{lemma}
\begin{proof}
    We want to show that given $g$ and $g$ defined as above, we have that $f * g = I$.

    $n=1$: \[g(1)\cdot f(1) = \frac{1}{f(1)}f(1) = 1\]

    $n > 1$: \begin{align*}
        \sum_{d\mid n}g(d) f(n/d) & = g(n)\cdot f(1) + \sum_{d\mid n, n<n}g(d) f(n/d)                   \\
                                  & = -\frac{1}{f(1)}\cdot \sum_{d\mid n, d < n} g(d) f(n/d) \cdot f(1)
        + \sum_{d\mid n, n<n}g(d) f(n/d)                                                                \\
                                  & = 0
    \end{align*}
    So $g$ is indeed an inverse of $f$ since they produce the identity function $I$.
\end{proof}

\subsection{M\"obius Inversion}
The motivation of this is: given a summatory function of multiplicative functions, can we recover the multiplicative function?
\begin{definition}[M\"obius $\mu$ Function]
    We define $\mu : \ZZ_+ \to \{-1, 0, 1\}$ given by
    \[\mu(n) = \begin{cases}
            (-1)^k & \text{if $n = p_1p_2\dots p_k$ if $p_i$ are pairwise distinct primes} \\
            0      & \text{otherwise}
        \end{cases}\]
    \emph{(We note that $\mu(1) = 1$.)}
\end{definition}

\begin{lemma}
    $\mu$ is a multiplicative function.
\end{lemma}
\begin{proof}
    Let $m, n\in\ZZ_+$ such that $(m, n) = 1$. We write
    \begin{align*}
        m & = p_1^{e_1}p_2^{e_2}\cdots p_k^{e_k} \\
        n & = q_1^{f_1}q_2^{f_2}\cdots q_l^{f_l} \\
    \end{align*}
    \begin{description}
        \item[Case 1]
            Some exponent $e_i$ or $f_i \geq 2$. Then we have that \[\mu(mn) = \mu(m)\mu(n) = 0\]
        \item[Case 2]
            We have that \begin{align*}
                m & =p_1p_2\cdots p_k \\
                n & =q_1q_2\cdots q_l
            \end{align*}
            where $p_i$ and $q_i$ are all pairwise distinct. Then $\mu(m) = (-1)^k$ and $\mu(n) = (-1)^l$, so $\mu(m) = \mu(n) = (-1)^{k+l}$.

            Since these are coprime $(m, n) = 1$, then we have that $\mu(mn) = (-1)^{k+l}$.
    \end{description}
    Which is as intended, giving that $\mu$ is a multiplicative function.
\end{proof}

\begin{lemma}
    We have the property:
    \begin{align*}
        \sum_{d\mid n}\mu(d) = 0 \qquad \forall n\geq 2.
    \end{align*}
    Which tells us that the summatory function of $\mu$ is $I$.
\end{lemma}
\begin{proof}
    We define
    \begin{align*}
        f(n) := \sum_{d\mid n}\mu(d) \qquad \text{is multiplicative}
    \end{align*}
    We check this on prime powers, for prime $p$ and $e\geq 1$:
    \begin{align*}
        f(p^e) & = \mu(1) + \mu(p) + \mu(p^2) + \cdots + \mu(p^e) \\
               & = 1 - 1 + 0 + \cdots + 0 = 0
    \end{align*}
    so we're done since $f$ is multiplicative and is $0$ for all power of primes.
\end{proof}

\begin{lemma}
    Let $i: \ZZ_+\to \{1\}$ be the constant $1$ function.
    \[i * \mu = \mu * i = I\]
\end{lemma}
\begin{proof} In the case of $n=1$, we have $i(1)\mu(1) = 1$.

    For $n > 1$, we have $(i * \mu)(n) = \displaystyle \sum_{d\mid n}\mu(d) = 0$ from above.
\end{proof}

We see here that summatory functions can be seen as Dirichlet products: the summatory function $F$ of $f$ is $F = f * i$. What we said about summatory functions being multiplicative boils down to Dirichlet convolutions preserving multiplicativity.

\recall that summatory functions inherit multiplicativity. In fact, this holds for Dirichlet products as well. If $f, g$ are multiplicative, then so is $f * g$.

The proof is parallel to the proof for summatory functions, for \cref{lemma:summatory-preserves-multiplicativity}.

\begin{theorem}[M\"obius Inversion]
    Let
    \begin{align*}
        F(n) & = \sum_{d\mid n} f(d)                         \\
        \intertext{Then we have }
        f(n) & = \sum_{d\mid n}\mu(d)\cdot F(n/d) = \mu * F.
    \end{align*}
\end{theorem}
\begin{proof}
    $F = f * i$, then \[F * \mu = (f * i) * \mu = f * (i * \mu) = f * I = f.\]
    which was simpler than I expected\dots
\end{proof}

\begin{corollary}\label{cor:summatory-multiplicativity-original-multiplicative}
    If $F$ is the summatory function of $f$, and $F$ is multiplicative, then $f$ is also multiplicative, as $f = \mu * F$ and $\mu$ is multiplicative and convolutions with multiplicative functions are multiplicative.
\end{corollary}
\begin{corollary}
    \Cref{cor:summatory-multiplicativity-original-multiplicative} gives another proof that $\phi$ is multiplicative, as \[\sum_{d\mid n}\phi(d) = \phi * i = \mathrm{id}.\]
\end{corollary}

\subsection{Applications of M\"obius Inversion}
\subsubsection{Cyclotomic Polynomials}

\recall the $n^\mathrm{th}$ cyclotomic polynomial $\Phi_n(x)$ is the unique irreducible polynomial in $\ZZ[x]$ dividing $x^n - 1$ but no $x^k - 1$ for $k < n$.

Thus \[\Phi_n(x) = \prod_{\substack{1\leq k < n \\ (k, n) = 1}} \left(x - e^{2\pi i k / n}\right)\]
as the roots of this polynomial are exactly the primitive $n^\mathrm{th}$ roots of unity. We have that \[\prod_{d\mid n}\Phi_d(x) = x^n - 1.\]
By M\"obius inversion, if
\begin{align*}
    G(n) & = \prod_{d\mid n}g(d),
    \intertext{then we have that}
    g(n) & = \prod_{d\mid n}G(d)^{\mu(n/d)}
\end{align*}
In particular, taking $G(n) = x^n - 1$ (with particular $x\in\CC$) as an arithmetic function, we have \[\Phi_n(x) = \prod_{d\mid n}(x^d - 1)^{\mu(n/d)}\qquad (x\in \CC)\]
Applying this identity for enough $x\in \CC$ yields this as an identity of polynomials.

\subsubsection{Dynatomic Polynomials}
The roots of cyclatomic polynomials are roots of unity. Dynatomic polynomials have as roots the periodic points (of certain periods) of a polynomial.

\begin{definition}
    Let $K$ be a field, and let $f\in K[x]$ of degree $d\geq 2$. Let \[f^n = \underbrace{f\circ f\circ \cdots \circ f}_{n\text{ times}}\]
    then $P\in \overline{K}$\footnote{Algebraic numbers in field $K$, the field you get by adjoining all roots of polynomials in $K[x]$.} is said to be \ul{periodic} under $f$ if
    \[f^n(P) = P\qquad \text{for some $n\geq 1$}\]
\end{definition}
\begin{example}
    Let $f(x) = x^2 - 1$. $0$ is a period point under $f$:
    \[0\longmapsto -1\longmapsto 0\]
    and its \ul{period} is $2$.

    \begin{remark*}
        If $n$ is the smallest positive integer such that $f^n(p) = p$ ($p$ periodic), then we call $n$ the \ul{exact period} of $p$ under $f$.
    \end{remark*}
\end{example}
\begin{definition}
    The $n^\mathrm{th}$ dynatomic polynomial of $f$ is
    \[\Phi_{f,n}(x) := \prod_{d\mid n}\left(f^d(x) - x\right)^{\mu(n/d)}\]
\end{definition}
We hope that $\Phi_{f, n}(x)$ has as its roots the points of exact period $n$\dots This hope is dashed\dots
\begin{example}
    $f(x) = x^2 - \frac{3}{4}$.
    \begin{align*}
        f^2(x) - x &= \left(x - \frac{3}{2}\right)\left(1 - \frac{1}{2}\right)^3 \\
        f(x) - x &= \left(x-\frac{3}{2}\right)\left(x + \frac{1}{2}\right)
    \end{align*}
    Thus \[\frac{f^2(x) - x}{f(x) - x} = \left(x + \frac{1}{2}\right)^2\]
    But $x=-\frac{1}{2}$ is \ul{fixed} under $f$.
\end{example}


\section{Dynamic Programming}
\lst{Knapsack}{$\mathcal{O}(n\sum_{i = 1}^n p_i)$}{dynamicProgramming/knapsack.cc}
\lst{TSP}{$\mathcal{O}(n 2^n)$}{dynamicProgramming/tsp.cc}
\lst{Subset Sum}{$\mathcal{O}(n\sum_{i = 1}^n v_i)$}{dynamicProgramming/subSetSum.cc}
\lst{Edit Distance}{$\mathcal{O}(nm)$}{dynamicProgramming/editDistance.cc}
\lst{Longest Increasing Subsequence}{$\mathcal{O}(n\log n)$}{dynamicProgramming/lis.cc}
\lst{Longest Common Subsequence}{$\mathcal{O}(nm)$}{dynamicProgramming/lcs.cc}


\subsection{Kernels}

This is a generalisation of the dot product function, where we want to find similarity between two vectors.

If we have data points \(v\) and \(w\) the distance is:

\(K(v, w)\)


\hypertarget{k-nearest-neighbor-k-nn}{%
\chapter{k-Nearest Neighbor (k-NN)}\label{k-nearest-neighbor-k-nn}}
\section{Introduction}
K-NN algorithm assumes the similarity between the new case/data and
available cases and put the new case into the category that is most
similar to the available categories. K-NN algorithm stores all the
available data and classifies a new data point based on the similarity.
This means when new data appears then it can be easily classified into a
well suite category by using K- NN algorithm. K-NN algorithm can be used
for Regression as well as for Classification but mostly it is used for
the Classification problems.

K-NN is a non-parametric algorithm, which means it does not make any
assumption on underlying data. It is also called a lazy learner
algorithm because it does not learn from the training set immediately
instead it stores the dataset and at the time of classification, it
performs an action on the dataset. K-NN algorithm at the training phase
just stores the dataset and when it gets new data, then it classifies
that data into a category that is much similar to the new data.
\section{Creating Theoretical Model}
Step-1: Select the number K of the neighbors. \(k\)=5 is preferred.

Step-2: Calculate the Euclidean distance of \(k\) number of neighbors.
\[
Euclidean\ Distance = \sqrt{(x_2-x_1)^2 + (y_2-y_1)^2}
\]

Step-3: Take the K nearest neighbors as per the calculated Euclidean
distance. We may use other means of finding sum of distance calculations
as well with Minkowski by setting \(p\) to \(1\) and \(2\) for using
Manhattan and Euclidean, respectivly.
\[
||x_1-x_2|| = (\sum_{i=1}^{n}|x_i-y_i|^p)^\frac{1}{p}
\]

Step-4: Among these k neighbors, count the number of the data points in
each category.

Step-5: Assign the new data points to that category where the neighbor is maximum.
\section{Implementation}
    \begin{tcolorbox}[breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder]
\prompt{In}{incolor}{1}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
\PY{k+kn}{import} \PY{n+nn}{numpy} \PY{k}{as} \PY{n+nn}{np}
\PY{k+kn}{import} \PY{n+nn}{pandas} \PY{k}{as} \PY{n+nn}{pd}
\PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{model\PYZus{}selection} \PY{k+kn}{import} \PY{n}{train\PYZus{}test\PYZus{}split}
\PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{preprocessing} \PY{k+kn}{import} \PY{n}{StandardScaler}
\PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{neighbors} \PY{k+kn}{import} \PY{n}{KNeighborsClassifier}
\PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{metrics} \PY{k+kn}{import} \PY{n}{confusion\PYZus{}matrix}\PY{p}{,} \PY{n}{accuracy\PYZus{}score}
\PY{k+kn}{import} \PY{n+nn}{matplotlib}\PY{n+nn}{.}\PY{n+nn}{pyplot} \PY{k}{as} \PY{n+nn}{plt}
\PY{k+kn}{from} \PY{n+nn}{matplotlib}\PY{n+nn}{.}\PY{n+nn}{colors} \PY{k+kn}{import} \PY{n}{ListedColormap}
\end{Verbatim}
\end{tcolorbox}

    \begin{tcolorbox}[breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder]
\prompt{In}{incolor}{2}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
\PY{n}{dataset} \PY{o}{=} \PY{n}{pd}\PY{o}{.}\PY{n}{read\PYZus{}csv}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{./data/ads.csv}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
\PY{n}{X} \PY{o}{=} \PY{n}{dataset}\PY{o}{.}\PY{n}{iloc}\PY{p}{[}\PY{p}{:}\PY{p}{,} \PY{p}{[}\PY{l+m+mi}{1}\PY{p}{,} \PY{l+m+mi}{2}\PY{p}{]}\PY{p}{]}\PY{o}{.}\PY{n}{values}
\PY{n}{y} \PY{o}{=} \PY{n}{dataset}\PY{o}{.}\PY{n}{iloc}\PY{p}{[}\PY{p}{:}\PY{p}{,} \PY{o}{\PYZhy{}}\PY{l+m+mi}{1}\PY{p}{]}\PY{o}{.}\PY{n}{values}
\PY{n}{dataset}\PY{o}{.}\PY{n}{head}\PY{p}{(}\PY{l+m+mi}{10}\PY{p}{)}
\end{Verbatim}
\end{tcolorbox}

            \begin{tcolorbox}[breakable, size=fbox, boxrule=.5pt, pad at break*=1mm, opacityfill=0]
\prompt{Out}{outcolor}{2}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
   Gender  Age  EstimatedSalary  Purchased
0    Male   19            19000          0
1    Male   35            20000          0
2  Female   26            43000          0
3  Female   27            57000          0
4    Male   19            76000          0
5    Male   27            58000          0
6  Female   27            84000          0
7  Female   32           150000          1
8    Male   25            33000          0
9  Female   35            65000          0
\end{Verbatim}
\end{tcolorbox}

    \begin{tcolorbox}[breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder]
\prompt{In}{incolor}{3}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
\PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{X\PYZus{}test}\PY{p}{,} \PY{n}{y\PYZus{}train}\PY{p}{,} \PY{n}{y\PYZus{}test} \PY{o}{=} \PY{n}{train\PYZus{}test\PYZus{}split}\PY{p}{(}\PY{n}{X}\PY{p}{,} \PY{n}{y}\PY{p}{,} \PY{n}{test\PYZus{}size} \PY{o}{=} \PY{l+m+mf}{0.25}\PY{p}{,} \PY{n}{random\PYZus{}state} \PY{o}{=} \PY{l+m+mi}{0}\PY{p}{)}
\end{Verbatim}
\end{tcolorbox}

    \begin{tcolorbox}[breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder]
\prompt{In}{incolor}{4}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
\PY{n}{sc} \PY{o}{=} \PY{n}{StandardScaler}\PY{p}{(}\PY{p}{)}
\PY{n}{X\PYZus{}train} \PY{o}{=} \PY{n}{sc}\PY{o}{.}\PY{n}{fit\PYZus{}transform}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{)}
\PY{n}{X\PYZus{}test} \PY{o}{=} \PY{n}{sc}\PY{o}{.}\PY{n}{transform}\PY{p}{(}\PY{n}{X\PYZus{}test}\PY{p}{)}
\end{Verbatim}
\end{tcolorbox}

    \begin{tcolorbox}[breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder]
\prompt{In}{incolor}{5}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
\PY{n}{classifier} \PY{o}{=} \PY{n}{KNeighborsClassifier}\PY{p}{(}\PY{n}{n\PYZus{}neighbors} \PY{o}{=} \PY{l+m+mi}{5}\PY{p}{,} \PY{n}{metric} \PY{o}{=} \PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{minkowski}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{p} \PY{o}{=} \PY{l+m+mi}{2}\PY{p}{)}
\PY{n}{classifier}\PY{o}{.}\PY{n}{fit}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{y\PYZus{}train}\PY{p}{)}
\end{Verbatim}
\end{tcolorbox}

            \begin{tcolorbox}[breakable, size=fbox, boxrule=.5pt, pad at break*=1mm, opacityfill=0]
\prompt{Out}{outcolor}{5}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
KNeighborsClassifier()
\end{Verbatim}
\end{tcolorbox}

    \begin{tcolorbox}[breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder]
\prompt{In}{incolor}{6}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
\PY{n}{y\PYZus{}pred} \PY{o}{=} \PY{n}{classifier}\PY{o}{.}\PY{n}{predict}\PY{p}{(}\PY{n}{X\PYZus{}test}\PY{p}{)}
\PY{n+nb}{print}\PY{p}{(}\PY{n}{y\PYZus{}pred}\PY{p}{)}
\PY{n+nb}{print}\PY{p}{(}\PY{n}{y\PYZus{}test}\PY{p}{)}
\end{Verbatim}
\end{tcolorbox}

    \begin{Verbatim}[commandchars=\\\{\}]
[0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 0 1 0 0 1 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0
 0 0 1 0 0 0 0 1 0 0 1 0 1 1 0 0 1 1 1 0 0 1 0 0 1 0 1 0 1 0 0 0 0 1 0 0 1
 0 0 0 0 1 1 1 1 0 0 1 0 0 1 1 0 0 1 0 0 0 0 0 1 1 1]
[0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 1 0 0 0 0 0 1 1 0 0 0 0
 0 0 1 0 0 0 0 1 0 0 1 0 1 1 0 0 0 1 1 0 0 1 0 0 1 0 1 0 1 0 0 0 0 1 0 0 1
 0 0 0 0 1 1 1 0 0 0 1 1 0 1 1 0 0 1 0 0 0 1 0 1 1 1]
    \end{Verbatim}

    \begin{tcolorbox}[breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder]
\prompt{In}{incolor}{7}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
\PY{n+nb}{print}\PY{p}{(}\PY{n}{confusion\PYZus{}matrix}\PY{p}{(}\PY{n}{y\PYZus{}test}\PY{p}{,} \PY{n}{y\PYZus{}pred}\PY{p}{)}\PY{p}{)}
\PY{n+nb}{print}\PY{p}{(}\PY{n}{accuracy\PYZus{}score}\PY{p}{(}\PY{n}{y\PYZus{}test}\PY{p}{,}\PY{n}{y\PYZus{}pred}\PY{p}{)}\PY{p}{)}
\end{Verbatim}
\end{tcolorbox}

    \begin{Verbatim}[commandchars=\\\{\}]
[[64  4]
 [ 3 29]]
0.93
    \end{Verbatim}

    \begin{tcolorbox}[breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder]
\prompt{In}{incolor}{8}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
\PY{n}{X\PYZus{}set}\PY{p}{,} \PY{n}{y\PYZus{}set} \PY{o}{=} \PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{y\PYZus{}train}
\PY{n}{X1}\PY{p}{,} \PY{n}{X2} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{meshgrid}\PY{p}{(}\PY{n}{np}\PY{o}{.}\PY{n}{arange}\PY{p}{(}\PY{n}{start} \PY{o}{=} \PY{n}{X\PYZus{}set}\PY{p}{[}\PY{p}{:}\PY{p}{,} \PY{l+m+mi}{0}\PY{p}{]}\PY{o}{.}\PY{n}{min}\PY{p}{(}\PY{p}{)} \PY{o}{\PYZhy{}} \PY{l+m+mi}{1}\PY{p}{,} \PY{n}{stop} \PY{o}{=} \PY{n}{X\PYZus{}set}\PY{p}{[}\PY{p}{:}\PY{p}{,} \PY{l+m+mi}{0}\PY{p}{]}\PY{o}{.}\PY{n}{max}\PY{p}{(}\PY{p}{)} \PY{o}{+} \PY{l+m+mi}{1}\PY{p}{,} \PY{n}{step} \PY{o}{=} \PY{l+m+mf}{0.01}\PY{p}{)}\PY{p}{,}
                     \PY{n}{np}\PY{o}{.}\PY{n}{arange}\PY{p}{(}\PY{n}{start} \PY{o}{=} \PY{n}{X\PYZus{}set}\PY{p}{[}\PY{p}{:}\PY{p}{,} \PY{l+m+mi}{1}\PY{p}{]}\PY{o}{.}\PY{n}{min}\PY{p}{(}\PY{p}{)} \PY{o}{\PYZhy{}} \PY{l+m+mi}{1}\PY{p}{,} \PY{n}{stop} \PY{o}{=} \PY{n}{X\PYZus{}set}\PY{p}{[}\PY{p}{:}\PY{p}{,} \PY{l+m+mi}{1}\PY{p}{]}\PY{o}{.}\PY{n}{max}\PY{p}{(}\PY{p}{)} \PY{o}{+} \PY{l+m+mi}{1}\PY{p}{,} \PY{n}{step} \PY{o}{=} \PY{l+m+mf}{0.01}\PY{p}{)}\PY{p}{)}
\PY{n}{plt}\PY{o}{.}\PY{n}{figure}\PY{p}{(}\PY{n}{dpi}\PY{o}{=}\PY{l+m+mi}{100}\PY{p}{)}
\PY{n}{plt}\PY{o}{.}\PY{n}{contourf}\PY{p}{(}\PY{n}{X1}\PY{p}{,} \PY{n}{X2}\PY{p}{,} \PY{n}{classifier}\PY{o}{.}\PY{n}{predict}\PY{p}{(}\PY{n}{np}\PY{o}{.}\PY{n}{array}\PY{p}{(}\PY{p}{[}\PY{n}{X1}\PY{o}{.}\PY{n}{ravel}\PY{p}{(}\PY{p}{)}\PY{p}{,} \PY{n}{X2}\PY{o}{.}\PY{n}{ravel}\PY{p}{(}\PY{p}{)}\PY{p}{]}\PY{p}{)}\PY{o}{.}\PY{n}{T}\PY{p}{)}\PY{o}{.}\PY{n}{reshape}\PY{p}{(}\PY{n}{X1}\PY{o}{.}\PY{n}{shape}\PY{p}{)}\PY{p}{,}
             \PY{n}{alpha} \PY{o}{=} \PY{l+m+mf}{0.5}\PY{p}{,} \PY{n}{cmap} \PY{o}{=} \PY{n}{ListedColormap}\PY{p}{(}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{red}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{green}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}\PY{p}{)}\PY{p}{)}
\PY{n}{plt}\PY{o}{.}\PY{n}{xlim}\PY{p}{(}\PY{n}{X1}\PY{o}{.}\PY{n}{min}\PY{p}{(}\PY{p}{)}\PY{p}{,} \PY{n}{X1}\PY{o}{.}\PY{n}{max}\PY{p}{(}\PY{p}{)}\PY{p}{)}
\PY{n}{plt}\PY{o}{.}\PY{n}{ylim}\PY{p}{(}\PY{n}{X2}\PY{o}{.}\PY{n}{min}\PY{p}{(}\PY{p}{)}\PY{p}{,} \PY{n}{X2}\PY{o}{.}\PY{n}{max}\PY{p}{(}\PY{p}{)}\PY{p}{)}
\PY{k}{for} \PY{n}{i}\PY{p}{,} \PY{n}{j} \PY{o+ow}{in} \PY{n+nb}{enumerate}\PY{p}{(}\PY{n}{np}\PY{o}{.}\PY{n}{unique}\PY{p}{(}\PY{n}{y\PYZus{}set}\PY{p}{)}\PY{p}{)}\PY{p}{:}
    \PY{n}{plt}\PY{o}{.}\PY{n}{scatter}\PY{p}{(}\PY{n}{X\PYZus{}set}\PY{p}{[}\PY{n}{y\PYZus{}set} \PY{o}{==} \PY{n}{j}\PY{p}{,} \PY{l+m+mi}{0}\PY{p}{]}\PY{p}{,} \PY{n}{X\PYZus{}set}\PY{p}{[}\PY{n}{y\PYZus{}set} \PY{o}{==} \PY{n}{j}\PY{p}{,} \PY{l+m+mi}{1}\PY{p}{]}\PY{p}{,} \PY{n}{color} \PY{o}{=} \PY{n}{ListedColormap}\PY{p}{(}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{red}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{green}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}\PY{p}{)}\PY{p}{(}\PY{n}{i}\PY{p}{)}\PY{p}{,} \PY{n}{label} \PY{o}{=} \PY{n}{j}\PY{p}{)}
\PY{n}{plt}\PY{o}{.}\PY{n}{xlabel}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{Age}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
\PY{n}{plt}\PY{o}{.}\PY{n}{ylabel}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{Estimated Salary}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
\PY{n}{plt}\PY{o}{.}\PY{n}{legend}\PY{p}{(}\PY{p}{)}
\PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
\end{Verbatim}
\end{tcolorbox}

    \begin{center}
    \adjustimage{max size={0.9\linewidth}{0.9\paperheight}}{./figures/KNN1.png}
    \end{center}
    { \hspace*{\fill} \\}

    \begin{tcolorbox}[breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder]
\prompt{In}{incolor}{9}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
\PY{k}{def} \PY{n+nf}{make\PYZus{}predection}\PY{p}{(}\PY{n}{age}\PY{p}{,} \PY{n}{salary}\PY{p}{)}\PY{p}{:}
    \PY{k}{return} \PY{n}{classifier}\PY{o}{.}\PY{n}{predict}\PY{p}{(}\PY{n}{sc}\PY{o}{.}\PY{n}{transform}\PY{p}{(}\PY{p}{[}\PY{p}{[}\PY{n}{age}\PY{p}{,}\PY{n}{salary}\PY{p}{]}\PY{p}{]}\PY{p}{)}\PY{p}{)}
\end{Verbatim}
\end{tcolorbox}

    \begin{tcolorbox}[breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder]
\prompt{In}{incolor}{10}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
\PY{n}{make\PYZus{}predection}\PY{p}{(}\PY{l+m+mi}{25}\PY{p}{,} \PY{l+m+mi}{130000}\PY{p}{)}
\end{Verbatim}
\end{tcolorbox}

            \begin{tcolorbox}[breakable, size=fbox, boxrule=.5pt, pad at break*=1mm, opacityfill=0]
\prompt{Out}{outcolor}{10}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
array([1], dtype=int64)
\end{Verbatim}
\end{tcolorbox}

    \begin{tcolorbox}[breakable, size=fbox, boxrule=1pt, pad at break*=1mm,colback=cellbackground, colframe=cellborder]
\prompt{In}{incolor}{11}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
\PY{n}{make\PYZus{}predection}\PY{p}{(}\PY{l+m+mi}{5}\PY{p}{,} \PY{l+m+mi}{100}\PY{p}{)}
\end{Verbatim}
\end{tcolorbox}

            \begin{tcolorbox}[breakable, size=fbox, boxrule=.5pt, pad at break*=1mm, opacityfill=0]
\prompt{Out}{outcolor}{11}{\boxspacing}
\begin{Verbatim}[commandchars=\\\{\}]
array([0], dtype=int64)
\end{Verbatim}
\end{tcolorbox}

MCA/Machine Learning/Project file/Tex source/k-Nearest Neighbor.tex


\FloatBarrier
\section{Combined Dynamics} \label{sec:combined}

It is possible to define a set of dynamics to run a combination of the dynamics.
The resulting dynamic is defined as
\begin{equation}
\dot{ x } = \sum_{d\in \mathcal{D}} \gamma_d V_d( x ),
\end{equation}
where $\mathcal{D}=\{ Logit, RD, Smith, BNN \}$ denotes the set of available dynamics, $V_d()$ is the differential equation of the $d\th$ dynamic and $\gamma_d$ is the weight assigned to it.
The dynamics should be defined in a cell array, e.g.,
\begin{lstlisting}
dynamics = {'bnn', 'rd'};
\end{lstlisting}
The combination is made making a linear combination between each dynamic listed in the cell array. The weight assigned to each dynamic is defined in the vector \verb|gamma|. In this case we assign
\begin{lstlisting}
gamma = [.25, .75];
\end{lstlisting}

Fig. \ref{fig:rps_combined} shows an example of the combined dynamics for the rock-paper-scissors game. Note that the evolution of the system is not confined to a limit cycle, as happened with the replicator dynamics in Fig.  \ref{fig:finite1}.

\begin{figure}[h]
  \centering
  \begin{subfigure}[b]{0.4\textwidth}
	  \includegraphics[width=\textwidth]{./images/test_combined.eps}
	  \caption{Simplex.}
	  \label{fig:test_combined_simplex}
  \end{subfigure}
  ~
  \begin{subfigure}[b]{0.45\textwidth}
	  \includegraphics[width=\textwidth]{./images/test_combined_ev.eps}
	  \caption{Evolution of the strategies in time.}
	  \label{fig:test_combined_ev}
  \end{subfigure}
  \caption{Evolution of the combination of replicator dynamics and BNN dynamics.}
  \label{fig:rps_combined}
\end{figure}


\section{Alternative Estimators}
In this section we present a detailed overview of two alternative estimators which in theory exhibit better finite sample properties.
\par Both of these estimators, the Jackknife IV estimator and the LIML estimator, fall under the broader class of k-class estimators. These estimators are partially robust ie less sensitive to weak instruments; with weak instruments, they are more reliable than 2SLS estimates. Conventional asymptotic approximations are used for their inference.

\subsection{Jackknife Instrumental Variables Estimator}
For an intuitive sense of the functioning of the jackknife instrumental variables estimator, consider the term `jackknife' as presented in statistical literature. The  jackknife estimator of a parameter is found by leaving out each observation from a dataset, calculating the estimate and computing the average of these calculations. Given a sample of size $N$, the jackknife estimate is found by aggregating the estimates of each sub-sample of size $(N-1)$. In a similar vein, the estimator we describe in this section replaces the usual fitted values from the reduced form regression of the two-stage least squares by `omit-one' fitted values.
\par The key feature of the JIVE estimator is that it eliminates the correlation between the fitted values and the structural equation errors, as explain in detail below. The fitted value of the standard 2SLS estimator is only asymptotically independent of the structural error ($\varepsilon_i$), but the JIVE is independent even in finite samples.
\par The derivation we present in this section is adopted from \cite{angrist1999jackknife}. To derive the expression formally, consider again the expressions for $\hat\beta_{OPT}$ and $\hat\beta_{2SLS}$ from section 2.2, equations (2.4) and (2.5). $\hat\beta_{OPT}$ employed the optimal instrument $\mathbf Z\pi$ and $\hat\beta_{2SLS}$ used an estimate of $\mathbf Z\pi$, denoted by $\mathbf Z\hat \pi$. Now we introduce the new estimator for $\beta$ by using a different estimate of the optimal instrument, $Z_{i}\tilde{\pi}$.

\par From the formulation of approximate bias of the 2SLS estimator we presented in section 2.4.2 (equation 2.11), it is clear that increasing the number of instruments while keeping the explanatory/predictive power of the instruments constant, leads to an increase in the bias of $\hat\beta_{2SLS}$. However, we can see that for the optimal estimator, increasing the number of instruments while keeping $Z_i\pi$ fixed will have no effect on the properties of $\hat\beta_{OPT}$. Thus, for finite samples in the presence of many instruments, $\hat\beta_{2SLS}$ fares much worse than $\hat\beta_{OPT}$.
\par Recall the representation of the 2SLS estimator using the projection matrix $P_Z$ from Section 2 (equation 2.8). We can see that the first-stage fitted values $\mathbf Z\hat{\pi}$ can be written as:

\begin{equation}
\mathbf Z\hat{\pi} = P_Z\mathbf X = \mathbf Z \pi + P_Z\eta
\end{equation}

The term $P_Z\eta$ in the above equation is correlated with the error term of the first-stage $\eta$ (see equation 1.2) and hence with the structural error term $\varepsilon$. Put differently, since $\hat \pi$ is estimated on the full sample which includes the $i$th observation, it is correlated with $\eta_i$, which is correlated with $\varepsilon_i$. This correlation is given by:

\begin{equation}
E[\varepsilon_{i}Z_{i}\hat{\pi}]
=E[E[\varepsilon_i Z_i\hat\pi|\mathbf Z]]
=E[Z_i(\mathbf Z^\prime \mathbf Z)^{-1} Z_i^\prime] \cdot E[\varepsilon_i \eta_i|\mathbf Z]]
=E[Z_i(\mathbf Z^\prime \mathbf Z)^{-1} Z_i^\prime \cdot \sigma_{\varepsilon\eta}^\prime]
=(K/N)\cdot\sigma_{\varepsilon\eta}'
\end{equation}

Due to this correlation between $\hat \pi$ and $\varepsilon_i$, $\hat\beta_{2SLS}$ is biased for $\beta$. While the correlation disappears asymptotically, it holds implications in finite samples.

\par In devising the new instrument, we attempt to keep the correlation in (4.2) equal to zero. The problem stems from $\hat \pi$ being estimated on the full sample which includes the $i$th observation. For the new estimator which has the constructed instrument $Z_{i}\tilde{\pi}$, $\tilde\pi$ is estimated not on the full sample but on the sample with the $i$th observation removed. Therefore, the new estimated instrument will be independent of $\varepsilon_i$ even in finite samples.

The $i$th row of the estimated instrument for 2SLS, $\mathbf Z\hat{\pi}$, where $\hat{\pi}=(\mathbf Z'\mathbf Z)^{-1}(\mathbf Z'\mathbf X)$, is given by
\begin{equation}
Z_{i}\hat{\pi}= Z_{i}({\mathbf Z}'\mathbf Z)^{-1}(\mathbf Z'\mathbf X)
\end{equation}

Remove the $i$th row from the matrices of regressors and instruments and denote them by $\mathbf X(i)$ and $\mathbf Z(i)$. Thus JIVE removes the dependence between $Z_{i}\hat{\pi}$ and the regressor $X_{i}$.

The corresponding estimate of $\pi$ and the constructed instrument will be:
\begin{equation}
\tilde{\pi}(i)=(\mathbf Z(i)'\mathbf Z(i))^{-1}(\mathbf Z(i)'\mathbf X(i))
\end{equation}

\begin{equation}
Z_{i}\tilde{\pi}(i)=Z_{i}(\mathbf Z(i)'\mathbf Z(i))^{-1}(\mathbf Z(i)'\mathbf X(i)Y)
\end{equation}
$\varepsilon_{i}$ and $X_{j}$ are independent when $i\neq j$, so it follows that
\begin{equation}
E[\varepsilon_{i}Z_{i}\tilde{\pi}(i)]=E[Z_{i}(\mathbf Z(i)'\mathbf Z(i))^{-1}(\mathbf Z(i)')E[\mathbf X(i)\varepsilon_{i}|\mathbf Z]]=0
\end{equation}

Hence, now we have $E[\varepsilon_{i}Z_{i}\tilde{\pi}(i)]=0$, so we have removed the correlation between the fitted values and the structural error presented in equation (4.2).
\par The JIVE estimator is equal to:

\begin{equation}
\hat{\beta}_{JIVE}=(\mathbf {\hat{X}}_{JIVE}'\mathbf X)^{-1}(\mathbf {\hat{X}}_{JIVE}'\mathbf Y)
\end{equation}
where $\hat{X}_{JIVE}$ is $N\times L$ dimensional matrix with the $i$ th row $Z_{i}\tilde{\pi}(i)$. \\

\par \cite{blomquist1999small} summarize the construction of the JIVE estimator by providing the following algorithm:

%\vspace{0.6cm}
\begin{algorithm}
\caption{Jackknife Instrumental Variables Estimator}\label{alg:euclid}
\begin{algorithmic}[1]
\State Use all observations but the $i$th to estimate parameters of the first-stage equation.
\State Combine the estimated first stage parameters with the instruments for the $i$th observation, $Z_i$, to construct a fitted value for the $i$th observation $X_i$.
\State Repeat steps 1 and 2 for all N observations.
\State Regress Y on the fitted values and the exogenous regressors.
\end{algorithmic}
\end{algorithm}

The estimator for $Z_i\pi$, $Z_{i}\tilde{\pi}(i)$ is consistent.
$\hat{\beta}_{JIVE}$ has the same probability limit and first-order asymptotic distribution as $\hat{\beta}_{OPT}$ and $\hat{\beta}_{2SLS}$, under conventional fixed model asymptotics (\cite{stock2002survey}).

\subsection{Limited Information Maximum-Likelihood Estimator}

Limited Information Maximum Likelihood (LIML) is an alternative method to estimate the parameters of the structural equation. It was formalized by \cite{anderson1949estimation}.\\
We follow the same model as described in Section 2, and extend it as shown below:
\begin{equation}
 \mathbf{Y} = \mathbf{X}\beta + \varepsilon = \mathbf{X_0}\beta_0 + \mathbf{X_1}\beta_1 + \varepsilon
\end{equation}
\begin{equation}
 \mathbf{X_1} = \mathbf{Z}\pi + \eta = \mathbf{Z_0}\pi_0 + \mathbf{Z_1}\pi_1 + \eta
\end{equation}
where $\mathbf{Z_0} = \mathbf{X_0}$.\\
Here, $\mathbf{X_0}$ is the matrix of exogenous variables and $\mathbf{X_1}$ of endogenous variables. $\mathbf{Z_0} = \mathbf{X_0}$ is the matrix of included exogenous variables, $\mathbf{Z_1}$ is the matrix of excluded exogenous variables.
Because the LIML estimator is based on the structural equation for $\mathbf{Y}$ combined with the first-stage equation for $\mathbf{X_1}$, it is called `limited information'. LIML estimator is given by:
\begin{equation}
\hat\beta_{LIML}=(\mathbf X'(\mathbf I-\hat\kappa \mathbf M_Z)\mathbf X)^{-1}\mathbf X'(\mathbf I-\hat\kappa \mathbf M_Z)\mathbf Y
\end{equation}
\par The LIML estimator has some excellent properties when the number of excluded instruments (number of columns in matrix $\mathbf X_1$) in the first-stage equation and the sample size are large. Although the LIML estimator and the 2SLS estimator are asymptotically equivalent in the standard large sample theory, they are quite different in case of many instruments or many weak instruments. It has no finite moments, which implies that its density tends to have very thick tails. \cite{anderson2010asymptotic} show that the LIML estimator shows asymptotic optimality with many weak instruments.


\section{Feature Extraction}

Feature extraction is mainly done with OpenCV.

\subsection{Contours}

Contours are probably the most starightforward feature one can think of. With the help of OpenCV, once we've found the \textbf{contour} of the target figure, we can calcuate the following:

\begin{itemize}
\item Perimeter: apply \textit{cv2.arcLength} on the contour gives us an approximation of the perimeter.
\item Area: apply \textit{cv2.contourArea} on the contour gives us an approximation of the area.
\end{itemize}

\subsection{SIFT}

SIFT allows us to easily find key points (corners) in the image. At first glance it seems to be a great feature, since it can distinguish circles / ellipses from triangles and rectangles / squares. However, the extraction of this feature is \textbf{highly unstable}. A little noise (e.g. a zigzag line) in the image could increase the number of key points by more than 20. Although it's possible to set some thresholds to tune out the undesired key points, still we couldn't put too much confidence on this feature.

\subsection{Feature Expansion}

In computer vision, when it comes to geometric figures, there are several frequently used features that we can easily obtain using the existing features. Some of them turn out to be useful for our task:

\begin{itemize}
\item $Thinness = \frac{P^2}{A}$: Thinness measures the degree to which the figure is ``economical'' in using its perimeter to enclose an area.
\item $Extent = \frac{A}{A_{br}}$: Extent measures the how much space the figure takes up in its bounding rectangle (the rectangle of the smallest area that bounds the target figure).
\end{itemize}

\subsection{Autonomous Feature Discovery}

We will discuss the possibility of autonomous rule derivation in Section 7.


\subsection{Convex hull in 3D}
\label{oct_sec:convex_hull}
\paragraph{}
The convex hull property of the NURBS surface indicates that all points on the surface must be contained within the convex hull constructed by its control points \citep{SELIMOVIC2009772}
The algorithm in 3D can be very similar to that in 2D as introduced in Sec.~\ref{qdt_sc:convex_hull}.
\begin{enumerate}
    \item Find the most left and right points (points with minimal and maximum $x$) since they are proved to be part of the convex hull.
    \item Connect these two points and use the line to separate other points into two group.
    \item Find the point with maximum distance to the line in step 2 in any group.
    \item Construct a triangle with two points in step 2 and the point in step 3.
    \item Find the point with maximum distance to the triangle in step 4. %5
    \item Construct a tetrahedron with the triangle in step 4 and the point in step 5.
    \item Follow the same procedures in Sec.~\ref{qdt_sc:convex_hull}
\end{enumerate}


\documentclass{article}
\usepackage{eecstex}
\usepackage{pgfplots}
\usepackage{physics}

\DeclareMathOperator{\rect}{rect}

\title{Homework 01}
\author{Bryan Ngo}
\date{2022-01-19}

\begin{document}

\maketitle

\setcounter{section}{1}

\section{}

\subsection{}

\begin{theorem}
    The system \(L\) is not time invariant.
\end{theorem}
\begin{proof}
    We can represent a time-delayed signal as a linear combination of the other 3 signals.
    We can represent the 3 input signals in a matrix
    \begin{equation}
        \bm{X} =
        \begin{bmatrix}
           -2 & -2 & 0 \\
           1 & 1 & 1 \\
           -2 & 0 & 1
        \end{bmatrix}
    \end{equation}
    Then, we can find the linear combination necessary to create a time delayed signal (for example, \(x_2[n - 1]\)) by solving the system of equations
    \begin{equation}
        \bm{Xa} =
        \begin{bmatrix}
            0 \\
            -2 \\
            1
        \end{bmatrix}
    \end{equation}
    where we get \(\bm{a} = \begin{bmatrix}
        -\frac{3}{2} & \frac{3}{2} & -2
    \end{bmatrix}^\top\).
    Thus, we have
    \begin{equation}
        x_2[n - 1] = -\frac{3}{2} x_1[n] + \frac{3}{2} x_2[n] - 2x_3[n]
    \end{equation}
    Calculating \(-\frac{3}{2} y_1[n] + \frac{3}{2} y_2[n] - 2y_3[n]\),
    we get the following plot:
    \begin{center}
        \begin{tikzpicture}
            \begin{axis}[
                xlabel=\(n\), ylabel={\(L\{x_2[n - 1]\}\)},
                title={Time invariance counterexample},
                axis lines=middle
            ]
            \addplot[
                ycomb,
                color=blue,
                mark=*
            ]
            coordinates {
                (-2, -4)
                (-1, 2)
                (0, 3)
                (1, -9)
                (2, -4)
                (3, -3)
            };
            \end{axis}
        \end{tikzpicture}
        \begin{tikzpicture}
            \begin{axis}[
                xlabel=\(n\), ylabel={\(y_2[n - 1]\)},
                title={Expected output signal},
                axis lines=middle
            ]
            \addplot[
                ycomb,
                color=blue,
                mark=*
            ]
            coordinates {
                (0, -1)
                (1, 1)
                (2, -3)
                (3, -0)
                (4, -1)
            };
            \end{axis}
        \end{tikzpicture}
    \end{center}
    So the system is \emph{not} time-invariant.
\end{proof}

\subsection{}

We can represent the delta function as
\begin{equation}
    \delta[n] = \frac{1}{2} (x_1[n] - x_2[n] + 2x_3[n])
\end{equation}
Using the properties of linear systems, we can find the impulse response
\begin{equation}
    h[n] = \frac{1}{2} (y_1[n] - y_2[n] + 2y_3[n])
\end{equation}
which gives us the following plot:
\begin{center}
    \begin{tikzpicture}
        \begin{axis}[
            xlabel=\(n\), ylabel={\(L\{\delta[n]\}\)},
            title={Impulse response},
            axis lines=middle
        ]
        \addplot[
            ycomb,
            color=blue,
            mark=*
        ]
        coordinates {
            (-2, 2)
            (-1, 1)
            (0, -2)
            (1, 3)
            (2, 2)
            (3, 1)
        };
        \end{axis}
    \end{tikzpicture}
\end{center}

\newpage
\section{}

\begin{align}
    y[n] - ay[n - 1] &= x[n] \\
    y[0] &= 1
\end{align}

\subsection{}

By counterexample, consider \(x[n] = \delta[n]\) and \(n = 1\),
\begin{equation}
    y[1] = a
\end{equation}
Then, consider \(x[n - 1] = \delta[n - 1]\) and \(n = 1\),
\begin{equation}
    y[1] - a y[0] = \delta[0] \implies y[1] = 1 + a
\end{equation}
whereas \(y[n - 1] = y[0] = 1\) by the initial condition.
So the system is not time-invariant.

\subsection{}

Given the initial condition \(y[0] = 1\), scaling the input does not change this fact, so the system \emph{cannot} be linear.

\subsection{}

The system is still not time-invariant due to the same counterexample as above.
The system is now linear as we can see because we no longer have an affine transformation at \(y[0]\).

\newpage
\section{}

\begin{equation}
    y[n] = h[n] \ast (\cos[\omega_0 n] x[n]) = \sum_{k \geqslant 0} \frac{1}{1 + k} \cos[\omega_0 (n - k)] x[n - k]
\end{equation}

\subsection{}
Proving linearity, suppose we are given the system responses \(x_1[n] \iff y_1[n]\) and \(x_2[n] \iff y_2[n]\),
\begin{align}
    T\{ax_1[n] + bx_2[n]\} &= h[n] \ast (\cosine[\omega_0 n] (ax_1[n] + bx_2[n])) \\
    &= h[n] \ast (\cos[\omega_0 n] ax_1[n] + \cos[\omega_0 n] bx_2[n]) \\
    &= a (h[n] \ast \cos[\omega_0 n] x_1[n]) + b (h[n] \ast \cos[\omega_0 n] x_2[n]) \\
    &= ay_1[n] + by_2[n]
\end{align}
For time invariance,
\begin{align}
    T\{x[n - n_0]\} &= h[n] \ast (\cosine[\omega_0 n] x[n - n_0]) \\
    y[n - n_0] &= h[n - n_0] \ast (\cosine[\omega_0 (n - n_0)] x[n - n_0])
\end{align}
The two are clearly not the same, so our system is not time invariant and thus not LTI.

\subsection{}

We can bound the cosine function with 1.
Similar to Example 2.18 in Oppenheim \& Schafer, if \(|x[n]| < B_x\),
\begin{equation}
    |y[n]| = \sum_{k \geqslant 0} \frac{1}{1 + k} B_x = B_x \sum_{k \geqslant 1} \frac{1}{k}
\end{equation}
which is a divergent harmonic series.

\subsection{}

The system is causal since the \(y[n]\) consists of a memoryless multiplication by a cosine function and convolution with a causal LTI system, which makes the system overall causal.

\newpage
\section{}

\subsection{}

We can model the frequency response as
\begin{equation}
    H(e^{j \omega}) =
    \begin{cases}
        1 & |\omega| < \frac{\pi}{2} \\
        0 & \text{elsewhere}
    \end{cases}
\end{equation}
Finding the inverse Fourier transform,
\begin{align}
    h[n] &= \frac{1}{2\pi} \int_{-\pi}^\pi H(e^{j \omega}) e^{j \omega n} \, d\omega \\
    &= \frac{1}{2\pi} \int_{-\frac{\pi}{2}}^{\frac{\pi}{2}} e^{j \omega n} \, d\omega = \frac{1}{2\pi} \eval{\frac{e^{j \omega n}}{jn}}_{-\frac{\pi}{2}}^{\frac{\pi}{2}} \\
    &= \frac{1}{2\pi j n} \left(e^{j \frac{\pi}{2} n} - e^{-j \frac{\pi}{2} n}\right) = \frac{\sin\left[\frac{\pi}{2} n\right]}{\pi n}
\end{align}

\subsection{}

The system is not causal, since \(h[-1] = \frac{1}{\pi} \neq 0\).

\subsection{}

This is a specific case of Example 2.18 in Oppenheim \& Schafer.
Each term of the absolute sum of the sinc function only decreases on the order of \(\frac{1}{n}\), so we can treat the sum as the harmonic series, which diverges.
Thus, the low pass filter is not BIBO stable.

\newpage
\section{}

\begin{equation}
    y[n] = \frac{1}{6} \sum_{k = 0}^5 x[n - k]
\end{equation}

\subsection{}

Using Equation 2.123 from Oppenheim \& Schafer with \(M = 5\),
\begin{align}
    H(e^{j \omega}) &= \frac{1}{6} \frac{\sin(3\omega)}{\sin\left(\frac{\omega}{2}\right)} e^{-j \frac{5}{2} \omega} \\
    |H(e^{j \omega})| &= \frac{1}{6} \left|\frac{\sin(3\omega)}{\sin\left(\frac{\omega}{2}\right)}\right| \\
    \angle H(e^{j \omega}) &= -\frac{5}{2} \omega \\
\end{align}
The zero crossings are \(\omega = \frac{k\pi}{3}\) for nonzero \(k \in \Z\), by solving for the equation \(\sin(3\omega) = 0\).
\begin{center}
    \begin{tikzpicture}
        \begin{axis}[
            xlabel=\(\omega\), ylabel={\(|H(e^{j \omega})|\)},
            title={Magnitude of Frequency Response},
            axis lines=middle,
            ymin=0, ymax=1,
            width=0.4\textwidth
        ]
        \addplot[color=blue] table[
            col sep=comma,
            x=n, y=q6a-mag
        ]{q6a-mag.csv};
        \end{axis}
    \end{tikzpicture}
    \begin{tikzpicture}
        \begin{axis}[
            xlabel=\(\omega\), ylabel={\(\angle H(e^{j \omega})\)},
            title={Phase of Frequency Response},
            axis lines=middle,
            width=0.4\textwidth
        ]
        \addplot[color=blue] table[
            col sep=comma,
            x=n, y=q6a-ang
        ]{q6a-ang.csv};
        \end{axis}
    \end{tikzpicture}
\end{center}

\subsection{}

Let \(\omega_1 = \frac{\pi}{3}\).
Then, the output \(y[n]\) is as shown:
\begin{center}
    \begin{tikzpicture}
        \begin{axis}[
            xlabel=\(n\), ylabel={\(y[n]\)},
            title={Convolution Sum},
            axis lines=middle,
            width=0.4\textwidth
        ]
        \addplot[ycomb, mark=*, color=blue] table[
            col sep=comma,
            x=n, y=q6b
        ]{q6b.csv};
        \end{axis}
    \end{tikzpicture}
\end{center}
The transient at the start of the convolution sum comes from at the beginning of the MAF, where the positive terms from the first 2 inputs of the cosine function.
The settling of the convolution sum comes from the fact that the sum of a sinusoid without a vertical shift over 1 period is 0. Since the MAF has a window of size 6, and the cosine has a period of 6 as well, once the window is filled by the cosine, it we get 0.
For any value of the convolution sum afterwards, it remains 0 due to the MAF now being saturated with a constant 1 period of the cosine, which has a sum of 0.

\newpage
\section{}

\begin{equation}
    y[n] = \mathcal{P}\{x[n - 2], x[n - 1], x[n], x[n + 1], x[n + 2]\}
\end{equation}

\subsection{}

We can transform the problem into a least-squares problem:
\begin{equation}
    \begin{bmatrix}
        x[-2] \\
        x[-1] \\
        x[0] \\
        x[1] \\
        x[2]
    \end{bmatrix}
    =
    \begin{bmatrix}
        1 & -2 & 4 \\
        1 & -1 & 1 \\
        1 & 0 & 0 \\
        1 & 1 & 1 \\
        1 & 2 & 4
    \end{bmatrix}
    \begin{bmatrix}
        a_0 \\
        a_1 \\
        a_2
    \end{bmatrix}
\end{equation}
By the least squares formula \(\bm{\hat{a}} = (\bm{P}^\top \bm{P})^{-1} \bm{P}^\top \bm{x}\), we get that
\begin{align}
    \begin{bmatrix}
        a_0 \\
        a_1 \\
        a_2
    \end{bmatrix}
    &=
    \frac{1}{35} \begin{bmatrix}
        -3 & 12 & 17 & 12 & -3 \\
        -7 & -3.5 & 0 & 3.5 & 7 \\
        5 & -2.5 & -5 & -2.5 & 5
    \end{bmatrix}
    \begin{bmatrix}
        x[-2] \\
        x[-1] \\
        x[0] \\
        x[1] \\
        x[2]
    \end{bmatrix} \\
    \implies y[0] &= a_0 = \frac{1}{35} (-3x[-2] + 12x[-1] + 17x[0] + 12x[1] - 3x[2])
\end{align}

\subsection{}

\begin{equation}
    \begin{bmatrix}
        x[-1] \\
        x[0] \\
        x[1] \\
        x[2] \\
        x[3]
    \end{bmatrix}
    =
    \begin{bmatrix}
        1 & -2 & 4 \\
        1 & -1 & 1 \\
        1 & 0 & 0 \\
        1 & 1 & 1 \\
        1 & 2 & 4
    \end{bmatrix}
    \begin{bmatrix}
        a_0 \\
        a_1 \\
        a_2
    \end{bmatrix}
\end{equation}
which gives us the solution
\begin{equation}
    y[1] = a_0 = \frac{1}{35} (-3x[-1] + 12x[0] + 17x[1] + 12x[2] - 3x[3])
\end{equation}

\subsection{}

The system is linear since it entirely consists of additions and scalar multiplications.
The system is time-invariant since for any shift \(n_0\), the corresponding \(x[n - n_0]\) is simply shifted and put into the solution, which is what is expected for a time-shifted output.
The system is stable since if \(x[n]\) is bounded by \(B_x\), \(y[n]\) is bounded by \(B_x\).
Since the system is LTI, the system has a frequency response.
We do not actually need to perform a regression for every \(n\), since the coefficient matrix multiplying the set of \(x[n - k]\) is constant.

\subsection{}

We can see that points further from the operating point have the largest weights, while weights further away have smaller and even negative weights.
This means that it better represents a distribution around a local region, and does not flatten out features as drastically as a standard MAF.

\end{document}