\section{What is data compression}

\singlespacing
In the context of data files, data compression is the process of encoding information using a smaller representation than the original file \cite{WikiDataCompression}. For data files, the size of the representation can be defined in bits or other derived units such as bytes.

\doublespacing
\singlespacing
Benefits of data compression occur from reducing the resources required to store and transmit data \cite{WikiDataCompression}. However, in the process of compressing and decompressing data computational resources are consumed which may impact benefits gained by compression.

\doublespacing
\singlespacing
The ratio of compression can be defined as:

\begin{equation*}
  C_{r} = \dfrac{d_{1}}{d_{2}}
\end{equation*}

\doublespacing
\singlespacing
The compressed ratio (C$_{r}$) is equal to the compressed file size ($d_{1}$) over the orginal file size ($d_{2}$) \cite{MasseyStudyGuide}.

\doublespacing
\singlespacing
\section{Lossless Compression}

\singlespacing

Lossless compression is a method of encoding a file with the goal of representing the file using a smaller amount of bits but not losing any information from the decoding process. The opposite of lossless compression is lossy compression where this is not the case. Huffman coding, the subject of this report, is used to implement lossless compression \cite{WikiHuffman}.

\doublespacing
\singlespacing
Lossless compression works because most real-world data contains statistical redundancy \cite{WikiDataCompression}. If a file contains symbols that occur more frequently than others we can use smaller representations to represent the frequent symbols and larger representations to represent the less frequent symbols. If a symbol doesn't occur at all in a file then no representation for that symbol is needed.

\doublespacing
\singlespacing
The two most common ways of constructing statistical models for the purposes of data compression are static and adaptive models. Static models build a model after reading all the data and store the model representation in the encoded file. Adaptive models update a model as the file is compressed \cite{WikiLossless}. The implementation of Huffman coding presented in this report is an example of a static model.

\doublespacing
\singlespacing
\section{Lossy Compression}

\singlespacing

Lossy compression works by removing less important information from a data file. Audio and Image data are examples of where lossy compression is applied. Human eyes are more sensitive to variations in luminance than they are to variations in color \cite{WikiDataCompression}. Because of this fact image compression can remove information from files that humans can not perceive thereby improving compression ratio for a file by reducing the number of bits needed to represent the said file. Similarly human limitations for hearing various audio signals present opportunities to decrease the required bit representation of audio files.

\doublespacing
\singlespacing
\section{Use cases}

\singlespacing

Various uses for data compression can be defined. General purpose data compression algorithms generally refer to compression algorithms that apply to standard text or binary files. Of these algorithms, some may be optimized for a particular type of input file such as text compression. As mentioned in previous sections specialized audio and image compression algorithms exist. Indeed for any particular domain, there may be a case for developing a special purpose compression algorithm, such is the case of HAPZIPPER a compression application for the purpose of compressing genetic data \cite{WikiLossless}.

\doublespacing
\singlespacing

Many of the lossless compression algorithms in use today combine Huffman Coding and other compression algorithms. DEFLATE combines Lempel-Ziv with Huffman coding and is used in zip files, gzip files and PNG images.

\doublespacing
\singlespacing
\section{Entropy}

\singlespacing

Information theory underpins the theoretical background of data compression. Claude Shannon published pioneering papers on the topic in the 1940s and 1950s \cite{WikiDataCompression}. Shannon came up with the idea of information entropy which can be defined by the following equation in regards to binary data \cite{MasseyStudyGuide}:

\begin{equation*}
  H = -\left(\sum\limits_{i=0}^n p_{i} log_{2}(p_{i})\right)
\end{equation*}

\doublespacing
\singlespacing
The negative of the sum of the probability of the symbol ($p_{i}$) occurring in a file times log base 2 of the probability of the symbol occurring in a file.

\doublespacing
\singlespacing
The implications of information entropy impose a limit on the potential compression ratio. If the likelihood of any symbol is the same entropy will equal the number of bits it takes to represent each symbol in the original file. A lower entropy allows more frequent symbols to be represented in a smaller number of bits.

\doublespacing
\singlespacing
There is no single lossless data compression algorithm that can compress any and all data. In fact, it is provably impossible to create such an algorithm \cite{WikiLossless}.

\section{Huffman Coding}

\doublespacing
\singlespacing
Huffman coding is an algorithm developed by David Huffman in 1951. He came up with his method as a student studying information theory at the Massachusetts Institute of Technology \cite{WikiHuffman}.

\doublespacing
\singlespacing
A Huffman code refers to a prefix code, an encoded representation of a symbol whereas Huffman coding is the process of creating and decoding Huffman codes. Huffman coding is a lossless data compression algorithm. There are two major tasks for the algorithm to perform. Encoding a data file and decoding an encoded file with the goal being achieving an optimal compression ratio.

\doublespacing
\singlespacing
Encoding a file involves building a Huffman tree, a type of top-down binary tree used to create prefix codes for symbols. Symbols that occur more frequently are closer to the root of the tree and ones that occur less frequently are found at lower levels of the tree.

\doublespacing
\singlespacing
Decoding a file involves using a built Huffman tree to decode a sequence of encoded symbols into their original form. A Huffman tree for any given file is not necessarily unique so a Huffman tree must also be extracted in the decoding process.


\lab{Pandas 1: Introduction}{Pandas 1: Introduction}
\objective{Though NumPy and SciPy are powerful tools for numerical computing, they lack some of the high-level functionality necessary for many data science applications.
Python's \emph{pandas} library, built on NumPy, is designed specifically for data management and analysis.
In this lab, we introduce pandas data structures, syntax, and explore its capabilities for quickly analyzing and presenting data.
}
\label{lab:pandas1}

% Just as NumPy is built on the \li{ndarray} data structure suited for efficient
% scientific and numerical computation, pandas is centered around a handful of
% core data structures custom built for data analysis: the \li{Series}, \li{DataFrame}, and \li{Panel}, which roughly correspond to one, two, and three-dimensional arrays.
% The \li{Panel} data structure is not used nearly

\section*{Series} % ===========================================================

A pandas \li{Series} is generalization of a one-dimensional NumPy array.
Like a NumPy array, every \li{Series} has a data type (\li{dtype}), and the entries of the \li{Series} are all of that type.
Unlike a NumPy array, every \li{Series} has an \emph{index} that labels each entry, and a \li{Series} object can also be given a name to label the entire data set.

\begin{lstlisting}
>>> import numpy as np
>>> import pandas as pd

# Initialize a Series of random entries with an index of letters.
>>> pd.Series(np.random.random(4), index=['a', 'b', 'c', 'd'])
<<a    0.474170
b    0.106878
c    0.420631
d    0.279713
dtype: float64>>

# The default index is integers from 0 to the length of the data.
>>> pd.Series(np.random.random(4), name="uniform draws")
<<0    0.767501
1    0.614208
2    0.470877
3    0.335885
Name: uniform draws, dtype: float64>>
\end{lstlisting}

The index in a \li{Series} is a pandas object of type \li{Index} and is stored as the \li{index} attribute of the \li{Series}.
The plain entries in the \li{Series} are stored as a NumPy array and can be accessed as such via the \li{values} attribute.

\begin{lstlisting}
>>> s1 = pd.Series([1, 2, 3, 4], index=['a', 'b', 'c', 'd'], name="some ints")

>>> s1.values                           # Get the entries as a NumPy array.
<<array([1, 2, 3, 4])>>

>>> print(s1.name, s1.dtype, sep=", ")  # Get the name and dtype.
<<some ints, int64>>

>>> s1.index                            # Get the pd.Index object.
<<Index(['a', 'b', 'c', 'd'], dtype='object')>>
\end{lstlisting}

The elements of a \li{Series} can be accessed by either the regular position-based integer index, or by the corresponding label in the index.
New entries can be added dynamically as long as a valid index label is provided, similar to adding a new key-value pair to a dictionary.
A \li{Series} can also be initialize from a dictionary: the keys become the index labels, and the values become the entries.

\begin{lstlisting}
>>> s2 = pd.Series([10, 20, 30], index=["apple", "banana", "carrot"])
>>> s2
<<apple     10
banana    20
carrot    30
dtype: int64>>

# s2[0] and s2["apple"] refer to the same entry.
>>> print(s2[0], s2["apple"], s2["carrot"])
<<10 10 30>>

>>> s2[0] += 5              # Change the value of the first entry.
>>> s2["dewberry"] = 0      # Add a new value with label 'dewberry'.
>>> s2
<<apple       15
banana      20
carrot      30
dewberry     0
dtype: int64>>

# Initialize a Series from a dictionary.
>>> pd.Series({"eggplant":3, "fig":5, "grape":7}, name="more foods")
<<eggplant    3
fig         5
grape       7
Name: more foods, dtype: int64>>
\end{lstlisting}

Slicing and fancy indexing also work the same way in \li{Series} as in NumPy arrays.
In addition, multiple entries of a \li{Series} can be selected by indexing a list of labels in the index.

\begin{lstlisting}
>>> s3 = pd.Series({"lions":2, "tigers":1, "bears":3}, name="oh my")
>>> s3
<<bears     3
lions     2
tigers    1
Name: oh my, dtype: int64>>

# Get a subset of the data by regular slicing.
>>> s3[1:]
<<lions     2
tigers    1
Name: oh my, dtype: int64>>

# Get a subset of the data with fancy indexing.
>>> s3[np.array([len(i) == 5 for i in s3.index])]
<<bears    3
lions    2
Name: oh my, dtype: int64>>

# Get a subset of the data by providing several index labels.
>>> s3[ ["tigers", "bears"] ]
tigers    1                     # Note that the entries are reordered,
bears     3                     # and the name stays the same.
<<Name: oh my, dtype: int64>>
\end{lstlisting}

\begin{comment} % Unnecessary, already getting too long.
We can create a \li{Series} having constant values in the following manner:
\begin{lstlisting}
>>> val = 4     #desired constant value of Series
>>> n = 6       #desired length of Series
>>> s3 = pd.Series(val, index=np.arange(n))
>>> s3
0    4
1    4
2    4
3    4
4    4
5    4
dtype: int64
\end{lstlisting}
\end{comment}

\begin{problem}
Create a pandas \li{Series} where the index labels are the even integers $0,2,\ldots,50$, and the entries are $n^2 - 1$, where $n$ is the entry's label.
Set all of the entries equal to zero whose labels are divisible by $3$.
\end{problem}

\subsection*{Operations with Series} % ----------------------------------------

A \li{Series} object has all of the advantages of a NumPy array, including entry-wise arithmetic, plus a few additional features (see Table \ref{table:pandas-numerical-methods}).
Operations between a \li{Series} $S_1$ with index $I_1$ and a \li{Series} $S_2$ with index $I_2$ results in a new \li{Series} with index $I_1\cup I_2$.
In other words, the index dictates how two \li{Series} can interact with each other.

\begin{lstlisting}
>>> s4 = pd.Series([1, 2, 4], index=['a', 'c', 'd'])
>>> s5 = pd.Series([10, 20, 40], index=['a', 'b', 'd'])
>>> 2*s4 + s5
<<a    12.0>>
<<b     NaN>>                           # s4 doesn't have an entry for b, and
<<c     NaN>>                           # s5 doesn't have an entry for c, so
<<d    48.0>>                           # the combination is Nan (np.nan / None).
<<dtype: float64>>
\end{lstlisting}

\begin{table}[H]
\begin{tabular}{r|l}
Method & Returns \\ \hline
\li{<<abs>>()}     & Object with absolute values taken (of numerical data) \\
\li{argmax()}  & The index label of the maximum value \\
\li{argmin()}  & The index label of the minimum value \\
\li{count()}   & The number of non-null entries \\
\li{cumprod()} & The cumulative product over an axis \\
\li{cumsum()}  & The cumulative sum over an axis \\
\li{<<max>>()}     & The maximum of the entries \\
\li{mean()}    & The average of the entries \\
\li{median()}  & The median of the entries \\
\li{<<min>>()}     & The minimum of the entries \\
\li{mode()}    & The most common element(s) \\
\li{prod()}    & The product of the elements \\
\li{<<sum>>()}     & The sum of the elements \\
\li{var()}     & The variance of the elements \\
\end{tabular}
\caption{Numerical methods of the \li{Series} and \li{DataFrame} pandas classes.
% Methods marked with a * are not methods of NumPy's \li{ndarray} class.
}
\label{table:pandas-numerical-methods}
\end{table}

Many \li{Series} are more useful than NumPy arrays primarily because of their index.
For example, a \li{Series} can be indexed by time with a pandas \li{DatetimeIndex}, an index with date and/or time values.
The usual way to create this kind of index is with \li{pd.date_range()}.

\begin{lstlisting}
# Make an index of the first three days in July 2000.
>>> pd.date_range("7/1/2000", "7/3/2000", freq='D')
<<DatetimeIndex(['2000-07-01', '2000-07-02', '2000-07-03'],
                dtype='datetime64[ns]', freq='D')>>
\end{lstlisting}

\begin{problem}
Suppose you make an investment of $d$ dollars in a particularly volatile stock.
Every day the value of your stock goes up by \$$1$ with probability $p$, or down by \$$1$ with probability $1-p$ (this is an example of a \emph{random walk}).

Write a function that accepts a probability parameter $p$ and an initial amount of money $d$, defaulting to $100$.
Use \li{pd.date_range()} to create an index of the days from 1 January 2000 to 31 December 2000.
Simulate the daily change of the stock by making one draw from a Bernoulli distribution with parameter $p$ (a binomial distribution with one draw) for each day.
Store the draws in a pandas \li{Series} with the date index and set the first draw to the initial amount $d$.
Sum the entries cumulatively to get the stock value by day.
Set any negative values to $0$, then plot the series using the \li{plot()} method of the \li{Series} object.

Call your function with a few different values of $p$ and $d$ to observe the different possible kinds of behavior.
\label{prob:pandas-random-walk}
\end{problem}

\begin{info}
The \li{Series} in Problem \ref{prob:pandas-random-walk} is an example of a \emph{time series}, since it is indexed by time.
Time series show up often in data science; we will explore them in more depth in another lab.
\end{info}

\begin{table}[H]
\begin{tabular}{r|l}
Method & Description \\ \hline
\li{append()} & Concatenate two or more \li{Series}. \\
\li{drop()} & Remove the entries with the specified label or labels \\
\li{drop_duplicates()} & Remove duplicate values \\
\li{dropna()} & Drop null entries \\
\li{fillna()} & Replace null entries with a specified value or strategy \\
\li{reindex()} & Replace the index \\
\li{sample()} & Draw a random entry \\
\li{shift()} & Shift the index \\
% \li{truncate()} & \\
\li{unique()} & Return unique values \\
% \li{where()} & Search for entires that satisfy some criteria \\
\end{tabular}
\caption{Methods for managing or modifying data in a pandas \li{Series} or \li{DataFrame}.}
\label{table:pandas-manage-data}
\end{table}

\section*{Data Frames} % ======================================================

A \li{DataFrame} is a collection of \li{Series} that share the same index, and is therefore a two-dimensional generalization of a NumPy array.
The row labels are collectively called the \emph{index}, and the column labels are collectively called the \emph{columns}.
An individual column in a \li{DataFrame} object is one \li{Series}.

There are many ways to initialize a \li{DataFrame}.
In the following code, we build a \li{DataFrame} out of a dictionary of \li{Series}.

\begin{lstlisting}
>>> x = pd.Series(np.random.randn(4), ['a', 'b', 'c', 'd'])
>>> y = pd.Series(np.random.randn(5), ['a', 'b', 'd', 'e', 'f'])
>>> df1 = pd.DataFrame({"series 1": x, "series 2": y})
>>> df1
<<   series 1  series 2
a -0.365542  1.227960
b  0.080133  0.683523
c  0.737970       NaN
d  0.097878 -1.102835
e       NaN  1.345004
f       NaN  0.217523>>
\end{lstlisting}

Note that the index of this \li{DataFrame} is the union of the index of \li{Series} \li{x} and that of \li{Series} \li{y}.
The columns are given by the keys of the dictionary \li{d}.
Since \li{x} doesn't have a label \li{e}, the
value in row \li{e}, column \li{1} is \li{NaN}.
This same reasoning explains the other missing values as well.
Note that if we take the first column of the \li{DataFrame} and drop the missing values, we recover the \li{Series} \li{x}:

\begin{lstlisting}
>>> df1["series1"].dropna()
<<a   -0.365542
b    0.080133
c    0.737970
d    0.097878
Name: series 1, dtype: float64>>
\end{lstlisting}

\begin{warn}
A pandas \li{DataFrame} cannot be sliced in exactly the same way as a NumPy array.
Notice how we just used \li{df1["series 1"]} to access a \emph{column} of the
the \li{DataFrame} \li{df1}.
We will discuss this in more detail later on.
\end{warn}

We can also initialize a \li{DataFrame} using a NumPy array, creating custom
row and column labels.
\begin{lstlisting}
>>> data = np.random.random((3, 4))
>>> pd.DataFrame(data, index=['A', 'B', 'C'], columns=np.arange(1, 5))

            1	        2	        3	        4
A	 0.065646	 0.968593	 0.593394	 0.750110
B	 0.803829	 0.662237	 0.200592	 0.137713
C	 0.288801	 0.956662	 0.817915	 0.951016
3 rows     4 columns
\end{lstlisting}

As with Series, if we don't specify the index or columns, the default is
\li{np.arange(n)}, where \li{n} is either the number of rows or columns.

\begin{comment} % Not necessary here. Too much info.
It is also possible to create multi-indexed arrays, for example:
\begin{lstlisting}
>>> grade=['eighth', 'ninth', 'tenth']
>>> subject=['math', 'science', 'english']
>>> myindex = pd.MultiIndex.from_product([grade, subject], names=['grade', 'subject'])
>>> myseries = pd.Series(np.random.randn(9), index=myindex)
>>> myseries
grade   subject
eighth  math       1.706644
        science   -0.899587
        english   -1.009832
ninth   math       2.096838
        science    1.884932
        english    0.413266
tenth   math      -0.924962
        science   -0.851689
        english    1.053329
dtype: float64
\end{lstlisting}

Multi-indexing is visually convenient, and will be explored further in Lab
\ref{lab:pandas3}, where we will discuss pandas pivot tables.
\end{comment}

\section*{Viewing and Accessing Data}

In this section we will explore some elementary accessing and querying
techniques that enable us to maneuver through and gain insight into our data.
Try using the \li{describe()} and \li{head()} methods for quick data summaries.

\subsection*{Basic Data Access}

We can slice the rows of a \li{DataFrame} much as with a NumPy array.

\begin{lstlisting}
>>> df = pd.DataFrame(np.random.randn(4, 2), index=['a', 'b', 'c', 'd'],
                      columns = ['I', 'II'])
>>> df[:2]

          I        II
a  0.758867  1.231330
b  0.402484 -0.955039

[2 rows x 2 columns]
\end{lstlisting}

More generally, we can select subsets of the data using the \li{iloc} and
\li{loc} indexers.
The \li{loc} index selects rows and columns based on
their \emph{labels}, while the \li{iloc} method selects them based on their
integer \emph{position}.
Accessing \li{Series} and \li{DataFrame} objects
using these indexing operations is more efficient than using bracket indexing, because the bracket indexing has to check many cases before it can determine how to slice the data structure.
Using \li{loc}/\li{iloc} explicitly, bypasses the extra checks.

\begin{lstlisting}
>>> # select rows a and c, column II
>>> df.loc[['a','c'], 'II']

a    1.231330
c    0.556121
Name: II, dtype: float64

>>> # select last two rows, first column
>>> df.iloc[-2:, 0]

c   -0.475952
d   -0.518989
Name: I, dtype: float64
\end{lstlisting}

Finally, a column of a \li{DataFrame} may be accessed using simple square
brackets and the name of the column, or alternatively by treating the label
as an object:
\begin{lstlisting}
>>> # get second column of df
>>> df['II']        # or, equivalently, df.II

a    1.231330
b   -0.955039
c    0.556121
d    0.173165
Name: II, dtype: float64
\end{lstlisting}

All of these techniques for getting subsets of the data may also be used to set
subsets of the data:

\begin{lstlisting}
>>> # set second columns to zeros
>>> df['II'] = 0
>>> df['II']
a    0
b    0
c    0
d    0
Name: II, dtype: int64

>>> # add additional column of ones
>>> df['III'] = 1
>>> df
            I  II  III
a   -0.460457   0    1
b    0.973422   0    1
c   -0.475952   0    1
d   -0.518989   0    1
\end{lstlisting}

% I FEEL LIKE THERE SHOULD BE A PROBLEM HERE!!

\begin{comment} % Moved random walk to Series section.
\subsection*{Plotting}

Plotting is often a much more effective way to view and gain understanding of a dataset than simply viewing the raw numbers.
Fortunately, pandas interfaces well with matplotlib, allowing relatively
painless data visualization.

Most of the functionality of plotting using pandas will be discussed in Lab \ref{lab:pandas2}.
 In this lab, we will simply show how to plot a \li{Series}.
Doing so is easy, as the \li{Series} object is equipped with its own plot
function.
% Change this example to use the titanic data, which will be cleaned up.

Let's start with visualizing a simple random walk.
By way of background, a \emph{random walk} is a
stochastic process used to model a non-deterministic path through some space.
It is a useful construct in
many fields and can be used to explain things like the motion of a molecule as it travels through a liquid to modeling the fluctuations
of stock prices.
Here we will simulate a one-dimensional symmetric random walk on the integers, which can be
described as follows.
\begin{enumerate}
  \item Start at 0.
  \item Flip a fair coin.
  \item If heads, move one unit to the right.
Otherwise, move one unit to the left.
  \item Go to Step 2.
\end{enumerate}
How can we simulate this random walk efficiently? Note that the walk is really characterized by the outcomes
of the coin flip.
If we represent heads by the number $1$ and tails by $-1$, then our position at a given moment
is just the cumulative sum of all previous outcomes.
Below, we simulate a sequence of coin flips, build the
resulting random walk, and plot the outcome.
\begin{lstlisting}
>>> import matplotlib.pyplot as plt
>>> N = 1000        # length of random walk
>>> s = np.zeros(N)
>>> s[1:] = np.random.binomial(1, .5, size=(N-1,))*2-1 #coin flips
>>> s = pd.Series(s)
>>> s = s.cumsum()  # random walk
>>> s.plot()
>>> plt.ylim([-50, 50])
>>> plt.show()
>>> plt.close()
\end{lstlisting}

The random walk is shown in Figure \ref{fig:PandasRandomWalk}.

\begin{figure}
\centering
\includegraphics[width=.7 \textwidth]{randomWalk.pdf}
\caption{Random walk of length 1000.}
\label{fig:PandasRandomWalk}
\end{figure}

\begin{problem}
Create five random walks of length 100, and plot them together in the same plot.

Next, create a ``biased'' random walk by changing the coin flip probability of head from 0.5 to 0.51.
Plot this biased walk with lengths 100, 10000, and then 100000.
Notice the definite trend that emerges.
Your results should be comparable to those in Figure \ref{pandas:biasedRandomWalk}.
\end{problem}

\begin{figure}
\centering
\includegraphics[width=.7 \textwidth]{biasedRandomWalk.pdf}
\caption{Biased random walk of length 100 (above) and 10000 (below).}
\label{pandas:biasedRandomWalk}
\end{figure}
\end{comment}

\section*{SQL Operations in pandas} % =========================================

The \li{DataFrame}, being a tabular data structure, bears an obvious resemblance to a typical relational
database table.
SQL is the standard for working with relational databases, and in this section we will
explore how pandas accomplishes some of the same tasks as SQL.
The SQL-like functionality of pandas is
one of its biggest advantages, since it can eliminate the need to switch between programming languages
for different tasks.
Within pandas we can handle both the querying \emph{and} data analysis.

For the following examples, we will use the following data:
\begin{lstlisting}
>>> #build toy data for SQL operations
>>> name = ['Mylan', 'Regan', 'Justin', 'Jess', 'Jason', 'Remi', 'Matt', 'Alexander', 'JeanMarie']
>>> sex = ['M', 'F', 'M', 'F', 'M', 'F', 'M', 'M', 'F']
>>> age = [20, 21, 18, 22, 19, 20, 20, 19, 20]
>>> rank = ['Sp', 'Se', 'Fr', 'Se', 'Sp', 'J', 'J', 'J', 'Se']
>>> ID = range(9)
>>> aid = ['y', 'n', 'n', 'y', 'n', 'n', 'n', 'y', 'n']
>>> GPA = [3.8, 3.5, 3.0, 3.9, 2.8, 2.9, 3.8, 3.4, 3.7]
>>> mathID = [0, 1, 5, 6, 3]
>>> mathGd = [4.0, 3.0, 3.5, 3.0, 4.0]
>>> major = ['y', 'n', 'y', 'n', 'n']
>>> studentInfo = pd.DataFrame({'ID': ID, 'Name': name, 'Sex': sex, 'Age': age, 'Class': rank})
>>> otherInfo = pd.DataFrame({'ID': ID, 'GPA': GPA, 'Financial_Aid': aid})
>>> mathInfo = pd.DataFrame({'ID': mathID, 'Grade': mathGd, 'Math_Major': major})
\end{lstlisting}

Before querying our data, it is important to know some of its basic properties,
such as number of columns, number of rows, and the datatypes of the columns.
This can be done by simply calling the \li{info()} method on the desired
\li{DataFrame}:

\begin{lstlisting}
>>> mathInfo.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Data columns (total 3 columns):
Grade         5 non-null float64
ID            5 non-null int64
Math_Major    5 non-null object
dtypes: float64(1), int64(1), object(1)
\end{lstlisting}

We can also get some basic information about the structure of the \li{DataFrame}
using the \li{head()} or \li{tail()} methods.

\begin{lstlisting}
>>> mathInfo.head()
   Grade  ID Math_Major  ID  Age  GPA
0    4.0   0          y   0   20  3.8
1    3.0   1          n   2   18  3.0
2    3.5   5          y   4   19  2.8
3    3.0   6          n   6   20  3.8
4    4.0   3          n   7   19  3.4
\end{lstlisting}

The method \li{isin()} is a useful way to find certain values in a
\li{DataFrame}.
 It compares the input (a list, dictionary, or \li{Series})
to the \li{DataFrame} and returns a boolean \li{Series} showing whether or
not the values match.
You can then use this boolean array to select appropriate
locations.
Now let's look at the pandas equivalent of some SQL SELECT statements.
\begin{lstlisting}
>>> # SELECT ID, Age FROM studentInfo
>>> studentInfo[['ID', 'Age']]

>>> # SELECT ID, GPA FROM otherInfo WHERE Financial_Aid = 'y'
>>> otherInfo[otherInfo['Financial_Aid']=='y'][['ID', 'GPA']]

>>> # SELECT Name FROM studentInfo WHERE Class = 'J' OR Class = 'Sp'
>>>  studentInfo[studentInfo['Class'].isin(['J','Sp'])]['Name']
\end{lstlisting}

\begin{problem}
The example above shows how to implement a simple WHERE condition, and it is easy
to have a more complex expression.
Simply enclose each condition by parentheses,
and use the standard boolean operators \li{\&} (AND), \li{\|} (OR), and \li{\~} (NOT) to
connect the conditions appropriately.
Use pandas to execute the following query:
\begin{lstlisting}[language=SQL]
SELECT ID, Name from studentInfo WHERE Age > 19 AND Sex = 'M'
\end{lstlisting}
\end{problem}

Next, let's look at JOIN statements.
In pandas, this is done with the \li{merge} function,
which takes as arguments the two \li{DataFrame} objects to join, as well as keyword arguments specifying
the column on which to join, along with the type (left, right, inner, outer).

\begin{lstlisting}
>>> # SELECT * FROM studentInfo INNER JOIN mathInfo ON studentInfo.ID = mathInfo.ID
>>> pd.merge(studentInfo, mathInfo, on='ID') # INNER JOIN is the default
   Age Class  ID    Name Sex  Grade Math_Major
0   20    Sp   0   Mylan   M    4.0          y
1   21    Se   1   Regan   F    3.0          n
2   22    Se   3    Jess   F    4.0          n
3   20     J   5    Remi   F    3.5          y
4   20     J   6    Matt   M    3.0          n
[5 rows x 7 columns]

>>> # SELECT GPA, Grade FROM otherInfo FULL OUTER JOIN mathInfo ON otherInfo.ID = mathInfo.ID
>>> pd.merge(otherInfo, mathInfo, on='ID', how='outer')[['GPA', 'Grade']]
   GPA  Grade
0  3.8    4.0
1  3.5    3.0
2  3.0    NaN
3  3.9    4.0
4  2.8    NaN
5  2.9    3.5
6  3.8    3.0
7  3.4    NaN
8  3.7    NaN
[9 rows x 2 columns]
\end{lstlisting}

\begin{problem}
Using a join operation, create a \li{DataFrame} containing the ID, age, and GPA of all male individuals.
You ought to be able to accomplish this in one line of code.
\end{problem}

\begin{comment}
This may be going into too much detail for an introductory lab...
It is sometimes desirable to join to DataFrames on their indexes rather than on a particular column.
The \li{join} method makes this operation convenient.
\begin{lstlisting}
>>> #create new DataFrame
>>> sibs = [0, 1, 0, 5, 2, 9]
>>> sibInfo = pd.DataFrame({'Siblings':sibs})
>>> sibInfo
   Siblings
0         0
1         1
2         0
3         5
4         2
5         9
[6 rows x 1 columns]

>>> #now join studentInfo with sibInfo
>>> studentInfo.join(sibInfo)
   Age Class  ID    Name Sex  Siblings
0   20    Sp   0    Mylan   M         0
1   21    Se   1   Regan   F         1
2   18    Fr   2     Justin   M         0
3   22    Se   3   Jess   F         5
4   19    Sp   4     Jason   M         2
5   20     J   5  Remi   F         9
6   20     J   6   Matt   M       NaN
7   19     J   7  Alexander   M       NaN
8   20    Se   8     JeanMarie   F       NaN
[9 rows x 6 columns]
\end{lstlisting}
The default is a left join, but this can be altered.
When attempting to join to DataFrames that share common columns on their indexes,
we must rename the columns to avoid contradictions.
We do this by simply appending
characters to the existing column names, specified by the \li{lsuffix} and \li{rsuffix}
parameters (for the left and right DataFrames, respectively).
\begin{lstlisting}
>>> #join studentInfo and mathInfo on indexes
>>> studentInfo.join(mathInfo, lsuffix='_left', rsuffix='_right', how='inner')
   Age Class  ID_left   Name Sex  Grade  ID_right Math_Major
0   20    Sp        0   Mylan   M    4.0         0          y
1   21    Se        1  Regan   F    3.0         1          n
2   18    Fr        2    Justin   M    3.5         5          y
3   22    Se        3  Jess   F    3.0         6          n
4   19    Sp        4    Jason   M    4.0         3          n
[5 rows x 8 columns]
\end{lstlisting}

The final SQL-like operation we will discuss is the UNION, or concatenation of DataFrames.
In the simplest setting, this operation is useful when we have two DataFrames that have the
same columns but possibly different rows.
\begin{lstlisting}
>>> #create another df holding more math info
>>> mathID2 = [0, 5, 2, 4]
>>> mathGd2 = [4.0, 3.5, 2.0, 3.0]
>>> major2 = ['y', 'y', 'n', 'y']
>>> mathInfo2 = pd.DataFrame({'Grade':mathGd2, 'ID':mathID2, 'Math_Major':major2})
>>> pd.concat([mathInfo, mathInfo2], ignore_index=True).drop_duplicates()
   Grade  ID Math_Major
0    4.0   0          y
1    3.0   1          n
2    3.5   5          y
3    3.0   6          n
4    4.0   3          n
7    2.0   2          n
8    3.0   4          y
[7 rows x 3 columns]
\end{lstlisting}
The \li{ignore_index=True} argument means we discard the indexes of the two input DataFrames and create
a new one for the concatenated DataFrame.
The \li{drop_duplicates} method simply drops duplicate rows.
\end{comment}

Be aware that other types of SQL-like operations are also possible, such as UNION.
When you find yourself unsure of how to carry out a more involved SQL-like operation, the online
pandas documentation will be of great service.

\section*{Analyzing Data}
Although pandas does not provide built-in support for heavy-duty statistical analysis of data, there
are nevertheless many features and functions that facilitate basic data manipulation and computation,
even when the data is in a somewhat messy state.
We will now explore some of these features.

\subsection*{Basic Data Manipulation}
Because the primary pandas data structures are subclasses of the \li{ndarray}, they are valid input
to most NumPy functions, and can often be treated simply as NumPy arrays.
For example, basic
vectorized operations work just fine:
\begin{lstlisting}
>>> x = pd.Series(np.random.randn(4), index=['a', 'b', 'c', 'd'])
>>> y = pd.Series(np.random.randn(5), index=['a', 'b', 'd', 'e', 'f'])
>>> x**2
a    1.710289
b    0.157482
c    0.540136
d    0.202580
dtype: float64
>>> z = x + y
>>> z
a    0.123877
b    0.278435
c         NaN
d   -1.318713
e         NaN
f         NaN
dtype: float64
>>> np.log(z)
a   -2.088469
b   -1.278570
c         NaN
d         NaN
e         NaN
f         NaN
dtype: float64
\end{lstlisting}
Notice that pandas automatically aligns the indexes when adding two \li{Series} (or \li{DataFrames}),
so that the the index of the output is simply the union of the indexes of the two inputs.
The default
missing value \li{NaN} is given for labels that are not shared by both inputs.

It may also be useful to transpose \li{DataFrames}, re-order the columns or rows, or sort according to a
given column.
Here we demonstrate these capabilities:
\begin{lstlisting}
>>> df = pd.DataFrame(np.random.randn(4,2), index=['a', 'b', 'c', 'd'], columns=['I', 'II'])
>>> df
          I        II
a -0.154878 -1.097156
b -0.948226  0.585780
c  0.433197 -0.493048
d -0.168612  0.999194

[4 rows x 2 columns]

>>> df.transpose()
           a         b         c         d
I  -0.154878 -0.948226  0.433197 -0.168612
II -1.097156  0.585780 -0.493048  0.999194

[2 rows x 4 columns]

>>> # switch order of columns, keep only rows 'a' and 'c'
>>> df.reindex(index=['a', 'c'], columns=['II', 'I'])
         II         I
a -1.097156 -0.154878
c -0.493048  0.433197

[2 rows x 2 columns]

>>> # sort descending according to column 'II'
>>> df.sort_values('II', ascending=False)
          I        II
d -0.168612  0.999194
b -0.948226  0.585780
c  0.433197 -0.493048
a -0.154878 -1.097156

[4 rows x 2 columns]
\end{lstlisting}

\begin{comment}
\subsection*{Basic Statistical Functions}

The pandas library allows us to easily calculate basic summary statistics of our data,
useful when we want a quick description of the data.
The \li{describe()} function
outputs several such summary statistics for each column in a \li{DataFrame}:
\begin{lstlisting}
>>> df.describe()
              I        II
count  4.000000  4.000000
mean  -0.209630 -0.001308
std    0.566696  0.964083
min   -0.948226 -1.097156
25%   -0.363516 -0.644075
50%   -0.161745  0.046366
75%   -0.007859  0.689133
max    0.433197  0.999194

[8 rows x 2 columns]
\end{lstlisting}

Functions for calculating means and variances, the covariance and correlation matrices, and other
basic statistics are also available.
Below, we calculate the means of each row, as well as the
covariance matrix:
\begin{lstlisting}
>>> df.mean(axis=1)
a   -0.626017
b   -0.181223
c   -0.029925
d    0.415291
dtype: float64

>>> df.cov()
           I        II
I   0.321144 -0.256229
II -0.256229  0.929456

[2 rows x 2 columns]
\end{lstlisting}
\end{comment}

\subsection*{Dealing with Missing Data} % -------------------------------------

Missing data is a ubiquitous problem in data science.
Fortunately, pandas is particularly well-suited to
handling missing and anomalous data.
As we have already seen, the pandas default for a missing value is \li{NaN}.
In basic arithmetic operations, if one of the operands is \li{NaN}, then the output is also \li{NaN}.
The following example illustrates this concept:
\begin{lstlisting}
>>> x = pd.Series(np.arange(5))
>>> y = pd.Series(np.random.randn(5))
>>> x.iloc[3] = np.nan
>>> x + y
0    0.731521
1    0.623651
2    2.396344
3         NaN
4    3.351182
dtype: float64
\end{lstlisting}

If we are not interested in the missing values, we can simply drop them from the data altogether:

\begin{lstlisting}
>>> (x + y).dropna()
0    0.731521
1    0.623651
2    2.396344
4    3.351182
dtype: float64
\end{lstlisting}

This is not always the desired behavior, however.
It may well be the case that missing data actually corresponds to
some default value, such as zero.
In this case, we can replace all instances of \li{NaN} with a specified value:
\begin{lstlisting}
>>> # fill missing data with 0, add
>>> x.fillna(0) + y
0    0.731521
1    0.623651
2    2.396344
3    1.829400
4    3.351182
dtype: float64
\end{lstlisting}

Other functions, such as \li{<<sum()>>} and \li{mean()} ignore NaN values in the computation.
When dealing with missing data, make sure you are aware of the behavior of the pandas functions you are using.

\section*{Data I/O} % =========================================================

Being able to import and export data is a fundamental skill in data science.
Unfortunately, with the multitude of data formats and conventions, importing data can often be a painful task.
The pandas library seeks to reduce some of the difficulty by providing file readers for various types of formats,
including CSV, Excel, HDF5, SQL, JSON, HTML, and pickle files.

\begin{table}[H]
\begin{tabular}{r|l}
Method & Description \\ \hline
\li{describe()}  & Return a \li{Series} describing the data structure \\
\li{head()}      & Return the first $n$ rows, defaulting to 5 \\
\li{tail()}      & Return the last $n$ rows, defaulting to 5 \\
\li{to_csv()}    & Write the index and entries to a CSV file \\
\li{to_json()}   & Convert the object to a JSON string \\
\li{to_pickle()} & Serialize the object and store it in an external file \\
\li{to_sql()}    & Write the object data to an open SQL database \\
\end{tabular}
\caption{Methods for viewing or exporting data in a pandas \li{Series} or \li{DataFrame}.}
\label{table:pandas-view-or-export}
\end{table}

The CSV (comma separated values) format is a simple way of storing tabular data
in plain text.
Because CSV files are one of the most popular file formats for
exchanging data, we will explore the \li{read_csv()} function in more detail.
To learn to read other types of file formats, see the online pandas
documentation.
To read a CSV data file into a \li{DataFrame}, call the
\li{read_csv()} function with the path to the CSV file, along with the
appropriate keyword arguments.
Below we list some of the most important keyword
arguments:
\begin{itemize}
\item \li{delimiter}:
This argument specifies the character that separates data fields, often a
comma or a whitespace character.

\item \li{header}:
The row number (starting at 0) in the CSV file that contains the column names.

\item \li{index_col}:
If you want to use one of the columns in the CSV file as the index for the
\li{DataFrame}, set this argument to the desired column number.

\item \li{skiprows}:
If an integer $n$, skip the first $n$ rows of the file, and then start reading
in the data.
If a list of integers, skip the specified rows.

\item \li{names}:
If the CSV file does not contain the column names, or you wish to use other
column names, specify them in a list assigned to this argument.

\end{itemize}
There are several other keyword arguments, but this should be enough to get you
started.

When you need to save your data, pandas allows you to write to several different
file formats.
A typical example is the \li{to_csv()} function method attached to
\li{Series} and \li{DataFrame} objects, which writes the data to a CSV file.
Keyword arguments allow you to specify the separator character, omit writing the
columns names or index, and specify many other options.
The code below
demonstrates its typical usage:
\begin{lstlisting}
>>> df.to_csv("my_df.csv")
\end{lstlisting}

\begin{problem} % Crime data.
The file \texttt{crime\_data.csv} contains data on types of crimes committed in the United States from 1960 to 2016.
\begin{itemize}
\item Load the data into a pandas \li{DataFrame}, using the column names in the file and the column titled
``Year'' as the index.
Make sure to skip lines that don't contain data.

\item Insert a new column into the data frame that contains the crime rate by year (the ratio of ``Total'' column
to the ``Population'' column).

\item Plot the crime rate as a function of the year.

\item List the 5 years with the highest crime rate in descending order.

\item Calculate the average number of total crimes as well as burglary crimes between 1960 and 2012.

\item Find the years for which the total number of crimes was below average, but the number of burglaries
was above average.

\item Plot the number of murders as a function of the population.

\item Select the Population, Violent, and Robbery columns for all years in the 1980s, and save
this smaller data frame to a CSV file \texttt{crime\_subset.csv}.
\end{itemize}
\end{problem}

\begin{problem}
In 1912 the RMS \emph{Titanic} sank after colliding with an iceberg.
The file \texttt{titanic.csv} contains data on the incident.
Each row represents a different passenger, and the columns describe various features of the passengers (age, sex, whether or not they survived, etc.)

Start by cleaning the data.
\begin{itemize}
    \item Read the data into a \li{DataFrame}.
    Use the first row of the file as the column labels, but do not use any of the columns as the index.
    \item Drop the columns \li{"Sibsp"}, \li{"Parch"}, \li{"Cabin"}, \li{"Boat"}, \li{"Body"}, and \li{"home.dest"}.
    \item Drop any entries without data in the \li{"Survived"} column, then change the remaining entries to \li{True} or \li{False} (they start as 1 or 0).
    \item Replace null entries in the \li{"Age"} column with the average age.
    \item Save the new \li{DataFrame} as \texttt{titanic\_clean.csv}.
\end{itemize}
Next, answer the following questions.
\begin{itemize}
    \item How many people survived? What percentage of passengers survived?
    \item What was the average price of a ticket? How much did the most expensive ticket cost?
    \item How old was the oldest survivor? How young was the youngest survivor? What about non-survivors?
\end{itemize}

\end{problem}


\def\err{{\mathop{\overline{\mathsf{err}}}}}

\section{AdaBoost}

%\subsection{Boosting}

\begin{frame}
  \frametitle{Boosting Methods}

  \begin{itemize}
    \item Boosting is one of the most powerful learning techniques introduced in the last twenty years.
    \item Combines output of many \structure{\textit{weak} classifiers} to produce a powerful committee.
    \item The most popular boosting algorithm is called \structure{\textit{AdaBoost}} \\
      (Freund and Schapire, 1997).
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Boosting Methods \cont}

  \begin{citeblock}{Definition}

    A \structure{weak classifier} is one whose error rate is only slightly better than random guessing.
  \end{citeblock}
  \pspread

  \structure{Idea of boosting:}

  \begin{itemize}
    \item Sequentially apply the \emph{weak} classifier to repeatedly modified versions of the data.
    \item This produces a sequence of classifiers.
    \item Weighted majority vote yields the final prediction.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Boosting Methods \cont}

  \begin{itemize}
    \item Consider a two-class problem with $y \in \{ -1, +1\}$. \\[.25cm] %\pause
    \item Given a set of observations $\mathcal{D} = \{(\vec{x}_i, y_i); \quad i=1,\ldots,N\}$ and a classifier $G(\vec x)$, the \structure{error rate} on the training sample is:
      \begin{displaymath}
        \err = \frac{1}{N} \sum_{i=1}^N I(y_i \neq G(\vec{x}_i))
      \end{displaymath}
      where $I$ is the \structure{indicator function}.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Boosting Methods \cont}

  \begin{itemize}
    \item Sequentially applied weak classifiers produce a sequence $G_m(\vec x)$ with $m=1,2,\ldots,M$ %\pause
    \item The \structure{combined prediction} is then:
      \begin{displaymath}
        G(\vec x) = \sign \left( \sum_{m=1}^M \alpha_m G_m(\vec x) \right)
      \end{displaymath}
      %\pause
    \item The weighting factors $\alpha_1, \ldots, \alpha_M$ are computed by the boosting algorithm. %\pause
    \item Each $\alpha_m$ weights the output of the corresponding classifier $G_m(\vec x)$.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Boosting Methods}

  \structure{Schematic of boosting procedure:}

  \begin{center}
    \resizebox{.35\linewidth}{!}{
      \input{\texfigdir/boosting_schematic.pstex_t}
    }
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Boosting Methods \cont}

  \structure{Modifications on the data:}

  \begin{itemize}
    \item Each boosting step consists of applying weights $w_1, w_2, \ldots, w_N$ to the training samples. %\pause
    \item Initially, the weights are set to $w_i = 1/N$. %\pause
    \item Thus the first classifier in the sequence is trained the usual way. %\pause
    \item For $m \geq 2$ the weights are individually modified. %\pause
    \item The classifiers $G_m$, with $m \geq 2$ are trained on differently-weighted samples.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Boosting Methods}

  \structure{Weighting scheme:}

  \begin{itemize}
    \item At step $m$ the misclassified observations from $G_{m-1}$ have increased weights.
    \item The weights for correctly classified samples from $G_{m-1}$ have decreased weights.
  \end{itemize}
  \pspread

  \structure{Effects of the weighting scheme:}

  \begin{itemize}
    \item Observations that are difficult to classify correctly get ever-increasing influence.
    \item Each successive classifier is forced to concentrate more on those observations that were misclassified by the previous one.
  \end{itemize}
\end{frame}

\subsection{AdaBoost}

\begin{frame}
  \frametitle{AdaBoost}

  \structure{AdaBoost algorithm:}
%
  \begin{centernss}
    \small
    \resizebox{.85\linewidth}{!}{
      \begin{struktogramm}(100,50)
        \assign{Initialize weights: $w_i =\gets 1/N ~,~i=1, \ldots, N$}
        \assign{$m \gets 1$}
        \until{$ m = M$}
          \assign{\structure{Fit classifier $G_m(\vec x)$ to training data using $\vec w$}}
          \assign{\structure{Compute classification error:}\\
            $ \mathsf{err}_m = \frac{\sum_{i=1}^N w_i I(y_i \neq G_m(x_i))}{\sum_{i=1}^N w_i}$
          }
          \assign{\structure{Compute classifier weights:}\\
            $ \alpha_m = \log \left( \frac{1- \mathsf{err}_m}{\mathsf{err}_m} \right) $
          }
          \assign{\structure{Compute new sample weights:}\\
            $ w_i \leftarrow w_i \exp \left[ \alpha_m I(y_i \neq G_m(x_i)) \right] ~,~ i = 1, \ldots, N $
          }
          \assign{$m \gets m + 1$}
        \untilend
        \assign{\structure{Output:} $G(\vec x) = \sign \left( \sum_{m=1}^M \alpha_m G_m(\vec x) \right) $}
      \end{struktogramm}
    }
  \end{centernss}

%   \begin{center}
%     \small
%     \begin{struktogramm}{10cm}{0.7cm}
%       \BLOCK {Initialize weights: $w_i = 1/N ~,~i=1, \ldots, N$}
%       \BLOCK {Set $m:=1$}
%       \REPEAT {
%         \BLOCK {\structure{Fit classifier $G_m(\vec x)$ to training data using $\vec w$}}
%         \BLOCK {\structure{Compute classification error:}\\
%           $ \mathsf{err}_m = \frac{\sum_{i=1}^N w_i I(y_i \neq G_m(x_i))}{\sum_{i=1}^N w_i}$
%         }
%         \BLOCK {\structure{Compute classifier weights:}\\
%           $ \alpha_m = \log \left( \frac{1- \mathsf{err}_m}{\mathsf{err}_m} \right) $
%         }
%         \BLOCK {\structure{Compute new sample weights:}\\
%           $ w_i \leftarrow w_i \exp \left[ \alpha_m I(y_i \neq G_m(x_i)) \right] ~,~ i = 1, \ldots, N $
%         }
%         \BLOCK {Set $m:= m + 1$}
%       }
%       UNTIL {$ m = M$}
%       \BLOCK {\structure{Output:} $G(\vec x) = \sign \left( \sum_{m=1}^M \alpha_m G_m(\vec x) \right) $}
%     \end{struktogramm}
%   \end{center}
\end{frame}

\begin{frame}
  \frametitle{AdaBoost \cont}

  \structure{Notes:}

  \begin{itemize}
    \item This version of AdaBoost is called \structure{discrete}, because each $G_m(\vec x)$ returns a discrete class label. %\pause
    \item AdaBoost can be modified to return also a real-valued prediction in the interval $[-1,+1]$.  %\pause
    \item Instead of taking just any classifier for $G_m(\vec x)$, that classifier may be used that results in the smallest error at step $m$.
  \end{itemize}
  \pspread

  AdaBoost dramatically increases the performance of even a very weak classifier.
\end{frame}

\begin{frame}
  \frametitle{AdaBoost \cont}

  \structure{Example from J.\ Matas and J.\ \v{S}ochman \\(Centre for Machine Perception, Technical University, Prague):}
%
  \begin{center}
    \resizebox{!}{6.5cm}{
      \input{\texfigdir/adaboost1.pstex_t}
    }
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{AdaBoost \cont}

  \structure{Example from J.\ Matas and J.\ \v{S}ochman \\(Centre for Machine Perception, Technical University, Prague):}

  \begin{center}
    \resizebox{!}{6.5cm}{
      \alt<2->{
        \input{\texfigdir/adaboost3.pstex_t}
      }{
        \input{\texfigdir/adaboost2.pstex_t}
      }
    }
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{AdaBoost \cont}

  \structure{Example from J.\ Matas and J.\ \v{S}ochman \\(Centre for Machine Perception, Technical University, Prague):}
%
  \begin{center}
    \resizebox{!}{6.5cm}{
      \input{\texfigdir/adaboost4.pstex_t}
    }
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{AdaBoost \cont}

  \structure{Example from J.\ Matas and J.\ \v{S}ochman \\(Centre for Machine Perception, Technical University, Prague):}

  \begin{columns}[c, onlytextwidth]
    \begin{column}{0.5\textwidth}
      \begin{figure}
        \centering
        \resizebox{.7\linewidth}{!}{
          \alt<8->{
            \input{\texfigdir/adaboost-classifier.pstex_t}
          }{\alt<7>{
            \input{\texfigdir/adaboost-classifier7.pstex_t}
          }{\alt<6>{
            \input{\texfigdir/adaboost-classifier6.pstex_t}
          }{\alt<5>{
            \input{\texfigdir/adaboost-classifier5.pstex_t}
          }{\alt<4>{
            \input{\texfigdir/adaboost-classifier4.pstex_t}
          }{\alt<3>{
            \input{\texfigdir/adaboost-classifier3.pstex_t}
          }{\alt<2>{
            \input{\texfigdir/adaboost-classifier2.pstex_t}
          }{
            \input{\texfigdir/adaboost-classifier1.pstex_t}
          }}}}}}}
        }
        \caption{\footnotesize Final classifier (based on a perceptron).}
      \end{figure}
    \end{column}\begin{column}{0.5\textwidth}
      \begin{figure}
        \centering
        \resizebox{\linewidth}{!}{
          \alt<8->{
            \input{\texfigdir/adaboost-error.pstex_t}
          }{\alt<7>{
            \input{\texfigdir/adaboost-error7.pstex_t}
          }{\alt<6>{
            \input{\texfigdir/adaboost-error6.pstex_t}
          }{\alt<5>{
            \input{\texfigdir/adaboost-error5.pstex_t}
          }{\alt<4>{
            \input{\texfigdir/adaboost-error4.pstex_t}
          }{\alt<3>{
            \input{\texfigdir/adaboost-error3.pstex_t}
          }{\alt<2>{
            \input{\texfigdir/adaboost-error2.pstex_t}
          }{
            \input{\texfigdir/adaboost-error1.pstex_t}
          }}}}}}}
        }
        \caption{\footnotesize Reduction of the classification error w.\,r.\,t.\ the sequence length.}
      \end{figure}
  \end{column}
  \end{columns}
\end{frame}

\input{nextTime.tex}

\subsection{Exponential Loss}

\begin{frame}
  \frametitle{Exponential Loss}

  \begin{itemize}
    \item Boosting fits an \structure{additive model} in a \structure{set of elementary basis functions}: \\[.1cm]
      \begin{center}
        \tikz[baseline]{
          \node[fill=bl1!100, anchor=base, rounded corners=3pt, inner sep=1mm] (d1) {
            \color{bl3}
            $
              \displaystyle
              f_M(\vec x) =
              \sum_{m=1}^M \beta_m \, b(\vec x ; \vec \gamma_m)
            $
          };
        }
      \end{center}
      \vspace{.1cm}
      where $\beta_m$ are expansion coefficients and $b(\vec x ; \vec \gamma_m)$ is a basis function given a set of parameters $\vec \gamma_m$. \\[.3cm] \pause
    \item Additive expansions are very popular in learning techniques:
      \begin{itemize}
        \item single-hidden-layer neural networks (perceptron)
        \item wavelets
        \item classification trees
        \item etc.
      \end{itemize}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Exponential Loss \cont}

  \begin{itemize}
    \item Expansion models are typically fit by \structure{minimizing a loss function $L$} averaged over the training data:
      \begin{equation}
        \label{eq:loss}
        \min_{\{ \beta_m, \vec \gamma_m \}_1^M } ~
        \left\{
          \sum_{i=1}^N L \big( y_i, f_M(\vec{x}_i) \big) =
          \sum_{i=1}^N L \left( y_i, \sum_{m=1}^M \beta_m \, b(\vec{x}_i ; \vec \gamma_m) \right)
        \right\}
      \end{equation}
      \\[.3cm] \pause
    \item \structure{Forward stagewise modeling} approximates the solution to \eqref{eq:loss}: \\[.1cm]
      \begin{itemize}
        \item New basis functions are sequentially added. \\[.1cm] %\pause
        \item Parameters and coefficients of already added functions are not changed. \\[.1cm] %\pause
        \item At each iteration, the subproblem of fitting just a single basis function is solved.
      \end{itemize}
%       \begin{displaymath}
%         \min_{\beta, \vec \gamma} ~\sum_{i=1}^N L(y_i, \beta b(\vec{x}_i ; \vec{\gamma}))
%        \end{displaymath}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Exponential Loss \cont}

  \begin{itemize}
    \item The \structure{$m$-th subproblem} may be rewritten as: \\[.3cm]
      \begin{center}
        \tikz[baseline]{
          \node[fill=bl1!100, anchor=base, rounded corners=3pt, inner sep=1mm] (d1) {
            \color{bl3}
            $
              \displaystyle
              (\beta_m, \vec \gamma_m) = \argmin_{\beta, \vec \gamma} ~\sum_{i=1}^N L\big(y_i, f_{m-1}(\vec{x}_i) + \beta \, b(\vec{x}_i ; \vec{\gamma})\big)
            $
          };
        }
      \end{center}
      \vspace{.3cm} \pause
    \item AdaBoost is equivalent to a forward stagewise additive modeling using an \structure{exponential loss function}:
      \begin{displaymath}
        L(y, f(\vec x)) = \exp (-y \, f(\vec x))
      \end{displaymath}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Exponential Loss \cont}

  \structure{Proof:}

  \begin{itemize}
    \item For AdaBoost, the basis functions are the classifiers $G_m(\vec x) \in \{-1,+1\}$. %\pause
    \item Using the \structure{exponential loss function}, one must solve at each step:
      \begin{eqnarray*}
        (\beta_m, G_m) = \argmin_{\beta, G} ~\sum_{i=1}^N \exp \left[ -y_i \big(f_{m-1}(\vec{x}_i) + \beta G(\vec{x}_i)\big) \right]
      \end{eqnarray*}
      %\pause
    \item Using the weight $w_i^{(m)} = \exp(-y_i \, f_{m-1}(\vec{x}_i))$ this can be rewritten as:
      \begin{eqnarray*}
        (\beta_m, G_m) = \argmin_{\beta, G} ~\sum_{i=1}^N w_i^{(m)} \exp (-\beta y_i G(\vec{x}_i))
      \end{eqnarray*}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Exponential Loss \cont}

  \structure{Observations:}

  \begin{itemize}
    \item Since $w_i^{(m)}$ is independent of $\beta$ and $G(\vec x)$, it can be seen as a weight applied to each observation. %\pause
    \item However, this weight depends on $f_{m-1}$, so the weights change with \\
      each iteration $m$.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Exponential Loss \cont}

  \vspace{-.5cm}
  \begin{eqnarray*}
    (\beta_m, G_m)
      &=& \argmin_{\beta, G} ~\sum_{i=1}^N w_i^{(m)} \exp (-\beta y_i G(\vec{x}_i)) \\ %\pause
      &=& \argmin_{\beta, G} ~ e^{-\beta} \sum_{y_i = G(\vec{x}_i)} w_i^{(m)} + e^{\beta} \sum_{y_i \neq G(\vec{x}_i)} w_i^{(m)} \\ %\pause
      &=& \argmin_{\beta, G} ~ \left( e^{\beta} - e^{-\beta} \right) \sum_{i=1}^N w_i^{(m)} I(y_i \neq G(\vec{x}_i)) + e^{-\beta} \sum_{i=1}^N w_i^{(m)}
  \end{eqnarray*}
  \pause

  \begin{itemize}
    \item Thus, for any value $\beta > 0$ the solution for $G_m(\vec x)$ is: \\[.25cm]
      \begin{center}
        \tikz[baseline]{
          \node[fill=bl1!100, anchor=base, rounded corners=3pt, inner sep=1mm] (d1) {
            \color{bl3}
            $
              \displaystyle
              G_m = \argmin_G \sum_{i=1}^N w_i^{(m)} \, I(y_i \neq G(\vec{x}_i))
            $
          };
        }
      \end{center}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Exponential Loss \cont}

  \begin{itemize}
    \item Plugging the reformulated $G_m(\vec x)$ into the objective function and solving for $\beta_m$ yields: \\[.15cm]
      \begin{center}
        \tikz[baseline]{
          \node[fill=bl1!100, anchor=base, rounded corners=3pt, inner sep=1mm] (d1) {
            \color{bl3}
            $
              \displaystyle
              \beta_m = \frac{1}{2} \log \frac{1 - \mathsf{err}_m}{\mathsf{err}_m}
            $
          };
        }
      \end{center}
      \vspace{.25cm}

      with $\mathsf{err}_m$ being the \structure{minimized weighted error rate}: \\[.15cm]

      \begin{displaymath}
        \mathsf{err}_m = \frac{\sum_{i=1}^N w_i^{(m)} I(y_i \neq G_m(\vec{x}_i))}{\sum_{i=1}^N w_i^{(m)}}
      \end{displaymath}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Exponential Loss \cont}

  \begin{itemize}
    \item From the update formula of the approximation:
      \begin{eqnarray*}
        f_m(\vec x) = f_{m-1}(\vec x) + \beta_m \, G_m(\vec x)
      \end{eqnarray*}
      we can calculate the \structure{weights for the next iteration}:
      \begin{eqnarray*}
        w_i^{(m+1)}
          &=& w_i^{(m)} ~ e^{-\beta_m y_i G_m(\vec{x}_i)} \\ \vspace{1cm} %\pause
          & & \mbox{\small(using: $-y_i G_m(\vec{x}_i) = 2 I (y_i \neq G_m(\vec{x}_i)) - 1$)} \\[.5cm] %\pause
          &=& w_i^{(m)} e^{\alpha_m I(y_i \neq G_m(\vec{x}_i))} e^{-\beta_m}
      \end{eqnarray*}
      with $\alpha_m = 2 \beta_m$.
  \end{itemize}

  \hfill \qed
\end{frame}

\begin{frame}
  \frametitle{Exponential Loss \cont}

  \structure{Compare result to initial AdaBoost algorithm:}

  \begin{itemize}
    \item \structure{Exponential loss:}
      \begin{eqnarray*}
        \beta_m     &=& \frac{1}{2} \log \frac{1 - \mathsf{err}_m}{\mathsf{err}_m} \\
        \alpha_m    &=& 2 \beta_m \\
        w_i^{(m+1)} &=& w_i^{(m)} ~ e^{\alpha_m I(y_i \neq G_m(\vec{x}_i))} ~ e^{-\beta_m}
      \end{eqnarray*}
      \pspread
    \item \structure{AdaBoost:}
      \begin{eqnarray*}
        \alpha_m &=&          \log \frac{1- \mathsf{err}_m}{\mathsf{err}_m} \\
        w_i      &\leftarrow& w_i \, e^{\alpha_m I(y_i \neq G_m(\vec{x}_i))}
      \end{eqnarray*}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Exponential Loss \cont}

  \begin{center}
    \resizebox{.85\linewidth}{!}{
      \alt<4->{
        \input{\texfigdir/exploss4.pstex_t}
      }{\alt<3>{
        \input{\texfigdir/exploss3.pstex_t}
      }{\alt<2>{
        \input{\texfigdir/exploss2.pstex_t}
      }{
        \input{\texfigdir/exploss1.pstex_t}
      }}}
    }
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Exponential Loss \cont}

  \structure{Discussion:}

  \begin{itemize}
    \item The equivalence of AdaBoost to forward stagewise additive modeling was discovered five years after its invention. %\pause
    \item The AdaBoost criterion yields a monotone decreasing function of the \structure{margin}: $y\,f(\vec x)$. \\[.5cm] %\pause
    \item In $\{-1, 1\}$ classification, the margin plays a role similar to the \\
      residuals $y - f(\vec x)$ in regression:
      \begin{itemize}
        \item Observations with $y_i \, f(\vec{x}_i) > 0$ are classified correctly
        \item Observations with $y_i \, f(\vec{x}_i) < 0$ are misclassified
        \item The decision boundary is at $f(\vec x) = 0$
     \end{itemize}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Exponential Loss \cont}

  \structure{Discussion (cont.):}

  \begin{itemize}
    \item The goal of the classification algorithm is to produce positive margins as frequently as possible. %\pause
    \item Thus, any loss criterion should penalize negative margins more heavily than positive ones! %\pause
    \item The exponential criterion concentrates much more influence on observations with large negative margins.
  \end{itemize}
  \pspread

  Due to the exponential loss, AdaBoost performance is known to degrade rapidly:
  \begin{itemize}
    \item in situations of noisy data
    \item when there are wrong class labels in the training data
  \end{itemize}
\end{frame}

\input{nextTime.tex}

%\subsection{Face Detection}

\begin{frame}
  \frametitle{Face Detection}

  Famous algorithm developed by \structure{Viola and Jones} in 2001.
  \spread

  \structure{Contributions:}

  \begin{itemize}
    \item Integral images for feature computation.
    \item Usage of AdaBoost for boosting.
    \item Classifier cascade for fast rejection of non-face regions.
  \end{itemize}
  \spread

  Various implementations available: for example look into \textit{OpenCV FaceDetector} sample code.
\end{frame}

\begin{frame}
  \frametitle{Face Detection \cont}

  \structure{Features:}

  \begin{itemize}
    \item Features are adapted from Haar basis functions. %\pause
    \item They are calculated by subtracting the sum of the pixels in the white from the sum of the pixels in the gray rectangles: \\[.3cm] %\pause
  \end{itemize}

  \structure{4 types of features used:}

  \begin{center}
    \resizebox{.4\linewidth}{!}{
      \input{\texfigdir/haar_features.pstex_t}
    }
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Face Detection \cont}

  \begin{itemize}
    \item The exhaustive set of features is very large: $> 45\,000$ for a 380x280 image. %\pause
    \item Rectangle features can be efficiently computed using an \\
     \structure{Integral Image $II$}:
      \begin{displaymath}
        II(x,y) = \sum_{x' \leq x, y' \leq y} I(x', y')
      \end{displaymath}
  \end{itemize}
  \spread

  \begin{center}
    \resizebox{.4\linewidth}{!}{
      \input{\texfigdir/integral_image.pstex_t}
    }
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Face Detection \cont}

  \structure{Computation of the sum of intensities for any rectangle D  with just \\ 3 basis operations based on the integral image:} \\[.5cm]

  \begin{center}
    \resizebox{.4\linewidth}{!}{
      \input{\texfigdir/integral_image2.pstex_t}
    }
  \end{center}

  \begin{itemize}
    \item Value at 1: A %\pause
    \item Value at 2: A + B %\pause
    \item Value at 3: A + C %\pause
    \item Value at 4: A + B + C + D %\pause
    \item Sum within D: $4 + 1 - (2 + 3)$
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Face Detection \cont}

  \structure{Boosting:}

  \begin{itemize}
    \item The classification functions are restricted to each depend on a single feature only. %\pause
    \item This way, AdaBoost can be interpreted as an effective feature selecting algorithm:
      \begin{itemize}
        \item The single rectangle feature is selected which best separates the observations. %\pause
        \item For each feature, the weak learner determines the optimal threshold classification function.
      \end{itemize}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Face Detection \cont}

  \begin{itemize}
    \item A weak classifier $G_j(\vec x)$ consists of a \structure{feature $x_j$}, an \structure{optimal threshold $\theta_j$}, and a \structure{parity $s_j$} to indicate the direction of the inequality:
      \begin{displaymath}
        G_j(\vec x) = \begin{cases}
                        1 \quad \mbox{if} ~s_j f_j(\vec x) < s_j \theta_j \\
                        0 \quad \mbox{otherwise}
                      \end{cases}
      \end{displaymath}
      with $\vec x$ being a sub-window of an image.
      \pspread
    \item Very aggressive process to discard the vast majority of features.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Face Detection \cont}

  \begin{itemize}
    \item Best feature selected by AdaBoost:
  \end{itemize}

  \begin{center}
    \begin{figure}
            \copyrightbox[b]{
      \makebox[.6\linewidth]{
          \includegraphics[width=0.4\linewidth]{\pngdir/bestfeature.\png}
      }
    }{Christian Hacker, Anton Batliner, and Elmar Noeth. \href{https://www5.informatik.uni-erlangen.de/Forschung/Publikationen/2006/Hacker06-AYL.pdf}{Are You Looking at Me, Are You Talking with Me: Multimodal Classification of the Focus of Attention.}}
  \end{figure}
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Face Detection \cont}

  \structure{Classifier Cascade:}

  \begin{itemize}
    \item Despite the efficiency of the feature computation, the complexity of the problem is still huge. %\pause
    \item Evaluating the full AdaBoost sequence on all sub-windows of the image takes too much time.
    \pspread
    \item Idea of cascaded classifiers:
    \begin{itemize}
      \item Simpler classifiers used to reject the majority of sub-windows. %\pause
      \item Each stage is again created by AdaBoost. %\pause
      \item Stage 1 uses the two features shown before; detects 100\,\% faces with \\ false positive rate (FPR) of around 40\,\%.
    \end{itemize}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Face Detection \cont}

  \begin{itemize}
    \item The cascade has the overall shape of a degenerated decision tree: \\[.5cm]
      \begin{center}
        \resizebox{.7\linewidth}{!}{
          \input{\texfigdir/cascade.pstex_t}
        }
      \end{center}
      \pspread
    \item A negative classification at any stage yields an immediate rejection of the sub-window.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Face Detection \cont}

  \begin{itemize}
    \item Structure of cascade reflects the (usually) overwhelming majority of negative sub-windows in an image. %\pause
    \item Goal is to reject as many negatives as possible at the earliest stage of the processing.
  \end{itemize}
  \spread

  \structure{Training of the cascade:}

  \begin{itemize}
    \item Subsequent classifiers are trained using only those examples which pass through all the previous stages. %\pause
    \item The next classifier faces a more difficult task than the previous one. %\pause
    \item Trade-off between more features achieving higher detection rates and lower false positive rates while requiring more computation time.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Face Detection \cont}

  \structure{Final classifier:}

  \begin{itemize}
    \setlength\itemsep{0.5cm}
    \item 32 layer cascade of increasingly strong boosted classifiers

    \begin{center}
      \begin{tabular}{r@{}l r c c}
        \multicolumn{2}{c}{\structure{Layer}} &
        \structure{\# Features} &
        \structure{TPR} &
        \structure{FPR}\\
        \hline
           &1  &   2~~~~~ & 100\,\% & 40\,\% \\
           &2  &   5~~~~~ & 100\,\% & 20\,\% \\
         3-&5  &  20~~~~~ & \\
         6-&7  &  50~~~~~ & \\
         8-&12 & 100~~~~~ & \\
        13-&32 & 200~~~~~ & \\ %\pause
      \end{tabular}
    \end{center}

    \item Number of stages and features adapted until false positive rate on validation was nearly zero while maintaining high correct rate.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Face Detection \cont}

  \structure{Example images from the training set:}

  \begin{center}
  \begin{figure}
            \copyrightbox[b]{
      \makebox[.6\linewidth]{
          \includegraphics[width=0.45\linewidth]{\pngdir/FaceTraining.\png}
      }
    }{Paul Viola and Michael Jones. \href{https://www.researchgate.net/publication/3940582_Rapid_Object_Detection_using_a_Boosted_Cascade_of_Simple_Features}{Rapid Object Detection using a Boosted Cascade of Simple Features. IEEE Conf Comput Vis Pattern Recognit.}}
  \end{figure}
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Face Detection \cont}

  \structure{Results on test set:}
  \begin{center}
  \begin{figure}
            \copyrightbox[b]{
      \makebox[.6\linewidth]{
          \includegraphics[width=0.5\linewidth]{\pngdir/facedetection.\png}
      }
    }{Christian Hacker, Anton Batliner, and Elmar Noeth. \href{https://www5.informatik.uni-erlangen.de/Forschung/Publikationen/2006/Hacker06-AYL.pdf}{Are You Looking at Me, Are You Talking with Me: Multimodal Classification of the Focus of Attention.}}
  \end{figure}
  \end{center}
\end{frame}

%\subsection{Lessons Learned}

\begin{frame}
  \frametitle{Lessons Learned}

  \begin{itemize}
    \item Boosting of weak classifiers can yield a powerful combined classifier \\[.3cm]
    \item AdaBoost algorithm \\[.3cm]
    \item Weights of the classifiers and on the data are adjusted \\[.3cm]
    \item AdaBoost minimizes an exponential loss function \\[.3cm]
    \item Viola \& Jones face detection algorithm
  \end{itemize}
\end{frame}

\subsection{Further Reading}

\begin{frame}
  \frametitle{Further Reading}

  Most of the content of this lecture has been taken from: \\[.25cm]

  \small
  \begin{itemize}
    \item T.\ Hastie, R.\ Tibshirani, J.\ Friedman: \\
      \structure{The Elements of Statistical Learning}, \\
      2nd Edition, Springer, 2009.\\%[.15cm]
    \item Y.\ Freund, R.\,E.\ Schapire: \\
      \structure{A decision-theoretic generalization of on-line learning and \\
	    an application to boosting},\\
      Journal of Computer and System Sciences, 55(1):119-139, 1997.\\%[.15cm]
    \item P.\,A.\ Viola, M.\,J.\ Jones:
      \structure{Robust Real-Time Face Detection}, \\
      International Journal of Computer Vision 57(2): 137-154, 2004.\\%[.15cm]
    \item J.\ Matas and J.\ \v{S}ochman:
      \structure{AdaBoost}, \\
      Centre for Machine Perception, Technical University, Prague. \\%[.15cm]
      \structure{\point\url{http://cmp.felk.cvut.cz/~sochmj1/adaboost_talk.pdf}}
  \end{itemize}
\end{frame}


% Document Head
\documentclass[11pt, oneside]{book}
\usepackage{geometry}
\geometry{letterpaper}
\usepackage[parfill]{parskip}
\usepackage{graphicx}

% Essential Packages
\usepackage{ragged2e}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{mathrsfs}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage[hyperref]{ntheorem}

% Theorem Style Customization
\setlength\theorempreskipamount{2ex}
\setlength\theorempostskipamount{3ex}

% hyperref Package Settings
\usepackage{hyperref}
\hypersetup{
	colorlinks = true,
	linkcolor = magenta
}

% ntheorem Declarations
\theoremstyle{break}
\newtheorem{thm}{Theorem}[section]
\newtheorem*{proof}{Proof}
\newtheorem{crly}{Corollary}[thm]
\newtheorem{lemma}[thm]{Lemma}
\newtheorem{propo}{Proposition}[section]
\newtheorem*{remark}{Remark}
\newtheorem*{note}{Note}
\newtheorem{defn}{Definition}[section]
\newtheorem{eg}{Example}[section]
\newtheorem{axiom}{Axiom}[chapter]
\newtheorem{principle}{Principle}[chapter]

% ntheorem listtheorem style
\makeatletter
\def\thm@@thmline@name#1#2#3#4{%
        \@dottedtocline{-2}{0em}{2.3em}%
                   {\makebox[\widesttheorem][l]{#1 \protect\numberline{#2}}#3}%
                   {#4}}
\@ifpackageloaded{hyperref}{
\def\thm@@thmline@name#1#2#3#4#5{%
    \ifx\#5\%
        \@dottedtocline{-2}{0em}{2.3em}%
            {\makebox[\widesttheorem][l]{#1 \protect\numberline{#2}}#3}%
            {#4}
    \else
        \ifHy@linktocpage\relax\relax
            \@dottedtocline{-2}{0em}{2.3em}%
                {\makebox[\widesttheorem][l]{#1 \protect\numberline{#2}}#3}%
                {\hyper@linkstart{link}{#5}{#4}\hyper@linkend}%
        \else
            \@dottedtocline{-2}{0em}{2.3em}%
                {\hyper@linkstart{link}{#5}%
                  {\makebox[\widesttheorem][l]{#1 \protect\numberline{#2}}#3}\hyper@linkend}%
                    {#4}%
        \fi
    \fi}
}
\makeatother
\newlength\widesttheorem
\AtBeginDocument{
  \settowidth{\widesttheorem}{Proposition A.1.1.1\quad}
}

\theoremlisttype{allname}

% Shortcuts
\newcommand{\bb}[1]{\mathbb{#1}}			% using bb instead of mathbb
\newcommand{\floor}[1]{\lfloor #1 \rfloor}	% simplifying the writing of a floor function
\newcommand{\ceiling}[1]{\lceil #1 \rceil}	% simplifying the writing of a ceiling function
\newcommand{\dotp}{\, \cdotp}				% dot product to distinguish from \cdot

% Main Body
\title{MATH145 Readings}
\author{Johnson Ng}

\begin{document}
\maketitle

\chapter*{Introduction}
\textit{Below is an introduction written by the author of the referred notes, Kenneth R. Davidson.}

As a student entering university to study mathematics, you probably have encountered prime bumbers. Chances are great that you believe that every integer factors uniquely into a product of primes but have not seen a proof. This important fact, known as the Fundamental Theorem of Algebra, is of crucial importance in the theory of numbers. It is not easy to prove. More importantly, it is not \textit{intuitively obvious}. Indeed, its significance is only realized with very large numbers beyond our real experience. The crucial fact that enables us to prove this with relative ease is the Euclidean Algorithm for finding greatest common divisors. The first two chapters deal with these basic properties of the integers and modular arithmetic.

It is worth noting that there are number systems not very much different from the intergers in which this unique factorization into primes fails. Far from being a disaster, this is an opportunity to investigate why this phenomena occurs. It shows us which properties of the integers themsevles are crucial to make the theory work. That is why we make a foray into quadratic number domains in chapter 3. Naturally, we merely touch the surface here as this theory lies in a rather deep connection between algebra and number theory.

A nice application of modular arithmetic is the Rivest-Shamir-Adelman (RSA) public key cryptography scheme. This code allows the author to publish the method of \textit{encoding} a message in a public place, while keeping the method of \textit{decoding} the message secret. This is a rather different idea of coding, as for all previously known codes, the method of decoding merely reversed the endoing method. The secret here is that it is very easy (with a computer) to find large primes (say 100-200 digits) but very difficult to factor the product of two large primes. If you believe that checking whether a number is prime involes trial division by all numbers up to the square root, it might be hard to imagine why determining if a number if prime should be any easier than finding factors. So we delve more deeply into methods used to determine whether a number is prime or composite. It quickly becomes clear that simple tests can determine beyond any doubt that a number is composite without giving any information at all about the factors.

In chapter 5, we introduce the complex numbers. There is a tacit assumption that the student is already reasionably familiar with the real numbers from studying calculus. However, a section is devoted to a brief discussion of how the real numbers are developed. No attempt is made to separate algebra from analysis. Indeed, all of mathematics is integrated and it is foolish to pretend that there are natural walls. In particular, we prove the Fundamental Theorem of Algebra that every complex polynomial factors into a product of linear terms. In spite of its name, this is a theorem of analysis because it relies on the (topological) completeness of the complex numbers. The proof we give is one of the simplest, and relies on the Extreme Value Theorem. We also develop the complex exponential function. This again is really a theorem of analysis. But it is included because it plays such an important role in other applications of the compelx numbers.

In chapter 6, we show that the same theory applies to the algebra of polynomials. In particular, there is a Euclidean Algorithm and unique factorization into irreducible polynomials. We exanime various tests for irreducibility, and study connections with irrationality of the roots. Then we do a few special topics about real and complex polynomials such as Sturm's Theorem for counting real roots, and the formula for solving cubics. Then in chapter 7, we study finite fields in some detail. This is just doing modular arithmetic modulo an irreducible polynomial instead of modulo a prime integer. Many of the results for $\bb{Z}_p$ carry over to finite fields with the same ideas. A rather beautiful application of these ideas is an algorithm for factoring polynomials over the rationals. This algorithm is based on a method for factoring polynomial of degree \textit{d} mod \textit{p} is much easeier than factoring a \textit{d} digit base \textit{p} number.

\tableofcontents

\chapter*{List of Definitions}
\theoremlisttype{all}
\listtheorems{defn}

\chapter*{List of Theorems}
\theoremlisttype{all}
\listtheorems{lemma,thm,crly,propo}

\chapter*{Axioms and Principles}
\listtheorems{axiom,principle}

% =================================
% 		The Integers
% =================================
\chapter{The Integers}

\section{Basic Properties}

The following list of properties is what a student taught in the `modern' style would come up with when talking about properties of integers. We give the name Commutative Rings for sets of numbers that satisfy these properties.

\begin{defn}[Commutative Rings]\label{defn-commutative-rings}
	A commutative ring is a set of numbers that satisfy these properties.
	\begin{enumerate}
		\item The \textbf{integers} consist of a set $\bb{Z}$ together with two binary operations \textbf{addition} (+) and \textbf{multiplication} ($\cdot$)
		\item (\textbf{commutativity of addition}) $\forall a, b \in \bb{Z} \quad a + b = b + a$
		\item (\textbf{associativity of addition}) $\forall a, b, c \in \bb{Z} \quad (a + b) + c = a + (b + c)$
		\item (\textbf{additive identity}) $\exists 0 \in \bb{Z} \enspace \forall a \in \bb{Z} \quad a + 0 = a = 0 + a$
		\item (\textbf{additive inverse}) $\forall a \in \bb{Z} \enspace \exists (-a) \in \bb{Z} \quad a + (-a) = 0$
		\item (\textbf{commutativity of multiplication}) $\forall a, b \in \bb{Z} \quad a \cdot b = b \cdot a$
		\item (\textbf{associativity of multiplication}) $\forall a, b, c \in \bb{Z} \quad (a \cdot b) \cdot c = a \cdot (b \cdot c)$
		\item (\textbf{multiplicative identity}) $\exists 1 \in \bb{Z} \enspace \forall a \in \bb{Z} \quad a \cdot 1 = a = 1 \cdot a$
		\item (\textbf{distributive law}) $\forall a, b, c \in \bb{Z} \enspace (a + b) \cdot c = a \cdot c + b \cdot c$
	\end{enumerate}
\end{defn}

But this does not fully distinguish integers from many other sets, for example:
\begin{enumerate}
	\item The real numbers $\bb{R}$.
	\item The set $\bb{Z}[\sqrt{3}] = \{a + b\sqrt{3} \; | \; a, b \in \bb{Z}\}$ \\
		See Exercise 1.1 for proof.
	\item The set $\bb{Z} \oplus \bb{Z} := \{(a, b) \, | \, a, b \in \bb{Z}\}$ with coordinate-wise addition and multiplication.
\end{enumerate}

\begin{defn}[Ring]
	A ring is a set of numbers that satisfy all properies of a commutative ring except for property 6, \textbf{commutativity of multiplication}, and added with an additional distributive law
	\begin{equation*}
		\forall a, b, c \in \bb{Z} \quad a \cdot (b + c) = a \cdot b + a \cdot c
	\end{equation*}
\end{defn}

\begin{eg}
	An example of a non-commutative ring is the set of $2 \times 2$ matrices with integer entries. While addition is coordinate-wise, multiplication is defined by the rule: $\forall a, b, c, d, w, x, y, z \in \bb{Z}$
	\begin{equation*}
		\begin{bmatrix}
			a & b \\
			c & d
		\end{bmatrix}
		\begin{bmatrix}
			w & x \\
			y & z
		\end{bmatrix}
		=
		\begin{bmatrix}
			aw + by & ax + bz \\
			cw + dy & cx + dz
		\end{bmatrix}
	\end{equation*}
	which is clearly non-commutative since
	\begin{equation*}
		\begin{bmatrix}
			w & x \\
			y & z
		\end{bmatrix}
		\begin{bmatrix}
			a & b \\
			c & d
		\end{bmatrix}
		=
		\begin{bmatrix}
			aw + cx & bw + dx \\
			ay + cz & by + dz
		\end{bmatrix}
	\end{equation*}
\end{eg}

\begin{eg}
	Another important example is the set of integers \textbf{modulo} n. Consider the ring $\bb{Z}_2$ consisting of two elements $\{0, 1\}$ with operations given by the tables: \\
	\begin{center}
		\begin{tabular}{c|c c}
			$+$	&	0	&	1 \\
			\hline
			0	&	0	&	1 \\
			1	&	1	&	0
		\end{tabular}
		$\quad$
		\begin{tabular}{c|c c}
			$\times$	&	0	&	1 \\
			\hline
			0			&	0	&	0 \\
			1			&	0	&	1
		\end{tabular}
	\end{center}
\end{eg}

The above example gives us a clue about how to add further properties to our list in Definition \ref{defn-commutative-rings} to ensure that the integers become our only example.

For instance, we declare the following property so as to distinguish many other commutative rings from the integers.

\begin{enumerate}
	\setcounter{enumi}{9}
	\item $\bb{Z}$ is generated by $\{0, 1\}$
\end{enumerate}

With this, we have laid the foundations of starting our set starting with 0 and 1, then form the rest of the elements of $\bb{Z}$ so as to provide the \textit{minimal} collection satisfying all our properties. In particular, since a ring is \textit{closed} under addition and multiplication, we need all the numbers of the form $1, 1 + 1, 1 + 1 + 1, 1 + 1 + 1 + 1, ...$.

Note that such a construction ensures that the set is indeed closed under addition and multiplication. The set is closed under addition since
\begin{gather*}
	\forall a, b \in \bb{Z} \\
	a + b = \underbrace{\underbrace{1 + 1 + \hdots + 1}_\text{a many 1's} + \underbrace{1 + 1 + \hdots + 1}_\text{b many 1's}}_\text{(a + b) many 1's}
\end{gather*}
which thus implies that $a + b \in \bb{Z}$ since a ring is closed under addition. The set is closed under multiplication since
\begin{align*}
	a \cdot b &= (\underbrace{1 + 1 + \hdots + 1}_\text{a many 1's}) \cdot (\underbrace{1 + 1 + \hdots + 1}_\text{b many 1's}) \\
			  &= \underbrace{1 \cdot (\underbrace{1 + 1 + \hdots + 1}_\text{b many 1's}) + 1 \cdot (\underbrace{1 + 1 + \hdots + 1}_\text{b many 1's}) + \hdots + 1 \cdot (\underbrace{1 + 1 + \hdots + 1}_\text{b many 1's})}_\text{a times}
\end{align*}
by the distributive law, and since each $1 \cdot (\underbrace{1 + 1 + \hdots + 1}_\text{b many 1's}) \in \bb{Z}$, we have, once again, that
\begin{equation*}
	a \cdot b = \underbrace{1 \cdot (\underbrace{1 + 1 + \hdots + 1}_\text{b many 1's}) + 1 \cdot (\underbrace{1 + 1 + \hdots + 1}_\text{b many 1's}) + \hdots + 1 \cdot (\underbrace{1 + 1 + \hdots + 1}_\text{b many 1's})}_\text{a times} \quad \in \bb{Z}
\end{equation*}
and thus our construction is closed under multiplication.

Next, in order to satisfy the \textbf{additive inverse property}, we \textit{may} have to add in $-1, -(1 + 1), -(1 + 1 + 1), ...$ The proof that such a collection is enough to satisfy all the properties of a commutative ring is similar to our exercise above.

Our legwork is not done yet, since we still have, for example, $\bb{Z}_2$ also being generated by its 0 and 1. We can, fortunately, eliminate this group of sets by stating that $1, 1 + 1, 1 + 1 + 1, ...$ are all unique. If such a property holds in any ring, then the collection $S = \{1, 1 + 1, 1 + 1 + 1, ...\}$ will be indistinguishable from the \textbf{natural numbers $\bb{N}$} (without the 0) by any mathematical property. In fact, to ensure that all the elements are distinct, it is important that none of them are 0, since for example:
\begin{equation*}
	0 = 1 + 1 + 1 + 1 \implies (1 + 1 + 1 + 1) + 1 = 1 \; \text{but} \; (1 + 1) + (1 + 1 + 1) = 5
\end{equation*}
by the distributive law.

With that said, then $-S \cap S = \emptyset$, and thus $R = S \cup \{0\} \cup -S$ is a ring that has indistinguishable properties from $\bb{Z}$. To clarify the explanation above, we state the following as a property:

\begin{enumerate}
	\setcounter{enumi}{10}
	\item No nontrivial sum of 1's is equal to 0.
\end{enumerate}

Some questions come into attention here (and will be hinted in Exercise 1.4 below):
\begin{enumerate}
	\item What does it mean to say that two mathematical objects are the same?
	\item What does it mean to say that two mathematical objects have the same properties?
\end{enumerate}

\begin{defn}[Order]
	We define order as follows:
	\begin{gather*}
		a < b \qquad \text{if} \qquad b - a \in \bb{N} \\
		a = b \qquad \text{if} \qquad b - a = 0 \\
		a > b \qquad \text{if} \qquad b - a \in \bb{N}
	\end{gather*}
	This order satisfies some simple properties:
	\begin{enumerate}
		\item[O1] $\forall a, b, c \in \bb{Z} \land a < b, \; a + c < b + c$
		\item[O2] $\forall a, b, c \in \bb{Z} \land a < b \land c > 0, \; ac < bc$
	\end{enumerate}
\end{defn}

\textbf{Exercises}
\begin{enumerate}
	\item[1.1] Show that $\bb{Z}[\sqrt{3}]$ is a commutative ring.
	\item[1.2] Describe explicitly the ring $\bb{Z}[\sqrt[3]{5}]$.
	\item[1.3] What are the additive and multiplicative identities for $\bb{Z} \oplus \bb{Z}$? Show that there are non-zero elements which multiply to 0.
	\item[1.4] Consider the ring $R = \{2^n \; | \; n \in \bb{Z}\}$ with addition $\oplus$ given by $2^n \oplus 2^m = 2^{n + m}$, and multiplication $\odot$ given by $2^n \odot 2^m = 2^{nm}$. Show that this is a ring. Then show that the map taking $2^n$ to its logarithm base 2 (namely n) maps R one-to-one and onto $\bb{Z}$ and preverse addition and multiplication. We say that R is isomorphic to $\bb{Z}$.
	\item[1.5] What other properties of the integers can you think of? Can these properties be deduced from the ones given above?
	\item[1.6] Can an order be put on $\bb{Z}_2$ satisfying O1?
\end{enumerate}

\section{Well Ordering Principle}

\begin{principle}[Well Ordering Principle]
	Every non-empty subset of $\bb{N}$ has a least element.
\end{principle}

\begin{thm}[Generalized Principle of Induction]
	For all integers n, let P(n) be the statement about n propositions. Suppose that proposition P(1) is true. Furthermore, suppose that if P(k) is true for $1 \leq k < n$, then P(n) is true. Then P(n) is true for all $n \geq 1$.
\end{thm}

\begin{proof}
	Let S be the set of all n such that P(n) is false. If S is empty, then we have our desired conclusion, i.e. P(n) is true for all $n \geq 1$. Otherwise, S is non-empty. Then by the Principle of Induction, there exists an n in S such that n is the least element in S. By the hypothesis, $n \neq 1$. Since n is the smallest integer in S, we know that P(k) is true for all $1 \leq k < n$ (otherwise if there is an $1 \leq l < n$ such that P(l) is false, then $l \in S \land l < n$ will contradict the fact that n is the least element). But by the induction hypothesis, P(n) must be true. This contradicts that P(n) is false. Thus our assumption that S is non-empty is false. Therefore, S must be empty, and therefore P(n) is true for all $n \geq 1$.
\end{proof}

\begin{eg}[False Proof of Induction]
	Let P(n) be the statement that every set of n people have all the same hair colour. This is evident for n = 1. Now look at larger n. Suppose that P(n - 1) is true. Given a group of n people, apply the induction hypothesis to all but the last person in the group. This group will have all the same hair colour. Now repeat this argument with all but the first person. We find that all the people have the same hair colour by combining these two facts. By induction, all people have the same hair colour.
\end{eg}

\textbf{Exercises}
\begin{enumerate}
	\item[2.1] Find the error in the False Proof of Induction example above.
\end{enumerate}

I believe the problem lies in the execution of ``Now repeat this argument with all but the first person''. The problem is that if the last person has a different hair colour, then the base case would have been false from the beginning, thus rendering the entire induction to be false.

\begin{enumerate}
	\item[2.2] Prove by induction that
	\begin{equation*}
		\sum_{i=1}^{n} i^3 = \left(\sum_{i=1}^{n} i\right)^2.
	\end{equation*}
\end{enumerate}

The base case is true and easily verifiable, since $1^3 = (1)^2 = 1$. Now suppose that for all k where $1 \leq k < n$ such that
\begin{equation*}
	\sum_{i=1}^{k} i^3 = \left(\sum_{i=1}^{k} i \right)^2
\end{equation*}
is true. Now consider the case n = k + 1. Then
\begin{align*}
	\sum_{i = 1}^{k + 1} i^3
		&= \sum_{i=1}^{k} i^3 + (k + 1)^3 \\
		&= \left( \sum_{i=1}^{k} i \right)^2 + (k + 1)^3 \\
		&= \left( \sum_{i=1}^{k} i \right)^2 + k^3 + 3k^2 + 3k + 1
\end{align*}

\begin{align*}
	\left( \sum_{i=1}^{k + 1} i \right)^2
		&= \left( \sum_{i = 1}^{k} i + k + 1 \right)^2 \\
		&= \left( \sum_{i = 1}^{k} i \right)^2 + 2(k+1)\left( \sum_{i = 1}^{k} i \right) + (k + 1)^2 \\
		&= \sum_{i=1}^{k} i^3 + (k+1)\left(\sum_{i=1}^{k} 2i + k + 1 \right)
\end{align*}
So
\begin{align*}
	\sum_{i = 1}^{k + 1} i^3 - \left( \sum_{i=1}^{k + 1} i \right)^2
		&= (k + 1)^3 - 2(k+1)\left( \sum_{i = 1}^{k} i \right) - (k + 1)^2 \\
		&= (k + 1)\Bigg( k^2 + 2k + 1 - \sum_{i=1}^{k} 2i - k - 1 \Bigg)
\end{align*}

\begin{enumerate}
	\item[2.3] Consider the Fibonnaci sequence, given by F(0) = F(1) = 1, and for $n \geq 0$, F(n + 2) = F(n) + F(n + 1). Let $r = (\sqrt{5} + 1)/2$. Prove by induction that
	\begin{equation*}
		F(n) = \frac{r^{n + 1} - (\frac{-1}{r})^{n + 1}}{\sqrt{5}}.
	\end{equation*}
	\item[2.4] Define a sequence of real numbers by the rules
	\begin{equation*}
		s_0 = 0 \quad \text{and} \quad s_{n + 1} = \sqrt{3 + s_n} \enspace \text{for } n \geq 0
	\end{equation*}
	\begin{enumerate}
		\item Show by induction that $s_n < s_{n + 1} < 3$ for all $n \geq 0$.
		\item The least upper bound principle (chapter 5) shows that the sequence has a limit. Show that the limit should be $\sigma = \frac{1 + \sqrt{13}}{2}.$
		\item Obtain a formula for $\sigma - s_{n + 1}$ in terms of $\sigma - s_n$. Hence prove by induction that $0 < \sigma - s_n < \frac{3}{4^n}$ for all $n \geq 0$.
	\end{enumerate}
\end{enumerate}

\end{document}
% Document End


\documentclass{beamer}
\usetheme{Madrid}

\def\Re#1{\operatorname{Re}\left(#1\right)}

\title[No Largest Prime]{There Is No Largest Prime Number}
\subtitle{With an introduction to a new proof technique}

\author[Euclid, DDU]{
    Euclid of Alexandria\inst{1}
    \and
    DDU\inst{2}
}
\institute[Alexandria, ST]{
    \inst{1}
    \texttt{euclid@alexandria.edu}\\
    Department of Mathematics\\
    University of Alexandria
    \and
    \inst{2}
    \texttt{ddu6@protonmail.com}\\
    Simple Text Orgnazition
}
\date[ISPN '80]{27th International Symposium of Prime Numbers, --280}

\begin{document}

\frame{\titlepage}

\begin{frame}
    \frametitle{Outline}
    \tableofcontents[pausesections]
\end{frame}

\AtBeginSection[]
{
  \begin{frame}
    \frametitle{Outline}
    \tableofcontents[currentsection]
  \end{frame}
}

\section{Motivation}

\subsection{What Are Prime Numbers?}

\begin{frame}
    \frametitle{What Are Prime Numbers?}

    \begin{definition}
        A \alert{prime number} is a number that has exactly two divisors.
    \end{definition}

    \begin{example}
        \begin{itemize}
            \item 2 is prime (two divisors: 1 and 2).
            \pause
            \item 3 is prime (two divisors: 1 and 3).
            \pause
            \item 4 is not prime (\alert{three} divisors: 1, 2, and 4).
        \end{itemize}
    \end{example}
\end{frame}

\section{Results}

\subsection{Proof of the Main Theorem}

\begin{frame}
    \frametitle{Proof of the Main Theorem}
    \framesubtitle{The proof uses \textit{reductio ad absurdum}}

    \begin{theorem}
        There is no largest prime number.
    \end{theorem}

    \begin{proof}
        \begin{enumerate}
            \item<1-| alert@1> Suppose $p$ were the largest prime number.
            \item<2-> Let $q$ be the product of the first $p$ numbers.
            \item<3-> Then $q + 1$ is not divisible by any of them.
            \item<1-> But $q + 1$ is \only<4->{greater than $1$, thus }divisible by some prime number not in the first $p$ numbers.\qedhere
        \end{enumerate}
    \end{proof}

    \uncover<5->{The proof used \textit{reductio ad absurdum}.}
\end{frame}

\section{What's More}

\subsection{An Open Question}

\begin{frame}
    \frametitle{An Open Question}

    \begin{columns}
        \column{.45\textwidth}
        \begin{block}{Answered Questions}
            How many primes are there?
        \end{block}

        \column{.45\textwidth}
        \begin{block}{Open Questions}
            Is every even number the sum of two primes?
            \cite{Goldbach1742}
        \end{block}
    \end{columns}

    \begin{thebibliography}{10}
        \bibitem{Goldbach1742}[Goldbach, 1742]
        Christian Goldbach.
        \newblock A problem we should try to solve before the ISPN '43 deadline,
        \newblock \emph{Letter to Leonhard Euler}, 1742.
    \end{thebibliography}
\end{frame}

\subsection{Another Open Question}

\begin{frame}
    \frametitle{Another Open Question}

    In complex plane, define $\zeta(s)=\sum\limits_{n=1}^\infty\frac1{n^s}$. Then, are all zeros of $\zeta$ in the strip $0\leq\Re{s}\leq1$ lie on the line $\Re{s}=\frac12$?
    \pause

    \begin{block}{Some Observation}
        If $\Re{s}>1$\pause, then
        \begin{align*}
            \zeta(s)&=\sum_{n=1}^\infty\frac1{n^s}\\
            &=\prod_p\frac1{1-p^{-s}}.
        \end{align*}
        \pause
       Thus $\zeta$ does not vanish when $\Re{s}>1$.
    \end{block}
\end{frame}

\subsection{An Algorithm For Finding Primes Numbers}

\begin{frame}[fragile]
    \frametitle{An Algorithm For Finding Primes Numbers}

    \begin{semiverbatim}
        \uncover<1->{\alert<0>{int main (void)}}
        \uncover<1->{\alert<0>{\{}}
        \uncover<1->{\alert<1>{ \alert<4>{std::}vector<bool> is_prime (100, true);}}
        \uncover<1->{\alert<1>{ for (int i = 2; i < 100; i++)}}
        \uncover<2->{\alert<2>{ if (is_prime[i])}}
        \uncover<2->{\alert<0>{ \{}}
        \uncover<3->{\alert<3>{ \alert<4>{std::}cout << i << " ";}}
        \uncover<3->{\alert<3>{ for (int j = i; j < 100;}}
        \uncover<3->{\alert<3>{ is_prime [j] = false, j+=i);}}
        \uncover<2->{\alert<0>{ \}}}
        \uncover<1->{\alert<0>{ return 0;}}
        \uncover<1->{\alert<0>{\}}}
    \end{semiverbatim}

    \visible<4->{Note the use of \alert{\texttt{std::}}.}
\end{frame}

\end{document}