\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage[dvipsnames]{xcolor}
\usepackage{amsmath}
\usepackage{tcolorbox}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{proof}
\usepackage{float}
\usepackage{listings}
\lstset{
  basicstyle=\ttfamily,
  mathescape
}

\theoremstyle{definition}

\usepackage[
top    = 2.50cm,
bottom = 2.50cm,
left   = 2.75cm,
right  = 2.75cm]{geometry}
\usepackage{fancyhdr}
\pagestyle{fancy}
\lhead{Advanced computation}
\rhead{EPFL/Alp Ozen}
\newcommand{\floor}[1]{\lfloor #1 \rfloor}

\title{Advanced Computation 1 - CS 101}
\author{Alp Ozen}
\date{\vspace{-5ex}}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{example}{Example}
\newtheorem{remark}{Remark}
\newtheorem{definition}{Definition}
\numberwithin{equation}{subsection}
\numberwithin{remark}{subsection}
\newtheorem{question}{Question \textit{?}}[subsection]
\newtheorem{answer}{Answer}[subsection]
\begin{document}

\maketitle
\section{Propositional logic and notions of sets(Week 1 - Week 4)}
\subsection{Propositions}
A proposition is a declarative sentence that is either true or false.
\begin{tcolorbox}
How much does it cost? \textbf{is not a proposition}
\\
I like red \textbf{is a proposition}
\end{tcolorbox}

To make life easier, we represent propositional statements through letters such as $p$.
\\

The conditional statement $p\implies q$ appears very often. Thus, we have the \textit{converse,contrapositive and inverse} which are:
\\
\textbf{converse}: $q \implies p$
\\
\textbf{contrapositive}: $ \neg q \implies \neg p $
\\
\textbf{inverse}: $ \neg p \implies \neg q$
\\

We note that a conditional is logically equivalent to its contrapositive.

\begin{tcolorbox}
\centering
\begin{array}{cccccc}
 p & q & p $\implies$ q & \neg q & \neg p & \neg q $\implies$ \neg p \\
 \hline
t & t & t & f & f & t\\
t & f & f & t & f & f\\
f & t & t & f & t & t\\
f & f & t & t & t & t
\end{array}
\end{tcolorbox}

\subsection{Precedence of logical operators}

\begin{figure}[h]
    \centering
    \includegraphics{figures/precedence}
\end{figure}

\subsection{Fuzzy logic}

In fuzzy logic, truth values are between 0 and 1. So if the statement "I like riding a bike" has a value of 0.8, it's negation has 1 minus this value, in this case -0.2.

\subsection{Applications of logic}
\subsubsection{Logic gates}
Here are the basic logic circuits from which more complex circuits are made:

\begin{figure}[h]
    \centering
    \includegraphics[scale= 1]{epflSemesterOne/advancedComputation/figures/logicgates.JPG}
    \caption{Logic gates}
    \label{fig:my_label}
\end{figure}

Note that the OR and AND gates accept only and only 2 inputs and output one. This input may be as compounded as possible but not exceed 'two chunks'. Thus, when given a complex logical output and reverse engineering, we identify the outer most outer operation, branch it into two or one(if it is simply a negation) and so on.

\subsubsection{More on propositions}

\begin{tcolorbox}
\begin{itemize}
    \item \textbf{Tautology} is a compound proposition that is always true regardless of the truth value of its variables
    \item \textbf{Contradiction} is a compound proposition that is always false regardless of the truth value of its variables
    \item \textbf{Contingency} compound statement that is neither tautology nor contradiction

    \begin{example}
    \\
    $p \land \neg p$ is a contradiction
    \\
    $p \lor \neg p$ is a tautology
    \end{example}
\end{itemize}
\end{tcolorbox}

Here are some examples of logical calculus:
\\
Show that $\neg(p \lor (\neg p \land q )) \equiveq \neg p \land \neg q$
\begin{align*}
    \neg(p\lor\neg p \land p \lor q)\\
    \neg(T \land p \lor q)\\
    \neg(p \lor q)\\
    \neg p \land \neg q
\end{align*}

\subsubsection{Satisfiability}

A compound proposition is \textbf{satisfiable} if a truth assignment can be made to its variables that make it true making it either a tautology or a contingency. It is \textbf{unsatisfiable} if the negation of the compound statement is a contradiction.

\subsection{Logical calculus and useful equivalences}

\begin{definition}
If $A \iff B$ is a tautology, then A is logically equivalent to B.
\end{definition}
\\

Here are some useful logical equivalences(omitting most obvious ones):

\begin{align*}
    p \implies q \equiv \neg p \lor q \equiv \neg(\neg q \lor p) \equiv \neg(q \implies p) \equiv \neg q \implies \neg p \\
    p \lor ( q \land r) \equiv (p \lor q) \land (p \lor r) \\
    \land \ \text{distributes over} \lor \text{and vice versa}\\
    p \lor \neg p \equiv T\\
    p \land \neg p \equiv F\\
    P \land T \equiv p\\
    p \lor F \equiv p
    \ \text{both $\land$ and $\lor$ are associative}
\end{align*}

For more see this figure:

\begin{figure}[h]
    \centering
    \includegraphics[scale = 0.7]{epflSemesterOne/advancedComputation/figures/logic.JPG}
    \caption{Logic, yey!}
    \label{fig:my_label}
\end{figure}

\begin{definition}
A \textbf{rule of inference} is based on the tautology $p \land (p \implies q) \implies q$. That is, whenever we are given that both $p$ and $p \implies q$ is true, we infer that q must be true. That is :
\\

\infer{q}{p & (p \implies q)}
\end{definition}

Another important fact of logic is that we may boil down all of $\lor, \oplus, \implies, \iff$ to simply propositions involving$\neg, \land$:

\begin{align*}
    p \lor q \equiv \neg \neg (p \lor q) \equiv \neg(\neg p \land \neg q)\\
    p \oplus q \equiv \neg(p \land q) \land (p \lor q)\\
    p \implies q \equiv \neg p \lor q \equiv \neg (p \land \neg q)\\
    p \iff q \equiv \neg(\neg(p\land q) \land (\neg (\neg p \land \neg q))
\end{align*}

\begin{question}
Given propositional variables and truth values of the single variables for which the compound proposition takes a value, is there a way of deducing a compound proposition?
\end{question}
\newpage
\subsection{Lec.03 notes}

The \textbf{contrapositive} is the following statement:
\begin{align*}
    p \implies q \equiv \neg p \lor q\\
    \equiv q \lor \neg p\\
    \equiv \neg q \implies \neg p\\
    \therefore p \implies q \equiv \neg q \implies \neg p
\end{align*}
\\
Some useful logical equivalences involving implication:

\begin{align*}
    (p \implies q) \land (p \implies r) \equiv (\neg p \lor q) \land (\neg p \lor r)\\
    \equiv \neg p \lor (q \land r) \equiv p \implies (q \land r)
\end{align*}
And here's a more trivial one:
\begin{align*}
    (p \implies q) \lor (p \implies r) \equiv (\neg p \lor q) \lor (\neg p \lor r)\\
    \equiv \neg p \lor \neg p \lor q \lor r \equiv \neg p \lor (q \lor r)\\
    \equiv p \implies (q \lor r)
\end{align*}
And slightly more complicated involving De Morgan:
\begin{align*}
    (p \implies r) \land (q \implies r) \equiv (\neg p \lor r) \land (\neg q \lor r)\\
    \equiv \neg r \lor (\neg p \land \neg q) \equiv \neg r \lor (\neg(p \lor q)\\
    \equiv (p \lor q) \implies r
\end{align*}

We must also add some comments on base b systems of numbers and a general algorithm for conversion. Let's take an example in base 5. Suppose we want to convert $60_{10}$ to its base 5 representation. Well the largest power of 5 less than or equal to 60 is 25 and thus we know that 60 can be \textbf{uniquely} represented as a linear combination of the powers of 5 less than or equal to it. In fact, using powers of 5 less than or equal to 60, we may represent all numbers up to $5^k - 1$ as $4\cdot5^{k-1} + \ldots + 4\cdot5^{0}$. Thus, to represent some $l$ in base $b$ the algorithm is to find the largest power of $b^k < l$, perform $\floor{\frac{l}{b^k}}$ then repeat step 1 and proceed as $l - b^k\cdot\floor{\frac{l}{b^k}}$ and repeat until $l - b^i\cdot\floor{\frac{l}{b^i}} = 0$

\subsection{Lec.04 notes}
\subsubsection{More on CMOS}
This was a lecture with a steep curve and here is a summary. First of all, we first revisit some of the concepts of the pmos and nmos resistors. A very important convention is that pmos must never be used for pull-down(connected to GND) and nmos never used for pullup(connected to VDD).

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.7]{epflSemesterOne/advancedComputation/figures/cmos.JPG}
    \caption{Caption}
    \label{Nmos,Pmos,Inverter,Nand}
\end{figure}

\begin{tcolorbox}
Some principles of cmos gates:
\begin{itemize}
    \item Pmos goes to top, nmos to bottom.
    \item Never connect high voltage to low voltage to prevent a short circuit.
    \item Any circuit may be realized as a combination of the NAND and NOR ciruit.
    \item Each pmos must connect to an nmos.
\end{itemize}
\end{tcolorbox}
\\
The most challenging part of CMOS circuits is the circuit analysis itself. Consider this example to see a method of circuit analysis:

\begin{figure}[h!]
    \centering
    \includegraphics[scale = 0.6]{epflSemesterOne/advancedComputation/figures/cmosq.JPG}
    \caption{exam question}
    \label{exam question}
\end{figure}

\FloatBarrier

\begin{tcolorbox}
Now, first of all, realize that X is connected to VDD iff all of $Y_{6} \land Y_{5} \land Y_{4}$ are grounded. That is $Y_{6} \land Y_{5} \land Y_{4} = 0$. For symbolic purposes, supposing that $Y_{i} = 0 \equiv \neg Y_{i} $ we get that $X = 1 \iff \neg (Y_{4} \lor Y_{5} \lor Y_{6})$ Now given that this is a CMOS circuit, we know that the bottom part does the exact opposite of the upper part. Thus, we have that $X=0 \iff \neg(\neg (Y_{4} \lor Y_{5} \lor Y_{6})) = Y_{4} \lor Y_{5} \lor Y_{6} \equiv Y_{1} \lor Y_{2} \lor Y_{3}$. As a final step, for $\neg (A \lor B \lor C)$ to be true, we need that the output equals $\neg(A \lor B \lor C)$ and since $X=1 \iff \neg (Y_{4} \lor Y_{5} \lor Y_{6})$ we get that $Y_{1} = Y_{2} \ldots$

\end{tcolorbox}

\subsubsection{Binary addition circuit}

Given that we can construct any compound logical gate using CMOS, suppose we want to implement a binary addition calculator. Now here are all the possible cases for doing binary addition:

\begin{figure}[h]
    \centering
    \includegraphics[scale = 0.6]{epflSemesterOne/advancedComputation/figures/binary.png}
    \caption{Addition possibilities}
    \label{fig:my_label}
\end{figure}

Now suppose we want a function $f(a,b,c_{in})$ to evaluate $s$. Well notice that $s$ is only true when the parity of $a,b,c_{in}$ is odd. That is, we may describe this outcome with the function $a \oplus b \oplus c_{in}$. Similarly, devising $g(a,b,c_{in})$ to compute $c_{out}$ we notice that $c_{out}$ evaluates to 1 iff at least two variables are true. This is equivalent to $(a\land b) \lor (a \land c_{in}) \lor (b \land c_{in})$

\subsubsection{Fast Multiplication aka. Karatsuba}
We now ponder whether there is a quick way of multiplying some $v$ and $w$. Now notice that for $v$ and $w$ in base 10, $v = aX + b$ and $w = cX + d$. Now notice that $v \cdot w = (aX+b)(cX+d)$ which in turn is:

\begin{equation*}
    v\cdot w = acX^2 + (ad + bc)X + bd
\end{equation*}

And further notice that:

\begin{equation*}
    ad + bc = (ac + bd) - (a - b)(c - d)
\end{equation*}

to get:

\begin{equation*}
    v\cdot w = acX^2 + ((ac + bd) - (a - b)(c - d))X + bd
\end{equation*}

Now if for instance $v$ and $w$ were 2 digit numbers, we would normally perform 4 digit by digit multiplications but with this new method, we end up performing only 3 and a trivial subs traction.
\\
As a general result, for multiplication of two k by k digit numbers, we end up performing $3^{\log_{2}k}$ multiplications and $log_{2}k$ many additions. Finally, notice that Karatsuba is a recursive algorithm.
\subsubsection{Two's compliment}
Consider how a computer is to represent negative integers. A very smart way of doing so is \textbf{two's compliment}. That is given a binary representation, we invert all 1's with 0's and all 0's with 1's and then add 1. Note that now the 0's take the role of 1's and vice versa. The reason for adding 1 is that the most significant digit is reserved for the sign. A 1 is a negative, a 0 a positive.
\\
\begin{remark}
Note that when multiplying two numbers with non-matching number of digits, we simply pad both numbers with 0's until both have number of digits that are a power of 2.
\end{remark}

\subsection{Lec 05. notes}
\subsubsection{Main points}
In this lecture quantifiers and their properties were discussed along with common pitfalls.
\\
Consider defining a proposition on some sub-domain. That is take $D = \{0,1,2\}, S = \{2\}$ Now we want to express the proposition $\forall x \in S, P(x)$ where $P(x)$ is to mean that $x$ is even in the form $\forall x Q(x)$. A major mistake made is to try and express this as $x \in S \land P(x) \equiv Q(x)$ Now clearly $\forall x Q(x) \not \equiv \forall x \in S, P(x) $ as taking $x=1$ leads to the LHS being false. So here is the correct way to do this: Let $Q(x) \equiv x\in S \rightarrow P(x)$. It is now the case that $\forall x Q(x) \equiv \forall x \in S, P(x)$. Therefore we have that:

\begin{align}
    \forall x \in S, P(x) \equiv \forall x(x\in S \rightarrow P(x))\\
    \exists x \in T P(x) \equiv \exists x( x\in T \land P(x))
\end{align}

\begin{remark}
Note that both $\exists$ and $\forall$ have precedence over any other logical operator.
\end{remark}

We would also like to point out that whenever we have an empty domain, then both $\exists x P(x)$ and $\exists x \neg P(x)$ are both false. Similarly, $\forall x P(x)$ and $\forall x \neg P(x)$ are both true.
\\
We now ask, how do we negate the quantifiers?

\begin{align}
    \neg(\exists x P(X)) \equiv \forall x \neg P(x)\\
    \neg(\forall x P(X)) \equiv \exists x \neg P(x)
\end{align}
And we add the note that the same negation rules also hold for quantifiers over a sub-domain. Here is an example proof:

\begin{align*}
    \equiv \neg \forall x \in S \ P(x) \equiv \neg (\forall x \in S \rightarrow P(x))\\
    \equiv \exists x \neg(x \in S \rightarrow P(x))\\
    \equiv \exists x \neg(x \not \in S \lor P(x))\\
    \equiv \exists x (x \in S \land \neg P(x))\\
    \equiv \exits x \in S, \ \neg P(x)
\end{align*}

\subsection{Lec 06. notes}

First of all, we note that when solving CMOS problems, always check if the circuit is complementary. That is whenever the upper part is 1, lower part must be disconnected. Hence if the upper part evaluates $A\land B$, lower part must evaluate $\neg(A \land B)$

Here is a nice CMOS circuit problem from week 3:

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{epflSemesterOne/advancedComputation/figures/cmos2.JPG}
    \caption{PSET3 cmos problem}
    \label{fig:my_label}
\end{figure}

Here's how I solved it:
\\
Now notice that $X=1$ iff. $(B=0 \land \neg A = 0) \lor (\neg B = 0 \land A = 0)$ Thus we obtain:
\begin{align*}
    (\neg B \land A) \lor (B \land \neg A)\\
    \equiv \neg(B\lor \neg A) \lor \neg(\neg B \lor A)\\
    \equiv \neg(A \rightarrow B) \lor \neg(B \rightarrow A)\\
    \equiv \neg ((A \rightarrow B) \land (B \rightarrow A))\\
    \equiv \neg (A \iff B)
\end{align*}
\clearpage

And here is some more details on two's compliment:

\begin{figure}[H]
    \centering
    \includegraphics[scale = 1.2]{epflSemesterOne/advancedComputation/figures/twosc.JPG}
    \caption{More on two's compliment}
    \label{fig:my_label}
\end{figure}

And now let's say a little more about two's complement. That is, two's compliment simply represents the additive inverse of a number in $\bmod{2^k}$ That is, suppose we wanted some $b$ such that $a + b \equiv 0 \bmod{2^k}$, then we'd have that $b = 2^{k} - a$ Which is essentially the formula for two's complement for a given bit architecture.

\subsection{Lec 07. notes}

We consider how to negate the $\exists!$ expression. Realize that:
$$\exists!xP(x) \equiv \exists x(P(x)\land P(y) \rightarrow (x=y))$$
And hence the negation(using our negation laws gives):
$$\neg\exists ! xP(x) \equiv P(x) \rightarrow \exists y \not = xP(y))$$
\\
And two demonstrate the importance of nesting order on quantifiers, consider this excerpt:

\begin{figure}[H]
    \centering
    \includegraphics[scale = 0.8]{epflSemesterOne/advancedComputation/figures/nesting.JPG}
    \caption{Nesting order demo}
    \label{fig:my_label}
\end{figure}

And notice here how $\exists x \forall y S(x,y) \rightarrow \forall y \exists x S(x,y)$ This makes sense as whenever an x exists for each y, we have to have that for all y, there is an x.
\\
\clearpage
And now we come to rules of inference for quantifiers.

\begin{definition}(Inference laws for quantified statements)
\\
\textbf{Universal instantiation}
\begin{equation*}
    \infer{P(c)}{\forall x P(x)}
\end{equation*}
\textbf{Universal generalization}
\begin{equation*}
        \infer{\forall x P(x)}{P(x) \text{for arbitrary x}}
\end{equation*}
And we note that the same hold respectively for the existential quantifier.
\end{definition}

And finally we present an application of rules of inference. Suppose the following:
\\
$H(x):$ x is here
\\
$U(x):$ x likes C
\\
$L(x)$ x is a fan of D.R.(Denis Ritchie)
\\
Now we assert the following:
\\
(1) There is someone here who likes C
\\
(2) Everyone who likes C is a fan of D.R.
\\
Hence we get:
$$(1) \equiv \exists x(H(x) \land U(x))$$
$$(2) \equiv \forall x(U(x) \rightarrow L(x))$$
Now what may we infer from these?
\\
Well we know  $\exists x(H(x) \land U(x))$ hence we by instantiation we have $H(c) \land U(c)$ then $U(c),H(c)$ and since $ \forall x(U(x) \rightarrow L(x))$ to give $U(c) \rightarrow L(c), L(c)$ and finally $$H(c)\land L(c)$$
\\
And although it is not too useful to memorize their names, we include here a table of common rules of inference:

\begin{figure}[H]
    \centering
    \includegraphics[scale = 0.9]{epflSemesterOne/advancedComputation/figures/inference.JPG}
    \caption{Rules of inference}
\end{figure}

\subsection{Lec 08. notes}
\subsubsection{Proof methods}
A \textbf{proof} is a valid argument establishing the truth of a statement.
\\
We list some common proof methods and elaborate:
\\
\textbf{Direct proof:} When trying to prove a statement like $p \rightarrow q$ we consider the only case that $p \rightarrow q$ would be false and show that it can not happen. That is we take $p$ true and using our inference laws, reach that $q$ must be true as well
\\
\textbf{Proof by contraposition:} Since $p\rightarrow q \equiv \neg q \rightarrow \neg p$ we try to show that the contrapositive holds via a direct proof. And here's a mini-example: Consider the statement \textit{if $n$ is an integer then $3n+2$ is odd.} Now suppose $3n+2$ is even to give $3n+2 = 2k, \ k\in \mathbb{N}$ Then we have that $n=\frac{2k-2}{3}$ and taking $k=2$ suffices to show $n$ is not an integer.
\\
\textbf{Proof by contradiction:} Firstly, proof by contradiction is often confused with proof by contraposition. We use proof by contradiction to show that a statement $p$ is true. To do this, if we can show that $\neg p \rightarrow q$ is true where $q$ is always false, we have that $\neg p$ is false hence $p$ true. A common example is showing that $\sqrt{2}$ is irrational. We take the negation of $p$ that $\sqrt{2}$ is rational and derive the contradiction that whenever we try to write $\sqrt{2} = \frac{z}{k}$ with $\gcd{z}{k} = 1$ that $\gcd{z}{k} = 1 \land \gcd{z}{k} \not = 1$ Thus the falsity of $p$ implies a contradiction(false value) showing that $p$ itself must be true.
\\
Similarly suppose we have statement $p\equiv a \rightarrow b$. We know that $\neg p \rightarrow a \land \neg b$ and get that $a\land \neg b$ is always false showing that $p$ must be true.
\subsubsection{Graphs and planarity}
\textbf{A planar} graph is one that can be drawn without overlapping edges. Now a planar graph does not imply that for all possible drawing, overlapping edges wont exist. It simply means that a drawing could be made without overlapping edges. Now the two most important facts of graphs are:

 \begin{tcolorbox}[drop shadow, title=($K_{3,3}$ and $K_{5}$ are not planar),lower separated=true]
    Now notice that each connection we make partitions the plane into an inner and outer side following the \textbf{Jordan theorem}. We find out that when it comes to drawing the last vertex for $K_{3,3}$, the edge is surrounded by 4 curves making it impossible to not cross a curve.
    \tcblower
    \centering
        \includegraphics[scale = 1.1,valign=t]{epflSemesterOne/advancedComputation/figures/k3.png}
\end{tcolorbox}
\\
Now, \textbf{Kuratowski's theorem} states that a graph is non-planar iff it contains in one way $K_{3,3}$ or $K_{5}$
\clearpage
\subsubsection{Different surfaces}
We now consider different surfaces, those of the \textbf{Torus, Mobius strip, Klein bottle} shown respectively in the figure below.

\begin{figure}[H]
    \centering
    \includegraphics{epflSemesterOne/advancedComputation/figures/mobius.JPG}
    \caption{Rectangular abstraction}
\end{figure}

These surfaces are interesting because the laws of planarity no longer hold. For instance, on a mobius strip, $K_{3,3}$ is indeed planar as below.

 \begin{tcolorbox}[drop shadow, title=($K_{3,3}$ has become planar),lower separated=true]
    \centering
        \includegraphics[scale = 0.9,valign=t]{epflSemesterOne/advancedComputation/figures/mobiusk3.JPG}
\end{tcolorbox}

\subsection{Lec 09. notes}
\subsubsection{Sequences}
We define a \textbf{sequence} as a mapping $f: \mathbb{N} \to \mathbb{R}$. Some most essential sequences are:

\begin{align*}
    \textbf{constant sequence} \ \exists c \forall i \ a_{i} = c\\
    \textbf{arithmetic sequence} \ \exists c \forall i \ a_{i+1} - a_{i}  = c\\
    \textbf{geometric sequence} \ \exists c \forall i \ \frac{a_{i+1}}{a_{i}}= c
\end{align*}
\\
More generally, the constant and arithmetic sequence are a member of the family of consecutive polynomial sequences. That is, since a polynomial is defines as:

$$\sum_{i=0}^{j} a_{i}X^{i}$$ we may have sequences of higher degree.
\subsubsection{Summations}
Consider the summation of the first n terms of a constant sequence $a_{i} = c$. We would have $\sum_{i=1}^{n}a_{i}$ which is the same as $\sum_{i=1}^{n}c$ and taking $c=1$ we get n times 1 added which is $\sum_{i=1}^{n}c = 1$
\\
Let's now consider the summation of an arithmetic sequence by using an $nxn$ square as below:
\begin{figure}[H]
    \centering
    \includegraphics{epflSemesterOne/advancedComputation/figures/square.JPG}
    \caption{Summations, yey!}
\end{figure}

We agree that at each step there are $2i-1$ colored crosses which add up to $n^2$ many crosses. So we express it as $\sum_{i=1}^{n}(2i-1) = n^2$ and try to get a useful formula out of it.
\\
Well we have that $\sum_{i=1}^{n}(2i-1) = \sum_{i=1}^{n}(2i) - \sum_{i=1}^{n}(1)$ Which further gives $2\sum_{i=1}^{n}(i) - \sum_{i=1}^{n}(1) = n^2$ hence we obtain $$\sum_{i=1}^{n}(i) = \frac{n^2 + n}{2}$$

\textcolor{gray}{We now consider more interesting summations involving} \textbf{telescoping sequences}. A telescoping sequence is a sequence where during summation of the terms (parts of) consecutive terms cancel each other, so that only few (parts of) terms remain to be added.
To illustrate this, consider the sequence $\{b_i\}$ for $i>0$ with $$b_i=i-(i-1),$$
and let $\{a_i\}$ be a constant sequence with $a_i=1$.
Obviously, it follows that $b_i=1$ so that $b_i=a_i$ and thus $\sum_{i=1}^nb_i=\sum_{i=1}^na_i$. The latter sum was already calculated, which was a trivial exercise but in principle still required adding together $n$ terms all equal to~1. Using that $\sum_{i=1}^nb_i=\sum_{i=1}^na_i$ the same sum $\sum_{i=1}^na_i$ can be computed by computing $\sum_{i=1}^nb_i$ instead: due to the telescoping effect, this computation does not require any actual calculations at all, but just involves careful administration:
\begin{eqnarray*}
\sum_{i=1}^nb_i&=&b_n\,+\,b_{n-1}\,+\,b_{n-2}\,+\,\ldots\,+\,b_2\,+\,b_1\\
&=&\underbrace{n-(n-1)}_{b_n}\,+\,\underbrace{n-1-(n-2)}_{b_{n-1}}\,+\,\underbrace{n-2-(n-3)}_{b_{n-2}}\,+\,\ldots\,+\,\underbrace{2-(2-1)}_{b_{2}}\,+\,\underbrace{1-(1-1)}_{b_{1}}\\
&=&n\underbrace{-(n-1)+n-1}_{=0}\underbrace{-(n-2)+n-2}_{=0}\underbrace{-(n-3)+\ldots}_{=0}\ldots\underbrace{\ldots+2}_{=0}\underbrace{-1+1}_{=0}\,-\,0\\
&=&n.
\end{eqnarray*}
\clearpage
Yet a much more formal way of showing this as follows:

\begin{eqnarray*}
\sum_{i=1}^nb_i&=&\sum_{i=1}^n(i-(i-1))\\
&=&\Big(\sum_{i=1}^ni\Big)-\Big(\sum_{i=1}^n(i-1)\Big)\,\,\,\,\text{     (in 2nd sum replace $i-1$ by $j$)} \\
&=&\Big(\sum_{i=1}^{n}i\Big)-\Big(\sum_{j=0}^{n-1}j\Big)\,\,\,\,\text{     (replace $j$ by $i$)} \\
&=&\Big(\sum_{i=1}^{n}i\Big)-\Big(\sum_{i=0}^{n-1}i\Big)\,\,\,\,\text{     (identify the part that occurs in both sums)} \\
&=&\Big(n+\sum_{i=1}^{n-1}i\Big)-\Big(0+\sum_{i=1}^{n-1}i\Big)\\
&=&n-0+\Big(\sum_{i=1}^{n-1}i\Big)-\Big(\sum_{i=1}^{n-1}i\Big)\\
&=&n.
\end{eqnarray*}

We now consider a different approach to geometric sequences. We are interested in a closed form expression for $\sum_{i=0}^ng_i$, where for relevant values of $r$ it may even make sense to talk about the value of $\sum_{i=0}^\infty g_i$.
Because $$\sum_{i=0}^ng_i=\sum_{i=0}^ng_0r^i=g_0\sum_{i=0}^nr^i$$
the value~$g_0$ is just an uninteresting multiplicative factor, and we omit it (i.e., from now on we assume that $g_0=1$; note that the entire sum equals zero if $g_0=0$).
Obviously, the problem is not interesting if $r=1$: then we get (with $g_0=1$) that
$$\sum_{i=0}^ng_i=\sum_{i=0}^nr^i=\sum_{i=0}^n1^i=\sum_{i=0}^n1=n+1.$$
Also $r=-1$ is not inspiring: we get 0 if $n$ is odd and 1 if $n$ is even; but the final arguments below work for $r=-1$ as well, so $r=-1$ is not excluded below.

Assuming $r\ne1$, we want to find a closed form expression for $\sum_{i=0}^nr^i$, i.e., for $$r^0+r^1+\ldots+r^{n-2}+r^{n-1}+r^n.$$
It was noted that we are already familiar with a similar sum, namely $$r^n+r^{n-1}+r^{n-2}+\ldots+r^1+r^0$$
which is just the same, but in the reverse order.
How come we are familiar with something like $r^n+r^{n-1}+r^{n-2}+\ldots+r^1+r^0$?
Just look at the decimal number $111\ldots11$ that consists of $n+1$ ones:
that is precisely the same as our expression $r^n+r^{n-1}+r^{n-2}+\ldots+r^1+r^0$ if we assume that $r=10$ (the \emph{basis} or \emph{radix} of our decimal number system) because our decimal number ${\color{red}{1}}{\color{green}{1}}{\color{blue}{1}}\ldots{\color{pink}{1}}{\color{magenta}{1}}$ is just a convenient shorthand notation for
$${\color{red}{1}}\times10^n+{\color{green}{1}}\times10^{n-1}+{\color{blue}{1}}\times10^{n-2}+\ldots+{\color{pink}{1}}\times10^1+{\color{magenta}{1}}\times10^0:$$
that is how the value of a number in decimal notation is defined, where the present case is particularly simple because all the digits are equal to one.

Taking for instance $n=4$, we find the number $11111$, and we know that if we multiply it by $9$ (which was carefully chosen as $10-1$, i.e., as $r-1$) we get, without any effort, $99999$, so that if we add one we get 100000, for which we have the nice closed form expression $10^5$ (i.e., $r^{n+1}$). We thus found that $$(11111\times 9)+1=10^5$$
and more in general that
$$(\underbrace{111\ldots11}_{n+1 \text{ ones}}\times 9)+1=10^{n+1},$$
or even more in general that
$$(\underbrace{111\ldots11}_{n+1 \text{ ones}}\times (r-1))+1=r^{n+1},$$
where the final $111\ldots11$ should be liberally interpreted as a shorthand notation for $r^n+r^{n-1}+r^{n-2}+\ldots+r^1+r^0$
and where we use that $10$ was just a placeholder for $r$ and thus that $9$ was a placeholder for $r-1$. This final expression leads us to believe that
$$\underline{(r^n+r^{n-1}+r^{n-2}+\ldots+r^1+r^0)\times(r-1)}+1=r^{n+1}$$
and thus that
$$r^n+r^{n-1}+r^{n-2}+\ldots+r^1+r^0=\frac{r^{n+1}-1}{r-1}$$
(where you should remember that indeed $r\ne 1$). But this is precisely the closed formula for $\sum_{i=0}^nr^i$ that we were trying to find, so we are done -- except we have not proved anything yet other than by intimidation.

To actually prove this result that
$$\sum_{i=0}^nr^i=\frac{r^{n+1}-1}{r-1}$$
if $r\ne 1$, have a look at the underlined part
$$(r^n+r^{n-1}+r^{n-2}+\ldots+r^1+r^0)\times(r-1)$$
of the equation above:
apparently we need to prove that this underlined part is equal to $r^{n+1}-1$.
Pulling the factor $r-1$ inside the parentheses we get
$$(r-1)r^n+(r-1)r^{n-1}+(r-1)r^{n-2}+\ldots+(r-1)r^1+(r-1)r^0$$
and thus
$$r^{n+1}-r^n+r^n-r^{n-1}+r^{n-1}-r^{n-2}+\ldots+r^2-r^1+r^1-r^0.$$
\textcolor{gray}{This is yet another example of our earlier telescoping sum: all the intermediate terms cancel each other.}

In any case, the conclusion is that if $r\ne1$ then
$$\sum_{i=0}^nr^i=\frac{r^{n+1}-1}{r-1}$$
for any integer $n\ge0$.
It follows that if $|r|<1$ we have that
$$\sum_{i=0}^\infty r^i=\frac{1}{1-r}$$
because for $n\to\infty$ the term $r^{n+1}\to0$.
\\

We finally conclude by representing a useful identity as follows:

$$ \sum_{i=1}^{\infty}\sum_{j=1}^{i} \cdots =  \sum_{j=1}^{\infty}\sum_{i=j}^{\infty} \cdots$$
\clearpage

\subsubsection{Sum of $i^k$}
And yet another important point to mention is that all the sums of form $i^k$ may be found using the fact that the telescoping sequence given by $$f(k+1,i) = i^{k+1} - (i-1)^{k+1}$$ since using the telescoping property we have that $$\sum_{i=1}^{n} i^{k+1} - (i-1)^{k+1} = n^{k+1} $$ which means that $$\sum_{i=1}^{n} i^{k+1} = n^{k+1} + \sum_{i=1}^{n}(i-1)^{k+1} $$ where the $\sum_{i=1}^{n}(i-1)^{k+1}$ are recursively found.

\subsubsection{Hilbert's Hotel}

Some interesting questions to pose on this matter are listed below with answers.
\begin{question}
A bus carrying countably infinite number of guests arrives, can guests be accommodated?
\end{question}
\begin{answer}
Well simply assign each former guest to odd numbered room and each new guest to even numbered room.
\end{answer}

\begin{question}
Now suppose that countably infinite number of buses with countably infinite guests arrive what now?
\end{question}
\begin{answer}
Well, we move each guest in room $k$ to room $2k-1$. Now for new guests, $kth$ guest in bus 1 moves to $2^{1}(2k-1)$ hence in general $kth$ guest in bus $j$ moves to $2^{j}(2k-1)$ room.
\end{answer}
\clearpage
\section{Algorithms and more (Week 5 - )}
\subsection{Lec 10. notes}
To demonstrate how summations come in handy consider snipets of following pseudocode

\begin{verbatim}
    B <- n while(B>0){step;}
\end{verbatim}
Leads to $\infty$ steps.
\begin{lstlisting}
          B < - n while(B>0){for i=1 to B do step}){B = $\floor{\frac{B}{2}}$}
\end{lstlisting}
Now notice we first perform $n$, then approximately $\frac{n}{2}$ steps and so on. Thus in total we have $\sum_{i=0}^{\log_{2}{n} }(\frac{n}{2^j})$ many steps where this approximates to $2n$
\\
And now sum must now summation results:

\begin{figure}[H]
    \centering
    \includegraphics[scale = 0.7]{epflSemesterOne/advancedComputation/figures/sums.JPG}
    \caption{Summation formulae}
    \label{fig:my_label}
\end{figure}
And now we present \textbf{countability}:
\begin{definition} (Countabiltiy)
A set is countable iff. it is finite or there is a bijection from \mathbb{N} to the set.
\end{definition}

And some countability facts:
\begin{itemize}
    \item Union of countable sets is countable
    \item \mathbb{Z} is countable, just map each odd number to the negative integers and each even number to positive integers.
    \item Union of a finite number of countable sets is countable. That is for the finite $A_{k}$ $k$ many sets, first map all the $a_{0} \in A_{0}$ to 1, then $a_{1} \in A_{1}$ to 2 and hence $k+1 \to a_{1} \in A_{0} $
    \item Union of a countable number of countable sets is countable.
    \begin{figure}[H]
        \centering
        \includegraphics[scale = .6]{epflSemesterOne/advancedComputation/figures/walk.JPG}
        \caption{Enumeration}
        \label{fig:my_label}
    \end{figure}
    \item Cartesian product of two countable sets is countable (for which reason rational numbers are countable)
    \item Infinite union of bitsrings of finite length are countable. That is suppose we define $\bigcup_{i=0}^{j} S^{j}$ where $S = \{0,1\}$ Now $S^{0}$ has 1 element, $S^{1}$ has 1 distinct element, $S^{2}$ has 3 distinct elements and so on. Thus, we map $ 0 \to S^{0}$, then $1 \to 01 $ $2 \to 10$ $3 \to 11$. Since each $S^{k}$ has $2^{k}$ elements our mapping is defined.
    \item The set $\{0,1\}^{*}$ consisting of infinite bitstrings is \textbf{uncountable} as follows by Cantor diagonilization.
\end{itemize}
\\
A question from this week's pset asked:
\\
\begin{question}
Let $A$ be the set of real numbers with a finite number of 1 in the binary form and $B$ the set of all reals with a finite number of 1 in decimal form. Which of these sets is countable?
\end{question}
\begin{answer}
As it turns out, when we try enumerating members of A, we notice this is doable because for instance if we were to miss out enumerating one element it would have to be the case that it either contains a 0 which  we have already counted or a 1 which again we have already counted. Conversely for B, suppose we have enumerated all such numbers, but then define a new number with a digit differing in at least one decimal place of each element of B.
\end{answer}

\subsection{Lec 11. notes}
We start with the \textbf{Halting problem} which states that given a computer program $P$ with input $I$ can we write a program that always determines whether $P(I)$ will terminate or not? As it turns out the answer is no.
\begin{example*}
Now let us define a program $A(P,x)$ that determines whether the program $P$ with input $x$ terminates. Now define $Q(x)$ as a program calling $A(x,x)$. Then $Q(P)$ runs $A(P,P)$ and $Q(P)$ will stop if $A(P,P)$ returns. Now consider $Q(Q)$. We have that $A(Q,Q)$ gets called. Well, if this returns no meaning $Q(Q)$ does not determinate, we get that $Q(Q)$ should terminate and vice versa. Hence, we get that such a program $A(x)$ can not exists.
\end{example*}

\begin{definition}(Big $O$)
A function $f(x)$ is said to be $O(g(x))$ or informally $f$ is $O(g)$ if $$ \exists C,k \ \forall x>k, \ |f(x)| \leq C|g(x)|$$
\end{definition}

\begin{definition}(Big $\Omega$ )
A function $f(x)$ is said to be $\Omega(g(x))$ or informally $f$ is $O(g)$ if $$ \exists C,k \ \forall x>k, \ |f(x)| \qeq C|g(x)|$$
\end{definition}

\begin{definition}(Big $\Theta$)
if $f$ is both $O(g)$ and $\Omega(g$, then f is $\Theta(g)$ meaning that f and g are of the same order
\end{definition}

\end{document}


%%
%% Automatically generated file from DocOnce source
%% (https://github.com/hplgit/doconce/)
%%
%%
% #ifdef PTEX2TEX_EXPLANATION
%%
%% The file follows the ptex2tex extended LaTeX format, see
%% ptex2tex: http://code.google.com/p/ptex2tex/
%%
%% Run
%%      ptex2tex myfile
%% or
%%      doconce ptex2tex myfile
%%
%% to turn myfile.p.tex into an ordinary LaTeX file myfile.tex.
%% (The ptex2tex program: http://code.google.com/p/ptex2tex)
%% Many preprocess options can be added to ptex2tex or doconce ptex2tex
%%
%%      ptex2tex -DMINTED myfile
%%      doconce ptex2tex myfile envir=minted
%%
%% ptex2tex will typeset code environments according to a global or local
%% .ptex2tex.cfg configure file. doconce ptex2tex will typeset code
%% according to options on the command line (just type doconce ptex2tex to
%% see examples). If doconce ptex2tex has envir=minted, it enables the
%% minted style without needing -DMINTED.
% #endif

% #define PREAMBLE

% #ifdef PREAMBLE
%-------------------- begin preamble ----------------------

\documentclass[%
oneside,                 % oneside: electronic viewing, twoside: printing
final,                   % draft: marks overfull hboxes, figures with paths
10pt]{article}

\listfiles               %  print all files needed to compile this document

\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb}
\usepackage[table]{xcolor}
\usepackage{bm,ltablex,microtype}

\usepackage[pdftex]{graphicx}

\usepackage[T1]{fontenc}
%\usepackage[latin1]{inputenc}
\usepackage{ucs}
\usepackage[utf8x]{inputenc}

\usepackage{lmodern}         % Latin Modern fonts derived from Computer Modern

% Hyperlinks in PDF:
\definecolor{linkcolor}{rgb}{0,0,0.4}
\usepackage{hyperref}
\hypersetup{
    breaklinks=true,
    colorlinks=true,
    linkcolor=linkcolor,
    urlcolor=linkcolor,
    citecolor=black,
    filecolor=black,
    %filecolor=blue,
    pdfmenubar=true,
    pdftoolbar=true,
    bookmarksdepth=3   % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC
    }
%\hyperbaseurl{}   % hyperlinks are relative to this root

\setcounter{tocdepth}{2}  % levels in table of contents

% Tricks for having figures close to where they are defined:
% 1. define less restrictive rules for where to put figures
\setcounter{topnumber}{2}
\setcounter{bottomnumber}{2}
\setcounter{totalnumber}{4}
\renewcommand{\topfraction}{0.95}
\renewcommand{\bottomfraction}{0.95}
\renewcommand{\textfraction}{0}
\renewcommand{\floatpagefraction}{0.75}
% floatpagefraction must always be less than topfraction!
% 2. ensure all figures are flushed before next section
\usepackage[section]{placeins}
% 3. enable begin{figure}[H] (often leads to ugly pagebreaks)
%\usepackage{float}\restylefloat{figure}

% prevent orhpans and widows
\clubpenalty = 10000
\widowpenalty = 10000

% --- end of standard preamble for documents ---

% insert custom LaTeX commands...

\raggedbottom
\makeindex
\usepackage[totoc]{idxlayout}   % for index in the toc
\usepackage[nottoc]{tocbibind}  % for references/bibliography in the toc

%-------------------- end preamble ----------------------

\begin{document}

% matching end for #ifdef PREAMBLE
% #endif

\newcommand{\exercisesection}[1]{\subsection*{#1}}

% ------------------- main content ----------------------

% ----------------- title -------------------------

\thispagestyle{empty}

\begin{center}
{\LARGE\bf
\begin{spacing}{1.25}
Project 5, deadline  December 10, 2017
\end{spacing}
}
\end{center}

% ----------------- author(s) -------------------------

\begin{center}
{\bf Partial Differential equations${}^{}$} \\ [0mm]
\end{center}

\begin{center}
% List of all institutions:
\end{center}

% ----------------- end author(s) -------------------------

% --- begin date ---
\begin{center}
Fall semester 2017
\end{center}
% --- end date ---

\vspace{1cm}

\subsection{Theoretical background and description of the system}

This project is more numerically directed, with tests
of algorithms for solving partial differential equations using finite difference schemes.
Chapter 10 of the lecture notes provides the necessary theoretical background.

\textbf{For this project you can collaborate with fellow students and you can  hand in a common report.}
This project (together with projects 3 and 4) counts 1/3 of the final mark.

The project has a strong mathematical axis. However, for those interested, it should be straightforward to replace the dimensionless analysis in  parts 5a-5f) with specific boundary and initial conditions. This is done in part 5g), with a focus on the heat equation and geophysical processes. That part is optional, but gives an additional 30 points to a score of 100.

The physical problem can be that of the temperature gradient in a rod of length $L=1$ or that of channel flow
between two flat and infinite plates at $x=0$ and $x=1$, where the fluid is initially at rest
and the plate $x=1$ is given a sudden initial movement.  We are looking at a one-dimensional
problem

\begin{equation*}
 \frac{\partial^2 u(x,t)}{\partial x^2} =\frac{\partial u(x,t)}{\partial t}, t> 0, x\in [0,L]
\end{equation*}
or

\begin{equation*}
u_{xx} = u_t,
\end{equation*}
with initial conditions, i.e., the conditions at $t=0$,

\begin{equation*}
u(x,0)= 0 \hspace{0.5cm} 0 < x < L
\end{equation*}
with $L=1$ the length of the $x$-region of interest. The
boundary conditions are

\begin{equation*}
u(0,t)= 0 \hspace{0.5cm} t \ge 0,
\end{equation*}
and

\begin{equation*}
u(L,t)= 1 \hspace{0.5cm} t \ge 0.
\end{equation*}
The function $u(x,t)$  can be the temperature gradient of a  rod or represent the fluid velocity
in a direction parallel to the plates, that is normal to the $x$-axis. In the latter case,
for small $t$, only the part of the fluid
close to the moving plate is set in significant  motion, resulting in a thin boundary layer at $x=1$.
As time increases, the velocity approaches a linear variation with $x$. In this case, which can be derived
from the incompressible Navier-Stokes, the above equations constitute a model for
studying friction between moving surfaces separated by a thin fluid film.

In this project we want to study the numerical stability of three methods for partial differential equations
(PDEs).
These methods we will discuss are
the explicit forward Euler algorithm with discretized versions of time given by a forward formula and a centered difference in space resulting in
\begin{equation*}
u_t\approx \frac{u(x,t+\Delta t)-u(x,t)}{\Delta t}=\frac{u(x_i,t_j+\Delta t)-u(x_i,t_j)}{\Delta t}
\end{equation*}
and

\begin{equation*}
u_{xx}\approx \frac{u(x+\Delta x,t)-2u(x,t)+u(x-\Delta x,t)}{\Delta x^2},
\end{equation*}
or

\begin{equation*}
u_{xx}\approx \frac{u(x_i+\Delta x,t_j)-2u(x_i,t_j)+u(x_i-\Delta x,t_j)}{\Delta x^2}.
\end{equation*}
We will also discuss the implicit Backward Euler with
\begin{equation*}
u_t\approx \frac{u(x,t)-u(x,t-\Delta t)}{\Delta t}=\frac{u(x_i,t_j)-u(x_i,t_j-\Delta t)}{\Delta t}
\end{equation*}
and

\begin{equation*}
u_{xx}\approx \frac{u(x+\Delta x,t)-2u(x,t)+u(x-\Delta x,t)}{\Delta x^2},
\end{equation*}
or

\begin{equation*}
u_{xx}\approx \frac{u(x_i+\Delta x,t_j)-2u(x_i,t_j)+u(x_i-\Delta x,t_j)}{\Delta x^2},
\end{equation*}

Finally we use the implicit Crank-Nicolson scheme with  a time-centered scheme at $(x,t+\Delta t/2)$
\begin{equation*}
u_t\approx \frac{u(x,t+\Delta t)-u(x,t)}{\Delta t}=\frac{u(x_i,t_j+\Delta t)-u(x_i,t_j)}{\Delta t}.
\end{equation*}
The corresponding spatial second-order derivative reads

\begin{align*}
u_{xx}\approx &\frac{1}{2}\left(\frac{u(x_i+\Delta x,t_j)-2u(x_i,t_j)+u(x_i-\Delta x,t_j)}{\Delta x^2}\right. +\\
&\left. \frac{u(x_i+\Delta x,t_j+\Delta t)-2u(x_i,t_j+\Delta t)+u(x_i-\Delta x,t_j+\Delta t)}{\Delta x^2}\right).
\end{align*}
Note well that we are using a time-centered scheme wih $t+\Delta t/2$ as center.

\paragraph{Project 5a): Setting up the algorithms.}
Write down the algorithms for these three methods and the equations you need to implement.
For the implicit schemes show that the equations lead to a tridiagonal matrix system for the new values.

\paragraph{Project 5b): Truncation errors and analytic solutions.}
Find the truncation errors of these three schemes and investigate their stability properties.
Find also the analytic solution to the continuous problem.

\paragraph{Project 5c): Implementation.}
Implement the three algorithms in the same code and perform tests of the solution
for these three approaches
for $\Delta x=1/10$, $\Delta x=1/100$ using  $\Delta t$ as dictated by the stability limit of the explicit scheme.
Study the solutions at two time points $t_1$ and $t_2$ where $u(x,t_1)$ is smooth but still significantly curved
and $u(x,t_2)$ is almost linear, close to the stationary state.
Remember that forsolving the tridiagonal equations you can use your code from project 1.
\paragraph{Project 5d):}
Compare the solutions at $t_1$ and $t_2$ with the analytic result for the continuous problem.
Which of the schemes would you classify as the best?

\paragraph{Project 5e): Moving to two dimensions.}
Extend the code you have developed here to two
  dimensions.
It means that we deal with a $2+1$ dimensional problem. Our differential equation becomes
\[
 \frac{\partial^2 u(x,y,t)}{\partial x^2}+\frac{\partial^2 u(x,y,t)}{\partial y^2} =\frac{\partial u(x,y,t)}{\partial t}, t> 0, x,y\in [0,1],
\]
where we now have made a model with a square lattice for $x$ and $y$.
How would you extend the boundary conditions from one dimension to two dimensions? And can you
  find a closed form solution here as well?  It is left to you to decide upon what kind of boundary conditions
you deem appropriate.
\paragraph{Project 5f): Solving the two-dimensional equations numerically.}
Here you can choose between an explicit or an implicit scheme.
For the implicit scheme discussed in chapter 10 of the lecture notes, upi need to use for example an iterative method like Jacobi's.

Outline the algorithm for solving the two-dimensional diffusion equation and
implement the explicit or the implicit scheme as function of $\Delta x$ (assuming
$\Delta x = \Delta y$) and $\Delta t$. Solve the equations numerically and give a critical discussion of your results.
Compare your results with the closed-form answer. Discuss the stability
of the solution as function of different values of $\Delta x$  and $\Delta t$.

You should, if you have time, try to  parallelize the equations using MPI or OpenMP.

\paragraph{Project 5g) Temperature distribution in the lithosphere (this part is optional and gives an additional score of 30 points).}
The Scientific background is  based on geological evidence at the surface. Geologists have proposed
that there was an active subduction zone on the western coast of
Norway about 1Gy ago. When subducting, the oceanic lithosphere
releases water and other chemical components that go mostly to the
surface but may also be trapped in the mantle above the subducting
slab. This process is called the refertilization of the mantle wedge
(see Figure). These chemical components contain more radioactive
elements than the normal mantle. We can therefore expect that ancient
mantle wedges are enriched in radioactive elements and therefore
warmer than normal mantle. The question is how much warmer they are
and if we can find evidence for this enrichment in geophysical
studies.

\begin{figure}[!ht]  %
  \centerline{\includegraphics[width=0.6\linewidth]{figslides/slab.jpg}}
  \caption{
  Cartoon of the enrichment of the mantle by a subduction slab.
  }
\end{figure}
%\clearpage % flush figures

The purpose of this part is to
calculate the thermal evolution of the lithosphere up to present following the emplacement of radioactive elements in the mantle wedge 1 Gy ago.

The equation to compute is the time evolution of the temperature distribution is the heat equation, which will be used only in 2D here, namely

\begin{equation}
\vec \nabla(k \vec \nabla T) + Q = \rho c_p \frac{\partial T}{\partial t}
\label{eq:heat2D}
\end{equation}
where $T$ is the temperature, $\rho$ is the density, $k$ is the thermal conductivity, $c_p$ is the specific heat capacity and $Q$ is the heat production.

With the boundary conditions and initial conditions, you should rescale your equations in order to use the results from parts 5a-5f). The paramaters of the model are described here.

\paragraph{Parameters of the model.}
The boundary conditions are $8^{\circ}\,C$ at the surface and $1300^{\circ}C$ at the bottom at a depth of 120 km.

In this study, we assume a constant density for the lithosphere of $3.5 10^3\,Kg/m^3$,
a constant thermal conductivity of $2.5\,W/m/^{\circ}C$
and a constant specific heat capacity of $1000\,J/Kg/^{\circ}C^{-1}$.

The heat production, caused by radioactive decay, cannot be taken as uniform in the whole lithosphere as radioactive elements are much more abundant in the crust than in the mantle.
We need therefore to separate the lithosphere in 3 units: the upper crust from 0 to 20 km depth, the lower crust from 20 to 40 km depth and the mantle from 40 to 120 km depth, where the heat production is set respectively to $1.4\,\mu W/m^3$, $0.35\,\mu W/m^3$ and $0.05\,\mu W/m^3$.

\paragraph{Temperature distribution before radioactive enrichment.}
Before radioactive enrichment, the temperature is steady-state and depends only on depth. In order to test your results you should
try to obtain analytical results for  the temperature profile as function of depth neglecting the presence of radioactice sources and. Compare thereafter these results with your numerical results.  Include then the radioactive sources and compare eventual analytical results with numerical calculations.

\paragraph{Temperature distribution after radioactive enrichment.}
We assume that the lithosphere above the slab gets enriched in
radioactive elements U, Th and K 1 Gy ago and that this results in an
additional heat production of $0.5\mu W/m^3$ over the whole depth of
the mantle (not the crust) in a 150 km wide area above the
slab. Neglecting all other thermal perturbations that this slab could
produce, and assuming that the heat production will remain constant
over geological time, compute the thermal evolution of the lithosphere
from 1 Gy to present.

The radioactivity will decrease with time. Assuming that the
additional heat is produced at 40\% by U, 40\% by Th and 20\% by K,
which have halflives of 4.47 Gy, 14.0 Gy and 1.25 Gy respectively,
compute the thermal evolution of the lithosphere and compare with the
result obtained above.

\paragraph{References.}
A very good reference is the textbook by \href{{http://www.springer.com/us/book/9783540225515}}{Winther and Tveito on partial differential equations}. It is available online \href{{https://vpn2.uio.no/+CSCO+0h756767633A2F2F797661782E66636576617472652E70627A++/book/10.1007/b138016/page/1}}{from the University library}.

\subsection{Introduction to numerical projects}

Here follows a brief recipe and recommendation on how to write a report for each
project.

\begin{itemize}
  \item Give a short description of the nature of the problem and the eventual  numerical methods you have used.

  \item Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.

  \item Include the source code of your program. Comment your program properly.

  \item If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.

  \item Include your results either in figure form or in a table. Remember to        label your results. All tables and figures should have relevant captions        and labels on the axes.

  \item Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.

  \item Try to give an interpretation of you results in your answers to  the problems.

  \item Critique: if possible include your comments and reflections about the  exercise, whether you felt you learnt something, ideas for improvements and  other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.

  \item Try to establish a practice where you log your work at the  computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember  what a previous test version  of your program did. Here you could also record  the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.
\end{itemize}

\noindent
\subsection{Format for electronic delivery of report and programs}

The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file.  As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:

\begin{itemize}
  \item Use Devilry to hand in your projects, log in  at  \href{{http://devilry.ifi.uio.no}}{\nolinkurl{http://devilry.ifi.uio.no}} with your normal UiO username and password and choose either 'fys3150' or 'fys4150'. There you can load up the files within the deadline.

  \item Upload \textbf{only} the report file!  For the source code file(s) you have developed please provide us with your link to your github domain.  The report file should include all of your discussions and a list of the codes you have developed.  Do not include library files which are available at the course homepage, unless you have made specific changes to them.

  \item In your git repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parametxers.

  \item In this and all later projects, you should include tests (for example unit tests) of your code(s).

  \item Comments  from us on your projects, approval or not, corrections to be made  etc can be found under your Devilry domain and are only visible to you and the teachers of the course.
\end{itemize}

\noindent
Finally,
we encourage you to work two and two together. Optimal working groups consist of
2-3 students. For this specific report you can  hand in a common report.

% ------------------- end of main content ---------------

% #ifdef PREAMBLE
\end{document}
% #endif


\documentclass[letterpaper,11pt]{article}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage[hmargin=1.25in,vmargin=1in]{geometry}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{listings}
\usepackage{lmodern}
\usepackage{mathtools}
\usepackage{microtype}
\usepackage{minted}

\DeclareMathOperator*{\argmin}{arg\,min}

\author{Philip Pham}
\date{\today}
\title{CSE 547 - Assignment 2}

\begin{document}
\maketitle

\section*{Problem 0}

\begin{description}
\item[List of collaborators:] I have not collaborated with anyone.
\item[List of acknowledgements:] None.
\item[Certify that you have read the instructions:] I have read and understood
  these policies.
\end{description}

\section*{Problem 1: Generalization, Streaming, and SGD}

In class, we examined using Stochastic Gradient Descent (SGD) for empirical loss
minimization, where we have an $N$ sized training set $\mathcal{T}$. The
empirical loss considered was:
\begin{equation}
  F(w) = \frac{1}{N} \sum_{(x,y) \in \mathcal{T}} l\left(w,(x,y)\right).
\end{equation}

Here, gradient descent for the function $F$ is the algorithm:
\begin{enumerate}
\item Initialize at some point $w^{(0)}$.
\item Sample $(x,y)$ uniformly at random from the set $\mathcal{T}$.
  \label{item:sgd_sample}
\item Update the parameters:
  \begin{equation}
    w^{(k+1)} = w^{(k)} - \eta_k \cdot \nabla l\left(w^{(k)},(x,y)\right),
  \end{equation}
  and go back to \ref{item:sgd_sample}.
\end{enumerate}

We provided guarantees assuming that $F$ was smooth and the gradients in our
training set were uniformly bounded,
$\lVert \nabla l\left(w, (x,y)\right) \rVert \leq B$.

However, in practice, we care about generalization, that is, statements on how
well we do on the underlying distribution. Define:
\begin{equation}
  \mathcal{L}(w) = \mathbb{E}_{(x,y) \in \mathcal{D}}l\left(w, (x,y)\right),
\end{equation}
where $\mathcal{D}$ is the underlying distribution.

Suppose we sought a point where $\lVert \nabla \mathcal{L} \rVert^2$ was
small. Obtaining this quantity to be small even in expectation would be
acceptable for this problem Assume that $\mathcal{L}$ is smooth and that the
gradients are uniformly bounded,
$\lVert \nabla l\left(w, (x,y)\right)\rVert \leq B$ for all parameters and all
possible points $(x,y)$ (under $\mathcal{D}$).

\begin{enumerate}
\item Assume we have sampling access to our underlying distribution
  $\mathcal{D}$. Explain how we can make $\lVert \nabla \mathcal{L}(w) \rVert^2$
  small in expection. What can you guarantee if you obtain $m$ samples and how
  would you do this?

  \subsection*{Solution}

  If we sample from $\mathcal{D}$ (as opposed to from $\mathcal{T}$ in Step
  \ref{item:sgd_sample} in the original algorithm), then we can guarantee that
  $\lVert \nabla \mathcal{L}(w) \rVert^2$ is small in expectation. The proof is
  identical to that of Theorem 1.2 from Lecture 6 in class but we replace
  $\mathcal{T}$ with $\mathcal{D}$. Thus, each gradient becomes a sample of
  $\nabla\mathcal{L}(w)$.

\item Suppose we contruct an $N$ sized training set $\mathcal{T}$, where each
  point is sampled under $\mathcal{D}$; then we construct the empirical loss
  funciton $F(w)$; then we run SGD on $F$ for $K$ steps (suppose $K \geq N$). Is
  there an argument on this procedure that implies something non-trivial (and
  technically correct) about $\lVert \nabla\mathcal{L}(w)\rVert^2$, even in
  expectation?

  \subsection*{Solution}

  If we let $K = N$, and sample each point from $\mathcal{T}$ exactly once, we
  can get the guarantee
  \begin{equation}
    \min_{k < K}\mathbb{E}\left[
      \left\lVert \nabla\mathcal{L}\left(w^{(k)}\right)\right \rVert_2^2
    \right] = B\sqrt{\frac{2\left(F\left(w^{(0)}\right) - F_*\right)L}{K}}
  \end{equation}
  as in Theorem 1.2 from Lecture 6. This works because using each sample from
  $\mathcal{T}$ exactly once is the same as sampling from $\mathcal{D}$ $K$
  times. However, this falls apart as soon as $K > N$ because now our population
  is $\mathcal{T}$, not $\mathcal{D}$, so we can no longer make any guarantees
  about $\lVert \nabla\mathcal{L}(w)\rVert^2$.
\end{enumerate}

\section*{Problem 2: GD versus Adaptive GD}

Let us compare gradient descent (GD), with a constant stepsize $\eta$, to
Adagrad on two simple one-dimensional objective functions. Adagrad will use the
stepsize at iteration $k$ where:
\begin{equation}
  \eta_k = \frac{C}{\sqrt{\sum_{j = 0}^k \lVert \nabla F \left(
        w^{(j)}
      \right)\rVert^2}}.
\end{equation}

\begin{enumerate}
\item Consider $F(w) = \frac{1}{2}w^2$. Let us start at $w_0 = 1$. For this
  problem a constant stepsize certainly makes sense for GD. Let $\eta = 3/4$ and
  $C = 3/4$.

  Analytically, characterize the convergence rates of GD and Adagrad
  with these parameter settings.

  \subsection*{Solution}

  For GD, we can explicitly, write down the formula for $w^{(k)}$. Since
  $F^\prime(w) = w$, $w^{(k+1)} = (1 - C)w^{(k)}$, so
  $w^{(k)} = \left(1-C\right)^k = \left(1/4\right)^k$. So,
  $\nabla F\left(w^{(k)}\right) = \left(1/4\right)^k$.

  The global minimum occurs at $w^* = 0$. Since $\eta_k \leq C$ with equality
  only when $k = 0$, Adagrad converges strictly slower.

  However, it does no converge much slower. We can show that
  $w^{(k)} \leq \left(1/3\right)^k$. We show this by strong
  induction. $w^{(0)} = 0$ satisifies this. Now, we have that
  \begin{align*}
    w^{(k+1)}
    &= w^{(k)}\left(1 - \frac{3/4}{\sqrt{\sum_{i = 0}^k \left(w^{(i)}\right)^2}}\right) \\
    &\leq w^{(k)}\left(1 - \frac{3/4}{\sqrt{\sum_{i = 0}^k \left(\frac{1}{3}\right)^2}}\right) \\
    &\leq w^{(k)}\left(1 - \frac{3/4}{\sqrt{9/8}}\right) \approx 0.2928932 w^{(k)} \\
    &\leq \left(\frac{1}{3}\right)^{k + 1}.
  \end{align*}

  Thus, we have that $\left(1/4\right)^k \leq w^{(k)} \leq \left(1/3\right)^k$,
  so the two optimization procedures do not differ substantially.  In this case,
  GD and Adagrad differ by at most a constant factor of $3/4$.

\item Consider the non-smooth, convex function $F(w) = \lvert w \rvert$. Let us
  start with $w_0 = 1$. Use $\eta = 3/4$ and $C = 3/4$ as before.

  Analytically, characterize the convergence rates of GD and Adagrad
  with these parameter settings.

  \subsection*{Solution}

  We have that
  \begin{equation}
    F^\prime\left(w\right) = \begin{cases}
      1, & w > 0; \\
      -1, & w < 0;
    \end{cases}
  \end{equation}
  and $F^\prime(w)$ is not defined at $w = 0$.

  In this case, GD would not converge. The sequence would look like
  \begin{align*}
    w^{(0)} &= 1 \\
    w^{(1)} &= 1/4 \\
    w^{(2)} &= -1/2 \\
    w^{(3)} &= 1/4 \\
    w^{(4)} &= -1/2 \\
    w^{(5)} &= 1/4 \\
            &~~\vdots
  \end{align*}

  Adagrad on the other hand is $O\left(1/\sqrt{T}\right)$, where $T$ is the
  number of steps that we have taken. To see this, note that
  $\lVert \nabla F(w) \rVert^2_2 = 1$ We have two cases: (\ref{itm:2_w_gt})
  $w^{(k)} > 0$ and (\ref{itm:2_w_lt}) $w^{(k)} < 0$.
  \begin{enumerate}
  \item In this case, we'd have that
    \begin{equation*}
      w^{(k + 1)} = w^{(k)} - \frac{3}{4\sqrt{k + 2}}.
    \end{equation*}
    \label{itm:2_w_gt}
  \item Here,
    \begin{equation*}
      w^{(k + 1)} = w^{(k)} + \frac{3}{4\sqrt{k + 2}}.
    \end{equation*}
    \label{itm:2_w_lt}
  \end{enumerate}

  Suppose that $0 < w^{(k)} < (3/4)/\sqrt{k + 1}$. Certainly, this is true for
  $k = 1$, where $w^{(1)} = 1/4$. Then, we have that
  \begin{equation}
    -\frac{3}{4\sqrt{k + 2}} < w^{(k + 1)} < \frac{3}{4\sqrt{k+1}} - \frac{3}{4\sqrt{k + 2}} < \frac{3}{4\sqrt{k + 2}}.
  \end{equation}
  We can similarly show the same inequality holds if
  $-(3/4)/\sqrt{k + 1} < w^{(k)} < 0$. Thus, we have that
  $\left\lvert w^{(k)} \right\rvert < (3/4)/\sqrt{k + 1}$ for all $k \geq 1$, so
  Adagrad converges at the rate $O\left(1/\sqrt{T}\right)$.
\end{enumerate}

\section*{Problem 3: Understanding Non-Convexity}

\subsection*{Newton's Method}

The least squares problem can be written as:
\begin{equation}
  \min_{w}L(w),~\text{where}~L(w) \coloneqq \frac{1}{2N}\left\lVert Xw - Y\right\rVert^2,
\end{equation}
where $X$ is our $N \times d$ matrix and $Y$ is our $N \times 1$ output vector. If we define:
\begin{equation}
  \Sigma \coloneqq \frac{1}{N}X^\intercal X,~u = \frac{1}{N}X^\intercal Y,
\end{equation}
then, this expression can be written as
\begin{equation}
  L(w) = \frac{1}{2}w^\intercal\Sigma w - u^\intercal w + \frac{1}{2N}\lVert Y \rVert^2.
\end{equation}

Note that $\Sigma$ is a positive-definite matrix. Assume that $\Sigma$ is full
rank.

Newton's method is typically thought of a being better than gradient descent due
to it using second order information. At some $w_0$, the method first constructs
a second-order Taylor's approximation around $w_0$, and then it makes the new
iterate to be the point at which the gradient of this approximation is $0$, that
is,
\begin{equation}
  w \leftarrow w - \left[ \nabla^2 L(w) \right]^{-1} \nabla L(w).
  \label{eqn:newton_update}
\end{equation}

\begin{enumerate}
\item Starting at some $w_0$ write out one step of the Newton's method in terms
  of $\Sigma$ and $u$. If you take one step, what point do you get to?

  \subsubsection*{Solution}

  Since $\Sigma$ is positive-definite, we can write $\Sigma = PD^2P^\intercal$,
  where $D$ is diagonal. Using that
  $w^\intercal \Sigma w = \left(D P^\intercal
    w\right)^\intercal\left(DP^\intercal w\right)$ and the chain rule, we have
  that
  \begin{align}
    \nabla L(w)
    &= D\left(L(w)\right)^\intercal \nonumber\\
    &= \left(\frac{1}{2} \left(2w^\intercal PD\right)\left(D P^\intercal\right)  - u^\intercal\right)^\intercal \nonumber\\
    &= \left(w^\intercal \Sigma - u^\intercal\right)^\intercal \nonumber\\
    &= \Sigma w - u.
      \label{eqn:newton_grad}
  \end{align}

  Calculating the derivative again, we have obtain
  \begin{equation}
    \nabla^2 L\left( w \right) = \Sigma.
    \label{eqn:newton_hessian}
  \end{equation}

  Subtituting Equations \ref{eqn:newton_grad} and \ref{eqn:newton_hessian} into
  Equation \ref{eqn:newton_update}, we obtain
  \begin{equation}
    w_1 = w_0 - \Sigma^{-1}\left(
      \Sigma w_0 - u
    \right) = \boxed{\Sigma^{-1} u.}
    \label{eqn:newton_w1}
  \end{equation}

\item Comment on the loss of this point and how it compares to the minimal
  function value?

  \subsubsection*{Solution}

  Equation \ref{eqn:newton_w1} expands to
  \begin{equation}
    w_1 = \Sigma^{-1} u = \left(X^\intercal X\right)^{-1}X^\intercal Y,
  \end{equation}
  which is the solution to the least squares problem, so $L(w_1)$ is the global
  minimum function value.
\end{enumerate}

\subsection*{Non-convex Case}

Consider the objective function:
\begin{equation}
  F(w) = \frac{1}{2}w^\intercal A w - b^\intercal w + c,
\end{equation}
where $A$ is a symmetric matrix, $b$ is a vector, and $c$ is a scalar. Our goal
is to minimize the objective function. Assume that $\Sigma$ is full rank.

Assume that $A$ is still symmetric, yet suppose now that there is at least one
negative eigenvalue. Hence, the problem is non-convex. Again, suppose you start
at $w_0$ and take a step of Newton's method. Let this new point be $w_1$.

\begin{enumerate}
\item What is the point $w_1$ and the object value $F\left(w_1\right)$.

  \subsubsection*{Solution}

  The calcuation for the gradient is similar to that in Equation
  \ref{eqn:newton_grad}:
  \begin{equation}
    \nabla F(w) = Aw - b.
    \label{eqn:newton_grad2}
  \end{equation}
  It's notable that $D$ will now have at least one complex entry on its
  diagonal. However, we only $D$ for an intermediate step. The final solution
  will still contain real values.

  Then, the Hessian is constant-valued:
  \begin{equation}
    \nabla^2 F(w) = A.
    \label{eqn:newton_hessian2}
  \end{equation}

  Substituting Equations \ref{eqn:newton_grad2} and \ref{eqn:newton_hessian2}
  into Equation \ref{eqn:newton_update}, we get that
  \begin{equation}
    w_1 = w_0 - A^{-1}\left(Aw_0 - b\right) = \boxed{A^{-1}b.}
    \label{eqn:newtwon_w12}
  \end{equation}
  The objective value is
  \begin{align}
    F\left(w_1\right)
    &= \frac{1}{2} \left(b^\intercal A^{-1}\right)A\left(A^{-1}b\right)
      - b^\intercal \left(A^{-1}b\right) + c \nonumber\\
    &= \boxed{-\frac{1}{2}b^\intercal A^{-1}b + c.}
      \label{eqn:newton_min_object2}
  \end{align}

\item What is the gradient at $w_1$?
  \subsubsection*{Solution}

  Substituting Equation \ref{eqn:newtwon_w12} into Equation
  \ref{eqn:newton_grad}, we get that
  \begin{equation}
    \nabla F(w_1) = Aw_1 - b = \boxed{0.}
  \end{equation}
\item What is the minimal object value of $F\left(\cdot\right)$? Did we achieve
  it?

  \subsubsection*{Solution}

  The minimal objective value is $-\infty$. We did not achieve it, for Equation
  \ref{eqn:newton_min_object2} is finite.

  To see that the minimal objective value is $-\infty$. Let $v$ be an
  eigenvector of $A$ associated with eigenvalue $\lambda < 0$. Then, we have
  \begin{align*}
    F\left(v\right)
    &= \frac{1}{2}v^\intercal A v - b^\intercal v + c \\
    &= \frac{\lambda}{2}\lVert v \rVert^2 - b^\intercal v + c.
  \end{align*}
  $\left\lvert b^\intercal v \right\rvert \leq \lVert b \rVert \lVert v \rVert$,
  so by increasing the magnitude of $v$, we can push the function value towards
  infinity.
\item Suppose you now are at $w_1$. In terms of the eigendecomposition
  $A = UDU^\intercal$, what are the directions of movement which \emph{strictly}
  decrease the objective value from $w_1$? Why?

  \subsubsection*{Solution}

  Let $v$ be an eigenvector with a negative eigenvalue. Then, for any
  $\epsilon > 0$, we have that
  \begin{equation*}
    \nabla F(w_1 + \epsilon v) = A\left(w_1 + \epsilon v\right) - b = \epsilon\lambda v,
  \end{equation*}
  where $\lambda < 0$ is the eigenvalue associated with $v$.

  By definition,
  $D_vF(w_1 + \epsilon v) = \nabla F(w_1 + \epsilon v) \cdot v = \epsilon\lambda
  \lVert v\rVert^2 < 0$, so $F$ is always decreasing in the direction $v$
  relative to $w_1$. This will be true for any $v$ that is a linear combination
  of eigenvectors with negative eigenvalues.
\end{enumerate}

\section*{Problem 4: Empirical Optimization}

We will now consider the multi-label classification problem. In the multi-label
problem, there are multiple labels that could be ``on'' for each input $x$. You
will use either the square loss or the binary logistic loss and consider
training two models, namely (i) a linear model and (ii) a multi-layer perceptron
(MLP) with a number of hidden nodes that you will tune.

You will try out three methods in each of the following: (1) SGD with a
mini-batch size that you tune. You will use the same minibatch size for the
other algorithms; (2) try out Polyak’s ``heavy ball method'' (aka momentum) or
Nesterov’s accelerated gradient descent (NAG); and (3) either Adagrad or
Adam. You must tune all the parameters of these methods.

The dataset contains 18 total categories with a number of categories for each
supercategory (vehicle or animal). In the dataset provided, each image contains
objects of a single supercategory, say vehicle, and potentially multiple objects
from the supercategory, such as car, boat, etc. In this exercise we shall build
a classifier that learns to identify \emph{all the categories of objects}
present in each image, by optimizing either a square loss or a logistic loss
objective. For the purposes of learning these classifiers, we shall use the
dataset and features from the first homework. We shall also provide a larger
version of this dataset since we need to train more parameters for this model.

The object function we choos to optimize is
\begin{equation}
  L(w) = \frac{\lambda}{2}\lVert w \rVert^2 +
  \frac{1}{n}\sum_{i=1}^n\sum_{j=1}^k l\left(y_{ij},f_{ij}(w)\right),
\end{equation}
where $f_{ij}(w) = w_j^\intercal x_i$ and $w_j \in \mathbb{R}^d$ is the $j$th
column of $w \in \mathbb{R}^d \times \mathbb{R}^k$. Here, $w$ is the linear
model we wish to optimize over and $\lambda > 0$ is the strength of $l_2$
regularization. here $l$ is the loss function:
\begin{itemize}
\item $l\left(y,\hat{y}\right) = \frac{1}{2}\left(y - \hat{y}\right)^2$ is the square error
  loss.
\item
  $l\left(y,\hat{y}\right) = y\log\left(1 + \exp\left(-\hat{y}\right)\right) +
  (1-y)\log\left(1 + \exp\left(\hat{y}\right)\right)$ is the logistic loss where
  the true label $y \in \{0,1\}$.
\end{itemize}

Notice that we encode $y_i$ as binary vector of length $k = 18$ (the number of
categories) where a $1$ indicates the presence of a category and $0$ indicates
the absence.

Determine which loss function works better for a linear classifier and use that
loss throughout the question.

When using $MLP$,
$f_{ij}(w) = \left\langle w_j^{(2)},
  \operatorname{relu}\left(w^{(1)}x_i\right)\right\rangle$, where
$w^{(1)} \in \mathbb{R}^h \times \mathbb{R}^d$ are the weights in the first
layer and $h$ is the number of hidden nodes. Again $w_j^{(2)} \in \mathbb{R}^h$
is the $j$th column of $w^{(2)} \in \mathbb{R}^h \times \mathbb{R}^k$, the
weights of the second layer.

\subsection*{SGD and Linear Regression}

Now consider running stochastic gradient descent on $L(w)$.

\begin{enumerate}
\item What mini-batch size do you use? What stepsize did you use? What value of
  $\lambda$ did you use? Specify your stepsize scheme if you chose to decay your
  stepsize. Which loss function did you find works better?

  \subsubsection*{Solution}

  I used a mini-batch size of $8$. My learning rate was $10^{-4}$. I used
  $\lambda = 0.7$. I found logistic loss to work better.

\item Report your training and validation loss.

  \subsubsection*{Solution}

  The results for the linear model can be seen in Figure
  \ref{fig:linear_loss_sgd} and Table \ref{tab:linear_sgd}. Training loss was
  0.122861 and validation loss was 0.199776.

  \begin{figure}
    \centering
    \includegraphics{problem4/linear_loss_sgd.pdf}
    \caption{Some overfiiting likely occurred since training loss is
      significantly less that validation loss. Increasing $\lambda$ made the
      model perform worse, however.}
    \label{fig:linear_loss_sgd}
  \end{figure}

  \begin{table}
    \centering
    \begin{tabular}{lrr}
      \toprule
      {} &  Average Precision Score &      Loss \\
      \midrule
      Training   &                 0.871461 &  0.122861 \\
      Validation &                 0.563140 &  0.199776 \\
      Test       &                 0.636070 &  0.177882 \\
      \bottomrule
    \end{tabular}
    \caption{Best average precision score and loss when training the linear
      model with SGD.}
    \label{tab:linear_sgd}
  \end{table}

  The results for the MLP model can be seen in Figure \ref{fig:mlp_loss_sgd} and
  Table \ref{tab:mlp_sgd}. Training loss was 0.157085 and validation loss was
  0.190817. I used $\lambda = 0.05$ as my regularization parameter and 1 layer
  of 256 hidden units. The learning rate was 0.0005. Validation loss is better
  than the linear model, but test loss is worst.

  \begin{figure}
    \centering
    \includegraphics{problem4/mlp_loss_sgd.pdf}
    \caption{The model appears to not have converged yet.}
    \label{fig:mlp_loss_sgd}
  \end{figure}

  \begin{table}
    \centering
    \begin{tabular}{lrr}
      \toprule
      {} &  Average Precision Score &      Loss \\
      \midrule
      Training   &                 0.708156 &  0.157085 \\
      Validation &                 0.565152 &  0.190817 \\
      Test       &                 0.596515 &  0.181875 \\
      \bottomrule
    \end{tabular}
    \caption{Best average precision score and loss when training the MLP
      model with SGD.}
    \label{tab:mlp_sgd}
  \end{table}

\item Compute the average precision score in the same manner, and plot these
  quantities.

  \subsubsection*{Solution}

  The results are in Tables \ref{tab:linear_sgd} and \ref{tab:mlp_sgd}. See
  Figures \ref{fig:linear_average_precision_score_sgd} and
  \ref{fig:mlp_average_precision_score_sgd}

  \begin{figure}
    \centering
    \includegraphics{problem4/linear_average_precision_score_sgd.pdf}
    \caption{Average precision score by the number of steps for the linear
      model.}
    \label{fig:linear_average_precision_score_sgd}
  \end{figure}

  \begin{figure}
    \centering
    \includegraphics{problem4/mlp_average_precision_score_sgd.pdf}
    \caption{Average precision score by the number of steps for the multi-layer
      perceptron model.}
    \label{fig:mlp_average_precision_score_sgd}
  \end{figure}

\item Report the overall average precision score on the test set. Also report
  the average precision score for each of the 18 classes separately. Comment on
  how the AP scores relate to the class imbalance.

  \subsubsection*{Solution}

  Sorry, I ran out of time and didn't get this far :(.

\end{enumerate}

\subsection*{Nesterov's method}

I used Nesterov's method with a momentum parameter of 0.9. All other parameter
were the same as last section. The results can be seen in Tables
\ref{tab:linear_nesterov} and \ref{tab:mlp_nesterov}. The precision for the MLP
model is notably better with an average precision score of 0.629526 on the test
set.

The progress of the training runs can be seen in Figures
\ref{fig:linear_loss_nesterov}, \ref{fig:mlp_loss_nesterov},
\ref{fig:linear_average_precision_score_nesterov}, and
\ref{fig:mlp_average_precision_score_nesterov}. This method trained the fastest
out of the three that I tried. In particular the linear model converged very
quickly.

  \begin{table}
    \centering
    \begin{tabular}{lrr}
      \toprule
      {} &  Average Precision Score &      Loss \\
      \midrule
      Training   &                 0.759774 &  0.170063 \\
      Validation &                 0.521453 &  0.262688 \\
      Test       &                 0.581486 &  0.232664 \\
      \bottomrule
    \end{tabular}
    \caption{Best average precision score and loss when training the linear
      model with SGD and Nesterov's momentum.}
    \label{tab:linear_nesterov}
  \end{table}

    \begin{table}
      \centering
      \begin{tabular}{lrr}
        \toprule
        {} &  Average Precision Score &      Loss \\
        \midrule
        Training   &                 0.793293 &  0.141046 \\
        Validation &                 0.587959 &  0.188455 \\
        Test       &                 0.629526 &  0.177171 \\
        \bottomrule
      \end{tabular}
      \caption{Best average precision score and loss when training the MLP
        model with SGD and Nesterov's momentum.}
    \label{tab:mlp_nesterov}
  \end{table}

  \begin{figure}
    \centering
    \includegraphics{problem4/linear_loss_nesterov.pdf}
    \caption{Evaluations on the datasets while training the linear model with
      Nesterov's momentum.}
    \label{fig:linear_loss_nesterov}
  \end{figure}

    \begin{figure}
    \centering
    \includegraphics{problem4/mlp_loss_nesterov.pdf}
    \caption{Evaluations on the datasets while training the MLP model with
      Nesterov's momentum.}
    \label{fig:mlp_loss_nesterov}
  \end{figure}

  \begin{figure}
    \centering
    \includegraphics{problem4/linear_average_precision_score_nesterov.pdf}
    \caption{Average precision score by the number of steps for the linear
      model.}
    \label{fig:linear_average_precision_score_nesterov}
  \end{figure}

  \begin{figure}
    \centering
    \includegraphics{problem4/mlp_average_precision_score_nesterov.pdf}
    \caption{Average precision score by the number of steps for the multi-layer
      perceptron model.}
    \label{fig:mlp_average_precision_score_nesterov}
  \end{figure}

  \subsection*{Adagrad}

  For the linear model, I used the same learning rate of $10^{-4}$. The results
  were very similar to the previous results. The results can be seen in Tables
  \ref{tab:linear_adagrad}.

  For the MLP model, I had trouble getting convergence. I used Adagrad method
  with a learning rate of $0.0005$. Judging from the results, this was likely,
  too small. Making it larger didn't seem to helper, either. Perhaps, step size
  decays, too, quickly. All other parameters were the same as last section. The
  results can be found in \ref{tab:mlp_adagrad}. The precision for the MLP model
  is notably better with an average precision score of 0.629526 on the test set.

  The progress of the training runs can be seen in Figures
  \ref{fig:linear_loss_adagrad}, \ref{fig:mlp_loss_adagrad},
  \ref{fig:linear_average_precision_score_adagrad}, and
  \ref{fig:mlp_average_precision_score_adagrad}.

  \begin{table}
    \centering
    \begin{tabular}{lrr}
      \toprule
      {} &  Average Precision Score &      Loss \\
      \midrule
      Training   &                 0.820442 &  0.136633 \\
      Validation &                 0.542389 &  0.204523 \\
      Test       &                 0.609165 &  0.184826 \\
      \bottomrule
    \end{tabular}
    \caption{Best average precision score and loss when training the linear
      model with SGD and Adagrad's momentum.}
    \label{tab:linear_adagrad}
  \end{table}

    \begin{table}
      \centering
      \begin{tabular}{lrr}
        \toprule
        {} &  Average Precision Score &      Loss \\
        \midrule
        Training   &                 0.605249 &  0.186454 \\
        Validation &                 0.531518 &  0.199406 \\
        Test       &                 0.546195 &  0.196277 \\
        \bottomrule
      \end{tabular}
      \caption{Best average precision score and loss when training the MLP
        model with SGD and Adagrad's momentum.}
    \label{tab:mlp_adagrad}
  \end{table}

  \begin{figure}
    \centering
    \includegraphics{problem4/linear_loss_adagrad.pdf}
    \caption{Evaluations on the datasets while training the linear model with
      Adagrad's momentum.}
    \label{fig:linear_loss_adagrad}
  \end{figure}

    \begin{figure}
    \centering
    \includegraphics{problem4/mlp_loss_adagrad.pdf}
    \caption{Evaluations on the datasets while training the MLP model with
      Adagrad's momentum.}
    \label{fig:mlp_loss_adagrad}
  \end{figure}

  \begin{figure}
    \centering
    \includegraphics{problem4/linear_average_precision_score_adagrad.pdf}
    \caption{Average precision score by the number of steps for the linear
      model.}
    \label{fig:linear_average_precision_score_adagrad}
  \end{figure}

  \begin{figure}
    \centering
    \includegraphics{problem4/mlp_average_precision_score_adagrad.pdf}
    \caption{Average precision score by the number of steps for the multi-layer
      perceptron model.}
    \label{fig:mlp_average_precision_score_adagrad}
  \end{figure}

\end{document}
% Local Variables:
% TeX-command-extra-options: "-shell-escape"
% End: