\chapter{Proofs by Contradiction}
\label{chapter:indirect-proofs}
\marginurl{%
  Proofs by Contradiction:\\\noindent
  Introduction to Mathematical Reasoning \#3
}{youtu.be/bWP0VYx75DI}
\section{Proving Negative Statements}
The direct method is not very convenient when we need to prove the negation of
a statement.

For example, we may try to prove that $78 n + 102 m = 11$ does not have integer
solutions. It is not clear how to prove it directly since we can not consider
all possible $n$ and $m$. Hence, we need another approach. Let us assume that
such a solution $n$, $m$ exists. Note that $78 n + 102 m$ is even, but $11$ is
odd. In other words, an odd number is equal to an even number, which is
impossible. Thus, the assumption was false.

Consider another example, let us prove that if $p^2$ is even, then
$p$ is also even ($p$ is an integer). Assume the opposite; i.e. that $p^2$ is
even but $p$ is not.\footnote{%
  Note that we use here the statement that an integer $n$ is not even iff it is
  odd, which, formally speaking, should be proven.
}
Let $p = 2b + 1$. Note that $p^2 = (2b + 1)^2 = 2(2b^2 + 2b) + 1$.
Hence, $p^2$ is odd which contradicts to the assumption that $p^2$ is even.

Using this idea we may prove much more complicated results e.g. one may show
that $\sqrt{2}$ is irrational. For the sake of contradiction, let us assume
that it is not true. In other words there are $p$ and $q$ such that
$\sqrt{2} = \frac{p}{q}$ and $\frac{p}{q}$ is an irreducible fraction.

Note that $\sqrt{2} q = p$, so $2q^2 = p^2$. Which implies that $p$ is even
and $4$ divides $p^2$. Therefore $4$ divides $2q^2$ and $q$ is also even. As
a result, we get a contradiction with the assumption that $\frac{p}{q}$ is an
irreducible fraction.

\begin{template}
  \textbf{Template for proving a statement by contradiction.} \\

  Assume, for the sake of contradiction, that \emph{the statement} is false.
  Then \emph{present some argument that leads to a contradiction}. Hence, the
  assumption is false and \emph{the statement} is true.
\end{template}

\begin{exercise}
  Show that $\sqrt{3}$ is irrational.
\end{exercise}

\begin{exercise}
  Let $a$ and $b$ be some integers. Show that $a^2 - 4b - 2 \neq 0$.
\end{exercise}
\begin{solution}
  Assume, for the sake of contradiction, that such $a$ and $b$ exist. Note that
  $a^2 = 4b + 2$, hence, $a$ is even but $a^2$ is not divisible for $4$ which is a
  contradiction.
\end{solution}

\section{Proving Implications by Contradiction}
This method works especially well when we need to prove an implication.
Since the implication $A \implies B$ is false only when $A$ is true but $B$ is
false, it is enough to derive a contradiction from the fact that $A$ is true
and $B$ is false.

We have already seen such examples in the previous section, we proved that
$p^2$ is even implies $p$ is even, for any integer $p$. Let us consider another
example. Let $a$ and $b$ be real numbers such that $a > b$. We need to show that
$(ac < bc) \implies c < 0$. So we may assume that $ac < bc$ but $c \ge 0$. By
the multiplicativity of the inequalities we know that if $(a > b)$ and $c > 0$,
then $ac > bc$ which contradicts to $ac < bc$.

A special case of such a proof is when we need to prove the implication
$A \implies B$, assume that $B$ is false and derive that $A$ is false which
contradicts to  $A$ (such proofs are called proofs by contraposition); note
that the previous proof is a proof in this form.

\section{Proof of ``OR'' Statements}
Another important case is when we need to prove that at least one of two
statements is true. For example, let us prove that $ab = 0$ iff $a = 0$ or
$b = 0$. We start from the implication from the right to the left. Since if
$a = 0$, then $ab = 0$ and the same is true for $b = 0$ this implication is
obvious.

The second part of the proof is the proof by contradiction. Assume $ab = 0$,
$a \neq 0$, and $b \neq 0$. Note that $b = \frac{ab}{a} = 0$,
hence $b = 0$ which is a contradiction to the assumption.

\begin{chapterendexercises}
    \exercise[recommended] Prove that if $n^2$ is odd, then $n$ is odd.
    \exercise  In Euclidean (standard) geometry, prove: If two lines share a
        common perpendicular, then the lines are parallel.
        \begin{solution}
          Let us denote by $AB$ the common perpendicular. Assume that the lines
          are not parallel (note that these lines are different) i.e. that there
          is an intersection $C$ of these lines.

          Note that the angles $CAB$ and $CBA$ are right, hence, the angle $ACB$
          is equal to $0$ degrees. So the lines are the same, which is a
          contradiction.

          Hence, the assumption was incorrect i.e. the lines are parallel.
        \end{solution}
    \exercise[recommended] Let us consider four-lines geometry, it is a theory with
        undefined terms: point, line, is on, and axioms:
        \begin{enumerate}
            \item there exist exactly four lines,
            \item any two distinct lines have exactly one point on both of them, and
            \item each point is on exactly two lines.
        \end{enumerate}

        Show that every line has exactly three points on it.
        \begin{solution}
          Let us assume the opposite; i.e. there is a line $\ell$ with four
          points $p_1, \dots, p_4$ on it. By axiom~3, for each of these points
          $p_i$ there is a line $\ell_i \neq \ell$ such that $p_i$ is on $\ell$.
          Since we have only $4$ lines and $\ell_i \neq \ell$, then there are $i
          \neq j$ such that $\ell_i = \ell_j$. As a result, $p_i$ and $p_j$ are
          on $\ell$ and $\ell_i$, it is contradicting to axiom~2.
        \end{solution}
    \exercise Let us consider group theory, it is a theory with undefined
        terms: group-element and times (if $a$ and $b$ are group elements,
        we denote $a$ times $b$ by $a \cdot b$), and axioms:
        \begin{enumerate}
          \item $(a \cdot b) \cdot c = a \cdot (b \cdot c)$
            for every group-elements $a$, $b$, and $c$;
          \item there is a unique group-element $e$ such that
            $e \cdot a = a = a \cdot e$ for every group-element $a$
            (we say that such an element is the identity element);
          \item for every group-element $a$ there is a group-element $b$
            such that $a \cdot b = e$, where $e$ is the identity element;
          \item for every group-element $a$ there is a group-element $b$
            such that $b \cdot a = e$, where $e$ is the identity element.
        \end{enumerate}

        Let $e$ be the identity element. Show the following statements
        \begin{itemize}
          \item if $b_0 \cdot a = b_1 \cdot a = e$, then $b_0 = b_1$, for every
            group-elements $a$, $b_0$, and $b_1$.
          \item if $a \cdot b_0 = a \cdot b_1 = e$, then $b_0 = b_1$, for every
            group-elements $a$, $b_0$, and $b_1$.
          \item if $a \cdot b_0 = b_1 \cdot a = e$, then $b_0 = b_1$, for every
            group-elements $a$, $b_0$, and $b_1$.
        \end{itemize}
    \exercise Let us consider three-points geometry, it is a theory with
        undefined terms: point, line, is on, and axioms:
        \begin{enumerate}
            \item There exist exactly three points.
            \item Two distinct points are on exactly one line.
            \item Not all the three points are collinear i.e. they do not lay on the
                same line.
            \item Two distinct lines are on at least one point i.e. there is at
              least one point such that it is on both lines.
        \end{enumerate}

        Show that there are exactly three lines.

        \begin{solution}
            Let us denote the points by $p_1$, $p_2$, and $p_3$ (they exist by
            Axiom~1). By Axiom~2, there are lines $l_{1, 2}$,
            $l_{1, 3}$, and $l_{2, 3}$ such that $p_i$ and $p_j$ are on
            $l_{i, j}$ ($i \neq j$).

            Note that the lines $l_{1, 2}$, $l_{1, 3}$, and $l_{2, 3}$ are different.
            Indeed, assume the opposite, i.e., without loss of generality that
            $l_{1, 2} = l_{1, 3}$. Note that $p_1$, $p_2$, and $p_3$ are on $l_{1, 2}$
            which contradicts Axiom~3.

            Let us now prove that there are no other lines. Assume the
            opposite i.e. that there is another line $l$. There is a
            point that is on $l$ and $l_{1, 2}$. Without loss of generality,
            this point is $p_1$. Additionally there is a point $p_i$
            ($i \neq 1$) that is on $l$ and  $l_{2, 3}$. However, it means that
            $p_1$ and $p_i$ are on $l$ which contradicts Axiom~2.
        \end{solution}
    \exercise Show that there are irrational numbers $a$ and $b$ such that
        $a^b$ is rational.
        \begin{solution}
          Assume that $a^b$ is irrational for all irrationals $a$ and $b$. Since
          $\sqrt{2}$ is irrational $\sqrt{2}^{\sqrt{2}}$ is also irrational.
          However, $(\sqrt{2}^{\sqrt{2}})^{\sqrt{2}} = \sqrt{2}^2 = 2$, which
          contradicts to the assumption.

          As a result, there are irrational $a$ and $b$ such that $a^b$ is
          rational.
        \end{solution}
    \exercise[recommended] Show that there does not exist the largest integer.

    \exercise Let us consider Young's geometry, it is a theory with undefined
        terms: point, line, is on, and axioms:
        \begin{enumerate}
            \item there exists at least one line,
            \item every line has exactly three points on it,
            \item not all points are on the same line,
            \item for two distinct points, there exists exactly one line on both of
              them,
            \item if a point does not lie on a given line, then there exists exactly
              one line on that point that does not intersect the given line.
        \end{enumerate}

        Show that for every point, there are exactly four lines on that point.
        \begin{solution}
          First, we prove that for every point, there is a line not on that
          point. Let $p$ be some point. By the first axiom, there is a line
          $\ell$. Assume that $p$ is on $\ell$ (otherwise we proved the
          statement). By axiom 2, there are two other points $p_1$ and $p_2$ on
          this line. By axiom 3, there is a point $q$ not on $\ell$. Finally, by
          axiom 4, there is a line $\ell'$ on $p_1$ and $q$. Note that $\ell'
          \neq \ell$, thus $p$ is not on $\ell'$.

          Now we are ready to prove that there are at least four lines. Let $p$
          be a point and $\ell$ be a line such that $p$ is not on $\ell$. By
          axiom 2, there are three points $p_1$, $p_2$, and $p_3$ on $\ell$. By
          axiom 4, there are lines $\ell_1$, $\ell_2$, and $\ell_3$ such that
          $p$ is on all of them and $p_i$ is on $\ell_i$ for $i \in \range{3}$.
          By axiom 5, there is a line $\ell_4$ contain $p$ but $p_1$, $p_2$,
          and $p_3$  are not on $\ell_4$. Thus, there are at least four lines
          through $p$.

          Let us now prove that there is no other line $\ell_5$ such that $p$ is
          on $\ell_5$. By axiom 5, there is a point $p_i$ such that $p_i$ is on
          both $\ell_5$ and $\ell$. But this contradicts to axiom 4, since $p_i$
          is also on $\ell_i$.
        \end{solution}
    \exercise Let us consider five-point geometry, it is a theory with undefined
      terms: point, line, is on, and axioms:
      \begin{enumerate}
        \item there exist exactly five points,
        \item each two distinct points have exactly one line on both of them, and
        \item each line has exactly two points.
      \end{enumerate}

      Show that each point has exactly four lines on it.
      \begin{solution}
        Let us consider some point $p$ and let $p_1$, \dots, $p_4$ be all other
        points (they exist by axiom~1). Note that there are lines $l_1$, \dots,
        $l_4$ such that $l_i$ goes throw $p$ and $p_i$. Note that $l_i \neq l_j$
        for $i \neq j$ by axiom~3 since $p_i$ is not on $l_j$ and there are
        already two points on $l_i$. Thus we proved that any point is on at
        least $4$ lines.

        Let us now prove that there are no other lines. Assume that $p$ is on a
        line $l$ and $l$ is not equal to $l_i$ for all $i$. However, by axiom 2,
        there is a point $p_j$ on $\ell$. Thus $l = l_j$ which is a contradiction.
      \end{solution}
\end{chapterendexercises}


\section{Homotopy, star-shaped regions}
As homology is a functor $H_\ast:\mathbf{Top}\to\mathbf{Ab}$, it preserves isomorphisms: homeomorphic spaces have isomorphic homology. However, homology is not always able to distinguish between non-homeomorphic spaces. We introduce the looser notion of homotopy, which is a central concept of algebraic topology, and later we will show that homology is a homotopy invariant.
\begin{definition}
Let $f_0,f_1:X\to Y$ be two maps. A \emph{homotopy} from $f_0$ to $f_1$ is a map $h:X\times I\to Y$ such that $h(x,0)=f_0(x)$ and $f(x,1)=f_1(x)$. We say that $f_0$ and $f_1$ are \emph{homotopic} and write $f_0\sim f_1$.  This notation is justified because it is indeed an equivalence relation (transitivity follows from the gluing lemma).
\end{definition}
\begin{figure}
	\centering
	\includegraphics[width=0.5\linewidth]{assets/L05/05-homotopy}
	\caption{A homotopy $h$ between $f_0,f_1:X\to Y$.}
	\label{fig:05-homotopy}
\end{figure}

We denote by $[X,Y]$ the set $\mathbf{Top}(X,Y)/\sim$.

Suppose we have a map $g: Y\to Z$ and homotopic maps $f_0,f_1:X\to Y$, with a homotopy $h:f_0\sim f_1$. Then $g\circ h$ gives a homotopy $g\circ f_0 \sim g\circ f_1$. Similarly, if $g:W\to X$ is a map and $f_0,f_1:X\to Y$ are homotopic, then $f_0\circ g\sim f_1\circ g$.

If $g_0\sim g_1:Y\to Z$ and $f_0\sim f_1: X\to Y$, then $g_0\circ f_0\sim g_0 \circ f_1\sim g_1 \circ f_1$. Hence we are able to compose homotopy classes, giving the dotted arrow below.
\begin{equation*}
\xymatrix{\mathbf{Top}(Y,Z)\times\mathbf{Top}(X,Y)\ar[d]\ar[r] & \mathbf{Top}(X,Z)\ar[d]\\
[Y,Z]\times[X,Y]\ar@{-->}[r] & [X,Z]}
\end{equation*}

\begin{definition}
The \emph{homotopy category of topological spaces} is $\mathrm{Ho}(\mathbf{Top})$ whose objects are topological spaces and $\mathrm{Ho}(\mathbf{Top})(X,Y)=[X,Y]=\mathbf{Top}(X,Y)/\sim$.
\end{definition}
\begin{definition}
	A map $f:X\to Y$ is a \emph{homotopy equivalence} if $[f]\in[X,Y]$ is an isomorphism in $\htop$. In other words, there is $g:Y\to X$ such that $fg\sim 1_Y$ and $gf\sim 1_X$.
\end{definition}
%This is an interesting category because it has \textit{terrible} categorical properties\todo{This sentence seems a bit random atm; give examples of why it's terrible? or just remove?}.

In Section \ref{lec:6-homotopy-invariance-of-homology} we will prove:
\begin{theorem}[Homotopy invariance of homology]
	If $f_0\sim f_1$, then $ H_\ast(f_0)= H_\ast(f_1)$.
\end{theorem}
This theorem states that the homology functor $H_\ast:\mathbf{Top}\to\mathbf{Ab}$ factors as $\mathbf{Top}\to\mathrm{Ho}(\mathbf{Top})\to\mathbf{Ab}$. In particular, it cannot distinguish between homotopy equivalent spaces. (Caution: spaces with isomorphic homology need not be homotopy equivalent.)
\begin{example}\label{exa:homotopy-equivalence-sphere}
The inclusion $S^{n-1}\subseteq \mathbf{R}^n-\{0\}$ is a homotopy equivalence. The homotopy inverse $p:\mathbf{R}^n-\{0\}\to S^{n-1}$ can be obtained by dividing a (always nonzero!) vector by its length. Clearly $p\circ i=1_{S^{n-1}}$. A homotopy $i\circ p\sim 1_{\mathbf{R}^n-\{0\}}$ is given by $(v,t)\mapsto tv+(1-t)\frac{v}{||v||}$. This example shows that homotopy equivalence does not preserve compactness.
\end{example}
\begin{definition}
A space $X$ is \emph{contractible} if the map $X\to\ast$ is a homotopy equivalence.
\end{definition}

\begin{definition}
	A \emph{star-shaped region} is a subspace $X$ of Euclidean space containing $\{0\}$, such that for all $x\in X$ and $t\in[0,1]$, $tx\in X$.
\end{definition}
\begin{figure}
	\centering
	\includegraphics[width=0.4\linewidth]{assets/L05/05-star-shaped-region}
	\caption{A star-shaped region.}
	\label{fig:05-star-shaped-region}
\end{figure}

For example, any convex region containing the origin is star-shaped. An argument similar to the one in Example \ref{exa:homotopy-equivalence-sphere} shows that the inclusion of $\{0\}$ into a star-shaped region is a homotopy equivalence. Thus star-shaped regions are contractible. Our goal now is to prove the following.
\begin{theorem}\label{thm:star-shaped-homology}
	Let $X$ be a star-shaped region. The augmentation map $\epsilon: H_\ast(X)\to \mathbf{Z}$ is an isomorphism, i.e., $ H_0(X)\cong\mathbf{Z}$ and $ H_i(X)\cong 0$ for $i>0$.
\end{theorem}
%The strategy of proof is that $\epsilon$ is induced by sending $X\to \ast$. We'll look at the chain map $S_\ast(x)\to\mathbf{Z}\to S_\ast(X)$. We'll show that this composite induces the same map in homology as the identity map, which means that the identity map factors through $\mathbf{Z}$, so we're done. (probably not necessary to include this now that it's been moved to this section, given that the proof follows shortly)
Before proving this, we will give a notion of homotopy in the category of chain complexes.
\begin{definition}
Let $C_\bullet,D_\bullet$ be chain complexes, and $f_0,f_1:C_\bullet\to D_\bullet$ be chain maps. A \emph{chain homotopy} $h:f_0\sim f_1$ is a collection of homomorphisms $h:C_n\to D_{n+1}$ such that $\partial h+h\partial=f_1-f_0$.
\end{definition}
			\begin{equation*}
			\xymatrix{C_{n+2}\ar[r]^{f_1-f_0}\ar[d]^\partial & D_{n+2}\ar[d]^\partial\\
			C_{n+1}\ar@{-->}[ur]^h\ar[r]^{f_1-f_0}\ar[d]^\partial & D_{n+2}\ar[d]^\partial\\
			C_n\ar@{-->}[ur]^h\ar[r]_{f_1-f_0} & D_n}
			\end{equation*}
The following lemma shows the significance of this condition.
\begin{lemma}
	If $f_0,f_1:C_\bullet\to D_\bullet$ are chain homotopic, then $f_{0,\ast}=f_{1,\ast}: H(C)\to H(D)$.
\end{lemma}
\begin{proof}
	We show that $(f_1-f_0)_\ast=0$. Let $c\in Z_n(C_\bullet)(C)$, so that $\partial c=0$. Then $(f_1-f_0)_\ast c=(\partial h+h\partial)c=\partial hc+h\partial c=\partial hc$ is a boundary, which is zero in homology.
\end{proof}
\begin{proof}[Proof of Theorem \ref{thm:star-shaped-homology}]
	The maps $\{0\}\to X$ and $X\to \{0\}$ induce $\mathbf{Z}\xrightarrow{\eta}S_\ast(X)$ and $S_\ast(X)\xrightarrow{\epsilon}\mathbf{Z}$ respectively. It is clear that $\epsilon\eta = 1:\mathbf{Z}\to\mathbf{Z}$. We now proceed to show that $\eta\epsilon\sim 1:S_\ast(X)\to S_\ast(X)$. Note that the composite $\eta \epsilon$ kills all chains in dimensions greater than zero, and on zero-chains we have $\eta\epsilon(\sum a_ix_i)=(\sum a_i)c_0$ where $c_0$ is the zero-simplex at the origin.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.8\linewidth]{assets/L05/05-star-shaped-line-homotopy-pf}
	\caption{The map $h$ illustrated on a 1-simplex.}
	\label{fig:05-star-shaped-line-homotopy-pf}
\end{figure}

	For $\sigma\in\Sin_q(X)$, define $h:\Sin_q(X)\to\Sin_{q+1}(X)$ as follows:
	$$h\sigma(t_0,\cdots,t_{q+1})=(1-t_0)\sigma\left(\frac{(t_0,\cdots,t_{q+1})}{1-t_0}\right).$$
	This extends by linearity to a map $h:S_q(X)\to S_{q+1}(X)$.
	Observe that $d_0 h \sigma = \sigma$, and if $q \geq 1$, $d_i h \sigma = h d_{i-1} \sigma$. Then, if $\sigma \in \Sin_q(X)$ with $q \geq 1$,
	\begin{align*}
		\partial h \sigma &= \sum_{i=0}^{q+1} (-1)^{i} d_i h\sigma\\
		&= \sigma - h \sum_{i=0}^q (-1)^i d_i \sigma\\
		&= \sigma - h\partial \sigma\\
		\partial h \sigma + h \partial \sigma &= \sigma.
	\end{align*}
	If $\sigma \in \Sin_0(X)$, then $d_1 h \sigma = c_0$ and we instead get $\partial h \sigma + h \partial \sigma = \sigma - c_0$. Thus $h:S_q(X)\to S_{q+1}(X)$ is a chain homotopy $\eta \epsilon \sim 1$ as desired.
\end{proof}
% Continue indenting like this.


\documentclass[11pt,twoside]{article}

\usepackage{amsmath,amsthm}
\usepackage{listings}
\usepackage[headings]{fullpage}
\usepackage[utopia]{mathdesign}
\usepackage{color}
\usepackage{graphicx}

\lstset{basicstyle=\footnotesize\ttfamily,language=Matlab}

\pagestyle{myheadings}
\markboth{Volume}{Volume}

\input{../fncextra}

\begin{document}

\begin{center}
  \bf Depths of despair
\end{center}

Numerical integration of a function of one variable is called quadrature. The analogous situation of integrating a function of two variables is sometimes called \emph{cubature}. In this project you will explore the extension of the trapezoid formula to this situation, and apply it to find the volume of water in a patch of the ocean.

Consider the problem $\iint_R f(x,y)\,dx\,dy$, where $R$ is the rectangle $[x_0,x_m] \times [y_0,y_n]$. We discretize each variable using the space steps $h$ and $k$, respectively, so that $x_i = x_0 + ih$ for $i=0,\ldots,n$, and $y_j=y_0+jk$ for $j=0,\ldots,m$. (This requires that $h=(x_m-x_0)/m$ and $k=(y_n-y_0)/n$.) The result is a decomposition of $R$ into rectangles with the nodes at the corners. The strategy is to compute a volume over each little rectangle and sum them over the whole domain. Here's an example of how the discretization process works in MATLAB:

\input{CubatureSetup}

At the corners of each rectangle we have values of $z$ according to $Z_{ij}=f(x_i,y_j)$. We can model the surface $z=f(x,y)$ over one of these rectangles using the function $L_{ij}(x,y)=a + b (x-x_i) + c(y-y_j) + d(x-x_i)(y-y_j)$, where $a,b,c,d$ are determined by interpolation conditions at the four corners of the rectangle. (These coefficients depend on $i$ and $j$, but that is left out to make the notation clearer.) We get a solid defined over the rectangle looking like this:

\input{CubatureSolid}

\subsection*{Objectives}
\label{sec:objectives}

Mathematical content:
\begin{enumerate}
\item Using a symbolic math package if you want, derive exact expressions for the interpolant $L_{ij}$ over the rectangle whose lower left corner is $(x_i,y_j)$, in terms of $h$, $k$, and four values $Z_{ij}$, $Z_{i+1,j}$, $Z_{i,j+1}$, $Z_{i+1,j+1}$. (Show the steps involved, whether you use a computer or not.)
\item Integrate the interpolant over the rectangle to get another expression in terms of the same quantities. This is $V_{ij}$, the volume used to approximate the integral of $f$ over the rectangle.
\item Show that if the $V_{ij}$ are added up over all $i$ and $j$, the result is identical to applying the trapezoid formula in each dimension sequentially over the domain (i.e., as an iterated integral).
\end{enumerate}

Computational content:
\begin{enumerate}
\item Download and install the GeoMapApp at \texttt{www.geomappapp.org}.
\item Start the app and agree to the license. You should get a window with a projected map of the world, in latitude-longitude corrdinates as measured in degrees of arc, and some GUI widgets. Use the magnifying glass tool to select a small rectangle over the ocean. You should try to get it to include about 1-2 degrees of latitiude (the $y$ axis).
\item Click on the icon that looks like a grid. After a few seconds another window should open with a histogram in it. Make sure the dropdown menu at the top of this window says ``GMRT Grid'' with some version number. Click on the floppy disk icon, select ``Grid: .XYZ (ascii format)'', and save the file to your computer. At this point you can close GeoMapApp.
\item You can import this data into MATLAB by double-clicking on it in the ``Current Folder'' tab of the MATLAB Desktop, and using the resulting dialog. What you will get is three columns of $x$, $y$, and $z$ data on a regular rectangular grid, where $x$ and $y$ are reported in degrees of arc, and $z$ will be negative to represent depths below sea level.
\item \textbf{After converting all the data into km units}, use it and the formula you derived above to estimate the amount of water in your patch of ocean.
\end{enumerate}

\subsection*{Submission}

Submit a function
\begin{verbatim}
function volume = ocean(x,y,z)
\end{verbatim}
that will compute an approximation to the volume using three vectors of depth data as described in the previous section.

Prepare a report that includes the following.
\begin{enumerate}
\item The required mathematical content.
\item Details needed to acquire the same raw data from the app and compute your volume result.
\item Some form of quantitative check on whether your answer is reasonable, in the sense of being at least the right order of magnitude.
\item A brief statement of the contributions of each team member to the project. \textbf{Be as specific as possible.}
\end{enumerate}
Your report should read like a document understandable to anyone else in the class. All of the mathematical expressions in your document should be typeset properly as mathematical notation. You have several options: Word, OpenOffice, \LaTeX, \texttt{texmacs.org}, Mathematica. Submit your report as a single PDF document, along with your M-file.

\end{document}


\section{Code}

\begin{minted}{python}
# -*- coding: utf-8 -*-
"""
TRABAJO 1.
Autor Blanca Cano Camarero
Grupo 2
"""

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D # to display 3d function

np.random.seed(1)
def STOP_EXECUTION_TO_SEE_RESULT():
        input('\n--- End of a section, press any enter to continue ---\n')

print('GRADIENT DESCENDENT\n')
print('Exercise 1\n')

## 1

def gradient_descent(initial_point, loss_function, gradient_function,  eta, max_iter, target_error):
    '''
    initicial point: w_0
    E: error function
    gradient_function
    eta:  step size

    ### stop conditions ###
    max_iter
    target_error

    #### return ####
    (w,iterations)
    w: the coordenates that minimize E
    it: the numbers of iterations needed to obtain w

    '''

    iterations = 0
    error = E( initial_point[0], initial_point[1])
    w = initial_point

    while ( (iterations < max_iter) and(error > target_error)):

        w = w - eta * gradient_function(w[0], w[1])

        iterations += 1
        error = loss_function(w[0], w[1])

    return w, iterations

############  2 ###################################
def E(u,v):
    '''
    Function to minimize
    '''
    return np.float64(
        ( u**3 * np.e**(v-2) - 2*v**2 * np.e**(-u) )**2
    )

def dEu(u,v):
    '''
    Partial derivate of E with respect to the variable u
    '''
    return np.float64(
        2
        *( u**3 * np.e**(v-2) - 2*v**2 * np.e**(-u))
        *( 3* u**2 * np.e**(v-2) + 2*v**2 * np.e**(-u))
    )

def dEv(u,v):
    '''
    Partial derivate of E with respect to the variable v
    '''
    return np.float64(
        2
        *( u**3 * np.e**(v-2) - 2*v**2 * np.e**(-u) )
        *( u**3 * np.e**(v-2) - 4*v * np.e**(-u))
    )

def gradE(u,v):
    '''
        gradient of E
    '''
    return np.array([dEu(u,v), dEv(u,v)])

######  conditions
eta = 0.1
max_iter = 10000000000
target_error = 1e-14
initial_point = np.array([1.0,1.0])
w, it = gradient_descent( initial_point,E, gradE, eta, max_iter, target_error )

# DISPLAY FIGURE

x = np.linspace(-30, 30, 50)
y = np.linspace(-30, 30, 50)
X, Y = np.meshgrid(x, y)
Z = E(X, Y) #E_w([X, Y])
fig = plt.figure()
ax = Axes3D(fig)
surf = ax.plot_surface(X, Y, Z, edgecolor='none', rstride=1,
                        cstride=1, cmap='jet')
min_point = np.array([w[0],w[1]])
min_point_ = min_point[:, np.newaxis]
ax.plot(min_point_[0], min_point_[1], E(min_point_[0], min_point_[1]), 'r*', markersize=10)
ax.set(title='Ejercicio 1.2. Función sobre la que se calcula el descenso de gradiente')
ax.set_xlabel('u')
ax.set_ylabel('v')
ax.set_zlabel('E(u,v)')
plt.show()

######### Exercise 1, part 2 answers  ######
print('2 a) Function:  E(u,v) = (u^3 e^{(v-s)} - 2* v^2 e^{-u})^2')

print('dE_u = 2(u^3 e^{(v-s)} - 2* v^2 e^{-u})(3u^2e^{(v-2)} + 2 v^2 e^{-u} ), ')

print( 'dE_v =  2(u^3 e^{(v-s)} - 2* v^2 e^{-u})(u^3 e^{(v-2)} - 4 v e^{-u}')

print ('So de gradient is: \n nabla E(u,v) =(2(u^3 e^{(v-s)} - 2* v^2 e^{-u})(3u^2e^{(v-2)} + 2 v^2 e^{-u} ), 2(u^3 e^{(v-s)} - 2* v^2 e^{-u})(u^3 e^{(v-2)} - 4 v e^{-u}) )')

print ('2b) Numbers of iterations : ', it)
print ('2c) Final coodinates: (', w[0], ', ', w[1],')')

############################### EXERCISE 1 PART 3 #########

def f(x,y):
    '''
    Function to minimize
    '''
    return np.float64(
        (x+2)**2 + 2*(y-2)**2 + 2* np.sin( 2* np.pi * x)* np.sin( 2* np.pi * y)
    )

def dfx(x,y):
    '''
    Partial derivate of f with respect to the variable x
    '''
    return np.float64(
        2*( x+2 ) + 4* np.pi* np.cos(  2* np.pi * x )* np.sin(2* np.pi * y)
    )

def dfy(x,y):
    '''
    Partial derivate of f with respect to the variable y
    '''
    return np.float64(
        4*( y-2 ) + 4* np.pi* np.cos(  2* np.pi * y )* np.sin(2* np.pi * x)

    )

def gradF(x,y):
    '''
        gradient of E
    '''
    return np.array([dfx(x,y), dfy(x,y)])

######################## gradien descendent with trace  ############

def gradient_descent_trace(initial_point, loss_function, gradient_function,  eta, max_iter):
    '''
    initicial point: w_0
    loss_function: error function
    gradient_function
    eta:  step size

    ### stop conditions ###
    max_iter

    #### return ####
    (w,iterations)
    w: the coordenates that minimize loss_function
    it: the numbers of iterations needed to obtain w

    '''

    iterations = 0
    error = loss_function( initial_point[0], initial_point[1])
    w = [initial_point]

    while iterations < max_iter:

        new_w = w[-1] - eta * gradient_function(w[-1][0], w[-1][1])

        iterations += 1
        error = loss_function(new_w[0], new_w[1])
        w.append( new_w )

    return w, iterations

######  conditions
smaller_eta = 0.01
bigger_eta = 0.1
max_iter = 50
initial_point = np.array([-1.0,1.0])

#### run
smaller_w, smaller_it = gradient_descent_trace( initial_point,f, gradF, smaller_eta, max_iter)
bigger_w, bigger_it = gradient_descent_trace( initial_point,f, gradF, bigger_eta, max_iter)

images_smaller_eta = [ f(x[0],x[1]) for x in smaller_w ]
images_bigger_eta = [ f(x[0],x[1]) for x in bigger_w]

print(f'With eta = {smaller_eta}, coordenates (x,y)= {(smaller_w[-1][0],smaller_w[-1][1])}, the number of iterations: {smaller_it} and the image is f(x,y) = {images_smaller_eta[-1]}')

print(f'With eta = {bigger_eta}, coordenates (x,y)={(bigger_w[-1][0],bigger_w[-1][1])}, the number of iterations: {bigger_it} and the image is f(x,y) = {images_bigger_eta[-1]}')

########### PLOTTING
x_label = 'Number of iterations'
y_label = 'f(x,y)'
bigger_eta_label = '$\eta$ = '+str(bigger_eta)
smaller_eta_label = '$\eta$ = '+str(smaller_eta)

## bigger eta ####
plt.clf()
plt.plot(images_bigger_eta, label=bigger_eta_label)
plt.xlabel(x_label)
plt.ylabel(y_label)
plt.title(f'Gradient descent of f with $\eta =$ {bigger_eta}')

plt.legend()
plt.show()
STOP_EXECUTION_TO_SEE_RESULT()
## smaller eta ###

plt.clf()
plt.plot(images_smaller_eta, label=smaller_eta_label)
plt.xlabel(x_label)
plt.ylabel(y_label)
plt.title(f'Gradient descent of f with $\eta =$ {smaller_eta}')

plt.legend()
plt.show()

STOP_EXECUTION_TO_SEE_RESULT()
## comparation ###
plt.clf()
plt.plot(images_bigger_eta, label=r"$\eta$ = 0.1")
plt.plot( images_smaller_eta, label=r"$\eta$ = 0.01")

plt.xlabel('Iterations')
plt.ylabel('f(x,y)')

plt.title("Comparation of the gradient descendent for $f$ changing eta value  ")

plt.legend()
plt.show()
STOP_EXECUTION_TO_SEE_RESULT()

### exta experimental ####

epsilon_eta = 1e-14
max_iter = 50
initial_point = np.array([-1.0,1.0])

epsilon_eta_label = '$\eta$ = '+str(epsilon_eta)
#### run
epsilon_w, epsilon_it = gradient_descent_trace( initial_point,f, gradF, epsilon_eta, max_iter)
images_epsilon_eta = [ f(x[0],x[1]) for x in epsilon_w ]

print(f'With eta = {epsilon_eta}, coordenates (x,y)={(epsilon_w[-1][0],epsilon_w[-1][1])}, the number of iterations: {epsilon_it} and the image is f(x,y) = {images_epsilon_eta[-1]}')
STOP_EXECUTION_TO_SEE_RESULT()

plt.clf()
plt.plot(images_epsilon_eta, label=epsilon_eta_label)
plt.xlabel(x_label)
plt.ylabel(y_label)
plt.title(f'Gradient descent of f with $\eta =$ {epsilon_eta}')

plt.legend()
plt.show()

STOP_EXECUTION_TO_SEE_RESULT()
## smaller eta and tiny one ###

plt.clf()

plt.plot(images_epsilon_eta, label=epsilon_eta_label)
plt.plot( images_smaller_eta, label=r"$\eta$ = 0.01")

plt.xlabel('Iterations')
plt.ylabel('f(x,y)')

plt.title("Comparation of the gradient descendent for $f$ changing eta value  ")

plt.legend()
plt.show()
STOP_EXECUTION_TO_SEE_RESULT()
######################################################################

#### exercise 3.b

initial_points = map(np.array, [[-.5, -.5],[1, 1], [2.1, -2.1], [-3,3],  [-2, 2]])
minimum_value = np.Infinity

print('{:^17}  {:^17}  {:^9}'.format('Initial', 'Final', 'Value'))

for initial in initial_points:

  w, _ = gradient_descent_trace(initial, f, gradF, 0.01, 50)
  local_minimum = w[-1]
  local_value = f(local_minimum[0], local_minimum[1])

  print('{}  {}  {: 1.5f}'.format(initial, local_minimum, local_value))

STOP_EXECUTION_TO_SEE_RESULT()

"""
Exercise 2
Author: Blanca Cano Camarero
"""

def STOP_EXECUTION_TO_SEE_RESULT():
        input('\n--- End of a section, press any enter to continue ---\n')

print('_______LINEAR REGRESSION EXERCISE _______\n')
print('Exercise 1')
input('\n Enter to start\n')

label5 = 1
label1 = -1

def readData(file_x, file_y):
        '''
        function for read data
        '''
	# reads files
        datax = np.load(file_x)
        datay = np.load(file_y)
        y = []
        x = []
        # Solo guardamos los datos cuya clase sea la 1 o la 5
        for i in range(0,datay.size):
                if datay[i] == 5 or datay[i] == 1:
                        if datay[i] == 5:
                                y.append(label5)
                        else:
                                y.append(label1)
                        x.append(np.array([1, datax[i][0], datax[i][1]]))

        x = np.array(x, np.float64)
        y = np.array(y, np.float64)

        return x, y

def Error(x,y,w):
    '''quadratic error
    INPUT
    x: input data matrix
    y: target vector
    w:  vector to

    OUTPUT
    quadratic error >= 0
    '''
    error_times_n = np.float64(np.linalg.norm(x.dot(w) - y.reshape(-1,1))**2)

    return np.float64(error_times_n/len(x))

def dError(x,y,w):
    ''' gradient
    OUTPUT
    column vector
    '''
    return  2/len(x)*(x.T.dot(x.dot(w) - y.reshape(-1,1)))

def sgd(x,y, eta = 0.01, max_iter = 1000, batch_size = 32, error=10**(-10)):
        '''
        Stochastic gradient descent
        INPUT
        x: data set
        y: target vector
        eta: learning rate
        max_iter

        OUTPUT
        w: weight vector
        '''

        #initialize data
        w = np.zeros((x.shape[1], 1), np.float64)
        n_iterations = 0

        len_x = len(x)
        x_index = np.arange( len_x )
        batch_start = 0
        w_error = Error(x,y,w)

        while n_iterations < max_iter and w_error > error :

                #shuffle and split the same into a sequence of mini-batches
                np.random.shuffle(x_index)
                for batch_start in range(0,  len_x, batch_size):
                        iter_index = x_index[ batch_start : batch_start + batch_size]

                        w = w - eta* dError(x[iter_index, :], y[iter_index], w)

                n_iterations += 1
                w_error = Error(x,y,w)

        return w

def sgd_exact_number_iter(x,y, eta = 0.01, max_iter = 1000, batch_size = 32, error = 10**(-10)):
        '''
        Stochastic gradient descent
        INPUT
        x: data set
        y: target vector
        eta: learning rate
        max_iter
        OUTPUT
        w: weight vector
        '''
        #initialize data
        w = np.zeros((x.shape[1], 1), np.float64)

        n_iterations = 0
        batch_start = 0
        len_x = len(x)

        x_index = np.arange( len_x )
        w_error = Error(x,y,w)

        while n_iterations < max_iter and w_error > error:
                #shuffle and split the same into a sequence of mini-batches
                if batch_start == 0:
                        x_index = np.random.permutation(x_index)
                iter_index = x_index[ batch_start : batch_start + batch_size]

                w = w - eta* dError(x[iter_index, :], y[iter_index], w)

                n_iterations += 1

                batch_start += batch_size
                if batch_start >= len_x: # if end, restart
                        batch_start = 0

                w_error = Error(x,y,w)

        return w

def pseudoInverseMatrix ( X ):
    '''
    INPUT
    X: is a matrix (must be a np.array) to use transpose and dot method
    OUTPUT
    hat matrix
    '''

    '''
    #S =( X^TX ) ^{-1}
    simetric_inverse = np.linalg.inv( X.T.dot(X) )

    # S X^T = ( X^TX ) ^{-1} X^T
    return simetric_inverse.dot(X.T)
    '''
    return np.linalg.pinv(X)

# Pseudoinverse
def pseudoInverse(X, Y):
    '''
    INPUT
    X is the feature matrix
    Y is the target vector (y_1, ..., y_m)

    OUTPUT:
    w: weight vector
    '''
    X_pseudo_inverse = pseudoInverseMatrix ( X )
    Y_transposed = Y.reshape(-1, 1)

    w = X_pseudo_inverse.dot( Y_transposed)

    return w

# Evaluating the autput

def performanceMeasurement(x,y,w):
    '''Evaluating the output binary case

    INPUT
    X is the feature matrix
    Y is the target vector (y_1, ..., y_m)

    OUTPUT:
    w: weight vector
    OUTPUT:
    bad_negative, bad_positives, input_size
    '''

    # defference between the sign of the regression and the target vector
    sign_column = np.sign(x.dot(w)) - y.reshape(-1,1)

    bad_positives = 0
    bad_negatives = 0

    for sign in sign_column[:,0]:
        if sign > 0 :
                bad_positives += 1
        elif sign < 0 :
                bad_negatives += 1

    input_size = len(y)

    return bad_negatives, bad_positives, input_size

def evaluationMetrics (x,y,w, label = None):
    '''PRINT THE PERFORMANCE MEASUREMENT
    '''
    bad_negatives, bad_positives, input_size = performanceMeasurement(x,y,w)

    accuracy = ( input_size-(bad_negatives +bad_positives))*100 / input_size

    if label :
        print(label)
    print (f'For w^T = {w.reshape(1,-1)}')
    print ( 'Input size: ', input_size )
    print( 'Bad negatives :', bad_negatives)
    print( 'Bad positives :', bad_positives)
    print( 'Accuracy rate :', accuracy, '%')

## Draw de result

### scatter plot
def plotResults (x,y,w, title = None):
        label_5 = 1
        label_1 = -1

        labels = (label_5, label_1)
        colors = {label_5: 'b', label_1: 'r'}
        values = {label_5: 'Number 5', label_1: 'Number 1'}

        plt.clf()

        # data set plot
        for number_label in labels:
                index = np.where(y == number_label)
                plt.scatter(x[index, 1], x[index, 2], c=colors[number_label], label=values[number_label])

        # regression line
        # x = 0
        symmetry_for_cero_intensity = -w[0]/w[2]

        #  x = 1, 0 = w0 + w1 * w2 * x2
        # then y = (-w0 - w1) /w2
        symmetry_for_one_intensity= (-w[0] - w[1])/w[2]

        #plotting order
        plt.plot([0, 1], [symmetry_for_cero_intensity, symmetry_for_one_intensity], 'k-', label=(title+ ' regression'))

        if title :
                plt.title(title)
        plt.xlabel('Average intensity')
        plt.ylabel('Simmetry')
        plt.legend()
        plt.show()

def plotResultMultiplesLines(x,y,multiple_w, main_title = None, multiples_title = None):
        '''
        INPUT
        x featue matrx
        y labels vector
        multiple_w vector of different weight vector
        multiple_titles
        '''
        label_5 = 1
        label_1 = -1

        labels = (label_5, label_1)
        colors = {label_5: 'b', label_1: 'r'}
        values = {label_5: 'Number 5', label_1: 'Number 1'}

        plt.clf()

        # data set plot
        for number_label in labels:
                index = np.where(y == number_label)
                plt.scatter(x[index, 1], x[index, 2], c=colors[number_label], label=values[number_label])

        for i in range(len(multiple_w)):
                w = multiple_w[i]
                title = multiple_title[i]
                # regression line
                # x = 0
                symmetry_for_cero_intensity = -w[0]/w[2]

                #  x = 1, 0 = w0 + w1 * w2 * x2
                # then y = (-w0 - w1) /w2
                symmetry_for_one_intensity= (-w[0] - w[1])/w[2]

                #plotting order
                plt.plot([0, 1],
                         [symmetry_for_cero_intensity, symmetry_for_one_intensity],
                         #'k-',
                         label=(title+ ' regression'))

        if main_title :
                plt.title(main_title)
        plt.xlabel('Average intensity')
        plt.ylabel('Simmetry')
        plt.legend()
        plt.show()

### _____________ DATA ____________________

# Reading training data set
x, y = readData('datos/X_train.npy', 'datos/y_train.npy')
# Reading test data set
x_test, y_test = readData('datos/X_test.npy', 'datos/y_test.npy')

w_pseudoinverse = pseudoInverse(x, y)
print("\n___ Goodness of the Pseudo-inverse fit ___\n")
print("  Ein:  ", Error(x, y, w_pseudoinverse))
print("  Eout: ", Error(x_test, y_test, w_pseudoinverse))

evaluationMetrics (x,y,w_pseudoinverse, '\nEvaluating output training data set')
evaluationMetrics (x_test, y_test, w_pseudoinverse, '\nEvaluating output test data set')
plotResults(x,y,w_pseudoinverse, title = 'Pseudo-inverse')

print(f'\nThe weight vector is {w_pseudoinverse}')

STOP_EXECUTION_TO_SEE_RESULT()

print("\n___ Goodness of the Stochastic Gradient Descendt (SGD) fit ___\n")

batch_sizes =[1,32,200,len(y)] #batch sizes compared in the experiment

n_iterations = [50,300]
for iteration in n_iterations:
        multiple_w = [w_pseudoinverse]
        multiple_title = ['pseudo-inverse']

        for _batch_size in batch_sizes:
                w = sgd(x,y, eta = 0.01, max_iter = iteration, batch_size = _batch_size)

                _title = f'SGD, batch size {_batch_size}'
                print( '\n\t'+_title)
                print ("Ein: ", Error(x,y,w))
                print ("Eout: ", Error(x_test, y_test, w))
                evaluationMetrics (x,y,w, '\nEvaluating output training data set')
                evaluationMetrics (x_test, y_test, w, '\nEvaluating output test data set')
                #plotResults(x,y,w, title = _title)
                multiple_w.append(w)
                multiple_title.append(_title)

                STOP_EXECUTION_TO_SEE_RESULT()

        plotResultMultiplesLines(x,y,multiple_w,f'Comparative batch sizes, {iteration} iterations',multiple_title)

STOP_EXECUTION_TO_SEE_RESULT()

print ('Exercise 2\n')

#### EXPERIMIENTS ###########

## a)
print('\nEXPERIMENT (a) \n')
def simula_unif(N, d, size):
        ''' generate a trining sample of N  points
in the square [-size,size]x[-size,size]
'''
        return np.random.uniform(-size,size,(N,d))

### data

size_training_example = 1000
dimension = 2
square_half_size = 1

training_sample = simula_unif( size_training_example,
                               dimension,
                               square_half_size)

STOP_EXECUTION_TO_SEE_RESULT()

plt.clf()
plt.scatter(training_sample[:, 0], training_sample[:, 1], c='b')
plt.title('Muestra de entrenamiento generada por una distribución uniforme')
plt.title('Training sample generated by a uniform distribution')
plt.xlabel('$x_1$ value')
plt.ylabel('$x_2$ value')

plt.show()

STOP_EXECUTION_TO_SEE_RESULT()

## b)

print('\nEXPERIMENT (b) \n')
def f(x1, x2):
	return np.sign(
            (x1 -0.2)**2
            +
            x2**2 -0.6
        )

def noisyVector(y, percent_noisy_data):
        '''
        y target vector to introduce noise
        size_training_example: number of point generated in each experiment,
        percent_noisy_data
        '''
        len_y = len(y)

        index = list(range(len_y))
        np.random.shuffle(index)

        size_noisy_data = int((len_y*percent_noisy_data)/ 100 )

        noisy_y = np.copy(y)
        for i in index[:size_noisy_data]:
                noisy_y[i] *= -1
        return noisy_y

#labels
y = np.array( [f(x[0],x[1]) for x in training_sample ])

percent_noisy_data = 10.0

y = noisyVector(y,  percent_noisy_data)

## draw
labels = (1, -1)
colors = {1: 'blue', -1: 'red'}

plt.clf()

for l in labels:

	index = np.where(y == l)
	plt.scatter(training_sample[index, 0],
                    training_sample[index,1],
                    c=colors[l],
                    label=str(l))

plt.title('Labelled training sample before noise')
plt.xlabel('$x_1$ value')
plt.ylabel('$x_2$ value')
plt.legend()
plt.show()

STOP_EXECUTION_TO_SEE_RESULT()

plt.clf()

for l in labels:

	index = np.where(y == l)
	plt.scatter(training_sample[index, 0],
                    training_sample[index,1],
                    c=colors[l],
                    label=str(l))

plt.title('Labelled training sample after noise')
plt.xlabel('$x_1$ value')
plt.ylabel('$x_2$ value')
plt.legend()
plt.show()

STOP_EXECUTION_TO_SEE_RESULT()

#### C
print('\nEXPERIMENT (c) \n')
eta = 0.01
batch_size = 5
maximum_number_iterations = 1000

x = np.array( [
        np.array([ 1, x_n[0], x_n[1] ])
        for x_n in training_sample
])

y = np.array( [f(x_n[0],x_n[1]) for x_n in training_sample ])
y = noisyVector(y,  percent_noisy_data)
w = sgd_exact_number_iter(x, y, eta, maximum_number_iterations, batch_size )

_title = f'SGD, batch size {batch_size}'
print( '\n\t'+_title)
print ("Ein: ", Error(x,y,w))
evaluationMetrics (x,y,w, '\nEvaluating output training data set')

STOP_EXECUTION_TO_SEE_RESULT()

plt.clf()

for l in labels:

	index = np.where(y == l)
	plt.scatter(training_sample[index, 0],
                    training_sample[index,1],
                    c=colors[l],
                    label=str(l))

plt.title('Linear regression fit')
plt.xlabel('$x_1$ value')
plt.ylabel('$x_2$ value')

# regression line
# x_0 = -1
y_0 = (w[1] - w[0]) /w[2]
#x_1 = 1
y_1 = -( w[1]+w[0]) /w[2]

plt.plot([-1, 1], [y_0, y_1], 'k-', label=('SGD regression'))
plt.xlim([-1,1])
plt.ylim([-1,1])
plt.show()

STOP_EXECUTION_TO_SEE_RESULT()

## d
print('\n EXPERIMENT (d), lineal regression\n')

def experiment(featureVector,
               number_of_repetitions = 1000,
               size_training_example = 1000,
               percent_noisy_data = 10.0
               ):
        '''
        INPUT
        featureVector: function that return  np.array
        number_of_repetitions: experiment repetitions ,
        size_training_example: number of point generated in each experiment,
        percent_noisy_data:

        OUTPUT
        (error_in, error_out)
        '''
        total_in_error = 0
        total_out_error = 0

        for _ in range( number_of_repetitions):
        ## data generation
                training_sample = simula_unif( size_training_example,
                                               dimension,
                                               square_half_size)

                test_sample = simula_unif( size_training_example,
                                           dimension,
                                           square_half_size)
                test_y = np.array( [f(x[0],x[1]) for x in test_sample ])
                test_y = noisyVector(test_y, percent_noisy_data)

                y = np.array( [f(x[0],x[1]) for x in training_sample ])
                y = noisyVector(y,  percent_noisy_data)

                # fit
                x = np.array( [
                        featureVector(x_n)
                        for x_n in training_sample
                ])

                x_test = np.array( [
                        featureVector(x_n)
                        for x_n in test_sample
                ])

                w = sgd_exact_number_iter(x, y, eta, maximum_number_iterations, batch_size = 32)

                total_in_error += Error(x,y,w)
                total_out_error += Error(x_test, test_y, w)

        error_in = float(total_in_error / number_of_repetitions)
        error_out = float(total_out_error / number_of_repetitions)

        return error_in, error_out

def linearFeatureVector(x_n):
        return np.array( [
                1,
                x_n[0],
                x_n[1]
               ] )

_number_of_repetitions = 1000
error_in, error_out = experiment( linearFeatureVector,
                                  number_of_repetitions = _number_of_repetitions,
                                  size_training_example = 1000,
                                  percent_noisy_data = 10.0
               )
print(f'The mean value of E_in in all {_number_of_repetitions} experiments is: {error_in}')
print(f'The mean value of E_out in all {_number_of_repetitions} experiments is: {error_out}')

STOP_EXECUTION_TO_SEE_RESULT()

# e)

print('\nEXPERIMENT (e)\n' )
eta = 0.01
batch_size = 5
maximum_number_iterations = 1000

def quadraticFeatureVector(x_n):
        '''
        INPUT
         xn = (x1,x2) vector of coordinates

        '''
        return np.array([ 1,
                   x_n[0],
                   x_n[1],
                   x_n[0]*x_n[1],
                   x_n[0]* x_n[0],
                   x_n[1]* x_n[1]  ])

x = np.array( [
        quadraticFeatureVector(x_n)
        for x_n in training_sample
])
y = np.array( [f(x[0],x[1]) for x in training_sample ])
y = noisyVector(y,  percent_noisy_data)

for i in [10, 50, 100, 200, 500, 700, 1000]:
        maximum_number_iterations = i
        w = sgd_exact_number_iter(x, y, eta, maximum_number_iterations , batch_size = 17)

        print('\nFor one experiment:')
        _title = f'SGD, batch size {batch_size}, number iterations {maximum_number_iterations}'
        print( '\n',_title)
        print ("Ein: ", Error(x,y,w))
        evaluationMetrics (x,y,w, '\nEvaluating output training data set')
print('')

STOP_EXECUTION_TO_SEE_RESULT()

## plotting

def equation (x,y,w):
        '''
        INPUT
        x coordinate
        y coodinate
        w weights vector

        OUTPUT
        Real number, the scalar product of features vector dot weights vector
        '''
        return ( w[0,0]
                 + w[1,0] * x
                 + w[2,0] * y
                 + w[3,0] * x * y
                 + w[4,0] * x**2
                 + w[5,0] * y**2
                )
'''
PLOTING LINEAR REGRESSION

We are going to plot the (x,y) \in [-1,-1]^2 that their value after
the linear regression for classification  is near to 0.

That means that they are in the limit area.
'''
error = 10**(-2.1)
space = np.linspace(-1,1,100)

z = [[ equation(i,j,w) for i in space] for j in space ]

plt.contour(space,space, z, 0, colors=['black'],linewidths=2 )

for l in labels:

	index = np.where(y == l)
	plt.scatter(training_sample[index, 0],
                    training_sample[index,1],
                    c=colors[l],
                    label=str(l))

plt.title('Quadratic regression fit')
plt.xlabel('$x_1$ value')
plt.ylabel('$x_2$ value')
plt.legend( loc = 'lower left')
plt.show()

STOP_EXECUTION_TO_SEE_RESULT()

## EXPERIMENT
error_in, error_out = experiment( quadraticFeatureVector,
                                  number_of_repetitions = _number_of_repetitions,
                                  size_training_example = 1000,
                                  percent_noisy_data = 10.0
                                 )
print(f'The mean value of E_in in all {_number_of_repetitions} experiments is: {error_in}')
print(f'The mean value of E_out in all {_number_of_repetitions} experiments is: {error_out}')
print('==========================================')
STOP_EXECUTION_TO_SEE_RESULT()

"""
Exercise 3
Author: Blanca Cano Camarero
"""

def f(x,y):
    '''
    Function to minimize
    '''
    return np.float64(
        (x+2)**2 + 2*(y-2)**2 + 2* np.sin( 2* np.pi * x)* np.sin( 2* np.pi * y)
    )

def dfx(x,y):
    '''
    Partial derivate of f with respect to the variable x
    '''
    return np.float64(
        2*( x+2 ) + 4* np.pi* np.cos(  2* np.pi * x )* np.sin(2* np.pi * y)
    )

def dfy(x,y):
    '''
    Partial derivate of f with respect to the variable y
    '''
    return np.float64(
        4*( y-2 ) + 4* np.pi* np.cos(  2* np.pi * y )* np.sin(2* np.pi * x)

    )

def gradf(x,y):
    '''
        gradient of E
    '''
    return np.array([dfx(x,y), dfy(x,y)])

def ddfxx(x,y):
    return 2 - 8*np.pi**2*np.sin(2*np.pi*x)*np.sin(2*np.pi*y)

def ddfyy(x,y):

    return 4 - 8*np.pi**2*np.sin(2*np.pi*x)*np.sin(2*np.pi*y)

def ddfxy(x,y):
    return 8*np.pi**2*np.cos(2*np.pi*x)*np.cos(2*np.pi*y)

def hessianf(x, y):
    return np.array([
        ddfxx(x,y),
        ddfxy(x,y),
        ddfxy(x,y),
        ddfyy(x,y),
    ]).reshape((2, 2))

def newton_trace(initial_point, fun, grad_fun, hessian, eta, max_iter):
    """ Newton method
    INPUT
    - initial_point:
    - f: differential function
    - grad_fun: Gradient
    - hessian: hessian
    - eta: learning rate
    - max_iter: number of iterations

    OUTPUT
    w trace
    """

    w = initial_point
    w_list = [initial_point]
    iterations = 0

    while iterations < max_iter:
        w = w - eta *np.linalg.inv(hessian(w[0],w[1])).dot(grad_fun(w[0], w[1]))
        w_list.append(w)
        iterations += 1

    return np.array(w_list)

# Dicrease with the x-axis

## Exercise 1

######  conditions
smaller_eta = 0.01
bigger_eta = 0.1
max_iter = 50
initial_point = np.array([-1.0,1.0])

#### run
smaller_w = newton_trace( initial_point,f, gradf, hessianf, smaller_eta, max_iter)
bigger_w = newton_trace( initial_point,f, gradf, hessianf, bigger_eta, max_iter)

images_smaller_eta = [ f(x[0],x[1]) for x in smaller_w ]
images_bigger_eta = [ f(x[0],x[1]) for x in bigger_w]

print(f'With eta = {smaller_eta}, coordenates (x,y)= {(smaller_w[-1][0],smaller_w[-1][1])}, the number of iterations: {max_iter} and the image is f(x,y) = {images_smaller_eta[-1]}')

print(f'With eta = {bigger_eta}, coordenates (x,y)={(bigger_w[-1][0],bigger_w[-1][1])}, the number of iterations: {max_iter} and the image is f(x,y) = {images_bigger_eta[-1]}')

########### PLOTTING
x_label = 'Number of iterations'
y_label = 'f(x,y)'
bigger_eta_label = '$\eta$ = '+str(bigger_eta)
smaller_eta_label = '$\eta$ = '+str(smaller_eta)

## bigger eta ####
plt.clf()
plt.plot(images_bigger_eta, label=bigger_eta_label)
plt.xlabel(x_label)
plt.ylabel(y_label)
plt.title(f"Newton's adjust of f with $\eta =$ {bigger_eta}")

plt.legend()
plt.show()

## smaller eta ###

plt.clf()
plt.plot(images_smaller_eta, label=smaller_eta_label)
plt.xlabel(x_label)
plt.ylabel(y_label)
plt.title(f"Newton's adjust of f with $\eta =$ {smaller_eta}")

plt.legend()
plt.show()

## comparation ###
plt.clf()
plt.plot(images_bigger_eta, label=r"$\eta$ = 0.1")
plt.plot( images_smaller_eta, label=r"$\eta$ = 0.01")

plt.xlabel('Iterations')
plt.ylabel('f(x,y)')

plt.title("Comparation of the newton's method for $f$ changing eta value  ")

plt.legend()
plt.show()

STOP_EXECUTION_TO_SEE_RESULT()

#### exercise 3.b
eta = 0.1
max_iter = 50
initial_points = map(np.array, [[-.5, -.5],[1, 1], [2.1, -2.1], [-3,3],  [-2, 2]])
minimum_value = np.Infinity

plt.clf()
print('{:^17}  {:^17}  {:^9}'.format('Initial', 'Final', 'Value'))

for initial in initial_points:

  w = newton_trace( initial,f, gradf, hessianf, eta, max_iter)
  local_minimum = w[-1]
  local_value = f(local_minimum[0], local_minimum[1])
  #plot
  images = [ f(x[0],x[1]) for x in w ]
  plt.plot(images, label=f'Initial point {initial}')

  print('{}  {}  {: 1.5f}'.format(initial, local_minimum, local_value))

plt.xlabel('Iterations')
plt.ylabel('f(x,y)')

plt.title("Comparation of the newton's method for $f$ changing initial point ")

plt.legend()
plt.show()

## without the biggest

initial_points = map(np.array, [[-.5, -.5],[1, 1], [-3,3],  [-2, 2]])
minimum_value = np.Infinity

plt.clf()
print('{:^17}  {:^17}  {:^9}'.format('Initial', 'Final', 'Value'))

for initial in initial_points:

  w = newton_trace( initial,f, gradf, hessianf, eta, max_iter)
  local_minimum = w[-1]
  local_value = f(local_minimum[0], local_minimum[1])
  #plot
  images = [ f(x[0],x[1]) for x in w ]
  plt.plot(images, label=f'Initial point {initial}')

  print('{}  {}  {: 1.5f}'.format(initial, local_minimum, local_value))

plt.xlabel('Iterations')
plt.ylabel('f(x,y)')

plt.title("Comparation of the newton's method for $f$ changing initial point  ")

plt.legend()
plt.show()

STOP_EXECUTION_TO_SEE_RESULT()

# let see gradient values
points =  [[-0.38067878, -0.52778221],
           [1.06677195, 0.91078249],
           [ 3.26077803, -3.11750721] ,
           [-2.,  2.]]

for x,y in points :

    print(f'For {(x,y)}')
    print(f'\tThe gradient value is {gradf(x,y)}')
    print(f'\tThe inverse hessian is {np.linalg.inv(hessianf(x,y))}\n')

STOP_EXECUTION_TO_SEE_RESULT()

\end{minted}


\section{Linear Algebra}

\begin{definition}[Basis]
    Given a set of vectors in $\mathbf{R}^{n}, V$, which is linearly
    independent, the set $V$ is a \textit{basis} if you can span
    $\mathbf{R}^{n}$ using $V$.
\end{definition}

\begin{definition}[Backward substituion]\label{backwardsubstituion}
    Opposite of~\nameref{forwardsubstituion}.
\end{definition}

\begin{definition}[Condition number]\label{conditionnumber}
    A measure of how the output value for a function changes respective to
    input variables.
\end{definition}

\begin{definition}[Column space]
    All linear combinations of the columns of a matrix $A$.
\end{definition}

\begin{definition}[Consistent]
    $\iff$ the rightmost column of an augmented matrix is not a pivot column.
    I.e.\ in a 3x4 matrix, if the last row is zero and 2nd column has pivot
    column, then the system is still consistent.
\end{definition}

\begin{definition}[Diagonal Dominance]
    $\forall{i}, \exists{i} A_{i, i}, \forall i, \iff A_{i,i}
    \geq \sum\limits_{j = 0, j\neq i}^{m}$,
    then matrix $A$ is DD.
\end{definition}

\begin{definition}[Diagonlizable matrix]
    Given a matrix $A$, we can express it as $A = PDP^{-1}$.
    Then, we can compute:
    \begin{align}
        A^{2} = (PDP^{-1})(PDP^{-1}) = PD(P^{-1}P)DP^{-1} = PDDP^{-1} \\
        \text{Because} P \cdot P{-1} = I \\
        \dots A^{k} = PD^{k}P^{-1}
    \end{align}

    So, we use diagonal matrices to easily raise matrices to power $k$.

    To diagonalize a matrix $A^{n\times{n}}$, it is required that it has $n$ linearly
    independent eigenvectors.
\end{definition}

\begin{definition}[Divergence]
    In vector calculus, divergence measures the magnitude to the
    gradient of~\nameref{vectorfield}s at a given point.

\end{definition}

\begin{definition}[Dotting matrices]
    \includegraphics[scale=0.3]{mm.png}
\end{definition}

\begin{definition}[Echelon form]
    A matrix where:
    \begin{enumerate}
        \item All nonzero rows are above zero-rows
        \item Each pivot column is placed from left to right
    \end{enumerate}
\end{definition}

\begin{definition}[Eigenvector]\label{eigen}
    Given a square matrix A, when A is multiplied with an eigenvector $v$,
    the resulting matrix A${^\prime}$ is a multiple of $v$.
    The multipe is denoted by $\lambda$ and is called an eigenvalue.
    So, $Av = \lambda v$

\end{definition}

\begin{definition}[Forward substituion]\label{forwardsubstituion}
    In an iterative method, if we first solve an element $A_{i,j}$,
    then when we caluclate some $A_{i+a, j+b}$ in the same iteration, then we
    use the updated value of $A_{i,j}$. Opposite
    of~\nameref{backwardsubstituion}

\end{definition}

\begin{definition}[Gaussian Elimination]
    AKA ``Row reduction''. Add row $x$ to row $y, y \neq x$, to reduce $y$ to
    zeroes.

\end{definition}

\begin{definition}[Ill-conditioned matrix]
    A matrix is ill-conditioned if the~\nameref{conditionnumber} is large.
    If you make a small change in an ill conditioned matrix, there are usually
    large differences in results from calculations with this maxtrix.

\end{definition}

\begin{definition}[Inner product]\label{innerprod}
    The product $u^{T}v$. Given that $u, v \in R^{i\times{j}}$, then we need to transpose
    to perform normal dot products.

    Inner products are commutative: $u \dot v \equiv v \dot u$.
\end{definition}

\begin{definition}[Inverse]
    For a matrix $A \in \mathbf{R}^{2x2}$:
    $$
    A^{-1} = \frac{1}{|A|}
    \begin{bmatrix}
    d & -b \\
    -c & a
    \end{bmatrix}
    = \frac{1}{ad - bc}
    \begin{bmatrix}
    d & -b \\
    -c & a
    \end{bmatrix}
    $$
    For a matrix $A \in \mathbf{R}^{3x3}$: bcacab

    Properties:
    \begin{itemize}
    \item $(A^{-1})^{-1} = A$
    \end{itemize}
\end{definition}

\begin{definition}[Kernel]
    For a vector space given by a~\nameref{lintrans}, a kernel is the set
    of vectors $v \in V$, s.t. $T(u) = 0$.
\end{definition}

\begin{definition}[Length of Vector]\label{vectorlength}
    For a vector
    \begin{align*}
        v = [a_{1}, a_{2}, \dots , a_{n}] \\
        |v| = \sqrt{a^{2}_{1} + a^{2}_{2} + \dots + a^{2}_{n}}{}
    \end{align*}
\end{definition}

\begin{definition}[Linear dependence]
    For $V = {v_{1}, \dots, v_{n}}$, and $\forall v, v is vector$,
    if none of the vectors in V can be written as a linear combination
    from the other vectors in V, the set is linearly independent.
\end{definition}

\begin{definition}[Linear transformation]\label{lintrans}
    Take a vector space into another, s.t.\
    \begin{align}
        T(u + v) = T(u) + T(v) &\forall u,v \in v \\
        T(cu) = cT(u) &\forall u \in V
    \end{align}
\end{definition}

\begin{definition}[Matrix of observations]
    Collect a sample, subject it to tests, and for each test, you give a value.
    For $n$ tests and $m$ samples, you will result in $\Re^{n\times{m}}$
    \textit{matrix of observations}.
\end{definition}

\begin{definition}[Norm]
    The ``length'' of a vector, just remember to sum up over all dimensions.
    Each number is raised to $n$, and the total sum is then raised to

    \begin{description}
        \item[L1]
        \item[L2] Can be defined as:
            \begin{align}
                \phi^{2} = \phi \times \phi =
                \dots = \int{\phi (x)^{2}  dx} \\
                |x| = \sqrt{ \sum\limits_{k = 1}^{n}{x_{k}|^{2}}}
            \end{align}
    \end{description}

    $\frac{1}{p}$.
\end{definition}

\begin{definition}[Normalization of vectors]
    $ \hat{X} \equiv \frac{X}{|X|} $, where $|x|$ is the~\nameref{vectorlength}
    $X$ is, in this case, evaluated as the additive sum of it's entries.
\end{definition}

\begin{definition}[Null space]
    Given a matrix $A$, if you solve for that each row = 0,
    all possible values for each $x$ makes out the Null space.

    Note that here it is apossible to get free variables for
    some x, and bound to others.

\end{definition}

\begin{definition}[Numerical stability]
    This concept describes how changes in the input should not affect the result.
    An example could be that sorting a list should not change the outcome from
    applying a function to the input.
\end{definition}

\begin{definition}[Orthogonal]\label{orthogonal}
    Two lines that intersect each other at 90 degrees.\\
    \begin{itemize}
        \item Orthogonal matrices preserve dot products:
        given two vectors $u$, and $v$, and an orthogonal matrix Q,
        the following is true:
        $u \times v = Qu \times Qv$
        \item The determinant of an orthogonal matrix is always 1 or -1
        \item The transpose of $Q$ is equal to it's inverse, hence:
            $Q \times Q^{T} = I$
    \end{itemize}
\end{definition}

\begin{definition}[Orthonormal]
    In~\nameref{innerprod} space are orthonormal if they are
    orthogonal and unit vectors.
\end{definition}

\begin{definition}[Perpendicular]
    Similar to~\nameref{orthogonal}, but with lines.
\end{definition}

\begin{definition}[Pivoting]
    Finding the first non-zero element in an algorithm on linear systems.
    To do so, you can do things like gaussian elimination, etc.
    \begin{description}
        \item[Complete pivoting] find the largest absolute value of a pivot,
            considering all elements.
        \item[Partial pivoting] finds the largest absolute value in the pivot
            column.
        \item[Scaled pivoting] finds the largest absolute value of a pivot
            column relative to it's entries in the same row.
    \end{description}
\end{definition}

\begin{definition}[Plane]
    A flat, two-dimensional surface
\end{definition}

\begin{definition}[Positive semidefninite matrix]
    Properties:
    \begin{itemize}
        \item Nonnegative~\nameref{eigen}s
        \item $X = V^{T}V$ for some $V \in \mathbf{R}^{mxn}$
        \item $X = \sum\limits_{i=1}^{m}\lambda_{i}w_{i}w^{t}_{i}$
            for some $\lambda_{i} \geq 0$ and vectors $w_{i} \in \mathbf{R}^{n}$
            such that $w^{T}_{i}w = 1$ and $w^{T}_{i}w_{j} = 0$

    \end{itemize}
\end{definition}

\begin{definition}[Preconditoner]
    A matrix $P$ such that $P^{-1}A$ has a smaller~\nameref{conditionnumber}
    than $A$.

\end{definition}

\begin{definition}[Projection]
    define a vector $v$ and $u$.
    \begin{align*}
        L &= \left\{cv | c \in \mathbf{R} \right\}  \\
        proj(v) &= l \in L \text{\ such that\ } u - proj(v)
        \text{\ is~\nameref{orthogonal} to l}  \\
        \textit{I.e., } proj(v) &= cv, c \in \mathbf{R}
    \end{align*}

    Properties:
    \begin{itemize}
        \item $proj(v) = proj(v)^{2}$
        \item Linear independence on $u, v$ also relates for $v - proj(v), u$
        \item adding $proj(v)$ to $v$ gives you $u$
    \end{itemize}

\end{definition}

\begin{definition}[Relaxation methods]
     relaxation methods are iterative methods for solving systems of equations,
     including nonlinear systems. Examples are Gauss-Seidel, Jacboi, etc.

\end{definition}

\begin{definition}[Singular matrix]
    Any square matrix without an inverse.
    A matrix is singular $\iff$ its determinant is 0.

\end{definition}

\begin{definition}[Span of vectors]\label{vectorspan}
    All linear combinations of a set of vectors.
    \begin{align*}
        V &= \left\{v_{1}, \dots, v_n\right\} \\
        C &= \left\{c \in C | \mathbf{R}\right\} \\
        span &= c_{1}v_{1} + \dots + c_{n}v_{n}
    \end{align*}
\end{definition}

\begin{definition}[Spectrum of a matrix]
    The multiset of its eigenvalues
\end{definition}

\begin{definition}[Spectral Radius]
    The largest absolute eigenvalue of a matrix.
\end{definition}

\begin{definition}[Successive Over Relaxation]
    Gauss-Seidel is the same as SOR (successive over-relaxation) with $\omega=1$
\end{definition}

\begin{definition}[Symmetric]
    \begin{itemize}
        \item Length of rows is equal
        \item The transpose is equal to the originl
    \end{itemize}
\end{definition}

\begin{definition}[Tensor]
    Geometric objects that describe linear relations between vectors or scalars.
\end{definition}

\begin{definition}[Toeplitz matrix]
    $$
    \begin{bmatrix}
    a & b & c & d \\
    b & a & b & c \\
    c & b & a & b \\
    d & c & b & a
    \end{bmatrix}
    $$

    Noteworthingly, this kind of matrix can be inverted in $O(n \times \log{n})$
\end{definition}

\begin{definition}[Trace]
    Sum of all diagonal entries in a matrix.
    $A_{11} + A_{22} + A_{nn}$
\end{definition}

\begin{definition}[Transpose]
    Take column $i$ and make it into a column. Repeat.
\end{definition}

\begin{definition}[Unit vector]
    A vector who's length is 1.
\end{definition}

\begin{definition}[vector length]
    number of "steps" in a vector
\end{definition}