\documentclass{article}

\usepackage{minted}
\usepackage{amsmath}
\usepackage{amsfonts}

\title{ECE-210-A Homework Feedback}
\author{Jonathan Lam}
\date{Spring 2022}

\begin{document}

\maketitle{}

\section{Introduction to MATLAB}
\label{sec:hw1}

\begin{description}
\item[Submission format] Please submit in *.m (plaintext) file format, rather than *.mlx. I should have made this a requirement at the beginning. While *.mlx (live script) files may be more visually appealling, they are more difficult to grade and may not be as cross-compatible with different versions of MATLAB.
\item[Notes on style (in addition to the style notes on the assignment sheet)] Name your files consistently (e.g., \mintinline{text}|ece210_hw1_lam.m|), and use a header at the beginning of each file. It may look something like:
\begin{minted}{matlab}
% Homework 1
% Jonathan Lam
% ECE210
% 02/16/2022
clear(); clc(); close('all');

%% Q1
% ...
\end{minted}
  At the least, have your name and the commands to clear the workspace at the top of every script file\footnote{A script file is one you would run directly. This is as opposed to a file that defines a function or class, in which you do not want to call \mintinline{matlab}|clear(); clc(); close('all');| at the top of.}.
  \item[\mintinline{matlab}|linspace()| vs. the colon operator] Be wary of the use cases and what the problem specifies! One is better suited for the case of a specified number of samples, and the other is better for cases of a specified frequency or period.
\end{description}

\clearpage{}
\section{Vectorization and for loops}
\label{sec:hw2}

\begin{description}
\item[Suppress intermediate and long outputs.] I don't need to see the outputs in the command window to evaluate your HW -- there are other ways. Printing large matrices (and other I/O in general) is also very slow (it will definitely mess with timing).
\item[Keep your code DRY] (DRY = ``Don't repeat yourself,'' and is a common maxim in software engineering.) If you see a constant (e.g., 100, 1000) being repeated throughout your code, it's better (more maintainable, and better documented) to define it once as a variable and use that variable throughout. That way, you can easily change the variable by changing one value, or you can rename the variable using MATLAB's variable refactoring utilities.
\item[Use sections to separate code execution.] Sections are a double-edged sword. For one thing, they are very convenient for running a particular block of code, and they should be used to partition your code into logical segments. Being able to focus on and run a small bit of code, and inspect the workspace (environment) in the middle of your script is extremely helpful. On the other hand, it can cause problems that are addressed in the next bullet point.
\item[Make sure your code runs in a clean environment.] First of all, make sure your code compiles and runs. Submitting code that has multiple errors throughout is a big red flag, because coding is a feedback-driven problem-solving process: you are expected to run your code, and check that the results match your intuition. If your code is not in a state where it can be run, it shows that you haven't gone through this process. Additionally (related to sections), make sure that if you clear your workspace and run all sections in your code from start to finish, that it runs correctly without errors. This is important because when you were debugging, you may have accidentally changed the behavior of your program.
\item[Get familiar with standard debugging methods.] Understanding error messages is key. That should help you handle syntactical errors. For semantic errors (when your code compiles but doesn't do what you think it does), start with a minimal, working example and build up from there step by step, until something begins to fail. Learn how to build test cases and check the error from an expected or true value. Use the documentation until you get tired of it. This falls under the category of standard debugging tips that are common in software engineering.
\end{description}

\clearpage{}
\section{Logical indexing and functions}
\begin{description}
\item[Logical indexing is different than multiplying by a logical matrix] The first question involves multiplying by a matrix of logical values, which acts like a layer mask: it preserves the domain, and sets all the values corresponding to FALSE to 0. In the second question, one needs to extract the x- and y-values that meet a certain condition. In this case, we want to condense the vectors to only those that match the condition; if we multiply by the logical matrix, we end up with a lot of spurious \mintinline{matlab}|[x, y] = [0, 0]| values. In other words, make sure you understand the difference between:
\begin{minted}{matlab}
A = [ 1 2 3 ; 4 5 6 ];
B = logical([ 1 0 0 ; 0 1 0 ]);
masked = A .* B;
filtered = A(B);
\end{minted}

\item[Padding vectors] Don't pad a vector with arbitrary values to make it match the length of another vector. In the case of extending the the result of \mintinline{matlab}|diff()| to make it match the length of the original vector, do not pad with zeros -- this may create an anamolous discontinuity at the padded end. A more reasonable method is to pad with the first value at the beginning, or pad with the last value at the end. Alternatively, truncating the first or last value of the longer vector may be equally appropriate and doesn't involve introducing new datapoints.

  There may be times when padding with zeros is acceptable, e.g., when using the FFT, which expects a zero-padded vector of length $2^n,\ n\in\mathbb{N}$.

\item[The \mintinline{text}|sinc| function] As many of you discovered, the \mintinline{matlab}|sinc| function requires the Signal Processing Toolkit. With your school account, you may install this toolkit for free. (If you have trouble with installing a toolkit, please let me know.)

  You may also implement the \mintinline{matlab}|sinc| function on your own. The formula for the (normalized) \mintinline{matlab}|sinc| is well-known: \[\text{sinc }x = \frac{\sin\pi x}{\pi x}\] The problem is the discontinuity (due to a division by 0 which leaves an undefined value at $x=0$), which will cause issues with your calculus functions. Since the limit is well-defined at $x=0$, we can be more explicit: \[\text{sinc }x=
    \begin{cases}
      1, & x=0 \\
      \frac{\sin\pi x}{\pi x}, & x\ne 0
    \end{cases}
  \] We may implement this in MATLAB like so:
\begin{minted}{matlab}
x = % some vector
y = sin(pi * x) ./ (pi * x) ;
y(x == 0) = 1;  % note: logical indexing and broadcasting!
\end{minted}
\end{description}

\section{Gram Schmidt orthonormalization}
\begin{description}
\item[An open-ended problem] This assignment was the first (and only one) in this course that is really open-ended. You learned about an algorithm called Gram-Schmidt in Linear Algebra or in Signals and Systems. Now you're told to automate it. Take a specification and implement it. Word problems like this are common in computer programming.

You might wonder as you go through the process: \textit{What is the most efficient way to do this? Should this part be vectorized? Can this part even be vectorized?} What you might not realize, however, is that programming is inherently an iterative process, and you won't know until you try it the first time. And then when you run your code and get results you don't expect, you debug it and refactor until you get a polished piece of code. I will also give you feedback when you submit your code, and you are allowed to resubmit code (for credit) because that's when you expect your code to improve. Don't expect to know what you're doing when you code; treat it as an interactive process.

\item[To vectorize, or not to vectorize] Gram Schmidt is an algorithm that cannot be totally vectorized. This is because it is a sequenced algorithm; the second row depends on the first row, and the third depends on the first two, and so on and so forth. The columns are not independent and thus the algorithm is not parallelizable. Thus we need a \mintinline{matlab}{for} loop somewhere in the algorithm. However, it turns out that you can vectorize both \mintinline{matlab}{ortho_proj} and \mintinline{matlab}{is_orthonormal}, because they both don't have this restriction.

\item[Matrix multiplication as inner product] A common definition for inner product of two (complex\footnote{Did you consider the case of complex vectors?}) vectors is the following:
\[\langle x,y\rangle=\sum_ix_i^*y_i\]
Now, assume we have in MATLAB two column vectors \mintinline{matlab}{x} and \mintinline{matlab}{y}. We can write this inner product\footnote{In math notation, this would be $x^Hy$, where $^H$ indicates the Hermitian transpose (conjugate transpose).} as the expression \mintinline{matlab}{dot(conj(x), y)}. However, there's a simpler way: \mintinline{matlab}{x' * y}. The place where this really shines is if you have two matrices $A$ and $B$ (with the same interpretation as given in the problem set, where each column is a vector in the set), and want to take all the dot products $\langle A_i,B_j\rangle$ where $A_i$ indicates vector/column $i$ from matrix $A$. In this case, you can do this very simply: \[C=A^HB\] The beauty is that $C_{ij}=A_i^HB_j=\langle A_i,B_j\rangle$.

\item[Vectorizing \mintinline{matlab}{is_orthonormal}] Consider a set of vectors $M$. If the set of vectors is orthonormal, what do you expect the output of $M^HM$ to be? (Hint: see previous bullet point.)

\item[Vectorizing \mintinline{matlab}{ortho_proj}] The projection of a vector on a basis is the sum of the projections of the vector on each of the basis vectors. You can write orthogonal projection using the formula given in your first linear algebra class: \[\text{proj}_{\{\vec{a_i}\}}\vec{b}=\sum_i\text{proj}_{\vec{a_i}}\vec{b}=\sum_i\frac{\langle\vec{a_i},\vec{b}\rangle}{||\vec{a_i}||^2}\vec{a_i}\] However, we can greatly simplify this for our \mintinline{matlab}{ortho_proj} case. First of all, if we assume that $\{\vec{a_i}\}$ is an orthonormal set of vectors, then $||\vec{a_i}||=||\vec{a_i}||^2=1$. The rest is simply three operations: an inner product, a regular (element-wise) product, and a sum. Can you vectorize the composition of these operations?

\item[Additional error checking for Gram Schmidt] Consider the case when there are more vectors in the set of vectors than the dimensionality of the space (the number of elements per vector). (This will be the case when $M$ is wider than it is tall.) Then, some vectors will necessarily linearly dependent. This in turn will cause some of the vectors from Gram Schmidt orthonormalization to be zero. Do you handle this case? (Most submissions don't check if the norm is zero before normalizing, and they don't attempt to remove zero vectors from the basis during Gram Schmidt). Note that the assignment doesn't ask you to check these, but neither does it give you the assumption that the input vectors are necessarily linearly independent. (Luckily, a set of random vectors where $M$ is at least as tall as it is wide will very likely be a linearly-independent set.)

\end{description}

\section{Object-oriented programming and cell arrays}
Not much to say about this assignment, it should be pretty straightforward.

\section{Z-transform, polynomials, and basic filtering}
\begin{description}
\item[An intuitive reading of the pole-zero plot] Zeros near or on the unit circle cause increased attenuation (decreased gain), and poles near the unit circle (but hopefully not on the unit circle) caused increased gain. The rightmost part of the unit circle represents a zero frequency, whereas the leftmost part represents higher frequencies. We can use some combination of these facts to intuit what kind of filter (lowpass, highpass, bandpass, bandstop) a filter is by inspecting its pole-zero plot.
\end{description}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


\documentclass[11 pt]{scrartcl}
\usepackage[header, margin, koma]{tyler}

\newcommand{\hwtitle}{Discussion 2B Recap}

\tikzstyle myBG=[line width=3pt,opacity=1]

\newcommand{\drawLinewithBG}[2]
{
  \draw[white,myBG]  (#1) -- (#2);
  \draw[black,very thick] (#1) -- (#2);
}
\newcommand{\drawPolarLinewithBG}[2]
{
  \draw[white,myBG]  (#1) -- (#2);
  \draw[black,very thick] (#1) -- (#2);
}

\pagestyle{fancy}
\fancyhf{}
\fancyhead[l]{\hwtitle{}}
\fancyhead[r]{Tyler Zhu}
\cfoot{\thepage}

\begin{document}
\title{\Large \hwtitle{}}
\author{\large Tyler Zhu}
\date{\large September 11, 2020}

\maketitle

\section{Terminology}

A \textbf{graph} $G$ consists of \textbf{vertices} or \textbf{nodes}, the points, and \textbf{edges}, the lines. We frequently write $V(G)$ and $E(G)$ for the vertex and edge sets of $G$ respectively, and call $|V(G)|$ and $|E(G)|$ the \textbf{order} and \textbf{size} of the graph respectively.
\begin{figure}[!htb]
    \centering
        \begin{tikzpicture}[node distance = 2cm,
            thick,main node/.style={circle,draw}]
            \node[main node] (1) {$u$};
            \node[main node] (2) [above right of=1, yshift=-0.5cm] {$v$};
            \node[main node] (3) [below right of=1, yshift=0.5cm] {$w$};
            \node[main node] (4) [below right of=2, yshift=0.5cm] {$x$};
            \node[main node] (5) [right of=4] {$y$};

            \path
            (1) edge (2)
                edge (3)
                edge (4)
            (2) edge (4)
            (3) edge (4)
            (4) edge (5);
        \end{tikzpicture}
        \caption{A graph of order 5 and size 6.}
        \label{fig:simple}
\end{figure}

In Figure~\ref{fig:simple}, $V(G) = \{u,v,w,x,y\}$ and $E(G) = \{uv, ux, uw, vx, wx, xy\}$. If $e=uv$ is an edge of $G$, then $u$ and $v$ are said to be \textbf{joined} by the edge $e$. In this case, $u$ and $v$ are referred to as \textbf{neighbors} of each other.

%prove characterization of eulerian circuits, and maybe also do hamiltonians
For a connected graph $G$, any open trail that contains every edge of $G$ is an \textbf{Eulerian trail}. If $G$ contains a closed Eulerian trail, it is \textbf{Eulerian}

\begin{theorem}
    A nontrivial connected graph $G$ is Eulerian if and only if every vertex of $G$ has even degree.
\end{theorem}

With this, we can easily characterize graphs possessing an Eulerian trail.
\begin{corollary}
    A connected graph $G$ contains an Eulerian trail if and only if exactly two vertices of $G$ have odd degree. Furthermore, each Eulerian trail of $G$ begins at one of these odd vertices and ends at another.
    \label{cor:euler}
\end{corollary}

There is an analog to Eulerian trails. A path in a graph $G$ that contains every vertex of $G$ is called a \textbf{Hamiltonian path}.\footnote{We studied Eulerian \textit{trails} because such trails may need to have repeated vertices, unlike paths which necessarily have unique vertices.} Unfortunately, these are much less well-behaved.

\section{Banquet Arrangement}
Here's Problem 2 from the discussion worksheet.
\begin{problem}
    Suppose $n$ people are attending a banquet, and each of them has at least $m$ friends $(2\leq m\leq n)$, where friendship is mutual.
    Prove that we can put at least $m+1$ of the attendants on the same round table, so that each person sits next to his or her friends on both sides.
\end{problem}

\begin{proof}[Solution]

As an example, suppose Alice, Bob, Claire, and Dom are attending, and the friend pairings are (Alice, Claire), (Alice, Dom), (Bob, Claire), (Bob, Dom), so $n = 4$ and $m = 2$ in this case.
We actually can't choose exactly three of them so that they all sit next to their friends, but we can certainly choose to sit the four them in the order (Alice, Claire, Bob, Dom).

If you haven't realized already, it's helpful to recast this problem in terms of graphs to solve it in general.
A big tipoff is both the fact that there are people and mutual friendships which strongly hint to being vertices and edges respectively as well as the topic of this discussion. So we will set
\[ \text{ people }\mapsto \text{ vertices } \]
and
\[ \text{ friendship }\mapsto \text{ edges }.\]

If we were to represent the above problem in graphs, it'd look something like this:

\begin{figure}[!ht]
\begin{center}
\begin{tikzpicture}[>=stealth', auto, semithick, node distance=2cm]
\tikzstyle{every state}=[fill=white,draw=black,thick,text=black,scale=0.8]
\node[state]    (A)               {$A$};
\node[state]    (B)[below of=A]   {$B$};
\node[state]    (C)[right of=A]   {$C$};
\node[state]    (D)[right of=B]   {$D$};
\path
(A) edge    (C)
    edge    (D)
(B) edge    (C)
    edge    (D);
\end{tikzpicture}
\end{center}
\caption{Casting the problem in terms of graphs.}
\end{figure}

We also see that sitting people around a table so that they sit next to their friends is just asking to find a cycle of length at least $m+1$. Using the above graph, it's easy to find one such cycle, $ACBD$, by starting at $A$ and walking on the graph until you come back to $A$.

So that's the easy part of the problem. Now let's see how we would create such a cycle. Naively, we might start at an arbitrary vertex, say $v_0$, and try to walk along this graph. We know $v_0$ has at least $m$ neighbors, so let's visit one of them, $v_1$.

Now $v_1$ also has at least $m$ neighbors, but we've already been to one of them, $v_0$, so it really only has $m-1$ unvisited neighbors, so let's visit one of them, $v_2$. We can keep repeating this process of visiting unvisited neighbors until we're out of new neighbors to get a path

\[ P = v_0v_1\dots v_l.\]

One thing we should note is that in the worst case, the number of unvisited neighbors for each $v_i$ is at least $m-i$, which is if all of the $i$ vertices already in the path are neighbors of $v_i$.
So $l \geq m$, or else we would still have neighbors we could visit.

All that's left for us now is to get a cycle from this path. Since we're stuck when we reach $v_l$, all of its neighbors must be in $P$. So let's make a new path starting at $v_l$ by backpedaling from $v_l$ until we reach the last neighbor of $v_l$ in $P$, say $v_k$. Now we have a path
\[ P' = v_l v_{l-1} \dots v_k\]
where $P'$ is also at least $m+1$ since $v_l$ and all of its neighbors are in the path. But $v_k$ is neighbors with $v_l$, so this is also a cycle, and we're done.
\end{proof}

We used the principle of trying things until we get stuck, and trying to fix things until we're free. You'll find that this often works surprisingly well.

The official solution is quite slick as well, but less motivated.
It considers the \emph{longest} path in the graph, and constructs the cycle from that.
This is an example of the \textbf{Extremal Principle}, where one looks at either the largest or the smallest object satisfying some property.

\begin{proof}[Official Solution]
    Let $P=v_0v_1\dots v_l$ be a longest path in the graph.
    Such a path exists because the length of paths is bounded above by $n$.
    All neighbors of $v_0$ must be in $P$, since otherwise $P$ can be extended to be even longer by appending this edge at the beginning of path $P$.
    Let $k$ be the maximum index of neighbors of $v_0$ along $P$.
    Since $v_0$ has at least $m$ neighbors, we must have $k\geq m$.
    Then $v_0v_1\dots v_kv_0$ gives us the desired cycle.
\end{proof}

Here's a cute example of the Extremal Principle applied in another setting. Before you read on, think about it for a few minutes.

\begin{problem}
    There are $n$ students standing in a field such that the distance between each pair is distinct. Each student is holding a ball, and when the teacher blows a whistle, each student throws their ball to the nearest student. Prove that there is a pair of students that throw their balls to each other.
\end{problem}
\begin{proof}
    Consider the two students closest to each other. Since they are each other's nearest student, they must throw their balls to each other.
\end{proof}

Try this one for another extremal flavored graph problem. How would you do it without looking at the extreme cases?
\begin{exercise}
    Show that every tree $G$ contians at least two leaves, or vertices of degree 1.
\end{exercise}

\iffalse

%%%%%%%%%%%%% SECTION %%%%%%%%%%%%5
\section{Planarity Bounds}
A graph $G$ is called a \textbf{planar graph} if $G$ can be drawn in the plane so that no two of its edges cross each other. If $G$ is planar, then it divides the plane into pieces called \textbf{regions}. Recall that for any planar, connected graph $G$, if $G$ has $V$ vertices, $E$ edges, and $F$ faces, Euler's formula tells us that
    \[ V-E+F = 2. \]

You can imagine how hard it is to prove a graph to be non-planar: you can't possibly check every way of drawing the graph. Euler's Identity leads to some simple conditions on planarity.

We can derive many useful bounds using this formula that involve only two of the three quantities, which is helpful especially for showing that certain graphs are nonplanar. Usually we omit faces since those are tricky to quantify, so let's try to compare the number of faces to the number of edges.

\begin{figure}[!htb]
\begin{center}
\begin{tikzpicture}[>=stealth', auto, semithick, node distance=2cm]
\tikzstyle{every state}=[fill=white,draw=black,thick,text=black,scale=0.8]
\node[state]    (A)               {};
\node[state]    (B)[right of=A]   {};
\node[state]    (C)[right of=B]   {};
\node[state]    (D)[below of=A]   {};
\node[state]    (E)[right of=D]   {};
\path
(A) edge    node[below = 0.5cm] {$F_1$} (B)
    edge    (D)
(B) edge    node[below = 0.6cm, left] {$F_2$} (C)
    edge    (E)
(C) edge    node[below right = 0.5cm] {$F_3$} (E)
(D) edge    (E);
\end{tikzpicture}
\end{center}
\caption{An example planar graph $G_1$ with its three faces marked.}
\label{fig:graph}
\end{figure}

Let's look at the graph $G_1$ shown in Figure~\ref{fig:graph}. Notice that every edge is a part of exactly two faces. So we can count the number of edges also by looking at how many edges border each face. Let $|F_i|$ denote the number of edges that border the face $F_i$. Then,
\begin{align*}
    2E &= |F_1| + |F_2| + |F_3| \\
       &= 4 + 3 + 5 \\
       &\geq 3 + 3 + 3 \\
       &= 3F
\end{align*}
where we used the very deep fact that every face is bordered by at least 3 edges (how else do you have a face?). Hence, we've arrived at the inequality $2E \geq 3F$. Now if we solve our formula for $F$ and substitute, we get the all important bound

\[ 2 - V + E = F \leq \frac{2}{3} E \implies \boxed{E \leq 3V-6}.\]

This is useful now as a method for showing a graph is nonplanar, as otherwise we'd have to draw every possible configuration of a graph and show all of them have crossings, which is neither feasible nor convincing. We can use this to show that $K_5$, the complete graph on 5 vertices, is nonplanar. Since it has $5$ vertices and $10$ edges, $10 \not\leq 3\cdot 5 - 6 = 9$.

One extension of the above bound is to find a bound when $G$ is triangle-free, i.e. no face is bounded by 3 edges. I'll leave that as an exercise for you. Recall that the graph with six vertices, where three vertices are connected to all three other vertices, is the complete bipartite graph $K_{3,3}$. We can use this new bound to show $K_{3,3}$ is nonplanar.

\begin{exercise}
    Show that if $G$ is a connected, planar, triangle-free graph, then $E \leq 2V - 4$. Use this to show that $K_{3,3}$ is nonplanar.
\end{exercise}

As a hint, we proved that all bipartite graphs have no odd tours. So what do we know about the presence of triangles in such a graph?

\begin{figure}[!htb]
    \centering
    \begin{tikzpicture}[scale = 0.7]

  \begin{scope}[xshift=-6cm,yshift=-1cm]
    \foreach \x in {4,2,0} {
      \foreach \y in {0,2,4} {
        \drawLinewithBG{\x,0}{\y,2};
      }
    }

    \foreach \x in {0,2,4} {
      \foreach \y in {0,2} {
        \node at (\x,\y) [circle,fill=black] {};
      }
    }

    \node at (2,-1.5) {\Large$K_{3,3}$};
  \end{scope}

  \begin{scope}[xshift=3cm]
    \foreach \a in { 18, 90, 162, 234, 306 } {
      \foreach \b in { 18, 90, 162, 234, 306 } {
        \drawPolarLinewithBG{\a:2}{\b:2};
      }
    }

    \foreach \a in {18,90, 162, 234, 306 } {
      \node at (\a:2cm) [circle,fill=black] {};
    }
    \node at (0,-2.5) {\Large$K_5$};
  \end{scope}
\end{tikzpicture}
\caption{The Kuratowski graphs}
\end{figure}

We can already see how much trouble it can be to determine planarity for arbitrary graphs, let alone simple ones like $K_{3,3}$ and $K_5$. Surprisingly, the main enemies to planarity are precisely these two graphs, which leads to a simple check to see if a graph is planar, discovered in 1930. Before I can state Kuratowski's remarkable theorem, I need to state two notions.

A \textbf{subdivision} of a graph is constructed by replacing one edge by two edges with a vertex in between, and a \textbf{subgraph} is constructed by removing vertices or edges.

\begin{theorem}[Kuratowski]
    A graph $G$ is planar if and only if $G$ does not contain a subdivision of $K_5$ or $K_{3,3}$ as a subgraph.
\end{theorem}

In other words, $G$ is planar if it is not possible to subdivide the edges of $K_5$ or $K_{3,3}$, and then possibly add edges or vertices, to get $G$.

\subsection{To Take Home}
\begin{exercise}[Dis. Problem 2b]
    Consider graphs with the property $T$: For every three distinct vertices $v_1,v_2,v_3$ of graph $G$ , there are at least two edges among them.
    Prove that if $G$ is a graph on $\geq$ 7 vertices, and $G$ has property $T$, then $G$ is nonplanar.
\end{exercise}
Hint: Proof by contradiction when $v = 7$. What do we know about groups of five vertices in a planar graph?

\section{Induction on Edges and Vertices}
Problem 3 from the discussion sometimes gives people trouble, so I'll try to explain it some more in depth. Here is the problem, paraphrased.

\begin{problem}
An edge coloring of a graph is an assignment of colors to edges in a graph where any two edges incident to the same vertex have different colors.
\alphanum
    \ii Prove that any graph with maximum degree $d\geq 1$ can be edge colored with $2d-1$ colors.
    \ii Show that any tree with maximum degree $d\geq 1$ can be edge colored with $d$ colors.
\enumend
\end{problem}

The solution is to use induction on the number of edges in the first question, and induction on the number of vertices in the second. I'm not going to re-explain the solution (you can find it online) but I'll try to explain why it's fine to induct this way.

Imagine if I had the following problem:
\begin{question}
    Prove that any graph with maximum degree $4$ can be edge colored with $7$ colors.
\end{question}

Now induction, especially on the number of edges, seems like a viable approach. We're proving some statement about all graphs, so let's assume its true for a graph with $m$ edges and prove it for one with $m+1$ edges (both with max degree 4). What about this problem:

\begin{question}
    Prove that any graph with maximum degree $10$ can be edge colored with $19$ colors.
\end{question}

See what I'm getting at? Just because we have a variable $d$ in our original statement doesn't mean we need to induct over $d$. We can treat $d$ as a constant and show the statement is true for all graphs given a specific $d$, which then shows the statement is true for all $d$.

Another analogy: we don't need to induct over our variable $d$ much like how the number of vertices and edges are also variables, but we don't need to induct over both.

\fi

\end{document}


\documentclass[12pt,a4]{article}
\author{Yujie Lu \quad Jiabao Ji \quad Tong Chen}
\newcommand{\handoutdate}{2020-03-05}
\newcommand{\firstduedate}{2020-03-10}
\newcommand{\finalduedate}{2020-03-17 before class}
\usepackage{pythonhighlight}
\usepackage{import}
\usepackage[many]{tcolorbox}
\tcbuselibrary{skins, breakable, theorems}
\tcolorboxenvironment{exercise}{
  enhanced,
  borderline={0.4pt}{0.4pt}{black},
  boxrule=0.4pt,
  colback=white,
  coltitle=black,
  sharp corners,
}

\input{preamble}

\section{Bit Complexity, Recursion, and Dynamic Programming}

\subsection{Bit Complexity of Euclid's Algorithm}

We have proved that Euclid's algorithm for computing $\gcd(a,b)$ makes at most
$O(\log a)$ iterations. What is the overall running time? Each iteration computes
$u \mod v$ for some integers. This can be done by integer division. What is its running time?
There are very sophisticated algorithms, but python probably does not come with them.
Recall the ``school method'' for dividing integers. Have a look at the pdf slides on the
webpage for an illustration of the school method. It is especially simple if we are dealing
with binary numbers. If $a$ and $b$ have at most $n$ bits, then the school method
has complexity $O(n^2)$.
\begin{exercise}
  Show the following, more precise bound of the school method for integer division:
  If $a$ has $n$ bits and $b$ has $k$ bits, then the school method can be implemented
  to run in $O( k(n-k))$ operations.
\end{exercise}
\begin{proof}
First, we know $n > k$ since $a > b$. To get r, $r = a \% b, (0 \leq r < b)$,
so the bits of $r$, we write it as $b_r$. $b_r < k$.

Remind the procedure of brute-force algorithm of $a \% b$, in each iteration,
$n$ decreases(since we can drop the higher bits). In the final iteration, $n < k$,
also this $n$ is exactly $b_r$, so we don't need to do the iteration $n$ times,
but only $n - k + 1$ times. And each iteration costs $k$ subtraction operation, which is an integer operation,
in all we do no more than $k(n - k+ 1)$ times of basic operations. That is
$O\left( k\left( n-k \right)  \right) $ basic operations.
\end{proof}
\begin{exercise}
  Show that the bit complexity of Euclid's algorithm, using the school method
  to compute $a \mod b$, is $O(n^2)$. That is,
  if $a$ and $b$ have at most $n$ bits, then $\gcd(a,b)$ makes $O(n^2)$ bit operations.\\

  In order to do so, here is python code of the Euclidean algorithm:
\begin{verbatim}
  def euclid(a,b):
    while (b > 0):
        r = a % b # so a = bu+r
        if (r == 0):
            return b
        s = b % r # so b = rv + s
        a = r
        b = s
    return a
\end{verbatim}
Don't be afraid to introduce notation! I recommend to let $n$ denote the number of bits of $a$.
Take some other letters for the number of bits in $b$ and so on.
\end{exercise}
\begin{proof}
		Assume that $\gcd\left( a,b \right) $ takes $t$ iterations and $t \le  2n$. To better represent our ideas.  Let's introduce
		some notations. Conside a process of iterations as follows:
		\[
				\gcd\left( a,b \right) = \gcd\left( x_{t},x_{t-1} \right) \to
				\gcd\left( x_{t-1},x_{t-2} \right) \to \ldots \to \gcd\left( x_1,0 \right)
		.\]
		And $\forall 1\le k\le t$ let $x_{k}$ has $y_{k}$ bits. We know $y_{t}\le n$ plus $x_{k}>x_{k-1}$.
		Hence $\forall k, y_{k}\ge y_{k-1}$.

		Let the operations of all iterations be $T$. And the operations of  $k$th iteration be $T_{k}$.
		Now apply the conclusion of Exercise 1, we have
		\begin{align*}
				T &= \sum_{k=1}^{t} T_{k} \\
				  &\le \lambda \sum_{k=1}^{t} y_{k}\left( y_{k}-y_{k-1} \right) \\
				& \le \frac{\lambda}{2}\left( \sum_{k=1}^{t} \left( y_{k}-y_{k-1} \right)^2 + y_t^2 \right) \\
				  & \le  \frac{\lambda}{2} \left( \sum_{k=1}^{t} \left( y_{k}-y_{k-1} \right)  \right)^2+ \frac{\lambda}{2} y_{t}^2\\
				  & \le  \frac{\lambda}{2} n^2
		.\end{align*}
		Hence $\gcd\left( a,b \right) $ makes $O\left( n^2 \right) $ operations.
\end{proof}
\subsection{Computing the Binomial Coefficient}

Next, we will investigate the binomial coefficient ${n \choose k}$, which
you might also know by the notation $C^k_n$. The number ${n \choose k}$ is defined
as the number of subsets of $\{1,\dots,n\}$ which have size exactly $k$.
This immediately shows that ${n \choose k}$ is $0$ if $k$ is negative or larger than $n$.
You might have seen the following recurrence:
\begin{align*}
 {n \choose k} & = {n-1 \choose k-1} + {n-1 \choose k} \textnormal{ if } n,k \geq 0 \ .
\end{align*}

\begin{exercise}[A Recursive Algorithm for the Binomial Coefficient]
  Using pseudocode, write a recursive algorithm computing
  ${n \choose k}$. Implement it in python! What is
  the running time of your algorithm, in terms of $n$ and $k$? Would you say it is an efficient
  algorithm? Why or why not?
\end{exercise}
\begin{proof}
recursive code:
\begin{python}
def recursiveCompute(n, k):
    if (n <= k or k == 0):
        return 1
    else:
        return recursiveCompute(n - 1, k - 1) + recursiveCompute(n - 1, k)
\end{python}

Like recursive algorithm for computing Fib(n), we can get a recursive tree,
which has $ \binom n k$ leaves and $ \left(2 \binom n k - 1\right) $ nodes in all.
We can prove it by induction.

Firstly, prove it for a fixed n, to do that, we use an induction.
And suppose the equation is true for all $m < n$ (base $n = 1$ is trivial).\\
\textbf{Base:} $k = 0:$

Obviously, the recursive tree only has 1 node, which satisfies the equation.\\
\textbf{Induction:}

To compute $\binom n {k+1} $, we will add a node for $\binom n {k+1}$ as the new root,
and two sub recursive tree for computing $\binom {n - 1} {k}, \binom {n - 1}{k + 1}$,
whose nodes size is known by induction hypothesis.
The new tree has
\[
\binom {n-1} {k} + \binom{n - 1} {k + 1} = \binom{n}{k + 1}
.\]
leaves, and
\[
\left(2\binom{n - 1}{k} - 1 \right)+ \left(2\binom{n - 1}{k + 1} -1\right) + 1
= 2\binom n {k + 1} - 1
.\]
nodes.

Similar to the induction above, we can tell for any $n, k$, the result holds.
For the running time analysis, each node in the tree needs to do an addition,
so it's $O\left(2 \binom n k - 1\right)= O\left( \binom{n}{k}  \right) $.
If we consider the cost of addition at each node: $\log \binom{n}{k}$, we get
$O((\binom n k )log(\binom n k))$.

To sum up, the algorithm's time complexity is
\[
		\Omega\left(\binom n k\right)\text{ and } O\left(\binom n k \log\left
		(\binom n k\right)\right).
\]
It's not a good algorithm, since the time complexity is very high, and it does a lot of
redundant addition.
\end{proof}
\begin{exercise}[A Dynamic Programming Algorithm for the Binomial Coefficient]
  Using pseudocode, write a dynamic programming algorithm
  computing ${n \choose k}$. Implement it in python! What is it running time
  in terms of $n$ and $k$?
  Would you say your algorithm is efficient? Why or why not?
\end{exercise}
\begin{proof}
code:
\begin{python}
def dpCompute(n, k):
    a = [[0 for x in range(k + 1)] for y in range(n + 1)]
    for i in range (n + 1):
        a[i][0] = 1
        if i <= n - k:
            for j in range (1, min(i, k) + 1):
                a[i][j] = a[i - 1][j - 1] + a[i - 1][j]
        else:
            for j in range(i - (n - k), min(i, k) + 1):
                a[i][j] = a[i - 1][j - 1] + a[i - 1][j]
    return a[n][k]
\end{python}
The algorithm needs to do $n$ iterations, for iterations $i \le  n - k$,
we should do $\mathrm{min}  (i, k)$ additions, and for $i > n - k$, we should do $\mathrm{min} (i, k) - i - (n - k)$ additions
(since there is no node of $\binom i j $ if $ i > n - k, j < i - (n - k)$ in the recursive tree)
In all, we need to compute
$$
    \sum_{i = 1}^{n - k} \sum_{j = 1}^{min(i, k)} 1 +
    \sum_{i = n - k + 1}^{n + 1}\sum_{j = i - (n - k)}^{min(i, k) + 1} 1
    = n (n - k)
$$
additions.

This is not trivial, so we draw a picture to better visualize this
\begin{figure}[ht]
    \centering
	\incfig[0.8]{triangle}
    \caption{How much do we compute.}
    \label{fig:triangle}
\end{figure}

Consider the cost of each addition, the algorithm is $\Omega(n(n - k))$ and $O(n(n - k) log \binom n k)$.
It's much more efficcient than the recursive algorithm. I think it's efficcient.

\end{proof}

\begin{exercise}[Binomial Coefficient modulo 2]
  Suppose we are only interested in whether ${n \choose k}$ is even or odd,
  i.e., we want to compute ${n \choose k}  \mod 2$. You could do this by computing
  ${n \choose k}$ using dynamic programming and then taking
  the result modulo $2$. What is the running time? Would you say this algorithm
  is efficient? Why or why not?
\end{exercise}
\begin{proof}
  The running time is similar to Exer.4, but since we only needs to know whether the result modula 2 is 1 or 0,
  we don't need to compute addition at each node but only a bool compute. \\
  \hspace*{1em} So the dynamic algorithm is $\Theta (n(n - k))$.\\
  \hspace*{1em} To this question specifically,not efficcient,
  we don't have to compute the value of $\binom n k$ at all.\\
  Below is the python code.
  \newpage
  \begin{python}
  def hasNtwoFactor(asking):
      res = 0
      while (asking % 2 == 0):
          res += 1
          asking = asking / 2
      return res

  def isEvenOrOdd(n, k):
      numerator = 0
      dominator = 0
      for i in range(1, n + 1):
          numerator += hasNtwoFactor(i)
      for i in range(1, k + 1):
          dominator += hasNtwoFactor(i)
      for i in range(1, n - k + 1):
          dominator += hasNtwoFactor(i)
      if (numerator - dominator > 0):
          return bool(1)
      else:
          return bool(0)
  \end{python}
  it's an $\Theta(n log(n))$ algorithm, faster than dpCompute.

We also have another algorithm based on \textbf{Lucas Theorem}
\begin{python}
def C(n, k):
	if n & k == k:
        return 1;
    else:
        return 0;
\end{python}

Now, let's proof this simple algorithm why it's correct.
\begin{theorem}
	\textbf{(Lucas Theorem)} For non-negative integers \textit{m} and \textit{n} and a prime \textit{p}, the following congruence relation holds:
	$${m \choose n} \equiv \prod_{i=0}^k {m_i \choose n_i} ( mod\ p),$$
	where
	$$m=m_kp^k+m_{k-1}p^{k-1}+\cdots+m_1p+m_0$$,
	and
	$$n=n_kp^k+n_{k-1}p^{k-1}+\cdots+n_1p+n_0$$
	are the base p expansions of m and n respectively. This uses the convention that ${m\choose n}=0$ if $m<n$.
\end{theorem}
Here, we let p = 2, and we have ${0 \choose 0}={1\choose 0}={1 \choose 1}=1$ and ${0 \choose 1}=0$. Hence, we get ${n\choose k}\equiv 1 (mod\ 2)$ if and only if every bit of k is less than that of n in binary representation. That means $n\ and\ k = k$.

The time complexity of this algorithm is $\Theta(log(n))$
\end{proof}
\begin{exercise}
  Remember the ``period'' algorithm for computing $F'_n := (F_n \mod k)$ discussed in class:
  (1) find some $i,j$ between $0$ and $k^2$ for which
  $F'_{i} =  F'_{j}$ and $F'_{i+1} = F'_{j+1} $.
  Then for $d := j-i$ the sequence $F'_{n}$ will repeat every $d$ steps, as there will be a cycle.
  This cycle can either be a ``true cycle'' or a ``lasso'':
  \begin{center}
  \includegraphics[width=0.8\textwidth]{figures/cycle-and-lasso.pdf}
  \end{center}

  Show that a lasso cannot happen. That is, show
  that the smallest $i$ for which this happens is $0$, i.e, for some $j$ we have
  $F'_0 = F'_j$ and $F'_1 = F'_{j+1}$ and thus $F'_n = F'_{n \textnormal{ mod }  j}$.
\end{exercise}

\begin{proof}
		Consider a pair $g\left( i \right) =\left( F'_{i-1},F'_{i} \right) , i\ge 0$. We know that
		$\forall i, g\left( i \right) \in [0,n)\times [0,n)$ which is a finite set with $k^2$ members.

		Consider the following set
		\[
				A = \{ g\left( 1 \right) ,g\left( 2 \right) ,\ldots, g\left( k^2 \right) , g\left( k^2+1 \right) \}
		.\]
		By \textbf{Pigeonhole Principle}, we know that $\exists r< s\in \{1,2,\ldots,k^2+1\} $ such that
		$g\left( r \right) =g\left( s \right) $ which is
		\[
				F'\left( r-1 \right) = F'\left( s-1 \right) , \quad F'\left( r \right)  = F'\left( s \right)
		.\]
		Since $F'\left( r-2 \right)$ is determined by $F'\left( r \right) -F'\left( r-1 \right) $, we can imply that
		\[
				F'\left( 0 \right)  = F'\left( s-r \right) , \quad F'\left( 1 \right)  = F'\left( s-r+1 \right)
		.\]
		Hence this cycle is a ``true cycle".
\end{proof}

\newpage
{\huge\noindent \textbf{Appendix}   \\ \\ }
{\large Pseudocode for Exercise 3, 4}
\begin{algorithm}[h]
		\caption{A function $f$ using recursive algorithm.}
		\KwIn{input parameters $n,k$}
		\KwOut{$ {n\choose k }$}
		\LinesNumbered
		\If{$n=k$ or $k=0$} {
				return $1$
		}
		\Else {
				return $f\left( n-1,k-1 \right) + f\left( n-1,k \right) $
		}
\end{algorithm}

\begin{algorithm}[h]
		\caption{A function $g$ using dynamic algorithm.}
		\KwIn{input parameters $n,k$}
		\KwOut{$ {n\choose k }$}
		\LinesNumbered
		a is a 2d array to store results.

		\For{$i$ in  $\{1,2,\ldots,n+1\} $ } {
				a[i][0] = 1
				\If {$i\le n-k $ } {
						\For{$j\le   \mathrm{min}\left( i,k \right) $ }
						{
								update a[i][j]
						}
				}
				\Else {
						\For {$j \le   \mathrm{min}\left( i,k \right)  \} $} {
								update a[i][j]
						}
				}
		}
		return a[n][k]
\end{algorithm}
\end{document}


\documentclass[12pt]{article}
\usepackage{latexsym}

\def\Def{Definition}
\def\a{for every }
\def\A{For every }
\def\e{there exists }
\def\E{There exists }
\def\pf{{\em Proof. }}
\def\eps{\varepsilon}
\def\bx{\bar{x}}
\def\by{\bar{y}}
\def\vphi{\varphi}

 \begin{document}

\newtheorem{dfn}{Definition}[section]
\newtheorem{eqn}{Equation}[section]
\newtheorem{lem}{Lemma}[section]
\newtheorem{thm}{Theorem}[section]
\newtheorem{cor}{Corollary}[section]

\title{A Brief Overview of Lyapunov Stability for Ordinary Differential Equations}
\author{David Glickenstein}
\date{May 2, 1996}
\maketitle

\abstract{Lyapunov stability is a way to characterize the qualitative
idea of stability in models of physical systems.  A stable solution to
a differential equation is one which does not change much when the
initial conditions are changed slightly.  We shall look at two methods
of checking the stability of solutions to ordinary differential
equations.  The first works by characterizing the solutions of linear
systems to study their stability and then looking at nonlinear systems
which are close to the linear case.  The second uses an auxilliary
function to find stability without having to characterize the
solutions.  Finally we look at some of the limitations of the Lyapunov
notion of stability.}

\section{Introduction}

We are going to look at general ordinary differential equations of the
following type:
\begin{equation}
	x'(t) = f(t,x(t))
	\label{gen}
\end{equation}
where $x: \Re \rightarrow \Re^n$ and $f:\Re \times \Re^n \rightarrow
\Re^n$ are functions and $t$, which usually represents time, is the
only independent variable.  The derivative $x'(t)$ is simply the
derivatives of each of the component functions, so if $x(t) =
(x_1(t),...,x_n(t))$ where $x_i(t)$ are real-valued functions on $\Re$
then
$$x'(t) = (x_1'(t),...,x_n'(t)).$$
In some cases we will be considering the autonomous case
\begin{equation}
	x'(t) = f(x(t))
	\label{aut}
\end{equation}
where $f:\Re^n \rightarrow \Re^n$.  We are going to assume the
existence of continuous solutions and uniqueness of a solution with
the initial condition $x(t_0) = x_0$ where $t_0$ is a constant and
$x_0$ is a constant n-vector.  In the simplest case, a solution will
tend to a constant as $t$ goes to infinity.  In the autonomous case,
we can think of the solution as a continuous curve through n-space
(called phase space), and when the solution tends to a constant, the
curve will converge to a point in phase space.  If a constant function
$c$ satisfies the differential equation, then the the derivative
$x'(t) = f = 0$ for $x_0 = c$ and $c$ is called an equilibrium point.
Periodic solutions satisfy $x(t) = x(t + n\tau)$ for some $\tau$ and
any integer $n$.  Other, more complicated solutions also exist which
may or may not be stable.  Nonautonomous systems have analogous types
of solutions.

We are going to be studying the stability of (\ref{gen}).  Stability
is a qualitative notion that if we perturb the initial condition
slightly, say $x(t_0) = x_0 + \eps$, then the new solution will be
close to the original system.  Stable differential equations are more
useful in practice since $x_0$ is often a  measured value and does
not have infinite precision.  A stable differential equation would
insure that our model would come up with an answer reasonably close to
the actual answer an exact initial condition.

First, we will define our notion of stability.  What we shall refer to
as {\em stable} is more correctly termed {\em Lyapunov stable} since
there are many different mathematical notions of stability.

\begin{dfn}
  	A solution $x(t)$ is {\em stable} if \a $\eps >0$
	\e $\delta > 0$ such that if $\| x_0 - \bx_0 \| < \delta$ then
	$\| x(t) - \bx(t) \| < \eps$ \a $t \ge t_0$, where $\bx(t)$ is
	the solution to
	$$\bx'(t) = f(t, \bx(t))$$
with $x(t_0) = x_0$ and $\bx(t_0) = \bx_0$.
\end{dfn}

Sometimes our model works so well that if we change the initial
condition slightly, the solution will still converge to the exact
solution under the original initial conditions.  This means that when
we use a measured value, our approximate model (with an inexact
initial condition) not only stays close to the actual solution, but
gets closer and closer to the actual solution as $t$ increases.  This
notion is called {\em asymptotic stability}.

\begin{dfn}
	A solution $x(t)$ is {\em asymptotically stable} if it is stable
	and there is a $\rho > 0$ such that if $\| x_0 - \bx_0 \| < \rho$
	then
$$\lim_{t \rightarrow \infty} \| x(t) - \bx(t) \| = 0.$$
\end{dfn}

In general, asymptotic stability is more desirable, since we know that
no matter where we place our initial condition, we will converge to
the actual answer if we are close enough to the actual initial
condition.  Unfortunately, asymptotic stability is not always
possible, as we shall see in the fourth section.

We will primarily use these two notions of stability.  There
are two main methods used to study Lyapunov stability, known
as Lyapunov's First and Second (or Direct) Methods.

\section {Lyapunov's First Method}

Lyapunov's First Method works by characterizing solutions to
differential equations of the type (\ref{gen}) and using those
characterizations to learn what we can about stability.  Thus our
approach will begin with simple linear systems and move on to certain
kinds of nonlinear systems which exhibit similar behavior to the
linear systems.

We will first look at linear differential equations

\begin{equation}
	x'(t) = A(t)x(t)
    \label{linear}
\end{equation}

where $A: \Re ^n \rightarrow \Re ^n$ is linear and $x: \Re \rightarrow \Re^n.$

When working with linear systems of the form (\ref{linear}) we will
use the following theorem, without proof, to characterize solutions.
The theorem is presented as a matrix equation where the derivative
$C'(t)$ of a matrix function $C$ is simply defined as the derivative
of each of the entries of the matrix.

\begin{thm}
	If $\Phi '(t) = A(t) \Phi (t)$ with $\Phi$ an invertible $n \times
n$ matrix then all solutions of (\ref{linear}) are linear combinations
of the columns of $\Phi$ and, in particular, we can choose some $\Phi
(t)$ such that $\Phi (t_0) = I$ where $I$ is the identity matrix, so
that $x(t) = \Phi (t) x_0$ is a solution to (\ref{linear}).  \end{thm}

\pf Omitted.

\begin{dfn}
	$\Phi$ as described in the last theorem is called the {\em
fundamental matrix} of the differential equation.
\end{dfn}

The fundamental matrix depends only on the differential equation and
thus only on the matrix function $A(t)$.

We also will need to define the operator norm for a matrix A:

\begin{dfn}
	If $A$ is an $n \times n$ real matrix then
$$\| A \| = \sum^n_{i,j=1} |a_{ij}|$$
\end{dfn}

Note that $\| Ax \| \le \| A \| \  \|x\|$ where $\| A \|$ is the operator
norm and the other two norms are the usual Euclidean vector norm in $\Re^n$.

We are now ready to prove an important theorem.  It turns out that in
the linear case the property of solutions being stable is equivalent
to the property of solutions being bounded.

\begin{thm}
	All solutions of (\ref{linear}) are stable if and only if they are
bounded.
\end{thm}

% begin proof
\pf All solutions are of the form $\Phi (t) x_0$ where $\Phi$ is the
fundamental matrix such that $\Phi (t_0) = I$.  Let $x(t)$ and
$\bx(t)$ be solutions satisfying $x(t_0) = x_0$ and $\bx(t_0) =
\bar{x}_0$ respectively.

Assume that all solutions to (\ref{linear}) are bounded.  Thus we can
find an $M$ such that $\| \Phi (t) \| < M$ for all $t > 0$.  Given
some $\eps > 0$, if $\| x_0 - \bx_0 \| < {\eps \over M}$ then
\begin{eqnarray*}
	\| x(t) - \bx(t) \| & = & \| \Phi (t)x_0 - \Phi (t) \bx_0 \| \\
	& \le & \| \Phi (t) \| \ \| x_0 - \bx_0 \| \\
	& < & \eps.
\end{eqnarray*}

Conversely, assume that all solutions are stable.  Thus the particular
solution $x(t) \equiv 0$ is stable, so \a $\eps > 0$ \e $\delta > 0$
such that if $\| \bx_0 \| < \delta$ then
$$\| \bx (t) \| = \| \Phi (t) \bx_0\| < \eps.$$
In particular, let
$$\bx_0^{(i)} = {\delta \over 2} e_i $$
where $e_i$ is the ith standard basis vector of $\Re^n$.  Then
$$\| \Phi (t) \bx_0^{(i)} \| = \| \phi_i(t) \| {\delta \over 2} <
\eps$$
where $\phi_i$ is the ith column of $\Phi$.  We can sum these over all
i and get:
$$\| \Phi (t) \| \le \sum^n_{i=1} \| \phi_i(t) \| \le {2n \eps \over
\delta} = k$$
so $$\| x(t) \| = \| \Phi (t) x_0 \| \le k \| x_0 \|$$
and $x(t)$ is bounded. $\Box$
%end proof

Note that boundedness is not necessarily true for stable systems in
general.  For instance, take the differential equation $x'(t) = 1$
whose solutions are all stable but not bounded.

Lyapunov's First Method consists mainly of showing that solutions of linear
systems, especially if $A(t)$ is a constant funtion, are bounded and
hence stable.  It then looks at functions which are nonlinear, but
simply small perturbations of linear functions.  We shall state
without proof several theorems.  For a more detailed explanation with
proofs, see [2].

Consider the constant linear equation:
\begin{equation}
	x'(t) = A \ x(t)
	\label{constlin}
\end{equation}
where $A$ is a constant matrix.  We can easily show the following.

\begin{thm}
	If all eigenvalues of A have negative real parts then every
solution to (\ref{constlin}) is asymptotically stable.
\end{thm}

\pf Omitted.

We can easily extend this to equations with slight perturbations.  So
we characterize the following nonlinear equation

\begin{equation}
	x'(t) = A \ x(t) + f(t,x(t))
	\label{nonlinear}
\end{equation}

with the following theorem.

\begin{thm}
	If f satisfies:

\begin{enumerate}
\item $f(t,x)$ is continuous for $\| x \| < a, \ 0 \le t <\infty$ and
\item $\lim_{\| x \| \rightarrow 0} {\| f(x,t) \| \over \| x \|} = 0$
\end{enumerate}
then the solution $x(t) \equiv 0$ is asymptotically stable.
\end{thm}

It turns out that to characterize the general linear case where $A$ is a
function of time, we need a slightly stronger notion of stability
called uniform stability.  The reader is again referred to [2].

\section{Lyapunov's Second Method}

Lyapunov's Second or Direct Method is unique in that it does not
require a characterization of the solutions to determine stability.
This method often allows us to determine whether a differential
equation is stable without knowing anything about what the solutions
look like, so it is ideal for dealing with nonlinear systems.

The method uses a supplementary function called a Lyapunov function to
determine properties of the asymptotic behavior of solutions to a
differential equation of the general form (\ref{gen}).  We will need
the following definitions.

\begin{dfn}
	A function $V(t,x)$ is {\em positive [negative]
definite} if \e a real-valued function $\vphi (r)$ such that:

\begin{enumerate}
\item $\vphi (r)$ is strictly increasing on $0 \le r \le a$ and
$\vphi (0) = 0$ and
\item $V(t,x) \ge \vphi ( \| x \| ) \ [V(t,x) \le - \vphi ( \| x \| )]$
for all $(t,x) \in \{ (t,x): t_0 \le t < \infty, \| x \| \le b < a \}$.
\end{enumerate}

\end{dfn}

\begin{dfn}
	$$V' = \sum^n_{i=1} {\partial V \over \partial x_i} f_i(t,x) +
{\partial V \over \partial t}$$
\end{dfn}
Note that $V'(t,x(t))$ is the time derivative of $V$.

\begin{dfn}
	A real function $V(t,x)$ is said to admit an {\em infinitesimal upper bound} if \e
$h > 0$ and a continuous, real-valued, strictly increasing function
$\psi$ with $\psi (0) = 0$ such that
$$|V(t,x)| \le \psi( \| x \|) \ {\rm for} \ \| x \| < h \ {\rm and} \
t \ge t_0.$$
\end{dfn}

Using these definitions we shall construct a function $V$.  If we can
find such a $V$, then we can find stable solutions to the differential
equation.  Unfortunately, there is no general way to construct $V$
from the differential equation (\ref{gen}).  Note that we can define
$V$ in terms of $t$ and $x$ (not $x(t)$) and since our definition of $V'$
uses the function $f$, the existence of $V$ will have implications for
the differential equation.

\begin{thm}
	If a continuous real-valued function $V(t,x)$ exists such that:
\begin{enumerate}
\item $V(t,x)$ is positive definite and
\item $V'(t,x)$ is non-positive
\end{enumerate}
then $x(t) \equiv 0$ is a stable solution of (\ref{gen}).
\end{thm}

\pf We know that $V(t,x)$ is positive definite, so \e a strictly
increasing function $\vphi (r)$ such that
$$0 < \vphi (\| x \| ) \le V(t,x)$$
for $0 < \| x \| < b$ and $t > t_0$.

Given $\eps > 0$ let
$$m_{\eps} = \min_{\| x \| = \eps} \vphi (\| x \| ) = \vphi (\eps).$$ Notice that
$m_{\eps} > 0$.  Since $V$ is continuous and $V(t,0) = 0$ for all $t$
(by positive definiteness),
we can choose a $\delta > 0$ such that
$$V(t_0, x_0) < m_{\eps}$$
if $\|x_0 \| < \delta$.  Now
$$V'(t,x(t)) \le 0$$
for $t_1 \ge t_0$ and $\| x_0 \| < \delta$ implies
$$V(t_1, x(t_1)) \le V(t_0,x(t_0)) = V(t_0,x_0) < m_{\eps}$$

Now, suppose for some $t_1 \ge t_0$ that $\| x(t_1) \| \ge \eps$
whenever $\| x_0 \| < \delta$.  Then
$$V(t_1,x(t_1)) \ge \vphi (\| x(t_1) \| ) \ge \vphi (\eps) =
m_{\eps}$$
which is a contradiction, so \a $\eps > 0$ \e $\delta > 0$ such that
$\| x(t) \| < \eps$ if $\| x_0 \| < \delta$ for $t \ge t_0$,
i.e. $x(t) \equiv 0$ is stable.  $\Box$

We call a function $V$ which satisfies the hypotheses of the theorem a
{\em Lyapunov function}.  In physical systems, the Lyapunov function
used is often an expression for energy, for instance stability for the
differential equation for a spring:
\begin{eqnarray*}
x'(t) & = & y\\
y'(t) & = & -kx
\end{eqnarray*}
can be done with $V(x,y) = {1 \over 2} y^2 + {1 \over 2} k x^2$, which is the equation
for total energy of the system.

By strengthening the requirements on $V$ we can find conditions for
asymptotic stability.

\begin{thm}
	If a continuous function $V(t,x)$ exists satisfying:
\begin{enumerate}
\item $V(t,x)$ is positive definite,
\item $V(t,x)$ admits an infinitesimal upper bound, and
\item $V'(t,x)$ is negative definite
\end{enumerate}
then the solution $x(t) \equiv 0$ of (\ref{gen}) is asymptotically
stable.
\end{thm}

\pf  By the previous theorem, the solution must be stable.  Suppose
that $x(t)$ is not asymptotically stable, so \a $\eps >0$ there exist
$\delta > 0$ and $\lambda > 0$ such that any nonzero solution $x(t)$ where
$x(t_0) = x_0$ satisfies $\lambda \le \| x(t) \| < \eps$ if $t \ge
t_0$ and $\| x_0 \| < \delta$.

Since $V'(t,x)$ is negative definite, there is a strictly increasing function
$\vphi (r)$ vanishing at the origin such that $V'(t,x) \le -
\vphi(\| x \|)$.  We know that $\| x(t) \| \ge \lambda > 0$ for $t \ge
t_0$ so there is a $d > 0$ such that $V'(t,x(t)) \le -d$.  This
implies that
\begin{eqnarray*}
	V(t,x(t)) & = & V(t_0,x_0) + \int^t_{t_0} V'(s,x(s)) ds \\
	& \le & V(t_0,x_0) - (t- t_0) d
\end{eqnarray*}
which is less less than zero for large $t$, contradicting the
assumption that $V(t,x)$ is positive definite.

Thus no such $\lambda$ exists and since $V(t,x(t))$ is positive definite and
decreasing with respect to $t$
$$\lim_{t \rightarrow \infty} V(t,x(t)) = 0$$
which implies that
$$\lim_{t \rightarrow \infty} \| x(t) \| = 0,$$
since $V(t,x(t)) \le \psi (\| x(t) \|)$ where $\psi (r)$ is strictly
increasing and vanishes at the origin (by the second hypothesis).
Thus as $V(t,x(t)) \rightarrow 0$, $\psi(\| x(t) \|) \rightarrow 0$,
and $\| x(t) \| \rightarrow 0$ (because $\psi$ is strictly increasing
and vanishes at 0).  Thus $x(t) \equiv 0$ is asymptotically stable. $\Box$

We can also use the direct method to determine instability in some cases.

\begin{thm}
	If a continuous real-valued function $V(t,x)$ exists on some set
$S = \{ (t,x) : t \ge t_1 \ and \ \| x \| < a \} $ satisfying:
\begin{enumerate}
\item $V(t,x)$ admits an infinitesimal upper bound,
\item $V'(t,x)$ is positive definite on S, and
\item \E a $T > t_1$ such that if $t_0 \ge T$ and $h > 0$ then \e $c
\in \Re^n$ such that $\| c \| < h$ and $V(t_0,c) > 0$
\end{enumerate}
then the solution $x(t) \equiv 0$ is not stable.
\end{thm}

\pf Suppose $x(t)$ is a solution not identically zero.  Then, by
uniqueness, $ \| x(t) \| \neq 0$ for all $t \ge t_1$.  If $x(t)$ is
defined at $t_0$ then for every $t \ge t_0$ for which $x(t)$ is
defined
$$V(t,x(t)) - V(t_0, x(t_0)) = \int^t_{t_0} V'(s,x(s)) ds > 0$$
since $V'(t,x(t))$ is positive definite.

Let $t_0 \ge T$ and $\eps > 0$.  There exists a $c$ such that $\| c \|
< \min(a,\eps)$ and $V(t_0,c) > 0$ by the third hypothesis.  Since $V$
admits an infinitesimal upper bound and is continuous, there is a
$\lambda \in (0,a)$ such that $\| x \| < \lambda$ for $t \ge t_1$.
Therefore,
$$| V(t,x) | < V(t_0, c).$$

Let $x(t)$ satisfy the initial condition $x(t_0) = c$.  Then $V(t,x(t)
> V(t_0, x(t_0)) > 0$ (from the positive definiteness argument), so
$\| x(t) \| \ge \lambda$.  Also by the second hypothesis we know that
there is a strictly increasing function $\vphi(r)$ such that $V'(t,x)
\ge \vphi(\| x \|)$.  Let
$$ \mu = \min_{\| x \| \in [\lambda, a]} \vphi(\| x \|) = \vphi (\lambda).$$
then
$$V'(t,x(t)) \ge \vphi(\| x(t) \|) \ge \mu.$$
Using the integral inequality, we get
$$V(t,x(t)) \ge V(t_0, x(t_0)) + (t - t_0) \mu.$$ Thus we can make
$V(t, x(t))$ arbitrarily large.  Since V(t,x) admits an infinitesimal
upper bound, there exists a strictly increasing function $\psi (r)$
such that $|V(t,x(t))| < \psi (\| x \|)$ and $\psi (0) = 0$.  So since
$V(t,x(t))$ can be arbitrarily large, $\psi(\| x(t) \|)$ can also be
arbitrarily large, and thus $\| x(t) \|$ gets arbitrarily large and the
solution is not stable. $\Box$

\section {Limitations of Lyapunov Stability}

The Lyapunov notion of stability has some problems, some of which we
shall look into here.  Asymptotic stability as we have defined it does
not always work well for periodic systems, so often other notions such
as orbital stability are used when studying differential equations
which have periodic solutions.  In addition, certain classes of
systems such as Hamiltonian Systems do not react well to a Lyapunov
analysis.  Finally, there is the question of whether Lyapunov
stability actually reflects our qualitative idea of stability.

We will first look at how Lyapunov stability works with periodic
solutions to the autonomous system (\ref{aut}).  It turns out that our
notion of asymptotic stability is incompatible with such solutions,
since the points will not get closer together, even thought two
solutions may get closer and closer to the same final state (a
periodic cycle).  To prove this we will first need a Lemma that
solutions to autonomous systems are invariant under translation.

\begin{lem}
	If $x(t)$ is a solution to (\ref{aut}) then $x(t+h)$ is also a
solution for any real number $h$.
\end{lem}

\pf Let $\bx (t) = x(t+h)$ and $s = t+h$.  Then
\begin{eqnarray*}
	\bx ' (t) & = & {dx \over ds}{ds \over dt} \\
	& = & x'(s)\\
	& = & f(x(s))\\
	& = & f(x(t+h))\\
	& = & f(\bx(t))\\
\end{eqnarray*}
so $\bx(t) = x(t+h)$ is a solution to (\ref{aut}). $\Box$

Now we can use the fact that translations of solutions are also
solutions to show that nontrivial periodic solutions cannot be
asymptotically stable.  By nontrivial periodic we mean that the
solutions do not converge to a constant vector, i.e., $x(t) \neq x(t +
\eps)$ for some small $\eps > 0$.

\begin{thm}
	If $x(t)$ is a nontrivial periodic solution of an autonomous
	system (\ref{aut}) then $x(t)$ is not asymptotically stable.
\end{thm}

\pf Suppose $x(t)$ is a nontrivial periodic solution.  Thus \e $t_0$
such that $f(x(t_0)) \neq 0$ because if not, $x'(t) = 0$ for all $t$
and the solution is constant (trivially periodic).  Suppose also that
$x(t)$ is asymptotically stable.

The stability assumption says that given $\eps > 0$ \e $\hat{\delta}
> 0$ such that $\| x(t) - \bx(t) \| < \eps$ if $ \| x_0 - \bx_0 \| <
\hat{\delta}$ where $x(t)$ satisfies $x(t_0) = x_0$ and $\bx(t)$ is a
solution to the same equation satisfying the initial condition
$\bx(t_0) = \bx_0$.

By the asymptotic stability assumption we can find a $\rho > 0$ such
that
$$\lim_{t \rightarrow \infty} \| x(t) - \bx(t) \| = 0$$
if $\| x_0 - \bx_0 \| < \rho$.

Let $\delta = \min (\hat{\delta}, \rho)$.  By the continuity of $x(t)$
we can fix some small $\bar{t}$ so that
$$\| x(t_0) - x(t_0 + \bar{t}) \| < \delta.$$
By the Lemma, $x(t + \bar{t})$ is a solution to the differential
equation, so by our asymptotic stability assumption
$$\lim_{t \rightarrow \infty} \| x(t) - x(t + \bar{t})\|  = 0.$$

We know that $x'(t_0) \neq 0$ so we can find $r > 0$ so that $\|x(t_0)
- x(t_0 + \bar{t}) \| > r$.  Also $x(t)$ is periodic, so assume it has
period $\tau$.  Then
$$\| x(t_0 + n \tau) - x(t_0 + \bar{t} + n \tau) \| > r >0$$
for all $n$, so the limit cannot go to zero, which contradicts our
assumption of asymptotic stability.  Thus $x(t)$ cannot be
asymptotically stable. $\Box$

Another class of problems which for which asymptotic stability does
not work is Time-Independent Hamiltonian systems.  They are defined as follows.

\begin{dfn}
	A {\em time-independent Hamiltontian system (TIHS)} is the 2n
dimensional system
\begin{eqnarray*}
	x'(t) & = & H_y(x(t), y(t)) \\
	y'(t) & = & -H_x(x(t), y(t))
\end{eqnarray*}
where $x(t)$ and $y(t)$ are both functions from $\Re$ to $\Re^n$ and
$H(x,y)$ is a real valued function with continuous partial
derivatives.
\end{dfn}

Hamiltonian Systems are used to model a large class of physical systems
and have many applications.  It turns out that, like nontrivial
periodic solutions to autonomous systems, no solutions to a TIHS can be
asymptotically stable.

\begin{thm}
	If $(x(t), y(t))$ is a solution to a TIHS then it is not
asymptotically stable.
\end{thm}

\pf Suppose $(x(t),y(t))$ is an asymptotically stable solution.  Then
we can find $t_0$ and $\delta > 0$ such that if $(\bx(t), \by(t))$ is
a solution and if $\| (x(t_0),y(t_0)) - (\bx(t_0), \by(t_0)) \| < \delta$ then
$$\lim_{t \rightarrow \infty} \| (x(t),y(t)) - (\bx(t), \by(t)) \| =
0.$$

But also,
$${d \over dt} H(x,y) = {\partial H \over \partial x}{dx \over dt} +
{\partial H \over \partial y}{dy \over dt} = H_x (H_y) + H_y (-H_x) =
0$$
so $H(x,y)$ is constant on any solution $(x(t),y(t))$.  By the continuity of $H$,
it must be constant on some $\delta$-neighborhood N of
$(x(t_0),y(t_0))$, so every point in N is an equilibrium point.  Thus
$(x(t),y(t))$ cannot be asymptotically stable (for initial condition
$(\bx(t_0),\by(t_0)) \in {\rm N}$, the solution
$(\bx(t),\by(t))$ will not converge to $(x(t),y(t))$. $\Box$

It turns out that although Lyapunov stability is extremely useful in
mathematically defining our qualitative view of stability, it is
neither necessary nor sufficient for something to appear stable on a
physical level.  Some examples can be found at the end of Chapter 4 of [1].

\begin{thebibliography}{99}
\bibitem{Cronan} J. Cronin. {\em Differential Equations: Introduction
and Qualitative Theory.} Marcel Dekker, New York, 1994.
\bibitem{Sanchez} D.A. Sanchez. {\em Ordinary Differential Equations
and Stability Theory: An Introduction.}  Dover Publications, New York,
1979.
\end{thebibliography}

\end{document}


\newcommand{\of}{\texttt{.o}~}

\lab{Interfacing With Other Programming Languages Using Cython}{Interfacing With Other Programming Languages Using Cython}
\label{lab:cythonwrap}

\objective{Learn to interface with object files using Cython. This lab should be worked through on a machine that has already been configured to build Cython extensions using gcc or MinGW.}

Suppose you are writing a program in Python, but would like to call code written in another language.
Perhaps this code has already been debugged and heavily optimized, so you do not want to simply re-implement the algorithm in Python or Cython.
In technical terms, you want Python to \emph{interface} (or communicate) with the other language.
For example, NumPy's linear algebra functions call functions from LAPACK and BLAS, which are written in Fortran.

One way to have Python interface with C is to write a Cython \emph{wrapper} for the C function.
The wrapper is a Cython function that calls the C function.
This is relatively easy to do because Cython compiles to C.
From Python, you can call the wrapper, which calls the C function (see Figure TODO).
In this lab you will learn how to use Cython to wrap a C function.

\section*{Wrapping with Cython: an overview}
When you use Cython to wrap a C function, you just write a Cython function that calls the C function.
To actually do this, we need to understand a little bit more about how C works.

After you write a program in C, you pass it to a compiler.
The compiler turns your C code into machine code, i.e., instructions that your computer can read.
The output of the compiler is an \emph{object file}.

In our scenario, we have C code defining a single function that we would like to wrap in Cython.
This C code has already been compiled to an object file.
The protocol for calling the C code from Cython is similar to calling it from C:
\begin{enumerate}
\item we need to \emph{include a header file} for the C code in our Cython code, and
\item we must \emph{link to the object file} compiled from the C code when we compile the Cython code.
\end{enumerate}

A \emph{header} for a file contains the declaration of the function defined in the file.
We include the header in the Cython code so that the compiler can check that the function is called with the right number and types of arguments.
Then, when we tell the compiler to \emph{link to} the object file, we simply tell it where to find the instructions defining the function declared in the header.

The Cython code will compile to a second object file.
This object file defines a module that can be imported into Python. See Figure \ref{fig:cython_flowchart}.

%TODO: make figure
\begin{figure}
\begin{tikzpicture}[dot/.style={circle,fill=black,minimum size=3pt,inner sep=0pt,
            outer sep=-1pt}, >=stealth', thick]

\node[draw, minimum height=1.5cm, minimum width=3cm, align=center](ctri1)
	[]{\Large{ctridiag.h} \\ {\small declare \texttt{ctridiag()}}};

\node[draw, minimum height=1.5cm, minimum width=3cm, align=center, node
	distance=2.5cm](ctri2)[below of=ctri1]{\Large{ctridiag.c} \\ {\small define \texttt
	{ctridiag()}}};

\node[draw, minimum height=1cm, minimum width=2cm, align=center, node
	distance=2.2cm, color=white!40!black](ctri3)[below of=ctri2]{COMPILE};

\node[draw, minimum height=.7cm, minimum width=3cm, align=center, node
	distance=1.9cm](ctri4)[below of=ctri3]{ \Large{ctridiag.o} \\{\small object file callable} \\ {\small from C}};

%\node[draw, minimum height=1cm, minimum width=3cm, align=center, node
%	distance=.85cm](ctri5)[below of=ctri4]{object file  \\ callable by C};

\draw[-, color=white!40!black](ctri2)--(ctri3);
\draw[->, shorten >=3pt,color=white!40!black](ctri3)--(ctri4);

\node[draw, minimum height=1.5cm, minimum width=3cm, align=center, node
	distance=7cm](ctri6)[right of=ctri1]{\Large{cython\_ctridiag.pyx}
	 \\ {\small define \texttt{cytridiag()}}};

\node[draw, minimum height=1cm, minimum width=2cm, align=center, node
	distance=3.3cm,color=white!40!black](ctri7)[below of=ctri6]{COMPILE};

\node[draw, minimum height=1.5cm, minimum width=3cm, align=center, node
	distance=3.3cm](ctri8)[below of=ctri7]{\Large{ctridiag.so} \\ {\small object file
	callable} \\ {\small from Python}};

\draw[-, color=white!40!black](ctri6)--(ctri7);
\draw[->, shorten >=3pt,color=white!40!black](ctri7)--(ctri8);
\draw[->, dashed, shorten >=3pt,color=white!40!black](1.5,-6.95)--(6, -3.3);
\draw[->, shorten >=3pt,color=white!40!black](ctri1)--(ctri6);

\node[draw, minimum height=.7cm, minimum width=1cm, align=center, node
	distance=3.1cm,color=white!40!black, fill=white](include)[right of=ctri1]
	{INCLUDE};

\node[draw, minimum height=.7cm, minimum width=1cm, align=center, node
	distance=5.8cm,color=white!40!black, fill=white](include)at(3.7,-5.15){LINK};

\end{tikzpicture}
\caption{This diagram shows the relationships between the files used to write a Cython wrapper for \li{ctridiag()}.
In fact, the Cython file \texttt{cython\_ctridiag.pyx} compiles first to C code and then to the shared object file, but we have omitted this step from the diagram.}
\label{fig:cython_flowchart}
\end{figure}

\section*{Wrapping with Cython: an example}
As an example, we will wrap the C function \texttt{ctridiag()} below.
This function computes the solution to a tridiagonal system $A\v = \x$.
Its parameters are pointers to four 1-D arrays, which satisfy the following:
\begin{itemize}
\item Arrays \li{a} and \li{c} have length \li{n-1} and contain the first subdiagonal and superdiagonal of $A$, respectively.
\item Arrays \li{b} and \li{x} have length \li{n} and represent the main diagonal of $A$ and $\x$, respectively.
\end{itemize}
The array \li{c} is used to store temporary values and \li{x} is transformed into the solution of the system.

\lstinputlisting[style=fromfile, language=C, name=ctridiag.c]{./ctridiag/ctridiag.c}

This terminal command tells the C-compiler gcc to compiles \texttt{ctridiag.c} to an object file \texttt{ctridiag.o}.

\begin{lstlisting}
>>> gcc -fPIC -c ctridiag.c -o ctridiag.o
\end{lstlisting}
The \texttt{-fPIC} option is required because we will later link to this object file when compiling a shared object file.
The \texttt{-c} flag prevents the compiler from raising an error even though \texttt{ctridiag.c} does not have a \li{main} function.

\subsection*{Write a header for \texttt{ctridiag.c}}

The header file essentially contains a function declaration for \li{ctridiag()}.
It tells Cython how to use the object file \li{ctridiag.o}.
\lstinputlisting[style=fromfile, language=C, name=ctridiag.h]{./ctridiag/ctridiag.h}

\subsection*{Write a Cython wrapper}
Next we write a Cython file containing a function that ``wraps'' \li{ctridiag()}.
This file must include the header we wrote in the previous step.

\lstinputlisting[style=fromfile, language=Python, name=cython_ctridiag.pyx]{./ctridiag/cython_ctridiag.pyx}

Some comments on this code are in order.
First, including the header for \li{ctridiag()} allows us to call this function even though it was not defined in this file.
This is a little like importing NumPy at the start of your Python script.

Second, the arguments of the Cython function \li{cytridiag()} are not in bijection with the arguments of \li{ctridiag()}.
In fact, Cython does not need to know the size of the NumPy arrays it accepts as inputs because these arrays are objects that carry that information with them.
We extract this information and pass it to the C function inside the Cython wrapper.

However, it is possible to unintentionally access memory outside of an array by calling this function.
For example, suppose the input arrays \li{a}, \li{b}, \li{c}, and \li{x} are of sizes 4, 5, 3, and 5.
Since \li{x} is used to determine the size of the system, the function \li{ctridiag()} will expect \li{c} to have length 4.
At some point, \li{ctridiag()} will likely try to read or even write to the 4th entry of \li{c}.
This address does exist in memory, but it does not contain an entry of \li{c}!
Therefore, this function must be called by a responsible user who knows the sizes of her arrays.
Alternatively, you could check the sizes of the arrays in the Cython wrapper before passing them to the C function.

Finally, the C function expects a parameter \li{double* a}, meaning that \li{a} is a \emph{pointer to} (i.e., the address of) a \li{double}.
The funtion \li{ctridiag()} expects this double to be the first entry of the array \li{a}.
So instead of passing the object \li{a} to \li{ctridiag()}, we find the first entry of \li{a} with \li{a[0]}, and then take its address with the \li{&} operator.

\subsection*{Compile the Cython wrapper}

Now we can compile \texttt{cython\_ctridiag.pyx} to build the Python extension.
The following setup file uses distutils to compile the Cython file, and may be run on Windows, Linux, or Macintosh-based computers.
Notice that in line 28, we link to the existing object file \texttt{ctridiag.o}.

\lstinputlisting[style=fromfile, language=Python, name=ctridiag_setup_distutils.py]{./ctridiag/ctridiag_setup_distutils.py}

This setup file can be called from the command line with the following command.
\begin{lstlisting}[style=ShellInput]
python ctridiag_setup_distutils.py build_ext --inplace
\end{lstlisting}
The \texttt{--inplace} flag tells the script to compile the extension in the current directory.
The appendix at the end of this lab contains setup files that build the Python extension by hand on various operating systems.

\subsection*{Test the Python extension}
After running the setup file, you should have a Python module called \li{cython_cytridiag} that defines a function \li{cytridiag()}.
You can import this module into IPython as usual.
However, if you modify \li{ctridiag()} and then try to recompile the extension, you may get an error if the module is currently in use.
Hence, if you are frequently recompiling your extension, it is wise to test it  with a script.

The following script tests the module \li{cython_cytridiag}.

\lstinputlisting[style=fromfile, language=Python, name=ctridiag_test.py]{./ctridiag/ctridiag_test.py}

\begin{warn}
When passing arrays as pointers to C or Fortran functions, be \emph{absolutely sure} that the array being passed is contiguous.
This means that the entries of the array are stored in \emph{adjacent} entries in memory.
Passing one of these functions a strided array will result in out of bounds memory accesses and could crash your computer.
For an example of how to check an array is contiguous in a Cython wrapper, see Problem \ref{prob:c_wrap}.
\end{warn}

\section*{Another example}
%TODO: replace the C function for this problem with one that implements Monte Carlo integration

The C function \li{cssor()} below implements the Successive Over Relaxation algorithm for solving Laplace's equation, which in two dimensions is
\[
\frac{\partial^2 F}{\partial x^2} + \frac{\partial^2 F}{\partial y^2} = 0.
\]
In the Successive Over Relaxation algorithm, an input array is iteratively modified until it converges to the solution $F(x,y)$.

\lstinputlisting[style=fromfile, language=C, name=cssor.c]{./cssor/cssor.c}

The function \li{cssor()} accepts the following inputs.
\begin{itemize}
\item $U$ is a pointer to a Fortran-contiguous 2-D array of double precision numbers.
\item $m$ is an integer storing the number of rows of $U$.
\item $n$ is an integer storing the number of columns of $U$.
\item $\omega$ is a double precision floating point value between $1$ and $2$.
The closer this value is to $2$, the faster the algorithm will converge, but if it is too close the algorithm may not converge at all.
For this lab just use $1.9$.
\item tol is a floating point number storing a tolerance used to determine when the algorithm should terminate.
\item maxiters is an integer storing the maximum allowable number of iterations.
\item info is a pointer to an integer.
This function will set info to $0$ if the algorithm converged to a solution within the given tolerance.
It will set info to $1$ if it did not.
\end{itemize}

\begin{problem}\label{prob:c_wrap}
Wrap \li{cssor()} so it can be called from Python, following these steps.
\begin{enumerate}
\item Write a C header for \li{cssor.c}.
\item Write a Cython wrapper \li{cyssor()} for the function \li{cssor()} as follows.
    \begin{enumerate}
    \item Have \li{cyssor()} accept parameters \li{tol} and \li{maxiters} that default to $10e^{-8}$ and 10,000, respectively.
    What other arguments does \li{cyssor()} need to accept?
    \item Check that the input \li{U} is a C-contiguous array (this means that the array is stored in a certain way in memory).
    You can check if \li{U} is C-contiguous with the code \li{U.is_c_contig()}, which returns a Boolean value.
    If \li{U} is not C-contiguous, raise a ValueError as follows:
    \begin{lstlisting}
    raise ValueError('Input array U is not C-contiguous')
    \end{lstlisting}
    When raising a ValueError, you may replace the string \li{'Input array U is not C-contiguous'} with any desired text.
    For more about errors and exceptions in Python, see \url{https://docs.python.org/2/tutorial/errors.html}.
    \item Raise a ValueError if \li{cssor()} fails to converge---i.e., if \li{cssor()} sets \li{info} to 1.
    \end{enumerate}
\item Compile \li{cssor.c} to an object file and compile your Cython wrapper to a Python extension.
Be aware that you may compile \li{cssor.c} correctly and still receive warnings from the compiler.
\item Write a test script for your Cython function.
    \begin{enumerate}
    \item You can run the function with the following code.
    \begin{lstlisting}
	import numpy as np
	from cython_fssor import cyssor
	resolution = 501
	U = np.zeros((resolution, resolution))
	X = np.linspace(0, 1, resolution)
	U[0] = np.sin(2 * np.pi * X)
	U[-1] = - U[0]
	cyssor(U, 1.9)
	\end{lstlisting}
    \item Have your test script run the code above and plot the modified array \li{U} on the domain $[0,1]\times[0, 1]$.
    Note that $U$ is a $501\times 501$ array.
    Your plot should look like Figure \ref{fig:solution}.
    \end{enumerate}
\end{enumerate}

%TODO: mayavify this plot.
\begin{figure}[H]
\includegraphics[width=\textwidth]{solution.png}
\caption{Correct output for the test script of Problem \ref{prob:c_wrap}.}
\label{fig:solution}
\end{figure}

\end{problem}

\section*{Wrapping a Fortran function (Optional)}
We can also use Cython to wrap Fortran.
As an example, we will wrap the Fortran function below, which implements the same algorithm as \li{ctridiag()}.

We have used the C library \li{iso_c_binding} to make the function accept pointers to native C types.
If we were wrapping a function or subroutine that we did not write ourselves, we would have to define a Fortran function that uses the \li{iso_c_binding} library to accept pointers from C and then uses the values it receives to call the original function.
\lstinputlisting[style=fromfile, language=Fortran, name=ftridiag.f90]{./ftridiag/ftridiag.f90}

\begin{warn}
When interfacing between Fortran and C, you will have to pass pointers to \emph{all} the variables you send to the Fortran function as arguments.
Passing a variable directly will probably crash Python.
\end{warn}

\subsection*{Write a header for \texttt{ftridiag.f90}}
Here is a header that tells C how to interface with the function we have just defined.
\lstinputlisting[style=fromfile, language=C, name=ftridiag.h]{./ftridiag/ftridiag.h}

To compile \li{ftridiag.f90} to an object file you can run the following command in your command line:
\begin{lstlisting}[style=ShellInput]
gfortran -fPIC -c ftridiag.f90 -o ftridiag.o
\end{lstlisting}

\subsection*{Write a Cython wrapper, compile, and test it}
The Cython wrapper for this function is analogous to \li{cython_ctridiag.pyx}, except that every variable passed to \li{ftridiag} should be a pointer.
Aside from the use of gfortran instead of gcc, the rest of the compilation and testing process is entirely the same.
The setup files specific to Windows and Linux, the setup file using distutils, and the test script are included in this lab folder.

\begin{warn}
Fortran differs from C in that the columns of arrays are stored in contiguous blocks of memory instead of the rows.
The default for NumPy and for C arrays is to have rows stored in contiguous blocks, but in NumPy this varies.
You should make sure any code passing arrays between Fortran, C, and NumPy addresses this discrepancy.
Though by default Fortran assumes arrays are arranged in Fortran-order, for many routines you can specify whether to act on an array or its transpose.
\end{warn}

\begin{info}
The function \li{ftridiag()} may seem unusual because it is a Fortran function that is callable from C.
In fact, this is not an uncommon scenario.
Many larger Fortran libraries (for example, LAPACK) provide C wrappers for all their functions.
\end{info}

\subsection*{Another example}

Here is a Fortran implementation of the Successive Over Relaxation algorithm for solving Laplace's equation.
This function is the Fortran equivalent of \li{cssor.c}.
Its parameters are the same, with the exception that all parameters are \emph{points} to the values they reference.

\lstinputlisting[style=fromfile, language=Fortran, name=fssor.f90]{./fssor/fssor.f90}

\begin{problem}
Imitate Problem \ref{prob:c_wrap} to wrap \li{fssor()} with Cython.
\end{problem}

\section*{Appendix: compiling C extensions for Python}
If you know more about compilers, you may find it enlightening to manually compile a C extension for Python.
Here we show how to manually compile \texttt{cython\_ctridiag.pyx} on Windows, Linux, and Macintosh machines.

On Windows, using the compiler MinGW (a version of gcc for windows), the compilation can be performed by running the following setup file.

\lstinputlisting[style=fromfile, language=Python, name=ctridiag_setup_windows64.py]{./ctridiag/ctridiag_setup_windows64.py}

The following file works on Linux and Macintosh machines using gcc.

\lstinputlisting[style=fromfile, language=Python, name=ctridiag_setup_linux.py]{./ctridiag/ctridiag_setup_linux.py}


\documentclass[12pt,letterpaper,twoside]{article}

\usepackage{cme212}

\begin{document}

{\centering \textbf{Paper Exercise 2\\ Due Tuesday, February 11th by
    4:30 P.M. PST via GradeScope\\}}
\vspace*{-8pt}\noindent\rule{\linewidth}{1pt}

\paragraph{Question 0: } Write an implementation of {\tt compute\_median} below that computes and returns the median value of an array of {\tt double}s. The implementation should have an average time complexity no worse than $\O(n)$ and are only allowed to use functionality available in the header files \href{https://en.cppreference.com/w/cpp/algorithm}{{\tt <algorithm>}} and \href{https://en.cppreference.com/w/cpp/container/vector}{{\tt <vector>}}. For an even number of values, return the mean of the two middle values. HINT: Find a more general function in {\tt <algorithm>} that you can map this {\tt compute\_median} onto.
\begin{cpp}
#include <algorithm>
#include <vector>

/** Compute the median of an array of doubles.
 * @param[in] values  The array of values.
 * @param[in]      n  The number of values in the array.
 * @result The median value of the array.
 *
 * @pre @a n > 0
 * @post Let sorted_v be @a values sorted by op<.
 *         If @a n odd,  result ==  sorted_v[n/2]
 *         If @a n even, result == (sorted_v[n/2] + sorted_v[n/2-1])/2
 */
double compute_median(const double* values, unsigned n)
\end{cpp}

\paragraph{Question 1: Lecture 4} Consider a data type $U$ with abstract type $U =
\{u_0,\dots,u_{n-1}\}$, where each $u_i$ is an integer. $U$'s iterator
iterates over the values in \emph{any} order. Assume $U$ has been implemented with a \texttt{std::vector<int> value\_} that contains all the elements. Write an
implementation of \texttt{U::iterator U::erase(U::iterator it)} that
has $O(1)$ time complexity. The specification is not necessary and we
emphasize that the returned iterator references an unordered
collection of elements.

\paragraph{Question 2: Lecture 4} Use the \texttt{erase} method you wrote above to write a method with the
following specification. Never use an invalid iterator---all iterators to a
\texttt{vector} become invalid after any \texttt{insert} or
\texttt{erase}---and your method should work for other STL containers, such
as \href{https://en.cppreference.com/w/cpp/container/map}{\texttt{map}} and
\href{https://en.cppreference.com/w/cpp/container/list}{\texttt{list}}.

\begin{cpp}
/** Erase all elements of @a x for which @a pred returns true.
 * @param[in,out] x Container of elements.
 * @param[in] pred Predicate that takes an element and returns a bool.
 *
 * @post new @a x.size() <= old @a x.size()
 * @post new @a x contains exactly those elements e of old @a x for
 *     which @a pred(e) returned false, and in the same relative order.
 */
template <typename Container, typename Predicate>
void erase_if(Container& x, Predicate pred);
\end{cpp}

\paragraph{Question 3: Lecture 6}
Remember the \texttt{Book} class from Exercise 1?
\begin{cpp}
	class Book:
	{
		public:
		Book(std::string filename)
		{
			std::string sent;
			std::ifstream book_file(filename);
			while(std::getline(book_file, sent))
			{
				book_.push_back(split(sent, ' '));
			}
			book_file.close();
		}

	}
\end{cpp}

There is a problem with this class, it does not implement RAII. There are two points where we need to fix this.

\textbf{a}

In the constructor if anything goes wrong in the \texttt{while} loop the error will immediately propagate up the stack and the \texttt{close()} function will never be called. This can cause memory leak. Use exception handling to fix this issue.

\textbf{b}

Assume that we are reading in large books and are worried that we might run out of space on the stack so we are allocating \texttt{std::vector<std::vector<string>> book\_} on the heap. this would change the class outline as such.
	\begin{cpp}
	class Book:
	{
		public:
		Book(std::string filename)
		{
			std::string sent;
			std::ifstream book_file(filename);
			try{
				while(std::getline(book_file, sent))
				{
					book_->push_back(split(sent, ' '));
				}
				book_file.close();
			}
			catch(){
				book_file.close()
				std::cout<<"something went wrong while reading the book."<<std::endl;
				throw;
			}
		}
		private:
		std::vector<std::vector<string>> *book_;
	}
\end{cpp}

Now there is another problem. The default destructor does not clean up the heap allocated memory. This means that as soon as a book object goes out of scope we have a memory leak. Implement the \texttt{Book} class following RAII.

\paragraph{Question 4: Lectures 6 \& 7}
We have to be careful with resource allocation when it comes to inheritance. Consider the following polymorphism example.

\begin{cpp}
	class Student
	{
		private:
			std::string name_;
		public:
			Student(std::string name): name_(name){};
			virtual std::string print()
			{
				std::cout<<"Hello my name is "<< name_<<std::endl;
			}

	};

	class UndergradStudent: public Student
	{
		private:
		std::string uni_;
		public:
		Student(std::string name, std::string uni): Student(name), uni_(uni) {};

		virtual std::string print()
		{
			std::cout<<"Hello my name is "<< name_<<std::endl;
			std::cout<<"I am an undergrad at "<< uni_<<std::endl;
		}

	};

	class GradStudent: public Student
	{
		private:
		const fstream thesis_;

		public:
		GradStudent(std::string name, std::string uni, std::string thesis):
		Student(name, uni)
		{
			thesis_.open(thesis);
		};

		virtual std::string print()
		{
			std::cout<<"Hello my name is "<< name_<<std::endl;
			std::cout<<"I am a grad student at "<< uni_<<std::endl;
		};

		~GradStudent(){
			thesis_.close();
		};
	};

	int main()
	{
		std::vector<Student*> students;
		students.push_back(GradStudent('Amy','Stanford', 'Amys_thesis.txt'));
		students.push_back(Student('Karl','Berkley'));
		students.push_back(Student('Steve'));

		for(auto student= students.begin(); student != student.end(); student++)
		{
			student->print();
		}
		delete students;
	}
\end{cpp}

	What goes wrong in the code above and how do you fix it?

\end{document}