\lab{Profiling and Optimizing Python Code}{Profiling}
\objective{Identify which portions of the code are most time consuming using a profiler. Optimize Python code using Cython, Numba, and other tools.}
\label{lab:ProfilingCode}

The best code goes through multiple drafts.
In a first draft, you should focus on writing code that does what it is supposed to and is easy to read.
After writing a first draft, you may find that your code does not run as quickly as you need it to.
Then it is time to \emph{optimize} the most time consuming parts of your code so that they run as quickly as possible.

In this lab we will optimize the function \li{qr1()} that computes the QR decomposition of a matrix via the modified Gram-Schmidt algorithm (see Lab \ref{lab:QRdecomp}).
\begin{lstlisting}
import numpy as np
from scipy import linalg as la

def qr1(A):
    ncols = A.shape[1]
    Q = A.copy()
    R = np.zeros((ncols, ncols))
    for i in range(ncols):
        R[i, i] = la.norm(Q[:, i])
        Q[:, i] = Q[:, i]/la.norm(Q[:, i])
        for j in range(i+1, ncols):
            R[i, j] = Q[:, j].dot(Q[:, i])
            Q[:,j] = Q[:,j]-Q[:, j].dot(Q[:, i])*Q[:,i]
    return Q, R
\end{lstlisting}

\section*{What to Optimize}
Python provides a \emph{profiler} that can identify where code spends most of its runtime.
The output of the profiler will tell you where to begin your optimization efforts.

In IPython\footnote{If you are not using IPython, you will need to use the \li{cProfile} module documented here: \url{https://docs.python.org/2/library/profile.html}.},
you can profile a function with \li{\%prun}.
Here we profile \li{qr1()} on a random $300 \times 300$ array.
\begin{lstlisting}
In [1]: A = np.random.rand(300, 300)
In [2]: %prun qr1(A)
\end{lstlisting}

On this computer, we get the following output.

{\scriptsize
\begin{verbatim}
         97206 function calls in 1.343 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.998    0.998    1.342    1.342 profiling_hw.py:4(qr1)
    89700    0.319    0.000    0.319    0.000 {method 'dot' of 'numpy.ndarray' objects}
      600    0.006    0.000    0.012    0.000 function_base.py:526(asarray_chkfinite)
      600    0.006    0.000    0.009    0.000 linalg.py:1840(norm)
     1200    0.005    0.000    0.005    0.000 {method 'any' of 'numpy.ndarray' objects}
      600    0.002    0.000    0.002    0.000 {method 'reduce' of 'numpy.ufunc' objects}
     1200    0.001    0.000    0.001    0.000 {numpy.core.multiarray.array}
     1200    0.001    0.000    0.002    0.000 numeric.py:167(asarray)
        1    0.001    0.001    0.001    0.001 {method 'copy' of 'numpy.ndarray' objects}
      600    0.001    0.000    0.022    0.000 misc.py:7(norm)
      301    0.001    0.000    0.001    0.000 {range}
        1    0.001    0.001    0.001    0.001 {numpy.core.multiarray.zeros}
      600    0.001    0.000    0.001    0.000 {method 'ravel' of 'numpy.ndarray' objects}
      600    0.000    0.000    0.000    0.000 {method 'conj' of 'numpy.ndarray' objects}
        1    0.000    0.000    1.343    1.343 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
\end{verbatim}
}

The first line of the output tells us that executing \li{qr1(A)} results in almost 100,000 function calls.
Then we see a table listing these functions along with data telling us how much time each takes.
Here, \li{ncalls} is the number of calls to the function, \li{tottime} is the total time spent in the function, and \li{cumtime} is the amount of time spent in the function including calls to other functions.

For example, the first line of the table is the function \li{qr1(A)} itself.
This function was called once, it took 1.342s to run, and 0.344s of that was spent in calls to other functions.
Of that 0.344s, there were 0.319s spent on 89,700 calls to \li{np.dot()}.

With this output, we see that most time is spent in multiplying matrices.
Since we cannot write a faster method to do this multiplication, we may want to try to reduce the number of matrix multiplications we perform.

\section*{How to Optimize}
Once you have identified those parts of your code that take the most time, how do you make them run faster?
Here are some of the techniques we will address in this lab:
\begin{itemize}
\item Avoid recomputing values
\item Avoid nested loops
\item Use existing functions instead of writing your own
\item Use generators when possible
\item Avoid excessive function calls
\item Write Pythonic code
\item Use Cython or Numba
\item Use a more efficient algorithm
\end{itemize}
You should always use the profiling and timing functions to help you decide when an optimization is actually useful.

\subsection*{Avoid Recomputing Values}
In our function \li{qr1()}, we can avoid recomputing \li{R[i,i]} in the outer loop and \li{R[i,j]} in the inner loop.
The rewritten function is as follows:
\begin{lstlisting}
def qr2(A):
    ncols = A.shape[1]
    Q = A.copy()
    R = np.zeros((ncols, ncols))
    for i in range(ncols):
        R[i, i] = la.norm(Q[:, i])
        Q[:, i] = Q[:, i]/R[i, i] # this line changed
        for j in range(i+1, ncols):
            R[i, j] = Q[:, j].dot(Q[:, i])
            Q[:,j] = Q[:,j]-R[i, j]*Q[:,i] # this line changed
    return Q, R
\end{lstlisting}

Profiling \li{qr2()} on a $300 \times 300$ matrix produces the following output.

{\scriptsize
\begin{verbatim}
         48756 function calls in 1.047 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.863    0.863    1.047    1.047 profiling_hw.py:16(qr2)
    44850    0.171    0.000    0.171    0.000 {method 'dot' of 'numpy.ndarray' objects}
      300    0.003    0.000    0.006    0.000 function_base.py:526(asarray_chkfinite)
      300    0.003    0.000    0.005    0.000 linalg.py:1840(norm)
      600    0.002    0.000    0.002    0.000 {method 'any' of 'numpy.ndarray' objects}
      300    0.001    0.000    0.001    0.000 {method 'reduce' of 'numpy.ufunc' objects}
      301    0.001    0.000    0.001    0.000 {range}
      600    0.001    0.000    0.001    0.000 {numpy.core.multiarray.array}
      600    0.001    0.000    0.001    0.000 numeric.py:167(asarray)
      300    0.000    0.000    0.012    0.000 misc.py:7(norm)
        1    0.000    0.000    0.000    0.000 {method 'copy' of 'numpy.ndarray' objects}
      300    0.000    0.000    0.000    0.000 {method 'ravel' of 'numpy.ndarray' objects}
        1    0.000    0.000    1.047    1.047 <string>:1(<module>)
      300    0.000    0.000    0.000    0.000 {method 'conj' of 'numpy.ndarray' objects}
        1    0.000    0.000    0.000    0.000 {numpy.core.multiarray.zeros}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
\end{verbatim}
}

Our optimization reduced almost every kind of function call by half, and reduced the total run time by 0.295s.

Some less obvious ways to eliminate excess computations include moving computations out of loops, not copying large data structures, and simplifying mathematical expressions.

\subsection*{Avoid Nested Loops}
For many algorithms, the temporal complexity of an algorithm is determined by its loops. Nested loops quickly increase the temporal complexity.
The best way to avoid nested loops is to use NumPy array operations instead of iterating through arrays.
If you must use nested loops, focus your optimization efforts on the innermost loop, which gets called the most times.

\subsection*{Use Existing Functions Instead of Writing Your Own}
If there is an intuitive operation you would like to perform on an array, chances are that NumPy or another library already has a function that does it.
Python and NumPy functions have already been optimized, and are usually many times faster than the equivalent you might write.
We saw an example of this in Lab \ref{lab:NumPyArrays} where we compared NumPy array multiplication with our own matrix multiplication implemented in Python.

\subsection*{Use Generators When Possible}
When you are iterating through a list, you can often replace the list with a \emph{generator}.
Instead of storing the entire list in memory, a generator computes each item as it is needed.
For example, the code
\begin{lstlisting}
>>> for i in range(100):
...     print i
\end{lstlisting}
stores the numbers 0 to 99 in memory, looks up each one in turn, and prints it.
On the other hand, the code
\begin{lstlisting}
>>> for i in xrange(100):
...     print i
\end{lstlisting}
uses a generator instead of a list.
This code computes the first number in the specified range (which is 0), and prints it.
Then it computes the next number (which is 1) and prints that.

It is also possible to write your own generators.
See \url{https://docs.python.org/2/tutorial/classes.html#generators} and \url{https://wiki.python.org/moin/Generators} for more information.

In our example, replacing each \li{range} with \li{xrange} does not speed up \li{qr2()} by a noticeable amount.

\subsection*{Avoid Excessive Function Calls}
Function calls take time.
Moreover, looking up methods associated with objects takes time.
Removing ``dots'' can significantly speed up execution time.

For example, we could rewrite our function to reduce the number of times we need to look up the function \li{la.norm()}.

\begin{lstlisting}
def qr2(A):
    norm = la.norm # this reduces the number of function look ups.
    ncols = A.shape[1]
    Q = A.copy()
    R = np.zeros((ncols, ncols))
    for i in range(ncols):
        R[i, i] = norm(Q[:, i])
        Q[:, i] = Q[:, i]/R[i, i]
        for j in range(i+1, ncols):
            R[i, j] = Q[:, j].dot(Q[:, i])
            Q[:,j] = Q[:,j]-R[i, j]*Q[:,i]
    return Q, R
\end{lstlisting}
Once again, an analysis with \li{\%prun} reveals that this optimization does not help significantly in this case.

\subsection*{Write Pythonic Code}
Several special features of Python allow you to write fast code easily.
First, list comprehensions are much faster than for loops. These are particularly useful when building lists inside a loop.
For example, replace
\begin{lstlisting}
>>> mylist = []
>>> for i in xrange(100):
...     mylist.append(math.sqrt(i))
\end{lstlisting}
with
\begin{lstlisting}
>>> mylist = [math.sqrt(i) for i in xrange(100)]
\end{lstlisting}
When it can be used, the function \li{map()} is even faster.
\begin{lstlisting}
>>> mylist = map(math.sqrt, xrange(100))
\end{lstlisting}
The analog of a list comprehension also exists for generators, dictionaries, and sets.

Second, swap values with a single assignment.
\begin{lstlisting}
>>> a, b = 1, 2
>>> a, b = b, a
>>> print a, b
2 1
\end{lstlisting}

Third, many non-Boolean objects in Python have truth values.
For example, numbers are \li{False} when equal to zero and \li{True} otherwise.
Similarly, lists and strings are \li{False} when they are empty and \li{True} otherwise.
So when \li{a} is a number, instead of
\begin{lstlisting}
>>> if a != 0:
\end{lstlisting}
use
\begin{lstlisting}
>>> if a:
\end{lstlisting}

\begin{problem}
Using the profiling techniques discussed in this lab, find out which portions of the code below require the most runtime. Then, rewrite the function using some of the optimization techniques we have discussed thus far.
\begin{lstlisting}
def foo(n):
    my_list = []
    for i in xrange(n):
        num = np.random.randint(-9,9)
        my_list.append(num)
    evens = 0
    for j in xrange(n):
        if j%2 == 0:
            evens += my_list[j]
    return my_list, evens
\end{lstlisting}

Hint: If you are unsure where to begin optimizing, walk through the code line by line to determine what the code is accomplishing. Then, write your own function to perform the same task in a more efficient way.
\end{problem}

\subsection*{Use Cython or Numba}
Though it is much easier to write simple, readable code in Python, it is also much slower than compiled languages such as C. Compiled languages, in general, are much faster.
Cython and Numba are two tools that you can use to optimize your code. Cython code is compiled before execution whereas Numba uses \emph{just-in-time} (JIT) compilation. We will first discuss Cython.

\subsection*{Cython}
Cython code is basically Python with extra type declarations.
This code is then compiled into C, which---depending on the details---can run much faster than the Python equivalent.
In this section we will introduce Cython as a language and discuss how it can be used to speed up Python code.

\begin{comment}
\subsection*{Compiling Cython}
With a few exceptions, every Python program is also a Cython program.
For example, suppose you save the following script as \li{cymodule.pyx}.
\begin{lstlisting}
import numpy as np
from scipy import linalg as la

def qr(A):
    norm = la.norm
    ncols = A.shape[1]
    Q = A.copy()
    R = np.zeros((ncols, ncols))
    for i in range(ncols):
        R[i, i] = norm(Q[:, i])
        Q[:, i] = Q[:, i]/R[i, i]
        for j in range(i+1, ncols):
            R[i, j] = Q[:, j].dot(Q[:, i])
            Q[:,j] = Q[:,j]-R[i, j]*Q[:,i]
    return Q, R
\end{lstlisting}
We can compile this code within IPython or from the command line.

To compile this code as Cython, we need to write another script, call it \li{setup.py}.
\lstinputlisting[style=fromfile, language=Python]{setup.py}
To run the setup script, type the following in the command line.
\footnote{Note that \li{python} in this line will need to refer to your version of Python that comes with Cython.
Often, you will need to replace \li{python} with a path pointing to a specific distribution of Python.}
\begin{lstlisting}
$ python setup.py build_ext --inplace
\end{lstlisting}
Now start up IPython in the same directory where you ran the setup script.
You can import \li{cymodule} just as if it were a Python module.
\begin{lstlisting}
In [1]: import cymodule
In [2]: cymodule.qr(A)
\end{lstlisting}
\end{comment}

\subsubsection*{Compilation in IPython}
Cython code can be compiled and imported to IPython (and IPython Notebooks) with the Cython magic function.
Load this function with the following command:
\footnote{In older versions of IPython and Cython, you may need to use the command \li{\%load_ext cythonmagic}.}
\begin{lstlisting}
In [1]: %load_ext Cython
\end{lstlisting}
Now you can define any Cython function in IPython by prefacing it with \li{\%\%cython}.
\begin{lstlisting}
In [2]: %%cython
   ...: import numpy as np
   ...: from scipy import linalg as la
   ...: def qr(A):
   ...:     norm = la.norm
   ...:     ncols = A.shape[1]
   ...:     Q = A.copy()
   ...:     R = np.zeros((ncols, ncols))
   ...:     for i in range(ncols):
   ...:         R[i, i] = norm(Q[:, i])
   ...:         Q[:, i] = Q[:, i]/R[i, i]
   ...:         for j in range(i+1, ncols):
   ...:             R[i, j] = Q[:, j].dot(Q[:, i])
   ...:             Q[:,j] = Q[:,j]-R[i, j]*Q[:,i]
   ...:     return Q, R
   ...:
\end{lstlisting}
Note the import statements on lines 2 and 3. Even if these modules were already loaded in the current IPython session, these lines are still be necessary. Now the \li{qr()} function has been compiled in Cython and can be called from IPython just as any other function.

It is best to use this method of compilation if you are only using your optimized function once or you are in the stages of developing your optimized algorithm. It can be a little frustrating to fix errors in your function if you are using the IPython shell. The IPython Notebook interface is much easier to use for these types of problems.

\subsubsection*{Compilation from the Command Line}
Compiling Cython from the command line is particularly useful if you need an optimized module that you will use over and over again. As we will discuss in this section, once a function is written and compiled in Cython, it can be imported as a module.

Cython code is usually written in a \li{.pyx} file, which is then compiled to C.
Next the C is compiled to a Python extension written in machine code.
Figure \ref{cython:compilation} shows how Cython code is compiled and called.

\begin{figure}
\centering
\includegraphics[width=.7\textwidth]{compilation.pdf}
\caption{A diagram showing how Cython code is compiled and called.
The path from calling a Cython function in a Python file to its evaluation is shown in red.}
\label{cython:compilation}
\end{figure}

These two compilations (first to C and then to a Python extension) are accomplished by the script \li{setup.py} discussed earlier in this lab.
\lstinputlisting[style=fromfile, language=Python]{setup.py}
The call to \li{cythonize()} compiles \li{cymodule.pyx} to C, and the call to \li{setup()} converts the C code to a Python extension.

\begin{info}
You can learn more about the Python/C balance in your Cython file with the following line.
\begin{lstlisting}
$ cython -a cymodule.pyx
\end{lstlisting}
This will generate a \li{html} file that you can open with your web browser.
It will show the lines that use Python in bright yellow and lines that use C in white.
\end{info}

\subsubsection*{Speeding up Cython with Type Declarations}
So far, our Cythonized \li{qr()} function does not run any faster than the Python version.\footnote{Sometimes you will see speedup immediately after Cythonizing a Python function.}
The simplest way to take advantage of the C-compilation is to declare data types, as you would in a C program.
All Cython data types and their NumPy equivalents are listed in Table \ref{table:cython_types}.

\begin{warn}
Unlike Python integers, which can be arbitrarily large, Cython integers can overflow.
\end{warn}

\begin{table}
\begin{center}
\begin{tabular}{|p{2.8cm}|p{2.5cm}|p{6cm}|}
\hline
Cython Type & NumPy Type& Description \\
\hline
float & float32 & 32 bit floating point number \\
double & float64 & 64 bit floating point number \\
float complex & complex64 & 64 bit floating point complex number \\
double complex & complex128 & 128 bit floating point complex number \\
char & int8 & 8 bit signed integer \\
unsigned char & uint8 & 8 bit unsigned integer \\
short & int16 & 16 bit signed integer \\
unsigned short & uint16 & 16 bit unsigned integer \\
int & int32 & 32 bit signed integer \\
unsigned int & uint32 & 32 bit unsigned integer \\
long & int32 or int64 & 32 or 64 bit signed integer (depends on platform) \\
unsigned long & uint32 or uint64 & 32 or 64 bit unsigned integer (depends on platform ) \\
long long & int64 & 64 bit signed integer \\
unsigned long long & uint64 & 64 bit unsigned integer \\
\hline
\end{tabular}
\end{center}
\caption{Numeric types available in Cython.}
\label{table:cython_types}
\end{table}

In our example, when we declare the index \li{j} to be an \li{int}, the inner for loop is pushed into C (instead of Python), speeding up our code immensely.
We do this by modifying \li{cymodule.pyx} as follows.
\begin{lstlisting}
...
def qr():
    cdef int j
    ncols = A.shape[1]
    Q = A.copy()
    ...
\end{lstlisting}

We could also declare \li{i} to be an int, but the effect of doing so is negligible.

After modifying \li{cymodule.pyx}, we must recompile it by running the script \li{setup.py}, and then re-import it into Python.
When we profile this function, we see that it is indeed a good deal faster than its Python cousin, and all the speed up is in the function itself (where the for loops are).

{\scriptsize
\begin{verbatim}
   ...
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.949    0.949    0.961    0.961 {cymodule.qr}
   ...
       \end{verbatim}
       }
Similarly, you can speed up function calls in Cython by declaring the types of some or all of the arguments.
\begin{lstlisting}
def myfunction(double arg1, int arg2, arg3):
    ...
\end{lstlisting}

\subsubsection*{Arrays in Cython}
You can also declare types for arrays.
Doing so produces a \emph{typed memoryview}, or Cython array.
As with ordinary variables, typed memoryviews can be initialized when they are declared or later.
\begin{lstlisting}
# Define a NumPy array
A = np.linspace(0,1,6)

# Create typed memoryviews on A and B
cdef double[:] cA = A
cdef int[:,:] cB

# Initialize cB
cB = np.array([[1,2],[3,4],[5,6]], dtype=np.dtype("i"))
\end{lstlisting}

Accessing single entries from a memoryview is faster than accessing entries of a NumPy array.
Also, passing slices of a memoryview to a Cython function is fast.
However, memoryviews do not support mathematical operations.
These must be performed on NumPy objects, or by looping through the arrays.

\begin{info}
Memoryviews can be passed as arguments to most NumPy functions.
NumPy includes functions for all common arithmetic operations.
For example, you can add typed memoryviews with \li{c = np.add(a, b)}.
This may or may not be faster than looping through the array.
\end{info}

\subsubsection*{Putting it All Together}
The following Python function computes the dot product of two 1-D arrays.
\begin{lstlisting}
def pydot(A, B):
    tot = 0.
    for i in xrange(A.shape[0]):
        tot += A[i] * B[i]
    return tot
\end{lstlisting}
A C equivalent can be compiled by running the following code in IPython:
\begin{lstlisting}
%%cython
def cydot(double[:] A, double[:] B):
    cdef double tot=0.
    cdef int i
    for i in xrange(A.shape[0]):
        tot += A[i] * B[i]
    return tot
\end{lstlisting}

Figure \ref{cython:dot} compares the speed of \li{pydot()}, \li{cydot()}, and the \li{dot()} method of a NumPy array.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{dot.pdf}
\caption{The running times of \li{pydot()}, \li{cydot()}, and the \li{dot()} method of a NumPy array on vectors of length ``Array Size.''
The Cython version runs almost as fast as the version built into NumPy.}
\label{cython:dot}
\end{figure}

%problems
\begin{problem} \label{prob:add}
\leavevmode
\begin{enumerate}
\item Write the following function in Python without using the \li{<<sum()>>} builtin function.
\begin{lstlisting}
def pysum(X):
    """ Return the sum of the elements of X.

    INPUTS:
    X - a 1-D NumPy array
    """
\end{lstlisting}

\item Rewrite the same function in Cython using a typed for-loop, and a typed memoryview of \li{X}. Call this function \li{cysum()}. Assume the input array \li{X} is an array of double precision floating point numbers.

\item Compare the \li{pysum()} function, the \li{cysum()}, the Python builtin \li{<<sum()>>} function, and the \li{<<np.sum()>>} function. Print the timing for each function to the terminal.
\end{enumerate}
\end{problem}

\begin{problem}
In the Affine Transformations Lab (Lab \ref{lab:ChangeBasis}) you wrote a function to compute the LU decomposition of a matrix. One possible implementation is found in the following code:
\begin{lstlisting}
def LU(A):
    """Returns the LU decomposition of a square matrix."""
    n = A.shape[0]
    U = np.array(np.copy(A), dtype=float)
    L = np.eye(n)

    for i in xrange(1,n):
        for j in xrange(i):
            L[i,j] = U[i,j]/U[j,j]
            U[i,j:] -= L[i,j]*U[j,j:]
    return L,U
\end{lstlisting}
\begin{enumerate}
\item Cython does not recognize NumPy array slicing. To be able to optimize this function using Cython, all array slicing must be removed. Rewrite the LU decomposition function so it performs every operation element-by-element instead of using NumPy array operations.
\item Port your function from Part 1 to Cython.
Use typed for-loops and typed memoryviews.
You may assume that you are only dealing with real arrays of double precision floating point numbers.
\item Compare the speeds of the original code, your solution to Part 1, and your solution to Part 2. Print the results to the terminal.
\end{enumerate}
\end{problem}

\subsection*{Numba}
As mentioned before, Numba relies on a just-in-time (JIT) compiler. This means that the code is compiled right before it is executed. We will discuss this process a bit later in this section.
The API for using Numba is incredibly simple. All one has to do is import Numba and add the \li{@jit} function decorator to your function. The following code would be a Numba equivalent to Problem \ref{prob:add}.
\begin{lstlisting}
from numba import jit
@jit
def numba_sum(A):
    total = 0
    for i in xrange(len(A)):
        total += A[i]
    return total
\end{lstlisting}

Though this code looks very simple, a lot is going on behind the scenes. As we saw with Cython, our code will run faster if datatypes are assigned to all our variables. With Cython, we had to explicitly definte the datatypes of each variable. Rather than requiring us to define the datatypes explicitly, Numba attempts to \emph{infer} the correct datatypes based on the datatypes of the input.

In the code above, for example, say that our array \li{A} was full of integers. Though we have not explicitly defined a datatype for the variable \li{total}, Numba will infer that the datatype for total should also be an integer.

Once all the datatypes have been inferred and assigned, the code is translated to machine code by the LLVM library. Numba will then cache this compiled version of our code. This means that we can bypass this whole inference and compilation process the next time we run our function.

\subsubsection*{More Control Within Numba}
Though the inference engine within Numba does a good job, it's not always perfect. If you would like to have more control, you may specify datatypes explicity as demonstrated in the code below.

Also, if you add the keyword argument, \li{nopython=True} to the \li{jit} decorator, an error will be raised if Numba was unable to convert everything to explicit datatypes.

In this example, we will assume that the input will be doubles. Note that is necessary to import the desired datatype from the Numba module.

\begin{lstlisting}
from numba import double
@jit(nopython=True, locals=dict(A=double[:], total=double))
def numba_sum(A):
    total = 0
    for i in xrange(len(A)):
        total += A[i]
    return total
\end{lstlisting}
Notice that the jit function decorator is the only thing that changed. Note also that this means that we will not be allowed to pass an array of integers to this function. If we had not specified datatypes, the inference engine would allow us to pass arrays of any numerical datatype. In the case that our function sees a datatype that it has not seen before, the inference and compilation process would have to be repeated. As before, the new version will also be cached.

If your function is running slower than you would expect, you can find out what is going on under the hood by calling the \li{inspect_types()} method of the function.

\begin{lstlisting}
# Due to the length of the output, we will leave it out of the lab text.
>>> numba_sum.inspect_types()
\end{lstlisting}

If you are trying to achieve the fastest runtimes possible, it will be worth trying both Numba and Cython. Since Numba relies on an LLVM compiler and Cython relies on a C compiler, it is possible that these two techniques will produce different results.

\begin{problem}
The code below defines a Python function which takes a matrix to the $n$th power.
\begin{lstlisting}
def pymatpow(X, power):
    """ Return X^{power}.

    Inputs:
        X (array) - A square 2-D NumPy array
        power (int) - The power to which we are taking the matrix X.
    Returns:
        prod (array) - X^{power}
    """
    prod = X.copy()
    temparr = np.empty_like(X[0])
    size = X.shape[0]
    for n in xrange(1, power):
        for i in xrange(size):
            for j in xrange(size):
                tot = 0.
                for k in xrange(size):
                    tot += prod[i,k] * X[k,j]
                temparr[j] = tot
            prod[i] = temparr
    return prod
\end{lstlisting}

\begin{enumerate}
\item Port \li{pymatpow()} to Cython using typed for-loops and typed memoryviews. Call your function \li{cymatpow()}.
\item Create a compiled version of \li{pymatpow()} using Numba. Call this function \li{numbamatpow()}.
\item Compare the speed of \li{pymatpow()}, \li{cymatpow()}, \li{numbamatpow()} and the \li{np.dot()} function. Remember to time \li{numbamatpow()} on the second pass so the compilation process is not part of your timing. Print your results to the terminal.
\end{enumerate}
NumPy takes products of matrices by calling BLAS and LAPACK, which are heavily optimized linear algebra libraries written in C, assembly, and Fortran.
\end{problem}

\subsection*{A Caution}
NumPy's array methods are often faster than a Cython or Numba equivalent you could code yourself.
If you are unsure which method is fastest, time them.

Moreover, a good algorithm written with a slow language (like Python) is faster than a bad algorithm written in a fast language (like C).
Hence, focus on writing fast algorithms with good Python code, and only use Cython  or Numba when and where it is necessary.

\subsection*{Use a More Efficient Algorithm}
The optimizations discussed thus far will speed up your code at most by a constant.
They will not change the complexity of your code.
In order to reduce the complexity (say from $O(n^2)$ to $O(n \log(n))$), you typically need to change your algorithm.
We will address the benefits of using more efficient algorithms in Problem \ref{prob:tridiag}.

\begin{comment}
\begin{problem}
Optimize the following function using techniques described in this lab:
\begin{lstlisting}
# TODO: COME UP WITH SOME ALGORITHM TO DO HERE!!
\end{lstlisting}
It should also include a list of changes, the reasoning behind the changes, and the effect of the changes on runtime. On the author's computer, computing the LU-decomposition on a 1000x1000 matrix took over 2 and a half minutes. The optimized version took a little over a second.

Hint: The best way to approach this problem is to analyze what each piece of code is actually doing. Then, determine if there is a more efficient way to accomplish the same task. Specifically, look for ways to use array operations instead of for loops, ways to replace blocks of code with built-in Python functions, and ways to avoid recomputing values.
\end{problem}
\end{comment}

The correct choice of algorithm is more important than a fast implementation.
For example, suppose you wish to solve the following tridiagonal system.
\[\begin{bmatrix}
b_1 & c_1 & 0 & 0 & \cdots & \cdots & 0 \\
a_2 & b_2 & c_2 & 0 & \cdots & \cdots & 0 \\
0 & a_3 & b_3 & c_3 & \cdots & \cdots & 0 \\
\vdots & \vdots & \vdots & \vdots & \ddots & \ddots & \vdots \\
\vdots & \vdots & \vdots & \vdots & \ddots & \ddots & c_{n-1} \\
0 & 0 & 0 & 0 & \cdots & a_n & b_n
\end{bmatrix}
\begin{bmatrix}
x_1\\
x_2\\
x_3\\
\vdots\\
\vdots\\
x_n
\end{bmatrix}
=
\begin{bmatrix}
d_1\\
d_2\\
d_3\\
\vdots\\
\vdots\\
d_n
\end{bmatrix}\]
One way to do this is with the general \li{solve} method in SciPy's \li{linalg} module.
Alternatively, you could use an algorithm optimized for tridiagonal matrices.
The code below implements one such algorithm in Python. This is called the Thomas algorithm.

\begin{lstlisting}
def pytridiag(a,b,c,d):
    """Solve the tridiagonal system Ax = d where A has diagonals a, b, and c.

    Inputs:
        a, b, c, d (array) - All 1-D NumPy arrays of equal length.
    Returns:
        x (array) - solution to the tridiagonal system.
    """
    n = len(a)

    # Make copies so the original arrays remain unchanged
    aa = np.copy(a)
    bb = np.copy(b)
    cc = np.copy(c)
    dd = np.copy(d)

    # Forward sweep
    for i in xrange(1, n):
        temp = aa[i]/bb[i-1]
        bb[i] = bb[i] - temp*cc[i-1]
        dd[i] = dd[i] - temp*dd[i-1]

    # Back substitution
    x = np.zeros_like(a)
    x[-1] = dd[-1]/bb[-1]
    for i in xrange(n-2, -1, -1):
        x[i] = (dd[i]-cc[i]*x[i+1])/bb[i]

    return x
\end{lstlisting}

\begin{problem} \label{prob:tridiag}
\leavevmode
\begin{enumerate}
\item Port the above code to Cython using typed for-loops, typed memoryviews and appropriate compiler directives.
\item Compare the speed of your new function with \li{pytridiag()} and \li{scipy.linalg.solve()}.
To compare the first two functions, start with $1000000 \times 1000000$ sized systems.
When testing the SciPy algorithm, start with $1000 \times 1000$ systems. You may use the code below to generate the arrays, \li{a}, \li{b}, and \li{c}, along with the corresponding tridiagonal matrix \li{A}.
\begin{lstlisting}
def init_tridiag(n):
    """Initializes a random nxn tridiagonal matrix A.

    Inputs:
        n (int) : size of array

    Returns:
        a (1-D array) : (-1)-th diagonal of A
        b (1-D array) : main diagonal of A
        c (1-D array) : (1)-th diagonal of A
        A (2-D array) : nxn tridiagonal matrix defined by a,b,c.
    """
    a = np.random.random_integers(-9,9,n).astype("float")
    b = np.random.random_integers(-9,9,n).astype("float")
    c = np.random.random_integers(-9,9,n).astype("float")

    A = np.zeros((b.size,b.size))
    np.fill_diagonal(A,b)
    np.fill_diagonal(A[1:,:-1],a[1:])
    np.fill_diagonal(A[:-1,1:],c)
    return a,b,c,A
\end{lstlisting}
\item What do you learn about good implementation versus proper choice of algorithm?
\end{enumerate}
Note that an efficient tridiagonal matrix solver is implemented by \li{scipy.sparse.linalg.spsolve()}.
\end{problem}

\section*{When to Stop Optimizing}
You don't need to apply every possible optimization to your code.
When your code runs acceptably fast, stop optimizing. There is no need spending valuable time making optimizations once the speed is sufficient.

Moreover, remember not to prematurely optimize your functions. Make sure the function does exactly what you want it to before worrying about any kind of optimization.

\section*{More on Cython (Optional)}
This section has a more complete introduction to Cython.
For another reference, see \url{http://docs.cython.org/src/tutorial/cython_tutorial.html}.

\subsection*{More on Type Declarations}
Type declarations make Cython faster than Python because a computer can process numbers faster when it knows ahead of time what type they are.
Declaring a variable type in Cython makes the variable a native machine type instead of a Python object.

You can initialize a variable when it is declared or afterwards.
It is possible to initialize several variables at once, as demonstrated below.

\begin{lstlisting}
# Declare integer variables i, j, and k.
# Set k equal to 2.
cdef int i, j, k=2

cdef:
    # Declare and initialize m equal to 4 and n equal to 5.
    int m=4, n=5
    # Declare and initialize e equal to 2.71.
    double e = 2.71
    # Declare a double precision complex number a.
    double complex a
\end{lstlisting}

\subsection*{Compiler Directives}
When you access elements of an array, Cython checks that the indices are within bounds.
Cython also allows negative indexing the same way Python does.
These features slow down code execution.
After a program has been carefully debugged, they may be removed via \emph{compiler directives}.

Compiler directives in Cython can be included as comments or function decorators.
Directives included in comments will apply to the whole file, while function decorators will only apply to the function immediately following.
The comments to turn off bounds checking and negative indices are
\begin{lstlisting}
# cython: boundscheck=False
# cython: wraparound=False
\end{lstlisting}
To use the function decorators, first import the \li{cython} module by including the line \li{cimport cython} in your import statements.
The decorators are
\begin{lstlisting}
cimport cython
@cython.boundscheck(False)
@cython.wraparound(False)
\end{lstlisting}

Cython has many other compiler directives, including \li{cdivision}.
When \li{cdivision} is set to \li{True}, the \li{\%} operator returns a number with the sign of the first argument (like in C).
Also, division by 0 will no longer raise a \li{ZeroDivisionError}, which will increase the speed of your program.

\subsection*{More on Functions in Cython}
Speed up function calls in Cython by declaring the types of some or all of the arguments.
\begin{lstlisting}
def myfunction(double[:] X, int n, double h, items):
    ...
\end{lstlisting}
If we pass in a NumPy array for the argument \li{X}, Cython will convert it to a typed memoryview before it enters the function.
However, if we pass in a NumPy array for \li{items}, it will remain a NumPy array in the function.
Both typed and untyped arguments can be made into keyword arguments.

Cython also allows you to make C functions that are only callable within the C extension you are currently building.
They are not ported into the Python namespace.
These functions are declared using the same syntax as in Python, except the keyword \li{def} is replaced with \li{cdef}.
The keyword \li{cpdef} combines \li{def} and \li{cdef} by creating two versions of the function: one for Python and one for C.

You can also specify the return type for functions declared using \li{cdef} and \li{cpdef}, as in the code below.
\begin{lstlisting}
cpdef int myfunction(double[:] X, int n, double h, items):
    ...
\end{lstlisting}

\subsection*{More Examples}
For another example, we will write functions which compute $AA^T$ from $A$.
That is, given \li{A}, these functions compute \li{B} where
\li{B[i,j] = dot(A[i],A[j])}.

Here is the Python solution.
\begin{lstlisting}
def pyrowdot(A):
    B = np.empty((A.shape[0], A.shape[0]))
    for i in xrange(A.shape[0]):
        for j in xrange(i):
            B[i,j] = pydot(A[i], A[j])
        B[i,i] = pydot(A[i], A[i])
    for i in xrange(A.shape[0]):
        for j in xrange(i+1, A.shape[0]):
            B[i,j] = B[j,i]
    return B
\end{lstlisting}

Here is the Cython solution.
We changed the function \li{cydot()} to a C function since it will only be called by \li{cyrowdot()}.
Also, the Cython solution uses a typed memoryview of its input.

\begin{lstlisting}
import numpy as np
cimport cython
# cython: boundscheck=False
# cython: wraparound=False

cpdef double cydot(double[:] A, double[:] B):
    cdef double tot=0.
    cdef int i, n=A.shape[0]
    for i in xrange(n):
        tot += A[i] * B[i]
    return tot

def cyrowdot(double[:,:] A):
    cdef double[:,:] B = np.empty((A.shape[0], A.shape[0]))
    cdef int i, j, n=A.shape[0]
    for i in xrange(n):
        for j in xrange(i):
           B[i,j] = cydot(A[i], A[j])
        B[i,i] = cydot(A[i], A[i])
    for i in xrange(n):
        for j in xrange(i+1, n):
            B[i,j] = B[j,i]
    return np.array(B)
\end{lstlisting}

This can also be done in NumPy by running \li{A.dot(A.T)}.
The timings of \li{pyrowdot()}, \li{cyrowdot()}, and the NumPy command \li{A.dot(A.T)} are shown in Figure \ref{cython:rowdot}.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{rowdot.pdf}
\caption{The timings of \li{pyrowdot()}, \li{cyrowdot()}, and the NumPy command \li{A.dot(A.T)}.
The arrays used for testing were $n\times 3$ where $n$ is shown along the horizontal axis.}
\label{cython:rowdot}
\end{figure}

In both of these examples, NumPy's implementation was faster than the version we wrote in Cython.


\section{Optimization of Surrogates on Sparse Grids}
\label{sec:52method}

\minitoc{72mm}{5}

\noindent
The methods presented in the last section can be combined to a
``meta-method'' for surrogate optimization.
The surrogates are constructed as interpolants on spatially adaptive
sparse grids, which we explain in the following.

\subsection{Novak--Ritter Adaptivity Criterion}
\label{sec:521novakRitter}

The classic surplus-based refinement strategy for
spatially adaptive sparse grids is not tailored to optimization,
as this refinement strategy aims to minimize the overall $\Ltwo$ error.
However, in optimization, it is reasonable to generate more
points in regions where we suspect the global minimum to be
to increase the interpolant's accuracy in these regions.
Hence, we employ an adaptivity criterion proposed by
Novak and Ritter \cite{Novak96Global} for hyperbolic cross points.
The Novak--Ritter criterion has also been applied to sparse grids
\multicite{Ferenczi05Globale,Valentin14Hierarchische,Valentin16Hierarchical}.

\paragraph{$m$-th order children}

As usual, the criterion works iteratively:
Starting with an initial regular sparse grid of a very coarse level,
the criterion selects a specific point $\gp{\*l,\*i}$ in each iteration
and inserts all its children into the grid.
This process is repeated until a given number $\ngpMax$ of grid points is
reached,
since we evaluate $\objfun$ at every grid point once, and we assume that
function evaluations dominate the overall complexity.
The difference to common refinement criteria is that
a point may be selected multiple times, in which case
\term{higher-order children} are inserted.
The $m$-th order children $\gp{\*l',\*i'}$ of a grid point $\gp{\*l,\*i}$
satisfy
\begin{equation}
  \label{eq:indirectChild}
  \*l'_{-t} = \*l^{}_{-t},\;\,
  \*i'_{-t} = \*i^{}_{-t},\;\,
  l'_t = l^{}_t + m,\;\,
  i'_t \in
  \begin{cases}
    \{1\},&(l_t = 0) \land (i_t = 0),\\
    \{2^m - 1\},&(l_t = 0) \land (i_t = 1),\\
    \{2^m i_t - 1,\, 2^m i_t + 1\},&\hphantom{(}l_t > 0,
  \end{cases}
\end{equation}
where $m \in \nat$ and $t \in \{1, \dotsc, d\}$
(cf.\ \cref{eq:directAncestor} for $m = 1$).
The order is chosen individually for each child point to be inserted
as the lowest number $m$ such that $\gp{\*l',\*i'}$ does not yet exist
in the grid.

%\todo{add figure for m-th order children?}

\paragraph{Criterion}

The Novak--Ritter refinement criterion \cite{Novak96Global}
refines the grid point $\gp{\*l,\*i}$ that minimizes the product%
\footnote{%
  Compared to \cite{Novak96Global},
  we added one in the base of each factor to avoid ambiguities
  for $0^0$.
  In addition, we swapped $\gamma$ with $1-\gamma$
  to make $\gamma$ more consistent with its name as adaptivity.%
}
\begin{equation}
  (r_{\*l,\*i} + 1)^\gamma \cdot
  (\normone{\*l} + d_{\*l,\*i} + 1)^{1 - \gamma}.
\end{equation}
Here, $r_{\*l,\*i} \ceq \setsize{
  \{(\*l',\*i') \in \liset \mid
  \objfun(\gp{\*l',\*i'}) \le \objfun(\gp{\*l,\*i})\}
} \in \{1, \dotsc, \setsize{\liset}\}$ is the \term{rank} of $\gp{\*l,\*i}$
(where $\liset$ is the current set of level-index pairs of the grid), i.e.,
the place of the function value at $\gp{\*l,\*i}$
in the ascending order of the function values at all points
of the current grid.
We denote the \term{degree} $d_{\*l,\*i} \in \natz$ of $\gp{\*l,\*i}$
as the number of previous refinements of this point.
Finally, $\gamma \in \clint{0, 1}$ is the \term{adaptivity parameter.}
%By choosing $\gamma = 0$, the function values become irrelevant and the
%criterion produces regular-like sparse grids.
%If we choose $\gamma = 1$, then the criterion always refines the point with
%the lowest function value, which means that the criterion is easily stuck
%in local minima.
We have to choose a suitable compromise between exploration ($\gamma = 0$)
and exploitation ($\gamma = 1$).
The best choice of course depends on the objective function $\objfun$ at hand,
but for our purposes, we choose a priori a value of $\gamma = 0.15$.
However, it may be an option to adapt the value of $\gamma$ automatically
during the grid generation phase.

\subsection{Global Optimization of Sparse Grid Surrogates}
\label{sec:522method}

\paragraph{Global, local, and globalized optimization methods}

In \cref{sec:51algorithms}, we presented various optimization methods
for the unconstrained case,
divided into global gradient-free methods such as differential evolution and
local gradient-based methods, for example, gradient descent.
A subset of these methods has been implemented in \sgpp{}
\cite{Pflueger10Spatially}, see \cref{tbl:optimizationMethod}.
The gradient-based methods need an initial point, and
they may get stuck in local minima.
Hence, we additionally implemented globalized versions
of the gradient-based methods
via a multi-start Monte Carlo approach with $m \ceq \min(10d, 100)$
uniformly distributed pseudo-random initial points.%
\footnote{%
  We split the number of permitted function evaluations evenly
  among the $m$ parallel calls.%
  %of the gradient-based method%
}
This means there are three types of methods:

\begin{enumerate}[label=T\arabic*.,ref=T\arabic*,leftmargin=2.7em]
  \item
  \label{item:globalMethods}
  Global gradient-free methods listed as implemented in
  \cref{tbl:optimizationMethod}

  \item
  \label{item:localMethods}
  Local gradient-based methods listed as implemented in
  \cref{tbl:optimizationMethod}%
  %(need an initial point)%
  \footnote{%
     Excluding Levenberg--Marquardt, which is only applicable
     to least-squares problems.%
  }

  \item
  \label{item:globalizedMethods}
  Globalized versions of the methods of type \ref{item:localMethods}
  %(do not need an initial point)
\end{enumerate}

\paragraph{Unconstrained optimization of sparse grid surrogates}

Given the objective function $\objfun\colon \clint{\*0, \*1} \to \real$,
the maximal number $\ngpMax \in \nat$ of evaluations of $f$, and
the adaptivity parameter $\gamma \in \clint{0, 1}$,
we determine an approximation $\xoptappr \in \clint{\*0, \*1}$
of the global minimum $\xopt$ of $\objfun$ as follows:

\begin{enumerate}
  \item
  Generate a spatially adaptive sparse grid $\sgset$
  with the Novak--Ritter refinement criterion
  for $\objfun$, $\ngpMax$, and $\gamma$.

  \item
  Determine the sparse grid interpolant $\sgintp$ of $\objfun$
  by solving the linear system \eqref{eq:hierarchizationProblem}.

  \item
  Optimize the interpolant:
  First, find the best grid point
  $\*x^{(0)} \ceq \vecargmin_{\gp{\*l,\*i} \in \sgset} \objfun(\*x_{\*l,\*i})$.
  Second, apply the local methods of type \ref{item:localMethods}
  to the interpolant $\sgintp$ with $\*x^{(0)}$ as initial point.
  Let $\*x^{(1)}$ be the resulting point with minimal objective function value.
  Third, we apply the global and globalized methods
  of types \ref{item:globalMethods} and \ref{item:globalizedMethods}
  to the interpolant $\sgintp$.
  Again, let $\*x^{(2)}$ be the point with
  minimal $\objfun$ value.
  Finally, determine the point of $\{\*x^{(0)}, \*x^{(1)}, \*x^{(2)}\}$
  with minimal $\objfun$ value and return it as $\xoptappr$.
\end{enumerate}

\noindent
Note that the third step requires a fixed number of additional
evaluations of the objective function,
which can be neglected compared to $\ngpMax$.
By default, we use the cubic modified hierarchical not-a-knot B-spline basis
$\bspl[\nak,\modified]{\*l,\*i}{p}$ ($p = 3$)
for the construction of the sparse grid surrogate.
However, we could apply any of the hierarchical (B-)spline bases presented in
\cref{chap:30BSplines,chap:40algorithms}.

\paragraph{Comparison methods}

We use two comparison methods.
First, we apply the gradient-free methods
(type \ref{item:globalMethods}) to the sparse grid interpolant
using modified piecewise linear hierarchical basis functions
(i.e., $p = 1$) on the same sparse grid as the cubic B-splines.
We cannot employ gradient-based optimization as the objective function
should be continuously differentiable and
discontinuous derivatives are usually numerically problematic
for gradient-based optimization methods
(see, e.g., \cite{Huebner14Mehrdimensionale}).
Second, we apply the gradient-free methods
(type \ref{item:globalMethods}) directly to the objective function.
We cannot use the gradient-based methods here as the gradient of the
objective function is assumed to be unknown.
For both of the comparison methods,
we make sure that the objective function is evaluated at most $\ngpMax$ times
by splitting the $\ngpMax$ evaluations
evenly among all employed optimization methods.

\paragraph{Constrained optimization}

For optimization problems with constraints,
we proceed exactly as for unconstrained optimization,
except that for optimizing the interpolant, we use the
constrained optimization algorithms implemented in \sgpp
as listed in \cref{tbl:optimizationMethod}.
We only replace the objective function $\objfun$ with a sparse grid
surrogate $\sgintp$, and we assume that the constraint function
$\ineqconfun$ can be evaluated fast.
However, it would also be possible to replace $\ineqconfun$
with a sparse grid surrogate.
In this case, it cannot be guaranteed that the resulting optimal point
$\xoptappr$ is feasible, i.e., we could have
$\lnot(\ineqconfun(\xoptappr) \le \*0)$.


\documentclass[12pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amsthm,amssymb,amsfonts}
\usepackage{listings}

\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}

\newenvironment{problem}[2][Problem]{\begin{trivlist}
\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
%If you want to title your bold things something different just make another thing exactly like this but replace "problem" with the name of the thing you want, like theorem or lemma or whatever

\begin{document}

%\renewcommand{\qedsymbol}{\filledbox}
%Good resources for looking up how to do stuff:
%Binary operators: http://www.access2science.com/latex/Binary.html
%General help: http://en.wikibooks.org/wiki/LaTeX/Mathematics
%Or just google stuff

\title{Assignment 2\\ MEEN 357}
\author{Jacob Hartzer}
\maketitle

\section*{Task 7}
\begin{problem}{i} Taylor series of $tan^{-1}(x)$
\begin{center}
$f(x) = tan^{-1}(x)$\\
$f(tan\theta{}) = \theta{}$\\$
f'(tan\theta{})sec^2\theta{}=1$\\$
sec^2\theta{} =tan^2\theta+1$\\$
x = tan\theta{} \,\to\, sec^2 \theta{}=1+x^2$\\$
f'(x)=1/(1+x^2 )$\\$
f^{(1)} (c)=\frac{1}{1+x^2}$\\$
f^{(2)} (c)=\frac{-2x}{1+x^2)^2}$\\$
f^{(3)} (c)=\frac{8x^2}{(1+x^2)^3} \frac{-2}{(1+x^2)^2} $\\$
f^{(4)} (c)=\frac{-48x^3}{(1+x^2)^4}+\frac{24x}{(1+x^2)^3} $\\$
f^{(5)} (c)=\frac{384x^4}{1+x^2)^5}-\frac{288x^2}{1+x^2 )^4}+\frac{24}{(1+x^2)^3} $\\$
f^{(6)} (c)=\frac{-3840x^5}{1+x^2)^6} + \frac{3840x^3}{(1+x^2)^5} -\frac{720x}{(1+x^2)^4} $
\end{center}
Continuing, one finds that
\begin{center}
$tan^{-1} (x)=f^{(0)} (c)+\frac{f^{(1)} (c)}{1!}x^1+\frac{f^{(2)} (c)}{2!}x^2+\dots{}+\frac{f^{(n)} (c)}{n!} x^n$\\
$=x-\frac{x^3}{3}+\frac{x^5}{3}-\frac{x^7}{7'}+\frac{x^9}{9}-\dots{}$\\
$=\sum^{\inf}_{n=1}-1^{n-1}  \frac{x^{2n-1}}{2n-1}$
\end{center}
\end{problem}

\begin{problem}{ii} Find $R_5$
\begin{center}
$R_5 \leq \frac{Max|f^{5+`}(c)|}{(5+1)!}x^{5-1}=Max\Big[\frac{-16c^5}{3(1+x^2 )^6}+\frac{16c^3}{3(1+x^2)^5}-\frac{c}{1+x^2 )^4}  \Big]\big( \frac{\pi}{6} \big)^4$\\
$c = 0.2274$\\
$R_5 = 0.0105$
\end{center}
\end{problem}

\begin{problem}{iii} Find the number of terms requried to limit the truncation error.
\begin{lstlisting}

clc; clear all; close all;

syms x;
fun(x) = atan(x);

n = 1;
while true
    fun_diff = diff(fun,n+1);

    remainder = max(abs(double(fun_diff(linspace(0,pi/4,500)))))/...
        factorial(n+1)*power((pi/6),n-1);

    if remainder < power(10,-5) || n >= 50
        break
    end

    n = n + 1 ;
end
disp(n); disp(remainder);

\end{lstlisting}
$n=15$\\
$remainder = 6.7654e-06$
\end{problem}

\end{document}


    \filetitle{min}{Define loss function for optimal policy}{modellang/min}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
min(Disc) Expr;
\end{verbatim}

\paragraph{Syntax for exact non-linear
simulations}\label{syntax-for-exact-non-linear-simulations}

\begin{verbatim}
min#(Disc) Expr;
\end{verbatim}

\paragraph{Description}\label{description}

The loss function must be types as one of the transition equations. The
\texttt{Disc} is a parameter or an expression defining the discount
factor (applied to future dates), and the expression \texttt{Expr}
defines the loss fuction. The \texttt{Disc} expression must not contain
a comma.

If you use the \texttt{min\#(Disc)} syntax, all equations created by
differentiating the lagrangian w.r.t. individual variables will be
earmarked for exact nonlinear simulations provided the respective
derivative is nonzero. This only makes sense if the loss function is
other than quadratic, and hence its derivatives are nonlinear.

There are two types of optimal policy that can be calculated:
time-consistent discretionary policy, and time-inconsistent optimal
policy with commitment. Use the option \texttt{'optimal='} in the
function \href{model/model}{\texttt{model}} at the time of loading the
model file to switch between these two types of policy; the option can
be either \texttt{'discretion'} (default) or \texttt{'commitment'}.

\paragraph{Example}\label{example}

This is a simple model file with a Phillips curve and a quadratic loss
function.

\begin{verbatim}
!transition_variables
    x, pi

!transition_shocks
    u

!parameters
    alpha, beta, gamma

!transition_equations
    min(beta) pi^2 + lambda*x^2;
    pi = alpha*pi{-1} + (1-alpha)*pi{1} + gamma*y + u;
\end{verbatim}


\documentclass[letterpaper]{article}

\usepackage{fullpage}
\usepackage{nopageno}
\usepackage{amsmath}
\usepackage{amssymb}
\allowdisplaybreaks

\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}

\begin{document}
\title{Notes}
\date{August 27, 2014}
\maketitle
chapters 1-5 in 420, chapter 6 in 421.
chapter 1 should be review.
one of the hardest, if not the hardest course. much new material.
not necessary to solve all problems, but try and be able to discuss.
will need to present solutions at least 3 times.
assignments to be turned in will probably be selected from the group of problems that were assigned and discussed.

\section*{notation review}
integers: $\mathbb{Z}=\{...,-3,-2,-1,0,1,2,3,...\}$

natural numbers:$\mathbb{N}=\{0,1,2,3,...\}$ note this includes 0

divisibility $a,b\in\mathbb{Z}, b \vert a$ (b divides a) means $a=bc$ for some integer $c$

\subsection*{division algorithm}
given two integers $a,b\in\mathbb{Z}$ where $b>0$ there exists unique integers $q$ and $r$ such that $a=bq+r$ with $0\le r<b$

note that this is trivial without constraints on $r$ because one can set $q=0$ and $r=b$
\subsubsection*{proof}
take $\mathcal{C}=\{a-bq\mid q\in\mathbb{Z}\}$

$\mathcal{C}^+=\mathcal{C}\cap\mathbb{N}$
\subsubsection*{claim}
$\mathcal{C}^+$ is non-empty. why?

if $a\ge 0,$ then $a-b\cdot0\in\mathcal{C}^+$

if $a<0,$ then $a-b\cdot a=a(1-b)$. $b>0$ so $1-b\le0$ so $a-ba\ge0$

now take the \emph{well ordering principle} (you can always find a smallest or least element of a subset of natural numbers)

let $r$ be the smallest element of $\mathcal{C}^+$
\[r=a-bq\]
claim is that $r<b$
\subsubsection*{contradiction}
assume $r\ge b$. then $r-b\ge0$
\begin{align*}
  r-b&=(a-bq)-b\\
  &=a-b(q+1)\in\mathcal{C}
\end{align*}
this means $r-b\in\mathcal{C}^+$ which is a contradiction because it is smaller than $r$ which is supposed to be the smalled element in $\mathcal{C}^+$

we know that $r<b$ so there exists $q,r$ with $a=bg+r$, $0\le r<b$.

to prove uniqueness, assume $a=bq_1+r_1$ with $0\le r_1<b$ and $a=bq_2+r_2$ with $0\le r_2<b$

prove 1 and 2 are the same, prove q get remainder for free.

subtract the two quantities
\begin{align*}
  0&=b(q_1-q_2)+(r_1-r_2)\\
\end{align*}
lets talk about $r_1$ and $r_2$. distance between $r_1$ and $r_2$ is at most b. draw it on a number line with r's between 0 and b to be convinced
\begin{align*}
  r_2-r_1&=b(q_1-q_2)\\
  \left\lvert r_2-r_1\right\rvert&=b\left\lvert q_1-q_2\right\rvert\\
\end{align*}
since $\left\lvert r_1-r_2\right\rvert<b$ the difference is zero and $r_1=r_2$

\section*{gcd}
greatest common divisor
\subsection*{definition}
let $a,b\in\mathbb{Z}$ not both 0. we say that the positive integer $d$ is the greatest common divisor of $a$ and $b$ if
\begin{enumerate}
\item
$d\vert a$ and $d\vert b$ (always at least have one)
\item
any other common divisor of $a$ and $b$ is also a divisor of $d$

in other words, if $c\vert a$ and $c\vert b$ then $c\vert d$
\end{enumerate}
uniqueness is implied by saying \emph{the} greatest common divisor
\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% UMB-CS114-2015F: Introduction to Programming in Java
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/UMB-CS114-2015F
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\def \topDirectory {.}
\def \texDirectory {\topDirectory/src/main/tex}

\documentclass[12pt,letterpaper,twoside]{article}
\usepackage{\texDirectory/template/style/directives}
\usepackage{\texDirectory/template/style/assignment}
\input{\texDirectory/template/config}

\usepackage{amsmath}

\begin{document}

\doc{title}{Solution to Assignment 3}
\doc{date-pub}{Oct 19, 2015 at 5:30 PM}
\doc{date-due}{Nov 04, 2015 at 5:30 PM}
\doc{points}{8}

\prepare{header}

\section*{Question 1}

Write a program \texttt{HouseOfCards.java} that randomly selects two cards from a standard 52-card deck.
The program would then print the names of the cards based on their rank and suit.
Finally, your program should determine if the two cards share the same rank or the same suit.
Your program is expected to function as shown in following examples:

\begin{terminal}
$ java HouseOfCards
Queen of Diamonds
Jack of Spades
Cards do not share rank or suit.
$ java HouseOfCards
Ace of Hearts
8 of Hearts
Cards share the same suit.
\end{terminal}

\subsection*{Solution}

\lstset{language=Java,tabsize=4}
\begin{lstlisting}
public class HouseOfCards {
	public static void main(String[] args) {
		int[] card1 = generateCard();
		int[] card2 = generateCard();
		printCardName(card1[0], card1[1]);
		printCardName(card2[0], card2[1]);
		if (card1.equals(card2))
			System.out.println("Cards are the same");
		else if (card1[0] == card2[0])
			System.out.println("Cards share the same suit.");
		else if (card1[1] == card2[1])
			System.out.println("Cards share the same rank.");
		else
			System.out.println("Cards do not share rank or suit.");
	}
	public static int[] generateCard() {
		int[] card = new int[2];
		int num = (int) Math.floor(Math.random() * 52);
		card[0] = num / 13;
		card[1] = num % 13;
		return card;
	}
	public static void printCardName(int suitNum, int rankNum) {
		String[] suitNames = {
			"Spades", "Diamonds", "Clubs", "Hearts"
		};
		String[] cardNames = {
			"Ace", "Two", "Three", "Four", "Five",
			"Six", "Seven", "Eight", "Nine", "Ten",
			"Soldier", "Queen", "King"
		};
		System.out.printf("%s of %s\n", cardNames[rankNum], suitNames[suitNum]);
	}
}
\end{lstlisting}

\section*{Question 2}

Write a program \texttt{Deviation.java} that prompts user to enter a positive integer number $n$ and then asks user to enter $n$ floating-point numbers.
Your program should compute and display standard deviation of the numbers according to Equation \ref{eq1}, in the format shown below.

\begin{equation}
\text{deviation} = \sqrt{\frac{\sum_{i = 1}^{n} \left( x_i - \frac{\sum_{i = 1}^{n} x_i}{n} \right)^2 }{n - 1}}
\label{eq1}
\end{equation}

\begin{terminal}
$ javac Deviation.java
$ java Deviation
Enter count: 10
Enter 10 numbers: 1.9 2.5 3.7 2 1 6 3 4 5 2
Standard Deviation: 1.557
\end{terminal}

\newpage

\subsection*{Solution}

\lstset{language=Java,tabsize=4}
\begin{lstlisting}
import java.util.Scanner;
public class Deviation {
	public static void main(String[] args) {
		double[] numbers = getNumbers();
		double deviation = findDeviation(numbers);
		System.out.printf("Standard Deviation: %.3f\n", deviation);
	}
	public static double[] getNumbers() {
		Scanner input = new Scanner(System.in);
		System.out.printf("Enter count: ");
		int num = input.nextInt();
		double[] numbers = new double[num];
		System.out.printf("Enter %d Numbers: ", num);
		for (int i = 0; i < num; i++) {
			numbers[i] = input.nextDouble();
		}
		input.close();
		return numbers;
	}
	public static double findDeviation(double[] numbers) {
		double mean = findMean(numbers);
		double sum = 0;
		for (int i = 0; i < numbers.length; i++)
			sum += Math.pow(numbers[i] - mean, 2);
		double deviation = Math.sqrt(sum / (numbers.length - 1));
		return deviation;
	}
	public static double findMean(double[] numbers) {
		double sum = 0;
		for (int i = 0; i < numbers.length; i++)
			sum += numbers[i];
		double mean = sum / numbers.length;
		return mean;
	}
}
\end{lstlisting}

\newpage

\section*{Question 3}

Write a program \texttt{DuplicateTester.java} that prompts user for a positive integer $x$ and creates an array with $x$ randomly generted integer elements in range 0 to 9.
Use a method with signature \texttt{public static int[] removeDuplicates(int[] array);} to remove all duplicate elements in the array.
Your program is expected to terminate by displaying the new array.
Following is a sample expected run of the program.

\begin{terminal}
$ javac DuplicateTester.java
$ java DuplicateTester
Size of Array: 10
Generated Array:
3 4 2 8 2 4 3 1 6 7
Filtered Array:
3 4 2 8 1 6 7
\end{terminal}

\subsection*{Solution}

\lstset{language=Java,tabsize=4}
\begin{lstlisting}
import java.util.Scanner;
public class DuplicateTester {
	public static void main(String[] args) {
		int size = getSize();
		System.out.println("Generated Array:");
		int[] array = generateArray(size);
		display(array);
		System.out.println("Filtered Array:");
		int[] newArray = removeDuplicates(array);
		display(newArray);
	}
	public static int getSize() {
		Scanner input = new Scanner(System.in);
		System.out.print("Size of Array: ");
		int size = input.nextInt();
		input.close();
		return size;
	}
	public static int[] generateArray(int size) {
		int[] array = new int[size];
		for (int i = 0; i < size; i++)
			array[i] = (int) Math.floor(Math.random() * 10);
		return array;
	}
	public static int[] removeDuplicates(int[] array) {
		boolean[] flags = new boolean[10];
		int[] numbers = new int[10];
		for (int i = 0; i < flags.length; i++)
			flags[i] = false;
		int j = 0;
		for (int i = 0; i < array.length; i++) {
			if (flags[array[i]] == false) {
				flags[array[i]] = true;
				numbers[j++] = array[i];
			}
		}
		int[] newArray = new int[j];
		for (int i = 0; i < j; i++)
			newArray[i] = numbers[i];
		return newArray;
	}
	public static void display(int[] array) {
		int size = array.length;
		for (int i = 0; i < size; i++)
			System.out.printf("%d ", array[i]);
		System.out.println();
	}
}
\end{lstlisting}

\section*{Question 4}

Thomas, a CS114 student, had been asked to write a program that performs simple arithmetic operations (add and multiply) on two matrices $x$ and $y$ whose elements are randomly generated from range 0 to 9.
The program was supposed to prompt user for a number $r$ and generate two $r \times r$ matrices, $x$ and $y$.
The program would then display $x + y$ and $x \times y$ respectively, as shown in Figure 1.

\newpage

\renewcommand{\lstlistingname}{Figure}
\lstset{caption=Expected Output of \texttt{MatrixWorld.java}, label=terminal1}
\begin{terminal}
Size of Matrices: 3

Matrix A:
  0   3   2
  7   0   1
  1   6   5

Matrix B:
  7   8   9
  0   3   1
  8   9   6

Sum:
  7  11  11
  7   3   2
  9  15  11

Multiplication:
 16  27  15
 57  65  69
 47  71  45
\end{terminal}

Unfortunately, Thomas gave up while developing the requested program.
He has asked that you continue development of his program without modifying the code he has written.

\lstset{caption=}
\lstset{language=Java,tabsize=4}
\begin{lstlisting}
// written by: Thomas A. Anderson
// <thomas.anderson001@umb.edu>

import java.util.Scanner;
public class MatrixWorld {
	public static void main(String[] args) {
		int size = getSize();
		int[][] matrixA = matrixInit(size);
		int[][] matrixB = matrixInit(size);
		int[][] sum = matrixAdd(matrixA, matrixB);
		int[][] multiplication = matrixMultiply(matrixA, matrixB);
		matrixDisplay(matrixA, "Matrix A");
		matrixDisplay(matrixB, "Matrix B");
		matrixDisplay(sum, "Sum");
		matrixDisplay(multiplication, "Multiplication");
	}
	public static int getSize() {
		// To Be Developed
	}
	public static int[][] matrixInit(int size) {
		// To Be Developed
	}
	public static int[][] matrixAdd(int[][] matrixA, int[][] matrixB) {
		// To Be Developed
	}
	public static int[][] matrixMultiply(int[][] matrixA, int[][] matrixB) {
		// To Be Developed
	}
	public static void matrixDisplay(int[][] matrix, String name) {
		// To Be Developed
	}
}
\end{lstlisting}

\subsection*{Solution}

\lstset{language=Java,tabsize=4}
\begin{lstlisting}
// written by: Thomas A. Anderson
// <thomas.anderson001@umb.edu>

import java.util.Scanner;
public class MatrixWorld {
	public static void main(String[] args) {
		int size = getSize();
		int[][] matrixA = matrixInit(size);
		int[][] matrixB = matrixInit(size);
		int[][] sum = matrixAdd(matrixA, matrixB);
		int[][] multiplication = matrixMultiply(matrixA, matrixB);
		matrixDisplay(matrixA, "Matrix A");
		matrixDisplay(matrixB, "Matrix B");
		matrixDisplay(sum, "Sum");
		matrixDisplay(multiplication, "Multiplication");
	}
	public static int getSize() {
		Scanner input = new Scanner(System.in);
		System.out.print("Size of Matrices: ");
		int size = input.nextInt();
		input.close();
		return size;
	}
	public static int[][] matrixInit(int size) {
		int[][] matrix = new int[size][size];
		for (int i = 0; i < size; i++)
			for (int j = 0; j < size; j++)
				matrix[i][j] = (int) Math.floor(Math.random() * 10);
		return matrix;
	}
	public static int[][] matrixAdd(int[][] matrixA, int[][] matrixB) {
		int size = matrixA.length;
		int[][] sum = new int[size][size];
		for (int i = 0; i < size; i++)
			for (int j = 0; j < size; j++)
				sum[i][j] = matrixA[i][j] + matrixB[i][j];
		return sum;
	}
	public static int[][] matrixMultiply(int[][] matrixA, int[][] matrixB) {
		int size = matrixA.length;
		int[][] multiplication = new int[size][size];
		for (int i = 0; i < size; i++) {
			for (int j = 0; j < size; j++) {
				multiplication[i][j] = 0;
				for (int k = 0; k < size; k++) {
					multiplication[i][j] += matrixA[i][k] * matrixB[k][j];
				}
			}
		}
		return multiplication;
	}
	public static void matrixDisplay(int[][] matrix, String name) {
		int size = matrix.length;
		System.out.printf("\n%s:\n", name);
		for (int i = 0; i < size; i++) {
			for (int j = 0; j < size; j++)
				System.out.printf("%3d ", matrix[i][j]);
			System.out.println();
		}
	}
}
\end{lstlisting}

\newpage

\section*{Question 5}

Write a program \texttt{BackwardPrimes.java} that prompts user for a positive integer number $x$ and displays $x$ prime numbers whose reversal is also a prime.
Numbers 13, 31 and 157 are instances of such numbers.
Following is a sample run of your program.

\begin{terminal}
$ javac BackwardPrimes.java
$ java BackwardPrimes
Number of Primes? 15
13 17 31 37 71 73 79 97 107 113 149 157 167 179 199
\end{terminal}

\subsection*{Solution}

\lstset{language=Java,tabsize=4}
\begin{lstlisting}
import java.util.Scanner;
public class BackwardPrimes {
	public static void main(String[] args) {
		int i = 0;
		int num = 2;
		int count = getNum();
		while (i < count)
			if (checkNumber(num++) == true)
				i++;
		System.out.println();
	}
	public static int getNum() {
		Scanner input = new Scanner(System.in);
		System.out.print("Number of Primes: ");
		int num = input.nextInt();
		input.close();
		return num;
	}
	public static boolean checkNumber(int number) {
		boolean flag = false;
		int reversed = reverse(number);
		if (number != reversed) {
			if (isPrime(number) && isPrime(reversed)) {
				System.out.printf("%d ", number);
				flag = true;
			}
		}
		return flag;
	}
	public static boolean isPrime(int number) {
		boolean isPrime = true;
		for (int i = 2; i < number; i++) {
			if (number % i == 0) {
				isPrime = false;
				break;
			}
		}
		return isPrime;
	}
	public static int reverse(int number) {
		String str = Integer.toString(number);
		String strReversed = reverse(str);
		int numReversed = Integer.parseInt(strReversed);
		return numReversed;
	}
	public static String reverse(String str) {
		char[] strReversed = new char[str.length()];
		for (int i = 0; i < str.length(); i++)
			strReversed[i] = str.charAt(str.length() - i - 1);
		return new String(strReversed);
	}
}
\end{lstlisting}

\end{document}


\section{Modular Arithmatic}

\theo{http://services.artofproblemsolving.com/download.php?id=YXR0YWNobWVudHMvMy8yL2QzYjEzOGM0ODE3YzYwZGU4NGFmOTEwZDc0ZGNhODRjOGMyMzZlLnBkZg==&rn=dGh1ZS12NC5wZGY=}
{Thue's Lemma}{
    Let $n>1$ be an integer and a be an integer co-prime to $n$. Then there
    are integers $x, y$ with $0<|x|,|y|<\sqrt{n}$ so that
    \[ x \equiv a y \ (\mod\ n) \]
    Such a solution $(x, y)$ is called a ``small solution'' sometimes.
}

\begin{prooof}
    Let $r=\lfloor\sqrt{n}\rfloor$ i.e. $r$ is the unique integer for which
    $r^{2} \leq n<(r+1)^{2}$ The number of pairs $(x, y)$ so that $0 \leq x, y
    \leq r$ is $(r+1)^{2}$ which is greater than $n .$ Then there must be two
    different pairs $\left(x_{1}, y_{1}\right)$ and $\left(x_{2},
    y_{2}\right)$ so that
    \[\begin{aligned}
        x_{1}-a y_{1} &\equiv x_{2}-a y_{2} &(\mod\ n) \\
        x_{1}-x_{2} &\equiv a\left(y_{1}-y_{2}\right) &(\mod\ n) \\
    \end{aligned}\]

    Let $x=x_{1}-x_{2}$ and $y=y_{1}-y_{2},$ and we get $x \equiv a y\ (\mod\ n)$.
    Now, we need to show that $0<|x|,|y|<r$ and $x, y \neq 0 .$ Certainly, if
    one of $x, y$ is zero, the other is zero as well. If both $x$ and $y$ are
    zero, that would mean that two pairs $\left(x_{1}, y_{1}\right)$ and
    $\left(x_{2}, y_{2}\right)$ are actually same. That is not the case, and so
    both $x, y$ can not be $0 .$ Therefore, none of $x$ or $y$ is $0,$ and we are
    done.
\end{prooof}

\theo{}
{Generalization of Thue's Lemma}{
    Let $\alpha$ and $\beta$ are two real numbers so that $\alpha \beta \geq p$.
    Then for an integer $x,$ there are integers $a, b$ with $0<|a|<\alpha$
    and $0<|b|<\beta$ so that
    \[a \equiv x b \quad(\mod\ p) \]
    And we can even make this lemma a two dimensional one.
}

\thmbox{}
{Fermat's 4n+1 Theorem}{
    Every prime of the form $4n+1$ can be written as the sum of squares of two
    coprime integers.
}

\begin{prooof}
    We know that there is an $x$ such that
    \[x^2 \equiv -1 \ (\mod\ p)\]
    And by \autoref{theorem:Thue's Lemma}, there are $a, b$ with
    $0<\left|a\right|, \left|b\right|< \sqrt{n}$ for which
    \[\begin{aligned}
        a &\equiv xb \ &(\mod\ p)\\
        a^2 &\equiv x^2b^2 \ &(\mod\ p)\\
        a^2 + b^2 &\equiv 0 \ &(\mod\ p)
    \end{aligned}\]
    Since $a^2+ b^2 < 2p$, we are done.
\end{prooof}

\theo{}
{General Fermat's 4n+1 Theorem}{
    Let $n\in \left\{1, 2, 3\right\}$. If $-n$ is a quadratic residue modulo
    $p$, then there exists $a, b$ such that $a^2 + nb^2 = p$
}

\thmbox{}
{Factors are of the same form}{
    If $D\in \left\{1, 2, 3\right\}$ and $n = x^2 + Dy^2$ for some $x\perp y$,
    then all of the factors of $n$ are of the form $a^2 + Db^2$.
}

\begin{prooof}
    This is because the product of two numbers of such form is the same form
    as them:
    \[\begin{aligned}
        \left(a^2+Db^2\right)\left(c^2 + Dd^2\right) &=\left(ac-Dbd\right)^2 +
        D\left(ad+bc\right)^2\\
        &=\left(ac+Dbd\right)^2 + D\left(ad-bc\right)^2
    \end{aligned}\]
    And by \autoref{theorem:General Fermat's 4n+1 Theorem} the prime factors
    of $n$ are of the same form. And so all factors of $n$ are of the same form.
\end{prooof}

\thmbox{}
{Quadratic Residue -3}{
    $-3$ is a quadratic residue of modulo $p$ iff $p$ is of the form $3k+1$.
}
\begin{prooof}
    The only if part is easy with Thue's Lemma. For the if part, we have
    \[\begin{aligned}
        \left(\frac{p}{3}\right)\left(\frac{3}{p}\right) &=
        (-1)^{\left(\frac{p-1}{2}\right) \left(\frac{3-1}{2}\right)}=
        (-1)^{\left(\frac{p-1}{2}\right) }
    \end{aligned}\]
    Then we casework on $p\equiv 1, -1 \ (\mod\ 4)$ to show that in either
    case, \[\left(\frac{-3}{p}\right) = 1\]
\end{prooof}

\prob{}
{Thue's Lemma Note}{}{
    Let $p$ be prime number, prove that there exists $x, y$ such that
    $p=2x^2+3y^3$ iff $p\equiv 5, 11 \ (\mod\ 24)$.
}

\begin{solution}
    We need to show that $\frac{-3}{2}$ is a quadratic residue mod $p$, and
    the rest will follow from Thue's Lemma. We do that using the quadratic
    residue rules for $2$ and $-3$.
\end{solution}


%!TEX root = ../notes.tex
\section{March 8, 2022}
\subsection{Legendre Symbol \emph{continued}}
\begin{example}\label{example:legendre-symbol}
    Determine if $219$ is a quadratic residue mod $383$ (we note that $383$ is a prime).
    \begin{align*}
        \lege{219}{383} & = \lege{3}{383}\cdot\lege{73}{383}    \\
        \intertext{We now flip the Legendre Symbols using quadratic reciprocity:}
                        & = -\lege{383}{3} \cdot \lege{383}{73} \\
                        & = -\lege{2}{3} \cdot \lege{18}{73}    \\
                        & = 1\cdot \lege{18}{73}                \\
                        & = \lege{2}{73}\cdot\lege{9}{73}       \\
                        & = \lege{2}{73} = \boxed{1}
    \end{align*}
\end{example}
\begin{remark*}
    We must factor the top argument before beginning to flip using quadratic reciprocity.
\end{remark*}

\subsection{Proof of Quadratic Reciprocity}

Recall \cref{thm:qr}:
\begin{theorem*}[Law of Quadratic Reciprocity]
    Let $p, q\in\ZZ_+$ be distinct odd positive primes. Then
    \begin{equation*}
        \lege{p}{q}\lege{q}{p} = (-1)^{\frac{p-1}{2}\frac{q-1}{2}}
    \end{equation*}
    In other words,
    \[\lege{p}{q} = \lege{q}{p}\]
    if and only if at least one of $p, q$ is congruent to $1$ mod $4$.
\end{theorem*}

\begin{proof}[Proof of Quadratic Reciprocity (\cref{thm:qr}), using Gauss's Lemma (\cref{lemma:gauss-lemma})]
    \textsc{wlog}, let
    \begin{align*}
        P & = \left\{1, 2, \dots, \frac{p-1}{2}\right\},\quad N=-P, \\
        Q & = \left\{1, 2, \dots, \frac{q-1}{2}\right\}
    \end{align*}
    We write $\tilde P, \tilde N$ for $P\pmod{p}$ and $N\pmod{p}$ respectively, so that Gauss's lemma gives
    \[\lege{q}{p} = (-1)^\mu, \quad\text{where }\mu = |q\tilde P\cap\tilde N|\]
    In other words, $\mu$ is exactly the number of $x\in P$ such that $qx\equiv n\pmod{p}$ for some $n\in N$, and hence the number of $x\in P$ such that for $y\in \ZZ$, \[-\frac{p}{2} < qx - py < 0.\]
    We now specify more precisely which $y$ can possibly satisfy this condition. Solving these inequalities for $y$ gives
    \begin{align*}
        \frac{qx}{p} < y < \frac{qx}{p}+\frac{1}{2}.
    \end{align*}
    \otoh, since $x\leq \frac{p-1}{2}$ $\forall x\in P$, this gives
    \begin{align*}
        y < \frac{qx}{p}+\frac{1}{2} & \leq \frac{q(p-1)}{2p}+\frac{1}{2} \\
                                     & < \frac{q+1}{2}.
    \end{align*}
    Thus $0 < y < \frac{q+1}{2}$, which means that
    \[y\in Q = \left\{1, 2, \dots, \frac{q-1}{2}\right\}.\]
    We've shown that $\mu$ is the number of points $(x, y)\in P\times Q$ such that
    \[\frac{p}{2} < qx - py < 0.\]

    Switching $p$ and $q$, we also have
    \[\lege{p}{q} = (-1)^\eta\]
    where $\eta$ is the number of pairs
    \[(y, x)\in Q\times P\]
    such that
    \[-\frac{q}{2} < py - qx < 0\]
    which is exactly the number of pairs
    \[(x, y)\in P\times Q\]
    satisfying
    \[0 < qx - py < \frac{q}{2}\]
    (reflecting the inequality over $0$).

    We note that
    \[\lege{p}{q}\lege{q}{p} = (-1)^\mu (-1)^\eta = (-1)^{\mu+\eta}\]
    so all that remains is counting $\mu$ and $\eta$. And we have that $\mu + \eta$ is the number of ordered pairs $(x, y)\in P\times Q$ such that either
    \[-\frac{p}{2} < qx - py < 0 \text{ or } 0 < qx - py < \frac{q}{2}\]
    Noting that $qx - py\neq 0$ since $x$ and $y$ are from $P$ and $Q$ respectively, hence we can reduce this to
    \[-\frac{p}{2} < qx - py < \frac{q}{2}.\]
    Graphically, we are looking at:
    \begin{center}
        \includegraphics[width=0.8\textwidth]{images/qr-diagram.png}
    \end{center}

    where $\mu + \eta$ is the number of lattice points in the shaded region.

    If $\alpha$ is the number of lattice points in $A$ and $\beta$ the number of lattice points in $B$. Then
    \[\mu + \eta = \frac{p-1}{2}\frac{q-1}{2} - (\alpha + \beta)\]
    We show that $\alpha = \beta$ so that $\alpha + \beta \equiv 0\pmod{2}$.

    Let $\rho$ be the rotation given by rotating the rectangle about its center leaves it invariant.
    \[\rho(x, y) = \left(\frac{p+1}{2}-x, \frac{q+1}{2} - y\right)\]
    Quick check that
    \[qx - py < \frac{-p}{2} \Leftrightarrow qx' - py' > \frac{q}{2}\]

    Since $\rho$ maps lattice points to lattice points, then $\alpha = \beta$ which concludes the proof with a little extra handiwork.
\end{proof}

\subsection{Jacobi Symbol}
The Jacobi symbol generalizes the Legendre symbol.
\begin{definition}[Jacobi Symbol]
    Let $b$ be an odd positive integer and let $a\in\ZZ$. Write
    $b = p_1p_2\cdots p_m$, where $p_i$ are (not necessarily distinct) primes. Then we write
    \[\lege{a}{b} = \lege{a}{p_1}\lege{a}{p_2}\cdots \lege{a}{p_m}\]
    is called the \ul{Jacobi symbol}.
\end{definition}
We note some basic properties that the Jacobi symbol is totally multiplicative (on top and bottom!):
\begin{align*}
    \lege{a_1a_2}{b} & = \lege{a_1}{b}\lege{a_2}{b} \\
    \lege{a}{b_1b_2} & = \lege{a}{b_1}\lege{a}{b_2}
\end{align*}
\begin{remark*}
    Note that they're multiplicative \emph{fixing} either top or bottom. That is, they don't multiply like fractions.
\end{remark*}

\textbf{Warning!} $\lege{a}{b} = 1$ does not imply that $a$ is a quadratic residue modulo $b$ (since we could have $-1$'s from the factorization cancel out).

However, $\lege{a}{b} = -1$ \emph{does} imply that $a$ is a non-residue modulo $b$. (it is a non-residue mod at least one of prime factors of $b$).

\begin{example}
    \[\lege{2}{15} = \lege{2}{3}\lege{2}{5} = (-1)(-1) = 1\]
    but $2$ is not a quadratic residue modulo $15$.
\end{example}
\begin{proposition}[5.2.2 of Text]\label{prop:5.2.2}
    We have the following properties about the Jacobi symbol:
    \begin{enumerate}[(a)]
        \item \[\lege{-1}{b} = (-1)^{\frac{b-1}{2}}\]
        \item \[\lege{2}{b} = (-1)^{\frac{b^2 - 1}{8}}\]
        \item If $a, b\in\ZZ_+$, then
              \[\lege{a}{b}\lege{b}{a} = (-1)^{\frac{a-1}{2}\frac{b-1}{2}}\]
    \end{enumerate}
\end{proposition}


%  -----------------------------------------------------------------------------
%  Author         : Bimalka Piyaruwan Thalagala
%  GitHub         : https://github.com/bimalka98
%  Date Created   : 01.09.2020
%  Last Modified  : 06.03.2020
%  -----------------------------------------------------------------------------

\documentclass[a4paper,11pt]{article}%,twocolumn
\input{settings/packages}
\input{settings/page}
\input{settings/macros}
\usepackage[ framed, numbered]{matlab-prettifier}%framed,%
\usepackage{listings}
\usepackage{physics}
\usepackage{pdfpages}
\usepackage[toc,page]{appendix}

\begin{document}
\input{content/title_page}

\pagebreak

\tableofcontents
\listoffigures
\vfill
\begin{center}
	\textbf{\textit{*PDF is clickable}}
\end{center}

\textit{\textbf{Note:}}\\
\textit{All the materials and executable {\tt MATLAB R2018a} Live Script related to the project can also be found at \url{https://github.com/bimalka98/Digital-Signal-Processing}}
\pagebreak

\section{Question 1 and Question 2}
Refer the Appendix for the Code. For the generation of the AWGN, MATLAB's {\tt normrnd($\mu$,$\sigma$,[1, samples])} function which returns random samples from a normal distribution with mean $\mu$ and standard deviation $\sigma$ was used.\\

\vspace{10cm}
\begin{center}
	\textit{this page was intentionally left blank}
\end{center}

\pagebreak

\section{Question 3}
Plot of the sequence of Received signal and observations of the impact of the variance of noise on it by varying $\sigma^2=1$.

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.45]{figures/q3f1}
	\caption{Sequence of the Received signal when $\sigma^2 = 1$}
\end{figure}

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.45]{figures/q3f2}
	\caption{Impact of the variance of noise on the Received signal}
\end{figure}

As illustrated in the above figure when the variance of the noise increases the range of the values the received sample can  be in increases (observe the change in the range of Y axis). This essentially increases the bit errors at the receiving end.

\pagebreak
\section{Question 4}
Sketching and comparing the sequence of Y(signal recovered through threshold) with the transmitted signal.
\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.4]{figures/q4f1}
	\caption{Sequence of Y when the number of samples is 1000}
\end{figure}

Most of the transmitted bits have been identified correctly by the receiver after the thresholding(perfectly aligned curves). But some of the bits have been identified incorrectly after the thresholding. Change of signal levels due to the unwanted noise added during the process of transmission is the reason.

\section{Question 5}
Repeating the above steps for a sequence of length L = 100,000.
\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.4]{figures/q5f1}
	\caption{Sequence of Y when the number of samples is 100,000}
\end{figure}

The same observations made regarding the previous figure is applicable to this scenario as well. But probability of error has a significant reduction when transmitting a huge number of bits.

\pagebreak
\subsection{Histogram of the received sequence taking	the no of bins as 10}
\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.45]{figures/q5f2}
	\caption{Histogram of the received sequence using the user defined function: bins = 10}
\end{figure}

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.45]{figures/q5f3}
	\caption{Histogram of the received sequence using the MATLAB's built-in {\tt hist()} fucntion: bins = 10}
\end{figure}

\textbf{Comparison} : Histograms generated using the custom user defined function and the built-in function are almost the same. The reason for the difference in frequencies can be identified as the range differences of bins considered in the two algorithms. Code of the user defined fucntion can be found at the end of the Appendix.

\pagebreak
\subsection{Impact when the number of bins is changed from 10 to 100}

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.45]{figures/q5f4}
	\caption{Histogram of the received sequence using the user defined function: bins =100}
\end{figure}

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.45]{figures/q5f5}
	\caption{Histogram of the received sequence using the MATLAB's built-in {\tt hist()} fucntion: bins = 100}
\end{figure}
\textbf{Impact}: When the number of bins is changed from 10 to 100, shape of the normal distribution becomes clearly visible and differences between two histograms become negligible as the ranges of the bins become more similar.

\pagebreak
\subsection{Conditional PDFs when number of bins = 100 and A = 1}

Dependent axis represents the normalized frequencies calculated as follows. And therefore following figures illustrates the approximation for the required Probability Density Functions.
\[
Normalized ~Freq = \frac{Frequency}{Total~Number~of~Samples \times Width~of~a~Bin}
\]

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.45]{figures/q5f6}
	\caption{$f_{R|S}(r|S=A)$ when A = 1: Mean $\approx$ 1}
\end{figure}

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.45]{figures/q5f7}
	\caption{$f_{R|S}(r|S= -A)$ when A =1: Mean $\approx$ -1}
\end{figure}

Observe that when the distribution is conditioned on S, the mean of the distribution shifts along the amplitude axis towards the S.

\pagebreak
\subsection{Impact of A on the Conditional PDFs}

As illustrated in the following figures, when the distribution is conditioned on S =A , the mean of the distribution shifts along the amplitude axis towards the S = A.  But the variance of the distribution remains the same as it is only affected by the AWGN.

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.45]{figures/q5f8}
	\caption{Impact of A on the Conditional PDFs}
\end{figure}

\subsection{Expected Values of the distributions when A = 1}

The expected value of a continuous Random variable $X$ which has a PDF of $f_X(x)$ is given as,
\[
E[X] = \int_{-\infty}^{+\infty} x.f_X(x) ~dx
\]
The same formula can be redefined for discrete case as follows where $\Delta x$ represents the width of a bin which is equal in our cases.
\[
E[X] = \sum_{i = 1}^{N} x_i.P(x_i).\Delta x
\]

Therefore the requires expected values can be easily found using the {\tt Expected(x, px, bin\_width)} function defined in the appendix. The required parameters of that function can be found using the {\tt myHistogram(valueArray,numberOfBins, Normalized)} function. \\

\begin{tabular}[!h]{l c r}
	Expected value $E[R]$& = & -0.00073493\\
Expected value $E[R|S=A]$ &=& 1.0004\\
Expected value $E[R|S=-A]$& =& -1.0011\\

\end{tabular}

\pagebreak
\subsection{Marginal PDF of R $f_R(r)$}
Dependent axis represents the normalized frequencies calculated as follows as mentioned previously. Therefore, the following figure illustrates the approximation for the required Marginal Probability Density Functions of the continuous Random Variable R.
\[
Normalized ~Freq = \frac{Frequency}{Total~Number~of~Samples \times Width~of~a~Bin}
\]

Expected value of the Random Variable  ($E[R]$ = -0.00073493) reaches zero as the probabilities Pr(D = 0) = Pr(D = 1) = 1/2 are equal.

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.45]{figures/q5f9}
	\caption{Marginal PDF of R $f_R(r)$}
\end{figure}

\pagebreak
\section{Question 6: Effect of Interference}

Interference does nothing other than increasing the variance of the  distribution. When comparing the following figures with the figures of the previous section it is clearly visible. As the variance increases, to keep the area under the graph  a constant, the peak of the graph lowers. Therefore following two observations can be made when additional interference is added to the transmitted signal.
\begin{itemize}
	\item Variance of the distribution increases. Which eventually increases the probability of error.
	\item Peak value of the distribution decreases to keep the area under the curve constant.
\end{itemize}

These observations are common to all the figures and subsections of this section.

\subsection{Conditional PDFs when number of bins = 100, A = 1 and under additional interference}

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.38]{figures/q6f1}
	\caption{$f_{R|S}(r|S=A)$ when interference is added}
\end{figure}

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.38]{figures/q6f2}
	\caption{$f_{R|S}(r|S= -A)$  when interference is added}
\end{figure}

\pagebreak
\subsection{Impact of A on the Conditional PDFs under additional interference}
\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.45]{figures/q6f3}
	\caption{Impact of A on the Conditional PDFs under additional interference}
\end{figure}

\subsection{Expected Values of the distributions and under additional interference}

As previously described interference affects only to the variance of the distribution. Therefore the Expected Values remains almost the same.\\

\begin{tabular}[!h]{l c r}
	Expected value $E[R]$& = & -0.0012806\\
	Expected value $E[R|S=A]$ &=& 0.99798\\
	Expected value $E[R|S=-A]$& =& -0.99949\\

\end{tabular}

\subsection{Marginal PDF of R $f_R(r)$ and under additional interference}
\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.38]{figures/q6f4}
	\caption{Marginal PDF of R $f_R(r)$ under additional interference}
\end{figure}

\section{Question 7: Effect of Scaling when the number of bins = 100 and A = 1 and $\sigma^2 = 1$}

Effect of scaling of the pulse sequence(S), on the conditional probability distributions has the same effect as changing the signal level `A'. Therefore the figures and the subsections will be the same as that of Question 5.\\

Therefore only the behavior of Marginal distribution of R(received signal) is illustrated here. As it was not considered in earlier sections.

\begin{figure}[!h]
	\centering
	\includegraphics[scale=0.45]{figures/q7f1}
	\caption{Effect of Scaling on the Marginal PDF of R $f_R(r)$}
\end{figure}

The first sub-figure of the above figure shows the instance without any scaling. when the scale is increased, the distribution tends to divide in to two separate peaks centered around $+\alpha$ and $-\alpha$ as expected due to the lower variance than the scaling factor.

\begin{appendices}
	\section{Matlab Code for the Simulation}
	\includepdf[pages=-]{code/code.pdf}
\end{appendices}

%---------------------------------------------------------------------------
\end{document}
-


\subsection{Summary of Curve Sketching}\label{subsec:SummaryCurveSketching}

The following is a guideline for sketching a curve $y=f(x)$ by hand. Each item may not be relevant to the function in question, but utilizing this guideline will provide all information needed to make a detailed sketch of the function.

\begin{formulabox}[Guideline for Curve Sketching]
\begin{enumerate}
	\item	Domain of the function
	\item	$x$- and $y$-Intercepts
	\item	Symmetry
	\item	Vertical and Horizontal Asymptotes
	\item	Intervals of Increase/Decrease, and Local Extrema
	\item	Concavity and Points of Inflection
	\item	Sketch the Graph
\end{enumerate}
\end{formulabox}

\begin{example}{Graph Sketching}{graphsketching}
Sketch the graph of $y=f(x)$ where $\ds{f(x)=\frac{2x^2}{x^2-1}}$
\end{example}

\begin{solution}
\begin{enumerate}
\item  The domain is $\{x: x^2-1 \neq 0\}= \{x: x \neq \pm 1\}=(- \infty,-1) \cup (-1,1) \cup (1, \infty)$
\item  There is an $x$-intercept at $x=0$. The $y$ intercept is $y=0$.
\item  $f(-x)=f(x)$, so $f$ is an even function (symmetric about $y$-axis)
\item  $\ds{ \lim_{x \to \pm \infty} \frac{2x^2}{x^2-1} =\lim_{x \to \pm \infty} \frac{2}{1-1/x^2} = 2 }$, so $y=2$ is a horizontal asymptote.

Now the denominator is 0 at $x =\pm1$, so we compute:
\[\lim_{x \to 1^+} \frac{2x^2}{x^2-1} = + \infty, \;   \lim_{x \to 1^-} \frac{2x^2}{x^2-1} = - \infty, \;
\lim_{x \to -1^+} \frac{2x^2}{x^2-1} = - \infty, \;   \lim_{x \to -1^-} \frac{2x^2}{x^2-1} = + \infty.  \]
So the lines $x=1$ and $x=-1$ are vertical asymptotes.

\item  For critical values we take the derivative:
$$f'(x) = \frac{4x(x^2-1)-2x^2 \cdot 2x}{(x^2-1)^2} = \frac{-4x}{(x^2-1)^2}.$$
Note that $f'(x)=0$ when $x=0$ (the top is zero).
Also, $f'(x)=DNE$ when $x=\pm 1$ (the bottom is zero).
As $x=\pm 1$ is \ifont{not} in the domain of $f(x)$, the only critical number is $x=0$ (recall that to be a critical number we need it to be in the domain of the original function).

Drawing a number line and including \ifont{all} of the split points of $f'(x)$ we have:
$$\includegraphics[width=3.5in]{images/graphex1}$$
Thus $f$ is increasing on $(-\infty, -1)\cup(-1,0)$ and decreasing on $(0,1)\cup(1, \infty)$.

By the first derivative test, $x=0$ is a local max.

\item  For possible inflection points we take the second derivative:
$$f''(x)= \frac{12x^2+4}{(x^2-1)^3}$$
The top is never zero.
Also, the bottom is only zero when $x=\pm 1$ (neither of which are in the domain of $f(x)$).
Thus, there are no possible inflection points to consider.

Drawing a number line and including \ifont{all} of the split points of $f''(x)$ we have:
$$\includegraphics[width=2.25in]{images/graphex2}$$
Hence $f$ is concave up on $(- \infty,-1)\cup(1, \infty)$, concave down on $(-1,1)$.

\item  We put this information together and sketch the graph.

We combine some of this information on a single number line to see what \ifont{shape} the graph has on certain intervals:
$$\includegraphics[width=3.0in]{images/graphex3}$$
Note that there is a horizontal asymptote at $y=2$ and that the curve has $x$-int of $x=0$ and $y$-int of $y=0$.
Therefore, a sketch of $f(x)$ is as follows:
$$\includegraphics[width=3.5in]{images/graphex4}$$
\end{enumerate}
\end{solution}