\section{Algorithms}
Defining algorithms to find the discrete logarithm in finite fields can be done supposing that all the prime factors of $n - 1$ are small, without loss of generality. This allows to assume fast algorithms exist, using $\mathbb{Z}^*_n$ as field.

As from today, one of the best results in computing was achieved in 2019 by a group of researchers who announced the computation of the discrete logarithm of the number (RSA-240 + 49204), composed by 795 bit. This was obtained using a 2.1GHz CPU and took approximately 4000 core-years.

There are also reports of computing a discrete logarithm within a 768-bit prime field, taking 5300 core years, using the same algorithm as the previous (Number Field Sieve).

\subsection{Silver, Pohlig and Hellman algorithm}
The Silver-Pohlig-Hellman algorithm exploits a possible factorization of the order of a group through the \textbf{Chinese Remainder Theorem}.

This applies to groups whose order is either a smooth integer or a prime power, iteratively computing the digit of the discrete logarithm by shifting out all but one unknown digit in the exponent.

The prime factorization of a group with order $|G|$ can be written using its $p$-th roots as:
$$|G| = p_1^{e_1} \cdot p_2^{e_2} \cdot \dots p_l^{e_l}$$
Computing the discrete logarithm $x = \log_\alpha \beta$ in $G$ can be performed through a \textbf{divide-and-conquer} approach, finding smaller discrete logarithms $x_i \equiv x \mod p_i^{e_i}$ in the subgroups of order $p_i$ through some attack such as Pollard's method.

Then, all $x_i$ are used applying the Chinese Remainder Theorem, determining an unique $x$ thanks to pairwise co-prime divisors.

A consequence of the properties of prime factors is that one needs to know the factorization of the group order, which is not always easy.

\subsection{Index-calculus algorithm}
The index-calculus \textbf{probabilistic} algorithm has the particularity of working only under certain assumptions, such as cyclic groups $\mathbb{Z}^*_p$ and $GF(2^m)$, and has a lot in common with integer factorization although generally uses polynomials in rings.

However, it is one of the most efficient ways to compute discrete logarithm, due to its \textit{high parallelism} involving operations independent from one another.

Here, the term \textbf{index} is a commonly used word to define the discrete logarithm: $x = ind(a) \mod q$ for some base $b$, for $b^x \equiv a \mod m$ if $b$ is a primitive root of $q$ and $\gcd(a, q) = 1$.

Index-calculus depends on the property that a significant fraction of elements of $G$ can be efficiently expressed as \textbf{products of elements} of a small subset of $G$, assuming $|G|= q = p^n$ is a fairly large power of a small prime $p$.

This algorithm allows to find for any $x \in \mathbb{Z}^*_q$ the value of $x \mod q - 1$ such that $y = \alpha^x$, with $\alpha$ generator.

It works in two phases: the first one consists in precomputation, since it does not depend on the element $x$ to decrypt, and only has to be carried once. An irreducible element $f \in \mathbb{Z}^*_q$ is identified, and a subset $B \subset \mathbb{Z}^*_q$ is chosen, which will serve as ``basis'' and usually consists of all monic irreducible polynomials of arbitrary degree.

Then, all discrete logarithms f $a \in B$ are computed, i.e.\ a random integer $t \in [1, q - 2]$ is chosen and then $b^t$ is computed through repeated squaring. The algorithm finds all elements $c \in \mathbb{Z}^*_q$ such that:
$$c = b^t \mod f$$

Now, the index-calculus algorithm attempts to write $c$ as:
$$c = c_0 \prod_{a \in B} a ^{m}$$
$m$ is the highest power of $a$ which divides $c$. One way to determine this is to run through all $a \in B$ and divide $c$ successively by $a^m$. If the constant $c_0$ is the only one remaining after the division, then $c$ has the above form; otherwise, a different random integer $t$ is chosen.

Another good method to compute $c$ as products of elements $a \in B$ is using an \textit{integer factorization algorithm}, hence the close relation between these two problems.

Now, supposing some $c \equiv b^t \mod f$ has been found, and $c$ has the desired type of factorization, taking the discrete logarithm of both sides allows to obtain:
$$ind(c) - ind(c_0) \equiv \sum_{a \in B} m \cdot ind(a) \mod q - 1$$
The \textbf{modulo} operation is introduced since the discrete logarithm is only defined modulo $q - 1$. The left side of the equivalence is known, since $ind(c) = t$, and discrete logarithms of constants are assumed to be known as well.

The coefficients $m$ are also known, leaving only values $ind(a)$ and $a \in B$ to be found.

Assuming $|a| = h$, unknowns can be obtained through a linear system of equations with $h$ unknowns. Supposing integers $t$ can be chosen until a large number of different $c$ which factor into a product of $a$, eventually $h$ \textbf{independent} congruences will be found, written as:
$$t = ind(c_0) \equiv \sum_{a \in B} m \cdot ind(a) \mod q - 1$$
Here, \textit{independent} implies that the determinant of the coefficient matrix $\{m\}$ is prime to $q - 1$.

The system of unknowns can then be solved, completing the first stage of index-calculus. Precomputation gives a large set of all the discrete logs of $a \in B$, from which to calculate any other discrete log.

The final stage is performed supposing $x$ is the message whose discrete logarithm should be computed, and that the previous phase has already given all values of $ind(a),\ \forall\ a \in B$.

A random integer $t$ is once again chosen, and $y = xb^t$ is computed, i.e.\ the unique element $y \in \mathbb{Z}^*_q$ satisfying $y \equiv xb^t \mod f$.

As in the first stage, $y$ has to be factored into a constant $y_0$ times the product of powers of $a$, with $a \in B$. If not, another $t$ is picked, and so on, until an integer is obtained such that:
$$y = y_0 \prod_{a \in B} a ^{m}$$

As soon as this happens, the algorithm can terminate:
$$ind(x) = ind(y) - t \qquad \and \qquad ind(y) = ind(y_0) + \sum_{a \in B} m \cdot ind(a)$$
All the terms are known, therefore a message can be decrypted. Furthermore, advances in research have significantly sped up the process in finding discrete logs.

This algorithm allows to obtain a runtime which is not exponential in the bit length of the order, but \textbf{subexponential}. This can once again be compared to integer factorization: the order of magnitude of time needed to solve the discrete log problem for $q = p^n$ which is $k$ bits long is the same as factoring a $k$-bit number.

In order to provide a security of 80 bits, an attacker has thus to perform $2^{80}$ steps, therefore the prime $p$ of a discrete logarithm in $\mathbb{Z}^*_p$ should be at least 1024 bit long; for $GF(2^m)$, index-calculus is even more powerful, making these groups \textit{unusable} in practice.

\subsection{Number Field Sieve}
The Number Field Sieve algorithm, based on index-calculus, is currently the fastest known algorithm for finding both discrete logarithms and \textit{integer factoring}, and the method used to break the record on the 795-bit value previously mentioned.

The key idea of the Number Field Sieve, starting from integer factorization consists in \textbf{iteratively finding divisors within a field}. Assuming $q$ is a composite number, the algorithm generates pairs such that:
$$x \equiv y \mod q$$

$\gcd(x - y, q)$ is a non-trivial divisor of $q$ with probability 0.5, and after finding several pairs, $q$ can be factorized.

The previous equation can also be stated including prime numbers $p_i$ and their power within the factorization, i.e.:
$$x = p_1^{n_1} \cdot p_2^{n_2} \cdot \dots \cdot p_m^{n_m}$$

This approach is used while computing discrete logarithm: when enough elements have been found, the linear system of equations is solved, and the logarithm is consecutively applied such that $\log(p_i) = a_i$. Therefore, if some element $b$ can be expressed as $b = p_1^{n_1} \cdot p_2^{n_2} \cdot \dots \cdot p_k^{n_k}$, then its discrete logarithm consists in:
$$\log(b) = n_1\log(p_1) \cdot n_2\log(p_2) \cdot \dots \cdot n_k\log(p_k)$$

Given integers $a$ generator of $\mathbb{Z}^*_q$, $b$, $n$ and a prime number such that $p^q = n$, $x$ solving $a^x \equiv b \mod q$, the algorithm attempts to find two monic polynomials $y_1$ and $y_2$ having the property:
$$y_1 \equiv 0 \mod p \qquad \land \qquad y_2 \equiv 0 \mod p$$

Denoting with $\lambda_i \in \mathbb{C}$ the root of $y_i$, then holds that $y_i$ is the minimal polynomial of $\lambda_i$ and it is possible to define fields in which it is ensured the unique factorization.

Having $y_i(\lambda_i) = 0$, then it is again possible to express $y_i$ as a combination of powers which can be used to reduce polynomials and define their complex product within the algebraic \textit{number complex field}. This can be mapped to the original field, while trying to compute couples of integer values which are squares of another number, to also be able to factorize in $\mathbb{Z}^*_q$.

The system obtained with all the factors is then solved, usually using the Chinese Remainder Theorem.

Seminar - Algebraic Methods and Algorithms in Cryptology  /chapters/algorithms.tex


\subsection{Combinatorics}

\input{combinations}

\subsection{Simplices and barycentric coordinates}
\label{sec:barycentric}
\input{barycentric}

\subsection{Domains and function spaces}

\begin{Notation}{domain}
  By $\domain$ we denote a \define{domain} in $\R^d$, that is, an
  open, connected subset. The boundary of $\domain$ is denoted by
  $\d\domain$.

  Unless stated otherwise, we will always assume that the boundary of
  the domain consists of finitely many smooth surfaces meeting in
  ridges. Actually, we will often assume polygonal domains, such that
  the finite element mesh covers it exactly.
\end{Notation}

\begin{Notation}{vector-diff-operators}
  Differential operators for vector fields $u:\R^d\to\R^d$
  are defined as follows:
  \begin{xalignat}2
    \nabla \vu &=
    \begin{pmatrix}
      \d_1 u_1 & \cdots & \d_d u_1\\
      \vdots && \vdots \\
      \d_1 u_d & \cdots & \d_d u_d
    \end{pmatrix}
    &&\text{(gradient)}
    \\
    \div \vu &= \sum_{i=1}^d \d_i u_i
    &&\text{(divergence)}\\
    \strain u &= \frac{\nabla u + (\nabla u)^\transpose}{2}
    &&\text{(symmetric gradient)}
  \end{xalignat}

  For a tensor field $\sigma: \R^d\to \R^{d\times d}$, the divergence
  is a vector defined row-wise as
  \begin{gather}
    \div\sigma = \left(\sum_{j=1}^d \d_j \sigma_{ij}\right)_{i=1,\dots,d}
  \end{gather}
\end{Notation}

\begin{Notation}{lebesgue-spaces}
  For a domain $\domain\in\R^d$, we denote by $L^2(\domain)$\index{L2@$L^2(\domain)$} the \define{Lebesgue-space}
  of square integrable ``functions'' on $\domain$ with its norm\index{norm!L2@$\norm{\cdot}_{L^2}=\norm{\cdot}_0$}
  \begin{gather}
    \norm{u} = \norm{u}_0 = \norm{u}_{L^2(\domain)}.
  \end{gather}
  We denote the inner product of $L^2(\domain)$ by
  \begin{gather}
    \scal(u,v)_{L^2(\domain} = \form(u,v) = \int_\domain u v \dvx.
  \end{gather}
\end{Notation}

\begin{Notation}{sobolev-spaces}
  By $H^k(\domain)$\index{Hk@$H^k(\domain)$} we denote the
  \define{Sobolev space} of square integrable functions on $\domain$
  with square integrable distributional derivatives up to order
  $k$. Its norm is\index{norm!Hk@$\norm{\cdot}_{H^k}=\norm{\cdot}_{k}$}
  \begin{gather}
    \norm{u}_k = \norm{u}_{H^k(\domain)} = \sum_{\abs{\alpha} \le k}
    \norm{\d^\alpha u}_{L^2(\domain)}.
  \end{gather}
  We also use the $H^k$-seminorm\index{seminorm!Hk@$\abs{\cdot}_{H^k}$}
  \begin{gather}
    \abs{u}_k = \norm{u}_{H^k(\domain)} = \sum_{\abs{\alpha} \le k}
    \norm{\d^\alpha u}_{L^2(\domain)}.
  \end{gather}
\end{Notation}

\begin{Notation}{sobolev-spaces-2}
  By $H^1_0(\domain)$\index{H10@$H^1_0(\domain)$} we denote the
  completion of $C^\infty_0(\domain)$ with respect to the norm
  $\norm{\cdot}_{H^1(\domain)}$. Similarly,
  $H^1_{\Gamma}(\domain)$\index{H1gamma@$H^1_\Gamma(\domain)$} for any
  $\Gamma\subset\d\domain$ is the completion of all functions in
  $C^\infty(\domain)$ vanishing on $\gamma$.

  By $H^{-1}(\domain)$\index{H1m@$H^{-1}(\domain)$}, we denote the
  normed dual of $H^1_0(\domain)$, that is the space of bounded linear
  functionals on this space.
\end{Notation}

\begin{Notation}{vector-valued}
  We denote vector valued quantities by boldface letters like $\vx$,
  $\vu$, $\vv$. Lebesgue spaces of vector valued functions
  are denoted either by $L^2(\domain;\R^d)$ or by $\vL(\domain)$. The inner product of these is
  \begin{gather}
    \form(u,v) = \int_\domain \vu \cdot \vv \dx.
  \end{gather}
  Vector valued Sobolev spaces $\vH^k(\domain)$ are defined accordingly.
\end{Notation}

\begin{Notation}{transpose}
  Given two Hilbert spaces $V$ and $W$ and a mapping $A\colon V\to W^*$,
  we define the \define{adjoint} of $A$ by
  \begin{gather}
    \begin{split}
      A^\transpose\colon W &\to V^* \\
      \scal(Av,w)_{W^*\times W} &= \scal(v,A^\transpose w)_{V\times V^*}
      \qquad \forall v\in V, w\in W.
    \end{split}
  \end{gather}
  The adjoint with respect to the Euclidean inner product is called
  the \define{transpose} of $A$, denoted by $A^\transpose$ as
  well. This notation is used consistently for real and complex
  Hilbert spaces.
\end{Notation}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:


\section{Convolutional Neural Network}
\label{sec:theoretical_background:cnn}

\Acrlongpl{cnn} feature a specialized architecture to process data that has a known grid-like topology, such as images.
They are based on \acrlongpl{ann} and the mathematical convolution operation \cite[p.~326--331]{deeplearningbook}.

The main problems when working with color images is that they are high-dimensional and therefore require a lot of processing power.
For this reason, \acrlongpl{cnn} use special layers to detect features and reduce the amount of data.

% ------------------------------------------------------------------------------------------------------------------------------
\subsection{Architecture}
\label{subsec:theoretical_background:cnn:architecture}

A \acrlong{cnn} features a variety of different layers.
The three most relevant layers are described in this section.

% -----------------------------
\paragraph{Convolutional Layer}
Convolutional layers apply a set amount of filters to the input image.
Filters are used to recognize certain features (e.g. edges) and consist of as many so-called kernels as there are input channels.
A kernel is applied by sliding it along an input channel and calculating the scalar product of the overlapping region.
This generates new layers which are summed up to a feature maps. % feature map
The first convolutional layers are responsible for detecting low-level features, while following layers are responsible for detecting high-level features \cite[p.~327--330]{deeplearningbook}.

Convolutional layers require the specification of many hyperparameters:
\begin{enumerate}
  \item Number of filters
  \item Kernel size
  \item Padding
  \item Stride
  \item Activation
  \item Bias
\end{enumerate}

The number of filters corresponds to the number of convolved feature maps produced.
The kernel size specifies the height and the width of the convolution window used.
Padding can be used to preserve the spatial dimensions by adding layers of zeros to the input image.
Stride specifies the amount of movement between applications of the filter.
The activation parameter is used to specify an activation function (typically \acrshort{relu}).
Finally, the bias parameter specifies whether to use a bias term.

% -----------------------
\paragraph{Pooling Layer}
Pooling layers are used to reduce the spatial dimensions of the convolved feature maps.
This is done by replacing a rectangular region of the feature map with a summary of this region.
Maximum pooling yields the maximum value and average pooling yields the average value of a specific rectangular region.
Usually, max-pooling is used to preserve large values which correspond to detected features.
By doing so, the required computational power is significantly decreased and the dominant features are extracted \cite[p.~335--339]{deeplearningbook}.

Max-pooling layers mainly require two hyperparameters: the pool size and the strides value.
The pool size is the region over which the maximum is taken.
The strides value specifies how far the pooling window moves for each pooling step.

% -------------------------------
\paragraph{Fully-Connected Layer}
If every artificial neuron in layer $n$ is connected to every artificial neuron in layer $n + 1$, it is referred to as a fully-connected or dense layer.
Fully-connected layers are responsible for learning the non-linear combinations of the high-level convolved feature maps.
They are used at the end of \acrshortpl{cnn} to flatten the result of the convolution process into a vector of values.
The output layer usually contains as many artificial neurons as there are classes \cite{cnn_fc}.

The three main hyperparameters of fully-connected layers are the number of artificial neurons, the activation function and wheter to use a bias term.


\chapter{Similarity}
\section{Eigen-\textit{something}}
The prefix \textit{eigen}- is adopted from the German word \textit{eigen} for ``own-" or ``unique to", ``peculiar to". We will study about some \textit{unique} something, up to similarity.

\begin{defn}[eigenvalue, eigenvector, eigenspace] For a linear \textit{operator} $T \in \mathfrak L(V,V)$, if $$Tv = \lambda v$$ for a scalar $\lambda\in F$ and a \textit{nonzero} vector $v\in V$, we call $\lambda$ an \textbf{eigenvalue} and $v$ an \textbf{eigenvector} of $T$. The \textbf{eigenspace} of $\lambda$ is a subspace $$E_\lambda = \left \{v\in V: ~~ Tv= \lambda v\right\} = \ker(T-\lambda I)$$ of all vectors whose eigenvalue is $\lambda$.
\end{defn}
\begin{ex}
Check whether $E_\lambda$ is a subspace of $V$.
\end{ex}
\begin{prop}
If $f(t) \in F[t]$ and $v\in E_\lambda$, then $f(T)v = f(\lambda)v.$
\end{prop}
\begin{theorem}
TFAE(the followings are equivalent):
\begin{enumerate}[label={(\alph*)}]
    \item $\lambda$ is an eigenvalue of $T$.
    \item $T-\lambda I$ is singular, i.e., non-invertible.
    \item $\operatorname{det} (T-\lambda I) = 0.$
\end{enumerate}
\end{theorem}
\begin{proof}
We already(and MUST) know that (b) and (c) are equivalent. If $T-\lambda I$ is invertible, $$Tv = \lambda v ~~~~\Longleftrightarrow~~~~ (T-\lambda I)v = 0~~~~\Longleftrightarrow~~~~v=0$$and hence $\lambda$ is not an eigenvalue. And if $\lambda$ is an eigenvalue, $T-\lambda I$ is not bijective and hence singular.
\end{proof}
Thus, determinant of $\lambda I - T$ is important to decide whether or not $\lambda$ is an eigenvalue of $T$. Hence we define a \textit{polynomial}:
\begin{defn}[characteristic polynomial] The \textbf{characteristic polynomial} $\phi_T(t)$ is a polynomial defined by $$\phi_T(t) = \operatorname{det} (tI-T).$$ We will write $\chi\phi$ instead of the term `characteristic polynomial' since it is so long. \textsf{XD}

Here, $t$ behaves like a \textit{scalar} since it used instead of a scalar $\lambda$.
\end{defn}
Note that $\lambda$ is an eigenvalue iff $\phi_T (\lambda) = 0.$
\begin{ex}
Check if $\chi\phi$ is really a polynomial.
\end{ex}
\begin{ex}
Calculate the $\chi\phi$ of a matrix $$A = \begin{pmatrix}3&3&-1\\2&2&-1\\2&2&0\end{pmatrix}.$$ Find its eigenvalues and eigenspaces, and calculate $\phi_A (A)$. Note that $$f(T) = \sum a_n T^n$$ for a polynomial $f(t) = \sum a_n t^n \in F[t]$ and a linear operator $T$. \label[ex]{ec}
\end{ex}

\section{Diagonalizability}

Why we consider it? A big \textit{raison d'etre} of eigen-something is \textit{diagonalization} of a linear operator. First, from its name, we can define as follows:

\begin{defn}[diagonalization]
A \textbf{diagonalization} of a linear operator $T\in \mathfrak L(V,V)$ is a representation $T$ as a similar operator of a diagonal operator $D = \operatorname{diag}(d_1,\cdots,d_n).$ If there is a diagonalization of $T$, that is $T\sim D$ for a diagonal operator $D$, then we call $T$ is \textbf{diagonalizable}.
\end{defn}
\begin{ex}
Determine whether the following matrices are diagonalizable, where $F=\mathbb Q$:
$$A=\begin{pmatrix} -1 & 3 & -1 \\ -3 & 5 & -1 \\ -3 & 3 & 1 \end{pmatrix},\qquad B= \begin{pmatrix} 0 & 1 \\ -1 & 0 \end{pmatrix}. $$ If a matrix is not diagonalizable in given field, consider $F=\mathbb R$ and $F=\mathbb C$.
\end{ex}

If $T$ is diagonalizable, then $[T]_{\mathfrak B}^{\mathfrak B} = U^{-1}DU$ for a diagonal matrix $D$, supposing the basis $\mathfrak B$ is given for the vector space $V$; and there is another basis $\mathfrak C$ for $V$ such that $U = [\operatorname{id}_V]_{\mathfrak B}^{\mathfrak C}$. Evaluating this, we obtain $D=[T]_{\mathfrak C}^{\mathfrak C}.$ Since it is diagonal, we have $$[D]^{i} = D\mathbf e_i  = [T]_{\mathfrak C}^{\mathfrak C} [w_i]_{\mathfrak C}=[T w_i]_{\mathfrak C}$$ and $$D\mathbf e_i = d_i\mathbf e_i =[d_i w_i]_{\mathfrak C},$$ where $D=\operatorname{diag}(d_1,\cdots,d_n)$ and $\mathfrak C = \{w_1, \cdots, w_n\}.$ Hence we have $Tw_i = d_i w_i$, i.e., new basis must consist of eigenvectors, and the diagonal matrix contains corresponding eigenvalues. It is equivalent to the original definition. Hence we can re-define diagonalizability of a linear operator without matrices:
\begin{defn}[redefine of diagonalizability] A linear operator $T\in\mathfrak L(V,V)$ is diagonalizable if there is a basis for $V$ whose elements are all eigenvectors of $V$.
\end{defn}
Since eigenvectors span $V$, there are $n$ linearly independent eigenvectors.
\begin{prop}
If the eigenvalues of $T$ are mutually different, $T$ is diagonalizable.
\end{prop}
\begin{proof}
If $\lambda$'s are different, eigenvectors are linearly independent.
\end{proof}
\begin{prop}
If $H$ is Hermitian, that is $H=H^\dagger$, then $H$ can be diagonalized by a unitary operator $U$, i.e., $U^{-1} = U^\dagger.$
\end{prop}
\begin{proof}
Exercise.
\end{proof}
\begin{theorem}
Let $T\in \mathfrak L(V,V)$ and $\lambda_i$'s are eigenvalues of $T$. Then TFAE:
\begin{enumerate}[label={(\alph*)}]
    \item $T$ is diagonalizable,
    \item $\phi_T(t) = \prod (x-\lambda_i)^{e_i}$, $e_i = \dim E_{\lambda_i},$
    \item $V = \bigoplus E_{\lambda_i},$
    \item $\dim V = \sum \dim E_{\lambda_i}.$
\end{enumerate}
\end{theorem}
\begin{proof}
\begin{description}
\item [(a)$\Rightarrow$(b)] $\chi\phi$ is invariant under similarity, since $$tI - T = U^{-1}(tI - D)U,$$for example. Hence $$\phi_T(t) = \phi_D(t) = \prod (t-\lambda_i)^{e_i}.$$ A term due to a basis element appears once in the characteristic polynomial, hence the exponent of $t-\lambda_i$ is the (maximum) number of independent vectors in $E_{\lambda_i}$, i.e., dimension.
\item [(b)$\Rightarrow$(c)$\Rightarrow$(d)$\Rightarrow$(a)] ㅎㅎ. For (b) to (c), use dimension argument. Note that $\sum e_k = n$.
\end{description}
\end{proof}

\section{Cayley-Hamilton Theorem and Minimal Polynomial}

From \cref{ec}, we can know $\phi_A (A)$ for some matrices. Is it a general result? The answer is \textsf{YES}, and it is called \textit{Cayley-Hamilton theorem}!
\begin{theorem}[Cayley-Hamilton] $$\phi_T(T)=0.$$\end{theorem}
For $n=2$, let $A = \begin{pmatrix}a&b\\c&d\end{pmatrix}.$ Then $$\phi_A(t) = (t-a)(t-d)-bc = t^2 - (a+d)t + (ad-bc)$$ and we get a \textit{familiar}(?) form: $$T^2 - (a+d)T + (ad-bc)=0.$$
\begin{proof}[Proof(?)] Evaluating $t=T$, $$\phi_T(T) = \det(TI-T) = \det(T-T) = 0.$$

(\textbf{NOT A PROOF.}) \renewcommand{\qedsymbol}{$\lightning$}
\end{proof}
First, $t$ behaves as a scalar. And also $0$ is a zero matrix, rather than a scalar 0, in a formula $\phi_T(T)=0.$

Then how to prove it? We will consider $t^n \phi_T(t^{-1})$.

\begin{proof} Let $\phi_T(t) = \sum_{i=0}^n c_i t^i,$ then
  $$t^n \phi_T(t^{-1}) = \sum_{i=0}^n c_i t^{n-i}
  = t^n \operatorname{det}\left( t^{-1}I-T \right)
  = \operatorname{det}(I-tT).$$
From $$\operatorname{det}(A)I = A \cdot \operatorname{adj}A,$$
we get $$\operatorname{det}(I-tT)I = (I-tT)\operatorname{adj}(I-tT).$$
In order to `remove' $I-tT$ in the RHS, multiplying $\sum_{i=0}^m(tT)^i $ left,
$$\begin{aligned}\left(\sum_{i=0}^m (tT)^i \right) \left(\sum_{i=0}^n c_i t^{n-i}\right) &=
\left(\sum_{i=0}^m (tT)^i \right) \operatorname{det}(I-tT)I \\ &=
 \left(\sum_{i=0}^m (tT)^i \right) (I-tT)\operatorname{adj}(I-tT) \\ &
 = \left(I-(tT)^{m+1} \right) \operatorname{adj}(I-tT).\end{aligned}$$
By definition of classical adjoint, every entry of this matrix is a polynomial
of degree less than $n$. Hence RHS have terms of degree less than $n$ or greater
than or equal to $m$; for big $m$, the terms of degree $d\in [n,m)$ in LHS must
be vanished. Hence, with $m$ big enough, we obtain that the coefficient of the term of degree $n$ is zero. Now, observing the coefficient
of the term of degree $n$, we get
$$\sum_{i=0}^n c_i T^i = 0.$$
Hence $\phi_T(T) = 0.$
\end{proof}

\begin{defn}[annihilating ideal]
$$\mathcal I_T = \{p(t) \in F[t]:~~p(T) = 0\}.$$
A polynomial in $\mathcal I_T$ is called an \textbf{annihilating polynomial}.
\end{defn}
Since $\phi_T(t) \in \mathcal I_T$, by Cayley-Hamilton theorem, $\mathcal I_T \ne \emptyset.$
\begin{theorem}[minimal polynomial]
There is a monic annihilating polynomial which has the smallest degree. `Monic' means that the coefficient of the highest order term is 1. We call this polynomial the \textbf{minimal polynomial} $m_T(t).$ And also, $$m_T(t) | p(t), \qquad p(t) \in \mathcal I_T;$$especially, $m_T(t) | \phi_T(t).$
\end{theorem}
\begin{proof}
  Since $\operatorname{deg}\mathcal I_T$ is a subset of $\mathbb N$, there is the minimal degree $d$. If there is two different monic annihilating polynomial of degree $d$, denoting $m_1$ and $m_2$, we have $m_1 - m_2 \in \mathcal I_T$ which leads to a contradiction.

  If $m_T(t) \not\!|\; p(t)$ for every $p(t)\in\mathcal I_T,$ by division algorithm, we get that the remainder $r(t) = p(t)~\textrm{mod}~m_T(t)$ is also an annihilating polynomial which has the degree less than of $m_T(t),$ a contradiction.
\end{proof}
\begin{prop}
$\mathcal I_T$ is really an ideal. (Of a ring $F[t].$)
\end{prop}
\begin{proof}
  Exercise.
\end{proof}
\begin{ex}
Find the $\chi \phi$ and $m\phi$ of a matrix $$A=\begin{pmatrix}5&-6&-6\\-1&4&2\\3&-6&-4\end{pmatrix}.$$
\end{ex}
\begin{proof}[Answer]
$$\phi_A(t) = (t-1)(t-2)^2 ,\qquad m_A(t) = (t-1)(t-2).$$
\end{proof}
\section{Invariant and Triangularizability}
\begin{defn}[invariant subspace]
  For $T\in\mathfrak L(V,V)$ and $W\le V$, $W$ is called invariant under $T$ if $TW \le W.$
\end{defn}
\begin{ex}
  \begin{itemize}
    \item $F[t]$ is invariant under $D = \frac{\mathrm d}{\mathrm dt}.$
    \item Every space is invariant under a projection.
    \item Suppose there are two linear operator $T$ and $S$ on $V$, which commute, i.e., $TS = ST$. Let $W = \operatorname{im} S$ and $N = \ker S$, then $W$ and $N$ are invariant under $T$, since $TW = TSV = STV \le SV = W$ and $Sn = 0 \implies STn = TSn = 0.$
  \end{itemize}
\end{ex}
Let $W\le V$ be invariant under $T$, and $\mathfrak C = \{w_i\}_{i=1}^m$ be a basis of $W$. Then, $$[T]_{\mathfrak C}^{\mathfrak C} = \begin{pmatrix}[T\upharpoonright _W]^{\mathfrak C}_{\mathfrak C} & * \\ \mathbf 0 & *\end{pmatrix},$$ since, letting $\mathfrak B = \{w_i, v_j\}_{i=1,j=1}^{m,~n-m}\supseteq \mathfrak C$ be a basis of $V$, $$Tw_i = \sum a_iw_i + \sum 0 v_j.$$

\begin{theorem}
  Let $W\le V$ be invariant under $T$, then $$\phi_{T\upharpoonright _W} | \phi_T\qquad \textrm{and}\qquad m_{T\upharpoonright W} | m_T.$$
\end{theorem}
\begin{proof}
Let $W$ has a basis $\mathfrak C$ and $\mathfrak B$ is a basis of $V$ which is extended from $\mathfrak C$. Then we have $$[T]^{\mathfrak B}_{\mathfrak B} = \begin{pmatrix}[T\upharpoonright _W]^{\mathfrak C}_{\mathfrak C}& * \\ \mathbf 0 & *\end{pmatrix}.$$ For $\chi\phi$, $$0 = \phi_T \left([T]^{\mathfrak B}_{\mathfrak B}\right) = \begin{pmatrix}\phi_T \left([T\upharpoonright _W]^{\mathfrak C}_{\mathfrak C}\right)& * \\ \mathbf 0 & *\end{pmatrix}.$$ And by above, we have $$\phi_T \left([T]^{\mathfrak B}_{\mathfrak B}\right)  = 0 \implies \phi_T \left([T\upharpoonright _W]^{\mathfrak C}_{\mathfrak C}\right)  = 0,$$ which completes the remained part of proof.
\end{proof}

\begin{ex}
  Consider a diagonalizable transformation $T$, and let $W_i$'s be its eigenspaces, then it suits perfectly to above theorem, and it makes the `sufficient-necessary condition' of diagonalizability clear. But if $T$ is not diagonalizable, it cannot be adopted since we do not know the other components of given block matrix.
\end{ex}

% conductor
We define the following as a generalization of `annihilator ideal':
\begin{defn}[conductor (ideal)] Let W be an \textit{invariant} subspace for $T$ and let $v$ be a vector in $V$. The \textbf{T-conductor of v into W} is the set $S_T(v; ~W)$ which consists of all polynomials $g\in F[t]$ such that $g(T) v \in W.$

If $W=0$, we denote it as $\mathcal I_T(v) = S_T(v;~0)$ and call the \textbf{T-annihilator of v}. And $\mathcal I_T = \bigcap_v \mathcal I_T(v)$ is the $T$-annihilator of $V$, which annihilates all the vectors of $V$.
\end{defn}
\begin{prop}Conductor is an ideal in $F[t]$.\end{prop}
\begin{defn}[conductor (vector)] The monic generator of the ideal $S(v;~W)$ is also called the \textbf{conductor} of $v$ into $W$.
\end{defn}

Analogous proofs of one for uniqueness of minimal polynomial prove also for the conductors, trivially. And, since $\mathcal I_T$ is the \textit{strongest} polynomials, the $T$-conductors divide the minimal polynomial for $T$.

\section{Minimal Polynomials and Triangular-/Diagonal-izability}
\begin{lemma}
Suppose $$m_T(t) = \prod (t-c_i)^{r_i},\qquad c_i \in F,$$ and let $W\lneq V$ be invariant under $T$. Then there exists a vector $v\not\in W$ such that $$\exists \lambda \text{: eigenvalue of }T:~~~~(T-\lambda I)v \in W,$$ that is, a linear polynomial is a $T$-conductor for some $v$.
\label[lemma]{lc}
\end{lemma}
\begin{proof}Let $w\in V\setminus W,$ and $g$ be the $T$-conductor of $w$ into $W$. ($g(T)w = 0.$) Then $g | m_T$, and since $w \not\in W$, $g $ cannot be a constant. ($g(T)w = kw \in W \implies k = 0 = g$ which is contradict to the fact that $g$ is a generator of an nontrivial ideal.) Therefore $$g(t) = \prod(t-c_i)^{e_i}; \qquad \sum e_i > 0.$$ Choose $j$ so that $e_j > 0$, then $g = (t-c_j) h$ for some $h$. Since $v=h(T)w \not \in W$ ($g$ is minimal in the sense of degree) and $g(T)w = (T-cI)v \in W$, we just found $v$! Obviously, $c$ is an eigenvalue.
\end{proof}

We conclude(?) with the following necessary-sufficient condition of diagonalizability and trigonalizability(trivial meaning), in the sense of minimal polynomial:
\begin{theorem}$T$ is triangularizable iff $m_T = \prod (t-\lambda_i)^{e_i},$ where $\lambda_i$'s are distinct.
\label[thm]{tt}
\end{theorem}
\begin{proof}
($\Longleftarrow$) Let $W=0$, then above lemma says $\exists v \exists \lambda (T - \lambda I) v = 0.$ Hence it forms an eigenspace, and there is a basis $\mathfrak B$ of $V$ extending $\{v\}$; therefore we have $$[T]_\mathfrak B ^ \mathfrak B = \begin{pmatrix} \lambda & ** \\ \mathbf 0 & * \end{pmatrix}.$$ By an induction on the dimension of square matrix ($*$ for above), we obtain a triangularization of $T$: $$[T]_\mathfrak B ^ \mathfrak B = \begin{pmatrix}
a_{11} & a_{12} & \cdots & a_{1n} \\
      0     & a_{22} & \cdots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
      0     &       0     & \cdots & a_{nn} \\
\end{pmatrix}.$$
($\Longrightarrow$) Calculate it$\sim$.
\end{proof}
\begin{coro}Every linear operator is triangularizable if the given field is algebraically closed.\end{coro}
\begin{proof}[Another proof of Corollary: using induction]
There is an eigenvalue and an eigenvector since the field is algebraically closed. Hence let $\lambda$ and $v$ the chosen ones, extend $v$ to a basis $\mathfrak B$ of $V$, and denote $\mathfrak C = \mathfrak B - \{v\}$ and $W = \langle \mathfrak C \rangle$. Then:
$$[T]_\mathfrak B ^ \mathfrak B = \begin{pmatrix}
\lambda & ** \\
\mathbf 0 & * \\
\end{pmatrix}.$$ And we know that $$* = [\pi \circ T]_{\mathfrak C}^{\mathfrak C}$$ where $\pi$ `removes' $v$-component: $$\pi:~V\to W;\qquad av + \sum_{w_i\in \mathfrak C} b_i w_i \mapsto  \sum_{w_i\in \mathfrak C} b_i w_i.$$
\begin{center}
\leavevmode
\xy
\xymatrix {
W\ar[r]^{T} \ar[dr]_{\pi\circ T}& TW \ar[d]^{\pi}\\ & W
}
\endxy
\end{center}
Since $\pi\circ T$ is linear, an induction completes the proof.
\end{proof}

\begin{theorem}$T$ is diagonalizable iff $m_T = \prod (t-\lambda_i),$ where $\lambda_i$'s are distinct.\end{theorem}
\begin{proof}
($\Longrightarrow$) Trivial. Think as a linear transformation each of $T-\lambda_i I$'s.

($\Longleftarrow$) Let $W=\bigoplus E_{\lambda_i}$ be the space spanned by all of the eigenvectors of $T$, and suppose $W \ne V$. By \cref{lc}, there is a vector $v\not \in W$ and an eigenvalue $\lambda_j$ such that $w = (T-\lambda_j I)v \in W$. Since $w\in W$, it is represented by a linear combination of eigenvectors uniquely: $$w = \sum_{w_i \in  E_{\lambda_i}} w_i,$$noting that $T w = \sum \lambda_i w_i.$

Let $m_T = (t-\lambda c_j) g$ for some polynomial $g$, and $$g(t) - g(c_j) = (t - c_j)h(t)$$ for some polynomial $h$. Then we have $$g(T)v - g(c_j)v = h(T)(T-c_j I)v = h(T)w \in W$$ and $g(T)v \in W$ whence $g(c_j)v \in W.$ Since $v\not\in W$, $g(c_j) = 0.$ It contradicts the assumption that $m_T$ has distinct roots.
\end{proof}

\section{Simultaneous Triangular-/Diagonal-ization}
We want to find a basis which triangularizes all of the transformations in a family $\mathscr F$ \textit{simultaneously}.

The subspace $W$ is \textbf{invariant under} $ {\mathscr F}$ if $W$ is invariant under each operators.

Since all diagonal matrices commute, if $T$ and $S$ diagonalized simultaneously, then$$(U^{-1}TU)(U^{-1}SU) = (U^{-1}SU)(U^{-1}TU)$$ and hence $TS = ST.$ Therefore we consider only a family whose elements commute mutually, for simultaneous diagonalization.

For simultaneous triangularization, one does not have to satisfy the commutating condition; however it is a \textit{sufficient} condition for simultaneous triangularization, as we will see.

\begin{lemma}Let $\mathscr F$ be a commuting family of triangularizable linear operators on $V$. Let $W$ be a proper subspace of $V$ which is invariant under $\mathscr F$, then there is a vector $v\in V\setminus W$ such that $$\forall T\in\mathscr F, ~~~ Tv \in \langle v \rangle \oplus W.$$
\end{lemma}
\begin{proof}It is too taxing to deal with infinitely many operators; hence we use a basis: let $\{T_1, \cdots, T_r\}$ be `a'(need not to be unique) maximal linearly independent subset of $\mathscr F$; i.e. a basis for $\langle \mathscr F \rangle \le \mathfrak L(V,V).$ ($\mathfrak L(V,V)$ is a f.d.v.s.) Then it is sufficient to check for these basis elements only.

By \cref{lc}, for a single operator, we can find a vector $v_1\in V \setminus W$ and a scalar $\lambda_1$ such that $(T_1 - \lambda_1 I)v_1 \in W.$ Since $W$ is invariant under $T_1$, $$V_1 = \left\{v\in V:~~(T_1 - \lambda_1 I)v \in W \right\} \gneq W.$$ And $V_1$ is invariant under $\mathscr F$.

Now, in order to use induction, consider $V_1$ instead of $V$. Let $W$ be a proper subspace of $V_1$, and $U_2 = T_2 \upharpoonright _W$ instead of $T_1$ of above procedure. Since $m_{U_2} | m_{T_2}$, we may apply \cref{lc} to new $W$ and $U_2$ and consider as of $T_2$. We obtain a vector $v_2 \in V_1 \setminus W$ and a scalar $\lambda_2$ such that $(T_2 - \lambda_2 I)v_2 \in W.$ Note that, since $v_2 \in V_1,$ both of $(T_1 - \lambda_1 I)v_2$ and $(T_2 - \lambda_1 I)v_2$ belong to $W$. And let $$V_2 =\left\{v\in V_1:~~(T_2 - \lambda_2 I)v \in W\right\},$$ then $V_2$ is invariant under $\mathscr F$.

Continue this process by an induction, then we can find $v = v_r$ as the desired vector.
\end{proof}
\begin{theorem}Let $\mathscr F$ be a commuting family of triangularizable linear operators on $V$. Then it can be triangularized simultaneously.
\end{theorem}
\begin{proof}
Induction. Now it is easy. (Same with the proof of \cref{tt}.)
\end{proof}

Now, finish with diagonalization.

\begin{theorem}Let $\mathscr F$ be a commuting family of diagonalizable linear operators on $V$. Then it can be diagonalized simultaneously.
\end{theorem}
\begin{proof}
Almost same process as for triangularization, at this point, however, it is easier to proceed by induction on dim$V$.

If $\operatorname{dim}V = 1$, automatically proved. Let $\operatorname{dim}V = n$ and choose any $cI \ne T\in\mathscr F$. Let $\lambda_i$'s be the distinct eigenvalues of $T$ and let $W_i = E_{\lambda_i} = \ker (T-c_i I).$ $W_i$ is invariant under every operator which commutes with $T$; and each operator in $$\mathscr F_i = \left\{T\upharpoonright_{W_i} : ~~~ T\in\mathscr F \right\}$$ is diagonalizable since its minimal polynomial divides the minimal polynomial for the corresponding operator in $\mathscr F$. Operators in $\mathscr F_i$ can be diagonalized simultaneously since $\operatorname{dim}W_i < \operatorname{dim}V$ by a basis $\mathfrak B_i$. Then $\mathfrak B = (\mathfrak B_i)$ is a desired basis.
\end{proof}


% Preamble.
\documentclass[12pt]{article}
\usepackage[margin=1.25in]{geometry}
\usepackage[fleqn]{amsmath}
\usepackage{textcomp}
\usepackage{gensymb}
\usepackage{amsfonts}
\usepackage{enumitem}
%\usepackage{tikz}  % Include for figures.
%\usepackage{subfiles}  % Include for subfiles.

%% Title macros.
\newcommand{\HOMEWORKNUM}{27}
\newcommand{\NAME}{D. Choi}
\newcommand{\DATE}{2020-07-02}

\title{\vspace{-2\baselineskip}MATH 225 - Homework \#\HOMEWORKNUM}
\author{\NAME}
\date{\DATE}

%% Formatting options.
%\pagenumbering{gobble}  % Include for single-page document.

% Document.
\begin{document}
\maketitle

\section*{1.}
\textit{Solve:}
\begin{equation*}
	y^{\prime\prime} - 9y = 0, \quad
	y(0) = 2, \quad
	y^\prime(0) = -1.
\end{equation*}
The roots of the characteristic equation are
\begin{equation*}
	r^2 - 9 = 0 \quad \Rightarrow \quad r_1 = 3, \quad r_2 = -3.
\end{equation*}
With $2$ distinct real roots, the general solution is
\begin{equation*}
	y(t) = c_1 e^{3 t} + c_2 e^{-3 t}.
\end{equation*}
Using the system of equations given by the initial conditions,
\begin{alignat*}{2}
	y(0) &=& \quad 2 &= c_1 + c_2 \\
	y^\prime(0) &=& \quad -1 &= 3 c_1 - 3 c_2,
\end{alignat*}
the coefficients can be solved for as
\begin{equation*}
	c_1 = \frac{5}{6}, \quad
	c_2 = \frac{7}{6}.
\end{equation*}
Thus,
\begin{equation*}
	\boxed{
		y(t) = \frac{5}{6} e^{3 t} + \frac{7}{6} e^{-3 t}
	}.
\end{equation*}

\section*{2.}
\textit{Solve:}
\begin{equation*}
	y^{\prime\prime} + 6y^\prime + 9y = 0, \quad
	y(0) = 1, \quad
	y(1) = 1.
\end{equation*}
The roots of the characteristic equation are
\begin{equation*}
	r^2 + 6r + 9 = 0 \quad \Rightarrow \quad r_1 = -3, \quad r_2 = -3.
\end{equation*}
With a repeated real root, the general solution is
\begin{equation*}
	y(t) = c_1 e^{-3 t} + c_2 e^{-3 t} t.
\end{equation*}
Using the system of equations given by the initial conditions,
\begin{alignat*}{2}
	y(0) &=& \quad 1 &= c_1 \\
	y(1) &=& \quad 1 &= c_1 e^{-3} +  c_2 e^{-3},
\end{alignat*}
the coefficients can be solved for as
\begin{equation*}
	c_1 = 1, \quad
	c_2 = e^3 - 1.
\end{equation*}
Thus,
\begin{equation*}
	\boxed{
		y(t) = e^{-3 t} + (e^3 - 1) e^{-3 t} t
	}.
\end{equation*}

\section*{3.}
\textit{Solve:}
\begin{equation*}
	y^{\prime\prime} + 7y^\prime + 10y = 0, \quad
	y(0) = -1, \quad
	y^\prime(0) = 0.
\end{equation*}
The roots of the characteristic equation are
\begin{equation*}
	r^2 + 7r + 10 = 0 \quad \Rightarrow \quad r_1 = -2, \quad r_2 = -5.
\end{equation*}
With $2$ distinct real roots, the general solution is
\begin{equation*}
	y(t) = c_1 e^{-2 t} + c_2 e^{-5 t}.
\end{equation*}
Using the system of equations given by the initial conditions,
\begin{alignat*}{2}
	y(0) &=& \quad -1 &= c_1 + c_2 \\
	y^\prime(0) &=& \quad 0 &= -2 c_1 - 5 c_2,
\end{alignat*}
the coefficients can be solved for as
\begin{equation*}
	c_1 = -\frac{5}{3}, \quad
	c_2 = \frac{2}{3}.
\end{equation*}
Thus,
\begin{equation*}
	\boxed{
		y(t) = -\frac{5}{3} e^{-2 t} + \frac{2}{3} e^{-5 t}
	}.
\end{equation*}

\end{document}


\documentclass[11pt]{article}
\usepackage[margin=0.75in]{geometry}
\usepackage{amsmath}
\usepackage{amsfonts}

\title{Regularized logistic regression gradient with weighted samples}

\begin{document}

\maketitle

Scikit-learn's batch logistic regression (which uses LIBLINEAR) doesn't support weighted
samples. The SGD classifier does support weighted samples, but it can be tricky to tune.
For my application, solving the optimization problem in batch with L-BFGS worked best.

\section*{No regularization, no weights}

In other words, all samples have weight $1.0$. $M$ training examples and $N$ features,
one of which is a dummy feature for the intercept. \\

$\theta \in \mathbb{R}^N$ - coefficients

$y \in \{0, 1\}^M$ - response variable ($M x 1$ vector)

$X \in \{0, 1\}^{MxN}$ - $M x N$ design matrix

$f(x) = \frac{1}{1+e^{-x}}$ - logistic function

$l(\theta)$ - loss function

$l(\theta) = - e^T \left(y \odot \log(p) + (1-y) \odot \log(1-p)\right) $

where $p = f(X^T \theta)$

aka average log-loss

$$r = f(X^T \theta) - y$$

$$\nabla l = X^T r $$

\section*{Standard regularization}

No regularization on the intercept. $M$ training examples and $N$ features. \\

$\theta \in \mathbb{R}^N$ - coefficients

$\theta_0 \in \mathbb{R}$ - intercept

$y \in \{0, 1\}^M$ - response variable ($M x 1$ vector)

$w \in (0, \infty)^N $ - per example weight ($N x 1$ vector)

$X \in \{0, 1\}^{MxN}$ - $M x N$ design matrix

$f(x) = \frac{1}{1+e^{-x}}$ - logistic function

$l(\theta)$ - loss function

$l(\theta) = -w^T \left(y \odot \log(p) + (1-y) \odot \log(1-p)\right) / \sum w $

where $p = f(X^T \theta + \theta_0)$

aka average of the usual log-loss, weighted by $w$

$$r = f(X^T \theta + \theta_0) - y$$

$$\nabla_0 l = \frac{r^T w}{\sum w} $$

$$\nabla l = \frac{X^T (w \odot r) + \lambda \theta}{ \sum w} $$

\end{document}

regularized logistic regression gradient with weighted samples.tex


\documentclass[12pt]{cdblatex}
\usepackage{examples}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% ============================================================================================
\bgroup
\CdbSetup{action=hide}
\begin{cadabra}
   import cdblib
   checkpoint_file = 'tests/semantic/output/example-03.json'
   cdblib.create (checkpoint_file)
   checkpoint = []
\end{cadabra}
\egroup

\clearpage

% ============================================================================================
\section*{Example 3a The Riemann curvature tensor}

\begin{cadabra}
   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u#}::Indices(position=independent).

   \partial{#}::PartialDerivative.

   \Gamma^{a}_{b c}::TableauSymmetry(shape={2}, indices={1,2});

   ;::Symbol;  # Suggsted by Kasper as a way to make use of ; legal
               # see https://cadabra.science/qa/473/is-this-legal-syntax
               # this code works with and without this trick

   # rules for the first two covariant derivs of V^a

   deriv1 := V^{a}_{; b}      -> \partial_{b}{V^{a}}
                               + \Gamma^{a}_{c b} V^{c}.        # cdb (ex-03.101,deriv1)

   deriv2 := V^{a}_{; b ; c}  -> \partial_{c}{V^{a}_{; b}}
                               + \Gamma^{a}_{d c} V^{d}_{; b}
                               - \Gamma^{d}_{b c} V^{a}_{; d}.  # cdb (ex-03.102,deriv2)

   substitute (deriv2,deriv1)                   # cdb (ex-03.103, deriv2)

   Vabc := V^{a}_{; b ; c} - V^{a}_{; c ; b}.   # cdb (ex-03.104, Vabc)

   substitute (Vabc,deriv2)                     # cdb (ex-03.105, Vabc)

   distribute     (Vabc)                        # cdb (ex-03.106, Vabc)
   product_rule   (Vabc)                        # cdb (ex-03.107, Vabc)

   sort_product   (Vabc)                        # cdb (ex-03.108, Vabc)
   rename_dummies (Vabc)                        # cdb (ex-03.109, Vabc)
   canonicalise   (Vabc)                        # cdb (ex-03.110, Vabc)

   sort_sum       (Vabc)                        # cdb (ex-03.111, Vabc)
   factor_out     (Vabc,$V^{a?}$)               # cdb (ex-03.112, Vabc)

   checkpoint.append (Vabc)

   # create rule for Riemann, export later (for use by lib/dgeom)

   substitute (Vabc,$V^{a} -> -1$)              # cdb (ex-03.113, Vabc)
                                                # note use of -1 to get correct
                                                # signs when coupled with the rule
                                                # for Rabcd (next statement)

   Rabcd := R^{a}_{d b c} -> @(Vabc).           # cdb (ex-03.114, Rabcd) #

   foo   := R^{a}_{b c d}.                      # cdb (ex-03.115, foo)
   substitute (foo, Rabcd)                      # cdb (ex-03.116, foo)

   # update rule to use nice indices

   Rabcd := R^{a}_{b c d} -> @(foo).

   checkpoint.append (Rabcd)

\end{cadabra}

\begin{dgroup*}[spread={3pt}]
   \Dmath*{\Cdb*{ex-03.101}}
   \Dmath*{\Cdb*{ex-03.102}}
   \Dmath*{\Cdb*{ex-03.103}}
\end{dgroup*}
%
\begin{dgroup*}[spread={3pt}]
   \Dmath*{\cdb{ex-03.104} = \Cdb*[\V{5pt}\hskip2.0cm\hfill]{ex-03.105}
                           = \Cdb*[\V{5pt}\hskip2.0cm\hfill]{ex-03.106}
                           = \Cdb*[\V{5pt}\hskip2.5cm\hfill]{ex-03.107}
                           = \Cdb*[\V{5pt}\hskip2.5cm\hfill]{ex-03.108}
                           = \Cdb*[\V{5pt}\hskip2.5cm\hfill]{ex-03.109}
                           = \Cdb*[\V{5pt}\hfill]{ex-03.110}
                           = \Cdb*[\V{5pt}\hfill]{ex-03.111}
                           = \Cdb*[\V{5pt}\hfill]{ex-03.112}
                           = - R^{a}{}_{dbc} V^d\quad\text{(MTW)}}
\end{dgroup*}

\Dmath*{\cdb{ex-03.115} = \cdb{ex-03.116}}

\clearpage

% ============================================================================================
\section*{Example 3b The Riemann curvature tensor}

This differs from the above by not using the \verb|::TableauSymmetry| property.
It gives the same results as above but it does require a little bit more
housekeeping.

\begin{cadabra}
   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u#}::Indices(position=independent).

   \partial{#}::PartialDerivative.

   ;::Symbol;  # Suggsted by Kasper as a way to make use of ; legal
               # see https://cadabra.science/qa/473/is-this-legal-syntax
               # this code works with and without this trick

   # rules for the first two covariant derivs of V^a

   deriv1 := V^{a}_{; b}      -> \partial_{b}{V^{a}}
                               + \Gamma^{a}_{c b} V^{c}.        # cdb (ex-03.301,deriv1)

   deriv2 := V^{a}_{; b ; c}  -> \partial_{c}{V^{a}_{; b}}
                               + \Gamma^{a}_{d c} V^{d}_{; b}
                               - \Gamma^{d}_{b c} V^{a}_{; d}.  # cdb (ex-03.302,deriv2)

   substitute (deriv2,deriv1)                    # cdb (ex-03.303, deriv2)

   Vabc := V^{a}_{; b ; c} - V^{a}_{; c ; b}.    # cdb (ex-03.304, Vabc)

   substitute (Vabc,deriv2)                      # cdb (ex-03.305, Vabc)

   distribute     (Vabc)                         # cdb (ex-03.306, Vabc)
   product_rule   (Vabc)                         # cdb (ex-03.307, Vabc)

   # ------------------------------------------------------
   # trick to obtain a symmetric connection

   G_{a b}::Symmetric.

   substitute     (Vabc,$\Gamma^{a}_{b c} ->  G^{a} G_{b c}$)
   sort_product   (Vabc)                         # cdb (ex-03.308, Vabc)
   rename_dummies (Vabc)                         # cdb (ex-03.309, Vabc)
   canonicalise   (Vabc)                         # cdb (ex-03.310, Vabc)
   substitute     (Vabc,$G^{a} G_{b c} -> \Gamma^{a}_{b c}$,repeat=True)
   # ------------------------------------------------------

   sort_product   (Vabc)
   rename_dummies (Vabc)
   canonicalise   (Vabc)

   sort_sum       (Vabc)                         # cdb (ex-03.311, Vabc)
   factor_out     (Vabc,$V^{a?}$)                # cdb (ex-03.312, Vabc)

   checkpoint.append (Vabc)
\end{cadabra}

\begin{dgroup*}[spread={3pt}]
   \Dmath*{\Cdb*{ex-03.301}}
   \Dmath*{\Cdb*{ex-03.302}}
   \Dmath*{\Cdb*{ex-03.303}}
\end{dgroup*}
%
\begin{dgroup*}[spread={3pt}]
   \Dmath*{\cdb{ex-03.304} = \Cdb*[\V{5pt}\hskip2.0cm\hfill]{ex-03.305}
                           = \Cdb*[\V{5pt}\hskip2.0cm\hfill]{ex-03.306}
                           = \Cdb*[\V{5pt}\hskip2.5cm\hfill]{ex-03.307}
                           = \Cdb*[\V{5pt}\hskip2.5cm\hfill]{ex-03.308}
                           = \Cdb*[\V{5pt}\hskip2.5cm\hfill]{ex-03.309}
                           = \Cdb*[\V{5pt}\hfill]{ex-03.310}
                           = \Cdb*[\V{5pt}\hfill]{ex-03.311}
                           = \Cdb*[\V{5pt}\hfill]{ex-03.312}
                           = - R^{a}{}_{dbc} V^d\quad\text{(MTW)}}
\end{dgroup*}

\clearpage

% ============================================================================================
% export to json format

\bgroup
\CdbSetup{action=hide}
\begin{cadabra}
   for i in range( len(checkpoint) ):
      cdblib.put ('check{:03d}'.format(i),checkpoint[i],checkpoint_file)
\end{cadabra}
\egroup

\end{document}


\section{Methodology}
\label{sec:Methodology}

Given an original instance $x^0$, we attempt to find a counterfactual instance $x^\prime \coloneqq x^0 +d^\prime \in \R^n$.

We first define the following loss function that encourages the predicted class $t$ of the perturbed instance $x^\prime$ to be different than the predicted class $t_0$ of the
original instance $x^0$.

\begin{equation}
   f_p(x^\prime; \kappa) \coloneqq \max \llp f(x^\prime; t_0) - \max_{t \neq t_0} f(x^\prime; t), \kappa \rlp, \label{obj: prediction}
\end{equation}
where $f(x^\prime; t)$ is the $t$-th class prediction probability, and $\kappa > 0$ caps the divergence between $x^0$ and $x^\prime$.

We also define another objective term to generate a sparse counterfactual instance that is similar to current instance:

\begin{equation}
   f_d(x^0, x^\prime) \coloneqq \beta \| x^0 - x^\prime  \|_1 +  \| x^0 - x^\prime  \|_2, \label{obj: distance}
\end{equation}
where $\beta$ is a  positive parameter that balances the trade-off between the sparsity and similarity of the counterfactual instance $x^\prime$.

To ensure the data distribution of $x^\prime$ lies closer to all neighboring instances in the same class, we introduce a third objective term:

\begin{equation}
   f_d(x^0, \llp x^j  \mid j \in K_t \rlp) \coloneqq -(1/|K_t|)  \sum_{j \in K_t} (x^j - x^0)^T (x^\prime - x^0). \label{obj: projection}
\end{equation}
Note that \eqref{obj: projection} encourages a counterfactual instance $x^\prime$ with a larger sample variance associated with samples in $K_t$. Geometrically, it defines a direction $x^\prime - x^0$ in feature space along which these data vary the most.

To define $K_t$, we need a representative, unlabeled sample of the training dataset. First the predictive model is called to label the dataset with the classes predicted by the model. Then
for each class $t$ we encode the instances belonging to that class and order them by increasing $L_2$ distance to ENC($x_0$). The $K_t$ nearest instances in the latent space are included in $K_t$.

With \eqref{obj: prediction}--\eqref{obj: projection}, we now formally define our optimization model:

\begin{subequations} \label{model: main}
\begin{align}
  \min \quad & f_p(x^\prime; \kappa) + c_{d} f_d(x^0, x^\prime) - c_{p} f_p(x^\prime, \llp x^j  \mid j \in K_t \rlp) \\
  \text{s.t.} \quad
       & x^\prime = x^0 + \sum_{j=1}^{n_k} \lambda_j d^j, \\
       & d^j = x^j - x^0, j \in K_t  \\
       & \sum_{j=1}^{n_k} \lambda_j \leq \delta, \\
       & \lambda \geq 0.
\end{align}
\end{subequations}

\subsection{Trajectory towards counterfactuals}
\label{subsec:Trajectory towards counterfactuals}

Now we present an algorithm that finds an trajectory towards the counterfactual.

\begin{algorithm}[ht!]
\caption{\textit{ConeOpt}}
\label{algorithm:ConeOpt}
\begin{algorithmic}[1]
    \REQUIRE An instance $x^0 \in \mathcal{X}$ to explain, and an index set $K_t$.
    \ENSURE A set of instances $L$ that provides a trajectory towards the counterfactual instances.
    \STATE Let $L \coloneqq \llp \phi \rlp$.
    \WHILE {Termination criteria not met}
        \STATE Solve optimization problem \eqref{model: main} to obtain a new instance $x^\prime$.
        	\STATE Update trajectory $L \coloneqq L \cup \llp x^\prime \rlp$.
        	\STATE Let $x^0 \coloneqq x^\prime$.
        \STATE Calculate the new mean of each cluster.
    \ENDWHILE
\end{algorithmic}
\end{algorithm}


\chapter{Assignment-Numerical Techniques}
\begin{enumerate}
	\item From the Taylor's series for $y(x)$, find $y(0.05)$ correct to four decimal places if $y(x)$ satisfies $y^{\prime \prime}-x y^{\prime}-y=0$ with the conditions $y(0)=1$ and $y^{\prime}(0)=0$.
 \begin{tasks}(4)
	\task[\textbf{a.}]$1.0050$
	\task[\textbf{b.}]$0.0150$
	\task[\textbf{c.}] $0.0155$
	\task[\textbf{d.}]  $0.5025$
\end{tasks}
	\item The value of $I=\int_{0}^{1} \frac{1}{1+x} d x$ using Simpson $1 / 3$ rule with $h=0.5$ is
	 \begin{tasks}(4)
		\task[\textbf{a.}]$0.69$
		\task[\textbf{b.}]$0.59$
		\task[\textbf{c.}]$0.49$
		\task[\textbf{d.}]  $0.39$
	\end{tasks}
	\item The value of the integral $\int_{0}^{8} \frac{1}{x^{2}+5} d x$, evaluated using Simpson's $\frac{1}{3}$ rule with $h=2$ is
	 \begin{tasks}(4)
		\task[\textbf{a.}]$0.59$
		\task[\textbf{b.}]$0.69$
		\task[\textbf{c.}] $0.79$
		\task[\textbf{d.}] $0.89$
	\end{tasks}
	\item Consider differential equation $y^{\prime}=-y$ with the condition $y(0)=1$. Use $-h=0.01$. The value of $y(0.02)$
	 \begin{tasks}(4)
		\task[\textbf{a.}]$0.9606$
		\task[\textbf{b.}] $0.9703$
		\task[\textbf{c.}]$0.9801$
		\task[\textbf{d.}] $0.99$
	\end{tasks}
	\item Using Euler's method solve the differential equation $y^{\prime}=-y$ with the condition $y(0)=1$ with $h=0.01$. Find $y(0.04)$
	 \begin{tasks}(4)
		\task[\textbf{a.}]$0.99$
		\task[\textbf{b.}] $0.9801$
		\task[\textbf{c.}]$0.9703$
		\task[\textbf{d.}] $0.9606$
	\end{tasks}
	\item Given the initial value problem $y^{(1)}(t)=1-t y(t)$ with $y(0)=1$, approximation of $y(1)$ by Runge-Kutta method with $\mathrm{h}=1$ would be:
	 \begin{tasks}(4)
		\task[\textbf{a.}]$1.35$
		\task[\textbf{b.}]$1.31$
		\task[\textbf{c.}]$1.32$
		\task[\textbf{d.}] $1.34$
	\end{tasks}
	\item Using the Newton's forward interpolating polynomial, $P_{5}(x)$ for the following tabular data find interpolated value of the function at $x=0.0045$.\\\\
	\begin{tabular}{|c|c|c|c|c|c|c|}
		\hline$x$ & 0 & $0.001$ & $0.002$ & $0.003$ & $0.004$ & $0.005$ \\
		\hline$y$ & $1.121$ & $1.123$ & $1.1255$ & $1.127$ & $1.128$ & $1.1285$ \\
		\hline
	\end{tabular}
	 \begin{tasks}(4)
		\task[\textbf{a.}]$-1.12840045$
		\task[\textbf{b.}]$1.12840045$
		\task[\textbf{c.}]$1.128124$
		\task[\textbf{d.}] $0.12840045$
	\end{tasks}
	\item From the Taylor's series for $y(x)$, find $y(0.1)$ correct to four decimal places if $y(x)$ satisfies\\
	$y^{\prime \prime}-x y^{\prime}-y=0$ with the conditions $y(0)=1$ and $y^{\prime}(0)=0$.
	 \begin{tasks}(4)
		\task[\textbf{a.}]$1.0050$
		\task[\textbf{b.}]$1.0150$
		\task[\textbf{c.}]$1.0155$
		\task[\textbf{d.}]$1.0250$
	\end{tasks}
	\item Compute the value of $\frac{2}{\sqrt{\pi}} \int_{0}^{x} e^{-x^{2}} d x$ when $x=0 \cdot 6538$ using Gauss's forward formula also taking origin at $0.65, h=0.01$ and $x=0.6538$, Use the given table\\
	\begin{tabular}{|l|l|}
		\hline \multicolumn{1}{|c|}{$x$} & \multicolumn{1}{c|}{$y$} \\
		\hline $0.62$ & $0.6194114$ \\
		\hline $0.63$ & $0.6270463$ \\
		\hline $0.64$ & $0.6345857$ \\
		\hline $0.65$ & $0.6420292$ \\
		\hline $0.66$ & $0.6493765$ \\
		\hline $0.67$ & $0.6566275$ \\
		\hline $0.68$ & $0.6637820$ \\
		\hline
	\end{tabular}
	 \begin{tasks}(4)
		\task[\textbf{a.}]$0.6448325$
		\task[\textbf{b.}]$0.7448325$
		\task[\textbf{c.}]$0.8448325$
		\task[\textbf{d.}]$0.5448325$
	\end{tasks}
\end{enumerate}
\section{Jacobian}
\begin{enumerate}
	\item Let $f=y^{x}$ then the value of $\frac{\partial^{2} f}{\partial x \partial y}$ at $x=2, y=1$ is
	 \begin{tasks}(4)
		\task[\textbf{a.}]0
		\task[\textbf{b.}]$\ln 2$
		\task[\textbf{c.}]1
		\task[\textbf{d.}]  $\frac{1}{\ln 2}$
	\end{tasks}
	\item If $z=x y \ln (x y)$ then which of the following is correct
	 \begin{tasks}(2)
		\task[\textbf{a.}]$x \frac{\partial z}{\partial x}+y \frac{\partial z}{\partial y}=0$
		\task[\textbf{b.}]$y \frac{\partial z}{\partial x}+x \frac{\partial z}{\partial y}=0$
		\task[\textbf{c.}]$x \frac{\partial z}{\partial x}-y \frac{\partial z}{\partial y}=0$
		\task[\textbf{d.}]  $y \frac{\partial z}{\partial x}-x \frac{\partial z}{\partial y}=0$
	\end{tasks}
	\item Let $f(x, y)=\sum_{K=1}^{10}\left(x^{2}-y^{2}\right)^{K}$ for all $(x, y)$. Then for all $(x, y)$
	 \begin{tasks}(2)
		\task[\textbf{a.}]$x \frac{\partial f}{\partial x}(x, y)-y \frac{\partial f}{\partial y}(x, y)=0$
		\task[\textbf{b.}]$x \frac{\partial f}{\partial x}(x, y)+y \frac{\partial f}{\partial y}(x, y)=0$
		\task[\textbf{c.}]$y \frac{\partial f}{\partial x}(x, y)-x \frac{\partial f}{\partial y}(x, y)=0$
		\task[\textbf{d.}] $y \frac{\partial f}{\partial x}(x, y)+x \frac{\partial f}{\partial y}(x, y)=0$
	\end{tasks}
	\item The value of $\int_{z=0}^{1} \int_{y=0}^{z} \int_{x=0}^{y} x y^{2} z^{3} d x d y d z$ is
	 \begin{tasks}(4)
		\task[\textbf{a.}]$\frac{1}{90}$
		\task[\textbf{b.}]$\frac{1}{50}$
		\task[\textbf{c.}]$\frac{1}{45}$
		\task[\textbf{d.}] $\frac{1}{10}$
	\end{tasks}
	\item In a cylindrical coordinates $x=r \cos \theta, y=r \cos \theta$ and $z=z$ then $J\left(\frac{\partial(x, y, z)}{\partial(r, \theta, z)}\right)$ is given by
	 \begin{tasks}(4)
		\task[\textbf{a.}]$r$
		\task[\textbf{b.}]$z$
		\task[\textbf{c.}]$-r$
		\task[\textbf{d.}] $-z$
	\end{tasks}
	\item If $x=u(1-v), y=u v \quad$ then $J\left(\frac{\partial(x, y)}{\partial(u, v)}\right)$ is given by
	 \begin{tasks}(4)
		\task[\textbf{a.}]$u$
		\task[\textbf{b.}]$v$
		\task[\textbf{c.}]$-u(1-2 v)$
		\task[\textbf{d.}] $u(1-2 v)$
	\end{tasks}
	\item If $u=x^{2}-2 y^{2} \quad v=2 x^{2}-y^{2} . \quad x=r \cos \theta, y=r \sin \theta \quad J\left(\frac{\partial(u, v)}{\partial(r, \theta)}\right)$
	 \begin{tasks}(4)
		\task[\textbf{a.}]$6 r^{3} \sin 2 \theta$
		\task[\textbf{b.}]$12 r^{3} \sin 2 \theta$
		\task[\textbf{c.}]$6 r^{3} \sin \theta$
		\task[\textbf{d.}] $12 r^{3} \sin \theta$
	\end{tasks}

\end{enumerate}

CSIR- Mathematical Physics/chapter/Assignments/Assignment-Numerical Techniques.tex


\section{Long exact homology sequence, Excision, and Genealogy}
WARNING: I've probably messed up typing some indices. Hopefully not.
\subsection{5-lemma}
Suppose you have two exact sequences of abelian groups.
\begin{equation*}
\xymatrix{A_4\ar[r]^d\ar[d]^{f_4} & A_3\ar[r]^d\ar[d]^{f_3} & A_2\ar[r]^d\ar[d]^{f_2} & A_1\ar[r]^r\ar[d]^{f_1} & A_0\ar[d]^{f_0}\\
B_4\ar[r]^d & B_3\ar[r]^d & B_2\ar[r]^d & B_1\ar[r]^r & B_0}
\end{equation*}
When can we guarantee that $f_2$ is an isomorphism? We're going to diagram chase :-(. Just follow your nose.

Let $b_2\in B_2$. We want to show that there is something in $A_2$ that can be pushed forward to $b_2$, i.e., prove surjectivity of $f_2$. We can consider $db_2\in B_1$. Let's assume that $f_1$ is surjective. Then there's $a_1\in A_1$ such that $f_1(a_1)=db_2$. What is $da_1$? Well, $f_0(da_1)=d(f_1(a_1))=d(db)=0$. So we want $f_0$ to be injective. Then $da_1$ is zero, so by exactness of the top sequence, there is some $a_2\in A_2$ such that $da_2=a_1$. What is $f_2(a_2)$? What is $d(f_2(a_2))$? By commutativity, $d(f_2(a_2))=f_1(d(a_2))=f_1(a_1)=db_2$. Let's consider $b_2-f_2(a_2)$. This maps to zero under $d$. So by exactness, there is $b_3\in B_3$ such that $d(b_3)=b_2-f_2(a_2)$. If we assume that $f_3$ is surjective, then there is $a_3\in A_3$ such that $f_3(a_3)=b_3$. But now, $d(a_3)\in A_2$, and $f_2(d(a_3))=d(f_3(a_3))=b_2-f_2(a_2)$. But this means that $b_2=f(a_2+d(a_3))$, which guarantees surjectivity of $f_2$. This means that if $f_1$ is surjective, $f_0$ is injective, and $f_3$ is surjective, then $f_2$ is surjective.

A similar dual process says that $f_2$ is injective if $f_1$ is injective, $f_3$ is injective, and $f_4$ is surjective. If all of these conditions are satisfied, then $f_2$ is an isomorphism. This is the content of the five-lemma.
\subsection{Relative homology}
I guess I didn't really define this. Suppose you have a pair of spaces $(X,A)$. Then you have an sexseq of chain complexes $0\to S_\ast(A)\to S_\ast(X)\to S_\ast(X,A)\to 0$.
\begin{definition}
The relative homology of the pair $ H_\ast(X,A):= H(S_\ast(X,A))$.
\end{definition}
\begin{example}
$ H_\ast(X,\emptyset)= H_\ast(X)$ because $S_\ast(\emptyset)=0$. Another case is $ H_\ast(X,X)=0$ because $S_\ast(X,X)=S_\ast(X)/S_\ast(X)=0$.
\end{example}
\subsection{General study of homologies of sexseqs of chain complexes}
Suppose I have three chain complexes $A_\bullet\to B_\bullet\to C_\bullet$. By the way, this is an important announcement. Henceforth, differentials in chain complexes will be denoted $d$, no longer $\partial$. For some reason. (It's so much easier for typing as well.) Assume that this is an exact sequence of chain complexes.

Is $ H_\ast(A)\to H_\ast(B)\to H_\ast(C)$ exact? Let's push this a little further. Suppose I have a sexseq $0\to A_\bullet\to B_\bullet\to C_\bullet\to 0$. We can ask the same question as before. Let's write this out more explicitly.
\begin{equation*}
\xymatrix{0\ar[r] & A_{n+1}\ar[r]^f\ar[d]^d & B_{n+1}\ar[r]^g\ar[d]^d & C_{n+1}\ar[r]\ar[d]^d & 0\\
0\ar[r] & A_n\ar[r]^f\ar[d]^d & B_n\ar[r]^g\ar[d]^d & C_n\ar[r]\ar[d]^d & 0\\
0\ar[r] & A_{n-1}\ar[r]^f & B_{n-1}\ar[r]^g & C_{n-1}\ar[r] & 0}
\end{equation*}
Let $[b]\in H_n(B)$ such that $g([b])=0$. It's determined by some $b\in B_n$ such that $d(b)=0$. If $g([b])=0$, then there is some $\overline{c}\in C_{n+1}$ such that $d\overline{c}=gb$. Now, $g$ is surjective, so there is some $\overline{b}\in B_{n+1}$ such that $g(\overline{b})=\overline{c}$. Then we can consider $d\overline{b}\in B_n$, and $g(d(\overline{b}))=d(\overline{c})\in C_n$. What is $b-d\overline{b}$? This maps to zero in $C_n$, so by exactness there is some $a\in A_n$ such that $f(a)=b-d\overline{b}$. Is $a$ a cycle? Well, $f(da)=d(fa)=d(b-d\overline{b})=db-d^2\overline{b}=db$, but we assumed that $db=0$, so $f(da)=0$. This means that $da$ is zero because $f$ is an injection by exactness. Therefore $a$ is a cycle. What is $[a]\in H_n(A)$? Well, $f([a])=[b-d\overline{b}]=[b]$ because $d\overline{b}$ is a cycle. Is the composite $ H_n(A)\to H_n(B)\to H_n(C)$ zero? Yes, because the composite factors through zero. This proves exactness of $ H_n(A)\to H_n(B)\to H_n(C)$.

\begin{theorem}[lexseq in homology]
Let $0\to A_\bullet\to B_\bullet\to C_\bullet\to 0$ be a sexseq of chain complexes. Then there is a natural homomorphism $\partial: H_n(C)\to H_{n-1}(A)$ such that there's lexseq:
\begin{equation*}
\xymatrix{ & & \ar[dll]^\partial\\
 H_n(A)\ar[r] & H_n(B)\ar[r] & H_n(C)\ar[dll]^\partial\\
 H_{n-1}(A)\ar[r] & H_{n-1}(B)\ar[r] & H_{n-1}(C)\ar[dll]^\partial\\
 & & &}
\end{equation*}
\end{theorem}
\begin{proof}
We'll construct $\partial$, and leave the rest as an exercise. We have our sexseq:
\begin{equation*}
\xymatrix{0\ar[r] & A_{n+1}\ar[r]^f\ar[d]^d & B_{n+1}\ar[r]^g\ar[d]^d & C_{n+1}\ar[r]\ar[d]^d & 0\\
0\ar[r] & A_n\ar[r]^f\ar[d]^d & B_n\ar[r]^g\ar[d]^d & C_n\ar[r]\ar[d]^d & 0\\
0\ar[r] & A_{n-1}\ar[r]^f & B_{n-1}\ar[r]^g & C_{n-1}\ar[r] & 0}
\end{equation*}
Let $c\in C_n$ such that $dc=0$. The map $g$ is surjective, so pick a $b\in B_n$ such that $g(b)=c$. Then consider $db\in B_{n-1}$. But $g(d(b))=0=d(g(b))=dc$. So by exactness, there is some $a\in A_{n-1}$ such that $f(a)=db$. How many choices are there of picking $a$? One, because $a$ is injective. We need to check that $a$ is a cycle. What is $d(a)$? Well, $d^2b=0$, so $da$ maps to $0$ under $f$. But because $f$ is injective, $da=0$, i.e., $a$ is a cycle. This means we can define $\partial[c]=[a]$.

To make sure that this is well-defined, let's make sure that this choice of homology class $a$ didn't depend on the $b$ that we chose. Pick some other $b^\prime$ such that $g(b^\prime)=c$. Then there is $a^\prime\in A_{n-1}$ such that $f(a^\prime)=db^\prime$. We want $a-a^\prime$ to be a boundary, so that $[a]=[a^\prime]$. We want $\overline{a}\in A_n$ such that $d\overline{a}=a-a^\prime$. Well, $g(b-b^\prime)=0$, so by exactness, there is $\overline{a}\in A_n$ such that $f(\overline{a})=b-b^\prime$. What is $d\overline{a}$? Well, $d\overline{a}=d(b-b^\prime)=db-db^\prime$. But $f(a-a^\prime)=b-b^\prime$, so because $f$ is injective, $d\overline{a}=a-a^\prime$, i.e., $[a]=[a^\prime]$. What else do I have to check? It's an exercise to check that $\partial$ as defined here is a homomorphism. Also, left as an exercise to check that this doesn't depend on $c\in[c]$, and that $\partial$ actually makes the exact sequence above exact.
\end{proof}
\subsection{Mathematical Genealogy}
I'm not typing in anything here. It's a rather big tree:
\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{assets/math-family}
\caption{Mathematical genealogy, growing from Lefschetz, who was initially a chemist. The asterisks are meant to indicate that someone's at MIT.}
\end{figure}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Written By Michael Brodskiy
% Class: Analytic Geometry & Calculus III (Math-292)
% Professor: V. Cherkassky
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[12pt]{article}
\usepackage{alphalph}
\usepackage[utf8]{inputenc}
\usepackage[russian,english]{babel}
\usepackage{titling}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{enumitem}
\usepackage{amssymb}
\usepackage[super]{nth}
\usepackage{everysel}
\usepackage{ragged2e}
\usepackage{geometry}
\usepackage{fancyhdr}
\geometry{top=1.0in,bottom=1.0in,left=1.0in,right=1.0in}
\newcommand{\subtitle}[1]{%
  \posttitle{%
    \par\end{center}
    \begin{center}\large#1\end{center}
    \vskip0.5em}%

}
\usepackage{hyperref}
\hypersetup{
colorlinks=true,
linkcolor=blue,
filecolor=magenta,
urlcolor=blue,
citecolor=blue,
}

\urlstyle{same}

\title{Lecture IX Notes}
\date{\today}
\author{Michael Brodskiy\\ \small Professor: V. Cherkassky}

% Mathematical Operations:

% Sum: $$\sum_{n=a}^{b} f(x) $$
% Integral: $$\int_{lower}^{upper} f(x) dx$$
% Limit: $$\lim_{x\to\infty} f(x)$$

\begin{document}

\maketitle

\section{Multivariable Functions $-$ 14.1}

The domain of a multivariable function, $f(x,y)$, may be found by splitting parts of the function up.

\textit{Example:}

$$f(x,y)=\ln(9-x^2-9y^2)$$
$$D(f(x,y): 9-x^2-9y^2 \geq 0\Longrightarrow x^2+9y^2\leq 9 \Longrightarrow \frac{x^2}{9}+y^2\leq 1 $$

\end{document}


\section{Respiratory Sinus Arrhythmia from RR-Intervals}
\subsection{PSD of RRI data}
After processing the ECG data to RRI data in three trails, the standard and the Bartlett averange periodograms are plotted with rectangular window length $L \in$ [$50s$, $100s$, $150s$]. Based on the analysis in section 1.2.2, the standard periodogram has relatively large variance and less leakage. With the decreasing length of window, the periodogram is tending to be smooth, resulting in both low variance and precision.
\begin{figure}[htb]
     \centering
     \begin{subfigure}{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/15/15a1.eps}
     \end{subfigure}
     ~
     \begin{subfigure}{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/15/15a2.eps}
     \end{subfigure}
     ~
     \begin{subfigure}{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/15/15a3.eps}
     \end{subfigure}
     ~
     \begin{subfigure}{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/15/15a4.eps}
     \end{subfigure}
     ~
     \begin{subfigure}{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/15/15a5.eps}
     \end{subfigure}
     ~
     \begin{subfigure}{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/15/15a6.eps}
     \end{subfigure}
        \caption{PSD of RRI data}
        \label{fig:1_5_a}
\end{figure}
\subsection{Frequency of three trails}
The experiment recorded the ECG data of three trails with normal, fast and slow breathing respectively. Therefore, the breaths per minute (BMP) for three trails will be different. A range of $12 \sim 20$ BMP is considered as the reference range for the normal breath. The observed peaks for three trails are $0.192Hz$, $0.416Hz$ and $0.124Hz$, corresponding to 23.04, 49.92 and 14.88 in BMP respectively. However, the harmonics frequencies of Trail 1 are failed to be captured which is probably caused by no restriction of the breath experiment. Trail 2 illustrate one harmonic at $f=0.832Hz$ and a large response at $0.928Hz$ at the same time, reasons of which may be the noise and the resolution of window. In addition, Trail 3 detects three harmonics frequencies at $0.248Hz$, $0.372Hz$ and $0.496Hz$.
\subsection{AR modelling of RRI data}
Fig.\ref{fig:1_5_c} depicts the AR model estimations with the incremental of order $p$. For Trail 1, the optimal order $p=10$ which detects the approximate peak at $f=0.192Hz$. The order of Trail 2 at $p=6$ can only detect the fundamental peak while harmonics can be detected with higher order. As to Trail 3, only when the order $p\ge2$, the peaks can be identified whereas the harmonics are difficult to be detected even if the order is high. Hence, the under-modelling causes the failed detection of peak and the over-modelling leads to capture harmonics and noise peaks. Compared with standard and averaging periodogram methods, the AR model performs powerfully in detecting interest peak and reducing variance as long as the order is determined.
\begin{figure}[t]
     \centering
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/15/15c1.eps}
     \end{subfigure}
     ~
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/15/15c2.eps}
     \end{subfigure}
     ~
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/15/15c3.eps}
     \end{subfigure}
        \caption{AR Model of RRI data}
        \label{fig:1_5_c}
\end{figure}


\section{The Ising Model}

\subsection{The partition function of the Ising model}

\begin{frame}
  \begin{definition}[The Ising model]
    Let $G = (V, E)$ be a graph. The partition function of the Ising model on $G$ is
    \begin{equation*}
      Z_{\text{Ising}}(G; y, z) = \sum_{\sigma \colon V \to \{0,1\}} y^{m(\sigma)} z^{n_1(\sigma)}.
    \end{equation*}
  \end{definition}

  \begin{itemize}
   \item $m(\sigma)$ is the number of monochromatic edges of $\sigma$.
   \item $n_1(\sigma)$ is the number of vertices $v$ with $\sigma(v) = 1$.
  \item $y$ is \emph{the edge interaction}.
  \item $z$ is \emph{the external field}.
  \end{itemize}

  \begin{enumerate}
  	\item Setting 1: there is no external field ($z = 1$).  \\
  	We write $Z_{\text{Ising}}(G; y) = Z_{\text{Ising}}(G; y, 1)$.
  	\item Setting 2: there is an external field ($z \ne 1$).
  \end{enumerate}

\end{frame}

\subsection{Example}

\begin{frame}
	\vspace{-4mm}
  \begin{equation*}
    Z_{\text{Ising}}(G; y, z) = \sum_{\sigma \colon V \to \{0,1\}} y^{m(\sigma)} z^{n_1(\sigma)}.
  \end{equation*}
	\vspace{-4mm}
  \begin{example}[The Ising model]
  	\vspace{2mm}
    \begin{columns}
      \column{0.2\textwidth}
      \begin{tikzpicture}
        \begin{scope}[every node/.style={circle,thick,draw}]
          \node (A) at (0,0) {A}; \node (B) at (1,0.5) {B}; \node (C) at (2,0) {C};
        \end{scope}

\begin{scope}[>={Stealth[black]},
              every edge/.style={draw=black,very thick}]
              \path [-] (A) edge node {} (B); \path [-] (A) edge node {} (C); \path [-] (B) edge
              node {} (C);
            \end{scope}
          \end{tikzpicture}

          \invisible<1-2>{
          \begin{tikzpicture}
            \begin{scope}[every node/.style={circle,thick,draw}]
              \node[fill=ChetwodeBlue] (A) at (0,0) {A}; \node (B) at (1,0.5) {B}; \node (C) at
              (2,0) {C};
            \end{scope}

\begin{scope}[>={Stealth[black]},
              every edge/.style={draw=black,very thick}]
              \path [-] (A) edge node {} (B); \path [-] (A) edge node {} (C); \path [-] (B) edge
              node {} (C);
            \end{scope}
          \end{tikzpicture}}

          \invisible<1-3>{
          \begin{tikzpicture}
            \begin{scope}[every node/.style={circle,thick,draw}]
              \node[fill=ChetwodeBlue] (A) at (0,0) {A}; \node[fill=ChetwodeBlue] (B) at (1,0.5)
              {B}; \node (C) at (2,0) {C};
            \end{scope}

\begin{scope}[>={Stealth[black]},
              every edge/.style={draw=black,very thick}]
              \path [-] (A) edge node {} (B); \path [-] (A) edge node {} (C); \path [-] (B) edge
              node {} (C);
            \end{scope}
          \end{tikzpicture}}

          \invisible<1-4>{
          \begin{tikzpicture}
            \begin{scope}[every node/.style={circle,thick,draw}]
              \node[fill=ChetwodeBlue] (A) at (0,0) {A}; \node[fill=ChetwodeBlue] (B) at (1,0.5)
              {B}; \node[fill=ChetwodeBlue] (C) at (2,0) {C};
            \end{scope}

\begin{scope}[>={Stealth[black]},
              every edge/.style={draw=black,very thick}]
              \path [-] (A) edge node {} (B); \path [-] (A) edge node {} (C); \path [-] (B) edge
              node {} (C);
            \end{scope}
          \end{tikzpicture}}
          \vspace{5mm}
          \column{0.6\textwidth}
          \begin{align*}
            \invisible<1>{Z_{\text{Ising}}(G; y, z) & = y^3 + \cdots} \\ \\
            \invisible<1-2>{Z_{\text{Ising}}(G; y, z) & = y^3 + 3yz + \cdots} \\ \\
            \invisible<1-3>{Z_{\text{Ising}}(G; y, z) & = y^3 + 3yz + 3yz^2 + \cdots} \\ \\
            \invisible<1-4>{Z_{\text{Ising}}(G; y, z) & = y^3 + 3yz + 3yz^2 + y^3z^3} \\ \\
            \invisible<1-5>{
        	Z_{\text{Ising}}(G; y) & = 2 y^3 + 6y
           }
          \end{align*}
        \end{columns}
        \invisible<6>{}
        \vspace{-3mm}
      \end{example}

 \end{frame}
    \subsection{Computational problems on the Ising model}

\begin{frame}
  \begin{proposition}
    Computing the polynomial $Z_{\text{Ising}}(G; y)$ is $\# \mathsf{P}$-hard.
  \end{proposition}

  \begin{comproblem}[$\textsc{Ising}(y, z)$]
  	\textbf{Instance:} A (multi)graph $G$. \\
  	\textbf{Output:} $Z_{\text{Ising}}(G; y, z)$.
  \end{comproblem}

  \begin{comproblem}[$\textsc{Factor-}K \textsc{-NormIsing}(y, z)$]
  	\textbf{Instance:} A (multi)graph $G$. \\
  	\textbf{Output:} $\,$ A rational number $\hat{N}$ such that
  	\begin{equation*}
  	\frac{1}{K} \hat{N} \le \left| Z_{\text{Ising}}(G; y, z) \right| \le K \hat{N}.
  	\end{equation*}
  \end{comproblem}

{\color{TurkishRose} $\textsc{Factor-}K \textsc{-NormIsing}(y, z) \le_T \textsc{Ising}(y, z)$.}

%  \begin{problem}
%    \textbf{Name:} \ \ \ $\textsc{Distance-}\rho \textsc{-Arg-Ising}(y, z)$. \\
%    \textbf{Instance:} A (multi)graph $G$. \\
%    \textbf{Output:} $\,$ A rational number $\hat{A}$ such that
%    \begin{equation*}
%      \left| \hat{A} - \arg \left( Z_{\text{Ising}}(G; y, z) \right)\right| \le \rho.
%    \end{equation*}
%  \end{problem}

\end{frame}


\chapter{Transfer Function Parameters}
\section{Jet Transfer Functions}
\section{Muon Transfer Function}
\begin{equation}
W\left(\left( \frac{q}{p_t}\right)_{\mu}, \left( \frac{q}{p_t}\right)_{p}, \eta \right) =
\frac{1}{2\pi\sigma}\mathrm{exp}
\left\{-\frac{\left[\Delta
\left( \frac{q}{p_t} \right)\right]^2}
{2\sigma^2}\right\}
\end{equation}

where the Gaussian width is defined separately for two $\eta$ regions as shown in Eq.~\ref{muonerrortf}. The $\eta$~region dependence is a result of the limited $\eta$ coverage by central fiber tracker in the forward region.

\begin{equation}
\sigma  =  \left\{
\begin{array} {c@{\quad:\quad}l} \alpha_{1} &
|\eta| \le 1.6 \\
\sqrt{\alpha^{2}_{1} + [\alpha_{2}(|\eta| - 1.6)]^2} &
|\eta| > 1.6
\end{array} \right\}
\end{equation}

The parameters $\alpha_{1}$ and $\alpha_{2}$ in the transfer function parameterization contain a constant term and a term proportional to $\frac{1}{p_{T}}$.

\begin{table}[!h!tbp]
\begin{center}
\begin{tabular}{c|cc|cc|}
\multicolumn{5}{c}
{\underline{Muon Smearing Parameters}} \\
       & \multicolumn{2}{|c|}{$<$~Dec.~2004} & \multicolumn{2}{|c|}{$>$~Dec.~2004} \\
Muon Type				&	A		&	B		&	A		&	B		\\
\hline
$>1$ SMT Hit ($\eta<1.6$)	&	0.00313	&	-0.0563	&	0.00308	&	-0.0370	\\
$>1$ SMT Hit ($\eta>1.6$)	&	0.00273	&	-0.0491	&	0.00458	&	-0.0550	\\
$=0$ SMT Hits				&	0.00509	&	-0.0916	&	0.00424	&	-0.0509	\\
\end{tabular}
\vspace{-0.1 in}
\caption{Muon smearing parameters for the function $(A + \frac{B}{p_{T}})$ for two different run periods.}
\label{muonsmearparam}
\end{center}
\end{table}

\section{Electron Transfer Function}


\subsection{Introduction to RNNs, LSTM, and ConvLSTM}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Feed-forward neural networks such as traditional ANNs and CNNs cannot learn temporal features from the data and hence are not the best choice for sequential data processing.
To handle such problems, a recurrent neural network (RNN) was introduced, which was specifically developed to address sequential data~\cite{aggarwal2018neural, Lecun2015, goodfellow2016deep}.
RNNs contain loops among the different nodes in their architecture to retain information in the model for long periods.
RNNs employ the current input with the previous memory state.
This ability of memory-keeping enables RNNs to predict what comes next.
Furthermore, RNNs were designed to handle sequential data, which implies that updating the learnable weights must consider the extent of the time dimension.
Accordingly, the backpropagation~\cite{Rumelhart1986} algorithm responsible for updating the learnable weights needs some modification to work along with the time dimension.
To alleviate this problem, backpropagation through time (BPTT)~\cite{aggarwal2018neural, goodfellow2016deep} was introduced.
In basic RNNs, short-term memories are only preserved, therefore it becomes unfeasible in the case of dealing with long sequences of data hence it may suffer from issues like vanishing or exploding gradients~\cite{bengio1994learning}.
Therefore, the fine-tuning of the model parameters and training of RNNs becomes very hard.
To overcome such issues, Hochreiter and Schmidhuber developed the Long-Short Term Memory networks (LSTMs~\cite{Hochreiter1997}).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure} [!h]
	\centering
	\begin{subfigure}[b]{1\textwidth}
		\centering
		\includegraphics[scale=1]{LSTM_img.png}
		\caption{LSTM}
		\label{fig:LSTM}
	\end{subfigure}
	 \\
	 \hfill
	\begin{subfigure}[b]{1\textwidth}
		\centering
		\includegraphics[scale=1]{convlstm_image.png}
		\caption{ConvLSTM}
		\label{fig:ConvLSTM}
	\end{subfigure}
	\caption{LSTM and ConvLSTM architectures.}
	\label{fig:lstm_convlstm}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

LSTMs were developed to keep information related to long-term dependencies, and to solve the problem of vanishing/exploding gradients.
Further, LSTMs handle inputs or outputs of any length that makes LSTMs powerful for solving very complex sequential problems.
Basic LSTM architecture shown in Fig.~\ref{fig:LSTM}  consists of four units: an input gate, a cell state, a forget gate, and an output gate.
These gates help regulate the flow of information that is added to or removed from the cell state.
The hidden states in LSTM hold the short-term memory, while the cells state holds the long-term memory.

The purpose of the forget gate is to decide what information to keep and what to neglect.
The current input \(X_{t}\) and the previous hidden state  \(h_{t-1}\) are passed through a sigmoid function which will produce values between \(0\) and \(1\).
Then the outputs of the sigmoid are multiplied with the previous cell state \(C_{t-1}\), accordingly, (\(0\)) outputs are discarded.
The mathematical calculation at the forget gate ($f_t$) is depicted in Eq.~(\ref{eq:eq1}):
\begin{equation}
f_{t}=\sigma\left(W_{f} \cdot\left[h_{t-1}, X_{t}\right]+b_{f}\right)
\label{eq:eq1}
\end{equation}
where \(W\) represents the learnable weights, and \(b\) represents the bias term.

The input gate \(i_{t}\) takes the current input \(X_t\) with the previous hidden state \(h_{t-1}\) then apply the sigmoid function to get values in a range between 0 (not important) and 1 (important), then the
same current input \(X_t\), and the hidden state \(h_{t-1}\) are passed through a \(\tanh\) function at \(\tilde{C}_{t}\) that will regulate the network by transferring the values into a range between \(-1\) and \(1\).
Then, the outputs from the sigmoid and \(\tanh\) functions are multiplied point-by-point to eliminate \(0\) values.
Equation~(\ref{eq:eq2}) depicts the calculation at the input gate:
\begin{equation}
\begin{aligned}
i_{t} &=\sigma\left(W_{i} \cdot\left[h_{t-1}, X_{t}\right]+b_{i}\right)
\\
\tilde{C}_{t} &=\tanh \left(W_{s} \cdot\left[h_{t-1}, X_{t}\right]+b_{c}\right)
\end{aligned} \label{eq:eq2}
\end{equation}
At this point, the network has sufficient information obtained from the input and forget gates.
Hence, the current cell state \(C_t\) can be calculated by multiplying the previous cell state \(C_{t-1}\) with the output of the forget gate, then the result is added to the calculated input values as depicted in Eq.~(\ref{eq:eq3}):
\begin{equation}
C_{t}=f_{t} * C_{t-1}+i_{t} * \tilde{C}_{t}
\label{eq:eq3}
\end{equation}
The output gate \(o_{t}\) computes the next hidden state \(h_{t}\) which
holds information related to the current inputs.
Accordingly, the current input \(X_{t}\) and the previous hidden state \(h_{t-1}\) are passed through a third sigmoid function to produce values between \(0\) and \(1\).
The current cell state \(C_{t}\) is passed through a \(\tanh\) function and multiplied point-by-point with \(o_{t}\) to produce the new hidden state \(h_{t}\) which is transferred to the next timestamp.
Equation~(\ref{eq:eq4}) illustrates the calculations at the output gate:
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{equation}
\begin{aligned}
o_{t} &=\sigma\left(W_{o}\left[h_{t-1}, X_{t}\right]+b_{o}\right) \\
h_{t} &=o_{t} * \tanh \left(C_{t}\right)
\end{aligned}
\label{eq:eq4}
\end{equation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Recently, LSTMs have been widely used for large-scale learning of language translation models, speech recognition systems, chatbots, forecasting stock markets, text data analysis, and many more~\cite{graves2014towards, cho2014properties}.
However, LSTMs are inefficient regarding capturing spatial information by themselves when the time series inputs are consecutive images.
Accordingly, ConvLSTM layer which is a combination of CNN and LSTM unit was introduced by Shi et al.~\cite{xingjian2015convolutional} to solve such a problem.
For ConvLSTM, the convolution operations are applied both at the input-to-state transition and at the state-to-state transitions.
ConvLSTM shown in Fig.~\ref{fig:ConvLSTM} is a variation of the LSTM cell as it performs a convolution operation within the LSTM cell.
ConvLSTM is a combination of a convolution operation and an LSTM cell.
Thus, ConvLSTM can capture the time-correlated and spatial features in a series of consecutive images.
Equation~(\ref{eq:eq5}) depicts the ConvLSTM operations as the inputs \(X_1, \dots, X_t\), hidden states \(h_1, \dots, h_t\), cell states \(C_1, \dots, C_t\) and input, forget and output gates are represented as \(i_t, f_t\), and \(o_t\), respectively:
\begin{equation}
	\begin{aligned}
		i_{t} &=\sigma\left(W_{x i} * X_{t}+W_{h i} * h_{t-1}+W_{c i} \odot C_{t-1}+b_{i}\right)
		\\
		f_{t} &=\sigma\left(W_{x f} * X_{t}+W_{h f} * h_{t-1}+W_{c f} \odot C_{t-1}+b_{f}\right) \\
		C_{t} &=f_{t} \odot C_{t-1}+i_{t} \odot \tanh \left(W_{x c} * X_{t}+W_{h c} * h_{t-1}+b_{c}\right)
		\\
		o_{t} &=\sigma\left(W_{x o} * X_{t}+W_{h o} * h_{t-1}+W_{c o} \odot C_{t}+b_{o}\right) \\
		h_{t} &=o_{t} \odot \tanh \left(C_{t}\right)
	\end{aligned}
	\label{eq:eq5}
\end{equation}
where \(*\) indicates the convolution operation, and \(\odot\) represents the
Hadamard product.
Recently, ConvLSTM has become very popular and is increasingly being used in
more and more image processing applications.


\section{Task 2}

For this task we are given the following code:
\begin{figure}
    \begin{lstlisting}
for i from 0 to N-1 // outer loop
    accum = A[i,0] * A[i,0];
    B[i,0] = accum;
    for j from 1 to 63 // inner loop
        tmpA = A[i, j];
        accum = sqrt(accum) + tmpA*tmpA;
        B[i,j] = accum;
    \end{lstlisting}
    \caption{Pseudocode given for the task.}
    \label{fig:t2code1}
\end{figure}

\subsection{Task 2.a}

\begin{itemize}
    \item Because both \texttt{accum} and \texttt{tmpA} are declared outside the
    loop, the outer loop cannot be parallel since the different threads will
    attempt to write to the same memory locations.

    \begin{figure}
        \begin{lstlisting}
for i from 0 to N-1 // outer loop
    float accum = A[i,0] * A[i,0];
    B[i,0] = accum;
    for j from 1 to 63 // inner loop
        float tmpA = A[i, j];
        accum = sqrt(accum) + tmpA*tmpA;
        B[i,j] = accum;
        \end{lstlisting}
        \caption{Code rewritten for allow for parallelisation of the outer
            loop.}
        \label{fig:t2code2}
    \end{figure}

    \item Since \texttt{accum} is declared outside the inner loop, it is not
    parallelisable since different threads would all try to access the same
    instance of \texttt{accum}.

\end{itemize}

\subsection{Task 2.b OpenMP Code}
The code for this can be found in \textit{src/task2openMP.cpp} and
\textit{src/cpuFunc.cu.h}. It can be built from the makefile using the
texttt{make bonus} command/target. It can then be run by running the
\textit{task2openMP} file. Execution times for different matrix sizes can be
seen in Table \ref{tab:task2times} along with execution times for the following
tasks.

\subsection{Task 2.c Naive GPU Code}
The code for this task can be found in \textit{src/task2.cu},
\textit{src/cpuFunc.cu.h}, \textit{src/gpuFunc.cu.h}. The kernel that is used
for this task is \texttt{flatNaiveTask2Kernel}. Execution times for different
matrix sizes can be seen in Table \ref{tab:task2times} along with execution
times for the following tasks.

\subsection{Task 2.d Transposed GPU Code}
The code for this task can be found in \textit{src/task2.cu},
\textit{src/cpuFunc.cu.h}, \textit{src/gpuFunc.cu.h}. The kernel that is used
for this task is \texttt{flatTransposedTask2Kernel}. It also uses the
\texttt{flatSharedTransposeKernel} from Task 1. Execution times for different
matrix sizes can be seen in Table \ref{tab:task2times} along with execution
times for the following tasks.

\subsection{Execution Time}
All running times were taken over 100 repetitions of each matrix size for each
different method. All the times we're taken running on one of the servers we
were given access to as part of the course.

\begin{table}[H]
\centering
\begin{tabular}{|r|r|r|r|r|}
\hline
\textbf{Dimensions} & \textbf{CPU} & \textbf{CPU OpenMP} & \textbf{GPU Naive} & \textbf{GPU Transposed} \\ \hline
$10 \times 64$      & $12\mu s$    & $185\mu s$          & $51\mu s$          & $69\mu s$               \\
$20 \times 64$      & $23\mu s$    & $11\mu s$           & $67\mu s$          & $68\mu s$               \\
$30 \times 64$      & $34\mu s$    & $11\mu s$           & $85\mu s$          & $70\mu s$               \\
$40 \times 64$      & $45\mu s$    & $12\mu s$           & $88\mu s$          & $68\mu s$               \\
$50 \times 64$      & $57\mu s$    & $12\mu s$           & $87\mu s$          & $70\mu s$               \\
$60 \times 64$      & $68\mu s$    & $12\mu s$           & $89\mu s$          & $70\mu s$               \\
$70 \times 64$      & $80\mu s$    & $13\mu s$           & $88\mu s$          & $74\mu s$               \\
$80 \times 64$      & $91\mu s$    & $13\mu s$           & $89\mu s$          & $69\mu s$               \\
$90 \times 64$      & $102\mu s$   & $13\mu s$           & $88\mu s$          & $71\mu s$               \\
$100 \times 64$     & $114\mu s$   & $14\mu s$           & $88\mu s$          & $74\mu s$               \\
$2000 \times 64$    & $2277\mu s$  & $106\mu s$          & $166\mu s$         & $129\mu s$              \\
$3000 \times 64$    & $3402\mu s$  & $135\mu s$          & $188\mu s$         & $166\mu s$              \\
$4000 \times 64$    & $4555\mu s$  & $179\mu s$          & $210\mu s$         & $200\mu s$              \\
$5000 \times 64$    & $5687\mu s$  & $222\mu s$          & $143\mu s$         & $145\mu s$              \\
$10000 \times 64$   & $11373\mu s$ & $436\mu s$          & $382\mu s$         & $216\mu s$              \\ \hline
\end{tabular}
\caption{This table shows the execution times for different implementations of
    the algorithm in this task. Each time is the average over 100 runs.}
\label{tab:task2times}
\end{table}

The first reading for the openMP solution is off, I have tried to re do the
reading several times and also increase the number of repetitions in order to
try and eliminate delays caused by a cold cache, but the reading is persistent.
This leads me to conclude that the overhead for creating the openMP threads are
much higher than the time saved by parallelising for a matrix with only 10 rows.


%!TEX root =  ../../master.tex
\subsection{Dispersion L\"uscher in 2 dimensions}

The discussion above is valid only in the continuum.
For a discretized lattice, an additional length scale is introduced that must be accounted for.
As is the case in both 3-D and 1-D, there exists a dispersion L\"uscher equation that is valid for the contact interaction and accounts for the discretization.
In \Appref{3D dispersion} we derive this dispersion L\"uscher formula for 2D and only state the result here.

Identifying the lattice spacing $\epsilon=N/L$, we have
\begin{align}
    \frac{2}{\pi} \log \left(\frac{2\pi \tilde a_{2}}{L}\right)
    &=
    \frac{1}{\pi^2}
    \left(
        \sum_{n_x,n_y=-\frac{N}{2}}^{\frac{N}{2}-1}\frac{1}{\tilde{K}^N_{nn}-x^\dispersion}
        -2\pi \log \left(\mathcal{L}^\dispersion_2\frac{N}{2}\right)
    \right)
    \nonumber\\
    &\equiv\frac{1}{\pi^2}S^{\dispersion}_2\left(x^\dispersion\right)\, ,\label{eq:2d dispersion luscher}
	\qquad
    \mathcal{L}^\dispersion_{2}
    =
    \exp \left(\log (2)-G \frac{2}{\pi}\right)
    =
    1.116306393581637659468497 \ldots
\end{align}
where $G$ is Catalan's constant.

To demonstrate the success of this formula we tuned lattices with $N=10$, 20, and 40 to $\tilde a_{2}/L = {1}/{10}$, which allows for a bound state, using $S^{\dispersion}$.
In \Figref{luescher2d} the black points were analyzed through $S^\dispersion_2$, and lie on a flat line, indicating that our dispersion \Luscher formula has correctly accounted for discretization effects.
On the other hand, if we use the same energies but analyze them with the usual continuum \Luscher function $S^{\spherical}_2$, shown as colored points, we see induced momentum-dependence and the flat line behavior is lost.

\begin{figure}
    \center
    \scalebox{0.8}{\input{figure/2d.pgf}}
    \caption{
    	Phase shifts and zeta functions in the spherical and dispersion scenario in two dimensions.
		\\
		\textit{(top)}
        	Finite-spacing eigenvalues $x^\dispersion=2\mu E^\dispersion L^2/4\pi^2$ of the Schr\"odinger equation are inserted into respective zeta functions to obtain phase shifts.
			The eigenvalues are obtained for a contact interaction analytically determined for the spherical case~\eqref{C2} and the dispersion case~\eqref{C2-dispersion}.
			Both tunings are fixed to $\tilde a_{2}/L=1/10$ (closed symbols) and $\tilde a_{2}/L=0$ (open symbols).
			Different markers correspond to different discretizations: $N=4$ (triangles), 10 (squares), 12 (diamonds), and 14 (hexagons).
        	For analysis, the colored points are obtained using $S^\spherical_2(x^\dispersion)$; corresponding to $N=4$ (red), 10 (green), 20 (blue), and 40 (purple).
        	The thin colored lines are the derived induced momentum-dependent terms for each $N$ as given in \Tabref{induced terms in 2 d}.
        	The black points are obtained using the $N$-appropriate $S^{\dispersion}_2(x^\dispersion)$ and exhibit the correct flat-line behavior.
        	The dashed gray line is $S_2^\spherical$, as in the bottom panel.
        \\
        \textit{(bottom)}
        	Two one-dimensional zeta functions, the spherical function $S_2^{\spherical}$ (light gray) given in \eqref{2d luscher} and $S_2^{\dispersion}$ (red) given in \eqref{2d dispersion luscher} with $N=4$ and $\nstep=\infty$.
        	The difference between the dispersion and spherical curves is responsible for moving the red triangles to the black triangles in the top panel.
	}
    \label{fig:luescher2d}
\end{figure}

As was done in the three- and one-dimensional cases, we can derive the functional form of the induced momentum-dependent terms.
The derivation is identical to those cases; for concision we show only the end result.
Expanded around small $x^\dispersion$ one finds
\begin{equation}
    \label{eq:2D corrections}
    S^\bigcirc_{2}\left(x^\dispersion\right)
    =
    2\pi\log\left(2\pi \frac{\tilde a_{2}}{L}\right)
    + \alpha_1(N)
    + \alpha_2(N) x^\dispersion
    + \alpha_3(N) (x^\dispersion)^2
    + \ldots
\end{equation}
The coefficients $\alpha_i(N)$ have an implicit dependence on $N$ since the sums are restricted outside of the Brillouin zone.  Further, in 2-D the sums involved in $\alpha_i$ converge sufficiently fast and there exist various techniques for evaluating these sums (see Appendices A of \Ref{Fiebig:1994qi,Beane:2010ny}).
We provide the numerical values of $\alpha_i(N)$ in \Tabref{induced terms in 2 d} for the discretizations shown in \Figref{luescher2d}.
These functions were also used to calculate the thin colored lines in \Figref{luescher2d}, where we see the small-$x$ expansion lose accuracy quickly for $N=4$ (consider the bound state, for example) but hold deeper into the spectrum for larger $N$.

\begin{table}
    \caption{The coefficients $\alpha_i$ of the induced momentum-dependent terms in \eqref{2D corrections} due to a contact interaction using $S^\bigcirc_2(x^\dispersion)$ as a function of discretization $N$, assuming $\nstep=\infty$.
    }
    \label{tab:induced terms in 2 d}
    \begin{tabular}{S[table-format=2.1]S[table-format=2.6]S[table-format=2.7]S[table-format=2.8]}
{$N$} & {{$\alpha_1$}} & {$\alpha_2$} & {$\alpha_3$} \\ \midrule
    4   &   0.20642   &   0.726184 & 0.0937105           \\
    10  &   0.03402   &   0.105117 & 0.0018607           \\
    20  &   0.00086   &   0.025850 & 0.0001108           \\
    40  &   0.00851   &   0.006433 & 0.0000068    \\
\end{tabular}
\end{table}