\documentclass{article}

\usepackage{epsfig}
\usepackage[export]{adjustbox}% http://ctan.org/pkg/adjustbox
\usepackage{caption}
\usepackage{subcaption}
\usepackage{fullpage}
\usepackage{commath}
\usepackage{amssymb}
\usepackage[space]{grffile}
\usepackage{algorithm, algorithmic}

\def\E{\mathbb{E}}
\def\R{\mathbb{R}}
\def\bin{\text{bin}}
\def\bet{\text{beta}}
\def\range{\text{range}}
\def\vec{\text{vec}}

\newcommand\hide[1]{}

\title{HMM with Contexts in DNA Methylation Problem}

\author{Chicheng Zhang}

\begin{document}
\maketitle

\section{The Model}

We are given sequences $\cbr{m_t, c_t}$, where at each time $t$ the methylation count is $m_t \in \cbr{0,1,\ldots,n}$, and the context information is $c_t \in [c]$. We model the data using a hidden Markov model with context information, that is, the underlying dynamics of the hidden states is driven by a Markov chain $\cbr{y_t}$, where $y_t \in [m]$.

It can be described by parameters $(\pi, T, O)$, where $\pi$ is the initial probability distribution $\Pr(y_1)$ and $T$ is the transition matrix of the Markov chain $\Pr(y_{t+1} | y_t)$, $O$ is the observation matrix, namely the conditional probability of methylation given hidden state and context $\Pr(m_t | c_t, y_t)$. See Figure~\ref{fig:contexthmm} for illustration.

\begin{figure}[h]
\centering
\scalebox{0.3}
{
\includegraphics{hmm.png}
}
\caption{Modeling DNA methylation using a Hidden Markov model with context information.}
\label{fig:contexthmm}
\end{figure}

Our objective is to recover parameters $\pi$, $T$ and $O$ based on iid samples drawn from this model.

\section{Algorithm based on Method of Moments}

For each $i$, consider recovering a block of observation matrix $O^{2, i} = \Pr(m_t | c_t = i, y_t)$ separately. We make the assumption that $O^{2, i}$ is of full column rank. Then, the joint cooccurence tensor of $(m_1, m_2, m_3)$, when the context of observation 2 $c_2$ is $i$, has the following structure:
\[ \E [m_1 \otimes m_2 \otimes m_3 | c_2 = i] = \sum_j \omega_j O^{1}_j \otimes O^{2, i}_j \otimes O^{3}_j \]
where $\omega$ is the distribution of $y_2$ given $c_2$, $O^{1} = \E[m_1 | y_2]$, $O^{2,i} = \E[m_2 | y_2, c_2 = i]$, $O^{3} = \E[m_3 | y_2]$.
At this point, standard tensor decomposition method can readily be applied to extract the columns of $O^{2, i}$.

Alternatively, we have
\[ \E [m_1 \otimes m_3] = \sum_j \omega_j O^1_j \otimes O^3_j \]
and
\[ \E [m_1 \otimes m_2 | c_2 = i] = \sum_j \omega_j O^{1}_j \otimes O^{2, i}_j \]
Then standard symmetrization and orthogonalization routine can be applied to recover the columns of $O^{2, i}$.

\section{Binomial Hidden Markov Model}
In this section, we consider HMMs whose observation matrix has structured columns, that is, columns which are probability mass functions of binomial distribution. We first take a step back, considering HMMs without context information. (The approach can be easily generalized to the setting with context information.)

In this model, each element of $O$ can be written as $O_{i,j} = {n \choose i} p_j^i (1-p_j)^{n-i}$, i.e. the probability that there are $i$ heads out of $n$ trials when flipping a coin with bias $p_j$. Instead of recovering $O$, we now turn to the task of recovering $p_j, j = 1,2,\ldots,m$. A naive approach would be to first recover the columns of $O$, e.g. $O_j$, then get $p_j$ using the formula:

\[ p_j = \sum_{i=0}^n \frac{i}{n} {n \choose i} p_j^i (1-p_j)^{n-i} = \sum_{i=0}^n \frac{i}{n} O_{i,j} \]

\subsection{Exploiting Similarity between Observations}
Sometimes it will be beneficial to consider feature representations of observation other than one-hot encoding. Define feature map $\phi: \cbr{0,\ldots,n} \to \R^D$, where $D$ can potentially be very large, even (uncountably) infinite. For example, $\phi(m) = \cbr{t^m (1-t)^{n-m}}_{t \in [0,1]}$, $\phi(m) = \cbr{t^m (1-t)^{n-m}}_{t \in \cbr{0,0.1, \ldots, 1}}$, or $\phi(m) = \cbr{ \frac{1}{\sqrt{2\pi}} \exp(-(t-m)^2/2)}_{t \in \R}$, etc.

Then, standard method of moments can be applied, resulting in the following representation:
\[ \E [\phi(m_1) \otimes \phi(m_2) \otimes \phi(m_3)] = \sum_j \omega_j O^{1}_j \otimes O^{2}_j \otimes O^{3}_j \]
where $\omega$ is the marginal distribution of $y_2$, $O^{1} = \E[\phi(m_1) | y_2]$, $O^{2} = \E[\phi(m_2) | y_2]$, $O^{3} = \E[\phi(m_3) | y_2]$.

Now, to decompose this $D \times D \times D$ tensor, dimensionality reduction tricks should be applied. After getting an estimate of $O^{2} = \E[\phi(m_2) | y_2]$, we can recover $p_j$'s via algebraic manipulations.

\hide
{
\section{Experiment Results}
\subsection{Setting}
In this set of experiments, we focus on the methylation data in cell type E1 and chromosome 1. At this point we do not take into account the contexts, and we further assume that the coverage in different segments are non-uniform. Let $l$ be the length of the input sequence, $s$ be the number of base segments merged to a single segment, and $n$ be the number of hidden states assumed.

Although the ultimate goal is to recover the transition matrix and observation matrix of the hidden Markov model, at the current stage, we recover $\E[\phi(x, y)|h]$ for each value of $h$. Recall that $x = (c, \mu)$ is the (coverage, methylation) count pair in each segment. The value of $p_h$ for each $h$ can be straightforwardly extracted from $\E[\phi(x, y)|h]$. The transition matrix can also be computed subsequently. We leave the recovery of all the parameters for future work.

%In subsequent results, we show the entire $\E[\phi(x,y)|h]$, as opposed to a single number $p_h \in [0,1]$. The reason is that we wish to recover more information about the hidden states learned, to get a better understanding of the HMM model.

%-- what features $\phi$ are you looking at?
%-- why are you showing the entire $\phi$ vector and not a single number?
We focus on two types of feature mappings $\phi$, the first one is binning mapping:
\[ \phi_{\bin}(x,y) = \begin{cases} {I(\frac{\mu}{c} \in (y,y+h])} & c \neq 0 \\ 0 & c = 0\end{cases} \]
where $h$ is the width of the bins.

The second one is Beta map:
\[ \phi(x,y) = \frac{1}{B(\mu+1, c-\mu+1)}y^\mu (1-y)^{c-\mu} \]
In this mapping there is no explicit notion of ``bin width", but the value of $c$ can be thought of as a parameter controlling it: if the coverage $c$ is large, then the bin width is small, and vice versa.

In experiments, since it is impossible to write down $\E[\phi(x_1, \cdot) \otimes \phi(x_2, \cdot) \otimes \phi(x_3, \cdot)]$ and perform tensor decomposition, we use discretization: compute an empirical version of $\E[\phi(x_1, y_1) \otimes \phi(x_2, y_2) \otimes \phi(x_3, y_3)]$, where $y_1, y_2, y_3$ is in $\cbr{0,h,2h,\ldots,1}$. By decomposing this tensor we are able to recover $\E[\phi(x, y) | h]$, where $y$ is in $\cbr{0,h,2h,\ldots,1}$. By mild regularity assumption on the distribution of $x$, this can give a fairly accurate estimate of $\E[\phi(x, y) | h]$ where $y$ ranges in the $[0,1]$ interval.

\subsection{Effect of Sample Size $l$}
We fix $m = 4$, $s = 1$, and vary the sample size (number of triples collected) $l =$ 1000, 10000, 20000, 40000, 80000, 160000, 320000. The samples are drawn without replacement from the original dataset. Figure~\ref{fig:varyl} shows the result. Note that the value of $m$ is the number of hidden states assumed; for example if $m = 4$, then we get $4$ columns of $\E[\phi(x,y)|h]$. We observe that the output of the algorithm stablizes when $l \geq 80000$.

\begin{figure}[H]

    \begin{tabular}{cc}
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_l/cell = E1_chr = 1_l = 10000_s = 1_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$l = 10000$}
    \end{subfigure}
    &
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_l/cell = E1_chr = 1_l = 20000_s = 1_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$l = 20000$}
    \end{subfigure}
    \\
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_l/cell = E1_chr = 1_l = 40000_s = 1_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$l = 40000$}
    \end{subfigure}
    &
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_l/cell = E1_chr = 1_l = 80000_s = 1_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$l = 80000$}
    \end{subfigure}
    \\
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_l/cell = E1_chr = 1_l = 160000_s = 1_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$l = 160000$}
    \end{subfigure}
    &
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_l/cell = E1_chr = 1_l = 320000_s = 1_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$l = 320000$}
    \end{subfigure}
    \\
    \end{tabular}

    \caption{The observation columns recovered $\E[\phi(x,y)|h]$ in terms of $h$, for varying sample size $l$. The $x$-axis is the value of $t$, the $y$-axis is the value of $\E[\phi(x,y)|h]$.}
    \label{fig:varyl}
\end{figure}

\subsection{Effect of Specifying the Number of States}

We fix $l = 320000$, $s = 1$, and vary the number of states $m = 2,3,4,5,6,7,8,9$. Figure~\ref{fig:varym} shows the columns of $\E[\phi(x,y)|h]$ recovered. The spectral algorithm provides reasonable results when $m \leq 4$; when $m \geq 5$, the algorithm started to recover observation columns that has a lot of negative entries.

\begin{figure}[H]

    \begin{tabular}{cc}
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_l/cell = E1_chr = 1_l = 320000_s = 1_m = 1_n = 30_phi = beta_full.pdf}
        \caption{$m = 1$}
    \end{subfigure}
    &
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_l/cell = E1_chr = 1_l = 320000_s = 1_m = 2_n = 30_phi = beta_full.pdf}
        \caption{$m = 2$}
    \end{subfigure}
    \\
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_l/cell = E1_chr = 1_l = 320000_s = 1_m = 3_n = 30_phi = beta_full.pdf}
        \caption{$m = 3$}
    \end{subfigure}
    &
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_l/cell = E1_chr = 1_l = 320000_s = 1_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$m = 4$}
    \end{subfigure}
    \\
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_l/cell = E1_chr = 1_l = 320000_s = 1_m = 5_n = 30_phi = beta_full.pdf}
        \caption{$m = 5$}
    \end{subfigure}
    \end{tabular}

    \caption{The observation columns recovered $\E[\phi(x,y)|h]$ in terms of $h$, for varying number of states $m$. The $x$-axis is the value of $t$, the $y$-axis is the value of $\E[\phi(x,y)|h]$.}
    \label{fig:varym}
\end{figure}

\subsection{Binning Feature vs. Beta Feature}

We compare the experimental results using two types of feature maps $\phi_{\bin}$ and $\phi_{\bet}$. Although the two mappings are substantially different, their respective $\E[\phi(x,y)|h]$ have some similarities. In some sense, the beta mapping is performing a ``soft'' binning which takes into account the number count $c$ in $x$: fixing the value of methylation probability $m/c$, if $c$ is larger, then the beta mapping is closer to a binning mapping with smaller bin size. Figure~\ref{fig:varyphi} shows the columns of the resulting $\E[\phi(x,y)|h]$ recovered. Generally, $\phi_{\bet}$ produces smoother observation columns.

%-- double check this one experiment -- typo in file naming?

\begin{figure}[H]
    \begin{tabular}{cc}
        \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_phi/cell = E2_chr = 1_l = 320000_s = 1_m = 3_n = 30_phi = beta_full.pdf}
        \caption{$\phi = \phi_{\bet}$ is beta mapping, $m = 3$}
    \end{subfigure}
    &
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_phi/cell = E2_chr = 1_l = 320000_s = 1_m = 3_n = 30_phi = binning_igz.pdf}
        \caption{$\phi = \phi_{\bin}$ is binning mapping, $m = 3$}
    \end{subfigure}
    \\
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_phi/cell = E2_chr = 1_l = 320000_s = 1_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$\phi = \phi_{\bet}$ is beta mapping, $m = 4$}
    \end{subfigure}
    &
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_phi/cell = E2_chr = 1_l = 320000_s = 1_m = 4_n = 30_phi = binning_igz.pdf}
        \caption{$\phi = \phi_{\bin}$ is binning mapping, $m = 4$}
    \end{subfigure}
    \end{tabular}

    \caption{The observation columns recovered $\E[\phi(x,y)|h]$ in terms of $h$, for $m = 4$ and two types of feature map $\phi$. The $x$-axis is the value of $t$, the $y$-axis is the value of $\E[\phi(x,y)|h]$.}
    \label{fig:varyphi}
\end{figure}

\subsection{Effect of Number of Segments Combined}
We fix $l = 320000$, $m = 4$, and vary the number of merged segments $s = 1,2,3,4,5,6,7,8$. Figure~\ref{fig:varys} shows the columns of the resulting $\E[\phi(x,y)|h]$. It can be seen that the recovered result is fairly insensitive to the choice of $s$.

\begin{figure}[H]
    \begin{tabular}{cc}
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_s/cell = E1_chr = 1_l = 320000_s = 1_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$s = 1$}
    \end{subfigure}
    &
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_s/cell = E1_chr = 1_l = 320000_s = 2_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$s = 2$}
    \end{subfigure}
    \\
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_s/cell = E1_chr = 1_l = 320000_s = 3_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$s = 3$}
    \end{subfigure}
    &
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_s/cell = E1_chr = 1_l = 320000_s = 4_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$s = 4$}
    \end{subfigure}
    \\
    \begin{subfigure}[t]{0.45\textwidth}
        \includegraphics[width=\textwidth]{figs/vary_s/cell = E1_chr = 1_l = 320000_s = 5_m = 4_n = 30_phi = beta_full.pdf}
        \caption{$s = 5$}
    \end{subfigure}
    \end{tabular}

    \caption{The observation columns recovered $\E[\phi(x,y)|h]$ in terms of $h$, for $m = 4$ and two types of feature map $\phi$. The $x$-axis is the value of $t$, the $y$-axis is the value of $\E[\phi(x,y)|h]$.}
    \label{fig:varys}
\end{figure}
}

\section{Possible Variants of Model}

\begin{figure}
  \caption{Possible Variants of the Model}
  \centering
  \includegraphics[width=0.8\textwidth]{model_doubt.jpg}
  \label{fig:modeldoubt}
\end{figure}

In the data given, we are not able to observe the context information on exact positions.
Instead, the data is grouped every 100 base pairs, where in each group, we observe
32 numbers $\cbr{(m_{t,i}, c_{t,i})}_{i=1}^{16}$, two for each context, representing
total coverage and total methylation in context-location specific manner.

We can model the data in the following two graphical models (see Figure~\ref{fig:modeldoubt}). The first one models
the sequence in each context separately. The hidden states are not only context specific,
but also location specific. It is not clear -- at least at this moment -- if we should
align the hidden states across different contexts.

The second model is to put the hidden state as the shared information across different
contexts. Then it assumes that given the hidden states, the 16 observations are conditionally
independent. We suspect this is too strong, and it might be preferable to assume the
16 observations to have dependency and directly build a wholistic feature map based
on them altogether.

\section{Efficient Joint Parameter Estimation of Multiple HMMs}
In this section we study the following probem.
Suppose we are given sequences drawn from two HMMs, and we have the prior knowledge that
the parameters of the two HMMs are close to each other.
Furthermore, suppose we have computed the moments from training data
(denoted as $P_{ij}, \tilde P_{ij}$, $P_{ijk}, \tilde P_{ijk}$ for all $(i,j,k)$ pairs).
The goal is to perform method of moments estimating the parameters of the two HMMs.

It is known that
\[ P_{i,j} = \E[x_i \otimes x_j] = \sum_{l=1}^m w_l (O_i)_l \otimes (O_j)_l \]
and
\[ P_{i,j,k} = \E[x_i \otimes x_j \otimes x_k] = \sum_{l=1}^m (O_i)_l \otimes (O_j)_l \otimes (O_k)_l \]

\subsection{Algorithmic Primitive}
\paragraph{Matrix Regularized Least Squares} In subsequent discussions, the following optimization problem is often considered:
\[ \min_{B \in \R^{n \times k}} f(B) = \| M - A B^T \|_F^2 + \lambda \| B - \tilde{B} \|_F^2 \]
where $M \in \R^{m \times n}$, $A \in \R^{m \times k}$ are given.

This optimization problem an be solved by noticing it can be decomposed to functions with
respect to rows of $B$, that is, $B^{1}, \ldots, B^{m}$.
This gives rise to the following optimization problem:
\[ \min_{B^i \in \R^k} \| M_i - A (B^i)^T \|^2 + \lambda \| B^i - \tilde B^i \|^2 \]
Taking derivatives, we have
\[ 2 A^T A (B^i)^T - 2 A^T M_i + 2 \lambda ((B^i)^T - (\tilde B^i)^T) = 0 \]
that is,
\[ (B^i)^T = (\lambda I + A^T A)^{-1} (A^T M_i + \lambda (\tilde B^i)^T ) \]
horizonally stacking the solution, we get
\[ B^T = (\lambda I + A^T A)^{-1} (A^T M + \lambda \tilde B^T) \]
that is, $B = (M^T A + \lambda B)(\lambda I + A^T A)^{-1}$.
(As a sanity check, when $\lambda = 0$, $B = (A^+ M)^T$, which is the correct solution.)

\paragraph{Weighted Extensions} It is sometimes desirable to consider the following
optimization objective:
\[ \min_{B \in \R^{n \times k}} f(B) = \| M - A B^T \|_F^2 + \lambda \| C (B - \tilde{B}) \|_F^2 \]
where $C \in \R^{p \times n}$ is a linear transformation (the objective aims to measure the
$\ell_2$ similarity of transformed $B$.)
Unfortunately, the solution does not admit a succinct representation. But we can still
write the solution. First, let $\vec(B) = ((B^1)^T, (B^2)^T, \ldots, (B^n)^T)$.
We rewrite the objective as:
\[ f(B) = \sum_{i=1}^n \| M_i - A (B^i)^T \|_2^2 + \sum_{j=1}^k \| C B_j - C B_j \|_2^2 \]
Thus, we can formulate the problem as the following least squares problem:
\[
   \begin{bmatrix} M_{11} \\ \ldots \\ M_{m1} \\ \ldots \\ M_{1n} \\ \ldots \\ M_{mn} \\ C^1 \tilde B_1 \\ \ldots \\ C^{n'} \tilde B_1 \\ \ldots \\ C^1 \tilde B_k \\ \ldots \\ C^{n'} \tilde B_k \end{bmatrix}
   =
   \begin{bmatrix}
     & A^1 & \\ & \ldots & \\ & A^m & \\  & & & \ldots & & & \\ & & & & & A^1 \\ & & & & & \ldots \\ & & & & & A^m \\
     C_{1,1} & & \ldots & & C_{1,n} \\& & \ldots \\ C_{n',1} & & \ldots & & C_{n',n} \\ & & \ldots \\
      & & C_{1,1}  &  & \ldots & & C_{1,n} \\& & & & \ldots \\  & & C_{n',1}  &  & \ldots & & C_{n',n}
   \end{bmatrix} \cdot \vec(B)
\]
Letting the vector in the left hand side as $Y$ and the matrix on the right hand side as $H$, we get $\vec(B) = H^+ Y$.

\paragraph{Basic Weighted Least Squares}
Suppose we would like to solve the following least squares problem:
\[ \min f(\tau) = \| T - \sum_{i=1}^k \tau_i T_i \|_F^2 \]
This is equivalent to solving the following equation:
\[
  \begin{bmatrix} T \cdot T_1 \\ \ldots \\ T \cdot T_k \end{bmatrix}
  =
  \begin{bmatrix} T_1 \cdot T_1 & \ldots & T_1 \cdot T_k \\ \ldots & \ldots & \ldots \\ T_k \cdot T_1 & \ldots & T_k \cdot T_k \end{bmatrix} \cdot \tau
\]
Specifically, if $T_i = a_i \cdot b_i \cdot c_i$, then the equation becomes
\[
  \begin{bmatrix} T(a_1, b_1, c_1) \\ \ldots \\ T(a_k, b_k, c_k) \end{bmatrix}
  =
  \begin{bmatrix} (a_1 \cdot a_1) (b_1 \cdot b_1) (c_1 \cdot c_1) & \ldots & (a_1 \cdot a_k) (b_1 \cdot b_k) (c_1 \cdot c_k) \\ \ldots & \ldots & \ldots \\ (a_k \cdot a_1) (b_k \cdot b_1) (c_k \cdot c_1) & \ldots & (a_k \cdot a_k) (b_k \cdot b_k) (c_k \cdot c_k) \end{bmatrix} \cdot \tau
\]
that is,
\[ \tau = ((A^T A) \circ (B^T B) \circ (C^T C))^{-1}  \begin{bmatrix} T(a_1, b_1, c_1) \\ \ldots \\ T(a_k, b_k, c_k) \end{bmatrix} \]

\subsection{Basic Algorithm}
Define tensor $T = P_{123}(U_1,U_2,U_3) = \sum_i w_i (U_1^TO_1)_i \otimes (U_2^TO_2)_i \otimes (U_3^TO_3)_i$

Given rank parameter $m$, we define objective function $F$ as follows:
\[ F(\tau, A, B, C, \tilde\tau, \tilde A, \tilde B, \tilde C) = \| \sum_{i=1}^m \tau_i A_i \otimes B_i \otimes C_i - T \|_F^2 + \| \sum_{i=1}^m \tilde\tau_i \tilde A_i \otimes \tilde B_i \otimes \tilde C_i - \tilde T \|_F^2 + \lambda \| B - \tilde B \|_F^2 \]

We have the following alternating minimization algorithm for the objective function.

\begin{algorithm}
  \caption{Efficient Joint Parameter Estimation; Basic Algorithm}
\begin{algorithmic}[H]
\STATE Randomly initialize $A, B, C, \tau, \tilde A, \tilde B, \tilde C, \tilde \tau$.
\FOR{$t=1,2,\ldots$:}
\STATE Update $A$: $A \gets \del{\del{\lambda I + (\tau \tau^T) \circ (B^T B) \circ (C^T C)}^{-1} \del{\begin{bmatrix} \tau_1 T(I, B_1, C_1) \\ \ldots \\ \tau_k T(I, B_k, C_k) \end{bmatrix} + \lambda \tilde A^T}}^T$
\STATE Update $B$: $B \gets \del{\del{\lambda I + (\tau \tau^T) \circ (A^T A) \circ (C^T C)}^{-1} \del{\begin{bmatrix} \tau_1 T(A_1, I, C_1) \\ \ldots \\ \tau_k T(A_k, I, C_k) \end{bmatrix} + \lambda \tilde B^T}}^T$
\STATE Update $C$: $C \gets \del{\del{\lambda I + (\tau \tau^T) \circ (A^T A) \circ (B^T B)}^{-1} \del{\begin{bmatrix} \tau_1 T(A_1, B_1, I) \\ \ldots \\ \tau_k T(A_k, B_k, I) \end{bmatrix} + \lambda \tilde C^T}}^T$
\STATE Update $\tau$: $\tau \gets ((A^T A) \circ (B^T B) \circ (C^T C))^{-1}  \begin{bmatrix} T(A_1, B_1, C_1) \\ \ldots \\ T(A_k, B_k, C_k) \end{bmatrix}$
\STATE Update $\tilde A$: $\tilde A \gets \del{\del{\lambda I + (\tilde \tau \tilde \tau^T) \circ (\tilde B^T \tilde B) \circ (\tilde C^T \tilde C)}^{-1} \del{\begin{bmatrix} \tilde\tau_1 \tilde T(I, \tilde B_1, \tilde C_1) \\ \ldots \\ \tilde\tau_k \tilde T(I, \tilde B_k, \tilde C_k) \end{bmatrix} + \lambda {\tilde A}^T}}^T$
\STATE Update $\tilde B$: $\tilde B \gets \del{\del{\lambda I + (\tilde \tau \tilde \tau^T) \circ (\tilde A^T \tilde A) \circ (\tilde C^T \tilde C)}^{-1} \del{\begin{bmatrix} \tilde\tau_1 \tilde T(\tilde A_1, I, \tilde C_1) \\ \ldots \\ \tilde\tau_k \tilde T(\tilde A_k, I, \tilde C_k) \end{bmatrix} + \lambda {\tilde B}^T}}^T$
\STATE Update $\tilde C$: $\tilde C \gets \del{\del{\lambda I + (\tilde \tau \tilde \tau^T) \circ (\tilde A^T \tilde A) \circ (\tilde B^T \tilde B)}^{-1} \del{\begin{bmatrix} \tilde\tau_1 \tilde T(\tilde A_1, \tilde B_1, I) \\ \ldots \\ \tilde\tau_k \tilde T(\tilde A_k, \tilde B_k, I) \end{bmatrix} + \lambda {\tilde C}^T}}^T$
\STATE Update $\tilde \tau$: $\tilde \tau \gets ((\tilde A^T \tilde A) \circ (\tilde B^T \tilde B) \circ (\tilde C^T \tilde C))^{-1}  \begin{bmatrix} \tilde T(\tilde A_1, \tilde B_1, \tilde C_1) \\ \ldots \\ \tilde T(\tilde A_k, \tilde B_k, \tilde C_k) \end{bmatrix}$
\ENDFOR
\end{algorithmic}
\end{algorithm}

\subsection{Algorithm Exploiting Three Views}

\subsection{Symmetrization }

Let $U$ be the left singular vectors of $P_{i,1}$. It it known that $\range(U) = \range(O)$
This serves as the projection matrix for dimensionality reduction.
We operate on the following tensor:
\[ T = P_{123}(U,U,U)(S_1, I, S_3) \]
where $S_1 = (U^T P_{23} U)(U^T P_{13} U)^{-1}$ and $S_3 = (U^T P_{21} U)(U^T P_{31} U)^{-1}$.
It is known that
\[ T = \sum_i w_i \]

\end{document}


\section{Exponential Functions}\label{sec:ExpFunctions}
An \dfont{exponential function} is a function of the form $f(x)=a^x$, where $a$ is a constant.
Examples are $2^x$, $10^x$ and $(1/2)^x$.
To more formally define the exponential function we look at various kinds of input values.

It is obvious
that $\ds a^5=a\cdot a\cdot a\cdot a\cdot a$ and $\ds a^3=a\cdot a\cdot a$,
but when we consider an exponential function $\ds a^x$ we can't
be limited to substituting integers for $x$. What does $\ds a^{2.5}$ or
$a^{-1.3}$ or $\ds a^\pi$ mean? And is it really true that
$a^{2.5}a^{-1.3}=a^{2.5-1.3}$? The answer to the first question is
actually quite difficult, so we will evade it; the answer to the
second question is ``yes.''

We'll evade the full answer to the hard question, but we have to know
something about exponential functions. You need first to understand
that since it's not ``obvious'' what $\ds 2^x$ should mean, we are really
free to make it mean whatever we want, so long as we keep the behavior
that {\em is} obvious, namely, when $x$ is a positive integer.
What else do we want to be true about $\ds 2^x$? We
want the properties of the previous two paragraphs to be true for all
exponents: $\ds 2^x2^y=2^{x+y}$ and $\ds (2^x)^y=2^{xy}$.

After the positive integers, the next easiest
number to understand is 0: $\ds 2^0=1$. You have presumably learned this
fact in the past; why is it true?  It
is true precisely because we want $\ds 2^a2^b=2^{a+b}$ to be true about
the function $\ds 2^x$. We need it to be true that $\ds 2^02^x=2^{0+x}=2^x$,
and this only works if $\ds 2^0=1$. The same argument implies that $\ds a^0=1$
for any $a$.

The next easiest set of numbers to
understand is the negative integers: for example, $\ds 2^{-3}=1/2^3$.
We know that whatever $\ds 2^{-3}$ means it must be
that $\ds 2^{-3}2^{3}=2^{-3+3}=2^0=1$, which means that $\ds 2^{-3}$ must be
$1/2^3$. In fact, by the same argument, once we know what $\ds 2^x$ means
for some value of $x$, $\ds 2^{-x}$ must be $\ds 1/2^{x}$ and more generally
$a^{-x}=1/a^x$.

Next, consider an exponent $1/q$, where $q$ is a positive integer. We
want it to be true that $\ds (2^x)^y=2^{xy}$, so $\ds
(2^{1/q})^q=2$. This means that $\ds 2^{1/q}$ is a $q$-th root of 2,
$\ds 2^{1/q}=\root q\of{2\ }$. This is all we need to understand that
$2^{p/q}=(2^{1/q})^p=(\root q\of{2\ })^p$ and
$a^{p/q}=(a^{1/q})^p=(\root q\of{a\ })^p$.

What's left is the hard part: what does $\ds 2^x$ mean when $x$ cannot be
written as a fraction, like $\ds x=\sqrt{2\ }$ or $\ds x=\pi$? What we know so
far is how to assign meaning to $\ds 2^x$ whenever $x=p/q$. If we were to
graph $a^x$ (for some $a>1$) at points $x=p/q$ then we'd see something like this:
$$\includegraphics[width=2.5in]{images/exp1}$$

This is a poor picture, but it illustrates a series
of individual points above the rational numbers
on the $x$-axis. There are really a lot of ``holes'' in the curve,
above $x=\pi$, for example. But (this is the hard part) it is possible
to prove that the holes can be ``filled in'', and that the resulting
function, called $\ds a^x$, really does have the properties we want,
namely that $\ds a^xa^y=a^{x+y}$ and $\ds (a^x)^y=a^{xy}$.
Such a graph would then look like this:
$$\includegraphics[width=2.5in]{images/exp2}$$

\begin{formulabox}[Three Types of Exponential Functions]
There are \ifont{three kinds} of exponential functions $f(x)=a^x$ depending on whether $a>1$, $a=1$ or $0<a<1$:
$$\includegraphics[width=6in]{images/exp3}$$
\end{formulabox}

\subsection*{Properties of Exponential Functions}
The first thing to note is that if $a<0$ then problems can occur.
Observe that if $a=-1$ then $(-1)^x$ is not defined for every $x$.
For example, $x=1/2$ is a square root and gives $(-1)^{1/2}=\sqrt{-1}$ which is not a real number.

\begin{formulabox}[Exponential Function Properties]
\begin{itemize}
\item \ifont{Only defined for positive $a$:} $a^x$ is only defined for all real $x$ if $a>0$
\item \ifont{Always positive:} $a^x>0$, for all $x$
\item \ifont{Exponent rules: }
\begin{multicols}{2}
\begin{enumerate}
	\item $\ds{a^xa^y=a^{x+y}}$
	\item $\ds{\frac{a^x}{a^y}=a^{x-y}}$
	\item $\ds{\left(a^x\right)^y=a^{xy}=a^{yx}=\left(a^y\right)^x}$
	\item $\ds{a^xb^x=(ab)^x}$
\end{enumerate}
\end{multicols}
\item \ifont{Long term behaviour:} If $a>1$, then $a^x\to\infty$ as $x\to\infty$ and $a^x\to 0$ as $x\to-\infty$.
\end{itemize}
\end{formulabox}

The last property can be observed from the graph. If $a>1$, then as $x$ gets larger and larger, so does $a^x$.
On the other hand, as $x$ gets large and negative, the function approaches the $x$-axis, that is, $a^x$ approaches $0$.

\begin{example}{Reflection of Exponential}{ReflectionExponential}
Determine an equation of the function after reflecting $y=2^x$ about the line $x=-2$.
\end{example}

\begin{solution}
First reflect about the $y$-axis to get $y=2^{-x}$.
Now shift by $2\times 2=4$ units to the \ifont{left} to get $y=2^{-(x+4)}$.
Side note: Can you see why this sequence of transformations is the same as reflection in the line $x=-2$?
Can you come up with a general rule for these types of reflections?
\end{solution}

\begin{example}{Determine the Exponential Function}{DetermineExponentialFunction}
Determine the exponential function $f(x)=ka^x$ that passes through the points $(1,6)$ and $(2,18)$.
\end{example}

\begin{solution}
We substitute our two points into the equation to get:
$$x=1,y=6\to6=ka^1$$
$$x=2,y=18\to18=ka^2$$
This gives us $6=ka$ and $18=ka^2$.
The first equation is $k=6/a$ and subbing this into the second gives: $18=(6/a)a^2$.
Thus, $18=6a$ and $a=3$.
Now we can see from $6=ka$ that $k=2$.
Therefore, the exponential function is $$f(x)=2\cdot 3^x.$$
%\vspace{-1cm}
\end{solution}

There is one base that is so important and convenient that we give it a special symbol.
This number is denoted by $e=2.71828\ldots$ (and is an irrational number).
Its \ifont{importance} stems from the fact that it simplifies many formulas
of Calculus and also shows up in other fields of mathematics.

\begin{example}{Domain of Function with Exponential}{DomainFunctionExponential}
Find the domain of $\ds f(x)=\frac{1}{\sqrt{e^x+1}}$.
\end{example}

\begin{solution}
For domain, we cannot divide by zero or take the square root of negative numbers.
Note that one of the properties of exponentials is that they are always positive!
Thus, $e^x+1>0$ (in fact, as $e^x>0$ we actually have that $e^x+1$ is at least one).
Therefore, $e^x+1$ is never zero nor negative, and gives no restrictions on $x$.
Thus, the domain is $\R$.
\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:ExpFunctions}}

\begin{enumialphparenastyle}

%%%%%%%%%%
\begin{ex}
Determine an equation of the function $y=a^x$ passing through the point $(3,8)$.
\begin{sol}
$y=2^x$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Find the $y$-intercept of $f(x)=4^x+6$.
\begin{sol}
$y=7$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Find the $y$-intercept of $f(x)=2\left(\frac{1}{2}\right)^x$.
\begin{sol}
$y=2$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Find the domain of $\ds{y=e^{-x}+e^{\frac{1}{x}}}$.
\begin{sol}
$x\neq 0$
\end{sol}
\end{ex}

\end{enumialphparenastyle}


\subsection{Unitary time}

Why? What's the interpretation here? Is this an assumption, or just a modelling choice?

\(\Psi (t_b-t_a)^* \Psi (t_b-t_a)=e^{(t_b-t_a)X^*} e^{(t_b-t_a)X}\)

\(\Psi (t_b-t_a)^* \Psi (t_b-t_a)=e^{(t_b-t_a)(X^*+X)}\)

\(X=iH\)

\(\Psi (t_b-t_a)^* \Psi (t_b-t_a)=e^{(t_b-t_a)(-iH+iH)}=I\)

\(\Psi (t_b-t_a)=e^{(t_b-t_a)iH}\)


% --------------------------- %
% PolynomialTaskTest_Revisited Start
% --------------------------- %
\section{\textbf{PolynomialTaskTest}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Particular Case}
\par
\textbf{Note: We noticed that this problem is repeated. Actually, it should not
go here. It corresponds to the chapter of Futures. Including it here just to
follow the order of the programs in the web site.}
\par
The particular problem we want to solve in this exercise is how can we implement
an efficient polynomial parallel adder and multiplier.
\par
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Solution}
\par
The solution to this problem requires the use of a pool of threads, which are
called \textit{executor services} in Java.  In this case, we will use Runnable
objects. These Runnable objects return what are called Futures. Futures of
Runnables provide the \textit{get()} method. This method blocks the caller till
the computation finishes.
\par
Let us examine the code to understand how the Futures are used to solve the
Polynomial problem.
\par
Here is the \textit{MulTask} Runnable. It requires 3 references to polynomials:
the multipliers and the result. When it is executed, it will split each of the
multipliers into two, say $p_1$, $p_2$ and $q_1$, $q_2$. The concatenation of
$p_1$ and $p_2$ is the original $p$ polynomial. Given this partition, the result
of a multiplication should be $p_1*q_1 + p_1*q_2 + p_2*q_1 + p_2*q_2$. So, the
task of multiplying two polynomials is decomposed into $4$ multiplications and
$3$ sums.
\par
\hfill
\begin{lstlisting}[style=numbers]
  static class MulTask implements Runnable {
    Polynomial p, q, r;
    Polynomial[] temp;
    public MulTask(Polynomial p, Polynomial q, Polynomial r) {
      this.p = p; this.q = q; this.r = r;
      int newDegree = 2 * p.getDegree();
      temp = new Polynomial[6];
      for (int i  = 0; i < temp.length; i++) {
        temp[i] = new Polynomial(newDegree);
      }
    }
    public void run() {
      try {
        if (p.getDegree() == 0) {
          r.set(0, p.get(0) * q.get(0));
        } else {
          Polynomial[] pp = p.split();
          Polynomial[] qq = q.split();
          Future<?>[] future = (Future<?>[]) new Future[7];
          // launch parallel multiplications
          future[0] = exec.submit(new MulTask(pp[0], qq[0], temp[0]));
          future[1] = exec.submit(new MulTask(pp[0], qq[1], temp[1]));
          future[2] = exec.submit(new MulTask(pp[1], qq[0], temp[2]));
          future[3] = exec.submit(new MulTask(pp[1], qq[1], temp[3]));
          // wait for them to finish
          for (int i = 0; i < 4; i++)
            future[i].get();
          // do partial sums
          future[5] = exec.submit(new AddTask(temp[0], temp[1], temp[4]));
          future[6] = exec.submit(new AddTask(temp[2], temp[3], temp[5]));
          // wait for them to finish
          future[5].get();
          future[6].get();
          // final sum
          future[7] = exec.submit(new AddTask(temp[4], temp[5], r));
          future[7].get();
        }
      } catch (Exception ex) {
        ex.printStackTrace();
      }
    }
  }
\end{lstlisting}
\hfill
\par
Here we have now the \textit{AddTask}. This Runnable is simpler. It again
requires $3$ references to polynomials. The execution consists of splitting each
polynomial into two. After that, sub-polynomials are added one to one.
\par
\hfill
\begin{lstlisting}[style=numbers]
  static class AddTask implements Runnable {
    Polynomial p, q, r;
    public AddTask(Polynomial p, Polynomial q, Polynomial r) {
      this.p = p; this.q = q; this.r = r;
    }
    public void run() {
      try {
        int n = p.getDegree();
        if (n == 1) {
          r.set(0, p.get(0) + q.get(0));
        } else {
          Polynomial[] pp = p.split(), qq = q.split(), rr = r.split();
          Future<?>[] future = (Future<?>[]) new Future[2];
          // create asynchronous computations
          future[0] = exec.submit(new AddTask(pp[0], qq[0], rr[0]));
          future[1] = exec.submit(new AddTask(pp[1], qq[1], rr[1]));
          // wait for them to finish
          future[0].get();
          future[1].get();
        }
      } catch (Exception ex) {
        ex.printStackTrace();
      }
    }
  }
\end{lstlisting}
\hfill
\par
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Experiment Description}
\par
Now let us discuss the experiment. It is actually a sum of the
polynomial $x^7+x^6+x^5+x^4+x^3+x^2+x+1$ by itself. The result of this
sum should be $2x^7+2x^6+2x^5+2x^4+2x^3+2x^2+2x+2$.
\par
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Sample Results and Interpretation}
\par
First, we had to fix the test case. The right one is as follows:
\par
\hfill
\begin{lstlisting}[style=numbers]
  /**
   * Test of run method, of class steal.PolynomialTask.
   */
  public void testRun() throws InterruptedException, ExecutionException {
    System.out.println("run");

    int[] a = {1, 1, 1, 1, 1, 1, 1, 1};
    int[] b = {1, 1, 1, 1, 1, 1, 1, 1};
    Polynomial aa = new Polynomial(a, 0, 8);
    Polynomial bb = new Polynomial(b, 0, 8);
    Polynomial cc = PolynomialTask.add(aa, bb);
    for (int i = 0; i < cc.getDegree(); i++) {
      assertEquals(2, cc.get(i));
    }
    Polynomial dd = PolynomialTask.multiply(aa,bb);
  }
\end{lstlisting}
\hfill
\par
After this fix, we had the following failure:
\par
\hfill
\begin{verbatim}
[oraadm@gdlaa008 Steal]$ junit steal.PolynomialTaskTest
.run
java.lang.ArrayIndexOutOfBoundsException: 7
  at steal.PolynomialTask$MulTask.run(PolynomialTask.java:102)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:724)
...
java.lang.ArrayIndexOutOfBoundsException: 7
  at steal.PolynomialTask$MulTask.run(PolynomialTask.java:102)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:724)
java.lang.ArrayIndexOutOfBoundsException: 7
  at steal.PolynomialTask$MulTask.run(PolynomialTask.java:102)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:724)

Time: 0.13

OK (1 test)
\end{verbatim}
\hfill
\par
The problem is that the size of the \textit{futures} array was not big enough.
It is 7, but it actually required 8 elements. After that fix, we were able to
run the test successfully every time:
\par
\hfill
\begin{verbatim}
[oraadm@gdlaa008 Steal]$ junit steal.PolynomialTaskTest
.run

Time: 0.072

OK (1 test)
\end{verbatim}
\hfill
\par
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% --------------------------- %
% PolynomialTaskTest_Revisited End
% --------------------------- %


\documentclass{amsart}

\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{listings}
\usepackage{color}
\usepackage{courier}
\definecolor{mygreen}{RGB}{28,172,0} % color values Red, Green, Blue
\definecolor{mylilas}{RGB}{170,55,241}
\lstset{language=Matlab,%
    basicstyle=\footnotesize\ttfamily,
    breaklines=true,%
    morekeywords={matlab2tikz},
    keywordstyle=\color{blue},%
    morekeywords=[2]{1}, keywordstyle=[2]{\color{black}},
    identifierstyle=\color{black},%
    stringstyle=\color{mylilas},
    commentstyle=\color{mygreen},%
    showstringspaces=false,%without this there will be a symbol in the places where there is a space
    numbers=none,%
    numberstyle={\tiny \color{black}},% size of the numbers
    numbersep=9pt, % this defines how far the numbers are from the text
    emph=[1]{for,end,break},emphstyle=[1]\color{red}, %some words to emphasise
    %emph=[2]{word1,word2}, emphstyle=[2]{style},
}

\theoremstyle{definition}
\newtheorem{remark}{Remark}
\newtheorem{exercise}{\color{red}Exercise}
\newtheorem{example}{Example}

\title[Programming finite elements in Matlab]{Programming finite elements in high-level languages with a focus on Matlab}
\author{Tom Gustafsson}

\begin{document}

\maketitle

\tableofcontents

\section{Variational problems}

Finite element method for linear problems concerns the computer
solution of the following \emph{variational problem}: find $u
\in U$ such that
\begin{equation}
  a(u,v) = l(v),
\end{equation}
for every $v \in V$. Here $U$ and $V$ are Hilbert spaces with the
norms $\|\cdot\|_U$ and $\|\cdot\|_V$, $a \in \mathcal{L}(U \times V;
\mathbb{R})$ is a continuous bilinear form and $l \in
\mathcal{L}(V;\mathbb{R})$ is a continuous linear form.

Recall that if $U=V$ the existence and uniqueness of the solution
follows from the Lax--Milgram theorem if the bilinear form $a$ is
\emph{coercive} in $V$. If $a$ is not coercive or if $U\neq V$, a more
general theory is required, see Babu\v{s}ka--Aziz~\cite{BA72}.

\begin{remark}
    The uniqueness aspect has no effect on the practical programming
in the sense that using finite elements one can easily \emph{try} to
discretize and solve continuous problems that are not
well-posed. Usually such approaches lead to singular,
non-invertible, matrix systems.
\end{remark}

\section{Galerkin methods}

Suppose that there exist finite dimensional subspaces $U_N \subset U$
and $V_M \subset V$ with the dimensions $N$ and $M$, respectively. The
\emph{discrete variational problem} reads: find $u_N \in U_N$ such
that
\begin{equation}
  \label{eq:dweakf}
  a(u_N,v) = l(v),
\end{equation}
for every $v \in V_M$.

\begin{remark}
  If $U_N = V_M$, the resulting method is sometimes referred to as the
  \emph{Ritz--Galerkin method}. Otherwise, the term
  \emph{Petrov--Galerkin method} is used.
\end{remark}

Suppose that $U_N = \mathrm{span}\,\{\varphi_i\}_{i=1}^N$ and $V_N =
\mathrm{span}\,\{\xi_i\}_{i=1}^M$ for some basis functions $\varphi_i
\in U_N$, $i\in\{1,\cdots,N\}$ and $\xi_i \in V_M$,
$i\in\{1,\cdots,M\}$.
Then we may expand the solution as
\begin{equation}
  u_N = \sum_{j=1}^N u_j \varphi_j,
\end{equation}
where $u_j \in \mathbb{R}$, $j\in\{1,\cdots,N\}$, are the degrees of
freedom. Substituting this expanded solution to \eqref{eq:dweakf}
leads to the matrix system
\begin{equation}
  \label{eq:matrixsystem}
  A x = b,
\end{equation}
where $A \in \mathbb{R}^{M \times N}$, $A_{ij} = a(\varphi_j, \xi_i)$, $b \in \mathbb{R}^M$, $b_i = l(\xi_i)$ and $x \in \mathbb{R}^N$, $x_i = u_i$.

\begin{exercise}
  Derive \eqref{eq:matrixsystem} from \eqref{eq:dweakf}.
\end{exercise}

The programming aspect of the Galerkin method is focused on
understanding and implementing the \emph{assembly procedure} that
constructs the pair $(A,b)$ given a bilinear form $a$, linear form $l$
and a set of basis functions. It is now \emph{crucial} to understand
that given a basis and a bilinear form, we \emph{always} have a corresponding
matrix that approximates the continuous operator.

\begin{remark}
    In mathematical literature the finite dimensional spaces $U_N$ and
$V_M$ are sometimes defined by limiting the set of arbitrary functions
to, e.g., piecewise polynomial functions. In such cases the basis is
implicit in the sense that we must do additional work to find the
basis functions or evaluate them pointwise.  Thus, from a programming
point-of-view an explicit basis is usually more preferable.
\end{remark}

\subsection{On the choice of basis functions}

Let us briefly discuss through examples how the choice of the basis
affects to the properties of the matrix $A$. In the following examples
$U=V$ and $U_N = V_M$.

\begin{example}
  \label{ex:diagonal}
  Consider the bilinear and linear forms
  \begin{equation}
    a(w,v) = \int_0^1 w^\prime v^\prime\,\mathrm{d}x, \quad l(v) = \int_0^1 v\,\mathrm{d}x,
  \end{equation}
  and the basis $\{\sin (\pi (2n-1) x)\}_{n=1}^N$. In terms of Matlab
  code, we have
  \begin{lstlisting}
    syms x
    N=5;
    dphi=diff(sin(pi*(1:2:(2*N))*x),x);
    A=int(dphi'*dphi,x,0,1);
  \end{lstlisting}
  which outputs the matrix
  \begin{equation}
\left(\begin{array}{ccccc} \frac{{\pi}^2}{2} & 0 & 0 & 0 & 0\\ 0 & \frac{9\, {\pi}^2}{2} & 0 & 0 & 0\\ 0 & 0 & \frac{25\, {\pi}^2}{2} & 0 & 0\\ 0 & 0 & 0 & \frac{49\, {\pi}^2}{2} & 0\\ 0 & 0 & 0 & 0 & \frac{81\, {\pi}^2}{2} \end{array}\right).
  \end{equation}
  Notice that the matrix is diagonal. This means that the solution
will be quick.  The corresponding linear form can be assembled and
the system solved with
  \begin{lstlisting}
    b=int(sin(pi*(1:2:(2*N))*x),x,0,1)';
    x=A\b;
  \end{lstlisting}
\end{example}

\begin{exercise}
  Draw the solution given by the Galerkin method in the previous example.
\end{exercise}

\begin{remark}
  The usage of symbolic integration is not a good idea in general due
to a high computational cost in comparison to numerical integration.
\end{remark}

\begin{example}
  Consider the same bilinear form as in Example~\ref{ex:diagonal}
  and the polynomial basis $\{x^n(1-x)^n\}_{n=1}^N$. Now the matrix $A$ is
  \begin{equation}
    \left(\begin{array}{ccccc} \frac{1}{3} & \frac{1}{15} & \frac{1}{70} & \frac{1}{315} & \frac{1}{1386}\\ \frac{1}{15} & \frac{2}{105} & \frac{1}{210} & \frac{4}{3465} & \frac{5}{18018}\\ \frac{1}{70} & \frac{1}{210} & \frac{1}{770} & \frac{1}{3003} & \frac{1}{12012}\\ \frac{1}{315} & \frac{4}{3465} & \frac{1}{3003} & \frac{4}{45045} & \frac{1}{43758}\\ \frac{1}{1386} & \frac{5}{18018} & \frac{1}{12012} & \frac{1}{43758} & \frac{5}{831402} \end{array}\right).
  \end{equation}
  Notice that this time the matrix is full and hence the solution of
the linear system requires more effort.
\end{example}

Based on the examples, we can deduce that the choice of the basis (or,
equivalently, the subspace $V_N$) has an effect on
\begin{itemize}
  \item the \emph{sparsity
    pattern} of the resulting matrix, and
    \item the \emph{condition
number}\footnote{As given by the Matlab function \emph{cond}.} which for the
former matrix is 81 whereas the condition number of the latter matrix
is approximately $2.1 \cdot 10^8$.
\end{itemize}
Moreover, let $P_N : V \rightarrow V_N$ be an orthogonal
projection. Using the continuity and coercivity of $a$, we get
\begin{align*}
  \|u-u_N\|_V^2 &\leq C^\prime a(u-u_N,u-u_N) \\
                &= C^\prime a(u-u_N,u-P_N u) \\
  &\leq C \|u-u_N\|_V \|u-P_N u\|_V,
\end{align*}
for some $C, C^\prime >0$. This leads to
\begin{equation}
  \|u-u_N\|_V \leq C \|u-P_N u\|_V,
\end{equation}
which essentially means that the error (in the sense of $\|u-u_N\|_V$)
depends on \emph{how well the space $V_N$ can represent functions of
the space $V$}. In an ``ideal case'' the exact solution belongs to the
discrete space $V_N$ leading to zero error. In practice this is rarely
the case.

Everything we have discussed so far applies to all Galerkin methods.
Next we focus on a particular Galerkin method called the \emph{finite
element method}.

\section{Finite element method}

Finite element methods are special Galerkin methods where the basis
functions are defined using a \emph{mesh}. In the following we
consider two dimensional problems but many parts can be
straightforwardly generalized to three dimensional problems.

\subsection{Mesh}

Let $\Omega \subset \mathbb{R}^2$ a polygonal domain. We split the
domain $\Omega$ into non-overlapping triangles or \emph{elements} $T
\in \mathcal{T}$ that satisfy
\begin{equation}
  \overline{\Omega} = \bigcup_{T \in \mathcal{T}} \overline{T}.
\end{equation}
The collection of elements $\mathcal{T}$ is called a mesh.  One
frequently encounters the notion of a \emph{mesh parameter $h_T$ of a
triangle $T$} that describes the maximum edge length of $T$, i.e.,
\begin{equation}
  h_T = \max_{x,y \in T} \|x-y\|,
\end{equation}
where $\|\cdot\|$ is the Euclidean norm. The mesh parameter of a
mesh $\mathcal{T}$ is then
\begin{equation}
  h = \max_{T \in \mathcal{T}} h_T.
\end{equation}

Let $N$ be the number of vertices in the mesh and let $M$ be the
number of elements.  In high-level array-based languages such as
Matlab, a convenient way to represent triangular (and other) meshes is
to use two matrices $p \in \mathbb{R}^{2 \times N}$ and $t \in
\mathbb{R}^{3 \times M}$. In the former, each column corresponds to
$x$ and $y$ coordinates of a vertex and in the latter each column
corresponds to the three indices of the vertices of an element.

\begin{example}
  Using Matlab PDE Toolbox we can generate a triangular mesh for the
  unit square with $h\leq 0.1$ using the command
\begin{lstlisting}
    [p,~,t]=initmesh(decsg([3 4 0 1 1 0 0 0 1 1]'),'Hmax',0.1)
\end{lstlisting}
  The matrix $t$ given by \emph{initmesh} has one extra row which
  in case of multiple domains tells us the subdomain in which the
  triangle lies. We do not need this information and therefore we
  strip the last row as follows:
\begin{lstlisting}
    t=t(1:3,:);
\end{lstlisting}
  The mesh can now be visualized with
\begin{lstlisting}
    trimesh(t',p(1,:),p(2,:));
\end{lstlisting}
\end{example}

\begin{exercise}
  Visualize two meshes with different mesh parameters.
\end{exercise}

\subsection{Definition of the piecewise linear basis functions}

In this text we focus on the \emph{piecewise linear basis}.  Each
basis function is defined so that it has value 1 in one vertex of the
mesh and value 0 in all other vertices. Furthermore, the basis
function is linear inside each element.

\begin{example}
  Let us generate a mesh and visualize all the basis functions.
  \begin{lstlisting}
    [p,~,t]=initmesh(decsg([3 4 0 1 1 0 0 0 1 1]'),'Hmax',0.5);
    t=t(1:3,:);
    figure;
    % there is one basis function per node, loop over nodes
    for itr = 1:size(p,2)
        subplot(ceil(sqrt(size(p,2))),ceil(sqrt(size(p,2))),itr);
        % the basis function is zero everywhere else ...
        u=zeros(size(p,2),1);
        % ... but one node.
        u(itr)=1;
        % draw the resulting surface
        trisurf(t',p(1,:),p(2,:),u);
        colormap parula;
        shading interp;
        view(2);
    end
  \end{lstlisting}
  From the resulting figure you can probably see why these
  kind of basis functions are sometimes referred to as
  the \emph{hat functions}.
\end{example}

These kind of basis functions have some nice properties.
For example:
\begin{enumerate}
\item They are based on a mesh and hence any domain
  that can be meshed is amenable to discretization.
\item The basis functions are nonzero on a very small area.
  Therefore, most entries of the resulting system matrix are zero
  and the matrix will be sparse.
\item We can easily project an arbitrary function to the
  basis by evaluating it at the vertices of the mesh.
\end{enumerate}

\subsection{Assembly of finite elements}

Let us denote by $\varphi_i$, $i \in \{1, \cdots, N\}$ the set of
piecewise linear basis functions defined by the mesh $\mathcal{T}$
of the domain $\Omega$.
Consider the bilinear form
\begin{equation}
  a(w,v) = \int_\Omega \nabla w \cdot \nabla v\,\mathrm{d}x.
\end{equation}
Our goal is to construct the matrix $A \in \mathbb{R}^{N \times N}$
with the entries $A_{ij} = a(\varphi_j, \varphi_i)$.  Since $N$ can be
very large and a large portion of the values $A_{ij}$ are zero, it is
not a good idea to evaluate each $a(\varphi_j, \varphi_i)$
separately. Instead, we evaluate \emph{all nonzero terms} of the form
\begin{equation}
  \label{eq:integral}
    \int_T \nabla \varphi_j \cdot \nabla \varphi_i\,\mathrm{d}x, \quad T \in \mathcal{T}, \quad i,j \in \{1,\cdots,N\},
\end{equation}
and add them together appropriately to \emph{assemble} $A$.

\begin{exercise}
  Download the commented version of the Matlab class template with the command
  \begin{lstlisting}
    unzip('http://super.org/file.zip')
  \end{lstlisting}
  Read through \emph{LinearAssembler.m} and try to run the tests with
  \begin{lstlisting}
    run(LinearAssemblerTests)
  \end{lstlisting}
\end{exercise}

Next we start introducing necessary concepts to implement the methods
of the class \verb|LinearAssembly|. We first implement the static
methods, secondly we write the constructor which does some
precomputations and finally implement the methods that assemble
bilinear and linear forms.

\subsection{Reference element}

Evaluation of the integral \eqref{eq:integral} can be done
systematically by mapping it to the \emph{reference triangle}
$\widehat{T}$ defined as
\begin{equation}
  \widehat{T} = \{ (\widehat{x}, \widehat{y}) : \widehat{x} \geq 0,~ \widehat{y} \geq 0,~ \widehat{x}+\widehat{y}\leq 1 \}.
\end{equation}
We end up evaluating integrals of polynomials on the reference
triangle and for this we require \emph{quadrature rules}. The idea of
quadrature rules is to replace the integral with a sum, i.e., for
a given polynomial $p \in P_n(\widehat{T})$
and triplets $(\widehat{x}_k,\widehat{y}_k,w_k)$, $k \in \{1, \cdots, M\}$,
we write
\begin{equation}
  \int_{\widehat{T}} p(\widehat{x},\widehat{y}) \,\mathrm{d}\widehat{x}\,\mathrm{d}\widehat{y} \approx \sum_{k = 1}^{M} p(\widehat{x}_k,\widehat{y}_k) w_k.
\end{equation}
\begin{remark}
  For any given polynomial order $n$ the quadrature rule can be exact
  if $M$ and the triplets are chosen right. The construction of exact
  rules is, however, non-trivial. See, e.g., Dunavant~\cite{D85} where
  an exact rule for second order polynomials is given by the triplets
  \begin{equation}
    (\tfrac{1}{3},\tfrac{1}{3},\tfrac{1}{6}), \quad
    (\tfrac{2}{3},\tfrac{1}{3},\tfrac{1}{6}), \quad
    (\tfrac{1}{3},\tfrac{2}{3},\tfrac{1}{6}).
  \end{equation}
\end{remark}
\begin{exercise}
  Implement the method \verb|localQuadrature|.
\end{exercise}
Let $F_T : \widehat{T} \rightarrow T$ be an affine mapping of the form
\begin{equation}
  \begin{bmatrix} x \\ y \end{bmatrix} = F_T(\widehat{x},\widehat{y}) = B_T\begin{bmatrix} \widehat{x} \\ \widehat{y}\end{bmatrix}+c_T,
\end{equation}
where $B_T \in \mathbb{R}^{2 \times 2}$ and $c_T \in \mathbb{R}^{2 \times 1}$
depend on the location and the shape of the triangle $T$.
Notice that we denote by $(x,y)$ points in the global
coordinate system and by $(\widehat{x},\widehat{y})$ points in the
coordinate system of the reference element.
\begin{exercise}
  Find explicit expressions for $B_T$ and $c_T$ in terms of the
vertices of the element $T$ and implement the method \verb|affineMappings|.
\end{exercise}
\begin{exercise}
  Implement the method \verb|F| which maps points from the local coordinate
  system to the global coordinate system using $B_T$ and $c_T$.
\end{exercise}
\subsection{Transforming the local basis}
Let the local basis functions be defined as
\begin{equation}
  \widehat{\varphi}_1(\widehat{x},\widehat{y}) = 1 - \widehat{x} - \widehat{y}, \quad \widehat{\varphi}_2(\widehat{x},\widehat{y}) = \widehat{x}, \quad \widehat{\varphi}_3(\widehat{x},\widehat{y}) = \widehat{y}.
\end{equation}
Our goal is to describe the global basis functions $\varphi_i$,
$i\in\{1,\cdots,N\}$, using these local basis functions.
\begin{exercise}
  Implement the method \verb|localBasis|.
\end{exercise}
Let $m$ be the column corresponding to the triangle $T$ in the element
matrix $t$. The global basis functions that are nonzero in element $T$
can be now defined as
\begin{equation}
  \label{eq:phi}
  \varphi_{e_i^{\mathrm{T}}te_m}(F_T(\widehat{x},\widehat{y}))= \widehat{\varphi}_i(\widehat{x},\widehat{y}),
\end{equation}
for each $i \in \{1,2,3\}$. Using the chain rule, we deduce that the
derivatives transform as follows:
\begin{equation}
  \label{eq:dphi}
  \nabla \varphi_{e_i^{\mathrm{T}}te_m}(F_T(\widehat{x},\widehat{y})) = B_T^{-\mathrm{T}} \widehat{\nabla} \widehat{\varphi}_i(\widehat{x},\widehat{y}),
\end{equation}
\begin{exercise}
  Implement \verb|globalBasis| which maps the local basis to global
basis using \eqref{eq:phi} and \eqref{eq:dphi}.
\end{exercise}
Finally, the integral \eqref{eq:integral} is transformed as
\begin{equation}
  \label{eq:integraltransform}
    \int_T \nabla \varphi_{e_j^{\mathrm{T}}te_m} \cdot \nabla \varphi_{e_i^{\mathrm{T}}te_m}\,\mathrm{d}x = \int_{\widehat{T}} B_K^{-\mathrm{T}}\widehat{\nabla} \widehat{\varphi}_j \cdot B_K^{-\mathrm{T}}\widehat{\nabla}\widehat{\varphi}_i|\det B_K|\,\mathrm{d}\widehat{x}.
\end{equation}
\begin{exercise}
  Implement \verb|determinant| for computing the determinant in \eqref{eq:integraltransform}.
\end{exercise}

\begin{thebibliography}{1}
\bibitem{BA72} I.~Babu\v{s}ka, A.~K.~Aziz. The Mathematical Foundations of the Finite Element Method with Applications to Partial Differential Equations. Academic Press (1972).
\bibitem{D85} D.~A.~Dunavant. High degree efficient symmetrical Gaussian quadrature rules for the triangle (1985).
\end{thebibliography}

\end{document}


\subsection{Derivatives and Integrals}
\input{./backgroundReview/singleVariableCalc/derivatives.tex}
\input{./backgroundReview/singleVariableCalc/integrals.tex}


\documentclass{article}
\usepackage{ws_template}

\usepackage{amsmath}
\usepackage{listings}

\title{homework sheet 01}
\author{
	\name{Denys Sobchyshak}\\
	\imat{03636581}\\
	\email{denys.sobchyshak@gmail.com}
	\And
	\name{Sergey Zakharov}\\
	\imat{03636642}\\
	\email{ga39pad@mytum.de}
}

\begin{document}
	\maketitle

	\section{Problem}
	To find eigenvalues of the matrix A we have to solve $ det(A-\lambda I)=0 $ which cane be derived from an eigenvelue equation. \\
	\[
	det(A-\lambda I)= det
	\begin{pmatrix}
		2-\lambda & -1 & 0  \\
		-1 & 2-\lambda & -1 \\
		0 & -1 & 2-\lambda
	\end{pmatrix}
	=(2-\lambda)^3-2(2-\lambda)=(2-\lambda)((2-\lambda)^2-2)=0
	\]\\
	Thus we can see that the spectrum of the matrix is given by $\lambda(A)=\left\{2, 2-\sqrt{2},2+\sqrt{2}\right\}$. To find the corresponding eigenvectors we substitute our eigenvalues into the eigenvalue equation.\\
	\begin{enumerate}
		\item For $\lambda=2$:\\
		$
		A-2I=0 \Rightarrow
		\left(\begin{array}{ccc|c}
		2-2 & -1 & 0 & 0 \\
		-1 & 2-2 & -1 & 0\\
		0 & -1 & 2-2 & 0
		\end{array}\right)
		\Rightarrow \left\{
		\begin{array}{cc}
		x=-z\\
		y=0\\
		\end{array}\right.
		$\\
		Thus using $z=-1$ we get the next eigenvector $v_1=\begin{pmatrix}1 \\0\\ -1\end{pmatrix}$.

		\item For $\lambda=2-\sqrt{2}$:\\
		$
		A-2I=0 \Rightarrow
		\left(\begin{array}{ccc|c}
		2-2+\sqrt{2} & -1 & 0 & 0 \\
		0 & -1 & 2-2+\sqrt{2} & 0 \\
		-1 & 2-2+\sqrt{2} & -1 & 0
		\end{array}\right)
		\Rightarrow \left\{
		\begin{array}{cc}
		x=\frac{y}{\sqrt{2}}\\
		y=\sqrt{2}z\\
		z=\frac{y}{\sqrt{2}}\\
		\end{array}\right.
		$\\
		Thus using $z=1$ we get the next eigenvector $v_2=\begin{pmatrix}1 \\\sqrt{2}\\ 1\end{pmatrix}$.

		\item For $\lambda=2+\sqrt{2}$:\\
		$
		A-2I=0 \Rightarrow
		\left(\begin{array}{ccc|c}
		2-2-\sqrt{2} & -1 & 0 & 0 \\
		-1 & 2-2-\sqrt{2} & -1 & 0\\
		0 & -1 & 2-2-\sqrt{2} & 0
		\end{array}\right)
		\Rightarrow \left\{
		\begin{array}{cc}
		x=-\frac{y}{\sqrt{2}}\\
		y=-\sqrt{2}z\\
		z=-\frac{y}{\sqrt{2}}\\
		\end{array}\right.
		$\\
		Thus using $z=1$ we get the next eigenvector $v_3=\begin{pmatrix}1 \\-\sqrt{2} \\ 1\end{pmatrix}$.
	\end{enumerate}
	The corresponding python/numpy code looks as follows:\\
	\lstinputlisting[language=Python,firstline=21]{scripts/ws1-linear-algebra.py}

	\section{Problem}
	If $ x_1, x_2 ... x_n$ are linearly independent eigenvectors of $ B \in  R^{n \times n} $ and $ \lambda_i $ are its eigenvalues then the following equation is satisfied: $ Bx_i = \lambda_i x_i $.
	This equation can be rewritten as $ BU = UD $ from here we get matrix $ B = UDU^{-1} $ as follows:\\\\
	$
	BU=B\begin{bmatrix} X_{1} & X_{2} & \cdots & X_n \end{bmatrix}
	=\begin{bmatrix} BX_{1} & BX_{2} & \cdots & BX_n \end{bmatrix}
	=\begin{bmatrix} \lambda_1X_{1} & \lambda_2X_{2} & \cdots & \lambda_nX_n \end{bmatrix}\\
	=\begin{bmatrix}
	\lambda_1x_{1,1} & \lambda_2x_{2,1} & \cdots & \lambda_nx_{n,1} \\
	\lambda_1x_{1,2} & \lambda_2x_{2,2} & \cdots & \lambda_nx_{n,2} \\
	\vdots & \vdots & \ddots & \vdots \\
	\lambda_1x_{1,n} & \lambda_2x_{2,n} & \cdots & \lambda_nx_{n,n}
	\end{bmatrix}
	=\begin{bmatrix}
	x_{1,1} & x_{2,1} & \cdots & x_{n,1} \\
	x_{1,2} & x_{2,2} & \cdots & x_{n,2} \\
	\vdots & \vdots & \ddots & \vdots \\
	x_{1,n} & x_{2,n} & \cdots & x_{n,n}
	\end{bmatrix}
	\begin{bmatrix}
	\lambda_1 & 0 & \cdots & 0 \\
	0 & \lambda_2 & \cdots & 0 \\
	\vdots & \vdots & \ddots & \vdots \\
	0 & 0 & \cdots & \lambda_n
	\end{bmatrix}\\
	=UD \Leftrightarrow BUU^{-1}=B=UDU^{-1}
	$
	\section{Problem}
	\begin{itemize}
		\item $B \bar{x} = \lambda \bar{x} $. Suppose that $\lambda$ and $\bar{x}$ are possibly complex and  $x \not=0$  .
		We take conjugate transpose to get $\bar{\lambda} x^* = (\lambda x)^* = (Ax)^* =x^*A^* = x^*A$  (because A is real and symmetric). Then $\bar{\lambda}^*x = (x^*A)x = x^*(Ax) = \lambda x^*x$. $x^*x$ is nonzero and real. So we get $\bar{\lambda} = \lambda$, which means that $\lambda$ is real.
		\item Assume $Av =\lambda v$ and  $ Aw =\mu w$. Then  \[ \lambda(v, w) = (\lambda v, w) = (Av, w) = (v, A^T w) = (v, Aw) = (v, \mu w) = \mu (v, w) \] is possible only if $(v, w) = 0$.
	\end{itemize}

	\section{Problem}
	\begin{itemize}
		\item $ |B| =  |UDU^{-1}| = |DU^{-1}U| = |D| = \prod_i{\lambda_i} $
		\item $ tr(B) = tr(UDU^{-1}) = tr(DU^{-1}U) = tr(D) = \sum_i{\lambda_i} $
	\end{itemize}

	\section{Problem}
	\begin{itemize}
		\item $ h(x_0) = w_0 + w^T x_0 = 0 \Rightarrow w^T x_0 = -w_0 $
		\item $ h(x_1) = w_0 + w^T x_1 = 0 \\
		h(x_2) = w_0 + w^T x_2 = 0 \\
		h(x_1) - h(x_2) = w^T x_1 - w^T x_2 =  w^T (x_1 - x_2) = 0 $
		\item Normal vector is defined as $\nabla h(x) = w$. $ \frac{\nabla h(x)}{|| \nabla h(x)||} = \frac{w}{||w||}$ is a unit normal vector. So $ \hat{w} = \frac{w}{||w||}$ is a unit normal vector.
		\item Distance can be found by projecting $(x - x_0)$ onto normal vector $w$. \[ D = |proj_w(x-x_0)| =\frac{w^T (x - x_0)}{||w||} = \hat{w}(x-x_0) = \frac{w^T x - w^T x_0}{||w||}  \overset{w^T x_0 = -w_0}{= \joinrel = \joinrel = \joinrel = \joinrel = \joinrel = \joinrel =} \frac{w^T x + w_0}{||w||} \]
	\end{itemize}

\end{document}


\documentclass[12pt]{article}

\usepackage[top=1in, bottom=1in, left=1in, right=1in]{geometry}
\usepackage{graphicx}
\usepackage{setspace}
\usepackage{bm}
\usepackage{amsmath}
\usepackage{amssymb,amsmath}
\usepackage{listings}
\usepackage{color}
\usepackage{enumitem}
\usepackage{fancyvrb}
\usepackage{hyperref}
\usepackage{diagbox}
\usepackage{float}
\geometry{letterpaper}
\linespread{1.1}% \geometry{landscape} % rotated page geometry

\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}
\definecolor{outcolor}{rgb}{0.545, 0.0, 0.0}

\lstdefinestyle{mystyle}{
	backgroundcolor=\color{backcolour},
	commentstyle=\color{codegreen},
	keywordstyle=\color{magenta},
	numberstyle=\tiny\color{codegray},
	stringstyle=\color{codepurple},
	basicstyle=\footnotesize,
	breakatwhitespace=false,
	breaklines=true,
	captionpos=b,
	keepspaces=true,
	numbers=left,
	numbersep=5pt,
	showspaces=false,
	showstringspaces=false,
	showtabs=false,
	tabsize=2
}

\lstset{style=mystyle}

\title{HW 2: State Estimation in Oil \& Gas Well Drilling}
\date{16 Feb. 2018}
\author{Franklin Zhao \\ SID: 3033030808}

\begin{document}

	\maketitle
	\newcommand{\tabitem}{~~\llap{\textbullet}~~}
	\renewcommand\theequation{\arabic{equation}}
	\renewcommand{\figurename}{Fig.}
	\renewcommand\thesection{Problem \arabic{section}:}
	\renewcommand\thesubsection{(\alph{subsection})}
	\onehalfspacing

\section{Dynamic System Modeling}
\subsection{}
\textbf{Modeling objective:} The objective is to formulate a mathmetical model that estimates the drill bit velocity given the table torque $T(t)$.\\
\textbf{Controllable input:} Table torque $\bf{T(t)}$\\
\textbf{Uncontrollable input:} Frictional torque $\bf{T_f(t)}$\\
\textbf{Measured output:} Table velocity $\bf{\omega_T(t)}$\\
\textbf{Performance output:} Bit velocity $\bf{\omega_B(t)}$
\subsection{}
For the top/table:
\begin{equation}
J_T\ddot{\theta}_T(t)=-k(\theta_T(t)-\theta_B(t))-b\omega_T(t)+T(t)
\end{equation}
For the bottom/bit:
\begin{equation}
J_B\ddot{\theta}_B(t)=k(\theta_T(t)-\theta_B(t))-b\omega_B(t)-T_f(t)
\end{equation}
\subsection{}
\begin{equation}
\frac{\partial}{\partial t}\left[
\begin{array}{c}
\theta_T\\
\dot\theta_T\\
\theta_B\\
\dot\theta_B
\end{array}\right]=\underbrace{\left[
\begin{array}{cccc}
0&1&0&0\\
-\frac{k}{J_T}&-\frac{b}{J_T}&\frac{k}{J_T}&0\\
0&0&0&1\\
\frac{k}{J_B}&0&-\frac{k}{J_B}&-\frac{b}{J_B}
\end{array}\right]}_\text{\textbf{A}}\left[
\begin{array}{c}
\theta_T\\
\dot\theta_T\\
\theta_B\\
\dot\theta_B
\end{array}\right]+\underbrace{\left[
\begin{array}{c}
0\\
\frac{1}{J_T}\\
0\\
0
\end{array}
\right]}_\text{\textbf{B}}T+\left[
\begin{array}{c}
0\\
0\\
0\\
-\frac{1}{J_B}
\end{array}
\right]T_f
\end{equation}
\begin{equation}
\dot\theta_T=\underbrace{\left[
\begin{array}{cccc}
0&1&0&0
\end{array}
\right]}_\text{\textbf{C}}\left[
\begin{array}{c}
\theta_T\\
\dot\theta_T\\
\theta_B\\
\dot\theta_B
\end{array}
\right]
\end{equation}
\section{Observability Analysis}
\subsection{}
\lstinputlisting[language=Python]{2a.py}
\begin{Verbatim}
Rank of Observability Matrix for four-state system: 3
\end{Verbatim}
Since $rank(\mathcal{O})=3<4$, $(A,C)$ is not observable.
\subsection{}
\begin{equation}
\frac{\partial}{\partial t}\left[
\begin{array}{c}
\theta\\
\dot\theta_T\\
\dot\theta_B
\end{array}\right]=\underbrace{\left[
	\begin{array}{ccc}
	0&1&-1\\
	-\frac{k}{J_t}&-\frac{b}{J_t}&0\\
	\frac{k}{J_B}&0&-\frac{b}{J_B}
	\end{array}\right]}_\text{\textbf{A}}\left[
\begin{array}{c}
\theta\\
\dot\theta_T\\
\dot\theta_B
\end{array}\right]+\underbrace{\left[
	\begin{array}{c}
	0\\
	\frac{1}{J_T}\\
	0
	\end{array}
	\right]}_\text{\textbf{B}}T+\left[
\begin{array}{c}
0\\
0\\
-\frac{1}{J_B}
\end{array}
\right]T_f
\end{equation}
\begin{equation}
\dot\theta_T=\underbrace{\left[
	\begin{array}{cccc}
	0&1&0
	\end{array}
	\right]}_\text{\textbf{C}}\left[
\begin{array}{c}
\theta\\
\dot\theta_T\\
\dot\theta_B
\end{array}
\right]
\end{equation}
\subsection{}
\lstinputlisting[language=Python]{2c.py}
\begin{Verbatim}
Rank of Observability Matrix for three-state system: 3
\end{Verbatim}
Since $rank(\mathcal{O})=3$, $(A,C)$ is observable.
\section{Measurement Data}
\lstinputlisting[language=Python]{3.py}
The plots of table torque and measured table velocity versus time are shown in Figure~\ref{fig:3}.
\begin{figure}[H]
	\centering
	\includegraphics[width=\linewidth]{3.png}
	\caption{Table torque and measured table velocity versus time}
	\label{fig:3}
\end{figure}
\section{Luenberger Observer}
\subsection{}
Denote $\left[
\begin{array}{c}
\hat\theta\\
\hat{\dot\theta}_T\\
\hat{\dot\theta}_B
\end{array}
\right]$ by $\hat x(t)$, $\hat{\dot\theta}_T$ by $\hat y(t)$, $T(t)$ by $u(t)$, and the Luenberger observer equations will be:
\begin{equation}\label{eq:luen1}
\dot{\hat x}(t)=A\hat x(t)+Bu(t)+L[y(t)-\hat y(t)],\ \ \ \hat x(0)=\hat{x}_0
\end{equation}
\begin{equation}\label{eq:luen2}
\hat y(t)=C\hat x(t)
\end{equation}
\lstinputlisting[language=Python]{4a.py}
\begin{Verbatim}
Eigenvalues of open-loop system:
[-0.08338525+0.29860789j -0.08338525-0.29860789j -0.08322949+0.j        ]
\end{Verbatim}
\subsection{}
The eigenvalues I chose were $2.5\lambda(A)$. The choice was justified in the RMSE error to be computed and converge speed to be shown in part (d). Basically I chose \{5, 1, 2, 3, 2.5\} and did the validation. It turned out that $2.5\lambda(A)$ was the best choice.
\lstinputlisting[language=Python]{4b.py}
\begin{Verbatim}
L:
[[-25.125]
[  0.375]
[ -0.675]]
\end{Verbatim}
\subsection{}
Write Equation~(\ref{eq:luen1})(\ref{eq:luen2}) in LTI form, and we obtain $A_{lobs}$, $B_{lobs}$, $C_{lobs}$:
\begin{equation}
\dot{\hat x}(t)=\underbrace{(A-LC)}_{A_{lobs}}\hat x(t)+\underbrace{\left[
\begin{array}{cc}
B&L
\end{array}
\right]}_{B_{lobs}}\left[
\begin{array}{c}
u(t)\\
y(t)
\end{array}
\right]
\end{equation}
\begin{equation}
\hat y(t)=C_{lobs}\hat x(t)
\end{equation}
\subsection{}
The eigenvalues I chose were $2.5\lambda(A)$:
\begin{Verbatim}
array([-0.20846314+0.74651974j, -0.20846314-0.74651974j, -0.20807373+0.j])
\end{Verbatim}
\lstinputlisting[language=Python]{4d.py}
\begin{Verbatim}
Luenberger Observer RMSE: 0.0991557694635
\end{Verbatim}
The result of Luenberger observer is shown in Figure~\ref{fig:4}
\begin{figure}[H]
	\centering
	\includegraphics[width=\linewidth]{4.png}
	\caption{Luenberger observer result}
	\label{fig:4}
\end{figure}
\newpage
\section{Kalman Filter (KF) Design}
\subsection{}
\begin{equation}
\dot{\hat x}(t)=A\hat x(t)+Bu(t)+L(t)(y(t)-\hat y(t))
\end{equation}
\begin{equation}
\hat y(t)=C\hat x(t)
\end{equation}
\begin{equation}
L(t)=\Sigma(t)C^TN^{-1}
\end{equation}
where $\Sigma(t)$ is the solution of the following differential equation:
\begin{equation}
\dot\Sigma(t)=\Sigma(t)A^T+A\Sigma(t)+W-\Sigma(t)C^TN^{-1}C\Sigma(t),\ \ \ \Sigma(0)=\Sigma_0
\end{equation}
where $W$ and $N$ are covariance matrices of $w(t)$ and $n(t)$.
\subsection{}
\lstinputlisting[language=Python]{5b.py}
\begin{Verbatim}
Kalman Filter RMSE: 0.0560275348234 rad/s
\end{Verbatim}
The value of $W$ I tuned is listed as follow. Basically I determined the magnitude first. Then based on the plots in part (c), I tuned the diagonal elements, and finally tuned other elements. It turned out that the RMSE did not change too much, but $\Sigma_{33}$ changed dramatically.
\begin{Verbatim}
W = np.matrix([[0.0006, 0.005, 0.001],
	       [0.02,   0.003, 0.004],
	       [0.004,  0.001, 0.003]])
\end{Verbatim}
\subsection{}
\lstinputlisting[language=Python]{5c.py}
The result of Kalman Filter is shown in Figure~\ref{fig:5}.
\begin{figure}[H]
	\centering
	\includegraphics[width=\linewidth]{5.png}
	\caption{Kalman filter result}
	\label{fig:5}
\end{figure}
\subsection{}
\lstinputlisting[language=Python]{5d.py}
\begin{Verbatim}
array([-0.08707987+0.2849352j, -0.08707987-0.2849352j, -0.34475403+0.j])
\end{Verbatim}
The 1st and 2nd poles are very close to the eigenvalues I chose in the Luenberger observer, but the 3rd one is a bit more different.
\section{Extended Kalman Filter (EKF) Design}
Replace Hooke's law with nonlinear spring torque relationship, and the EKF can be obtained:
\begin{equation}
\underbrace{\frac{\partial}{\partial t}\left[
\begin{array}{c}
\theta\\
\dot\theta_T\\
\dot\theta_B
\end{array}
\right]}_{\hat x(t)}=\underbrace{\left[
\begin{array}{c}
\hat\theta_T - \hat\theta_B\\
\frac{-k_1\theta-k_2\theta^3-b\dot\theta_T+T(t)}{J_T}\\
\frac{k_1\theta+k_2\theta^3-b\dot\theta_B}{J_B}
\end{array}
\right]}_{f(x(t),u(t))}+w(t),\ \ \ x(0)=x_0
\end{equation}
\begin{equation}
\dot\theta_T=\underbrace{C\left[
\begin{array}{c}
\theta\\
\dot\theta_T\\
\dot\theta_B
\end{array}
\right]}_{h(x(t),u(t))}+n(t)
\end{equation}
Hence, $F(t)$ and $H(t)$ can be derived:
\begin{equation}
F(t)=\frac{\partial f}{\partial x}(\hat x(t),u(t))=\left[
\begin{array}{ccc}
0&1&-1\\
\frac{-k_1-3k_2\hat\theta^2}{J_T}&\frac{-b}{J_T}&0\\
\frac{k_1+3k_2\hat\theta^2}{J_B}&0&\frac{-b}{J_B}
\end{array}
\right]
\end{equation}
\begin{equation}
H(t)=\frac{\partial h}{\partial x}(\hat x(t),u(t))=C=\left[
\begin{array}{ccc}
0&1&0
\end{array}\right]
\end{equation}
\lstinputlisting[language=Python]{6.py}
\begin{Verbatim}
Extended Kalman Filter RMSE: 0.0458628285791 rad/s
\end{Verbatim}
The result of EKF is shown in Figure~\ref{fig:6}.
\begin{figure}[H]
	\centering
	\includegraphics[width=\linewidth]{6.png}
	\caption{EKF result}
	\label{fig:6}
\end{figure}
\end{document}

HW2 - State Estimation/ZHAO_FRANKLIN_HW2.tex


\subsection{A Gluing Construction}
In previous parts, we saw that we can make $\log$ is a complete analytic function.
But we are not entirely satisfied, as it is just a bunch of function elements related together, instead of a genuin function.
We shall construct a space as a ``bigger domain'' $R$ at which we can realise $\log$ as a geniune function.\\
The idea is to consider the function elements we defined earlier and glue them together.
We define
$$R=\left( \coprod_{n\in\mathbb Z}U_{I(n)} \middle)\right/\sim$$
where $z_1\in U_{I(m)}$ and $z_2\in U_{I(n)}$ have $z_1\sim z_2$ iff $z_1=z_2$ as elements of $\mathbb C$ and $f_{I(m)}(z_1)=f_{I(n)}(z_2)$.
We give $R$ the quotient topology.
One can imagine $R$ as an ``infinite multi-storey carpark'' that spirals up and down.
\begin{remark}
    Since $F_{I(m)}\approx F_{I(n)}$ for all $m,n\in\mathbb Z$, it follows that $R$ is path-connected by following the sequence of direct analytic continuations.
\end{remark}
Then, with this construction, we can extend all the $f_{I(n)}$ on the $U_{I(n)}$ to a global function $f:R\to\mathbb C$ by $f([z])=f_{I(n)}(z)$ for $z\in U_{I(n)}$.
\begin{proposition}
    $f$ is well-defined.
\end{proposition}
\begin{proof}
    Follows directly from our definition of the equivalence relation $\sim$.
\end{proof}
Similarly, the natural inclusions $U_{I(n)}\hookrightarrow\mathbb C_\star$ can be extended to a global function $\pi:R\to\mathbb C_\star$ with $\pi([z])=z$.
One can also easily verify that $\pi$ is well-defined.
There is a very nice relationship between $f,\pi$ and the usual exponential map.
Indeed, $\exp\circ f=\pi$, which basically tells us $f$ has basically everything we want from $\log$.\\
We can use these global functions together.
Define $\Phi([z])=(\pi([z]),f([z]))$, then $\Phi$ is injective by definition of $\sim$.
Therefore $R$ is Hausdorff as $\mathbb C^2$ is.
\begin{remark}
    $\Phi$ indeed identifies $R$ with the graph $\{(w,z)\in\mathbb C^2:w=\exp(z)\}$.
    So we can view $R$ alteratively as ``flipping'' the graph of $\exp$.
\end{remark}


\section{Empirical relevance}
\begin{frame}{Empirical relevance: definition}
    The \alert{empirical relevance} $r_i(t)$ of node $i$ at time $t$ is defined as:

    \[
        r_i(t) = \frac{n_i(t)}{n_i^{PA}(t)}
    \]

    \begin{itemize}
        \item $n_i(t) = \frac{\Delta k_i^{in}(t, \Delta t)}{L(t, \Delta t)}$: ratio between:

        \begin{itemize}
            \begin{footnotesize}
            \item $\Delta k_i^{in}(t, \Delta t)$: \# of incoming links received by node $i$ in the time window $[t, t+\Delta t]$
            \item $L(t, \Delta t)$: total \# of links created within the same time window
        \end{footnotesize}
        \end{itemize}

        \item $n_i^{PA}(t) = \frac{k_i^{in}(t)}{\sum_j k_j^{in}(t)}$: expected value of $n_i(t)$ according to preferential attachment alone
    \end{itemize}

    $r_i(t) > 1$ \alert{$(< 1)$}: node $i$ at time $t$ outperforms \alert{(underperforms)} in the competition for incoming links with respect to its preferencial attachment weight.
\end{frame}

\section{The Extended Fitness Model}

\begin{frame}{The Extended Fitness Model}
    \begin{itemize}
        \item \textbf{PageRank's under-performance in time-dependent networks is a general feature.}
        \item We can validate this using a model more compatible with the idea that \emph{a node is important if it's pointed by other important nodes.}
    \end{itemize}
    \begin{center}
        \alert{Extended Fitness Model} (EFM)
    \end{center}
    \begin{itemize}
        \item \textbf{High-fitness nodes are more sensitive to fitness} than low-fitness nodes, when choosing their outgoing links.
        \item High-fitness nodes are then more likely to be pointed by other high-fitness nodes than low-fitness nodes.
        \item EFM is \alert{more favorable} to PageRank than RM.
    \end{itemize}
\end{frame}

\begin{frame}{EFM:\@ sensitivity to fitness}
    Probability $\Pi_{i;j}^{in}(t)$ that a link created by node $j$ at time $t$ ends in node $i$:
    \[
        \Pi_{i;j}^{in}(t) \sim (k_i^{in}(t)+1)^{1-\eta_j} \, \eta_i^{\eta_j} \, f_R(t-\tau_i)
    \]
    \begin{itemize}
        \item Fitness $\eta \in [0, 1]$ to prevent negative exponents
        \item $\Pi^{in}$ depends on the fitness of the target \emph{and of the source} nodes (difference with RM).
        \item $k_i^{in}(t)$: indegree of node $i$ at time $t$.
    \end{itemize}
\end{frame}

\begin{frame}{PageRank vs.\ indegree: correlation with fitness in EFM}
    \begin{figure}
        \begin{columns}

        \column{0.6\textwidth}
            \includegraphics[width=1.0\textwidth]{figures/PageRankEFM_heatmap}

        \column{0.4\textwidth}
            \begin{footnotesize}
            \begin{itemize}
                \item $\rho(A) = 2A^{-3}, \; A \in [1, \infty]$
                \item $H$ nodes: high fitness; $\eta \in [10^{-5}, 1]$
                \item $(N-H)$ nodes: low fitness; $\eta \in [0, 10^{-5}]$
                \item $N=10000$, $H=250$
            \end{itemize}
            \end{footnotesize}
        \end{columns}
        \caption{Comparison of performance of PageRank and indegree (EFM data).}
    \end{figure}
\end{frame}


\documentclass[../../main.tex]{subfiles}
\begin{document}
\subsubsection{Append}

The operation $append$ will return a Collection with a Value added at a specified numeric Index.
\begin{schema}{Append[Collection, V, \nat]}
  coll?, coll! : Collection \\
  v? : V \\
  idx? : \nat \\
  append~\_ : Collection \cross V \cross \nat \bij Collection
  \where
  \#~idx? = 1 \\
  coll! = append(coll?, v?, idx?) @ \\
  \t1 let ~ ~ coll' == front(\{~ i : \nat ~| ~ i \in 0~..~idx?\} \extract coll?) \cat v? \\
  \t1 \ \ \ ~~~ coll'' == \{~ j : \nat ~| ~ j \in idx?~..~\#~coll?\} \extract coll? \\
  \t1 = coll' \cat coll'' \implies \\
  \t2 (front(coll')  \cat v? \cat  coll'') ~ \land \\
  \t2 (~ v? \mapsto idx? \in coll!) ~\land \\
  \t2 (~\#~ coll! = \#~ coll? + 1)
\end{schema}
$append$ results in the composition of $coll'$ and $coll''$ such that
$$coll! = coll' \cat coll'' ~ \land ~idx? \mapsto v? \in coll!$$
\begin{itemize}
\item $coll'$ is the items in $coll?$ up to and including $idx?$ but the value at $idx?$ is replaced with $v?$ such that
  $idx? \mapsto coll?_{idx?} \not \in coll'$
\item $coll''$ is the items in $coll?$ from $idx?$ to $\#~coll? \implies coll?_{idx?} \in coll''$
\end{itemize}
 The following example illustrates these properties.
\begin{argue}
  X = \langle x_{0}, x_{1}, x_{2} \rangle \\
  \t1 x_{0} = 0 \\
  \t1 x_{1} = foo \\
  \t1 x_{2} = \langle a, b, c \rangle \\
  \t1 v? = bar \\
  append(X, v?, 0) = \langle bar, 0, foo, \langle a, b, c \rangle \rangle \\
  append(X, v?, 1) = \langle 0, bar, foo, \langle a, b, c \rangle \rangle \\
  append(X, v?, 2) = \langle 0, foo, bar, \langle a, b, c \rangle \rangle \\
  append(X, v?, 3) = \langle 0, foo, \langle a, b, c \rangle, bar \rangle \\
  append(X, v?, 4) = append(X, v?, 3) \iff 3 \not \in \dom X
\end{argue}
\end{document}


\section{Defining data for Oriented TQFT}
This is the defining data for $(d + 1)$-dimensional oriented TQFT.
It Closed $d$-dimensional $\Sigma^d$
has vector space $V(\Sigma^d)$.
When $M^{d+1}$ is closed, meaning it has no boundary,
$Z(M^{d+1})\in\mathbb{C}$ is just a number.
If $M^{d+1}$ has a boundary,
then $Z(M^{d+1})\in V(\partial M^{d+1})$.
This is a state.
$Z$ is the path integral on the spacetime manifold.
You can think of it as a wave function over boundary conditions.

These are the axioms.
All manifolds have orientation.

\begin{enumerate}
    \item $Z$ and $V$ are functorial with respect to orientation-preserving
    diffeomorphisms of $M^{d+1}$ and $\Sigma^d$.
    \item $V$ should be ``involutory'' in the sense that
    $V(\bar{\Sigma^d}) = V(\Sigma^d)^*$.
    Here the $*$ means dual vector space.
    Over bar means hermitian conjugate.
    \item $Z$ and $V$ should be multiplicative.
    \item Non-triviality.
    \item $Z$ is hermitian.
\end{enumerate}

Let's explain the axioms.
\begin{axiom}[Axiom 1]
    Let $f:\Sigma_1^d \to \Sigma_2^d$ be a diffeomorphism.
    This defines an isomorphism
    $\mathcal{U}_f: V(\Sigma_1^d)\to V(\Sigma_2^d)$.
    If we also have $g: \Sigma_2^d \to \Sigma_3^d$,
    then
    $\mathcal{U}_g \mathcal{U}_f = \mathcal{U}_{gf}$.

    If $f$ extends to
    $\tilde{f}: M_1^{d+1}\to M_2^{d+1}$
    where $\partial M_1 = \Sigma_1$
    and $\partial M_2 = \Sigma_2$,
    then
    $\mathcal{U}_f\left( Z\left( M_1^{d+1} \right) \right)
    = Z\left( M_2^{d=1} \right)$.
\end{axiom}
Suppose we have a diffeomorphism from one circle $\Sigma_1 = S^1$ to another
circle.
Then there will be an induced map $\mathcal{U}_f$ from vector spaces between
then too.

\begin{question}
    Isn't $Z$ a path integral that is a number?
    Why is it a vector space?
\end{question}
You can think of $Z\left( M_1^{d+1} \right)$
as a state,
because like in path integrals you have boundary conditions,
which are variable.

\begin{question}
    What is a diffeomorphism?
\end{question}
A diffeomorphism $f:X_1 \to X_2$ is a smooth and continuous map.

\begin{question}
    What is a manifold?
\end{question}
Manifolds can be broken up into charts $\mathcal{U}_\alpha$
with coordinate charts $\phi: \mathcal{U}_\alpha\to \mathbb{R}^d$.

\begin{question}
What is the relation between $\mathcal{U}_{f}$ and
$\mathcal{U}_{\tilde{f}}$?
\end{question}
They are actually the same.

\begin{question}
    The extension from $f$ to $\tilde{f}$ is not unique?
\end{question}
It's not unique.

\begin{question}
    $\mathcal{U}_f$ acting on the partition function
    gives the other partition function.
    This is the sum of all paths.
\end{question}
Don't think of it like that.
This thing is only acting on the partition function.
Now you're saying $Z$ is the sum of something
and how does $U_{f}$ act on terms in the sum.
I'm hesitant to say anything general about that.
There's no general clean answer.
Just think of it as an abstract object $Z$
which gives a state if there is a boundary
and a number otherwise.

Let's talk about axiom 3.
Imagine if I build a manifold by gluing simpler manifolds.
Then I can build my vector spaces by some tensor products and multiplications.
Recall that $\sqcup$ is the disjoint union.
Then we can state the axiom.
\begin{axiom}[Axiom 3]
    Let
    \begin{align}
        V\left( \Sigma_1^d \sqcup \Sigma_2^d  \right)
        =
        V\left( \Sigma_1^d \right) \otimes
        V\left( \Sigma_2^d \right)
    \end{align}
    Suppose the boundary is
    \begin{align}
        \partial M^{d+1} =
        \Sigma_1 \sqcup \bar{\Sigma}_2
    \end{align}.
    Then
    \begin{align}
        Z\left( M_1^{d=1} \right) &\in
        V\left( \Sigma_1 \right) \otimes V\left( \bar{\Sigma}_2 \right)\\
        &\simeq V\left( \Sigma_1 \right) \otimes \overbar{V(\Sigma_2)}
    \end{align}
    so we can think of $Z\left( M^{d+1}_1 \right)$
    as a state in
    $V\left( \Sigma_1 \right)\otimes V\left( \overbar{\Sigma_2} \right)$
    or as a linear map from
    $V\left( \Sigma_2 \right)\to V\left( \Sigma_1 \right)$.
\end{axiom}
I can think of this as a state in a tensor product of two spaces.
\begin{align}
    Z\left( M_1^{d+1} \right)
    = \sum_{n,m} c_{n,m}
    \ket{n}_{V(\Sigma_1)}
    \otimes \ket{m}_{V(\overbar{\Sigma_2})}
\end{align}
or you can think of it as
\begin{align}
    Z\left( M_1^{d+1} \right)
    = \sum_{n,m} c_{n,m}
    \ket{n}_{V(\Sigma_1)}
    \bra{m}_{V(\overbar{\Sigma_2})}
\end{align}

\begin{question}
This only applies when the boundary is a disjoint union of two manifolds?
\end{question}
You can always think of the boundary as $\partial M^{d+1}=\Sigma_1 \sqcup
\emptyset$

\begin{question}
    Why do we need the complex conjugate?
\end{question}
It's a state on $V(\Sigma_1)\otimes V(\ovebar{\Sigma_2})$.
Actually maybe it shouldn't be there,
put a question mark there.
Actually, there should be no complex conjugate.

We can think of the path integral as a state on the vector space
$V(\Sigma_1)\otimes\overbar{V(\Sigma_2)}$
or as a map from
$V(\Sigma_2)\to V(\Sigma_1)$.

I can think of a state as a map from the vector space to the empty manifold.
I'm just moving pieces around no each side.

It's really nothing more than thinking of it as bras and kets.
Just think of this as a definition.
I have a state that lives on a tensor product of two vector spaces.
I can think of it as an operator from one vector space to another.
It's really just that simple.
There's nothing more to that.
Forget that we called it the path integral.
You can think of any tensor product state as an operator from one space to
another.

\begin{question}
    When you have 3 disjoint components of $\Sigma$,
    then how do you define orientation?
    How do you decided which orientation to use?
\end{question}
I don't think you need extra information about what the boundary orientation
looks like.
If this has + orientation, then are you saying that?

Actually I think there should be a star.
\begin{align}
    Z\left( M_1^{d+1} \right)
    = \sum_{n,m} c_{n,m}^*
    \ket{n}_{V(\Sigma_1)}
    \bra{m}_{V(\overbar{\Sigma_2})}
\end{align}
We'll come back to this.

The path integral is a state on the boundary.

We've done the multiplicative axiom for $V$.
We haven't gotten to the multiplicative axiom for $Z$ yet.
We want to glue together $M$s with common boundaries.

Suppose that we have some manifold $M_1$
and another manifold $M_2$.
The boundary $\partial M_1 = \Sigma_1 \sqcup \overbar{\Sigma_2}$
and the boundary
$\partial M_2 = \Sigma_2 \sqcup \overbar{\Sigma_3}$.
Then $M=M_1 \cup_{\Sigma_2} M_2$.
If you think of these $Z$ as linear maps,
then we just compose the maps.
\begin{align}
    Z(M) = Z(M_2)\circ Z(M_1): V(\Sigma_1) \to V(\Sigma_3)
\end{align}
or equivalently,
we can think of $Z(M)$ as the inner product of these two states,
where the inner product
\begin{align}
    Z(M) =
    {\langle Z(M_1), Z(M_1) \rangle}_{V(\Sigma_2)}
    \in V(\Sigma_1) \otimes V(\overbar{\Sigma_3})
\end{align}

\begin{question}
    The bar just means going counterclockwise right?
\end{question}
Yes.

Actually there are many ways of glueing manifolds.
When I identify points together,
I can actually identify with any points I want.
So in general we need a gluing map
$\phi: \Sigma_2 \to \Sigma_2$
with
$M = M_1 \cup_\phi M_2$.

I'm going to use Axiom 1
and this $\phi$ map is going to define a corresponding map
$\mathcal{U}_\phi: V(\Sigma_2)\to V(\Sigma_2)$
so
\begin{align}
    Z\left( M_1 \cup_\phi M_2 \right)
    =
    Z(M_2) \circ \mathcal{U}_\phi \circ Z(M_1).
\end{align}
Or equivalently
$\langle Z(M_2), \mathcal{U}_\phi Z(M_1)\rangle_{V(\Sigma_2)}$.
$\mathcal{U}_\phi$ is an isomorphism between these two vector spaces.

Closed $M^{d=1}$
can be obtained by gluing together
$M_1$, $M_2$ along the their common boundary.
For example,
if you wanted to calculate the path integral on a sphere,
you blue toegether the two on a common boundary.

If you want a torus,
ou glue
\begin{align}
    T^2 = (S_1\otimes I) \union (S^1\times I).
\end{align}

That was the only non-trivial axiom,
the rest are pretty simple.

Funnily enough,.
This is about what happens when we have a non-empty manifold.
the next axiom is called non-triviality

\begin{axiom}[Axiom 4: non-triviality]
    Consider a boundary $\Sigma^d = \emptyset$.
    Then
    V(\emptyset) = V(\emptyset) \otimes V(\emptyset)
    and
    \begin{align}
        V(\emptyset) =
        \begin{cases}
            0 & \text{ordinal}\\
            \mathbb{C} & 1-dimensional
        \end{cases}
        hence we have
        \begin{align}
            V(\phi) = \mathbb{C}
        \end{align}
    \end{align}
    If $M^{d+1}$ is closed,
    $\partial M^{d+1} = \emptyset$,
    then
    $Z(M^{d+1}) \in V(\emptyset) \simeq \mathbb{C}$.
    Consider the case
    \begin{align}
        M^{d+1} = \emptyset.
    \end{align}
    Then
    \begin{align}
        Z(\emptyset) = Z\emptyset) Z(\emptyset)
    \end{align}
    which implies
    \begin{align}
        Z(\emptyset) =
        \begin{cases}
            0 &\\
            1
        \end{cases}.
    \end{align}
    Take $Z(\emptyset)=1$.
\end{axiom}

consider $M^{d+1}= \sigma^d \times I$.
Then
\begin{align}
    Z\left( \Sigma^d \times I \right)
    =
    \begin{cases}
        0\\
        II_{V(\Sigma^d)}
    \end{cases}
\end{align}
Take
$Z(\Sigma^d\times I) = II_{V(\Sigma^d)}
= \sum_{n=1}^{\dim(V(\Sigma^d))}
\ket{n}\bra{n}$
where $\{\ket{n}\}$ is orthonormal basis in $V(\Sigma^d)$.
Or,
\begin{align}
    Z(\Sigma^d\times I) =
    \sum_n
    \ket{n}_{V(\Sigma^d)}
    \otimes
    \ket{n}_{V(\Sigma)}^d.
\end{align}

The last axiom is simple.
\begin{axiom}[Hermitian]
    $Z(\overbar{M^{d+1}}) = Z(M^{d+1})^\dagger$.
\end{axiom}

\section{Non-trivial consequences}
\subsection{Homotopy invariance of maps}
This has some non-trivial consequences.
Consider $M_1^{d+1}=\Sigma^d\times I$
and consider $M_2^{d+1}=\Sigma^2\times I$
with homotopy
$F: M_1\to M_2$.
Then
$\mathcal{F}(\Sigma^d, t)
= f_t(\Sigma^d)$
for $f_t: \Sigma_d\to \Sigma^d$.
This defines a continuous family
$\mathcal{U}_{f_t}: \Sigma^d\to \Sigma^d$ satisfying
$\mathcal{U}_{f_t} = \mathcal{U}_{f_{t'}}$.

[diagram here]

If I have a homotopy group with a homotopy of maps.

\subsection{Mapping Class Group}
That leads us to the important group for a manifold.
For every manifold, there is a mapping class group $\Sigma^d$.
Colloquially, it's the group of ``large'' diffeomorphisms.
You would take all diffeomorphisms and mod out all those that are continuously
\begin{align}
    \mathrm{MCG}(\Sigma^d) = \frac{\mathrm{Diff}(\Sigma^d)}{\mathrm{diff}_0(\Sigma^d)}
\end{align}
Two diffeomorphisms are equivalent if they can be connected to each other.

This is the group taking all possible diffeomorphisms
and modding out all the ones equivalent to the identity.

The mapping class group is related to the homotopy group,
but not the same thing.
You may not understand this,
but
\begin{align}
    \mathrm{MCG}(\Sigma^d)
    = \mathrm{out}(\pi_1(\Sigma^d))
\end{align}
It's isomorphic to the group of all outer maps between loops.
An example of such a map that maps a non-trivial loop on an torus
to the other type of non-trivial loop around the torus.

In TQFT,
all my maps are smooth.
If my maps aren't smooth,
I don't have something that takes me everywhere from the left to everywhere on
the right.
If I have a smooth map,
that's what allows me to take vector spaces on the left to vector space on the
right.
The only reason I can go from left to right is because $\mathcal{F}$ is a
diffeomorphism.

The punchline is this.
For each $g\in \mathrm{MCG}(\Sigma^d)$,
we get some $\mathcal{U}_g: V(\Sigma^d) \to V(\Sigma^d)$.
If you figure out the $\mathcal{U}_g$,
you can completely figure out the TQFT.
These $\mathcal{U}_g$ are very important,
and can basically characterise the TQFT.

Is this the $S$ and $T$ matrices?
Yes.
For every element in the MCG  of a torus,
there is a linear map that acts on the ground state subspace on the system,
hence the $S$ and $T$ matrices on the subspace.
Don't worry if you've never heard of it.

Do you have a concise reference to recommend for the last two lectures?
There's an article from Atiyah 1988.
It's basically called TQFT.
My whole representation is based on this article.

\begin{question}
    Is it readable?
\end{question}
It's as readable as my lectures are understandable.
If you don't understand my lecture,
then I'm not sure.
It's not too bad,
it doesn't assume you know that much.

We're basically done,
but there's one more consequence to talk about.
These TQFTs are very baby versions of TQFTs,
and in real applications, we need some extra structure.