\section{Deep Learning} \label{sec:bt/DNN}

Deep learning is a powerful extension of the simple learning algorithms detailed above. In theory, any problem which can be modeled as the mapping of an input vector to an output vector can be solved by deep learning \cite{goodfellow2016}. In other words, any function of any complexity can be approximated given sufficiently large models and sufficiently large labeled training examples.

\subsection{Deep Feedforward Neural Networks}

Feedforward neural networks constitute a branch of deep learning covering models with a one-directional data flow from input to output. They are called \textit{neural networks} because they are loosely inspired by the neural circuitry of the human brain. Extending this analogy, all human perception may be considered one vast deep feedforward neural network. It works by mapping sensory input from sight, touch, smell, taste, and hearing through countless layers of neurons to output what we perceive of the world. Artificial neural networks work similarly by mapping a vector of input data points to one or more output values.

Even in their simplest form, deep feedforward neural networks overcome many challenges that linear models face. Linear regression, for instance, may never accurately approximate a non-linear function or capture interactions between two or more input variables. This limitation may only be overcome by introducing a nonlinearity to the input $\bm{x}$. We encapsulate this nonlinearity as $\phi(\bm{x})$. Thus, the equivalent to equation \ref{eq:bt_linReg} becomes equation \ref{eq:bt/dnn/dfnn}.

\begin{equation}
    \label{eq:bt/dnn/dfnn}
    y=f(\bm{x};\bm{\theta},\bm{x})=\phi(\bm{x};\bm{\theta})^T\bm{w}
\end{equation}

We now have another set of trainable parameters $\bm{\theta}$, meaning that the final form of $\phi$ is yet to be determined. By allowing the model to experience a given set of labeled training data, $\bm{\theta}$ will gradually converge towards a value that enables the full model to approximate the data optimally.
% be trained from experiencing a given set of labeled training data.
The parameters defined by $\bm{w}$ may subsequently map from $\phi(\bm{x})$ to the desired output. $\phi$ is called a \textit{hidden layer} because its outputs are unrelated to either the input or output vectors. It can instead be thought of as a transformation of the input layer.
% We can think of $\phi$ as a transformation of the input layer.
Its output is hence a new representation or interpretation of the information provided by $\bm{x}$.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.5\textwidth]{figures/bt_dfnn.png}
    \caption{Example of a simple \acrshort{mlp}. Left representation depicts individual neurons of each layer and the edges between them. The right representation is more common, depicting each layer with vector notation. }
    \source{\cite{goodfellow2016}}
    \label{fig:bt_dfnn}
\end{figure}

\subsubsection{\acrlong{mlp}}

A \acrfull{mlp} is one form of feedforward neural network, the simplest implementation of which is illustrated in figure \ref{fig:bt_dfnn}. Here we have two input neurons, one hidden layer with two neurons, and one output layer with one neuron. With vector notation, we say that the hidden layer $\bm{h}$ is a function of the input layer $\bm{x}$, with the relation $\bm{h}=f^{(1)}(\bm{x};\bm{W}, \bm{c})$. $\bm{h}$ is then the input to a third layer (which in this case is also the output layer) with the relation $y=f^{(2)}(\bm{h};\bm{W}, \bm{c})$. We can keep adding depth and width to the model this way, further increasing its complexity and potential hypothesis space. The entire model may now be given by equation \ref{eq:bt/dnn/mlp1}.

\begin{equation}
    \label{eq:bt/dnn/mlp1}
    f(\bm{x};\bm{W},\bm{c},\bm{w},b)=f^{(2)}(f^{(1)}(\bm{x}))
\end{equation}

To provide the nonlinearity we want, we define $\bm{h}=g(\bm{W}^T\bm{x}+\bm{c})$. This functon is commonly known as the \textit{activation function} of the layer. For deeper networks, it would be added to the output of every hidden layer. There are many activation functions to choose from, but the default recommendation in the machine learning community is to use the \acrfull{relu} \cite{agarap2018}. It is defined as $g(z)=max\{0,z\}$ and has been made popular for its simplicity without trading off on optimization with gradient-based backpropagation. The complete network of figure \ref{fig:bt_dfnn} can thus be given by equation \ref{eq:bt/dnn/mlp3}.

\begin{equation}
    \label{eq:bt/dnn/mlp2}
    y=f(\bm{x};\bm{W},\bm{c}, \bm{w}, b)=\bm{w}^Tg(\bm{W}^T\bm{x}+\bm{c}) + b
\end{equation}

\begin{equation}
    \label{eq:bt/dnn/mlp3}
    y=f(\bm{x};\bm{W},\bm{c}, \bm{w}, b)=\bm{w}^Tmax\{\bm{W}^T\bm{x}+\bm{c}\} + b
\end{equation}

\newpage
\subsection{\acrlong{cnn}s} \label{sec:bt/DNN/CNN}

A \acrfull{cnn} is a specialized kind of deep feedforward neural network that has shown remarkable results in computer vision and language processing. When first proposed by \textcite{lecun1989} in 1989, it revolutionized machine learning problems where data had a grid-structured topology. Such data could be interpreted with more insight while allowing for much deeper and scaleable model architectures. Before its inception, most practitioners had little faith in the advent of neural networks for machine learning. The success of \acrshort{cnn}s triggered a wave of interest, which laid the foundation for further research in neural networks and deep learning \cite{goodfellow2016}.

\subsubsection{Convolution}

To understand \acrshort{cnn}s, we need to understand what a \textit{convolution} is. Used in many engineering disciplines, it is simply a mathematical operator which expresses how two functions overlap \cite{weisstein2003}. In signal processing, for instance, convolution may be used to mathematically replicate how a given sound signal would behave in any conceivable environment. All one would need is the impulse response of this environment, which could be the recording of a clap or similar fast transient sound. Convolving these two signals would have the effect of "placing" the sound in this environment, artificially reproducing all reverberations. The operation is typically denoted with an asterisk and is defined as equations \ref{eq:bt/dnn/conv1}-\ref{eq:bt/dnn/conv2}. This particular version is known as discrete convolution, which is what we will be encountering for neural networks.

\begin{equation}
    \label{eq:bt/dnn/conv1}
    s(t)=(x*k)(t)
\end{equation}

\begin{equation}
    \label{eq:bt/dnn/conv2}
    s(t)=\sum_{a=-\infty}^{\infty}x(a)k(t-a)
\end{equation}

With \acrshort{cnn} terminology, $x$ is the input to a convolutional layer and $k$ is its \textit{kernel}. Since the input to a \acrshort{cnn} is often an image, time series, or other signals, $x$ and $k$ are usually multidimensional. Taking image classification as an example, the convolutional operations performed on all input pixels would look like equation \ref{eq:bt/dnn/conv3}.

\begin{equation}
    \label{eq:bt/dnn/conv3}
    S(i,j)=(K*X)(i,j)=\sum_{m}^{}\sum_{n}^{}X(i-m,j-n)K(m,n)
\end{equation}

Each pixel is denoted by $i$ and $j$ for its spatial coordinate in the input image. The output from convolving a kernel $K$ with an image $X$ is called a \textit{feature map} because it is essentially a mapping to a new "image" where features represented by the kernel are highlighted. A visual representation of how a 2-dimensional convolutional operation may be applied to an input is displayed in figure \ref{fig:bt_conv}.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\textwidth]{figures/bt_conv.png}
    \caption{Example of 2-dimensional convolution operation. If this were image classification, values in the input matrix would represent individual pixel activations in the image. The final value of each output cell is calculated by the convolution operation given by equation \ref{eq:bt/dnn/conv3}.}
    \source{\cite{goodfellow2016}}
    \label{fig:bt_conv}
\end{figure}

The kernel $K$ is what defines all properties of a convolutional model. Instead of learning individual parameters for every edge between every neuron, as is the case for \acrshort{mlp}s, \acrshort{cnn}s learn the internal weights of all kernels in the network. As we will see, \acrshort{cnn}s usually contain several kernels for every convolutional layer, each capturing one feature in the input to each layer.

\subsubsection{Properties of \acrshort{cnn}s}

What makes \acrshort{cnn}s so efficient is based on three essential ideas. First, since parameters are only present in each kernel, as described above, the network does not need a single parameter for all neurons in every layer. This idea is known as \textit{parameter sharing} and will reduce the storage requirements of large models. Another consequence of the convolutional layers is that each input neuron is only connected to a few outputs. As can be seen in figure \ref{fig:bt_conv}, only four neurons of the input are required to calculate the value of a cell in the output. This idea is called \textit{sparse interactions}. Contrary to an \acrshort{mlp}, which requires connections from every input to every output, this feature drastically improves both computing efficiency and memory requirements during training.

Finally, a concept known as \textit{equivariant representations} allows for the detection of features that are independent of spatial (or temporal) position in the input. In other words, if there is an object in the input to a convolutional layer that causes high activations in a feature map, the same object will cause an equally high activation in the same feature map if it were in another position. This effect is another consequence of the way in which feature maps are created by "moving" the same kernel across the entire input. \acrshort{mlp}s, in contrast, lack this property since each neuron is associated with an individual learned weight.
% The kernel will eventually reach and generate the convolutional output for that respective feature regardless of its po

% \subsection{Terms} \label{sec:bt/DNN/terms}

% Much machine learning terminology is considered common knowledge in most literature and is often neglected to be explained in detail. Some such terms, many of which will be used frequently during the course of this thesis, are detailed below.

% \subsubsection{Batch Normalization}
% \subsubsection{Dropout}
% % \subsubsection{Perceptron}
% % \subsubsection{\acrlong{relu}}
% \subsubsection{Max Pooling}
% \subsubsection{Softmax}
% \subsubsection{Cross Entropy Loss}
% \subsubsection{Precision, Accuracy and Recall}


\lab{Kalman Filter}{Kalman Filter}
\objective{Understand how to implement the standard Kalman Filter. Apply to the problem of projectile tracking.}

Measured observations are often prone to significant noise, due to restrictions on measurement accuracy.
For example, most commercial GPS devices can provide a good estimate of geolocation, but only within a dozen meters or so.
A Kalman filter is an algorithm that takes a sequence of noisy observations made over time and attempts to get rid of the noise,
producing more accurate estimates than the original observations.
To do this, the algorithm needs information about the system being observed.

Consider the problem of tracking a projectile as it travels through the air.
Short-range projectiles approximately trace out parabolas, but a sensor that is recording measurements of the projectile's position over time will likely show a path that is much
less smooth.
Because we know something about the laws of physics, we can filter out the noise in the measurements using basic Newtonian mechanics, recovering a more accurate estimate
of the projectile's trajectory.
In this lab, we will simulate measurements of a projectile and implement a Kalman filter to estimate the complete trajectory of the projectile.

\section*{Linear Dynamical Systems}
The standard Kalman filter assumes that:
(1) we have a linear dynamical system,
(2) the state of the system evolves over time with some noise, and
(3) we receive noisy measurements about the state of the system at each iteration.
More formally, letting $\mathbf{x}_{k}$ denote the state of the system at time $k$, we have
\begin{equation}
\mathbf{x}_{k+1} = F_{k} \mathbf{x}_{k} + G_{k}\mathbf{u}_{k} + \mathbf{w}_{k}
\label{eq:state}
\end{equation}
where $F_{k}$ is a state-transition model, $G_{k}$ is a control-input model, $\mathbf{u}_{k}$ is a control vector, and $\mathbf{w}_{k}$ is the noise present in state $k$.
This noise is assumed to be drawn from a multivariate Gaussian distribution with zero mean and covariance matrix $Q_{k}$.
The control-input model and control vector allow the assumption that the state can be additionally influenced by some other factor than the linear state-transition model.

We further assume that the states are ``hidden,'' and we only get the noisy observations
\begin{equation}
\mathbf{z}_{k} = H_{k}\mathbf{x}_{k} + \mathbf{v}_{k}
\label{eq:obs}
\end{equation}
where $H_{k}$ is the observation model mapping the state space to the observation space, and $\mathbf{v}_{k}$ is the observation noise present at iteration $k$.
As with the aforementioned error, we assume that this noise is drawn from a multivariate Gaussian distribution with zero mean and covariance matrix $R_{k}$.

The dynamics stated above are all taken to be linear.
Thus, for our purposes, the operators $F_k$, $G_k$, and $H_k$ are all matrices, and $\mathbf{x}_k$, $\mathbf{u}_k$, $\mathbf{z}_k$, and $\mathbf{v}_k$ are all vectors.

We will assume that the transition and observation models, the control vector, and the noise covariances are constant, i.e. for each $k$, we will replace $F_{k}, H_k, \mathbf{u}_{k}, Q_{k},$ and $R_{k}$ with $F, H, \mathbf{u}, Q,$ and $R$.
We will also assume that $G = I$ is the identity matrix, so it can safely be ignored.

\begin{problem}
Begin implementing a \li{KalmanFilter} class by writing an initialization method that stores the transition and observation models, noise covariances, and control vector.
We provide an interface below:
\begin{lstlisting}
class KalmanFilter(object):
    def __init__(self,F,Q,H,R,u):
        """
        Initialize the dynamical system models.

        Parameters
        ----------
        F : ndarray of shape (n,n)
            The state transition model.
        Q : ndarray of shape (n,n)
            The covariance matrix for the state noise.
        H : ndarray of shape (m,n)
            The observation model.
        R : ndarray of shape (m,m)
            The covariance matric for observation noise.
        u : ndarray of shape (n,)
            The control vector.
        """
        pass
\end{lstlisting}
\end{problem}

We now derive the linear dynamical system parameters for a projectile traveling through $\mathbb{R}^2$ undergoing a constant downward gravitational force of 9.8 $m/s^2$.
The relevant information needed to describe how the projectile moves through space is its position and velocity.
Thus, our state vector has the form
\[
\mathbf{x} = \left( \begin{array}{c} s_{x} \\ s_{y} \\ V_{x} \\ V_{y} \end{array} \right),
\]
where $s_x$ and $s_y$ give the $x$ and $y$ coordinates of the position (in meters), and $V_x$ and $V_y$ give the horizontal and vertical components of the velocity (in meters per second), respectively.

How does the system evolve from one time step to the next?
Assuming each time step is $0.1$ seconds, it is easy enough to calculate the new position:
\begin{align*}
s_x' &= s_x + 0.1V_x\\
s_y' &= s_y + 0.1V_y.
\end{align*}
Further, since the only force acting on the projectile is gravity (we are ignoring things like wind resistance), the horizontal velocity remains constant:
\[
V_x' = V_x.
\]
The vertical velocity, however, does change due to the effects of gravity.
From basic Newtonian mechanics, we have
\[
V_y' = V_y - 0.1\cdot9.8.
\]
In summary, over one time step, the state evolves from $\mathbf{x}$ to $\mathbf{x}'$, where
\[
\mathbf{x}' = \left( \begin{array}{c} s_{x} + 0.1V_x \\ s_{y} + 0.1V_y \\ V_{x} \\ V_{y} - 0.98 \end{array} \right).
\]
From this equation, you can extract the state transition model $F$ and the control vector $u$.

We now turn our attention to the observation model.
Imagine that a radar sensor captures (noisy) measurements of the projectile's position as it travels through the air.
At each time step, the radar transmits the observation $z = (z_x, z_y)$ given by
\begin{align*}
z_x &= s_x + v_x\\
z_y &= s_y + v_y,
\end{align*}
where $(v_x, v_y)$ is a noise vector assumed to be drawn from a multivariate Gaussian with mean zero and some known covariance.
These equations indicate the appropriate choice of observation model.

\begin{problem}
Work out the transition and observation models $F$ and $H$, along with the control vector $\mathbf{u}$, corresponding to the projectile.
Assume that the noise covariances are given by
\begin{align*}
Q &= 0.1 \cdot I_4\\
R &= 5000 \cdot I_2.
\end{align*}
Instantiate a \li{KalmanFilter} object with these values.
\end{problem}

\begin{figure}
    \centering
    \begin{tabular}{cc}
    \includegraphics[width=.45\textwidth]{states_evolution} &
    \includegraphics[width=.45\textwidth]{obs_evolution}
    \end{tabular}
    \caption{State sequence (left) and sampling of observation sequence (right).}
    \label{fig:evolution}
\end{figure}

We now wish to simulate a sequence of states and observations from the dynamical system.
In addition to the system parameters, we need an initial state $\mathbf{x}_0$ to get started.
Computing the subsequent states and observations is simply a matter of following equations \ref{eq:state} and \ref{eq:obs}.

\begin{problem}
Add a method to your \li{KalmanFilter} class to generate a state and observation sequence by evolving the system from a given initial state (the function \li{numpy.random.multivariate_normal} will be useful).
To do this, implement the following:
\begin{lstlisting}
def evolve(self,x0,N):
    """
    Compute the first N states and observations generated by the Kalman system.

    Parameters
    ----------
    x0 : ndarray of shape (n,)
        The initial state.
    N : integer
        The number of time steps to evolve.

    Returns
    -------
    states : ndarray of shape (n,N)
        States 0 through N-1, given by each column.
    obs : ndarray of shape (m,N)
        Observations 0 through N-1, given by each column.
    """
    pass
\end{lstlisting}

Simulate the true and observed trajectory of a projectile with initial state
\[
\mathbf{x}_0 = \left( \begin{array}{c} 0\\ 0 \\ 300 \\ 600\end{array} \right).
\]
Approximately 1250 time steps should be sufficient for the projectile to hit the ground (i.e. for the $y$ coordinate to return to 0).
Your results should qualitatively match those given in Figure \ref{fig:evolution}.
\label{prob:simulation}
\end{problem}

\section*{State Estimation with the Kalman Filter}
The Kalman filter is a recursive estimator that smooths out the noise in real time, estimating each current state based on the past state estimate and the current measurement.
This process is done by repeatedly invoking two steps: Predict and Update.
The predict step is used to estimate the current state based on the previous state.
The update step then combines this prediction with the current observation, yielding a more robust estimate of the current state.

To describe these steps in detail, we need additional notation. Let
\begin{itemize}
    \item $\widehat{\mathbf{x}}_{n|m}$ be the state estimate at time $n$ given only measurements up through time $m$; and
    \item $P_{n|m}$ be an error covariance matrix, measuring the estimated accuracy of the state at time $n$ given only measurements up through time $m$.
\end{itemize}

The elements $\widehat{\mathbf{x}}_{k|k}$ and $P_{k|k}$ represent the state of the filter at time $k$, giving the state estimate and the accuracy of the estimate.

%\begin{figure}
%    \centering
%    \includegraphics[width=.90\textwidth]{norms}
%    \caption{Norms of the estimated accuracy matrices as the Kalman filter progresses. }
%    \label{fig:norms}
%\end{figure}

We evolve the filter recursively, as follows:
\begin{align*}
\textbf{Predict} & & \widehat{\mathbf{x}}_{k|k-1} & = F\widehat{\mathbf{x}}_{k-1|k-1} + \mathbf{u} \\
 & & P_{k|k-1} & = FP_{k-1|k-1}F^{T} + Q \\
\textbf{Update} & & \tilde{\mathbf{y}}_{k} & = \mathbf{z}_{k} - H\widehat{\mathbf{x}}_{k|k-1} \\
 & & S_{k} & = HP_{k|k-1}H^{T} + R \\
 & & K_{k} & = P_{k|k-1}H^{T}S_{k}^{-1} \\
 & & \widehat{\mathbf{x}}_{k|k} & = \widehat{\mathbf{x}}_{k|k-1} + K_{k}\tilde{\mathbf{y}}_{k} \\
 & & P_{k|k} & = (I - K_{k}H)P_{k|k-1}
\end{align*}

The more observations we have, the greater the accuracy of these estimates becomes (i.e the norm of the accuracy matrix converges to $0$).
%This is reflected in Figure \ref{fig:norms}.

\begin{problem}
Add code to your \li{KalmanFilter} class to estimate a state sequence corresponding to a given observation sequence and initial state estimate.
Implement the following class method:
\begin{lstlisting}
def estimate(self,x,P,z):
    """
    Compute the state estimates using the Kalman filter.
    If x and P correspond to time step k, then z is a sequence of
    observations starting at time step k+1.

    Parameters
    ----------
    x : ndarray of shape (n,)
        The initial state estimate.
    P : ndarray of shape (n,n)
        The initial error covariance matrix.
    z : ndarray of shape(m,N)
        Sequence of N observations (each column is an observation).

    Returns
    -------
    out : ndarray of shape (n,N)
        Sequence of state estimates (each column is an estimate).
    """
    pass
\end{lstlisting}
\end{problem}

\begin{figure}
    \centering
    \begin{tabular}{cc}
    \includegraphics[width=.45\textwidth]{estimate_macro} &
    \includegraphics[width=.45\textwidth]{estimate_micro}
    \end{tabular}
    \caption{State estimates together with observations and true state sequence (detailed view on the right).}
    \label{fig:estimates}
\end{figure}

Returning to the projectile example, we now assume that our radar sensor has taken observations from time steps $200$ through $800$ (take the corresponding slice of the
observations produced in Problem \ref{prob:simulation}).
Using these observations, we seek to estimate the corresponding true states of the projectile.
We must first come up with a state estimate $\widehat{\mathbf{x}}_{200}$ for time step $200$, and then feed this into the Kalman filter
to obtain estimates $\widehat{\mathbf{x}}_{201}, \ldots, \widehat{\mathbf{x}}_{800}$.
\begin{problem}
Calculate an initial state estimate $\widehat{\mathbf{x}}_{200}$ as follows:
For the horizontal and vertical positions, simply use the observed position at time $200$.
For the velocity, compute the average velocity between the observations $\mathbf{z}_k$ and $\mathbf{z}_{k+1}$ for
$k = 200,\ldots,208$, then average these 9 values and take this as the initial velocity estimate.
(Hint: the NumPy function \li{diff} is useful here.)

Using the initial state estimate, $P_{200} = 10^{6} \cdot Q$, and your Kalman filter, compute the next $600$ state estimates,
i.e. compute $\widehat{\mathbf{x}}_{201}, \ldots, \widehat{\mathbf{x}}_{800}$.
Plot these state estimates as a smooth green curve together with the radar observations (as red dots) and the entire true state sequence (as a blue curve).
Zoom in to see how well it follows the true path. Your plots should be similar to Figure \ref{fig:estimates}.
\label{prob:state_estimate}
\end{problem}

In the absence of observations, we can still estimate some information about the state of the system at some future time.
We can do this by recognizing that the expected state noise $\mathbb{E}\left[\boldsymbol{\epsilon}_{k}\right] = 0$ at any time $k$.
Thus, given a current state estimate $\widehat{\mathbf{x}}_{n|m}$ using only measurements up through time $m$, the expected state at time $n+1$ is
\begin{equation*}
\widehat{\mathbf{x}}_{n+1|m} = F \widehat{\mathbf{x}}_{n|m} + \mathbf{u}
\end{equation*}

\begin{problem}
Add a function to your class that predicts the next $k$ states given a current state estimate but in the absence of observations.
Do so by implementing the following function:
\begin{lstlisting}
def predict(self,x,k):
    """
    Predict the next k states in the absence of observations.

    Parameters
    ----------
    x : ndarray of shape (n,)
        The current state estimate.
    k : integer
        The number of states to predict.

    Returns
    -------
    out : ndarray of shape (n,k)
        The next k predicted states.
    """
    pass
\end{lstlisting}
\end{problem}

\begin{figure}
    \centering
    \begin{tabular}{cc}
    \includegraphics[width=.45\textwidth]{impact_macro} &
    \includegraphics[width=.45\textwidth]{impact_micro}
    \end{tabular}
    \caption{Predicted vs. actual point of impact (detailed view on right).}
    \label{fig:impact}
\end{figure}

We can use this prediction routine to estimate where the projectile will hit the surface.
\begin{problem}
Using the final state estimate $\widehat{\mathbf{x}}_{800}$ that you obtained in Problem \ref{prob:state_estimate}, predict the future states of
the projectile until it hits the ground.
Predicting approximately the next $450$ states should be sufficient.

Plot the actual state sequence together with the predicted state sequence (as a yellow curve), and observe how near the prediction is to the actual point of impact.
Your results should be similar to those shown in Figure \ref{fig:impact}.
\end{problem}

In the absence of observations, we can also reverse the system and iterate backward in time to infer information about states of the system prior to measured observations.
The system is reversed by
\begin{equation*}
\mathbf{x}_{k} = F^{-1}(\mathbf{x}_{k+1} - \mathbf{u} - \boldsymbol{\epsilon}_{k+1}).
\end{equation*}
Considering again that $\mathbb{E}\left[\boldsymbol{\epsilon}_{k}\right] = 0$ at any time $k$, we can ignore this term, simplifying the recursive estimation backward in time.

\begin{figure}[htb]
    \centering
    \begin{tabular}{cc}
    \includegraphics[width=.45\textwidth]{origin_macro} &
    \includegraphics[width=.45\textwidth]{origin_micro}
    \end{tabular}
    \caption{Predicted vs. actual point of origin (detailed view on right).}
    \label{fig:origin}
\end{figure}

\begin{problem}
Add a function to you class that rewinds the system from a given state estimate, returning predictions for the previous states.
Do so by implementing the following function:
\begin{lstlisting}
def rewind(self,x,k):
    """
    Predict the k states preceding the current state estimate x.

    Parameters
    ----------
    x : ndarray of shape (n,)
        The current state estimate.
    k : integer
        The number of preceding states to predict.

    Returns
    -------
    out : ndarray of shape (n,k)
        The k preceding predicted states.
    """
    pass
\end{lstlisting}
\end{problem}

Returning to the projectile example, we can now predict the point of origin.

\begin{problem}
Using your state estimate $\widehat{\mathbf{x}}_{250}$, predict the point of origin of the projectile along with
all states leading up to time step $250$. Note that you may have to take a few extra time steps to predict the point of origin.
(The point of origin is the first point along the trajectory where the $y$ coordinate is $0$.)
Plot these predicted states (in cyan) together with the original state sequence.
Zoom in to see how accurate your prediction is.
Your plots should be similar to Figure \ref{fig:origin}.
\label{prob:origin_pt}

Repeat the prediction starting with $\widehat{\mathbf{x}}_{600}$.  Compare to the previous results.  Which is better?  Why?
\end{problem}


\documentclass{article}
\usepackage{leonine,amsmath,amssymb,amsthm,graphicx}
\setkeys{Gin}{width=\linewidth,totalheight=\textheight,keepaspectratio}
\graphicspath{{graphics/}}
% Prints a trailing space in a smart way.
\usepackage{xspace}
% Inserts a blank page
\newcommand{\blankpage}{\newpage\hbox{}\thispagestyle{empty}\newpage}
% \usepackage{units}
% Typesets the font size, leading, and measure in the form of 10/12x26 pc.
\newcommand{\measure}[3]{#1/#2$\times$\unit[#3]{pc}}

\theoremstyle{definition}
\newtheorem{pred}[thm]{Prediction}

\title{Askesis: Introduction} \author{Eric Purdy}

\begin{document}

\maketitle

\section{Perceptron model of a neuron}

A perceptron has two possible outputs, $1$ (representing firing) and
$0$ (representing not firing). Let its inputs be denoted by $x_1,
\dots, x_n$. The rule is then
$$y = \begin{cases} 1 & \sum_i w_i x_i > \theta \\ -1 & \mbox{otherwise}
\end{cases}$$

The $x_i$ can usually be taken to be either $0$ (if the input neuron
fires) or $1$ (if the input neuron does not fire). Neurons that do not
fire are thus not included in the sum. Occasionally, it is helpful to
think about $x_i$ between zero and one, in which case it can be
encoded by the firing rate. So, if the neuron fires at its maximum
rate, that is represented by a one, and if it doesn't fire at all,
that is represented by a zero. If it fires at a tenth its firing rate,
that is represented by $0.1$.

The $w_i$ are called the synapse weights; each is a summary of
properties of the synapse that occurs between the $i$-th input neuron
and the output neuron. These properties include the physical distance
between the two neurons, as well as the concentration of
neurotransmitter receptors in the output neuron, and other things as
well. If $w_i > 0$, then we call the synapse ``excitatory'' - the
firing of the input neuron excites the output neuron. Most excitatory
synapses use the neurotransmitter glutamate. If $w_i < 0$, we call the
synapse ``inhibitory'' - the firing of the input neuron inhibits the
output neuron. Most inhibitory synapses use the neurotransmitter GABA
(Gamma-Aminobutyric acid). In actual neurons, a single neuron releases
a single main neurotransmitter, and is therefore only ever excitatory
(all synapses it outputs to are excitatory) or inhibitory (all
synapses it outputs to are inhibitory). The parameter $\theta$
controls how excitable the neuron is, and can be modified by
learning. This is called ``intrinsic plasticity''.

\footnote{I like to think of glutamate as the ``black ink'' and GABA as the
  ``red ink'' of the nervous system.}

A perceptron is much simpler than a neuron, but is a reasonable
first-pass approximation. Our model does not require anything more
complicated than perceptrons.

The word ``perceptron'' actually refers to two things: the model
above, and a learning algorithm. We will describe the learning
algorithm here. First, we add an $x_0$ that is always $1$, and let
$w_0 = - \theta$. The perceptron then fires if $\sum_{i=0}^n w_i x_i >
0$. We then suppose that we see a sequence of inputs $x_i(t)$, each
labeled either $1$ or $-1$ by a desired output $\widehat{y}(t)$. The
learning rule is then as follows: if the output $y$ of the neuron was
the same as $\widehat{y}(t)$, we do nothing. If $y(t) \ne
\widehat{y}(t)$, then we update the weights as follows:
$$w_i(t+1) = w_i(t) + \alpha (\widehat{y}(t) - y(t)) x_i(t).$$ If we
start with all $w_i$ equal to zero, then the $w_i(t)$ (considered as a
vector) will be equal to the sum of the $\mathbf{x}(t)$ for which
$\widehat{y}(t)=1,y(t)=0$, minus the sum of the $\mathbf{x}(t)$ for
which $\widehat{y}(t)=0, y(t)=1$. The parameter $\alpha$ is called
the learning rate; high values of $\alpha$ result in a system that
changes its parameters more quickly and thus learns more quickly. A
lower learning rate can be desirable, since the system will be less
likely to lose the information it already has.

The perceptron algorithm is impossible to implement in a single
neuron, since it requires the same synapse to be both excitatory and
inhibitory, depending on what the training data looks like. The
perceptron algorithm is still useful to know, since it can be
implemented by multiple neurons, and has long been thought to be a
large part of how the cerebellum works.

\section{Basic overview of the cerebellum}

\begin{figure}
\includegraphics[width=\linewidth]{organization.png}
\caption{Inputs and outputs of the cerebellum. The two arrows marked
  with double lines represent two neural pathways that convey copies
  of the same information.}
\label{fig-organization}
\end{figure}

\begin{figure}
\includegraphics[width=\linewidth]{nanohub/global.png}
\caption{How the cerebellum fits into the rest of the brain. Courtesy
  of
  http://what-when-how.com/wp-content/uploads/2012/04/tmp15F121.jpg}
\label{fig-global}
\end{figure}

\begin{figure}
\includegraphics[width=\linewidth]{nanohub/pathways.png}
\caption{Pathways into and out of the cerebellum. The fastigial,
  interposed, and dentate nuclei (and some others) are collectively
  referred to as the deep nuclei, as they lie deep within the
  cerebellum. Courtesy of
  http://www.dizziness-and-balance.com/anatomy/brain/cerebellum.htm}
\label{fig-pathways}
\end{figure}

\section{Cell types}

The cerebellum contains a number of different types of cells:
\begin{description}
\item[Mossy fiber] Mossy fibers are the axons of neurons outside of
  the cerebellum. They constitute one of the two sources of input of
  the cerebellum. One of the major sources of mossy fibers is the
  pontine nuclei, which relay information from the cerebrum.

  All mossy fibers are excitatory. The mossy fibers excite granular
  cells and Golgi cells, and collaterals (side branches) of the mossy
  fibers excite deep nuclear cells.

  Mossy fibers participate in both the positive and negative pathways.

\item[Deep nuclear cell] These are the main source of output from the
  cerebellum. (Some Purkinje cells project to the vestibular nuclei,
  which seem to be pretty similar to the deep nuclei.)

  Some of the deep nuclei are: interposed nucleus, fastigial nucleus,
  dentate nucleus.

  There are two kinds of deep nuclear cell, large and small. The large
  cells can be excitatory or inhibitory, and project to premotor
  areas. The small cells are inhibitory and project to the inferior
  olive.

  Deep nuclear cells receive input from mossy fiber collaterals,
  climbing fiber collaterals, and Purkinje cells.

  Deep nuclear cells participate in both the positive and negative
  pathways.

\item[Inferior olive cell] These are the second source of input to the
  cerebellum. They receive an ``efference'' copy of the signals sent
  from the cerebrum to the spinal cord.

  The inferior olive cells are excitatory and send climbing fibers
  into the cerebellum to make contact with the Purkinje cells,
  stellate cells, and basket cells. A single climbing fiber impulse
  will cause a Purkinje cell to fire.

  Collaterals of the climbing fiber make contact with the deep nuclear
  cells.

  Inferior olive cells participate in both the positve and negative
  pathways.

\item[Purkinje cell] The Purkinje cells are inhibitory, and send their
  output to the deep nuclear cells. They receive input mainly from the
  parallel fibers, which are the output of the granular cells. They
  also receive some input directly from nearby granular cells.

  Purkinje cells fire at a certain base rate even when not receiving
  any input.

  Purkinje cells participate in the negative pathway.

\item[Basket/stellate cell] Basket and stellate cells inhibit the
  Purkinje cells (and thus disinhibit the deep nuclear cells by
  proxy).

  The stellate cells occupy the outermost portions of the cerebellar
  cortex, while basket cells are slightly more inward. There is also
  an intermediate type of cell that occurs between the basket cells
  and stellate cells and has an appearance halfway between the two.

  Basket and stellate cells participate in the negative pathway.

\item[Granular cell] Granular cells receive input from 4-5 mossy
  fibers. Granular cells are excitatory. The output of the granular
  cell is the parallel fiber, which sends input to the Purkinje cells,
  basket cells, and stellate cells.

  Granular cells are generally thought to fire only when multiple of
  their inputs are active.

  Granular cells are the most numerous neurons in the brain; about
  three-fourths of the neurons in the brain are granular cells in the
  cerebellum. (Granular cells are very small, so the cerebellum only
  takes up about 10\% of the volume of the brain.)

  Granular cells participate in the negative pathway.
\end{description}

\begin{figure}
\includegraphics[width=\linewidth]{cells.png}
\caption{Cells of the cerebellum. Excitatory cells are shown in black,
  while inhibitory cells are shown in red. Inputs to a cell are shown
  as an empty half circle, while outputs are shown as a full half
  circle.  Cells of the inferior olive are shown in green, to
  emphasize that their output is used to train the targeted cells;
  they are excitatory. There are two kinds of deep nuclear cell: one
  excitatory and projecting either back into the cerebellum or to
  premotor areas; and one inhibitory and projecting to the inferior
  olive. Since the cells receive the same input, we have simplified
  the picture by identifying them together. It is also worth noting
  that state cells are actually represented twice here: they are both
  ``mossy fiber cells'' and ``deep nuclear cells''.}
\label{fig-cells}
\end{figure}

\begin{figure}
\includegraphics[width=\linewidth]{nanohub/028.png}
\caption{Cells of the cerebellum. Courtesy of
  https://nanohub.org/resources/18948/watch?resid=19060}
\label{fig-physical-1}
\end{figure}

\begin{figure}
\includegraphics[width=\linewidth]{nanohub/029.png}
\caption{Cells of the cerebellum. Courtesy of
  https://nanohub.org/resources/18948/watch?resid=19060}
\label{fig-physical-2}
\end{figure}

\begin{figure}
\includegraphics[width=\linewidth]{nanohub/image004.png}
\caption{Cells of the cerebellum. Courtesy of http://www.scritub.com/limba/engleza/health/CEREBELLUM25854.php}
\label{fig-physical-3}
\end{figure}

\begin{figure}
\includegraphics[width=\linewidth]{nanohub/nature.png}
\caption{Cells of the cerebellum. Courtesy of
  http://www.nature.com/nrn/journal/v9/n4/images/nrn2332-f3.jpg.
PCN:pontine nuclei.
MF: mossy fiber.
Lg: Lugaro cell.
5-HT fibres: beaded fibers.
UB: unipolar brush cells (found only in the vestibulocerebellum).
Go: Golgi cell.
Gr: Granular cell.
N-C: state cell pathway.
CN/VN: deep nuclei and vestibular nuclei.
PF: parallel fiber.
LTD: long-term depression, the learning mechanism of the Purkinje cell.
SC: stellate cell.
BC: basket cell.
CF: climbing fiber.
N-O: training suppression cell pathway to inferior olive.
IO: inferior olive.
R-O: red nucleus to inferior olive pathway.
pRN: parvocellular (small cells) red nucleus.
Pd: unknown, seems to refer to another kind of beaded fiber.
}
\label{fig-physical-4}
\end{figure}

\section{Learning}

As a brief aside to anyone not versed in machine learning or neuroscience,
``learning'' is a term that describes changing the structure or parameters of a
system in order to make that system better achieve some goal or perform some
purpose correctly. The best understood mechanism for learning in the brain is
changing the strength of synapses, the gaps between two neurons, usually by
changing the number of receptors that respond to neurotransmitters.

\section{Thoughts}

\begin{itemize}
\item Discuss microzones. In particular, how big are they? What
  constraints do they impose?
\item Give sigmoid rule with probability.

\item Hebbian learning

\item Explain electrical synapses in the inferior olive?
\end{itemize}

\end{document}


\documentclass{ut-thesis}

%**************************** General libraries *******************************
\usepackage{mathtools}
\usepackage{cancel}
\usepackage[utf8]{inputenc}
\usepackage{amssymb}
\usepackage{ntheorem}
\usepackage{tensor}
%\usepackage{physics}
\usepackage[italicdiff]{physics}
\usepackage{calc}
\usepackage{caption}
%\usepackage{subcaption}
\usepackage{tcolorbox}
\usepackage{chngcntr}
\usepackage{titlesec}
\usepackage {graphicx,float}
\usepackage{subfig}
\usepackage{siunitx}
\usepackage{xcolor}
\usepackage{etoolbox} %ifthen
\usepackage[outline]{contour} % glow around text

%**************************** Bibliography management *******************************
\usepackage{biblatex} %Imports biblatex package
\addbibresource{sample.bib} %Import the bibliography file

%**************************** Tikz libraries *******************************
\usepackage{tikz}
\usetikzlibrary{shapes,arrows,arrows.meta,spy}
\usetikzlibrary{calc}% needed for BB
\usepackage{pgfplots}
\usetikzlibrary{math} % for \tikzmath
\usetikzlibrary{angles,quotes} % for pic (angle labels)
\usetikzlibrary{decorations.pathmorphing,decorations.markings}
\usetikzlibrary{decorations.pathreplacing} % for curly braces
\usetikzlibrary {decorations.markings,shapes.arrows}
\usetikzlibrary{patterns}
\usepackage{tikz-3dplot}

%\usepackage[margin=0cm,nohead]{geometry}
%\usepackage[active,tightpage]{preview}
%\usepackage[pdf]{pstricks}

%**************************** custom macro's *******************************
\newsavebox\CBox
\newcommand\hcancel[2][0.5pt]{%
  \ifmmode\sbox\CBox{$#2$}\else\sbox\CBox{#2}\fi%
  \makebox[0pt][l]{\usebox\CBox}%
  \rule[0.5\ht\CBox-#1/2]{\wd\CBox}{#1}}
%\newcommand{\edal}{\end{align}}
%\newcommand{\bgal}{\begin{align}\end{align}}
\newcommand{\Lagr}{\mathcal{L}}
\newcommand{\questeq}{\overset{?}{=}}
%\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\renewcommand\thelemma{\unskip}
\newcommand{\christ}[3]{\ensuremath{\Gamma^{#1}_{#2#3}}}
\newcommand{\half}{\ensuremath{\frac{1}{2}}}
\newcommand{\kwart}{\ensuremath{\frac{1}{4}}}
\newcommand\del{\overline{\boldsymbol{{\triangledown}}}}
\newcommand\maal{\boldsymbol{\times}}
\newcommand\spatie{\quad\quad\quad\quad}
\newcommand\RAr{\quad\Rightarrow\quad}
%\newcommand{\myabsdv}[3]{{\frac{\delta^2 \ensuremath{{#1}}}{{\delta %%\ensuremath{{#2}}}{\delta\ensuremath{{#3}}}}}}
\newcommand\myabsdv[3][1]{{\frac{{\delta^{2}}{#1}}{\delta {#2} \delta {#3}}}}

%**************************** layout setings *******************************
\counterwithin*{equation}{chapter}
\counterwithin*{equation}{section}
\counterwithin*{equation}{subsection}
\renewcommand{\theequation}{\arabic{equation}}
\titleformat{\chapter}{\normalfont\huge}{\thechapter.}{20pt}{\huge\it}
\titleformat{\chapter}[display]
  {\normalfont\bfseries}{}{0pt}{\Huge}
\titlespacing*{\chapter}{0pt}{50pt}{*2}

%**************************** Tikz  setings *******************************
\begin{comment}
\tikzset{
    circ/.style={draw, circle,inner sep=0pt,minimum size=8mm, font=\scriptsize},
    triangle/.tip={Computer Modern Rightarrow[open,angle=120:3pt]}}

\tikzset{
  every point/.style = {circle, inner sep={.75\pgflinewidth}, opacity=1, draw, solid, fill=white},
  point/.style={insert path={node[every point, #1]{}}}, point/.default={},
  colored point/.style = {point={fill=#1}},
  point name/.style = {insert path={coordinate (#1)}},
  inherit/.style = {point/.style={insert path={node[circle, inner sep={.75\pgflinewidth}, draw, fill, #1]{}}}}
}
\end{comment}

%**************************** Where to find images *****************************
\graphicspath{ {D:/MathLatex/images/} }

%**************************** Document itself *******************************
\author{Bernard Carrette}
\gradyear{2020}
\title{Tensor Calculus\\J.L. Synge and A.Schild (Dover Publication)\\ Solutions to exercises}
\begin{document}
\maketitle

\section*{Remarks and warnings}
You're welcome to use these notes, but they may contain errors, so proceed with caution. If you do find an error, however, I'd be happy to receive bug reports, suggestions, and the like through Github.
\section*{Some notation conventions}
\begin{center}
\begin{tabular}{ c c  }
$\partial_r \equiv \pdv{}{x^r}$ & \\\\
$\Gamma^r_{mn} \equiv
\begin{Bmatrix}
r\\
m n\\
\end{Bmatrix}$ & Christoffel symbol of the second kind\\\\

\end{tabular}
\end{center}

\tableofcontents
\listoffigures
\include{chapter1}
\include{chapter2}
\include{chapter3}
\include{chapter4}
%\include{chapter5}
\printbibliography %Prints bibliography
\end{document}


\subsection*{Stream Support}

The following stream processing functions are supported:

\begin{itemize}
\item \lstinline{stream_tail(x)}: \textit{primitive}, assumes that the tail (second component) of the
  pair \lstinline{x} is a nullary function, and returns the result of
  applying that function.\\
\emph{Laziness:}  Yes: \lstinline{stream_tail} only forces the direct tail of a given
stream,
but not the rest of the stream, i.e. not the tail of the tail, etc.
\item \lstinline{stream(x1, x2,..., xn)}: \textit{primitive}, returns a stream with $n$ elements. The
first element is \lstinline{x1}, the second \lstinline{x2}, etc.\\
\emph{Laziness:}  No: In this implementation, we generate first a
           complete list, and then a stream using \lstinline{list_to_stream}.
\item \lstinline{is_stream(x)}: Returns \lstinline{true} if
  \lstinline{x} is a stream as defined in the lectures, and
  \lstinline{false} otherwise.\\
\emph{Laziness:}  No: \lstinline{is_stream} needs to force the given stream.
\item \lstinline{list_to_stream(xs)}: transforms a given list to a stream.\\
\emph{Laziness:}  Yes: \lstinline{list_to_stream} goes down the list only when forced.
\item \lstinline{stream_to_list(s)}: transforms a given stream to a list.\\
\emph{Laziness:}  No: \lstinline{stream_to_list} needs to force the whole stream.
\item \lstinline{stream_length(s)}: Returns the length of the stream
  \lstinline{s}.\\
\emph{Laziness:}  No: The function needs to force the whole stream.
\item \lstinline{stream_map(f, s)}: Returns a stream that results from stream
  \lstinline{s} by element-wise application of \lstinline{f}.\\
\emph{Laziness:}  Yes: The argument stream is only explored as forced by
           the result stream.
\item \lstinline{build_stream(n, f)}: Makes a stream with \lstinline{n}
elements by applying the unary function \lstinline{f} to the numbers 0
to \lstinline{n - 1}.\\
\emph{Laziness:}  Yes: The result stream forces the applications of fun
            for the next element.
\item \lstinline{stream_for_each(f, s)}: Applies \lstinline{f} to every
  element of the stream \lstinline{s}, and then returns
  \lstinline{true}.\\
\emph{Laziness:}  No: \lstinline{stream_for_each} forces the exploration of the entire stream.
\item \lstinline{stream_reverse(s)}: Returns finite stream \lstinline{s} in reverse
  order. Does not terminate for infinite streams.\\
\emph{Laziness:}  No: \lstinline{stream_reverse} forces the exploration of the entire stream.
\item \lstinline{stream_append(xs, ys)}: Returns a stream that results from
appending the stream \lstinline{ys} to the stream \lstinline{xs}.\\
\emph{Laziness:}  Yes: Forcing the result stream activates the actual append operation.
\item \lstinline{stream_member(x, s)}: Returns first postfix substream
whose head is equal to
\lstinline{x} (\lstinline{===}); returns \lstinline{null} if the
element does not occur in the stream.\\
 \emph{Laziness:}  Sort-of: \lstinline{stream_member} forces the stream only until the element is found.
\item \lstinline{stream_remove(x, s)}: Returns a stream that results from
given stream \lstinline{s} by removing the first item from \lstinline{s} that
is equal (\lstinline{===}) to \lstinline{x}.
Returns the original list
if there is no occurrence.\\
\emph{Laziness:}  Yes: Forcing the result stream leads to construction of each next element.
\item \lstinline{stream_remove_all(x, s)}: Returns a stream that results from
given stream \lstinline{s} by removing all items from \lstinline{s} that
are equal (\lstinline{===}) to \lstinline{x}.\\
\emph{Laziness:}  Yes: The result stream forces the construction of each next element.
\item \lstinline{stream_filter(pred, s)}: Returns a stream that contains
only those elements for which the one-argument function
\lstinline{pred}
returns \lstinline{true}.\\
\emph{Laziness:}  Yes: The result stream forces the construction of
            each next element. Of course, the construction
            of the next element needs to go down the stream
            until an element is found for which \lstinline{pred} holds.
\item \lstinline{enum_stream(start, end)}: Returns a stream that enumerates
numbers starting from \lstinline{start} using a step size of 1, until
the number exceeds (\lstinline{>}) \lstinline{end}.\\
 \emph{Laziness:}  Yes: Forcing the result stream leads to the construction of
            each next element.
\item \lstinline{integers_from(n)}: Constructs an infinite stream of integers
starting at a given number \lstinline{n}.\\
 \emph{Laziness:}  Yes: Forcing the result stream leads to the construction of
            each next element.
\item \lstinline{eval_stream(s, n)}: Constructs the list
of the first \lstinline{n} elements
of a given stream \lstinline{s}.\\
\emph{Laziness:}  Sort-of: \lstinline{eval_stream} only forces the computation of
                the first \lstinline{n} elements, and leaves the rest of
                the stream untouched.
\item \lstinline{stream_ref(s, n)}: Returns the element
of stream \lstinline{s} at position \lstinline{n},
where the first element has index 0.\\
 \emph{Laziness:}  Sort-of: \lstinline{stream_ref} only forces the computation of
                the first \lstinline{n} elements, and leaves the rest of
                the stream untouched.
\end{itemize}


\section{Test}
The corpus we use comprise of three different speakers: two men and one woman.
Each speaker has 460 samples, where the average duration of each sample is
about 4 seconds.  For each speaker, we randomly choose $M$ samples to train GMM
model. For more deep inspection of the algorithm, M is enumerated from 2 to 5.
We then feed the whole corpus to to the algorithm, and test its accuracy.
Further more, for the robustness of the result, we repeat the test process 10
times, and average the result.

\subsection{Result}
\begin{figure}[!ht]
\centering
\includegraphics[width=\textwidth]{res/plot.pdf}
\caption{Accuracy vs Number of samples used for training}
\label{fig:result}
\end{figure}

\begin{table}[!ht]
	\centering
	\begin{tabular}{|c|c|c|c|c|}
		\hline
        \#Training sample & 2 & 3 & 4 & 5 \\\hline
		Accuracy & 0.968555 & 0.991293 & 0.997549 & 0.997380 \\\hline
	\end{tabular}
	\label{table:result}
\end{table}

As we can see from the curve ploted, the performance of the algorithm increases as
the number of training samples given increases. The accuracy when
just using two training samples for each user has reached $96.55\%$, which
is supprisingly high with respect to using such small amount of training samples.

When using 4 to 5 samples per speaker, the accuracy is above $99.73\%$, which
strongly confirmed the effectivenes of MFCC feature and GMM modeling for each
speaker.

However, further inspection on misclassified samples showed limitation on our
test case: all the misclassified samples are within two men speakers, indicates
the weakness of the algorithm.


\documentclass{scrartcl}
\usepackage[a4paper,left=1in,right=1in,top=1.2in,bottom=1in]{geometry}
\usepackage{siunitx}
\usepackage{graphicx}
\usepackage{mathtools}
\setkomafont{disposition}{\normalfont\bfseries}
\newcommand*\diff{\mathop{}\!\mathrm{d}}
\newcommand*\Diff[1]{\mathop{}\!\mathrm{d^#1}}
\newcommand*\colvec[3][]{
    \begin{pmatrix}\ifx\relax#1\relax\else#1\\\fi#2\\#3\end{pmatrix}
}

%title
\title{Exercise 02:\\State space analysis}
\subtitle{Theoretical Neuroscience II}
\author{Johannes G\"atjen \and Lorena Morton}

%use these for structure/overview
\newcommand\Question{%
  \textbf{Question:}%
}
\newcommand\Answer{%
  \textbf{Answer:}%
}

\begin{document}
\maketitle

\section{Task A}

We investigate a linear recurrent network with 2 populations. The time-development is given by the equation

\begin{align*}
\frac{\diff \mathbf{v}}{\diff t} = \mathbf{A} \cdot \mathbf{v} + \mathbf{b} \qquad \text{ with } \qquad \mathbf{A} =  \left( \begin{array}{cc}
-2 & 3 \\
-3 & 2 \end{array} \right).
\end{align*}

We determine the input vector $\mathbf{b}$ such that the steady state of the network is at

\begin{align*}
\mathbf{v}_{ss} = \colvec{10}{10} \qquad \text{ by solving the equation } \qquad \mathbf{A} \cdot \mathbf{v}_{ss} + \mathbf{b} = 0. \\
\shortintertext{Substituting our values we obtain:}
\left( \begin{array}{cc}
-2 & 3 \\
-3 & 2 \end{array} \right) \cdot \colvec{10}{10} + \mathbf{b} &= \colvec{10}{-10} + \mathbf{b} = 0 \\
\Leftrightarrow  \mathbf{b} &= \colvec{-10}{10} \\
\shortintertext{With this we get the full dynamic equation:}
\colvec{\frac{\diff v_1}{\diff t}}{\frac{\diff v_2}{\diff t}} =  \left( \begin{array}{cc}
-2 & 3 \\
-3 & 2 \end{array} \right) \cdot \colvec{v_1}{v_2} + \colvec{-10}{10} \\
\shortintertext{And we get two line equations when we set the gradient for both populations to zero:}
\text{$v_1$-isocline:} & -2 v_1 + 3 v_2 - 10 = 0 \\
\Leftrightarrow & v_2 = \frac{2}{3} v_1 + \frac{10}{3} \\
\text{$v_2$-isocline:} & -3 v_1 + 2 v_2 + 10 = 0 \\
\Leftrightarrow & v_2 = \frac{3}{2} v_1 - 5 \\
\shortintertext{Get the intersection point, by solving the system of linear equations for $v_1$ and $v_2$.}
& \frac{2}{3} v_1 + \frac{10}{3} = \frac{3}{2} v_1 - 5 \\
\Leftrightarrow & v_1 = 10 \\
\shortintertext{Replace the value in one of the original equations.}
v_2 = \frac{3 \cdot 10}{2} - 5 = 10 \\
\shortintertext{And we have confirmed that the steady state is}
\mathbf{v}_{ss} = \colvec{10}{10}.
\end{align*}

\section{Task B}

We obtain the eigenvalues of the connectivity matrix by solving the characteristic equation.
\begin{align*}
&(-2 - \lambda) \cdot (2 - \lambda) + 9 = 0 \\
\Leftrightarrow & -4 + 2\lambda - 2\lambda + \lambda^2 + 9 = 0 \\
\Leftrightarrow & \lambda^2 = -5 \\
\Leftrightarrow & \lambda_1 = i\sqrt{5} \qquad \lambda_2 = -i\sqrt{5}
\end{align*}
Because $Im(\lambda_1) = - Im(\lambda_2)$ we expect a spiraling behavior from the dynamical system, and because $Re(\lambda_1) = Re(\lambda_2) = 0$ it is neutrally stable.

\begin{figure}
\centering
\includegraphics[trim = {1.3cm 0 0.5cm 0.2cm}, width=0.7\textwidth, clip]{../pics/traj}
\caption{Three trajectories of the population activity in the state space for the linear recurrent network.}
\label{label}
\end{figure}

\section{Task C}

As $\kappa$ increases, the frequency of the oscillations do not change, but the steady state value for the amplitude of the oscillations gets smaller, such that the trajectory is damped more strongly. The steady state also appears to change from neutrally stable to stable with higher $\kappa$.

\begin{figure}
\centering
\includegraphics[trim = {0.8cm 0 0.5cm 0.2cm}, width=0.7\textwidth, clip]{../pics/nonlin}
\caption{The evolution of the activity of the two populations over time for the non-linear recurrent network, with $\kappa = 2$.}
\label{label}
\end{figure}

\begin{figure}
\centering
\includegraphics[trim = {0.8cm 0 0.5cm 0.2cm}, width=0.7\textwidth, clip]{../pics/nonlintraj}
\caption{The trajectory of the population activity in the state space for the non-linear recurrent network, with initial condition at $(7, 11)$.}
\label{label}
\end{figure}

%\operatorname{\mathbf{v}}(t) =

%include picture
%\begin{figure}
%\centering
%\includegraphics[trim = {1.3cm 0 2cm 0.9cm}, width=\textwidth, clip]{../pics/picname}
%\caption{caption text}
%\label{label}
%\end{figure}
\end{document}


\section{Coulomb Integrals with error function Operator}

In a lot of applications it's useful to modify the normal Coulomb operator with
error function($\omega$ is some given constant):
\begin{equation}\label{erf_coulomb_integral:1}
 \frac{1}{r_{12}} = \frac{erf(\omega r_{12})}{r_{12}} + \frac{erfc(\omega r_{12})}{r_{12}}
\end{equation}
where the error function is:
\begin{align}\label{erf_coulomb_integral:2}
 erf(\omega r_{12})  &= \frac{2}{\sqrt{\pi}}\int_{0}^{\omega r_{12}} e^{-  t^{2}} dt \nonumber \\
 erfc(\omega r_{12}) &= \frac{2}{\sqrt{\pi}}\int_{\omega r_{12}}^{+\infty} e^{- t^{2}} dt
\end{align}
therefore $erf(\omega r_{12}) + erfc(\omega r_{12}) = 1$. Based on the decomposition of the Coulomb operator,
now the normal four body integral could be rewritten as:
\begin{equation}\label{erf_coulomb_integral:3}
 I_{ijkl} = \int \chi_{i}(r_{1})\chi_{j}(r_{1})\frac{erf(\omega r_{12})}{r_{12}}
\chi_{k}(r_{2})\chi_{l}(r_{2}) dr_{1} dr_{2}
\end{equation}

How to integrate the $I_{ijkl}$? By following the idea in \ref{os_eri}, we need to transform
the operator with erf or erfc function into another form, just like we use Laplace transformation
for the $r_{12}$ operator:
\begin{equation}\label{erf_coulomb_integral:5}
 \frac{1}{|r-r^{'}|} = \frac{2}{\pi^{1/2}}\int^{\infty}_{0}
e^{(r-r^{'})^{2}u^{2}}du
\end{equation}
in \ref{os_eri}. However, here we have a better idea; just to rewrite the normal erf/erfc function
in the following way:
\begin{align}\label{erf_coulomb_integral:4}
 erf(\omega r_{12})
 &= \frac{2}{\sqrt{\pi}}\int_{0}^{\omega r_{12}} e^{- t^{2}} dt \quad \text{set t=u*$\omega r_{12}$}
 \Rightarrow \nonumber \\
 &= \frac{2}{\sqrt{\pi}}\int_{0}^{\omega r_{12}} e^{- (u\omega r_{12})^{2}} d(u\omega r_{12}) \nonumber \\
 &= \frac{2\omega}{\sqrt{\pi}}\int_{0}^{1} r_{12}e^{- u^{2}\omega^{2}r_{12}^{2}} du
\end{align}
here the trick is to use variable $u$ to replace the variable $t$, so that to transform the
integral limit from $\omega r_{12}$ to 1. By using this form of erf function, we can get:
\begin{equation}\label{erf_coulomb_integral:6}
 \frac{erf(\omega r_{12})}{r_{12}} =
 \frac{2\omega}{\sqrt{\pi}}\int_{0}^{1} e^{- u^{2}\omega^{2}r_{12}^{2}} du
\end{equation}
so the $r_{12}$ inside the integrate of \ref{erf_coulomb_integral:4} is canceled with the
one in the denominator. It's interesting that the result in \ref{erf_coulomb_integral:6}
is same with operator transformation in \ref{erf_coulomb_integral:5}, except that here
the integration goes from 0 to 1 and in \ref{erf_coulomb_integral:5} it goes from 0 to
$+\infty$. Similarly, the operator with erfc function is:
\begin{equation}\label{erf_coulomb_integral:7}
 \frac{erfc(\omega r_{12})}{r_{12}} =
 \frac{2\omega}{\sqrt{\pi}}\int_{1}^{+\infty} e^{- u^{2}\omega^{2}r_{12}^{2}} du
\end{equation}

Such derivation leads to an important idea, that the integral with operator
$\dfrac{erf(\omega r_{12})}{r_{12}}$ etc. can be used for the recurrence relation described
in \ref{os_eri} without any problem. For example, the result in \ref{OS_ERI_eq:2}
can be rewritten as for operator with erf function:
\begin{equation}
 \begin{split}
 (ab|\frac{erf(\omega r_{12})}{r_{12}}|cd) &= \frac{2\omega}{\pi^{1/2}}\int^{1}_{0} du
\int dr \int dr^{'} \chi_{a}(r)\chi_{b}(r) e^{(r-r^{'})^{2}(u\omega)^{2}}
\chi_{c}(r^{'})\chi_{d}(r^{'}) \\
&= \frac{2\omega}{\pi^{1/2}}\int^{1}_{0} du
\int dr^{'}\chi_{c}(r^{'})\chi_{d}(r^{'})
\left( \int dr  \chi_{a}(r)\chi_{b}(r) e^{(r-r^{'})^{2}(u\omega)^{2}}\right) \\
&= \frac{2\omega}{\pi^{1/2}}\int^{1}_{0} du (ab|u\omega|cd)
 \end{split}
\end{equation}
Let's name $u^{'} = u\omega$. By repeating all of same procedure in \ref{os_eri} in terms of
$u^{'}$, we can get the same recurrence relation for ERI as shown in \ref{OS_ERI_result}(here
below we omit the operator of $\dfrac{erf(\omega r_{12})}{r_{12}}$):
\begin{equation}
 \begin{split}
((a+\iota_{i})b|cd)^{(m)} &= (P_{i} - A_{i})(ab|cd)^{(m)} +
\left(W_{i} -P_{i}\right)(ab|cd)^{(m+1)} \\
&+\frac{N_{i}(A)}{2\epsilon}\left(((a-\iota_{i})b|cd)^{(m)}-\frac{\rho}{
\epsilon }((a-\iota_{i})b|cd)^{(m+1)}\right)  \\
&+\frac{N_{i}(B)}{2\epsilon}\left((a(b-\iota_{i})|cd)^{(m)}-\frac{\rho}{
\epsilon }(a(b-\iota_{i})|cd)^{(m+1)}\right)  \\
&+\left(\frac{N_{i}(C)}{2}\right)\frac{1}{\epsilon+\eta}
(ab|(c-\iota_{i})d)^{(m+1)} \\
&+\left(\frac{N_{i}(D)}{2}\right)\frac{1}{\epsilon+\eta}
(ab|c(d-\iota_{i}))^{(m+1)}
\end{split}
\label{erf_coulomb_integral:8}
\end{equation}
where the integral of $(ab|cd)^{(m)}$ in \ref{OS_ERI_eq:28} is expressed as:
\begin{equation}
\label{erf_coulomb_integral:9}
 (ab|\frac{erf(\omega r_{12})}{r_{12}}|cd)^{(m)} = \frac{2\omega}{\sqrt{\pi}}\int^{1}_{0} du \left(
 \frac{\omega^{2}u^{2}}{\rho+\omega^{2}u^{2}}\right)^{m}(ab|\omega u|cd)
\end{equation}
The difference here is that the integrand is integrated from 0 to 1, not $+\infty$.

For the bottom integral of $(00|00)^{(m)}$, by repeating the same procedure from
\ref{OS_ERI_eq:30} to \ref{OS_ERI_eq:33}, it can see that:
\begin{equation}
 \begin{split}
 (00|\frac{erf(\omega r_{12})}{r_{12}}|00)^{(m)} &=\frac{2\omega}{\sqrt{\pi}}(0_{A}|0_{B})(0_{C}|0_{D}) \\
 &\int_{0}^{1} du \left( \frac{\omega^{2}u^{2}}{\rho+\omega^{2}u^{2}}\right)^{m}
 \left(\frac{\rho}{\rho+\omega^{2}u^{2}}\right)^{\frac{3}{2}}
 e^{-\frac{\rho \omega^{2}u^{2}}{\rho+ \omega^{2}u^{2}}|PQ|^{2}}
 \end{split}
\label{erf_coulomb_integral:10}
\end{equation}
Still if we set $t^{2} = \frac{\omega^{2}u^{2}}{\rho+ \omega^{2}u^{2}}$, and through
\ref{OS_ERI_eq:35} and set $u^{'} = \omega u$ it's easy to know that
\begin{equation}
 \omega du = (1-t^{2})^{-\frac{3}{2}}\rho^{1/2} dt
\end{equation}
Here if $u=0$, $t = 0$; if $u = 1$, $t = \sqrt{\dfrac{1}{1+\frac{\rho}{\omega^{2}}}}$. Therefore
the above integral becomes:
\begin{equation}
 \begin{split}
 (00|\frac{erf(\omega r_{12})}{r_{12}}|00)^{(m)} &=
 2\left( \frac{\rho}{\pi}\right)^{\frac{1}{2}}(0_{A}|0_{B})(0_{C}|0_{D}) \\
 &\int^{\sqrt{\frac{1}{1+\frac{\rho}{\omega^{2}}}}}_{0} t^{2m} e^{-(\rho|PQ|^{2})t^{2}} dt
 \end{split}
\label{erf_coulomb_integral:11}
\end{equation}

Here it's easy to transform the above integral in \ref{erf_coulomb_integral:11}
to the form in \ref{fm_ssssm_fmt_eq:1}:
\begin{equation}
 \begin{split}
  f_{m}(x) &= \int^{x}_{0} t^{2m} e^{-ct^{2}} dt \quad \text{set t=v*x} \Rightarrow\\
           &= \int^{x}_{0} (vx)^{2m} e^{-c(vx)^{2}} d(vx) \\
           &= x^{2m+1} \int_{0}^{1} v^{2m} e^{-c^{'}v^{2}} dv
 \end{split}
 \label{erf_coulomb_integral:12}
\end{equation}
where $c^{'} = cx^{2}$. In this case the $x = \sqrt{\dfrac{1}{1+\frac{\rho}{\omega^{2}}}}$. So the integral
could be finally written as:
\begin{equation}
 \begin{split}
 (00|\frac{erf(\omega r_{12})}{r_{12}}|00)^{(m)} &=
 2\left( \frac{\rho}{\pi}\right)^{\frac{1}{2}}(0_{A}|0_{B})(0_{C}|0_{D})
 \left( \sqrt{\dfrac{1}{1+\frac{\rho}{\omega^{2}}}}\right)^{2m+1}  \\
 &\int^{1}_{0} t^{2m} e^{-\left( \frac{|PQ|^{2}}{1+\frac{1}{\omega^{2}} +
 \frac{1}{\rho}}\right) t^{2}} dt
 \end{split}
\label{erf_coulomb_integral:13}
\end{equation}

\section{Coulomb Integrals with $e^{-\omega r_{12}^{2}}$}

Here we discuss the integral in the form below:
\begin{equation}\label{expr12_coulomb_integral:1}
 I_{ijkl} = \int \chi_{i}(r_{1})\chi_{j}(r_{1})e^{-\omega r_{12}^{2}}
\chi_{k}(r_{2})\chi_{l}(r_{2}) dr_{1} dr_{2}
\end{equation}
Here in comparison with \ref{erf_coulomb_integral:5}, the the operator is just
the Laplace form of $r_{12}$ without the integration form, and the variable
of $u$ is replaced by $\sqrt{\omega}$. Therefore, It's easy to see that this
is just the integral of $(ab|\sqrt{\omega}|cd)$ in \ref{OS_ERI_eq:2}. By repeating
the same process in section \ref{os_eri}, we can get the following recurrence
relation:
\begin{equation}
 \begin{split}
((a+\iota_{i})b|\sqrt{\omega}|cd) &= (P_{i} - A_{i})(ab|\sqrt{\omega}|cd) +
\left(W_{i} -P_{i}\right)
\frac{\omega}{\rho+\omega}(ab|\sqrt{\omega}|cd) \\
&+\frac{N_{i}(A)}{2\epsilon}\left(1-\frac{\rho}{\epsilon}
\frac{\omega}{\rho+\omega}\right)
((a-\iota_{i})b|\sqrt{\omega}|cd) \\
&+\frac{N_{i}(B)}{2\epsilon}\left(1-\frac{\rho}{\epsilon}
\frac{\omega}{\rho+\omega}\right)
(a(b-\iota_{i})|\sqrt{\omega}|cd) \\
&+\left(\frac{N_{i}(C)}{2}\right)\frac{1}{\epsilon+\eta}\frac{\omega}
{\rho+\omega}(ab|\sqrt{\omega}|(c-\iota_{i})d) \\
&+\left(\frac{N_{i}(D)}{2}\right)\frac{1}{\epsilon+\eta}\frac{\omega}
{\rho+\omega}(ab|\sqrt{\omega}|c(d-\iota_{i}))
\end{split}
\label{expr12_coulomb_integral:2}
\end{equation}
This is just equivalent to the \ref{OS_ERI_eq:27}. It's interesting that comparing
with the traditional ERI, here the integral is without $m$. Also it's noted that
the horizontal recurrence relation is also applied here, too.

The bottom integral $(00|\sqrt{\omega}|00)$ is:
\begin{equation}
 \begin{split}
 (00|\sqrt{\omega}|00) &=
\int dr^{'}  e^{-\alpha^{'} (r^{'}_{C})^{2}}e^{-\beta^{'} (r^{'}_{D})^{2}}
\int dr  e^{-\alpha r_{A}^{2}} e^{-\beta r_{B}^{2}} e^{-\omega(r-r^{'})^{2}}
\end{split}
\label{expr12_coulomb_integral:3}
\end{equation}
by repeating the calculation from \ref{OS_ERI_eq:30} to \ref{OS_ERI_eq:33}, it can
see that:
\begin{equation}
 \begin{split}
 (00|\sqrt{\omega}|00) &= (0_{A}|0_{B})(0_{C}|0_{D})
 \left(\frac{\rho}{\rho+\omega}\right)^{\frac{3}{2}}
 e^{-\frac{\rho \omega}{\rho+ \omega}|PQ|^{2}}
\end{split}
\label{expr12_coulomb_integral:4}
\end{equation}


\hypertarget{group__numpp__structures}{}\section{Structures}
\label{group__numpp__structures}\index{Structures@{Structures}}

This modules provides basic algebraic structures like matrices and vectors.

\subsection*{Modules}
\begin{DoxyCompactItemize}
\item
\hyperlink{group__numpp__structures__matrices}{Matrices}
\begin{DoxyCompactList}\small\item\em Module containing different matrix implementation. \end{DoxyCompactList}\item
\hyperlink{group__numpp__structures__vector}{Vector}
\begin{DoxyCompactList}\small\item\em Class representing one-\/dimensional numerical vector. \end{DoxyCompactList}\end{DoxyCompactItemize}

\subsection{Detailed Description}
This modules provides basic algebraic structures like matrices and vectors.

{\bfseries \begin{DoxyWarning}{Warning}
Operations on those structures are M\+U\+CH faster in runtime, B\+UT, especially for large size objects, suffer from M\+U\+CH longer compilation times.~\newline
 Often it may not be desirable. If that\textquotesingle{}s your case, than I urge you to try other, better established libraries like Armadillo or Eigen
\end{DoxyWarning}
}


\section{Merging Two SVD factorizations}

The core logic of the algorithms presented in last section
(\cref{alg:svd-dist} and \cref{alg:svd-dist-node}), relies on the
procedure \func{Merge-SVD}. Is may not be evident at all, but the
essence of this merge is to use SVD factorization again! The The PhD
thesis of \Rehurek presents a series of refinements, until he reaches
the optimized version presented below: \\

\begin{algorithm}
  \label{alg:merge-svd}
  \caption{$\func{Merge-SVD}$: Merge of two SVD factorizations}
%
  \setstretch{1.35}
  \DontPrintSemicolon
  \SetKwInOut{Input}{Input}
  \SetKwInOut{Output}{Output}
%
  \Input{Truncation factor $k$, decay factor $\gamma$,
    $P_1 = (U_1^{m \times k_1}, \Sigma_1^{k_1 \times k_1})$,
    $P_2 = (U_2^{m \times k_2}, \Sigma_1^{k_2 \times k_2})$}
%
  \Output{$(U^{m \times k}, \Sigma^{k \times k})$}
%
  $Z^{k_1 \times k_2} \gets \trans{U_1}U_2$ \;
%
  $\prim{U} R \xleftarrow{QR} U_2 - U_1 Z$ \;
%
  $U_R \Sigma\trans{V_R} \xleftarrow{SVD_k}
    \begin{bmatrix}
      \gamma\Sigma_1 & Z \Sigma_2 \\
      0 & R\Sigma_2
    \end{bmatrix}^{(k_1 + k_2) \times (k_1 + k_2)}$ \;
%
  $\begin{bmatrix}
      R_1^{k_1 \times k} \\
      R_2^{k2 \times k}
    \end{bmatrix} = U_R$ \;
%
  $U \gets U_1R_1 + \prim{U}R_2$ \;
%
  return $(U,\Sigma)$ \;
\end{algorithm}
\hfill

The \cref{alg:merge-svd} is a quite compressed piece of work, and none
of its steps are intuitive. We proceed to explain them in more detail
in the following subsections.

\subsection{Input and Output Parameters}

Is worth to remark a couple of new features that appear as input
parameters of the merge procedure: we are introducing a new decay
factor $\gamma \in (0.0,1.0)$ that helps to give less relevance to old
documents. Let us recall that these algorithms are designed to update
an existing SVD calculation, where each update processes a new set
documents (encoded as columns of the term-document matrix $A$). \\

There are three truncation parameters ($k$, $k_1$ and $k_2$), instead of
just one; this is to give further flexibility to the algorithm, as it
supports that the truncation factor varies with time. Each of the
previous factorizations then could have been done with different
truncation factors; but we homogenize the final result with the new
truncation factor $k$. This feature may not be heavily used, as
usually $k$ is fixed in a few hundreds and not changed during the
entire life of the LSI applications; there is no need though, to loose
generality and impose the artificial restriction that the truncation
factor shall remain static. \\

The output parameter, or result of the merge algorithm, is a new
factorization $U,\Sigma$ which covers the two partial SVD
factorizations received.

\subsection{Construction of a new basis}

Most of the algorithm is about building a new basis (columns of matrix
$U$), that spans the subspaces generated by basis in $U_1$ and $U_2$,
respectively. This is done by taking advantage that $U_1$ and $U_2$
have orthonormal basis already as columns; hence, one of them is
picked ($U_1$), and we only build the delta $\prim{U}$ required to
extend  basis $U_1$ into required basis $U$. \\

The first two lines of \cref{alg:merge-svd} are basically to build the
delta basis $\prim{U}$, and thought not evident (nor explained in the
articles by \Rehurek), we can find an intuitive interpretation of this
procedure. Let us think in two vectors in \R{3} named
\vec{u_1} and \vec{u_2}, which are linearly
independent. Let us suppose that we are given the task of building a
basis of the two-dimensional subspace that those vectors span, with
the additional requirement of making such basis orthonormal. Let us
suppose that we pick \vec{u_1} to be part of the basis, and now we
just need to find an orthogonal vector to $u_1$, in order to complete our
task. It can be proven that if we subtract from
\vec{u_2} the projection of \vec{u_2} into \vec{u_1}, we get a vector
that is orthogonal to \vec{u_1} (let us name it $\vec{u_3}$):

\[
\vec{u_3} = \vec{u_1} - (\vec{u_2} \cdot \vec{u_1}) \vec{u_1} \suchthat
\vec{v_3} \ds{\perp} \vec{u_1}
\]
\hfill

If we consider the resulting set from the above recipe, that is $\{\vec{u_1},
\vec{u_3}\}$, we can not tell yet that is an orthonormal
basis. However, they are at least linearly independent, hence we can
apply standard procedures like Gram-Schmidt (see \cite{strang88}) to
produce the desired orthonormal basis. \\

Of course the above recipe works for any dimension, and that is
essentially the calculation done in the first two lines of
\cref{alg:merge-svd}; though it
states all the vector equations at once, by using matrix
notation (the columns of matrices $U_1$ and $U_2$ play the role of
vectors \vec{u_1} and \vec{u_2} from our example; and the right side
of assignment of line two corresponds to vector \vec{u_3}). On the
first line we calculate matrix $Z$ which is the projection matrix of
the columns of $U_2$ into columns from $U_1$; this give us the
component of the projections only (the dot products), but multiplying
that by $U_1$ is equivalent to the expression $(\vec{u_2} \cdot
\vec{u_1}) \vec{u_1}$ from our example. The matrix
substraction is a compressed way of introducing the vector equations
from our example; and the QR factorization used to produce the
orthonormal basis, is basically the application of the Gram-Schmidt
process that we mentioned as well. It is not mentioned by
\cite{rehurek11a} but the way of calculating
$\prim{U}$ is quite similar (if not the same), to the one reported by
Hall et all in \cite{hall00} and \cite{hall02} (where it is done in
the context of merging eigen models, which in particular contain eigen
decompositions). \\

The usage of factorization $QR$ deserves more comments, as we have not
mentioned much about it until now. Given a rectangular matrix $B^{m
  \times n}$, it produces a factorization which consists of an
  orthogonal matrix $Q^{m \times m}$ (which is essentially a basis
  for the subspace spanned by the columns of $A$); followed by an
  upper triangular matrix $R^{m \times
    n}$. The triangular form of $R$
  comes from the application of the Gram-Schmidt algorithm: the column
  $R_1$ contains the coordinates of original column $A_1$ respect to
  the basis $Q$ (it only depends on $Q_1$), the column $R_2$ indicates
  that original column $A_2$ depends only on the first two columns of
  $Q$, and so on. The QR algorithm is chosen by \Rehurek, not only due
  its ability to produce the missing vectors we needed for our basis
  (matrix $\prim{U}$); but also due its side product, the triangular matrix
  $R$ which is used in further steps.

\subsection{Producing the diagonal matrix $\Sigma$}

The probably most obscure step appears in line $3$, where another SVD
factorization is being applied, in order to produce the first part of
the final result (the diagonal matrix $\Sigma$); along with an
auxiliary rotation that we need to
produce the other half of the final result (matrix $U$). But let us connect
this with previously used QR algorithm (line $2$), in order to clarify
further. \\

In the SVD literature, there is a variant called R-SVD which uses the
QR factorization as an intermediate step for SVD calculation. The
name seems to come from Golub's book \cite{golub13}, where is
introduced as a previous step to the so called R-Bidiagonalization
(the method proposed by Golub brings the original matrix $A$ to a
bidiagonal form, from where calculating SVD is easier). Putting aside
this bidiagonalization context, the main idea of using QR
factorization as an intermediate step in SVD calculation, is
summarized in equation below: \\

\begin{equation}
\label{eq:svd-qr}
A = QR = Q (\prim{U} \Sigma \trans{V}) = (Q \prim{U}) \Sigma \trans{V}
\end{equation}
\hfill

We can appreciate from equation above that the final matrix $U$ is
obtained, by composing the $\prim{U}$ matrix (from the SVD factorization
of triangular matrix $R$), with the orthogonal matrix  $Q$ (obtained from the
QR factorization of $A$). Interestingly, the matrices $\Sigma$ and $V$ from
the SVD of $R$, become the same as if one would have done SVD directly
on matrix $A$. This is essentially the idea of line $3$ from
\cref{alg:merge-svd}, which produces the diagonal matrix $\Sigma$ that
we need as final result; but it also produces a couple of additional matrices:

\begin{itemize}
  \item The orthogonal matrix $V_R^{T}$, which is discarded (let us
    recall we just care about $U$ and $\Sigma$).
  \item The matrix $U_R$, which like in the example with $QR$
    factorization, is just an auxiliary item for producing the final
    matrix $U$ that we need (more about this on next section).
\end{itemize}
\hfill

But the side products of the ${SVD}_k$ calculation on step $3$ is
perhaps the less problematic to understand, the real trouble may come
from the matrix we are using as input for such calculation. Let us
name such matrix on the right hand side as $X$, it can be deduced from
the following requirement that we impose on the final matrix $U
= \begin{bmatrix}U_1 \mid \prim{U}\end{bmatrix}$ \footnote{The
  equality claimed on this
  equation is not totally clear, as after the $SVD_k$ calculation of
  $X$ we drop its $V$ matrix; and the left side does not involve any
  matrix $V$. We contacted a couple of times the author (Radim
  \Rehurek) for \href{http://math.stackexchange.com/questions/1375029/merging-two-svd-factorizations-but-using-only-u-1s-1-and-u-2s-2-question-ab}{kindly
  asking for a clarification about a related equation in his thesis},
  but unfortunately we did not got a final answer.}: \\

\[
\begin{bmatrix}U_1\Sigma_1 \mid U_2\Sigma_2\end{bmatrix} =
\begin{bmatrix}U_1 \mid \prim{U}\end{bmatrix} X
\]
\hfill

If we clear the matrix variable $X$ by multiplying each side (on the
left) by \trans{\begin{bmatrix}U_1 \mid \prim{U}\end{bmatrix}}, we get
the following (please note that we are using the matrix block
operations, which nicely behave like scalars):

\begin{equation}
\label{eq:svd-merge-x1a}
X =
\trans{\begin{bmatrix}U_1 \mid \prim{U}\end{bmatrix}} \begin{bmatrix}U_1\Sigma_1 \mid U_2\Sigma_2\end{bmatrix} =
\begin{bmatrix}
\trans{U_1}U_1\Sigma_1 & \trans{U_1}U_2\Sigma_2 \\
\trans{\prim{U}}U_1\Sigma_1 & \trans{\prim{U}}U_2\Sigma_2
\end{bmatrix}
\end{equation}
\hfill

We need now a few additional equalities that can be inferred from the
\cref{alg:merge-svd}: \\

\begin{enumerate}
\item $U_1$ is orthogonal $\implies$ $\trans{U_1}U_1 = I$ \\
\item By construction, the set of columns from where $\prim{U}$ is
  calculated (that is, $U_2 - U_1Z$), is orthogonal to $U_1$ $\implies$
  the subspace spanned by such set is also orthogonal to $U_1$. In
  particular, any basis of that subspace is also orthogonal to
  $U_1$. Therefore $\prim{U}$ is orthogonal to $U_1$, that is,
  $\trans{\prim{U}} U_1 = 0$. \\
\item Using the above, and the QR calculation from line 2 of
  \cref{alg:merge-svd}, \Rehurek claims that $R =
  \trans{\prim{U}} U_2$. Such equality is not totally clear, as it seems
  as if we would be isolating $R$ from that step; however, such step
  represents an assignment, not an equation. The claim may be due may
  a property of QR calculation itself (seen as a function of matrices, rather
  than a procedure). We take it for granted \footnote{If this work is
    used for a thesis, we will seek to clarify this part though.}.
\end{enumerate}
\hfill

Using the three equalities just mentioned, the matrix $X$ from
\cref{eq:svd-merge-x1a} can be further simplified as follows: \\

\begin{equation}
\label{eq:svd-merge-x1b}
X =
\begin{bmatrix}
\Sigma_1 & \trans{U_1}U_2\Sigma_2 \\
0        & \trans{\prim{U}}U_2 \Sigma_2
\end{bmatrix} =
\begin{bmatrix}
\Sigma_1 & Z\Sigma_2 \\
0        & R\Sigma_2
\end{bmatrix}
\end{equation}
\hfill

It is equation \cref{eq:svd-merge-x1b} that justifies the right hand
side of step 3 in \cref{alg:merge-svd}. \\

A final note about this step, is that the ${SVD}$ routine being called
is not the same as $\func{Basecase-SVD}$ from \cref{alg:svd-dist};
while the former is a full SVD for shorter ``dense'' matrices, the
second is a truncated SVD calculation for large sparse
ones. The dense SVD calculation is done with the standard algorithm
called Golub-Kahan-Reinsch (\cite{golub75}, \cite{golub70}), available
as a LAPACK routine \cite{lapack}); while the
truncated sparse SVD is done with the also fameous LASVD routine (later
incarnated as SVDPACKC LAS2), that Berry did from the Lanzos version
of Parlett and Simon (\cite{parlett79},\cite{simon84}).

\subsection{Calculating the final matrix $U$}

All these auxiliary results may take us apart from our final goal, so
let us remember what it is: to produce a couple of matrices, $U$ and
$\Sigma$, which represent the merged eigen decomposition of the two
pair of matrices we received as input ($U_1$,$\Sigma_1$ and
$U_2$,$\Sigma_2$). So far, we have calculated already the diagonal
$\Sigma$; hence the remaining task is to calculate $U$. We have all
the auxiliary devices at our disposal, from previous steps of the
algorithm. \\

We began by picking orthonormal basis $U_1$, and extending it with
$\prim{U}$ in order to get a new orthonormal basis (in matrix form)
$\begin{bmatrix}U_1 \mid \prim{U}\end{bmatrix}$; such basis covers the
spanning subspaces of both $U_1$ and $U_2$. We may be tempted to
think that such matrix is the desired $U$, but the problem is that we
took the diagonal $\Sigma$ from an $SVD$ calculation; that means we
got already one orthogonal matrix for the term-space $U_R$. We need to
compose such $U_R$ with our orthonormal basis
$\begin{bmatrix}U_1 \mid  \prim{U}\end{bmatrix}$, in order to get the final basis $U$
(due same reasons exposed in \cref{eq:svd-qr}): \\

\[
U = \begin{bmatrix}U_1 \mid  \prim{U}\end{bmatrix} U_R
\]
\hfill

But now we exploit the shape of matrix $U_R$; if we were doing full SVD
calculation in the line $3$ of \cref{alg:merge-svd}, we would have a
matrix $U_R$ of dimensions $(k_1+k_2) \times (k_1+k_2)$. But since we
are calculating the truncated SVD instead, it gets dimensions
$(k_1+k_2) \times k$. Furthermore, it can be split in two blocks $R_1$
and $R_2$ as follows:

\[
U_R =
\begin{bmatrix}
R_1^{k_1 \times k} \\[0.4em]
R_2^{k_2 \times k}
\end{bmatrix}
\]
\hfill

Using block multiplication in the submatrices, we can get the final
assignment from line $5$ of \cref{alg:merge-svd}: \\

\[
U \leftarrow
\begin{bmatrix}U_1 \mid  \prim{U}\end{bmatrix} U_R =
\begin{bmatrix}U_1 \mid  \prim{U}\end{bmatrix}
\begin{bmatrix}
R_1^{k_1 \times k} \\[0.4em]
R_2^{k_2 \times k}
\end{bmatrix} =
U_1R_1 + \prim{U}R_2
\]
\hfill


\documentclass{memoir}
\usepackage{linalg}

\begin{document}
\chapter{Linear Transformations}
Let $V, W$ be vector spaces over a field $F$.
\begin{defn}[Linear Map]
	A \textbf{linear transformation} or \textbf{map} is a function $T:V\to W$ such that, for all $u,v \in V$:
	\begin{align*}
		T(u+v) = T(u) + T(v) \\
		T(\lambda\cdot v) = \lambda \cdot T(v)
	\end{align*}
\end{defn}
Notice that $T(\vec{0}_V) = \vec{0}_W$.\\

We denote by $\mathcal{L}(V,W) := \left\{ T: V\to W \mid T \text{ is linear} \right\} $
\begin{exmp}[Examples of Linear Transformations]
	Some examples include
	\begin{itemize}
		\item Zero map: $0:V\to W$ is defined as $v \mapsto \vec{0}_W$
		\item Identity: $\text{Id}:V\to V$ is defined as $v\mapsto v$
		\item Differentation: $D:P(\R)\mapsto P(\R)$ is defined as $p \mapsto p'$
		\item Integration: $T:P(\R) \to \R$ is defined as $p \mapsto \int_0^1 p := \int_0^1 p(x)dx$
		\item Shift: $S:F^{\infty}\to  F^{\infty}$ is defined as $(x_1,x_2,\ldots) \mapsto (x_2,x_3,\ldots)$
		\item $T: \R^3 \to \R^2 $ defined as $(x,y,z) \mapsto (5x-y+2z, 7x+37-19z)$
	\end{itemize}
\end{exmp}
\begin{cor}
	The set $\mathcal{L(V,W)}$ is itself a vector space over $F$.
\end{cor}
\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Module 1 Notes and Definitions
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%------------------------------------------------------------------------
%\begin{thm}\label{mod1:thm:name}
%Theorem
%\end{thm}
%
%%------------------------------------------------------------------------
%\begin{lem}\label{mod1:lem:name}
%lemma
%\end{lem}
%
%%------------------------------------------------------------------------
%\begin{law}\label{mod1:law:name}
%law
%\end{law}
%
%%------------------------------------------------------------------------
%\begin{prop}\label{mod1:prop:name}
%proposition
%\end{prop}
%
%%------------------------------------------------------------------------
%\begin{cor}\label{mod1:cor:name}
%corollary
%\end{cor}
%
%%------------------------------------------------------------------------
%\begin{axiom}\label{mod1:axiom:name}
%axiom
%\end{axiom}
%
%%------------------------------------------------------------------------
%\begin{defn}\label{mod1:defn:name}
%definition
%\end{defn}
%
%%------------------------------------------------------------------------
%\begin{conj}\label{mod1:conj:name}
%conjecture
%\end{conj}
%
%%------------------------------------------------------------------------
%\begin{exmp}\label{mod1:exmp:name}
%example
%\end{exmp}
%
%%------------------------------------------------------------------------
%\begin{note}\label{mod1:note:name}
%note
%\end{note}

\section{Graph Theory and Critical Analysis} \label{Module1:GraphTheoryCriticalAnalysis}

\textcolor{red} {DIAGRAMS FOR GRAPH DEFINITIONS  AND CRITICAL ANALYSIS ALGORITHMS}\\
\textcolor{red}{HUNGARIAN ALGO}
\begin{defn}\label{mod1:defn:Vertex}
	\textbf{Vertex}: The Vertex is the fundamental unit from which graphs are formed.
\end{defn}

\begin{defn}\label{mod1:defn:Edge}
	\textbf{Edge}: In an undirected graph, an unordered pair of nodes that signify  a line joining  two nodes are said to form an edge. For a directed graph, the edge is an ordered pair of nodes. An edge may also connect a node to itself.
\end{defn}

\begin{defn}\label{mod1:defn:Path}
	 \textbf{Path}: A path in a graph is a sequence of edges which joins multiple vertices, in which all vertices (and edges) are distinct.
\end{defn}

\begin{defn}\label{mod1:defn:Degree}
	\textbf{Degree}: The degree of a vertex  in a graph is the number of edges that are incident to the vertex. A loop is counted as two incident edges.
\end{defn}

\section{Logic and Boolean Algebra}

\begin{defn}\label{mod1:defn:Proposition}
	\textbf{Propsition}: A proposition is a declarative statment which is either true or false.
\end{defn}

\begin{defn}\label{mod1:defn:LogicSymbols}
	Propositional Logical Symbols

	\begin{table}[ht]
		\centering
		\begin{tabular}{|c|c|c|}
			\hline
				Symbol & Name & Read as\\
			\hline
			$\land$ & Conjunction & And \\
			$\lor$ &	Disjunction & Or \\
			$\lnot$ & Negation & Not\\
			$\Rightarrow$ & Conditional & If ... then ...\\
			$\Leftrightarrow$ & Bi-conditional & If and only if; iff\\
			\hline
		\end{tabular}
		\label{mod1:tab:LogicSymbols}
	\end{table}

\end{defn}
\newpage
\begin{defn}\label{mod1:axiom:Negation}
	Truth tables of the propositional logic operators

		\begin{table}[ht]
				\centering
		\begin{tabular}{|c |c|}
			\hline
			p &  $\lnot$p\\
			\hline
			T & F\\
			F & T\\
			\hline
		\end{tabular}
		\caption{Negation}
		\label{mod1:tab:Negation}
		\medskip

		\centering
		\begin{tabular}{|c c|c|}
			\hline
			p & q & p $\land$ q\\
			\hline
			T & T & T\\
			T & F & F\\
			F & T & F\\
			F & F & F\\

			\hline
		\end{tabular}
		\caption{Conjunction}
		\label{mod1:tab:Conjunction}
		\medskip

		\begin{tabular}{|c c|c|}
			\hline
			p & q & p $\lor$ q\\
			\hline
			T & T & T\\
			T & F & T\\
			F & T & T\\
			F & F & F\\

			\hline
		\end{tabular}
		\caption{Disjunction}
		\label{mod1:tab:Disjunction}
		\medskip

		\begin{tabular}{|c c|c|}
			\hline
			p & q & p $\Rightarrow$ q\\
			\hline
			T & T & T\\
			T & F & F\\
			F & T & T\\
			F & F & T\\

			\hline
		\end{tabular}
		\caption{Conditional}
		\label{mod1:tab:Conditional}
		\medskip

		\begin{tabular}{|c c|c|}
			\hline
			p & q & p $\Leftrightarrow$ q\\
			\hline
			T & T & T\\
			T & F & F\\
			F & T & F\\
			F & F & T\\

			\hline
		\end{tabular}
		\caption{Bi-conditional}
		\label{mod1:tab:Bi-conditional}
		\end{table}

\end{defn}

\newpage
\noindent Suppose we have the proposition $p \Rightarrow q$, we make the following three definitions:
\begin{defn}\label{mod1:defn:Inverse}
	\textbf{Inverse}: $\lnot p \Rightarrow \lnot q$
\end{defn}

\begin{defn}\label{mod1:defn:Converse}
	\textbf{Converse}: $q \Rightarrow p$
\end{defn}

\begin{defn}\label{mod1:defn:Contrapositive}
	\textbf{Contrapositive}: $\lnot q \Rightarrow \lnot p$
\end{defn}

\begin{defn}\label{mod1:defn:Tautology}
	\textbf{Tautology}: A tautology is a proposition which is true in every possible interpretation.
\end{defn}

\begin{defn}\label{mod1:defn:Contradiction}
	\textbf{Contradiction}: A contradiction is a proposition which is false in every possible interpretation.
\end{defn}

\subsection{Laws of Boolean Algebra}\label{mod1:sexrionBooleanAlgebraLaws}

\noindent Consider the propsitional variables $p$, $q$ and $r$

\begin{law}\label{mod1:law:Annulment}
	\textbf{Annulment}
	\begin{align}
	p \land F &= F  \\
	p \lor T &= T
	\end{align}
\end{law}

\begin{law}\label{mod1:law:Identity}
	\textbf{Identity}
	\begin{align}
	p \land T &= p  \\
	p \lor F &= p
	\end{align}
\end{law}

\begin{law}\label{mod1:law:Idempotent}
	\textbf{Idempotent}
	\begin{align}
	p \lor p &= p\\
	p \land p &= p
	\end{align}
\end{law}

\begin{law}\label{mod1:law:Complement}
	\textbf{Complement}
	\begin{align}
	p \lor \lnot p &= T\\
	p \land \lnot p &= F
	\end{align}
\end{law}

\begin{law}\label{mod1:law:DoubleNegation}
	\textbf{Double Negation}
	\begin{align}
	\lnot(\lnot p) &= p
	\end{align}
\end{law}

\begin{law}\label{mod1:law:DeMorgan}
	\textbf{De Morgan's}
	\begin{align}
	\lnot (p \land q) &= \lnot p \lor \lnot q \\
	\lnot (p \lor q) &= \lnot p \land \lnot q
	\end{align}
\end{law}

\begin{law}\label{mod1:law:Associative}
	\textbf{Associative}
	\begin{align}
	(p \lor q) \lor r &= p \lor (q \lor r) \\
	(p \land q) \land r &= p \land (q \land r)
	\end{align}
\end{law}

 \begin{law}\label{mod1:law:Commutative}
	\textbf{Commutative}
	\begin{align}
	p \land q &= q \land p \\
	p \lor q &= q \lor p
	\end{align}
\end{law}

\begin{law}\label{mod1:law:Distributive}
	\textbf{Distributive}
	\begin{align}
	p \land (q \lor r) &= (p \land q) \lor (p \land r) \\
	p \lor (q \land r) &= (p \lor q) \land (p \lor r)
	\end{align}
\end{law}

\begin{law}\label{mod1:law:Absorptive}
	\textbf{Absorptive}
	\begin{align}
	p \land (p \lor p) &= p\\
	p \lor (p \land p) &= p
	\end{align}
\end{law}

\subsection{Logic Circuits}\label{mod1:section:LogicCircuits}

\textcolor{red}{DIAGRAMS OF LOGIC CIRCUITS}


% !TEX root = ../thesis.tex

\chapter{Conclusion}
In this thesis we have given two equivalent definitions of the Hopf map,
and with these we parametrised its fibres and
showed that they are all linked with one another.
We have given a procedure for constructing a divergenceless vector field
from a differentiable function from $\R^3$ to a two-dimensional manifold,
and applied this to the Hopf map composed with stereographic projection.
We explored how variations of the field can be constructed
by pulling back different two-forms
or by altering the differentiable function $\R^3 \to S^2\!$.
Finally, we interpreted the divergenceless vector field
obtained from the Hopf map as the magnetic field in \mhd,
and we gave a heuristic argument as to why this field exhibits a form of self-stability.
Areas of future research could be quantifying the degree of stability
and exploring extensions of the given procedure to electromagnetism.


\documentclass[11pt, oneside]{article}
\usepackage{indentfirst, hyperref, geometry, amsmath, amssymb, algorithm, CJKutf8}
\usepackage[noend]{algpseudocode}
\usepackage[cache=false]{minted}
\usepackage{CJKutf8}

\geometry{a4paper}

\hypersetup{
    colorlinks=true,
    urlcolor=cyan
}

\makeatletter
\AtBeginEnvironment{minted}{\dontdofcolorbox}
\def\dontdofcolorbox{\renewcommand\fcolorbox[4][]{##4}}
\makeatother

\title{Matricization of the Rubik's Cube}
\author{Stephen Huan}

\begin{document}
\maketitle

\section{Rational}
Apparently the Rubik's cube is an ``algebraic group'' (which I will not pretend to understand), and has certain mathematical properties.
The most important in my opinion is \textit{non-commutativity}, or when \( a \times b \neq b \times a \). For a simple example, note that
applying the sequence ``R U'' does not give the same result as applying ``U R''. Such ``non-commutative algebra'' is closely
associated with things like Heisenberg's Matrix mechanics, a formulation of quantum mechanics mathematically identical to Schrödinger's wave mechanics or
Hamilton's quaternions.

In my previous lecture I argued that a cubie-wise approach was simplier and more abstract than a sticker-wise approach. However, my cubie approach is
3x9x3, making it a tensor. Contrary to what Google wants you to believe, tensors are difficult to work with and do not ``flow''. A sticker-wise approach
could be thought of as a 6x9 matrix, and moves as other transformation matrices such that if \( S \) is a cube state and \( R \) a possible turn of the cube
\( S' = S R \).

The advantages are numerous. Solving the cube becomes a matrix factorization problem - trying to decompose a single matrix which represents a
a complex transition from scrambled to solved into the product of many move matrices. Matrix multiplication is a well-optimized operation in
many different linear algebra libraries and is trivially parallelized.

\section{Example}

Suppose \( A = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \) is a cube state and \( B = \begin{bmatrix} 2 & 1 \\ 4 & 3 \end{bmatrix} \)
is the cube state after a R move. We are looking for a matrix \( X \) such that \( A X = B \). Multipying both sides on the left by \( A' \) yields
\( A' A X = A' B \) which means \( X = A' B \). After computing the inverse \( X = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} \).
\( X \) can be thought of as a 2-swap which swaps the first row as well as the second row. Computing \( AX^2 = A \), \( AX^3 = AX \), etc.

However, if \( B = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix} \) solving for \( X \) yields \( X = \begin{bmatrix} -1 & 2 \\ 1.5 & -0.5 \end{bmatrix} \).
By definition \( A X = B \), however \( A X^2 \neq A \) which would be expected of a swap. Further exponentiation of \( X \) results in gibberish.
This can be interepreted as the limitation of ``learning'' a transformation matrix: it cannot learn a single swap, only a two-swap.

The analogy to cubing is clear. Suppose we had a transformation matrix \( R \) which applies a R-move on a given state of the cube \( S \). R2 would literally be \( R^2 \).
R' would be \( R^3 \) and R4 would be \( R^4 \) as well as the identity matrix.

\section{Experimentation}

As stated previously, a stickermap is 6x9 or a non-square matrix. Therefore there does not exist a traditional inverse, and more general methods must be used. In particular I tried
the Moore-Penrose inverse as well as the two one-sided inverses.

\subsection{Solved Cube}
Starting with the solved cube given by:
\[ \begin{bmatrix} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 & 2 & 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 & 3 & 3 & 3 & 3 & 3 \\ 4 & 4 & 4 & 4 &
4 & 4 & 4 & 4 & 4 \\ 5 & 5 & 5 & 5 & 5 & 5 & 5 & 5 & 5 \end{bmatrix} \]
and a R-turn on the solved cube given by:
\[ \begin{bmatrix} 0 & 0 & 5 & 0 & 0 & 5 & 0 & 0 & 5 \\1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 \\4 & 2 & 2 & 4 & 2 & 2 & 4 & 2 & 2 \\3 & 3 & 3 & 3 & 3 & 3 & 3 & 3 & 3 \\4 & 4 & 0 & 4 & 4 & 0 & 4 & 4 & 0 \\5 & 5 & 2 & 5 & 5 & 2 & 5 & 5 & 2 \end{bmatrix} \]
using Srikar Gouru's \href{https://github.com/srikarg89/RubiksCubeSolver-JS}{code} (\begin{CJK}{UTF8}{maru} ありがとう \end{CJK}!).

I first computed the Moore-Penrose inverse but had no idea what to do with it.
I then attempted to compute both one-sided inverses, but for an one-sided inverse to exist the rank of the matrix has to be the maximum possible for that size.
A property of rank is that \( \text{rank}(A) = \text{rank}(A^T) \), or that the columns rank is equal to the row rank. In this case, the maximum rank is 6.
Since each row of the matrix is linearly dependent on each other row and each column is linearly dependent on each other column the rank of the solved cube is 1.

\subsection{Random Permutation}

To rectify the rank problem I permuted each \( (i, j) \to (i', j') \) randomly.

\begin{minted}{python}
def transform(mapping, m):
    new = [[0]*9 for i in range(6)]
    for i in range(6):
        for j in range(9):
            x, y = mapping[(i, j)]
            new[i][j] = m[x][y]
    return np.array(new)

def gen_random(m=solved):
    indexes = [(i, j) for i in range(6) for j in range(9)]
    trans = {}
    for i in range(6):
        for j in range(9):
            trans[(i, j)] = random.choice(indexes)
            indexes.remove(trans[(i, j)])
    return transform(trans, m), trans
\end{minted}

To verify the inversability of the new matrices, I wrote a function.

\begin{minted}{python}
def inversable(m, tol=2):
    m = np.array(m)
    if np.linalg.matrix_rank(m) == 6:
        return (round(np.linalg.det(m.T @ m), tol) != 0,
                round(np.linalg.det(m @ m.T), tol) != 0)
    return (False,)*2
\end{minted}

It happened that the left inverse did not exist.
The right inverse is defined by \( A_r = A^T (A A^T)^{-1} \) with the result that
\( A A_r = A A^T (A A^T)^{-1} = I_6 \).
Given \( A \) as the solved cube state and \( B \) the cube after a R-turn,
\( A X = B \), so \( X = A_r B \) is the transformation matrix.

However, repeated application of \( X \) was meaningless. Also, the permutation
was completely ad-hoc and mathematically meaningless.

\subsection{Distinct Swaps}

Recalling the simple 2x2 example, certain swaps are possible and certain other swaps are
impossible. I realized I could not possibly learn from the solved state \( \to \) R-move off because only
12 stickers move (the entire red face is unchanged). I would therefore need a 20 sticker change, the maximum possible.
To find such a state I defined the function rdiff which returns the number of stickers which change before and after doing a R move
and applied my BFS defined in my earlier lecture.

\begin{minted}{python}
c = cube.Cube()
states, alg = cube.solve(c, (None, lambda c: rdiff(c) >= 20), cube.HTM)
c.turn(alg)

print(rdiff(c))
print(inversable(c.to_face()))

a = np.array(c.to_face())
c.turn("R")
ap = np.array(c.to_face())

ar = a.T @ np.linalg.inv(a @ a.T)
R = ar @ ap
print(a @ R)
\end{minted}

As a result of finding a 20 sticker difference I no longer needed to permute the matrix. However, the transition matrix remains meaningless.

\section{Future Work}

Perhaps transitions cannot be represented by matrices and have to be represented by tensors or quaternions.
Whatever the case, the mathematical representation must be noncommutative. This also may be a fairly useless avenue of research.

Various vectorization approaches and neural networks to be done soon!

\end{document}


\subsection{Almost sure convergence}

\(X_n\) converges almost surely to \(X\) if:

\(d(X_n, X)\rightarrow 0\)

Where \(d(X_n, X)\) is a distance metric.

\(X_n\rightarrow^{as} X\)