\chapter{Algorithms}
In this chapter, we will list some important algorithms which may be used in the solution of each subproblem.

\section{Shortest Path Between Two Nodes}

Algorithms for the calculation of the Shortest Path between two points are very important for the solution of this problem. They will be used to calculate the shortest path between multiple points when calculating a vehicle's path, such as the shortest path between the garage and a pick up point. To do this, there are many algorithms at our disposal. We will evaluate some of them and make a decision on which is(are) best for the situation at hand.

\subsection{Dijkstra's Algorithm}

\subsubsection{Description}
This algorithm was conceived by Edsger W. Dijkstra's in 1956 and is used to calculate shortest paths in directed or undirected graphs, so long there are no edges with negative weight. Upon utilization of this algorithm, a tree holding the shortest paths from the origin node to all others is formed.

\subsubsection{Extra Data, Data Structures and Algorithms Required}
\begin{itemize}
    \item Minimum distance to the origin node from each of the graph's nodes (dist)
    \item Node that comes before it in its path (path)
    \item Priority queue to hold the nodes to be processed next
    \item Decrease-Key function, to maintain the nodes with shortest distance in the top of the queue
\end{itemize}

\input{Algorithms/Dijkstra.tex}

\subsubsection{Analysis}
The first part of the algorithm's goal is to prepare the data (lines 1-6): paths are set to null, distances to infinity, the priority queue is initialized and the origin node is inserted. Then, a breadth-first search is performed; a check is made to every node found in order to understand if its path can be shortened by the use of the edge in analysis. After that, if the node is not already in the queue, it must be inserted, to be eventually processed. If it is, its key must be reduced. The items of the priority queue are ordered by their key. This system is used so that the nodes which are closest to the origin are processed first. This procedure ensures that the distance of already processed nodes remains intact, boosting the algorithm's efficiency and making this a greedy algorithm. \\
%\begin{itemize}
%    \item \textbf{Loop invariant} - $ priorityQueue \neq \emptyset $, as each time a node is processed, its shortest path is defined and all nodes will be processed when the queue is empty
%    \item \textbf{Loop variant} - the number of nodes to be processed, since the algorithm would stop when the queue is empty
%\end{itemize}

\subsubsection{Efficiency}
\begin{itemize}
    \item \textbf{Temporal Complexity} - $ O((|N|+|E|) \times \log(|N|) $, where E is the set of edges and N the set of nodes of the graph. Decrease-Key's time efficiency is $ O(|E| \times log(|N|)$. It can be $ O(1) $ if Fibonacci Heaps are used instead of regular Priority Queues, making the whole algorithm $ O(|N| \times log(|N|) $
    \item \textbf{Spatial Complexity} - Depends on the implementation, but usually $ O(|N|) $
\end{itemize}

\subsubsection{Usability}
This algorithm is one to consider using in the problem, but there may be slightly more efficient ones.

\subsection{Bellman-Ford Algorithm}

\subsubsection{Description}
The Bellman-Ford Algorithm is another algorithm used to calculate the shortest path between nodes in a graph. It was first proposed by Alfonso Shimbel in 1955 but ended up being named after Richard Bellman and Lester Ford, who later officially published it independently. This algorithm is useful to calculate shortest paths in graphs with negative weight edges and to detect negative edge cycles in them.

\subsubsection{Extra Data, Data Structures and Algorithms Required}
\begin{itemize}
    \item Minimum distance to the origin node from each of the graph's nodes (dist)
    \item Node that comes before it in its path (path)
\end{itemize}

\input{Algorithms/Bellman-Ford.tex}

\subsubsection{Analysis}
In this algorithm, the first step is to prepare the graph (lines 1-4). Then, all edges are analyzed $|N| - 1 $ times(lines 6-7), checking if they can be used to shorten the path from s to the destination node of the edge(line 8). This process is repeated this amount of times because it is the maximum length in edges for a path. In the end, a check is made, to see if the algorithm worked. If it did not, the cycle as negative weight loops (lines 14-18). This algorithm is considered to implement dynamic programming.

\subsubsection{Efficiency}
\begin{itemize}
    \item \textbf{Temporal Complexity} - $ O(|N| \times |E|) $, where E is the set of edges and N the set of nodes of the graph
    \item \textbf{Spatial Complexity} - Depends on the implementation, but in this case $ O(1) $
\end{itemize}

\subsubsection{Usability}
This algorithm poses little interest to this problem given that our edges' weights represent the length of a road, which would never be negative.

\subsection{Bidirectional Dijkstra}

\subsubsection{Description}
Ira Pohl was the first one to design and implement a bidirectional heuristic search algorithm in 1971.
Bidirectional Dijkstra's that implements the idea of a bidirectional search into Dijkstra's algorithm. It finds a shortest path from an initial node to a goal node in a directed or undirected graph. It runs two simultaneous searches: one forward from the initial state, and one backward from the goal, alternating between the two. The stopping criteria is one that must be very well implemented to guarantee the shortest path. Compared to normal Dijkstra's there's a speedup of 2x.

\subsubsection{Extra Data, Data Structures and Algorithms Required}
\begin{itemize}
    \item Minimum distances to the origin and goal nodes from each of the graph's nodes (dist)
    \item Nodes that come before each node in both forward and backward paths (not implemented in pseudocode)
    \item Two priority queues to hold the nodes to be processed next in each direction
    \item Way of search (way) (1 if it was encountered by forward search first, 2 if it was encountered by backward search first, 0 default)
    \item Decrease-Key function, to maintain the nodes with shortest distance in the top of the queue
\end{itemize}

\input{Algorithms/Bidirectional.tex}
\uline{Note:} The pseudocode is not complete, missing the parts to calculate the actual path and to calculate the transposed graph (needed for the backward search because this is a directed graph). These parts were excluded in order to not make the algorithm's pseudocode even more complex and spacious.

\subsubsection{Analysis}
This algorithm basically performs two Dijkstra's algorithms: one starting from the initial node and going forward(lines 16-23); another one starting from the ending node and going backward (lines 24-31). Each time they cross, the distance is registered (lines 25-26 for example). The stopping criteria is a tricky problem in this case. In many situations, the algorithm ends when the two sets collide (forward search and backward search). We found that this criteria did not guarantee the shortest path. Instead, we chose to end the algorithm when there can be no longer any better distances because the nodes being processed are already too far apart.
%\begin{itemize}
%    \item \textbf{Loop invariant} - $ dist(n) + distR(v) > bestDist $, where best distance is the best Distance for a complete path obtained yet, dist(n) is the distance of the node being processed in forward way to the starting node and $ distR(v) $ the distance of the node being processed in backward way to the ending node
%    \item \textbf{Loop variant} - the number of nodes to be processed in each way, since the algorithm would stop when each search had got no more nodes to process on the queue
%\end{itemize}

\subsubsection{Efficiency}
\begin{itemize}
    \item \textbf{Temporal Complexity} - $ O((|N|+|E|) \times \log(|N|) $, where E is the set of edges and N the set of nodes of the graph.
    \item \textbf{Spatial Complexity} - Depends on the implementation, but usually $ O(|N|) $
\end{itemize}
Even though there's a speed up, the complexities will be the same as in Dijkstra's, since the difference is multiplication and division by constants, which is ignored in Big O order.

\subsubsection{Usability}
Bidirectional Dijkstra's algorithm is one of the fastest algorithms when it comes shortest paths. It has the down-side of requiring the previous knowledge of the ending/destination node, but in this scenario that is not a problem.

\subsection{A* Algorithms}
\subsubsection{Description and Analysis}
The first time a A* Algorithm came up was in \textit{Shakey Project}, project of the first general purpose mobile robot capable of reasoning about is own actions. In an A* (pronounced A-star) Search Algorithm, the 'level' of a node in a graph becomes a sum of two different characteristics. For instance, applied to this problem, the distance a node is from the origin in the search algorithm is substituted for the sum of:
\begin{itemize}
    \item $d_{sv}$ - actual distance to the origin
    \item $\pi_{nt}$ - estimated guess of the distance to the goal node (heuristic)
\end{itemize}
The reliability of this technique depends on the efficacy of the heuristic. In this case, because the weight of edges is distance in meters, the efficacy is great if we assume $\pi$ to be the euclidean distance. This algorithm can be implemented into both Bidirectional and normal Dijkstra's.
\begin{itemize}
    \item \textbf{Upside} - Speeds up the process immensely
    \item \textbf{Downside} - The algorithm does not absolutely guarantee the best solution.
\end{itemize}

\subsubsection{Usability}
This is a technique that shows to be very promising and a good idea to implement in the solution of this problem.

\newpage
\section{Shortest Path Between All Nodes}
This section is dedicated to algorithms that calculate the shortest path between all pairs of points. Such algorithms may be used in the solution, specially in the pre-processing part.

\subsection{Floyd-Warshall}

\subsubsection{Description}
This algorithm was published by Robert Floyd in 1962, yet it was virtually the same as some algorithms published by Stephen Warshall and Bernard Roy, 1962 and 1959 respectively. It takes an adjacency matrix that represents the graph as an input (empty in the beginning) and calculates the shortest paths between all nodes. The value of a path between to vertices is the sum of all edge's weights through that path. It is registered in the matrix as such: distance of shortest path between 1 and 4 would be matrix[1][4]. Edges may have negative values but the graph can't have a negative cycle.
% For each pair of vertices, the shortest path is calculated. Time complexity is O \begin{math}(|V| ^3).\end{math}, and space complexity is O \begin{math}(|V| ^2).\end{math}

\subsubsection{Extra Data, Data Structures and Algorithms Required}
\begin{itemize}
    \item $ |N| \times |N| $ matrix (another one if paths are needed), where $ |N| $ is the number of vertices
\end{itemize}

\input{Algorithms/Floyd-Warshall.tex}
\uline{Note:} The pseudocode is not complete, missing the matrix to calculate the paths and respective instructions. This was done in order to simplify the algorithm and minimize it.

\subsubsection{Analysis}
First part is basically the preparation of the matrix: fill it with $\infty$ distances; for the edges existant (lines 1-3), change the values in the matrix to their weight (lines 4-6). Then, all pairs of nodes will be checked on if there is any node the path can pass through that would minimize its cost (lines 7-14).

\subsubsection{Efficiency}
\begin{itemize}
    \item \textbf{Temporal Complexity} - $ O(|N|^3) $, where N is the set of nodes of the graph.
    \item \textbf{Spatial Complexity} - Depends on the implementation, but usually $ O(|N|^2) $, for the matrix of size $|N| \times |N|$
\end{itemize}

\subsubsection{Usability}
This algorithm is an interesting algorithm to be used in the beginning of the program, to leave all shortest paths processed. However, this might take a long time, and it might be a better idea to simply calculate the shortest path each time it is needed.

\subsection{Dijkstra's Algorithm}

\subsubsection{Description and Analysis}
Dijkstra's can also be used to calculate all shortest paths. This is done by simply executing the algorithm for every node in the graph as origin.

\subsubsection{Efficiency}
\begin{itemize}
    \item \textbf{Temporal Complexity} - $ O(|N| \times (|N|+|E|) \times \log(|N|) $, where E is the set of edges and N the set of nodes of the graph. Same as Dijkstra's but multiplied by $|N|$
    \item \textbf{Spatial Complexity} - Depends on the implementation, but usually $ O(|N|) $, for the priority queue, same as Dijkstra's
\end{itemize}

\subsubsection{Usability}
This is a better option for less dense graphs compared to Floyd-Warshall. As this is not the case for a road network, this algorithm is relatively irrelevant.

\newpage
\section{Connectivity}
The algorithms in this section check the connectivity of the graph. By other words, this section is dedicated to algorithms that detect if any nodes are inaccessible to others, dividing the graph in strongly connected components. A graph will be strongly connected if there is a path between all pairs of points, or if the graph is a strongly connected component itself. Algorithms to test connectivity may be useful to solve the problem of unpredictability.

\subsection{Brute Force with Floyd-Warshall}

\subsubsection{Description and Analysis}
One way to test the connectivity of a graph is by analyzing the results of Floyd-Warhsall in a certain graph. If there are any pairs of nodes which the distance remains $\infty$, they do not belong to the same strongly connected component.

\subsubsection{Efficiency}
\begin{itemize}
    \item \textbf{Temporal Complexity} - $ O(|N|^3) $, where N is the set of nodes of the graph, same as Floyd-Warshall
    \item \textbf{Spatial Complexity} - Depends on the implementation, but usually $ O(|N|^2) $, for the matrix of size $|N| \times |N|$
\end{itemize}

\subsubsection{Usability}
This algorithm works yet is not a good option. Even if the Floyd-Warshall is going to be utilized anyway, the complexity of traversing the whole matrix is $O(|N|^2)$. Therefore, there are much better algorithms for the job.

\subsection{Kosaraju's algorithm}

\subsubsection{Description}
The algorithm was first published in 1981 by Micha Sharir but gets is name from whom first mencioned it in 1971, Sambasiva Rao Kosaraju. This algorithm allows to identify strongly connected components in a directed graph, testing its connectivity.

\subsubsection{Extra Data, Data Structures and Algorithms Required}
\begin{itemize}
    \item boolean identifying if a node has been visited or not (visited)
    \item identifier of the strong component the node belongs to (strongComponent)
    \item function to invert the edges of the graph
    \item a queue to hold the order of the visits
\end{itemize}

\input{Algorithms/Kosaraju.tex}

\subsubsection{Analysis}
The algorithm performs two depth-first searches in the graph. In the first one, it marks the nodes by visit order (lines 5-9). The second one starts on the nodes with lowest order and is performed on the graph with the edges inverted (lines 10-14). This second one will generate multiple expansion trees (or just one if the graph is strongly connected), each being a strongly connected component. If two nodes are not in the same strongly connected component, one of them cannot be reached from the other.

\subsubsection{Efficiency}
\begin{itemize}
    \item \textbf{Temporal Complexity} - $ O(|N|+|E|) $, where N is the set of nodes of the graph and E the set of edges. Each dfs is $O(|N|+|E|)$
    \item \textbf{Spatial Complexity} - Depends on the implementation, but in this one it is $ O(|N|) $, being the size of the queue $|N|$
\end{itemize}

\subsubsection{Usability}
This algorithm is a serious candidate to check the connectivity of the graph.

\subsection{Tarjan's Algorithm}

\subsubsection{Description}
The Tarjan's Algorithm was named after its inventor, Robert Tarjan, and is another algorithm capable of identifying the strongly connected components in a given graph. It does so, similarly to Kosaraju's, with the help of dfs, only this time the graph is only traversed once.

\subsubsection{Extra Data, Data Structures and Algorithms Required}
\begin{itemize}
    \item boolean identifying if a node has been visited or not (visited)
    \item integer identifying the counter when a node is first found (num)
    \item integer identifying the lowest num reachable by a node (low)
\end{itemize}

\input{Algorithms/Tarjan.tex}

\subsubsection{Analysis}
The algorithm is fairly simple: the graph is traversed by a depth first search (lines 5-6); the $num$ and $low$ of a node are set to the $numCounter$(lines 12-13); each time an already visited node is found, all nodes with $low > that nodes low$ will be changed (lines 16-18). This way, all vertices with the same $low$ belong to the same strongly connected component.

\subsubsection{Efficiency}
\begin{itemize}
    \item \textbf{Temporal Complexity} - $ O(|N|+|E|) $, where N is the set of nodes of the graph and E the set of edges. The dfs is $O(|N|+|E|)$. This is the same complexity as Kosaraju's, but the algorithm is twice as fast since it only performs one dfs.
    \item \textbf{Spatial Complexity} - Depends on the implementation, but in this one it is $ O(1) $.
\end{itemize}

\subsubsection{Usability}
This algorithm is an even better candidate than the last one, since it is supposedly twice as fast.

\newpage
\section{Clustering Algorithms}

A Clustering is a machine learning task which involves the recognition of natural grouping in data. Such algorithms can be used in the distribution of orders per vehicle.

\subsection{Agglomerative Hierarchical Clustering}

\subsubsection{Description}

This algorithm is one of the most common Hierarchical Clustering algorithms. It is used to form groups of objects of data based on their similarity. In this case, we will use it to form N groups of data nodes with great similarity.

\input{Algorithms/Clustering.tex}
\uline{Note:} the implementation presented above is just a rough sketch, the one to be implemented will be different.

\subsubsection{Analysis}
The algorithm begins to find the pair of points which are most similar. In this case, similarity is measured by distance (lines 6-14). After he has found one, he merges the two sets (. In between sets, there are many criteria to calculate similarity. The one we will implement is taking the similarity between two sets as the similarity of the most similar pair of points where one belong to the first set and other to the second set. This algorithm adopts a Bottom-Up strategy (as do all agglomerative hierarchical clustering algorithms), as it start by processing each node and joining them to obtain the final result.

\subsubsection{Efficiency}
\begin{itemize}
    \item \textbf{Temporal Complexity} - $ O(|S|^2*(|S| - n)) $, where S is the set of points. This is only an estimated guess.
    \item \textbf{Spatial Complexity} - Depends on the implementation, but in this one it is $ O(|S|) $.
\end{itemize}

\subsubsection{Usability}
This is the only algorithm analysed in this cathegory, thus it is the chosen one to perform the task.

\section{Extras}
There are other algorithms, simpler or less relevant, that may (or may not) be used in our solution:
\begin{itemize}
    \item Breadth-First Search (bfs) - Graph searching algorithm
    \item Depth-First Search (dfs) - Graph searching algorithm
    \item Decrease-Key Function - Priority changing function on a priority queue
    \item Graph Transposing Function - Inversion of the edges in a directed graph
    \item Fibonacci Heaps (Data structure) - Improved priority queue
    \item Euclidian Distance function - To calculate the euclidian (straight-line) distance between two points
\end{itemize}


\subsection{Node-tree manipulation and the decoder time complexity}
In the following we detail the routines to grow and merge node-trees. These completes the description of the decoder as it is analogous to the original Union-Find algorithm with the addition of these two steps.

The original Union-Find algorithm had a complexity driven by the time complexity of merging the union-find data structures representing the cluster-trees. This is known to be $\mathcal{O}(n\alpha(n)$, where $\alpha(n)$ is the inverse Ackermann function, a sublogarithmic growing function.

The worst-case time complexity of the Union-Find Partitioned-Growth decoder of \Cref{algo:ufbb} follows directly from the complexity of the growth and merge of the node trees. They are still quasilinear. In particular, in sections \ref{} and \ref{}, we describe respectively the routines for the growth and the merge of the node-trees and show that they have an amortized time-complexity $\mathcal{O}(n\log n)$. In consequence, the complexity of the total algorithm is driven by these two steps and the amortized time-complexity is $\mathcal{O}(n\log n)$.

\subsubsection{Growing node-trees}\label{sec:grownodetrees}

The Union-Find decoder grows clusters by adding half-edges to all boundary vertices of the cluster. Instead, here we propose to grow only the nodes with minimal $PMW$. For this, it is necessary to save the list of boundary vertices to each node (\Cref{fig:nodetypes}c). In this algorithm, to \emph{grow a node} means to grow all edges connected to its boundary vertices a half-edge per the Union-Find decoder. The proposed growth algorithm consists of 3 depth-first-searches starting from a reference node $R$. Since the minimal PMW can be found without the PWM of the reference node $R$ itself, any node $N\in\nset$ may be chosen as the reference node $R$.

\begin{algorithm}[h]
  \BlankLine
  \KwData{Node tree $\nset$}
  \KwResult{$\nset$}
  \BlankLine

  Choose a reference node $R \in \nset$\;
  Calculate node parities via tail-recursive DFS in $\nset$ from $R$ via \Cref{eq:snodeparity,eq:jnodeparity}\;
  Calculate PMW with respect to $R$ via head-recursive DFS in $\nset$ from $R$ via \Cref{eq:pmwdif}\;
  Grow all nodes with minimal PMW in a DFS in $\nset$ from $R$\;

  \caption{Grow node-tree (naive)}\label{algo:grow}
\end{algorithm}

When two clusters merge on vertices $a$ and $b$, their respective node-trees must also be merged. This is performed by creating an edge between nodes to which vertices $a$ and $b$ belong to.

\textcolor{cyan}{
  It is possible to reduce the cost of \Cref{algo:grow} by two implementations. First, the DFS's need not be performed during every growth iteration. For this, we introduce the \emph{node delay} $\delta_\nset$, the difference in PMW with the minimal PMW in the node-tree.
}

\begin{equation}\label{eq:nodedelay}
  \delta_N = \text{PWM}(N) - \min_{N\in\nset}{\text{PWM}(N) }
\end{equation}

\textcolor{cyan}{
  With the node delay, the first two DFS's for the calculation of the parity and PMW need only to be performed during the first growth iteration after a merger, during which we now additionally sort the nodes with boundary vertices in the node-tree by the node delay. In the first growth iteration after a merger, we iterate over the sorted nodes and grow all nodes with delay 0, and add 1 to the delay, until a node with delay 1 is encountered. In the second iteration, we grow all nodes with delay 1, and so on. The number of operations for iterating through the sorted nodes is bounded by the number of growth operations of boundary edges, and its contribution to the complexity is thus defined within the original Union-Find decoder. The additional cost of growing a node-tree $\nset$ while not merging is thus $\abs{\nset}$.
}

\textcolor{cyan}{
  A second reduction in the cost is possible by a selection procedure for the reference node $R_{\text{merged}}$ in the merged node-tree $\nset_{\text{merged}}$. Since $\nset_{\text{merged}}$ must be odd to grow, the node-trees involved its merger must consist of at least one odd node-tree $\nset_O$ and a number of other node-trees. If the reference node $R_O$ of $\nset_O$ is chosen for $R_{\text{merged}}$ after the merger, the parity and delays within the sub-tree of nodes in $R_O$ do not change during the merger. Thus, the DFS's can be limited all other node-trees but $\nset_O$. The greatest reduction in cost is achieved if the reference node of the largest odd node-tree in the merger is chosen as the new reference node.
}

After calculating the node delay, each subsequent growth iteration is performed by a single DFS. In this growth DFS, each node with zero delay is grown, and the delay of other nodes are subtracted by $\nicefrac{1}{2}$. This procedure is effectively equivalent to growing the nodes with minimal PMW. If a merge with another node-tree occurs, the PMW's within the merged node-tree change, and a recalculation of the node parities and reference PMW's is required. The reduced cost algorithm is described in \Cref{algo:growdelay}.

\begin{algorithm}[h]
  \BlankLine
  \KwData{Node tree $\nset$, growth iteration $i$ after merge}
  \KwResult{$\nset$}
  \BlankLine

  \If(){$i=0$}{
    Choose a reference node $R \in \nset$.\;
    Calculate node parities via tail-recursive DFS in $\nset$ from $R$ via \Cref{eq:snodeparity,eq:jnodeparity}.\;
    Calculate PMW with respect to $R$ via head-recursive DFS in $\nset$ from $R$ via \Cref{eq:pmwdif}.\;
    Calculate node delay $\delta_N$ in $\nset$ via \Cref{eq:nodedelay}.\;
    Sort nodes with boundary edges by their node delay in a list $\mathcal{L}$.\;
  }
  \For(){node $N$ in $\mathcal{L}$}{
    \eIf(){$\delta_N =i$}{
      Grow boundary edges of $N$. Add 1 to $\delta_N$.
    }(){
      Break loop.
    }
  }
  \caption{Grow node-tree (low-cost)}\label{algo:growdelay}
\end{algorithm}

Note that while the cost is reduced in \Cref{algo:growdelay}, the complexity of the growth procedure is identical to \Cref{algo:grow}, which is proportional to the size of the node-tree due to the growth DFS. However, \Cref{algo:growdelay} allows us to solve a problem described in the next section.

% An addition reduction in cost is possible by retaining the same root node through several mergers, such that the delay and PMW DFS's need only to be performed on a partition of the node-tree. However, these optimizations through implementation do not reduce the overall complexity of the algorithm, and will thus not be covered in this article.

% Equations \Cref{eq:pmwdif,eq:pseudodelay,eq:snodeparity,eq:jnodeparity} show that the differences in PMW with respect to the root node can be calculated through two depth-first searches (DFS) of $\nset$. In the first DFS, the \emph{parity DFS}, we calculate the node parity $\pi_N$ a via tail-recursive function using \Cref{eq:snodeparity,eq:jnodeparity}. In the second DFS, the \emph{PMW DFS}, we calculate the PMW differences $\delta_N$ via a head-recursive function using \Cref{eq:pmwdif,eq:pseudodelay}.

% \todo[inline]{MOVE? or expand, growth DFS is not yet introduced}During this second DFS, the minimal PMW difference within $\nset$ can be kept track of by comparison for calculating the node delay $\delta_N$ at a later instance during the \emph{growth DFS}.

% The Union-Find decoder grows clusters by adding half-edges to all boundary vertices of the cluster. Instead, here we propose to grow only the nodes with minimal $PMW$.
%A single growth iteration, which is applied in the Union-Find decoder by adding half-edges to all boundary vertices of the cluster, is now replaced by another DFS of $\nset$ from the same root $R$ as before.
%During this DFS, $\delta_N$ of every node $N$ is calculated via \Cref{eq:delaypseudo} and $N$ is conditionally grown --- adding half-edges to the boundary vertices in the current node and adding $\nicefrac{1}{2}$ to its radius $\rho_N$ --- if $\delta_N = 0$.
% For this, it is necessary to save the list of boundary vertices to each node (\Cref{fig:nodetypes}c). %When the delays $\delta_N$ for all nodes in $\nset$ are zero, all nodes are grown simultaneously within the same iteration.

% As the nodes grow, the values for $\delta_N$ across $\nset$ change. %This would mean that the DFS's of node parity and delay need to be done after each growth iteration.
% Fortunately, if no new nodes are added to $\nset$ after a growth iteration, the parities %$\pi_N$ on iteration $t+\theta$
% can be calculated via previous values of the PMW differences. %pseudo-delays in iteration $t$ by the introduction of the node \textbf{wait} parameter $\omega_N$.
% During the growth DFS, if $\delta_N \neq 0$, add 1 to $\omega_N$. If no nodes are added to $\nset$, the node delay in the next iteration are
% \begin{multline}\label{eq:delay}
%     \delta_N(t+\theta) = \tilde{\delta}_N(t) - \min_{X \in \nset(t)}{\tilde{\delta}_X} - \omega_N(t+\theta-1),  \\
%     \hspace{.3cm} N \in \nset, \hspace{.3cm} \nset(t+\theta) = \nset(t).
% \end{multline}

% Note that we have not stated which node in $\nset$ should be the root node. In fact, any node in $\nset$ could have been picked as the root of the node-tree. The only requirement is that the DFS of cluster growth must be performed along the same direction as the DFS's of the parity and delay calculations. If no cluster mergers occur, the node delays can be reused in the node suspension calculation prior to node growth. The node-tree is constructed by storing all neighbors of a node to a list. This way, the DFS's' direction can be determined by simply saving the root node, the starting point of the DFS's, to the cluster. All node variables are depicted in \Cref{fig:nodetypes}c.
% In the next section, we expand upon this idea of "reusing" some intermediate parameters to calculate the node suspensions after a cluster merger.

% \subsection{Joining node-trees}\label{sec:nodejoin}

% \Figure[b](topskip=0pt, botskip=0pt, midskip=0pt){figures/tikz/build/main-figure5.pdf}{
%     \emph{(a)} An odd cluster $\nset_o=\{A, B, O\}$ with root $A$ joins with an even cluster $\nset_e=\{C, E\}$ with root $C$ on nodes $O, E$, respectively, to a joined node-tree. If we choose to \emph{(b)}, make $E$ a child of $O$, the parities and delays the subtree of $\nset_o$ can unchanged, and we only have to perform partial parity and delay calculations over the subtree of $\nset_e$. If we choose to \emph{(c)}, make $O$ a child of $E$, parities and delays have to be recalculated in the entire joined node-tree. \label{fig:inversion}}

\Figure[tb](topskip=0pt, botskip=0pt, midskip=0pt){figures/tikz/build/main-figure4.pdf}{
    The relevant data structures. \emph{(a)} The cluster-tree of the Union-Find data structure, whose elements are vertices $v$. The path from a vertex to the root of the cluster-tree is traversed to find the root element, which differentiates clusters. Next to the cluster size and parity, the root node $R$ of the node-tree $\nset$ is stored at the root of the cluster-tree. Each vertex $v$ additionally stores to which node $N$ it belongs. \emph{(b)} A spanning-tree $\vset$ with 3 non-trivial vertices. As $\vset$ is strictly acyclic, the cluster's edges must be maintained such that no cycles are created. This is done during growth by removing edges (red dotted lines) if a cycle is detected. \emph{(c)} The node-tree $\nset$, which consists of the syndrome-nodes $\dot{A}, \dot{B}, \dot{C}$ with primer vertices $a, b, c$, and junction-nodes $\bar{D}, \bar{I}$ with roots $d,i$. At each node $N$ and the root $R=\dot{B}$ various variables are stored.\label{fig:nodetypes}}

% As clusters grow in size they merge with other clusters.
% %In the Union-Find (UF) algorithm, odd clusters
% %parity clusters of an odd number of non-trivial vertices, which are elements of $\sigma$,
% %grow %in size
% %repeatedly and merge with other clusters until all clusters are even.
% %The data structures of the merged cluster can be obtained by merging the data structures of the original clusters: %During these mergers, the node-trees and the cluster-trees are merged. %of the Partitioned-Growth data structure are combined.
% %Let us first make a  distinction between the merging protocols of the underlying data structures;
% %the cluster-trees are merged with the $\Union$ function, whereas
% In the following we describe how to obtain the node-tree of the merged cluster from the original node-trees. % are merged with the $\Nodejoin$ function which we describe in the following. %After a join of multiple node-trees, the node delays within the combined node-tree change. Therefore, the $\Nodejoin$ protocol's focus is to minimize the DFS's of the recalculation of the node parity and delays in the combined node-tree.

% %Let the parity $\nset$ be the number of syndrome-nodes in $\nset$, which is equivalent to the parity of the cluster of $\nset$.

% Since only odd clusters grow, only two types of clusters can merge: even with odd and odd with odd.

% For even node-trees, the PMW is undefined since a perfect matching would not be possible when merging with a non-trivial vertex. Consequently, PMW differences are undefined for even node-trees and do not need to be calculated.

% The second type of merger is between an even and an odd cluster. %The combined cluster is odd, and its growth is continued. Thus, its node delays must be computed.
% In order to define the new node-tree, we need to choose a new root. We consider two different options, choosing the root of the odd cluster and choosing the root of the even cluster:

% Let $\nset_o,\nset_e$ be the odd and even node-trees that are to be joined, and let $O\in \nset_o,E \in \nset_e$ be the nodes connected (\Cref{fig:inversion}\emph{a}).
% If $\nset_o$'s root becomes the root of the joined node-tree (\Cref{fig:inversion}\emph{b}), $E$ becomes a child node of $O$.
% Since $\nset_e$ contains an even number of syndrome-nodes, the node parities in $\nset_o$ do not change.
% Hence, the node parity DFS is only necessary in the subtree $\nset_e$, which now has $E$ as subroot potentially different than the original root.
% Furthermore, as the node PMW difference depends only on its own properties and its parent's, the node PMW DFS is also only required from node $E$ and within the subtree of $\nset_e$.
% Hence, only partial DFS's within the even cluster are required. % of the node-tree are precisely what was required, as the node parity and delays in $\nset_e$ were undefined.

% Alternatively, if $\nset_e$'s root becomes the root of the combined tree (\Cref{fig:inversion}\emph{c}), an odd number of syndrome-nodes are attached to $E$. In consequence the parities of the nodes on the path from $E$ to the root are changed. This choice of root requires the DFS's on the entire combined node-tree to calculate for node parities and PMW differences.

% For this reason, when merging node-trees we keep the root of the odd node-tree. We dub this strategy \textbf{Odd-Rooted Join}.

% In addition, a cluster can be subjected to multiple mergers within the same growth iteration, during which the parity of the merged cluster changes depending on the number of mergers and the parities of the clusters involved. The DFS's related to the parities and PMW differences should not be calculated directly after the joining of node-trees. %After all, it may be possible for the cluster to merge again such that the parities and delays become invalid.
% \textcolor{red}{To prevent redundant calculations, subroots of the even subtrees are stored in a list at the root of the node-tree (\Cref{fig:nodetypes}\emph{c}). When multiple mergers occur, the root node that stores the now redundant subroots is replaced by a new root with a new list of even subtrees subroots. If a cluster is selected for growth, we loop over the list of the subroots in the list at the new root node and initiate the DFS's from these subroots.} We call this the \textbf{Root List Replacement}.


%!TEX root=report.tex
\subsection{Time Series Analysis}

In time series analysis one attempts to find a model for predicting future observations
(this is different form the OLS analysis which tries to estimate the trends in the data, such as velocity and acceleration).
 Once such a model is found, it should be validated by analyzing the residuals. These residuals must be approximately white noise distributed \cite[p.~130]{time-series-analysis}, otherwise the model is invalid.

\subsubsection{ARIMA}

ARIMA is the name of the stochastic model used. The theory is very complex so it will not be discussed in detail here, instead we refer to \cite[p.~130]{time-series-analysis}.

In short ARIMA uses the previous values with some weight to predict future values. Which values are used is denoted by the following notation:
\begin{equation}
(p, d, q) \times (P, D, Q)_s
\end{equation}

$p$ is the highest lags of actual measurements used and $q$ is the maximum amount of lags in residuals used. The $d$ part indicates how many difference operators which should be used to transform the data. $(P, D, Q)$ is completely similar, but steps not by one lag but by $s$ lags ($s=$season length). This allows for seasons trends such as an yearly pattern.

\subsubsection{AutoCorrelation function (ACF) and Partial AutoCorrelation Function (PACF)}

ACF and PACF are measures of correlation between different time lags and are used to make a qualified guess about how the prediction model should look like.
For more detail on how these are estimatied, please see \cite[p.~146]{time-series-analysis}.

When ACF and PACF have been estimated, rules \cite[table~6.1]{time-series-analysis} for how the stochastic model should look can be applied.
Typically when dealing with complex models, this becomes an iterative and subjective process - different analysts might reach different models.
The iterative step consist of estemating parameters, then calculate ACF and PACF, interpet them, estimate parameters, ... , and so it continues until the ACF and PACF suggest that no more lag terms(weights) are needed.

\subsection{Ljung-Box test}

The null hypothesis for the Ljung-Box is ``The data is independently distributed''.
In the ARIMA case it is applied to model residuals; so in other words it tests whether residuals at different lags have any correlation with each other.
 Thus a low p-value means that the residuals aren't white noise.

The Ljung-Box is a $\chi^2$ test with the statistical value:
\begin{equation}
L = n \cdot (n + 2) \sum_{k=1}^h \frac{\hat{\rho}_k^2}{n - k}
\end{equation}

$n$ is the number of observations. The parameter $h$ is the highest lag in the ACF which should be considered. The $\chi^2$ statistics, have $h - (p + q + P + Q)$ degrees of freedom.


\documentclass[10pt,english]{article}

%\usepackage[T1]{fontenc}
%\usepackage[utf8]{inputenc}
\usepackage[a4paper,left=0.4in, right=0.4in,top=0.5in,bottom=0.6in]{geometry}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{subcaption}
\usepackage{epstopdf}
\usepackage{graphicx}
\usepackage{mathrsfs}
\usepackage{epstopdf}
\usepackage{mathtools}
\usepackage{textcomp}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{epstopdf}
%\usepackage{multicol}
%\usepackage{tikz}
\usepackage{hyperref}

% code listing settings
\usepackage{listings}
\lstset{
    language=Python,
    basicstyle=\ttfamily\small,
    aboveskip={1.0\baselineskip},
    belowskip={1.0\baselineskip},
    columns=fixed,
    extendedchars=true,
    breaklines=true,
    tabsize=4,
    prebreak=\raisebox{0ex}[0ex][0ex]{\ensuremath{\hookleftarrow}},
    frame=lines,
    showtabs=false,
    showspaces=false,
    showstringspaces=false,
    keywordstyle=\color[rgb]{0.627,0.126,0.941},
    commentstyle=\color[rgb]{0.133,0.545,0.133},
    stringstyle=\color[rgb]{01,0,0},
    numbers=left,
    numberstyle=\small,
    stepnumber=1,
    numbersep=10pt,
    captionpos=t,
    escapeinside={\%*}{*)}
}
\author{Steven Porretta \\ Student \# 100756494}
\title{Assignment \#4}
\date{}
\begin{document}
    \maketitle
    Code can be found at \url{https://github.com/0xSteve/learning_automata_simulator}
    \section{Question 1}
        In this section we will examine some code snippets from the first question.
    \begin{lstlisting}[label={list:first},caption=Testbench code for the Tsetlin.]
    c2 = 0.7
    c1 = 0.05
    for i in range(0, 7):
        print("c1 = " + str(c1) + ", c2 = " + str(c2) + ", N = 13.")
        a = la.Tsetlin(13, 2, [c1, c2])
        a.simulate(50, 30001)
        b = ala.Tsetlin.stationary_probability_analytic([c1, c2], 13)
        c = ala.Tsetlin.number_of_states_estimate([c1, c2])
        print("Tsetlin P1(infinity) = " + str(b) + "(Analytic)")
        print("Tsetlin P1(infinity) = " + str(a.action_average[0]) + "(Simulated)")
        print("Tsetlin # of states required = " + str(c) + "(Estimate)")
        c1 += 0.1
        c1 = round(c1, 2)
\end{lstlisting}
This excerpt of code generates the entire quantity of required code for this question.  As can be seen in the following code-snippet.

\begin{lstlisting}[label={list:first},caption=Testbench output.]
c1 = 0.05, c2 = 0.7, N = 13.
Tsetlin P1(infinity) = 0.9999999890725503(Analytic)
Tsetlin P1(infinity) = 1.0(Simulated)
Tsetlin # of states required = 3(Estimate)
c1 = 0.15, c2 = 0.7, N = 13.
Tsetlin P1(infinity) = 0.9999778874501014(Analytic)
Tsetlin P1(infinity) = 1.0(Simulated)
Tsetlin # of states required = 4(Estimate)
c1 = 0.25, c2 = 0.7, N = 13.
Tsetlin P1(infinity) = 0.9990142374252533(Analytic)
Tsetlin P1(infinity) = 0.999998000067(Simulated)
Tsetlin # of states required = 6(Estimate)
c1 = 0.35, c2 = 0.7, N = 13.
Tsetlin P1(infinity) = 0.9865794150962881(Analytic)
Tsetlin P1(infinity) = 0.999797340089(Simulated)
Tsetlin # of states required = 9(Estimate)
c1 = 0.45, c2 = 0.7, N = 13.
Tsetlin P1(infinity) = 0.9151468144874294(Analytic)
Tsetlin P1(infinity) = 0.980783973868(Simulated)
Tsetlin # of states required = 18(Estimate)
c1 = 0.55, c2 = 0.7, N = 13.
Tsetlin P1(infinity) = 0.7453193640776348(Analytic)
Tsetlin P1(infinity) = 0.786514449518(Simulated)
Tsetlin # of states required = 0(Estimate)
c1 = 0.65, c2 = 0.7, N = 13.
Tsetlin P1(infinity) = 0.5680008401435711(Analytic)
Tsetlin P1(infinity) = 0.570874970834(Simulated)
Tsetlin # of states required = 0(Estimate)
\end{lstlisting}
Now that it is seen working as one would expect, considering rounding errors from python 3.6, it is time to take a look at the useful snippets of code governing the functionality of the Tsetlin, and Krylov automata.

\begin{lstlisting}[label={list:first},caption=Tsetlin core code.]
     def next_state_on_reward(self):
        '''Find the next state of the learner, given that the teacher
           rewarded.'''
        if (self.current_state mod (self.N / self.R) != 1):
            self.current_state -= 1

    def next_state_on_penalty(self):
        '''Find the next state of the learner, given that the teacher
           penalized.'''
        if(self.current_state mod (self.N / self.R) != 0):
            self.current_state += 1
        elif(self.current_state mod (self.N / self.R) == 0):
            # Don't really add states, just cycle through N, 2N, 4N, etc.
            if(self.current_state != self.N):
                a = (self.N / self.R) mod self.N
                self.current_state = a + self.current_state
            else:
                self.current_state = self.N / self.R

    # Determine the next state as the teacher.
    def environment_response(self):
        '''Determine the next state of the learner from the perspective
        of the teacher.'''
        response = uniform(0, 1)
        penalty_index = 1
        if(self.current_state <= self.n):
            self.actions[0] += 1
            penalty_index = 0
        else:
            self.actions[1] += 1

        if(response > self.c[penalty_index]):
            # Reward.
            self.next_state_on_reward()
        else:
            # Penalty.
            self.next_state_on_penalty()
\end{lstlisting}
The above is the core code of the Tsetlin machine, governing state translations and action choices.  Essentially, whenever it is in the range 1 to N, it chooses action $\alpha_1$ and $\alpha_2$ otherwise.  This code is essentially the same for the Krylov machine, which we will see in the next section.

\section{Question 2}
\begin{lstlisting}[label={list:first},caption=testbench for the Krylov 2-action.]
    for i in range(0, 7):
    print("c1 = " + str(c1) + ", c2 = " + str(c2) + ", N = 13.")
    a = la.Tsetlin(13, 2, [c1/2, c2/2])
    a.simulate(50, 30001)
    b = ala.Tsetlin.stationary_probability_analytic([c1, c2], 13)
    c = ala.Tsetlin.number_of_states_estimate([c1, c2])
    d = la.Krylov(13, 2, [c1, c2])
    d.simulate(10, 50000)
    e = ala.Tsetlin.stationary_probability_analytic([c1, c2], 13)
    f = ala.Tsetlin.number_of_states_estimate([c1, c2])
    print("Tsetlin P1(infinity) = " + str(b) + "(Analytic)")
    print("Tsetlin P1(infinity) = " + str(a.action_average[0]) + "(Simulated)")
    print("Tsetlin # of states required = " + str(c) + "(Estimate)")
    print("Krylov P1(infinity) = " + str(e) + "(Analytic)")
    print("Krylov P1(infinity) = " + str(d.action_average[0]) + "(Simulated)")
    print("Krylov # of states required = " + str(f) + "(Estimate)")
    c1 += 0.1
    c1 = round(c1, 2)
\end{lstlisting}
As can be seen from the code, this test bench looks very similar to the test bench of question 1, however, note the $c$ vector for the Tsetlin automaton is now $c_1 / 2 $, $c_2/2$.  As is expected, both automata behave in the same manner. as can be seen from the output code snippet.
\begin{lstlisting}[label={list:first},caption=testbench output for the Krylov 2-action.]
    c1 = 0.05, c2 = 0.7, N = 13.
    Tsetlin P1(infinity) = 0.9999999890725503(Analytic)
    Tsetlin P1(infinity) = 1.0(Simulated)
    Tsetlin # of states required = 3(Estimate)
    Krylov P1(infinity) = 0.9999999890725503(Analytic)
    Krylov P1(infinity) = 1.0(Simulated)
    Krylov # of states required = 3(Estimate)
    c1 = 0.15, c2 = 0.7, N = 13.
    Tsetlin P1(infinity) = 0.9999778874501014(Analytic)
    Tsetlin P1(infinity) = 0.999999333356(Simulated)
    Tsetlin # of states required = 4(Estimate)
    Krylov P1(infinity) = 0.9999778874501014(Analytic)
    Krylov P1(infinity) = 1.0(Simulated)
    Krylov # of states required = 4(Estimate)
    c1 = 0.25, c2 = 0.7, N = 13.
    Tsetlin P1(infinity) = 0.9990142374252533(Analytic)
    Tsetlin P1(infinity) = 1.0(Simulated)
    Tsetlin # of states required = 6(Estimate)
    Krylov P1(infinity) = 0.9990142374252533(Analytic)
    Krylov P1(infinity) = 1.0(Simulated)
    Krylov # of states required = 6(Estimate)
    c1 = 0.35, c2 = 0.7, N = 13.
    Tsetlin P1(infinity) = 0.9865794150962881(Analytic)
    Tsetlin P1(infinity) = 1.0(Simulated)
    Tsetlin # of states required = 9(Estimate)
    Krylov P1(infinity) = 0.9865794150962881(Analytic)
    Krylov P1(infinity) = 1.0(Simulated)
    Krylov # of states required = 9(Estimate)
    c1 = 0.45, c2 = 0.7, N = 13.
    Tsetlin P1(infinity) = 0.9151468144874294(Analytic)
    Tsetlin P1(infinity) = 1.0(Simulated)
    Tsetlin # of states required = 18(Estimate)
    Krylov P1(infinity) = 0.9151468144874294(Analytic)
    Krylov P1(infinity) = 1.0(Simulated)
    Krylov # of states required = 18(Estimate)
    c1 = 0.55, c2 = 0.7, N = 13.
    Tsetlin P1(infinity) = 0.7453193640776348(Analytic)
    Tsetlin P1(infinity) = 0.983473884204(Simulated)
    Tsetlin # of states required = 0(Estimate)
    Krylov P1(infinity) = 0.7453193640776348(Analytic)
    Krylov P1(infinity) = 0.999998(Simulated)
    Krylov # of states required = 0(Estimate)
    c1 = 0.65, c2 = 0.7, N = 13.
    Tsetlin P1(infinity) = 0.5680008401435711(Analytic)
    Tsetlin P1(infinity) = 0.670734975501(Simulated)
    Tsetlin # of states required = 0(Estimate)
    Krylov P1(infinity) = 0.5680008401435711(Analytic)
    Krylov P1(infinity) = 0.870114(Simulated)
    Krylov # of states required = 0(Estimate)
\end{lstlisting}
Observing the code for the Krylov machine, one notices that most of the code is inherited from the Tsetlin machine, except the state translations. It is incredible, that the only major distinction is that a penalty is treated as a penalty with 50\% probability and a success otherwise. Literally, all other code for the Krylov machine is inherited from the Tsetlin.
\clearpage
\begin{lstlisting}[label={list:first},caption=Krylov core code.]
def next_state_on_penalty(self):
        '''Find the next state of the learner, given that the teacher
           penalized.'''

        # If this number is greater than 0.5, then penalize the learner.
        is_penalty = uniform(0, 1)

        if(is_penalty >= 0.5):
            Tsetlin.next_state_on_penalty(self)
        else:
            Tsetlin.next_state_on_reward(self)

\end{lstlisting}

\section{Question 3}

First let us consider state changes in the $L_{R-I}$, since there really are no states, but instead just an interval, $\lbrace0, 1\rbrace$, of possibilities.  Consider the following code-snippet.

\begin{lstlisting}[label={list:first},caption=State Translation in the $L_{R-I}$  automaton.]
 def do_reward(self, action):
        if(action == 2):
            self.p1 = self.k_r * self.p1
        else:
            self.p1 = 1 - (self.k_r * self.p2)
        self.p2 = 1 - self.p1

    def do_penalty(self):
        pass
\end{lstlisting}
It can be seen from the above code-snippet, that when the environment rewards, an action is updated based on the action selected.  When the environment issues a penalty, nothing happens.  The python command \textit{pass} is command that simply does nothing, and is usually used for prototyping.  In this case, pass is included to explicitly show that a penalty does nothing.

From this automaton some interesting things can be observed.  First, let us consider the output of the testbench.py file.  Time complexity has been measured in terms of discrete steps instead of actual time spent in the processor. The physical time, does not represent the number of actions being computed, as many operations can be processed simulatneously, in the background of a system, the time is not accurate.  A corolary to this event is that the number of discrete calls to the program, accurately represent the time complexity, when each action is considered a time unit.  To better understand this, observe the following code snippet.
\begin{lstlisting}[label={list:first},caption=State Translation in the $L_{R-I}$  automaton.]
=============================================================
The optimal K_r value is: 0.74995
The optimal lambda_r value is: 0.25005
The accuracy for k_r = 0.74995 is: 0.963
The computation time in iterations is: 18
=============================================================
=============================================================
The optimal K_r value is: 0.7811937499999999
The optimal lambda_r value is: 0.2188062500000001
The accuracy for k_r = 0.7811937499999999 is: 0.958
The computation time in iterations is: 26
=============================================================
=============================================================
The optimal K_r value is: 0.7811937499999999
The optimal lambda_r value is: 0.2188062500000001
The accuracy for k_r = 0.7811937499999999 is: 0.965
The computation time in iterations is: 16
=============================================================
=============================================================
The optimal K_r value is: 0.8124374999999999
The optimal lambda_r value is: 0.18756250000000008
The accuracy for k_r = 0.8124374999999999 is: 0.953
The computation time in iterations is: 45
=============================================================
=============================================================
The optimal K_r value is: 0.874925
The optimal lambda_r value is: 0.12507500000000005
The accuracy for k_r = 0.874925 is: 0.976
The computation time in iterations is: 57
=============================================================
=============================================================
The optimal K_r value is: 0.90616875
The optimal lambda_r value is: 0.09383125000000003
The accuracy for k_r = 0.90616875 is: 0.969
The computation time in iterations is: 160
=============================================================
=============================================================
The optimal K_r value is: 0.96865625
The optimal lambda_r value is: 0.031343750000000004
The accuracy for k_r = 0.96865625 is: 0.977
The computation time in iterations is: 1298
=============================================================
\end{lstlisting}
It is observable that as $c1$ approaches $c2$ the number of operations increase exponentially.  It is also observable that as $n \to \infty$, then $\lambda_R \to 0$. Corolary: $c_1 \to c_2$, then $\lambda_R \to 0$ or $k_R \to \infty$.  To get a better understanding of this effect, observe the following graph, Figure \ref{blah}. It can be seen that $\lambda_R$ approaches $0$ as the time complexity goes to infinity.  This is representative of the difficulty of the system.  When the percent difference between penalties of choosing action 1 and action 2 approaches 0, then the time complexity approaches infinity, and the $\lambda_R$ required becomes an undesirably low value.
\begin{figure}[h!]
    \centering
    \includegraphics{plot.png}
    \caption{$\lambda_R$ Vs. $n$} \label{blah}
\end{figure}
\end{document}


As well as in logistic regression as in neural networks, minimizing(or maximizing depending on your setup) the cost function is a central problem, and can to some extent be stated as \textit{the} problem. Having a method which is both efficient and converges towards the minimum is, luckily, not a new problem in computer science. In the logistic regression optimization, three main methods will be utilized, while in the neural network we will focus on the stochastic conjugate gradient descent. For logistic regression we will use a learning rate optimized gradient descent method, and

Gradient descent measures the local gradient of the cost function, with regards to the weights $\bm{w}$(or $\beta$. Since the gradient goes in the direction of fastest increase, we will go in the opposite direction, i.e. negative gradient. We start by choosing random values for $\bm{w}$ (since our cost function is convex any choice should give correct results), calculate the gradient, update the $\bm{w}$ values, and do this iteratively until the algorithm converges to a minimum. The size of the steps is important, and is determined by the learning rate. If the learning rate is too small, we will need many iterations which is time consuming. However, if the learning rate is too high, we might overshoot and miss the minimum. One way to choose the learning rate is too let it depend on the size of the gradient. If the gradient is large, i.e a steep slope, the learning rate can be relatively high. When the gradient is small, the learning rate is also small.

\subsubsection{Gradient descent}
Gradient descent is set up in a general fashion as, utilizing an optimized learning rate $\eta_k$ find and a scaling parameter $\gamma$. The scaling parameter $\gamma$ is introduced in order to prevent results from blowing up.
\begin{algorithm}[H]
    \caption{Gradient descent.}
    \label{alg:gd}
    \begin{algorithmic}[1]
        \State Input: $\bm{X}$, $\bm{y}$, $\bm{w}_0$, $\eta$
        \State Set initial weights, $\bm{w} = \bm{w}_0$
        \State Set previous gradient with current, $\nabla_0 = \nabla_1$
        \While{$i < N_\mathrm{max}$ and $||\nabla\mathcal{C}(\bm{w})||<\varepsilon$}
            \State $z = \bm{X}\cdot \bm{w}$
            \State $p = \sigma(z)$
            \State $\nabla_0 = \nabla_1$, set previous gradient to current.
            \State $\nabla_1 = - \bm{X}^T \cdot (\bm{y} - \bm{p}) / \gamma + \lambda f_L(\bm{w}) / \gamma$
            \State $\bm{w}_0 = \bm{w}$
            \State Update learning parameter, $\eta_k$
            \State $\bm{w} = \bm{w}_0 - \eta_k \nabla_1$, update weights.
        \EndWhile
        \State Return $\bm{w}$
    \end{algorithmic}
\end{algorithm}
The full implementation with the optimized learning rate $\eta_k$ can be seen in the article \citet{optimal-learning-rate}.

\subsubsection{Stochastic gradient descent}
Stochastic gradient descent is similar to gradient descent, expect that we first randomly shuffle our data, then divide our data into mini batches, $N_\mathrm{mb}$. Then we run gradient descent on each of the mini batches, and take the average of the outputted gradients as our gradient descent step. We repeat this $N_\mathrm{epochs}$.
\begin{algorithm}[H]
    \caption{Stochastic gradient descent(SGD).}
    \label{alg:sgd}
    \begin{algorithmic}[1]
        \For $i_e$ in $N_\mathrm{epochs}$ epochs
            \State Shuffle data $\bm{X}_\mathrm{train}$, $\bm{y}_\mathrm{train}$
            \State Split into mini batches, $\bm{X}_\mathrm{train}$, $\bm{y}_\mathrm{train}$
            \For $i_\mathrm{mb}$ in mini batches,
                \State Perform gradient descent step, and retrieve $\nabla\bm{w}_{i_\mathrm{mb}}$
            \EndFor
            \State Take the average of the $\nabla\bm{w}_{i_\mathrm{mb}}$ and update the weight matrix $\bm{w}$.
        \EndFor
    \end{algorithmic}
\end{algorithm}

% To minimize this cost function we use gradient descent. \husk{Gradient descent measures the local gradient of the cost function, with regards to $\beta$ in our case. Since the gradient goes in the direction of fastest increase, we will go in the opposite direction, i.e. negative gradient. We start by choosing random values for $\beta$ (since our cost function is convex any choice should give correct results), calculate the gradient, update the $\beta$ values, and do this iteratively until the algorithm converges to a minimum. The size of the steps is important, and is determined by the learning rate. If the learning rate is too small, we will need many iterations which is time consuming. However, if the learning rate is too high, we might overshoot and miss the minimum. One way to choose the learning rate is too let it depend on the size of the gradient. If the gradient is large, i.e a steep slope, the learning rate can be relatively high. When the gradient is small, the learning rate is also small.}

% \sjekk{Returning to the logistic regression problem, the derivative of the cost function is
% \begin{equation}\label{eqT:diff_cost}
% \begin{split}
% \frac{\partial J(\beta)}{\partial \beta} &=-\frac{1}{N}\bm{X}^T(\bm{y}-\bm{p})\\
%  &= \frac{1}{N}\bm{X}^T(\bm{p}-\bm{y}),
% \end{split}
% \end{equation}
% where $\bm{X}$ is the $N\times(p+1)$ matrix of $x_i$ values, $\bm{p}$ is the vector of fitted probabilities with $i$th element $p(x_i;\beta)$ and $\bm{y}$ is the vector of $y_i$ values. The new $\beta$ using gradient descent is then\\
% $\beta_{new} = \beta_{old}\ -\ \frac{\partial J(\beta)}{\partial \beta}lr$, where $lr$ is the learning rate (step size).} \husk{This is done iteratively until we reach the set max iterations or $\frac{\partial J(\beta)}{\partial \beta}$ is within a given tolerance of zero.}

% \sjekk{Like we introduced Lasso and Ridge regression to avoid overfitting in Project 1, we can add a penalty term to the cost function in Equation \ref{eq:Geron_cost}. In our project we used two different penalties: $L1 = \lambda|\beta|$ and $L2 = \lambda||\beta||^2$. When fitting the model we need to include the derivatives of the penalty term in Equation \ref{eqT:diff_cost}.} \husk{The gradient with the penalty term is,
% \begin{equation}
% \begin{split}
% \frac{\partial J(\beta)}{\partial \beta} &=\frac{1}{N}\bm{X}^T(\bm{p}-\bm{y})\ + \lambda\cdot{sign(\beta)},\ for\ L1\ regularization\\
%  &\mathrm{or}\\
% \frac{\partial J(\beta)}{\partial \beta} &=\frac{1}{N}\bm{X}^T(\bm{p}-\bm{y})\ + \lambda\cdot{2\beta},\ for\ L2\ regularization.
% \end{split}
% \end{equation}
% }


\subsection{Kessler Parameterization}
SCALE implements a one-moment bulk microphysical scheme, which treats only warm clouds (cloud and rain). This scheme predicts the mixing ratio of cloud ($Q_{cloud}$) and rain ($Q_{rain}$). Cloud microphysical processes treated in this scheme are saturation adjustment (corresponding to nucleation, evaporation, and cloud condensation), evaporation, auto-conversion, accretion, and sedimentation. The tendency of $Q_{cloud}$, $Q_{rain}$, and $Q_{v}$ (vapor mixing ratio) is as follows:

\begin{eqnarray}
\frac{\partial Q_{cloud}}{\partial t}&=&dQ|_{sat}-dQ|_{auto}-dQ|_{acc}\\
\frac{\partial Q_{rain}}{\partial t}&=&dQ|_{auto}+dQ|_{acc}-dQ|_{evap}-F_{Q_{r}}|_{sed}\\
\frac{\partial Q_{v}}{\partial t}&=&dQ|_{evap}-dQ|_{sat}
\end{eqnarray}

where $dQ|_{sat}$, $dQ|_{auto}$, $dQ|_{acc}$, and $dQ|_{evap}$ represent the mixing ratio tendency by saturation adjustment, auto-conversion, accretion, and evaporation, respectively. $F_{Q_{r}}|_{sed}$ represents flux of $Q_{r}$ by sedimentation.\\ $dQ|_{auto}$, $dQ|_{acc}$, and $dQ|_{evap}$ are given as:

\begin{eqnarray}
dQ_{auto}&=&\left\{
\begin{array}{ll}
Q_{cloud}*10^{-3} & (Q_{cloud}>10^{-3})\\
0 & (else)\\
\end{array}\right.\\
dQ_{acc}&=&2.2\times Q_{cloud}\times Q_{rain}^{0.875}\\
dQ_{evap}&=&\left\{
\begin{array}{ll}
f_{vent} \frac{q_{s}-Q_{cloud}}{q_{s}\rho}\frac{(\rho*Q_{rain})^{0.525}}{5.4\times 10^{5}+\frac{2.55\times 10^{8}}{pq_{s}}} & ( q_{s} > Q_{cloud})\\
0 & (else)\\
\end{array} \right.
\end{eqnarray}

where $f_{vent}$ is the ventilation factor ($f_{vent}=1.6+124.9(\rho Q_{rain})^{0.2046}$), and the unit of $dQ_{***}$ is [kg/kg/s]. $p$, $q_{s}$, and $\rho$ are pressure, saturation vapor mixing ratio, and total density, respectively.\\
$dQ|_{sat}$ is given as:

\begin{eqnarray}
dQ|_{sat}=Q_{v}-q_{s}.
\end{eqnarray}

Terminal velocities of cloud ($V_{t,c}$) and rain ($V_{t,r}$) are given as:

\begin{eqnarray}
V_{t,c}&=&0\\
V_{t,r}&=&36.34(\rho Q_{rain})^{0.1364} [m/s]
\end{eqnarray}