\documentclass[11pt,a4paper]{report}
\usepackage{a4wide}
\usepackage{color}
\usepackage{graphicx}
\usepackage{fixltx2e}
% Default margins are too wide all the way around. I reset them here
%\setlength{\topmargin}{-.5in}
%\setlength{\textheight}{9in}
%\setlength{\oddsidemargin}{.125in}
%\setlength{\textwidth}{175mm}
\setlength{\parindent}{0cm}
\begin{document}
\title{COSC3500\\Final Report}
\author{Anthony Vaccaro\\University of Queensland}

%\today
\maketitle
\begin{abstract}
This report introduces the concept of modelling forest fires with cellular
automata, and discusses an implementation of this model.
\end{abstract}

\newpage
\tableofcontents
\newpage

\chapter{Initial Research and Serial Implementation}

\newpage
\section{The Forest-Fire Model}

The Forest-Fire model is a cellular automaton-based model of a wild fire
spreading through a forest of trees or other plants. The model has a few
variants but typically follows these rules:\\

\begin{itemize}
\item An empty cell becomes a tree with probability p.
\item A tree becomes a burning tree with probability f.
\item A tree becomes a burning tree if its neighbours are burning.
\item A burning tree becomes an empty cell.
\end{itemize}
where variables p and f can be adjusted to produce different patterns.

These simple rules can produce complex patterns and have lead to research
into the area of modelling forest fires.

\newpage
\section{Algorithm used}

The algorithm chosen for this report and corresponding program is based on the
rules given above, with a few minor adjustments:\\

\begin{itemize}
\item Burning trees take l timestep to become empty.
\item Trees with burning neighbours have probability c to catch fire.
\end{itemize}

This gives provides the simulation with four main variables to adjust - p, f,
l, and c.

\includegraphics{tex/flowchart}

\newpage
\section{Implementation details}

The implementation was written in C, based on the student's past
experience with programming. The implementation did not require any complex
programming techniques, although a few key points are worth discussing.\\

As much of this simulation relies on probabilities, the use of a random number
generator requires some thought. the \texttt{random()} function of the C
standard library was selected due to its ease of use and availability, but as
it is only a pseudo-random number generator, perhaps this could be changed in
later efforts. There are websites that provide "true randomness" based on
atmospheric noise\cite{random}, and perhaps data from this service could be
used and compared against the output of \texttt{random()}.\\

Exporting the output of the program was done via the use of
libpng\cite{libpng}, which can easily convert 2d arrays into pictures. Sample
code was used from a third party website in order to assist with this
process\cite{png}. After each timestep was completed, the grid of cells was
exported to an image, with pixels being coloured based on their status.

\newpage
\section{Verification of program's output}

The implementation currently uses \texttt{random()} for its random number
generation, and this can be made deterministic by using a predefined seed with
the \texttt{srandom()} function. This will be used to compare the results of
later optimisations and multiprocessing techniques.

\newpage
\section{Algorithm's performance while scaling with forest size}

As the algorithm operates on neighbours, which remain constant with an
increasing grid size, performance should scale linearly with number of cells in
the grid.

\begin{verbatim}

$ time ./forest 100 0 100
real    0m1.179s
user    0m0.433s
sys     0m0.046s

$ time ./forest 200 0 100
real    0m2.524s
user    0m1.802s
sys     0m0.113s

$ time ./forest 300 0 100

real    0m5.395s
user    0m4.500s
sys     0m0.172s

\end{verbatim}

\newpage
\section{Further Considerations}

\newpage

\chapter{Optimisation and Parallel Implementation}

\newpage

\section{Optimisation of Serial Algorithm}

Before commencing with parallelisation of the \texttt{forest} program, the code
was examined for optimisations that could be made. An initial profile of the
program\footnote{The program was compiled without any optimisation flags, and
run on a 400 by 400 cell grid for 1000 timesteps.} with the \texttt{gprof} tool gave the following output:

\begin{verbatim}
Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
 51.29      6.30     6.30     1000     6.30     8.04  out_png
 28.22      9.77     3.47 160000000     0.00     0.00  cell_auto
  8.32     10.79     1.02     1000     1.02     1.38  save_png_to_file
  6.28     11.57     0.77     1000     0.77     4.24  timestep
  5.83     12.28     0.72 320000000     0.00     0.00  pixel_at
  0.29     12.32     0.04                             print_usage
  0.00     12.32     0.00        1     0.00     0.00  alloc_forest
  0.00     12.32     0.00        1     0.00     0.00  parse_args

\end{verbatim}

This profile showed that approximately 65\% of execution time was spent
creating the PNG images and writing them to disk.\\

From this profile, two immediate conclusions were drawn: There should be an
alternate output method that does not incur any I/O overhead, and the PNG
output module should be optimimised.\\

a ``null'' output module was implemented, which allowed for benchmarking of the
code without worrying about the performance of third party libraries or I/O.
Terminal output via the \texttt{ncurses} library was also implemented, for easy
examination of code output changes.\\

The \texttt{timestep()} function was later moved into \texttt{main()} to save
unnecessary function call overhead.
\newpage
\subsection{PNG Module}

Inside the PNG module, several immediate optimisations were recognised: the
simple function \texttt{pixel\_at()} could be easily inlined, and the
intermediary struct \texttt{bitmap} was unnecessary and could be bypassed
completely. The module was rewritten and showed a considerable
performance increase:

\begin{verbatim}
Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
 51.53      5.11     5.11     1000     5.11     5.11  out_png
 40.72      9.15     4.04 160000000     0.00     0.00  cell_auto
  7.88      9.93     0.78     1000     0.78     4.82  timestep
  0.10      9.94     0.01                             main
  0.00      9.94     0.00        1     0.00     0.00  alloc_forest
  0.00      9.94     0.00        1     0.00     0.00  parse_args

\end{verbatim}

As can be seen from the above profile, the rewritten \texttt{out\_png()}
function takes approximately 5ms per call, as opposed to 8ms in the original
version. The overhead from unnecessary function calls was removed by combining
procedures into one function, and the total execution time spent on output was
now 51\%, down from 65\%.

\subsection{Memory Allocation}

Initially, the two-dimensional grid that was used to store cell data was
allocated using three tiers of pointers: a top-level array containing double
pointers, inner arrays of pointers, and finally structs were allocated
individually.\\

This presented two problems: examining a cell would require four or even five
memory lookups to de-reference all the pointers, and with individually allocated
cell structs, the physical location of neighbouring cells in memory could be
quite distant, resulting in cache misses, particularly with larger grids.\\

By switching to a one-dimensional array that was allocated in one call to
\texttt{malloc()}, both of these problems were solved. Performance was shown to
increase, although this was not measured as other optimisations took place
around this time.\\

\section{Parallelisation}

After the code was found to be sufficiently optimised in a serial environment,
methods to improve performance via parallelisation were investigated.\\

The main iteration loop over cells in the grid was reimplemented with two grids
instead of one, so that changes could be made to a "new" grid while reading an
"old" grid. This solved a problem with correctness in early versions of the
program, where fires would spread in one direction more than others, due to
examining neighbouring cells that were from the current timestep rather than
the previous timestep.\\

After moving to the two-grid implementation, iterating over one section of the
grid did not depend on other sections, and the loop could easily be
parallelised with OpenMP. Using MPI to further parallelise via domain
decomposition was also identified as a method to increase performance.

\subsection{OpenMP}

OpenMP allows for simple division of problems into smaller chunks which can be
processed independently. This is performed via multithreading.\\

OpenMP is implemented via the use of \texttt{pragma} directives added before
sections of code. To parallelise the main iteration loop in the \texttt{forest}
program, the \texttt{pragma omp parallel for} directive was added before the
loop initialisation, as follows:

\begin{verbatim}
    for (i = 0; i < f->simLength; i++) {
            #pragma omp parallel for
            for (x = 0; x < s; x++) {
                for (int y = 0; y < t; y++) {
                    int rand;
                    random_r(&stuff[omp_get_thread_num()], &rand);
                    cell_auto(f, x, y, f->oldGrid[(y * t) + x].status, &rand);
                    }
                }
\end{verbatim}

This split the loop at each row, giving each thread a row of the grid to
operate on.\\

Initial performance when using OpenMP showed a significant performance
decrease. Using the \texttt{OMP\_NUM\_THREADS} environment variable to limit
exectution to four threads showed more than a tenfold increase in execution time compared to
execution when using one thread:

\begin{verbatim}
$ export OMP_NUM_THREADS=1
$ time ./forest 400 400 null 1000
6.207 sec

$ export OMP_NUM_THREADS=4
$ time ./forest 400 400 null 1000
84.831 sec
\end{verbatim}

While examining the code, it was found that the \texttt{random()} function
used was not multithread-capable, as it relies on a global buffer to hold state
information between calls. However, the glibc library provides a reentrant random number
generator which is capable of being used in a multithreaded environment, called
\texttt{random\_r()}. The code was modified to use \texttt{random\_r()} with
individual state buffers for each thread, and performance was shown to scale
with threads:

\begin{verbatim}
$ export OMP_NUM_THREADS=1
$ time ./forest 400 400 null 1000
5.571 sec

$ export OMP_NUM_THREADS=2
$ time ./forest 400 400 null 1000
4.613 sec

$ export OMP_NUM_THREADS=4
$ time ./forest 400 400 null 1000
3.206 sec
\end{verbatim}

These tests were performed on a desktop computer with an Intel i7 2600k CPU.
When switching to a node on the Barrine cluster, performance still degraded
when using multiple threads:

\begin{verbatim}
forest with 400 x 400 grid on 1 threads took: 6.627 sec
forest with 400 x 400 grid on 2 threads took: 12.016 sec
forest with 400 x 400 grid on 4 threads took: 8.013 sec
forest with 400 x 400 grid on 8 threads took: 7.210 sec
\end{verbatim}

The cause of this performance decrease is as of yet unexplained, however it
could be due to differing CPU architectures. The Barrine node used has an Intel
Xeon L5520 CPU, which is from an earlier generation than the desktop CPU used.

\subsection{MPI}

Using MPI to parallelise programs involves running multiple instances of the
same program on multiple computers, or nodes, and communicating between each
node to transfer data. Each node running the MPI-enabled program is known as a
"rank".\\

A suitable method for improving performance with MPI would be to divide the
grid into multiple smaller sections, and have each rank process one section. At
the end of each timestep, each rank could pass their section to neighbouring
ranks, allowing an updated copy of the entire grid to exist on each rank. Then,
processing could continue. OpenMP could be used to divide the work on each rank
up into smaller subsections, allowing for all CPUs on a rank to be utilised
fully.\\

The MPI implementation of the program was written to use four ranks, each
working on one quarter of the grid. To keep implementation as simple as
possible, the entire grid was passed between ranks at the end of each timestep,
however it would also be possible to pass just the bordering rows/columns of
cells. This would keep transferred data to a minimum and improve performance,
but this technique was not able to be implemented in time.

\newpage
\section{Verification of implementation output}

To ensure that the program's output could be verified, it had to be able to
produce deterministic output. A verification output method was added, which
altered the seeding method for random number generation to use a static seed.
The module examines the contents of the cell grid at the final timestep of the
simulation, and produces an MD5 checksum of the contents. This was implemented
using glibc's \texttt{crypt()} function.\\

If the program's output can be determined at runtime, it should be possible to
reproduce the same output by running the program with the same arguments,
multiple times. This was found to be the case when using one OpenMP thread, but
when using multiple threads, the program's output changed between runs:

\begin{verbatim}

$ export OMP_NUM_THREADS=1
$ ./forest 240 240 verify 500
Grid checksum is: $1$cosc3500$Zz63shoT031bnGE.f1UVF/
$ ./forest 240 240 verify 500
Grid checksum is: $1$cosc3500$Zz63shoT031bnGE.f1UVF/
$ ./forest 240 240 verify 500
Grid checksum is: $1$cosc3500$Zz63shoT031bnGE.f1UVF/

$ export OMP_NUM_THREADS=2
$ ./forest 240 240 verify 500
Grid checksum is: $1$cosc3500$q.T.ss7LAG/oZpycVU9Ig0
$ ./forest 240 240 verify 500
Grid checksum is: $1$cosc3500$ibx89tXbp/9Jer53wZkPH.
$ ./forest 240 240 verify 500
Grid checksum is: $1$cosc3500$7R05CaitedeiVLqruuDFw0

\end{verbatim}

The reason for the non-determinism when using multiple threads is due to the
technique of pseudo-random number generators relying on a seed value and
producing random numbers in a sequence. When working in a single-threaded
context, one sequence is used over the entire grid - for example, a cell
at position \textbf{n} in a grid would have the \textbf{n}\textsuperscript{th}
number in a random sequence.\\

However, in a context where multiple threads are working on the same grid,
each thread requires its own sequence so as to avoid contention. This creates
the issue observed above.

\newpage
\section{Test Plan}

The \texttt{forest} program's output should be made deterministic when working
with multiple threads, but once this has been accomplished, it can be run on
large grids for long simulation lengths.\\

One area which would be worthwhile exploring is the number of trees and burning
trees in a forest over time. As fires burn more rapidly when there are more
trees, it would be interesting to see the effect of fires that occur in
sparsely-populated forests, compared to densely populated forests. Collection
of these statistics is easily accomplished with the current program, and they
could be graphed using third-party software.

\newpage

\chapter{``Big Runs''}

\section{Performance Scaling on a HPC Cluster}

With an implementation that can be run on multiple computer nodes using MPI, as well as
being able to take advantage of multi-core CPUs with OpenMP, some consideration should be
given to performance when run on different configurations of threads and
ranks.\\

Because there is some overhead when synchronising between threads and ranks, it
is important to use a large grid size, so that there is sufficient work to be
done by each thread/rank between timesteps. Using a small grid size would
almost certainly result in the overhead generated by parallelisation
outweighing the benefits of the added processing power.\\

The OpenMP, MPI, and serial implementations were each run on a grid of
2560x1440 size, for 1000 timesteps. OpenMP-enabled implementations were each
run with 1, 2, 4, and 8 threads. the MPI implementation was fixed to 4 ranks
due to implementation restraints.\\
\\
\begin{tabular}{| l | l | l | l | l | l |}
\hline
Libraries used & OpenMP & MPI  & Total threads & Execution & \% \\
 & threads used & ranks used & & time  & \\ \hline
None & n/a & n/a & 1 & 113.661s & 100\% \\ \hline
OpenMP & 1 & n/a & 1 & 117.157s & 103\% \\ \hline
OpenMP & 2 & n/a & 2 & 204.803s & 180\% \\ \hline
OpenMP & 4 & n/a & 4 & 200.665s & 176\% \\ \hline
OpenMP & 8 & n/a & 8 & 139.342s & 122\% \\ \hline
MPI/OpenMP & 1 & 4 & 4 & 170.443s & 149\% \\ \hline
MPI/OpenMP & 2 & 4 & 8 & 171.454s & 150\% \\ \hline
MPI/OpenMP & 4 & 4 & 16 & 125.797s & 110\% \\ \hline
MPI/OpenMP & 8 & 4 & 32 & 120.502s & 106\% \\ \hline
\end{tabular}

\newpage
Unfortunately, it appears that the naive serial implementation is still faster
than all parallel implementations. As mentioned earlier, there is an overhead
when using OpenMP that was found to be absent when using a desktop CPU. This
may be the result of different OpenMP implementations or extra instructions
provided by newer-generation CPUs.\\

This demonstrates an issue with parallelisation that will become increasingly
important as computers begin to scale with core count, rather than with
clock speed. Parallelisation is not an ``easy fix'' that can be applied to
increase performance of any algorithm or procedure; it relies on careful
consideration and optimisation in order to provide benefits. Simply attempting
to give a program more CPUs will not increase its performance linearly, and in
many cases will reduce performance. Some algorithms cannot be parallelised, and
others may require too much overhead to be able to provide a performance
increase.\\

The \texttt{forest} program's algorithm should be able to benefit from
parallelisation in theory, but the implementation of MPI used includes large
amounts of unnecessary data transfer. This reduces the performance to a point
where the serial implementation is the best option to use, which should not be
the case.

\section{Testing Process Used}

The source package for the \texttt{forest} program includes a Makefile which
allows compilation with OpenMP or MPI functionality enabled. Also included are
PBS scripts to allow running of test runs with varying configurations of
MPI/OpenMP on a HPC cluster. These scripts also output the execution time of
each run.\\

\texttt{forest} has had a logging feature added. This allows the count of trees
and burning trees in the forest to be recorded in a text file at the end of
each timestep. This allows for the easy graphing of these numbers.\\

PBS jobs were submitted to the Savanna cluster, and execution times were
averaged over three runs, in order to account for CPU contention with other
jobs.\\

\newpage
\section{Scientific Experimentation}

Simulating forest fires is a potentially useful area of computer science,
because these fires are a large hazard in many countries, including Australia.
Learning how to control fires and understanding the safety features required to
protect people living in forest areas.\\

Unfortunately, many of the features that would be required to investigate these
areas were not implemented in \texttt{forest}. Much of the development cycle
was spent on parallelisation techniques, instead of improving the simulation.
However, the logging feature can be used to demonstrate the self-organised
criticality of the forest fire model:\\

\includegraphics[scale=0.8]{tex/graph}

\newpage

\appendix
\chapter{Appendix One: Commands used during testing}

The following is a sample of the PBS job used during performance testing runs:
\begin{verbatim}
#!/bin/bash
#PBS -N mpi-test
#PBS -q workq
#PBS -A uq-COSC3500
#PBS -lwalltime=00:10:00
#PBS -lnodes=4:ppn=8
#PBS -lmem=1gb

. /usr/share/modules/init/bash

module load intel-mpi intel-cc-13

cd $PBS_O_WORKDIR
export TIMEFORMAT="%E sec"

export OMP_NUM_THREADS=8
export DIM=2560
export DIM2=1440
export LENGTH=1000

echo -n "forest with $DIM x $DIM2 grid for $LENGTH timesteps on
$OMP_NUM_THREADS threads using 4 MPI nodes took: " >&2
time mpirun -envall -np 4 forest-mpi $DIM $DIM2 null $LENGTH
\end{verbatim}

The following is a sample of the output of the previous job:
\begin{verbatim}
forest with 2560 x 1440 grid for 1000 timesteps on 8 threads WITHOUT MPI took:
139.342 sec
\end{verbatim}

\begin{thebibliography}{99}

\bibitem{png}
"Write a PNG file using C and libpng", 2010 Ben Bullock
http://www.lemoda.net/c/write-png/

\bibitem{libpng}
"libpng Home Page", 2013 Greg Roelofs
http://www.libpng.org/pub/png/libpng.html

\bibitem{random}
"RANDOM.ORG - True Random Number Service", 2012 RANDOM.ORG
http://www.random.org/

\end{thebibliography}
\end{document}


\minesp draws upon ideas from time series compression, time series forecasting, integer compression, general-purpose compression, and high-performance computing. From a technical perspective, \minesp is unusual or unique in its abilities to:
\begin{enumerate}
    \item Bit pack with extremely small block sizes
    \item Bit pack low-bitwidth integers effectively
    \item Efficiently exploit correlation between successive samples in \textit{multivariate} time series
    \item Naturally integrate both run-length encoding and bit packing
    \item Exploit vectorized hardware through forecaster, learning algorithm, and bit packing method co-design
\end{enumerate}
% Certain other methods do one of these, but we are not aware of an existing algorithm that does two or more.
From an application persective, \minesp is distinct in that it enables higher-ratio lossless compression with far less memory and latency than competing methods.

    % \item Quickly and losslessly compress a wide variety of time series while obtaining excellent compression ratios

% TODO also reiterate what's different in terms of what it enables here

% ------------------------------------------------
\subsection{Compression of Time Series}
% ------------------------------------------------

Most work on compressing time series has focused on lossy techniques. The most common approach is to approximate the data as a sequence of low-order polynomials \cite{swab, lemireSegmentation, tsCompressSmartGrid, iotCompressCrap, apca, paa}. An alternative, commonly seen in the data mining literature, is to discretize the time series using Symbolic Aggregate Approximation (SAX) \cite{sax} or its variations \cite{isax, isax2}. These approaches are designed to preserve enough information about the time series to support indexing or specific data mining algorithms (e.g. \cite{isax, fastShapelet, hotSax}), rather than to compress the time series \textit{per se}. As a result, they are extremely lossy; a hundred-sample time series might be compressed into one or two bytes, depending on the exact discretization parameters. % This is a result of these methods being intended to facilitate certain tasks, rather compress data \textit{per se}. % , but do construct representations that can be understood as lossy encodings of the data.

% Other lossy approaches include Adaptive Piecewise Constant Approximation \cite{apca}, and Piecewise Aggregate Approximation \cite{paa}.%, and numerous other methods  that approximate time series as sequences of low-order polynomials.

For audio time series specifically, there are a large number of lossy codecs \cite{vorbis, shorten, aac, opus}, as well as a small number of lossless \cite{flac, alac} codecs. In principle, some of these could be applied to non-audio time series. However, modern codecs make such strong assumptions about the possible numbers of channels, sampling rates, bit depths, or other characteristics that it is infeasible to use them on non-audio time series.

Many fewer algorithms exist for lossless time series compression. For floating-point time series, the only algorithm of which we are aware is that of the Gorilla database \cite{gorilla}. This method XORs each value with the previous value to obtain a diff, and then bit packs the diffs. In contrast to our approach, it assumes that time series are univariate and have 64-bit floating-point elements. % The same work also describes a method of compressing integer timestamps. This consists of first delta-delta coding and then applying a similar bit packing compression approach.
% Like other databases \cite{influxDB, berkeleyTreeDB}, they delta-delta code before applying compression %Most time series databases use some form of integer compression (c.f. next section) [], generic predictive coding \cite{akumuli}, or floating-point compression methods.

For lossless compression of integer time series (including timestamps), existing approaches include directly applying general-purpose compressors \cite{respawnDB, openTSDB, chronicleDB, kairosDB, druid}, (double) delta encoding and then applying an integer compressor \cite{influxDB, gorilla}, or predictive coding and byte-packing \cite{akumuli}. These approaches can work well, but tend to offer both less compression and less speed than \mine.

% A final noteworthy method is Blosc \cite{blosc}. While not intended solely for time series (or integers), Blosc's assumption that every $k$ bytes (or bits \cite{bitshuf}) will be correlated for some $k$ is a natural fit for multivariate time series. %grouping of correlated bytes and/or bits makes it well-suited for multivariate time series.

% ------------------------ parquet
%   -currently supports snappy, GZIP, lzo
%   -hdfs also supports these; prolly others also
%   -https://www.cloudera.com/documentation/enterprise/5-6-x/topics/impala_parquet.html#parquet_compression
% ------------------------ Akumuli
%   -DFCM predictor (for both floats and ints?); XOR with prediction, then do something to pack it
%   -"The timestamp can be a simple integer or datetime in ISO 8601 format"
%   -handles int or float values
%   -"This data-structure consists of 4KB blocks"
%   -"They require large amount of memory per data stream to maintain a sliding window of previously seen samples. The larger the context size the better compression ratio gets."
%   -"if we're dealing with 100'000 time-series we'll need about 1GB of memory only for compression contexts."
% ------------------------ RespawnDB
%   -GZIP
%   -"Enabling compression adds a factor of six slowdown in BTDS performance"
% ------------------------ OpenTSDB
%   -LZ0 or snappy for floats, varbyte (1, 2, 4, or 8 bytes) for ints
%   -http://opentsdb.net/faq.html
% ------------------------ Gorilla
%    -custom F64 for values, custom delta-delta for timestamps
% ------------------------ ChronicleDB
%   -lz4
% ------------------------ BerkeleyTreeDB
%   -custom delta-delta
% ------------------------ KairosDB
%     -off-the-shelf compressors: lzo, snappy, probably others
%     -https://github.com/kairosdb/kairosdb/search?utf8=✓&q=compression&type=
% ------------------------ Druid
%   -"The timestamp and metric columns are simple: behind the scenes each of these is an array of integer or floating point values compressed with LZ4."
%       -"metrics" are scalars (the time series)
%   -they also have "dimensions" which are categorical cols; assign each a numeric ID and map IDs to lists of places they occur
%   -http://druid.io/docs/latest/design/segments.html
% ------------------------ InfluxDB
%   -"Timestamp encoding is adaptive and based on the structure of the timestamps that are encoded. It uses a combination of delta encoding, scaling, and compression using simple8b run-length encoding"
%   -"Floats are encoded using an implementation of the Facebook Gorilla paper."
%   -For integers: "If all ZigZag encoded values are less than (1 << 60) - 1, they are compressed using simple8b encoding. If any values are larger than the maximum then all values are stored uncompressed in the block. If all values are identical, run-length encoding is used."
%   -"Strings are encoding using Snappy compression"
%   -and before compaction (where all the above methods get used): "When a write comes in the new points are serialized, compressed using Snappy, and written to a WAL file"
%       -and note: "This means that batching points together is required to achieve high throughput performance. (Optimal batch size seems to be 5,000-10,000 points per batch for many use cases.)"
% ------------------------ LittleTable
%   -LZO1X (LZO) compression http://www.oberhumer.com/opensource/lzo/
% ------------------------ RocksDB (not a ts database, but whatever)
%   -snappy, lz4, zstd

\subsection{Compression of Integers}

% Integer compression is not as well-studied as general-purpose compression, but has seen great progress in recent years.

The fastest methods of compressing integers are generally based on bit packing---i.e., using at most $b$ bits to represent values in $\{0, 2^b-1\}$, and storing these bits contiguously \cite{bbp, pfor, fastpfor}. Since $b$ is determined by the largest value that must be encoded, naively applying this method yields limited compression. To improve it, one can encode fixed-size blocks of data at a time, so that $b$ can be set based on the largest values in a block instead of the whole dataset \cite{kGamma, pfor, fastpfor}. A further improvement is to ignore the largest few values when setting $b$ and store their omitted bits separately \cite{pfor, fastpfor}.

\minesp bit packing differs significantly from existing methods in two ways. First, it compresses much smaller blocks of samples. This reduces its throughput as compared to, e.g., \cite{fastpfor}, but significantly improves compression ratios (c.f. Section~\ref{sec:results}). This is because large values only increase $b$ for a few samples instead of for many. Second, \minesp is designed for 8 and 16-bit integers, rather than 32 or 64-bit integers. Existing methods are often inapplicable to lower-bitwidth data (unless converted to higher-bitwidth data) thanks to strong assumptions about bitwidth and data layout.
% $b$ that is even one too large significantly alters the ratio for low-bitwidth data.

A common \cite{flac, shorten} alternative to bit packing is Golomb coding \cite{golomb}, or its special case Rice coding \cite{rice}. The idea is to assume that the values follow a geometric distribution, often with a rate constant fit to the data. %, and therefore make the encoding cost linear in the magnitude of the encoded value.

Both bit packing and Golomb coding are bit-based methods in that they do not guarantee that encoded values will be aligned on byte boundaries. When this is undesirable, one can employ byte-based methods such as 4-Wise Null Suppression \cite{kGamma}, LEB128 \cite{dwarf}, or Varint-G8IU \cite{varintG8IU}. These methods reduce the number of bytes used to store each sample by encoding in a few bits how many bytes are necessary to represent its value, and then encoding only that many bytes. Some, such as Simple8B \cite{simple8b} and SIMD-GroupSimple \cite{groupSimd}, allow fractional bytes to be stored while preserving byte alignment for groups of samples. % These methods allow for efficient universal codes---that is, codes that can represent any possible integer. Universal

% Before applying any of these coding schemes, it is common to apply some transform to the raw data to make the values closer to 0. Such transforms include delta encoding, \cite{fastpfor, bbp}, delta-delta encoding \cite{influxDB}, and linear predictive coding (LPC) \cite{flac}. LPC deterministically generates a prediction for each sample based on the previous samples, and stores the error in the prediction instead of the raw value; when the errors are small, the integers stored are closer to 0. Delta coding and delta-delta coding are special cases wherein each sample is predicted to be the previous sample, or a linear extrapolation from the previous two samples, respectively.

% ------------------------------------------------
\subsection{General-Purpose Compression}
% ------------------------------------------------
% While \minesp is only designed to compress integer time series, a
A reasonable alternative to using a time series compressor would be to apply a general-purpose compression algorithm, possibly after delta coding or other preprocessing. Thanks largely to the development of Asymmetric Numeral Systems (ANS) \cite{ans} for entropy coding, general purpose compressors have advanced greatly in recent years. In particular, Zstd \cite{zstd}, Brotli \cite{brotli}, LZ4 \cite{lz4} and others have attained speed-compression tradeoffs significantly better than traditional methods such as GZIP \cite{gzip}, LZO \cite{lzo}, etc. However, these methods have much higher memory requirements than \minesp and, empirically, often do not compress as well and/or decompress as quickly.

% Also of note is Blosc \cite{blosc}, which is especially applicable to multivariate time series as a result of its grouping correlated bits and/or bytes together during preprocessing.

% ------------------------------------------------
\subsection{Predictive Filtering}
% ------------------------------------------------

% TODO move delta, double, LPC descriptions to here. Also talk about \fire.

% Predictive coding in some form is a common element of many compression algorithms. This consists of having some forecaster predict the values of the next byte(s) to be compressed and only storing the prediction error, rather than the original value. When the forecaster is better than chance, the errors will be drawn from a lower-entropy distribution than that of the raw data. In particular, they will often be tightly concentrated around zero \cite{shorten}. The original data can be reconstructed from the errors by having the decoder run the same forecaster and add the encoded errors to its predictions.

For numeric data such as time series, there are four types of predictive coding commonly in use: predictive filtering \cite{png}, delta coding \cite{fastpfor, bbp}, double-delta coding \cite{influxDB, gorilla}, and XOR-based encoding \cite{gorilla}. In predictive filtering, each prediction is a linear combination of a fixed number of recent samples. This can be understood as an autoregressive model or the application of a Finite Impulse Response (FIR) filter. When the filter is learned from the data, this is termed ``adaptive filtering.'' Many audio compressors use some form of adaptive filtering \cite{shorten, flac, aac}.

Delta coding is a special case of predictive filtering where the prediction is always the previous value. Double-delta coding, also called delta-delta coding or delta-of-deltas coding, consists of applying delta coding twice in succession. XOR-based encoding is similar to delta coding, but replaces subtraction of the previous value with the XOR operation. This modification is often desirable for floating-point data \cite{gorilla}.

Our forecasting method can be understood as a special case of adaptive filtering. While adaptive filtering is a well-studied mathematical problem in the signal processing literature, we are unaware of a practical algorithm that attains speed within an order of magnitude of our own. I.e., our method's primary novelty is as a vectorized \textit{algorithm} for fitting and predicting multivariate time series, rather than as a mathematical \textit{model} of multivariate time series. That said, it does incorporate different modeling assumptions than other compression algorithms for time series in that it reduces the model to one parameter and omits a bias term. % It considers training speed and vectorization in a way that is orthogonal to all adaptive filtering work of which we are aware. It also differs from the adpative filtering in audio codecs in that there is only one coefficient to learn and it is learned online. %. or that accounts for the subtleties of low-precision integers.

% Indeed, the only methods

% most similar method is likely that of \cite{neuralBranchPredictor}, in which a

% Moreover, existing databases almost universally use delta, double-delta, or XOR encoding, suggesting that simple  % suggesting that these approaches are the relevant benchmarks.

% Before applying any of these coding schemes, it is common to apply some transform to the raw data to make the values closer to 0. Such transforms include delta encoding, \cite{fastpfor, bbp}, delta-delta encoding \cite{influxDB}, and linear predictive coding (LPC) \cite{flac}. LPC deterministically generates a prediction for each sample based on the previous samples, and stores the error in the prediction instead of the raw value; when the errors are small, the integers stored are closer to 0. Delta coding and delta-delta coding are special cases wherein each sample is predicted to be the previous sample, or a linear extrapolation from the previous two samples, respectively.


\documentclass{article}

%% preamble
\usepackage{hyperref}
\usepackage{verbatim}
\usepackage{color}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{cancel}

\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 9.in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in
\newcommand{\myhrule}{ \begin{center}\rule{.9\linewidth}{.25mm}\end{center} }
\definecolor{darkgray}{rgb}{0.95,0.95,0.95}
\definecolor{heavygray}{rgb}{0.05,0.05,0.05}
\definecolor{foo}{rgb}{.8,0,.8}
\newcommand{\pad}{\vspace{8pt}\noindent}
\newcommand{\red}[1]{{\color{red}#1\color{black}}}
\newcommand{\myhref}[2]{\href{#1}{\color{foo}\underline{#2}\color{black}}}

\newcommand{\Mod}[1]{\ \mathrm{mod}\ #1}

\begin{document}

\title{CMDA 3634 Spring 2018 Homework 03}

%% change this to your name
\author{Pim Silpacharn}
\vspace{-64pt}\maketitle
\begin{center}\underline{You must complete the following task by 5pm on Tuesday 03/27/18.}\end{center}
Your write up for this homework should be presented in a {\LaTeX} formatted PDF document. You may copy the \LaTeX{} used to prepare this report as follows

\begin{enumerate}
\item Click on this  \myhref{https://www.sharelatex.com/read/tkfwfqfjqqgs}{link}
\item Click on Menu/Copy Project.
\item Modify the HW03.tex document to respond to the following questions.
\item Remember: click the Recompile button to rebuild the document when you have made edits.
\item Remember: Change the author.

\end{enumerate}

\pad \emph{Each student} must individually upload the following files to the CMDA 3634 Canvas page at \myhref{https://canvas.vt.edu}{https://canvas.vt.edu}

\begin{enumerate}
\item \verb|firstnameLastnameHW03.tex| {\LaTeX} file.
\item Any figure files to be included by \verb|firstnameLastnameHW03.tex| file.
\item \verb|firstnameLastnameHW03.pdf| PDF file.
\end{enumerate}

\noindent In addition, all source code must be submitted to an online git repository as follows:
\begin{enumerate}
    \item While on the webpage for you git repository, go to Settings $\rightarrow$ Collaborators.
    \item Add nchalmer@vt.edu and zjiaqi@vt.edu as collaborators.
    \item Make a folder named HW03 in you repository and store all relevant source files to this assignment in this folder. Ensure your assignment files compile with {\verb make }.
\end{enumerate}

\pad You must complete this assignment on your own.

\vspace{16pt}
\begin{center}
\underline{\bf 160 points will be awarded for a successful completion.}
\vspace{8pt}\underline{\bf Extra credit will be awarded as appropriate.}
\end{center}

\newpage

\vspace*{1em}
\noindent{\bf Q0.1}(30 points) 30 points is awarded for attending class on March 13th and submitting ``Computing pi in parallel using OpenMP'' code to GitHub. All code and any plotting results should be saved to your GitHub repository. Any additional comments can be included here in this homework.
\vspace*{1em}

\vspace*{1em}
\noindent{\bf Q0.2}(30 points) 30 points is awarded for attending class on March 15th and submitting ``adding threading to a Mandelbrot set generator'' code to GitHub. All code and any plotting results should be saved to your GitHub repository. Any additional comments can be included here in this homework.

\vspace*{1em}

\section*{ElGamal Public-key Cyptography}

In the previous assignment we added some safe modular products and exponentiation functions, and used these to program functions which can identify very large prime numbers $p$ and even easily find generators of $\mathbb{Z}_p$ for some specific primes. With these tools, we can now reliably setup an ElGamal cryptographic system. In this assignment, we will implement the ElGamal encryption and decryption algorithms and begin exploring how to use parallelism if we wish to determine the secret key used in an ElGamal cryptographic system.

\subsection*{Part 1}
In the {\verb HW03 } folder you will find two folders, {\verb Part1 } and {\verb Part2 }. This section concerns the contents of the {\verb Part1 } folder.

\pad {\bf Setting up an ElGamal Cryptographic System}:
Recall that in the description of the ElGamal cryptographic system in Homework 1 we considered three different people, Alice, Bob, and Eve. In the system, Alice sets up an ElGamal cryptographic system and shares her \emph{public-key} which consists of the prime number $p$, generator $g$, and number $h$. She also secretly selects a number $x$, keeping this number private. Using the public-key, Bob is able to send messages that only Alice can read by encrypting his messages $m$ using $p$, $g$, and $h$ in such a way that only Alice is able to decrypt the resulting cyphertext.

Let us consider a single program which can emulate this process. In the source file {\verb main.c } we have begun a program which must be run with MPI using at least 2 ranks. We will label rank 0 as Alice and rank 1 as Bob.

\vspace*{1em}
\noindent{\bf Q1.1}(5 points) The program should begin with Alice setting up her ElGamal cryptographic system. Since this the same process each time, we have added the function {\verb setupElGamal } in the {\verb functions.c } file. Write this function so it calculates and returns the necessary values for an ElGamal cryptographic system.

\vspace*{1em}
\noindent{\bf Q1.2}(5 points) Alter the {\verb main } function so that only Alice inputs a bit length from the user and sets up an ElGamal cryptographic system.
\vspace*{1em}

\vspace*{1em}
\noindent{\bf Q1.3}(5 points) Use MPI to broadcast the values which are part of Alice's public-key to all MPI processes.
\vspace*{1em}

\pad {\bf Encryption and Decryption}:
Once the ElGamal system is set up, the crucial steps that remain are the encryption and decryption functions. The {\verb main } function currently creates an array of {\verb Nmessages=5 } messages, encrypts and then decrypts them, outputting their values along the way.

\vspace*{1em}
\noindent{\bf Q2.1}(10 points) In the file {\verb functions.c } we have begun implementing the {\verb ElGamalEncrypt } function. Use the description of the ElGamal encryption process from Homework 1 to complete this function.
\vspace*{1em}

\vspace*{1em}
\noindent{\bf Q2.2}(10 points) In the file {\verb functions.c } we have begun implementing the {\verb ElGamalDecrypt } function. Use the description of the ElGamal decryption process from Homework 1 to complete this function.
\vspace*{1em}

\vspace*{1em}
\noindent{\bf Q2.3}(15 points) Alter the {\verb main } function so that only Bob populates the message array and encrypts it. After encryption, use MPI to send the encrypted messages to Alice. Alice should then receive the messages and decrypt them. Run your program and confirm that Alice obtains Bob's original messages after decryption.
\vspace*{1em}

\subsection*{Part 2}
In the {\verb HW03 } folder you will find two folders, {\verb Part1 } and {\verb Part2 }. This section concerns the contents of the {\verb Part2 } folder. Before completing this section, copy the contents of your {\verb setupElGamal }, {\verb ElGamalEncrypt }, and {\verb ElGamalDecrypt } functions completed above to {\verb functions.c } in this folder.

\pad {\bf Cracking the ElGamal Cryptographic System}:
Once the messages $m$ are encrypted, is very difficult to decrypt them without knowledge of the random number $y$ used in the encryption algorithm, or the secret key $x$. In the case where we have many messages, finding each $y$ is not feasible and we must instead determine the secret key. This amounts to finding the number $x$ such that
\[
h = g^x \Mod{p}.
\]
Trying to solve this equation by brute force (that is, literally trying every value $x$) is, luckily, embarrassingly parallel. In this part we will use MPI to accelerate the process of cracking the ElGamal cryptographic system by distributing the job of trying each value of $x$ amoungst many MPI processes.
\vspace*{1em}

\noindent{\bf Q3.1}(5 points) Alter the {\verb main } function so that only rank 0 sets up the ElGamal cryptographic system and then broadcast the public key information to all MPI processes.
\vspace*{1em}

\noindent{\bf Q3.2}(15 points) In the {\verb main } function, all MPI processes currently loop through all values of $x$. Partition this loop between all MPI processes by setting values for {\verb start } and {\verb end } so that all values of $x$ are tested, and the amount of work each MPI process performs is roughly equal.
\vspace*{1em}

\pad {\bf Parallel performance}:

\noindent{\bf Q4.1}(10 points) What would be a good measure of the work performed inside the loop in your program's {\verb main } function? Use {\verb MPI_Wtime() } to compute the total time this loop takes to run inside your program and use your measurement of work to compute your program's throughput. Output both runtime and throughput quantities at the end of your program.
\vspace*{1em}

\noindent{\bf Q4.2}(20 points) Using New River, run your program several times for $N = 1, 2, 4, 8, 12, 16,$ and 20 MPI processes. You may need to alter your main file to fix the number of bits $n$, rather than inputting it from the user. Plot the average runtime of your program vs $N$ for a variety of bit lengths. In a separate plot, show the average throughput numbers reported by your program. Give some comments on how this program's performance scales with the number of MPI processes.
\vspace{5mm} \\
\textbf{Comments}: As the number of MPI processes increases, the runtime decreases. Increasing the bit size numbers increases the runtime. I noticed that the throughput quantities increased throughout but I am unsure I did this process correctly.
\vspace*{1em}

\vspace*{1em}
\noindent{\bf Bonus:}(20 points) In practice, we would like to stop searching for the secret key $x$ once it is found, rather than continuing the loop. Alter the loop in your {\verb main } function so that at regular intervals, say every {\verb Ninterval } iterations, you determine if the secret key has been found, and stop iterating if so. Remake your runtime and throughput plots in Q3.3. If you averaged over a large number of runs, how much faster would you expect this updated version of the code to run? Does the frequency at which you check if the secret has been found, i.e. the size of {\verb Ninterval } affect the performance?

%\myhrule

%\begin{thebibliography}{9}
%\bibitem{gol} Wikipedia -- description of the Game of Life
%\url{https://en.wikipedia.org/wiki/Conway's_Game_of_Life}

%\end{thebibliography}

\end{document}