%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%% DYMOLA INTERFACE  %%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Dymola Interface}
Modelica is "a non-proprietary, object-oriented, equation-based language to conveniently model complex physical systems containing, e.g., mechanical, electrical, electronic, hydraulic,
thermal, control, electric power or process-oriented subcomponents."\footnote{\url{http://www.modelica.org}}.  Modelica models (with a file extension of .mo) are built, translated (compiled), and simulated in Dymola (http://www.modelon.com/p-
roducts/dymola/), which is a commercial modeling and simulation environment based on the Modelica modeling language.
A standard Modelica example called BouncingBall, which simulates the trajectory of an object falling in one dimension from a height, is shown as an example:
\begin{lstlisting}
model BouncingBall
  parameter Real e=0.7 "coefficient of restitution";
  parameter Real g=9.81 "gravity acceleration";
  parameter Real hstart = 10 "height of ball at time zero";
  parameter Real vstart = 0 "velocity of ball at time zero";
  Real h(start=hstart,fixed=true) "height of ball";
  Real v(start=vstart,fixed=true) "velocity of ball";
  Boolean flying(start=true) "true, if ball is flying";
  Boolean impact;
  Real v_new;
  Integer foo;

equation
  impact = h <= 0.0;
  foo = if impact then 1 else 2;
  der(v) = if flying then -g else 0;
  der(h) = v;

  when {h <= 0.0 and v <= 0.0,impact} then
    v_new = if edge(impact) then -e*pre(v) else 0;
    flying = v_new > 0;
    reinit(v, v_new);
  end when;

  annotation (uses(Modelica(version="3.2.1")),
    experiment(StopTime=10, Interval=0.1),
    __Dymola_experimentSetupOutput);

end BouncingBall;
\end{lstlisting}

\subsubsection{Files}
When a modelica model, e.g., BouncingBall model, is implemented in Dymola, the platform dependent C-code from a Modelica model and the corresponding executable code
(i.e., by default dymosim.exe on the Windows operating system) are generated for simulation.  After the executable is generated, it may be run multiple times (with Dymola license).
A separate TEXT file (by default dsin.txt) containing model parameters and initial conditions are also generated as part of the build process.  The RAVEN Dymola interface
modifies input parameters by changing copies of this file.  Both the executable and TEXT parameter file (or simulation initialization file) names must be provided to RAVEN. The TEXT parameter file must be of type 'DymolaInitialisation'.  In the case of
the BouncingBall model previously mentioned on the Windows operating system, the \textless Files\textgreater  specification would look like:
\begin{lstlisting}[style=XML]
<Files>
  <Input name='dsin.txt' type='DymolaInitialisation'>dsin.txt</Input>
</Files>
\end{lstlisting}

The Dymola interface can only pass scalar values into the TEXT parameter file. If the user wants to pass vector information to Dymola, he can do so by providing an optional TEXT vector file to Dymola. This file must have the type 'DymolaVectors'. This additional file can then be read by the Dymola model. If vecor data is passed from RAVEN to the Dymola interface and the TEXT vector file is not specified, the interface will display an error and stop the Dymola execution. If the TEXT vector file is specified (and vector data is passed to the interface), the interface will write the datd into the specified file, but also display a warning, saying that the Dymola interface found vector data to be passed and if this data is supposed to go into the simulation initialisation file of type 'DymolaInitialisation' the array must be split into scalars. The \textless Files\textgreater specification for the vector data look as follows:
\begin{lstlisting}[style=XML]
<Files>
  <Input name='timeSeriesData.txt' type='DymolaVectors'>timeSeriesData.txt</Input>
</Files>
\end{lstlisting}

\subsubsection{Models}
An executable (dymosim.exe) and a simulation initialization file (dsin.txt) can be generated after either translating or simulating the
Modelica model (BouncingBall.mo) using the Dymola Graphical User Interface (GUI) or Dymola Application Programming Interface (API)-routines.
To generate an executable and a simulation initialization file, use the Dymola API-routines (or Dymola GUI) to translate the model as follows:
\lstset{
    frame=single,
    breaklines=true,
    postbreak=\raisebox{0ex}[0ex][0ex]{\ensuremath{\color{red}\hookrightarrow\space}}
}
\begin{enumerate}
\item Change to the directory containing the .mo file to generate an executable.  In Dymola GUI, this corresponds to File/Change Directory in menus:
\begin{lstlisting}
>> cd("C:/msys64/home/KIMJ/projects/raven/framework/CodeInterfaces/Dymola");
C:/msys64/home/KIMJ/projects/raven/framework/CodeInterfaces/Dymola
 = true
\end{lstlisting}
\item Reads the specified file and displays its window.  In Dymola GUI, this corresponds to File/Open in the menus:
\begin{lstlisting}
>> openModel("BouncingBall.mo")
 = true
\end{lstlisting}
\item Compile the model (with current settings), and create the model executable and the corresponding simulation initialization file.  In Dymola GUI, this corresponds to Translate Model in the menus:
\begin{lstlisting}
>> translateModel("BouncingBall");
 = true
\end{lstlisting}
At this point the model executable and the simulation initialization file should have been created in the same directory as the original model file.
Additionally, they could be created by simulating the model.  The following command corresponds to Simulate in the menus in Dymola GUI:
\begin{lstlisting}
>> simulateModel("BouncingBall", stopTime=10, numberOfIntervals=0, outputInterval=0.1, method="dassl", resultFile="BouncingBall");
 = true
\end{lstlisting}
The file extension (.mat) is automatically added to a output file (resultFile), e.g., BouncingBall.mat.  If the generated executable code is triggered directly from a
command prompt, the output file is always named as "dsres.mat".
\end{enumerate}
The model executable is specified to RAVEN using the \textless Models\textgreater  section of the input file as follows:
\begin{lstlisting}[style=XML]
<Simulation>
    ...
  <Models>

      <executable>dymosim.exe</executable>

  </Models>
    ...
</Simulation>
\end{lstlisting}
RAVEN works best with Comma-Separated Value (CSV) files.  Therefore, the default
.mat output type needs to be converted to .csv output.
The Dymola interface will automatically convert the .mat output to human-readable
forms, i.e., .csv output, through its implementation of the finalizeCodeOutput function.
\\In order to speed up the reading and conversion of the .mat file, the user can specify
the list of variables (in addition to the Time variable) that need to be imported and
converted into a csv file minimizing
the IO memory usage as much as possible. Within the \xmlNode{Code} the following
XML
node (in addition ot the \xmlNode{executable} one) can be inputted:

\begin{itemize}
   \item \xmlNode{outputVariablesToLoad}, \xmlDesc{space separated list, optional
   parameter}, a space separated list of variables that need be exported from the .mat
   file (in addition to the Time variable). \default{all the variables in the .mat file}.
\end{itemize}
For example:
\begin{lstlisting}[style=XML]
<Simulation>
    ...
  <Models>

      <executable>dymosim.exe</executable>
      <outputVariablesToLoad>var1 var2 var3</outputVariablesToLoad>

  </Models>
    ...
</Simulation>
\end{lstlisting}


% JuliaCon Proceedings
% Template with instructions: https://www.overleaf.com/read/dqjbrhqxjpwq

\documentclass{juliacon}
\setcounter{page}{1}
\usepackage{url}
\usepackage{amsmath}
\usepackage[]{algorithm2e}
\newcommand\EatDot[1]{}

\begin{document}

\input{header}

\maketitle

\begin{abstract}

\small{\texttt{StateSpaceModels.jl}} is an open-source Julia package for modeling, forecasting and simulating time series in a state-space framework. The package represents a straightforward tool that can be useful for a wide range of applications that deal with time series. In addition, it contains features that are not present in related commercial software, such as Monte Carlo simulation and the possibility of setting any user-defined linear model.

\end{abstract}

\section{Motivation} \label{sec:introduction}

State-space modeling is a classical framework in control engineering that represents a system through the definition of input, state, and output variables \cite{zadeh2008linear}. Input variables are external entities that are inserted into the system and can serve as control inputs or noise. State variables represent unobserved components that evolve through time following a given state equation and also depending on the values of the input variables. Finally, output variables result from the realization of the state plus noise factors and represent the observable outcome of the system. In general, state-space models make use of the Kalman filter \cite{kalman1960new, welch1995introduction} to obtain predictive estimates for the state.

Due to its comprehensive form and wide range of potential applications, state-space models found a niche in time-series modeling, forecasting, and simulation, representing a flexible framework for time-series analysis with time-varying parameters \cite{durbin2012time}. The ability to conveniently define the evolution of a time series allows the characterization of stochastic components, such as trend and seasonality, that are non-trivial to model in other frameworks.

There are several packages focused on state-space models for time-series analysis in other languages, such as \texttt{KFAS} in R \cite{kfas} and \texttt{Statsmodels} in Python \cite{seabold2010statsmodels}, as well as commercial software such as \texttt{STAMP} \cite{koopman2000stamp}. Among related Julia packages, we highlight:
\begin{itemize}
    \item ~\texttt{Kalman.jl} \cite{kalmanjl}, which implements Kalman filtering and smoothing.
    \item ~\texttt{StateSpace.jl} \cite{statespacejl}, a control-oriented package with several versions of the Kalman filter.
    \item ~\texttt{ControlSystems.jl} \cite{controlsystemsjl}, a package aimed at control applications with numerous features, including state-space modeling.
\end{itemize}

The contributions brought by these packages notwithstanding, there are currently no state-space model packages aimed at time-series analysis in Julia. Furthermore, some have not been updated for a long time and are not functional in Julia 1.0. Thus, the main objectives of \texttt{StateSpaceModels.jl} \cite{statespacemodels} are:

\begin{enumerate}
    \item to fill this gap by implementing a general and intuitive framework in Julia for modeling, estimating, forecasting and simulating time series with state-space models;
    \item to provide an open-source package that is capable of performing the same functions as related commercial software such as \texttt{STAMP} and more;
    \item to develop a package for time-series analysis with state-space models that is fully implemented in Julia, contrary to related packages in other languages that depend on C or Fortran \mbox{routines} \cite{kfas}.
\end{enumerate}

The remainder of this paper is organized as follows. In Section \ref{sec:statespace}, the Gaussian state-space framework is introduced. In Section \ref{sec:modeling}, the procedures for model specification are presented. Section \ref{sec:estimation} explains the filtering, estimation and smoothing processes. Section \ref{sec:diagnostics} discusses the available diagnostics to evaluate the model specification and estimation. In Section \ref{sec:forecasting}, the forecasting and simulation procedures are presented. Section \ref{sec:applications} contains examples of applications that illustrate the use of the package. Finally, conclusions and future work are provided in Section \ref{sec:conclusion}.

\section{Gaussian state-space framework} \label{sec:statespace}

For the sake of consistency and readability, we utilize in the package the same notation as \cite{durbin2012time}. Suppose we have a series of observations $y_{1}, \dots, y_{n}$. In a state-space framework, it is assumed that the observations depend on a set of unobserved components denominated states, or simply the state, and denoted by $\alpha_{1}, \dots, \alpha_{n}$. The state components often have physical interpretations, such as trend and seasonality in a time-series context or position and speed in a control setting.

The main idea behind state-space modeling is to define the evolution of the state and its relation with the observed variables. This is done through the following two equations, denoted the observation equation and the state equation, respectively:
%
\begin{align}
    y_{t} &= Z_{t} \alpha_{t} + \varepsilon_{t}, \hspace{0.08\columnwidth} \varepsilon_{t} \sim N(0, H_{t}), \label{eq:obs_equation} \\
    \alpha_{t+1} &= T_{t} \alpha_{t} + R_{t} \eta_{t}, \quad \eta_{t} \sim N(0, Q_{t}), \label{eq:state_equation}
\end{align}
%
\noindent where $y_{t}$ is a $p \times 1$ vector of observations and $\alpha_{t}$ is an unobserved $m \times 1$ vector representing the state at instant $t$. Note that the behavior of the system over time is determined by $\alpha_{t}$ as defined in \mbox{Eq. \eqref{eq:state_equation}}, but $\alpha_{t}$ cannot be directly observed, contrary to $y_{t}$.

Matrices $Z_{t}$, $T_{t}$, and $R_{t}$ are the ones that define how the observations relate to the state and how the state evolves over time, and are generally assumed to be known. The error terms $\varepsilon_{t}$ and $\eta_{t}$ are supposed to be serially independent and independent of each other. Finally, $H_{t}$ and $Q_{t}$ are the covariance matrices of the error terms. In this package, it is assumed that $T$, $R$, $H$, and $Q$ are not time-varying, as this is the case for the vast majority of practical applications. Conversely, $Z_{t}$ is allowed to vary over time.

Next, for illustration purposes, we present an example model and show how it can be inserted into the state-space framework. Consider the following model, called the linear trend model:
%
\begin{align}
    y_{t} &= \mu_{t} + \varepsilon_{t}, \hspace{0.115\columnwidth} \varepsilon_{t} \sim N(0, \sigma^{2}_{\varepsilon}), \label{eq:lineartrend1} \\
    \mu_{t+1} &= \mu_{t} + \nu_{t} + \xi_{t}, \quad \xi_{t} \sim N(0, \sigma^{2}_{\xi}), \label{eq:lineartrend2} \\
    \nu_{t+1} &= \nu_{t} + \zeta_{t}, \hspace{0.12\columnwidth} \zeta_{t} \sim N(0, \sigma^{2}_{\zeta}), \label{eq:lineartrend3}
\end{align}
%
\noindent where $\mu_{t}$ represents the trend component and $\nu_{t}$ the slope component. Note that the we can rewrite this model in the framework of Eq. \eqref{eq:obs_equation}--\eqref{eq:state_equation} in the following manner:
%
\begin{align}
    y_{t} &= \begin{bmatrix} 1 & 0 \end{bmatrix} \alpha_{t} + \varepsilon_{t}, \hspace{0.15\columnwidth} \varepsilon_{t} \sim N(0, \sigma^{2}_{\varepsilon}), \\
    \alpha_{t+1} &= \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} \alpha_{t} + \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}\!\eta_{t}, \quad \eta_{t} \sim N \bigg(\!\begin{bmatrix} 0 \\ 0 \end{bmatrix}\!, \begin{bmatrix} \sigma^{2}_{\xi} & 0 \\ 0 & \sigma^{2}_{\zeta} \end{bmatrix}\!\bigg).
\end{align}
%
Similarly, more complex and sophisticated models can also be modeled in the state-space framework.

\section{Model specification} \label{sec:modeling}

Model specification is the step wherein the user defines the state-space model they want to consider. This is done through the creation of a \texttt{StateSpaceModel} structure that contains the observations $y_{t}$ as well as matrices $Z_{t}$, $T$, and $R$.

\subsection{Predefined models} \label{subsec:predefined}

A set of predefined classical models is available, namely the local level model, the linear trend model, the basic structural model, and the structural model with exogenous variables. These models can be conveniently defined by calling a function and providing the observations $y_{t}$. Note that all models automatically extend to the multivariate case.

\subsubsection{Local level model}

The local level model consists of a stochastic level component that is defined by a random walk:
%
\begin{align}
    y_{t} &= \mu_{t} + \varepsilon_{t}, \quad \varepsilon_{t} \sim N(0, \sigma^{2}_\varepsilon), \label{eq:locallevel1} \\
    \mu_{t+1} &= \mu_{t} + \xi_{t}, \quad \xi_{t} \sim N(0, \sigma^{2}_\xi). \label{eq:locallevel2}
\end{align}
%
A local level \texttt{StateSpaceModel} can be created using:
%
\begin{lstlisting}[language = Julia]
model = local_level(y)
\end{lstlisting}
%
\noindent where \texttt{y} is an observation vector in the univariate case or a matrix in column-wise fashion in the multivariate case, i.e., each column representing a variable and each line representing a time period.
%

\subsubsection{Linear trend model}

The linear trend model consists of a local level model with the addition of a stochastic slope term, as seen in Eq. \eqref{eq:lineartrend1}--\eqref{eq:lineartrend3}. A linear trend \texttt{StateSpaceModel} can be created using:
%
\begin{lstlisting}[language = Julia]
model = linear_trend(y)
\end{lstlisting}

\subsubsection{Structural model}

The basic structural model consists of three stochastic components: trend, slope, and seasonality, defined in the following manner:
%
\begin{align}
    y_{t} &= \mu_{t} + \gamma_{t} + \varepsilon_{t}, \hspace{0.09\columnwidth} \varepsilon_{t} \sim N(0, \sigma^{2}_\varepsilon), \label{eq:structural1} \\
    \mu_{t+1} &= \mu_{t} + \nu_{t} + \xi_{t}, \hspace{0.095\columnwidth} \xi_{t} \sim N(0, \sigma^{2}_\xi), \label{eq:structural2} \\
    \nu_{t+1} &= \nu_{t} + \zeta_{t}, \hspace{0.18\columnwidth} \zeta_{t} \sim N(0, \sigma^{2}_\zeta), \label{eq:structural3} \\
    \gamma_{t+1} &= \sum_{j=1}^{s-1} \gamma_{t+1-j} + \omega_{t}, \quad \omega_{t} \sim N(0, \sigma^{2}_\omega). \label{eq:structural4}
\end{align}
%
In this model, there is a stochastic trend that contains a slope component, similarly to the linear trend model. Furthermore, a stochastic seasonal component is modeled so that its sum over the seasonality period $s$ is equal to zero except for a noise factor. A basic structural \texttt{StateSpaceModel} can be created using:
%
\begin{lstlisting}[language = Julia]
model = structural(y, s)
\end{lstlisting}

Additionally, a structural model can contain exogenous variables, also known as explanatory variables. Exogenous variables represent external factors that are correlated to the phenomenon in question and can, thus, be used to improve the estimation and forecasting by increasing the available information. This model is identical to \eqref{eq:structural1}--\eqref{eq:structural4} except for the observation equation, which has an additional exogenous factor:
%
\begin{align}
    y_{t} &= \mu_{t} + \gamma_{t} + \theta^{\top} X_{t} + \varepsilon_{t}, \quad \varepsilon_{t} \sim N(0, \sigma^{2}_\varepsilon) \label{eq:structural_exogenous}
\end{align}
%
A structural model with exogenous variables can be created with:
%
\begin{lstlisting}[language = Julia]
model = structural(y, s; X = X)
\end{lstlisting}
%
\noindent where \texttt{X} is a matrix containing the exogenous variables observations in column-wise fashion.

\subsection{User-defined models}

Apart from the well-known predefined models presented in Section \ref{subsec:predefined}, the package allows the input of any user-defined linear model through the definition of matrices $Z_{t}$, $T$, and $R$ through the following constructor:
%
\begin{lstlisting}[language = Julia]
model = StateSpaceModel{Float64}(y, Z, T, R)
\end{lstlisting}

In the simpler case where $Z_{t}$ is constant, the argument \texttt{Z} can be input as an \texttt{Array\{Float64, 2\}}. Conversely, if $Z_{t}$ is time-varying, it must be input as an \texttt{Array\{Float64, 3\}} where the third dimension represents the time periods. Any desired float precision can be employed within the model.

\section{Filtering, estimation, and smoothing} \label{sec:estimation}

Next, the model needs to be estimated. In this step, values for the covariance matrices $H$ and $Q$, initially assumed to be unknown, are obtained via maximum likelihood estimation \cite{casella2002statistical}. In order to do that, the Kalman filter needs to be utilized. Additionally, a smoother is applied in order to obtain the so-called smoothed estimates for the state.

After specifying the model, the estimation step can be called with the function \texttt{statespace}:
%
\begin{lstlisting}[language = Julia]
ss = statespace(model; verbose = 1)
\end{lstlisting}

In this step, the Kalman filtering, the maximum likelihood estimation of the fixed parameters, and the smoothing are conducted. The optional argument \texttt{verbose}, which defaults to 1, specifies the verbosity: 0 for no output, 1 for some progress information, 2 for progress information and summary of the optimization result and 3 for progress information as well as the full optimization log.

An example of the estimation log with default verbosity can be seen below.
%
\begin{lstlisting}[language = Julia]
julia> ss = statespace(model)
==============================================================
                  StateSpaceModels.jl v0.2.0
 (c) Raphael Saavedra, Guilherme Bodin, and Mario Souto, 2019
--------------------------------------------------------------
            Starting state-space model estimation.
    Initiating maximum likelihood estimation with 3 seeds.
--------------------------------------------------------------
             Seed 0 is aimed at degenerate cases.
--------------------------------------------------------------
||    seed    |     log-likelihood      |      time (s)     ||
||       0    |        -16217.4939      |          0.21     ||
||       1    |         -1350.4763      |          2.65     ||
||       2    |         -1350.4763      |          4.08     ||
||       3    |         -1350.4763      |          5.69     ||
--------------------------------------------------------------
           Maximum likelihood estimation complete.
                  Log-likelihood: -1350.4763
             End of state-space model estimation.
==============================================================
\end{lstlisting}

\subsection{Filtering}

The filtering step derives the predictive and filtered states as well as their covariance matrices at every time period via the Kalman filter. The predictive state and its variance are given by
%
\begin{align}
    a_{t+1} &= \mathbb{E}[\alpha_{t+1} | Y_{t}], \\
    P_{t+1} &= \mathbb{V}[\alpha_{t+1} | Y_{t}],
\end{align}
%
while the filtered state and its variance are given by
%
\begin{align}
    a_{t|t} &= \mathbb{E}[\alpha_{t} | Y_{t}], \\
    P_{t|t} &= \mathbb{V}[\alpha_{t} | Y_{t}],
\end{align}
%
where $Y_{t}$ denotes the set of time periods $y_{1}, \dots, y_{t}$.

Furthermore, the filter also computes the innovations, or prediction errors, as well as their covariance matrix at every time period:
%
\begin{align}
    v_{t} &= y_{t} - Z a_{t}, \\
    F_{t} &= \mathbb{V}[v_{t}].
\end{align}
%

The package contains an implementation of the standard Kalman filter and the square-root Kalman filter \cite{durbin2012time}. The latter is a variant which utilizes Cholesky decomposition to ensure that the computed covariance matrices are positive semidefinite, thus avoiding numerical errors.

Additionally, in order to provide a flexible framework, we allow the utilization of any user-implemented variant of the Kalman filter through an abstract type called \texttt{AbstractFilter}:
%
\begin{lstlisting}[language = Julia]
struct MyKalmanFilter <: AbstractFilter
\end{lstlisting}

The choice of the filter type must be done in the estimation step as follows:
%
\begin{lstlisting}[language = Julia]
ss = statespace(model; filter_type = SquareRootFilter{Float64})
\end{lstlisting}

Filters allow any desired float precision. If no \texttt{filter\_type} is provided, the standard type \texttt{KalmanFilter\{Float64\}} is utilized as a default option. Further information on Kalman filtering can be found in Chapter 4.3 of \cite{durbin2012time}.

\subsection{Estimation}

The Kalman filter is responsible for deriving estimates for the predictive state and for its covariance matrix at each time period, but it is dependent on the values of the noise covariances $H$ and $Q$. Thus, these constant parameters need to be estimated via maximum likelihood \cite{casella2002statistical}. The log-likelihood function is given by
%
\begin{align}
    \ell(Y_{n}) = -\frac{np}{2} \log 2 \pi - \frac{1}{2} \sum_{t=1}^{n} (\log |F_{t}| + v_{t}^{\top} F_{t}^{-1} v_{t}),
\end{align}
%
which depends on the innovations and their covariance matrix at each time period. Therefore, at each optimization iteration, the Kalman filter needs to be executed.

The implemented optimization method \texttt{RandomSeedsLBFGS} generates random initial values for the parameters and then uses the L-BFGS algorithm \cite{liu1989limited} to maximize the log-likelihood function via the unconstrained optimization package \texttt{Optim.jl} \cite{mogensen2018optim}.

Additionally, similar to the filtering step, users are able to define any desired optimization method through abstract type \texttt{AbstractOptimizationMethod}:
%
\begin{lstlisting}[language = Julia]
struct MyOptimizationMethod <: AbstractOptimizationMethod
\end{lstlisting}

As with the filter type, the choice of optimization method must be done in the estimation step:
%
\begin{lstlisting}[language = Julia]
ss = statespace(model; optimization_method = RandomSeedsLBFGS())
\end{lstlisting}

More advanced parameters, such as optimizer tolerances and the number of seeds, can be set within the \texttt{RandomSeedsLBFGS} constructor.

\subsection{Smoothing}

The smoothing step is conducted after estimation is complete. It consists in obtaining the so-called smoothed state and its covariance matrix at each time period, i.e.,
%
\begin{align}
    \hat{\alpha}_{t} &= \mathbb{E}[\alpha_{t} | Y_{n}], \\
    V_{t} &= \mathbb{V}[\alpha_{t} | Y_{n}].
\end{align}
%
Similar to the filtering step, the smoothing is conducted through an iterative process. The procedure is done backwards, starting from the final time period $n$. The smoothed state estimates are useful for analyzing the behavior of the state components. Further information on state smoothing can be found on Chapter 4.4 of \cite{durbin2012time}.

\subsection{Missing observations}

One interesting property of the Kalman filter and smoother is their ability to treat missing observations. Suppose we have a time series spanning time periods $t = 1, \dots, n$, but for some periods $\tau, \dots, \tau^{*}$ we have no observations available. This represents an obstacle for estimating a model in several frameworks. Nonetheless, the Kalman filter and smoother allow the derivation of minimum variance linear unbiased estimates for the missing observations so that a completion of the time series is possible. Conveniently, the only necessary modification in the filtering and smoothing equations is considering $Z_{t} = 0$ for the missing period.

In the package, this is implemented so that any \texttt{NaN} values in the observations are considered missing observations and automatically treated in the filtering and smoothing steps. In this way, the estimation is successfully conducted even when the time series of interest is incomplete.

Consider the following simulated time series composed by a growing trend plus a Gaussian noise. Additionally, we remove observations 10 through 20 to treat them as missing:
%
\begin{lstlisting}[language = Julia]
# Create growing trend series with Gaussian noise
y = collect(1:0.25:20) + 0.5*randn(77)
# Remove observations 10 through 20
y[10:20] .= NaN
# Specify the state-space model and estimate it
model = linear_trend(y)
ss = statespace(model)
\end{lstlisting}
%
The results of the estimation step are presented in Fig. \ref{fig:missing_observations}. The predictive, filtered, and smoothed state estimates can be accessed in \texttt{ss.filter.a}, \texttt{ss.filter.att}, and \texttt{ss.smoother.alpha}, \mbox{respectively}. It is clear in Fig. \ref{fig:missing_observations} that the filter and smoother are able to effectively capture the growing linear trend of the series even in the period where observations are missing.
%
\begin{figure}[h]
	\centering
	\includegraphics[width=\columnwidth]{Figures/missing_obs.pdf}
	\caption{Automatic completion of missing observations in the filtering and smoothing.}
	\label{fig:missing_observations}
\end{figure}

\subsection{Estimation outputs}

At the end of the filtering, estimation, and smoothing routine executed by the \texttt{statespace} function, a \texttt{StateSpace} structure is returned. It contains six fields:

\begin{itemize}
    \item ~\texttt{model}: the previously defined \texttt{StateSpaceModel}.
    \item ~\texttt{filter}: a \texttt{FilterOutput} structure representing the result of the Kalman filter. It contains the predictive and filtered states as well as the innovations and their covariance matrices at each time period. Furthermore, there is a flag indicating if steady state was attained and the time period when it was attained.
    \item ~\texttt{smoother}: a \texttt{SmoothedState} structure representing the result of the smoothing procedure. It contains the smoothed state as well as its covariance matrix at each time period.
    \item ~\texttt{covariance}: a \texttt{StateSpaceCovariance} structure which contains the covariance matrices of the observation and state noises.
    \item ~\texttt{filter\_type}: the defined filter type.
    \item ~\texttt{optimization\_method}: the defined optimization method.
\end{itemize}

\section{Diagnostics} \label{sec:diagnostics}

As a way of evaluating the model specification and estimation, the package contains statistical diagnostics that can be conducted over the residuals. The available diagnostics are the Jarque-Bera normality test \cite{jarque1980efficient}, the Ljung-Box independence test \cite{box1970distribution}, and an homoscedasticity test \cite{durbin2012time}. The diagnostics can be run through function \texttt{diagnostics}.

An example of the diagnostics log can be seen below.
%
\begin{lstlisting}[language = Julia]
julia> diagnostics(ss)
==============================================================
                    Running diagnostics...
                Jarque-Bera: p-value = 0.89683
                  Ljung-Box: p-value = 0.95855
           Homoscedasticity: p-value = 0.30752
==============================================================
\end{lstlisting}

\section{Forecasting and Simulation} \label{sec:forecasting}

Finally, after the model has been estimated, forecasting and Monte Carlo simulation can be conducted. In the case of forecasting, estimates for the future values of the time series are computed along with their probability distributions. Alternatively, it is possible to simulate an arbitrary number of future scenarios of the series, which is useful for a wide range of applications such as stochastic optimization.

The minimum mean square error forecasts for $y_{t}$ can be obtained by treating future values of $y_{t}$ as missing values. Forecasting is conducted with the function \texttt{forecast}, which receives a \texttt{StateSpace} structure coming from the estimation step and the number of time periods ahead to be forecast, and outputs the minimum square error forecasts and the predictive distributions of $y_{t}$ at each time period:
%
\begin{lstlisting}[language = Julia]
pred, dist = forecast(ss, N)
\end{lstlisting}
%
Alternatively, another powerful tool is the simulation of future scenarios, since these are used as input by several applications, such as certain stochastic optimization models. Performing Monte Carlo simulation in a time-series state-space framework involves sampling several scenarios for the observation and state errors utilizing the estimated variances $H$ and $Q$ at each time period, and then computing the state-space recursions for each scenario.

Monte Carlo simulation can be conducted with the use of the function \texttt{simulate}, which receives a \texttt{StateSpace} structure coming from the estimation step, the number of time periods ahead $N$, and the number of scenarios $S$ to be simulated, and outputs an $N \times S$ matrix of scenarios for $y_{t}$:
%
\begin{lstlisting}[language = Julia]
scenarios = simulate(ss, N, S)
\end{lstlisting}

\section{Applications} \label{sec:applications}

In this section, we present several practical applications which can be addressed via state-space modeling. We utilize \texttt{StateSpaceModels.jl} to tackle the presented problems and provide the results. Furthermore, for the sake of reproducibility, the code of some of the examples is in the folder \texttt{examples} in the \texttt{StateSpaceModels.jl} repository.

\subsection{Airline passengers}

As a first example, let us use the classical monthly airline passengers time series. In order to avoid multiplicative effects, we use the well-known approach of taking the log of the series. Fig. \ref{fig:log_airline_forecast} shows the log-airline passengers time series. We can estimate a structural \texttt{StateSpaceModel} as follows.
%
\begin{lstlisting}[language = Julia]
# Load the AirPassengers dataset
AP = CSV.read("AirPassengers.csv")
# Take the log of the series
logAP = log.(Vector{Float64}(AP[:Passengers]))
# Specify the state-space model
model = structural(logAP, 12)
# Estimate the state-space model
ss = statespace(model)
\end{lstlisting}

By estimating a structural model, we can analyzed the individual components of the series, such as trend and seasonality. These components are presented in Fig. \ref{fig:log_airline_components} and represent dimensions 1 and 3 of the smoothed state, respectively.

We can also forecast the following two years of the time series by using the \texttt{forecast} function. The result is displayed in Fig. \ref{fig:log_airline_forecast}.
%
\begin{lstlisting}[language = Julia]
# Number of months ahead to be forecast
N = 24
# Perform forecasting
pred, dist = forecast(ss, N)
\end{lstlisting}
%
\begin{figure}[h]
	\centering
	\includegraphics[width=\columnwidth]{Figures/log_airline_forecast.pdf}
	\caption{Log-airline passengers historical data and forecasting.}
	\label{fig:log_airline_forecast}
\end{figure}
%
\begin{figure}[h]
	\centering
	\includegraphics[width=\columnwidth]{Figures/log_airline_components.pdf}
	\caption{Smoothed trend and seasonal components of the log-airline passengers time series.}
	\label{fig:log_airline_components}
\end{figure}

\subsection{Electricity consumption}

Another practical example is the study of monthly electricity consumption in a given area. In this example, we will use real data from a Brazilian distribution company. In addition, we will consider a temperature series, which is highly correlated to electricity consumption in Brazil, as an exogenous variable. A similar study using \texttt{StateSpaceModels.jl} was conducted in \cite{saavedra2018simulating}.

In this case, our objective is to simulate a large set of future scenarios that can be used as inputs in a stochastic optimization problem with the goal of obtaining the best contracting strategy for a distribution company. To this end, the function \texttt{simulate} will be used to perform Monte Carlo simulation. We will also use the square-root Kalman filter for this application.
%
\begin{lstlisting}[language = Julia]
# Specify the state-space model
model = structural(consumption, 12; X = temperature)
# Estimate the state-space model
ss = statespace(model; filter_type = SquareRootFilter{Float64})
# Number of months ahead to be simulated
N = 24
# Number of scenarios to be simulated
S = 1000
# Perform simulation
sim = simulate(ss, N, S)
\end{lstlisting}

The time series and the resulting simulation can be seen in Fig. \ref{fig:energy_simulation}. The future scenarios are graphically represented by their mean, which is identical to the forecast at each time period, as well as the 5\% and 95\% quantiles.
%
\begin{figure}[h]
	\centering
	\includegraphics[width=\columnwidth]{Figures/energy_simulation.pdf}
	\caption{Monthly energy consumption historical data and simulation of future scenarios.}
	\label{fig:energy_simulation}
\end{figure}

\subsection{Vehicle tracking}

Finally, in order to illustrate one application that does not fall into any of the predefined models, thus requiring a user-defined model, let us consider an example from control theory. More precisely, we are going to use \texttt{StateSpaceModels.jl} to track a vehicle from noisy sensor data. In this case, $y_t$ is a $2 \times 1$ observation vector representing the corrupted measurements of the vehicle's position on the two-dimensional plane in instant $t$. Since sensors collect the observations with the presence of additive Gaussian noise, we need to filter the observation in order to obtain a better estimate of the vehicle's position.

The position and speed in each dimension compose the state of the vehicle. Let us refer to $x_t^{(d)}$ as the position on the axis $d$ and to $\dot{x}^{(d)}_t$ as the speed on the axis $d$ in instant $t$. Additionally, let $\eta^{(d)}_t$ be the input drive force on the axis $d$, which acts as state noise. For a single dimension, we can describe the vehicle dynamics as
\begin{equation}
\begin{aligned}
    & x_{t+1}^{(d)} = x_t^{(d)} + \Big( 1 - \frac{\rho \Delta_t}{2} \Big) \Delta_t \dot{x}^{(d)}_t + \frac{\Delta^2_t}{2} \eta_t^{(d)}, \\
    & \dot{x}^{(d)}_{t+1} = (1 - \rho) \dot{x}^{(d)}_{t} + \Delta_t \eta^{(d)}_t,
\end{aligned}\label{eq_control}
\end{equation}
where $\Delta_t$ is the time step and $\rho$ is a known damping effect on speed.

We can cast the dynamical system (\ref{eq_control}) as a state-space model in the following manner:
\begin{align*}
    y_t &= \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix} \alpha_{t+1} + \varepsilon_t, \\
    \alpha_{t+1} &= \begin{bmatrix} 1 & (1 - \tfrac{\rho \Delta_t}{2}) \Delta_t & 0 & 0 \\ 0 & (1 - \rho) & 0 & 0 \\ 0 & 0 & 1 & (1 - \tfrac{\rho \Delta_t}{2}) \\ 0 & 0 & 0 & (1 - \rho) \end{bmatrix} \alpha_{t} + \begin{bmatrix} \tfrac{\Delta^2_t}{2} & 0 \\ \Delta_t & 0 \\ 0 & \tfrac{\Delta^2_t}{2} \\ 0 & \Delta_t \end{bmatrix} \eta_{t},
\end{align*}
where $\alpha_t = (x_t^{(1)}, \dot{x}^{(1)}_{t}, x_t^{(2)}, \dot{x}^{(2)}_{t})^{\top}$ and $\eta_t = (\eta^{(1)}_t, \eta^{(2)}_t)^{\top}$.

\vspace{0.3cm}

We can formulate the vehicle tracking problem in the \texttt{StateSpaceModels.jl} framework as:

\begin{lstlisting}[language = Julia]
# State transition matrix
T = kron(Matrix{Float64}(I, p, p), [1 (1 - ρ * Δ / 2) * Δ; 0 (1 - ρ * Δ)])
# Input matrix
R = kron(Matrix{Float64}(I, p, p), [.5 * Δ^2; Δ])
# Output (measurement) matrix
Z = kron(Matrix{Float64}(I, p, p), [1 0])
# User defined model
model = StateSpaceModel{Float64}(y, Z, T, R)
# Estimate vehicle speed and position
ss = statespace(model)
\end{lstlisting}

In this example, we define the noise variances $H$ and $Q$, generate the noises and simulate a random vehicle trajectory using the state-space equations.
%
\begin{lstlisting}[language = Julia]
# Generate random actuators
Q = .5 * Matrix{Float64}(I, q, q)
η = MvNormal(zeros(q), Q)
# Generate random measurement noise
H = 2. * Matrix{Float64}(I, p, p)
ε = MvNormal(zeros(p), H)
# Simulate vehicle trajectory
α = Matrix{Float64}(undef, n + 1, m)
y = Matrix{Float64}(undef, n, p)
for t in 1:n
    y[t, :] = Z * α[t, :] + rand(ε)
    α[t + 1, :] = T * α[t, :] + R * rand(η)
end
\end{lstlisting}

An illustration of the results can be seen in Fig. \ref{fig:vehicle_tracking}. It can be seen that the measurements are reasonably noisy when compared to the true position. Furthermore, the estimated positions, represented by the smoothed state, effectively estimate the true positions with small inaccuracies.
%
\begin{figure}[h]
	\centering
	\includegraphics[width=\columnwidth]{Figures/vehicle_tracking.pdf}
	\caption{Vehicle tracking using a state-space model.}
	\label{fig:vehicle_tracking}
\end{figure}

\section{Conclusion} \label{sec:conclusion}

\texttt{StateSpaceModels.jl} is a flexible package for time-series modeling, forecasting, and simulating that is fully implemented in Julia. The package contains an implementation of the Kalman filter and smoother as well as their square-root variants. Additionally, users have the ability to use any implemented filter or optimization method. Missing observations in the form of \texttt{NaN} values are automatically treated in the filtering and smoothing steps.

Besides comprising several predefined classical models, it is also possible to define any linear model with the \texttt{StateSpaceModel} constructor. Forecasting and Monte Carlo simulation of future scenarios are also available. Finally, the package documentation contains a manual with straightforward examples that are simple to reproduce.

\input{bib.tex}

\end{document}


\chapter{Neural Networks}
\label{cha:neural_networks}
\epigraph{In the recent years Neural Networks (NN) have solved more and more
  tasks that have previously been too difficult or simply too tedious to solve
  with traditionally coded algorithms. They have been successfully applied to a
  variety of problems such as pattern recognition, image classification and
  prediction.  As the NN algorithms learn from data, they seem to be good
  candidates for finding anomalies without any prior knowledge about the
  given dataset, as long as it is big enough. By applying online learning
  algorithms that learn and adapt continuously it should in theory be possible
  to create an automated, adaptive outlier detection.\\ This chapter describes
  the mechanics of NNs from the ground up. Starting with the widely used
  \emph{feedforward networks} (FNN), we go on to \emph{recurrent neural nets}
  (RNN). Similar to biological neural networks, they have cyclic connections,
  and are capable of processing sequences. After that, a special kind of RNN is
  introduced, that dramatically cuts computational costs and solves some
  notorious problems that arise during RNN training. In addition to the general
  difficulty of training NNs, they often rely on hyper-parameters, that are
  typically set by manually tuning the network.  This chapter is closed with a
  brief description of hyper-parameter optimization techniques which were used
  to automate this task.
}

\section{Feedforward Neural Networks}
\label{sec:feedforward_neural_networks}

The idea for artificial neural networks is based on the human brain,
which is a highly complex, non-linear, and parallel computer.  Just like the
brain, neural networks are constructed from small units that are connected to each other
with weights. In analogy to its biological counterpart these units are also
called \emph{neurons}. Older literature also often refers to them as
\emph{perceptrons}.  They are capable of processing and passing on incoming
information.
Traditional \emph{feedforward neural networks} (FNN) implement static
input-output mappings, which mathematically makes them pure functions of the
input signals. Figure \ref{fig:perceptron} shows a simple schematic of a single
neuron.  It generally has $n$ inputs $u_j$ and one output.  Each of the inputs
has an assigned weight $w_j$, which determines the contribution of a given
input to the output.  The output of a neuron will further be referred to as
\emph{activation} $y$:
\begin{equation}
  \label{eq:ffn_eq}
  y = \varphi \left( \sum_j (w_j u_j) + b  \right)
       = \varphi (\vec{w} \cdot \vt{u} + b).
\end{equation}
The sum over all the inputs multiplied by their weights can conveniently be
represented by a dot product.  Often a bias term $b$ is included, which can be
taken as a measure of how easy it is to activate the neuron.  The function
$\varphi$ is called activation function and can have various different forms,
from a simple binary function to an arbitrary (non-linear), monotonically
increasing function.  A frequently used activation function in FNNs is the
sigmoid function
\begin{equation}
  \sigma (z) = \frac{1}{1 + e^{-z}},
\end{equation}
which is shown in Fig.~\ref{fig:sigmoid}. To create a \emph{layer} of neurons
they are simply stacked on top of each other:
\begin{equation}
  \vt{y} = \varphi (\wmatr{} \vt{u} + \vec{b}).
\end{equation}

\begin{figure}
  \begin{minipage}{.42\textwidth}
    \centering
    \Perceptron
    \vspace{4.5mm}
    \caption{Schematic of a neuron [\cite{Nielsen2015}].
    The activation function is represented by the circle.}
    \label{fig:perceptron}
  \end{minipage}
  \hspace{.02\textwidth}
  \begin{minipage}{.54\textwidth}
    \centering
    \includegraphics[width=\linewidth]{sigmoid_tanh.pdf}
    \caption{Sigmoid function $\varphi$. Typically used as an activation function
    in feedforward neural nets.}
    \label{fig:sigmoid}
  \end{minipage}
\end{figure}

The final FNN with multiple layers (such as in
Fig.~\ref{fig:feedforward_network}) is created by feeding the output of one
layer to another one until the last layer of the network is reached.  The
\emph{universal approximation theorem} [\cite{uni_approx_theorem}] states the
ability of FNNs to approximate an arbitrary non-linear function $f$
\begin{equation}
   \vt{d} = f(\vt{u})
\end{equation}
with arbitrary precision. This means that given any $\epsilon > 0$ we can find
an FNN $F$, such that
\begin{equation}
  ||F(\Theta, \vt{u}) - f(\vt{u})||_2 < \epsilon
\end{equation}

for all possible inputs $\vt{u}$. The exact form of the approximation $F$
depends on the network architecture, but generally it depends on a set of
parameters $\Theta$ called weights (and biases), which represent several layers
of the FNN.  Of course, the universal approximation theorem does not say
anything about the practical learnability of a task, which will be discussed in
Sec.~\ref{sub:gradient_descent}.

Each layer of an FNN can be represented by a matrix, and therefore only has the
computational expressibility of linear functions. This linearity is only broken
by the activation function. Without the non-linear activation an arbitrary
number of layers would not be more effective than a single one.  Through the
organization in layers FNNs are able to model not only arbitrary functions but
also to separate datasets that are not linearly separable.
Fig.~\ref{fig:feedforward_network} shows an FNN where the first (input) layer
represents the data that is fed into the network, followed by one or more
hidden layers, and an output layer.  Hidden layers act as a non-linear
transform that distorts the input in such a way that its classes become
linearly separable by the output layer.  An interactive explanation of how this
works can be found in a blog post by [\cite{colah_topology}].

\begin{figure}
  \centering
  \FeedForwardNet{2}
  \caption{Fully connected feedforward neural network with a single hidden
    layer and two output neurons.}
  \label{fig:feedforward_network}
\end{figure}

Every network that has more than two or three hidden layers is typically called
a \emph{deep neural network}. It is fundamentally not different from the basic
architecture of the described feedforward networks but holds the potential of
more powerful transformations of the input. The type of neural network that is
shown in Fig.~\ref{fig:feedforward_network} is called \emph{feedforward
network}, because the input data is entering the network at the input layer and
then passed through the network towards the output layer.  Feedforward nets are
often applied to classification tasks, such as the recognition of a certain
shape in an image.

The goal of the machine learning approach is to find a weight configuration
that captures the \emph{essence} of the presented dataset, meaning that it
generalizes well, including over inputs that it has not been trained with.
This is typically approached by some variation of Gradient Descent (GD). GD
algorithms try to minimize a certain loss or cost function with respect to a
given weight configuration (more detailed description in
Sec.~\ref{sub:gradient_descent}).  However, the pursued generalization of the
network is highly dependent on the training dataset. It should ideally include
the total variability of possible inputs.  This is very hard to achieve in
practice and the available datasets are typically split randomly into three
categories: \emph{training}, \emph{validation}, and \emph{test} set.  The
training set is used to optimize the network. Its loss is minimized by a GD
algorithm. The validation set is used to determine the performance of the
network on inputs that are not included in the training set.  The optimization
should be stopped as soon as the error on the validation set is not decreasing
any more in order to reduce the risk of overfitting. This is one of many
regularization methods that are applied in ML in order to achieve a
generalization over previously unseen datapoints. Because the information of
the validation set is leaking into the network via the early stopping
criterion, a third dataset is needed to evaluate the actual performance and
generalization of the network.  This third set is called the test set.

Feedforward networks outperform conventional algorithms at tasks such as image
recognition and other classification and pattern recognition.  They are,
however, not very suitable to model time series, as they cannot model correlations
of previous inputs. This means that they are not designed for the prediction of
sequences.  To be able to process sequences and make predictions,
\emph{recurrent} weight connections are introduced in
Sec.~\ref{sec:recurrent_neural_networks}.  Despite the compelling results that
can be achieved by FNNs, they have a few drawbacks.  Backpropagation algorithms
typically need very large amounts of data to train the network. The gradient
descent steps have to be small enough to not jump over the desired minima,
which leads to very long training times.  Additionally the nature of the
training data can lead to biases of the resulting network if it does not
properly represent the possible parameter space.  It is hard to infer
afterwards how the network came to a specific result, as the weight matrices do
not represent a traceable way of reasoning or logic.  A trained network is like
a black box, which does not come with an obvious way of determining how or why
a certain classification was made.

\subsection{Convolutional Neural Nets}%
\label{sub:convolutional_neural_nets}

A breakthrough in image classification performance was achieved by the
introduction of a subtype of FNNs, the \emph{convolutional neural networks}
(CNN), which can leverage the spatial structure of the input data.  They
combine the concept of filters from signal processing with the ML approach by
learning their own filters (often referred to as kernels as well) for feature
detection in images.  The convolutional operation of a given filter $F$ and an
image $H$ is defined by:
\begin{align}
  G &= H * F \text{, where}\\
  G[i,j] &= \sum_{u=-k}^k \sum_{v=-k}^k H[i-u, j-v]F[u,v]
\end{align}

A well known example of a filter for edge detection looks like this:
\begin{equation}
  F = \begin{bmatrix}
    -1 & -1 & -1 \\
    -1 & 8  & -1 \\
    -1 & -1 & -1
  \end{bmatrix}
\end{equation}

Applying it to a (black and white) image $H$
\begin{equation}
  H = \begin{bmatrix}
	  1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\
	  1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\
	  1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\
	  1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\
	  1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\
	  1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\
	\end{bmatrix}
\end{equation}

and padding the convoluted image $G$ to obtain the original dimensions results
in a filtered image $R$, which is zero everywhere except at the location of the
edge:

\begin{equation}
  \newcommand{\nan}{-}
  \newcommand{\minus}{\text{-}}
	R = \begin{bmatrix}
    \nan & \nan & \nan & \nan & \nan & \nan  & \nan & \nan & \nan\\
    \nan & 0 & 0 & 0 & 3 & \minus 3 & 0 & 0 & \nan\\
    \nan & 0 & 0 & 0 & 3 & \minus 3 & 0 & 0 & \nan\\
    \nan & 0 & 0 & 0 & 3 & \minus 3 & 0 & 0 & \nan\\
    \nan & 0 & 0 & 0 & 3 & \minus 3 & 0 & 0 & \nan\\
    \nan & \nan & \nan & \nan & \nan & \nan  & \nan & \nan & \nan\\
  \end{bmatrix}
\end{equation}

A convolutional layer consists of multiple kernels, which are learned via GD.
Each kernel can be thought of as a neuron that sees only a small part of the
input image at a time and slides over the whole image. Its output therefore
represents the local structure of said part of the image, which makes
a CNN capable of exploiting spatial structures in images (see schematic in
Fig.~\ref{fig:conv_layer}).  The kernels which are sliding over the input
images naturally lead to a translation invariance of the learned patterns
[\cite{lecun1995convolutional}]. A pattern that is recognized by a filter in one
region of the image can just as well be detected somewhere else, which stands
in contrast to fully connected layers for which the same pattern in different
parts of an image looks completely different.  Scale invariance can be achieved
by using multiple kernels of different size.  Feedforward nets, which are fully
connected to the input cannot leverage the spatial information as efficiently.
Additionally, CNNs decrease the number of parameters in comparison to a normal
FNN, which reduces the risk of overfitting and thus leads to a better
generalization of NNs.
\begin{figure}
  \centering
  \includegraphics[width=0.4\linewidth]{conv_layer.png}
  \caption{The 2D input image on the left is transformed into a 3D output
  volume by applying multiple convolutions to the input [\cite{conv_layer_wiki}].}
  \label{fig:conv_layer}
\end{figure}

An interesting example of a CNN outside the realm of image classification is
AtomNet [\cite{dzamba1510atomnet}], which is used to predict bioactivity of
molecules for drug discovery by exploiting the local structure of biochemical
interactions.

\subsection{Gradient Descent}
\label{sub:gradient_descent}

The most common technique to train neural networks is Gradient Descent.
It is based on minimizing a \emph{loss} (cost or penalty) function $\mathcal{L}$
\begin{equation}
  \label{eq:batch_loss}
  \mathcal{L}(\Theta) = \sum_{\vt{u} \in \mathcal{U}}
                        ||F(\Theta, \vt{u}) - \vt{d}||_2,
\end{equation}
which defines how close the network is to producing the
desired results.  The loss function that is used throughout this thesis is
given by Eq.~\ref{eq:batch_loss}, where the target outputs, given a certain
input $\vec{u}$ out of all training examples $\mathcal{U}$, are denoted by
$\vt{d}$.

Initially, the weights of the network are set randomly and are to be adjusted
in the optimization phase, also called learning or training.  The loss is, of
course, dependent on all the weights and biases, which is where the gradient
descent comes into play.  The partial derivatives of the loss function are
taken with respect to all the weight and biases of the network to find the
gradient which points in the direction of steepest descent.  With this
calculated direction we can step the weights towards the nearest local minimum
and like that gradually increase the performance of the network.
\begin{equation}
  \label{eq:gradient_descent}
  \Theta' = \Theta + \eta \dd{\mathcal{L}}{\Theta}
\end{equation}
The size of the steps is defined by the \emph{learning rate} $\eta$, which has
to be chosen carefully.  If it is too large, the algorithm will oscillate
around the desired minimum, but if chosen too small, the training times will
become too long.  Several adaptive gradient computation algorithms address this
issue. One promising algorithm is called \emph{Adam} (Adaptive Moment
Estimation). Adam combines adaptive gradient descent methods with
momentum based algorithms [\cite{ADAM}]. Momentum-based optimizers add a fraction
of the previously used weight update to simulate an acceleration of the
gradient descent.  The method of applying the gradient descent algorithm to
multi-layer (deep) neural networks is called \emph{backpropagation} (BP),
because the gradient calculation is started at the last layer and iteratively
propagated back through the whole network by applying the chain rule. A more
detailed description of BP is given in Sec.~\ref{sub:backpropagation}.

There are three different variations of gradient descent, which only differ in
the way the sum in Eq.~\ref{eq:batch_loss} is interpreted.  Summing over all
available training examples $\vt{u}$, namely the whole \emph{batch}, is called
{\em batch GD}. Calculating the loss only for a single randomly chosen example,
is called \emph{stochastic gradient descent} (SGD). The compromise of the two,
mini-batch GD, uses a subset of the training examples, performs a weight
update, and then goes on to the next mini-batch.  Both stochastic and
mini-batch GD end up calculating an approximate loss $L$ from a subset $U
\subset \mathcal{U}$:
\begin{equation}
  \mathcal{L}(\Theta) \approx L(\Theta) =
    \sum_{\vt{u} \in U} ||F(\Theta, \vt{u}) - \vt{d}||_2,
\end{equation}

which also leads to an approximated gradient. The updates that are performed
with the approximate gradient are hoped to enable the optimizer to jump out of
shallow local minima and saddle points, which is discussed further in the next
paragraph.

\subsubsection{Local Minima of the Error Surface}%
\label{ssub:local_minima_of_the_error_surface}

As BP is a gradient-based method, there is no guarantee at all, that the
algorithm will find the global minimum of the error surface.  To illustrate
this, Fig.~\ref{fig:error_surface_bgd} depicts a two dimensional error surface
with a global and a local minimum.  Depending on where on the surface the
optimization is started, it ends up in the local or global minimum.  It was
shown that for linear activation functions, the error surface contains only a
single minimum, the global one, with all other locations of zero gradient being
saddle points [\cite{BALDI}]. Momentum-based optimizers can find the global
minimum quite efficiently in such cases. However, this cannot be generalized to
the practically used NNs that almost exclusively use non-linear activations.
Optimizers that include momentum like the Adam optimizer can still sometimes
yield better results, as depicted in Fig.~\ref{fig:error_surface_bgd}.  The two
plots both show a two dimensional error surface, where the coordinates $x$ and
$y$ are regarded as the weights that have to be optimized.  The black, dotted
lines show different runs with varying initial values of $x$ and $y$.  The left
plot shows the convergence paths of plain batch GD algorithm as described
above, which converges towards the global minimum in three out of five cases.
As GD is a purely gradient based method, the path always advances in the
direction of steepest descent.  With the more advanced Adam optimizer, the
global minimum is found in four out of five cases. The momentum towards the
global minimum that is gained in the beginning carries the optimization away
from the local minimum.  Of course there are numerous cases where this approach
will not yield a global minimum. Additionally the Adam algorithm is much more
computationally expensive than plain GD and needs more steps, as can be seen
from the denser dots on the black lines.
\begin{figure}
  \begin{minipage}[b]{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{gradient_descent.pdf}
    GD
  \end{minipage}
  \hspace{.02\textwidth}
  \begin{minipage}[b]{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{adam.pdf}
    Adam
  \end{minipage}
  \caption{On the left we can see a plain GD optimizer while on the right the
    Adam optimizer was used.  Depending on the starting values of the two
    variables, there are cases in which the optimization ends in the local
    minimum on the right. The GD optimizer needs fewer steps, while Adam finds
    the global minimum in one more case.}
  \label{fig:error_surface_bgd}
\end{figure}

\begin{figure}
  \begin{minipage}[b]{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{stochastic_gradient_descent.pdf}
    stochastic GD
  \end{minipage}
  \hspace{.02\textwidth}
  \begin{minipage}[b]{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{stochastic_adam.pdf}
    stochastic Adam
  \end{minipage}
  \caption{Convergence paths of stochastic GD algorithms. The noisy paths are
    due to the approximation of the gradient through mini-batches.}
  \label{fig:error_surface_sgd}
\end{figure}
To date, the most commonly used technique to encourage GD to converge to the
global minimum is mini-batch gradient descent.  Instead of evaluating the
\emph{true} loss of the whole training set as indicated by
Eq.~\ref{eq:batch_loss}, SGD adapts the weights after every mini-batch $X$.
This leads to an approximation of the gradient and causes oscillations in the
convergence path, which should make it more probable to escape local minima.
Fig.~\ref{fig:error_surface_sgd} again compares plain GD and Adam, but now with
the perturbing effect of evaluating an approximate loss. The number of training
steps that have to be taken increase significantly, but plain SGD is able to
find the global minimum in four out of five cases. Stochastic Adam almost
reaches the minimum in all five cases, but seems to get stuck right before
the global minimum in the last case.  The conclusion we can draw from the above
examples is that care has to be taken with respect to the convergence of NNs.
Different optimizers, gradient descent variations, and initializations can
yield vastly different results. However, experience has shown that in most
pattern recognition tasks, SGD algorithms can find a local minimum that yields
satisfactory performance, even though it is not guaranteed that this minimum is
global.  In most applications, an increase in the network connections is
supposed to create paths around suboptimal local minima [\cite{rumelhart1986}].
Additionally, in most applications the training is actually stopped early,
meaning at the time the validation error does not decrease further.  This is
done because a network that is fitted perfectly to the training data would most
probably not generalize well over new inputs.  In most cases it is therefore
acceptable or even desirable to remain in a \emph{good} local minimum that
results in a good generalization of the network.

\subsection{Backpropagation}%
\label{sub:backpropagation}

Backpropagation is the basic algorithm that carries out the GD minimizations
that were introduced previously. It was invented in the 1970s but not widely
used until the paper by~[\cite{rumelhart1986}], which marks a breakthrough in the
field of machine learning.  For an easy to read but in depth description of the
subtleties of BP the reader is referred to the book \emph{Neural Networks and
Deep Learning} by [\cite{Nielsen2015}].

Fundamentally, BP is nothing more than the application of the chain rule to the
cost function of a neural network, which can be solved by an \emph{automatic
differentiation} (AD) algorithm. In fact, BP is a special case of AD called
\emph{reverse mode} AD~[\cite{autodiff}].  To illustrate the BP algorithm we will
consider a network with $L$ feedforward layers, where the activations of the
first layer are just the inputs to the network ($\vec{u} = \vec{y}_1$) and the
last layer contains the network outputs $\vec{y}_L$. The components $y^j_l$ of
the vector $\vec{y}_l$ that contains all activations of a layer $l$ are
calculated based on the previous layer:
\begin{equation}
\begin{aligned}
  z^j_l &= \sum_k w^{jk}_l y^k_{l-1} + b^k_L \\
  y^j_l &= \varphi(z^k_l) \label{eq:forward_pass},
\end{aligned}
\end{equation}
where $\vec{z}_l$ are called the \emph{weighted inputs}.  The system above may
be altered such that a given activation depends on any \emph{previous} layer
activation, but \emph{not} on activations of any \emph{next} layer. This step is
named \emph{forward pass}.  During the forward pass, all activations are
calculated starting from the first layer. The goal of the \emph{backward pass}
is to adjust all the weights $w^{jk}_l$ (and biases $b^k_L$) such that the loss $L$
\begin{equation}
  L = \frac{1}{2}\sum_k (d^k - y^k_L)^2
\end{equation}
of a single input example $\vec{u}$ is minimized. BP solves this by iteratively
calculating the gradients needed for Eq.~\ref{eq:gradient_descent} starting
from the last layer.  Before directly calculating the necessary gradients it is
easier to compute the \emph{error} $\delta_L$ of the weighted inputs for each
unit $j$ of the last layer:
\begin{equation}
  \delta^j_L = \dd{L}{z^j_L} = \frac{1}{2}\sum_k \dd{}{z^j_L}(d^k - y^k_L)^2
             = \dd{L}{y^j_L}\dd{y^j_L}{z^j_L}
             = (d^j - y^j_L) \varphi'(z^j_L) \label{eq:delta_last_layer},
\end{equation}
where the sum vanishes because $y^k_L$ only depends on $z^j_L$ if $j = k$.
Now the error of any layer $l$ can be expressed in terms of the error of the
next layer $l+1$:
\begin{equation}
  \delta^j_{l} = \dd{L}{z^j_l} = \sum_k \dd{L}{z^j_{l+1}}\dd{z^j_{l+1}}{z^j_l}
  = \sum_k \delta^j_{l+1} \dd{z^j_{l+1}}{z^j_l}
  = \sum_k \delta^j_{l+1} w^{kj}_{l+1} \varphi'(z^j_l). \label{eq:delta_other_layer}
\end{equation}

The desired error gradients can be found to be:
\begin{equation}
\begin{aligned}
  \dd{L}{b^j_l} &= \delta^j_l, \\
  \dd{L}{w^{jk}_l} &= y^j_{l-1} \delta^j_l. \label{eq:error_grads}
\end{aligned}
\end{equation}
They can be used to incrementally update the weights and biases according
to Eq.~\ref{eq:gradient_descent}.

\subsubsection{The Backpropagation Algorithm}%
\label{ssub:the_backpropagation_algorithm}

\begin{enumerate}
  \item \emph{Input}: $\vec{u}$ is the activation vector of the first layer
    $\vec{y}_1$
  \item \emph{Forward pass}: Compute activations $\vec{y}_l$ for each
    $l=2,3,...,L$ (Eq.~\ref{eq:forward_pass})
  \item \emph{Output error}: Compute error $\delta_L$ of last layer
    (Eq.~\ref{eq:delta_last_layer})
  \item \emph{Backpropagate}: Compute $\delta_{l-1}$ based on error of layer
    $l$ (Eq.~\ref{eq:delta_other_layer})
  \item \emph{Output}: Obtain the gradients of the cost function
    (Eq.~\ref{eq:error_grads})
\end{enumerate}


\documentclass[letterpaper,10pt]{article}
\usepackage{epsfig,comment,amssymb,listings,amsmath}
\setlength{\parskip}{1.8mm}
\setlength{\parindent}{0mm}
\begin{document}
%\date{}
\title{\Large \bf Commutation relations in Fock space}

\author{
{\rm Daniel Tang}\\
}

\maketitle

% Use the following at camera-ready time to suppress page numbers.
% Comment it out when you first submit the paper for review.
%\thispagestyle{empty}

\section{Commutation identities}
%##########################################################

\begin{equation}
AB = BA + [A,B]
\end{equation}

\begin{equation}
[A, BC] = [A,B]C + B[A,C]
\end{equation}

\begin{equation}
[AB, C] =  A[B,C] + [A,C]B
\end{equation}

\begin{equation}
[A,B + C] = [A,B] + [A,C]
\label{sumcommute}
\end{equation}
\begin{equation}
[A,mB] = m[A,B]
\end{equation}
where $m$ is a scalar multiplicative constant.
\begin{equation}
[A,B] = -[B,A]
\end{equation}

\begin{equation}
[ab,AB] = a[b,A]B + [a,A]bB + Aa[b,B] + A[a,B]b
\end{equation}
\begin{equation}
 = a[b,A]B  + [a,A]Bb + aA[b,B] + A[a,B]b
\end{equation}

\begin{equation}
[A,B^n] = \sum_{q=1}^n B^{q-1}[A,B]B^{n-q}
\end{equation}

\begin{equation}
[A,B^{n+1}] = [A,B]B^n + [A,B^n]B - [[A,B^n],B]
\end{equation}

\begin{multline}
[[A,C],[B,D]]=[[[A,B],C],D]+[[[B,C],D],A]
\\+[[[C,D],A],B]+[[[D,A],B],C]
\end{multline}
(source Wikipedia "Commutator")

\begin{equation}
[[A,H],[B,H]]=[[[A,B],H],H]+[[[B,H],H],A]-[[[A,H],B],H]
\end{equation}
so
\begin{multline}
[[^nA,H],[^mB,H]]=[^2[[^{n-1}A,H],[^{m-1}B,H]],H]
\\+[[^{m+1}B,H],[^{n-1}A,H]]-[[[^nA,H],[^{m-1}B,H]],H]
\end{multline}

If we define $[^nA,H] = [\dots [[A,H],H] \dots ,H]$ to be the $n$-fold commutation:

\begin{equation}
[^n AB,H] = \sum_{m=0}^n {n \choose m}[^mA,H][^{n-m}B,H]
\end{equation}

\begin{equation}
[^n ABC,H] = \sum_{a+b+c = n} \frac{n!}{a!b!c!}[^aA,H][^bB,H][^cC,H]
\end{equation}

\section{Annihilation and creation operators}
%##########################################################

Beginning with

\[
[a_\psi,a_\phi^\dag] = a_\psi a_\phi^\dag - a_\phi^\dag a_\psi =
\begin{cases}
 1 & \text{if }\psi=\phi \\
 0 & \text{otherwise} \\
 \end{cases}
\]

\section{$a^\dag$ and $a$ operators}
%##########################################################

\begin{equation}
[a^m, a^\dag] = ma^{m-1}
\end{equation}

\begin{equation}
[a,a^{\dag m}] = ma^{\dag(m-1)}
\end{equation}
the above holds for all m, even -ve.
\begin{equation}
[a^-,a^m] = ((a^- + a)^m - a^m)a^-
\end{equation}

\begin{equation}
[a^n,a^{\dag m}] = \sum_{q=1}^{\min(m,n)} \frac{m!n!}{q!(m-q)!(n-q)!} a^{\dag m-q}a^{n-q}
\end{equation}

\begin{equation}
[a^{\dag p}a^m, a^{\dag q}a^n] =
a^{\dag p}[a^m, a^{\dag q}] a^n -
a^{\dag q} [a^n, a^{\dag p}]a^m
\end{equation}

\section{$a^-$ operator}
%##########################################################

Define the $a^-$ operator such that
\[
a^-a^\dag = I
\]
where $I$ is the identity operator. Given this we can see immediately that $[a^\dag,a^-a^\dag] = 0$, so
\[
a^\dag a^-a^\dag -a^-a^\dag a^\dag = [a^\dag,a^-]a^\dag = 0
\]
So, for all states, $S$ other than the ground state,
\[
[a^\dag,a^-]S = 0
\]
For the ground state, $\emptyset$, in order to ensure $[a^\dag,a^-]\emptyset = 0$ we define
\[
a^\dag (a^-\emptyset) = \emptyset
\]
However, such terms as $a^-\emptyset$ will never arise through annihilation operators as they will always be multipllied by zero.

\section{$L_{ir}$ operator}
%##########################################################

Changes $\lambda_i$ to $(1-r)\lambda_i$

\begin{equation}
[L_{ir}, a_i^\dag] = 0
\end{equation}

\begin{equation}
[L_{ir}, a_i] = r\lambda_i L_{ir}
\end{equation}

\begin{equation}
[L_{ir}, a_i^n] =  L_{ir}(a_i^n - (a_i-r\lambda_i)^n)
\end{equation}

\section{$g_{ir}$ operator}
%##########################################################

Multiplies each basis by $(1-r)^{\Delta_i}$

\[
g_{ir}D_0 = 1
\]

\begin{equation}
[a_i^\dag, g_{ir}] = ra_i^\dag g_{ir}
\end{equation}

\begin{equation}
[a_i, g_{ir}] = \frac{r(\lambda - a_i)}{1-r}g_{ir}
\end{equation}
so
\begin{equation}
g_{ir}a_i^{\dag n} = (1-r)^na_i^{\dag n}g_{ir}
\end{equation}
and
\begin{equation}
[g_{ir}, a_i^{\dag n}] = ((1-r)^n - 1)a_i^{\dag n}g_{ir}
\end{equation}
also
\begin{equation}
g_{ir}a_i = \frac{a_i-r\lambda_i}{1-r}g_{ir}
\end{equation}
so
\begin{equation}
g_{ir}a_i^n = \left(\frac{a_i-r\lambda_i}{1-r}\right)^ng_{ir}
\end{equation}
\begin{equation}
g_{ir}a_i^n = (1-r)^{-n}\sum_{m=0}^n{n \choose m} a_i^m(-r\lambda_i)^{n-m}g_{ir}
\end{equation}

and
\begin{equation}
[g_{ir},a_i^n] = \left((1-r)^{-n}\sum_{m=0}^n{n \choose m} a_i^m(-r\lambda_i)^{n-m} - a_i^n\right) g_{ir}
\end{equation}
so
\begin{equation}
[g_{ir},a_i^{\dag n}a_i^m] = \left((1-r)^{n-m}\sum_{l=0}^m{m \choose l} a_i^{\dag n}a_i^l(-r\lambda_i)^{m-l} - a_i^{\dag n}a_i^m\right) g_{ir}
\end{equation}

\section{$Lg_{ir}$ operator}
%##########################################################

We can join the $L$ and $g$ operators into a compound operator that multiplies bases by $(1-r)^\Delta_i$ and multiplies $\lambda_i$ by $(1-r)$. It has the properties:

\begin{equation}
[Lg_{ir}, a_i^\dag] = -ra_i^\dag Lg_{ir}
\end{equation}

\begin{equation}
[Lg_{ir}, a_i] = \frac{ra_i}{1-r} Lg_{ir}
\end{equation}

So
\begin{equation}
Lg_{ir}a_i^{\dag n}a_i^m = (1-r)^{n-m}a_i^{\dag n}a_i^mLg_{ir}
\end{equation}
and
\begin{equation}
[Lg_{ir},a_i^{\dag n}a_i^m] = ((1-r)^{n-m} - 1)a_i^{\dag n}a_i^mLg_{ir}
\end{equation}

%{\footnotesize \bibliographystyle{acm}
%\bibliography{sample}}

%\theendnotes

\end{document}


\documentclass[twoside]{MATH77}
\usepackage{multicol}
\usepackage[fleqn,reqno,centertags]{amsmath}
\begin{document}
\begmath 12.1 One-Dimensional Table Look Up, Interpolation, \hbox{and Differentiation}

\silentfootnote{$^\copyright$1997 Calif. Inst. of Technology, \thisyear \ Math \`a la Carte, Inc.}

\subsection{Purpose}

Given a table, $(x_i$, $y_i)$, of independent variable values and the
corresponding dependent variable values, this subroutine finds the
points in the table closest to a given value of the independent
variable and uses these points to interpolate for the corresponding
value of the dependent variable.  Error estimates, Hermite
interpolation (i.e.\ using $(x_i$, $y_i$, $y_i^\prime)$), different
look up methods, and the computation of derivative information are
available.

\subsection{Usage}

\subsubsection{Program Prototype, Single Precision}

\begin{description}
\item[INTEGER]  \ {\bf NTAB, NDEG, LUP, IOPT}$(\geq k_1)$\newline
[$k_1 \geq 2$ depends on options used.]

\item[REAL]  \ {\bf X, Y, XT}$(\geq $NTAB){\bf , YT}$(\geq $NTAB){\bf ,\newline
EOPT}$(\geq k_2$) [$k_2 \geq 1$ depends on options used .]
\end{description}
\begin{center}
\fbox{\begin{tabular}{@{\bf }c}
CALL SILUP (X, Y, NTAB, XT, YT,\\
NDEG, LUP, IOPT, EOPT)\\
\end{tabular}}
\end{center}

\subsubsection{Argument Definitions}

\begin{description}
\item[X]  [in] Independent variable where value of interpolant is desired.

\item[Y]  [out] Value of interpolant.

\item[NTAB]  [in] Number of points in the table, $\geq \text{NDEG}+k$,
  where $k=1$ if NDEG is odd and 2 if NDEG is even, and must be one
  bigger than this if an error estimate is requested.

\item[XT()]  [in] Array of independent variable values, XT($i)=x_i$. Must be
monotone increasing or monotone decreasing, but this is not checked
for. (XT(I$)=$ XT(I+1) is permitted, but has a special meaning, see
Section~C.) If the $x_i$ are equally spaced one should set LUP (below) to~3,
and provide $x_1$ in XT(1) and $x_2-x_1$ in XT(2).

\item[YT()]  [in] Array of dependent variable values. YT($i)=y_i.$

\item[NDEG]  [in] Nominal degree of the polynomial to be used in the
interpolation. Require $0\leq $ NDEG $\leq 15.$

For NDEG $=0$, Y is set to YT($k)$, where $k$ minimizes $|\text{X} -x_k|.$

For NDEG odd, Y is the result of standard polynomial interpolation of degree
NDEG using NDEG + 1 points. The interpolating function is continuous, but
the first derivative is usually discontinuous at tabular points.

For NDEG $>$ 0 and even, Y is obtained from a function that is a linear
combination of two polynomials of degree NDEG. This function is a polynomial
of degree NDEG + 1 using NDEG + 2 points that interpolates all but the two
outer points. It has a continuous first derivative but the second derivative
is usually discontinuous at tabular points.

Usually $|error|$ in the interpolant will tend to decrease in a fairly
regular way for increasing values of NDEG until either errors due to a lack
of precision in YT, or the inherent instability of high degree polynomial
interpolation causes errors to get worse. If one wants a continuous first
derivative, even values of NDEG $>$ 0 should be used.

\item[LUP]  [inout] Defines the type of look up method. (Changed only if LUP
$\leq 0$ on input.)

\begin{itemize}
\item[$\leq 0$]  \ If LUP $=0$, start with a binary search, else do a
sequential search starting with an index of $-$LUP. On exit, LUP is set to $%
-k$, where $k$ minimizes $|$X $-$ XT($k)|.$
\item[$= 1$]  Binary search. Use this when accesses are not sequential, and
XT values are not close to being equally spaced.
\item[$= 2$]  Start a sequential search with an index $=[1.5+($NTAB$-$1)
(X $-$ XT(1))/(XT(NTAB) $-$ XT(1))]. Use this when points are almost equally
spaced and accesses are not sequential.
\item[$= 3$]  YT(I) corresponds to XT(1) + (I$-$1)*XT(2), no search is
required to do the look up and XT can have dimension XT(2). This is
recommended when the XT values are equally spaced since it takes the least
space and should usually be fastest.
\item[$= 4$]  Internal information connected with the $x_k$ values used in
the last interpolation is reused. (Don't use this value if there is any
chance for an intervening call to this subroutine.) Only YT should be
changed; IOPT is not examined. This option saves time when interpolating
components after the first of a vector valued function.
\end{itemize}

\item[IOPT()]  [inout] IOPT(1) is used to return a status as follows:

\begin{itemize}
\item[$-$10 ]  \ An option index is out of range.
\item[$-$9 ]  \ NTAB is outside allowed limits.
\item[$-$8 ]  \ NDEG is outside allowed limits.
\item[$-$7 ]  \  LUP $>$ 3 when program was not ready for it.
\item[$-$6 ]  \ Option 3 (compute derivatives), has requested more than
15~derivatives.
\item[$-$5 ]  \ LUP = 3, and XT(2) = 0.
\item[$-$4 ]  \ XT(1) = XT(NTAB), and NTAB is not~1.
\item[$-$3 ]  \ Bad points (see option 6) mean only 0 or 1 points were
available for interpolation..
\item[$-$2 ]  \ There is only one table entry; the estimated error that was
requested was not computed.
\item[$-$1 ]  \ The accuracy requested was not obtained.
\item[0 ]  \ Normal return, no exceptional conditions.
\item[1 ]  \ X was outside the domain of the table, extrapolation used.
\item[2 ]  \ Available table values were so few that this restricted the
degree of the polynomial.  A valid error estimate is not returned in this
case.
\end{itemize}

Starting with IOPT(2) options are specified by integers in the range 0 to~7,
followed in some cases by integers providing argument(s) for the option.
Each option, with its arguments if any, is followed in IOPT by the next
option or by a~0.  If an option index is specified more than once,
only the last specification is used.

\begin{itemize}
\item[0]  End of the option list; this must always be the last option
specified in IOPT.

\item[1]  An error estimate is to be returned in EOPT(1).

\item[2]  (Argument: K2) K2 gives the polynomial degree to use when
extrapolating. The default for K2 is NDEG if NDEG $\leq $ 2 or if NDEG is
even, else it is NDEG~$-$~1.

\item[3]  (Arguments: K3, L3) Save $(k^{th}$ derivative of interpolating
polynomial$)/k!$ in EOPT(K3+$k-$1) for $k=1$, 2, ..., L3. These values are
the coefficients of the polynomial in the monomial basis expanded about X.
Require $0\leq $ L3 $\leq 15.$

\item[4]  (Argument K4) The absolute and relative errors expected in YT
entries are specified in EOPT(K4) and EOPT(K4+1) respectively. The values
provided here are used in estimating the error in the interpolation. An
error estimate is returned in EOPT(1).

\item[5]  (Argument K5, L5) Do the interpolation to the accuracy requested
by the absolute error tolerance specified in EOPT(K5) and the relative error
tolerance in EOPT(K5+1) respectively. An attempt is made to keep the final
error $<$ EOPT(K5) + EOPT(K5+1)\thinspace ($|$YT($k)|$ + $|$YT($k^{\prime
})|)$, where $k$ and $k^{\prime }$ are indices for table values close to X.
Standard polynomial interpolation is done, but here NDEG gives the maximal
degree polynomial to use in the interpolation. The actual degree used in
doing the interpolation is stored in the space for the argument L5. If both
EOPT's specified are $\leq $ 0, IOPT(1) is not set to $-$1, and no error
message is generated due to an unsatisfied accuracy request. An error
estimate is returned in EOPT(1).

\item[6]  (Argument K6) Do not use point (XT$(k)$, YT($k))$ in the
interpolation if YT($k$) = EOPT(K6). This option is useful if one has a table
with equally spaced points, but with some bad data points, and may also be
used by the multiple dimensional interpolation subroutine. Points are
selected for use in the interpolation as if points flagged with YT($k$) =
EOPT(K6) were not present.

\item[7] (Argument K7) YT(K7+$k$) gives the first derivative
corresponding to the function value in YT($k$).  These derivatives
are to be used in doing the interpolation.  One gets a continuous
interpolant only for NDEG = 3, 7, 11, and 15; these cases also give a
continuous first derivative in the interpolant.  The interpolating
polynomial satisfies $p$(XT($k$)) = YT($k$), $p^\prime $(XT($k$)) =
YT(K7+$k$) for values of $k$ that give values of XT close to X.
(Section~D describes how points are selected.) If NDEG is even, a
value of YT($k$) is used without using the corresponding value of
YT(K7+$k$).

\end{itemize}

\item[EOPT]  [inout] Array used to return an error estimate and used for
options.

\begin{description}
\item[EOPT$(1)$]  [out] contains an estimate of the error in the
interpolation if an error estimate has been requested by setting
option 1, 4 or~5.

\item[EOPT$(>1)$]  [in or out] for use by options~3--6.
\end{description}
\end{description}

\subsubsection{Modifications for Double Precision}

Change SILUP to DILUP, and the REAL type statement to DOUBLE PRECISION.

\subsection{Examples and Remarks}

DRSILUP below is a sample program that interpolates in a table of $\sin
(x)$ given with a spacing of~0.5 for NDEG = 2 to~10, and obtains an error
estimate. ODSILUP below gives the results of running DRSILUP on an IBM PC,
which uses IEEE 32~bit floating point arithmetic.

The user is reminded that polynomial interpolation of high degree is
hazardous, where ``high" depends strongly on the kind of data being
interpolated. The error estimates provided by the program are usually
greater than the actual error, but on any given interpolation may be much
too small. Similarly, the order selected by the program to satisfy a given
error criterion will usually do a good job, but will (infrequently) use too
low a degree due to an overly optimistic error estimate or an overly
pessimistic conclusion that the corrections are starting to diverge. When
many interpolations are going to be done in a table we recommend trying
several degrees at enough points to cover the kinds of functional behavior
in different areas of the table and examining the error estimates. Then
later interpolations can be done using the fixed degree that appears best
for the accuracy desired.

If one has a table with discontinuities, one can set XT(I) = XT(I+1) =
the point of the discontinuity. The $x_j$'s that are used in the
interpolation will be selected so that all $j$ satisfy $j \leq $ I, or
$j \geq$ I + 1.  If the discontinuity lies between two points, use
option 6 to define the value at the point of discontinuity.

\subsection{Functional Description}

The look up process identifies the point in the table, XT($k)$, nearest to
the input value X.  To obtain continuity in the interpolant, remaining
points are selected one at a time, keeping the number of points on either
side of X balanced as long as this is possible.  Thus XT($k)$ and
XT($k^{\prime})$, the second entry selected, will bracket X if
extrapolation is not required.  When NDEG $>0$ is even (and Hermite
interpolation is not being used), a linear combination of the polynomials
of degree NDEG interpolating the left NDEG + 1 and the right NDEG + 1
points is used.  (A total of NDEG + 2 points is used.) If it is not
possible to select $(\text{NDEG}+2) / 2$ points on either side of X, then
just NDEG + 1 points are used to obtain a polynomial of degree NDEG as in
the odd case.  When extrapolating, the default action is to use standard
polynomial interpolation of degree $2\times \max (1,\lfloor
$NDEG$/2\rfloor )$.  This default can be changed using an option value
of~2 in IOPT().  Interpolations are done using the Newton divided
difference form of the interpolating polynomial as described for
Algorithms I (NDEG odd) and IV (NDEG$>0$, even) by Krogh in
\cite{Krogh:1970:EAP}.  The derivatives are computed as described for
Algorithm V in \cite{Krogh:1970:EAP}.

Define $E_{Min}$ by the following:
\begin{equation*}
E_{Min}=E_{Abs}+E_{Rel\,}(|\text{YT}(i)|+|\text{YT}(i^{\prime })|),
\end{equation*}
where $E_{Abs}$ = EOPT(K4) if option~4 is used, and is otherwise~0; $E_{Rel}$
= EOPT(K4+1) if option~4 is used and is otherwise the smallest positive
floating point number that gives a number different from~1 when added to~1;
and $i$ and $i^{\prime}$ are the indices for the first two table entries
selected for use in the interpolation.

Then the error is estimated by

\begin{equation*}
\hspace{-5pt}\text{error est.}=1.5\left( |y_c-P_{k+1}|+
\frac{|P_k-P_{k-1}|}{32}\right)+E_{Min},
\end{equation*}
where $y_c$ is the value being returned for Y, $P_k$ is the result obtained
by standard polynomial interpolation of degree $k$, and $k$ is NDEG unless
the degree has been reduced because of extrapolation in which case $k$ is
the degree actually used. If there are not sufficient points available to
compute $P_{k+1}$, then the error estimate is given by~1.5 $%
|P_k-P_{k-1}|+E_{Min}.$

The choice of degree for option~5 makes use of the following quantities.
\begin{align*}
e_0 &= |P_0|\\
\overline{e}_1&=\max (.75,\ e_1/(e_1+e_0)\\
e_j &= |P_j - P_{j-1}|,\quad j = 1,\ 2,\ ...\\
\overline{e}_j&=\frac 12\overline{e}_{j-1}+e_j/(e_j+e_{j-1}),\quad j=2,
 3,\ ...\\
\widehat{e}_j&=\frac 12e_{j-1}\overline{e}_{j-1}/j,\quad j=2,\ 3,\ ...
\end{align*}
The value of $j$ is allowed to increase until either $e_j+e_{j-1}\leq $
error request, or $e_j\geq \widehat{e}_j$ for two successive steps with $j>1$%
. The quantity, $\overline{e}_j$ indicates the rate of decrease in $e_j$ for
successive values of $j$. A good rate of decrease in the past causes a
better rate to be required in the future. Also, better convergence is
required as the degree gets larger. The final error estimate is given by~1.5
$(e_j+.0625e_{j-1})+E_{Min\text{.}}$ When derivatives are being computed,
the error request is decreased by the factor $|(x-x_i)/(x_i-x_{i^{\prime }})|
$, where $i$ and $i^{\prime }$ are the indices for the first two table
entries selected.

\bibliography{math77}
\bibliographystyle{math77}

\subsection{Error Procedures and Restrictions}

Values of IOPT$(1) < 0$ ordinarily cause an error message to be printed, and
those $< -3$ do not ordinarily result in a return to the user. One can
change the action on errors by calling the message/error routine MESS
of Chapter~19.3 before calling this routine.

\subsection{Supporting Information}

The source language is ANSI Fortran~77.

\begin{tabular}{@{\bf}l@{\hspace{5pt}}l}
\bf Entry & \hspace{.35in} {\bf Required Files}\vspace{2pt} \\
DILUP & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, DILUP, DMESS, MESS\rule[-5pt]{0pt}{8pt}}\\
SILUP & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, MESS, SILUP, SMESS}\\
\end{tabular}

Subroutine designed and written by: Fred T. Krogh, JPL, May~1991.
Hermite interpolation added December~1994.

\begcodenp
\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRSILUP}\vspace{10pt}
\lstinputlisting{\codeloc{silup}}
\newpage

\vspace{30pt}\centerline{\bf \large ODSILUP}\vspace{10pt}
\lstset{language={}}
\lstinputlisting{\outputloc{silup}}
\end{document}


\section{08/24}
This course will involve only a few techniques with a variety of applications.
Many domains, such as Distributed Computing, Networks, etc., all depend on
randomized algorithms.

We will focus on a few techniques that lead to understanding.
\begin{enumerate}
    \item Union Bound
    \item Linearity of Expectation
    \item Markov's Inequality
    \item Chernoff Bounds
\end{enumerate}
These four things prove very useful in the design and understanding of
algorithms.

\subsection{Basic Definitions}
\begin{definition}{}{}
    A \emph{Sample Space} is a set $S$ whose elements consist of \emph{simple
    events} (also called \emph{elementary} events). When $S$ is finite or
    countably infinite, we say it is a \emph{Discrete} sample space.
\end{definition}

\begin{definition}{}{}
    An \emph{event} in a sample space $S$ is a subset of $S$.
\end{definition}

\begin{definition}{}{}
    A \emph{Probability Distribution} on $S$ is a function $\Pr: 2^S \to [0, 1]$
    that satisfies
    \begin{enumerate}
        \item $\prob{S} = 1$
        \item If $E_1$, $E_2$, $\dots$ are pairwise disjoint events (i.e., $E_i
        \cap E_j = \emptyset$ for all pairs $i, j$, also called mutually
        exclusive), ed by some finite or countably infinite set $I$, then
        \[\prob{\bigcup_{i \in I} E_i} = \sum_{i \in I} \prob{E_i}\]
    \end{enumerate}
\end{definition}

\subsection{Conditional Probability}
The expression $\prob{E_2 \given E_1}$ is read ``the probability of $E_2$ given
$E_1$.'' For example, if we randomly select a person from Texas, we might write
\begin{align*}
    E_1 &= \hbox{person chosen is in Houston}\\
    E_2 &= \hbox{person chosen is a UH student}
\end{align*}
in which case $\prob{E_2 \given E_1}$ is simply the probability that a randomly
selected person from Texas is a UH student \emph{given that they are in
Houston}.

\begin{definition}{Conditional Probability}{}
    The conditional probability $\prob{E_2 \given E_1}$ is defined
\[\prob{E_2 \given E_1} = \frac{\prob{E_2 \cap E_1}}{\prob{E_1}}\]
\end{definition}

Intuitively, this can be thought of as taking the probability that $E_2$ and
$E_1$ occur and ``normalizing it'' by dividing by the probability that $E_1$
occurs.

\subsection{Independence}
\begin{definition}{Independent Events}{}
Two events, $E_1$ and $E_2$, are \emph{independent} if
\[\prob{E_1 \cap E_2} = \prob{E_1}\prob{E_2}\]
or, equivalently, if
\[\prob{E_1 \given E_2} = \prob{E_1}\]
\end{definition}
For example, suppose $C_1$ and $C_2$ are the outcomes of two fair coin tosses.
These are independent, since
\[\prob{C_1 = H \cap C_2 = T} = \prob{C_1 = H}\prob{C_2 = T} = \frac{1}{4}\]
Independence is not always related to physical independence. For example, say we
are given a fair die and let
\begin{align*}
    E_1 &= \hbox{roll is even}\\
    E_2 &= \hbox{roll is less than or equal to 4}
\end{align*}
In this case, we can enumerate the sample space and explicitly determine
$\prob{E_1}$, $\prob{E_2}$, $\prob{E_1 \cap E_2}$, and $\prob{E_1}\prob{E_2}$,
to see if the events are independent:
\begin{align*}
    E_1 &= \set{2, 4, 6}\\
    E_2 &= \set{1, 2, 3, 4}\\
    E_1 \cap E_2 &= \set{2, 4}
\end{align*}
Then
\begin{align*}
    \prob{E_1} &= \frac{3}{6} = \frac{1}{2}\\
    \prob{E_2} &= \frac{4}{6} = \frac{2}{3}\\
    \prob{E_1 \cap E_2} &= \frac{2}{6} = \frac{1}{3}\\
    \prob{E_1}\prob{E_2} &= \frac{1}{2} \cdot \frac{2}{3} = \frac{1}{3}
\end{align*}
Thus, we see the events are independent. On the other hand, if $E_2$ is the
event that the roll is \emph{strictly less} than 4, we have
\begin{align*}
    E_1 &= \set{2, 4, 6}\\
    E_2 &= \set{1, 2, 3}\\
    E_1 \cap E_2 &= \set{2}
\end{align*}
Then
\begin{align*}
    \prob{E_1} &= \frac{3}{6} = \frac{1}{2}\\
    \prob{E_2} &= \frac{3}{6} = \frac{1}{2}\\
    \prob{E_1 \cap E_2} &= \frac{1}{6}\\
    \prob{E_1}\prob{E_2} &= \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}
\end{align*}
and we see that the events are \emph{not independent}.

\subsection{The Inclusion-Exclusion Principle}
%TODO insert venn diagram
A basic result in set theory is that
\begin{align*}
    \card{A \cup B} &= \card{A} + \card{B} - \card{A \cap B}\\
    \card{A \cup B \cup C} &= \card{A} + \card{B} + \card{C} - \card{A \cap B} - \card{A \cap C} - \card{B \cap C} + \card{A \cap B \cap C}
\end{align*}
which can be generalized to an arbitrary finite union by
\[\bigcup_{i=1}^n A_i = \sum_{k=1}^n (-1)^{k+1} \left(\sum_{1 \leq i_1 < \dots < i_k \leq n} \card{A_{i_1} \cap \cdots \cap A_{i_k}} \right)\]
or equivalently
\[\card{\bigcup_{i=1}^n A_i} = \sum_{\emptyset \neq J \subseteq \set{1,\dots,n}}
(-1)^{\card{J}+1} \card{\bigcap_{j\in J} A_j}\]
This yields the probability formulas
\begin{theorem}{Inclusion-Exclusion Principle}{}
    For any events $E_1$, $E_2$, \dots, $E_n$,
\begin{align*}
    \prob{E_1 \cup E_2} &= \prob{E_1} + \prob{E_2} - \prob{E_1 \cap E_2}\\
    \prob{\bigcup_{i=1}^n E_i} &= \sum_{\emptyset \neq J \subseteq \set{1,\dots,n}}
(-1)^{\card{J}+1} \prob{\bigcap_{j\in J} E_j}
\end{align*}
\end{theorem}

\subsection{Union Bound}
\begin{theorem}{Union Bound}{}
    Let $E_1$, $E_2$, \dots, be any countable set of events. Then
\[\prob{\bigcup E_i} \leq \sum\prob{E_i}\]
\end{theorem}
While this bound is often not very precise, it is useful in many cases where the
events $E_i$ are ``bad'' and we can bound the likelihood of a single $E_i$. This
allows us to bound the likelihood of \emph{any} $E_i$.

\subsection{Conditioning on Multiple Events (Chain Rule)}
By repeatedly applying the definition of conditional probability, we have
\begin{align*}\prob{E_1 \cap E_2 \cap \dots \cap E_n}
    &= \prod_{i=1}^n \prob{E_i \given E_1 \cap E_2 \cap \dots \cap E_{i-1}}\\
    &= \prob{E_1}\prob{E_2 \given E_1}\cdots \prob{E_n \given E_1 \cap E_2 \cap \dots \cap E_{n - 1}}\\
\end{align*}

\subsection{Birthday Paradox}
\begin{problem}{Birthday Paradox}{}
Suppose $n$ people are in a room and they have birthdays chosen uniformly at
random from the 365 calendar days. What is the probability that two people share
a birthday?
\end{problem}

The probability that \emph{no two} people share a birthday can be calculated by
considering the following events:

\begin{align*}
    E_1 &= \hbox{person 1 has a birthday}\\
    E_2 &= \hbox{person 2 has a different birthday than person 1}\\
    \phantom{..}&\vdots\\
    E_i &= \hbox{person $i$ has a different birthday than people 1 through $i - 1$}
\end{align*}

Specifically, the probability that no two people share a birthday is
\begin{align*}\prob{E_1 \cap E_2 \cap \dots \cap E_n}
    &= \prob{E_1}\prob{E_2 \given E_1}\cdots \prob{E_n \given E_1 \cap E_2 \cap \dots \cap E_{n - 1}}\\
    &= 1 \times \left(1 - \frac{1}{365}\right) \times \left(1 - \frac{2}{365}\right) \times \cdots \times \left(1 - \frac{365 - n + 1}{365}\right)
\end{align*}

For what value of $n$ is the above probability less than $\sfrac{1}{2}$?


\section{Mathematics components}
\comp{const}
{Output a constant}
{}{out:float}
Outputs a constant value.
\startparams
val & float & value to output\\
\end{tabularx}

\comp{add}
{Adds the inputs together}
{in1:float,in2:float}{$mul1\times(in1+add1) + mul2\times(in2+add2)$}
Adds together the inputs, after they have been processed through
some parameters.
\startparams
add1 & float & added to input 1 before multiplication by mul1\\
add2 & float & added to input 2 before multiplication by mul2\\
mul1 & float & multiplies $in1+add1$\\
mul2 & float & multiplies $in2+add2$\\
\end{tabularx}

\comp{mul}
{Multiplies the inputs together.}
{in1:float,in2:float}{$mul1\times(in1+add1) \times mul2\times(in2+add2)$}
Multiplies together the inputs, after they have been processed through
some parameters. Yes, the two multiplies are redundant.
\startparams
add1 & float & added to input 1 before multiplication by mul1\\
add2 & float & added to input 2 before multiplication by mul2\\
mul1 & float & multiplies $in1+add1$\\
mul2 & float & multiplies $in2+add2$\\
\end{tabularx}

\comp{mul}
{Clamps the input to a range}
{in:input}{output clamped to range}
Clamps the output to ensure it is within a given range.
\startparams
min & float & minimum value\\
max & float & maximum value\\
\end{tabularx}

\clearpage
\comp{func}
{Performs one of a number of functions on the sum of the inputs.}
{in1:float,in2:float}{$addout+mulout\times f \big(mul1(in1+add1)+ mul2(in2+add2)\big)$}
Takes the two inputs and processes them through multiply and add
parameters, puts the result through a function, and then puts the
output through another multiply and add stage. If an input is
not connected, it is zero. Thus, the output is:
\[
p_{addout} + p_{mulout} \times f\Big((p_{add1}+in_1)\times p_{mul1} +
(p_{add2}+in_2)\times p_{mul2} \Big)
\]
Functions supported include: sine, cosine, gaussian(x+y), x mod y, abs(x+y).
\startparams
add1 & float & added to input 1 before multiplication by mul1\\
mul1 & float & multiplies $in1+add1$\\
add2 & float & added to input 2 before multiplication by mul2\\
mul2 & float & multiplies $in2+add2$\\
func & enum & function to perform\\
mulout & float & multiplies function output\\
addout & float & added to output
\end{tabularx}

\clearpage
\comp{perlin}{Perlin coherent noise generator (input is values or current time)}
{x:float,y:float}{out:perlin(x+y),zerocrosstrig:int}
Calculates
\[
p_{addout} + p_{mulout} \times \operatorname{perlin}\Big((p_{add1}+in_1)\times p_{mul1} +
(p_{add2}+in_2)\times p_{mul2} \Big)
\]
where \emph{perlin} is the Perlin noise function with range [-1,1].
Thus, it generates
smoothly varying coherent random noise. If neither input is connected,
$in_1$ is the current time. If the output crosses zero, i.e. the previous
output was one side of zero and the new output is the other side,
the \emph{zerocrosstrig} output is set high. This is typically used
with no inputs to generate randomly timed events, with \emph{mul1} set
to control the frequency.
\startparams
octaves & int & number of octaves in noise function -- the higher, the
more ``fractal'' the noise and the ``busier'' it becomes\\
persistence & float & the noise persistence -- how much the amplitudes
decrease for each successive octave\\
add1 & float & added to input 1 (or time if no inputs are connected)
before multiplication by mul1\\
mul1 & float & multiplies $in1+add1$\\
add2 & float & added to input 2 before multiplication by mul2\\
mul2 & float & multiplies $in2+add2$\\
mulout & float & multiplies function output\\
addout & float & added to output
\end{tabularx}
For more information on \emph{octaves} and \emph{persistence}, and Perlin
noise is general, see the documentation for \texttt{libnoise}.