\chapter{O-C Diagrams}
\label{Chapter_oc}
\section{O-C Basics}
Observed minus Calculated diagrams is a diagnostic tool and involves the evaluation and interpretation
of the disagreement between the measure of an observable event and its
predicted value \citep{Sterken2005basic}.
The idea that random processes may be important in determining O-C behaviour is not new, dating back at least to
\cite{Eddington1929}. Before dealing with the consequences for the statistics of
O-C diagrams, it is worthwhile to review possible physical causes of random
cycle-to-cycle period variations \citep{Koen2005stat}.

In astronomy, O-C usually implies a temporal aspect, and is used when
discussing cyclic phenomena where the times of occurrence of a given event is irregular.
The O-C diagram is then constructed by plotting the quantity O-C as a function of time, the correct interpretation
of these deviations leads to a better model (and a new O-C diagram).

In variable-star studies, O-C is sometimes expressed as deviations of phase
in the cycle of variability, whereas the time axis is the cycle number (commonly
indicated by $E$). In such studies, the O-C diagram mostly refers to rather
simple C formalisms, viz. linear or quadratic ephemeris formulae, sometimes
combined with a trigonometric periodic term \citep{Sterken2005basic}.

%Period of a recurrent phenomenon is the time interval after which the event
%goes exactly through a same cycle again. In reality, we deal with cycles, rhythms, waves, and
%pseudo-periods or characteristic times, in short, with processes that repeat themselves
%in a more or less regular way.

We are dealing with processes that repeat themselves in a more or less regular way.
Any attempt to construct a reliable O-C diagram will fail if a wrong
value of period $P$ is used. But it is not always easy to derive a period from the observational data:
$P$ is not directly observable, it follows from the
determination of at least two moments of time of the same reference phase (epochs).
The observation of two such epochs $T_{1}$ and $T_{2}$ immediately provide us an upper limit for $P$.
When more than two such times are available, the period can be derived by a least-squares solution of a set of
equations
\begin{equation} \label{eq:period}
T_{i} - T_{j} = nP
\end{equation}
where $n$ is an integer, commonly called the cycle number $E$(epoch).

Phase is a position on the cycle of variation, a convenient periodic measure of
elapsed time: $\varphi (t)$ is the fraction of $P$ that elapsed since the occurrence of the
reference time $T_{0}$ and is given by

\begin{equation} \label{eq:phase}
\varphi = \frac{T-T_{0}}{P} ~\bmod~ 1
\end{equation}

The most common approach to the O-C procedure is the one in which one reference
phase is selected, and where the timings of this reference phase are studied
and interpreted.
Several methods exist to determine the reference phase in a variability curve.
This methods are discussed in Chapter \ref{Chapter_minima_det}.

\section{Constant and Variable Period}
If $P$ is constant and if its value is known, equation (\ref{eq:period}) leads to

%\begin{equation} \label{eq:Tmin}
%T_{max} = T_{0} + P E
%\end{equation}
%or,
\begin{equation} \label{eq:Tmin2}
T_{min} = T_{0} + P E
\end{equation}
where $T_{min}$ is the time of minimum light, $T_{0}$
is the zero epoch and $E$ is the number of cycles elapsed since the zero epoch. $T_{0}$ and
$P$ are obtained through a least-squares solution. The longer the time interval (in
cycles) over which the data have been collected, the higher will be the accuracy
of the solution for $P$: the uncertainty in $P$ is inversionally proportional to the
number of cycles, and proportional to the r.m.s. scatter of the data.

It is certainly not a trivial task to conclude from experimental data that a
significant period change has occurred. In principle, changes of period could
be described by any mathematical formula expressing $P$ as a function of time.
Thus, the time of epoch $T_{m}$ is

\begin{equation} \label{eq:P_var}
T_{m} = T_{0} + \int P(t)dt   ~~~~~\mathrm{or}~~~~~   T_{m} = T_{0} + \int P(E)dE
\end{equation}

In most cases, relation (\ref{eq:P_var}) is restricted to linear variations, cyclic variations, or
a combination of both of them.

Many causes can lead to period variations.
%the reference phase is seen systematically earlier (negative O-C) or later (positive O-C),
%depending whether eclipsing variable is nearer or further from the observer.
%Such meandering of the reference phase will result in a cyclic O-C diagram.
%Apsidal motion in an elliptical orbit can re-orient the orbit with
%respect to the observer and cause an effect similar to the previous case.
Transfer of matter (between stars in a multiple system) or mass ejection (from a system)
can provoke period changes. Periodic variations of O-C can indicate presents of other body in binary system.

If period changes linear with time, we will write $P$ as $P = a+bt$, where $t$ is the time, and $a,~b$ are constants.
Let $P_{0}$ be the period at $t=0$ and $\bar{P}$ is the average period over the whole time of observations, then

\begin{equation} \label{eq:P_lin_P}
P = a + b \bar{P}E
\end{equation}
and
\begin{equation} \label{eq:P_lin_Pmid}
\bar{P} = a + \frac{1}{2}bt
\end{equation}
so
\begin{equation} \label{eq:P_lin_Tm}
T_{m} = T_{0} + aE + \frac{1}{2}b \bar{P}E^{2}
\end{equation}
with
\begin{equation} \label{eq:P_lin_add}
P_{0} = a,  ~~~~  \frac{dP}{dE} = b\bar{P},   ~~~~   \frac{dP}{dt} = b
\end{equation}
expected time
\begin{equation} \label{eq:P_lin_Tm_res}
T_{m} = T_{0} + P_{0}E + \frac{1}{2} \frac{dP}{dt} \bar{P}E^{2}
\end{equation}

\begin{equation} \label{eq:P_lin_OC_res}
O-C = \frac{1}{2} \frac{dP}{dt} \bar{P}E^{2}
\end{equation}

%Equation (\ref{eq:P_lin_Tm}) is the most-frequently applied equation in O-C discussions.
%Unfortunately, it is often misunderstood. Worse even, the factor $1/2$ is occasionally
%forgotten. The numerical value of $\frac{dP}{dt}$ is obtained through a quadratic fit, which
%also yields $P_{0}$.

When we now look at the O-C diagram for BW Vul in Fig.\ref{fig_oc}, we see that its
shape suggests a possible parabolic form, which seems to stand for a linear period
change. The least-squares parabolic fit to equation (\ref{eq:P_lin_Tm}) yields $P_{0} = 0.2010274$
and $\frac{dP}{dt}= 1.8 \cdot 10^{−10}$ days per cycle, or $~9.1 \cdot 10^{−10}$ day per day.

\begin{figure}[!ht]
\vspace{0cm}
\centerline{\includegraphics[width=0.65\textwidth]{oc_example.png}}
\caption{O-C diagram of some $T_{min}$ of BW Vul with best fit parabola. \citep{Sterken2005basic}}
\label{fig_oc}
\end{figure}

\begin{figure}[!ht]
\vspace{0cm}
\centerline{\includegraphics[width=0.65\textwidth]{oc_example_2.png}}
\caption{Differential O-C diagram after prewhitening with the fitted parabola. \citep{Sterken2005basic}}
\label{fig_oc2}
\end{figure}

When looking at Fig.\ref{fig_oc}, one notices that there are stretches of the O-C parabolic
fit where the data at one time are systematically above the fitted curve, and at
other times remain below the curve. Figure \ref{fig_oc2} shows the differences that were
obtained by removing the parabolic trend, and shows wave shape with
variable amplitude \citep{Sterken2005basic}. Such a trend, in particular when it repeats itself,
can indicate presence of other body in binary system.
In such case, O-C can be fitted with additional term corresponding to 3rd body orbit parameters:

\begin{equation} \label{eq:P_lin_OC_sin}
T_{m} = T_{0} + P_{0}E + \frac{1}{2} \frac{dP}{dt} \bar{P}E^{2} +
\dfrac{a_{12}\sin i}{c}   \left[  \dfrac{1-e^2}{1+e \cos\nu}   \sin(\nu + \omega)  + e \sin \omega  \right]
\end{equation}

%\begin{equation} \label{eq:P_lin_OC_sin}
%T_{m} = T_{0} + P_{0}E + \frac{1}{2} \frac{dP}{dt} \bar{P}E^{2} + A\sin(2\frac{\pi}{\Pi}E+\phi)
%\end{equation}
%
%\begin{equation}\label{eq:4_mmm}
%T_{min} = JD_{0}+P \times E + Q \times E^2 + \dfrac{a_{12}\sin i}{c}   \left[  \dfrac{1-e^2}{1+e \cos\nu}   \sin(\nu + \omega)  + e \sin \omega  \right]
%\end{equation}
where $a_{12} \sin i$ is the projected semi-major axis, $e$ is the eccentricity,
$\omega$ is the longitude of the periastron, $\nu$ is the true anomaly of the EB orbit around the common centre of the mass of
the whole system and $c$ is the velocity of light \citep{irwin1959}.


% Preamble.
\documentclass[12pt]{article}
\usepackage[margin=1.25in]{geometry}
\usepackage[fleqn]{amsmath}
\usepackage{textcomp}
\usepackage{gensymb}
\usepackage{amsfonts}
\usepackage{enumitem}
\usepackage{siunitx}
%\usepackage{subfiles}  % Include for subfiles.
%\usepackage{tikz}  % Include for handmade figures.
%\usepackage{graphicx}  % Include for external figures.

%% Title macros.
\newcommand{\HOMEWORKNUM}{3}
\newcommand{\NAME}{D. Choi}
\newcommand{\DATE}{2020-09-11}

\title{\vspace{-2\baselineskip}PHYS 151 - Written Homework \#\HOMEWORKNUM}
\author{\NAME}
\date{\DATE}

%% Formatting options.
%\pagenumbering{gobble}  % Include for single-page document.

% Document.
\begin{document}
\maketitle

\section*{1.}
\textit{The figure shows a sailing ship approaching a hostile harbor. At time
$t = 0$, a cannon placed atop a lighthouse, at height $h$ above the deck of
the hostile ship, fires a projectile at the oncoming ship, which is moving at
a speed of $v_1$ toward the shore. The cannonball is fired at a speed of $v_2$
at an angle $\theta$ above the horizontal.}
\begin{enumerate}[label=(\alph*)]
	\item \textit{For this cannonball to impact exactly the front of the ship,
	at what distance $D$ from the dock should the ship be when the equipment
	is thrown? Ignore air resistance.} \\[\baselineskip]
	The ship's horizontal and vertical positions as functions of time can be
	modeled by
	\begin{gather*}
		x_s(t) = -v_1t + D, \\
		y_s(t) = 0.
	\end{gather*}
	In turn, the projectile's horizontal and vertical positions can be modeled
	by
	\begin{gather*}
		x_p(t) = v_2\cos(\theta)t, \\
		y_p(t) = -\frac{g}{2}t^2 + v_2\sin(\theta)t + h.
	\end{gather*}
	The projectile must hit the ship at a time $t_f$ when both
	$x_s(t_f) = x_p(t_f)$ and $y_s(t_f) = y_p(t_f)$. Because $y_s(t_f)$ and
	$y_p(t_f)$ are both known, $t_f$ can be solved for with
	\begin{gather*}
		y_s(t_f) = y_p(t_f), \\
		0 = -\frac{g}{2}{t_f}^2 + v_2\sin(\theta)t_f + h, \\
		t_f = \frac{v_2\sin(\theta) + \sqrt{(v_2\sin(\theta))^2 + 2gh}}{g}.
	\end{gather*}
	Using $t_f$, $D$ can be solved for with
	\begin{gather*}
		x_s(t_f) = x_p(t_f), \\
		-v_1t_f + D = v_2\cos(\theta)t_f, \\
		D = v_2\cos(\theta)t_f + v_1t_f, \\
		D = (v_2\cos(\theta) + v_1) t_f.
	\end{gather*}
	In summary, the expanded formula to find the distance $D$ is
	\begin{gather*}
		\boxed{D = (v_2\cos(\theta) + v_1) t_f}
	\end{gather*}
	where
	\begin{equation*}
		\boxed{
			t_f = \frac{v_2\sin(\theta)
			+ \sqrt{(v_2\sin(\theta))^2 + 2gh}}{g}
		}.
	\end{equation*}
	\item \textit{What is the total time of flight of the cannonball? \\
	Use $v_1 = 40.0\si{\frac{cm}{s}},\ v_2 = 16.0\si{\frac{m}{s}},\
	\theta = \ang{55.0}$, and $h = 9.25\si{m}$.}
	\begin{align*}
		t_f
		&= \frac{v_2\sin(\theta) + \sqrt{(v_2\sin(\theta))^2 + 2gh}}{g} \\
		&= \frac
			{
				(16.0\si{\frac{m}{s}}) \sin(\ang{55.0})
				+ \sqrt{
					((16.0\si{\frac{m}{s}}) \sin(\ang{55.0}))^2
					+ 2 (9.81\si{\frac{m}{s^2}}) (9.25\si{m})
				}
			}
			{9.81\si{\frac{m}{s^2}}}
			\\
		&= \boxed{3.25\si{s}}.
	\end{align*}
	\item \textit{Using the numbers given in (b.), what are the $x$ and $y$
	components of the velocity of the cannonball just before impacting the
	deck of the ship?} \\[\baselineskip]
	Let $v_x(t)$ and $v_y(t)$ represent the horizontal and vertical components
	of the projectile's velocity as functions of time, respectively.
	\begin{align*}
		v_x(t_f)
		&= v_2\cos(\theta) \\
		&= (16.0\si{\frac{m}{s}}) \cos(\ang{55.0}) \\
		&= \boxed{9.18\si{\frac{m}{s}}}.
	\end{align*}
	\begin{align*}
		v_y(t_f)
		&= -gt_f + v_2\sin(\theta) \\
		&= -(9.81\si{\frac{m}{s^2}}) (3.25\si{s})
			+ (16.0\si{\frac{m}{s}}) \sin(\ang{55.0}) \\
		&= \boxed{-18.8\si{\frac{m}{s}}}.
	\end{align*}
\end{enumerate}

\end{document}


\begin{itemize}
\item For an L-shape gene it is verified that:
\begin{itemize}
\item The ratio $r=\frac{\min\{\mathit{cMI}(t)\}}{\mathit{cMI}(0)}$ is small,
\item $t^{\ast} = \mathrm{argmin}\{ \mathit{cMI}(t) \}$ is the \textbf{optimal threshold} for
dichotomizing the methylation data of this gene.
\end{itemize}
\end{itemize}

%To estimate the MI terms we use a kernel-based estimator by applying a %Gaussian kernel to each data point:
%\[
%I(X,Y) = \frac 1M \sum_{i=1}^M \log\frac{M\sum_{j=1}^M e^{-\frac{1}{2h^2}%((x_i-x_j)^2+(y_i-y_j)^2)}}{%
      %                                \sum_{j=1}^M e^{-\frac{1}{2h^2}(x_i-%x_j)^2} \sum_{j=1}^M e^{-\frac{1}{2h^2}(y_i-y_j)^2}}
%\]
%where $h$ is a tuning parameter for the kernel width empirically set to
% $h=0.3$.

\subsection{Based on Spline regression}
\begin{itemize}
\item As an alternative to the previous method we suggest that spline regression \cite{racine} can be used for scatter plot clustering.
\item In spline regression a curve $y=s(x)$ is represented as $\mathbf{y}_i=\mathbf{B}_i\mathbf{c}$ where
\begin{itemize}
\item $\mathbf{B}_i =\left[ B_{1p}\mathbf{x}_i,B_{2p}\mathbf{x}_i,\dots,B_{Lp}\mathbf{x}_i \right]$ the spline basis matrix and
\item $\mathbf{c}$ is the vector of spline coefficients.
\end{itemize}

\item This suggests the following method (and algorithm) for detecting L--shaped genes based on \textbf{Clustering Spline Coefficients}:
\begin {enumerate}
\item Select genes with significant correlation.
\item For each selected gene fit a cubic splines regression model.
\item Obtain a distance matrix between all genes using the $1-\rho$ distance computed on spline coefficients.
\item Perform a hierarchical clustering and
\item Select genes in the \textit{L-shaped cluster(s)}.
\end{enumerate}
\end{itemize}


\subsection{Field extensions}\label{subsec:field_extensions}

\begin{definition}\label{def:splitting_field}
  A \term{splitting field} for a nonconstant polynomial \( f(X) \in \Bbbk[X] \) of degree \( n \) is the smallest \hyperref[def:field_extension]{field extension} \( \BbbK \) of \( \Bbbk \) in which \( f(X) \) has \( n \) \hyperref[def:polynomial_root]{roots}. That is,
  \begin{equation*}
    \BbbK \cong \Bbbk(a_1, \ldots, a_n),
  \end{equation*}
  where \( a_1, \ldots, a_n \) are roots of \( f(X) \).

  By \fullref{thm:splitting_field_existence}, splitting fields exist and are unique up to an isomorphism.
\end{definition}

\begin{proposition}\label{thm:splitting_field_existence}
  There exists a unique up to an (possibly nonunique) isomorphism \hyperref[def:splitting_field]{splitting field} for every nonconstant polynomial in one indeterminate over a field.
\end{proposition}
\begin{proof}
  \SubProof{Proof of existence}\mcite[thm 9.10 \\ thm. 9.12]{Knapp2016BasicAlgebra} We use induction on the degree of the polynomial \( f(X) = \sum_{k=0}^n a_k x^k \) over \( \Bbbk \). In the base case \( n = 1 \), \( f(X) \) is already linear, and hence \( \Bbbk \) is itself a splitting field for \( f(X) \).

  Suppose that there exist splitting fields for polynomials over \( \Bbbk \) of degree \( n - 1 \). By \fullref{thm:maximal_ideal_theorem}, the \hyperref[def:semiring_ideal]{principal ideal} \( \braket{ f(X) } \) is contained in some maximal ideal \( M \). By \fullref{thm:quotient_by_maximal_ideal}, the quotient of \( R[X] \) by \( M \) is a field.

  Define \( u_n \coloneqq \braket{ X } + M \). We have \( f(u_n) = \braket{ f(X) } + M \), hence \( u_n \) is a root of \( f \) in \( M \). Then
  \begin{equation*}
    f(X) = (X - u_n) q(X)
  \end{equation*}
  for some polynomials \( q(X) \) and \( r(X) \), both of degree less than \( n \) (or \( r(X) = 0 \)).

  We can now apply the inductive hypothesis to obtain a splitting field of \( q(X) \). Let \( u_1, \ldots, u_{n-1} \) be the roots of \( q(X) \) in this field. We can then adjoin \( u_1, \ldots, u_n \) to the field \( \Bbbk \) to obtain a splitting field \( \Bbbk(u_1, \ldots, u_n) \) of \( f(X) \). Denote this field by \( \BbbK \).

  \SubProof{Proof of uniqueness} Suppose that, given our previous construction, \( \BbbL \) is also a splitting field for \( f(X) \).

  Again, we use induction on the degree \( n \) of \( f(X) \). The case \( n = 1 \) is again obvious.

  Suppose that any two splitting fields for polynomials over \( \Bbbk \) of degree \( n - 1 \) are isomorphic. Let \( b_n \) be a root of \( f(X) \) in \( \BbbL \) and let
  \begin{equation*}
    f(X) = (X - b_n) r(X).
  \end{equation*}

  Let \( b_1, \ldots, b_{n-1} \) be the roots of \( r(X) \). Let \( \varphi \) be an isomorphism between the subfield \( \Bbbk(a_1, \ldots, a_{n-1}) \) of \( \BbbK \) and the corresponding subfield \( \Bbbk(b_1, \ldots, b_{n-1}) \) of \( \BbbL \). It follows that
  \begin{equation*}
    \underbrace{\prod_{k=1}^{n-1} (X - b_k)}_{r(X)} = \underbrace{\prod_{k=1}^{n-1} (X - \varphi(a_k))}_{q^\varphi(X)}.
  \end{equation*}

  Therefore, we can extend \( \varphi \) to an isomorphism \( \widehat{\varphi}: \BbbK \to \BbbL \) by putting \( \widehat{\varphi}(a_n) \coloneqq b_n \).
\end{proof}

\begin{theorem}[Classification of finite fields]\label{thm:finite_fields}
  \hfill
  \begin{thmenum}
    \thmitem{thm:finite_fields/characteristic} The \hyperref[def:ring_characteristic]{characteristic} of a \hyperref[def:field]{field} with \( q \) elements is a \hyperref[def:prime_number]{prime number} \( p \), and \( q \) is a power of \( p \).

    The fields of prime cardinality are sometimes called \term{prime fields}.

    \thmitem{thm:finite_fields/prime_field} For a prime number \( p \), the ring \hyperref[thm:ring_of_integers_modulo]{\( \BbbZ_p \)} of integers modulo \( p \) is a field.

    \thmitem{thm:finite_fields/splitting} All \hyperref[def:field]{fields} with \( q \) elements are \hyperref[def:field/homomorphism]{isomorphic} as \hyperref[def:splitting_field]{splitting fields} for the polynomial
    \begin{equation*}
      X^q - X \in \BbbZ_p[X].
    \end{equation*}

    Utilizing the general conventions of identifying isomorphic objects in algebra, we denote by \( \BbbF_q \) \enquote{the} finite field with \( q \) elements. Finite fields are also called \term{Galois fields}.

    Every member of \( \BbbF_q \) is a root of \( X^q - X \).
  \end{thmenum}
\end{theorem}
\begin{proof}
  \SubProofOf{thm:finite_fields/characteristic} Let \( \BbbK \) be a field with \( q \) elements and let \( p \) be the \hyperref[def:ring_characteristic]{characteristic} of \( \BbbK \). Then \( \BbbZ_p \) is a subring of \( \BbbK \). By \fullref{thm:multiplicative_group_of_integers_modulo}, \( \BbbZ_p \) is a field.

  By \fullref{thm:quotient_by_maximal_ideal}, \( \braket{ p } \) is a maximal ideal in \( \BbbZ \), and, by \fullref{thm:def:semiring_ideal/maximal_is_prime}, \( p \) is a prime number.

  By \fullref{thm:lagranges_theorem_for_groups}, \( p \) divides \( q \). But \( \BbbK / \BbbZ_p \) is again a field by \fullref{thm:quotient_ideal_lattice_theorem}, and again has prime characteristic. Continuing by induction, we eventually obtain a sequence \( p_1, \ldots, p_n \) of prime numbers such that
  \begin{equation*}
    q = p_1 \cdots p_n.
  \end{equation*}

  By \fullref{thm:lagranges_theorem_for_groups}, \( q \) cannot contain subgroups of prime cardinalities \( p_1 \) and \( p_2 \) unless \( p_1 = p_2 \). Hence, again by induction, we conclude that
  \begin{equation*}
    p_1 = \cdots = p_n.
  \end{equation*}

  Therefore, \( q = p^n \).

  \SubProofOf{thm:finite_fields/prime_field} Follows from \fullref{thm:multiplicative_group_of_integers_modulo}.

  \SubProofOf{thm:finite_fields/splitting} Let \( \BbbK \) be a field with \( q \) elements with characteristic \( p \). We will show that every element of \( \BbbK \) is a root of \( X^q - X \in \BbbZ_p[X] \).

  The multiplicative group of \( \BbbK \) has order \( q - 1 \). The order of a non-zero element \( a \in \BbbK \) divides \( q - 1 \) by \fullref{thm:def:group_order/divides}, hence \( a^{q - 1} = 1 \pmod q \). We also have \( 0^q = 0 \). Therefore, for every element of \( \BbbF_q \), we have \( a^q = a \).

  Then
  \begin{equation*}
    X^q - X = \prod_{u \in \BbbK} (X - u).
  \end{equation*}
\end{proof}

\begin{proposition}\label{thm:functions_over_prime_fields}
  For every \hyperref[thm:finite_fields]{finite field} \( \BbbF_q \) and every \hyperref[def:polynomial_algebra]{polynomial ring} \( \BbbF_q[X_1, \ldots, X_n] \) in finitely many indeterminates, there exists an \( \BbbF_q \)-\hyperref[def:algebra_over_ring]{algebra} isomorphism
  \begin{equation*}
    \frac {\BbbF_q[X_1, \ldots, X_n]} {\braket{ X_i^q - X_i \given i = 1, \ldots, n }} \cong \fun(\BbbF_q^n, \BbbF_q),
  \end{equation*}
  where \( \fun(\BbbF_q^n, \BbbF_q) \) is the \hyperref[thm:functions_over_algebra]{\( \BbbF_q \)-algebra of all functions} from \( \BbbF
  _q^n \) to \( \BbbF_q \).

  Furthermore, every coset of polynomials has a unique representative given by \fullref{thm:finite_field_lagrange_interpolation}.
\end{proposition}
\begin{proof}
  Consider the \hyperref[thm:polynomial_semiring_universal_qroperty]{functional evaluation homomorphism}
  \begin{equation*}
    \Phi: \BbbF_q[X_1, \ldots, X_m] \to \fun(\BbbF_q^m, \BbbF_q).
  \end{equation*}

  By \fullref{thm:finite_field_lagrange_interpolation}, \( \Phi \) is surjective. Then, by \fullref{thm:quotient_algebra_universal_property},
  \begin{equation*}
    \BbbF_q[X_1, \ldots, X_n] / \ker \Phi \cong \fun(\BbbF_q^m, \BbbF_q).
  \end{equation*}

  We will now prove that \( \ker \Phi \) equals
  \begin{equation*}
    I \coloneqq \braket{ X_i^q - X_i \given i = 1, \ldots, n }.
  \end{equation*}

  First, let \( e: \mscrX \to \BbbF_q \) be the variable assignment that assigns \( u_1, \ldots, u_n \) to the corresponding indeterminates. By \fullref{thm:finite_fields/splitting}, every member of \( \BbbF_q \) is a root of \( X_i^q - X_i \). Then, for any indeterminate \( X_i \),
  \begin{equation*}
    \Phi_e(X_i^q - X_i) = u_i^q - u_i = 0 \pmod q.
  \end{equation*}

  Hence, the polynomial function \( \Phi(X_i^q - X_i) \) is the zero constant function. It follows that any linear combination of the polynomials \( X_i^q - X_i \) for \( i = 1, \ldots, n \) is also the zero function. Therefore, \( I \subseteq \ker \Phi \).

  We will prove the converse inclusion via induction on \( n \).

  In the case of a single indeterminate \( X \), for every polynomial \( f(X) \in \ker \Phi \), we know that the entirety of \( \BbbF_q \) are roots of \( f(X) \). By \fullref{thm:def:integral_domain/root_limit}, \( f(X) \) has at most \( q \) roots. Hence, \( X - u \) divides \( f(X) \) for every \( u \in \BbbF_q \). We have
  \begin{equation*}
    \underbrace{\prod_{u \in \BbbF_q} (X - u)}_{\mathclap{ X^q - X \T*{by \fullref{thm:finite_fields/splitting}}}} \mid f(X),
  \end{equation*}
  and hence \( f(X) \in \braket{ X^q - X } \).

  We have, up until now, shown that the entire proposition holds for the case of one indeterminate. Suppose that the proposition holds for \( n - 1 \) indeterminates and let \( f \in \BbbF_q[X_1, \ldots, X_n] \) be a nonconstant polynomial such that \( \Phi(f) \) is the zero function. Due to \fullref{thm:def:polynomial_algebra/iterated}, we can regard \( f \) as a univariate polynomial in \( X_n \)over \( \BbbF_q[X_1, \ldots, X_{n-1}] \). Thus,
  \begin{equation*}
    f(X_1, \ldots, X_n) = \sum_{k =0}^\infty \underbrace{\parens*{ \sum_\gamma a_{(k,\gamma)} X_1^{\gamma_1} X_2^{\gamma_1} \cdots X_{n-1}^{\gamma_{n-1}} }}_{s_k(X_1, \ldots, X_{n-1})} {X_n}^k,
  \end{equation*}
  where \( \gamma \) is a multi-index over the first \( n - 1 \) indeterminates.

  As a polynomial in \( X_n \), \( f \) has \( m \coloneqq (n-1)p \) roots \( s_1, \ldots, s_m \), which are themselves polynomials from \( \BbbF_q[X_1, \ldots, X_{n-1}] \). For some \( c \), we have
  \begin{equation*}
    f(X_1, \ldots, X_n) = c(X_1, \ldots, X_{n-1}) \prod_{j=1}^m (X_n - s_j(X_1, \ldots, X_{n-1}))
  \end{equation*}
  and
  \begin{equation*}
    0 = \Phi(f) = \Phi(c) \cdot \prod_{j=1}^m \parens[\Big]{ \Phi(X_n) - \Phi(s_j) }.
  \end{equation*}

  Since \( \BbbF_q[X_1, \ldots, X_{n-1}] \) is \hyperref[def:entire_semiring]{entire}, we conclude that either \( \Phi(c) \) is the zero function or \( \Phi(X_n) = \Phi(s_j) \) for at least one index \( 1 \leq j \leq m \). The latter is impossible, because \( \Phi(X_n) \) is linearly independent from polynomials in the first \( n - 1 \) variables.

  The inductive hypothesis holds for the polynomial \( c \), and \( \Phi(c) \) being the zero function implies
  \begin{equation*}
    c \in \braket{ X_i^q - X_i \given i = 1, \ldots, n - 1 } \subsetneq I.
  \end{equation*}

  Therefore, \( f \in I \) since \( f \) divides \( c \). We have chosen \( f \) to be an arbitrary member of \( \ker \Phi \), which implies \( \ker \Phi \subseteq I \).

  We have already shown that \( I \subseteq \ker \Phi \). We thus conclude that \( I = \ker \Phi \) and
  \begin{equation*}
    \BbbF_q[X_1, \ldots, X_m] / I \cong \fun(\BbbF_q^m, \BbbF_q).
  \end{equation*}
\end{proof}

\begin{definition}\label{def:transcendetal_element}
  We say that the element \( a \in \BbbK \) of the field extension \( \BbbK \) of \( \Bbbk \) is \term{transcendental} over \( \BbbK \) if it is \hyperref[def:algebraic_dependence]{algebraically independent}.

  If \( a \) is not transcendental, we say that it is \term{algebraic}. If every element of \( \BbbK \) is algebraic over \( \Bbbk \), we say that \( \BbbK \) is an \term{algebraic extension} of \( \Bbbk \).
\end{definition}

\begin{proposition}\label{thm:field_is_algebraic_over_itself}
  Every field is an \hyperref[def:transcendental_element]{algebraic extension} of itself.
\end{proposition}
\begin{proof}
  Every element \( a \in \BbbK \) is a root of the polynomial \( X - a \).
\end{proof}

\begin{theorem}[Euler's constant is transcendental]\label{thm:eulers_constant_is_transcendental}
  \hyperref[def:exponential_function]{Euler's constant} \( e \) is \hyperref[def:transcendetal_element]{transcendental} over \( \BbbQ \).
\end{theorem}

\begin{theorem}[Pi is transcendental]\label{thm:pi_is_transcendental}\mcite[454]{Knapp2016BasicAlgebra}
  The number \hyperref[def:pi]{\( \pi \)} is \hyperref[def:transcendetal_element]{transcendental} over \( \BbbQ \).
\end{theorem}

\begin{example}\label{ex:polynomials_over_pi}
  \Fullref{thm:pi_is_transcendental} implies that the polynomials \( \BbbQ[X] \) can be embedded into \( \BbbR \) via \( \Phi_\pi: \BbbQ[X] \to \BbbR \). We can identify a polynomial
  \begin{equation*}
    p(X) = \sum_{i=0}^n a_k X^k
  \end{equation*}
  with rational coefficients with the number
  \begin{equation*}
    p(\pi) = \sum_{i=0}^n a_k \pi^k.
  \end{equation*}
\end{example}

\begin{definition}\label{def:finite_field_extension}
  If \( \BbbK \) is \hyperref[thm:vector_space_dimension]{finite-dimensional vector space} over \( \Bbbk \), we say that \( \BbbK \) is a \term{finite extensions} of \( \Bbbk \).
\end{definition}

\begin{lemma}\label{thm:finite_field_extensions_are_algebraic}
  Every \hyperref[def:finite_field_extension]{finite field extension} is \hyperref[def:transcendetal_element]{algebraic}.
\end{lemma}
\begin{proof}
  Let \( \BbbK \) be a field extension of \( \Bbbk \). Consider the evaluation map \( \Phi_a: \Bbbk[X] \to \Bbbk[u] \) for some \( u \in \BbbK \).

  Since the polynomials \( X^k \) for \( k = 0, 1, 2, \ldots \) form a basis for \( \Bbbk[X] \). If \( \Phi_a \) is injective, then \( \Phi_a(X_k) \) are linearly independent over \( \BbbK \). But \( \BbbK \) has finite dimension over \( \Bbbk \).

  The obtained contradiction shows that \( \Phi_a \) is not injective.
\end{proof}

\begin{definition}\label{def:algebraically_closed_field}\mcite[prop. 9.20]{Knapp2016BasicAlgebra}
  We say that the field \( \BbbK \) is algebraically closed if any of the equivalent conditions are satisfied:
  \begin{thmenum}
    \thmitem{def:algebraically_closed_field/trivial_algebraic_extensions} \( \BbbK \) has no nontrivial algebraic \hyperref[def:transcendental_element]{extensions}.
    \thmitem{def:algebraically_closed_field/linear_irreducible_polynomials} Every irreducible polynomial in \( \BbbK[X] \) is linear.
    \thmitem{def:algebraically_closed_field/at_least_one_root} Every nonconstant polynomial in \( \BbbK[X] \) has at least one root in \( \BbbK \).
    \thmitem{def:algebraically_closed_field/factorization} Every polynomial in \( \BbbK[X] \) \hyperref[def:irreducible_factorization]{factors} into a product of linear polynomials.
    \thmitem{def:algebraically_closed_field/exactly_n_roots} Every polynomial in \( \BbbK[X] \) of degree \( n \) has exactly \( n \) roots in \( \BbbK \), counting the root multiplicities.
  \end{thmenum}
\end{definition}
\begin{proof}
  \ImplicationSubProof{def:algebraically_closed_field/trivial_algebraic_extensions}{def:algebraically_closed_field/linear_irreducible_polynomials} Let \( p(X) \) be an irreducible polynomial in \( \BbbK[X] \).

  Since \( \BbbK[X] \) is a unique factorization domain, it satisfies \fullref{def:unique_factorization_domain/primes_and_ideals}, and hence \( p(X) \) is a prime element. Thus, \( \braket {p(X)} \) is a \hyperref[def:semiring_ideal/prime]{prime ideal} in \( \BbbK[X] \).

  Since \( \BbbK[X] \) is a principal ideal domain, by \fullref{thm:def:principal_ideal_domain/prime_ideal_is_maximal}, \( \braket{ p(X) } \) is also a maximal ideal. By \fullref{def:semiring_ideal/maximal/quotient}, the quotient \( Q \coloneqq \BbbK[X] / \braket{ p(X) } \) is a field. The vectors \( 1, X, X^2, \cdots, X^n \) for a basis of \( Q \) over \( \BbbK \), where \( n \) is the degree of \( p(X) \).

  By \fullref{thm:finite_field_extensions_are_algebraic}, \( Q \) is an algebraic extension of \( \BbbK \). Since \( \BbbK \) has no nontrivial algebraic extensions, it follows that \( \BbbK = Q \). Thus, \( Q \) has dimension \( 1 \), and we have already discussed that \( \dim Q = \deg p \). Therefore, \( p \) is a linear polynomial.

  \ImplicationSubProof{def:algebraically_closed_field/linear_irreducible_polynomials}{def:algebraically_closed_field/at_least_one_root} Suppose that every irreducible polynomial is linear.

  By \fullref{thm:def:unique_factorization_domain/polynomial_ring}, \( \BbbK[X] \) is a unique factorization domain, and thus there exist irreducible polynomials \( q_1(X), \ldots, q_n(X) \) and a unit \( a \) such that
  \begin{equation*}
    p(X) = a q_1(X) \cdots q_n(X).
  \end{equation*}

  By assumption, the irreducible polynomials are linear, and hence have roots. Therefore, \( p(X) \) has at least one root.

  \ImplicationSubProof{def:algebraically_closed_field/at_least_one_root}{def:algebraically_closed_field/factorization} Suppose that \( u_1 \) is a root of \( p(X) \). Then \( p(X) \) is divisible by \( (X - u_1) \). Using induction on the degree of \( p(X) \), we can factor \( p(X) \) into
  \begin{equation*}
    p(X) = a (X - u_1) (X - u_2) \cdots (X - u_n),
  \end{equation*}
  where \( a \) is a unit of \( \BbbK \). This is the desired factorization.

  \ImplicationSubProof{def:algebraically_closed_field/factorization}{def:algebraically_closed_field/exactly_n_roots} Follows from the equivalence in \fullref{def:polynomial_root} by induction on the polynomial degree. By \fullref{thm:def:integral_domain/root_limit}, the number of roots is bounded by \( n \).

  \ImplicationSubProof{def:algebraically_closed_field/exactly_n_roots}{def:algebraically_closed_field/trivial_algebraic_extensions} Suppose that every nonconstant polynomial of degree \( n \) has exactly \( n \) roots in \( \Bbbk \) and let \( \BbbK \) be an algebraic extension of \( \Bbbk \).

  By \fullref{thm:def:integral_domain/root_limit}, every polynomial in \( \BbbK[X] \) has at most \( n \) roots. By assumption, every root of every polynomial is contained in \( \Bbbk \). Since \( \BbbK \) is algebraic over \( \Bbbk \), it follows that every element of \( \BbbK \) is a root of some polynomial. Therefore, \( \BbbK = \Bbbk \).
\end{proof}

\begin{proposition}\label{thm:no_finite_extensions_of_closed_fields}
  An \hyperref[def:algebraically_closed_field]{algebraically closed field} has no nontrivial finite extension fields.
\end{proposition}
\begin{proof}
  Follows from \fullref{thm:finite_field_extensions_are_algebraic} applied to \fullref{def:algebraically_closed_field/trivial_algebraic_extensions}.
\end{proof}

\begin{theorem}[{Weak \term[en=zero locus theorem]{nullstellensatz}}]\label{thm:weak_nullstellensatz}
  Let \( \mscrK \) be an \hyperref[def:algebraically_closed_field]{algebraically closed field} and let \( \BbbK[X_1, \ldots, X_n] \) be its \hyperref[def:polynomial_algebra]{polynomial ring} in \( n \) indeterminates.

  The ideal \( M \) of \( \BbbK[X_1, \ldots, X_n] \) is \hyperref[def:semiring_ideal/maximal]{maximal} if and only if there exist elements \( u_1, \ldots, u_n \) of \( \BbbK \) such that
  \begin{equation*}
    M = \braket{ X_1 - u_1, \ldots, X_n - u_n }.
  \end{equation*}
\end{theorem}


\section{On-policy Prediction with Approximation}

In this section we consider the applications of function approximation techniques in reinforcement learning, to learn mappings from states to values. Typically we will consider parametric functional forms, in which case we can achieve a reduction in dimensionality of the problem (number of parameters smaller than state space). In this way, the function generalises between states, as the update of one state impacts the value of another.\\

Function approximation techniques are applicable to partially observable problems, in which the full state space is not available to the agent. A function approximation scheme which ignores certain aspects of the space behaves just as if those aspects are unobservable.

\subsection{Value-function Approximation}
Many techniques from supervised learning are applicable to learning value functions from experience, but not all are equipped to deal with the non-stationarity that often occurs in RL. In RL it is also important to be able to learn online.\\

\subsection{The Prediction Objective ($\VE{}$)}
Define a state distribution $\mu(s) \geq 0$, $\sum_s \mu(s) = 1$ that represents how much we care about each state $s$. Given an estimator $\hat{v}(s, \vec{w})$ of $v_\pi(s)$, parameterised by $\vec{w}$, we define our objective function as the \emph{Mean Squared Value Error}
\begin{equation}
    \VE{} \doteq \sum_{s \in \S{}} \mu(s) \left[ v_\pi(s) - \hat{v}(s, \vec{w})\right]^2.
\end{equation}\\

Often we choose $\mu(s)$ to be the fraction of time spent in $s$. Under on-policy training this is referred to as the \emph{on-policy distribution}. In continuing tasks, this is the stationary distribution under $\pi$.\\

At this stage it is not clear that we have chosen the correct (or even a good) objective function, since the ultimate goal is a good policy for the task. For now, will continue with $\VE{}$ nonetheless.

\subsubsection*{The on-policy distribution in episodic tasks}
In an episodic task the on-policy distribution depends on how the initial states of the episode are chosen. Let $h(s)$ be the probability that an episode begins in state $s$ and $\eta(s)$ be the expected time spent in $s$ per episode. Note that you can either start in $s$ or transition there from $\bar{s}$, so
\[
    \eta(s) = h(s) + \sum_{\bar{s}} \eta(\bar{s}) \sum_a \pi(a \vert{} \bar{s}) p(s \vert{} \bar{s}, a) \quad \forall s \in \S{}.
\]
One can solve this system for $\eta$, then take the on-policy distribution as
\[
    \mu(s) = \frac{\eta(s)}{\sum_{s'}\eta(s')} \quad \forall s \in \S{}.
\]
This is the natural choice without discounting. With discounting we consider it a form of termination and include a factor of $\gamma$ in the second term of the recurrence relation above.

\subsection{Stochastic-gradient and Semi-gradient Methods}
\subsubsection*{(Stochastic) Gradient Descent}
We assume that states appear in examples with the same distribution $\mu(s)$, in which case a good strategy is to minimise our loss function on observed examples. \emph{Stochastic gradient-descent} moves the weights in the direction of decreasing $\VE{}$:
\begin{align}
    \vec{w}_{t+1} &=  \vec{w}_t - \frac12 \alpha \grad_{\vec{w}} \left[v_\pi(S_t) - \hat{v}(S_t, \vec{w})\right]^2 \\
                  &= \vec{w}_t + \alpha \left[ v_\pi(S_t) - \hat{v}(S_t, \vec{w})\right] \grad_{\vec{w}} \hat{v}(S_t, \vec{w}).
\end{align}
Of course, we might not know the true value function exactly, we will likely only have access to some approximation of it $U_t$, possibly corrupted by noise or got from bootstrapping with our latest estimate. In these cases we cannot perform the above computation, but we can still make the general SGD update
\begin{equation}
    \vec{w}_{t+1} = \vec{w}_t + \alpha \left[ U_t - \hat{v}(S_t, \vec{w})\right] \grad_{\vec{w}} \hat{v}(S_t, \vec{w})
\end{equation}
If $U_t$ is an unbiased estimate of the state value for each $t$, then the sequence $\vec{w}_t$ is guaranteed to converge to a local optimum under the usual stochastic approximation conditions for decreasing $\alpha$.\\

The Monte Carlo target $U_t = G_t$ is an unbiased estimator, so locally optimal convergence is guaranteed in this case. Algorithm is given below.\\

\includegraphics[width=\textwidth]{\NotesImages/gradient_monte_carlo.png}\\

\subsubsection*{Semi-Gradient Descent}
We don't get the same convergence guarantees if we use bootstrapping estimates of the value function in our update target, for instance if we had used the TD(0) update $U_t = R_{t+1} + \gamma \hat{v}(S_{t+1}, \vec{w})$. This is because the target now depends on the parameters $\vec{w}$, so the gradient is not exactly the gradient of our loss function -- it only takes into account the change on our estimate with respect to $\vec{w}$. For this reason we call updates such as this \emph{semi-gradient methods}.\\

Semi-gradient methods are often preferable to pure gradient methods since they can offer much faster learning, in spite of not giving the same convergence guarantees. A prototypical choice is the TD(0) update, an algorithm for which is given in the box below.\\

\includegraphics[width=\textwidth]{\NotesImages/semi_gradient_td0.png}\\

\subsubsection*{State Aggregation}
\emph{State aggregation} is a simple form of generalising in which we group together states and fix them to have the same estimated value.

\subsection{Linear Methods}
As always, linear methods of function approximation are an important special case
\begin{equation}
    \hat{v}(s, \vec{w}) = \vec{w}^\top \vec{x}(s)
\end{equation}
where $\vec{x}(s)$ are feature vectors, vectors of functions (features) $x_i: \S{} \to \mathbb{R}$. The SGD update for the linear model is
\begin{equation}
    \vec{w}_{t+1} = \vec{w}_t + \alpha \left[ U_t - \hat{v}(S_t, \vec{w})\right] \vec{x}(s).
\end{equation}
Naturally, the linear case is the most studied and the majority of convergence results for learning systems are for this case (or simpler). In particular, there is the benefit that there is a unique global optimum for our loss function (in the non-degenerate case).

\subsubsection*{Convergence of Linear TD(0)}
The semi-gradient TD(0) algorithm is known to converge under linear function approximation. The point converged to is not the global optimum, but a point near the local optimum. We consider this case in more detail. First write $\vec{x}_t = \vec{x}(S_t)$ then rearrange the update
\begin{align}
    \vec{w}_{t+1} &= \vec{w}_t + \alpha \left(R_{t+1} + \gamma \vec{w}_t^\top \vec{x}_{t+1} - \vec{w}_{t}^\top \vec{x}_t\right)\vec{x}_t \\
                  &= \vec{w}_t + \alpha \left(R_{t+1}\vec{x}_t - \vec{x}_t(\vec{x}_t -  \gamma\vec{x}_{t+1})^\top\vec{w}_t\right).
\end{align}
Now note that we can write
\[
    \E{}[\vec{w}_{t+1} \vert{} \vec{w}_t] = \vec{w}_t + \alpha (\vec{b} - \mathrm{A}\vec{w}_t)
\]
where $\vec{b} = \E{}[R_{t+1}\vec{x}_t]$ and $\mathrm{A} = \E{}\left[\vec{x}_t(\vec{x}_t - \gamma\vec{x}_{t+1})^\top\right]$. It's clear now that in a steady state we must have (can be shown that $\mathrm{A}$ positive definite and so invertible)
\[
    \vec{w}_{\text{TD}} = \mathrm{A}^{-1}\vec{b}.
\]
We call this point the \emph{TD fixed point}, linear semi-gradient TD(0) converges to this point. (In the notes there is a box with some details.)\\

At the TD fixed point (in the continuing case) it has been proven that $\VE$ is within a bounded expansion of the lowest possible error
\begin{equation}
    \VE(\vec{w}_{\text{TD}}) \leq \frac{1}{1 - \gamma} \min_{\vec{w}} \VE(\vec{w}).
\end{equation}
It is often the case that $\gamma$ is close to 1, so this region can be quite large. The TD method has substantial loss in asymptotic performance. Regardless of this, it still has much lower variance than MC methods and can thus be faster. The desired update method will depend on the task at hand.\\

\subsubsection*{Other Linear Updates}
Linear semi-gradient DP $U_t = \sum_a \pi(a \vert{} S_t) \sum_{s', r} p(s', r \vert{} S_t, a) [r + \gamma \hat{v}(s', \vec{w}_t)]$ with updates according to the on-policy distribution also converged to the TD fixed point. There are convergence results for other step methods we have considered too. Critical to all of these is that updates are taken according to the on-policy distribution. For other update distributions, bootstrapping methods can diverge to infinity. $n$-step semi-gradient TD is given in the box below.\\

\includegraphics[width=\textwidth]{\NotesImages/semi_gradient_tdn.png}\\

\subsection{Feature Construction for Linear Methods}
Discussed in this section
\begin{itemize}
    \item Polynomial Basis
    \item Fourier Basis
    \item One could use other orthogonal function bases but they are yet to see application in RL.
    \item Radial Basis Functions. (Offer little advantage over coarse coding with circles, but greatly increases computational complexity)
\end{itemize}

\setcounter{subsubsection}{2}
\subsubsection{Coarse Coding}
One way of promoting generalisation between states is to cover the state-space in overlapping regions, with each region representing a feature. If the state is being considered, then all regions that contain this state will be activated. The amount of overlap of the receptive fields (the states which can activate a feature) dictates the breadth of generalisation.

\subsubsection{Tile Coding}
A \emph{tiling} of a continuous state space is form of coarse-coding that creates a partition of the state space (all of the state space is covered but elements of the tiling do not overlap). We call a sub-region of a tiling a \emph{tile}. One might introduce multiple overlapping tilings to incorporate generalisation.\\

\includegraphics[width=\textwidth]{\NotesImages/tile_coding.png}\\

An advantage of tilings is that, because each tiling forms a partition, the total number of features active at any one time is just the total number of tilings used. So $\alpha = \frac{1}{kn}$, where $n$ is the number of tilings, results in $k$-trial learning. That is, on average the learning asymptotes after $k$ presentations of each state (assuming all updates use the same, constant target).\\

Tile coding is computationally efficient and may be the most practical feature representation for modern sequential digital computers.\\

A useful trick for reducing memory requirements is \emph{hashing}. One can essentially hash the state space, then tile the hashed values. This means that each tile in the hashed space will represent (multiple) pseudo-randomly distributed tiles in the original space. Since only a small proportion of the state space needs to have high resolution value estimates, this can be a good way to reduce memory with little loss in performance.

\subsection{Selecting Step-Size Parameters Manually}
In the tabular case, taking $\alpha = \frac1\tau$ will mean that the estimate for a state will approach the mean of its targets, with the most recent targets having the greatest effect, in about $\tau$ experiences.\\

With function approximation, there is not a clear notion of the number of visits to a state because of continuous degrees of generalisation. However, a sensible consideration for learning from $\tau$ presentations is
\begin{equation}
    \alpha = \frac{1}{\tau \E{}[\vec{x}^\top\vec{x}]}.
\end{equation}

\subsection{Nonlinear Function Approximation: Artificial Neural Networks}
These of course see a lot of application in RL, especially with deep learning. There are some good review articles on the web.

\subsection{Least-Squares TD}
We saw earlier that TD(0) with linear function approximation converges to the TD fixed point
\[
    \vec{w}_{\text{TD}} = \mathrm{A}^{-1}\vec{b},
\]
where $\vec{b} = \E{}[R_{t+1}\vec{x}_t]$ and $\mathrm{A} = \E{}\left[\vec{x}_t(\vec{x}_t - \gamma\vec{x}_{t+1})^\top\right]$. Previously we computed the solution iteratively, but this is a waste of data! We could compute the MLE of $\mathrm{A}$ and $\vec{b}$ and then use those. This is the \emph{Least-Squares TD Algorithm}, it uses the estimators
\begin{equation}
    \hat{\mathrm{A}}_t = \sum_{k=0}^{t-1} x_k ( x_k - \gamma x_{k+1})^\top + \epsilon \mathrm{I} \quad \mathrm{and} \quad \hat{\vec{b}}_t = \sum_{k=0}^{t-1} R_{t+1} x_k
\end{equation}
where we introduce $\epsilon > 0$ to ensure that the sequence of $\hat{\mathrm{A}}_t$ are each invertible. (These are estimates of $t\mathrm{A}$ and $t\vec{b}$ but the $t$ cancel out.)\\

This is the most data efficient form of TD(0), but it is also more computationally intensive. Implementing incrementally and with tricks to do the matrix inverse (because of the particular form of $\mathrm{A}$ as sum of outer products), one can do this in $O(d^2)$ computations, where $d$ is the number of parameters/features (note that this is independent of $t$). (For comparison, the semi-gradient TD(0) method needs $O(d)$ computations.) The formula for $\mathrm{A}$ is
\begin{align}
    \hat{\mathrm{A}}_t &= \left(\hat{\mathrm{A}}_{t-1} + x_t ( x_t - \gamma x_{t+1})^\top\right)^{-1} \\
                      &= \hat{\mathrm{A}}_{t-1}^{-1} - \frac{\hat{\mathrm{A}}_{t-1}^{-1} x_t( x_t - \gamma x_{t+1})^\top \hat{\mathrm{A}}_{t-1}^{-1}}{1 + x_t ( x_t - \gamma x_{t+1})^\top \hat{\mathrm{A}}_{t-1}^{-1} x_t}
\end{align}
To store $\hat{A}_{t-1}$ LSTD also needs $O(d^2)$ memory. LSTD has no step-size parameter, which means that it never forgets -- this can be a blessing or a curse depending on the application. The choice between LSTD and semi-gradient TD will depend on the application, for instance the computation available and the importance of learning quickly. Pseudocode for LSTD is given below.\\

\includegraphics[width=\textwidth]{\NotesImages/lstd.png}\\

\subsection{Memory-based Function Approximation}
As an alternative to the parametric approaches discussed above, we might instead store all the training examples and execute an algorithm on the whole dataset when required, such as LOESS or nearest neighbour averaging. This approach is sometimes called \emph{lazy learning}. The methods that go with this are non-parametric function approximation schemes. One can often evaluate the function approximation locally in the neighbourhood of the current state, which helps with the curse of dimensionality.

\subsection{Kernel-based Function Approximation}
Using kernels to define similarities between states for generalisation, e.g. kernel regression for state values.

\subsection{Looking Deeper at On-policy Learning: Interest and Emphasis}
Sometimes we are not equally interested in each state, so limited resources can be better spent than to treat every state equally. For instance, in discounted episodic problems we might be more interested in starting states because later rewards are discounted. \\

Introduce the scalar random variable $I_t \geq 0 $ called $interest$, the degree of interest we have in accurately valuing the state at time $t$. If we don't care at all about the state then $I_t = 0$, if we fully care then it might be 1 (but it is formally allowed to take any non-negative value). The interest can be set in any causal way. The distribution in our loss function $\VE$ is then defined as the distribution of states encountered when following the target policy, weighted by the interest.\\

We also introduce the scalar random variable $M_t \geq 0$, called the \emph{emphasis}. The emphasis multiplies the learning update at each time-step. For general $n$-step learning
\begin{equation}
    \vec{w}_{t+n} = \vec{w}_{t + n - 1} + \alpha M_t [G_{t:t+n} - \hat{v}(S_t, \vec{w}_{t+n-1})] \grad_{\vec{w}} \hat{v}(S_t, \vec{w}_{t+n-1}) \quad 0 \leq t < T,
\end{equation}
with the emphasis defined recursively as
\begin{equation}
    M_t = I_t + \gamma^n M_{t-n}
\end{equation}
with $M_t = 0$ $\forall t < 0$.


%!TEX root = inversion.tex
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Copyright (c) 2003-2018 by The University of Queensland
% http://www.uq.edu.au
%
% Primary Business: Queensland, Australia
% Licensed under the Apache License, version 2.0
% http://www.apache.org/licenses/LICENSE-2.0
%
% Development until 2012 by Earth Systems Science Computational Center (ESSCC)
% Development 2012-2013 by School of Earth Sciences
% Development from 2014 by Centre for Geoscience Computing (GeoComp)
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{DC resistivity inversion: 3D}\label{sec:forward DCRES}
This section will discuss DC resistivity\index{DC forward} forward modelling, as well as an \escript
class which allows for solutions of these forward problems. The DC resistivity
forward problem is modelled via the application of Ohm's Law to the flow of current
through the ground. When sources are treated as a point sources and Ohm's Law
is written in terms of the potential field, the equation becomes:
\begin{equation} \label{ref:dcres:eq1}
\nabla \cdot (\sigma \nabla \phi) = -I \delta(x-x_s) \delta(y-y_s) \delta(z-z_s)
\end{equation}
Where $(x,y,z)$ and $(x_s, y_s, z_s)$ are the coordinates of the observation and source
points respectively. The total potential, $\phi$, is split into primary and secondary
potentials $\phi = \phi_p + \phi_s$, where the primary potential is analytically calculated
as a flat half-space background model with conductivity of $\sigma_p$.
The secondary potential is due to conductivity deviations
from the background model and has its conductivity denoted as $\sigma_s$.
This approach effectively removes the singularities of the Dirac delta
source and provides more accurate results \cite{rucker2006three}.
An analytical solution is available for the primary potential of a uniform half-space due to a single pole source and is given by:
\begin{equation} \label{ref:dcres:eq2}
\phi_p = \frac{I}{2 \pi \sigma_1 R}
\end{equation}
Where $I$ is the current and $R$ is the distance from the observation points to the source.
In \escript the observation points are the nodes of the domain and $R$ is given by
\begin{equation} \label{ref:dcres:eq3}
R = \sqrt{(x-x_s)^2+(y-y_s)^2 + z^2}
\end{equation}
The secondary potential, $\phi_s$, is given by
\begin{equation}\label{ref:dcres:eq4}
-\mathbf{\nabla}\cdot\left(\sigma\,\nabla \phi_s \right)  =
 \mathbf{\nabla}\cdot\left( \left(\sigma_p-\sigma\right)\,\nabla \phi_p  \right)
\end{equation}
where $\sigma_p$ is the conductivity of the background half-space.
The weak form of above PDE is given by multiplication of a suitable test function, $w$, and integrating over the domain $\Omega$:
\begin{multline}\label{ref:dcres:eq5}
-\int_{\partial\Omega} \sigma\,\nabla \phi_s  \cdot \hat{n} w\,ds +
 \int_{\Omega} \sigma\,\nabla \phi_s  \cdot \nabla w\,d\Omega =\\
-\int_{\partial\Omega} \left(\sigma_p-\sigma\right)\,\nabla \phi_p
\cdot \hat{n} w\,ds + \int_{\Omega} \left(\sigma_p-\sigma\right)\,\nabla \phi_p  \cdot \nabla w\,d\Omega
\end{multline}
The integrals over the domain boundary provide the boundary conditions which are
implemented as Dirichlet conditions (i.e. zero potential) at all interfaces except the
top, where Neumann conditions apply (i.e. no current flux through the air-earth interface).
From the integrals over the domain, the \escript coefficients can be deduced: the
left-hand-side conforms to \escript coefficient $A$, whereas the right-hand-side agrees
with the coefficient $X$ (see User Guide).

A number a of different configurations for electrode set-up are available \cite[pg 5]{LOKE2014}.
An \escript class is provided for each of the following survey types:
\begin{itemize}
\item Wenner alpha
\item Pole-Pole
\item Dipole-Dipole
\item Pole-Dipole
\item Schlumberger
\end{itemize}

These configurations are comprised of at least one pair of current and potential
electrodes separated by a distance $a$. In those configurations which use $n$,
electrodes in the currently active set may be separated by $na$. In the classes
that follow, the specified value of $n$ is an upper limit. That is $n$ will
start at 1 and iterate up to the value specified.

\subsection{Usage}
The DC resistivity forward modelling classes are specified as follows:

\begin{classdesc}{WennerSurvey}{self, domain, primaryConductivity, secondaryConductivity,
current, a, midPoint, directionVector, numElectrodes}
\end{classdesc}

\begin{classdesc}{polepoleSurvey}{domain, primaryConductivity, secondaryConductivity,
current, a, midPoint, directionVector, numElectrodes}
\end{classdesc}

\begin{classdesc}{DipoleDipoleSurvey}{self, domain, primaryConductivity, secondaryConductivity,
current, a, n, midPoint, directionVector, numElectrodes}
\end{classdesc}

\begin{classdesc}{PoleDipoleSurvey}{self, domain, primaryConductivity, secondaryConductivity,
current, a, n, midPoint, directionVector, numElectrodes}
\end{classdesc}

\begin{classdesc}{SchlumbergerSurvey}{self, domain, primaryConductivity, secondaryConductivity,
current, a, n, midPoint, directionVector, numElectrodes}
\end{classdesc}

\noindent Where:
\begin{itemize}
\item \texttt{domain} is the domain which represent the half-space of interest.
it is important that a node exists at the points where the electrodes will be placed.
\item \texttt{primaryConductivity} is a data object which defines the primary conductivity
it should be defined on the ContinuousFunction function space.
\item \texttt{secondaryConductivity} is a data object which defines the secondary conductivity
it should be defined on the ContinuousFunction function space.
\item \texttt{current} is the value of the injection current to be used in amps this is a currently a
constant.
\item \texttt{a} is the electrode separation distance.
\item \texttt{n} is the electrode separation distance multiplier.
\item \texttt{midpoint} is the centre of the survey. Electrodes will spread from this point
in the direction defined by the direction vector and in the opposite direction, placing
half of the electrodes on either side.
\item \texttt{directionVector} defines as the direction in which electrodes are spread.
\item \texttt{numElectrodes} is the number of electrodes to be used in the survey.
\end{itemize}

When calculating the potentials the survey is moved along the set of electrodes.
The process of moving the electrodes along is repeated for each consecutive value of $n$.
As $n$ increases less potentials are calculated, this is because a greater spacing is
required and hence some electrodes are skipped. The process of building up these
pseudo-sections is covered in greater depth by Loke (2014)\cite[pg 19]{LOKE2014}.
These classes all share common member functions described below. For the surveys
where $n$ is not specified only one list will be returned.

\begin{methoddesc}[]{getPotential}{}
Returns 3 lists, each made up of a number of lists containing primary, secondary and total
potential differences. Each of the lists contains $n$ sublists.
\end{methoddesc}

\begin{methoddesc}[]{getElectrodes}{}
Returns a list containing the positions of the electrodes
\end{methoddesc}

\begin{methoddesc}[]{getApparentResistivityPrimary}{}
Returns a series of lists containing primary apparent resistivities one for each
value of $n$.
\end{methoddesc}

\begin{methoddesc}[]{getApparentResistivitySecondary}{}
Returns a series of lists containing secondary apparent resistivities one for each
value of $n$.
\end{methoddesc}

\begin{methoddesc}[]{getApparentResistivityTotal}{}
Returns a series of lists containing total apparent resistivities, one for each
value of $n$. This is generally the result of interest.
\end{methoddesc}

The apparent resistivities are calculated by applying a geometric factor to the
measured potentials.


%------------------------------------------------------------------------------
\section{Covariance matrix propagation}
\label{sec:propagation-covariance}
%------------------------------------------------------------------------------

\subsection{State space representation}
\label{sec:state-space-notation}
%------------------------------------------------------------------------------

The orbit determination problem is typically approached by using a state space representation. For a very detailed description of the associated concepts, referencing the textbook
of \citet{tapley2004} is highly recommended, which also served as the basis for the following derivation.

The \gls{sym:stateDim}-dimensional state vector $\xt$ of a satellite can be defined as
\begin{equation}
 \xt = \left(\rt^T, \vt^T, \gls{sym:cvec}^T \right)^T,
\end{equation}
with \gls{sym:cvec} being a multi-dimensional vector of constant parameters to be solved for in the orbit determination process, which might, for example,
include the drag and \acrshort{acr:srp} coefficients. Due to the non-linear nature of orbital motion, the conversion of the second-order differential equations of motion into a vector of first-order differential equations is written as
\begin{equation}
 \frac{d}{dt} \xt = \gls{sym:functionvec}\left(\gls{sym:state},\gls{sym:t}\right). \label{eq:non-linear-system}
\end{equation}
As the state, in general, cannot be observed directly, an additional equation, the \textit{measurement model} \citep{maybeck1979}, is required to map the state vector to the observations:
\begin{equation}
 \yt = \gls{sym:functiongvec}\left(\gls{sym:state},\gls{sym:t}\right) + \boldsymbol{\gls{sym:obsError}}, \label{eq:non-linear-obs}
\end{equation}
here, \gls{sym:obsError} is the error in the observation. With {\yt} being \gls{sym:obsDim}-dimensional, and, in general, $\gls{sym:obsDim}<\gls{sym:stateDim}$, for the orbit determination the number of
observations \gls{sym:numObs} guarantees that $\gls{sym:numObs}\times\gls{sym:obsDim} >> \gls{sym:stateDim}$.

A linearization is obtained by introducing a reference trajectory, denoted by an index \gls{idx:ref}, and work with the deviations
\begin{align}
 \Delta\xt &= \xt - \xref, \label{eq:deviation-ref} \\
 \Delta\yt &= \yt - \yref, \label{eq:deviation-ref-y}
\end{align}
which are assumed to be small - a concept familiar to Encke's method for perturbed orbits. Expanding \eq{eq:non-linear-system} and \eq{eq:non-linear-obs} about the reference state gives
\begin{align}
 \xdot &= \gls{sym:functionvec}\left(\gls{sym:state}_{\gls{idx:ref}},\gls{sym:t}\right) + \left(\frac{\partial \gls{sym:functionvec}\left(\gls{sym:state},\gls{sym:t}\right)}{\partial
\xt}\right)_{\gls{idx:ref}} \cdot \left(\xt - \xref\right) + \mathcal{O}\left(\Delta\xt^2\right), \\
\yt &= \gls{sym:functiongvec}\left(\gls{sym:state}_{\gls{idx:ref}},\gls{sym:t}\right) + \left(\frac{\partial \gls{sym:functiongvec}\left(\gls{sym:state},\gls{sym:t}\right)}{\partial
\xt}\right)_{\gls{idx:ref}} \cdot \left(\xt - \xref\right) + \mathcal{O}\left(\Delta\xt^2\right) + \boldsymbol{\gls{sym:obsError}},
\end{align}
or, combined with \eq{eq:deviation-ref} and \eq{eq:deviation-ref-y} and neglecting higher-order terms:
\begin{align}
 \Delta \xdot &= \left(\frac{\partial \xdot}{\partial \xt}\right)_{\gls{idx:ref}} \Delta \xt, \\
 \Delta \yt &= \left(\frac{\partial \gls{sym:functiongvec}\left(\gls{sym:state},\gls{sym:t}\right)}{\partial \xt}\right)_{\gls{idx:ref}} \Delta \xt + \gls{sym:obsError}.
\end{align}
Defining the system matrix $\gls{sym:systemMat}\left(\gls{sym:t}\right)$ as
\begin{equation}
 \gls{sym:systemMat}\left(\gls{sym:t}\right) \equiv \left(\frac{\partial \xdot}{\partial \xt}\right)_{\gls{idx:ref}}
\end{equation}
and the output matrix as
\begin{equation}
 \gls{sym:outputMat}\left(\gls{sym:t}\right) \equiv \left(\frac{\partial \gls{sym:functiongvec}\left(\gls{sym:state},\gls{sym:t}\right)}{\partial \xt}\right)_{\gls{idx:ref}}
\end{equation}
the homogeneous part of the state space representation is obtained for the linearized system:
\begin{align}
 \Delta \xdot &= \gls{sym:systemMat}\left(\gls{sym:t}\right) \Delta \xt, \label{eq:diff-homo} \\
 \Delta \yt   &= \gls{sym:outputMat}\left(\gls{sym:t}\right) \Delta \xt + \gls{sym:obsError}. \label{eq:meas-model-linearized}
\end{align}
The general solution for \eq{eq:diff-homo} is:
\begin{align}
 \Delta \xt &= \frac{\partial \Delta \xt}{\partial \Delta\gls{sym:state}_0} \Delta \gls{sym:state}_0 \\
            &= \frac{\partial \left(\xt - \xref\right)}{\partial \left(\gls{sym:state}_0 - \gls{sym:state}_{\gls{idx:ref},0}\right)} \Delta \gls{sym:state}_0
\label{eq:ref-constant}\\
            &= \frac{\partial \xt}{\partial \gls{sym:state}_0} \Delta \gls{sym:state}_0 \label{eq:set-homo}
\end{align}
given the initial condition $\gls{sym:state}\left(\gls{sym:t}_0\right)=\gls{sym:state}_0$ and noting that the reference state is constant in the partial derivative in
\eq{eq:ref-constant}. Now the state error transition matrix \gls{sym:set} can be introduced, which translates the state error $\Delta \xt$ from $\gls{sym:t}_0$ to $\gls{sym:t}$:
\begin{equation}
 \set{\gls{sym:t}}{\gls{sym:t}_0} \equiv \frac{\partial \xt}{\partial \gls{sym:state}_0}
\end{equation}
Differentiating \eq{eq:set-homo} provides
\begin{equation}
 \Delta \xdot = \dot{\gls{sym:set}}\left(\gls{sym:t},\gls{sym:t}_0\right) \Delta \gls{sym:state}_0,
\end{equation}
which, together with \eq{eq:set-homo}, can be substituted into \eq{eq:diff-homo}:
\begin{equation}
 \dot{\gls{sym:set}}\left(\gls{sym:t},\gls{sym:t}_0\right) \Delta \gls{sym:state}_0 = \gls{sym:systemMat}\left(\gls{sym:t}\right)
\set{\gls{sym:t}}{\gls{sym:t}_0} \Delta \gls{sym:state}_0,
\end{equation}
finally providing a differential equation for the state error transition matrix, which will later be integrated numerically for the propagation of the covariance matrix:
\begin{equation}
 \dot{\gls{sym:set}}\left(\gls{sym:t},\gls{sym:t}_0\right) = \gls{sym:systemMat}\left(\gls{sym:t}\right) \set{\gls{sym:t}}{\gls{sym:t}_0} \label{eq:set-ode}
\end{equation}

The main advantage of solving for the state error transition matrix using \eq{eq:set-ode} over the direct solution of \eq{eq:diff-homo}
is that the state error transition matrix allows for a simple formulation of the covariance matrix propagation and the determination of
the best estimate of the state vector \citep{tapley2004}.

The influence of system or process noise, which is characterized by the unmodeled accelerations, is provided by another term leading to the general formulation of the state
space representation:
\begin{equation}
 \Delta \xdot = \gls{sym:systemMat}\left(\gls{sym:t}\right) \Delta \xt + \gls{sym:inputMat} \left(\gls{sym:t}\right) \gls{sym:inputVec}\left(\gls{sym:t}\right),
\label{eq:state-space}
\end{equation}
here, $\gls{sym:inputVec}\left(\gls{sym:t}\right)$ is the process noise, while $\gls{sym:inputMat}\left(\gls{sym:t}\right)$ is the input matrix, which converts the unmodeled
accelerations into the quantities of the state vector.

A particular solution to \eq{eq:state-space} can be found via the \textit{variation of constants} method, starting with:
\begin{equation}
 \Delta \xt = \set{\gls{sym:t}}{\gls{sym:t}_0} \gls{sym:vop}\left(\gls{sym:t}\right). \label{eq:vop}
\end{equation}
Differentiating:
\begin{equation}
 \Delta \xdot = \dot{\gls{sym:set}}\left(\gls{sym:t},\gls{sym:t}_0\right) \gls{sym:vop}\left(\gls{sym:t}\right) + \set{\gls{sym:t}}{\gls{sym:t}_0}
\dot{\gls{sym:vop}}\left(\gls{sym:t}\right),
\end{equation}
and substituting into \eq{eq:state-space} provides
\begin{equation}
 \dot{\gls{sym:set}}\left(\gls{sym:t},\gls{sym:t}_0\right) \gls{sym:vop}\left(\gls{sym:t}\right) + \set{\gls{sym:t}}{\gls{sym:t}_0}
\dot{\gls{sym:vop}}\left(\gls{sym:t}\right) = \gls{sym:systemMat}\left(\gls{sym:t}\right) \Delta \xt + \gls{sym:inputMat} \left(\gls{sym:t}\right)
\gls{sym:inputVec}\left(\gls{sym:t}\right).
\end{equation}
This equation can be re-written by substituting the already known relationships for the time derivative of the state error transition matrix, \eq{eq:set-ode},
and \eq{eq:vop}:
\begin{equation}
 \gls{sym:systemMat}\left(\gls{sym:t}\right) \set{\gls{sym:t}}{\gls{sym:t}_0} \gls{sym:vop}\left(\gls{sym:t}\right) + \set{\gls{sym:t}}{\gls{sym:t}_0}
\dot{\gls{sym:vop}}\left(\gls{sym:t}\right) = \gls{sym:systemMat}\left(\gls{sym:t}\right) \set{\gls{sym:t}}{\gls{sym:t}_0} \gls{sym:vop}\left(\gls{sym:t}\right)  +
\gls{sym:inputMat} \left(\gls{sym:t}\right) \gls{sym:inputVec}\left(\gls{sym:t}\right),
\end{equation}
which results in
\begin{equation}
 \set{\gls{sym:t}}{\gls{sym:t}_0} \dot{\gls{sym:vop}}\left(\gls{sym:t}\right) = \gls{sym:inputMat} \left(\gls{sym:t}\right) \gls{sym:inputVec}\left(\gls{sym:t}\right).
\end{equation}
The solution is now obtained by integration:
\begin{equation}
 \gls{sym:vop}\left(\gls{sym:t}\right) = \gls{sym:vop}_0 + \int_{\gls{sym:t}_0}^{\gls{sym:t}} \gls{sym:set}^{-1}\left(\gls{sym:xi},\gls{sym:t}_0\right)
\gls{sym:inputMat}\left(\gls{sym:xi}\right)\gls{sym:inputVec}\left(\gls{sym:xi}\right) d\gls{sym:xi}. \label{eq:result-ct}
\end{equation}

Substituting the result for $\gls{sym:vop}\left(\gls{sym:t}\right)$ from \eq{eq:result-ct} into \eq{eq:vop} results in:
\begin{equation}
 \Delta \xt = \set{\gls{sym:t}}{\gls{sym:t}_0} \gls{sym:vop}_0 + \int_{\gls{sym:t}_0}^{\gls{sym:t}} \set{\gls{sym:t}}{\gls{sym:t}_0}
\gls{sym:set}^{-1}\left(\gls{sym:xi},\gls{sym:t}_0\right)
\gls{sym:inputMat}\left(\gls{sym:xi}\right)\gls{sym:inputVec}\left(\gls{sym:xi}\right) d\gls{sym:xi},
\end{equation}
which can be simplified using the following properties of the state error transition matrix:
\begin{equation}
 \set{\gls{sym:t}}{\gls{sym:t}_0}\gls{sym:set}^{-1}\left(\gls{sym:xi},\gls{sym:t}_0\right) = \set{\gls{sym:t}}{\gls{sym:t}_0} \set{\gls{sym:t}_0}{\gls{sym:xi}}
 = \set{\gls{sym:t}}{\gls{sym:xi}},
\end{equation}
as well as the initial condition $\gls{sym:vop}_0 = \gls{sym:state}_0$ to finally provide the general solution for the inhomogeneous \eq{eq:state-space}:
\begin{equation}
 \Delta \xt = \set{\gls{sym:t}}{\gls{sym:t}_0} \Delta \gls{sym:state}_0 + \int_{\gls{sym:t}_0}^{\gls{sym:t}} \set{\gls{sym:t}}{\gls{sym:xi}}
\gls{sym:inputMat}\left(\gls{sym:xi}\right)\gls{sym:inputVec}\left(\gls{sym:xi}\right) d\gls{sym:xi}. \label{eq:state-space-general-solution}
\end{equation}

The result in \eq{eq:state-space-general-solution} is also referred to as the \textit{matrix superposition integral} \citep{gelb1974}, where the second term describes how an input
(here: process noise \gls{sym:inputVec}) at a time \gls{sym:xi} translates into a state vector change at time \gls{sym:t}.

%------------------------------------------------------------------------------
\subsection{General formulation using the state transition matrix}
\label{sec:propagation-covariance-theory}
%------------------------------------------------------------------------------

Using the state space representation, as introduced in \sect{sec:state-space-notation}, with its general solution given by \eq{eq:state-space-general-solution},
 the covariance matrix at a time \gls{sym:t} follows as:
\begin{alignat}{3}
 \gls{sym:P}\left(\gls{sym:t}\right) = \gls{sym:expVal}\left[\Delta\gls{sym:state}\left(\gls{sym:t}\right)\cdot\Delta\gls{sym:state}\left(\gls{sym:t}
\right)^T\right]&=
\gls{sym:expVal}\Bigl[&&\left(\gls{sym:set}\left(\gls{sym:t},\gls{sym:t}_0\right)\cdot
\Delta\gls{sym:state}\left(\gls{sym:t}_0\right)+
\int_{\gls{sym:t}_0}^{\gls{sym:t}}\gls{sym:set}\left(\gls{sym:t},\gls{sym:xi}\right)\cdot
\gls{sym:inputMat}\left(\gls{sym:xi}\right)\cdot\gls{sym:inputVec}\left(\gls{sym:xi}\right) d\gls{sym:xi}\right)\times \notag \\
& &&\left(\gls{sym:set}\left(\gls{sym:t},\gls{sym:t}_0\right)\cdot
\Delta\gls{sym:state}\left(\gls{sym:t}_0\right)+ \int_{\gls{sym:t}_0}^{\gls{sym:t}}\gls{sym:set}\left(\gls{sym:t},\gls{sym:eta}\right)\cdot
\gls{sym:inputMat}\left(\gls{sym:eta}\right)\cdot\gls{sym:inputVec}\left(\gls{sym:eta}\right) d\gls{sym:eta}\right)^T\Bigr] \notag \\
&= &&\gls{sym:set}\left(\gls{sym:t},\gls{sym:t}_0\right)\cdot\gls{sym:expVal}\left[\Delta\gls{sym:state}\left(\gls{sym:t}_0\right)\cdot
\Delta\gls{sym:state}\left(\gls{sym:t}_0\right)^T\right]\cdot \gls{sym:set}\left(\gls{sym:t},\gls{sym:t}_0\right)^T + \label{eq:full-cov-prop} \\
& &&
\gls{sym:set}\left(\gls{sym:t},\gls{sym:t}_0\right)\cdot\int_{\gls{sym:t}_0}^{\gls{sym:t}}\gls{sym:expVal}\left[\Delta\gls{sym:state}\left(\gls{sym:t}_0\right)
\cdot\gls{sym:inputVec}\left(\gls{sym:eta}\right)^T\right]\cdot\gls{sym:inputMat}\left(\gls{sym:eta}\right)\cdot
\gls{sym:set}\left(\gls{sym:t},\gls{sym:eta}\right) d\gls{sym:eta}+ \notag \\
& &&
\left(\gls{sym:set}\left(\gls{sym:t},\gls{sym:t}_0\right)\cdot\int_{\gls{sym:t}_0}^{\gls{sym:t}}\gls{sym:expVal}\left[\Delta\gls{sym:state}\left(\gls{sym:t}_0\right)
\cdot\gls{sym:inputVec}\left(\gls{sym:eta}\right)^T\right]\cdot\gls{sym:inputMat}\left(\gls{sym:eta}\right)\cdot
\gls{sym:set}\left(\gls{sym:t},\gls{sym:eta}\right) d\gls{sym:eta}\right)^T + \notag \\
& &&
\int_{\gls{sym:t}_0}^{\gls{sym:t}} \int_{\gls{sym:t}_0}^{\gls{sym:t}} \gls{sym:set}\left(\gls{sym:t},\gls{sym:xi}\right) \cdot
\gls{sym:inputMat}\left(\gls{sym:xi}\right) \cdot
\gls{sym:expVal}\left[\gls{sym:inputVec}\left(\gls{sym:xi}\right)\cdot \gls{sym:inputVec}\left(\gls{sym:eta}\right)^T\right]
\gls{sym:inputMat}\left(\gls{sym:eta}\right)^T \cdot
\gls{sym:set}\left(\gls{sym:t},\gls{sym:eta}\right)^T d\gls{sym:xi} d\gls{sym:eta}. \notag
\end{alignat}
Analysing the individual terms in the above sum, the first one is the time update of the covariance matrix from $\gls{sym:t}_0$ to \gls{sym:t}, with
\begin{equation}
  \gls{sym:P}\left(\gls{sym:t}_0\right) = \gls{sym:expVal}\left[\Delta\gls{sym:state}\left(\gls{sym:t}_0\right)\cdot\Delta\gls{sym:state}\left(\gls{sym:t}_0\right)^T\right ].
\end{equation}

The second and third term in \eq{eq:full-cov-prop} are similar in that the one is obtained from the other by transposing. That contribution results
from the cross-correlation of the state vector at $\gls{sym:t}_0$ and the process noise and shall be defined as \gls{sym:Qxu}:
\begin{equation}
 \gls{sym:Qxu}\left(\gls{sym:t}\right) =
\gls{sym:set}\left(\gls{sym:t},\gls{sym:t}_0\right)\cdot\int_{\gls{sym:t}_0}^{\gls{sym:t}}\gls{sym:expVal}\left[\Delta\gls{sym:state}\left(\gls{sym:t}_0\right)
\cdot\gls{sym:inputVec}\left(\gls{sym:eta}\right)^T\right]\cdot\gls{sym:inputMat}\left(\gls{sym:eta}\right)\cdot
\gls{sym:set}\left(\gls{sym:t},\gls{sym:eta}\right) d\gls{sym:eta}
\end{equation}

The last term in \eq{eq:full-cov-prop} shall be denoted as $\gls{sym:Quu}\left(\gls{sym:t}\right)$, being the second moment of process noise matrix:
\begin{equation}
 \gls{sym:Quu}\left(\gls{sym:t}\right) = \int_{\gls{sym:t}_0}^{\gls{sym:t}} \int_{\gls{sym:t}_0}^{\gls{sym:t}} \gls{sym:set}\left(\gls{sym:t},\gls{sym:xi}\right) \cdot
\gls{sym:inputMat}\left(\gls{sym:xi}\right) \cdot
\gls{sym:expVal}\left[\gls{sym:inputVec}\left(\gls{sym:xi}\right)\cdot \gls{sym:inputVec}\left(\gls{sym:eta}\right)^T\right]
\gls{sym:inputMat}\left(\gls{sym:eta}\right)^T \cdot
\gls{sym:set}\left(\gls{sym:t},\gls{sym:eta}\right)^T d\gls{sym:xi} d\gls{sym:eta}.
\end{equation}

It is now possible to write \eq{eq:full-cov-prop} in a more condensed way:
\begin{equation}
 \gls{sym:P}\left(\gls{sym:t}\right) = \set{\gls{sym:t}}{\gls{sym:t}_0} \gls{sym:P}_0 \set{\gls{sym:t}}{\gls{sym:t}_0}^T + \gls{sym:Qxu}\left(\gls{sym:t}\right) +
\gls{sym:Qxu}\left(\gls{sym:t}\right)^T + \gls{sym:Quu}\left(\gls{sym:t}\right).
\end{equation}

%------------------------------------------------------------------------------
\subsection{SET matrix integration}
\label{sec:propagation-covariance-set-integration}
%------------------------------------------------------------------------------
The state vector \gls{sym:state} is defined as
\begin{equation}
 \gls{sym:state} = \left(\gls{sym:radvec}, \gls{sym:velvec}\right)^T
                 = \left(\rx, \ry, \rz, \gls{sym:v}_{\gls{idx:x}}, \gls{sym:v}_{\gls{idx:y}},
\gls{sym:v}_{\gls{idx:z}}\right)^T
\end{equation}

The partial derivative matrix \gls{sym:matrixF}, without any solve-for parameters, is defined as
\begin{equation}
 \gls{sym:matrixF}\left(\gls{sym:tg}\right) = \frac{\partial \dot{\gls{sym:state}}}{\partial \gls{sym:state}} =
 \begin{pmatrix}
    % first row
    \frac{\partial \gls{sym:v}_{\gls{idx:x}}}{\partial \rx} &
    \frac{\partial \gls{sym:v}_{\gls{idx:x}}}{\partial \ry} &
    \frac{\partial \gls{sym:v}_{\gls{idx:x}}}{\partial \rz} &
    \frac{\partial \gls{sym:v}_{\gls{idx:x}}}{\partial \gls{sym:v}_{\gls{idx:x}}} &
    \frac{\partial \gls{sym:v}_{\gls{idx:x}}}{\partial \gls{sym:v}_{\gls{idx:y}}} &
    \frac{\partial \gls{sym:v}_{\gls{idx:x}}}{\partial \gls{sym:v}_{\gls{idx:z}}} \\
    % second row
    \frac{\partial \gls{sym:v}_{\gls{idx:y}}}{\partial \rx} &
    \frac{\partial \gls{sym:v}_{\gls{idx:y}}}{\partial \ry} &
    \frac{\partial \gls{sym:v}_{\gls{idx:y}}}{\partial \rz} &
    \frac{\partial \gls{sym:v}_{\gls{idx:y}}}{\partial \gls{sym:v}_{\gls{idx:x}}} &
    \frac{\partial \gls{sym:v}_{\gls{idx:y}}}{\partial \gls{sym:v}_{\gls{idx:y}}} &
    \frac{\partial \gls{sym:v}_{\gls{idx:y}}}{\partial \gls{sym:v}_{\gls{idx:z}}} \\
    % third row
    \frac{\partial \gls{sym:v}_{\gls{idx:z}}}{\partial \rx} &
    \frac{\partial \gls{sym:v}_{\gls{idx:z}}}{\partial \ry} &
    \frac{\partial \gls{sym:v}_{\gls{idx:z}}}{\partial \rz} &
    \frac{\partial \gls{sym:v}_{\gls{idx:z}}}{\partial \gls{sym:v}_{\gls{idx:x}}} &
    \frac{\partial \gls{sym:v}_{\gls{idx:z}}}{\partial \gls{sym:v}_{\gls{idx:y}}} &
    \frac{\partial \gls{sym:v}_{\gls{idx:z}}}{\partial \gls{sym:v}_{\gls{idx:z}}} \\
    % fourth row
    \frac{\partial \gls{sym:a}_{\gls{idx:x}}}{\partial \rx} &
    \frac{\partial \gls{sym:a}_{\gls{idx:x}}}{\partial \ry} &
    \frac{\partial \gls{sym:a}_{\gls{idx:x}}}{\partial \rz} &
    \frac{\partial \gls{sym:a}_{\gls{idx:x}}}{\partial \gls{sym:v}_{\gls{idx:x}}} &
    \frac{\partial \gls{sym:a}_{\gls{idx:x}}}{\partial \gls{sym:v}_{\gls{idx:y}}} &
    \frac{\partial \gls{sym:a}_{\gls{idx:x}}}{\partial \gls{sym:v}_{\gls{idx:z}}} \\
    % fifth row
    \frac{\partial \gls{sym:a}_{\gls{idx:y}}}{\partial \rx} &
    \frac{\partial \gls{sym:a}_{\gls{idx:y}}}{\partial \ry} &
    \frac{\partial \gls{sym:a}_{\gls{idx:y}}}{\partial \rz} &
    \frac{\partial \gls{sym:a}_{\gls{idx:y}}}{\partial \gls{sym:v}_{\gls{idx:x}}} &
    \frac{\partial \gls{sym:a}_{\gls{idx:y}}}{\partial \gls{sym:v}_{\gls{idx:y}}} &
    \frac{\partial \gls{sym:a}_{\gls{idx:y}}}{\partial \gls{sym:v}_{\gls{idx:z}}} \\
    % sixth row
    \frac{\partial \gls{sym:a}_{\gls{idx:z}}}{\partial \rx} &
    \frac{\partial \gls{sym:a}_{\gls{idx:z}}}{\partial \ry} &
    \frac{\partial \gls{sym:a}_{\gls{idx:z}}}{\partial \rz} &
    \frac{\partial \gls{sym:a}_{\gls{idx:z}}}{\partial \gls{sym:v}_{\gls{idx:x}}} &
    \frac{\partial \gls{sym:a}_{\gls{idx:z}}}{\partial \gls{sym:v}_{\gls{idx:y}}} &
    \frac{\partial \gls{sym:a}_{\gls{idx:z}}}{\partial \gls{sym:v}_{\gls{idx:z}}}
  \end{pmatrix} \label{eq:partial-derivative-matrix}
\end{equation}

%------------------------------------------------------------------------------
\subsubsection{Partial derivatives for the geopotential}
\label{sec:propagation-covariance-set-integration-geopotential}
%------------------------------------------------------------------------------

The partial derivatives of the acceleration contribution due to the non-spherical Earth in the body fixed frame, are
determined according to \cite{long1989}.
As they are only a function of the radius vector, the partial derivate matrix contribution of the geopotential results in (see
\eq{eq:partial-derivative-matrix}):
\begin{equation}
 \gls{sym:matrixF}_{\gls{idx:geop}}\left(\gls{sym:tg}\right) =
 \begin{pmatrix}
    % first row
    0 & 0 & 0 & 1 & 0 & 0 \\
    % second row
    0 & 0 & 0 & 0 & 1 & 0 \\
    % third row
    0 & 0 & 0 & 0 & 0 & 1 \\
    % fourth row
    \frac{\partial \gls{sym:a}_{\gls{idx:x}}}{\partial \rx} &
    \frac{\partial \gls{sym:a}_{\gls{idx:x}}}{\partial \ry} &
    \frac{\partial \gls{sym:a}_{\gls{idx:x}}}{\partial \rz} &
    0 & 0 & 0 \\
    % fifth row
    \frac{\partial \gls{sym:a}_{\gls{idx:y}}}{\partial \rx} &
    \frac{\partial \gls{sym:a}_{\gls{idx:y}}}{\partial \ry} &
    \frac{\partial \gls{sym:a}_{\gls{idx:y}}}{\partial \rz} &
    0 & 0 & 0 \\
    % sixth row
    \frac{\partial \gls{sym:a}_{\gls{idx:z}}}{\partial \rx} &
    \frac{\partial \gls{sym:a}_{\gls{idx:z}}}{\partial \ry} &
    \frac{\partial \gls{sym:a}_{\gls{idx:z}}}{\partial \rz} &
    0 & 0 & 0
  \end{pmatrix} \label{eq:partial-derivative-matrix-geopotential}
\end{equation}
The individual derivatives are based on a differentiation of the acceleration vector components derived for the geopotential
in \eq{eq:geopotential-acceleration-1} through \eq{eq:geopotential-acceleration-2} and are computed as follows:
\begin{equation}
  \begin{aligned}
 \frac{\partial \gls{sym:avec}_{\gls{idx:geop},\gls{idx:bodyfixed}}}{\partial \rbf} &=
  \frac{\partial}{\partial\rbf} \left(\frac{\partial\gls{sym:potential}}{\partial\gls{sym:r}}\right)
  \frac{\partial\gls{sym:radvec}}{\partial\rbf} +
  \frac{\partial}{\partial\rbf} \left(\frac{\partial\gls{sym:potential}}{\partial\phigc}\right)
  \frac{\partial\phigc}{\partial\rbf} +
  \frac{\partial}{\partial\rbf} \left(\frac{\partial\gls{sym:potential}}{\partial\gls{sym:long}}\right)
  \frac{\partial\gls{sym:long}}{\partial\rbf}  \\
  &+ \frac{\partial\gls{sym:potential}}{\partial\gls{sym:r}} \frac{\partial^2\gls{sym:r}}{\partial\rbf} +
      \frac{\partial\gls{sym:potential}}{\partial\phigc} \frac{\partial^2\gls{sym:lat}}{\partial\rbf} +
      \frac{\partial\gls{sym:potential}}{\partial\gls{sym:long}} \frac{\partial^2\gls{sym:long}}{\partial\rbf}
\label{eq:geop-acc-derivative}
\end{aligned}
\end{equation}

The partial derivatives of the first three terms in \eq{eq:geop-acc-derivative}, $\partial\gls{sym:potential}/\partial\gls{sym:r}$,
$\partial\gls{sym:potential}/\partial\phigc$, $\partial\gls{sym:potential}/\partial\gls{sym:long}$, with respect to the
body-fixed radius vector, $\gls{sym:r}_{\gls{idx:bodyfixed}}$, are obtained by differentiating
\eq{eq:geopotential-acceleration-1} through \eq{eq:geopotential-acceleration-2}
using the following notation \citep{long1989}:
\begin{equation}
 \frac{\partial}{\partial\rbf}
 \begin{bmatrix}
  \frac{\partial\gls{sym:potential}}{\partial\gls{sym:r}} \\[0.5em]
  \frac{\partial\gls{sym:potential}}{\partial\phigc} \\[0.5em]
  \frac{\partial\gls{sym:potential}}{\partial\gls{sym:long}}
 \end{bmatrix}
 =
 \begin{pmatrix}
  %first row
  \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:r}^2} &
  \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:r}\partial\phigc} &
  \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:r}\partial\gls{sym:long}} \\
  % second row
  \frac{\partial^2\gls{sym:potential}}{\partial\phigc\partial\gls{sym:r}} &
  \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:lat}^2_{\gls{idx:gc}}} &
  \frac{\partial^2\gls{sym:potential}}{\partial\phigc\partial\gls{sym:long}} \\
  % third row
  \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:long}\partial\gls{sym:r}} &
  \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:long}\partial\gls{sym:r}} &
  \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:long}\partial\phigc}
 \end{pmatrix}
 \begin{bmatrix}
  \frac{\partial\gls{sym:radvec}}{\partial\gls{sym:r}_{\gls{idx:bodyfixed}}} \\[0.5em]
  \frac{\partial\phigc}{\partial\rbf} \\[0.5em]
  \frac{\partial\gls{sym:long}}{\partial\rbf}
 \end{bmatrix} \label{eq:second-derivative-potential}
\end{equation}
For example, the first element in \eq{eq:second-derivative-potential} would be read as:
\begin{equation}
 \frac{\partial}{\partial\rbf} \left(\frac{\partial\gls{sym:potential}}{\partial\gls{sym:r}}\right) =
 \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:r}^2} \frac{\partial\gls{sym:r}}{\partial\rbf} +
 \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:r}\partial\phigc}
\frac{\partial\phigc}{\partial\rbf} +
\frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:r}\partial\gls{sym:long}} \frac{\partial\gls{sym:long}}{\partial\rbf}
\end{equation}
The matrix in \eq{eq:second-derivative-potential} is symmetric, so that only six different second derivatives have to be computed, based on the first
derivatives in \eq{eq:deriv-geopotential}, as well as the derivatives of the associated Legendre functions (\eq{eq:deriv-legendre-phi} and
\eq{eq:deriv-legendre-phi-2}) with respect to the  independent variable (being the sine of the geocentric latitude):
\begin{align}
%
% d^2U/dr^2
%
 \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:r}^2} = &\frac{\gls{sym:grav}}{\gls{sym:r}^3}
             \sum\limits_{\gls{sym:geo_n}=2}^{\infty} \left(\frac{\re}{\gls{sym:r}}\right)^{\gls{sym:geo_n}}
\left(\gls{sym:geo_n}+2\right)\left(\gls{sym:geo_n}+1\right) \times \notag \\
      &\qquad \qquad \times \sum\limits_{\gls{sym:geo_m}=0}^{\gls{sym:geo_n}} \left(\cnm{}{} \cos
\left(\gls{sym:geo_m}\gls{sym:long}\right) + \snm{}{} \sin \left(\gls{sym:geo_m}\gls{sym:long}\right)\right)
\legphi{}{} \displaybreak[0] \\[1em]
%
% d^2U/drdphi
%
 \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:r}\partial\phigc} = &
 \frac{\partial^2\gls{sym:potential}}{\partial\phigc\partial\gls{sym:r}} = -\frac{\gls{sym:grav}}{\gls{sym:r}^2}
             \sum\limits_{\gls{sym:geo_n}=2}^{\infty}
\left(\frac{\re}{\gls{sym:r}}\right)^{\gls{sym:geo_n}} \left(\gls{sym:geo_n}+1\right)
\sum\limits_{\gls{sym:geo_m}=0}^{\gls{sym:geo_n}} \left(\cnm{}{} \cos
\left(\gls{sym:geo_m}\gls{sym:long}\right) + \snm{}{} \sin \left(\gls{sym:geo_m}\gls{sym:long}\right)\right) \times \notag
\\
 & \qquad \qquad \times \left(\legphi{}{+1} - \gls{sym:geo_m} \tan \phigc \legphi{}{} \right) \displaybreak[0] \\[1em]
%
% d^2U/drdlambda
%
 \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:r}\partial\gls{sym:long}} = &
 \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:long}\partial\gls{sym:r}} = -\frac{\gls{sym:grav}}{\gls{sym:r}^2}
             \sum\limits_{\gls{sym:geo_n}=2}^{\infty}
\left(\frac{\re}{\gls{sym:r}}\right)^{\gls{sym:geo_n}} \left(\gls{sym:geo_n}+1\right)
\sum\limits_{\gls{sym:geo_m}=0}^{\gls{sym:geo_n}} \gls{sym:geo_m} \left(\snm{}{} \cos
\left(\gls{sym:geo_m}\gls{sym:long}\right) - \cnm{}{} \sin \left(\gls{sym:geo_m}\gls{sym:long}\right)\right) \times \notag
\\
 & \qquad \qquad \times \legphi{}{} \displaybreak[0] \\[1em]
%
% d^2U/dphi^2
%
 \frac{\partial^2\gls{sym:potential}}{\partial\phigc^2} = &\frac{\gls{sym:grav}}{\gls{sym:r}}
 \sum\limits_{\gls{sym:geo_n}=2}^{\infty} \left(\frac{\re}{\gls{sym:r}}\right)^{\gls{sym:geo_n}} \sum\limits_{\gls{sym:geo_m}=0}^{\gls{sym:geo_n}}
 \left(\cnm{}{} \cos \left(\gls{sym:geo_m}\gls{sym:long}\right) + \snm{}{} \sin \left(\gls{sym:geo_m}\gls{sym:long}\right)\right)
\left(\tan\phigc\legphi{}{+1} \right. + \notag \\  & \qquad \qquad + \left. \left(\gls{sym:geo_m}^2\sec^2\phigc - \gls{sym:geo_m}\tan^2\phigc -
\gls{sym:geo_n}\left(\gls{sym:geo_n}+1\right)\right)\legphi{}{}\right) \displaybreak[0]  \label{eq:d2Udphi2}\\[1em]
%
% d^2U/dphidlambda
 \frac{\partial^2\gls{sym:potential}}{\partial\phigc\partial\gls{sym:long}} = &
 \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:long}\partial\phigc} = \frac{\gls{sym:grav}}{\gls{sym:r}} \sum\limits_{\gls{sym:geo_n}=2}^{\infty}
\left(\frac{\re}{\gls{sym:r}}\right)^{\gls{sym:geo_n}} \sum\limits_{\gls{sym:geo_m}=0}^{\gls{sym:geo_n}} \gls{sym:geo_m} \left(\snm{}{} \cos
\left(\gls{sym:geo_m}\gls{sym:long}\right) - \cnm{}{} \sin \left(\gls{sym:geo_m}\gls{sym:long}\right)\right) \times \notag \\
 & \qquad \qquad \times \left(\legphi{}{+1} - \gls{sym:geo_m} \tan \phigc \legphi{}{} \right) \displaybreak[0] \\[1em]
%
% d^2U/dlambda^2
%
 \frac{\partial^2\gls{sym:potential}}{\partial\gls{sym:long}^2} = &
 -\frac{\gls{sym:grav}}{\gls{sym:r}^2} \sum\limits_{\gls{sym:geo_n}=2}^{\infty} \left(\frac{\re}{\gls{sym:r}}\right)^{\gls{sym:geo_n}}
\sum\limits_{\gls{sym:geo_m}=0}^{\gls{sym:geo_n}} \gls{sym:geo_m}^2 \left(\cnm{}{} \cos \left(\gls{sym:geo_m}\gls{sym:long}\right) + \snm{}{} \sin
\left(\gls{sym:geo_m}\gls{sym:long}\right)\right) \legphi{}{}
\end{align}
Note that there is no discontinuity for $\phigc=90^\circ$ in \eq{eq:d2Udphi2}, as for $\gls{sym:geo_m}=0$ the tangent is reduced by a cosine function provided
by the \legphi{}{+1} function and for $\gls{sym:geo_m}=1$ one obtains for the combination of secant and tangent:
\begin{equation}
 \sec^2\phigc - \tan^2\phigc = 1
\end{equation}
Finally, for all $\gls{sym:geo_m}>1$, there is always a $\cos^2\phigc$ term, which reduces the secant (to unity) and the squared tangent (to $\sin^2\phigc$).

The second part of \eq{eq:geop-acc-derivative} contains second derivatives of \gls{sym:r}, \phigc and \gls{sym:long} with respect to the radius in the
body-fixed frame. These derivatives are obtained by differentiating \eqsto{eq:deriv-spherical-1}{eq:deriv-spherical-3} \citep{long1989}:
\begin{align}
 \frac{\partial^2\gls{sym:r}}{\partial\rbf^2}    =& \frac{1}{\gls{sym:r}}\left(\gls{sym:identity}-\frac{\rbf\rbf^T}{\gls{sym:r}^2}\right) \\[1em]
 \frac{\partial^2\phigc}{\partial\rbf^2}         =&
-\frac{1}{\left(\rx^2+\ry^2\right)^{3/2}}\left(\left(\frac{\partial\rz}{\partial\rbf}\right)^T-\frac{\rz\rbf}{\gls{sym:r}^2}
\right)\left(\rx\left(\frac{\partial\rx}{\partial\rbf}\right)+\ry\left(\frac{\partial\ry}{\partial\rbf}\right)\right) - \notag \\[1em]
 &
-\frac{1}{\gls{sym:r}^2\sqrt{\rx^2+\ry^2}}\left(\rbf\left(\frac{\partial\rz}{\partial\rbf}\right)+\rz\gls{sym:identity}-\frac{2\rz}{\gls{sym:r}^2}
\rbf\rbf^T\right) \\[1em]
 \frac{\partial^2\gls{sym:long}}{\partial\rbf^2} =& \frac{1}{\left(\rx^2+\ry^2\right)^2}
 \begin{bmatrix}
 -\ry \\
 \rx  \\
  0
 \end{bmatrix}
 \left(\rx\left(\frac{\partial\rx}{\partial\rbf}\right)+\ry\left(\frac{\partial\ry}{\partial\rbf}\right)\right) +
 \frac{1}{\rx^2+\ry^2}
 \begin{bmatrix}
  0 & -1 & 0 \\
  1 & 0  & 0 \\
  0 & 0  & 0
 \end{bmatrix}
\end{align}
While the above equations appear to be quite complex, they can be significantly simplified, considering the fact that the partial derivatives of the radius
components are unit vectors:
\begin{align}
 \frac{\partial\rx}{\partial\rbf} = \left[1,0,0\right] \notag \\
 \frac{\partial\ry}{\partial\rbf} = \left[0,1,0\right] \notag \\
 \frac{\partial\rz}{\partial\rbf} = \left[0,0,1\right] \notag
\end{align}

Now all quantities are available to compute the inertial acceleration $\gls{sym:avec}_{\gls{idx:bodyfixed}}$ in the body-fixed (\acrshort{acr:itrf}) frame.
In order to obtain the derivative of the acceleration in the \acrshort{acr:gcrf} with respect to the radius vector in the \acrshort{acr:gcrf},
the relationship from \eq{eq:ns-trafo} is used:
\begin{equation}
 \frac{\partial\gls{sym:avec}_{\gls{idx:ns}}}{\partial\gls{sym:radvec}} =
\kot{itrf}{gcrf}\frac{\partial\gls{sym:avec}_{\gls{idx:bodyfixed}}}{\partial\rbf}\frac{\partial\rbf}{\partial\gls{sym:radvec}}=
\kot{itrf}{gcrf}\frac{\partial\gls{sym:avec}_{\gls{idx:bodyfixed}}}{\partial\rbf}\left(\kot{itrf}{gcrf}\right)^T
\end{equation}

%------------------------------------------------------------------------------
\subsubsection{Partial derivatives for atmospheric drag contributions}
\label{sec:propagation-covariance-set-integration-drag}
%------------------------------------------------------------------------------

The computation of partial derivatives due to atmospheric drag is based on \eq{eq:drag-acceleration}, the latter being a function of the relative velocity,
which deviates from the inertial velocity due to the rotating atmosphere and horizontal wind. A similar procedure to the one shown in the following was
presented in \cite{long1989}. Provided that the relative velocity is already available in the \gls{acr:gcrf}, the derivatives of the acceleration vector with
respect to the inertial velocity vector can be determined as follows:
\begin{equation}
 \frac{\partial\gls{sym:avec}_{\gls{idx:drag}}}{\partial\gls{sym:velvec}} =
   \frac{\partial\gls{sym:avec}_{\gls{idx:drag}}}{\partial\relvel}\frac{\partial\relvel}{\partial\gls{sym:velvec}} =
   \frac{\partial\gls{sym:avec}_{\gls{idx:drag}}}{\partial\relvel} =
-\frac{\density}{2}\frac{\cd\crossSection}{\gls{sym:mass}}\left(\gls{sym:identity}\left|\relvel\right|+\frac{\relvel\relvel^T}{\left|\relvel\right|}\right).
\end{equation}
The partial derivative of the relative velocity with respect to the inertial velocity is one, which is due to the definition of the relative
velocity in the \gls{acr:gcrf}:
\begin{equation}
 \relvel = \gls{sym:velvec} + f\left(\gls{sym:radvec},t\right)
 \label{eq:relvel-fr}
\end{equation}
The additional terms, designated with $f\left(\gls{sym:radvec},t\right)$ are due to the already mentioned rotating atmosphere as well as horizontal wind and
are only functions of the radius vector and time.

Neglecting horizontal wind contributions, the relative velocity can be given as a function of the rotating atmosphere alone, being assumed to co-rotate with
the Earth, resulting in:
\begin{equation}
 \relvel = \gls{sym:velvec} - \gls{sym:rot}_{,\gls{acr:gcrf}}\times\gls{sym:radvec}
 \label{eq:relvel-rot}
\end{equation}

The angular rotation vector of the Earth, \gls{sym:rot}, is defined in the \gls{acr:tirs} as
$\gls{sym:rot}_{,\gls{acr:tirs}}=\left[0,0,\omega_{\gls{idx:earth}}\right]^T$. As the relative velocity is evaluated in the \gls{acr:gcrf}, one has to take care
of the appropriate frame transformation.

The associated partial derivatives of the acceleration vector with respect to the radius vector thus result as:
\begin{equation}
 \frac{\partial\gls{sym:avec}_{\gls{idx:drag}}}{\partial\gls{sym:radvec}} =
   \frac{\partial\gls{sym:avec}_{\gls{idx:drag}}}{\partial\relvel}\frac{\partial\relvel}{\partial\gls{sym:radvec}} =
  -\frac{\partial\gls{sym:avec}_{\gls{idx:drag}}}{\partial\relvel}\left(\kot{tirs}{gcrf} \gls{sym:rot}_{,\gls{acr:tirs}}\right)
\end{equation}
Making use of the fact, that $\gls{sym:rot}_{,\gls{acr:tirs}}$ has only a non-zero component in \gls{sym:zg}-direction, the first rotation in the
transformation from \gls{acr:tirs} to \gls{acr:gcrf} (reduction of the \acrfull{acr:era}, see \eq{eq:ERA}) is not required (identity).