%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Comparing the Maximal Updated Density (MUD) and MAP points}\label{sec:estimation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We formally define the maximal updated density (MUD) point as
\begin{equation}\label{eq:mudpt_inital_defn}
	\mudpt := \argmax {\updated}(\param).
\end{equation}
We motivate the use of the MUD point as an alternative to the MAP point for parameter estimation problems.
In this section, we assume linear (or affine) QoI maps with Gaussian distributions, which are often used in the UQ literature to provide a common framework for comparing methods and their solutions.

This section is structured into several subsections to help focus the interpretations and results.
In Section~\ref{subsec:Motivation}, we present some useful details, notation, and terminology used for this comparison framework.
To build intuition, we compare both the MUD and MAP points in Section~\ref{subsec:low-d-example} using a low-dimensional example.
A unifying perspective is provided for affine maps in Section~\ref{subsec:unifying-perspective} along with derivations of closed form expressions for the MUD and MAP points in this comparison framework.
These results are summarized in a theorem of existence and uniqueness of the MUD point in this comparison framework.
%Subsection~\ref{subsec:MUD-MAP-generalized-contours} provides some useful interpretations of the MUD and MAP points in terms of their connections to points on generalized contours and relationships to least-squares solutions.
%Finally, in Subsection~\ref{subsec:MUD-theorem}, we summarize the theory of existence and uniqueness for the MUD point.
%along with several numerical results that compare features of the MUD point to two other point estimates: the MAP point obtained from a Bayesian formulation and the solutions to a least-squares problem.

\subsection{MUD and MAP points for the ``linear Gaussian'' case}\label{subsec:Motivation}

%{\bf Regularization/deterministic optimization connections.
%The type of problem is now different involving parameter identification under uncertainty}

Let $\norm{\mathbf{x}}_C^2 := (\mathbf{x}, \mathbf{x})_C = \mathbf{x}^T C \mathbf{x} $ denote the square of the induced norm associated with a positive-definite operator $C : \RR^n \to \RR^n$ and the usual (Euclidean) inner product.
In what follows, the inverse covariances associated with non-degenerative multivariate Gaussian distributions will play the role of $C$.
%
%either attached to the Gaussian initial (prior) $\mathcal{N}(\param_0, \initialCov)$, observed $\mathcal{N}(\observedMean, \observedCov)$, or predicted $\mathcal{N}(Q(\param_0), \predictedCov)$ distributions.
%So, $C$ will play the roles of $\observedCov^{-1}$, $\initialCov^{-1}$, and $\predictedCov^{-1}$.

Suppose that the initial and prior densities are both given by the same $\mathcal{N}(\param_0, \initialCov)$ distribution.
Additionally, suppose the map $Q$ is linear and that the data-likelihood and observed densities are both given by the same $\mathcal{N}(\observedMean, \observedCov)$ distribution.

The linearity of $Q$ implies that $Q(\param)=A\param$ for some $A\in\RR^{d\times p}$, and that the predicted density follows a $\mathcal{N}(Q(\param_0), \predictedCov)$ distribution where
\begin{equation}\label{eq:predictCov}
	\predictedCov := A\initialCov A^\top.
\end{equation}
While it is not technically necessary to ensure that the predictability assumption holds (i.e, that $\updated$ is in fact a density) in order to formally define a MUD point using~\eqref{eq:mudpt_inital_defn}, it is useful when discussing certain theoretical results involving data-constructed QoI maps as we see in Sections~\ref{subsec:unifying-perspective} and~\ref{sec:data-maps}.
%The predictability assumption holds whenever
%\begin{equation}\label{eq:predict_cov_assump}
%	\predictedCov \geq \observedCov,
%\end{equation}
%where the inequality is interpreted as holding term-wise.
%We therefore refer to the condition of~\eqref{eq:predict_cov_assump} as the predictability assumption for the linear Gaussian case.
Conceptually, the predictability assumption holds when the predicted variance is larger in all directions than the observed variance.
This condition is consistent with the desired outcome of a parameter estimation problem where the incorporation of observed data produces not only an improvement to the initial point estimate but also serves to reduce the uncertainty in this point estimate (quantified in this case by the covariance).
Mathematically, this occurs when the smallest eigenvalue value of $\predictedCov$ is larger than the largest eigenvalue value of $\observedCov$.

When $\observedCov$ is non-degenerative (i.e., the smallest eigenvalue is positive), the predictability assumption can always be satisfied if $d\leq p$ and $A$ is full rank by choosing $\initialCov$ to have sufficiently large eigenvalues (i.e., if we choose initial variances to be sufficiently large).
To simplify the theoretical presentation of this section, we assume these conditions are met so that the predictability assumption holds and $\updated$ does in fact define a density.
However, in Section~\ref{sec:high-dim-linear-example}, we still compute the formal MUD point for a high-dimensional example involving rank-deficient $A$ to demonstrate the overall usefulness of the MUD point even in situations where $\updated$ may fail to be an actual density.

\begin{table}
\centering
\begin{tabular}{|c|c|}
\hline
  & \\
  Tikhonov & $T(\param):=\norm{Q(\param)-\observedMean}_{\observedCov^{-1}}^2 +
      \norm{\param-\initialMean}_{\initialCov^{-1}}^2$
  \\ & \\ \hline & \\
  Data-Consistent & $J(\param):=T(\param) - \norm{Q(\param)-Q(\initialMean)}_{\predictedCov^{-1}}^2$
  \\ & \\
  \hline
\end{tabular}
\caption{Functionals to minimize to obtain $\param$ that maximizes  the updated PDF (bottom) and the Bayesian posterior PDF (top).
Here, $T(\param)$ is the typical functional often associated with Tikhonov regularization, and the $J(\param)$ has an additional term subtracted from $T(\param)$ coming from the predicted density that serves as ``unregularization'' in data--informed directions.}
  \label{tab:func_comparisons}
\end{table}

With these assumptions, the parameters that maximize the posterior and updated densities are described as the arguments that minimize certain quadratic functionals.
Table~\ref{tab:func_comparisons} presents a scaling of these functionals defined by the negative logarithm of the associated posterior and updated densities.
Note that the functional, $T(\param)$, obtained from the posterior density is immediately identified as the typical functional used in Tikhonov regularization \citep{Tarantola_book}.

The data-mismatch term given by
\begin{equation*}
	\norm{Q(\param)-\observedMean}_{\observedCov^{-1}}^2
\end{equation*}
comes from the data-likelihood/observed density whereas the regularization term defined by
\begin{equation*}
	 \norm{\param-\initialMean}_{\initialCov^{-1}}^2
\end{equation*}
comes from the prior/initial density.
We refer to this term as the Tikhonov regularization term.

The functional, $J(\param)$, obtained from the updated density is written as a modification of the $T(\param)$ function where the subtraction of
\begin{equation*}
	 \norm{Q(\param)-Q(\initialMean)}_{\predictedCov^{-1}}^2
\end{equation*}
comes from the predicted density.

\vfill
% \pagebreak
\subsection{A Low-Dimensional Example}\label{subsec:low-d-example}
To build intuition around the fundamental differences of MAP and MUD points, we consider an example where the linear QoI map is defined by $A=\mat{cc}{1 & 1}$, i.e., the parameter space is 2-dimensional while the data space is 1-dimensional.

In this example, the parameters in the initial and observed densities are given by
\[
	\initialMean = \mat{cc}{0.25 & 0.25}^\top, \initialCov = \mat{cc}{1 & -0.25 \\ -0.25 & 0.5}, \ \observedMean=1, \text{ and } \observedCov = \mat{c}{0.25}.
\]

The top row of Fig.~\ref{fig:regularization} shows contour plots in the parameter space for the data-mismatch term (left), Tikhonov regularization term (middle), as well as the functional $T(\param)$ (right).
Conceptually, the regularization term is a radially symmetric function that penalizes parameters that are far away from the initial mean.

\begin{figure}
  \centering
  \begin{tabular}{|ccc|}
    \hline
      \subf{\includegraphics[width=0.3\linewidth]{figures/contours/data_mismatch_contour.png}}
      {Data Mismatch}
    &
      \subf{\includegraphics[width=0.3\linewidth]{figures/contours/tikonov_contour.png}}
      {Regularization}
    &
      \subf{\includegraphics[width=0.3\linewidth]{figures/contours/classical_solution.png}}
      {Bayesian Posterior}
    \\
    \hline
      \subf{\includegraphics[width=0.3\linewidth]{figures/contours/data_mismatch_contour.png}}
      {Data mismatch}
    &
      \subf{\includegraphics[width=0.3\linewidth]{figures/contours/consistent_contour.png}}
      {Modified Regularization}
    &
      \subf{\includegraphics[width=0.3\linewidth]{figures/contours/consistent_solution.png}}
       {Updated Density}
    \\
    \hline
  \end{tabular}

  \caption{Gaussian data mismatch over 2-D parameter space for a 2-to-1 linear map (left plots). Gaussian initial/prior lead to different regularization terms associated with updated/Bayesian PDFs (middle plots), which lead to different optimization functions (right plots) and parameter estimates that produce maximum PDF values for update/Bayesian PDF (red dot in right plots).}
  \label{fig:regularization}
\end{figure}

The bottom row of Fig.~\ref{fig:regularization} shows contour plots in the parameter space for the data-mismatch term (left), modified regularization term (middle), as well as the functional $J(\param)$ (right).
Here, we see that the modified regularization term only penalizes the movement of parameters in certain directions away from the initial parameter mean.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{A Unifying Perspective and Closed Form Solutions}\label{subsec:unifying-perspective}

Assume that the QoI map, $Q$, now takes the slightly more general form
\begin{equation}\label{eq:qoi_map}
Q(\param) = A \param + \mathbf{b}
\end{equation}
where $\mathbf{b} \in \RR^d$ may be viewed as a bias in the QoI map.
The inclusion of this term makes is relevant for drawing conclusions involving the data-constructed QoI maps presented in Section~\ref{sec:data-maps}.
Using the same Gaussian distribution assumptions as described in Section~\ref{subsec:Motivation}, we again identify the MAP and MUD points as the values that minimize the functionals $T(\param)$ and $J(\param)$, respectively, shown in Table~\ref{tab:func_comparisons}.

The posterior covariance is formally given by
\begin{equation}\label{eq:map_cov}
\Sigma_\text{post} := ( A^\top {\Sigma}_\text{obs}^{-1} A + \initialCov^{-1} )^{-1}.
\end{equation}
Applying the Woodbury matrix identity and~\eqref{eq:predictCov}, we rewrite the posterior covariance as
\begin{equation}\label{eq:map_cov_analytical}
\Sigma_\text{post} = \initialCov - \initialCov A^\top \left[\predictedCov + \observedCov\right]^{-1} A \initialCov,
\end{equation}
which allows us to interpret $\Sigma_\text{post}$ as a rank $d$ correction (or update) of $\initialCov$.
Note that $\predictedCov + \observedCov$ is invertible because it is the sum of two symmetric positive definite matrices.
With either version of $\Sigma_\text{post}$ given above, we rewrite the closed form expression for the MAP poing given in \cite{Tarantola_book} as
\begin{equation}\label{eq:map-point-analytical}
\param^{\text{MAP}} = \param_0 + \Sigma_\text{post} A^\top \observedCov^{-1} (\observedMean - b - A\param_0).
\end{equation}
%where the posterior covariance is given by
%\begin{equation}\label{eq:map_cov}
%\Sigma_\text{post} = ( A^\top {\Sigma}_\text{obs}^{-1} A + \initialCov^{-1} )^{-1},
%\end{equation}
%which is

We now derive an alternative representation of $J(\param)$ to draw comparisons to the posterior covariance and MAP point.
First, define
\begin{equation}\label{eq:eff_reg}
	R := \nolinebreak \initialCov^{-1} - \nolinebreak A^\top \predictedCov^{-1} A.
\end{equation}
Using this $R$, rewrite $J(\param)$ as
\begin{equation}\label{eq:dci-objective-alt}
J(\param):= \norm{\observedMean - Q(\param)}_{\observedCov^{-1}}^2 + \norm{\param - \param_0}_{R}^2.
\end{equation}
In this form, we identify $R$ as the {\em effective regularization} in $J(\param)$ due to the formulation in the data-consistent framework.

Observe that if $d=p$, then, by the assumption that $A$ is full-rank, $A$ is invertible.
In this case, $R$ is the $p\times p$ zero matrix and~\eqref{eq:dci-objective-alt} reduces to the data-discrepancy term so that the MUD point is recognizable as the least squares solution, i.e., the point that minimizes the data-discrepancy term.
Moreover, in this case we can immediately identify that $\mudpt = A^{-1}(\observedMean-b)$.
This is also evident from the perspective of the densities.
Specifically, in this case, $\updated$ is defined by applying a change of variables formula to $\observed$.
%Subsequently, the MUD point corresponds exactly to the point that maximizes $\observed$.

Suppose instead that $d<p$ so that the inverse-problem is under-determined.
In this case, we observe that constructing $R$ only requires specification of the initial/prior density and the QoI map, i.e., $R$ may be defined prior to any collection of data on the QoI.
Subsequently, we can interpret $J(\param)$ as coming from a modified Bayesian inverse problem with a prior defined by a $N(\param_0,\Sigma_R)$ distribution where $\Sigma_R=R^{-1}$.
In other words, the MUD and MAP points can both be interpreted as solutions to different Bayesian inverse problems.

However, $\Sigma_R$ is in fact a degenerative covariance, i.e., $R$ is not technically invertible.
This implies that $\Sigma_R$ cannot be directly substituted in for $\initialCov$ in~\eqref{eq:map_cov_analytical} to define a closed form expression for $\updatedCov$.
We therefore first substitute  $\Sigma_\text{post}$ and $\initialCov^{-1}$ in~\eqref{eq:map_cov_analytical} with $\updatedCov$  and $R$, respectively, to get
\begin{equation}
	\updatedCov := \left(A^\top \observedCov^{-1} A + R\right)^{-1}.
\end{equation}
Since $R$ is not invertible, Woodbury's identity cannot be applied (yet).
Using~\eqref{eq:eff_reg}, we can form
\begin{equation}
	\updatedCov = \left(A^\top \observedCov^{-1} A +  \initialCov^{-1} - \nolinebreak A^\top \predictedCov^{-1} A\right)^{-1},
\end{equation}
which is re-arranged as
\begin{equation}
	\updatedCov = \left(A^\top \left[\observedCov^{-1} - \predictedCov^{-1}\right]A + \initialCov^{-1}\right)^{-1}.
\end{equation}
Recall from Section~\ref{subsec:Motivation} that the predictability assumption in this case is that the smallest eigenvalue of $\predictedCov$ is larger than the largest eigenvalue of $\observedCov$.
The roles are reversed when we consider the inverses of these matrices.
Subsequently, $\observedCov^{-1}-\predictedCov^{-1}$ is a symmetric positive definite matrix and thus invertible.
Applying the Woodbury identity yields
\begin{equation}\label{eq:updated_cov_almost}
	\updatedCov = \initialCov - \initialCov A^\top\left( \left[\observedCov^{-1} - \predictedCov^{-1}\right]^{-1} + \predictedCov\right)^{-1} A\initialCov.
\end{equation}
Applying Hua's identity and simplifying gives
\begin{equation}\label{eq:Hua}
	\left( \left[\observedCov^{-1} - \predictedCov^{-1}\right]^{-1} + \predictedCov\right)^{-1} = \predictedCov^{-1}\left[\predictedCov - \observedCov\right]\predictedCov^{-1}.
\end{equation}
Substituting~\eqref{eq:Hua} into~\eqref{eq:updated_cov_almost} gives
\begin{equation}\label{eq:updatedCov_final}
	\updatedCov = \initialCov - \initialCov A^\top \predictedCov^{-1}\left[\predictedCov-\observedCov\right]\predictedCov^{-1}A\initialCov.
\end{equation}
We can now modify the expression for the MAP point given in~\eqref{eq:map-point-analytical} by substituting $\updatedCov$ for $\Sigma_\text{post}$ to write the MUD point that minimizes $J$ as
\begin{equation}\label{eq:mud-point-analytical-alt}
\param^{\text{MUD}} = \param_0 + \updatedCov A^\top \observedCov^{-1} (\observedMean - b - A\param_0).
\end{equation}
Substituting~\eqref{eq:updatedCov_final} into~\eqref{eq:mud-point-analytical-alt} and simplifying, we have
\begin{equation}\label{eq:mud-point-analytical-final}
	\mudpt = \param_0 + \initialCov A^\top \predictedCov^{-1}(\observedMean - b - A\param_0).
\end{equation}

Comparing~\eqref{eq:mud-point-analytical-final} to~\eqref{eq:map-point-analytical}, we
see that the MUD point does not depend on the observed covariance whereas the MAP point does.
%Both $\updatedCov$ and $\Sigma_\text{post}$ depend on $\observedCov$.
%Conceptually, this implies that the MUD point is not directly sensitivity to the amount of noise in observational data, but the noise directly impacts $\updatedCov$ in the data-informed directions.
%However, when $\observedMean$ is estimated by noisy data, this will impact the MUD and MAP points in similar ways.
Moreover, applying $Q$ to \eqref{eq:mud-point-analytical-final} and substituting accordingly reveals that $Q(\mudpt) = \observedMean$.

Overall, this motivates the MUD point as an \emph{alternative parameter estimate} with predictive accuracy and properties directly correlated to the relationship between $\observedMean$ and the true signal for which noisy data are generated.

These ideas are explored further in Section~\ref{sec:high-dim-linear-example} and utilized in the analysis of QoI maps constructed from noisy measurement data associated with a true parameter value in Section~\ref{sec:data-maps}.
We end this section by summarizing the above results in the following theorem stating the existence and uniqueness of a MUD point for the linear Gaussian case.

\begin{thm}\label{thm:MUD_existence_uniqueness}
Suppose  $Q(\param)=A\param+b$ for some full rank $A\in\RR^{d\times p}$ with $d\leq p$ and $b\in\RR^d$.
If $\initial \sim N(\param_0,\initialCov)$, $\observed\sim N(\observedMean,\observedCov)$, and the predictability assumption holds, then
\begin{enumerate}[(a)]
\item There exists a unique parameter, denoted by $\mudpt$, that maximizes $\updated$.
\item $Q(\mudpt) = \observedMean$.
\item If $d=p$, $\mudpt$ is given by $A^{-1}$. If $d<p$, $\mudpt$ is given by~\eqref{eq:mud-point-analytical-final} and the covariance associated with this point is given by~\eqref{eq:updatedCov_final}.
\end{enumerate}
\end{thm}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Higher-Dimensional Linear Gaussian Examples}\label{sec:high-dim-linear-example}

%For under-determined QoI maps, $\paramref$ is un-recoverable due to the set-valued nature of the set $Q^{-1}(\observedMean)$.
%While the incorporation of prior beliefs through the use of an initial or prior density allows us to determine a single ``best'' estimate of $\paramref$, this is not necessary.
%For instance, the least squares (LS) solution produces a single point estimate of $\paramref$ by enforcing a requirement of minimal norm.
%The LS solution will lie at the intersection of hyper-balls centered at the origin and the set-valued hyperplanes associated with $Q^{-1}(\observedMean)$, which is visible in Figure~\ref{fig:regularization} in 2-dimensions.

%By contrast, the Tikhonov-regularized solution selects a point that is biased by an initial covariance on $\Lambda$.
%The data-consistent solution is an update to the initial mean in this same direction but always lies on the contour $Q^{-1}(\observedMean)$.
%The trouble is, none of these regularization approaches actually guarantee that in under-determined problems, the unique solutions that are selected are close to $\paramref$.

%There may or may not be good reasons for us to impose a requirement on the norm of the solution.
%If there are physical reasons relating to the magnitude of parameters involved, then the least-squares solution may very well be more accurate than the other two.
%However, not every problem lends itself to such structure.
%By contrast, if the initial beliefs are well-informed by the experience of the modeling team, it is possible that the MUD or MAP point out-perform Least-Squares.
%In situations where the beliefs are already really good (the mean is near the true value), and the data is really bad, the MAP solution may (by design) out-perform the MUD solution.
%If some beliefs are correct but others are not (the covariance incorporates some incorrect directions), the MUD point may dramatically out-perform the MAP point, especially in situations with few available measurements and strongly-held initial beliefs.

%Below we demonstrate with a pair of complementary examples involving linear maps which leverage the analytical equations presented in the previous section.
%We show that regardless of how well-informed your beliefs are, the convergence rate of the MUD solutions as more data is incorporated\---either by dimension or rank)\---will match those of the Least-Squares solutions.
%Moreover, the MUD solutions are not sensitive to scaling of the initial covariance (how strongly initial beliefs are held), the way that the MAP solution is.
%This provides a strong motivating factor for the consideration of the data-consistent approach within the standard set of solution methods available to scientists and modelers who seek to perform parameter-identification.

%\subsection{MUD and MAP points and generalized contours}\label{subsec:gen_contours}

We first describe the relationship of MUD, MAP, and least squares estimates to the set-valued inverses of $Q$ in order to establish a conceptual framework for interpreting the numerical results that follow.
While this discussion is somewhat abstract, we refer to Figure~\ref{fig:regularization} to make these ideas more clear.

The MUD point exists on the generalized contour defined by $Q^{-1}(\observedMean)$.
We say generalized contour in this case because if $A$ is a $p$-to-$d$ full rank linear map with $d<p$, then $Q^{-1}(\observedMean)$ exists as a $(p-d)$-dimensional linear hyperplane in $\pspace$.

This means that the MUD point retains the ``predictive precision'' of a least squares solution to the inverse problem (i.e., a parameter that minimizes the data-mismatch term $\norm{\observedMean - Q(\param)}_{\observedCov^{-1}}^2$) while incorporating the flexibility of prior beliefs in directions not informed by the QoI.
This is illustrated in the bottom right plot of Figure~\ref{fig:regularization}.
For under-determined or ill-conditioned problems, this suggests that ``good'' prior beliefs may be used to produce a MUD point that is more accurate than a least squares solution.
This is explored in the following examples involving high-dimensional linear maps.

By contrast, the MAP point exists on a line connecting the initial mean, $\param_0$, and the generalized contour defined by $Q^{-1}(\observedMean)$.
Substituting \eqref{eq:map_cov_analytical} into \eqref{eq:map-point-analytical}, we see that this line is in the direction of the orthogonal nullspace of the image of $A$ under $\initialCov$; i.e., $\mathcal{N}(\initialCov A)^\perp$.
In fact, this line intersects the generalized contour defined by $Q^{-1}(\observedMean)$ precisely at the MUD point.
If one parameterizes the line between $\param_0$ and $\param^{\text{MUD}}$, then one can also identify $\param^{\text{MAP}}$ as a convex sum of these two points.
The weights of this convex sum, which determine the position of the MAP point on this line, are determined by the ``precision of data'' (i.e., on $\observedCov$) and the ``strength of prior beliefs'' (i.e., on $\Sigma_{\text{init}}$).
This is seen by comparing the location of the MAP point in the top right plot of Figure~\ref{fig:regularization} to the line segment connecting the initial mean to the MUD point in the bottom right plot of this same figure.
The impact of this is also explored in the following examples.

\begin{figure}[htbp]
  \includegraphics[width=0.475\linewidth]{figures/lin/lin-dim-cov-convergence}
  \includegraphics[width=0.475\linewidth]{figures/lin/lin-rank-cov-convergence}
\caption{
	Relative errors between $\paramref$ and (i) the least squares solution obtained through {\tt numpy}'s {\tt linalg.pinv} module, (ii) the closed-form solution for the MUD point given in Eq~\eqref{eq:mud-point-analytical-final}, and (iii) the MAP point.
  (Left): Error for increasing dimensions of $D$ for $A$ taken to be a Gaussian Random Map.
  (Right): Error for increasing row-rank of $A$, generated with Gaussian vectors and a SVD.
}
\label{fig:lin-error}
\end{figure}

\subsection{Impact of Output Dimension}%: Random $\RR^{k\times 100}$ matrix for $k=1,2,\ldots,100$}

%Here, we demonstrate how the various estimates of a true parameter $\paramref$ are impacted by the number of available QoI.%, especially in problems where the dimension of $\pspace$ is high.
%As more QoI are incorporated, the expectation is that the accuracy of $\paramref$ improves as well.
%In the following example, we demonstrate that the MUD point does indeed perform this way, and compare its ability to approximate $\paramref$ against two other popular solution methods: the least-squares solution and Tikhonov-regularized solution, i.e. the MAP point.

We consider QoI defined by $A\param+\mathbf{b}$ for $A\in\RR^{k\times 100}$ where $k=1,2,\ldots,100$ to demonstrate how the various estimates of a true parameter $\paramref$ are impacted by the number of available QoI.
To generate the matrices, we first generate $10,000$ independent identically distributed (i.i.d.) random numbers from a $N(0,1)$ distribution that are arranged into a reference $\RR^{100\times 100}$ matrix.

The same distribution is also used for generating the components of the 100-dimensional vectors defining a reference bias vector $\mathbf{b}$ and reference parameter $\paramref$.
A multivariate Gaussian distribution is used for the initial density, with zero mean and $\initialCov$ chosen as a diagonal covariance with random entries drawn from $U[0.5, 1.5]$ and sorted in descending order. %\footnote{A random seed of $271$ is used so that the covariance in this example matches that in the following, where we perform a similar study but vary the row-rank of $A$ instead of the dimension.}
%We then attempt to recover $\paramref$ with the three solution methods: MUD, MAP, and Least Squares, leveraging analytical solutions for all three since we have posed a linear problem under Gaussian assumptions.
The prior density is also a zero mean multivariate Gaussian distribution.
However, to demonstrate the impact of the strength of prior beliefs on the MAP point, we choose the prior covariance to be $\alpha\initialCov$  for $\alpha=0.001, 0.01, 0.1, $ and $10$.
Here, smaller values of $\alpha$ correspond to a ``stronger'' belief in the prior since the prior density becomes more concentrated near the prior mean.

To study the impact of dimension on the MUD, MAP, and least squares estimates, we solve a
sequence of inverse problems by truncating the rows of the reference matrix and bias vector.
%
%We scale the initial covariance $\initialCov$ by a factor $\alpha$ for $\alpha = 0.01, 0.1, 1, 10 $ to simulate varying levels of attachment to the specification of the initial density.
The results are summarized in the left plot of Figure~\ref{fig:lin-error}, which shows convergence towards $\paramref$ for all the problems considered with the exception of several MAP estimates corresponding to strongly-held beliefs in the prior.

%We demonstrate that the MUD solution retains the accuracy of least squares solutions while simultaneously offering the flexibility of specifying initial beliefs.
%Normally in order to incorporate such beliefs, one would perform Tikhonov regularization, usually with the inclusion of a hyper-parameter which scales the additional parameter-space norm in the objective function.
%Mathematically, this scaling factor applied to the norm is equivalent to scaling the matrix representation of the initial (prior) covariance.
%Increasing this scaling factor is interpreted as having less confidence in these initial assumptions.
%Conversely, decreasing it is equivalent to putting more emphasis on the prior beliefs than the evidence provided by the data, which causes MAP solutions to drift away from the solution contour (equivalence class) to which $\paramref$ belongs.
%The MUD point, by contrast, is not impacted by such a scaling of the initial covariance, providing \emph{consistent} solutions which demonstrate levels of accuracy that MAP points only exhibit for larger values of scaling factors.

%To illustrate this with a concrete example, we consider the problem $A\param + b = y$ by first generating a square matrix $A$ of dimension $100$ with standard Gaussian entries.
%The same distribution is used for the components of vector $b$ and $\paramref$ as well.
%A Gaussian distribution is used for the initial density, with mean at the origin (to complement the structure imposed by the least-squares framework), and a diagonal covariance with entries drawn from $U[0.5, 1.5]$ and sorted in descending order.\footnote{A random seed of $271$ is used so that the covariance in this example matches that in the following, where we perform a similar study but vary the row-rank of $A$ instead of the dimension.}
%We then attempt to recover $\paramref$ with the three solution methods: MUD, MAP, and Least Squares, leveraging analytical solutions for all three since we have posed a linear problem under Gaussian assumptions.
%To study the impact of dimension, we solve inverse problems for a sequence of maps generated by truncating the rows of $A$.
%Finally, several scalings of the initial covariance are chosen to simulate a common procedure in the Bayesian literature: hyper-parameter optimization.
%We scale the initial covariance $\initialCov$ by a factor $\alpha$ for $\alpha = 0.01, 0.1, 1, 10 $ to simulate varying levels of attachment to the specification of the initial density.
%The results are summarized in Figure~\ref{fig:lin-error}, which shows convergence towards $\paramref$ for all the problems considered with the exception of several Tikhonov solutions corresponding to strongly-held beliefs in initial assumptions about parameters.

We note that the MUD solution is the same for all choices of $\alpha$ and corresponds to the same level of accuracy that the MAP point achieves when $\alpha$ is chosen to be large.
In other words, the MUD point is not impacted by a scaling of the initial covariance, providing \emph{consistent} solutions which demonstrate levels of accuracy that MAP points only exhibit for larger values of scaling factors.
%The accuracy of the Tikhonov solution is highly dependent on the specification of $\initialCov$ and in some cases leads to nearly-divergent scenarios.

Of interest is also that the MUD point can sometimes out-perform the least squares estimate while generally achieving similar levels of accuracy.
This suggests that the MUD point has several favorable qualities.
Not only is it robust to the specification of prior assumptions, but it manages to offer the flexibility of incorporating good prior specifications without paying the additional cost of hyper-parameter optimization (i.e., choosing an appropriate $\alpha$) that would be required for the MAP estimates to achieve comparable results.

While omitted in the interest of space, if $\initialCov$ is chosen as $\alpha I$, where $I$ denotes the identity matrix of appropriate dimensions, then the MUD point will always agree with the least squares estimate.
Taking these results together, this implies that only a good ``relative spatial structure'' of prior beliefs is required to improve the MUD point's accuracy over both MAP and least squares estimates.

\subsection{Impact of Rank: One Hundred (Deficient) $\RR^{100 \times 100}$ matrices}

Here, we investigate whether the previous dimension-dependent example extend to matrices $A$ which are of a fixed dimension but varying rank.
This is of interest in applications where many QoI are available to construct an operator but a great deal of redundancy may be present in the data collected, and feature-engineering new quantities is somehow prohibitive (perhaps due to gradient estimation).

%The theory implies that a full-rank operator $A$ whose input and output dimensions are equal would ``turn off'' the data-consistent regularization, since the predicted covariance $\predictedCov = A^T \initialCov A$ is invertible.

The rank of $A$ corresponds to the number of unique directions of information present in the operator, i.e., how many directions in the parameter space are informed by the QoI map.
%, in other words, the mutual-distinctness of the information in the Quantities of Interest.
The operators in the previous example were all full rank, so the dimension of each map also corresponded to the rank of $A$.
When $A$ is rank-deficient, $\predictedCov$ is non-invertible, so we must modify the form of \eqref{eq:mud-point-analytical-final} to substitute a pseudo-inverse for the predicted covariance.
% we are curious to see if the same behaviors are observed on this related problem which violates one of the stated assumptions in the theory presented in section [TK - section with theory in it].

In this example, the dimension of the data space remains fixed at $d=100$ across all experiments.
However, we sequentially increase the row-rank of $A$ from $r=1, \ldots, 100$.
To control the rank of $A$, we first construct a reference $\RR^{100\times 100}$ matrix as in the previous example using 10,000 i.i.d. $N(0,1)$ random numbers.
We then compute a singular value decomposition of this reference matrix of the form $USV^\top$ and construct 100 rank-1 matrices of the form $A_i=\mathbf{u}_i s_i \mathbf{v}_i^\top$ for $i=1,\ldots,100$ where $\mathbf{u}_i$ and $\mathbf{v}_i$ denote the $i$th columns of $U$ and $V$, respectively and $s_i$ denotes the $i$th singular value.
Then, we analyze the impact of $A = \sum_i^r A_i$ for $r=1,\ldots,100$.
%\footnote{We validated by alternatively generating rank-1 matrices from random Gaussian vectors $a_i \in \RR^100$ to form $A_i = a_i a_i^T$, and included assertion checks in code to validate correct rank with our generating functions.}.
Aside from the differing construction of $A$, the rest of the choices involved in the experiment ($\paramref$, the reference bias vector, and the distributions involved) is identical to the previous example.

In the right plot of Figure~\ref{fig:lin-error}, we
%super-impose the pseudo-inverse solution in light blue and see that it matches the black line associated with the MUD error much of the time.
again find that the MUD point is generally as accurate as the least squares estimate, but incorporates an initial description of uncertainty, which may allow it to outperform the least squares estimate.
Also, we again see that the MAP estimates are impacted by the strength of prior beliefs.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Data-constructed QoI maps and MUD points}\label{sec:data-maps}

Suppose there exists $d$ measurement devices for which repeated noisy data are obtained.
For each $1\leq j\leq d$, denote by $\mathcal{M}_j(\paramref)$ the $j$th measurement device, and denote by $N_j$ the number of noisy data obtained for $\mathcal{M}_j(\paramref)$.
Let $d_{j,i}$ denote the $i$th noisy datum obtained for the $j$th measurement where $1\leq i\leq N_j$.
To simplify the presentation and some of the resulting notation, we assume an unbiased additive error model for the measurement noise with independent identically distributed (i.i.d.) Gaussian errors so that
\begin{equation}\label{eq:obs_data_error}
	d_{j,i} = M_j(\param^\star) + \xi_i, \ \xi_i\sim N(0,\sigma_j^2), \ \ 1\leq i\leq N_j.
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{The Weighted Mean Error Map}

We now construct a $d$-dimensional vector-valued map from data obtained on the $d$ measurement devices.
The data-defined QoI map we consider in this work is referred to as the weighted mean error (WME) map, denoted by $Q_\text{WME}(\param)$ with $j$th component, denoted by $Q_{\text{WME},j}(\param)$, given by
\begin{equation}\label{eq:qoi_WME}
	Q_{\text{WME},j}(\param) := \frac{1}{\sqrt{N_j}} \sum_{i=1}^{N_j} \frac{M_j(\param)-d_{j,i}}{\sigma_j}.
\end{equation}
%%\begin{equation}\label{eq:qoi_WME}
%%	Q_{(\param,\xi) := \frac{1}{\sqrt{S}} \sum_{j=1}^S \frac{M_j(\param)-d_j(\xi_j)}{\sigma_j}.
%%\end{equation}
By a substitution of \eqref{eq:obs_data_error} into \eqref{eq:qoi_WME} and rationalizing the denominator of the multiplicative factor, $Q_{\text{WME},j}(\paramref)$ is identified as the sample average of $N_j$ random draws from an i.i.d.~$N(0,N_j)$ distribution.
By assumption, the observed data are generated according to the fixed true physical parameter vector given by $\paramref$ in \eqref{eq:obs_data_error}.
Subsequently, each component of $Q_\text{WME}(\paramref)$ is a random draw from an $N(0,1)$ distribution.
Therefore, with this choice of data-defined QoI map, we specify $\observed$ as a $N(\mathbf{0}_{d\times 1},\mathbf{I}_{d\times d})$ distribution.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Analysis of MUD point for the WME map}\label{sec:MUD_analysis}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Below, we consider issues of existence, uniqueness, and convergence of $\mudpt$ for the WME map under certain assumptions.
In~\ref{sec:conclusions}, we provide some remarks on how we plan to generalize this analysis in future work.

For each $1\leq j\leq d$, assume that the observable maps $M_j$ are linear maps of $\param$ (affine maps require minor modifications to the results below).
For notational simplicity below, assume that $M_j$ is written explicitly as a $1\times p$ row vector and that the $d$-vectors form a linearly independent set.
Then, it is possible to rewrite $Q_{\text{WME}}(\param)$ as
\begin{equation}
	Q_\text{WME}(\param) = A(\mathbf{N})\param + b(\mathbf{N}),
\end{equation}
where the $j$th component of $\mathbf{N}\in\RR^d$ is given by $N_j$ and the $j$th row of $A(\mathbf{N})\in\RR^{d\times p}$ is given by
\begin{equation}
	\frac{1}{\sqrt{N_j}} \sum_{i=1}^{N_j} \frac{M_j}{\sigma_j} = \frac{\sqrt{N_j}}{\sigma_j}M_j,
\end{equation}
and the bias vector, $\mathbf{b}(\mathbf{N})\in\RR^d$, is defined by the data, with $j$th component, denoted by $\mathbf{b}_j$, given by
\begin{equation}
	\mathbf{b}_j(\mathbf{N}) = -\frac{1}{\sqrt{N_j}} \sum_{i=1}^{N_j} \frac{d_{j,i}}{\sigma_j}.
\end{equation}
Since $A(\mathbf{N})\initialCov A(\mathbf{N})^\top$ defines a predicted covariance, and the observed covariance is the identity map, the predictability assumption is immediately satisfied if each diagonal component of the predicted covariance is significantly greater than $1$.

The off-diagonal components of the predicted covariance (which are dictated in large part by the structure of the measurement operators $M_j$) dictate how much larger than $1$ each diagonal component must be to ensure the predictability assumption holds.
However, we demonstrate below that this will happen once a minimum number of data points $N_\text{min}$ are obtained for each measurement.

First, observe that the $j$th diagonal component of the predicted covariance matrix is given by the predicted variance associated with using the scalar-valued map $Q_{\text{WME},j}$.
Then, the associated predicted variance is given by
\begin{equation}
	\frac{N_j}{\sigma_j^2} M_j\initialCov M_j^\top
\end{equation}
Since $\initialCov$ is assumed to be non-degenerative and $M_j$ is a non-trivial row vector, this predicted variance grows linearly with $N_j$.
In other words, the $j$th diagonal component of the predicted covariance has the form $\beta_j N_j$ for some $\beta_j>0$.
Let $N_{\text{min},j}$ denote the minimum $N_j$ for $1\leq j\leq N$ necessary to make the $j$th diagonal components sufficiently large so that the smallest eigenvalue of the predicted covariance is larger than $1$ (which is the repeated eigenvalue for the observed covariance).

The following result is now an immediate consequence of Theorem~\ref{thm:MUD_existence_uniqueness},

\begin{corollary}\label{cor:MUD_wme}
If $\initial \sim N(\param_0,\initialCov)$ and data are obtained for $d$ linearly independent measurements on $\pspace$ with an additive noise model with i.i.d. Gaussian noise for each measurement, then there exists a minimum number of data points obtained for each of the measurements such that there exists a unique $\mudpt$ and $Q_\text{WME}(\mudpt) = 0$.
\end{corollary}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{MUD points for nonlinear maps: Examples}\label{sec:Parameter-identification}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We now use MUD points as parameter estimates for nonlinear data-constructed maps using simulated noisy temporal and spatial data associated with solutions to differential equations.
The previously derived closed form expressions for $\mudpt$ do not apply in these examples.
Instead, in each example we use a fixed set of i.i.d.~samples drawn from the initial density to approximate the updated density and subsequently choose the sample that maximizes this approximation.
Future work will consider how to leverage more sophisticated optimization approaches for estimating $\mudpt$, e.g., through iterative quasi-Newton approaches based on repeated approximate linearizations that can exploit the closed form expressions for $\mudpt$.
%that are iteratively improved upon, e.g., by combining this work with a quasi-Newton type approach.

The two examples below have distinct points of emphasis.
The first example focuses on the improved accuracy and precision of scalar $\mudpt$ estimates as more data are incorporated into the construction of $Q_\text{WME}$.
The second example focuses on how the utilization of subsets of the data into different components of a vector-valued $Q_\text{WME}$ can improve the estimates across multiple components of a $\mudpt$ estimate for a parameter vector.
Despite these differences, both examples share the following details.
\begin{itemize}
  \item Sets of i.i.d.~samples from the initial densities are used to both approximate the predicted density (and thus the updated density) and to form a discrete space in which to search for the MUD point.
  \item A reference parameter is chosen and used to establish a (noiseless) reference solution (in space or time).
  \item An i.i.d. additive noise model with $\xi \sim N(0,\sigma^2)$ is used to perturb the reference temporal and spatial solutions.
  \item Since the magnitude of solutions in both examples is $\mathcal{O}(1)$, noise variances are chosen so that $\mathbb{P}( \abs{\xi} < \tau ) = 99\%$, with $\tau=0.1$. In other words, we simulate an ``engineering accuracy'' by permitting measurement noise to vary by  approximately 10\% of the signal magnitude.
%  \item Vertical axes on convergence plots represent the absolute error between $\paramref$ and $\mudpt$, with bounds chosen to be fixed across plots within a figure.
%  \item Errors are reported as the average over 20 sets of independently generated noisy data.
%  \item To estimate convergence rates, we compute first-order linear regressions for the logarithm (base 10) of the mean and variances for the absolute error. The coefficient for the mean in the regression appears as an annotation in the figures.
\end{itemize}

\FloatBarrier

\subsection{ODE Example}\label{subsec:ode-example}
Consider the exponential decay problem with uncertain decay rate $\param$:
$$
\begin{cases}
\frac{\partial u}{\partial t} & = \param u(t), \ 0<t\leq 3, \\ u(0) &= 0.75,
\end{cases}
$$
with solution
\begin{equation}
u(t;\param) = u_0\exp(-\param t), \; u_0 = 0.75 ,
\end{equation}
and measurements begin at $t=1$.

The initial uncertainty in the decay rate is given by a uniform density on $\param \in \Lambda = (0, 1)$, and $10,000$ samples from this density are used to estimate the push-forward and updated densities as well as the MUD points.
Noisy data are simulated over the interval $t \in [1,3]$ assuming a $100$Hz measurement sensor.
An example of this setup is shown in Figure~\ref{fig:ode-reference}, with the solid line representing the ``true'' signal obtained by evaluating the solution for $\paramref = 0.5$, and a particular realization of noisy data shown as perturbed points around it.

\begin{figure}[htb]
  \includegraphics[width=\linewidth]{figures/ode/ode_20_reference_solution.png}
  \includegraphics[width=\linewidth]{figures/ode/ode_200_reference_solution.png}
  \caption{Curves for exponential decay model with various decay coefficients. Dashed curves denote 99\% probability intervals for noise. The true signal is shown in solid black.
 Curves associated with the MUD estimates for the decay coefficient computed from 20 trials are shown in light red.
 Both sets of these curves encompass the true signal for $N=20$ (top plot) and $N=200$ (bottom plot) data points.
 The light red curves are almost indistinguishable in the bottom plot as they all lie nearly on the true signal which demonstrates the overall reduction in variance in MUD estimates around the true signal when using $N=200$ data points.
  }
  \label{fig:ode-reference}
\end{figure}

For 20 different realizations of $N=5, 10, 15, 20, 25, 50, 100, \text{ and } 200$ noisy measurement data, we construct $Q_\text{WME}$ with $u(t;\param)$ replacing $M_j(\param)$ in~\eqref{eq:qoi_WME}, estimate $\mudpt$, and analyze the statistical error in these estimates of $\paramref$.
In the left plot of Figure~\ref{fig:ode-convergence}, we observe that as more data are used that the accuracy (defined by the mean error between $\mudpt$ and $\paramref$) decreases.

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.45\linewidth]{figures/ode/ode_convergence_mud_obs_mean.png}
  \includegraphics[width=0.45\linewidth]{figures/ode/ode_convergence_mud_obs_var.png}
  \caption{The mean (left) and variance (right) of absolute errors in MUD estimates as a function of the number of data points used. These statistics are computed over 20 trials.
  }
  \label{fig:ode-convergence}
\end{figure}

The right plot demonstrates that the precision (i.e., the variance in the error) is also reduced by using more data.
These results are also visualized in the plots of Figure~\ref{fig:ode-reference} where the distinct red curves in the top plot are the signals associated with the 20 $\mudpt$ estimates associated with the 20 independent trials of $N=20$ noisy measurements.
However, in the bottom plot, these $20$ red curves are fairly indistinguishable to the naked eye from the true signal when using $N=200$ noisy measurements.
In Appendix~\ref{ext:ode-example}, we repeat this problem with measurement equipment that operates at double the temporal frequency and show how this change impacts the MUD solutions.

% \FloatBarrier
\vfill
\subsection{PDE Example}\label{subsec:pde-example}

In this problem, the uncertain model parameter is now described by an unknown function defining the boundary data to a stationary PDE.
The focus is on demonstrating that a vector-valued QoI can be constructed from noisy data to produce a MUD point that more accurately reconstructs features of the unknown function.
As described below, there is an intuitive way to separate the spatial data to construct the distinct components of the vector-valued QoI map.
While beyond the scope of this work, a future work will consider how to incorporate clustering analysis on the data for less obvious cases where we seek to determine which data points should be utilized to construct the various components of a vector-valued QoI map.
In the interest of clarity, we present representative results for a fixed finite-dimensional representation of the parameter space and forgo the more complicated convergence analysis that requires increasing both the number of data available as well as the dimension of the parameter space.

Consider the Poisson problem:
\begin{equation}\label{eq:pde-equation}
\begin{cases}
\hfill -\nabla \cdot \nabla u &= f(x), \quad\text{on } x\in \Omega, \\
\hfill u &= 0, \quad\text{ on } \Gamma_T \cup \Gamma_B, \\
\hfill \frac{\partial u}{\partial \mathbf{n}} &= g(x_2), \quad\text{ on } \Gamma_L, \\
\hfill \frac{\partial u}{\partial \mathbf{n}} &= 0, \quad\text{ on } \Gamma_R,
\end{cases}
\end{equation}
where $x=(x_1, x_2) \in \Omega = (0,1)^2$ is the spatial domain; $\Gamma_T$, $\Gamma_B$, $\Gamma_L$, and $\Gamma_R$, denote the top, bottom, left, and right boundaries of this domain, respectively, and $\frac{\partial u}{\partial \mathbf{n}}$ denotes the usual outward normal derivative.
The forcing function $f$ is taken to be $10\exp\left ( \norm{x - 0.5}^2 / 0.02 \right )$.

Here, we assume that $g(x_2)$ is unknown, and the goal is to use noisy data to estimate this unknown boundary data.
In other words, the parameter $\param$ now represents an uncertain function.
To generate the noisy data, we use a reference $g(x_2)\propto x_2^2(x_2-1)^5$ with a constant of proportionality chosen to produce a minimum of $-3$ at $x_2=\frac{2}{7}$.
Then, we compute a reference solution using piecewise-linear finite elements on a triangulation of a $36\times36$ mesh.
Random noise is then added to every degree of freedom of this reference solution, and the spatial data are subsequently computed from a fixed set of 100 randomly placed sensors in the subdomain $(0.05, 0.95)^2 \subset \Omega$.
This process is repeated $20$ times to study the subsequent variation in MUD points due to different realizations of noisy data.
See the plot of Figure~\ref{fig:pde-Q} for a representative noisy response surface and location of spatial data used across all 20 trials.

\begin{figure}[htbp]
\centering
\includegraphics[width=0.65\linewidth]{figures/pde-highd/pde-highd_sensors_D2.png}
\caption{
A representative noisy perturbation of the reference response surface. Locations of the randomly chosen spatial data used to construct both $Q_{1D}$ and $Q_{2D}$ are shown as black and red dots.
}
\label{fig:pde-Q}
\end{figure}

%We simulate measurement noise as described in the preamble to Sec.~\ref{sec:Parameter-identification} and use this data to construct a QoI map with which we solve a parameter identification problem which characterizes $g$.
To construct a finite-dimensional parameter space describing the initial uncertainty of $g(x_2)$, we first assume that it is known that $g$ is non-positive and bounded below by $-4$.
We further assume that $g$ is smooth enough to be reasonably approximated by a piecewise-linear continuous spline with four knots and that $g(0)=g(1)=0$.
Thus, the uncertainty is described by the values of the splines at the two interior knot points chosen as the equispaced points $1/3$ and $2/3$.
This defines a finite-dimensional parameter space described by $\pspace = [0,-4]^2$.
We generate $1000$ samples from an initial uniform density on $\pspace$ to (1) generate random spline functions and compute the (noise-free) data from solutions associated with these splines; and (2) estimate the push-forward and updated densities along with the MUD estimate of $g(x_2)$.
The randomly generated splines are shown in the left plot of Figure~\ref{fig:pde-MUD} along with the reference $g$ and its interpolant on these spline knots.
These same $1000$ samples are used across all $20$ realizations of random noisy data.

\begin{figure}[htbp]
\centering
    \includegraphics[width=0.325\linewidth]{figures/pde-highd/pde-highd_init_D2.png}
    \includegraphics[width=0.325\linewidth]{figures/pde-highd/pde-highd_pair_D2-1_m100.png}
    \includegraphics[width=0.325\linewidth]{figures/pde-highd/pde-highd_pair_D2-2_m100.png}
\caption{
In the left plot, the reference $g(x_2)$ is shown as the solid black curve with its interpolant onto the spline basis shown as a dotted red curve.
The dashed blue line represents the sample from parameter space which most closely predicts noiseless data, which we refer to as the projection of $g$.
The purple curves in the center and right plots show the variability in MUD estimates of $g(x_2)$ for the 20 different realizations of noisy data.
The center plot uses $Q_{1D}$ and the right plot uses $Q_{2D}$ to construct the MUD estimates.
}
\label{fig:pde-MUD}
\end{figure}

Two separate types of data-constructed $Q_\text{WME}$ maps are constructed from the data.
The first map, which for simplicity we denote by $Q_{1D}$, uses all 100 spatial data points to construct a scalar-valued QoI map.
Given the location of the spline knots, a second map, which for simplicity we denote by $Q_{2D}$, uses the data points above (below) the mid-line of the spatial domain to form the first (second) component of the 2-dimensional vector-valued QoI map.
This separation of spatial data is illustrated in the top-left plot in Figure~\ref{fig:pde-Q}.
For reference, the interpolant of the reference $g(x_2)$ on the spline basis is also shown in these plots.
The sample from $\pspace$ which best predicts the hundred noiseless measurements (in 2-norm) is also shown, referred to as the projection of $g(x_2)$.

\begin{figure}[htbp]
\centering
    \includegraphics[width=0.45\linewidth]{figures/pde-highd/pde-highd_convergence.png}
\caption{
A plot of the variance in 2-norm of MUD solutions acquired from solving the SIP with $Q_{1D}$ and $Q_{2D}$ as more measurements are incorporated.
The solid black line represents $Q_{1D}$ and the dashed blue line represents $Q_{2D}$.
}
\label{fig:pde-convergence}
\end{figure}

The center and right plots in Figure~\ref{fig:pde-MUD} shows the variability in the $20$ MUD estimates of $g(x_2)$ obtained from the different realizations of noisy data using $Q_{1D}$ and $Q_{2D}$, respectively.
A comparison of these plots suggests the MUD estimates associated with $Q_{2D}$ have less spatial variability than those associated with $Q_{1D}$, while also being better point-wise estimates for the interpolant or projection of $g(x_2)$.
To quantify this reduction in variability, the convergence with respect to the number of measurements incorporated is illustrated in Figure~\ref{fig:pde-convergence}.
There we see that using $Q_{2D}$ for inversion results in a slow but noticeable reduction in variance while $Q_{1D}$ exhibits a slight upward trend.
The MUD solutions for $Q_{1D}$ appear unable to resolve which of the two parameters represents a minimum value of $g$, while $Q_{2D}$ MUD solutions concentrate around the $\lambda_i$ which best predicts the measurements.
The latter solution captures more features of the solution we seek (an estimate of $g$).


\section{Numerical Results}
\label{sec:numerical}

In this section, we demonstrate the analytical
claims above with numerical examples and
highlight the performance of the BIE approach
with demonstrations on domains of analytical
and practical interest. The software used to
generate the figures is available online
\footnote{\texttt{https://doi.org/10.5281/zenodo.2641296}}.

\subsection{Numerical methods}

First, we describe the numerical tools needed
to compute Stokes eigenvalues in a BIE
framework.

\subsubsection{Discretizing the BIE}

In order to turn the BIEs analyzed
above into discrete linear systems,
we require some standard techniques
from the BIE literature.

Let the boundary be divided into $N_p$
panels.
%
We parameterize panel $j$ as
$\bx_j(t)$, with $t$ ranging over the
interval $[-1,1]$.
%
Each component of $\bx_j$ is taken to be
a polynomial interpolant over the
standard 16th-order Legendre nodes on
$[-1,1]$, denoted by $t_n$, so that
the total number of discretization
points is $N=16N_p$.
%
See \cref{fig:panels} for an example
discretization.

\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{fig/annulus_discretization.pdf}
\caption{Sample discretization of an annulus, where the inner
circle, $r=1$ is discretized using $6$ panels and the outer circle $r=1.7$
is discretized using $10$ panels.}
\label{fig:panels}
\end{figure}
%
Another important quantity below is the
arc-length density of a panel, which
we denote by $s_j(t) := |\bx'_j(t)|$.
%
Finally, we denote the set of panels
which are adjacent to panel $j$
by $A(j)$. On a closed curve,
$A(j)$ contains two integers.

The integral kernels of the single and
double layer potentials have weak
singularities of the form $|\xx-\yy|^p\log |\xx-\yy|$
for some $p \in \N_0$, which require special
quadrature rules to achieve high-order accuracy.
%
In the examples below, we use generalized
Gaussian quadrature (GGQ)~\cite{bremer2010}.
%
To demonstrate the idea, we consider
evaluating the convolution of a kernel
$K$ with a density $\sigma$
at the boundary node $\xx_j(t_l)$.
%
GGQ is a Nystr\"{o}m-type discretization ---
the density is approximated
by its values at the discretization nodes,
which we denote by
$\sigma_{qp} := \sigma(\xx_q(t_p))$.
%
The basis of a GGQ rule is a set of
support nodes and weights for the
contribution to the integral from the
``self'' panel (panel $j$) and the adjacent
panels (with index in $A(j)$).
%

For the self panel, there is a special set
of nodes and weights for each interpolation
point. Denote the nodes and weights
for interpolation point $l$ by $t^{(l)}_{n}$
and $w^{(l)}_{n}$, respectively, with
$1\leq l \leq 16$ and $1\leq n \leq N_s$.
%
The adjacent panels are handled by a single
set of over-sampled support nodes and weights.
We denote these nodes and weights
by $\tilde{t}_n$ and $\tilde{w}_n$, respectively,
for $1 \leq n \leq N_a$.
%
For the 20th-order rule we used, $N_s = 16$ and
$N_a = 48$.
%
The contribution of other panels is assumed
to be given to high accuracy by the standard
Gauss-Legendre weights, which we denote
by $w_n$.
%
Adding these contributions together, we obtain the
quadrature

\begin{multline}
  \int_\Gamma K(\xx_i(t_l),\yy) \, \sigma(\yy)
  \, dS(\yy) \approx \\
  \sum_{p=1}^{16} \sum_{n=1}^{N_s}
  w^{(l)}_{n} K(\xx_i(t_l),\xx_i({t}^{(l)}_{n}))
  s_i({t}^{(l)}_{n}) B^{(l)}_{np} \sigma_{ip} \quad \textrm{(self)}
  \\
  + \sum_{q\in A(i)} \sum_{p=1}^{16} \sum_{n=1}^{N_a}
  \tilde{w}_jK(\xx_i(t_l),\xx_q(\tilde{t}_n)) s_q(\tilde{t}_n)
  C_{np} \sigma_{qp}
  \quad \textrm{(adjacent)} \\
  + \sum_{q\neq i, q\not\in A(i)} \sum_{p=1}^{16}
  w_p K(\xx_i(t_l),\xx_q(t_p)) s_q(t_p)
  \sigma_{qp} \quad \textrm{(far)} \nonumber \; ,
\end{multline}
where $\bB^{(l)}$ and $\bC$ are interpolation
matrices from the standard Legendre nodes
to the self and adjacent panel support nodes,
respectively. Observe that the quadrature is
linear in $\sigma_{qp}$. In practice, we pre-compute
and store the self and adjacent matrix entries for each
interpolation point, which is a parallelizable
$O(N)$ calculation. The ``far'' interactions
are computed on-the-fly.

\begin{remark}
  \label{rmk:levelrestrict}
  We ensure that ``far'' interactions
  are handled to high precision by requiring that
  no two adjacent panels differ in length
  by more than a factor of 2. On a domain which does
  not nearly self-intersect this
  guarantees that no ``far'' interactions occur
  which are much closer than 1/2 of a panel away
  (assuming panels are relatively flat).
  Because the location of the singularity is
  bounded away from the panel and the smooth
  rule is of high order, we obtain a quadrature
  rule with sufficient precision.

  The overall order of accuracy of the GGQ we use
  is 20th-order, up to the precision of the ``far''
  interactions.
\end{remark}

\subsubsection{Fast determinant method}

Once the discretization is set, we can form
a compressed representation of the system matrix
using recursive skeletonization~\cite{ho2012fast}.
%
We use the implementation of this procedure
included in the fast linear algebra in
MATLAB (\texttt{FLAM}) package
\cite{hoFLAM_1253582}.
%
At low-to-medium frequencies, the scaling
of the recursive skeletonization algorithm
is $O(N\log N)$ in operation count and
storage and, by using a generalization
of the Sylvester determinant formula,
allows for a fast determinant
calculation in $O(N\log N)$ time as a
follow-up step.
%
At higher-frequencies,
the recursive skeletonization procedure,
which is based on the assumption that off-diagonal
blocks of the matrix are of low rank,
breaks down and does not offer a speed advantage.
%
These algorithms take a precision parameter
$\epsflam$ which determines the
accuracy to which any sub-blocks of the matrix
should be compressed. In all experiments,
we set $\epsflam = 10^{-14}$.

The compressed representation also allows
for fast applications of the system matrix,
its transpose, the inverse of the system
matrix, and the inverse transpose to
vectors.
%
In particular, this allows us to estimate the
smallest singular values by performing
randomized subspace iteration, see
\cite[Algorithm 4.4]{halko2011finding},
on the inverse operator.
%
Below, we use the smallest singular value
as a measure of the quality of the
eigenvalues found by approximating the
roots of the determinant.
%
We also evaluate the second smallest singular
value if the root finding procedure suggests a
possible double root.

\subsubsection{Interpolation and root-finding}

To estimate the eigenvalues, we fit a Chebyshev
interpolant to the discretized determinant as a
function of $k$ on intervals.
%
This is done adaptively so that the Chebyshev
coefficients of the determinant have decayed
to the point that the ratio of the last
coefficient to the largest coefficient is below
some threshold.
%
In all experiments, we set this threshold
as $\epscheb = 10^{-13}$.
%
We perform this fit using the \texttt{chebfun}
utility in the package of the same name
\cite{driscoll2014chebfun}
so that we can make use
of the \texttt{roots} utility to approximate
the roots of the determinant.

The \texttt{roots} utility returns the roots
of the polynomial in the complex plane, with
some minimal internal processing to remove
spurious roots.
%
Because our numerical determinant evaluation
is somewhat noisy and we fit the function up to
precision $\epscheb$, we perform some
further post-processing to eliminate remaining
spurious roots.
%
Let $k^{(l)}_\cheb$ denote the roots of the interpolants.
%
We ignore any of the returned roots with $|\imag(k^{(l)}_\cheb)|
> \sqrt{\epscheb}$, as these are too far from real-valued
to be non-spurious.
%
For the remaining roots, we consider the
properties of $\real(k^{(l)}_\cheb)$.
%
We inspect any pairs of roots $(k^{(p)}_\cheb,k^{(q)}_\cheb)$
for which $|\real(k^{(p)}_\cheb-k^{(q)}_\cheb)| < \sqrt{\epscheb}$,
as these are possibly spurious double roots.
%
For these pairs, we compare the right singular
vector of the appropriate BIE operator corresponding
to the smallest singular value for each of $\real(k^{(p)}_\cheb)$
and $\real(k^{(q)}_\cheb)$, which we denote by
$\bv_p$ and $\bv_q$.
%
If $\|\bv_p - \bv_q \bv_q^* \bv_p \| < 10^{-5}$, then
we consider the pair to be spurious.
%
For these near double roots, we also check
that there is no two dimensional null-space
corresponding to the root by estimating the
second smallest singular value of the BIE operator.
%
If this is larger than $10^{-5}$, then we declare
it to be a simple root.

We can obtain an a posteriori estimate of the
error in a computed root as follows.
%
Let $f$ denote an analytic function,
$P$ be the polynomial interpolant
of that function over some interval,
$\delta f = f-P$ be the difference,
and $k_\cheb$ denote a computed root
of $P$ which is simple (i.e. assume
that $P'(k_\cheb) \ne 0$).
%
The algorithm used by \texttt{chebfun}
to approximate the roots of $P$ is
backward stable~\cite{noferini2017chebyshev}.
Therefore the error in the roots will be
small relative to the error of the fit and
we set $P(k_\cheb) = 0$ below.
%
Suppose that
$f(k_\cheb + \delta k) = 0$ for some
small $\delta k$. Then

\begin{align*}
  0 &= f(k_\cheb + \delta k) \\
  0 &= P(k_\cheb + \delta k) + \delta f(k_\cheb + \delta k) \\
  \delta k &= -\frac{\delta f(k_\cheb + \delta k)}{P'(k_\cheb)} + O(\delta k^2) .
\end{align*}
In practice, we can obtain an approximate upper
bound for $|\delta f(k_\cheb+\delta k)|$
as $\epscheb \|P\|_\infty$ so that $\epscheb \|P\|_\infty/
|P'(k_\cheb)|$ provides an approximate upper bound
for the error in the root.

\subsection{Eigenvalues of an annulus}
We test our numerical machinery and validate our analytical
and numerical claims
by comparing the results to the true eigenvalues on the annulus
which are known analytically (see~\cref{sec:annul_dir_exact}).
In all of the examples below, we work on the annulus $r_{1}<r<r_{2}$
with $r_{1} = 1$ and $r_{2} = 1.7$.
If the inner boundary is discretized using $N_{1}$ panels,
then the outer boundary is discretized using $N_{2} =
\lceil r_{2}/r_{1} N_{1} \rceil +1$ panels to ensure that the
panels are approximately the same length on both the boundaries.
The total number of discretization points
is then given by $N = 16(N_{1} + N_{2})$.
Let $D^{N}_{k}$ denote the linear system corresponding
to the Nystr\"{o}m discretization of
$-2\cD_{k} -2\cW$ using generalized Gaussian quadrature,
and let $C^{N}_{k}$ denote the linear system corresponding
to the Nystr\"{o}m discretization of
$-2\cD_{k} - 2i\cS_{k} -2\cW$.
Let $f_{D}^{N}(k) = \text{det}(I+D^{N}_{k})$, and
$f_{C}^{N}(k) = \text{det}(I+C^{N}_{k})$.

\subsubsection{Convergence study}
\label{subsec:convannulus}
We demonstrate that
for sufficiently large $N$, if $k_{0}$ is a Dirichlet
eigenvalue of the annulus,
then $f_{D}^{N}(k_{D}) = 0$ and $f_{C}^{N}(k_{C}) = 0$
where $|k_{D} - k_{0}| \lesssim N^{-20}$,
and $|k_{C} - k_{0}| \lesssim N^{-20}$.
Recall that the GGQ we use has an expected order
of convergence of $N^{-20}$ for evaluating convolutions
with such integral kernels,
so that the error of the roots is observed to have the
same order as the quadrature rule.
In~\cref{fig:conv}, we show this result for $k_{0} = 13.48025717955055$
and plot the errors $|k_{D}-k_{0}|$ and $|k_{C}-k_{0}|$ as a function
of $N$.

\begin{figure}
  \centering
  \includegraphics[width=0.98\textwidth]{fig/conv-studies.pdf}
  \caption{Convergence studies for the Dirichlet eigenvalues
  computed using the integral equations
  $\cI - 2\cD_{k} - 2\cW$ (left) and
  $\cI - 2\cD_{k} - 2i \cS_{k} - 2\cW$ (right).}
  \label{fig:conv}
\end{figure}

\subsubsection{Spurious eigenvalues}
\label{subsec:spurannulus}
As noted in~\cref{subsec:dlanalysis},
if $k_{0}$ is a Neumann eigenvalue corresponding to the interior
inclusion, which in our case is the disk $r\leq r_{1}$, then
$f_{D}^{N}(k_{D}) = 0$ with $|k_{D}-k_{0}| = O(\varepsilon)$, even though $k_{0}$
is not a Dirichlet eigenvalue of the annulus, i.e.
the integral equation $-\cI - 2\cD_{k} -2\cW$ has a spurious eigenvalue.
In~\cref{fig:spur}, we demonstrate this result and also show that
$f_{C}^{N}(k_{0}) \neq 0$, i.e., the combined field representation is
robust and invertible at all values of $k$ which are not the Dirichlet
eigenvalues of the annulus.

\begin{figure}
\centering
\includegraphics[width=0.98\linewidth]{fig/det_spur.pdf}
\caption{The values of the discretized determinants $f_{C}^{N}(k)$ (left)
and $f_{D}^{N}(k)$ on the interval $k=[14,15]$ with $N=368$ (the
solid lines are the real parts of the determinant and the dashed lines
are the imaginary parts of the determinant). The vertical
dotted line denotes the spurious eigenvalue $k_{0} = 14.79595178235126$
and $f_{D}^{N}(k_{D}) = 0$ with $|k_{D}-k_{0}|=6.8\times 10^{-12}$.
On the other hand $|f_{C}^{N}(k_{0})|=0.42$, and thus $\cI-2\cD_{k}-2i\cS_{k}
-2\cW$ has no
spurious eigenvalue in the neighborhood of $k=k_{0}$.}
\label{fig:spur}
\end{figure}

\subsubsection{Speed}
\label{subsec:speed}
In this section, we demonstrate the $O(N\log{N})$ scaling of evaluating
$f^{N}_{C}(k)$ as long as $N$ is large enough to resolve the interactions
at the Helmholtz parameter $k$.
When $N$ is smaller than that, we observe a worse scaling since the
assumption that far-interactions are low-rank is no longer valid at
the tolerance of FLAM.
We plot the timing results corresponding to three different values of
$k$ in~\cref{fig:speed}. The times are as recorded for a laptop with
16Gb of RAM and an Intel Core i7-6600U CPU at 2.60GHz with 4 cores.
\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{fig/speed_res.pdf}
\caption{Time taken $(t)$ in seconds to evaluate the determinant $f_{C}^{N}(k)$ as
a function of $N$ for three different values of $k$.}
\label{fig:speed}
\end{figure}

\subsection{Eigenvalues of a barbell-shaped domain}
\label{subsec:barbell}

\begin{figure}
  \centering
  \begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{fig/ex_barbell_001_bdry}
    \caption{A barbell-shaped domain.}
    \label{subfig:barbell_bdry}
  \end{subfigure}
  ~
  \begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{fig/ex_many_holes_004_bdry}
    \caption{A domain with several inclusions.}
    \label{subfig:many_inclusions_bdry}
  \end{subfigure}
  \caption{Computational domains.}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[width=\textwidth]{fig/barbell_gallery}
  \caption{Vorticity plots of the first 119 eigenfunctions
    of the barbell-shaped domain.}
  \label{fig:barbell_gallery}
\end{figure}

\begin{figure}
  \centering
  \begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{fig/ex_barbell_001_sings_plot}
    \caption{Smallest singular value of the BIE operator.}
    \label{subfig:barbell_sings}
  \end{subfigure}
  ~
  \begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{fig/ex_barbell_001_coeffs_plot}
    \caption{Normalized Chebyshev coefficients of the
      determinant on 3 intervals in $k$.}
    \label{subfig:barbell_coeffs}
  \end{subfigure}
  \caption{Diagnostics for the first 119 barbell eigenvalues.}
  \label{fig:barbell_diagnostics}
\end{figure}

We consider the barbell-shaped domain in \cref{subfig:barbell_bdry}.
This domain is the union of a square of side-length 6,
a square of side-length 3, and a ``bridge'' connecting
them of height 1 and width 5/2.
%
For the sake of simplicity, we round the corners of the domain
to obtain a smooth object.
%
Applying the approach described in~\cite{epstein2016smoothed},
the corners of the domain are rounded by convolving with
the Gaussian kernel
\begin{equation}
  \nonumber
\phi(x) = \frac{1}{\sqrt{2\pi h}} e^{-x^2/(2 h^2)} \, ,
\end{equation}
with $h\approx 0.06$. This leaves the domain unperturbed
to high precision outside of a radius of $0.1$ around
each corner.
%
The eigenfunctions of such a domain display the well-known
localization property~\cite{trefethen2006computed}:
many of the eigenfunctions are approximately supported
within one of the squares.
%
We compute these eigenfunctions corresponding to
eigenvalues $k^2$ with $k$ in the range
$0.5 \leq k \leq 6.5$.

%
The panels are divided adaptively so that the smallest
panels in the rounded corners are smaller than $10^{-2}$,
which keeps the panels relatively flat.
%
This results in $N_p = 412$ after enforcing the
level-restriction property described in
\cref{rmk:levelrestrict}
and enforcing that no panel is larger than
one wavelength for the largest $k$
(here $\lambda=2\pi/6.5$).

As this is a simply-connected domain,
the eigenvalues are estimated by finding the values
$k$ for which $\cI-2\cDk-2\cW$ is non-invertible.
%
Let $f^N(k) = \det (\cI^N-2\cDk^N-2\cW^N)$.
To find the roots of $f^N(k)$, we fit a \texttt{chebfun}
representation of $f^N(k)$ on each of the intervals
$[j/2,(j+1)/2]$ for $j = 1,\ldots,12$.
%
We plot the absolute value of the Chebyshev coefficients
(normalized by the absolute value of the first coefficient)
of $f^N(k)$ on the intervals $[0.5,1.0]$, $[3.0,3.5]$,
and $[6.0,6.5]$ in \cref{subfig:barbell_coeffs}.
%
As expected, the coefficients decay exponentially
to zero, with more terms required at higher
frequencies.
%

We compute the roots of these Chebyshev interpolants
and apply the post-processing described above.
%
There were 135 total roots: 3 were removed because
the imaginary part was too large and 13 pairs were found
with values within $\sqrt{\epscheb}$ of each other.
%
For these 13 pairs, none represented two distinct
eigenvalues or a double root.
%
This leaves 119 roots in the range $0\leq k \leq 6.5$.
%
We plot the smallest singular value of
$\cI^N-\cDk^N-2\cW^N$ for each of these roots in
\cref{subfig:barbell_sings}
and plot the vorticity of the eigenfunctions
in \cref{fig:barbell_gallery}.
%
The singular values suggest that the quality of the
eigenvalues is good.
From the plots, we see that localization occurs
until about the 100th eigenvalue.

%

%\subsection{Robustness on a nearly multiply connected domain}
%\label{subsec:crescent}

\subsection{Eigenvalues of a domain with several inclusions}

\begin{figure}
  \centering
  \includegraphics[width=\textwidth]{fig/many_inclusions_gallery}
  \caption{Vorticity plots of the eigenfunctions corresponding
  to the first 76 eigenvalues of a domain with several inclusions.}
  \label{fig:many_inclusions_gallery}
\end{figure}

\begin{figure}
  \centering
  \begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{fig/ex_many_holes_004_sings_plot}
    \caption{Smallest singular value of the BIE operator
      corresponding to the first 76 eigenvalues of a
      domain with several inclusions.}
    \label{subfig:many_inclusions_sings}
  \end{subfigure}
  ~
  \begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{fig/ex_many_holes_004_coeffs_plot}
    \caption{Normalized Chebyshev coefficients of $f^N(k)$ on
      3 different intervals in $k$.}
    \label{subfig:many_inclusions_coeffs}
  \end{subfigure}
  \caption{Diagnostics for the eigenvalues of a domain
    with several inclusions.}
  \label{fig:many_inclusions_diagnostics}
\end{figure}

We now consider the multiply-connected domain in
\cref{subfig:many_inclusions_bdry}.
%
The domain is defined by a smooth rectangular region
of width 3 and height 2,
with an array of randomly rotated ``starfish'' shapes
removed.
%
Such shapes are of interest in
materials design, see, for instance, \cite{overvelde2012compaction}.
%
We compute the eigenfunctions corresponding to
eigenvalues $k^2$ with $k$ in the range
$3 \leq k \leq 9$ (this range includes the smallest
eigenvalue).

%
For this smooth shape, ensuring that no panel is
larger than one wavelength for the largest $k$
(here $\lambda=2\pi/9$) is sufficient to resolve
the object to high precision.
%
After enforcing the
level-restriction property described in
\cref{rmk:levelrestrict}, we end up with
$N_p = 224$.

As this is a multiply-connected domain,
the eigenvalues are estimated by finding the values
$k$ for which $\cI-2\cDk-2i\cSk-2\cW$ is non-invertible.
%
Let $f^N(k) = \det (\cI^N-2\cDk^N-2i\cSk^N-2\cW^N)$.
To find the roots of $f^N(k)$, we fit a \texttt{chebfun}
representation of $f^N(k)$ on each of the intervals
$[j/2,(j+1)/2]$ for $j = 6,\ldots,13$ and the intervals
$[j/4,(j+1)/4]$ for $j = 28,\ldots,35$.
%
It should be noted that, due to the relative sizes
of the domains,
this represents a lower frequency problem than
that for the barbell when measured in the number
of wavelengths across the object.
%
Thus, the use of a finer grid in frequency results from
the difficulty in resolving the Fredholm determinant
for this problem, which has a larger dynamical range
than that for the barbell.
%
We plot the absolute value of the Chebyshev coefficients
of $f^N(k)$ on the intervals $[3,3.5]$, $[6.5,7]$,
and $[8.75,9]$ in \cref{subfig:barbell_coeffs}.
%
As expected, the coefficients decay exponentially
to zero, with more terms required at higher
frequencies (note that the interval $[8.75,9]$ is
smaller than the others).
%

We compute the roots of these Chebyshev interpolants
and apply the post-processing described above.
%
There were 103 total roots: 21 were removed because
the imaginary part was too large and 6 pairs were found
with values within $\sqrt{\epscheb}$ of each other.
%
For these 6 pairs, none represented two distinct
eigenvalues or a double root.
%
This leaves 76 roots in the range $0\leq k \leq 9$.
%
We plot the smallest singular value of
$\cI^N-\cDk^N-2i\cSk^N-2\cW^N$ for each of these roots in
\cref{subfig:many_inclusions_sings}
and plot the vorticity of the eigenfunctions
in \cref{fig:many_inclusions_gallery}.
%
In the vorticity plots, we observe a different type of
localization property than that seen in the barbell,
with many of the eigenfunctions
approximately supported in a small, connected subset
of the domain. This is consistent with other studies
\cite{filoche2009strong,lindsay2018boundary}.

%

\begin{figure}
  \centering
  \begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{fig/ex_many_holes_004_sings_plot_west}
    \caption{Smallest singular value of the BIE operator
      for the computed roots on the original intervals (\textbullet)
      and an estimate of the error (dotted).}
    \label{subfig:many_inclusions_sings_west}
  \end{subfigure}
  ~
  \begin{subfigure}[t]{0.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{fig/ex_many_holes_004_sings_plot_ref}
    \caption{Smallest singular value of the BIE operator
      for the computed roots on the original intervals (\textbullet)
      and the values obtained on the refined interval ($\times$).}
    \label{subfig:many_inclusions_sings_ref}
  \end{subfigure}
  \caption{Further diagnostics for the eigenvalues of a domain
    with several inclusions.}
  \label{fig:many_inclusions_diagnostics_2}
\end{figure}

The singular values suggest that the quality of the
eigenvalues is good, with a few outliers.
%
To explain these outliers, we consider two quantities
which affect the singular value at a computed root.
%
As described above, we can approximate the
error in the computed root at $k_\cheb$ by
$\epscheb \|P\|_\infty/|P'(k_\cheb)|$,
where $P$ is the interpolating polynomial.
%
The singular value estimate itself is affected by
the error incurred in applying the inverse of the
compressed BIE matrix, which can be hard to quantify
\cite{ho2012fast}.
%
We approximate this error by $O(\sqrt{N})\epsflam$
and assume this is the order of the error in
the singular value estimate.
We plot the maximum of these two estimates
along with the computed singular values in
\cref{subfig:many_inclusions_sings_west}.
There is a reasonably good correlation between
the maximum of the error estimates and the
observed smallest singular value for the BIE,
especially for larger errors.

The worst outliers are from the left half
of the interval $[4.5,5]$.
%
Because the determinant is much larger on the
right half than the left half of $[4.5,5]$,
we can improve the estimate for the error
in the roots by subdividing the interval.
We plot the smallest singular value of the
BIE for the roots obtained by fitting a polynomial
on $[4.5,4.75]$ in \cref{subfig:many_inclusions_sings_ref};
the roots on the refined interval
are of significantly higher quality.

\begin{remark}
  The above experience suggests that
  the ratio $\|P\|_\infty/|P'(k_\cheb)|$
  is a useful diagnostic for performing
  automated eigenvalue estimation.
  Note that at a multiple root, this ratio
  will be more difficult to bound.
\end{remark}
%

%


\section{Materials and methods} \label{sec:metodos}
    In this section we will explain in more detail the problem modeling, ie the machine learning techniques implemented in the project and how they were specified.

    \subsection{Related Work - Deep VO}
        The paper \cite{wang2017deepvo} presents a novel end-to-end framework for monocular VO by using deep Recurrent Convolutional Neural Networks (RCNNs) \cite{liang2015recurrent}. Since it is trained and deployed in an end-to-end manner, it infers poses directly from a sequence of raw RGB images (videos) without adopting any module in the conventional VO pipeline. Based on the RCNNs, it not only automatically learns effective feature representation for the VO problem through Convolutional Neural Networks, but also implicitly models sequential dynamics and relations using deep Recurrent Neural Networks. The framework has been tested under the Kitti Dataset \footnote{\url{http://www.cvlibs.net/datasets/kitti/eval_odometry.php}}, showing great results.

        The model is mainly composed of CNN based feature extraction, as previous presented in \cite{chen2016deep}, and RNN based sequential modelling, as shown in \cite{chung2015recurrent}. The architecture of the proposed VO system is shown in Figure \ref{fig:8}, it takes a monocular image sequence as input. Two consecutive images are stacked together to form a tensor for the deep RCNN to learn how to extract motion information and estimate poses. The VO system develops over time and estimates new poses as images are captured.

        The image feature extraction of is done by a CNN, which configuration is shown at Figure \ref{fig:9}. The CNN takes raw RGB images instead of pre-processed counterparts, such as optical flow or depth images, as input because the network is trained to learn an efficient feature representation with reduced dimensionality for the VO.

        Following the CNN, a deep RNN is designed to conduct sequential learning, i.e., to model dynamics and relations among a sequence of CNN features. Since the RNN is capable of modelling dependencies in a sequence, it is well suited to the VO problem which involves temporal model (motion model) and sequential data (image sequence).

        To learn the hyperparameters $\theta$ of the DNNs, the Euclidean distance between the ground truth pose ($p_k$ ,$\varphi_k$) at time $k$ and its estimated one ($\text{\^{p}}_k$ ,$\hat{\varphi_k}$) is minimised. The loss function is composed of Mean Square Error (MSE) of all positions p and orientations :
        \begin{equation}
            \theta^* = \underset{\theta}{\mathrm{argmin}} \frac{1}{N} \sum_{i=1}^{N}\sum_{k=1}^{t} \left\| \hat{p_k} - p_k \right\|^{2}_2 + k \cdot \left\| \hat{\varphi_k} - \varphi_k \right\|^{2}_2
            \label{eq:MSE}
        \end{equation}

         In Figure \ref{fig:10}, the losses results of the models are given for both training and validate datasets. As shown in the fig., the loss significantly reduces with time, and, once the training and validation losses shown a similar behavior we can also say that is free of overfiting, so, the model is well-fit.

         The Figures \ref{fig:11} and \ref{fig:12} show us the obtained maps for the framework for both training and test datasets. As we can see in the figures, the obtained odometry from the system is very close to the ground truth position, showing how great the system performs. The system is analysed according to the KITTI VO/SLAM evaluation metrics, i.e., averaged Root Mean Square Errors (RMSEs) of the translational and rotational errors for all subsequences of lengths ranging. The average RMSEs of the estimated VO are given in Figure \ref{fig:13}. Al- though the result of the DeepVO is worst than that of the stereo VISO2 (VISO2\_S), it is consistently better than the monocular VISO2 (VISO2\_M).

         The paper presents a novel end-to-end monocular VO algorithm based on Deep Learning, that does not depend on any module in the conventional VO algorithms (even camera calibration) for pose estimation and it is trained in an end-to-end manner, there is no need to carefully tune the parameters of the VO system. Based on the KITTI VO benchmark, it is verified that it can produce accurate VO results with precise scales and work well in completely new scenarios.

         For this project, we aim to reproduce the paper approach using images and poses obtained at the AirSim, an open source simulator for autonomous vehicles built on Unreal Engine/Unity, from Microsoft AI\&Research \footnote{\label{ref:note1}\url{https://github.com/microsoft/AirSim}}.

    \subsection{Project Dataset}
        \subsubsection{Data Collection}
            In order to create our own \emph{dataset} to perform the article model we had use the Microsoft AirSim Simulator to generate both images and poses.

            Six sequences were generated under three different scenarios: Neighborhood(Figure \ref{fig:neig}), Africa (Figure \ref{fig:africa}) and LandscapeMountains(Figure \ref{fig:mountain}).

            The simulator were used at the \emph{ComputerVision} mode and the cameras controlled via keyboard. Each sequence has around four hundred pictures, making a total of around two thousand images for the training set.

        \subsubsection{Images Data Prerocessing}
            The first stage of image preprocessing was eliminate all failed pictures given by the simulator, excluding them from both pictures and pose lists.

            After that, as we read the images we convert them to grayscale and normalize their pixels values, in a range that goes from -0.5 to 0.5 in order to be able to use the pretrained weight of FlowNet, all of it were done using the OpenCV library.

            At least, we concatenated the pictures two by two, once that the two images will going to work as input for our Neural Network.

        \subsubsection{Poses Data Preprocessing}
            The AirSim recording mode, as we further explain, give us a text file with the following informations for each image: timestamp, x\_position, y\_position, z\_position, quaternions\_x, quaternions\_y, quaternions\_z, quaternions\_w and a filename of the image associated to that pose.

            As explained in \citeonline{quat_angle}, a quaternion is a four-element vector that can be used to encode any rotation in a 3D coordinate system.  Technically, a quaternion is composed of one real element and three complex elements, and it can be used for much more than rotations. For our work, we need the angles to be at an Euler representation. Euler angles provide a way to represent the 3D orientation of an object using a combination of three rotations about different axes.

            In order to transform the given angles to an Euler representaion we need to use the following equations:

            \begin{equation}
                \phi = arctan(\frac{2(ab+cd)}{a^2 - b^2 - c^2 + d^2})
            \end{equation}
            \begin{equation}
                \theta = -arcsin(2(bd-ac))
            \end{equation}
            \begin{equation}
                \Psi = arctan(\frac{2(ad+bc)}{a^2 + b^2 - c^2 - d^2}
            \end{equation}

            This can be easily done in \emph{Python} using the \emph{Scipy} \footnote{\url{https://docs.scipy.org/doc/scipy-1.2.1/reference/generated/scipy.spatial.transform.Rotation.html}} library.

            Another thing we had to do in order to process the information was make the first pose our start point, so we subtract all the other poses from the first one value. In addition, in addition we subtract all the poses, with the except of the firts, from the previous pose value, so, the network has to predict only the dislocate between two images.

    \subsection{Deep VO Implementation}
        In order to simulate Wang's\cite{wang2017deepvo} paper for our dataset, we implemented the same RCNN model showed at Figure \ref{fig:2}.

        For implementing the model we used the PyTorch \footnote{\url{https://pytorch.org/}} library. All the convolutionals layers has a \emph{Conv2d} structure, followed by a \emph{BatchNorm2d} (batch normalization), and then a \emph{LeakyReLU} function (a Rectified Linear Activation Function) and in the end, a \emph{Dropout}.

        A convolution is the simple application of a filter to an input that results in an activation, the batch normalization reduces the amount by what the hidden unit values shift around (covariance shift). The activation function is responsible for transforming the summed weighted input from the node into the activation of the node or output for that input. And in the dropout stage, at each training stage, individual nodes are either dropped out of the net with probability 1-p or kept with probability p, so that a reduced network is left; incoming and outgoing edges to a dropped-out node are also removed.

        For the CNN part we tested two different configurations, which results will be further commented. We first fitted the model without any previous information and, in a second stage, we loaded pretrained weight of FlowNet network \footnote{\url{https://towardsdatascience.com/a-brief-review-of-flownet-dca6bd574de0}}.

        For the RNN part of the network, we had two \emph{LSTM} layers followed by a linear output, that receives 100 features as input and outputed 6 features (our pose). The Long Short-Term Memory (LSTM) networks are a modified version of recurrent neural networks, which makes it easier to remember past data in memory.

        To define the loss function we used the Mean Square Error, same as Wang's paper and which equations is defined at Equation \ref{eq:MSE}. The used optimizer was the Adagrad, an algorithm for gradient-based optimization that does just this: It adapts the learning rate to the parameters, performing smaller updates (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. high learning rates) for parameters associated with infrequent features.


\chapter{Inner product spaces}

One structure that we often use in $\R^n$, but which is missing from
abstract vector spaces, is the dot product. In
Chapter~\ref{cha:vectors-rn}, we saw how to use dot products to
compute the length of a vector, the angle between two vectors, and to
decide when two vectors are orthogonal. We also used dot products to
define the projection of one vector onto another, and to find shortest
distances between various objects (such as points and planes, two
lines, etc).

In this chapter, we will consider inner product spaces. An inner
product space is essentially an abstract vector spaces that has been
equipped with an operation that works ``like'' the dot product.


\chapter{Survival analysis}
\section{Survivor functions for whole sample}
In the fourth exercise we do survival analysis with data from lung cancer patients. The three variables used here are \texttt{PRE30}, indicating whether the patient is a smoker, \texttt{AGE}, encoding the age at surgery, and \texttt{Risk1Y}, indicating whether the patient died within one year after the surgery. The first observations are shown in Table \ref{5table}. Hence, this data can be understood as interval censored survival data. In this exercise we were asked to assume failure times at \texttt{AGE}+1, if the patient died and fit survivor functions to model $S(t)=P(T>t)$, where $T$ is a random variable encoding the failure time of an object, here the time of death of a patient.
\begin{table}[ht]
\centering
\begin{tabular}{llrl}
  \hline
 & PRE30 & AGE & Risk1Y \\
  \hline
1 & TRUE &  60 & FALSE \\
  2 & TRUE &  51 & FALSE \\
  3 & TRUE &  59 & FALSE \\
  4 & FALSE &  54 & FALSE \\
  5 & TRUE &  73 & TRUE \\
  6 & FALSE &  51 & FALSE \\
   \hline
\end{tabular}
\caption{First observations of the lung cancer patient data set.}
\label{5table}
\end{table}

To assess which parametric model is reasonable for the data, we use non-parametric estimators for the survival curve. Therefore we draw the Kaplan-Meier estimator, given by $$\hat{S}_{KM}(t)=\prod_{\{j:\tau_j<t\}}(1-\frac{d_j}{r_j}),$$ where $0\leq \tau_1<\tau_2<...$ denote the ordered uncensored failure times (here \texttt{AGE}+1), $r_i$ and $d_i$ the number of units still at risk and failures at $\tau_i$ respectively. A similar estimator, especially in those areas where still many cases are at risk, is the Fleming-Harrington estimator $$\hat{S}_{FH}(t)=\prod_{\{j:\tau_j<t\}}\exp(-\frac{d_j}{r_j}).$$ Both are shown in Figure \ref{5nonpar} where we can see that they only differ obviously at ages $>80$. In the area of lower risk both seem to be identical, why we can't see the orange line. We can also see that the confidence band spreads as the survival probability decreases. Both curves are fitted using \texttt{survfit} from the package \textit{survival} which also computes pointwise confidence intervals using Greenwood's formula.
\begin{figure}[!t]
\centering
\includegraphics[width=0.7\textwidth, keepaspectratio]{ex4/survfits.png}
\caption{Nonparametric estimators for the survivor function. KM = Kaplan-Meier drawn in blue; FH=Fleming-Harrington in orange. Dashed lines show the confidence bands.}
\label{5nonpar}
\end{figure}
\begin{figure}[!b]
\centering
\includegraphics[width=0.7\textwidth, keepaspectratio]{ex4/paramsurvfits.png}
\caption{KM estimator and parametric fits for the survivor function. Dashed lines represent the confidence band of the KM estimator. Exponential fit in orange; Weibull fit in blue.}
\label{5parfit}
\end{figure}

With the function \texttt{survreg} we can fit parametric models to the data. Typical models are the exponential, Weibull and log-logistic models. Because we were asked to fit the first two in this exercise, I only give more details on them. The exponential model assumes a constant Hazard function, i.e. rate of failure, of $\lambda$ over time resulting in an exponential survivor function $S_{exp}(t)=\exp(-\lambda t)$. With the Weibull distribution we can handle varying Hazard rates by taking $h(t)=\alpha \lambda ^{\alpha} t^{\alpha-1}$ resulting in a survivor function $S(t)=\exp(-(\lambda t)^\alpha)$. At $t=1$ we have $h(1)=\alpha\lambda^\alpha$, so $\lambda$ is still the parameter of failure rate, defined for the unit time and modified by the shape parameter. Obviously we get the exponential model for $\alpha=1$, an increasing Hazard function for $\alpha>1$ and an decreasing Hazard or $0<\alpha<1$. The Weibull distribution is often more appropriate when we can't assume constant Hazard rates. In clinical studies concerning relapses of psychological diseases for example a relapse, i.e. failure, gets often less probable with time since recovery. Here a decreasing Hazard function would be appropriate. But in general older living organisms have a smaller survivor probability over time than younger ones indicating the use of an increasing Hazard. In this example the fits (see Figure \ref{5parfit}) compared to the nonparametric KM estimator suggest that an exponential model can be rejected quite obviously, since the shape does not at all represent the step function. The Weibull model on the other hand seems very good as it follows the KM estimation pretty directly. Although it lies beneath the lower confidence bound at an age around 78, the overall shape suits pretty well.

\section{Comparison of smokers and non-smokers}
Now we split the data set into two parts - smokers and non-smokers - according to the variable \texttt{PRE30}. Fitting a KM estimate for each group leads to Figure \ref{5nonpargroups}. We see very clear that the curve of smokers is stepper than the other. The curve for nonsmokers drops in the end since the only patient with age 81 in the nonsmokers group, who also was the oldest one, died. But all in all the survival probability of nonsmokers seems higher. In the plot we also see that there are more steps in the curve for the smokers. This is because of the unbalanced sample sizes. With 386 smokers and 84 nonsmokers the proportion of smokers is 82.12$\%$. It is well known that smoking increases the probability to get cancer, so this is not a surprising observation here. An inferential comparison of the two groups to check whether the visual differences are statistically significant is done with a log-rank test. With the R function \texttt{survdiff} in the same package we get the results $\chi ^2=2.7$ and $p=.1$. This indicates a difference between the groups, too, which most researchers won't interpret as significant. Since I am no expert in this area, I can't finally judge it, but would emphasize that the difference may be statistically relevant in this case.
\begin{figure}[!htb]
\centering
\includegraphics[width=0.8\textwidth, keepaspectratio]{ex4/smokers.png}
\caption{Kaplan-Meier estimate for smokers (red) and nonsmokers (blue) with confidence bands.}
\label{5nonpargroups}
\end{figure}

Now we want to check whether the assumption of a Weibull model is still justified when we look at the groups separately. Therefore we fitted a Weibull curve for each of the groups (see Figure \ref{5paragroups}). For the group of smokers we see like above a very good fit. For nonsmokers the Weibull fit looks quite well but is probably distorted by the sudden drop down in the end. It seems like a less steep curve would fit better to the data in the beginning. This may be solved by an extended sample where especially more older patient are included. Of course, this is sometimes hard to accomplish in practice. Still the Weibull curve of the smokers group decreases faster than the other and I would finally conclude, also because it seems quite reasonable (although a statistician should argue rather with data than with common sense), that smoking decreases the probability of surviving one year after the surgery.
\begin{figure}[!h]
\centering
\includegraphics[width=\textwidth, keepaspectratio]{ex4/groupsparam}
\caption{Kaplan-Meier estimate with dashed confidence bands (black) and Weibull fit (blue) for patient groups of smokers (left) and nonsmokers (right).}
\label{5paragroups}
\end{figure}