\section{Proof of Proposition 1}
\label{sec:supp-proof}

Adversarial training usually results in reduced accuracy on clean examples. In practice, training on $50\%$ clean and  $50\%$ adversarial examples instead of training \emph{only} on adversarial examples allows to control the robustness-accuracy trade-off to some extent, \ie, increase accuracy while sacrificing robustness. The following proposition shows that there exist problems where adversarial training is unable to reconcile robustness and accuracy, while \ConfTrain is able to obtain \emph{both}:

\begin{proposition}\label{prop:toy-example}
    We consider a classification problem with two points $x=0$ and $x=\epsilon$ in $\mR$ with deterministic labels, \ie,
    $p(y=2|x=0)=1$ and $p(y=1|x=\epsilon)=1$, such that the problem is fully determined by the probability $p_0=p(x=0)$.
    The Bayes error of this classification problem is zero. Let the predicted probability distribution over classes be $\tilde{p}(y|x)=\frac{e^{g_y(x)}}{e^{g_1(x)}+e^{g_2(x)}}$, where $g:\R^d \rightarrow \R^2$ is the classifier and we assume that the function $\lambda:\mR_+ \rightarrow [0,1]$ used in \ConfTrain is monotonically decreasing and $\lambda(0)=1$. Then, the error of the Bayes optimal classifier (with cross-entropy loss) for
    \vspace*{-8px}
    \begin{itemize}
        \item adversarial training on $100\%$ adversarial examples
        is $\min\{p_0,1-p_0\}$.
        \vspace*{-3px}
        \item adversarial training on $50\%$/$50\%$ adversarial/clean examples per batch
        is $\min\{p_0,1 - p_0\}$.
        \vspace*{-3px}
        \item \ConfTrain on $50\%$ clean and $50\%$ adversarial examples
        is \emph{zero}
        if $\lambda(\epsilon) < \min\left\{\nicefrac{p_0}{1-p_0},\nicefrac{1-p_0}{p_0}\right\}$.
        \vspace*{-6px}
    \end{itemize}
\end{proposition}

\begin{proof}
    First, we stress that we are dealing with three different probability distributions over the labels:
	the true one $\Pr(y|x)$, the imposed one during training $\hat{p}(y|x)$ and the predicted one $\tilde{p}(y|x)$. We also note that $\hat{p}$ depends on $\lambda$ as follows:
    \begin{align}
        \hat{p}(k) = \lambda p_y(k) + (1 - \lambda) u(k)
    \end{align}
    where $p_y(k)$ is the original one-hot distribution, \ie, $p_y(k) = 1$ iff $k = y$ and $p_y(k) = 0$ otherwise with $y$ being the true label, and $u(k) = \nicefrac{1}{K}$ is the uniform distribution. We note that this is merely an alternative formulation to the target distribution as outlined in the main paper.
    Also note that $\lambda$ itself is a function of the norm $\|\delta\|$; here, this dependence is made explicit by writing $\hat{p}(\lambda)(y|x)$.
	This makes the expressions for the expected loss of \ConfTrain slightly more complicated. We first derive the Bayes optimal classifier and its loss for \ConfTrain.
    We introduce
    \begin{align}
        a = g_1(0)-g_2(0), \quad b=g_1(\epsilon)-g_2(\epsilon). \label{eq:supp-a-b}
    \end{align}
    and express the logarithm of the predicted probabilities (confidences) of class $1$ and $2$ in terms of these quantities.
	\begin{align*}
		\hspace*{-10px}
		-\log \tilde{p}(y=2|x=x)&= -\log\Big(\frac{e^{g_2(x)}}{e^{g_1(x)} + e^{g_2(x)}}\Big) = \log\big(1 + e^{g_1(x)-g_2(x)}\big)
     	=\begin{cases} \log\big(1+e^a\big) & \textrm{ if } x=0\\ \log(1+e^b) & \textrm{ if } x=\epsilon \end{cases}.\\
    	\hspace*{-10px}
    	-\log \tilde{p}(y=1|x=x)&=-\log\Big(\frac{e^{g_1(x)}}{e^{g_1(x)} + e^{g_2(x)}}\Big) = \log\big(1 + e^{g_2(x)-g_1(x)}\big) =
    	\begin{cases} \log(1+e^{-a}) & \textrm{ if } x=0\\ \log\big(1+e^{-b}\big) & \textrm{ if } x=\epsilon\end{cases}.
    \end{align*}

    We consider the approach by \cite{MadryICLR2018} with $100\%$ adversarial training. The expected loss can be written as
	\begin{align*}
		&\Exp\Big[\max_{\norm{\delta}_\infty\leq \epsilon} L(y,g(x+\delta)\Big]= \Exp\Big[\Exp\Big[\max_{\norm{\delta}_\infty\leq \epsilon} L(y,g(x+\delta)|x\Big]\Big]\\
		=&\Pr(x=0)\Pr(y=2|x=0)\max\left\{-\log\big(\tilde{p}(y=2|x=0)\big),-\log\big(\tilde{p}(y=2|x=\epsilon)\big)\right\}\\
		&+(1-\Pr(x=0))\Pr(y=1|x=\epsilon)\max\left\{-\log\big(\tilde{p}(y=1|x=0)\big),-\log\big(\tilde{p}(y=1|x=\epsilon)\big)\right\}\\
		=&\Pr(x=0)\max\left\{-\log\big(\tilde{p}(y=2|x=0)\big),-\log\big(\tilde{p}(y=2|x=\epsilon)\big)\right\}\\
	 	&+ (1-\Pr(x=0))\max\left\{-\log\big(\tilde{p}(y=1|x=0)\big),-\log\big(\tilde{p}(y=1|x=\epsilon)\right\}
	\end{align*}
	This yields in terms of the parameters $a,b$ the expected loss:
    \begin{align*}
    	L(a,b)=&  \max\Big\{ \log(1+e^a),\log(1+e^b)\Big\}p_0 + \max\Big\{ \log(1+e^{-a}),\log(1+e^{-b})\Big\}(1-p_0)
    \end{align*}
   	The expected loss is minimized if $a=b$ as then both maxima are minimal. This results in the expected loss
   	\begin{align*}
    	L(a)= \log(1+e^a)p_0 + \log(1+e^{-a}) (1-p_0).
    \end{align*}
    The critical point is attained at $a^*=b^*=\log\Big(\frac{1-p_0}{p_0}\Big)$.
    Thus
    \begin{align*}
        a^*=b^*=\begin{cases} >0 & \textrm{ if } p_0<\frac{1}{2},\\ <0 & \textrm{ if } p_0>\frac{1}{2}.\end{cases}
    \end{align*}
    Thus, we classify $x=0$ correctly, if $p_0>\frac{1}{2}$ and $x=\epsilon$ correctly if $p_0<\frac{1}{2}$. As result, the error of $100\%$ adversarial training is given by
    $\min\{p_0,1-p_0\}$ whereas the Bayes optimal error is zero as the problem is deterministic.

    Next we consider $50\%$ adversarial plus $50\%$ clean training. The expected loss
	\begin{align*}
		\Exp\Big[\max_{\norm{\delta}_\infty\leq \epsilon} L(y,g(x+\delta)\Big] + \Exp\Big[ L(y,g(x+\delta)\Big],
	\end{align*}
	can be written as
    \begin{align*}
	    L(a,b)=&  \max\Big\{ \log(1+e^a),\log(1+e^b)\Big\}p_0 \\
	           &+ \max\Big\{ \log(1+e^{-a}),\log(1+e^{-b})\Big\}(1-p_0)\\
	           &+ \log(1+e^a)p_0 + \log(1+e^{-b})(1-p_0)
    \end{align*}
    We make a case distinction. If $a\geq b$, then the loss reduces to
    \begin{align*}
	    L(a,b)=&    \log(1+e^a)p_0 + \log(1+e^{-b})(1-p_0) \\
	           & + \log(1+e^a)p_0 + \log(1+e^{-b})(1-p_0)\\
	           &\geq L(a,a)\\
	           &=2\log(1+e^a)p_0 + 2\log(1+e^{-a})(1-p_0)
    \end{align*}
    Solving for the critical point yields $a^*=\log\Big(\frac{1-p_0}{p_0}\Big)=b^*$. Next we consider the set $a\leq b$.  This yields the loss
    \begin{align*}
	    L(a,b)=& \log(1+e^b)p_0 + \log(1+e^{-a})(1-p_0)\\
	           &+ \log(1+e^a)p_0 + \log(1+e^{-b})(1-p_0)
    \end{align*}
    Solving for the critical point yields $a^*=\log\Big(\frac{1-p_0}{p_0}\Big)=b^*$ which fulfills $a\leq b$. Actually, it coincides with the solution found already for $100\%$ adversarial training and thus resulting error of $50\%$ adversarial, $50\%$ clean training is again $\min\{p_0,1-p_0\}$ which is not equal to the Bayes optimal error.

    For our confidence-calibrated adversarial training one first has to solve
	\begin{align*}
		\delta_x^*(j)=\argmax_{\norm{\delta}_\infty \leq \epsilon} \max\limits_{k\neq j} \tilde{p}(y=k\,|\,x+\delta).
	\end{align*}
	We get with the expressions above (note that we have a binary classification problem):
    \begin{align*}
	    \delta^*_0(1)=\argmax_{\norm{\delta}_\infty \leq \epsilon} \tilde{p}(y=2|{0}+\delta) = \begin{cases} 0 & \textrm{ if } a<b \\ \epsilon & \textrm{ else}.\end{cases}.\\
	    \delta^*_0(2)= \argmax_{\norm{\delta}_\infty \leq \epsilon} \tilde{p}(y=1|{0} +\delta) = \begin{cases} \epsilon & \textrm{ if }  a<b  \\ 0 & \textrm{ else}.\end{cases}.\\
			\delta^*_\epsilon(1)= \argmax_{\norm{\delta}_\infty \leq \epsilon} \tilde{p}(y=2|\epsilon+\delta) = \begin{cases} -\epsilon & \textrm{ if }  a<b  \\0 & \textrm{ else}.\end{cases}.\\
	    \delta^*_\epsilon(2)=\argmax_{\norm{\delta}_\infty \leq \epsilon} \tilde{p}(y=1|\epsilon +\delta) = \begin{cases} 0 & \textrm{ if }  a<b  \\ -\epsilon & \textrm{ else}.\end{cases}.
    \end{align*}
    Note that the imposed distribution $\hat{p}$ over the classes depends on the true label $y$ of $x$ and $\delta^*_x(y)$ and then $\lambda(\norm{\delta^*_x(y)}_\infty)$. Due to the simple structure of the problem it holds that $\norm{\delta^*_x(y)}_\infty$ is either $0$ or $\epsilon$.

	In \ConfTrain we use for $50\%$ of the batch the standard cross-entropy loss and for the other $50\%$ we use the following loss:
	\begin{align*}
		L\left(\hat{p}_y\big(\lambda\big(\norm{\delta^*_x(y)}\big)\big)(x),\tilde{p}(x)\right)= - \sum_{j=1}^2 \hat{p}_y\big(\lambda\big(\norm{\delta^*_x(y)}\big)\big)(y=j\,|\,x=x) \log\left(\tilde{p}\big(y=j\,|\,x=x+\delta^*_x(j)\big)\right).
	\end{align*}
	The corresponding expected loss is then given by
	\begin{align*}
		&\Exp\Big[ L\Big(\hat{p}_y\big(\lambda\big(\norm{\delta^*_x(y)}\big)\big)(x),\tilde{p}(x)\Big)\Big]
		=\Exp\Big[\,\Exp\Big[ L\Big(\hat{p}_y\big(\lambda\big(\norm{\delta^*_x(y)}\big)\big)(x),\tilde{p}(x)\Big) \Big| \,x\Big]\Big] \\
		=&\Pr(x=0)\,\Exp\Big[ L\Big(\hat{p}_y\big(\lambda\big(\norm{\delta^*_0(y)}\big)\big)(0),\tilde{p}(0)\Big)\Big|\,x=0\Big] + \Pr(x=\epsilon)\,\Exp\Big[ L\Big(\hat{p}_Y\big(\lambda\big(\norm{\delta^*_\epsilon(y)}\big)\big)(\epsilon),\tilde{p}(\epsilon)\Big)\Big|\,x=\epsilon\Big],
	\end{align*}
	where $\Pr(x=0)=p_0$ and $\Pr(x=\epsilon)=1-\Pr(x=0)=1-p_0$. With the true conditional probabilities $\Pr(y=s\,|\,x)$ we get
	\begin{align*}
		&\Exp\Big[ L\Big(\hat{p}_y\big(\lambda(\delta)\big)(x),\tilde{p}(x)\Big)\Big|\,x\Big]
		=\sum_{s=1}^2 \Pr(y=s\,|\,x) \, L\Big(\hat{p}_s\big(\lambda\big(\norm{\delta^*_x(s)}\big)\big)(x),\tilde{p}(x)\Big)\\
		=&- \sum_{s=1}^2 \Pr(y=s\,|\,x) \sum_{j=1}^2 \hat{p}_s\big(\lambda\big(\norm{\delta^*_x(s)}\big)\big)(y=j\,|\,x) \log\big(\tilde{p}(y=j\,|\,x=x+\delta^*_x(s))\big)
	\end{align*}
	For our problem it holds with $\Pr(y=2\,|\,x=0)=\Pr(y=1\,|\,x=\epsilon) =1$ (by assumption). Thus,
	\begin{align*}
		\Exp\Big[ L\Big(\hat{p}_y\big(\lambda(\delta)\big)(x),\tilde{p}(x)\Big)\Big|x=0\Big]=-\sum_{j=1}^2 \hat{p}_2\big(\lambda\big(\norm{\delta^*_0(2)}\big)\big)(y=j\,|\,x=0) \log\big(\tilde{p}(y=j\,|\,x=x+\delta^*_0(2))\big)\\
		\Exp\Big[ L\Big(\hat{p}_y\big(\lambda(\delta)\big)(x),\tilde{p}(x))\Big|\,x=\epsilon\Big]=-\sum_{j=1}^2 \hat{p}_1\big(\lambda\big(\norm{\delta^*_\epsilon(1)}\big)\big)(y=j\,|\,x=\epsilon) \log\big(\tilde{p}(y=j\,|\,x=x+\delta^*_\epsilon(1))\big)
	\end{align*}
	As $\norm{\delta^*_x(y)}_\infty$ is either $0$ or $\epsilon$ and $\lambda(0)=1$ we use in the following for simplicity the notation $\lambda=\lambda(\norm{\epsilon}_\infty)$. Moreover, note that
	\begin{align*}
		\hat{p}_y(\lambda)(y=j|x=x)=\begin{cases} \lambda + \frac{(1-\lambda)}{K} & \textrm{ if } y = j,\\ \frac{(1-\lambda)}{K} & \textrm{ else }\end{cases},
	\end{align*}
	where $K$ is the number of classes. Thus, $K=2$ in our example and we note that $\lambda + \frac{(1-\lambda)}{2}=\frac{1+\lambda}{2}$.

    With this we can write the total loss (remember that we have half normal cross-entropy loss and half the loss for the adversarial part with the modified ``labels'') as
    \begin{align*}
    	L(a,b)=&  p_0\Big[\log(1+e^a) \Id_{a\geq b}
           + \Id_{a<b}\Big( \frac{(1+\lambda)}{2}\log(1+e^b)+\frac{(1-\lambda)}{2}\log(1+e^{-b})\Big)\Big]\\
					 &+ (1-p_0)\Big[\log(1+e^{-b})   \Id_{a\geq b} +\Id_{a<b}\Big( \frac{(1+\lambda)}{2}\log(1+e^{-a})+\frac{(1-\lambda)}{2}\log(1+e^a)\Big)\Big]\\
           &+ \log(1+e^a)p_0 + \log(1+e^{-b})(1-p_0),
    \end{align*}
    where we have omitted a global factor $\frac{1}{2}$ for better readability (the last row is the cross-entropy loss).  We distinguish two sets in the optimization. First we consider the case $a\geq b$. Then it is easy to see that in order to minimize the loss we have $a=b$.
    \begin{align*}
    	\partial_a L &= 2\frac{e^a}{1+e^a}p_0-\frac{e^{-a}}{1+e^{-a}}(1-p_0)
    \end{align*}
    This yields $e^a=\frac{1-p_0}{p_0}$ or $a=\log\big(\frac{1-p_0}{p_0}\big)$ and the minimum for $a\geq b$ is attained on the boundary of the domain of $a\leq b$. The other case is $a\leq b$. We get
    \begin{align*}
    	\partial_a L =& \Big[\frac{(1+\lambda)}{2}\frac{-e^{-a}}{1+e^{-a}}+\frac{(1-\lambda)}{2}\frac{e^a}{1+e^a}\Big](1-p_0)
                   + p_0 \frac{e^a}{1+e^a}\\
    	\partial_b L =& \Big[\frac{(1+\lambda)}{2}\frac{e^{b}}{1+e^{b}}+\frac{(1-\lambda)}{2}\frac{-e^b}{1+e^{-b}}\Big]p_0
                  + (1-p_0) \frac{-e^{-b}}{1+e^{-b}}
    \end{align*}
    This yields the solution
    \begin{align*}
    	a^* = \log\left(\frac{\frac{1+\lambda}{2}(1-p_0)}{p_0 + \frac{1-\lambda}{2}(1-p_0)}\right), \qquad
    	b^* = \log\left(\frac{\frac{1-\lambda}{2}p_0 + (1-p_0)}{\frac{1+\lambda}{2}p_0}\right)
    \end{align*}
    It is straightforward to check that $a^*<b^*$ for all $0<p_0<1$, indeed we have
	\begin{align*}
		\frac{\frac{1+\lambda}{2}(1-p_0)}{p_0 + \frac{1-\lambda}{2}(1-p_0)} &= \frac{\frac{1+\lambda}{2}(1-p_0)}{p_0 \frac{1+\lambda}{2} +\frac{1-\lambda}{2}}
		=\frac{1-p_0 - \frac{(1-\lambda)}{2}(1-p_0)}{p_0 \frac{1+\lambda}{2} +\frac{1-\lambda}{2}} < \frac{\frac{1-\lambda}{2}p_0 + (1-p_0)}{\frac{1+\lambda}{2}p_0}
	\end{align*}
	if $0<p_0<1$ and note that $\lambda<1$ by assumption. We have $a^*<0$ and thus $g_2(0)>g_1(0)$ (Bayes optimal decision for $x=0$) if
	\begin{align*}
    	1 > \frac{1-p_0}{p_0}\lambda,
	\end{align*}
   	and $b^*>0$ and thus $g_1(\epsilon)>g_2(\epsilon)$ (Bayes optimal decision for $x=\epsilon$) if
   	\begin{align*}
	    1 > \frac{p_0}{1-p_0}\lambda.
   	\end{align*}
    Thus we recover the Bayes classifier if
    \begin{align*}
    	\lambda < \min\Big\{\frac{1-p_0}{p_0},\frac{p_0}{1-p_0}\Big\}.
   	\end{align*}
\end{proof}


\subsection{Dynamic Time Warping} \label{dynamic_time_warping}
Dynamic Time Warping (DTW) is a widely used and robust distance measure for time series, \textit{allowing similar shapes
to match even if they are out of phase in the time axis} \cite{keogh2002exact}. The following explanation to calculate
the DTW distance is based on \cite{sart2010accelerating}.

Given are two time series $Q = (q_1,\allowbreak q_2,\allowbreak \dots,\allowbreak q_i,\allowbreak \dots,\allowbreak q_l)$
with length $l$, $C = (c_1,\allowbreak c_2,\allowbreak \dots,\allowbreak c_j,\allowbreak \dots,\allowbreak c_k)$ with
length $k$ over the domain set $\mathbb{U}$ and a distance measure function $d$ with
$d: \mathbb{U} \times \mathbb{U} \to \mathbb{R}$. Calculating the DTW distance between the two time series $Q$ and $C$
can be achieved by calculating a matrix $M$ of size $l \times k$ with the following rule.
\begin{equation} \label{eq:dtw}
    M_{i, j} = \begin{cases}
        d(q_i,c_j) & \text{if } i = 1 \wedge j = 1\\
        M_{i,j-1} + d(q_i,c_j) & \text{if } i = 1 \wedge j \neq 1\\
        M_{i-1,j} + d(q_i,c_j) & \text{if } i \neq 1 \wedge j = 1\\
        min(M_{i-1,j}, M_{i-1,j-1}, M_{i,j-1}) + d(q_i,c_j) & \text{if } i \neq 1 \wedge j \neq 1
    \end{cases}
\end{equation}
The DTW distance between the two time series $Q$ and $C$ is the entry $M_{l,k}$ of the resulting matrix.
\begin{equation}
    DTW(Q, C) = M_{l,k}
\end{equation}
The detection of the warping path as a result of the backtracking is irrelevant for the aim of this bachelor thesis.
Figure \ref{fig:dynamictimewarping} illustrates DTW for two time series $Q$ and $C$ that contain recorded data from one
acceleration sensor. DTW, shown as above, has time and space complexity of $\mathcal{O}(lk)$. When ignoring the
warping path as a result the algorithm can easily reduce the space complexity to $\mathcal{O}(min(l, k))$. This can be
achieved by keeping only the last important entries in space that are necessary to calculate the final entry $M_{l,k}$
of the matrix.

\begin{figure}
    \begin{center}
        \resizebox {\textwidth} {!} {
            \begin{tabular}{cc}
                \resizebox* {!} {0.3\textwidth} {
                    \begin{tikzpicture}
                        \begin{axis}[
                            xmin=0,
                            xmax=47,
                            xlabel=time,
                            ylabel=acceleration,
                            width=\axisdefaultwidth,
                            height=0.7*\axisdefaultheight,
                            reverse legend,
                            legend pos=south east]
                            \addplot[gray, quiver={u=\thisrow{u}, v=\thisrow{v}}] table {../data/fig/dynamictimewarping/path.dat};
                            \addplot[red, thick, mark=none] table {../data/fig/dynamictimewarping/q.dat};
                            \addlegendentry{Q}
                            \addplot[blue, thick, mark=none] table {../data/fig/dynamictimewarping/c.dat};
                            \addlegendentry{C}
                        \end{axis}
                    \end{tikzpicture}
                } & \quad
                \resizebox* {!} {0.3\textwidth} {
                    \begin{tabular}[b]{ll}
                        \begin{turn}{90}
                            \begin{tikzpicture}
                                \begin{axis}[
                                    xmin=0,
                                    xmax=47,
                                    ymin=-100,
                                    ymax=0,
                                    hide x axis,
                                    hide y axis,
                                    width=\axisdefaultwidth,
                                    height=0.7*\axisdefaultheight]
                                    \addplot[red, ultra thick, mark=none] table {../data/fig/dynamictimewarping/q.dat};
                                \end{axis}
                            \end{tikzpicture}
                        \end{turn} \hspace*{3em} &
                        \begin{tikzpicture}
                            \begin{axis}[
                                enlargelimits=false,
                                ymin=0,
                                ymax=47,
                                hide x axis,
                                hide y axis,
                                width=\axisdefaultwidth,
                                height=\axisdefaultwidth,
                                colorbar,
                                colormap/viridis high res]
                                \addplot[matrix plot*,
                                    mesh/cols=48,
                                    point meta=explicit] table[meta=C] {../data/fig/dynamictimewarping/matrix.dat};
                                \addplot[white, ultra thick, mark=*, mark size=1] table {../data/fig/dynamictimewarping/matrix_path.dat};
                            \end{axis}
                        \end{tikzpicture} \\
                        &
                        \\[1em]
                        &
                        \begin{tikzpicture}
                            \begin{axis}[
                                xmin=0,
                                xmax=47,
                                ymin=-100,
                                ymax=0,
                                hide x axis,
                                hide y axis,
                                width=\axisdefaultwidth,
                                height=0.7*\axisdefaultheight]
                                \addplot[blue, ultra thick, mark=none] table {../data/fig/dynamictimewarping/c.dat};
                            \end{axis}
                        \end{tikzpicture}
                    \end{tabular}
                }
            \end{tabular}
        }
    \end{center}
    \caption{Two time series $Q$ and $C$ containing recorded and compressed data from one acceleration sensor. On the
    left plot are both time series graphs, the gray lines are representing the warping path of plain DTW. The right plot
    shows the associated matrix containing the distances between the time series data points. Starting in the lower left
    corner and ending in the upper right corner, the warping path is illustrated as a white graph.}
    \label{fig:dynamictimewarping}
\end{figure}

\input{background_and_notation/dynamic_time_warping/sakoe-chiba_band.tex}
\input{background_and_notation/dynamic_time_warping/time_series_normalization.tex}


\subsection{Regression}

We can simply regress outcomes on variables, including treatment.

This assumes treatment effects are constant.

This also assumes that outcomes \(y_{1i}\) and \(y_{0i}\) are independent of \(D_i\), conditional on \(X\).

If we are missing variables in \(X\) then we will have biased estimates.

This also assumes the effects of \(X\) are linear.

We assume: \(E[y_{0i}|\mathbf x_{i}, D_i]=\mathbf x_i \theta\).


\section{The Model | Methodology} \label{sec:03}
Define the model, give some equations and explain them:
\begin{equation}
TSO(\tau_1, \tau_2) = \int_{\tau_1}^{\tau_2} \frac{WPL(s)}{C(s)}ds = \int_{\tau_1}^{\tau_2} U(s)ds.
\end{equation}

\subsection{Subsection}
Add aligned equations:
\begin{align}
U_t &= \Lambda_t + Y_t\\
d\Lambda_t &= \Lambda_t dt\\
Y_t &= {\bf b}^\top {\bf X}_t\\
d{\bf X}_t &= ({\bf A}{\bf X}_t +  {\bf e}_p\sigma_t\theta_t)dt +  {\bf e}_p \sigma_t dB_t^{\theta},
\end{align}
with matrices and vectors
\[{\bf A} = \left( \begin{array}{ccccc}
0 & 1 & 0 & \ldots& 0\\
0 & 0 & 1 & \ddots& \vdots\\
\vdots &  & \ddots & \ddots &0\\
0 & \ldots & \ldots & 0 & 1\\
-\alpha_p & -\alpha_{p-1} & \ldots & & -\alpha_1 \end{array} \right)\quad {\bf e}_p =  \left( \begin{array}{c}
0\\
0\\
\vdots\\
0\\
1
\end{array} \right) \]
Integrals:
\begin{align*}
U_s &=  \int_t^{s}\tilde{U_u}^{\eta_1}\exp({\bf A}(s-u)){\bf x}du +  \int_t^{s}\tilde{U_u}^{\eta_1}\exp({\bf A}(s-u))\Lambda_u du   \\
&+  \int_t^{s}\tilde{U_u}^{\eta_1}\exp({\bf A}(s-u)) {\bf e}_p \sigma_u dB_u^\theta
\end{align*}


\section{Segment coordinate transform}

For the following calculations we define the following points of the wing segment: $\vec p_1$ is the innermost point of the leading edge, $\vec p_2$ the outermost point of the leading edge, $\vec p_3$ the innermost point of the trailing edge, and $\vec p_4$ the outermost point of the trailing edge.

\begin{figure}[htb]
  \centering
  \includegraphics[width = 10cm]{gfx/bilinearSurface}
	\caption{Mathematically, a the chord face of a wing segment is a bilinear surface}
	\label{fig:bilin_surf}
\end{figure}

\subsection{Parametrization of the chord surface}

For simplicity, lets rename $\alpha := \eta$, $\beta := \xi$. Each point $\vec p$ on the surface then be expressed by

\begin{align}
\vec p(\alpha, \beta) &= \left( \vec {p_1} (1-\alpha) + \vec {p_2} \alpha \right) (1-\beta) \\
                  &+ \left( \vec {p_3} (1-\alpha) + \vec {p_4} \alpha \right) \beta \\
                  &= \alpha \underbrace{(-\vec p_1 + \vec p_2)}_{\vec a} + \beta\underbrace{(-\vec p_1 + \vec p_3)}_{\vec b} + \alpha \beta \underbrace{(\vec p_1 - \vec p_2 - \vec p_3 + \vec p_4)}_{\vec c} + \underbrace{\vec p_1}_{\vec d}
\label{eq:param}
\end{align}

This formula defines the transformation between the segments coordinates $(\alpha, \beta)$ to cartesian coordinates $(x, y, z)$. Mathematically, this form is called called bilinear surface. To sum up:

\begin{align}
\vec p (\alpha, \beta) &= \alpha   \vec a + \beta \vec b + \alpha \beta \vec c + \vec d \, , \quad
\textrm{with}\\
\vec a &= -\vec p_1 + \vec p_2 \nonumber \\
\vec b &= -\vec p_1 + \vec p_3 \nonumber \\
\vec c &= \vec p_1 - \vec p_2 - \vec p_3 + \vec p_4 \nonumber \\
\vec d &= \vec p_1 \nonumber
\end{align}

\subsection{Projecting a point onto the chord surface}
The projection of a point $\vec x$ onto the surface is defined as the point $\vec p(\alpha, \beta)$ so that $\vec p (\alpha, \beta) - \vec x$ is orthogonal to the surface. Mathematically it can be shown, that the orthogonality requirement is equivalent to finding the point $\vec p(\alpha, \beta)$ that has a smallest distance to $\vec x$. \par
This is of course an optimization problem which can be defined as:
\begin{equation}
\min_{\alpha, \beta} f(\alpha, \beta) := \min_{\alpha, \beta} \Vert \vec p(\alpha, \beta) - \vec x \Vert^2
\end{equation}
This is an almost quadratic problem (it is only quadratic, if $\vec c = 0$) and can be thus solved with Newton's optimization method. Newtons method is an iterative procedure. In our case, we can adapt it as follows:

\begin{algorithm}[htb]
 %\SetAlgoLined
 $(\alpha; \beta)_0 \leftarrow (0; 0)$ \\
 $k \leftarrow 0$\\
 \While{not converged}{
  $(\alpha, \beta)_{k+1} \leftarrow (\alpha, \beta)_{k} - s [\nabla^2 f(\alpha, \beta) ]^{-1} \cdot \vec \nabla f(\alpha, \beta),\quad \textrm{with}\, s \leq 1$\\
  $k \leftarrow k + 1$ \\
 }
 \caption{Newton's optimization algorithm}
\end{algorithm}

In order to perform this projection, we need the gradient $\vec \nabla f(\alpha, \beta)$ and the hessian matrix $\nabla^2 f(\alpha, \beta)$, which are defined as the first and second order derivative of $f(\alpha, \beta)$. \\

\paragraph{Gradient}
Using the chain rule, we get for the gradient:
\begin{equation}
\vec \nabla f(\alpha, \beta) = 2 J_p (\alpha, \beta)^T \cdot (\vec p(\alpha, \beta) - \vec x),
\end{equation}
where $J_p (\alpha, \beta)$ is the Jacobian of $\vec p(\alpha, \beta)$. In components this is:
\begin{align}
\frac {\partial f} {\partial \alpha}(\alpha, \beta) &= 2 \left(\frac{\partial \vec p}{\partial \alpha}(\alpha, \beta)\right)^T (\vec p(\alpha, \beta) - \vec x) \nonumber\\
&= 2\left( \vec a + \beta \vec c \right)^T (\vec p(\alpha, \beta) - \vec x)
\end{align}
and
\begin{align}
\frac {\partial f} {\partial \beta}(\alpha, \beta) &= 2 \left(\frac{\partial \vec p}{\partial \beta}(\alpha, \beta)\right)^T (\vec p(\alpha, \beta) - \vec x) \nonumber\\
&= 2\left( \vec b + \alpha \vec c \right)^T (\vec p(\alpha, \beta) - \vec x)
\end{align}

\paragraph{Hessian}
A derivative of the gradient gives as the Hessian matrix:
\begin{align}
\nabla^2 f(\alpha, \beta) = 2 J_p (\alpha, \beta)^T J_p (\alpha, \beta) + 2 (\nabla^2 \vec p (\alpha, \beta))^T \cdot (\vec p(\alpha, \beta) - \vec x)
\end{align}
Thus for the diagonal elements of the Hessian, we get
\begin{align}
\frac {\partial^2 f} {\partial \alpha^2}(\alpha, \beta) &= 2(\vec a + \beta \vec c)^T(\vec a + \beta \vec c) = 2 \Vert \vec a + \beta \vec c \Vert^2 \\
\frac {\partial^2 f} {\partial \beta^2}(\alpha, \beta) &= 2(\vec b + \alpha \vec c)^T(\vec b + \alpha \vec c) = 2 \Vert \vec b + \alpha \vec c \Vert^2
\end{align}
and for the off-diagonals
\begin{align}
\frac {\partial^2 f} {\partial \alpha \partial  \beta}(\alpha, \beta)
&= 2(\vec a + \beta \vec c)^T(\vec b + \alpha \vec c) + 2(\vec p (\alpha, \beta) - \vec x)^T \vec c.
\end{align}
Due to the symmetry of the Hessian, we get the same result for the other off-diagonal element $\frac {\partial^2 f} {\partial \beta \partial  \alpha}(\alpha, \beta)$.


  The basic examples relevant to us are:
  \begin{description}
    \item[0-dim: ]
      A 0-dimensional topological space is a collection of points.
    \item[1-dim: ]
      A 1-dimensional topological space is a graph - a collection of edges glued together at vertices.
    \item[2-dim: ]
      A 2-dimensional topological is a graph with faces.
  \end{description}
  All the maps between topological spaces that we'll consider will be continuous.
  A map $f:X \rightarrow Y$ between topological spaces is a \emph{homeomorphism} if it has a continuous inverse.

  \begin{ex}
    Examples of covering spaces.
    \todo{examples of covering spaces.}
  \end{ex}

  \begin{mdframed}
    Assume from now on that all our spaces are \emph{path-connected}.
  \end{mdframed}

  \subsection{Covering spaces}

  $X$ is called a \emph{cover} of $Y$.
  The set $\pi^{-1}(y)$ is called the fiber over $y$
  If $\cali$ is finite then $\pi$ is called a \emph{finite cover}.

  \begin{qbox}
    Show that every cover of a graph (resp. 2-graph) is a graph (resp. 2-graph).
  \end{qbox}

  \subsection{Group action}
  A left group action of $G$ on a space $X$, denoted $G \groupaction X$, is a collection of continuous maps
  \begin{align*}
    g \cdot - : X &\longrightarrow X\\
    x &\longmapsto gx
  \end{align*}
  satisfying $ex = x$, where $e$ is the identity element in $G$ and $g(hx) = (gh)x$ for all $g,h \in G$.
  \begin{definition}
    We say that $G \groupaction X$ is \emph{properly discontinous} if every point $x \in X$ has an open neighborhood $U$ such that
    \begin{equation*}
      g U \cap U = \varnothing
    \end{equation*}
    for all $g \in G$.
  \end{definition}
  \begin{qbox}
    Show that if $g U \cap U = \varnothing$ for all $g \in G$ then $g U \cap hU = \varnothing$ for all $g,h \in G$.
  \end{qbox}
  \begin{ex}
    add example of maps between $S^1$: rotation and reflection.
    \todo{add example of maps between $S^1$: rotation and reflection.}
  \end{ex}
  \begin{qbox}
    Show that if $G \groupaction X$ is free then $X \rightarrow G \backslash X$ is a cover.
  \end{qbox}


\documentclass[main.tex]{subfiles}
\begin{document}

\subsection{Exercise 8}

For clarity, we denote with Greek indices those ranging from 1 to \(N\), the size of the vector of data; and with Latin indices those ranging from 1 to \(M\), the number of templates.

We are assuming that the data have a Gaussian distribution with a covariance matrix \(C\), and we are modelling their mean \(\mu_\alpha  \) as a sum of templates \(t_{i \alpha}\) with coefficients \(A_i\):
%
\begin{align}
\mu _\alpha = t_{i \alpha } A_i
\,,
\end{align}
%
where the Einstein summation convention has been used.
Therefore, the likelihood is proportional to
%
\begin{align}
\mathscr{L}(d_\alpha | A_i) \propto \exp(- \frac{1}{2} \qty(d_\alpha - A_{i} t_{i \alpha }) C^{-1}_{\alpha \beta }
\qty(d_\beta - A_j t_{j \beta }))
\,.
\end{align}

The normalization only depends on the covariance matrix \(C_{\alpha \beta }\), which we assume is fixed.
Therefore, maximizing the likelihood\footnote{Which is equivalent to maximizing the posterior if we are using a flat prior.} is equivalent to minimizing the \(\chi^2\), which reads
%
\begin{align}
\chi^2 = \qty(d_\alpha - A_{i} t_{i \alpha }) C^{-1}_{\alpha \beta }
\qty(d_\beta - A_j t_{j \beta })
\,.
\end{align}

We want to minimize this as the amplitudes vary: therefore, we set the derivative with respect to \(A_k\) to zero,\footnote{The fact that the stationary point we will find is indeed a minimum can be checked by looking at the second derivative of \(\chi^2\):
%
\begin{align}
\pdv[2]{\chi ^2}{A_k}{A_m} = 2 t_{k \alpha } C_{\alpha \beta }^{-1} t_{m \beta }
\,,
\end{align}
%
and recalling that the inverse of the covariance matrix is positive definite.
}
%
\begin{align}
\pdv{\chi^2}{A_k} = -2 t_{k \alpha } C^{-1}_{\alpha \beta } \qty(d_\beta - A_j t_{j \beta }) = 0
\,,
\end{align}
%
which means that
%
\begin{align}
t_{k \alpha } C^{-1}_{\alpha \beta } d_\beta = (t_{k \alpha } C^{-1}_{\alpha \beta }  t_{j \beta }) A_j
\,,
\end{align}
%
a linear system of \(M\) equations (indexed by \(k\)) in the \(M\) variables \(A_j\).
If we denote the evaluations of bilinear forms in the data (\(N\)-dimensional) space with brackets, as \(a_\alpha C_{\alpha \beta } b_\beta \overset{\text{def}}{=} (a | C |b)\), this reads
%
\begin{align}
(t | C^{-1} | d)_k &= (t | C^{-1} | t)_{kj} A_j  \\
\qty[(t | C^{-1} | t)^{-1}]_{mk} (t | C^{-1} | d)_k  &=
\underbrace{ \qty[(t | C^{-1} | t)^{-1}]_{mk}
(t | C^{-1} | t)_{kj}}_{= \delta_{mj}} A_j = A_m  \\
A_m &= \qty[(t | C^{-1} | t)^{-1}]_{mk} (t | C^{-1} | d)_k
\,, \label{eq:template-fitting}
\end{align}
%
where the inverse of \((t | C^{-1} | t)\) is to be computed in the \(M\)-dimensional vector space.

\subsection{Exercise 9}

Our model for the mean value is in the form \(\mu (\Theta , A) = A \overline{x}(\Theta )\), where \(\overline{x}\) is a generic function of \(\Theta \), while \(A\) is our scale parameter.\footnote{This is not specified in the problem, but it seems natural to think that \(\abs{\overline{x}(\Theta )}\) is a constant for varying \(\Theta \). }
Our likelihood then reads
%
\begin{align}
\mathscr{L}(x | \Theta , A) = \underbrace{\frac{1}{(2\pi )^{N/2} \sqrt{\det C}}}_{B_1 }
\exp(- \frac{1}{2} (x - A \overline{x}(\Theta ))^{\top} C^{-1} (x- A \overline{x}(\Theta )))
\,.
\end{align}

If the priors for both \(A\) and \(\Theta \) are flat, this corresponds to the joint posterior \(P (\Theta , A | x)\).
We want to marginalize over \(A\), which amounts to integrating over it: dropping the dependence on \(\Theta \) of \(\overline{x}\) and defining \(V = C^{-1}\) we find
%
\begin{align}
P(\Theta | x)
&= B_1  \int \exp(- \frac{1}{2} (x - A \overline{x})^{\top} V (x- A \overline{x})) \dd{A}  \\
&= B_1  \int \exp(- \frac{1}{2} \qty(x^{\top} V x -2 A \overline{x}^{\top} V x + A^2 \overline{x}^{\top} V \overline{x})) \dd{A}
\marginnote{Used the symmetry of \(V\).}
\,.
\end{align}

The amplitude being negative makes little sense in a typical physical context, however the Gaussian integral can be done analytically only over the whole of \(\mathbb{R}\).

In order to get analytical results, here we will marginalize by integrating over negative amplitudes as well (\(A \in \mathbb{R}\)); the last figure (\ref{fig:marginalization}) will show how only integrating over positive amplitudes only would have looked (by numerical calculation) in a simple case.
In general if one wishes to perform the integral over \(A \in (0, + \infty )\) the tabulated values of the error function may be used.

Applying the formula for the single-variable Gaussian integral \eqref{eq:single-variable-gaussian-integral} (the bilinear forms are all evaluated to yield scalars, we are only integrating over the scalar \(A\)!) we then get
%
\begin{align}
P(\Theta | x) &= \underbrace{B_1  \exp(- \frac{1}{2} x^{\top} V x )}_{B_2 }
\exp( \frac{1}{2} \frac{(\overline{x}^{\top} V x)^2}{(\overline{x}^{\top} V \overline{x})})
\sqrt{ \frac{2 \pi }{\overline{x}^{\top} V \overline{x}}}  \\
&= B_2 \sqrt{\frac{2 \pi}{\overline{x}^{\top}V \overline{x}}}
\exp( \frac{1}{2} \frac{\overline{x}^{\top} \Omega \overline{x}}{\overline{x}^{\top}V \overline{x}})
\,,
\end{align}
%
where we defined the bilinear form \(\Omega = V x x^{\top} V^{\top}\).\footnote{With explicit indices, \(\Omega_{im} = V_{ij} x_j x_k V_{km}\).}

% We can observe that all information about the scale of \(\overline{x}\) has been lost: if we map \(\overline{x} \to A \overline{x}\) the argument of the exponential does not change, therefore the only change comes from the factor in front, and the posterior \(P\) is mapped to \(P / A\).

\subsubsection{An application of posterior marginalization in this fashion}

Let us consider a simple example of this as a sanity check: suppose that \(x\) is two-dimensional, and \(\overline{x}(\Theta ) = (\cos \Theta , \sin \Theta )^{\top}\); further, suppose that \(V\) is diagonal, so that
%
\begin{align}
V = \left[\begin{array}{cc}
\sigma_x^{-2} & 0 \\
0 & \sigma _y^{-2}
\end{array}\right]
\,.
\end{align}

Also, suppose that the observed data parameter is
%
\begin{align}
x = A_x \left[\begin{array}{c}
\cos \varphi  \\
\sin \varphi
\end{array}\right]
\,.
\end{align}

Then, the multiplicative constant in front of the marginalized posterior reads
%
\begin{align}
B_2 = B_1 \exp(- \frac{1}{2} A_x^2 \qty( \frac{\cos^2\varphi}{\sigma^2_x} + \frac{\sin^2\varphi}{\sigma^2_y}))
\,;
\end{align}
%
while the bilinear form \(\Omega \) is
%
\begin{align}
\Omega &= A_x^2
\left[\begin{array}{cc}
\sigma_x^{-2} & 0 \\
0 & \sigma _y^{-2}
\end{array}\right]
\left[\begin{array}{cc}
\cos^2 \varphi  & \cos \varphi \sin \varphi  \\
\cos \varphi \sin \varphi  & \sin^2 \varphi
\end{array}\right]
\left[\begin{array}{cc}
\sigma_x^{-2} & 0 \\
0 & \sigma _y^{-2}
\end{array}\right]  \\
&= A_x^2\left[\begin{array}{cc}
\cos^2 \varphi / \sigma_x^{4} & \cos \varphi \sin \varphi / \sigma_x^2 \sigma _y^2 \\
\cos \varphi \sin \varphi / \sigma_x^2 \sigma _y^2 & \sin^2 \varphi / \sigma_y^{4}
\end{array}\right]
\,.
\end{align}

Then, when we  evaluate the marginalized posterior we will find something in the form
%
\begin{align}
P(\Theta | x) &= B_1 \sqrt{2 \pi } \qty( \frac{\cos^2\Theta }{\sigma_x^2} + \frac{\sin^2 \Theta }{\sigma _y^2})^{-1/2}
\exp( A_x^2 F(\Theta , \varphi ))
\,,
\end{align}
%
where \(F (\Theta , \varphi )\) is some function whose specific form does not really matter.\footnote{For completeness, here is the full expression:
%
\begin{align}
\begin{split}
F(\Theta , \varphi ) &=
- \frac{1}{2} \qty( \frac{\cos^2\varphi}{\sigma^2_x} + \frac{\sin^2\varphi}{\sigma^2_y})+  \\
&\phantom{=}\
+
\qty( \frac{\cos^2 \Theta}{\sigma _x^2} + \frac{\sin^2 \Theta }{\sigma _y^2})^{-1}
\qty[
    \frac{\cos^2 \Theta \cos^2 \varphi }{\sigma _x^{4}}
    +2\frac{\cos \Theta \sin \Theta \cos \varphi \sin \varphi  }{\sigma _x^{2} \sigma _y^{2}}
    +\frac{\sin^2 \Theta \sin^2 \varphi }{\sigma _y^{4}}
]
\,.
\end{split}
\end{align}
%
}

The amplitude of the observed data vector, \(A_x\), appears in a rather simple way, as a multiplicative prefactor in the exponent: it can affect the shape of the distribution, but not its mean.
Specifically, we can see that scaling \(A_x\) is equivalent to scaling \(\sigma _x\) and \(\sigma _y\) simultaneously in the opposite direction --- this is rather intuitive, since the angular size of the distribution as seen from the origin is smaller if it is further away.

% Therefore, we see that by marginalizing over \(A\) we have ``forgotten'' any scaling information about \(x\).

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{figures/marginalization.pdf}
\caption{Marginalization: the left plot shows the full likelihood in terms of \(A\) and \(\Theta \); the middle plot shows the result of marginalization as shown in the previous calculation (the posterior as a function of \(\Theta \)); the right plot shows the result of the more physically meaningful marginalization over \(A \in (0, + \infty )\) only.
Here the likelihood is a diagonal Gaussian with \(\sigma _x = \num{1.2}\) and \(\sigma _y = \num{1.8}\), centered in \(A_x = \num{2.5}\) and \(\varphi = \SI{1}{rad}\).}
\label{fig:marginalization}
\end{figure}

\subsubsection{Likelihood marginalization}

So far we have considered the posterior \(P(\Theta | x)\), the marginalized posterior, a function of the parameter(s) \(\Theta \); however we may also be interested in the marginalized likelihood \(\mathscr{L}(x | \Theta )\), whose expression is the same as the one we found for \(P(\Theta | x)\).
Let us write it in a way which makes the dependence on \(x\) more explicit:
%
\begin{align} \label{eq:marginalized-likelihood}
\mathscr{L}(x | \Theta ) = \underbrace{B_1 \sqrt{\frac{2 \pi }{\overline{x}^{\top} V \overline{x}}}}_{B_3 } \exp(- \frac{1}{2} x^{\top} V x +
\frac{1}{2}
\frac{(\overline{x}^{\top} V x)^2}{\overline{x}^{\top} V \overline{x}})
\,,
\end{align}
%
which can be simplified by making use of the fact that the best-fit template amplitude we found in the last exercise (equation \eqref{eq:template-fitting}) can be applied here, with the single template \(t = \overline{x}\), the single amplitude \(A\), the data \(d = x\), and the inverse covariance matrix \(C^{-1}= V\): the fitting value for \(A \) is
%
\begin{align}
\hat{A} = \frac{\overline{x}^{\top} V x}{\overline{x}^{\top} V \overline{x}}
\,;
\end{align}
%
therefore the likelihood is
%
\begin{align}
\mathscr{L}(x | \Theta ) = B_3 \exp(- \frac{1}{2} x^{\top} V x +
\frac{1}{2}
 \hat{A} \overline{x}^{\top} V x)
\,.
\end{align}

This can be rewritten in the canonical MVN form by making use of the matrix square completion formula \eqref{eq:square-completion}, with \(A = -V\) and \(\vec{b}^{\top} = \hat{A} \overline{x}^{\top} V\):
%
\begin{align}
\begin{split}
- \frac{1}{2} x^{\top} V x + \frac{1}{2} \hat{A} \overline{x}^{\top} V x
&= - \frac{1}{2}
\qty(x - \frac{1}{2} V^{-1} \hat{A} (\overline{x}^{\top} V)^{\top})^{\top} V
\qty(x - \frac{1}{2} V^{-1} \hat{A} (\overline{x}^{\top} V)^{\top}) \\
&\phantom{=}\
+ \frac{1}{8} \hat{A}^2 (\overline{x}^{\top} V) V^{-1} (\overline{x}^{\top} V)^{\top}
\end{split}\\
&= - \frac{1}{2}
\qty(x - \frac{1}{2} \hat{A} \overline{x})^{\top}
V
\qty(x - \frac{1}{2} \hat{A} \overline{x})
+
\frac{1}{8} \hat{A}^2 \overline{x}^{\top} V \overline{x}
\,.
\end{align}

Therefore, the marginalized likelihood reads
%
\begin{align}
\mathscr{L}(x | \Theta ) = B_3 \exp( \frac{1}{8} \hat{A}^2 \overline{x}^{\top} V \overline{x}) \exp(- \frac{1}{2} \qty(x - \frac{1}{2} \hat{A} \overline{x})^{\top} V \qty( \frac{1}{2} x - \hat{A} \overline{x}))
\,.
\end{align}

We must be careful with this expression: it looks like a multivariate normal in \(\overline{x}\), however \(\hat{A}\) is not in general independent of it.
If neither the estimate for \(\hat{A}\) nor the expression \(\overline{x}^{\top} V \overline{x}\) vary significantly in the range of \(\overline{x}\) we are interested in, then we can consider this expression for the likelihood as Gaussian in \(x\), or as a marginalized posterior which is Gaussian in \(\overline{x}\).

% A clearer way to see that this is indeed still a MVN is to come back to the original expression \eqref{eq:marginalized-likelihood}, and to write it as
% %
% \begin{align}
% \mathscr{L} (x | \Theta ) = B_3 \exp(- \frac{1}{2} x^{\top} \qty(V - \frac{V \overline{x} \overline{x}^{\top} V}{\overline{x}^{\top} V \overline{x}}) x )
% \,,
% \end{align}
% %
% thus showing that the likelihood is a \emph{zero-mean} MVN with covariance given by
% %
% \begin{align}
% \qty[V - \frac{V \overline{x} \overline{x}^{\top} V}{\overline{x}^{\top} V \overline{x}}]^{-1}
% \,.
% \end{align}

\end{document}


\documentclass[main.tex]{subfiles}
\begin{document}

We set $c=1$.

Here we will often use (anti)symmetrization of indices, which makes some calculations much easier. The idea of symmetrization is to sum over all permutation of the selected indices, with a minus sign for the odd permutation if the case of anti symmetrization. So, for instance, \(F_{\mu \nu }\) can be antisymmetrized into \(F_{[\mu \nu ]} = \frac[i]{1}{2} \qty(F_{\mu \nu }- F_{\nu \mu })\) and symmetrized into \(F_{(\mu \nu )}= \frac[i]{1}{2} \qty(F_{\mu \nu } + F_{\nu \mu })\).

The factor \(\frac[i]{1}{2} \) is in general \(1/n!\), where \(n\) is the number of antisymmetrized indices. This is included because in general we will be summing \(n!\) terms, and we want to write things like: ``\(F_{\mu \nu }\) is antisymmetric means \(F_{\mu \nu} = F_{[\mu \nu ]}\)'', so we need to rescale the sum to make it into an average.

The general formulas are then:
%
\begin{subequations}
\begin{align}
  F_{[\mu_{1} \dots \mu_{n}]} &= \frac{1}{n!} \sum _{\sigma  \in \mathfrak{S}_n} \sign{\sigma} F_{\sigma (\mu_1)\dots \sigma(\mu_n)} \\
  F_{(\mu_{1} \dots \mu_{n})} &= \frac{1}{n!} \sum _{\sigma  \in \mathfrak{S}_n}F_{\sigma (\mu_1)\dots \sigma(\mu_n)}
\,,
\end{align}
\end{subequations}
%
where \(\mathfrak{S}_n\) is the \emph{symmetric group} of permutations of \(n\) elements, and the sign of a permutation \(\sigma \in \mathfrak S_n\) is \(\pm 1\), depending on the parity of pair swaps that are needed to get that configuration (we fix (\(\sign(\mathbb{1}) = 1\))).

If we want to symmetrize indices which are not next to each other, we will denote the end of the (anti)symmetrized indices by a vertical bar.

A useful mnemonic for the Riemann tensor: we'd like to write the formula \(R = \partial \Gamma  + \Gamma \Gamma \) keeping the indices in the same order on either side: the way to do it is this:
%
\begin{align}
  R^{\mu }_{\alpha \beta \gamma } = -2 \qty(
    \Gamma^{\mu }_{\alpha [\beta , \gamma ]}
    + \Gamma^{\sigma }_{\alpha [\beta } \Gamma^{\mu }_{\gamma ] \sigma }
  )
\,,
\end{align}
%
which is much easier to remember: one writes down the indices in the same order, adds a dummy index in the second Christoffel symbol on the last term, which must be contracted with the upper index on the other symbol.

Then, since the Riemann tensor is antisymmetric in the last two indices, we antisymmetrize in those.

Another useful relation is given by
%
\begin{align}
  R_{\mu \nu } = \partial_{\gamma } \Gamma^{\gamma }_{\mu \nu } - \Gamma^{\alpha }_{\mu \beta } \Gamma^{\beta }_{\nu \alpha } - \nabla_{\mu } \qty(\partial_{\nu } \log \sqrt{ \abs{g}})
\,,
\end{align}
%
where \(g\) is the determinant of the metric.

\section{Sheet 1}

\subsection{Lorentz transformations }

\subsubsection{Inverses}

We can consider a Lorentz boost with velocity $v$ in the $x$ direction, and we look at its representation in the $(t, x)$ plane (since the $y$ and $z$ directions are unchanged). Its matrix expression looks like:
%
\begin{equation}\label{LorBoost}
    \Lambda = \begin{bmatrix}
        \gamma & -v \gamma \\
        -v \gamma & \gamma
    \end{bmatrix}\,,
\end{equation}
%
where $\gamma = 1 / \sqrt{1 - v^2}$. The inverse of this matrix can be computed using the general formula for a 2x2 matrix:

\begin{equation}
    A^{-1}=
    \begin{bmatrix}
        a & b \\
        c & d
    \end{bmatrix}^{-1}
    =
    \frac{1}{\det(A)}
        \begin{bmatrix}
        d & -b \\
        -c & a
    \end{bmatrix}\,.
\end{equation}

The determinant of $\Lambda $ is equal to $\gamma^2 (1-v^2) = 1$, therefore the inverse matrix is:
%
\begin{equation}
    \Lambda = \begin{bmatrix}
        \gamma & v \gamma \\
        v \gamma & \gamma
    \end{bmatrix}\,.
\end{equation}

\subsubsection{Invariance of the spacetime interval}

Our Lorentz transformation is

\begin{subequations}
\begin{align}
    \dd{t}' &= \gamma (\dd{t} - v \dd{x}) \\
    \dd{x}' &= \gamma (-v\dd{t} + \dd{x}) \\
    \dd{y}' &= \dd{y} \\
    \dd{z}' &= \dd{z}
\end{align}
\end{subequations}
%
and we wish to prove that the spacetime interval, defined by $\dd{s^2} = \eta_{\mu\nu} \dd{x^\mu}\dd{x^\nu}$ is preserved: $\dd{s'\,^2} = \dd{s^2}$.
Let us write the claimed equality explicitly:
%
\begin{subequations}
\begin{align}
    -\dd{t^2} + \dd{x^2}+ \dd{y^2}+ \dd{z^2}
    &= \gamma (\dd{t} - v \dd{x})
\end{align}
\end{subequations}

\subsubsection{Tensor notation pseudo-orthogonality}

The invariance of the spacetime interval $\dd{s'\,^2} = \dd{s^2}$ can be also written as \(\eta_{\mu\nu} \dd{x^\mu} \dd{x^\nu} = \eta_{\mu\nu} \dd{x'\,^\mu} \dd{x'\,^\nu}\). By making the primed differentials explicit we have:
%
\begin{equation}
  \eta_{\mu\nu} \dd{x^\mu} \dd{x^\nu}
  =
  \eta_{\mu\nu} \tensor{\Lambda}{^\mu_\rho} \dd{x^\rho} \tensor{\Lambda}{^\nu_\sigma} \dd{x^\sigma} \,,
\end{equation}
%
but the dummy indices on the LHS can be changed to \(\rho\) and \(\sigma\), so that both sides are proportional to  \(\dd{x^\rho}\dd{x^\sigma}\). Doing this we get:
%
\begin{equation}
  \eta_{\rho\sigma}
  =
  \eta_{\mu\nu} \tensor{\Lambda}{^\mu_\rho} \tensor{\Lambda}{^\nu_\sigma}
  =\tensor{(\Lambda^\top)}{_\rho^\mu} \eta_{\mu\nu} \tensor{\Lambda}{^\nu_\sigma} \,,
\end{equation}
%
or, in matrix form, \(\eta = \Lambda^\top \eta \Lambda\).

% \subsubsection{Pseudo orthogonality}
% Defining $dx^\mu=(cdt,dx,dy,dz)^T$, and $dx_\mu=(cdt,dx,dy,dz)=\eta_{\mu\nu}dx^\nu$, we will have
\begin{subequations}
% \begin{align}
% &ds^2=dx^\mu dx_\mu=dx^\mu\eta_{\mu\nu}dx^\nu\nonumber\\
% &\dd{s,^{2\prime}}=dx^{\mu\prime} dx_\mu'=dx^{\mu\prime}\eta_{\mu\nu}dx^{\nu\prime}=dx^\rho\Lambda_\rho^\mu\eta_{\mu\nu}dx^\sigma\Lambda_\sigma^\nu
% \end{align} and $\dd{s'\,^2} = \dd{s^2}$ is equivalent to our thesis.
\end{subequations}

% \subsubsection{Pseudo orthogonality by matrix product}

% Let $\Lambda$ be the Lorentz transformation in \eqref{LorBoost}.bIn this case we have
\begin{subequations}
% \begin{align}
%     \Lambda ^T\eta\Lambda=& \begin{bmatrix}
%         \gamma & -v \gamma & 0 & 0\\
%         -v \gamma & \gamma & 0 & 0\\
%         0 & 0 & 1 & 0\\
%         0 & 0 & 0 & 1\\
%     \end{bmatrix}\cdot \begin{bmatrix}
%         -1 & 0 & 0 & 0\\
%         0 & 1 & 0 & 0\\
%         0 & 0 & 1 & 0\\
%         0 & 0 & 0 & 1\\
%     \end{bmatrix}\cdot \begin{bmatrix}
%         \gamma & -v \gamma & 0 & 0\\
%         -v \gamma & \gamma & 0 & 0\\
%         0 & 0 & 1 & 0\\
%         0 & 0 & 0 & 1\\
%     \end{bmatrix}=\nonumber\\
%      =&\begin{bmatrix}
%         -\gamma & -v \gamma & 0 & 0\\
%         v \gamma & \gamma & 0 & 0\\
%         0 & 0 & 1 & 0\\
%         0 & 0 & 0 & 1\\
%     \end{bmatrix}\cdot
%     \begin{bmatrix}
%         \gamma & -v \gamma & 0 & 0\\
%         -v \gamma & \gamma & 0 & 0\\
%         0 & 0 & 1 & 0\\
%         0 & 0 & 0 & 1\\
%     \end{bmatrix}=\nonumber\\
%     =& \begin{bmatrix}
%         -\gamma^2+v^2\gamma^2 & 0 & 0 & 0\\
%         0 & -v^2\gamma^2+\gamma^2 & 0 & 0\\
%         0 & 0 & 1 & 0\\
%         0 & 0 & 0 & 1\\
%     \end{bmatrix}=\begin{bmatrix}
%         -1 & 0 & 0 & 0\\
%         0 & 1 & 0 & 0\\
%         0 & 0 & 1 & 0\\
%         0 & 0 & 0 & 1\\
%     \end{bmatrix}=\eta
% \end{align}
\end{subequations}
% Where we used the fact that $\gamma^2(1-v^2)=\frac{\gamma^2}{\gamma^2}=1$.

\subsubsection{Explicit pseudo-orthogonality}

For simplicity but WLOG we consider a boost in the \(x\) direction with velocity \(v\) and Lorentz factor \(\gamma\). The matrix expression to verify is:
%
\begin{subequations}
\begin{align}
  \begin{bmatrix}
  \gamma    & - v \gamma  \\
    -v \gamma & \gamma
  \end{bmatrix}
  \begin{bmatrix}
    -1 & 0 \\
    0 & 1
  \end{bmatrix}
  \begin{bmatrix}
    \gamma & -v \gamma \\
     -v \gamma&  \gamma
  \end{bmatrix}
  &\overset{?}{=}
  \begin{bmatrix}
  -1   & 0 \\
  0   & 1
\end{bmatrix} \\
  \begin{bmatrix}
  \gamma    & - v \gamma  \\
    -v \gamma & \gamma
  \end{bmatrix}
  \begin{bmatrix}
    -\gamma & v \gamma \\
    -v \gamma&  \gamma
  \end{bmatrix}
  &\overset{?}{=}
  \begin{bmatrix}
  -1   & 0 \\
  0   & 1
\end{bmatrix} \\
  \begin{bmatrix}
  -\gamma^2 + \gamma^2 v^2    & v \gamma^2 - v \gamma^2  \\
    v \gamma^2 -v \gamma^2 & -v\gamma^2 +\gamma^2
  \end{bmatrix}
  &=
  \begin{bmatrix}
  -1   & 0 \\
  0   & 1
\end{bmatrix}\,,
\end{align}
\end{subequations}
%
which by \(\gamma^2 = 1/ (1-v^2)\) confirms the validity of the expression.

\subsection{Muons}

\subsubsection{Nonrelativistic approximation}

The survival probability is given by \(\mathbb{P} (t) = \exp(-t/ \SI{2.2e-6}{s})\). If the ground is \( h =\SI{15}{km} \) away, then the muon will reach it in \(t = h/v = \SI{15}{km} / (0.995c) \approx \SI{5.03e-05}{s}\), therefore \(\mathbb{P}(t) \approx \num{1.2e-10} \).

\subsubsection{Relativistic effects: ground perspective}

The observer on the ground will see the muon having to traverse the whole \(h = \SI{15}{km} \), but the muon's time will be dilated for them by a factor \(\gamma_v \approx 10\): therefore the survival probability
will be \(\mathbb{P}(t) = \exp(- t / (\gamma_v \times \SI{2.2e-6}{s})) \approx 0.1\).

\subsubsection{Relativistic effects: muons perspective}

The muons in their system will observe length contraction, which can be calculated by applying a Lorentz boost, by a factor \(\gamma_v \approx 10\): therefore the survival probability
will be \(\mathbb{P}(t) = \exp(- t / (\gamma_v \times \SI{2.2e-6}{s})) \approx 0.1\).
This result is the same of the one predicted by the ground observer: the relativity principle is respected.

\subsection{Radiation}

\subsubsection{New angle}
In the source frame the radiation velocity components are \( u_x' = \cos\theta', u_y' = \sin\theta' \). From the composition of velocities we obtain:

\begin{subequations}
\begin{align}
    u_y = \sin\theta &= \frac{\dd{y}}{\dd{t}} = \frac{\dd{y'}}{\gamma_v(\dd{t'} + v\dd{x'})} = \frac{\sin\theta'}{\gamma_v(1+v\cos\theta')} \\
    u_x = \cos\theta &= \frac{\dd{x}}{\dd{t}} = \frac{\gamma_v(\dd{x'} + v\dd{t'})}{\gamma_v(\dd{t'} + v\dd{x'})} = \frac{\cos\theta' + v}{1+v\cos\theta'}\,,
\end{align}
\end{subequations}
%
hence:
%
\begin{equation}
\frac{1}{\tan\theta} = \frac{\gamma_v}{\tan\theta'} + \frac{\gamma_vv}{\sin\theta'} \,.
\end{equation}

\subsubsection{Angle plot and relevant limits}
See the jupyter notebook in the \texttt{python} folder for plots.
For $v=0$ we have $\theta=\theta'$ as we expected, while for $v=1$, $\theta = 0$.

\subsubsection{Radiation speed invariance}

Are the components of the velocity, which we called \(\sin \theta\) and \(\cos \theta \), actually normalized? Let us check:

\begin{subequations}
\begin{align}
    \sin^2\theta + \cos^2\theta &= \frac{(\frac{\sin\theta'}{\gamma_v})^2 + (\cos\theta' + v)^2}{(1 + v\cos\theta')^2} \\
    &= \frac{(1-v^2)\sin^2\theta' + \cos^2\theta' + v^2 + 2v\cos\theta'}{(1 + v\cos\theta')^2} \\
    &= \frac{1+v^2(1-\sin\theta')+2v\cos\theta'}{(1 + v\cos\theta')^2} = 1 \,,
\end{align}
\end{subequations}
%
therefore the square modulus of the speed of the radiation is still \(c\), as we could have assumed earlier.

\subsubsection{Isotropic emission}

Since the angular distribution of emission varies when changing inertial reference, we might suppose that every system in relative motion respect to $O$ with $v\neq 0$ observes nonisotropic emission.

This can be seen by noticing that for $v\simeq 1$ we have that in the observer system there is almost only emission at an angle $\theta = 0$.
In general, since there is a Lorentz \(\gamma\) factor multiplying a function of the angle in the radiation emission frame \(O'\), the cotangent of the angle in the observation frame \(O\) must get larger and larger as the relative velocity \(v\) increases, therefore the radiation gets compressed towards angles with large cotangents: \(\theta\sim 0\).

See the jupyter notebook in the \texttt{python} folder for interactive plots :)

\end{document}


\chapter{Position, Speed, and Acceleration}

Let's say that you get in a car and drive 45 km/hour for two hours in
one direction. Your position will be $2 \times 45 = 90$ km from where
you started. We could plot your velocity and your position during those two hours:

\includegraphics[width=0.7\textwidth]{speed_simple.pdf}

\includegraphics[width=0.7\textwidth]{position_simple.pdf}

Notice that the area under the velocity line is $2 \times 45$, the total
distance traveled. In fact, at any point in your drive, your distance
from the starting point is equal to the area under the velocity line.
% Diagram needed, explain why it is the area under the curve
% KA: https://www.khanacademy.org/math/ap-calculus-ab/ab-diff-contextual-applications-new/ab-4-2/e/interpret-motion-graphs

What if you:
\begin{itemize}
\item drive 50 km/hour for 45 minutes
\item rest for 15 minutes
\item continue driving 90 km/hour for 45 minutes
\item turn around and head back at 120 km/hour for 15 minutes
\end{itemize}
% KA: https://www.khanacademy.org/math/ap-calculus-ab/ab-diff-contextual-applications-new/ab-4-2/v/one-dimensional-motion-with-calculus

Now the graphs look a little more complex:

\includegraphics[width=0.7\textwidth]{speed_stops.pdf}

\includegraphics[width=0.7\textwidth]{position_stops.pdf}

It is still true that at any time, the position is equal to the
area under the graph up to that time. Notice that when the velocity goes
negative, it is subtracting from the position( negative velocity means going backwards).

Notice that when you are going faster, the slope of the position line
is steeper. When the slope of the position is negative, the velocity is
negative( the velocity is decreasing).

So we end up with two important ideas:
\begin{itemize}
\item If we plot the velocity, the position is equal to the area under the speed line.
\item If we plot the position, the velocity is equal to the slope of the position line.
\end{itemize}

\section{Acceleration}

Just as velocity represents the change in position over time,
acceleration represents the change in velocity over time.

What if, from a standstill, you accelerate smoothly 45 km/hour each
hour for one hour and 30 minutes. Then you stop accelerating for 15
minutes. Then you decelerate (or accelerate toward home) at a rate of
80 km/hour each hour for 15 minutes. Now we have three graphs: the
acceleration, the velocity, and the position

\includegraphics[width=0.7\textwidth]{acceleration_asp.pdf}

\includegraphics[width=0.7\textwidth]{speed_asp.pdf}

\includegraphics[width=0.7\textwidth]{position_asp.pdf}

So we end up with two important ideas:
\begin{itemize}
\item If we plot the acceleration, the velocity is equal to the area under the acceleration line.
\item If we plot the velocity, the acceleration is equal to the slope of the velocity line.
\end{itemize}

\section{Differentiation and Integration}

In calculus, we talk a lot about differentiation and integration.

\textit{Differentiation} is just finding the slope of a curve. If you give me
the position of an object over time, I will differentiate that to find
its velocity at any time. Once I have the velocity, I can differentiate
again to find its acceleration.

\textit{Integration} is just finding the area under a curve. If you give me
the initial speed of an object and its acceleration over time, I will
integrate that to find its velocity at any time. Then, if I know the
starting point of the object, I can integrate the velocity to find its
position at any time.

\section{Speed vs. Velocity, Distance vs. Position}

In casual conversation, we tend to use the words ``speed'' and
``velocity'' interchangeably. When we are solving problems, velocity
represents the change in position, and that can be pretty specific,
like ``It's velocity is 12 m/s due west.'' Velocity, then, can
have a direction and can be negative.

Speed, is just the magnitude of the velocity, like ``12 m/s''.  It is
a number that is never negative.
% ADD: Scalar vs Vector
% KA: https://www.khanacademy.org/math/ap-calculus-ab/ab-diff-contextual-applications-new/ab-4-2/v/one-dimensional-motion-with-calculus

Similarly, a position can be quite complex like ``12 km east of
Lexington.'' Distance is just a number, like ``12 km from Lexington.''
Distance is never negative.

% ADD: Distance vs Displacement
% KA: https://www.khanacademy.org/science/high-school-physics/one-dimensional-motion-2/distance-displacement-and-coordinate-systems-2/a/relative-motion-review-article

% ADD: Example interpreting all three graphs
% Video: https://www.google.com/url?sa=i&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DnUb7xfkc0Ac&psig=AOvVaw2cfsXUgDpoYlIRVKcfIMJP&ust=1653186487204000&source=images&cd=vfe&ved=0CAwQjRxqFwoTCPCdxtzF7_cCFQAAAAAdAAAAABAD


% -----------------------------------------------------------------------------------
% Section   :
% -----------------------------------------------------------------------------------

\def\H{\mathcal{H}}
\def\Z{\pmb{Z}}
\def\X{\pmb{X}}
\def\ci{c_{i,\H_i(x)}}

\section*{Proof}

Intuitively, the the use of different hash functions ensures that
actually the chance of getting an estimate that is far from some average is small. Now we make this precise. Since for every $x$ the value $\ci \ge f_x$ we have:
	\[
		f_x \le \widetilde{f_x} = min(c_{1,\H_1(x),...c_{d,\H_d(x)}})
	\]

 We show that if $w = \frac{2}{\eps}$ and $d = \log_2 \frac{1}{\delta}$ then:
	\[
		\Pr { f_x \le \widetilde{f_x} \le f_x + \eps m } \ge 1 - \delta
	\]

Let's define random variables $\Z_1,\ldots,\Z_d$ so that $\ci = f_x + \Z_i$, where
	\[
		\Z_i = \sum_{y \ne x, \H_i(y) = \H_i(x)} f_y,
	\]

define $\pmb{X}_{i,h} = 1$ if $\H_i(y) = \H_i(x)$ and $0$ otherwise:

	\[
		\Z_i = \sum_{y \ne x} f_y \pmb{X}_{i,y},
	\]

Since the $\H$'s are pairwise independent, we have:
	\[
		\E{\Z_i} = \sum_{y\ne x} f_y \E{ X_{i,y}} = \sum_{y\ne x} Pr{ \H_i(x) = \H_i(x)} \le \frac{m}{w},
	\]

by Markov we have:
	\[
		\Pr{ \Z_i \ge \eps m} \le \frac{1}{w \eps} = \frac{1}{2}..
	\]

Since all $Z_i$ are independent, we know:

	\[
		\Pr{Z_i \ge \eps m \quad \forall \quad 1 \le i \le d} \le \bigg(\frac{1}{d}\bigg)^d = \delta.
	\]

Now with probability $1-\delta$ there is some $j$ so that $Z_j \le \eps m$:

	\[
		\widetilde{f_x} = min (f_x + \Z_1, ..., f_x + Z_d) \le f_x + \eps m.
	\]

Hence CountMin esitmates $x$ up to $+/- \eps m$ with space $O \bigg(  \frac{log m log \frac{1}{\delta}}{\eps} \bigg)$.


\chapter{Diffusion}
\emph{Stochastic everything.}
\newpage

\section{Describing a Diffusion}
    \subsection{An Aside on Calculus}
        Let's forget all the stochastic process stuff for a moment. Consider a smooth function $F(x):\mathbb{R} \mapsto \mathbb{R}$. Let $f(x) = F'(x)$.

        \[ F(x) = F(0) + \int_0^x f(t) \mathrm{d}t \]

        So any smooth function $F$ can be uniquely determined by an initial condition and its derivative $f$.

        \[ f(x) = \lim_{h \to 0} \frac{F(x+h)-F(x)}{h} \]

        That is, we assume the function can be approximated by a linear function near $x$. This gives us some food for thought.

    \subsection{Diffusion}
        Similar to the derivative in calculus, we assume an arbitrary stochastic process can be approximated by a Brownian motion near $x+h$ as $h \to 0$.
        \[ x_h = x_0 + \mu(x_0)h + \sigma(x_0)\sqrt{h}z_1 \quad z_1 \sim \mathcal{N}(0,1) \]

        As is in calculus, we define two functions $\mu(t,x)$ and $\sigma^2(t,x)$,
        \[ \mu(t,x) = \lim_{h \to 0}\frac{1}{h}\mathbb{E}[X(t+h)-X(t)|X(t)=x] \]
        \[ \sigma^2(t,x) = \lim_{h \to 0}\frac{1}{h}\var[X(t+h)-X(t)|X(t)=x] \]

        A diffusion is \textbf{Time-homogeneous} if $\mu(t,x) = \mu(x)$, $\sigma(t,x) = \sigma(x)$. From now on we only consider time-homogeneous diffusions.

        Once $\mu$ and $\sigma$ are specified,
        \begin{equation}\label{eq:DiffusionDE} \mathrm{d}X_t = \mu(t,X_t)\mathrm{d}t + \sigma(t,X_t)\mathrm{d}W_t \end{equation}
        where intuitively $W_t$ is the ``movement of a Brownian motion in a very small interval of time''. But since a Brownian motion is not differentiable, defining $W_t$ is non-trivial. However, if we take the integral of Equation \ref{eq:DiffusionDE}.
        \[ X_t - X_0 = \int_0^t \mu(X_s)\mathrm{d}s + \int_0^t \sigma(X_s)\mathrm{d}W_s \]
        where
        \[ \int_0^t \sigma(X_s)\mathrm{d}s \]
        is called the \textbf{Ito's Integral}. Due to its complexity we will not give any further details on Ito's Integral, and we will not operate on it in the following parts.

\section{Langevin Dynamics}
    \emph{“大家听说过Langevin吗？肯定听说过。郎之万。--Chihao.”}
    \subsection{Recap: Gradient Descent}
        Recall the update rule of Gradient Descent
        \[ X_t = X_{t-1} - \eta\nabla f(X_{t-1}) \Longrightarrow X_t - X_{t-1} = -\nabla f(X_{t-1})\]

        In the continuous case,
        \[ \mathrm{d}X_t = -\nabla f(X_t)\mathrm{d}t \]

    \subsection{Langevin Dynamics}
        A \textbf{Langevin Dynamics} is a continuous version of gradient descent, with an additional Gaussian noise\footnote{“下降完了它还要抖一下--Chihao”}
        \[ \mathrm{d}X_t = -\nabla f(X_t)\mathrm{d}t + \sqrt{2}\mathrm{d}W_t \]

        \begin{theorem}[Convergence of Langevin Dynamics]\label{thm:ConvergenceOfLangevinDynamics}
            For any function $f$, $X_t \sim p(X)$, where $p(X)$ is given by
            \[ p(x) = \frac{e^{-f(x)}}{\sum_x e^{-f(x)}} \]
        \end{theorem}

        The following sections will be devoted to the convergence analysis. The proofs will be given in 1D cases $X_t \in \mathbb{R}$, but the results apply to $\mathbb{R}^n$. And we further asuume $\sigma(X_t) \in \mathbb{R}$.

    \subsection{Fokker-Planck Equation}
        Fokker-Planck Equation is the continuous version of Kolmogorov Forward Equation \ref{eq:KolmogorovForwardEquation}.

        Let $p(t,y)$ denote the density of $X_t = y$.

        \begin{equation}\label{eq:Fokker-PlanckEquation}
            \frac{\partial p(t,y)}{\partial t} = - \sum_{i=1}^n \frac{\partial \mu_i(y)p(t,y)}{\partial y_i} + \frac{1}{2}\sigma^2(y)\sum_{i=1}^n\frac{\partial^2 p(t,y)}{\partial y_i^2}
        \end{equation}
        Where $y_i$ and $\mu_i(y)$ denote the $i$-th element of vectory $y$ and $\mu(y)$ respectively.

        To prove \ref{eq:Fokker-PlanckEquation}, we first introduce the backward probability. Fix $y \in \mathbb{R}^n$, let
        \[ q(t,x) \triangleq p_{X_t}(y|X_0=x) \]
        Then
        \begin{equation}\label{eq:Fokker-PlanckBackwardEq}
            \frac{\partial q(t,x)}{\partial t} = \mu(x)\sum_{i=1}^n \frac{\partial q(t,x)}{\partial x_i} + \frac{1}{2}\sum_{i=1}^n\frac{\partial^2 \sigma^2(x)q(t,x)}{\partial x_i^2}
        \end{equation}

        \subsubsection{Proof of Backward Equation in 1D}
            \begin{align*}
                \partial_t q(t,x) &= \mu(x)\partial_xq(t,x) + \frac{1}{2}\sigma^2(x)\partial_{xx}q(t,x)
            \end{align*}
            We prove a more general case. Let $\rho()$ be some function defined on the state space, then we define $g(t,x)$ by
            \[ g(t,x) = \mathbb{E}_x[\rho(X_t)] \quad \mathbb{E}_x[\cdot]=\mathbb{E}[\cdot|X_0=x] \]
            Therefore
            \[ g(t+h,x) = \mathbb{E}_x[\rho(X_{t+h})] = \mathbb{E}_x[\mathbb{E}_x[\rho(X_{t+h})|X_h]] = \mathbb{E}_x[g(t, X_h)] \]

            \begin{align*}
                LHS &= \lim_{h \to 0}\frac{1}{h}\mathbb{E}_x[g(t,X_h) - g(t,x)]\\
                &= \lim_{h \to 0}\frac{1}{h}\mathbb{E}_x[g'(t,x)(X_h - x) + \frac{1}{2}g''(t,x)(X_h-x)^2 + o(h)]\\
                &= \mu(x)g'(t,x) + \frac{1}{2}\sigma^2(x)g''(t,x)
            \end{align*}

            And our desired backward equation can be proved by letting $\rho(z) = \mathbb{I}[z \le y]$ and take derivative w.r.t. $y$.
            \[ g(t,x) = \mathbb{E}_x[\rho(X_t)] = \mathbb{P}_x[X_t \le y] \triangleq F(t,x,y) \]

            So $F(t,x,y)$ satisfies the backward equation, and notice that $\partial_y F(t,x,y) = q(t,x)$, so simply take partial derivative of $F$ w.r.t. $y$ yields the desired result.

        \subsubsection{Proof of Forward Equation in 1D}
            By Markov property,
            \[ g(s,y) = \mathbb{E}[\rho(X_{t+s})|X_t=y] \]

    \subsection{Proof of Convergence}
        Let the stationary distribution be $\pi(x)$. Plug $\pi(x)$ into Equation \ref{eq:Fokker-PlanckEquation}.
        \[ 0 = -\frac{\partial}{\partial x}\left( f'(x) \cdot \pi(x) \right) + \frac{1}{2}\frac{\partial}{\partial x}\pi'(x)\]

        Therefore
        \[ \pi'(x) = -f'(x)\pi(x) \]

        This is an ordinary differential equation, with solution
        \[ \pi(x) \sim e^{-f} \]

    \subsection{Convergence Analysis}
        Suppose the function $f$ is $m$-strongly convex. Recall that an $m$-strongly convex function has a quadratic lower bound
        \[ f(y) \ge f(x) + \nabla f(x)^T(y-x) + \frac{m}{2}(y-x)^T\nabla^2f(x)(y-x) \]

        \begin{theorem}
            For an $m$-strongly convex $f$, the Langevin dynamics converges to its stationary distribution exponentially fast.

            Let $x_t$, $y_t$ be two Langevin Dynamics,
            \[ \mathbb{E}[\|x_t-y_t\|] \le e^{-mt}\|x_0-y_0\| \]
        \end{theorem}
        \begin{proof}
            We construct a coupling. Let $X_t$ and $Y_t$ be two Langevin Dynamics, with the same Gaussian noise. Since they have the same noise, the remaining analysis is very similar to that of an ordinary gradient descent.

            Consider $\|x_t-y_t\|^2$ . To show that it converges very fast, we only need to show that its derivative drops very fast. Thank you, continuity!\footnote{``Everybody should be able to take the gradient of a quadratic function. -- Bo Jiang''}
            \begin{align*}
                \mathrm{d}\|x_t-y_t\|^2 &= 2\left(\frac{\mathrm{d}}{\mathrm{d}t}(x_t-y_t)^T(x_t-y_t)\right)\\
                &= 2(\nabla f(y_t) - \nabla f(x_t))^T(x_t-y_t)
            \end{align*}

            By strong convexity
            \[ f(y_t) - f(x_t) \ge \nabla f(x_t)^T(y_t-x_t) + \frac{m}{2}\|x_t-y_t\|^2 \]
            and
            \[ f(x_t) - f(y_t) \ge \nabla f(y_t)^T(x_t-y_t) + \frac{m}{2}\|x_t-y_t\|^2 \]

            Add the two equations up
            \[ 0 \ge (\nabla f(y_t) - \nabla f(x_t))^T(x_t-y_t) + m\|x_t-y_t\|^2 \]

            Therefore
            \[ \mathrm{d}\|x_t-y_t\|^2 \le -2m\|x_t-y_t\|^2 \]

            Integrate on both sides yields our desired result
            \[ \|x_t-y_t\|^2 \le e^{-2m}\|x_0-y_0\|^2 \]
        \end{proof}

    \subsection{Discrete Implementation of Langevin Dynamics}
        \[ Z_{t+1} = Z_t - \eta\nabla f(Z_t) + \sqrt{2\eta}\xi_t \]
        where
        \[ \xi_t \sim \mathcal{N}(0,1) \]
        \begin{remark}
            The choice of $\eta$ matters.
        \end{remark}

        The convergence analysis in the discrete case is beyond the scope.


\chapter{Simple Machines}

As mentioned earlier, physicists define work to be the force applied
times the distance it is applied over. So, if you pushed your car 100
meters with 17 newtons of force, you have done 1700 joules of work.

Humans have always had to move really heavy things, so many centuries
ago we developed simple machines to decrease the amount of force
necessary to execute those tasks. These include things like:
\begin{itemize}
\item Levers
\item Pulleys
\item Ramps
\item Gears
\item Hydraulics
\item Screws
\end{itemize}

\includegraphics[width=0.8\textwidth]{Simple_Machines.png}

While these machines can decrease the force needed, they don't change
the amount of work that must be done. So if the force is decreased to
a third, the distance that you must apply the force is increased by a
factor of three.

``Mechanical gain'' is what we call the increase in force.

\section{Levers}

A lever rotates on a fulcrum. To decrease the necessary force, the load
is placed nearer to the fulcrum than where the force is applied.

In particular, physicists talk about the \newterm{torque} created by a
force. When you push on a lever, the torque is the product of the
force you exert and the distance from the point of rotation.

Torque is typically measured in newton-meters.

To balance two torques, the products must be the same. So, assuming
that the forces are applied in the proper direction,

$$R_L F_L = R_A F_A$$

where $R_L$ and $R_A$ are the distance from the fulcrum to the where
the load's force and the applied force (respectively) are applied, and
$F_L$ and $F_A$ are the amounts of the forces.

\begin{Exercise}[title={Lever}, label=lever]

Paul, who weighs 70 kilograms, sits on a see-saw 4 meters from the
fulcrum.  Jan, who weighs 50 kilograms, wants to balance. How far
should Jan sit from the fulcrum?

\end{Exercise}
\begin{Answer}[ref=lever]
  Paul is exerting $(70)(9.8)$ newtons of force at 4 meters from the
  fulcrum, so he is creating a torque of 2,744 newton-meters of torque
  on the see-saw.  Jan is creating $(50)(9.5) = 490$ newtons of
  force.

  If $r$ is the distance from the fulcrum to Jan's seat, to balance
  $490 r = 2744$, so $r = 5.6$ meters.
\end{Answer}
% KA: https://www.khanacademy.org/science/physics/discoveries/simple-machines-explorations/a/lever

\includegraphics[width=0.8\textwidth]{WD=WD.png}

\section{Ramps}

Ramps, or include planes, let you roll or slide objects up to a higher
level. Steeper ramps give you less mechanical gain. For example, it is much easier
to roll a ball up a wheelchair ramp than on a skateboard ramp.
% diagram neeeded

Assuming the ramp has a constant steepness, the mechanical gain is
equal to the ratio of the length of the ramp divided by the amount
that it rises.

If you assume there is no friction, the force that you push a weight up the ramp will be:

$$F_A = \frac{V}{L} F_G$$

Where $F_A$ is the force you need to push. $L$ is the length of the
ramp, $V$ is the amount of vertical gain and $F_G$ is the force of
gravity on the mass.

(We haven't talked about the sine function yet, but in case you already know about it: Note that

$$\frac{V}{L} = \sin{\theta}$$

where $\theta$ is the angle between the ramp and level.)

\begin{Exercise}[title={Ramp}, label=ramp]
A barrel of oil weighs 136 kilograms. You can push with a force of
up to 300 newtons. You have to get the barrel onto a platform that is 2
meters. What is the shortest board that you can use as a ramp?
\end{Exercise}
\begin{Answer}[ref=ramp]
  To lift the barrel would require $136 \times 9.8 = 1,332.8$ newtons of force.

  Letting $L$ be the length of the ramp:

  $$300= \frac{2}{L} 1332.8$$

  So $L = 8.885$ meters.
\end{Answer}

\section{Gears}

Gears (which might have a chain connecting them like on a bicycle)
have teeth and come in pairs. You apply torque to one gear, and it
applies torque to another. The torque is increased or decreased based
on the ratio between the teeth on the gears.
% ADD: Driver, Driven, Idler

\includegraphics[width=0.8\textwidth]{Gears.png}

If $N_A$ is the number of teeth on the gear you are turning with a
torque of $T_A$, and $N_L$ is the number of teeth on the gear it is
turning, the resulting torque is:

$$T_L = \frac{N_A}{N_L} T_A$$

\begin{Exercise}[title={Gears}, label=gear]

The bicycle is an interesting case because we are not trying to get
mechanical gain. We want to spin the pedals slower with more force.

You like to pedal your bike at 70 revolutions per minute. The
chainring that is connected to your pedals has 53 teeth. The
circumference of your tire is 2.2 meters. You wish to ride a 583 meters
per minute.

How many teeth should the rear sprocket have?

\end{Exercise}
\begin{Answer}[ref=ramp]

  $$583 = (70)(2.2)\frac{53}{n}$$

Thus $n = 14$ teeth.
\end{Answer}
% KA: https://www.khanacademy.org/science/physics/discoveries/simple-machines-explorations/a/simple-machines-and-how-to-use-this-tutorial

\section{Hydraulics}

In a hydraulic system, like the braking system of a car, you exert
force on a piston filled with fluid. The fluid carries that pressure
into another cylinder. The pressure of the fluid pushes the piston in
that cylinder out.

\includegraphics[width=0.8\textwidth]{Hydraulics.png}

The pressure in the hose can be measured in pounds per square inch
(PSI) or newtons per square meter (Pascals or Pa). We will use Pascals.
% ADD: Create a page in the back of the book with units

To figure out how much pressure you create, you divide the force by
the area of the piston head you are pushing.

To figure out how much force that creates on the other end, you
multiply the pressure times the area of the piston head that is
pushing the load.

\begin{Exercise}[title={Hydraulics}, label=hydraulics]

Your car has disc brakes. When you put 2,500,000 pascals of pressure on the
brake fluid, the car stops quickly. As the car designer, you would like
that to require 12 newtons of force from the driver's foot.

What should the radius of the master cylinder (the one the driver is pushing on) be?
\end{Exercise}
\begin{Answer}[ref=hydraulics]
  We are looking for $r$, the radius of the piston head in meters. The area of the piston head is $\pi r^2$.

  The pressure in pascals of the brake fluid is given by $12 / (\pi r^2)$.

  $$2,500,000 = \frac{12}{\pi r^2}$$

  So $r = \sqrt{\frac{12}{\pi \times 2.5 \times 10^6}} = 0.001236077446474$ meters.

\end{Answer}
% KA: https://youtu.be/Pn5YEMwQb4Y


\problemname{Quality-Adjusted Life-Year}

%% Image URL: https://www.pexels.com/photo/sunset-sunshine-travel-wings-103127/
%% Image License: https://www.pexels.com/photo-license/

\illustration{0.33}{balcony.jpg}{~}

The Quality-Adjusted Life-Year (QALY) is a way to measure a person's
quality of life that includes both the quality and the quantity of
life lived.

The quality of life lived can be quantified as a number between $0$ and
$1$.  If someone is living with perfect health, the quality of life is
$1$.  If someone is dead, then the quality of life is $0$.  The quality of
life may increase or decrease due to medical treatements, sickness,
etc.

The QALY for each period in which the quality of life is constant is
simply the product of the quality of life and the length of the period
(in years).  We wish to know the amount of QALY accumulated by a
person at the time of death, given the complete history of this
person.

\section*{Input}

The first line of input contains a single integer $N$~($1 \leq N \leq 100$), which is the number of periods of constant quality of life during the person's lifetime.

The next $N$ lines describe the periods of life. Each of these lines contains two real numbers $q$~($0 < q \leq 1$), which is the quality of life in this period, and $y$~($0 < y \leq 100$), which is the number of years in this period. All real numbers will be specified to exactly one decimal place.

\section*{Output}

Display the QALY accumulated by the person. Your answer will be considered correct if its absolute error does not exceed $10^{-3}$.


\documentclass[11pt]{amsart}
\usepackage{amssymb}
\usepackage{showkeys}
\usepackage{amsmath}
\usepackage{float}
\usepackage{subcaption}
\usepackage{graphicx}
\usepackage{stix}

\newtheorem{thm}{Theorem}[section]
\newtheorem{prop}[thm]{Proposition}
\newtheorem{lem}[thm]{Lemma}
\newtheorem{cor}[thm]{Corollary}

\theoremstyle{definition}
\newtheorem{definition}[thm]{Definition}
\newtheorem{example}[thm]{Example}

\theoremstyle{remark}

\newtheorem{remark}[thm]{Remark}

\newcommand{\R}{\mathbf{R}}  % The real numbers.
\DeclareMathOperator{\dist}{dist} % The distance.

\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

\begin{document}
\title{Signal Denoising and Error Bounding using Analytic Methods}
\author{Noah Stockwell}

\begin{abstract}
It is shown that with normalized convolution kernels, one can find smooth derivatives that interpret any continuous system sampled at discrete points.
\end{abstract}

\maketitle
\tableofcontents

\section{Introduction}
This paper provides a specific formulation of a method to find smooth trends in discrete data for both the Riemann integral and then the Lebesgue integral with bound consideration. The motivation is the existence of the engineering question: \textit{what do real-world processes look like when they are sampled and how do we remove sampling error without destroying the signal?} This paper considers the result of applying a convolution to a set of 2-dimensional discrete points, and how accurate we can expect the convolution to be compared to the process that is being sampled.
\newpage
\section{Formulation for $S\in C_0(\mathbb{R})$ using the Riemann Integral}
\begin{thm}Given a set $T=(T_x, T_y)$ of points separated by an identical $T_x$ distance $\lambda$, the generating function $S\in C_0(\mathbb{R})$ can be approximated by a smooth function.
\end{thm}
\begin{proof}
First, define the piecewise function made up of constant functions that passes through all points in $T$:
$$
f(x)\mid (x,f(x))\in T
$$
With
$$
f(t)\equiv T_{y_m} \qquad\forall\,\, t\in [T_{x_m}, T_{x_{m+1}})
$$
Next, define the function with consideration width $w$:
\begin{equation}
\delta_n(x, w):=
\begin{cases}
\left(1-\left(\frac{x}{w}\right)^n\right)^2 & x\in (-w, w)\\
0&x\not\in (-w, w)\\
\end{cases}
\end{equation}
Normalization is trivial:
$$
N(w):=\int_{-w}^w\delta_n(t, w)\,dt =C(w)
$$where $C$ is a constant. Define and note that
\begin{equation}
\phi(x,w) := \frac{\delta_n(x,w)}{N(w)}
\end{equation}
Is now a probability function: the total area under $\phi$ is now 1 across $\mathbb{R}$. Define the convolution:
$$
f\approx(f\circledast\phi)(x, w) = \int_{-\infty}^\infty f(\tau)*\phi(x-\tau, w)\,d\tau
$$
Which is exactly equivalent to
\begin{equation}
(f\circledast\phi)(x, w) =
\begin{cases}
\displaystyle \int_I f(\tau)*\phi(x-\tau, w)\,d\tau &x\in I\\
0 &x\not\in I\\
\end{cases}
\end{equation}
For $I=(\min(T_x)-w,\max(T_x)+w)$. Because $f$ is constant and piecewise, we can redefine the integral part as such:
\begin{equation}
\displaystyle \int_I f(\tau)*\phi(x-\tau)\,d\tau = \displaystyle\sum_{a\in T_x} f(a) \int_a^{a+\lambda} \phi(x-\tau, w)\,d\tau
\end{equation}
We do not care about extrapolation which means we can throw out the extra convolution width $w$ and tails rewriting the convolution as:
$$
f(x)\approx(f\circledast\phi)(x, w) =\displaystyle\sum_{a\in T_x} f(a)\displaystyle\int_a^{a+\lambda} \phi(x-\tau, w)\,d\tau
$$
For $x\in [\min(T_x),\max(T_x)]$. By construction, $\phi$ is smooth and thus a linear combination of $\phi$'s is smooth.
\end{proof}
\begin{cor}Given a set $T=(T_x, T_y)$ of points separated by an identical $T_x$ distance $\lambda$, the generating derivative can be approximated by a smooth function.
\end{cor}
\begin{proof}
From our last result,
\begin{align}
f(x)&\approx \displaystyle\sum_{a\in T_x} f(a)\displaystyle\int_a^{a+\lambda} \phi(x-\tau, w)\,d\tau\\
\implies \frac{d}{dx}\,f(x)&\approx \displaystyle\sum_{a\in T_x} f(a)\,\,* \frac{d}{dx}\int_a^{a+\lambda} \phi(x-\tau, w)\,d\tau\\
\implies f'(x) &\approx \displaystyle\sum_{a\in T_x} f(a) * \left( \phi(x-a-\lambda, w)-\phi(x-a, w) \right)
\end{align}
The approximation for $f'$ can be convolved with an arbitrary probability kernel $h$, as well, to find a smoother interpretation of the derivative:
\begin{equation}
f'(x)\approx \left(\frac{d}{dx} (f\circledast\phi)(x,w)\,\circledast\, h\right)(x)
\end{equation}
\end{proof}
It is clear that if we choose not only normalized functions $\phi_n$ and $h$ to be approximations to the identity, we have that our representation will approach the function itself at every point \cite{stein_real_2006}.
\newpage

\section{Motivation for $S\in C_0(\mathbb{R})$ using the Lebesgue Integral}
Of the many properties of the Lebesgue integral that we do not have with the Riemann integral is the ability to ignore points on a set of measure zero. It is clear that a process of engineering interest is happening continuously in the real world but can be only sampled at finitely many points. Furthermore, it is possible to have significant noise or sensor malfunctions at some points that drive $S$ incredibly far away from the generating signal.\\

\begin{figure}[H]
\centering
\begin{subfigure}[t]{0.5\textwidth}
	\centering
	\includegraphics[height=2.0in]{/Users/noahstockwell/Documents/TeX/Proj544/Images/NoisyThermocouple.png}
	\caption{The signal over multiple samples}
\end{subfigure}%
~
\begin{subfigure}[t]{0.5\textwidth}
	\centering
	\includegraphics[height=2.0in]{/Users/noahstockwell/Documents/TeX/Proj544/Images/SmoothThermocouple.png}
	\caption{The signal over few samples}
\end{subfigure}
\caption{A sample thermocouple response at different sampling levels}
\end{figure}

Above in Figure 1(A), it is clear to us as humans the general shape of the generating signal. However, the true data points being collected look more like Figure 2(B) and it is often a trade-off between maintaining a close relationship to the original generating signal's value and maintaining a close relationship to the original generating signal's behavior. For example, as seen in section (2), the convolution with $\delta_n$ will either produce a function that has very few quick changes in its derivative, or a function that approximates the original function closely by value. It is in the Engineer's interest to ask the question: \textit{how much data can I reasonably destroy without jeopardizing the conclusion?}\\

With this in mind, we will construct a method and identify how `bad' it can be: how far can it be away from the generating signal?

\section{Formulation for $S\in C_0(\mathbb{R})$ using the Lebesgue Integral}
\subsection{Construction of problem}
Let $S$ be our observed signal. We can visualize $S$ as $S(t) = f(t) + \eta(t) + \rho(t)$ where $f$ is the exact representation of the real process, $\eta$ is white noise upon the signal, and $\rho$ is extreme instantaneous variation induced by the signal. To replicate the real world, let $\rho$ vanish everywhere except on a set of Lebesgue measure 0. The motivation for this representation is that it is possible to have extreme variation when recording a signal that has no physical meaning and does not repeat. As white noise on a signal is zero in the time domain, for all $E$ of sufficient measure, we would like $\displaystyle\int_E\eta=0$. We only need to consider $S$ supported on a set of finite measure as the time values sampled are discrete and thus extrapolation beyond local areas is not desired. As such, we only require $S$ to belong to $L^1_{\text{loc}}$.
\subsection{Requirements}
Without loss of generality, we can split $\eta$ into $\eta^+$ and $\eta^-$ such that:
\begin{align*}
\int S&=\int f+\int \eta\\\\
\int S&=\int f+\int \eta^+-\int \eta^-
\end{align*}
Where $\eta^+$ and $\eta^-$ are both strictly nonnegative functions on the signal interval I, and since we know $\eta$ should be zero in the time domain \cite{saito_naoki_simultaneous_1994}, we want to create a process such that:$$\int_I\eta^+=\int_I\eta^-$$
Since the $\eta^\pm$ are zero when the other is positive, we can let $$I=E^+\bigcup E^-$$ with $E^+$ and $E^-$ disjoint and thus we have:
$$\int_I\eta^i =\int_I\chi_{E^i}\cdot\eta^i= \int_{E^i}\eta^i$$
Since $E^i\subset I$. We can begin constructing the function by defining:
\begin{align*}
E_a^+ &= \{x: S(x)-f(x)>a\}\\
E_a^- &= \{x: f(x)-S(x)>a\}
\end{align*}
\vfill
Clearly, $E_a^+\nearrow E^+$ and $E_a^- \nearrow E^-$ as $a\to 0$. For almost everywhere compactly supported functions, we have $$m(E^\pm_a)\le m(E^\pm)\quad \forall\,\, a>0$$Similarly, for finitely supported functions, we have $$|E^\pm_a|\le |E^\pm|\quad \forall\,\, a>0$$
\vfill
With these, we want to optimize $f$ such that:
\begin{enumerate}
	\item The quantity of noise is largely time independent:
	$$\lim_{a\to 0} \left\lvert m(E_a^+) - m(E_a^-) \right\rvert < \epsilon_a$$
	\item The amount of noise is largely time independent:
	$$\lim_{a\to 0}\left\lvert \int_{E_a^+} \eta - \int_{E_a^-} \eta\right\rvert<\epsilon_a$$
\end{enumerate}
We can rewrite (2) as:
$$\lim_{a\to 0}\left\lvert \int_{E_a^+} S(x)-f(x)\,dx - \int_{E_a^-} f(x)-S(x)\,dx\right\rvert<\epsilon_a$$
With these in hand, we now want to see how we can bound this difference. It is clear that we desire some type of average over value and time so that we can balance both requirement (1) and (2). This paper makes the choice to consider a convolution $\phi$ that is normalized and can be considered part of a sequence approaching an approximation to the identity. In fact, we will chose the exact $\phi_n$ that was selected earlier:

\begin{align*}
	\delta_n(x, w)&:=
	\begin{cases}
	\left(1-\left(\frac{x}{w}\right)^n\right)^2 & x\in (-w, w)\\
	0&x\not\in (-w, w)\\
	\end{cases}\\\\
	N(w)&:=\int_{-w}^w\delta_n(t, w)\,dt = C(w)\\\\
	\phi(x,w)&:= \frac{\delta_n(x,w)}{N(w)}
\end{align*}

The question is how far away from $f$ is $S \circledast \phi$? Two bounding options are considered.

\subsection{Error Bounding using a Naive Approach}
Because we require our convolution kernels to be normalized, we have:
\begin{align*}
	(S\circledast \phi_n) &= \int_E S(\tau)*\phi(x-\tau, w)\,d\tau\\
	 &\le \sup_{x\in E} S(x)\quad\\
\end{align*}
However, this bound is really not helpful. A smarter approach to bounding should be considered. One possible method would be to restrict ourselves to $A \subset E$:
\begin{align*}
(S\circledast \phi_n) &= \int_A S(\tau)*\phi(x-\tau, w)\,d\tau\\
&\le \sup_{x\in A_w} S(x)\quad\\
\end{align*}
Where $A_w$ is a $w$-dilation of $A$. This is a little more helpful but again $A$ has to be large enough to absorb $\eta$.

\subsection{Error Bounding using the Maximal Function}
The Hardy-Littlewood maximal function $S^*$ is defined to be:
\[
S^*(t) := \sup_{B_r(t)\,\, \forall t\in \mathbb{R}} \,\,\intbar_B S(t)\,dt
\]
It is possible to put a bound on the maximal function, especially because we only consider a single-dimensional independent variable on $E$ with m$(E) < \infty$ \cite{stein_real_2006}:
\[
\text{m}({x\in E: S^*(x) > \alpha}) \le \frac{3}{\alpha} \norm{S}_{L^1(E)}
\]
It is apparent that we can consider the maximal function as the upper bound for the convolution if we require that the convolution kernel has most density around the origin and is only a small value away from vanishing outside of a small ball centered at the origin. In fact, with our choice of $\phi_n$ have:
\[
(S \circledast \phi_n)(t) \le (1+c) \,S^*(t)
\]
With $c>0$ small. In fact, because our convolution vanishes outside of a ball of radius $w$ around the origin and our convolution kernel is less than the box kernel $K$ of width $\le w$ outside of a smaller ball, we have:
\begin{align*}
(S\circledast \phi_n) &= \int_E S(\tau)*\phi(x-\tau, w)\,d\tau\\
& \le (1+c) \sup_{\tilde{w} \in (0,w]} \int_E S(\tau)*K(x-\tau, \tilde{w})\,d\tau\\
&=  (1+c) \sup_{B_r(t)\,\, \forall t\in (0, w]} \,\,\intbar_B S(t)\,dt\\
&= (1+c)\,S^*_{w}(x)
\end{align*}
Where $S^*_{w}(x)$ is the maximal function taken over balls of size up to $w$ around $x$ and $c>0$ small. This provides us a much nicer bound on the convolution.\\
It was claimed earlier that $S$ could be represented as the sum of three functions and $\rho$ vanishes everywhere except a set of Lebesgue measure 0. There are multiple methods to remove outliers for data: apply one or multiple and now we have:
\[
\int_E S = \int_E f + \eta
\]
as the Lebesgue integral ignores values on a set of measure zero. We can now assume that we can simply sum the convolution as we did in the beginning:
\begin{align*}
S(x)\approx(S\circledast\phi)(x, w) &\le \epsilon_0(x) + \sum_{a_n\in T_x} S\left(a_n\right)\int_{a_n}^{a_{n+1}} \phi(x-\tau, w)\,d\tau\\
&= \epsilon_0(x) + \sum_{a_n\in T_x} \left(f\left(a_n\right) + \eta\left(a_n\right)\right)\int_{a_n}^{a_{n+1}} \phi(x-\tau, w)\,d\tau\\
\end{align*}
If we set
\[
b_{{a_n}(x), w} = \int_{a_n}^{a_{n+1}} \phi(x-\tau, w)\,d\tau
\]
we have
\begin{align*}
(S\circledast\phi)(x, w) &\le \epsilon_0(x) +  \sum_{a_n\in T_x} \left(f\left(a_n\right) + \eta\left(a_n\right)\right)\ * b_{{a_n}, w}(x)\\
&= \epsilon_0(x) +  \left(\sum_{a_n\in T_x} f\left(a_n\right) * b_{{a_n}, w}(x)\right) + \left(\sum_{a_n\in T_x} \eta\left(a_n\right) * b_{{a_n}, w}(x)\right)\\
\end{align*}
Since $\eta$ sums to zero in the time domain, we get:
\[
\sum_{a_n\in T_x} \eta\left(a_n\right) * b_{{a_n}, w}(x) \equiv 0
\]
Leaving us with only
\[
(S\circledast\phi)(x, w) \le \epsilon_0(x) +  \sum_{a_n\in T_x} f\left(a_n\right) * b_{{a_n}, w}(x)
\]
Which leads us back to the Riemann conclusion. At this point, the question is can we recover $f(t)$ exactly? Clearly we want equality, so the process would have to start with identifying $\epsilon_0$:
\[
\epsilon_0(x) = \min\left(\left|(1+c)*S^*_{w}(x) - S(x)\right|, \left|\sup_{\tilde{x}\in B_w(x)}S(\tilde{x}) - S(x)\right|\right)
\]
It should be clear at least with this choice of bounding, $f(t)$ cannot be recovered exactly. However, for certain classes of functions, we can approximate $f$ very closely. A further question of importance is given a certain type of function, is it possible to significantly improve the bound, i.e. given a first-order ODE, is it possible to be within only an epsilon range of $f$?
\newpage
\section{Conclusion}
We do not have the ability to recreate $f$ with this formulation, however we may be able to get close for certain functions, especially simple functions. This process was implemented in MATLAB(\texttrademark) on an experiment involving temperature measurements of a thermocouple being moved from one environment to another \cite{engr_132}:
\begin{figure}[H]
	\centering
	\includegraphics[height=2.0in]{/Users/noahstockwell/Documents/TeX/Proj544/Images/Comparison.png}
	\caption{Thermocouple response: Noisy and Convolved}
\end{figure}
Figure 2 demonstrates the potential of a well-tuned convolution in denoising signals for Engineering analysis. It is clear that a correct choice of parameters allows for a close interpretation of the signal. For the mathematical question of how far away is the convolution, more work should be done to better identify the constant $c$ and if superior bounding methods exist. Furthermore, it may be of interest to study only certain classes of functions, as we are modeling real-world processes and we should expect a process to be near a simple PDE, for example. \\
A potential method to attempt to analyze $S$ in the future could be trying to interpret $f+\eta^+$ and $f-\eta^-$ because these functions would appear to be a sawtooth-wave style upon a smooth function. It could then be possible to use the property that the sawtooth frequency and amplitude should be equal in the time domain to reduce the error in the approximation to $f$.
\newpage
\bibliography{bibFile}
\bibliographystyle{plain}
\end{document}