\chapter{Acoustic Tubes}\label{ch:brass}

The dynamics of woodwind and brass instruments is based on wave propagation in acoustic tubes. Although the physical processes that generate the sound are fundamentally different from those in strings, the underlying models have many similarities. The main difference between acoustic tubes and the (ideal) strings, is that tubes have a varying cross-sectional area, causing wave dispersion and greatly influencing the modal frequencies and modal shapes present in the system.

In this work, planar wave propagation is assumed (rather than spherical), such that the behaviour of the systems can be approximated using 1D systems. Although higher-dimensional models might better capture some physical effects (see e.g. \cite{Kemp2002}), 1D systems already show good agreement between model and measurement \cite{Eveno2012}. Moreover, looking towards real-time implementation of these models, the choice to simplify to 1D has been made due to the low relative computational cost.

This chapter first presents Webster's equation, which extends the 1D wave equation presented in Section \ref{sec:1DWave} by introducing a spatially varying cross-section. Although not used for the contributions in Part \ref{part:contributions}, Webster's equation forms a good basis for the second part of this chapter, which decomposes Webster's equation into a system of two coupled first-order PDEs. This has been used to model the trombone in paper \citeP[H].

\section{Webster's equation}\label{sec:webstersEq}
For an (axially symmetric) acoustic tube of length $L$ (in m), where the wavelengths of the frequencies at interest are much larger than the radius of the tube, one can simplify the system to be one-dimensional \cite{Bilbao2018}. For low-amplitude vibrations, the air propagation in this tube can be described using \textit{Webster}'s equation \cite{Webster1919}
\begin{equation}\label{eq:webstersPDE}
    S\partial_t^2\Psi = c^2\partial_x(S\partial_x\Psi),
\end{equation}
with \textit{acoustic potential} $\Psi = \Psi(x,t)$ (in m$^2$/s), the cross-sectional area along the tube, or bore profile $S = S(x)$ (in m$^2$) and the speed of sound in air $c$ (in m/s). The state variable $\Psi$ is defined for $t\geq 0$ and $x \in \D$ where domain $\D = [0, L]$. If $S(x)$ is constant, Eq. \eqref{eq:webstersPDE} reduces to the 1D wave equation in Eq. \eqref{eq:1DwavePDE}. This shows that for a cylindrical acoustic tube, the fundamental frequency is not affected by the cross-sectional area, but solely relies on length $L$ and wave speed $c$ according to Eq. \eqref{eq:fundamentalFreq}. The acoustic potential can be related to pressure $p = p(x,t)$ (in Pa) and particle velocity $v = v(x,t)$ (in m/s) according to \cite{Bilbao2018}
\begin{equation}\label{eq:pressureVelocityWebster}
    p = \rho_0 \pt \Psi, \qaq v = -\px \Psi,
\end{equation}
with air density $\rho_0$ (in kg/m$^3$).

The interesting thing about the presence of a variable cross-section, is that it causes dispersive or scattering behaviour, especially at locations of high (spatial) variation of $S$. See Figure \ref{fig:websterPropagation}. Contrary to frequency dispersion as happens in a stiff string (see Chapter \ref{ch:stiffString}), all frequencies travel at the same speed, but some components of the wave get reflected due to the geometry of the tube.

\def\figWidth{0.32}
\begin{figure}[h]
    \centering
    \subfloat[$t = 1$ ms.\label{fig:websterPropagation1}]{\includegraphics[width=\figWidth\textwidth]{figures/resonators/brass/websterPropagation1.eps}}\hfill
    \subfloat[$t = 5$ ms.\label{fig:websterPropagation2}]{\includegraphics[width=\figWidth\textwidth]{figures/resonators/brass/websterPropagation2.eps}}\hfill
    \subfloat[$t = 8$ ms.\label{fig:websterPropagation3}]{\includegraphics[width=\figWidth\textwidth]{figures/resonators/brass/websterPropagation3.eps}}
    \caption{Wave propagation and dispersion in an acoustic tube of varying cross-section (shown in grey) modelled by Webster's equation in Eq. \eqref{eq:webstersPDE}. Positive acoustic potential $\Psi$ is shown in red and negative in blue, which is also plotted for clarity. \label{fig:websterPropagation}}
\end{figure}
%test for git

\subsubsection{Boundary conditions}
The choices for boundary conditions in an acoustic tube are open and closed, defined as \cite{Bilbao2018}\footnote{The Dirichlet condition is identical to the one shown in Eq. \eqref{eq:contDirichlet}, but is derived from $p(0, t) = p(L, t) = 0$ and the time-derivative has thus been kept here.}
\begin{subequations}\label{eq:contBoundariesBrass}
    \begin{align}
        \partial_t\Psi(0, t) &= 0, & \partial_t\Psi(L, t) &= 0, & &\text{(Dirichlet, open)},\label{eq:contDirichletBrass}\\
        \partial_x\Psi(0, t) &= 0, & \partial_x\Psi(L, t) &= 0, & &\text{(Neumann, closed)}\label{eq:contNeumannBrass}.
    \end{align}
\end{subequations}
This might be slightly counter-intuitive when compared to the 1D wave equation, as ``closed" might imply the ``fixed" or Dirichlet boundary condition. The opposite can be intuitively shown by imagining a wave front with a positive acoustic potential (and thus positive pressure according to Eq. \eqref{eq:pressureVelocityWebster}) moving through a tube and hitting a closed end. What reflects is also a wave front with a positive acoustic potential, i.e., the sign of the potential does not flip. This also happens using the free or Neumann condition for the 1D wave equation (see Figure \ref{fig:boundaryCondsCont}).
Here, the following boundaries are chosen
\begin{equation}\label{eq:openClosed}
    \partial_x\Psi(0, t) = 0, \quad \text{and} \quad \partial_t\Psi(L, t) = 0,
\end{equation}
i.e. closed at the left end and open at the right end.

\subsection{Discrete time}
The state variable is discretised to the grid function $\Psi_l^n$ and is defined for $n\in \mathbb{N}^0$ and $l = \{0, \hdots, N\}$, where $N$ is the number of intervals between the grid points.
As the cross-section is distributed in space, $S(x)$ needs to be discretised to a grid function as well, albeit only in space (as it is not time-varying). Following \cite{Bilbao2018}, it is useful to introduce \textit{interleaved grid points} at $l-1/2$ and $l+1/2$ for $S$ and are defined as
\begin{equation}\label{eq:sHalf}
    \Sm = \mxm S(x=lh), \qaq \Sp = \mxp S(x=lh),
\end{equation}
and approximate a `true' (possibly measured) bore profile $S(x)$ sampled at $x=lh$ with grid spacing $h$ (see Figure \ref{fig:variableCrossSection}). Using these definitions, one can discretise Eq. \eqref{eq:webstersPDE} to the following FD scheme \cite{theBible}\footnote{Notice that in \cite{theBible}, Webster's equation is $\Sbar_l \delta_{tt}\Psi^n_l = c^2\dxp(\Sm(\dxm\Psiln))$ but is identical to Eq. \eqref{eq:discWebster}. This discretisation has been chosen for a more straightforward energy analysis in Section \ref{sec:energyAnalysisWebster}.}
\begin{equation}\label{eq:discWebster}
    \Sbar_l \delta_{tt}\Psi^n_l = c^2\dxm\left(\Sp(\dxp\Psiln)\right),
\end{equation}
where
\begin{equation}\label{eq:Sbar}
    \Sbar_l = \mxp\Sm = \mxm\Sp = \mxx S(x=lh),
\end{equation}
the choice of which will become apparent in Section \ref{sec:stabilityEnergyWebster}.
The right-hand side of the scheme contains an operator applied to two grid functions ($S$ and $\Psi$) multiplied onto each other. In order to expand this, the product rule must be used. Recalling Eq. \eqref{eq:productRule} and applying this to backwards spatial operators instead yields
\begin{equation}
    \dxm (u_l^nw_l^n) = (\dxm u_l^n)(\mxm w_l^n) + (\mxm u_l^n)(\dxm w_l^n).
\end{equation}
Using the product rule, the right-hand side of Eq. \eqref{eq:discWebster} can be expanded to
\begin{equation*}
    \Sbar\dtt\Psiln = c^2\left[(\dxm \Sp)(\mxm (\dxp \Psiln)) + (\mxm \Sp)(\dxm (\dxp \Psiln))\right],
\end{equation*}
and solving for $\Psinp$ yields the following update equation (see Appendix \ref{app:webstersUpdateEq}):
\begin{equation}
    \Psinp = 2(1-\lambda^2)\Psiln-\Psinm+ \frac{\lambda^2\Sp}{\Sbar_l}\Psilp + \frac{\lambda^2\Sm}{\Sbar_l}\Psilm,\label{eq:webstersUpdateEq}
\end{equation}
with
\begin{equation}
    \lambda = \frac{ck}{h},
\end{equation}
and similar to the 1D wave equation in Section \ref{sec:1DWaveDisc} needs to abide
\begin{equation}\label{eq:CFLwebster}
    \lambda \leq 1,
\end{equation}
in order for the scheme to be stable. See Section \ref{sec:stabilityEnergyWebster} for a derivation. The number of grid points $N$ can then be calculated in the same way as for the 1D wave equation in Eq. \eqref{eq:orderOfCalc}, and the stencil is similar to Figure \ref{fig:stencil1DWave}.

Notice that at the boundaries, Eq. \eqref{eq:webstersUpdateEq} requires values of $S$ outside of the defined domain through its definition in Eq. \eqref{eq:Sbar} (i.e., $S_{N+1/2}$ and $S_{-1/2}$). To solve this, one can set $\Sbar_0 = S(0)$ and $\Sbar_N = S(L)$ from which $S_{-1/2}$ and $S_{N+1/2}$ can be calculated according to
\begin{subequations}
    \begin{align}
        \Sbar_0 = \frac{1}{2}(S_{1/2} + S_{-1/2}) \ &\Rightarrow \ S_{-1/2} = 2\Sbar_0 - S_{1/2},\\
        \Sbar_N = \frac{1}{2}(S_{N+1/2} + S_{N-1/2}) \ &\Rightarrow \ S_{N+1/2} = 2\Sbar_N - S_{N-1/2}.\label{eq:Snph}
    \end{align}
\end{subequations}
Although these values will not be needed when discretising the boundary conditions in Eq. \eqref{eq:contBoundariesBrass}, they will be useful at a later point.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/resonators/brass/variableCrossSection.eps}
    \caption{Approximations to $S(x)$ used in Eq. \eqref{eq:discWebster}. Dashed lines indicate the interleaved grid on which $S$ is sampled (Eq. \eqref{eq:sHalf}) and solid lines indicate $\Sbar$ which are averages of these (Eq. \eqref{eq:Sbar}). \label{fig:variableCrossSection}}
\end{figure}

\subsubsection{Boundary conditions}
One can discretise the continuous boundary conditions in Eq. \eqref{eq:openClosed} (closed at $x=0$, open at $x=L$) using centred difference operators for higher accuracy
\begin{subequations}\label{eq:discBoundariesBrass}
    \begin{align}
        \dxd\Psi_0^n &= 0 \quad \Rightarrow \quad\Psi_{-1}^n = \Psi_1^n ,\quad \!\!\!\!\!\!&&\text{(Neumann, closed)}, \label{eq:closedLeftBrass}\\
        \dtd\Psi_N^n &= 0\quad\Rightarrow \quad \Psi_N^n = 0, \quad \!\!\!\!\!\!&&\text{(Dirichlet, open)}\label{eq:openRightBrass}.
    \end{align}
\end{subequations}
At the left boundary, Eq. \eqref{eq:webstersUpdateEq} can be expanded to:
\begin{equation*}
    \begin{aligned}
        \Psi_0^{n+1} &= 2(1-\lambda^2)\Psi_0^n-\Psi_0^{n-1}+ \frac{\lambda^2S_{1/2}}{\bar S_0}\Psi_1^n + \frac{\lambda^2S_{-1/2}}{\bar S_0}\Psi_{-1}^n,\\[-0.5em]
        \xLeftrightarrow{\mystrut\ \text{Eq. \eqref{eq:closedLeftBrass}}\ }\quad\Psi_0^{n+1} &= 2(1-\lambda^2)\Psi_0^n-\Psi_0^{n-1}+ \frac{\lambda^2(S_{1/2}+S_{-1/2})}{\bar S_0}\Psi_1^n,
    \end{aligned}
\end{equation*}
and as $\Sbar_0 = \frac{1}{2}(S_{1/2}+S_{-1/2})$ through Eq. \eqref{eq:Sbar}, this can be solved to
\begin{equation}\label{eq:leftBoundaryWebster}
    \Psi_0^{n+1} = 2(1-\lambda^2)\Psi_0^n-\Psi_0^{n-1}+ 2\lambda^2\Psi_1^n.
\end{equation}
One can implement the right boundary condition by simply reducing the range of operation to $l = \{0, \hdots, N-1\}$, as $\Psi_N^n = 0$ according to Eq. \eqref{eq:openRightBrass}. A more realistic boundary condition for the open end is presented in the following.

\subsection{Radiation}\label{sec:radiating}
One of the ways that an acoustic tube loses energy is through radiation. The right boundary condition presented in Eq. \eqref{eq:openClosed} can be changed to be radiating according to \cite{theBible}
\begin{equation}\label{eq:radCont}
    \partial_x\Psi(L,t) = -a_1\partial_t\Psi(L,t)-a_2\Psi(L,t),
\end{equation}
where, for a tube terminating on an infinite plane \cite{Atig2004}
\begin{equation}
    a_1 = \frac{1}{2(0.8216)^2c}\ , \quad \text{and} \quad a_2 = \frac{L}{0.8216\sqrt{S_0S(1)/\pi}}\ ,
\end{equation}
which determine the amount of loss and inertia at the radiating boundary respectively.

The radiating boundary in Eq. \eqref{eq:radCont} can then be discretised to \cite{theBible}
\begin{equation}\label{eq:centRadBound}
    \delta_{x\cdot}\Psi_N^n = -a_1\dtd\Psi_N^n - a_2\mu_{t\cdot}\Psi_N^n,
\end{equation}
which can be expanded and solved for $\Psi_{N+1}^n$ according to
\begin{equation}
    \Psi_{N+1}^n = h\left(-\frac{a_1}{k}(\Psi_N^{n+1} - \Psi_N^{n-1}) - a_2(\Psi_N^{n+1} + \Psi_N^{n-1})\right) + \Psi_{N-1}^n.
\end{equation}
Substitution into Eq. \eqref{eq:webstersUpdateEq} at the $l=N$ yields the following update equation
\begin{equation}
    \Psi_N^{n+1} = \frac{2(1-\lambda^2)\Psi_N^n-\Psi_N^{n-1}+\alpha_-\Psi_N^{n-1} + 2\lambda^2\Psi_{N-1}^n}{\left(1+\alpha_+\right)},
\end{equation}
where
\begin{equation}
    \alpha_\pm = h\left(\frac{a_1}{k}\pm a_2\right)\frac{\lambda^2S_{N+1/2}}{\bar S_N}.
\end{equation}
One can observe that $S_{N+1/2}$ is needed which is outside the defined domain. As mentioned before, setting $\Sbar_N = S(L)$, one can calculate $S_{N+1/2}$ using Eq. \eqref{eq:Snph} to solve the issue.

\subsection{Excitation}\label{sec:webstersExcitation}
Although excitations will be discussed more in-depth in Part \ref{part:exciters}\todo{maybe refer to chapter/section instead}, a simple way to excite Webster's equation will be presented here.

Following \cite{Bilbao2018}, one can create an input signal $v_\text{in} = v_\text{in}(t)$ that interacts with the particle velocity of the tube. As this relates to the acoustic potential as in Eq. \eqref{eq:pressureVelocityWebster}, one can change the boundary condition of the left boundary to
\begin{equation}
    \px \Psi(0, t) = - v_\text{in}.
\end{equation}
Discretising this using the centred spatial operator, yields
\begin{equation}
    \dxd \Psi_0^n = -v_\text{in}^n \quad \Rightarrow \quad \Psi_{-1}^n = 2h v_\text{in}^n + \Psi_1^n,
\end{equation}
and can be substituted into the update equation in Eq. \eqref{eq:webstersUpdateEq} at $l=0$ to get
\begin{align}
    \Psi_0^{n+1} &= 2(1-\lambda^2)\Psi_0^n-\Psi_0^{n-1} \frac{\lambda^2S_{1/2}}{\Sbar_0}\Psi_1^n + \frac{\lambda^2S_{-1/2}}{\Sbar_0}\left(2h v_\text{in}^n + \Psi_1^n\right),\nonumber\\
    \Psi_0^{n+1} &= 2(1-\lambda^2)\Psi_0^n-\Psi_0^{n-1} +2\lambda^2\Psi_1^n + \frac{2h\lambda^2S_{-1/2}}{\Sbar_0} v_\text{in}^n.
\end{align}
The input signal is arbitrary, but looking towards lip excitation, and following \cite{theBible}, one can set the input to a pulse train as shown in Figure \ref{fig:inputWebster}. More details can be found in Chapter \ref{ch:physInspExcitations}.
\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{figures/resonators/brass/inputWebster.eps}
    \caption{A pulse train with a frequency of $213$ Hz a duty cycle of 50\% and an attack of $22$ ms, used to generate the output in Figure \ref{fig:outputWebster}. \label{fig:inputWebster}}
\end{figure}

% At the left boundary, one can sed $\bar S_N = S_N$ from which $S_{N+1/2}$ can be calculated:
% \begin{equation}
%     S_N = \frac{1}{2}(S_{N+1/2} + S_{N-1/2}) \ \Rightarrow \ S_{N+1/2} = 2S_N - S_{N-1/2}.
% \end{equation}
% % \begin{equation}
% %         S_0 = \frac{1}{2}(S_{1/2} + S_{-1/2}) \ \Rightarrow \  S_{-1/2}
% %         = 2S_0 - S_{1/2}
% % \end{equation}

% The same can be done for the right boundary ($\bar S_N = S_N$) if this is chosen to be anything else but open (e.g., closed or radiating -- see Section \ref{sec:radiating}):
% \begin{equation}
%     S_N = \frac{1}{2}(S_{N+1/2} + S_{N-1/2}) \ \Rightarrow \ S_{N+1/2} = 2S_N - S_{N-1/2}.
% \end{equation}
% For now though, we follow the conditions given in \eqref{eq:openClosed} and we can simply set the right boundary to its initial state
% \begin{equation}
%     \Psi_N^n = \Psi_N^0
% \end{equation}
% which is normally $0$. A more realistic open end is a radiating one, which can be found below.

\subsection{Matrix form and output}\label{sec:outputWebster}
One can write scheme \eqref{eq:discWebster} in matrix form by saving the state in a vector $\boldPsi^n = [\Psi_0^n, \hdots, \Psi_N^n]^T$ and creating a $\Dxx$ matrix that includes the effect of the cross-sectional area $S$. Assuming Neumann boundary conditions yields
\begin{equation}
    \Dxx = \frac{1}{h^2}\begin{bmatrix}
        -2& 2 &  & & & \mathbf{0}& \\
        \frac{S_{1/2}}{\Sbar_1} & -2 &\frac{S_{3/2}}{\Sbar_1} & & & & \\
        & \ddots & \ddots & \ddots &  & & \\
        & & \frac{S_{l-1/2}}{\Sbar_l}& -2 & \frac{S_{l+1/2}}{\Sbar_l} & & \\
        & & & \ddots & \ddots & \ddots & \\
        & & &  & \frac{S_{N-3/2}}{\Sbar_{N-1}}  & -2 & \frac{S_{N-1/2}}{\Sbar_{N-1}} \\
        & \mathbf{0} & & & & 2 & -2 \\
    \end{bmatrix}.
\end{equation}
Notice that there are no appearances of $S$ at the boundaries as these vanish due to the boundary conditions as in Eq. \eqref{eq:leftBoundaryWebster}.
Using $\I_N$ as the $N\times N$ identity matrix, one can write scheme \eqref{eq:discWebster} in matrix form as
\begin{equation}\label{eq:matrixFormWebsters}
    \A\boldPsi^{n+1} = \B\boldPsi^n + \C \boldPsi^{n-1} + \mathbf{v}^n
\end{equation}
where
\begin{equation*}
    \begin{gathered}
    \A = \begin{bmatrix}
        \I_{N}& \mathbf{0}\\\
        \mathbf{0} & 1 + \alpha_+\
    \end{bmatrix}, \quad \B = 2\I + c^2 k^2 \Dxx, \quad \text{and} \quad \C = \begin{bmatrix}
        -\I_N & \mathbf{0}\\\
        \mathbf{0}& -1 + \alpha_-
    \end{bmatrix},
    \end{gathered}
\end{equation*}
and the $(N+1)\times 1$ input vector $\mathbf{v}^n$ consists of zeros except for the first index:
\begin{equation}
    \mathbf{v}_i^n =
    \begin{cases}
        \frac{2h \lambda^2 S_{-1/2}}{\Sbar_0}v_\text{in}^n, & \text{if } i = 1,\\
        0,& \text{otherwise}.
    \end{cases}
\end{equation}
%
Notice how the radiation is included by changing the last entry of matrices $\A$ and $\C$.
The output of an implementation of Webster's equation is shown in Figure \ref{fig:outputWebster}. The parameters used for the scheme, the input signal and the geometry used to obtain the output can be found in Table \ref{tab:websterParams}, Figure \ref{fig:inputWebster}, and Figure \ref{fig:geometryWebster} respectively.
\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{figures/resonators/brass/outputWebster.eps}
    \caption{The output of Webster's equation at $\Psi_N^n$ using the input in Figure \ref{fig:inputWebster}, the parameters in Table \ref{tab:websterParams}, and the geometry in Figure \ref{fig:geometryWebster}. \label{fig:outputWebster}}
\end{figure}

\begin{table}[h]
    \begin{center}
    \begin{tabular}{|l|c|c|}
        \hline
        Name & Symbol (unit) & Value\\ \hline
        Length & $L$ (m) & $\approx 3$\\
        Wave speed & $c$ (m/s) & 343\\
        Cross-sectional area & $S(x)$ & See paper \citeP[H]\\\hline
        \end{tabular}
    \caption{Parameters for the implementation of Webster's equation. The length is slightly below $3$ m to yield $\lambda = 1$ in Eq. \eqref{eq:CFLwebster}.\label{tab:websterParams}}
    \end{center}
\end{table}
{\renewcommand{\arraystretch}{1}

\hspace{0.1\textwidth}
\begin{figure}[h]
    \centering
    \includegraphics[width=0.6\textwidth]{figures/resonators/brass/geometryWebster.eps}
    \caption{The geometry used for the implementation. See paper \citeP[H] for more details. \label{fig:geometryWebster}}
\end{figure}

\subsection{Energy analysis}\label{sec:energyAnalysisWebster}
The energy analysis of Webster's equation with a radiating boundary might seem straightforward. However, due to the varying cross-sectional area, the energy balance deserves a more detailed treatment, especially at the boundaries. For this analysis, (centred) Neumann boundary conditions are used for both boundaries (for generality) and the input is ignored. This section follows the steps presented in Section \ref{sec:energyAnalysis}.% to retain generality. %These can then easily be adapted to Dirichlet if needed.

\subsubsection{Step 1: Obtain $\dtp \h$}
Usually, to ensure vanishing boundary terms when using centred Neumann boundary conditions, the primed inner product in Eq. \eqref{eq:primedInnerProd} is chosen. However, as the system has a spatially varying cross-section, the more general weighted inner product in Eq. \eqref{eq:weightedInnerProd} has to be chosen instead.

Taking an inner product weighted by free parameters $0 < \el, \er \leq 2$ at the left and right boundary respectively, of scheme \eqref{eq:discWebster} with $(\dtd \Psiln)$ over discrete domain $d$, yields
\begin{equation}\label{eq:powerBalanceWebster}
    \dtp \h = \langle \dtd\Psiln ,\Sbar_l \delta_{tt}\Psi^n_l\rangle_d^{\el,\er} - c^2\langle\dtd\Psiln, \dxm(\Sp(\delta_{x+}\Psiln))\rangle_d^{\el,\er} = 0.
\end{equation}

\subsubsection{Step 2: Identify energy types and isolate $\dtp$}
As the right boundary is set to be radiating according to Eq. \eqref{eq:centRadBound}, the energy balance will eventually be of the following form:
\begin{equation}\label{eq:radiatingEnergyForm}
    \delta_{t+}(\mathfrak{h}+\mathfrak{h}_\text{b}) = \b-\mathfrak{q}_\text{b},
\end{equation}
where $\mathfrak{h}_\text{b}$ is the energy stored by the radiating boundary through the inertia term, $\mathfrak{q}_\text{b}$ describes the the energy losses through radiation and $\b$ is the general boundary term.

% As one can rewrite
% \begin{equation}\label{eq:switchSignsWebster}
%     \dxp\left(\Sm(\delta_{x-}\Psiln)\right) \ \Longleftrightarrow\  \dxm\big(\Sp(\dxp\Psiln)\big),
% \end{equation}
Starting at Eq. \eqref{eq:powerBalanceWebster}, the last term can -- using identity \eqref{eq:weightedIdentityMinus} -- be rewritten to
\begin{equation*}
    c^2\langle\Sp \dtd \dxp \Psiln, (\dxp \Psiln)\rangle_{\underline{d}} + \b_\text{r} - \b_\text{l},
\end{equation*}
where
\begin{subequations}
    \begin{align}
     \mathfrak{b}_\text{r} &= c^2(\dtd\Psi_N^n)\left(\frac{\epsilon_\text{r}}{2}S_{N+1/2}(\dxp \Psi_N^n) + \left(1-\frac{\epsilon_\text{r}}{2}\right)S_{N-1/2}(\dxm\Psi_N^n)\right), \\
     \mathfrak{b}_\text{l} &= c^2(\dtd\Psi_0^n)\left(\frac{\epsilon_\text{l}}{2}S_{-1/2}(\dxm\Psi_0^n)+\left(1-\frac{\epsilon_\text{l}}{2}\right)S_{1/2}(\dxp \Psi_0^n))\right),
    \end{align}
\end{subequations}
are the right and left boundary term respectively (notice that $\b_\text{l}$ is subtracted). One can immediately observe that the boundary terms vanish if Dirichlet boundary conditions would be used.

Then, using identities \eqref{eq:prodIdentity1} and \eqref{eq:prodIdentity2} yields
\begin{equation*}
    \dtp \h = \b_\text{r} - \b_\text{l}
\end{equation*}
where
\begin{equation}\label{eq:energyBalanceWebster}
    \begin{gathered}
        \h = \t + \v, \qwiq \t = \frac{1}{2} \left(\lVert\sqrt{\Sbar_l} \dtm \Psiln \rVert_d^{\el, \er}\right)^2 \quad \text{and} \\
        % \v =c^2\langle\dtd \dxp \Psiln, \Sp(\dxp \Psiln)\rangle_{\underline{d}}\ .
        \v = \frac{c^2}{2}\langle \Sp\dxp \Psiln, e_{t-}\dxp\Psiln\rangle_{\underline{d}}.
    \end{gathered}
\end{equation}
Notice that $\Sbar_l$ is included in the norm by using its square-root.

The next step is to find definitions for $\el$ and $\er$ such that the boundary terms vanish if radiation were to be ignored. In other words, the boundary terms need to be rewritten such that
\begin{align*}
    \dxd \Psi_0^n &= 0\  \Rightarrow\ \b_\text{l} = 0 \\
    \dxd \Psi_N^n &= 0\  \Rightarrow\ \b_\text{r} = 0
\end{align*}
for the left and right boundary respectively. It can be shown that, for the special cases of $\epsilon_\text{r} = S_{N-1/2}/\mu_{xx}S_N$ and $\epsilon_\text{l} = S_{1/2}/\mu_{xx}S_0$, the boundary terms vanish in this case
\begin{subequations}\label{eq:centStrictDissip}
\begin{align}
    \mathfrak{b}_\text{r} &= c^2 (\dtd\Psi_N^n)S_{N-1/2}(2-\epsilon_\text{r})(\delta_{x\cdot}\Psi_N^n)\label{eq:centStrictDissipRight},\\
    \mathfrak{b}_\text{l} &= c^2 (\dtd\Psi_0^n)S_{1/2}(2-\epsilon_\text{l})(\delta_{x\cdot}\Psi_0^n).\label{eq:centStrictDissipLeft}
\end{align}
\end{subequations}
See Appendix \ref{app:boundaryWebster} for a derivation of this. %Also note that $\el, \er \leq 2$ to the boundary terms to be non-negative.

To add the energy stored and dissipated by the radiating boundary, its definition in Eq. \eqref{eq:centRadBound} can be substituted into the right boundary term $\b_\text{r}$ in Eq. \eqref{eq:centStrictDissipRight} as
\begin{align*}
    \mathfrak{b}_\text{r} &= c^2 (\dtd\Psi_N^n)S_{N-1/2}(2-\epsilon_\text{r})(-a_1\dtd\Psi_N^n - a_2\mu_{t\cdot}\Psi_N^n),\\
    &= c^2 S_{N-1/2}(2-\epsilon_\text{r})\left(-a_1(\dtd\Psi_N^n)^2 - a_2(\dtd\Psi_N^n)(\mu_{t\cdot}\Psi_N^n)\right).
\end{align*}
This can then be decomposed in $\h_\text{b}$ and $\q_\text{b}$ used in Eq. \eqref{eq:powerBalanceWebster}.
% These can be obtained by substituting the definition of the radiating boundary Eq. \eqref{eq:centRadBound} into \eqref{eq:centStrictDissipRight} to get

Using identity \eqref{eq:prodIdentity4} yields the definitions for $\h_\text{b}$ and $\q_\text{b}$ in Eq. \eqref{eq:radiatingEnergyForm}
\begin{equation}
    \h_\text{b} = \frac{c^2 S_{N-1/2}(2-\epsilon_\text{r}) a_2}{2} \mtm(\Psi_N^n)^2, \qaq \q_\text{b} = c^2 S_{N-1/2}(2-\epsilon_\text{r})a_1(\dtd\Psi_N^n)^2.
\end{equation}
Finally, $\b = \b_\text{l}$ and can be shown to vanish for both Dirichlet and (centred) Neumann conditions.

\subsubsection{Step 3: Check units}
\SWcomment[It seems like in order for the units to make sense, one must write Eq. \eqref{eq:webstersPDE} as
\begin{equation}
    \frac{S\rho}{c^2} \ptt\Psi = \frac{B}{c^2}\px(S(\px\Psi)),
\end{equation}
where $B$ is the bulk modulus of air (in N/m$^2$)]
\subsubsection{Step 4: Implementation}
Figure \ref{fig:energyWebsters} shows the energetic output of Webster's equation with a radiating boundary at $x=L$. To highlight the effect of the radiation, the parameters are set to $L = 1$ and $S(x) = 0.01$ for all $x\in \D$.  The system is excited with a raised cosine close to the left boundary, and when the excitation reaches the radiating boundary, the total energy in the system decreases due to the losses. The energy stored by the boundary $\h_\text{b}$ is also shown and indeed increases when the wave reaches the boundary.
\begin{figure}[h]
    \centering
    \begin{tikzpicture}[->,node distance=3cm,
        thick,main node/.style={circle,draw}]

        \node[] (image) at (0,0) {
        \includegraphics[width=\textwidth]{figures/resonators/brass/energyWebster2.eps}
        };

        \node[] (he) at (0.2,0.5) {\small $\mathfrak{h}_\text{e}$};

        \node[] (h) at (-5.75, 1) {\small $\mathfrak{h}$};
        \node[] (v) at (-5.75, 0.5) {\small $\color{red}\mathfrak{v}$};
        \node[] (t) at (-5.75, 0) {\small $\color{blue}\mathfrak{t}$};
        \node[] (hb) at (-5.75, -0.5) {\small $\color[HTML]{00DB00}\mathfrak{h}_\text{b}$};
    \end{tikzpicture}
      \caption{The kinetic (blue), potential (red), and total (black) energy as well as the energy stored by the radiation condition (green) of an implementation of Webster's equation are plotted in the left panel. Notice that the energy decreases between $n=60$ and $n=70$ as the excitation reached the boundary where damping is included. The right panel shows the normalised energy (according to Eq. \eqref{eq:normalisedEnergyDamping}) and shows that the deviation of the energy is within machine precision. \label{fig:energyWebsters}}
\end{figure}

\subsection{Stability through energy analysis}\label{sec:stabilityEnergyWebster}
Frequency domain analysis as presented in Section \ref{sec:stabilityAnalysis}, or more specifically, von Neumann analysis, can not be performed on Webster's equation due to the varying cross-section of the system \cite{theBible}. Instead, stability conditions can be obtained through energy analysis explained in Section \ref{sec:stabilityAnalysisEnergy}.

Consider the following scheme
\begin{equation}
    [S]_l \delta_{tt}\Psi^n_l = c^2\dxm(\Sp(\delta_{x+}\Psiln)),
\end{equation}
where $[S]_l$ is a still undetermined second-order approximation to the true geometry of the acoustic tube and will be shown to be $\Sbar_l$ below. As done for the 1D wave equation in Section \ref{sec:stabilityAnalysisEnergy}, the potential energy $\v$ in Eq. \eqref{eq:energyBalanceWebster} can be rewritten using identity \eqref{eq:prodIdentityEnergyStab} as
\begin{equation*}
    \v = \frac{c^2}{2}\left(\lVert\sqrt{S_{l+1/2}}\mtm\dxp\Psiln\rVert_{\underline{d}}^2-\frac{k^2}{4}\lVert\sqrt{S_{l+1/2}}\dtm\dxp \Psiln\rVert_{\underline{d}}^2\right).
\end{equation*}
For spatially varying systems, one can use the following extension of the bound given in Eq. \eqref{eq:spatialBound} \cite{theBible}
\begin{equation}\label{eq:spatialBoundWebster}
    \lVert \sqrt{\phi_l}\dxp \uln \rVert_{\underline{d}} \leq\frac{2}{h}\lVert \sqrt{\mxm\phi_l} \uln \rVert_d,
\end{equation}
where spatially varying function $\phi_l > 0$ is defined over the same domain as $u$. The following condition can then be put on $\v$
\begin{align*}
    \v &\geq \frac{c^2}{2}\left(\lVert\sqrt{S_{l+1/2}}\mtm\dxp\Psiln\rVert_{\underline{d}}^2 - \frac{k^2}{4}\left(\frac{2}{h}\lVert\sqrt{\mxm \Sp}\dtm\Psiln\rVert_{d}\right)^2\right),\\
    &\geq \frac{c^2}{2}\left(\lVert\sqrt{S_{l+1/2}}\mtm\dxp\Psiln\rVert_{\underline{d}}^2 - \frac{k^2}{h^2}\lVert\sqrt{\mxx S_l}\dtm\Psiln\rVert_{d}^2\right)\\
    & \geq \frac{c^2}{2}\left(\lVert\sqrt{S_{l+1/2}}\mtm\dxp\Psiln\rVert_{\underline{d}}^2 - \frac{k^2}{h^2}\left(\lVert\sqrt{\mxx S_l}\dtm\Psiln\rVert_{d}^{\el,\er}\right)^2\right),
\end{align*}
where the last step is possible because $0 < \el, \er \leq 2$.
% Assuming that $\el, \er \leq 2$, the following is true
% \begin{equation}
%     \lVert \uln \rVert^{\el, \er}_d\leq \lVert \uln\rVert_d
% \end{equation}
% \begin{equation}
%     \t \geq
% \end{equation}
Substituting this into the energy balance in Eq. \eqref{eq:energyBalanceWebster} yields
\begin{align*}
    \h = \t + \v \geq &\  \frac{1}{2} \left(\lVert\sqrt{[S]_l} \dtm \Psiln \rVert_d^{\el, \er}\right)^2 \\
    &+ \frac{c^2}{2}\left(\lVert\sqrt{S_{l+1/2}}\mtm\dxp\Psiln\rVert_{\underline{d}}^2 - \frac{k^2}{h^2}\left(\lVert\sqrt{\mxx S_l}\dtm\Psiln\rVert_{d}^{\el,\er}\right)^2\right),
\end{align*}
and as $\lVert\sqrt{S_{l+1/2}}\mtm\dxp\Psiln\rVert_{\underline{d}}^2$ is non-negative, the following is also true
\begin{equation*}
    \h = \t + \v \geq \frac{1}{2} \left(\lVert\sqrt{[S]_l} \dtm \Psiln \rVert_d^{\el, \er}\right)^2- \frac{\lambda^2}{2}\left(\lVert\sqrt{\mxx S_l}\dtm\Psiln\rVert_{d}^{\el,\er}\right)^2.
\end{equation*}
This can be written as
\begin{equation}
    \h \geq \frac{1}{2} \sum_d \left(\sqrt{[S]_l} - \lambda^2\sqrt{\mxx S_l}\right)(\dtm \Psiln)^2
\end{equation}
which is non-negative if
\begin{align*}
    \text{min}\Big(\sqrt{[S]_l} - \lambda^2\sqrt{\mxx S_l}\Big) \geq 0,\\
    \lambda \leq \text{min}\left(\sqrt{\frac{[S]_l}{\mxx S_l}}\right).
\end{align*}
For the special choice of $[S]_l = \mxx S_l$, this condition reduces to
\begin{equation}
    \lambda \leq 1,
\end{equation}
also given in \eqref{eq:CFLwebster}. This choice of $[S]_l$ is equal to $\Sbar_l$ through Eq. \eqref{eq:Sbar}, hence its choice in Eq. \eqref{eq:discWebster}.

\subsection{Modal analysis}
Following Section \ref{sec:oneStepForm}one can perform a modal analysis of the system by writing Eq. \eqref{eq:matrixFormWebsters} in one-step  form (ignoring the input vector). The variable cross-section causes the modes of the system to vary in interesting ways. If the tube is perfectly cylindrical and $S(x)$ is thus constant, Webster's equation reduces to the 1D wave equation and the modal frequencies are integer multiples of the fundamental frequency.
For low cross-sectional variations, the modes generally follow a linear pattern. See Figure \ref{fig:webstersModes}. The damping per mode, however, follows a different pattern, where higher damping occurs around $\fs/4$, and is due to the comb-filtering effect that the radiation has on the system. \SWcomment[is this true?]

\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{figures/resonators/brass/webstersModes.eps}
    \caption{The result of a modal analysis of Webster's equation with parameters and geometry given in \ref{sec:outputWebster}. Notice the higher damping for modes around $\fs/4$. \label{fig:webstersModes}}
\end{figure}

\section{First-order system}\label{sec:firstOrderSystem}
Until now, only PDEs that are second-order in time have been presented, i.e., that are dependent on the acceleration of the state variable. This section presents a system of two coupled first-order PDEs which are instead dependent on the velocity. State-of-the-art research on brass instruments in the context of FDTD methods also uses this coupled system (see e.g.\cite{Bilbao2016, Harrison2018}) and has been used in this project to model the trombone in paper \citeP[H].\todo{look at wording}

\subsection{Continuous time}
Using the same variables as before for cross-sectional area $S=S(x)$, wave speed $c$, and air density $\rho_0$, a system of PDEs that describes air propagation in an acoustic tube can be defined as follows:
\begin{subequations}\label{eq:firstOrderSystem}
\begin{align}
    \frac{S}{\rho_0 c^2}\partial_t p &= -\partial_x(Sv),\label{eq:contPressure}\\
    \rho_0\partial_tv &= -\partial_xp\label{eq:discVelocity},
\end{align}
\end{subequations}
where pressure $p = p(x,t)$ (Pa) and particle velocity $v = v(x,t)$ (m/s) are defined for $x\in \D$ with domain $\D = [0, L]$ and tube length $L$ (in m). These state variables are related to the acoustic potential $\Psi$ as shown in Eq. \eqref{eq:pressureVelocityWebster} as
\begin{equation}\label{eq:pressureVelocityFirstOrder}
    p = \rho_0\partial_t \Psi, \quad v = -\partial_x\Psi.
\end{equation}
Indeed it can be shown by substituting these definitions into Eq. \eqref{eq:contPressure}, Webster's equation can be obtained:
\begin{equation}
  \nonumber
        \frac{S}{\rho_0 c^2}\partial_t(\rho_0 \partial_t\Psi) = -\partial_x(S(-\partial_x\Psi))\quad \Longrightarrow \quad S\partial_t^2\Psi = c^2\partial_x(S\partial_x\Psi).
\end{equation}

\subsubsection{Boundary conditions}
For the first-order PDE system in Eq. \eqref{eq:firstOrderSystem}, the boundary conditions are defined as follows:
\begin{subequations}
    \begin{align}\label{eq:firstOrderBoundaryConditionsCont}
        p(0,t) &= 0, &  p(L,t) &= 0, & &\quad\text{(Dirichlet, open)},\\
        S(0)v(0) &= 0, & S(L)v(L) &= 0, & &\quad \text{(Neumann, closed)},
    \end{align}
\end{subequations}
which, through Eq. \eqref{eq:pressureVelocityFirstOrder}, relate to the boundary conditions of Webster's equation in Eq. \eqref{eq:contBoundariesBrass}.

\subsection{Discrete time}\label{sec:firstOrderDiscrete}
It is useful to place either $p$ or $v$ on an interleaved grid (see Figure \ref{fig:interleavedGrid}). Following \cite{Harrison2018}, $v$ is placed on this interleaved grid both in space and time.
Accordingly, system \eqref{eq:firstOrderSystem} is discretised as
\begin{subequations}\label{eq:firstOrderFDS}
    \begin{align}
        \frac{\bar S_l}{\rho_0 c^2}\delta_{t+}p_l^n &= -\delta_{x-}(S_{l+1/2}v_{l+1/2}^{n+1/2}),\label{eq:discPressure}\\
        \rho_0 \delta_{t-}v_{l+1/2}^{n+1/2}&=-\delta_{x+}p_l^n,\label{eq:discVelocity}
    \end{align}
\end{subequations}
after which the update equations become
\begin{subequations}
    \begin{align}
        p_l^{n+1} &= p_l^n - \frac{\rho_0 c \lambda}{\bar{S}_l}(S_{l+1/2}v_{l+1/2}^{n+1/2}-S_{l-1/2}v_{l-1/2}^{n+1/2}),\label{eq:pressureUpdate}\\
        v_{l+1/2}^{n+1/2} &= v_{l+1/2}^{n-1/2}-\frac{\lambda}{\rho_0 c}(p_{l+1}^n - p_l^n),\label{eq:velocityUpdate}
    \end{align}
\end{subequations}
where (again) $\lambda = ck/h \leq 1$ for stability. The pressure is defined for $l=\{0, \hdots, N\}$ and the velocity for $l=\{0, \hdots, N-1\}$ where $N$ is the number of intervals between the grid points on the pressure grid. Notice that the range of calculation for the particle velocity is one fewer grid point than that of the pressure.

An advantage of using an interleaved grid like this, is that the forward and backward FD operators are second-order accurate, and can be shown through a Taylor series expansion as done in \ref{sec:FDoperators} (also see \cite{Harrison2018}).

\begin{figure}
    \centering
    \includegraphics[width=0.7\textwidth]{figures/resonators/brass/interleavedGridFigure.pdf}
    \caption{The interleaved grid used for the system of FD schemes in Eq. \eqref{eq:firstOrderFDS}. Grid points on the regular grid (in black) are used for pressure $p$, while points on the interleaved grid (in white) are used for particle velocity $v$.\label{fig:interleavedGrid}}
\end{figure}

\subsubsection{Boundary conditions}\label{sec:boundariesFirstOrder}
The boundary conditions in Eq. \eqref{eq:firstOrderBoundaryConditionsCont} can be discretised as follows
\begin{subequations}
    \begin{align}\label{eq:firstOrderBoundaryConditions}
        p_0^n &= 0, & p_N^n &= 0, & &\text{(Dirichlet, open),}\\
        \mu_{x-}(S_{1/2}v_{1/2}^n) &= 0, & \mu_{x+}(S_{N-1/2}v_{N-1/2}^n) &= 0, & &\text{(Neumann, closed)}.
    \end{align}
\end{subequations}
\todo{stencil?}
\subsection{Matrix form}
System \eqref{eq:firstOrderFDS} can be written in matrix form, by saving the states of $p_l^n$ and $v_{l+1/2}^{n+1}$ in vectors as
\begin{equation}
    \mathbf{p}^n = [p_0^n, \hdots, p_N^n]^T, \qaq \mathbf{v}^{n+1/2} = [v_{1/2}^{n+1/2}, \hdots, v_{N-1/2}^{n+1/2}]^T
\end{equation}
which are of sizes $(N+1)\times 1$ and $N\times 1$ respectively. One may then write the scheme in matrix form as
\begin{subequations}
    \begin{align}
        \mathbf{v}^{n+1/2} &= \mathbf{v}^{n-1/2} + \B_p \mathbf{p}^n\\
        \mathbf{p}^{n+1} &= \mathbf{p}^n + \B_v \mathbf{v}^{n+1/2}
    \end{align}
\end{subequations}
where
\begin{equation}
    \B_v = \frac{\rho_0 c}\lambda \begin{bmatrix}
        -\frac{2S_{1/2}}{\Sbar_0}& & & \mathbf{0}\\
        \frac{S_{1/2}}{\Sbar_1} & -\frac{S_{3/2}}{\Sbar_1} & & \\
        & \ddots & \ddots & \\
        & & \frac{S_{N-3/2}}{\Sbar_{N-1}} & -\frac{S_{N-1/2}}{\Sbar_{N-1}} \\
        \mathbf{0} & & & \frac{2 S_{N-1/2}}{\Sbar_N} \\
    \end{bmatrix}.
\end{equation}
is of size $(N+1) \times N$.

\begin{equation}
    \B_p = \frac{\lambda}{\rho c}\begin{bmatrix}
        1& -1& & & \mathbf{0} &\\
         & 1 & -1 & & & \\
       & & \ddots & \ddots & &\\
         & & & 1& -1& \\
        & \mathbf{0} & & & 1& -1 \\
    \end{bmatrix}.
\end{equation}
is of size $N \times (N+1)$.

Alternatively, one can write the scheme in one-step form by concatenating the states of the pressure and particle velocity into one vector. The matrix form will then be
\begin{equation}
    \underbrace{\begin{bmatrix}
        \mathbf{p}^{n+1}\\
        \mathbf{v}^{n+1/2}
    \end{bmatrix}}_{\u^{n+1}} = \B\underbrace{\begin{bmatrix}
        \mathbf{p}^{n}\\
        \mathbf{v}^{n-1/2}
    \end{bmatrix}}_{\u^{n}}
\end{equation}
where
\begin{equation}
    \B = \begin{bmatrix}
        \I_{N+1}+\B_v\B_p & \B_v  \\
        \B_p & \I_{N}
    \end{bmatrix},
\end{equation}
is of size $(2N + 1) \times (2N + 1)$ and may be directly used as matrix $\Q$ for the modal analysis using a one-step form described in Section \ref{sec:oneStepForm}.

Finally the concatenated vector is defined as,
\begin{equation}
    \u^n = [p_0^n, \hdots, p_N^n, v_{1/2}^{n-1/2}, \hdots, v_{N-1/2}^{n-1/2}]^T
\end{equation}
and will be of size $(2N + 1) \times 1$.

\subsection{Energy analysis}
This section presents an energy analysis of the first-order system presented above using the techniques presented in Section \ref{sec:energyAnalysis}. For the bulk of the analysis, \cite{Harrison2018} is followed.

\subsubsection{Step 1: Obtain $\dtp \h$}
To obtain the correct energy balance, an inner product of Eq. \eqref{eq:discPressure} with $\mu_{t+}p_l^n$ needs to be taken over discrete domain $d = \{0, \hdots, N\}$. Using the primed inner product in Eq. \eqref{eq:primedInnerProd} and, after taking all terms to the left-hand side, yields\footnote{The primed rather than the weighted inner product can be used here as the eventual boundary terms can be shown to vanish when using the boundary conditions in Eq. \eqref{eq:firstOrderBoundaryConditions}.}
\begin{equation}\label{eq:firstOrderPrimed}
    \delta_{t+}\mathfrak{h} = \frac{1}{\rho_0 c^2}\langle \mu_{t+}p_l^n, \Sbar \delta_{t+}p_l^n \rangle_{d}' +\langle \mu_{t+}p_l^n, \dxm(S_{l+1/2}v_{l+1/2}^{n+1/2})\rangle_{d}' = 0.
\end{equation}

\subsubsection{Step 2: Identify energy types and isolate $\dtp$}
For the rest of the analysis, the following superscripts and subscripts will be assumed unless denoted otherwise: $n$ and $l$ for $p$, $l$ for $\bar S$, $l+1/2$ for $S$ and $l+1/2$ and $n+1/2$ for $v$. After performing summation by parts of the last term using identity \eqref{eq:primedIdentityMinus}, Eq. \eqref{eq:firstOrderPrimed} becomes
\begin{equation}\label{eq:discEnergyFirstOrder}
    \delta_{t+}\mathfrak{h} = \frac{1}{\rho_0 c^2}\langle \mu_{t+}p, \bar S \delta_{t+}p \rangle_{d}' -\langle \mu_{t+}\dxp p, Sv\rangle_{\underline{d}} = -\mathfrak{b} \quad \end{equation}
where the boundary term is
\begin{align}
    \mathfrak{b} &= \mathfrak{b}_\text{r} - \mathfrak{b}_\text{l}, \quad \text{with} \nonumber\\
    \mathfrak{b}_\text{r} &= (\mu_{t+}p_N)\mu_{x+}(S_{N-1/2}v_{N-1/2})\quad \text{and}\label{eq:firstOrderRightBoundary}\\
    \mathfrak{b}_\text{l} &= (\mu_{t+}p_0)\mu_{x-}(S_{1/2}v_{1/2})\label{eq:firstOrderLeftBoundary},
\end{align}
and can be shown to vanish under the boundary conditions shown in Eq. \eqref{eq:firstOrderBoundaryConditions}. Then, Eq. \eqref{eq:discVelocity} can be substituted into Eq. \eqref{eq:discEnergyFirstOrder} to get
\begin{align}
    \delta_{t+}\mathfrak{h} &= \frac{1}{\rho_0 c^2}\langle \mu_{t+}p, \bar S \delta_{t+}p \rangle_{d}' -\langle \mu_{t+}(-\rho_0\delta_{t-}v), Sv\rangle_{\underline{d}} = 0\\
    &= \frac{1}{\rho_0 c^2}\langle \mu_{t+}p, \bar S \delta_{t+}p \rangle_{d}' + \rho_0 \langle \delta_{t\cdot}v, Sv\rangle_{\underline{d}} = 0.
\end{align}
Finally, one can use identities \eqref{eq:prodIdentity3} and \eqref{eq:prodIdentity2} for the first and second term respectively to get
\begin{equation}\label{eq:energyBalanceFirstOrder}
    \begin{gathered}
        \mathfrak{h} = \mathfrak{t} + \mathfrak{v}\quad \text{where}\\
       \mathfrak{t} = \frac{\rho_0}{2}\langle Sv, e_{t-}v\rangle_{\underline{d}}\quad \text{and} \quad \mathfrak{v} = \frac{1}{2\rho_0 c^2}\left(\lVert\sqrt{\bar S }p\rVert'_d\right)^2.
    \end{gathered}
\end{equation}
\subsubsection{Step 3: Check units}
Writing the terms in Eq. \eqref{eq:energyBalanceFirstOrder} in their units yields
\begin{align*}
    \t = \frac{\rho_0}{2}\langle Sv, e_{t-}v\rangle_{\underline{d}}\ \overset{\text{in units}}{\xrightarrow{\hspace*{1cm}}}& \ \text{kg}\cdot \text{m}^{-3}\cdot \text{m} \cdot (\text{m}^2 \cdot \text{m}\cdot\text{s}^{-1} \cdot \text{m}\cdot\text{s}^{-1}),\\
    & = \text{kg} \cdot \text{m}^2 \cdot \text{s}^{-2},\\
    \v = \frac{1}{2\rho_0 c^2}\left(\lVert\sqrt{\bar S }p\rVert'_d\right)^2\ \overset{\text{in units}}{\xrightarrow{\hspace*{1cm}}}&\  (\text{kg} \cdot \text{m}^{-3} \cdot \text{m}^2 \cdot \text{s}^{-2})^{-1} \cdot(\text{m} \cdot \text{kg} \cdot \text{m}^{-1}\cdot \text{s}^{-2})^2,\\
    &= \text{kg} \cdot \text{m}^2 \cdot \text{s}^{-2},
\end{align*}
and indeed have the correct units.
\subsubsection{Step 4: Implementation}
Figure \ref{fig:energyFirstOrder} shows the energetic output of an implementation of the first order system in Eq. \eqref{eq:firstOrderFDS}, and shows that the energy is within machine precision.

\begin{figure}[h]
    \centering
    \begin{tikzpicture}[->,node distance=3cm,
        thick,main node/.style={circle,draw}]

        \node[] (image) at (0,0) {
        \includegraphics[width=\textwidth]{figures/resonators/brass/energyFirstOrder.eps}
        };

        \node[] (he) at (0.2,0.5) {\small $\mathfrak{h}_\text{e}$};

        \node[] (h) at (-5.75, 1) {\small $\mathfrak{h}$};
        \node[] (v) at (-5.75, 0.5) {\small $\color{red}\mathfrak{v}$};
        \node[] (t) at (-5.75, 0) {\small $\color{blue}\mathfrak{t}$};
    \end{tikzpicture}
      \caption{The kinetic (blue), potential (red), and total (black) energy of an implementation of the first-order system in Eq. \eqref{eq:firstOrderFDS} are plotted in the left panel. The right panel shows the normalised energy (according to Eq. \eqref{eq:normalisedEnergy}) and shows that the deviation of the energy is within machine precision. \label{fig:energyFirstOrder}}
\end{figure}

\subsection{Adding radiation}
\def\r{\text{r}}
\def\one{{(1)}}
Following \cite{Harrison2018}, radiation  can be added to the schemes using a circuit representation of the Levine and Schwinger radiation model (See Figure \ref{fig:circuit}). This section will provide a derivation \SWcomment[I'll actually move most of this to an appendix...]

\begin{figure}[h]
    \centering
    \includegraphics[width=0.7\textwidth]{figures/resonators/brass/circuit.pdf}
    \caption{The circuit representation of the Levine and Schwinger radiation model. \label{fig:circuit}}
\end{figure}

The system can be described as
\begin{subequations}\label{eq:barVPSystem}
    \begin{align}
        \bar v &= \mu_{t+}v_\one + \frac{1}{R_2}\mu_{t+}p_\one + C_\r \delta_{t+}p_\one,\label{eq:barV}\\
        \bar p &= L_\r \delta_{t+}v_\one,\label{eq:barP1}\\
        \bar p &= \left(1+\frac{R_1}{R_2}\right)\mu_{t+}p_\one+ R_1 C_\r\delta_{t+}p_\one\label{eq:barP2},
    \end{align}
\end{subequations}
where $\bar p^{n+1/2}$ and $\bar v^{n+1/2}$ are placed on the interleaved temporal grid and are related to the tube by
\begin{equation}\label{eq:barVars}
    \bar p = \mu_{t+}p^n_N, \quad \bar S_N \bar v = \mu_{x-}\left(S_{N+1/2}v_{N+1/2}^{n+1/2}\right).
\end{equation}
This can be applied to the right boundary of the tube by evaluating Eq. \eqref{eq:pressureUpdate} at $l = N$
\begin{equation}
    p_N^{n+1} = p_N^n - \frac{\rho_0 c \lambda}{\bar{S}_N}\left(S_{N+1/2}v_{N+1/2}^{n+1/2}-S_{N-1/2}v_{N-1/2}^{n+1/2}\right),
\end{equation}
and, rewriting this to
\begin{align}
    p_N^{n+1} &= p_N^n - \frac{\rho_0 c \lambda}{\bar{S}_N}\left(2\mu_{x-}\left(S_{N+1/2}v_{N+1/2}^{n+1/2}\right)-2S_{N-1/2}v_{N-1/2}^{n+1/2}\right)\nonumber,\\
    p_N^{n+1} &= p_N^n - \frac{2\rho_0 c \lambda}{\bar{S}_N}\left(\bar S_N \bar v-S_{N-1/2}v_{N-1/2}^{n+1/2}\right)\label{eq:preSolutP}.
\end{align}
A definition for $\bar v$ can then be found by first expanding Eq. \eqref{eq:barV} to
\begin{equation}\label{eq:vBarExpanded}
    \bar v = \frac{1}{2}\left(v_\one^{n+1} + v_\one^n\right) + \left(\frac{1}{2R_2} + \frac{C_\r}{k}\right) p_\one^{n+1} +\left(\frac{1}{2R_2} - \frac{C_\r}{k}\right)p_\one^n
\end{equation}
after which it should be made solely dependent on the known values $v_\one^n$, $p_\one^n$ and $p_N^n$ and the unknown $p_N^{n+1}$ (as this can be obtained using Eq. \eqref{eq:preSolutP}).

Equation \eqref{eq:barP1} can be expanded to
\begin{equation}\label{eq:voneNext}
    v_\one^{n+1} = \frac{k}{L_\r}\bar p + v_\one^n ,
\end{equation}
and Eq. \eqref{eq:barP2} to
\begin{align*}
    &\bar p =\left(1+\frac{R_1}{R_2}\right)\mu_{t+}p_\one+ R_1 C_\r\delta_{t+}p_\one,\\
    &\bar p =\frac{1}{2}\left(1+\frac{R_1}{R_2}\right)\left(p_\one^{n+1} + p_\one^n\right) + \frac{R_1C_\r}{k}\left(p_\one^{n+1} - p_\one^n\right),\\
    \left(\frac{1}{2}+\frac{R_1}{2R_2} + \frac{R_1C_\r}{k}\right)&p_\one^{n+1} = \bar p + \left(\frac{R_1C_\r}{k} - \frac{1}{2} - \frac{R_1}{2R_2}\right)p_\one^n
\end{align*}
and finally solved for $p_\one^{n+1}$ as
\begin{equation}\label{eq:poneNext}
    p_\one^{n+1} = \underbrace{\left(\frac{2R_2k}{2R_1R_2C_\r + k(R_1 + R_2)}\right)}_{\zeta_1}\bar p + \underbrace{\left(\frac{2R_1R_2C_\r - k(R_1 + R_2)}{2R_1R_2C_\r + k(R_1 + R_2)}\right)}_{\zeta_2} p_\one^n .
\end{equation}
Equations \eqref{eq:voneNext} and \eqref{eq:poneNext} can then be substituted into Eq. \eqref{eq:vBarExpanded} and, using the definition of $\bar p$ from Eq. \eqref{eq:barVars}, yields
\begin{align}
    \bar v &= \frac{1}{2}\left(\frac{k}{L_\r}(\mu_{t+}p_N^n) + 2v_\one^n\right)+\left(\frac{1}{2R_2} + \frac{C_\r}{k}\right)\zeta_1\mu_{t+}p_N^n\nonumber \\
    & \qquad\qquad+ \left(\frac{1}{2R_2} + \frac{C_\r}{k}\right)\zeta_2p_\one^n + \left(\frac{1}{2R_2} - \frac{C_\r}{k}\right)p_\one^n\nonumber,\\
    \bar v &= \underbrace{\left(\frac{k}{2L_\r} + \frac{\zeta_1}{2R_2}+\frac{C_\r\zeta_1}{k}\right)}_{\zeta_3}\mu_{t+}p_N^n + v_\one^n + \underbrace{\left(\frac{\zeta_2+1}{2R_2} + \frac{C_\r\zeta_2 - C_\r}{k}\right)}_{\zeta_4}p_\one^n.
\end{align}
Finally, substituting this definition for $\bar v$ into Eq. \eqref{eq:preSolutP} yields
\begin{align}
    p_N^{n+1} &= p_N^n - \frac{2\rho_0c\lambda}{\bar S_N}\left(\bar S_N
    \left[\zeta_3\left(\frac{p_N^{n+1} + p_N^n}{2}\right) + v_\one^n + \zeta_4p_\one^n\right] - S_{N-1/2}v_{N-1/2}^{n+1/2}\right)\nonumber\\
    p_N^{n+1} &= p_N^n - \rho_0c\lambda\left(\zeta_3(p_N^{n+1} + p_N^n) + 2(v_\one^n + \zeta_4p_\one^n)-\frac{2S_{N-1/2}v_{N-1/2}^{n+1/2}}{\bar S_N}\right)\nonumber
\end{align}
and yields a definition for $p_N^{n+1}$ based on known values of the system
\begin{equation}
    p_N^{n+1} = \frac{1 - \rho_0c\lambda\zeta_3}{1+\rho_0c\lambda\zeta_3}p_N^n - \frac{2\rho_0c\lambda}{1+\rho_0c\lambda\zeta_3} \left( v_\one^n+\zeta_4p_\one^n - \frac{S_{N-1/2}v_{N-1/2}^{n+1/2}}{\bar S_N}\right).
\end{equation}
\subsection{Energy}
\SWcomment[not done yet...]
Recalling the condition at the right boundary from \eqref{eq:firstOrderRightBoundary}
\begin{equation}
    \mathfrak{b}_\r = (\mu_{t+}p_N)\underbrace{\mu_{x+}(S_{N-1/2}v_{N-1/2})}_{\mu_{x-}S_{N+1/2}v_{N+1/2}},
\end{equation}
using Eq. \eqref{eq:barVars} one can rewrite this to
\begin{equation}
    \mathfrak{b}_\r = \bar p\bar S_N\bar v.
\end{equation}
then we can


\chapter{The Game of Nim}
\marginurl{%
    The game of Nim:\\\noindent
    Introduction to Combinatorial Game Theory \#3
}{youtu.be/H-SyB0NK3H8}

This chapter discusses probably the most famous combinatorial game, the game
of \emph{Nim}.
In this game there are several piles of chips on the table. On each turn
the current player may remove some number of chips from \emph{one} of the piles;
however, the player should remove \emph{at least one chip}.
We say that a game of Nim is a $k$-pile game of Nim if there are $k$ piles.
\marginurl{%
  You can play Nim on this website
}{dotsphinx.com/games/nim/}

We start from analysis of the game when we have one pile of chips. It is clear
that the first player to move wins since he/she may remove all the chips.

Consider a more complicated case when we have two piles of size $n$ and $m$
respectively. We need to consider two cases:
\begin{enumerate}
  \item If $n = m$, then the second player to move wins. Indeed, we can
    use the symmetric strategy; i.e., if the first player removes $s$
    chips from one pile we also remove $s$ chips from the other pile.
    It is clear that we can always make a move as long as the first player can.
  \item Otherwise, the first player wins because it can move to the state
    with two equal piles.
\end{enumerate}

The case of three piles is even more complicated. So we spend the rest of the
chapter studying it.

\section{Nim Sum}

We start from a definition of the XOR operation
$\lxor : \set{0, 1} \times \set{0, 1} \to \set{0, 1}$,
also known as ``exclusive or''), this operation is defined as follows:
$a \lxor b = 1$ iff $a \neq b$.

It is well-known that any number $n \in \N_0$
can be represented as a binary number ($\N_0$ denotes nonnegative integers);
\nomenclature[S]{$\N_0$}{denotes the set of nonnegative integers}
we write $n = (a_\ell, \dots, a_0)_2$ if $n = \sum_{i = 0}^\ell a_i 2^i$.
For example,
$5 = 4 + 1 = 1 \cdot 2^2 + 0 \cdot 2^1 + 1 \cdot 2^0 = (1, 0, 1)_2$ and
$6 = 4 + 2 = 1 \cdot 2^2 + 1 \cdot 2^1 + 0 \cdot 2^0 = (1, 1, 0)_2$.
So we can define the Nim sum $\bitwisexor : \N_0 \times \N_0 \to \N_0$, also
known as bitwise xor, as follows:
$(a_\ell, \dots, a_0)_2 \bitwisexor (b_\ell, \dots, b_0)_2 =
    (a_\ell \lxor b_\ell, \dots, a_0 \lxor b_0)_2$.
For example, $5 \bitwisexor 6 = (1, 0, 1)_2 \bitwisexor (1, 1, 0)_2 =
(1 \lxor 1, 0 \lxor 1, 1 \lxor 0)_2 = (0, 1, 1)_2$.
\nomenclature[N]{$\bitwisexor$}{denotes the Nim sum}

\begin{exercise}
  Show that $a \bitwisexor (b \bitwisexor c) = (a \bitwisexor b) \bitwisexor c$
  for any $a, b, c \in \N_0$.
\end{exercise}
Hence, we are going to write $a \bitwisexor b \bitwisexor c$ instead of
$a \bitwisexor (b \bitwisexor c)$ and $(a \bitwisexor b) \bitwisexor c$.

\section{Bouton's Theorem}

Now we may notice that $a \bitwisexor b = 0$ iff $a = b$. So
our result about $2$-pile Nim can be rephrased:
a position $(a, b)$ in the $2$-pile Nim is a P-position iff
$a \bitwisexor b = 0$. Which leads us to the next theorem.
\begin{theorem}[Bouton]
\label{theorem:bouton}
  A position $(a, b, c)$ in $3$-pile Nim is a P-position iff
  $a \bitwisexor b \bitwisexor c = 0$
\end{theorem}
\begin{proof}
  We prove the statement using structural induction.
  First note that the only terminal position the $3$-pile Nim is $(0, 0, 0)$
  and $(0, 0, 0)$ and $0 \bitwisexor 0 \bitwisexor 0 = 0$.

  Let us consider
  some $(a, b, c)$ such that $a \bitwisexor b \bitwisexor c \neq 0$.
  We need to show that there is a move from this position to a P-position.
  Let $a \bitwisexor b \bitwisexor c =
      (0, \dots, 0, 1, r_{k - 1}, \dots, r_0)_2$. So among $a$, $b$, and $c$
  there is a number that has $1$ in the $k$th position.
  Note that without loss of generality
  $a = (p_\ell, \dots, p_{k + 1}, 1, p_{k - 1}, \dots, p_0)_2$.
  Consider $a' = (p_\ell, \dots, p_{k + 1}, 0,
    p_{k - 1} \lxor r_{k - 1}, \dots, r_0 \lxor p_0)_2$. It is clear that
  $a' < a$ and $a' \bitwisexor b \bitwisexor c = 0$.
  Hence, $(a', b, c)$ is a P-position and therefore, $(a, b, c)$ is an
  N-position.

  Finally, let us consider $(a, b, c)$ such that
  $a \bitwisexor b \bitwisexor c = 0$. Assume that there is a move to a
  position $(a', b, c)$ such that $a' \bitwisexor b \bitwisexor c = 0$.
  This implies that
  $(a' \bitwisexor b \bitwisexor c) \bitwisexor
      (a \bitwisexor b \bitwisexor c) =  a \bitwisexor a' = 0$, whence $a = a'$.
\end{proof}

\begin{exercise}
  Prove that a position $(a_1, \dots, a_n)$ in $k$-pile Nim is a P-position
  iff $a_1 \bitwisexor \dots \bitwisexor a_k = 0$.
\end{exercise}

Using Bouton's theorem we can analyse games that do not look like the game of
Nim. An important example of such a game is the \emph{turning turtles} game.
\begin{game}[Turning Turtles Game]
  Given a horizontal line of $m$ coins with some coins showing heads and some
  tails. Each turn, a player have to flip one coin from head to tail, and in the
  same time (if he/she wants), flip one more coin to the left of it.
\end{game}
The game is equivalent to the game of Nim with each coin showing head in $k$th
position equals to a pile of $k$ chips.
\begin{enumerate}
  \item Flip one coin of position $k$ from head to tail. This move is equivalent
    with removing all stones from a pile with $k$ stones.
  \item Flip one coin of position $k$ from head to tail and flip another coin
    (from tail to head) to the left of it in position $t$. This move is equivalent
    with removing some stones from a pile with $k$ stones leaving $t$ stones in
    that pile.
  \item Flip one coin of position $k$ from head to tail and flip another coin
    (from head to tail) to the left of it in position $t$. This move is equivalent
    with removing some stones from a pile with $k$ stones leaving $t$ stones in that
    pile. Note that having two piles with a same number of stones is the same as
    having none of both piles by Bouton's theorem and the fact that
    $x \bitwisexor x = 0$.
\end{enumerate}

\section{Error-correcting Codes}
Using developed methods we may solve the following question about an extension
of the game discussed in \Cref{chapter:strong-induction}:
Alice has chosen a number from $1$ to $1000$. Bob wants to
guess the number so he is asking Alice ``yes'' or ``no'' questions.
How many questions does Bob need to ask to determine the number in the
worst-case scenario if Alice may lie to one of Bob's questions?

It is clear that Bob can use the same strategy as if Alice cannot lie bu ask
every question twice and if the answer for the same question are different ask
it third time. Hence, the number of questions necessary is at most $21$. In this
section we show how Bob can guess the number using only $15$ questions.

To solve this question we need to introduce the notion of error-correcting
codes. To illustrate this notion consider completely different scenario.
Two people wish to send a number from $\range{n}$ by sending $m$ bits;
unfortunately the channel connecting them is not unreliable so among $m$ sent
bits $d$ may be corrupted, but they still want to be able to reconstruct the
original message. How can they do this? To answer this question we need to
introduce error-correcting codes.
\begin{definition}
  Let $x, y \in \set{0, 1}^m$ we say that the Hamming distance
  $\hammingDist{x}{y}$between $x$ and
  $y$ is the number of positions where these two strings are different.

  A function $C : \range{n} \to \set{0, 1}^m$ is an error-correcting code
  correcting $d$ errors if for any $z \in \set{0, 1}^m$ there is at most one
  $i$ such that $\hammingDist{C(i)}{z} \le d$.
\end{definition}
It is clear that if they know such a code, they can send a number from
$\range{n}$ and reconstruct it back. We may also note that if Bob knows such a
code for $n = 1000$ and $d = 1$, then Bob may guess the Alice's number using $m$
questions.

Hence, to prove that it is enough for Bob to ask 15 questions, we need to show
that there is an error-correcting code with $n = 1000$, $m = 15$, and $d = 1$.
One may notice that instead of constructing an error correcting code, it is
enough to construct a set $S \subseteq \set{0, 1}^m$ such that $S$ has $n$
elements and $\hammingDist{x}{y} \ge 2d + 1$ for any $x, y \in S$. Indeed, assume
such a set exists; without loss of generality $S = \set{x_1, \dots, x_n}$.
Consider $C : \range{n} \to \set{0, 1}^m$ such that $C(i) = a_i$. We claim that
$C$ is an error-correcting code correcting $d$ errors. Assume the opposite;
i.e., that for some $z$ there are $i$, $j$ such that
$\hammingDist{C(i)}{z}, \hammingDist{C(j)}{z} \le d$. This implies that
$\hammingDist{C(i)}{C(j)} = \hammingDist{a_i}{a_j} \le 2d + 1$ which is a
contradiction.

In the rest of the section we construct such a set $S$ for $n \ge 1000$, $m =
15$, and $d = 1$. Note that any two P-positions in turning turtles are different
in at least $3$ positions. Hence, let us consider the set
$S \subseteq \set{0, 1}^m$ of P-positions in turning turtles with $m$ coins
(heads are represented by $0$ and tails are represented by $1$). To finish the
construction of the set, we use the following lemma that we prove in
\Cref{chapter:principles}.
\begin{lemma}
\label{lemma:turning-turtles-number-P}
  There are at least $2^{2^r - r - 1}$ P-positions in turning turtles with
  $2^r - 1$ coins.
\end{lemma}

Combining this together we prove the following theorem.
\begin{theorem}
  Let $r$ be an integer, let $n = 2^{2^r - r - 1}$, and $m = 2^r - 1$.
  There is an error-correcting code $C : \range{n} \to \set{0, 1}^m$
  correcting $1$ error.\footnote{%
    Nonetheless that these constructions seem very artificial and theoretical,
    in fact a code based on a very similar idea, the binary Golay code, was used
    to encode pictures of Jupiter and Saturn sent by Voyager~1 and Voyager~2.
  }
\end{theorem}
The code constructed in this section is called \emph{Hamming code} and in
\Cref{chapter:principles} we will show that it is optimal; i.e., we cannot
reduce $m$ without reducing $n$.

\begin{chapterendexercises}
  \exercise \emph{Nimble} is a game played on a board made of a line of squares
    labeled $0$, $1$, \dots A finite number of coins is placed on squares with
    possibly more than one coin on a square. A move consists in taking one coin
    and moving it to any square to the left, possibly over some coins and
    possibly onto a square containing some coins. Players, as usual, alternate
    moves and the game ends when all the coins are on the square $0$. The last
    player to move wins. Determine P- and N-positions in this game.
  \exercise In \emph{staircase Nim}, a staircase of $n$ steps contains coins of
    some of the steps. We can describe any position by a tuple $(a_1, \dots, a_n)$,
    where $a_i$ is the number of coins on the $i$th step. A move in staircase
    Nim consists of moving any number of coins from the $i$th step to
    $(i - 1)$th step. Coins reaching the ground (step $0$) are removed from the
    play. The game ends when all the coins are removed. Players alternate moves;
    the last player to move wins. Show that $(a_1, \dots, a_n)$ is a P-position
    if coins on odd steps $(x_1, x_3, \dots)$ form a P-position in the game of
    Nim.
  \exercise In \emph{index-$k$ Nim} players can remove chips from at least
    one but up to $k$ different piles. Show that $(a_1, \dots, a_n)$ is a
    P-position in index-$k$ Nim iff
    $\sum_{j = 1}^n b_{j, i} \equiv 0 \pmod{k + 1}$ for every
    $i \in \set{0, 1, \dots, \ell}$, where $(b_{j, \ell} \dots b_{j, 0})_2$ is
    the binary representation of $a_j$.
\end{chapterendexercises}


\section{Linear Scalable Transformer
model}

As stated earlier, the proposed model proposes the replacement of the
scaled dot-product attention from original Transformer architecture by a
kernelized attention with relative positional encoding

The proposed model replaces the scaled-dot-product-attention by a
kernelized attention with RPE. Following the observations of
\citet{shaw2018selfattention} that
accumulating absolute positional encoding with relative positional
encoding yield no benefits, the positional encoding is also removed -
although for some specific applications it might be beneficial to
maintain it. The algorithm used to calculate each term is chosen
dynamically to occupy the least memory depending on the sequence lengths
and embedding dimensions - as memory usage is easier to evaluate
precisely than execution time.

In this work we have chosen the following formulation, with
$\phi(x) = elu(x) + 1$.

\begin{equation}
A = \frac{\left( \phi(Q) \times \phi(K^T) + S^{rel} \right)}{\sum_j \left( \phi(Q) \times \phi(K^T) + S^{rel} \right)} \times V
\end{equation}

This is essentially a combination of two terms: the kernelized attention
proposed by  \citet{katharopoulos2020transformers}, and the relative positional encoding proposed by
\citet{shaw2018selfattention}. The left
term is the score matrix of shape $(L_Q, L_K)$, with a denominator
which scales all rows so that they sum to 1. For the sake of the
implementation, the multiplication must be distributed as:

\begin{equation}
A = \frac{\left( \phi(Q) \times \phi(K^T) \times V \right) + \left( S^{rel} \times V\right)}{\sum_j \left( \phi(Q) \times \phi(K^T) \right) + \sum_j \left( S^{rel} \right)}
\end{equation}

The denominator can be easily calculated by applying the (naive or
linear complexity) algorithms with V replaced by a matrix of shape
$(L_K, 1)$ full of 1.

For each case (masked/bidirectional) the algorithm is chosen between
naive and linear complexity to occupy the least memory.

\begin{itemize}
\item
for the masked $Q \times K^T \times V$ term, the memory occupied by
the naive algorithm is $L_QL_K$ while the linear complexity
algorithm occupies $d^2 \times max(L_Q, L_K)$
\item
for the bidirectional $Q \times K^T \times V$ term, the memory
occupied by the naive algorithm is $L_QL_K$ while the linear
complexity algorithm occupies $d^2$
\item
for the $S_{rel} \times V$ term (masked and bidirectional), the
memory occupied by the naive algorithm is $L_QL_K$ while the linear
complexity algorithm occupies $L_Q \times (2k+1 + 4)$
\end{itemize}

\endinput


\documentclass[12pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amsthm,amssymb,amsfonts}
\usepackage{enumitem}
\usepackage{tabu}
\usepackage{xcolor}
\usepackage{mathtools}
\usepackage{tcolorbox}
\usepackage{changepage}
\usepackage{kpfonts}
\usepackage{picture}
\usepackage{venndiagram}
\usepackage{graphicx}

\newcommand{\prob}[1]{\mathbb{P}(#1)}
\newcommand{\condprob}[2]{\mathbb{P}(#1 \text{ } \lvert \text{ } #2)}

\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\field}{\mathcal{F}}

\begin{document}
% ================================================================================================================================
% ================================================================================================================================
\section*{CHAPTER 1}

\subsection*{Conjuction Fallacy}
\noindent
The probability of the joint realization of two events, say $A$ and $B$, cannot be larger than the probability of either of the two events, considered on its own. Denoting the probabilities of the events by $\mathbb{P}(A)$, $\mathbb{P}(B)$, and $\mathbb{P}(A \cap B)$, the basic law of probability law takes the form,

\begin{align*}
\mathbb{P}(A \cap B) \leq \mathbb{P}(A) \enspace \text{  or  } \enspace \mathbb{P}(A \cap B) \mathbb{P}(B).
\end{align*}

\subsection*{Binary Relations}
\noindent
\textbf{Definition} A \textit{Binary Relation}, given sets $X$ and $Y$, is a set $R$ such that

\begin{equation*}
R \subseteq X \times Y.
\end{equation*}

\noindent
A binary relation is a set of ordered pairs $xy \in X \times Y$, where $xy$ is an abbreviation for $(x,y)$. If $X=Y$, then $R$ is said to be a binary relation on $X$. A binary relations $R$ is a \textit{quasi order} on a set $X$ if it is \textit{reflexive} and \textit{transitive}. That is, for all $x$, $y$, and $z$ in $X$,

\begin{align*}
xRx && (\text{reflexivity}) \\
xRy \text{ } \& \text{ } yRz \implies xRz && (\text{reflexivity})
\end{align*}

\noindent
A binary relation $R$ is an \textit{equivalence relation} on a set $X$ if it is \textit{reflexive}, \textit{transitive}, and \textit{symmetric} on $X$. for all $x$, $y$, and $z$ in $X$,

\begin{equation*}
xRy \iff yRx.
\end{equation*}

\subsection*{Partitions}
\noindent
The family $\mathcal{X} = \big \{ [x] \lvert x \in X \big \}$ of subsets is called a \textit{partition} of $X$ induced by $\sim$. Any partition $\mathcal{X}$ of $X$ satisfies the following three properties,

\begin{enumerate}
\item $Y \in X$ implies $Y \neq \emptyset$;
\item $Y,Z \in \mathcal{X}$ and $Y \neq Z$ imply $Y \cap Z \neq \emptyset$;
\item $\cup \mathcal{X} = X$.
\end{enumerate}

\noindent
Conversely, any family $\mathcal{X}$ of subsets of a set $X$ satisfying $[1]$, $[2]$, and $[3]$ is called a partition of $X$.

% ================================================================================================================================
% ================================================================================================================================
\section*{CHAPTER 2: Sample Spaces}

\subsection*{The Sample Space}
\noindent
What is critical is that all feasible outcomes have a description in the sample space. It is of no importance that the sample space contains outcomes that never happen in practice: in the framework of a probabilistic model, such outcomes may be assigned zero probability. \\

\noindent
A sample space which is either \textbf{finite} or \textbf{countable} is called \textit{discrete}.  \\

\begin{tcolorbox}
\begin{center}
Finite vs. Countable
\end{center}

\textbf{Finite}: A definite number. Not infinite. In other words it could be measured, or given a value.

\vspace*{.5cm}

\textbf{Countable}: Either finite or a countably infinite (implying that the cardinality of a countable set is a subset of $\N$). Elements of a countable set can always be counted one at a time and, although the counting may never finish, every element of the set is associated with a unique natural number.
\end{tcolorbox}

\subsection*{Concept of an Event}
\noindent
Fundamentally, an \textit{Event} is a subset of the sample space. In general, each event pertaining to a particular experiment may be identified with a subset of the relevant sample space. However, with respect to the converse, the general assumption that all subsets of the sample space are events to which a probability could be assigned would create mathematical difficulties.

\subsection*{Indicator Functions}
\noindent
\textbf{Definition}: An \textit{Indicator Function} is a function defined on a set $\Omega$ that indicates membership of an element in a subset of $X$ of $\Omega$. Let $\Omega$ be a non-empty set and $X$ a subset of $\Omega$. The indicator function on $X$ is a function $I_x : \Omega \rightarrow \{ 0,1 \}$ by,

\begin{equation*}
I_x(w) =  \begin{cases}
      0 & w \in X \\
      1 & w \in \overline{X} \text{  } (\text{or } w \not \in X)
   \end{cases}
\end{equation*}

\noindent
Properties of Indicator Functions:

\begin{itemize}
\item For subsets $X, Y$ of $\Omega$, $I_{X \cap Y}=I_X \cdot I_Y$, i.e. for every $w \in \Omega$, $I_{X \cap Y}(w)=I_X(w) \cdot I_Y(w)$.
\item $I_{X \cup Y}=I_X + I_Y - I_{X \cap Y}$ (if $X,Y$ are disjoint, $I_{X \cup Y}=I_X + I_Y$)
\item $I_{A^C}=1-I_A$
\item If $X \subseteq Y$, then $I_X \subseteq I_Y$, i.e. $I_X(w) \leq I_Y(w)$ for every $w \in \Omega$.
\end{itemize}

% ================================================================================================================================
% ================================================================================================================================
\section*{CHAPTER 3: Probability and Area}

\subsection*{Axioms for a Field}
\noindent
\textbf{Definition}: Let $\Omega$ be a set. Let $\field$ be a nonempty collection of subsets of $\Omega$ satisfying the axioms

\begin{adjustwidth}{2.5em}{0pt}
\textbf{[F1]} For all $A, B \in \field$, $A \cup B \in \field$.

\vspace{.1cm}
\noindent
\textbf{[F2]} For each $A \in \field$, $\overline{A} \in \field$ also.
\end{adjustwidth}

\vspace{.5cm}
\noindent
\textbf{Theorem 3.2:} Let $\field$ be a field of subsets of $\Omega$. Then,
\begin{frame}{}
\begin{enumerate}[label=(\roman*)]
\item $\Omega \in \field$
\item $\emptyset \in \field$
\item For all $A, B \in \field$, $A \setminus B = A \overline{B} \in \field$
\item $\bigcup\limits_{i=1}^{n} A_i \in \field$
\item $\bigcap\limits_{i=1}^{n} A_i \in \field$
 \makebox(0,0){\put(3,5\normalbaselineskip){%
               $\left.\rule{0pt}{2.2\normalbaselineskip}\right\}$ $\field$ is closed under finite union \textbf{and} finite intersection}}
\end{enumerate}
\end{frame}

\subsection*{Infinite Sample Spaces and $\sigma$-Fields}
\noindent
\textbf{Definition}: A field $(\Omega, \field)$ is called a $\sigma$-field if it is \textit{closed under countable union}, that is, if for any countable collection $A_1, \ldots, A_n, \ldots$ of sets in $\field$, we have $\bigcup_{n=1}^{\infty} A_n \in \field$. Note that for any sets $A_1, \ldots, A_n, \ldots$, we have

\begin{equation*}
\bigcap_{n=1}^{\infty} A_n = \overline{\bigcup_{n=1}^{\infty} \overline{A_n}}
\end{equation*}

\noindent
As a $\sigma$-field, $\field$ is closed under both countable union and complementation. We conclude that $\field$ is also closed under countable intersection.

\subsection*{Borel Fields}
\noindent
Recall that a set $S$ of real numbers is an \textit{open} set of $\R$ if for any $x \in S$, there exists some $\delta > 0$ such that whenever $\lvert x - y \rvert < \delta$, then $y \in S$. The standard family of events for $\R$ is a distinguished $\sigma$-field $\mathcal{B}$ containing all the open sets of $\R$. In fact, $\mathcal{B}$ is the `smallest' $\sigma$-field containing these open sets. The collection $\mathcal{B}$ is called the \textit{Borel Field} of $\R$, and the events in $\mathcal{B}$ are referred to as the \textit{Borel sets} of $\R$. These definitions extend naturally to the case where all the sample space is $\R^n$.

\begin{center}
\includegraphics[width=9cm, height=5cm]{probthoerycommonterms}
\end{center}

\noindent
\subsection*{General Strategy}
We start with a field of sets $\field$ on a set $\Omega$. Then, we introduce the concept of a probability measure $\mathbb{P}$ as a function assigning a number $\mathbb{P}(A)$, with $0 \leq \mathbb{P}(A) \leq 1$, to every event $A$ in $\field$. This function $\mathbb{P} : \field \rightarrow [0,1]$ will be assumed to satisfy a number of axioms (given in Ch. 4). The probability measure $\mathbb{P}$ is not assumed in general to be defined on all subsets of the sample space due to the case of an uncountable sample space leading to a contradiction.

% ================================================================================================================================
% ================================================================================================================================
\section*{CHAPTER 4: Probability Measures}
We shall define a probability measure $\mathbb{P}$ as a function assigning a number $\mathbb{P}(A)$ to each event in $A$ in a field $\field$. For all events $A$, $\mathbb{P}$ will satisfy the following conditions,

\begin{enumerate}[label=(\roman*)]
\item $\mathbb{P}(\Omega) = 1$;
\item $0 \leq \mathbb{P}(A) \leq 1$;
\item $\mathbb{P}(A \cup B) = \mathbb{P}(A) + \mathbb{P}(B)$ for incompatible events $A,B$;
\item $\mathbb{P}(\bigcup_{i=1}^{n}) = \sum_{i=1}^{n} \mathbb{P}(A_i)$ for pairwise incompatible events $A_1, \ldots, A_n$ (\textit{finite additivity});
\end{enumerate}

\noindent
By requiring that if $A_1, \ldots, A_i, \ldots$ is a countably infinite sequence of pairwise incompatible events, then $\bigcup_{i=1}^{n} A_i$ is an event. Moreover,

\begin{equation}
\mathbb{P} \big ( \bigcup_{i=1}^{\infty} \big ) = \sum_{i=1}^{\infty} \mathbb{P}(A_i).
\end{equation}

\subsection*{Finitely Additive Probability Space}
Let $\field$ be a field of sets on $\Omega$. Then, the triple $(\Omega, \field, \mathbb{P})$ is a \textit{finitely additive probability space} iff $\mathbb{P}$ is a real valued function on $\field$ satisfying the following three conditions: For all $A,B \in \field$,

\begin{itemize}
\item $[K1] \mathbb{P}(\Omega) = 1$;
\item $[K2] \mathbb{P}(A) \geq 0$;
\item $[K3]$ if $A \cap B = 0$, then $\mathbb{P}(A \cup B) = \mathbb{P}(A) + \mathbb{P}(B)$;
\end{itemize}

\noindent
The function $\mathbb{P}$ is called a \textit{probability measure}. In the special case where $\field$ is a $\sigma$-field,

\begin{equation}
\mathbb{P} \big ( \bigcup_{i=1}^{\infty} \big ) = \sum_{i=1}^{\infty} \mathbb{P}(A_i)
\end{equation}

\noindent
holds for any countable family $\big \{ A_i \lvert i \in \N \big \}$ of pairwise disjoint events, then $(\Omega, \field, \mathbb{P})$ is called a \textit{probability space}. \\

\subsection*{Probability and Counting Measure}
\noindent
Suppose that $\Omega$ is a finite set, and let $\field$ be a field of its subsets. For any event $A \in \field$, we define

\begin{equation*}
\prob{A} = \frac{\lvert A \rvert}{\lvert \Omega \rvert}
\end{equation*}

\noindent
The value of the probability measure on $\mathbb{P}$ for a particular event $A$ is the ratio of the number of elements of $A$ to the total number of elements in the sample space $\Omega$. The probability measure is sometimes called the \textit{counting measure} and is appropriate when it makes sense to attribute an equal weight or likelihood to each point of the sample space.
\subsection*{Probability Distribution}

\noindent
\textbf{Definition}: Let $\Omega$ be a finite or countable sample space. Then $p$ is a \textit{probability distribution} on $\Omega$ iff $p$ is a real valued function on $\Omega$ satisfying

\begin{align*}
p(x) \geq 0 \text{ for all } x \in \Omega; \\
\sum_{x \in \Omega} p(x) = 1.
\end{align*}

\noindent
\textbf{Theorem}: Let $\field$ be a field of sets on a finite or countable sample space $\Omega$, and let $p$ be a probability distribution on $\Omega$. Define a function $\mathbb{P}$ on $\field$ by

\begin{equation}
\mathbb{P}(A) = \sum_{x \in A} p(x),
\end{equation}

\noindent
for all $A \in \field$. Then $(\Omega, \field, \mathbb{P})$ is a \textit{probability space}. When $(A_i)_{i \in I}$ is a sequence of pairwise events, with the index set $I$ being finite or countable, and each of the events $A_i$ being finite or countable,

\begin{align*}
\sum_{i \in I} \mathbb{P}(A_i) & = \sum_{i \in I} \sum_{x \in A_i} p(x) \\
& = \sum_{x \in \bigcup_{i \in I} A_i} p(x) \\
& = \mathbb{P}(\bigcup_{i \in I} A_i)
\end{align*}

\subsection*{Remarks}
\noindent
The material in this chapter suggests that the notions of a field of events and of a probability measure are only considered with finite or countable sample spaces because the probability of any subset of the sample space could be computed from the probability distribution. With uncountable sample spaces, the notion of a probability distribution is useless. It turns out that `probability density functions' is a conceptual notion related to that of probability distributions.

% ================================================================================================================================
% ================================================================================================================================
\section*{CHAPTER 5: Basic Rules of Probability Calculus}
\noindent
\textbf{Theorem}: Let $\Omega$ be a sample space, and $(\Omega, \field, \mathbb{P})$ a finite additive probability space. We do note assume that $\Omega$ or $\field$ are finite. Then, for any events $A,B,C \in \field$,

\begin{enumerate}[label=(\roman*)]
\item $\prob{A} + \prob{\overline{A}} = 1$
\item $\prob{\emptyset}=0$
\item If $A_1, \ldots, A_n$ are incompatible events in $\field$, then

\begin{equation*}
\mathbb{P} \Big ( \sum_{j=1}^{n} A_j \Big ) = \sum_{j=1}^{n} \prob{A_j}
\end{equation*}

\item If $A \subseteq B$, then $\prob{A} \leq \prob{B}$
\item $\prob{A \cup  B} = \prob{A} + \prob{B} - \prob{A \cap B}$
\item

\begin{align*}
\prob{A \cup B \cup C} =  \enspace & \prob{A} + \prob{B} + \prob{C} \\
& - \prob{AB} - \prob{AC} - \prob{BC} \\
& + \prob{ABC}
\end{align*}

\item $\prob{A\overline{B}} = \prob{A \setminus B} = \prob{A} - \prob{AB}$
\end{enumerate}

\subsection*{Poincar\'{e}'s Identity}
\noindent
We write $S(k,n)$ for the collection of subsets of the set $\{ 1, \ldots, n \}$ containing exactly $k$ elements, with $1 \leq k \leq n$. For any finite collection $A_1, \ldots, A_n$ of events, we have

\begin{equation*}
\prob{\bigcup_{i=1}^{n} A_i} = \sum_{i=0}^{n-1} (-1)^i \sum_{J \in S(i_1, n)} \prob{\bigcap_{k \in J} A_k}
\end{equation*}

% ================================================================================================================================
% ================================================================================================================================
\section*{CHAPTER 6}
\subsection*{Sampling with Replacement and with Ordering}
\noindent
Let $A_1, A_2, \ldots, A_n$ be $n$ finite sets containing $m_1, m_2, \ldots, m_n > 0$ points, respectively. Then,

\begin{equation}
\lvert A_1, A_2, \ldots, A_n \rvert = m_1 \cdot m_2 \ldots \cdot m_n
\end{equation}

\noindent
In a set containing $n > 0$ elements, there are exactly $n^m$ ordered samples of size $m \geq 1$, with replacement.

\subsection*{Sampling without Replacement and with Ordering}
The number of ordered samples of size $m$ without replacement, in a set of $n \geq m > 1$ elements, is equal to

\begin{equation*}
(n)_m = n(n-1) \cdot \ldots \cdot (n-m+1) = \frac{n!}{(n-m)!}
\end{equation*}

\noindent
The first equality defines the notation $(n)_m$. We have thus $(n)_n = n!$.

\subsection*{Stirling's Formula}
\noindent
For any positive integer $n$, we have

\begin{equation}
n! \sim (2 \pi)^{\frac{1}{2}} n^{n + \frac{1}{2}} e^{-n}.
\end{equation}
% ================================================================================================================================
% ================================================================================================================================
\section*{CHAPTER 7}
\subsection*{Binomial Coefficient}
\noindent
In a set of size $n \geq 0$, the number of subsets of size $m \geq 0$ is equal to

\begin{equation*}
{n \choose m} = \frac{n!}{m! (n-m)!}.
\end{equation*}

\noindent
The coefficient ${n \choose m}$ is called the binomial coefficient and is used in the \textit{binomial theorem}. It represents the number of ways of splitting a set of $n$ elements into two subsets containing $n-m$ and $m$ elements with $0 \leq m \leq n$.

\subsection*{Binomial Theorem}
\noindent
Let $a$ and $b$ be any two real number, and let $n$ be a positive integer. Then,

\begin{equation*}
(a+b)^n = \sum_{k=0}^{n} {n \choose k} a^k b^{n-k}.
\end{equation*}

\noindent
We can generalize this theorem to satisfy the following when $0 \leq m \leq n$,

\begin{align*}
{n \choose m} & = {n \choose n-m}; \\
{n \choose m} & = {n-1 \choose m-1} + {n-1 \choose m}; \\
2^n & = {n \choose 0} + {n \choose 1} + \cdots + {n \choose n} = \sum_{k=0}^{n} {n \choose k}. \\
\end{align*}

\subsection*{Multinomial Coefficient}
\noindent
In attempts to generalize the binomial coefficient, we define the \textit{multinomial coefficient}. It will later be used in the definition of the \textit{multinomial distribution}. There are exactly

\begin{equation*}
{n \choose m_1 m_2 \ldots m_k} = \frac{n!}{m_1! \cdot m_2! \cdot \ldots \cdot m_k!}
\end{equation*}

\noindent
ways of splitting a set containing $n \geq 0$ elements into $k \geq 0$ subsets containing $m_1, m_2, \ldots, m_k$ elements with $m_1 + m_2 + \ldots + m_k = n$ and $m_i \geq 0$ for $i = 1, 2, \ldots, k$.

\subsection*{Multinomial Theorem}
\noindent
\noindent
Let $a_1, a_2, \ldots, a_k$ be any $k$ real numbers, and let $n$ be a positive integer. Then,

\begin{equation*}
(a_1 + a_2 + \ldots + a_k)^n = \sum_{(m_1, m_2, \ldots, m_k)} {n \choose m_1 m_2 \ldots m_k} a_1^{m_1} a_2^{m_2} \cdots a_k^{m_k},
\end{equation*}

\noindent
where the summation runs over all $k$-tuples $(m_1, m_2, \ldots, m_k)$ of non-negative integers $m_i$, $0 \leq i \leq k$, satisfying $\sum_{i=1}^{k} m_1 = n$.

% ================================================================================================================================
% ================================================================================================================================
\section*{CHAPTER 8: Discrete Distributions}
Five cases of discrete distributions are considered in this chapter. In the first three cases, the sample space is finite.

\subsubsection*{Bernoulli Distribution}
\noindent
Recall that a \textit{probability distribution} is a real valued function $p$ defined on a finite or countable sample space $\Omega$ and satisfying the two conditions:

\begin{enumerate}
\item $p(x) \geq 0$ for all $x \in \Omega$;
\item $\sum_{x \in \Omega} p(x) = 1$.
\end{enumerate}

\noindent
Let $\Omega = \{ 0,1 \}$ be a sample space, and let $\alpha$ be any real number, with $0 \leq \alpha \leq 1$. The probability distribution $p$ defined on $\Omega$ by

\begin{equation*}
p(x) =  \begin{cases}
      \alpha & \text{if } x = 1 \\
      1-\alpha & \text{if } x = 0
      \end{cases}
\end{equation*}

\noindent
is called a \textit{Bernoulli distribution} with parameter $\alpha$. Any probability distribution on the sample space $\Omega$ is a Bernoulli distribution because there is always some parameter $\alpha \geq 0$ satisfying the above equation.

\subsubsection*{Bernoulli Distribution: Empirical Situation}
\noindent
An experimenter is selecting a single ball from an urn containing only black and white balls in proportions $\alpha$ and $1-\alpha$, respectively. When a black ball is selected, $1$ is recorded; otherwise, $0$ is recorded.

\subsubsection*{Binomial Distribution}
Let $\Omega = \{ 0, 1, \ldots, n \}$ be a sample space, with $n$ a positive integer, and let $\alpha$ be any real number, with $0 \leq \alpha \leq 1$. The probability distribution $p$ defined on $\Omega$ by

\begin{equation*}
p(m) = {n \choose m} a^m (1 - \alpha)^{n-m}
\end{equation*}

\noindent
is called a \textit{binomial distribution} with parameters $\alpha$ and $n$.

\subsubsection*{Binomial Distribution: Empirical Situation}
\noindent
An experimenter is selecting $n$ balls, with replacement, from an urn containing only black and white balls. The probability of getting a black ball if a single ball is selected is equal to $\alpha$. The number $m$ of black balls obtained is recorded.

\subsubsection*{Multinomial Distribution}
\noindent
Let $n$ and $k$ be two integers with $n \geq k > 0$. Let the sample space $\Omega$ be the set of all $k$-tuples $(m_1, m_2, \ldots, m_k)$ of non-negative integer satisfying $\sum_{i=1}^{k} m_i=n$. Let $\alpha_1, \ldots, \alpha_k$ be non-negative real numbers satisfying $\sum_{k=1}^{i=1} \alpha_i = 1$. The probability distribution $p$ defined on $\Omega$ by the equation

\begin{equation*}
p(m_1, m_2, \ldots, m_k) = {n \choose m_1 \text{ } m_2 \cdots m_k } a_1^{m_1} a_2^{m_2} \cdots a_k^{m_k}
\end{equation*}

\noindent
is called the \textit{multinomial distribution} with parameters $\alpha_1, \ldots, \alpha_k$ and $n$.

\subsubsection*{Multinomial Distribution: Empirical Situation}
\noindent
An experimenter is sampling balls with replacement from an urn containing $k$ different types of balls, numbered $1 \ldots k$. If a single ball is selected from the urn, the probability that it is a ball of type $i$ $(1 \leq i \leq k)$ is equal to $\alpha_i$; thus, $(0 \leq i \leq k)$. Suppose $n$ balls are sampled, and that the experimenter records the number $m_i$ of balls of each type.

\subsubsection*{Geometric Distribution}
\noindent
Let $\Omega = \{ 1, 2, \ldots, n, \ldots \}$ be the sample space, and let $\alpha$ be a real number with $0 < \alpha < 1$. The function $p$ defined on $\Omega$ by the equation

\begin{equation*}
p(n) = \alpha ( 1-\alpha )^{n-1}
\end{equation*}

\noindent
is called the \textit{geometric distribution} with parameter $\alpha$. The function $p$ is clearly non-negative.

\subsubsection*{Geometric Distribution: Empirical Situation}
\noindent
An experimenter is drawing balls with replacement from an urn containing black and white balls. We suppose that the probability of drawing a black ball is a constant $\alpha$. The drawing is continued until the first black ball is drawn. The experimenter only records the number of the particular trial where this occurs. Thus, if the first black ball is drawn on trial $5$, the experimenter records the number $5$ as the outcome of the experiment.

\subsubsection*{Poisson Distribution}
\noindent
Let $\Omega = \{ 0, 1, 2, \ldots, n, \ldots \}$ be the sample space, and let $\lambda$ be a positive real number. The probability distribution $p$ defined on $\Omega$ by the equation

\begin{equation*}
p(k) = e^{- \lambda} \frac{\lambda^k}{k!}.
\end{equation*}

\noindent
is called a \textit{Poisson distribution} with parameter $\lambda$.

\subsubsection*{Poisson Distribution: Empirical Situation}
\noindent
An experimenter is watching a display for the appearance of signals of some kind. We suppose that these signals have the following characteristics: they are punctual; the occurrence of a signal has no effect on the occurrence of the next one; and the occurrence of the signals is random but homogeneous, in the sense that the signal is just as likely to occur within one interval of time of a given length within some other interval of the same length.

% ================================================================================================================================
% ================================================================================================================================
\section*{CHAPTER 9: Conditional Probability}

% TODO: Look into this more...

\noindent
The conditional probability of some event $A$ `given' an event $B$ is the probability that $A$ occurs when we know for sure that $B$ must also occur (or has occurred). The usual notation is $\condprob{A}{B}$. Let $(\Omega, \field, \mathbb{P})$ be a finite additive probability space. For all events $A,B$ such that $\prob{B} \neq 0$, we define

\begin{equation*}
\condprob{A}{B} = \frac{\prob{A \cap B}}{\prob{B}}.
\end{equation*}

\subsection*{Some Consequences of Conditional Probability}
\noindent
\textbf{Theorem}: Suppose that $(\Omega, \field, \mathbb{P})$ is a finitely additive probability space. For all events $A,B,$ and $C$, such that $\prob{C} \neq 0$, we have

\begin{enumerate}[label=(\roman*)]
\item $\condprob{\Omega}{C} = 1$;
\item $\condprob{A}{C} \geq 0$;
\item if $A \cap B \neq \emptyset$, then $\condprob{A \cup B}{C} = \condprob{A}{C} + \condprob{B}{C}$;
\item if $A \cap B \neq \emptyset$, then $\condprob{A}{C} = 0$;
\item if $\prob{A} \neq 0$, then $\condprob{A}{C} = \frac{\prob{A} \condprob{C}{A}}{\prob{C}}$.
\end{enumerate}

\noindent
\textbf{Theorem}: Suppose that $(\Omega, \field, \mathbb{P})$ is a finitely additive probability space. Let $C$ be some event such that $\prob{C} \neq 0$. For any $A \in \field$, define $\mathbb{P}_C(A) = \condprob{A}{C}$. Then $(\Omega, \field, \mathbb{P}_C)$ is a finitely additive probability space.

% What is the proof for this?
\noindent
When we condition by some event $C \in \field$, with $C \neq \Omega$, it is almost as if we were restricting consideration to a smaller sample space $C \subseteq \Omega$. For example, suppose that $A \cap C$ and $B \cap C$ are two incompatible events, and that $\prob{C} \neq 0$. It is not difficult to show that

\begin{equation*}
\condprob{A \cup B	}{C} = \condprob{A}{C} + \condprob{B}{C},
\end{equation*}

\noindent
which is essentially the defining additivity property of probabilities holding for smaller sample space $C$.

\subsection*{Theorem of Total Probabilities}
\noindent
The idea here is that the probability of an event $A$ can in many cases be decomposed additively in terms of the probabilities of other events that \textit{cover} $\Omega$. The idea is to decompose the probability of $A$ through the sum

\begin{equation*}
\prob{A} = \prob{A \cup H_1} + \prob{A \cup H_2} + \ldots + \prob{A \cup H_n} .
\end{equation*}

\noindent
Note that $\condprob{A}{H_i} \prob{H_i} = \prob{A \cap H_i}$ for $1 \leq i \leq n$. For events $E_1, E_2, \ldots, E_n$ that are pairwise incompatible, the notation $\sum_{i=1}^{n} E_i$ has the same meaning as the notation $\bigcup_{i=1}^{n} E_i$. Thus, we can more formally define the \textit{Theorem of Total Probabilities}. \\

\noindent
\textbf{Theorem}: Let $(H_i)_{1 \leq i \leq n}$ be a family of pairwise incompatbile events in a finitely additive probability space $(\Omega, \field, \mathbb{P})$, such that $\sum_{i=1}^{n} H_i = \Omega$. Then, for all $A \in \field$,

\begin{equation*}
\prob{A} = \sum_{i=1}^{n} \prob{A \cap H_i}.
\end{equation*}

\noindent
If $\prob{H_i} \neq 0$, for all $i$, $1 \leq i \leq n$, then

\begin{equation*}
\prob{A} = \sum_{i=1}^{n} \condprob{A}{H_i} \prob{H_i}.
\end{equation*}

\subsection*{Remarks}
\noindent
The theorem presented in this section represents results based on a finite set of events. Versions of this theorem also hold in the cases of countable and uncountable collections of events $H_i$, but the probabilities of joint events $A \cap E_i$ are replaced by a joint density function.

% ================================================================================================================================
% ================================================================================================================================
\section*{CHAPTER 10: Independence \& Bayes' Theorem}
\noindent
In a finite additive probability space $(\Omega, \field, \mathbb{P})$, two events $A,B$ are independent if

\begin{equation*}
\prob{A \cap B} = \prob{A}\prob{B}.
\end{equation*}

\noindent
An event $A$ is \textit{independent} of some event $B$ when knowing that $B$ is realized does not affect the probability of $A$. It is not required that $\prob{A}, \prob{B} > 0$. However, if $\prob{A} = 0$, then $\prob{A \cap B} = 0$, since $A \cap B \subseteq A$, and $\prob{A \cap B} = \prob{A}\prob{B} = 0$. An event of probability zero is thus independent of any other event. Suppose that $A$ and $B$ are two independent events in a finite additive probability space $(\Omega, \field, \mathbb{P})$. Then,

\begin{enumerate}[label=(\roman*)]
\item $\overline{A}$ and $B$ are independent;
\item if $\prob{B} \neq 0$, then $\prob{A \lvert B} = \prob{A}$.
\end{enumerate}

\subsection*{Bayes Theorem}
Let $(H_i)_{1 \leq i \leq n}$ be a family of pairwise incompatible events in a finitely additive probability space $(\Omega, \field, \mathbb{P})$, satisfying $\bigcup_{i=1}^{n} H_i = \Omega$ and $\prob{H_i} \neq 0$, for all $i, 1 \leq i \leq n$. If $D \in \field$ is any event such that $\prob{D} \neq 0$, then, for $1 \leq i \leq n$,

\begin{equation*}
\prob{H_i \lvert D} = \frac{\prob{D \lvert H_i}\prob{H_i}}{\sum_{j=1}^{n} \prob{D \lvert H_j}\prob{H_j}}.
\end{equation*}

\subsection*{Remarks}
\noindent
The events $H_i$ are called \textit{hypotheses} and $\prob{H_i}$ is called \textit{a priori probability} of hypothesis $H_i$. The conditional probability $\condprob{H_i}{D}$ is referred to as the \textit{a posteriori probability} provided by the realization of the event $D$. Suppose that some experiment has been performed, and some `data' (the event $D$) have been collected. A number of alternative `theories' are considered. An intuitively appealing question is: what are the `respective probabilities' of the various theories, given the data? Bayes Theorem suggests that such `probabilities' might perhaps be recomputed from the a priori probabilities of these hypotheses via the system of equations for $1 \leq i \leq n$. For instance, in what sense can we talk about the `probability' of a theory?
\end{document}


\subsection{Bias and variance of the Robinson estimator}

robinson: can't have confounded in dummy. but can in real. general result of propensity stuff?

Framing: Partialling out is an alternative to OLS where \(n<<p\) doesn't hold. alterntive to LASSO etc

\(\hat \theta \approx N(\theta, V/n)\)

\(V=(E[\hat D^2)^{-1}E[\hat D^2\epsilon^2 ](E[\hat D^2])^{-1}\)

These are robust standard errors.

\subsubsection{Moments of the Robinson estimator}

If IID then

\(Var (\hat \theta) =\dfrac{\sigma^2_\epsilon }{\sum_i(x_i-\hat X_i)^2}\)

Otherwise, can use GLM

What are the properties of the estimator?

\(E[\hat \theta ]=E[\dfrac{\sum_i (X_i-\hat X_i)(y_i-\hat y_i)}{\sum_i(x_i-\hat X_i)^2}]\)

\subsection{Non-linear treatment effects in the Robinson estimator}

Page on reformulating as non-linear. can do it. show can be estimated using arg min
https://arxiv.org/pdf/1712.04912.pdf

\subsection{DML}

in DML. page on orthogonality scores, page on constructing them; page on using them to estimate parameters (GMM)

We have \(P(X)=f(\theta , \rho)\)
\(\hat \theta = f(X, n)\)
\(\theta = g(\rho , X)\)

So error is:
\(\hat \theta - \theta=f(X, n)-g(\rho , X)\)

Bias is defined as:
\(Bias(\hat \theta, \theta ) = E[\hat \theta - \theta]=E[\hat \theta ] - \theta \)
\(Bias = E[\hat \theta - \theta]=E[f(X, n)-g(\rho , X)]\)
\(Bias = E[\hat \theta - \theta]=E[f(X, n)]-g(\rho ,X)\)

double ML: regression each parametric parameter on ML of other variables.
eg: get \(e(x|z)\)
\(e(d|x)\)
\(d=m(x)+v\)
\(d\) is correlated with \(x\) so bias.
\(v\) is corrleated with \(d\) but not \(x\). use as "iv".
Still need estimate for \(g(x)\).

for iterative, process is:
+ estimate \(g(x)\)
+ plug into other and estimate theta
+ this section should be in sample splitting. rename iterative estimation. separate pages for bias, variance
+ how does this work?? paper says random forest regression and OLS. intialise \(\theta \) randomly?
+ page on bias, variance, efficiency?
+ page on sample splitting, why?

+ page on goal: \(x\) and \(z\) orthogonal for split sampling
+ page on \(X=m_0(Z)+\mu\), first stage machine learning, synthetic instrumental variables? h3 on that for multiple variables on interest. regression for each

\subsection{DML1}
Divide into \(k\).

For each do ML on nuicance (how???) use all instances outside of sample

Then do GMM using orthogonality condition to calculate \(\theta \). (how??) use instances in sample

Average \(\theta \) from each class

\subsection{Last stage Robinson}

Separate page for last stage: note we can do OLS, GLS etc with choice of \(\Omega \).