\SecDef{longtrail}{The Long-Trail Strategy}

Linear and differential cryptanalysis are powerful methods of attacking block ciphers. It became a standard for new designs to be accompanied with arguments for security against linear and differential cryptanalysis.

The goal of linear and differential cryptanalysis is to find a \emph{distinguisher} of a cryptographic function.

\begin{definition}[Linear and Differential Distinguishers]
Let $E\colon \field{n} \to \field{n}$.

A pair $\alpha_{in},\alpha_{out}\in \field{n}$ is called a \emph{linear distinguisher} of $E$ with linear correlation
\eq{
& \LC{E}(\alpha_{in}, \alpha_{out}) \eqdef 2^{-n} \LAT{E}(\alpha_{in}, \alpha_{out}), \\
\text{if}~ &\pabs{\LC{E}((\alpha_{in}, \alpha_{out})} \gg 2^{-n/2}.
}

A pair $\alpha_{in},\alpha_{out}\in \field{n}$ is called a \emph{differential distinguisher} of $E$ with differential probability
\eq{
&\DP{E}(\alpha_{in}, \alpha_{out}) \eqdef \DDT{E}(\alpha_{in},\alpha_{out}), \\
\text{if}~ &\DP{E}(\alpha_{in}, \alpha_{out}) \gg 2^{-n}.
}

For a keyed permutation $E_k(x)\colon \field{n} \times \field{\kappa} \to \field{n}$, a pair $\alpha_{in},\alpha_{out}\in \field{n}$ is a linear/differential distinguisher if for a large enough fraction of keys $k \in \field{\kappa}$, $(\alpha_{in}, \alpha_{out}$ is a linear/differential distinguisher of $E_k$.
\end{definition}

Cryptographic functions are in most cases built in an iterated way.  Intermediate values are analyzed and included in the distinguisher. The iterations of the round function are \emph{assumed to be independent} and linear/differential distinguishers of each round are linked in a chain, called \emph{a trail}.

\newcommand\vf{\mathbf{f}}
\newcommand\valpha{\boldsymbol{\alpha}}

\begin{definition}[Linear and Differential trail]
$ $\newline
%Let $\vf = (f_1,\ldots,f_r)$, $f_i\colon \field{n} \times \field{\kappa} \to \field{n}$ and for $k \in \field{\kappa}$ let $f_i^k\colon \field{n} \to \field{n}$ be such that $f_i^k(x) = f_i(x, k)$.
Let $\vf = (f_1,\ldots,f_r)$, $f_i\colon \field{n} \to \field{n}$.
%Let \emph{trace} of $x \in \field{n}$ with respect to $\vf$ and $\vk = (k_1, \ldots, k_r), k_i \in \field{\kappa}$ be defined as
% \eq{
% &\Tr_{\vf}^{\vk} \colon \field{n} \to (\field{n})^{r+1},\\
% &\Tr_{\vf}^{\vk}(x) \eqdef \proundd{
% x, f_1^{k_1}(x), f_2^{k_2}\circ f_1^{k_1}(x), \ldots, f_r\circ f_{r-1} \circ \ldots\circ f_2 \circ f_1(x)
% }.
% }
A \emph{trail} over $\vf$ is a sequence $\valpha$ of $r+1$ vectors:
$$
\valpha = (\alpha_0, \ldots, \alpha_r), \alpha_i \in \field{n}.
$$

The \emph{expected differential probability} of the trail $\valpha$ is defined as
\eq{
\EDP{\vf}(\valpha) \eqdef
    \prod_{i = 1}^r \DP{f_i}(\alpha_{i-1}, \alpha_i).
}

The \emph{expected linear correlation} of the trail $\valpha$ is defined as
\eq{
\ELC{\vf}(\valpha) \eqdef
    \prod_{i = 1}^r \LC{f_i}(\alpha_{i-1}, \alpha_i).
}
\end{definition}

\Todo{motivation: trails add up to differential/linear distinguisher (with independent rounds, e.g. with markov assumption}

In order to ensure that a cryptographic primitive is secure against trail-based differential/linear cryptanalysis, it is necessary to prove an upper-bound of the maximum $\EDP{}$ and $\ELC{}$ among all trails.

\begin{definition}
Let $\vf = (f_1,\ldots,f_r)$, $f_i\colon \field{n} \to \field{n}$.

The \emph{maximum expected differential trail probability} of $\vf$ is denoted $\MEDP{\vf}$ and is equal to:
$$
\MEDP{\vf} \eqdef \max_{\valpha \in (\field{n})^{r+1}, \valpha \ne 0} \EDP{\vf}(\valpha).
$$

The \emph{maximum expected linear trail correlation} of $\vf$ is denoted $\MELC{\vf}$ and is equal to:
$$
\MELC{\vf} \eqdef \max_{\valpha \in (\field{n})^{r+1}, \valpha \ne 0} \ELC{\vf}(\valpha).
$$
\end{definition}

\SubSecDef{widetrail}{The Wide-Trail Argument}

The wide-trail strategy is the main method of proving an upper bound on the $\MEDP{}$ and $\MELC{}$ of a cryptographic primitive. It was introduced by Daemen and Rijmen~\cite{WideTrail} and was used to argue about the security of AES against linear and differential attacks.

I describe the argument for the differential trail cryptanalysis, the linear case is completely analogous.

Consider an SPN structure and a trail $\valpha$ with a nonzero $\MEDP{}$. Any difference propagates through the linear layer of the structure with probability 1. Furthermore, a zero difference propagates through an S-Box to a zero difference with probability 1. It follows that the $\MEDP{}$ of the trail depends only on the differential probabilities of S-Boxes with nonzero input/output differences in the trail. Such S-Boxes are called \emph{active} S-Boxes.

The idea of the wide-trail strategy is to prove a lower bound on the number of active S-Boxes in a trail. Then, the differential uniformity of the S-Box is used to obtain an upper bound on the expected differential probability of a trail, i.e. the $\MEDP{}$. This is done by simply raising the minimum differential probability of the S-Box to the power of the minimum number of active S-Boxes.
The first step is usually done by proving strong diffusion properties of the linear layer. For example, the MixColumns operation in the AES has branch number 5 and this already proves that every 2 rounds of AES have at least 5 active S-Boxes. The second step suggests that an S-Box with a low differential uniformity (and low linearity) should be used.

Assume that we want to design an ARX-based block cipher with provable security against linear and differential trail-based cryptanalysis. We can use an existing ARX-based block cipher with a small block as a (keyed) S-Box. We then have to use the $\MEDP{}$ of the small block cipher instead of the differential uniformity. Indeed, for small block sizes, the $\MEDP{}$ can be obtained for example using the Matsui search algorithm~\cite{MatsuiAlgo}. This evaluation was performed by Biryukov~\etal{} in~\cite{BVL16} for the block ciphers \speck{32} up to \speck{64}. See~\TabRef{speckey-bounds} for the results on 32-bit block size.

\begin{remark}
In order to justify the assumption of independent rounds in trails, the authors of~\cite{BVL16} consider \speckey{}, a slightly modified variant of \speck{32}. The only difference is that, in \speckey{}, the round keys are added to the whole state. In this way, the independence assumption is lifted from the block cipher structure to the key schedule. In \sparx{}, we used \speckey{} in order to have better justified provable security.
\end{remark}

\begin{table}[ht]
    \setlength{\tabcolsep}{4pt}
    \footnotesize
    \begin{center}
        \begin{tabular}{c|rrrrrrrrrrr}
            \toprule
            $r$    &  $1$ &  $2$ &  $3$ &  $4$ &  $5$ &  $6$ &  $7$ &  $8$ &  $9$ & $10$ & \\
            \midrule
            $\MEDP{}$ & $-0$ & $-1$ & $-3$ & $-5$ & $-9$ & $-13$ & $-18$ & $-24$ & $-30$ & $-34$ & \\
            $\MELC{}$ & $-0$ & $-0$ & $-1$ & $-3$ & $-5$ &  $-7$ &  $-9$ & $-12$ & $-14$ & $-17$ & \\
            \bottomrule
        \end{tabular}
    \end{center}
    \TabDef{speckey-bounds}{$\protect\MEDP{}$ and $\protect\MELC{}$ of \speck{32} / \speckey{} ($\log_2$ scale); $r$ is the number of rounds.}
\end{table}

Consider using 1 round of \speck{32} as the keyed S-Box. Note that it has a differential with probability $1=2^{-0}$. Therefore, the bound on $\MEDP{}$ obtained from the wide-trail argument will be trivial, i.e. $\MEDP{} \le 1$.

Now consider using 3 rounds of \speck{32} as the keyed S-Box $A$. Assume that we design a block cipher $E$ with 128-bit block, i.e. with 4 parallel \speck{32}-based S-Boxes. Assume that the linear layer is a $4\times 4$ MDS matrix over $\fielde{32}$, i.e. it has branching number 5. Then at least 5 S-Boxes are active every two rounds and each S-Box has $\MEDP{A} = 2^{-3}$. It follows that for the $r$ round block cipher $E_r$, the wide-trail argument provides bound $\MEDP{E_r} \le (2^{-3})^{5r/2}$. In order to get $\MEDP{E_r} \le 2^{-128}$, we need $r \ge 128/7.5 \approx 17.07$. Therefore, at least 18 rounds of SPN are needed, i.e. 54 rounds of \speck{32} repeated four times in parallel. Such a block cipher would be very inefficient.

Using the novel \emph{long-trail} strategy, we show that it is possible to build much more efficient block ciphers with ARX-based S-Boxes and provable security against linear and differential trail-based cryptanalysis.

\SubSecDef{longtrailsub}{The Long-Trail Argument}

Observe that in the ARX-based block ciphers the $\MEDP{}$ grows slower at the first few rounds and grows faster afterwards. For example, the $\MEDP{}$ of the 10 round \speck{32} is $2^{-34}$, which is much less than the $\MEDP{}$ of the 5 round \speck{32} squared: $(2^{-9})^2 = 2^{-18}$. The wide-trail strategy does not exploit this fact and uses the worse bound. Indeed, in general, the better bound can not be used, because the 10 rounds of \speck{32} are not always isolated inside the trail structure. Therefore, each concrete trail structure must be analyzed separately. We call \emph{a long trail} such an isolated chain of (keyed) S-Boxes.

\newcommand\LT[1]{\mathsf{LT}({#1})}
\begin{definition}[Long Trail]
Consider an SPN-based block cipher and a fixed trail $\valpha$. A \emph{long trail (LT)} is a chain of active S-Boxes in the trail interleaved with key additions, such that no difference comes into the chain from outside (i.e., the linear layers do not mix in differences into the chain).

Consider a partition of active S-Boxes in the trail into long trails. The multiset of lengths of long trails in any such partition is called a \emph{long trail decomposition} of the trail $T$, denoted $\LT{\valpha}$.
\end{definition}

\begin{proposition}[Long-Trail Bound]
\PropLabel{longtrailbound}
Let $\vf$ be round function of an SPN-based block cipher with an S-Box $S$ and let $\valpha$ be a trail over $\vf$. Then
\eq{
&\EDP{\vf}(\valpha) \le \prod_{r^{(m)} \in \LT{\valpha}} \pround{\MEDP{S^r}}^m,\\
&\ELC{\vf}(\valpha) \le \prod_{r^{(m)} \in \LT{\valpha}} \pround{\MELC{S^r}}^m,
}
where $r^{(m)}$ means that element $r$ repeats $m$ times in the multiset $\LT{\valpha}$, and $\MEDP{S^r}$ (resp. $\MELC{S^r}$) denote the $\MEDP{}$ of $r$ rounds of $S$ (resp. $\MELC{}$).
\end{proposition}

\begin{proof}
Recall that in the definition of $\EDP{}$ and $\ELC{}$ all rounds are considered independent. Therefore, all S-Boxes are independent as well. Hence, $\EDP{\vf}(\valpha)$ is a product of some $\DDT{}$ entry of each S-Box (depending on the trail $\valpha$). The proposition simply replaces a subset of these factors by the upper bound on their product, which does not depend on the exact trail $\valpha$, only on the fact that it is a non-zero trail. The same reasoning applies to the case of linear trails.
\end{proof}

This proposition gives an idea of improving a bound on $\MEDP{}$ and $\MELC{}$ of a block cipher. Instead of enumerating all valid \emph{exact} trails, we only need to enumerate all valid \emph{truncated} trails telling whether each S-Box is active or not. For each such trail, we need to obtain a preferably optimal long-trail decomposition, which leads to an upper bound on $\EDP{}$ or $\ELC{}$ of all exact trails fitting the current truncated trail. By taking the maximum bound among all truncated trails, we obtain an upper bound on $\MEDP{}$ and $\MELC{}$ of the block cipher.

For the sake of completeness, I express the wide-trail bound in the same way to highlight that it is a special case of the long-trail bound. Indeed, the long-trail partition of any trail into chains of length 1 is equivalent to counting the number of active S-Boxes. This, in turn, requires less information about each trail and allows to obtain a simple mathematical argument. On the contrary, the long-trail bound requires algorithmic evaluation.

\begin{proposition}[Wide-Trail Bound]
Let $\vf$ be round function of an SPN-based block cipher with an S-Box $S$ and let $\valpha$ be a trail over $\vf$. Then
\eq{
&\EDP{\vf}(\valpha) \le \prod_{r^{(m)} \in \LT{\valpha}} \pround{\MEDP{S}}^{rm},\\
&\ELC{\vf}(\valpha) \le \prod_{r^{(m)} \in \LT{\valpha}} \pround{\MELC{S}}^{rm}.
}
\end{proposition}

\SubSecDef{alg-decomposition}{An Algorithm for Long-Trail Decomposition}

The most straightforward way to apply the long-trail argument to bound the $\MEDP{}$ and $\MELC{}$ of a cipher is as follows:
\begin{enumerate}
    \item enumerate all possible truncated trails composed of active/inactive S-boxes;
    \item find an optimal decomposition of each trail into long trails (LT);
    \item bound the probability of each trail using the product of the $\MEDP{}$ (resp. $\MELC{}$) of all active long trails i.e. by applying the Long Trail Argument (see~\PropRef{longtrailbound});
    \item the maximum bound over all trails is the final upper bound.
\end{enumerate}

Note that this approach is feasible only for a small number of rounds, because the number of truncated trails grows exponentially.

In this section, I sketch an algorithm for the only non-trivial step, step $(2)$, i.e. an algorithm for finding an optimal decomposition of a given truncated trail into long trails.

First, note that the trail can be represented as a graph, where nodes are active S-Boxes and an edge corresponds to a possible connection of two S-Boxes in a long trail. Moreover, this graph is a forest. Indeed, an S-Box can't receive two edges from the previous round, because it contradicts a definition of long trail - there must be a single difference coming in. For each tree in the forest, we choose the root to be the S-Box from the earliest round, which is determined uniquely by the same reason. Then, for any node its children may only be in the next round.

The goal then is to cover all nodes with disjoint ``vertical'' paths, such that the product of the paths' probabilities is minimal. By the path probability we understand the respective long trail's probability. The simplest (and the worst) solution is to choose paths consisting of single nodes. Note that this solution already gives some upper bound and by finding a better decomposition we improve this bound.

I propose an algorithm based on recursive dynamic programming approach. For each node, we recursively solve the sub-problem for the subtree rooted at that node. However, we need to compute some additional information apart from the best decomposition of the subtree. Consider the optimal decomposition of the whole forest into such paths and consider the long trail which goes through the current subtree's root. Clearly, if we fix this long trail, the rest of the subtree becomes completely independent and has to be decomposed optimally. Therefore, from the subtree we need to know only the probability of this decomposition and the length of the long trail's part in the subtree. We don't know the optimal length beforehand, therefore we store the best probabilities for all possible lengths. Another view on this is that we group all possible subtree decompositions by length of the long trail which goes through the subtree root and for each such length we greedily choose the minimum probability. Then, when we obtain such tables for all children of some node, we can easily compute the table for the node itself - we check all possible ways to choose a child of the node and the length of the long trail which goes through the child and we try to join the current node to that long trail. Then the corresponding probability is the product of the best probabilities of the other children with the probability corresponding to the children's long trail and the probability stored in the children's table respectively for that length.

\paragraph{Complexity.}
The complexity is dominated by computing the table for each node. One of the $w$ children has to be selected for the continuation of the trail, and the size of its child's table is limited by the number of rounds $r$.
Therefore, each node's contribution to the complexity is at most $\OO(wr)$. The total complexity of the algorithm then is $O(w^2r^2)$, where $w$ is the number of S-Boxes in parallel, and $r$ is the number of rounds. Note that $wr$ corresponds to the total number of S-Boxes in the cipher.

Despite the reasonable efficiency of the algorithm, the amount of all truncated trails for which the algorithm has to be run adds a large factor to the complexity of the evaluation of a block cipher. In the next section, I will describe an algorithm which completes the whole evaluation in a much more efficient way, under a special condition on the linear layer.

\SubSecDef{alg-special}{Efficient Algorithm for Special Linear Layers}

The most complicated step in the above procedure is finding an optimal decomposition of a given truncated trail into long trails. The difficulty arises from the so-called \emph{branching}: situation in which a long trail may be extended in more than one way. The definition of long trail relies on the fact that there is no linear transformation on a path between two S-Boxes in a long trail. Therefore, branching happens only when some output word of the linear layer receives two or more active input words without modifications.

In order to cut off the branching effect (and thus to make finding the optimal decomposition of a long trail trivial), we can put some additional linear functions that will modify the contribution of some of the input words. Equivalently, when choosing a linear layer we simply do not consider layers which cause branching of long trails. As we will show later, this restriction has many advantages.

To simplify our study of the linear layer, we introduce a matrix representation for it. In an SPN-based block cipher operating on $w$ words, the linear layer may be expressed as a $w\times w$ block matrix. We will denote the zero and the identity sub-matrices by $0$ and $1$ respectively and an unspecified (arbitrary) sub-matrix by $L$. This information is sufficient for analyzing the high-level structure of a cipher. Using this notation, the linear layers to which we restrict our analysis have matrices in which each column has at most one element $1$.

For the special subset of linear layers outlined above, I present an algorithm for obtaining $\MEDP{}$ and $\MELC{}$ bounds, based on a dynamic programming approach. Since there is no branching, any truncated trail consists of disjoint sequences of active S-Boxes. We can treat each such sequence as a long trail to obtain an optimal decomposition. More importantly, because of this simplification, we can avoid enumerating all trails by grouping them in a particular way.

We proceed round by round and maintain a set of best truncated trails up to an equivalence relation, which is defined as follows. For all S-Boxes at the current last round $s$, we assign a number, which is equal to the length of the long trail that covers this S-Box, or zero if the S-Box is not active. We say that two truncated trails for $s$ steps are equivalent if the tuples consisting of those numbers (lengths of long trails) are the same for both truncated trails. This equivalence captures the possibility to replace some prefix of a trail by an equivalent one without breaking the validity of the trail or its LT decomposition. The total probability, however, can change. The key observation is that from two equivalent trails we can keep only the one with the highest current probability. Indeed, if the optimal truncated trail for all $r$ rounds is an extension of the trail for $s$ rounds with lower probability, we can take the first $s$ rounds from the trail with higher probability without breaking validity and obtain a better trail, which contradicts the assumed optimality.

The pseudo-code for the algorithm is given in~\AlgRef{special}.
Note that in the case of the $\MELC{}$ bound, the matrix of the linear layer has to be inverted and transposed. However, instead of inversion, we can build up the trails in the reverse direction: from the ciphertext side to the plaintext side. In this way, it is sufficient to only transpose the linear layer.

\FigTex{alg-special.tex}

\paragraph{Complexity.}
The complexity of the algorithm can be upper-bounded as follows. The size of the set $S_i$ is upper-bounded by the number of all $w$-tuples of integers in $\seg{0}{i}$, i.e. $(i+1)^w$. Generating extensions of an element $s \in S_i$ requires $w2^w$ operations. Repeating this for $r$ rounds results in complexity $\OO(r \cdot w 2^w \cdot (r+1)^w)$. In practice, only a small subset of all possible $w$-tuples is possible. Note that this algorithm implicitly already performs the enumeration of all truncated trails and therefore, this is the complexity of the full evaluation of the $\MEDP{}$ and $\MELC{}$ of the block cipher.


\section{Information Theory and Dimensionality Reduction}
The following section provides a short introduction into information theory and optimal coding.
For a more comprehensive introduction it is recommended to look into
\cite[Chapter 6 \& 7]{Applebaum2008}. In the previous section we have talked about
model comparison and we will follow this conception from an information theoretic
point of view.
We have earlier used the likelihood as a measure for model comparison and we will use it now
to introduce an information theoretic measure, the cross-entropy. Strictly speaking, the
likelihood function maps a set of possible models to a real value, given an observed data
set. We recognize that the likelihood function is not a probability density function, as it is
not normalized.

\begin{proposition}[Cross-entropy]
Given: We have observed a set of data point $D$ generated from a random variable $X \sim \rho(x)$, with
$\rho(x)$ being unknown to us, and we want to represent the data using the model $\hat{\rho}(x)$.
The probability of i.i.d. samples $\{x_1, x_2, x_2, \dots x_N\}$
under the model $\hat{\rho}(x)$ is given by
\begin{align*}
	p(D|\hat{\rho}) = \prod_{n=1}^N \hat{\rho}(x_n)
\end{align*}
and
\begin{align*}
	\log L(D;\hat{\rho}) = \log p(D|\hat{\rho}) = \sum_{n=1}^N \log \hat{\rho}(x_n)
\end{align*}
is the log-likelihood. We find the empirical expectation of a function by using the normalized
sum over all observations, such that for the expected log-likelihood we find
\begin{align}
	\begin{split}
	   \Exest{}{\log \hat{\rho}(x)} &= \frac{1}{N} \sum_{n=1}^N \log \hat{\rho}(x_n) \\
	   							     &= \Ex{\rho_{emp}(x)}{\log \hat{\rho}(x)}
	\end{split}
\end{align}
where
\begin{align*}
	\rho_{emp}(x) = \frac{1}{N} \sum_n \delta(x - x_n)
\end{align*}
is the empirical distribution of $x$. That is, the empirical expectation of the log-likelihood
is an expectation with respect to the empirical distribution $\rho_{emp}(x)$. We notice that
if the number of observations approaches infinity the empirical distribution is equal to
$\rho(x)$:
\begin{align*}
	\lim_{N \rightarrow \infty} \rho_{emp}(x) = \rho(x)
\end{align*}
Following this observation the expected negative log-likelihood for infinitely many observations
can be written as:
\begin{align*}
	\lim_{N \rightarrow \infty} \left( \Ex{\rho_{emp}(x)}{- \log \hat{\rho}(x)} \right)
				&= \Ex{\rho(x)}{- \log \hat{\rho}(x)}
\end{align*}
where
\begin{align}
	\Ex{\rho(x)}{- \log \hat{\rho}(x)} \stackrel{\text{cont.}}{:=} - \int_{X}\rho(x) \log \hat{\rho}(x) \, \mathrm{d}x
\end{align}
is called the cross-entropy of using probability distribution $\hat{\rho}(x)$ to represent data
from the true distribution $\rho(x)$. For the \emph{discrete case} the cross-entropy is given by
\begin{align}
	\Ex{p(x)}{- \log p(x)}
		&\stackrel{\text{disc.}}{:=} - \sum_{x \epsilon X} p(x) \log \hat{p}(x)
\end{align}
In information theory the cross-entropy is a measure of the expected code length needed to respresent
events from the probability distribution $\rho(x)$ using another probability distribution
$\hat{\rho}(x)$. We will later see how to understand this definition.
\end{proposition}

%The cross-entropy
%is a measure of the expected information needed to identify an event $A$ from the
%sample space $S$ given we code random variable $X$  by $\hat{\rho}(x)$ rather than $\rho(x)$.
%For the \emph{continuous case} we define the cross-entropy as
%	\begin{align}
%		\mathrm{E}_{\rho(x)}\left[-\log \rho (x) \right]
%			&\stackrel{\text{cont.}}{:=} - \int_{X}\rho(x) \log \hat{\rho}(x) \, \mathrm{d}x
%	\end{align}
%where
%	\begin{align}
%		\mathrm{I}(A) = -log_b\rho(A)
%	\end{align}
%is called the information content of an event $A$. Due to the monotonicity of the logarithm,
%we see that $\mathrm{I}(A) \rightarrow 0$ for $\rho(A) \rightarrow 1$. That is, information can be seen
%as a measure of 'surprise'. If an event if very likely it's information content is low, whereas
%if an event is very unlikely it's information content is  high.
%It should be noted that the logarithm is taken to the
%base $b$ dependent on the coding scheme which defines the unit of information. $b = 2$ for example
%corresponds to a coding scheme in \emph{bits} and $b = \mathrm{e}$ corresponds to a coding scheme
%in \emph{nats}. Similar to the continuous case we can measure the cross-entropy for the
%\emph{discrete case} using
%	\begin{align}
%	 		\mathrm{E}_{p(x)}\left[-\log p (x) \right]
%			&\stackrel{\text{disc.}}{:=} - \sum_{x \epsilon X} p(x) \log \hat{p}(x)
% 	\end{align}
%In general the cross-entropy is a measure of the expected code length needed to respresent
%events from the probability distribution $\rho(x)$ using another probability distribution
%$\hat{\rho}(x)$.
%\end{proposition}
%
%Training set for different classes
%
%$c = 0, \dots, 9 $
%
%$\hat{\rho}(\hat{x} | c) =
%      \mathcal{N}(\underbrace{\mathbf{U}_c\TT \mathbf{x}}_{\mathbf{s}} |
%                  \boldsymbol \mu_c, \sigma_{1c}^2	, \dots, \sigma_{dc}^2)$
%
%The log-likelihood of the data is equal to tha negative cross-entropy of the model given the data.
%
%Test sets:
%$\rho(\mathbf{x} | c^*) =\frac{1}{N} \sum_{\mathbf{x_k} \, \epsilon \, \text{testset}} \delta(\mathbf{x} - \mathbf{x_k})$
%
%$ \Rightarrow \text{cross-entropy} \hat{=}
%    - \sum_{\mathbf{x_k} \, \epsilon \, \text{testset*}} \log \hat{\rho} (\mathbf{x_k}|c)$
%
%$\mathrm{E}_{\rho_{emp}} \left[ f(\mathbf{x}) \right] = \frac{1}{N} \sum_{k = 1}^N f(\mathbf{x_k}) $
%
%exp(neg cross-entropy) $:=$ likelihood of the model for the entire test set

\subsection{Consider the best model}
We have learned earlier that the true model $\rho(x)$ has largest likelihood
given the data is generated from $\rho(x)$. That is, if the expected log-likelihood is maximized
then the expectation of the negative log-likelihood will be minimal such that
\begin{align}
   &\lim_{N \rightarrow \infty} \left( \frac{1}{N} \sum_{k = 1}^N -\log \rho(\mathbf{x_k}) \right)
   = - \int \rho (\mathbf{x}) \log \rho(\mathbf{x}) \mathrm{d}\mathbf{x}
\end{align}
is minimal for the true generating model $\rho(x)$. We call
\begin{align}
	  \mathrm{h}[X] &\stackrel{\text{cont.}}{:=} - \int \rho (\mathbf{x}) \log \rho(\mathbf{x}) \mathrm{d}\mathbf{x}
\end{align}
the \emph{differential entropy} of a continuous random variable X with
$\mathrm{h}[X] \, \epsilon \, \mathbb{R}$. An eqivalent definition for the case of discrete
random variables has the form
\begin{align}
	\mathrm{H}[x] = - \sum_{k=1}^K p(x_k) \log p(x_k)
\end{align}
with $\mathrm{H}[x] \, \epsilon \, \mathbb{R}^+$. This is called the \emph{discrete entropy}.
The entropy measures the minimal discription length, or the most compact description, of a random
variable. The description length of a random variable will be minimal if the code which describes
the random variable comes from true distribution of that random variable. The negative log-probability
of an observation approximates the code word length for describing that observation.

\subsection{Information}
In information theory the negative log probabilty of an event $A$ under a model $\rho(x)$
	\begin{align}
		\mathrm{I}(A) = -log_b\rho(A)
	\end{align}
is called the information content of an event $A$. Due to the monotonicity of the logarithm,
we see that $\mathrm{I}(A) \rightarrow 0$ for $\rho(A) \rightarrow 1$. That is, information can be seen
as a measure of 'surprise'. If an event if very likely it's information content is low, whereas
if an event is very unlikely it's information content is  high.
It should be noted that the logarithm is taken to the
base $b$ dependent on the number of symbols in the alphabet which defines the unit of information.
$b = 2$ for example corresponds to a binary code which is represented in \emph{bits}, whereas
$b = \mathrm{e}$ corresponds to a coding scheme in \emph{nats}.

\begin{exbox}{Discrete distribution (uniform)}
Given a discrete random variable $X$ such that
\begin{align*}
	x \, \epsilon \, \{1,2,3,4\} \qquad p(x) = \frac{1}{4}
\end{align*}
Then for a binary code the entropy is given by:
\begin{flalign*}
	\mathrm{H}[p(x)] &= - \sum_{x = 1,2,3,4} p(x) \log p(x) \\
	 				 &= - \log_2 \frac{1}{4} \\
	 				 &= \log_2 4 = 2 [bits]
\end{flalign*}
Whereas for a terniary code with three symbols the entropy is given by:
\begin{flalign*}
\mathrm{H}[p(x)] &= - \sum_{x = 1,2,3,4} p(x) \log p(x) \\
	 				 &= - \log_3 \frac{1}{4} \\
	 				 &= \log_3 4 \approx 1.26 [trits]
\end{flalign*}

Binary alphabet: $p(x) = \frac{1}{8} \Rightarrow \mathrm{H}[p(x)] = 3\mathrm{bits}.$ \\
Ternary alphabet: $p(x) = \frac{1}{9} \Rightarrow \mathrm{H}[p(x)] = 2\mathrm{trits}.$ \\

\end{exbox}

\subsubsection{Optimality}
We understand optimality in the sense that for \emph{infinitely long} sequences of observations
the description length takes a minimum and that this minimum description length is tightly bounded
by the entropy such that:
\begin{align}
	\mathrm{H}[x] \leq \frac{\# \text{symbols}}{\# \text{words}} \leq \mathrm{H}[x] + 1
\end{align}
where $\frac{\# \text{symbols}}{\# \text{words}}$ is the the average code word length. This
is called "Shannon's noiseless coding theorem".

\subsection{Quantization}
Problem: The description length of a continuous random variable is infinite. However, we can introduce a quantization which divides the variable space into equally sized bins of size $\Delta$. By this we are mapping a continuous RV into a discrete space such that we can calculate the discrete Entropy $\mathrm{H}_\Delta$ under the quantization $\Delta$.

\begin{wrapfigure}{r}{0.5\textwidth}
	\centering
	\includegraphics[width=0.45\textwidth]{./lecture12/gauss_quant.pdf}
	\caption{Quantization of a normally distributed random variable.}
\end{wrapfigure}
The
\begin{align}
	{H}_\Delta[x] \approx \mathrm{h}[x] - \log \Delta
\end{align}

It can be shown that:
\begin{align}
	\mathrm{h}[x] = \lim_{\Delta \rightarrow 0} \left( \mathrm{H}_\Delta[x] + \log \Delta \right)
\end{align}

Maximum likelihood classification follows the minimum description length principle,
i.e. find the model that corresponds to the most compact description of the data.

\subsection{Joint entropy}
\begin{definition}[Joint entropy]
Discrete case:
\begin{align*}
		\mathrm{H}[X,Y] = -\sum_{x_k,y_j} p(x_k,y_j) \log p(x_k,y_j)
\end{align*}

Continuous case:
\begin{align*}
		\mathrm{h}[X,Y] = - \mathop{\int \! \! \! \int} \rho(x,y) \log \rho(x,y) \, \mathrm{d}x \mathrm{d}y
\end{align*}
\end{definition}

\subsection{Mutual Information}

\begin{align*}
	\underbrace{\mathbf{X}}_{\text{Source}} \stackrel{\text{channel}}{\longrightarrow} \underbrace{\mathbf{Y}}_{\text{Receiver}}
\end{align*}

\begin{example}[Channel coding with shared bits]
	Given: A bit code with 8 possible words
	\begin{align*}
		\mb{b} \, \epsilon \, \{0,1\}^3 = \{000,001,010,011,100,101,110,111\}
	\end{align*}
	and equal probabilities of word occurences
	\begin{align*}
		\qquad p(\mathbf{b}) = \frac{1}{8}
	\end{align*}
	Source and Receiver share one bit such that
	\begin{align*}
 		\mb{x} = \begin{pmatrix} b_1 \\ b_2 \end{pmatrix}; \qquad
		\mb{y} = \begin{pmatrix} b_2 \\ b_3 \end{pmatrix}
	\end{align*}
	The optimal code word length for source and receiver is given by
	\begin{align*}
		\mathrm{H}[X] = 2 \, \mathrm{bits}; \qquad \mathrm{H}[Y] = 2 \, \mathrm{bits}
	\end{align*}
	and for the shared code
	\begin{align*}
			\mathrm{H}[X,Y] = 3 \, \mathrm{bits}
	\end{align*}
	We see that $\mathrm{H}[X,Y] \leq \mathrm{H}[X] + \mathrm{H}[Y]$ and that
	\begin{align*}
		\mathrm{I}[X:Y] = \mathrm{H}[X] + \mathrm{H}[Y] - \mathrm{H}[X,Y] = 1 \, \mathrm{bits}
	\end{align*}
	is the number of shared bits between source $X$ and receiver $Y$. This is called the mutual
	information of $X$ and $Y$.
\end{example}

\begin{definition}[Mutual information]
\begin{align*}
		\mathrm{I}[X:Y]
		      & \stackrel{\text{cont.}}{:=} \mathrm{h}[X] + \mathrm{h}[Y] - \mathrm{h}[X,Y] \\
   		      & \stackrel{\text{disc.}}{:=} \mathrm{H}[X] + \mathrm{H}[Y] - \mathrm{H}[X,Y]
\end{align*}
\end{definition}

\begin{figure}
	\centering
	\includegraphics[width=0.45\textwidth]{./lecture12/venn.pdf}
	\caption{A Venn diagram for the relationship of information theoretic measures.}
\end{figure}

\subsubsection*{Reexpressing mutual information}
\begin{align}
\begin{split}
	\mathrm{I}[X:Y] &= \mathrm{h}[X] + \mathrm{h}[Y] - \mathrm{h}[X,Y] \\
	                &= - \int \rho (\mathbf{x}) \log \rho(\mathbf{x}) \mathrm{d}\mathbf{x}
	                   - \int \rho (\mathbf{y}) \log \rho(\mathbf{y}) \mathrm{d}\mathbf{y}
	                   + \mathop{\int \! \! \! \int} \rho(x,y) \log \rho(x,y) \, \mathrm{d}x \mathrm{d}y \\
	                &= \mathop{\int \! \! \! \int} \rho(x,y) \log \frac{\rho(x,y)}{\rho(x)\rho(y)} \, \mathrm{d}x \mathrm{d}y \\
	                &= \mathop{\int \! \! \! \int} \rho(x,y) \log \frac{\rho(x|y)}{\rho(x)} \, \mathrm{d}x \mathrm{d}y \\
	                &= \mathop{\int \! \! \! \int} \rho(x,y) \log \frac{\rho(y|x)}{\rho(y)} \, \mathrm{d}x \mathrm{d}y \\
	                &= \mathrm{h}[X] + \mathop{\int \! \! \! \int} \rho(x|y) \rho(y) \log \rho(x|y) \, \mathrm{d}x \mathrm{d}y \\
	                &= \mathrm{h}[X] - \underbrace{\int \rho(y)
	                     \left(- \int \rho(x|y) \log \rho(x|y) \, \mathrm{d}x \right) \mathrm{d}y}_{\text{conditional entropy}} \\
	                &= \mathrm{h}[X] - \mathrm{h}[X|Y] \\
	                &= \mathrm{h}[Y] - \mathrm{h}[Y|X]
\end{split}
\end{align}
The conditional entropy $\mathrm{h}[Y|X]$ is the entropy of a random variable $Y$ conditioned on
any possible outcome of random variable $X$. It corresponds to the number of additional bits
needed on average to code for $Y$ given that $X$ has been observed. It is a reflection of the uncertainty that
is introduced by the channel.

\begin{exbox}{Differential entropy of a Gaussian random variable}
	\begin{align*}
	\mathrm{h}[X] &= \frac{1}{2} \log_\mathrm{e} \left( 2 \pi \mathrm{e} \mathrm{Var}[X] \right) \text{in nats} \\
				  &= \frac{1}{2} \log_2 \left( 2 \pi \mathrm{e} \mathrm{Var}[X] \right) \text{in bits}
	\end{align*}
\end{exbox}


\chapter{Sets and Tables}

\textsc{Perspective}: Sets and tables are data aggregates that are
very useful for a number of common programming tasks.  Nevertheless,
few programming languages support these data types, with the notable
exceptions of Sail (Reiser 1976) and SETL (Dewar, Schonberg, and
Schwartz 1981). There are many reasons why these obviously useful data
types are not found in most programming languages, but perceived
implementation problems certainly rank high among them. If only for
this reason, their implementation in Icon is worth studying.

Historically, tables in Icon were inherited from SNOBOL4 and SL5. Sets
came later, as an extension to Icon, and were designed and implemented
as a class project. Although sets were a late addition to Icon, they
are simpler than tables.  Nonetheless, they present many of the same
implementation problems that tables do. Consequently, sets are
considered here first.

Sets and the operations on them support the familiar mathematical
concepts of finite sets: membership, the insertion and deletion of
members, and the operations of union, intersection, and
difference. What is interesting about a set in Icon is that it can
contain members of any data type. This is certainly a case where
heterogeneity significantly increases the usefulness of a data
aggregate without adding to the difficulty of the implementation,
\textit{per se.}

The ability of a set to grow and shrink in size influences the
implementation significantly. Efficient access to members of a set,
which is needed for testing membership as well as the addition and
deletion of members, is an important consideration, since sets can be
arbitrarily large.

Tables have more structure than sets. Abstractly, a table is a set of
pairs that represents a many-to-one relationship-a function. In this
sense, the default value of a table provides an extension of the
partial function represented by the entry and assigned value pairs to
a complete function over all possible entry values. Programmers,
however, tend to view tables in a more restricted way, using them to
tabulate the attributes of a set of values of interest. In fact,
before sets were added to Icon, tables were often used to simulate
sets by associating a specific assigned value with membership.

\section{Sets}
\subsection{Data Organization for Sets}

Hash lookup and linked lists are used to provide an efficient way of
locating set members. For every set there is a set-header block that
contains a word for the number of members in the set and slots that
serve as heads for (possibly empty) linked lists of set-element
blocks. The number of slots is an implementation parameter. In version
6 of Icon there were thirty-seven slots in table-header blocks on
computers with large address spaces but only thirteen slots on
computers with small address spaces.This worked well for small hash
tables, but performance degraded for large tables due to the long hash
chains in each slot.

Now, the hash table is segmented; each hash table starts out with a
single segment containing a fixed number of slots (typically eight),
but the number of slots doubles repeatedly as the number of segments
grows. Small hash tables benefit from reduced memory requirements,
while large tables show dramatic performance gains. The price paid is
one extra indirection to get to the lists of members. The maximum
number of segments in a set header block is a configuration parameter:
The default is six on machines with small address spaces and ten
otherwise.
{\color{blue} Unicon allows up to twenty segments, which places an
upper limit on the maximum possible number of slots (albeit quite a
large one of $slots \times 2^{19}$).
}
To reduce the size of the diagrams the maximum number of segments is
assumed to be six in the figures that follow but, even with this low
number, the maximum number of slots is $256$ ($8 + 8 + 16 + 32 + 64
+ 128$), which is still a respectable increase compared to version 6.

The structure for an empty set, produced by

\iconline{
\>s := set()
}

\noindent is

\begin{picture}(300,250)(0,-32)
\put(120,32){\wordpile{5}{\nullptrbox{}}}
\put(120,32){\blboxlabel{segment 5}}
\put(120,94){\leftboxlabels{segment 0}{segment 1}}
\put(120,112){\wordboxptr{60}{}}
\begin{picture}(0,0)(-20,32)
\put(240,94){\rightboxlabels{slot 0}{slot 1}}
\put(360,80){\vdots}
\put(240,0){\brboxlabel{slot 7}}
\put(240,0){\wordpile{8}{\nullptrbox{}}}
\put(240,128){\blkbox{slots}{40}}
\put(240,128){\brboxlabel{size}}
\end{picture}
\put(80,80){\vdots}
\put(120,128){\blkbox{\textit{id}}{7}}
\put(120,128){\brboxlabel{mask}}
\put(120,160){\blkbox{set}{0}}
\put(120,160){\brboxlabel{number of elements in the set}}
\put(0,176){\dvboxptr{set}{np}{40}{}}
\put(0,176){\tlboxlabel{\texttt{s}}}
\end{picture}

Each member of a set is contained in a separate set-element
block. When a value is looked up in a set (for example, to add a new
member), a hash number is computed from this value. The absolute value
of the remainder resulting from dividing the hash number by the number
of slots is used to select a slot.

Each set-element block contains a descriptor for its value, the
corresponding hash number, and a pointer to the next set-element
block, if any, on the linked list. For example, the set-element block
for the integer 10 is:

\begin{picture}(300,90)
\put(120,0){\dvbox{integer}{n}{10}}
\put(120,0){\trboxlabel{member value}}
\put(120,32){\wordbox{129}{}}
\put(120,32){\brboxlabel{hash number}}
\put(120,48){\nullptrbox{next set-element block}}
\put(120,64){\wordbox{selem}{}}
\end{picture}

As illustrated by this figure, the hash number for an integer is not
the value of the integer (in Icon Version 6 it was). The hash number
for an integer is the result of multiplying it by eight times the
golden ratio using fixed point arithmetic. Hash computation is
discussed in detail in Sec. 7.3.

The structures for the set

\iconline{
\ \ s := set([10,23])
}

\noindent are

\begin{picture}(300,240)(30,0)
%\put(0,0){\graphpaper{40}{23}}
%set header
\put(0,70){\begin{picture}(0,0)
\put(0,96){\blkbox{\textit{id}}{7}}
\put(0,96){\brboxlabel{mask}}
\put(0,128){\blkbox{set}{2}}
\put(0,128){\brboxlabel{number of elements in the set}}
\put(0,0){\wordpile{5}{\nullptrbox{}}}
\put(0,80){\wordboxptr{60}{}}
\end{picture}
}
%segment 0
\put(140,8){\begin{picture}(0,0)
\put(0,0){\brboxlabel{slot 7}}
\put(0,0){\wordpile{6}{\nullptrbox{}}}
\put(0,96){\wordboxptr{40}{}}
\put(0,112){\nullptrbox{slot 0}}
\put(0,128){\blkbox{slots}{40}}
\put(0,128){\brboxlabel{size}}
\end{picture}
}
% set element 1
\put(260,40){\begin{picture}(0,0)
\put(0,0){\dvbox{integer}{n}{10}}
\put(0,32){\wordbox{129}{}}
\put(0,32){\brboxlabel{}}
\put(0,48){\wordboxptr{40}{}}
\put(0,64){\wordbox{selem}{}}
\end{picture}
}
% set element 2
\put(380,24){\begin{picture}(0,0)
\put(0,0){\dvbox{integer}{n}{23}}
\put(0,32){\wordbox{297}{}}
\put(0,32){\brboxlabel{}}
\put(0,48){\nullptrbox{}}
\put(0,64){\wordbox{selem}{}}
\end{picture}
}

\end{picture}

This example was chosen for illustration, since both 10 and 23 go in slot 1.

In searching the list, the hash number of the value being looked up is
compared with the hash numbers in the set-element blocks. If a match
is found, the value in the set-element block may or may not be the same
as the value being looked up, since collisions in the hash computation
are unavoidable. Thus, if the hash numbers are the same, it is
necessary to determine whether or not their values are equivalent. The
comparison that is used is the same one that is used by the
source-language operation \texttt{x === y}.

To improve the performance of the lookup process, the set-element
blocks in each linked list are ordered by their hash numbers. When a
linked list of set-element blocks is examined, the search stops if a
hash number of an element on the list is greater than the hash number
of the value being looked up.

If the value is not found and the lookup is being performed to insert
a new member, a set-element block for the new member is created and
linked into the list at that point. For example,

\iconline{
\>insert(s, 2)
}

\noindent inserts a set-element block for 2 at the head of the list
in slot 1, since its hash value is 25. The word in the set-header
block that contains the number of members is incremented to reflect
the insertion.

\subsection{Set Operations}

The set operations of union, intersection, and difference all produce
new sets and do not modify their arguments.

In the case of union, a copy of the larger set is made first to
provide the basis for the union. This involves not only copying the
set-header block but also all of its set-element blocks. These are
linked together as in the original set, and no lookup is
required. After this copy is made, each member of the set for the
other argument is inserted in the copy, using the same technique that
is used in insert. The larger set is copied, since copying does not
require lookup and the possible comparison of values that insertion
does. The insertion of a member from the second set may take longer,
however, since the linked lists in the copy may be longer.

In the case of intersection, a copy of the smaller argument set is
made, omitting any of its members that are not in the larger set. As
with union, this strategy is designed to minimize the number of
lookups.

For the difference of two sets, a copy of the first argument set is
made, adding only elements that are not in the second argument. This
involves looking up all members in the first argument set in the
second argument set.

\section{Tables}
\subsection{Data Organization for Tables}

The implementation of tables is similar to the implementation of sets,
with a header block containing slots for elements ordered by hash
numbers. A table-header block contains an extra descriptor for the
default assigned value.
{\color{blue} As with lists (but {\em not} sets), Unicon replaces the
  terminating null pointer used by Icon with a pointer to the table
  header block.
}

An empty table with the default assigned value 0 is produced by

\iconline{
\>t := table(0)
}

The structure of the table-header is

\begin{picture}(400,250)(0,-32)
%\put(0,-32){\graphpaper{40}{25}}
\put(120,0){\dvbox{integer}{n}{0}}
\put(120,0){\blboxlabel{default assigned value}}
\put(120,32){\wordpile{5}{\nullptrbox{}}}
\put(120,32){\blboxlabel{segment 5}}
\put(120,94){\leftboxlabels{segment 0}{segment 1}}
\put(120,112){\wordboxptr{60}{}}
\begin{picture}(0,0)(-20,32)
\put(240,94){\rightboxlabels{slot 0}{slot 1}}
\put(360,80){\vdots}
\put(240,0){\brboxlabel{slot 7}}
\put(240,0){\wordpile{8}{\hdrnullptrbox{}}}
\put(240,128){\blkbox{slots}{40}}
\put(240,128){\brboxlabel{size}}
\end{picture}
\put(80,80){\vdots}
\put(120,128){\blkbox{\textit{id}}{7}}
\put(120,128){\brboxlabel{mask}}
\put(120,160){\blkbox{table}{0}}
\put(120,160){\brboxlabel{number of elements in the table}}
\put(0,176){\dvboxptr{table}{np}{40}{}}
\put(0,176){\tlboxlabel{\texttt{t}}}
\end{picture}

Table lookup is more complicated than set lookup, since table elements
contain both an entry value and an assigned value. Furthermore, table
elements can be referenced by variables. A new table element is
created as a byproduct of assignment to a table reference with an
entry value that is not in the table.

The result of evaluating an assignment expression such as

\iconline{
\>t[10] := 1
}

\noindent illustrates the structure of a table-element block:

\begin{picture}(300,120)(0,-32)
\put(120,-32){\dvbox{integer}{n}{1}}
\put(120,-32){\trboxlabel{assigned value}}
\put(120,0){\dvbox{integer}{n}{10}}
\put(120,0){\trboxlabel{entry value}}
\put(120,32){\wordbox{129}{}}
\put(120,32){\brboxlabel{hash number}}
\put(120,48){\hdrnullptrbox{next table-element block}}
\put(120,64){\wordbox{telem}{}}
\end{picture}

%% \begin{picture}(300,150)(50,-10)
%% \put(120,0){\dvbox{integer}{n}{1}}
%% \put(120,0){\trboxlabel{assigned value}}
%% \put(120,32){\dvbox{null}{n}{39}}
%% \put(120,32){\trboxlabel{entry value}}
%% \put(120,64){\dvbox{null}{n}{0}}
%% \put(120,64){\trboxlabel{next table-element block}}
%% \put(120,96){\blkbox{telem}{39}}
%% \put(120,96){\brboxlabel{hash number}}
%% \end{picture}

In the case of a table reference such as \texttt{t[x]}, the hash
number for the entry value x is used to select a slot, and the
corresponding list is searched for a table-element block that contains
the same entry value. As in the case of sets, comparison is first made
using hash numbers; values are compared only if their hash numbers are
the same.

If a table-element block with a matching entry value is found, a
variable that points to the corresponding assigned value is
produced. For example, if 10 is in t as illustrated previously,
\texttt{t[10]} produces

\begin{picture}(300,120)(0,-32)
%\put(0,-32){\graphpaper{30}{15}}
\put(140,-32){\dvbox{integer}{n}{1}}
\put(140,-32){\trboxlabel{assigned value}}
\put(140,0){\dvbox{integer}{n}{10}}
\put(140,0){\trboxlabel{entry value}}
\put(140,32){\wordbox{129}{}}
\put(140,32){\brboxlabel{hash number}}
\put(140,48){\hdrnullptrbox{next table-element block}}
\put(140,64){\wordbox{telem}{}}
%
\put(0,-16){\dvboxptr{5}{npv}{40}{}}
\put(120,-8){\line(0,1){80}}
\put(120,72){\vector(1,0){20}}
\multiput(120,-8)(4,0){4}{\line(1,0){2}}
\put(136,-8){\vector(1,0){4}}
\end{picture}

If this variable is dereferenced, as in

\iconline{
\>write(t[10])
}

\noindent the value 1 is written. On the other hand, if an assignment
is made to this variable, as in

\iconline{
\>t[10] +:= 1
}

\noindent the assigned value in the table-element block is changed:

\begin{picture}(300,120)(0,-32)
\put(140,-32){\dvbox{integer}{n}{2}}
\put(140,-32){\trboxlabel{assigned value}}
\put(140,0){\dvbox{integer}{n}{10}}
\put(140,0){\trboxlabel{entry value}}
\put(140,32){\wordbox{129}{}}
\put(140,32){\brboxlabel{hash number}}
\put(140,48){\hdrnullptrbox{next table-element block}}
\put(140,64){\wordbox{telem}{}}
\end{picture}

If a table element with a matching entry value is not found, the
situation is very similar to that in a subscripted string: the
operation to be performed depends on whether the table reference is
used in a dereferencing or assignment context. In a dereferencing
context, the default value for the table is produced, while in an
assignment context, a new element is added to the table.

The approach taken is similar to that for subscripted strings: a
trapped variable is created. As with substring trapped variables,
table-element trapped variables contain the information that is
necessary to carry out the required computation for either
dereferencing or assignment.

Suppose, for example, that the entry value \texttt{36} is not in the
table \texttt{t}. Then \texttt{t[36]} produces the following result:

\begin{picture}(300,100)
\put(120,0){\dvbox{integer}{n}{36}}
\put(120,0){\trboxlabel{entry value}}
\put(120,32){\wordbox{465}{}}
\put(120,32){\brboxlabel{hash number}}
\put(120,48){\blkboxptr{tvtbl}{40}{table header block for t}}
\put(0,64){\dvboxptr{tvtbl}{nptv}{40}{}}
\end{picture}

%% [DonW]  None of this is true, post V6
%% Note that the size of a table-element trapped-variable block is the
%% same as the size of a table-element block. The last descriptor in the
%% table-element trapped-variable block is reserved for subsequent use,
%% as described below.

If this trapped variable is dereferenced, as in

\iconline{
\>write(t[36])
}

\noindent the default assigned value, 0, which is in the table-header
block for \texttt{t}, is produced. Unfortunately, the situation is not
always this simple. It is possible for elements to be inserted in a
table between the time the table-element trapped-variable block is
created and the time it is dereferenced. An example is

\iconline{
\>write(t[36] , t[36] := 2)
}

Since functions do not dereference their arguments until all the
arguments have been evaluated, the result of dereferencing the first
argument of write should be 2, not 0. In order to handle such cases,
when a table-element trapped variable is dereferenced, its linked list
in the table must be searched again to determine whether to return the
assigned value of a newly inserted element or to return the default
value.

If an assignment is made to the table reference, as in

\iconline{
\>t[36] +:= 1
}

%----- post V6 processing
\noindent the table-element trapped-variable block is copied to a new
table-element block with the assigned value stored in the new block.
%----- which replaces the V6 text
%% \noindent the table-element trapped-variable block is converted to a
%% table-element block with the assigned value stored in the reserved
%% descriptor of the table-element trapped-variable block. The
%% table-element block is then linked in the appropriate place. Note that
%% the structures of table-element blocks and table-element
%% trapped-variable blocks are the same, allowing this conversion without
%% allocating a new table-element block.

It then is necessary to search the linked list for its slot to % again to
determine the place to insert the table-element block. As in the case
of dereferencing, elements may have been inserted in the table between
the time the table-element trapped variable was created and the time a
value is assigned to it. Normally, no matching entry is found, and the
%table-element trapped-variable block, transformed into a table-element
new table-element
block, is inserted with the new assigned value.  If a matching entry
is found, its assigned value is simply changed, and the block is
discarded.

Note that reference to a value that is not in a table requires only
one computation of its hash value, but two lookups are required in the
linked list of table-element blocks for its slot.

\section{Hashing Functions}

\PrimaryIndexBegin{Hash computations}
Ideally, a hash computation should produce a different result for
every different value to which it is applied, and the distribution of
the remainder on division by the number of slots should be
uniform. Even approaching this ideal requires an impractical amount of
computation and space. In practice, it is desirable to have a fast
computation that produces few collisions.

The subject of hash computation has been studied extensively.  In
general, there is a trade-off between faster lookup, on the average,
and more storage overhead. Beyond this, there is a substantial body of
knowledge concerning useful techniques (Knuth 1973, pp. 506-549). For
example, there are fewer collisions if the number of slots is a prime
that is not close to a power of two. Originally, this consideration
motivated the choices of 37 and 13 for number of hash table slots on
computers with large and small address spaces, respectively. As
computer memory sizes grew, the one-size-fits-all strategy was
replaced with one that increases the number of slots as needed
(Griswold and Townsend 1993).

In most situations in which hashing techniques are used, all the
values for which hash computations are performed are strings. In Icon,
however, any kind of value can be the member of a set or the entry
value in a table. The hash computation must, therefore, apply to any
type of value. The support routine for computing hash numbers first
checks for a string, and then has a switch statement to handle all
the other types.

The string hashing implementation is important and deserves extra
scrutiny. The Unicon implementation differs from Icon and is shown
in blue; the \#ifdef is included for exposition and is not in the code.

\index{C functions!\texttt{hash}}%
\begin{iconcode}
uword hash(dp)\\
dptr dp;\\
\>\{\\
\>register char *s;\\
\>register uword i;\\
\>register word j, n;\\
\>register unsigned int *bitarr;\\
\>double r;\\
\>if (Qual(*dp)) \{\\
\end{iconcode}
%
\begin{specialcode}{\tt\color{blue}}
\#ifdef Unicon\\
\>hashstring:\\
\>\> /*\\
\>\>\  * Compute the hash value for the string based on a scaled sum\\
\>\>\  *  of its first and last several characters, plus its length.\\
\>\>\  *  Loops are unrolled.\\
\>\>\  */\\
\>\> i = 0;\\
\>\> s = StrLoc(*dp);\\
\>\> n = StrLen(*dp);\\
\\
\>\> switch(n)\{\\
\>\>\> case 20:  i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\> case 19:  i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\> case 18:  i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\> case 17:  i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\> case 16:  i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\> case 15:  i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\> case 14:  i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\> case 13:  i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\> case 12:  i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\> case 11:  i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\> case 10:  i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\> case 9:   i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\> case 8:   i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\> case 7:   i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\> case 6:   i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\> case 5:   i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\> case 4:   i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\> case 3:   i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\> case 2:   i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\> case 1:   i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\> case 0:   break;\\
\>\>\> default:\\
\>\>\>\>i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\>\>i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\>\>i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\>\>i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\>\>i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\>\>i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\>\>i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\>\>i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\>\>i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\>\>i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\>\>\\
\>\>\>\>s += n - 20;\\
\\
\>\>\>\>i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\>\>i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\>\>i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\>\>i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\>\>i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\>\>i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\>\>i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\>\>i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\>\>i \^{}= (i <{}< 7)\^{}(*s++)\^{}(i >{}> 3);\\
\>\>\>\>i \^{}= ~(i <{}< 11)\^{}(*s++)\^{}(i >{}> 5);\\
\>\>\}\\
\>\> i += n;\\
\>\}\\
\#else /* Icon */
\end{specialcode}%blue
%
\begin{iconcode}
\>hashstring:\\
\>\>/*\\
\>\>\ * Compute the hash value for the string based on a scaled \\
\>\>\ * \ sum of its first ten characters, plus its length.\\
\>\>\ */\\
\>\>i = 0;\\
\>\>s = StrLoc(*dp);\\
\>\>j = n = StrLen(*dp);\\
\>\>if (j > 10)\ \ /* limit scan to first ten characters */\\
\>\>\>j = 10;\\
\>\>while (j-{}- > 0) \{\\
\>\>\>i += *s++ \& 0xFF;\ \ /* add unsigned version of char */\\
\>\>\>i *= 37;\ \ \ \ /* scale by a nice prime number */\\
\>\>\}\\
\>\>i += n;\ \ \ \ \ \ /* add (untruncated) string length */\\
\>\>\}\\
\end{iconcode}
%
\begin{specialcode}{\tt\color{blue}}
\#endif\>\>\> /* Icon / Unicon */\\
\end{specialcode}
%
\begin{iconcode}
\>else \{\\
\>\>switch (Type(*dp)) \{\\
\>\>\>/*\\
\>\>\>\ * The hash value of an integer is itself times eight \\
\>\>\>\ * \ times the golden ratio. \ We do this calculation in \\
\>\>\>\ * \ fixed point. \ We don't just use the integer itself, \\
\>\>\>\ * \ for that would give bad results with sets having\\
\>\>\>\ * \ entries that are multiples of a power of two.\\
\>\>\>\ */\\
\>\>\>case T\_Integer:\\
\>\>\>\>i = (13255 * (uword)IntVal(*dp)) >> 10;\\
\>\>\>\>break;\\
\>\>\>/*\\
\>\>\>\ * The hash value of a bignum is based on its length and \\
\>\>\>\ * \ its most and least significant digits.\\
\>\>\>\ */\\
\>\>case T\_Lrgint:\\
\>\>\>\{\\
\>\>\>struct b\_bignum *b = \&BlkLoc(*dp)->bignumblk;\\
\>\>\>i = ((b->lsd - b->msd) <{}< 16) \^{}\\
\>\>\>\>(b->digits[b->msd] <{}< 8) \^{}\\
\>\>\>\>b->digits[b->lsd];\\
\>\>\>\}\\
\>\>\>break;\\
\>\>\>/*\\
\>\>\>\ * The hash value of a real number is itself times a \\
\>\>\>\ * \ constant, converted to an unsigned integer. \ The \\
\>\>\>\ * \ intent is to scramble the bits well, in the case of \\
\>\>\>\ * \ integral values, and to scale up fractional values \\
\>\>\>\ * \ so they don't all land in the same bin. The constant\\
\>\>\>\ * \ below is 32749 / 29, the quotient of two primes,\\
\>\>\>\ * \ and was observed to work well in empirical testing.\\
\>\>\>\ */\\
\>\>\>case T\_Real:\\
\>\>\>\>GetReal(dp,r);\\
\>\>\>\>i = r * 1129.27586206896558;\\
\>\>\>\>break;\\
\>\>\>/*\\
\>\>\>\ * The hash value of a cset is based on a convoluted \\
\>\>\>\ * \ combination of all its bits.\\
\>\>\>\ */\\
\>\>\>case T\_Cset:\\
\>\>\>\>i = 0;\\
\>\>\>\>bitarr = BlkLoc(*dp)->cset.bits + CsetSize - 1;\\
\>\>\>\>for (j = 0; j < CsetSize; j++) \{\\
\>\>\>\>\>i += *bitarr-{}-;\\
\>\>\>\>\>i *= 37;\ \ \ \ \ \ /* better distribution */\\
\>\>\>\>\>\}\\
\>\>\>\>i \%= 1048583;\ \ \ \ /* scramble the bits */\\
\>\>\>\>break;\\
\>\>\>/*\\
\>\>\>\ * The hash value of a list, set, table, or record is \\
\>\>\>\ * \ its id, hashed like an integer.\\
\>\>\>\ */\\
\>\>\>case T\_List:\\
\>\>\>\>i = (13255 * BlkLoc(*dp)->list.id) >> 10;\\
\>\>\>\>break;\\
\>\>\>case T\_Set:\\
\>\>\>\>i = (13255 * BlkLoc(*dp)->set.id) >> 10;\\
\>\>\>\>break;\\
\>\>\>case T\_Table:\\
\>\>\>\>i = (13255 * BlkLoc(*dp)->table.id) >> 10;\\
\>\>\>\>break;\\
\>\>\>case T\_Record:\\
\>\>\>\>i = (13255 * BlkLoc(*dp)->record.id) >> 10;\\
\>\>\>\>break;\\
\ \  case T\_Proc:\\
\ \  \ \ \ dp = \&(BlkLoc(*dp)->proc.pname);\\
\ \  \ \ \ goto hashstring;\\
\>\>\>default:\\
\>\>\>\>/*\\
\>\>\>\>\ * For other types, use the type code as the hash\\
\>\>\>\>\ * \ value.\\
\>\>\>\>\ */\\
\>\>\>\>i = Type(*dp);\\
\>\>\>\>break;\\
\>\>\>\}\\
\>\>\}\\
\>return i;\\
\>\}
\end{iconcode}

To hash a string, its characters are combined mathematically as
integers. At most the first ten characters are used. {\color{blue}
Unicon uses up to the first ten and the last ten, to distinguish
long strings such as identifiers with differences at either end}.
The limit on the number of characters used in the hash is important
because strings can be very long and adding all the characters does
not improve the hashing sufficiently to justify the time spent in the
computation. A maximum of ten or twenty is, however, \textit{ad hoc}. To
provide a measure of discrimination between strings with the same
initial substring, the length of the string is added to the sum of the
characters.

Icon's technique for hashing strings is not sophisticated, and others
that produce better hashing results are known. However, the
computation is simple, easy to write in C, and works well on most data.
Unicon's technique is adapted from one due to Arash Partow
(http://www.partow.net/programming/hashfunctions/), with length
caps and loop unrolling.  It was validated empirically on a wide
range of data sets.

For a numeric type, the hash value is derived from the number. In the
case of a cset, the words containing the bits for the cset are
combined using the exclusive-or operation.

The remaining data types pose an interesting problem. Hash computation
must be based on attributes of a value that are invariant with
time. Some types, such as files, have such attributes. On the other
hand, there is no time-invariant attribute that distinguishes one list
from another. The size of a list may change, the elements in it may
change, and even its location in memory may change as the result of
garbage collection. For a list, its only time-invariant attribute is
its type.

This presents a dilemma{---}the type of such a value can be used as its
hash number, but if that is done, all values of that type are in the
same slot and have the same hash number. Lookup for these values
degenerates to a linear search.  The alternative is to add some
time-invariant attribute, such as a serial number, to these
values. Icon does this, at the cost of increasing the size of every
such value.
\PrimaryIndexEnd{Hash computations}

\textsc{Retrospective}: Few programming languages support sets or
tables with Icon's generality. The implementation of sets and tables
provides a clear focus on the generality of descriptors and the
uniformity with which different kinds of data are treated in Icon.

Since sets and tables may be very large, efficient lookup is an
important concern. The hashing and chaining technique used is only one
of many possibilities. However, there must be a mechanism for
determining the equivalence of values independent of the structure in
which they are stored.

The fact that elements in tables are accessed by subscripting
expressions introduces several complexities. In particular, the fact
that the contents of the table that is subscripted may change between
the time the subscripting expression is evaluated and the time it is
dereferenced or assigned to introduces the necessity of two lookups
for every table reference.

Hashing a variety of different types of data raises interesting
issues. The hashing techniques used by Icon are not sophisticated and
there is considerable room for improvement. The trade-offs involved
are difficult to evaluate, however.

\bigskip

\noindent\textbf{EXERCISES}

\liststyleLvii
\begin{enumerate}
\item \begin{enumerate}

\item Contrast sets and csets with respect to their implementation,
their usefulness in programming, and the efficiency of operations on
them.

\item Give an example of a situation in which the heterogeneity of
sets is useful in programming.

\item How much space does an empty set occupy?

\item Diagram the structures resulting from the evaluation of the
following expressions:\newline
 t := table()\newline
 t[t] := t

\item There are many sophisticated data structures that are designed
to ensure efficient lookup in data aggregates like sets and tables
(Gonnet 1984). Consider the importance of speed of lookup in sets and
tables in Icon and the advantages that these more sophisticated data
structures might supply.

\item Some of the more sophisticated data structures mentioned in the
preceding exercise have been tried experimentally in Icon and either
have introduced unexpected implementation problems or have not
provided a significant improvement in performance. What are possible
reasons for these disappointing results?

\item Icon goes to a lot of trouble to avoid adding table-element
blocks to a table unless an assignment is made to them.  Suppose a
table-element block were simply added when a reference was made to an
entry value that is not in the table.
\end{enumerate}
\end{enumerate}
\liststyleLviii
\begin{itemize}
\item How would this simplify the implementation?

\item What positive and negative consequences could this change have
on the running speed and space required during program execution?

\item Give examples of types of programs for which the change would
have positive and negative effects on performance, respectively.

\item Would this change be transparent to the Icon programmer, not
counting possible time and space differences?
\end{itemize}
\liststyleLix
\begin{enumerate}
\item \begin{enumerate}

\item There is space in a table-element trapped-variable block to put
the default value for the table. Why is this not done?

\item
What is the consequence of evaluating the following expressions?
\end{enumerate}
\end{enumerate}
\begin{iconcode}
\>t := table(0)\\
\>t[37] := 2\\
\>write(t[37], t := table(1))
\end{iconcode}

What would happen if the last line given previously were

\iconline{
\>write(t[37],t := list(100,3))
}

or

\iconline{
\>write(t[37], t := "hello")
}

\liststyleLx
\begin{enumerate}
\item \begin{enumerate}

\item Give examples of different strings that have the same hash numbers.

\item Design a method for hashing strings that produces a better
distribution than the the current one.

\item What attribute of a table is time-invariant?

\item What kinds of symptoms might result from a hashing computation
 based on an attribute of a value that is not time-invariant?

\end{enumerate}
\end{enumerate}


% !TEX root = Main.tex
\section{Matrix Approximation \& Reconstruction}

$\min_{rank(B)=k}[\sum_{(i,j)\in I}{(a_{ij}-b_{ij})^2}], I=\{(i,j): \mathit{ob.}\}$
\subsection*{Alternating Least Squares}
$f(U,v_i) = \sum_{(i,j)\in I} (a_{i,j} - \langle u_j, v_i \rangle)^2$\\
$f(u_i,V) = \sum_{(i,j)\in I} (a_{i,j} - \langle u_j, v_i \rangle)^2$\\
Convex when fixed one.

% YZF: delete?
% \subsection*{Coordinate Descent}
% 1. init: $\mathbf{x}^{(0)} \in \mathbb{R}^D$\\
% 2. for $t = 0 \ \text{to} \ \mathit{maxIter}$:\\
% 3. sample $d \in_{u.a.r.} \{1, \ldots, D\}$\\
% 4. $u^\star = \argmin_{u \in \mathbb{R}} f(x_1^{(t)}, .., x_{d-1}^{(t)}, u, x_{d+1}^{(t)}, .., x_D^{(t)})$\\
% 5. $\mathbf{x}_d^{(t+1)} = u^\star$ and $\mathbf{x}_i^{(t+1)} = \mathbf{x}_i^{(t)}$ for $i \neq d$

% \subsection*{Projected Gradient Descent (Constrained Opt.)}
% minimize $f(x)$, $x \in Q$ (constraint).\\
% \textbf{Project} $x$ onto $Q$: $P_Q(\mathbf{x}) = \argmin_{y \in Q} \|\mathbf{y} - \mathbf{x}\|$,\\
% \textbf{Update}: $\mathbf{x}^{(t+1)} = P_Q[\mathbf{x}^{(t)} - \gamma \nabla f(\mathbf{x}^{(t)})]$,\\
% $\mathbf{x}^{(t+1)}$ is unique if $Q$ convex.

% YZF: need revising
\subsection*{Convex Optimization}
Def.: $\{(x,t)|x \in dom f, f(x) \leq t\}$, $f : \mathbb{R}^D \rightarrow \mathbb{R}$ is convex, if $dom\ f$ is a convex set, and if $\forall \mathbf{x}, \mathbf{y} \in dom\ f$, and $\forall \alpha\in[0,1]$: $f(\alpha \mathbf{x} + (1 - \alpha)\mathbf{y}) \leq \alpha f(\mathbf{x}) + (1-\alpha)f(\mathbf{y})$.
Convex $\iff$ Hessian p.s.d $\iff$ local=global \\
Positive semi-definite: all principal minors (same-indexed rows and columns) $\geq$ 0\\
Positive definite: leading principal minors $>$ 0

\subsection*{Convex Relaxation}
Replace non-convex rank constraints by convex norm constraints (superset). Then project optimum back (hopefully still optimal).\\
$\min_{\mathbf{B}\in P_k}{\|\mathbf{A-B}\|^2_G}, P_k=\{\mathbf{B}:\|\mathbf{B}\|_{*}\leq k\}\supseteq Q_k=\{\mathbf{B}:\mathit{rank}(\mathbf{B})\leq k\}$ (in fact tightest convex lowerbound $\mathit{rank}(\mathbf{B})\geq \|\mathbf{B}\|_{*}, for \|\mathbf{B}\|_2 \leq 1$)

\subsection*{SVD Thresholding}
% TBA
$\mathbf{B}^{*}=\mathit{shrink}_\tau(\mathbf{A})=\argmin_{\mathbf{B}}{\{\|\mathbf{A-B}\|^2_F + \tau\|\mathbf{B}\|_{*}\}}$\\
Then with SVD $\mathbf{A=UDV_T}, \mathbf{D}=\mathit{diag}(\sigma_i)$, holds $\mathbf{B^*=UD_\tau V^T, D_\tau} = \mathit{diag}(\max\{0,\sigma_i - \tau\})$ \\
Iteration: $\mathbf{B}_{t+1}=\mathbf{B}_t + \eta_t \Pi(\mathbf{A} - \mathit{shrink}_\tau(\mathbf{B}_t))$


\section{The matrix of a linear transformation II}

%Requires Linear Transformations.

\begin{outcome}
  \begin{enumerate}
  \item Find the matrix of a linear transformation with respect to
    general bases.
  \end{enumerate}
\end{outcome}

We begin this section with an important lemma.

\begin{lemma}{Mapping of a basis}{mapping-basis}
Let $T: \R^n \to \R^n$ be an isomorphism.  Then $T$ maps any basis of
$\R^n$ to another basis for $\R^n$.

Conversely, if $T:
\R^n \to \R^n$ is a linear transformation which
maps a basis of $\R^n$ to another basis of $\R^n$,
then it is an isomorphism.
\end{lemma}

\begin{proof}
First, suppose $T:\R^n \to \R^n$ is a linear
transformation which is one to one and onto. Let $\set{
\vect{v}_{1},\ldots,\vect{v}_{n}} $ be a basis for
$\R^n$. We wish to show that $\set{T(\vect{v}_{1}),\ldots,
T(\vect{v}_{n})} $ is also a basis for $\R^n$.

First consider why it is linearly independent. Suppose
$\sum_{k=1}^{n}a_{k}T(\vect{v}_{k})=\vect{0}$. Then by linearity we have $T\paren{
\sum_{k=1}^{n}a_{k}\vect{v}_{k}} =\vect{0}$ and since $T$ is one
to one, it follows that $\sum_{k=1}^{n}a_{k}\vect{v}_{k}=\vect{0}$.
This requires that  each $a_{k}=0$ because $\set{\vect{v}_{1},\ldots,
\vect{v}_{n}} $ is independent, and it follows that $\set{
T(\vect{v}_{1}),\ldots, T(\vect{v}_{n})} $ is linearly
independent.

Next take $\vect{w}\in \R^n$. Since $T$ is onto, there exists
$\vect{v}\in \R^n$ such that $T(\vect{v})=\vect{w}$. Since
$ \set{\vect{v}_{1},\ldots,\vect{v}_{n}}$ is a basis, in particular it
is a spanning set and there are scalars $b_{k}$ such that
$T\paren{\sum_{k=1}^{n}b_{k}\vect{v} _{k}} =T(\vect{v})
=\vect{w}$. Therefore $\vect{w} =\sum_{k=1}^{n}b_{k}T(\vect{v}_{k})$
which is in the
$\sspan\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})}$. Therefore,
$\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})}$ is a basis as
claimed.

Suppose now that $T: \R^n \to \R^n$ is a linear
transformation such that $T(\vect{v}_{i})=\vect{w}_{i}$ where
$\set{\vect{v} _{1},\ldots,\vect{v}_{n}} $ and $\set{
\vect{w}_{1},\ldots, \vect{w}_{n}} $ are two bases for
$\R^n$.

To show that $T$ is one to one, let $T\paren{
\sum_{k=1}^{n}c_{k}\vect{v}_{k}} =\vect{0}$. Then
$\sum_{k=1}^{n}c_{k}T(\vect{v}_{k})=\sum_{k=1}^{n}c_{k}\vect{w}_{k}=\vect{
0}$. It follows that each $c_{k} = 0$ because it is given that
$\set{\vect{w} _{1},\ldots,\vect{w}_{n}} $ is linearly
independent. Hence $T\paren{\sum_{k=1}^{n}c_{k}\vect{v}_{k}}
=\vect{0}$ implies that $\sum_{k=1}^{n}c_{k}\vect{v}_{k}=\vect{0}$ and
so $T$ is one to one.

To show that $T$ is onto, let $\vect{w}$ be an arbitrary vector in
$\R^n$. This vector can be written as $\vect{w} =
\sum_{k=1}^{n}d_k\vect{w}_k =
\sum_{k=1}^{n}d_{k}T(\vect{v}_{k})=T\paren{\sum_{k=1}^{n}d_{k}
\vect{v}_{k}}$.  Therefore, $T$ is also onto.
\end{proof}

Consider now an important definition.

\begin{definition}{Coordinate vector}{coordinate-vector}
Let $B = \set{\vect{v}_1, \vect{v}_2,\ldots, \vect{v}_n }$
be a basis for $\R^n$ and let $\vect{x}$ be an arbitrary
vector in $\R^n$. Then $\vect{x}$ is uniquely represented as
$\vect{x} = a_1\vect{v}_1 +
a_2\vect{v}_2 + \ldots + a_n\vect{v}_n$ for scalars $a_1,\ldots,
a_n$.

The  \textbf{coordinate vector}\index{vector!coordinate vector}\index{coordinate vector} of $\vect{x}$ with respect to the
basis $B$, written $C_B(\vect{x})$ or  $[\vect{x}]_B$,  is given by
\[
C_B(\vect{x}) =  C_B (a_1\vect{v}_1 + a_2\vect{v}_2 + \ldots + a_n\vect{v}_n) = \begin{mymatrix}{c}
a_1 \\
a_2 \\
\vdots \\
a_n
\end{mymatrix}
\]
\end{definition}

Consider the following example.

\begin{example}{Coordinate vector}{coordinate-vector}
Let $B = \set{\begin{mymatrix}{r}
1 \\
0
\end{mymatrix}, \begin{mymatrix}{r}
-1 \\
1
\end{mymatrix} }$ be a basis of $\R^2$ and let $\vect{x} = \begin{mymatrix}{r}
3 \\
-1
\end{mymatrix}$ be a vector in $\R^2$. Find $C_B(\vect{x})$.
\end{example}

\begin{solution}
First, note the order of the basis is important so label the vectors in the basis $B$ as
\[
B = \set{\begin{mymatrix}{r}
1 \\
0
\end{mymatrix}, \begin{mymatrix}{r}
-1 \\
1
\end{mymatrix} } = \set{\vect{v}_1, \vect{v}_2 } \]
Now we need to find $a_1, a_2$ such that $\vect{x} = a_1 \vect{v}_1 + a_2 \vect{v}_2$, that is:
\[
\begin{mymatrix}{r}
3 \\
-1
\end{mymatrix}
=
a_1
\begin{mymatrix}{r}
1 \\
0
\end{mymatrix}
+ a_2
\begin{mymatrix}{r}
-1 \\
1
\end{mymatrix}\]
Solving this system gives $a_1 = 2, a_2 = -1$. Therefore the coordinate vector of $\vect{x}$ with respect to the basis $B$ is
\[
C_B(\vect{x})
=
\begin{mymatrix}{r}
a_1 \\
a_2
\end{mymatrix}
= \begin{mymatrix}{r}
2 \\
-1
\end{mymatrix}
\]
\end{solution}

Given any basis $B$, one can easily verify that the coordinate function is actually an isomorphism.

\begin{theorem}{$C_B$ is a linear transformation}{coordinate-linear-transformation}
For any basis $B$ of $\R^n$, the coordinate function
\[ C_B: \R^n  \rightarrow \R^n  \]
is a linear transformation, and moreover an isomorphism.
\end{theorem}

We now discuss the main  result  of this section, that is how
to represent a linear transformation with respect to different
bases.

\begin{theorem}{The matrix of a linear transformation}{matrix-linear-transformation-bases}
Let $T: \R^n \to \R^m$ be a linear transformation,
and let $B_1$ and $B_2$ be bases of $\R^{n}$ and
$\R^{m}$ respectively.

Then the following holds
\begin{equation}
C_{B_2} T = M_{B_{2} B_{1}} C_{B_1}   \label{matrix-equation}
\end{equation}
where $M_{B_{2} B_{1}}$  is a unique $m \times n$-matrix.

If the basis $B_1$ is given by $B_1=\set{\vect{v}_1,\ldots, \vect{v}_n}$ in this order, then

\[  M_{B_{2} B_{1}} = \mat{C_{B_2}(T(\vect{v}_1)), C_{B_2}(T(\vect{v}_2)), \ldots, C_{B_2}(T(\vect{v}_n)) } \]
\end{theorem}

\begin{proof}
The above equation {\eqref{matrix-equation}} can be represented by the following diagram.
\begin{equation*}
\begin{array}{rrcll}
&  & T &  &  \\
& \R^n & \rightarrow  & \R^m & \\
& C_{B_{1} }\downarrow  & \circ  & \downarrow C_{B_{2} } &  \\
& \R^{n} & \rightarrow  & \R^{m} &  \\
&  & M_{B_{2} B_{1} } &  &
\end{array}
\end{equation*}

Since $C_{B_1}$ is an isomorphism, then the matrix we are looking for is the matrix of the linear transformation
\[   C_{B_2} T C^{-1}_{B_1} : \R^n \to \R^m. \]
By Theorem~\ref{thm:matrix-of-linear-transformation}, the columns are
given by the image of the standard basis $\set{\vect{e}_1,
\vect{e}_2,\ldots, \vect{e}_n }$. But since $C^{-1}_{B_1}( \vect{e}_i) = \vect{v}_i$, we readily obtain that

\[ \begin{array}{ll}
M_{B_{2} B_{1}}
& = \mat{C_{B_2}T C^{-1}_{B_1} (\vect{e}_1), C_{B_2}T C^{-1}_{B_1} (\vect{2}_2), \ldots, C_{B_2}T C^{-1}_{B_1} (\vect{e}_n) } \\
& = \mat{C_{B_2}(T(\vect{v}_1)), C_{B_2}(T(\vect{v}_2)), \ldots, C_{B_2}(T(\vect{v}_n)) }
\end{array}\]
and this completes the proof.
\end{proof}

Consider the following example.

\begin{example}{Matrix of a linear transformation}{matrix-linear-transformation}
Let $T: \R^2 \to \R^2$ be a linear transformation defined by $T \paren{\begin{mymatrix}{r}
a \\
b
\end{mymatrix}} = \begin{mymatrix}{r}
b \\
a
\end{mymatrix}$.

Consider the two bases
\[
B_1 = \set{\vect{v}_{1}, \vect{v}_{2} } = \set{\begin{mymatrix}{r}
1 \\
0
\end{mymatrix}, \begin{mymatrix}{r}
-1 \\
1
\end{mymatrix}
}
\]
 and
\[
B_2 = \set{\begin{mymatrix}{r}
1 \\
1
\end{mymatrix}, \begin{mymatrix}{r}
1 \\
-1
\end{mymatrix}
}
\]

Find the matrix $M_{B_2,B_1}$ of $T$ with respect to the bases $B_1$ and $B_2$.
\end{example}

\begin{solution}
By Theorem~\ref{thm:matrix-linear-transformation-bases}, the columns of $M_{B_{2} B_{1}}$ are the
coordinate vectors of $T(\vect{v}_{1}), T(\vect{v}_{2})$ with respect
to $B_2$.

Since \[
T \paren{
\begin{mymatrix}{r}
1 \\
0
\end{mymatrix}}
= \begin{mymatrix}{r}
0 \\
1
\end{mymatrix} ,\]
a standard calculation yields
\[
 \begin{mymatrix}{r}
0 \\
1
\end{mymatrix}
 =
\paren{\frac{1}{2}}\begin{mymatrix}{r}
1 \\
1
\end{mymatrix}
+
\paren{-\frac{1}{2}}
\begin{mymatrix}{r}
1 \\
-1
\end{mymatrix},
\]
the first column of $M_{B_{2} B_{1}}$ is $\begin{mymatrix}{r}
\vspace{0.05in}\frac{1}{2}\\
\vspace{0.05in}-\frac{1}{2}
\end{mymatrix}$.

The second column is found in a similar way. We have
\[
T \paren{
\begin{mymatrix}{r}
-1 \\
1
\end{mymatrix}}
= \begin{mymatrix}{r}
1 \\
-1
\end{mymatrix} , \]
and with respect to $B_2$ calculate:
\[
\begin{mymatrix}{r}
1 \\
-1
\end{mymatrix}
=
0 \begin{mymatrix}{r}
1 \\
1
\end{mymatrix}
+
1
\begin{mymatrix}{r}
1 \\
-1
\end{mymatrix}
\]
Hence the second column of $M_{B_{2} B_{1}}$ is given by $\begin{mymatrix}{r}
0 \\
1
\end{mymatrix}$. We thus obtain
\[
M_{B_{2} B_{1}} = \begin{mymatrix}{rr}
\vspace{0.05in}\frac{1}{2} & 0 \\
\vspace{0.05in}-\frac{1}{2} & 1
\end{mymatrix} \]

We can verify that this is the correct matrix $M_{B_{2} B_{1}}$ on the specific example
\[
\vect{v} = \begin{mymatrix}{r}
3 \\
-1
\end{mymatrix} \]
First applying $T$ gives
\[
T( \vect{v} ) =
T \paren{
\begin{mymatrix}{r}
3 \\
-1
\end{mymatrix}} = \begin{mymatrix}{r}
-1\\
3
\end{mymatrix}
\]
and one can compute that
\[ C_{B_2}
 \paren{
\begin{mymatrix}{r}
-1 \\
3
\end{mymatrix}} = \begin{mymatrix}{r}
1\\
-2
\end{mymatrix} .\]

On the other hand, one compute $C_{B_1}( \vect{v})$ as
\[ C_{B_1}
 \paren{
\begin{mymatrix}{r}
3 \\
-1
\end{mymatrix}} = \begin{mymatrix}{r}
2\\
-1
\end{mymatrix} ,\]
and finally applying $M_{B_1 B_2}$ gives

\[\begin{mymatrix}{rr}
\vspace{0.05in}\frac{1}{2} & 0 \\
\vspace{0.05in}-\frac{1}{2} & 1
\end{mymatrix}
\begin{mymatrix}{r}
2 \\
-1
\end{mymatrix}
= \begin{mymatrix}{r}
1 \\
-2
\end{mymatrix} \]
as above.

We see that the same vector results from either method, as suggested by Theorem~\ref{thm:matrix-linear-transformation-bases}.
\end{solution}

If the bases $B_1$ and $B_2$ are equal, say $B$, then we write $M_{B}$ instead of  $M_{B B}$.
The following example illustrates how to compute  such a matrix. Note that this is what we did earlier when we considered only
$B_1=B_2$ to be the standard basis.

\begin{example}{Matrix of a linear transformation with respect to an arbitrary   basis}{arbitrary-bases}

Consider the basis $B$ of $\R^3$ given by
\begin{equation*}
B = \set{\vect{v}_1 , \vect{v}_2,  \vect{v}_3} =
\set{
\begin{mymatrix}{r}
1 \\
0 \\
1
\end{mymatrix} ,\begin{mymatrix}{r}
1 \\
1 \\
1
\end{mymatrix} ,\begin{mymatrix}{r}
-1 \\
1 \\
0
\end{mymatrix} }
\]

And let $T :\R^{3}\to \R^{3}$ be the linear transformation
defined on $B$ as:
\begin{equation*}
T\begin{mymatrix}{r}
1 \\
0 \\
1
\end{mymatrix} =\begin{mymatrix}{r}
1 \\
-1 \\
1
\end{mymatrix} ,T \begin{mymatrix}{c}
1 \\
1 \\
1
\end{mymatrix} =\begin{mymatrix}{r}
1 \\
2 \\
-1
\end{mymatrix} ,T\begin{mymatrix}{r}
-1 \\
1 \\
0
\end{mymatrix} =\begin{mymatrix}{r}
0 \\
1 \\
1
\end{mymatrix}
\end{equation*}

\begin{enumerate}
\item Find the matrix  $M_{B}$ of $T$ relative to the basis $B$.
\item Then find the usual matrix of $T$ with respect to the standard basis of $\R^{3}$.
\end{enumerate}

\end{example}

\begin{solution}

Equation  {\eqref{matrix-equation}}  gives $ C_BT=M_{B}C_B$, and thus
$M_{B} = C_BTC^{-1}_B$.

Now $C_B(\vect{v}_i) = \vect{e}_i$, so the matrix of $C_B^{-1}$ (with respect to the standard basis) is given by
\[ \mat{C_B^{-1}(\vect{e}_1) \;\; C_B^{-1}(\vect{e}_2) \;\; C_B^{-1}(\vect{e}_2) } =
\begin{mymatrix}{rrr}
1 & 1 & -1 \\
0 & 1 & 1 \\
1 & 1 & 0
\end{mymatrix}
\]
Moreover the matrix of  $T C_B^{-1}$ is given by
\[ \mat{TC_B^{-1}(\vect{e}_1) \;\; TC_B^{-1}(\vect{e}_2) \;\; TC_B^{-1}(\vect{e}_2) } =
\begin{mymatrix}{rrr}
1 & 1 & 0 \\
-1 & 2 & 1 \\
1 & -1 & 1
\end{mymatrix}
\]
Thus
\[ \begin{array}{ll}
M_{B} & =  C_BTC^{-1}_B =  [C^{-1}_B]^{-1} [TC^{-1}_B] \\
	& =
\begin{mymatrix}{rrr}
1 & 1 & -1 \\
0 & 1 & 1 \\
1 & 1 & 0
\end{mymatrix} ^{-1}\begin{mymatrix}{rrr}
1 & 1 & 0 \\
-1 & 2 & 1 \\
1 & -1 & 1
\end{mymatrix} \\
&=\begin{mymatrix}{rrr}
2 & -5 & 1 \\
-1 & 4 & 0 \\
0 & -2 & 1
\end{mymatrix}
\end{array}
\]

Consider how this works. Let $\vect{b} = \begin{mymatrix}{r}
b_1 \\
b_2 \\
b_3
\end{mymatrix}$ be an arbitrary vector in $\R^3$.

Apply $C^{-1}_{B}$ to $\vect{b}$ to get
\begin{equation*}
b_1\begin{mymatrix}{r}
1 \\
0 \\
1
\end{mymatrix} + b_2\begin{mymatrix}{r}
1 \\
1 \\
1
\end{mymatrix} + b_3\begin{mymatrix}{r}
-1 \\
1 \\
0
\end{mymatrix}
\end{equation*}
Apply $T$ to this linear combination to obtain
\begin{equation*}
b_1\begin{mymatrix}{r}
1 \\
-1 \\
1
\end{mymatrix} + b_2\begin{mymatrix}{r}
1 \\
2 \\
-1
\end{mymatrix} + b_3\begin{mymatrix}{r}
0 \\
1 \\
1
\end{mymatrix} =\begin{mymatrix}{c}
b_1+b_2 \\
-b_1 + 2b_2+ b_3 \\
b_1-b_2+b_3
\end{mymatrix}
\end{equation*}
Now take the matrix $M_{B}$ of the transformation (as found above) and multiply it by $\vect{b}$.
\begin{equation*}
\begin{mymatrix}{rrr}
2 & -5 & 1 \\
-1 & 4 & 0 \\
0 & -2 & 1
\end{mymatrix} \begin{mymatrix}{c}
b_1 \\
b_2 \\
b_3
\end{mymatrix} =\begin{mymatrix}{c}
2b_1-5b_2+b_3 \\
-b_1 + 4b_2 \\
-2b_2 + b_3
\end{mymatrix}
\end{equation*}
Is this the coordinate vector of the above relative to the given basis? We check as follows.
\begin{equation*}
(2b_1-5b_2+b_3) \begin{mymatrix}{c}
1 \\
0 \\
1
\end{mymatrix} +(-b_1 + 4b_2) \begin{mymatrix}{c}
1 \\
1 \\
1
\end{mymatrix} +(-2b_2+b_3) \begin{mymatrix}{c}
-1 \\
1 \\
0
\end{mymatrix}
\end{equation*}
\begin{equation*}
= \begin{mymatrix}{c}
b_1+b_2 \\
-b_1 + 2b_2+b_3 \\
b_1-b_2+b_3
\end{mymatrix}
\end{equation*}
You see it is the same thing.

Now let us find the matrix of $T$ with respect to the standard basis. Let $A$ be
this matrix. That is, multiplication by $A$ is the same as doing $T$. Thus
\begin{equation*}
A\begin{mymatrix}{rrr}
1 & 1 & -1 \\
0 & 1 & 1 \\
1 & 1 & 0
\end{mymatrix} =\begin{mymatrix}{rrr}
1 & 1 & 0 \\
-1 & 2 & 1 \\
1 & -1 & 1
\end{mymatrix}
\end{equation*}
Hence
\begin{equation*}
A=\begin{mymatrix}{rrr}
1 & 1 & 0 \\
-1 & 2 & 1 \\
1 & -1 & 1
\end{mymatrix} \begin{mymatrix}{rrr}
1 & 1 & -1 \\
0 & 1 & 1 \\
1 & 1 & 0
\end{mymatrix} ^{-1}=\begin{mymatrix}{rrr}
0 & 0 & 1 \\
2 & 3 & -3 \\
-3 & -2 & 4
\end{mymatrix}
\end{equation*}
Of course this is a very different matrix than the matrix of the linear
transformation with respect to the non-standard basis.
\end{solution}


\section{Single-mode squeezing}

Let
\begin{equation}
\widehat{b}^-
= \frac{\widehat{a}^- - \beta \widehat{a}^+}{\sqrt{1-\vbr{\beta}^2}},\qquad
\widehat{b}^+ = \rbr{\widehat{b}^-}^\dagger
= \frac{\widehat{a}^+ - \beta^*\widehat{a}^-}{\sqrt{1-\vbr{\beta}^2}},
\label{eq:res-bogolyubov-b}
\end{equation}
so that
\begin{equation}
\sbr{\widehat{b}^-, \widehat{b}^+}_- =
\rbr{1-\vbr{\beta}^2}^{-1}\rbr{\sbr{\widehat{a}^-, \widehat{a}^+}_-
+ \beta\beta^*\sbr{\widehat{a}^+, \widehat{a}^-}_-} = \widehat{1}.
\end{equation}
One seeks the state $\Ket{\beta}$ satisfying
\begin{equation}
\widehat{b}^-\Ket{\beta} = 0.
\label{eq:squeezed-state-def-1}
\end{equation}

Note that \cref{eq:res-bogolyubov-b} is a restricted \emph{Bogolyubov
transformation}, in that the coefficient of $\widehat{a}^-$ for $\widehat{b}^-$
is real.

\subsection{Single-mode squeeze operator}
One attempts
\begin{equation}
\rfun{\widehat{S}}{z}\widehat{a}^-\rfun{\widehat{S}^{-1}}{z},
\end{equation}
where
\begin{align}
	\rfun{\widehat{S}}{z} &\coloneqq \cfun{\exp}{\frac{1}{2}\rbr{z
		\rbr{\widehat{a}^+}^2 - z^*\rbr{\widehat{a}^-}^2}} \nonumber \\
&\equiv \cfun{\expi}%
{\Im z\rbr{\frac{m\Omega}{2}\widehat{x}^2 - \frac{2}{m\Omega}\widehat{p}^2}
-\Re z \frac{\widehat{x}\widehat{p}+\widehat{p}\widehat{x}}{2}}
\end{align}
is the \emph{single-mode squeeze operator}. Note that
\begin{equation}
\rfun{\widehat{S}^{-1}}{z} = \rfun{\widehat{S}}{-z}
= \rfun{\widehat{S}^\dagger}{z}.
\end{equation}

The parameterisation $z = r\ee^{\ii\phi}$ will also be used, where $r =
\vbr{z}$, $\phi = \arg z$.

\begin{namedthm}{Proposition}
	\label{nthm:prop-ad-aminus}
Let $X = \rbr{z\rbr{\widehat{a}^+}^2 - z^*\rbr{\widehat{a}^-}^2}/2$. Then
\begin{align}
	\ad_X^{(2n)} \widehat{a}^- &= \vbr{z}^{2n} \widehat{a}^-
	= r^{2n} \widehat{a}^-,\\
	\ad_X^{(2n+1)} \widehat{a}^-
	&= -\vbr{z}^{2n+1}\frac{z}{\vbr{z}}\widehat{a}^+
	= - r^{2n+1} \ee^{\ii\phi} \widehat{a}^+,
	\qquad \forall n \ge 0.
\end{align}
\end{namedthm} % Proposition
\begin{proof}
	\begin{equation}
		\ad_X^{(0)}\widehat{a}^- = \widehat{a}^-;
	\end{equation}
	\begin{align}
		\ad_X^{(1)}\widehat{a}^- &=
		\sbr{\frac{z}{2}\rbr{\widehat{a}^+}^2, \widehat{a}^-}_-
		= \frac{z}{2}\rbr{\widehat{a}^+ \sbr{\widehat{a}^+, \widehat{a}^-}_-
		+ \sbr{\widehat{a}^+, \widehat{a}^-}_- \widehat{a}^+}
		= -z\widehat{a}^+ \nonumber \\
		&= -\frac{z}{\vbr{z}}\vbr{z}^1\widehat{a}^+
		= -\ee^{\ii\phi} r^1 \widehat{a}^+,
	\end{align}
	\begin{align}
		\ad_X^{(2)}\widehat{a}^- &=
		\sbr{-\frac{z^*}{2}\rbr{\widehat{a}^-}^2, -z\widehat{a}^+}_-
		= \frac{\vbr{z}^2}{2} 2\widehat{a}^- \nonumber \\
		&= \vbr{z}^2\widehat{a}^- = r^2 \widehat{a}^-;
	\end{align}
	\begin{align}
		\ad_X^{(3)}\widehat{a}^- &=
		\sbr{\frac{z}{2}\rbr{\widehat{a}^+}^2, \vbr{z}^2\widehat{a}^-}_-
		= \frac{z\vbr{z}^2}{2} \rbr{-2} \widehat{a}^+ \nonumber \\
		&= -\frac{z}{\vbr{z}}\vbr{z}^3\widehat{a}^+
		= -\ee^{\ii\phi} r^3 \widehat{a}^+.
	\end{align}
	Prop.~(\ref{nthm:prop-ad-aminus}) can then be proved by induction for
	even and odd integers, respectively.
\end{proof}

\begin{namedthm}{Proposition}
\begin{align}
	\rfun{\widehat{S}}{z} \widehat{a}^- \rfun{\widehat{S}^\dagger}{z}
	&= \widehat{a}^-\cosh \vbr{z}
		- \widehat{a}^+ \frac{z}{\vbr{z}} \sinh \vbr{z}
	\nonumber \\
	&= \widehat{a}^-\cosh r - \widehat{a}^+ \ee^{\ii\phi} \sinh r,
	\label{eq:squeezed-aminus-parameter}
\end{align}
so that $z$ or $\rbr{r, \phi}$ is a parameterisation of $\beta$ in
\cref{eq:res-bogolyubov-b} in that
\begin{equation}
	\vbr{\beta} = \tanh\vbr{z} = \tanh r,\qquad \arg \beta = \arg z = \phi.
	\label{eq:beta-parameter}
\end{equation}
\end{namedthm} %Proposition
\begin{proof}
	By prop.~\ref{nthm:prop-ad-aminus} and \cref{eq:bch-sandwich},
	\begin{equation} % \begin{align}
		\rfun{\widehat{S}}{z} \widehat{a}^- \rfun{\widehat{S}^\dagger}{z}
		= \sum_{n=0}^{+\infty}\rbr{
			\frac{\vbr{z}^{2n}}{\rbr{2n}!} \widehat{a}^-
- \frac{z}{\vbr{z}}\frac{\vbr{z}^{2n+1}}{\rbr{2n+1}!}\widehat{a}^+}
		%\nonumber \\
		%&= \widehat{a}^-\cosh \vbr{z}
		%- \widehat{a}^+ \frac{z}{\vbr{z}} \sinh \vbr{z} \nonumber \\
		%&= \widehat{a}^-\cosh r - \widehat{a}^+ \ee^{\ii\phi} \sinh r,
		\label{eq:squeezed-aminus-parameter-proof}
	\end{equation} % \end{align}
	which proves \cref{eq:squeezed-aminus-parameter}. Comparing
	\cref{eq:squeezed-aminus-parameter-proof} with \cref{eq:res-bogolyubov-b},
	one gets
	\begin{equation}
		\left\{\begin{array}{l}
			\rbr{1-\vbr{\beta}^2}^{-1/2} = \cosh \vbr{z} = \cosh r, \\
			\beta (1-\vbr{\beta}^2)^{-1/2}
			= \frac{z}{\vbr{z}}\sinh \vbr{z} = \ee^{\ii\phi}\sinh r,
		\end{array}\right.
	\end{equation}
	which gives \cref{eq:beta-parameter}.
\end{proof}

\begin{namedthm}{Squeezed ground state}
	$\Ket{\beta}$ in \cref{eq:squeezed-state-def-1} is the $\beta$-squeezed
ground state, namely
\begin{equation}
	\Ket{\beta} = \rfun{\widehat{S}}{z}\Ket{0}.
\end{equation}

\end{namedthm} % Squeezed ground state
\begin{proof}
Note that
\begin{equation}
a^-\Ket{0} \coloneqq 0 \eqqcolon \rfun{S^\dagger}{\beta} b^- \Ket{0}
= a^- \rfun{S^\dagger}{\beta} \Ket{\beta}.
\end{equation}
With $\rfun{S}{\alpha}$ unitary in mind, one finds
\begin{equation}
\Ket{\beta} = \ee^{\ii\theta}\rfun{S}{\beta}\Ket{0}.
\end{equation}
One can fix the phase $\theta$ to be zero.
\end{proof}

\subsection{Time evolution}

\begin{namedthm}{Proposition}
	Let $X = \ii\Omega t a^+ a^-$, $Y = \frac{1}{2}\rbr{z
		\rbr{a^+}^2 - z^*\rbr{a^-}^2}$.
	\begin{equation}
		\ad_X^{(n)}Y = \frac{1}{2}\rbr{z \rbr{+2\ii\Omega t}^n
		\rbr{a^+}^2 - z^*\rbr{-2\ii\Omega t}^n\rbr{a^-}^2}.
	\end{equation}

\end{namedthm} % Proposition
\begin{proof}
\begin{equation}
\ad_X^{(0)}Y = Y = \rbr{2\ii t}^0 Y;
\end{equation}

\begin{align}
	\ad_X^{(1)}Y &= \sbr{\ii\Omega t a^+ a^-, \frac{1}{2}\rbr{z
		\rbr{a^+}^2 - z^*\rbr{a^-}^2}}_- \nonumber \\
&= \ii \Omega t\rbr{z\rbr{a^+}^2 + z^*\rbr{a^-}^2} \nonumber \\
&= \frac{1}{2}\rbr{z \rbr{+2\ii\Omega t}^1
		\rbr{a^+}^2 - z^*\rbr{-2\ii\Omega t}^1\rbr{a^-}^2};
\end{align}
\begin{align}
	\ad_X^{(2)}Y &= \sbr{\ii\Omega t a^+ a^-, \ii \Omega t
\rbr{z\rbr{a^+}^2 + z^*\rbr{a^-}^2} }_- \nonumber \\
&= 2\rbr{\ii \Omega t}^2\rbr{ z\rbr{a^+}^2 - z^*\rbr{a^-}^2} \nonumber \\
&= \frac{1}{2}\rbr{z \rbr{+2\ii\Omega t}^2
		\rbr{a^+}^2 - z^*\rbr{-2\ii\Omega t}^2\rbr{a^-}^2};
\end{align}
Can be proved by induction.
\end{proof}

\begin{namedthm}{Proposition}
\begin{equation}
\rfun{\mscrU}{t}\rfun{S}{z}	= \rfun{S}{z^t}\rfun{\mscrU}{t},
\end{equation}
where
\begin{equation}
z^t \coloneqq z\ee^{2\ii\Omega t}.
\end{equation}

\end{namedthm} % Proposition
\begin{proof}
Sum up.
\end{proof}

Note one also has $\beta^t \coloneqq \beta \ee^{2\ii\Omega t}$.

\subsection{Particle numbers}

\begin{namedthm}{Lemma}[Factorising $\rfun{\widehat{S}}{z}$]
\begin{alignat}{3}
\rfun{\widehat{S}}{z} &=&&
\cfun{\exp}{+\frac{1}{2}\rbr{\widehat{a}^+}^2\ee^{+\ii\theta}\tanh r}
\nonumber \\
&&\cdot& \cfun{\exp}{\rbr{\frac{1}{2}+\widehat{a}^+\widehat{a}^-}\ln\sech r}
\cfun{\exp}{-\frac{1}{2}\rbr{\widehat{a}^-}^2\ee^{-\ii\theta}\tanh r}
\\
&=&&\cfun{\exp}{\frac{\beta}{2} \rbr{\widehat{a}^+}^2}
\cfun{\exp}{\rbr{\frac{1}{2}+\widehat{a}^+\widehat{a}^-}
\ln\sqrt{1-\vbr{\beta}^2}}
\cfun{\exp}{-\frac{\beta^*}{2}\rbr{\widehat{a}^-}^2}.
\end{alignat}
\end{namedthm} % Lemma

\begin{nameddef}{Particle number representation of $\Ket{\beta}$}
\begin{align}
\Braket{2n | \beta} &= \rbr{1-\vbr{\beta}^2}^{1/4}
\frac{\sqrt{\rbr{2n}!}}{2^n n!}\beta^n, \\
\Braket{2n+1 | \beta} &= 0.
\end{align}

\end{nameddef}

\begin{equation}
	\Braket{\beta | \widehat{a}^+\widehat{a}^- | \beta}
	= \Braket{0 | \rfun{S^\dagger}{z}\widehat{a}^+
	\widehat{a}^-\rfun{S}{z} | 0}
	= \sinh^2 r = \frac{\vbr{\beta}^2}{1-\vbr{\beta}^2}.
\end{equation}

\begin{equation}
	\Braket{0 | \widehat{b}^+\widehat{b}^- | 0}
	= \Braket{0 | \rfun{S}{z}\widehat{a}^+\rfun{S^\dagger}{z}
	\rfun{S}{z}\widehat{a}^-\rfun{S^\dagger}{z} | 0}
	= \sinh^2 r = \frac{\vbr{\beta}^2}{1-\vbr{\beta}^2}.
\end{equation}

\subsection{Wave function}

\begin{namedthm}{Lemma}[Factorising $\rfun{\widehat{S}}{z}$]
\begin{equation}
	\rfun{\widehat{S}}{z} =
	\rfun{\expi}{\frac{m\Omega}{2}\sigma_z \, x^2}
	\rfun{\expi}{-\ln \lambda_z\,\frac{\widehat{x}\widehat{p} +
			\widehat{p}\widehat{x}}{2}}
	\rfun{\expi}{-\frac{\sigma_z}{2m\Omega}\, p^2},
\end{equation}
with
\begin{align}
	\lambda_z &\coloneqq \cosh r + \cos\phi\,\sinh r \equiv
	\ee^{+r} \cos^2\frac{\phi}{2}+\ee^{-r}\sin^2\frac{\phi}{2}, \\
	\sigma_z &\coloneqq \frac{1}{\cot\phi+\coth r\,\csc\phi} \equiv
	\frac{\sin\phi\,\sinh r}{\lambda_z}.
	\end{align}
\end{namedthm} % Lemma

\begin{namedthm}{Proposition}
\begin{equation}
\lambda_z^2\rbr{1+\sigma_z^2} = \lambda_{2z}
\end{equation}

\end{namedthm} % Proposition

\begin{nameddef}{Wave function of $\Ket{\beta}$}
\begin{align}
	&\phantom{{}={}} \Braket{x | \ee^{\ii b_1 \widehat{x}^2} \ee^{\ii b_2
		\rbr{\widehat{x} \widehat{p} + \widehat{p} \widehat{x}}/2} \ee^{\ii b_3
		\widehat{p}^2} | \psi} \nonumber \\
	&= \ee^{\ii b_1 x^2} \Braket{x | \ee^{\ii b_2
		\rbr{\widehat{x} \widehat{p} + \widehat{p} \widehat{x}}/2} \ee^{\ii b_3
		\widehat{p}^2} | \psi} \nonumber \\
	&= \ee^{\ii b_1 x^2 + b_2/2} \Braket{x \ee^{b_2} | \ee^{\ii b_3
	\widehat{p}^2} | \psi} \nonumber \\
	&= \frac{\ee^{\ii b_1 x^2 + b_2/2}}{\rbr{-4\pp\ii b_3}^{1/2}}
		\int_{-\infty}^{+\infty}\dif x'\,\cfun{\expi}{-\frac{\rbr{x\ee^{b_2}
		-x'}^2}{4b_3}}\Braket{x' | \psi}.
\end{align}
For $\rfun{\widehat{S}}{z}$, $b_1 = m\Omega\sigma_z/2$, $b_2 = -\ln \lambda_z$,
$b_3 = -\sigma_z/2m\Omega$, and $\Ket{\psi} = \Ket{0}$, so that
	\begin{align}
	&\phantom{{}={}} \rbr{\frac{m\Omega}{\pp}}^{-1/4} \Braket{x |
		\rfun{\widehat{S}}{z} | 0} \nonumber \\
	&= \rbr{\lambda_z\rbr{1+\ii\sigma_z}}^{-1/2}
	\cfun{\exp}{-\frac{m\Omega x^2}{2}
	\rbr{\frac{1}{\lambda_z^2\rbr{1+\ii\sigma_z}} - \ii\sigma_z}} \nonumber \\
	&= \rbr{\lambda_z^2\rbr{1+\sigma_z^2}}^{-1/4}
	\cfun{\exp}{-\frac{m\Omega x^2}{2 \lambda_z^2\rbr{1+\sigma_z^2}}}
	\cfun{\expi}{\frac{m\Omega x^2 \sigma_z}{2 \lambda_z^2\rbr{1+\sigma_z^2}}
		- \frac{\rfun{\arg}{1+\ii\sigma_z}}{2}} \nonumber \\
	&= \lambda_{2z}^{-1/4}
	\cfun{\exp}{-\frac{m\Omega}{2 \lambda_{2z}} x^2}
	\cfun{\expi}{\frac{m\Omega\sigma_{2z}}{2} x^2 - \frac{1}{2}
	\rfun{\arg}{1+\ii\sigma_z}}.
	%\nonumber \\
	\end{align}
\end{nameddef} % Wave function of $\Ket{\beta}$

\subsection{Density matrix and Wigner function}

\begin{nameddef}{Density matrix in the particle number representation}
\begin{align}
\Braket{2n_1 | \widehat{\rho} | 2n_2} &= \sqrt{1-\vbr{\beta}^2}
\frac{\sqrt{\rbr{2n_1}!\rbr{2n_2}!}}{2^{n_1+n_2}n_1!n_2!}
\beta^{n_1} \rbr{\beta^*}^{n_2} \nonumber \\
&= \sqrt{1-\vbr{\beta}^2}
\frac{\sqrt{\rbr{2n_1}!\rbr{2n_2}!}}{2^{n_1+n_2}n_1!n_2!}
\ee^{\ii\rbr{n1-n2}\phi}\rbr{\tanh r}^{n_1+n_2}.
\end{align}
\end{nameddef} % Density matrix in the particle number representation

\begin{nameddef}{Density matrix in the position representation}
\begin{equation}
\Braket{x_1 | \widehat{\rho} | x_2} = \sqrt{\frac{m\Omega}{2\lambda_{2z}}}
\cfun{\exp}{-\frac{m\Omega}{2\lambda_{2z}}\rbr{x_1^2 + x_2^2}}
\cfun{\expi}{\frac{m\Omega\sigma_{2z}}{2}\rbr{x_1^2-x_2^2}}.
\end{equation}
\end{nameddef} % Density matrix in the position representation

\begin{nameddef}{Wigner function}
\begin{align}
	&\phantom{{}={}} \rfun{W}{x, p} \nonumber \\
	&= \pp^{-1} \cfun{\exp}{- m\Omega\lambda_{-2z} x^2
	+ 2 \lambda_{2z} \sigma_{2z} x p - \lambda_{2z} \frac{p^2}{m\Omega}} \\
	&\equiv \pp^{-1} \cfun{\exp}{-m\Omega \frac{X^2}{\ee^{+2r}} -
		\frac{1}{m\Omega}\frac{P^2}{\ee^{-2r}}},
\end{align}
where
\begin{equation}
	\begin{pmatrix} \rbr{m\Omega}^{+1/2} X \\
		\rbr{m\Omega}^{-1/2} P \end{pmatrix} \coloneqq
	\begin{pmatrix}
	\cos\phi/2 & \sin\phi/2 \\ -\sin\phi/2 & \cos\phi/2
	\end{pmatrix}
	\begin{pmatrix} \rbr{m\Omega}^{+1/2} x \\
		\rbr{m\Omega}^{-1/2} p \end{pmatrix}
\end{equation}
are principle coordinates of the Wigner ellipse. One sees that
$\phi/2$ rotates the canonical coordinates for the Wigner function, while
$\ee^{+r}$ ($\ee^{-r}$) stretches (squeezes) the principle axis of $X$ ($P$),
if $r > 0$.
\end{nameddef} % Wigner function