\chapter{Factor analysis}
\label{factanal}

\section{Role of factor analysis}
\label{rolefa}

Most of the development of factor analysis has taken place outside the statistical community, most often in terms of Psychometrics which may partly reflect its origins in the study of intelligence.   The earliest cited reference is \cite{Spearman:1904}.  Factor Analysis and Principal Components Analysis are often confused with each other, just to be really awkward there is one method of performing Factor Analysis called Principal Component Extraction.   The two methods should never be confused.   Principal Components seeks orthogonal projections of the data according the variance maximisation with the hope of achieving some dimension reduction.   Factor analysis is all about studying the co-variance (or correlation) and is based on a statistical model.  We hope to describe the covariance relationships between many variables in terms of a few underlying, unobservable random quantities called factors.   If there is a group of highly correlated variables, which in turn are uncorrelated with other variables, perhaps these represent realisations of some underlying phenomena that is responsible for for the observed correlations.   It is an attempt to approximate the covarariance matrix $\boldsymbol{\Sigma}$.  It is not highly regarded by many statisticians, but it is used by many others.   There are currently many variations on a theme (such as Structural Equation Modelling) which are also very common in many applied literatures.

Whilst we can consider one model for factor analysis, there are two very different fitting methods, neither of which is entirely satisfactory.   Having found a solution there are a large number of possible rotations of the solution each of which aim to give the most interpretable solution.   In other words, don't be surprised if different computer programs give different ``Factor Analysis'' solutions for the same data.

\begin{figure}
\begin{picture}(100,200)(0,0)

\put(0,150){\fbox{f1}}
\put(30,150){\vector(2,-1){95}}
\put(30,150){\vector(2,1){95}}
\put(30,150){\vector(1,-1){95}}

\put(0,50){\fbox{f2}}
\put(30,50){\vector(2,-1){95}}
\put(30,50){\vector(2,1){95}}
\put(30,50){\vector(1,1){95}}

\put(150,0){\fbox{X1}}
\put(150,100){\fbox{X2}}
\put(150,200){\fbox{X3}}

\end{picture}
\caption{Factor analysis, dependence between three variables represented by two latent variables - is this sensible}
\end{figure}

Factor Analysis is normally carried out with a view to reification: the investigator usually has a conceptual model of some underlying entity which cannot be measured directly.   These latent, or hidden, variables are the factors in factor analysis.   The aim of factor analysis is that each of the $p$ observed variables can be represented by means of $q<p$ mutually uncorrelated common factors.   This will leave some uncorrelated residual specific to each of the observed variables, the uniqueness, which is not correlated with any of the remaining $p-1$ variables \footnote{Note that the diagonal of a correlation matrix is 1. This statement implies that only part of this 1 is due to the $q<p$ latent variables - this part is known as the communality.}.   It is possible to rotate the $q$ axes of common factors to new orthogonal or obligue axes to make the factor solution fit with existing theoretical ideas regarding the model.

\section{The factor analysis model}
\label{factanalmodel}

The orthogonal model underlying Factor Analysis can be described as follows:

\begin{displaymath}
\label{factanal}
\boldsymbol{x} = \boldsymbol{\mu} + \boldsymbol{\Gamma} \boldsymbol{\phi} + \boldsymbol{\zeta}
\end{displaymath}

Where $\boldsymbol{x}$ is an $1 \times p$ random vector.   $\boldsymbol{\mu}$ represents a vector of unknown constants (mean values), $\boldsymbol{\Gamma}$ is an unknown $p \times q$ matrix of constants referred to as the \textit{loadings}.   $\boldsymbol{\phi}$ is a $q \times 1$ unobserved random vector referred to as the \textit{scores} assumed to have mean $\boldsymbol{0}$ and covariance $\boldsymbol{\Sigma}_{\phi}$, it is commonly assumed that $\boldsymbol{\Sigma}_{\phi} = \boldsymbol{I}$.   $\boldsymbol{\zeta}$ is $1 \times p$ unobserved random error vector having mean $\boldsymbol{0}$ and by assumption a diagonal covariance $\boldsymbol{\psi}$ referred to as the \textit{uniqueness} or \textit{specific variance}.

With these assumptions, $cov(\boldsymbol{\phi}, \boldsymbol{\zeta}) = 0$, if $\boldsymbol{\Sigma}_{\phi} = \boldsymbol{I}$ then $cov(\boldsymbol{x}, \boldsymbol{\phi}) = \boldsymbol{\Gamma}$.   It is worth emphasising that unlike many multivariate techniques covered here, factor analysis is a statistical model for our observations, with the following distributional form:

\begin{displaymath}
\boldsymbol{x} \sim Normal(\boldsymbol{\mu}, \boldsymbol{\Gamma} \boldsymbol{\Gamma}^{T} + \boldsymbol{\psi})
\end{displaymath}

It may be slightly clearer to consider the way a vector of observations $\boldsymbol{x} = x_{1}, \ldots, x_{p}$ are modelled in factor analysis:

\begin{eqnarray*}
x_{1} &=& \mu_{1} + \sum_{k=1}^{q} \gamma_{1k} \phi_{k}  + \zeta_{1}\\
x_{2} &=& \mu_{2} + \sum_{k=1}^{q} \gamma_{2k} \phi_{k}  + \zeta_{2}\\
&\vdots&\\
x_{p} &=& \mu_{p} + \sum_{k=1}^{q} \gamma_{pk} \phi_{k}  + \zeta_{p}
\end{eqnarray*}

Note that under the terms of this model:

\begin{equation}
\label{communalities}
var(x_{j}) = \gamma_{j1}^{2} + \gamma_{j2}^{2} + \ldots + \gamma_{jq}^{2} + var(\zeta_{j})
\end{equation}

One potential problem with this model should be immediately obvious, there can be rather more parameters than data.   For example, note that the covariance matrix $\boldsymbol{\Sigma}$ has $p(p+1)/2$ parameters, the factor model $ \boldsymbol{\Gamma} \boldsymbol{\Gamma}^{T} + \boldsymbol{\psi})$ has $qp - q(q-1)/2 + p$ parameters.  One issue arises whereby a factor analsis model must be constrained in order to ensure identifiability.  Clearly, $p(p+1)/2 \geq qp - q(q-1)/2 + p$, or:

\begin{equation}
\label{qp}
q \leq \frac{2p + 1 - \sqrt{8p-1}}{2}
\end{equation}

This gives some maximum values of $q$ for given values of $p$:

\begin{tabular}{rr}
\hline
$p$ & max $q$ \\
\hline
1 & 0\\
2 & 0\\
3 & 1\\
4 & 1\\
5 & 2\\
6 & 3\\
7 & 3\\
8 & 4\\
9 & 5\\
10 & 6\\
\hline
\end{tabular}

Where $q<p$, the right hand side of \ref{communalities} indicates how much of $var(x_{j}$ is explained by the model, a concept referred to as the communality.   Consideration of the order of the model leads on to a point we will consider later, degrees of freedom after fitting a $q$ factor model:

\begin{equation}
\label{dffact}
df = \frac{p(p+1)}{2} - qp + \frac{q(q-1)}{2} - p = \frac{(p-q)^{2} - (d+m)}{2}
\end{equation}

\subsection{Centred and standardised data}

In practice it is often much simpler to centre the data, so that we model:

\begin{equation}
\label{facentre}
x_{j} - \mu_{j} = \sum_{k=1}^{q} \gamma_{k} \phi_{k} + \zeta_{j}; j = 1, \ldots, p
\end{equation}

or even to standardise the variables so that in effect we are modelling the correlation matrix rather than the covariance matrix.

\begin{equation}
\label{fastandardise}
\frac{x_{j} - \mu_{j}}{\sigma_{jj}} = \sum_{k=1}^{q} \gamma_{k} \phi_{k} + \zeta_{j}; j = 1, \ldots, p
\end{equation}

Regardless of the data matrix used, factor analysis is essentially a model for $\boldsymbol{\Sigma}$, the covariance matrix of $\boldsymbol{x}$,

\begin{displaymath}
\label{covdecomp}
\boldsymbol{\Sigma} = \boldsymbol{\Gamma}\boldsymbol{\Gamma}^{T} + \boldsymbol{\psi}
\end{displaymath}

\subsection{Factor indeterminacy}

We now consider another problem with factor analysis.   It is a very indeterminate model, specifically it is unchanged if we replace $\boldsymbol{\Gamma}$ by $\boldsymbol{K} \boldsymbol{\Gamma}$ for any orthogonal matrix $\boldsymbol{K}$.   However, this can be turned to our advantage, with sensible choice of a suitable orthogonal matrix $\boldsymbol{K}$ we can achieve a rotation that may yield a more interpretable answer.  Factor analysis therefore requires an additional stage, having fitted the model we may wish to consider rotation of the coefficients.

\subsection{Strategy for factor analysis}

 To fit the model, we therefore need to:

\begin{itemize}
\item Estimate the number of common factors $q$.
\item Estimate the factor loadings $\boldsymbol{\Gamma}$
\item Estimate the specific variances $\boldsymbol{\psi}^{2}$
\item On occasion, estimate the factor scores $\boldsymbol{\phi}$
\end{itemize}

We will now consider fitting methods for factor analysis.   It will be obvious that the preferred method in R is the maximum likelihood method, but we will first consider methods based around principal components to reinforce some ideas about the model.

\section{Principal component extraction}

We have already used the spectral decomposition to obtain one possible factoring of the covariance matrix $\boldsymbol{\Sigma}$.

\begin{displaymath}
\boldsymbol{\Sigma} = \boldsymbol{E} \boldsymbol{\Lambda}\boldsymbol{E}^{T}
\end{displaymath}

which can be expanded:

\begin{eqnarray*}
\boldsymbol{\Sigma} &=& \lambda_{1} \boldsymbol{e}_{1} \boldsymbol{e}_{1}^{T} +  \lambda_{2} \boldsymbol{e}_{2} \boldsymbol{e}_{2}^{T} + \ldots  \lambda_{p} \boldsymbol{e}_{p} \boldsymbol{e}_{p}^{T} \\
 &=& \left( \sqrt{\lambda_{1}} \boldsymbol{e}_{1}, \sqrt{\lambda_{2}} \boldsymbol{e}_{2}, \ldots, \sqrt{\lambda_{p}} \boldsymbol{e}_{p} \right) \left( \begin{array}{c} \sqrt{\lambda_{1}} \boldsymbol{e}_{1} \\  \sqrt{\lambda_{2}} \boldsymbol{e}_{2} \\ \vdots \\  \sqrt{\lambda_{p}} \boldsymbol{e}_{p} \end{array} \right)
\end{eqnarray*}

Of course, in practice we don't know $\boldsymbol{\Sigma}$ and we use  $S$ (or we standardise the variables and use $R$ - it should be remembered that this is rather a big decision when working with principal components).   Referring back to our data, it should be remembered that the spectral decomposition yields linear principal components as follows:

\begin{eqnarray*}
z_{1} &=& e_{11}x_{1} + e_{12}x_{2} + \ldots + e_{1p} x_{p}; var(z_{1})=\lambda_{1}\\
z_{2} &=& e_{21}x_{2} + e_{22}x_{2} + \ldots + e_{2p} x_{p}; var(z_{2})=\lambda_{2}\\
&\vdots&\\
z_{p} &=& e_{p1}x_{1} + e_{p2}x_{1} + \ldots + e_{1p} x_{p}; var(z_{p})=\lambda_{p}
\end{eqnarray*}

which in matrix notation this can be expressed as:

\begin{equation}
\label{pcfact}
\boldsymbol{Z} = \boldsymbol{E}\boldsymbol{X}
\end{equation}

where  $\boldsymbol{Z} = \left( \begin{array}{c} z_{1} \\ z_{2} \\ \vdots \\ z_{p} \end{array} \right)$,  $\boldsymbol{X} = \left( \begin{array}{c} X_{1} \\ X_{2} \\ \vdots \\ X_{p} \end{array} \right)$ and  $\boldsymbol{E} = \left( \begin{array}{cccc} e_{11} & e_{12} & \hdots & e_{1p} \\
e_{21}& e_{22} & \hdots & e_{2p} \\
 \vdots & \vdots & \ddots & \vdots \\
e_{p1} & e_{p2} & \hdots & e_{pp}  \end{array} \right)$.

Multiplying both sides of \ref{pcfact} by $\boldsymbol{E}^{-1}$gives:

\begin{equation}
\boldsymbol{E}^{-1} \boldsymbol{Z} = \boldsymbol{X}
\end{equation}

We know orthogonal matrices generally that $\boldsymbol{E}^{-1} = \boldsymbol{E}^{T}$ so we can invert the transformation by using

\begin{equation}
\boldsymbol{X} = \boldsymbol{E}^{T} \boldsymbol{Z}
\end{equation}

which can be expanded as:

\begin{eqnarray*}
x_{1} &=& e_{11}z_{1} + e_{21} z_{2}  + \ldots + e_{p} z_{p}  \\
x_{2} &=& e_{12}z_{2} + e_{22} z_{2} + \ldots + e_{p} z_{p}  \\
&\vdots& \\
x_{p} &=& e_{1p}z_{1} + e_{2p} z_{2} + \ldots + e_{pp} z_{p}
\end{eqnarray*}

which we could express as;

\begin{eqnarray*}
x_{1} &=& (e_{11} \sqrt{\lambda_{1}}) \frac{z_{1}}{\sqrt{\lambda_{1}}} + (e_{21} \sqrt{\lambda_{2}}) \frac{z_{2}}{\sqrt{\lambda_{2}}} + \ldots +  (e_{p1} \sqrt{\lambda_{p}}) \frac{z_{p}}{\sqrt{\lambda_{p}}} \\
x_{2} &=& (e_{12} \sqrt{\lambda_{1}}) \frac{z_{1}}{\sqrt{\lambda_{1}}} + (e_{12} \sqrt{\lambda_{1}}) \frac{z_{1}}{\sqrt{\lambda_{1}}} + \ldots +  (e_{p2} \sqrt{\lambda_{p}}) \frac{z_{p}}{\sqrt{\lambda_{p}}} \\
&\vdots& \\
x_{p} &=& (e_{1p} \sqrt{\lambda_{1}}) \frac{z_{1}}{\sqrt{\lambda_{1}}} + (e_{2p} \sqrt{\lambda_{2}}) \frac{z_{2}}{\sqrt{\lambda_{2}}} + \ldots +  (e_{pp} \sqrt{\lambda_{p}}) \frac{z_{p}}{\sqrt{\lambda_{p}}}
\end{eqnarray*}

and if we set $\gamma_{jk} = (e_{jk} \sqrt{\lambda_{j}})$ and $\phi_{j} = z_{j} / \sqrt{\lambda_{j}}$ we have a clear link with the factor analysis model given in equation \ref{factanal}.   If we try writing this in matrix terminology, our loadings matrix $\boldsymbol{\Gamma}$ is the $p \times p$ matrix where the $j$th column is given by  $\sqrt{\lambda_{j}} \boldsymbol{e}_{j}$ we now have:

\begin{displaymath}
\boldsymbol{S} = \boldsymbol{\Gamma} \boldsymbol{\Gamma}^{T}
\end{displaymath}

which is getting us part of the way to our factor analysis model.   Before going any further we will reinforce this procedure by considering how to obtain these values from within R.  Note that under the principal component solution, the estimated loadings do not alter as the number of factors is increased or decreased. We are going to load the economic data, and carry out a decomposition of the correlation matrix $\boldsymbol{R}$.

\singlespacing
\begin{verbatim}
> econ <- read.csv("econ.csv", row.names = 1)
> econ.cor <- cor(econ)
> econ.ev <- eigen(econ.cor)
> loadings <- matrix(0,9,9)
> for (i in 1:9){
> loadings[,i] <- sqrt(econ.ev$values[i]) * econ.ev$vectors[,i]
> }
> econ.cor - loadings %*% t(loadings) ## should equal zero
\end{verbatim}
\onehalfspacing

Clearly we don't actually want to use a decomposition with with $q=p$ variables.   As might be rather obvious bearing in mind our earlier use of principal components, we wish to partition $\boldsymbol{\Lambda}$ into $\boldsymbol{\Lambda}_{1} = \lambda_{1}, \lambda_{2}, \ldots, \lambda_{q}$ and $\boldsymbol{\Lambda_{2}} = \lambda_{q+1}, \ldots, \lambda_{p}$ with the corresponding eigenvectors.   As a consequence, we reduce the size of our $\boldsymbol{\Gamma}$ matrix, i.e. to neglect the contribution of  $\lambda_{q+1} \boldsymbol{e}_{q+1} \boldsymbol{e}_{q+1}^{T} + \ldots  \lambda_{p} \boldsymbol{e}_{p} \boldsymbol{e}_{p}^{T}$.   So when considering our model for the data, we wish to partition our factors as follows:

\begin{eqnarray*}
x_{1} &=& e_{11}z_{1} + e_{21} z_{2}  + \ldots + e_{q1} z_{q} +  e_{q+1,1} z_{q+1} + \ldots +  e_{p1} z_{p} \\
x_{2} &=& e_{12}z_{2} + e_{22} z_{2} + \ldots + e_{q2} z_{q} +  e_{q+1,2} z_{q+1} + \ldots + e_{p2} z_{p} \\
&\vdots& \\
x_{p} &=& e_{1p}z_{1} + e_{2p} z_{2} + \ldots + e_{qp} z_{q} + e_{q+1,p} z_{q+1} + \ldots + e_{pp} z_{p}
\end{eqnarray*}

and if we set $ e_{q+1,j} z_{q+1} + \ldots +  e_{pj} z_{p} = \zeta_{j}; j = 1, \ldots, p$ we can rewrite this as:

\begin{eqnarray*}
x_{1} &=& e_{11} z_{1} + e_{21} z_{2} + \ldots + e_{q1} z_{q} +  \zeta_{1}\\
x_{2} &=& e_{12} z_{1} + e_{22} z_{2} + \ldots + e_{q2} z_{q} +  \zeta_{2}\\
&\vdots& \\
x_{p} &=& e_{1p} z_{1} + e_{2p} z_{1} + \ldots + e_{qp} z_{q} + \zeta_{p}
\end{eqnarray*}

As earlier, we can expressed this as:

\begin{eqnarray*}
x_{1} &=& (e_{11} \sqrt{\lambda_{1}}) \frac{z_{1}}{\sqrt{\lambda_{1}}} + (e_{21} \sqrt{\lambda_{2}}) \frac{z_{2}}{\sqrt{\lambda_{2}}} + \ldots +  (e_{q1} \sqrt{\lambda_{q}}) \frac{z_{q}}{\sqrt{\lambda_{q}}} +  \zeta_{1}\\
x_{2} &=& (e_{12} \sqrt{\lambda_{1}}) \frac{z_{1}}{\sqrt{\lambda_{1}}} + (e_{12} \sqrt{\lambda_{1}}) \frac{z_{1}}{\sqrt{\lambda_{1}}} + \ldots +  (e_{q2} \sqrt{\lambda_{q}}) \frac{z_{q}}{\sqrt{\lambda_{q}}} +  \zeta_{2}\\
&\vdots& \\
x_{p} &=& (e_{1p} \sqrt{\lambda_{1}}) \frac{z_{1}}{\sqrt{\lambda_{1}}} + (e_{2p} \sqrt{\lambda_{2}}) \frac{z_{2}}{\sqrt{\lambda_{2}}} + \ldots +  (e_{qp} \sqrt{\lambda_{q}}) \frac{z_{q}}{\sqrt{\lambda_{q}}} +  \zeta_{p}
\end{eqnarray*}

where $\gamma_{jk} = (e_{jk} \sqrt{\lambda_{j}})$ and $\phi_{i} = z_{i} / \sqrt{\lambda_{i}}$ as before, notice as stated at the outset that $var(\boldsymbol{\zeta}) = \boldsymbol{\psi}$.   If we consider this in terms of the decomposition of the covariance matrix we have:

\begin{equation}
\boldsymbol{\Sigma} = \left( \sqrt{\lambda_{1}} \boldsymbol{e}_{1}, \sqrt{\lambda_{2}} \boldsymbol{e}_{2}, \ldots, \sqrt{\lambda_{q}} \boldsymbol{e}_{q} \right) \left( \begin{array}{r} \sqrt{\lambda_{1}} \boldsymbol{e}_{1} \\  \sqrt{\lambda_{2}} \boldsymbol{e}_{2} \\ \vdots \\  \sqrt{\lambda_{q}} \boldsymbol{e}_{q} \end{array} \right) +
\left[ \begin{array}{rrrr} \psi_{1} & 0 & \hdots & 0\\
0 & \psi_{2} & \hdots & 0\\
\vdots & \vdots & \ddots & \vdots\\
0 & 0 & \hdots & \psi_{p}
\end{array} \right]
\end{equation}

Where now $\psi_{j} = var(\zeta_{j}) =  \sigma_{jj} - \sum_{k=1}^{q} \gamma_{jk}^{2}$ for $k = 1, 2, \ldots, q$.

Estimates of the specific variances are given by diagonal elements of the matrix $\boldsymbol{\hat{\Sigma}} - \boldsymbol{\hat{\Gamma}}\boldsymbol{\hat{\Gamma}}^{T}$, i.e:

\begin{equation}
\boldsymbol{\hat{\psi}} =
\left[ \begin{array}{rrrr} \psi_{1} & 0 & \hdots & 0\\
0 & \psi_{2} & \hdots & 0\\
\vdots & \vdots & \ddots & \vdots\\
0 & 0 & \hdots & \psi_{p}
\end{array} \right]
\mbox{with}  \psi_{j} = \sigma_{jj} - \sum_{k=1}^{q} \gamma_{jk}^{2}
\end{equation}

So, when using the principal component solution of $\boldsymbol{\hat{\Sigma}}$, it is specified in terms of eigenvalue-eigenvector pairs ($\hat{\lambda}_{1}, \hat{\boldsymbol{e}}_{1}$), ($\hat{\lambda}_{2}, \hat{\boldsymbol{e}}_{2}$), $\ldots$, ($\hat{\lambda}_{p}, \hat{\boldsymbol{e}}_{p}$), where $\hat{\lambda}_{1} \geq \hat{\lambda}_{2} \geq \ldots \geq \hat{\lambda}_{p}$.   If we wish to find a $q<p$ solution of common factors, then the estimated factor loadings are given by:

\begin{displaymath}
\boldsymbol{\hat{\Gamma}} = \left( \sqrt{\lambda_{1}} \boldsymbol{e}_{1}, \sqrt{\lambda_{2}} \boldsymbol{e}_{2}, \ldots, \sqrt{\lambda_{q}} \boldsymbol{e}_{q} \right)
\end{displaymath}

As with the factor analysis model given earlier, the factors $\boldsymbol{\phi}$ have identity covariance matrix

\begin{displaymath}
var(\boldsymbol{\phi}) = var \left(\sqrt{\Lambda_{1}} \boldsymbol{\Gamma}_{1}^{T}(\boldsymbol{x} - \boldsymbol{\mu}) \right) = \boldsymbol{I}_{q},
\end{displaymath}

 and are uncorrelated with the residuals:

\begin{displaymath}
cov(\boldsymbol{\phi}, \boldsymbol{\zeta}) = cov \left( \sqrt{\Lambda_{1}} \boldsymbol{\Gamma}_{1}^{T}(\boldsymbol{x} - \boldsymbol{\mu}),  \boldsymbol{\Gamma}_{2}\boldsymbol{\Gamma}_{2}^{T}(\boldsymbol{x} - \boldsymbol{\mu}) \right) = \sqrt{\boldsymbol{\Lambda_{1}}} \boldsymbol{\Gamma}_{1}^{T} \boldsymbol{\Sigma} \boldsymbol{\Gamma}_{2} \boldsymbol{\Gamma}_{2}^{T} = 0
\end{displaymath}

However, one major objection to this principal component ``solution'' is that it can also be seen that each $\zeta_{i}$ contains the same $z_{i}$ so they are not mutually unrelated.   Hence the latent variables obtained using the principal component method do not explain all the correlation structure in our data $\boldsymbol{X}$.   The covariance matrix for the errors is now:

\begin{displaymath}
var(\boldsymbol{\zeta}) = \boldsymbol{\Gamma}_{2} \boldsymbol{\Lambda}_{2} \boldsymbol{\Gamma}_{2}^{T}
\end{displaymath}

This additional step can be carried out fairly easily in R.   We only need to discard the unwanted components and estimate the uniquenesses:

\singlespacing
\begin{verbatim}
> loadings4 <- matrix(0,9,4)
> for (i in 1:4){
> loadings4[,i] <- sqrt(econ.ev$values[i]) * econ.ev$vectors[,i]
> }
> LLt <- loadings4 %*% t(loadings4)
> unique <- diag(econ.cor - LLt)
> error <- econ.cor - (LLt + unique)
\end{verbatim}
\onehalfspacing

and so \texttt{loadings4} gives us the matrix of loadings, \texttt{unique} gives us an estimate of the uniquenesses.  It should be noted that the loadings are unaltered as the number of factors $q$ is changed.   It may be noted that the diagonal elements of $\boldsymbol{\hat{\Sigma}}$ are given by the diagonal elements of $\boldsymbol{\Gamma}\boldsymbol{\Gamma}^{T} + \boldsymbol{\psi}$, but this is not true of the off-diagonal elements.  There are error terms associated with our decomposition of the covariance matrix, these can be easily found from \texttt{error}.  Clearly these are values we wish to see minimised.

We will consider interpretation of factor structure in more detail later.   However, for now it may be of interest to examine the four factor solution.   It would appear that the first factor represents some kind of contrast between agriculture and other industries (with the exception of finance and mining).

\singlespacing
\begin{verbatim}
\loadings4
                                   [,1]        [,2]        [,3]        [,4]
Agriculture                -0.978199871 -0.07760625  0.05173168 -0.02899271
Mining                     -0.002342834 -0.90214224 -0.21179672 -0.06592893
Manufacture                 0.645370678 -0.52159027 -0.15703856  0.34982446
PowerSupplies               0.476333161 -0.37897538 -0.58769654 -0.39731951
Construction                0.608061420 -0.07694001  0.15838634  0.66387307
ServiceIndustries           0.707975893  0.51045159 -0.12126845  0.05137022
Finance                     0.138717720  0.66237521 -0.61559512  0.05147600
SocialAndPersonalServices   0.723602099  0.32374238  0.32749903 -0.40851903
TransportAndCommunications  0.684640120 -0.29451591  0.39342807 -0.31637790
\end{verbatim}
\onehalfspacing

\subsection{Diagnostics for the factor model}

We can define a residual matrix as:

\begin{equation}
\boldsymbol{\epsilon} = \boldsymbol{{S}} - \left(\boldsymbol{L}\boldsymbol{L}^{T} + \boldsymbol{\psi} \right)
\end{equation}

By construction, the diagonal elements of this residual matrix will be zero.   A decision to retain a particular $q$ factor model could be made depending on the size of the off-diagonal elements.   Rather conveniently, there is an inequality which gives us:

\begin{equation}
\left[ \boldsymbol{\epsilon} = \boldsymbol{\hat{\Sigma}} - \left(\boldsymbol{L}\boldsymbol{L}^{T} + \boldsymbol{\psi} \right) \right] \leq \hat{\lambda}_{q+1}^{2} + \cdots + \hat{\lambda}_{p}^{2}
\end{equation}

So it is possible to check the acceptability of fit in terms of a small sum of squares of neglected eigenvalues.

In a similar manner to that used in principal components, it is possible to use the eigenvalues to indicate the proportion of variance explained by any given factor.   So instead of examining discarded components we could examine those we intend to retain.   Bearing in mind that $trace(\boldsymbol{\Sigma} = \sigma_{11} + \sigma_{22} + \ldots + \sigma_{pp}$, we know that the amount of variation explained by the first factor $\gamma_{11}^{2} + \gamma_{21}^{2} + \ldots + \gamma_{p1}^{2} = (\sqrt{\lambda_{1}} \boldsymbol{e}_{1})^{T}(\sqrt{\lambda_{1}} \boldsymbol{e}_{1}) = \lambda^{1}$.

So we know that the $j$-th factor explains the following proportion of total sample variance:

\begin{equation}
\frac{\lambda_{j}}{trace(\boldsymbol{S})}
\end{equation}

which reduces to $\frac{\lambda_{j}}{p}$ when using standardised variables (the correlation matrix).

It is actually in the context of factor analysis that the Kaiser criterion was developed.   This is implemented by default in a number of computer programs, basically we retain factors which are explaining more than the average amount of variance; if we are decomposing the correlation matrix we retain all factors where the corresponding eigenvalues are greater than one.   We can consider the number of components to retain from our earlier eigen analysis.   The following R output gives the eigenvalues, and the proportion and cumulating proportion explained by each possible factor.

\singlespacing
\begin{verbatim}
> econ.ev$values
[1]  3.482820 2.132332 1.098373 0.9984261 0.5450933
[6] 0.3836385 0.2224905 0.1367327 0.0000930

> econ.ev$values / 0.09
[1] 38.698003578 23.692581759 12.204146983 11.093622860  6.056592340
[6]  4.262650255  2.472116648  1.519252072  0.001033506

> cumsum(econ.ev$values/0.09)
[1]  38.69800  62.39059  74.59473  85.68836  91.74495  96.00760  98.47971
[8]  99.99897 100.00000
\end{verbatim}
\onehalfspacing

Considering the eigenvalues first, using the Kaiser criterion would lead us to select three components, but it should be noted that the fourth component is only just below 1 (0.998) giving perhaps some warning as to the arbitrariness of this device.   There are 9 variables, so we divide by 9 (and multiply by 100 to express the proportions as a percentage).   Cumulative values are also given.   We require five components to explain over 90\% of the variation.   Remember that according to formula \ref{qp} this is the largest value of $q$ that can be contemplated with nine manifest variables.

\subsection{Communalities}

Another important concept are the communalities.   In the case of standardised variables, these indicate the proportion of variance of a manifest variable explained by its relevant factor structure.   These are simply estimated as:

\begin{equation}
\xi_{jk}^{2} = \gamma_{j1}^{2} +  \gamma_{j2}^{2} + \cdots +  \gamma_{jq}^{2}
\end{equation}

Terminology can now be supplied for the decomposition of the variance of $\boldsymbol{x}$ given earlier in \ref{communalities} to reflect the reduced dimensionality.

\begin{equation}
\label{communality}
var(x_{j}) = \underbrace{\gamma_{j1}^{2} + \gamma_{j2}^{2} + \ldots + \gamma_{jq}^{2}}_{communality\ of\ x_{j}} + \underbrace{\psi_{i}}_{specificity\ of\ x_{j}}
\end{equation}

For standardised variables, $var(x_{j}) = 1$, therefore: $\gamma_{i1}^{2} + \gamma_{i2}^{2} + \ldots + \gamma_{i1}^{q} \leq 1$ and $-1 \leq \gamma_{jk} \leq 1$.

These are fairly simply extracted from our matrix of loadings by squaring all entries and summing by row:

\singlespacing
\begin{verbatim}
> row.names(loadings4) <- row.names(econ.cor)
> apply(loadings4^2, 1, sum)
               Agriculture                     Mining
                 0.9664145                  0.8630706
               Manufacture              PowerSupplies
                 0.8355980                  0.8737656
              Construction          ServiceIndustries
                 0.8414721                  0.7791356
                   Finance  SocialAndPersonalServices
                 0.8395906                  0.9025525
TransportAndCommunications
                 0.8103523
\end{verbatim}
\onehalfspacing

These appear to be reasonably high for most variables which would suggest a plausible fit for the factor model.

\subsection{Principal Factor solution}

We might have been worried about the way our model above doesn little to account for the off-diagonal elements of $\boldsymbol{\hat{\Sigma}}$.   Principal factoring (which seems to have rather fewer advocates) considering decomposing a reduced matrix.   We know that the diagonal elements of our covariance matrix are given by $\sigma_{jj} = \xi_{j}^{2} + \psi_{j}$, so having determined the number $q$ of common factors needed, we can decompose the reduced covariance matrix.   If we obtain some initial estimates of $\boldsymbol{\psi}$, we can re-estimate the remaining part of the decomposition $\boldsymbol{\Gamma} \boldsymbol{\Gamma}^{T}$.

\begin{equation}
\sigma_{jj} = \xi_{j}^{2} + \psi_{j}
\end{equation}

If we had some initial estimate of $\boldsymbol{\psi}$, $\widetilde{\boldsymbol{\psi}}$ say,  we could obtained a ``reduced'' covariance matrix

\begin{displaymath}
\boldsymbol{S} = \left( \begin{array}{rrrr}
\widetilde{\xi}_{1}^{2} & s_{12} & \cdots & s_{1p}\\
s_{21} & \widetilde{\xi}_{2}^{2} & \cdots & s_{2p}\\
\vdots & \vdots & \ddots & \vdots \\
s_{p1} & s_{p2} & \cdots & \widetilde{\xi}_{p}^{2}
\end{array} \right)
\end{displaymath}
and carry out an eigendecomposition of this matrix, updating our estimates of the uniqueness and repeat until convergence.

So all we need is an initial estimate of $\widetilde{\boldsymbol{\psi}}$.   Many programs conduct a multiple regression of each manifest variable on each other, and use $s_{jj} r_{j}^{2}$.   We then conduct a principal component analysis on $\boldsymbol{S} - \boldsymbol{\psi}$to find $\boldsymbol{\Gamma}$.   $\boldsymbol{\psi}$ can then be recalculated as the diagonal of $\boldsymbol{S} - \boldsymbol{\Gamma} \boldsymbol{\Gamma}^{T}$ and we extract a further set of principal components.   These latter steps are repeated until convergence, which can be slow if it happens at all.   As we are working with the correlation matrix, it's easy enough to find these intial values:

\singlespacing
\begin{verbatim}
> r2s <- vector("numeric", 9)
>
> for (i in 1:9){
+ y <- econ[,i]
+ x <- econ[,-i]
+ mod <- lm(y~as.matrix(x))
+ r2s[i] <- summary(mod)$r.squared
+ }
>
>
> unique <- diag(1-r2s)
> diag(unique)
[1] 0.0001429627 0.0420230140 0.0006887118 0.1325518542 0.0138610514
[6] 0.0016432160 0.0043879128 0.0007569780 0.0161276459
\end{verbatim}
\onehalfspacing

And all that is now required is to repeatedly implement the loop stage.   This is presented as a function in \texttt{mvmmisc.R}, but it is worth pasting through this manually to see how the procedure works.

\singlespacing
\begin{verbatim}
> new <- econ.cor - unique
> new.ev <- eigen(new)
>
> loadings4pf <- matrix(0,9,4)
> for (i in 1:4){
+ loadings4pf[,i] <- sqrt(new.ev$values[i]) * new.ev$vectors[,i]
+ }
>
> LLt <- loadings4pf %*% t(loadings4pf)
>
> unique.f <- econ.cor - LLt
> diag(unique) <- diag(unique.f)
>
> diag(unique)
[1] 0.02890147 0.14965321 0.15824274 0.25101144 0.14818053 0.21896192 0.15269320
[8] 0.09280381 0.20001567
>
> loadings4pf
              [,1]        [,2]        [,3]         [,4]
 [1,] -0.980556270 -0.06951113  0.06899117 -0.004043332
 [2,] -0.007596438 -0.88584019 -0.20248258  0.156770695
 [3,]  0.644457570 -0.53242421 -0.27602489 -0.258391989
 [4,]  0.453749164 -0.35005068 -0.40926760  0.503055476
 [5,]  0.607710009 -0.08927793 -0.03546047 -0.687953501
 [6,]  0.711408766  0.50862889 -0.12606387 -0.018444548
 [7,]  0.140377243  0.67201322 -0.59955488  0.128581528
 [8,]  0.727082131  0.31778092  0.43171651  0.301966740
 [9,]  0.681111481 -0.30230726  0.45690884  0.189515460
\end{verbatim}
\onehalfspacing

It should be noted that in addition to slow (or no) convergence, different results will be obtained depending on whether correlation or covariance matrix is used.  However, this approach does not require any distributional assumptions so may be of some use of multivariate normality cannot be claimed, even by refuge to the central limit theorem.   \cite{Harmon:1967} does indicate further fundamental differences between the principal component and this solution.

Little more needs to be said about this method of factor analysis, we now turn our attention to a more promising approach, maximum likelihood.

\section{Maximum likelihood solutions}
\label{mlfact}

Obvious conclusions might be drawn by noting that R only offers this method of fitting factor analysis models, see the helpfile for the relevant function \texttt{?factanal} as well as \cite{Venables+Ripley:2002}.   It should be noted from the outset that this method is invariant to changes in scale, a proof given in \cite{Seber:1984}.   In other words, it doesn't matter whether the correlation or the covariance matrix are used, or indeed whether any other scale changes are applied.   There are a number of other advantages associated with maximum likelihood fitting, but the problem of Heywood cases still remains, whereby some of the unique variances are estimated with a negative value.

We also need to impose an additional assumption over and above the factor analysis assumptions set out earlier, namely that the following matrix:

\begin{equation}
\label{diagconstraint}
\boldsymbol{\Gamma}^{T} \boldsymbol{\Psi}^{-1} \boldsymbol{\Gamma}
\end{equation}

must be diagonal to enable model fitting.   Having fitted the model, as we will find out later we are free to rotate the solution.

If the maximum likelihood method is so superior, the obvious question arises as to either of the principal component based methods have remained in use for so long.   There is in fact a long and far from trouble free history in terms of trying to develop a maximum likelihood solution for factor analysis, details of an earlier approach to maximum likelihood fitting are given in \cite{Morrison:1976}.   In any case, we well assume that our data follows a multivariate normal distribution, which will have the following likelihood:

\begin{equation}
\label{mvnlike}
L(\boldsymbol{x}; \boldsymbol{\mu}, \boldsymbol{\Sigma}) = (2 \pi)^{-\frac{np}{2}} |\boldsymbol{\Sigma}|^{-\frac{n}{2}} e^{-\frac{1}{2}tr\left( \boldsymbol{\Sigma}^{-1} ( \sum_{i=1}^{n}(\boldsymbol{x}_{i} - \boldsymbol{\bar{x}})(\boldsymbol{x}_{i} - \boldsymbol{\bar{x}})^{T} + n(\boldsymbol{\bar{x}} - \boldsymbol{\mu})(\boldsymbol{\bar{x}} - \boldsymbol{\mu})^{T} ) \right)}
\end{equation}
we wish to solve this in terms of our factor analysis model and therefore need to find an expression for the likelihood of $L((\boldsymbol{x}; \boldsymbol{\mu}, \boldsymbol{\Gamma}, \boldsymbol{\psi})$.

$\boldsymbol{\mu}$ is a nuisance parameter for our purposes here, we can either get rid of it by using the estimate $\boldsymbol{\hat{\mu}} = \boldsymbol{\bar{x}}$ and hence use the profile likelihood to find $\boldsymbol{\hat{\Gamma}}$ and  $\boldsymbol{\hat{\psi}}$ , or we can factorise the likelihood as  $L(\boldsymbol{S}; \boldsymbol{\bar{x}}, \boldsymbol{\Sigma}) L(\boldsymbol{\bar{x}}; \boldsymbol{\mu}, \boldsymbol{\Sigma})$.   In this latter case, $\boldsymbol{bar{x}}$ and $\boldsymbol{S}$ are the joint sufficient statistics for $\boldsymbol{\mu}$ and $\boldsymbol{\Sigma}$ respectively, for the purposes of factor analysis we only require the first part of the factorised likelihood which can be estimated by conditional maximum likelihood.   Note that as $\boldsymbol{bar{x}}$ and $\boldsymbol{S}$ are independent this is also the marginal likelihood.

Taking logs of \ref{mvnlike}, and collecting constant terms into $c_{1}$ and $c_{2}$ we can say that we wish to maximise:

\begin{equation}
\label{fall}
\ln L = c_{1} - c_{2} \left( \ln |\boldsymbol{\Gamma} \boldsymbol{\Gamma}^{T} + \boldsymbol{\psi}| + trace(\boldsymbol{\Gamma} \boldsymbol{\Gamma}^{T} + \boldsymbol{\psi})^{-1}\boldsymbol{S} \right)
\end{equation}

By taking this likelihood, along with the diagonality contraints indicated in \ref{diagconstraint} all we need is a procedure for estimation.

An intial estimate of $\widetilde{\boldsymbol{\psi}}$ has to be made as before, \cite{Lawley+Maxwell:1971} give maximum likelihood solutions for the uniquenesses.  \cite{Joreskog:1967}  noted that for fixed $\boldsymbol{\psi}>0$, the likelihood equations require:

\begin{equation}
\boldsymbol{\hat{\Gamma}} = \sqrt{\boldsymbol{\psi}} \boldsymbol{E}_{1} \sqrt{(\boldsymbol{\Lambda}_{1} - \boldsymbol{I})}
\end{equation}

where $\boldsymbol{\Lambda}_{1}$ contains the $q$ largest eigenvalues of $\sqrt{\boldsymbol{\psi}} \boldsymbol{S} \sqrt{\boldsymbol{\psi}}$, and  $\boldsymbol{E}_{1}$ the corresponding eigenvectors.   This is used to estimate $\boldsymbol{\hat{\Gamma}}$ given a value of $\boldsymbol{\hat{\psi}}$.   Now, the log likeihood is maximised with respect to $\boldsymbol{\hat{\psi}}$ given an estimate of  $\boldsymbol{\hat{\Gamma}}$.

As stated, this method is implemented in R, and therefore it is quite easy to try to fit a model to our economics data:

\singlespacing
\begin{verbatim}
> econ <- read.csv("econ.csv", row.names = 1)
> econ.fact <- factanal(econ, factors = 4, rotation = "none")
\end{verbatim}
\onehalfspacing

We can consider the residual matrix for our maximum likelihood solution

\singlespacing
\begin{verbatim}
> loadml <- loadings(econ.fact)
> class(loadml) <- "matrix"
> uniqueml <- econ.fact$uniquenesses
> resid <- econ.cor - ( loadml%*% t(loadml) + diag(uniqueml) )
> resid
\end{verbatim}
\onehalfspacing

It will be seen that these are considerably smaller than those residuals obtained from the principal component method used earlier.   One gain from using the maximum likelihood method is that classical multivariate work provide a test for the adequacy of model fit.   If take our null hypothesis as belief that our factor analysis model is an adequate representation of the covariance matrix we will test the following:

\begin{eqnarray*}
H_{0}&:& \boldsymbol{\Sigma} = \boldsymbol{\Gamma} \boldsymbol{\Gamma}^{T} + \boldsymbol{\psi}\\
H_{1}&:&  \boldsymbol{\Sigma}\ is\ any\ other\ positive\ definite\ matrix
\end{eqnarray*}

This (eventually) yields a likelihood ratio statistic:

\begin{equation}
-2 \ln \Lambda = -2 \ln \left(  \frac{|\boldsymbol{\hat{\Sigma}}|}{|\boldsymbol{S}|} \right) + n \left( tr(\boldsymbol{\hat{\Sigma}}^{-1}\boldsymbol{S}) - p \right)
\end{equation}

with $\frac{1}{2} \left( (p-q)^{2} - p - q \right)$ degrees of freedom.

It can be shown (not here) that $ tr(\boldsymbol{\hat{\Sigma}}^{-1}\boldsymbol{S}) - p = 0$ at the maximum likelihood  so this term can be removed and we can consider that

\begin{equation}
\label{faqtest}
-2 \ln \Lambda = n \ln \left(  \frac{|\boldsymbol{\hat{\Sigma}}|}{|\boldsymbol{S}|} \right)
\end{equation}

All that remains is to add a correction suggested by \cite{Bartlett:1951,Bartlett:1954}.   We need to replace $n$ with something slightly more elaborate, the exact formula chosen varies amongst many multivariate tests, in the current R function the correction applied is:

\begin{displaymath}
n - 1 - \frac{2p + 5}{6} - \frac{2q}{3}
\end{displaymath}

%JW $n - 1 - \frac{(2p + 4q + 5)}{6}$.
%WK $n - \frac{2p + 11}{6} - \frac{2q}{3}$

Hence we are going to test:

\begin{equation}
n - 1 - \frac{2p + 5}{6} - \frac{2q}{3} \ln  \left(  \frac{|\boldsymbol{\hat{\Gamma}} \boldsymbol{\hat{\Gamma}}^{T} + \boldsymbol{\hat{\psi}}|}{|\boldsymbol{S}|} \right) > \chi^{2}_{\left((p-q)^{2} - p - q \right) / 2, \alpha}
\end{equation}

The idea might be to start with $q$ small (anticipating the rejection of $H_{0}$), and increase $q$ until $H_{0}$ is no longer rejected.   As with all such tests, there are many reasons for rejecting $H_{0}$, not all of these may concern us.   In addition, \cite{Johnson+Wichern:2002} suggest that if $n$ is large and $q$ is small relative to $p$, it will tend to reject $H_{0}$ even though $\boldsymbol{\hat{\Sigma}}$ is close to $\boldsymbol{S}$.  So the situation can arise whereby we can claim ``statistical significance'' for the inclusion of additional factors in our model, but they actually add little to the model.   This tends to reinforces the exploratory aspects of multiviariate analysis (for some sense of exploratory).

We can extract the communalities from our model as easily as before:

\singlespacing
\begin{verbatim}
> apply(loadml^2, 1, sum)
               Agriculture                     Mining
                 0.9954167                  0.5937541
               Manufacture              PowerSupplies
                 0.9950263                  0.9950022
              Construction          ServiceIndustries
                 0.4852833                  0.8360147
                   Finance  SocialAndPersonalServices
                 0.4786655                  0.9950786
TransportAndCommunications
                 0.4676025
\end{verbatim}
\onehalfspacing

The R output has already given us information on the proportion of variance explained by each of the factors:

\singlespacing
\begin{verbatim}
               Factor1 Factor2 Factor3 Factor4
SS loadings      3.270   1.519   1.189   0.864
Proportion Var   0.363   0.169   0.132   0.096
Cumulative Var   0.363   0.532   0.664   0.760
\end{verbatim}
\onehalfspacing

suggesting that 76\% of variation is explained by our four factors (under the maximum likelihood solution).   We reckoned on 10\% points more for the principal component solution.   This  would be expected due the variance maximising properties of principal components generally (whether used appropriately or for factor analysis).

It is now important to turn our attention to rotation.   The maximum likelihood solution is constrained by the diagonality constraint, and it is particularly important here that rotations are considered.

\section{Rotation}
\label{rotfa}

It was stated earlier that one of the potential disadvantages of factor analysis was a certain rotational indeterminancy, indeed in the maximum likelihood fitting method it is necessary to add a constraint specifically deal with this.   We are now going to consider one of the benefits of rotation; to yield a more interpretable factor structure.   In short, we seek a rotation:

\begin{equation}
\boldsymbol{\hat{\Gamma}}^{(R)} = \boldsymbol{\hat{\Gamma}} \boldsymbol{T}
\end{equation}
such that we obtain easy-to-interpret factor loadings.   One definition of ``easy'' is that where possible some components would be large, others small.   The most obvious way to do this is actually to carry out the exercise by eye, and to rotate the axes around the origin so that some factor loadings become small.   It is also easy to suggest a two dimensional rotation matrix:

\begin{displaymath}
\boldsymbol{T} = \left( \begin{array}{rr} \cos \phi & \sin \phi \\
- \sin \phi & \cos \phi \end{array} \right)
\end{displaymath}

for rotation angle $\phi; -\pi \leq \phi \leq \phi$.   All we need to do is find a suitable value for $\phi$.   This becomes slightly more difficult in every sense where $q>2$, indeed it is possible to carry out the whole procedure by eye with pencil and paper (do you remember what they are).

For orthogonal rotations, two objective criteria are most commonly used to determine the optimal rotation: the Varimax procedure \citep{Kaiser:1958} and the Quartimax procedure \cite{Neuhaus+Wrigley:1954}.   The former is currently available within R and will be considered here, as usual it is worth checking the definitive entry in \texttt{?varimax}.   This looks for a rotation which maximises the objective V:

\begin{equation}
V = \frac{1}{p^{2}} \sum_{k=1}^{q} \left( p \sum_{j=1}^{p} \left[\frac{\gamma_{jk}^{2}}{\xi_{j}^{2}} \right]^{4} - \left[ \sum_{j=1}^{p} \left[\frac{\gamma_{jk}^{2}}{\xi_{j}^{2}}\right] \right]^{2} \right)
\end{equation}

where $\xi_{i}^{2} = \sum_{k=1}^{q} \gamma_{jk}^{2}$ is the communality for each of the $j$ variables as before.

%Another index to be maximised is the so-called quartimax rotation \cite{Neuhaus+Wrigley:1954}
%
%\begin{equation}
%Q = \sum_{j=1}^{p} \sum_{k=1}^{q} \gamma_{jk}^{4} - \frac{1}{pq} \left(\sum_{j=1}^{p} \sum_{k=1}^{q} \gamma_{jk}^{2} \right)^{2}
%\end{equation}

%It needs to be stressed that this whole point of this exercise is interpretability!.

Earlier we called \texttt{factanal()} with the argument \texttt{rotation = "none"}, hence the default is to carry out a rotation.   It is also possible to obtain a promax rotation.   However, it is useful for our purposes to carry out the rotations directly on the loadings matrices we have generated earlier, the following call:
\begin{verbatim}
> varimax(loadings4)
\end{verbatim}
will supply a varimax rotation of our four principal component factors.

In many books dealing with topic it is conventional to consider this subject by visually rotating the axis, leaving the loadings in the same position.   However, inverting this procedure we can very simply plot the rotated and unrotated loadings as follows:

\singlespacing
\begin{verbatim}
> plot(loadings4[,c(1:2)], pch = as.character(c(1:9)),
    xlab = expression(paste(gamma,"1")), ylab = expression(paste(gamma,"2")),
    main = "First and second loadings",
    xlim = c(-1,1), ylim = c(-1,1))
> points(varimax(loadings4)$loadings[,c(1:2)],
    pch = letters[c(1:9)], col = "red")
> abline(h = 0)
> abline(v = 0)
\end{verbatim}
\onehalfspacing

where the numbers 1-9 represent the unrotated loadings for variables 1 to 9, and the letters a-i represent the rotated loadings for variables 1 to 9 on the first two factors.   This is depicted in figure \ref{farotation}.

\begin{figure}
\begin{center}
\includegraphics[width = 0.5\textwidth]{images/farotation}
\caption{Plot overlaying the co-ordinates of factor loadings 1 and 2 before and after rotation optimised by the varimax criterion}
\label{farotation}
\end{center}
\end{figure}

Although is is more difficult to see what is going on with $q=4$, we can see for example that the eighth variable (Social and Personal Services) has in increased loading in terms of $\gamma_{18}$, and a much decreased loading in terms of the second factor ($\gamma_{28}$ is virtually zero.   Thus we may feel that we have achieved some simplification of our factor structure.

\section{Factor scoring}

Finally, there are occasions where we may wish to estimate values for $\boldsymbol{\phi}_{i}$ for a given individual $i$.   These values are referred to as the scores, the process of estimating them, which has to be carried out after $\boldsymbol{\Gamma}$ and $\boldsymbol{\psi}$ have been estimated is therefore referred to as scoring.

Two methods are available in R for scoring, Thomson and Bartlett's.   The default is that no scoring takes place (it requires a data matrix).   By including \texttt{scores = "Bartlett")} or \texttt{scores = "regression"} these estimates are obtained.

\cite{Bartlett:1937,Bartlett:1938} propsed a method based upon weighted least squares.

Once we have estimates

\begin{eqnarray*}
x_{1} - \bar{x}_{1}  &=& \sum_{k=1}^{q} \hat{\gamma}_{1k} \phi_{1}  + \zeta_{1}\\
x_{2} - \bar{x}_{2} &=&  \sum_{k=1}^{q} \hat{\gamma}_{2k} \phi_{2}  + \zeta_{2}\\
&\vdots&\\
x_{p} - \bar{x}_{p} &=& \sum_{k=1}^{q} \hat{\gamma}_{pk} \phi_{p}  + \zeta_{p}
\end{eqnarray*}

we need to estimate $\phi_{j}$ for $j=1, \ldots, q$, however as $var(\zeta_{j}) = \psi_{j} $ are not equal he argued that weighted least squares was the most appropriate technique.

The weighted least squares estimates thus obtained are:

\begin{equation}
\boldsymbol{\hat{\phi}}_{i} = (\boldsymbol{\Gamma}^{T} \boldsymbol{\Psi}^{-1} \boldsymbol{\Gamma})\boldsymbol{\Gamma}^{T} \boldsymbol{\Psi}(\boldsymbol{x}_{i} - \boldsymbol{\bar{x}})
\end{equation}

\cite{Thomson:1951} is based on assuming that both $\boldsymbol{\phi}$ and $\boldsymbol{\zeta}$ are multivariate normal, thus a concatenation of the manifest ($\boldsymbol{x}$) and latent ( $\boldsymbol{\phi}$) variables $\boldsymbol{y}^{T} = (\boldsymbol{\phi}^{T}, \boldsymbol{x}^{T})$ will also be normal with dispersion matrix:

\begin{displaymath}
var(\boldsymbol{y}) = \left( \begin{array}{cc} \boldsymbol{I} & \boldsymbol{\Gamma}^{T} \\
\boldsymbol{\Gamma} & \boldsymbol{\Gamma} \boldsymbol{\Gamma}^{T} + \boldsymbol{\psi} \end{array} \right)
\end{displaymath}

The mean of $\boldsymbol{\phi}$ is zero by definition, therefore:

\begin{displaymath}
E(\boldsymbol{z} | \boldsymbol{x}_{0}) =  \boldsymbol{\Gamma}^{T}(\boldsymbol{\Gamma})\boldsymbol{\Gamma}^{T} + \boldsymbol{\Psi})^{-1}(\boldsymbol{x}_{o} - \boldsymbol{\mu})
\end{displaymath}

which gives the estimate for the scores as:

\begin{equation}
\boldsymbol{z} =  \boldsymbol{\hat{\Gamma}}^{T}(\boldsymbol{\hat{\Gamma}})\boldsymbol{\hat{\Gamma}}^{T} + \boldsymbol{\hat{\psi}})^{-1}(\boldsymbol{x}_{i} - \boldsymbol{\hat{mu}})
\end{equation}

It might be clear that factor scoring takes no account of uncertainty in the estimates of $\boldsymbol{\hat{\Gamma}}$ and $ \boldsymbol{\hat{\psi}}$, this is one area where Bayesian methods are coming to the fore \citep{Aitkin+Aitkin:2005}

%As stated, Factor Analysis is very common in application areas such as Psychometrics.   The canonical example is an analysis of intelligence testing.  We will use an example from Smith, G. A. and Stanley G. (1983) ``Clocking g: relating intelligence and measures of timed performance''. Intelligence, 7, 353-368 (presented in Bartholomew, 1990).   If we load up the \texttt{ability.cov} object we have a covariance matrix arising from six tests given to 112 individuals.   The six tests are general: a non-verbal measure of general intelligence using Cattell's culture-fair test, picture: a picture-completion test, blocks: block design, maze: mazes, reading: reading comprehension and finally vocab: vocabulary.   The underlying theory is that there exists some kind of ``general intelligence'' ($g$); the more $g$ an individual has the higher they will tend to score in all of the six tests.

%The following code will fit a Factor Analysis model by maximum likelihood to this covariance matrix; a set of calls to \texttt{update} will increase the number of latent variables to two, and will also fit different rotations.   Part of the rationale for carrying out the rotations is to produce more readily interpreted factors.   In the case of promax, the rotation yields a number of loadings which are essentially zero.

%Bartholomew gives both covariance and correlation matrices, but these are inconsistent. Neither are in the original paper.

%Barthlomew, D. J. (1987) Latent Variable Analysis and Factor Analysis. Griffin.
%Barthlomew, D. J. and Knott, M. (1990) Latent Variable Analysis and Factor Analysis. Second Edition, Arnold.

%\singlespacing
%\begin{verbatim}
%> ability.cov ## have a look at the covariance matrix
%> ?ability.cov ## get some details on the ``data''
%> ability.FA <- factanal(factors = 1, covmat=ability.cov, rotation = "none")
%> ability.FA ## have a look at the first model
%> update(ability.FA, factors=2) ## what about two latent variables
%> update(ability.FA, factors=2, rotation="varimax")
%> update(ability.FA, factors=2, rotation="promax")
%\end{verbatim}
%\onehalfspacing

%\textit{How do you interpret the various models?}   Most of our work in factor analysis will be considered with rotation and reificaton (interpreting the hidden variables).

%%% Local Variables: ***
%%% mode:latex ***
%%% TeX-master: "../book.tex"  ***
%%% End: ***


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{indentfirst}
\usepackage{amssymb}
\usepackage{fancyhdr}

\title{LX331: Assignment 3}
\author{Duy Nguyen}
\date{17 February 2017}

\pagestyle{fancy}
\fancyhf{}
\rhead{Nguyen \thepage}

\begin{document}

\maketitle

\section{The syntax and semantics of Propositional Logic}
\subsection*{A. B.}
(1) $\sim s \& r$ : Stuart is not in the kitchen and Fred left.

\begin{tabular}{cc|ccccc}
r & s & $\sim s$ \& $\sim s \& r$ \\
\hline
T & T & F & F \\
T & F & T & T \\
F & T & F & F \\
F & F & T & F \\
\end{tabular}

(2) $\sim (p \lor q)$ : It is not true that Mary or Sue is at home.

\begin{tabular}{cc|ccccc}
p & q & $p \lor q$ \& $\sim (p \lor q)$ \\
\hline
T & T & T & F \\
T & F & T & F \\
F & T & T & F \\
F & F & F & T \\
\end{tabular}

(3) $\sim (p \& s) \rightarrow q$ : If it is not true that Marry is at home and Stuart is in the kitchen, then Sue is at home.

\begin{tabular}{ccc|ccccc}
p & q & s & $p \& s$ & $\sim (p \& s)$ & $\sim (p \& s) \rightarrow q$ \\
\hline
T &T & T & T & F & T \\
T &T & F & F & T & T\\
T &F & T & T & F & T\\
T &F & F & F & T & F\\
F &T & T & F & T & T\\
F &T & F & F & T & T\\
F &F & T & F & T & F\\
F &F & F & F & T & F\\
\end{tabular}

(4) $\sim q \& ((s \lor \sim r) \rightarrow \sim s)$ : Sue isn't at home and if Stuart is in the kitchen or Fred didn't left then Stuart is not in the kitchen.

\begin{tabular}{ccc|ccccc}
p &r & s & $s \lor \sim r$ & $(s \lor \sim r) \rightarrow \sim s$ & $\sim q \& ((s \lor \sim r) \rightarrow \sim s)$ \\
\hline
T &T & T & T & F & F \\
T &T & F & F & T & F\\
T &F & T & T & F & F\\
T &F & F & T & T & F\\
F &T & T & T & F & F\\
F &T & F & F & T & T\\
F &F & T & T & F & F\\
F &F & F & T & T & T\\
\end{tabular}

\subsection*{C. D.}
(5) Stuart is not in the kitchen and Fred didn’t leave.

$\sim s \& \sim r$

We can rewrite this logic formula as $\sim (s \lor r)$ due to DeMorgan's law.

\begin{tabular}{cc|c}
s & r & $\sim s \& \sim r \equiv \sim (s \lor r)$ \\
\hline
T & T & F  \\
T & F & F \\
F & T & F  \\
F & F & T  \\
\end{tabular}

(6) If Sue is at home or Fred didn’t leave, then Stuart is not in the kitchen.

$(q \lor \sim r) \rightarrow \sim s$

\begin{tabular}{ccc|cc}
p & r & s & $(q \lor \sim r)$ & $(q \lor \sim r) \rightarrow \sim s$ \\
\hline
T & T & T & T & F  \\
T & T & F & F & T \\
T & F & T & T & F \\
T & F & F & F & T \\
F & T & T & F & T \\
F & T & F & F & T \\
F & F & T & F & F \\
F & F & F & F & T \\
\end{tabular}

(7) It's not the case that Mary and Sue are both at home.

$\sim (p \& q)$

\begin{tabular}{cc|c}
p & q & $\sim (p \& q)$ \\
\hline
T & T & F  \\
T & F & T \\
F & T & T  \\
F & F & T  \\
\end{tabular}

(8) Neither Mary nor Sue is at home.

$\sim (p \lor q)$

\begin{tabular}{cc|c}
p & q & $\sim (p \lor q)$ \\
\hline
T & T & F  \\
T & F & F \\
F & T & F  \\
F & F & T  \\
\end{tabular}

\section{Representing semantic ambiguity in Propositional Logic}

(9) I didn't talk to Fred and Barney

We can interpret two understanding from (9).

(a) I did not talk to Fred and Barney at the same time.

$\sim (p \& q)$

(b) I did not talk to Fred, and I did not talked to Barney

$\sim p \& \sim q$

If we look into the truth table, we can see that the two interpretations of the English sentence is not equivalent.

\begin{tabular}{cc|cc}
p & q & $\sim (p \& q)$ & $\sim p \& \sim q$\\
\hline
T & T & F & F  \\
T & F & T & F\\
F & T & T & F\\
F & F & T & T \\
\end{tabular}

\section{A new logical connective}

I can't type the connective arrow so $->$ is a substitution.

\begin{tabular}{cc|cl}
A & B & A $->$ B & Comment\\
\hline
T & T & F & It's the last week so there can't be homework  \\
T & F & T & It's not the last week so there have to be homework\\
F & T & T & It's the last week, so it is true that there isn't homework\\
F & F & F & It's not the last week, so if there is no homework it is false. \\
\end{tabular}

\section{Logical relations between sentences}

\subsection*{A.}
So we have that $p \rightarrow q \equiv \sim p \lor q \equiv \sim (p \& \sim q)$. So $p \rightarrow q$ and $\sim (p \& \sim q)$ should be equivalent. Let's check with the truth table.

\begin{tabular}{cc|cc}
p & q & $p \rightarrow q$ & $\sim (p \& \sim q)$ \\ \hline
T & T & T & T \\
T & F & F & F \\
F & T & T & T \\
F & F & T & T \\
\end{tabular}

\subsection*{B.}
Logical compatible means that there are rows in that both formulae is both true or both false.

\begin{tabular}{cc|cc}
r & s & $\sim r \& \sim s$ & $\sim (r \lor \sim s)$ \\ \hline
\textbf{T} & \textbf{T} & \textbf{F} & \textbf{F} \\
\textbf{T} & \textbf{F} & \textbf{F} & \textbf{F} \\
F & T & F & T \\
F & F & T & F \\
\end{tabular}

As we can see, the first two row are show that these two formulae are compatible.

\subsection*{C.}
One formula entails the other when the truth of the first "forces" the truth of the second. The table for two formulae is as follow:

\begin{tabular}{cc|cc}
p & s & $ \sim (p \& s) $ & $ \sim (p \lor s) $ \\ \hline
T & T & F & F \\
T & F & T & F \\
F & T & T & F \\
\textbf{F} & \textbf{F} & \textbf{T} & \textbf{T} \\
\end{tabular}

From the truth table, we can see that $ \sim (p \& s) $ does not entails $ \sim (p \lor s) $. However, the last line show us that $ \sim (p \lor s) $ entails $ \sim (p \& s) $.

\end{document}


\section{The image of a map and the replacement axiom}\label{chap:image}

The idea of the image of a map $f:A\to X$ is that it is, in a way, the least subtype of $X$ that contains all the values of $f$. More precisely, the image of $f$ is an embedding $i:\im(f)\hookrightarrow X$ that fits in a commuting triangle
\begin{equation*}
  \begin{tikzcd}[column sep=tiny]
    A \arrow[rr,"q"] \arrow[dr,swap,"f"] & & \im(f) \arrow[dl,hook,"i"] \\
    \phantom{\im(f)} & X
  \end{tikzcd}
\end{equation*}
and satisfies the \emph{universal property} of the image of $f$. The universal property of the image of $f$ asserts that if a subtype $B\hookrightarrow X$ contains all the values of $f$, then it contains the image of $f$.
%In other words, for  asserts that there is a unique map $h:\im(f)\to B$ for which the tetrahedron
%\begin{equation*}
%  \begin{tikzcd}[column sep=large]
%    A \arrow[rr] \arrow[dr,"q"] \arrow[dddr,swap,"f"] & & B \arrow[dddl,"m"] \\
%    & \im(f) \arrow[ur,densely dotted,"h"] \arrow[dd,"i"] \\ \\
%    & X
%  \end{tikzcd}
%\end{equation*}
%commutes.
The image of a map can be constructed using the propositional truncation operation. In fact, we can also go the other way around: The propositional truncation of a type $A$ is the image of the map $A\to\unit$.

The final topic of this section is the type theoretic replacement axiom. A specific instance of the replacement axiom asserts that the image of any map $f:A\to\UU$ is equivalent to a type in $\UU$, provided that $A$ is equivant to a type in $\UU$. This property will be used to construct quotients in type theory, much in the same way as quotients are constructed in set theory.

We should note that the existence of the propositional truncation operation and the replacement axiom will be assumed for now. However, once we assume that universes are closed under pushouts, we will be able to construct the propositional truncations and we will be able to prove the replacement axiom. These constructions will be given in \cref{sec:join-construction}.

\subsection{The image of a map}\label{sec:image-construction}
 %Note that there is quite a lot of information in this diagram: not only are there the three small commuting triangles; there is also the large commuting triange in the back, and there is a three-dimensional solid filling the space between the four triangles. We make the following definition, in order to express the universal property of the image efficiently.

\begin{defn}
  Let $f:A\to X$ and $g:B\to X$ be maps. A \define{morphism} from $f$ to $g$ over $X$ consists of a map $h:A\to B$ equipped with a homotopy $H:f\htpy g\circ h$ witnessing that the triangle
\begin{equation*}
\begin{tikzcd}[column sep=tiny]
A \arrow[rr,"h"] \arrow[dr,swap,"f"] & & B \arrow[dl,"g"] \\
& X
\end{tikzcd}
\end{equation*}
commutes. Thus, we define the type
\begin{equation*}
\mathrm{hom}_X(f,g)\defeq\sm{h:A\to B}f\htpy g\circ h.
\end{equation*}
Composition of morphisms over $X$ is defined by
\begin{equation*}
  (k,K)\circ (h,H) \defeq (k\circ h,\ct{H}{(K\cdot h)}).
\end{equation*}
\end{defn}

\begin{defn}
Consider a commuting triangle
\begin{equation*}
\begin{tikzcd}[column sep=tiny]
A \arrow[rr,"q"] \arrow[dr,swap,"f"] & & I \arrow[dl,"i"] \\
& X
\end{tikzcd}
\end{equation*}
with $H:f\htpy i\circ q$, where $i$ is an embedding\index{embedding}.
We say that $i$ has the \define{universal property of the image of $f$}\index{universal property!of the image} if the map
\begin{equation*}
\blank\circ(q,H) : \mathrm{hom}_X(i,m)\to\mathrm{hom}_X(f,m)
\end{equation*}
is an equivalence for every embedding $m:B\to X$.
\end{defn}

\begin{rmk}
  Consider a commuting triangle
\begin{equation*}
\begin{tikzcd}[column sep=tiny]
A \arrow[rr,"q"] \arrow[dr,swap,"f"] & & I \arrow[dl,"i"] \\
& X
\end{tikzcd}
\end{equation*}
with $H:f\htpy i\circ q$, where $i$ is an embedding. Then it is not hard to see that the embedding $i$ satisfies the universal property of the image inclusion if and only if for every commuting triangle
\begin{equation*}
  \begin{tikzcd}[column sep=tiny]
    A \arrow[dr,swap,"f"] \arrow[rr,"g"] & & B \arrow[dl,"m"] \\
    & X
  \end{tikzcd}
\end{equation*}
with $G:f\htpy m\circ g$, where $m$ is an embedding, the type of quadruples $(h,K,L,M)$ consisting of
\begin{enumerate}
\item a map $h:I\to B$,
\item a homotopy $K:i\htpy m\circ h$ witnessing that the triangle
  \begin{equation*}
    \begin{tikzcd}[column sep=tiny]
      I \arrow[rr,"h"] \arrow[dr,swap,"i"] & & B \arrow[dl,"m"] \\
      & X
    \end{tikzcd}
  \end{equation*}
  commutes,
\item a homotopy $L:g\htpy h\circ q$ witnessing that the triangle
  \begin{equation*}
    \begin{tikzcd}[column sep=tiny]
      A \arrow[rr,"q"] \arrow[dr,swap,"g"] & & I \arrow[dl,"h"] \\
      & B
    \end{tikzcd}
  \end{equation*}
  commutes,
\item a homotopy $M:\ct{H}{(K\cdot q)}\htpy\ct{G}{(m\cdot L)}$ witnessing that the square
  \begin{equation*}
    \begin{tikzcd}
      f \arrow[d,swap,"H"] \arrow[r,"G"] & m\circ g \arrow[d,"m\cdot L"] \\
      i\circ q \arrow[r,swap,"K\cdot q"] & m\circ h\circ g
    \end{tikzcd}
  \end{equation*}
  commutes,
\end{enumerate}
is contractible. However, the situation is in fact much simpler, because the type $\mathrm{hom}_X(f,m)$ is a proposition whenever $m$ is an embedding.
\end{rmk}

\begin{rmk}
  Suppose that the map $f:A\to X$ has a section. Then the identity function
  \begin{equation*}
    \idfunc:X\to X
  \end{equation*}
  satisfies the universal property of the image of $f$.
\end{rmk}

\begin{rmk}
  Suppose that $f:A\to X$ is already an embedding. Then $f$ itself satisfies the universal property of the image of $f$.
\end{rmk}

\begin{lem}
For any $f:A\to X$ and any embedding\index{embedding} $m:B\to X$, the type $\mathrm{hom}_X(f,m)$ is a proposition.
\end{lem}

\begin{proof}
  Recall from \cref{ex:triangle_fib} that the type $\mathrm{hom}_X(f,m)$ is equivalent to the type
  \begin{equation*}
    \prd{x:X}\fib{f}{x}\to\fib{m}{x}.
  \end{equation*}
  Therefore it suffices to show that this type is a proposition. Recall from \cref{cor:prop_emb} that a map is an embedding if and only if its fibers are propositions.
  Thus we see that the type $\prd{x:X}\fib{f}{x}\to\fib{m}{x}$ is a product of propositions, hence it is a proposition by \cref{thm:trunc_pi}.
\end{proof}

\begin{prp}\label{prp:simplifly-universal-property-image}
  Consider a commuting triangle
  \begin{equation*}
    \begin{tikzcd}[column sep=tiny]
      A \arrow[rr,"q"] \arrow[dr,swap,"f"] & & I \arrow[dl,"i"] \\
      & X
\end{tikzcd}
  \end{equation*}
  with $H:f\htpy i\circ q$, where $i$ is an embedding. Then the following are equivalent:
  \begin{enumerate}
  \item The embedding $i$ satisfies the universal property of the image inclusion of $f$.
  \item For every embedding $m:B\to X$ there is a map
    \begin{equation*}
      \mathrm{hom}_X(f,m)\to\mathrm{hom}_X(i,m).
    \end{equation*}
  \end{enumerate}
\end{prp}

\begin{proof}
Since $\mathrm{hom}_X(f,m)$ is a proposition for every every embedding $m:B\to X$, the claim follows immediately by \cref{ex:prop_equiv}.
\end{proof}

Just as in the cases for pullbacks and pushouts, the universal property of the image implies that the image is determined uniquely. We will show here that the type of image factorizations of any map is a proposition. In \cref{sec:image-construction} we will construct the image, after constructing the propositional truncation.

\begin{prp}
  Let $f$ be a map, and consider two commuting triangles
  \begin{equation*}
    \begin{tikzcd}[column sep=tiny]
      A \arrow[dr,swap,"f"] \arrow[rr,"q"] & & B \arrow[dl,"i"] &[2em] A \arrow[dr,swap,"f"] \arrow[rr,"{q'}"] & & B' \arrow[dl,"{i'}"] \\
      & X & & \phantom{B'} & X
    \end{tikzcd}
  \end{equation*}
  with $I:f\htpy i\circ q$ and $I':f\htpy i'\circ q'$, in which $i$ and $i'$ are assumed to be embeddings. Moreover, consider
  \begin{equation*}
    (h,H):\mathrm{hom}_X(i,i')
  \end{equation*}
  equipped with an identification $(h,H)\circ(q,I)=(q',I')$ in $\mathrm{hom}_X(f,i')$. Then, if any two of the following properties hold, so does the third:
  \begin{enumerate}
  \item The embedding $i$ satisfies the universal property of the image inclusion of $f$.
  \item The embedding $i'$ satisfies the universal property of the image inclusion of $f$.
  \item The map $h$ is an equivalence.
  \end{enumerate}
\end{prp}

\begin{proof}
  Consider an embedding $m:C\to X$. Then we have a commuting triangle
  \begin{equation*}
    \begin{tikzcd}[column sep=-1em]
      \mathrm{hom}_X(i',m) \arrow[rr,"{\blank\circ(h,H)}"] \arrow[dr,swap,"{\blank\circ(q',I')}"] & & \mathrm{hom}_X(i,m) \arrow[dl,"{\blank\circ(q,I)}"] \\
      & \mathrm{hom}_X(f,m), & \phantom{\mathrm{hom}_X(i',m)}
    \end{tikzcd}
  \end{equation*}
  so it follows that if any two of these maps are equivalences, then so is the third. The claim now follows by the observation that $\blank\circ(h,H)$ is an equivalence for every embedding $m:C\to X$ if and only if $h$ is an equivalence.
\end{proof}

\begin{cor}\label{cor:uniqueness-image}
  Consider two image factorizations
  \begin{equation*}
    \begin{tikzcd}[column sep=tiny]
      A \arrow[dr,swap,"f"] \arrow[rr,"q"] & & B \arrow[dl,"i"] &[2em] A \arrow[dr,swap,"f"] \arrow[rr,"{q'}"] & & B' \arrow[dl,"{i'}"] \\
      & X & & \phantom{B'} & X
    \end{tikzcd}
  \end{equation*}
  of a map $f$, with $I:f\htpy i\circ q$ and $I':f\htpy i'\circ q'$. Then the type of $(e,H):\mathrm{hom}_X(i,i')$ in which $e$ is an equivalence, equipped with an identification
  \begin{equation*}
    (e,H)\circ(q,I)=(q',I')
  \end{equation*}
  in $\mathrm{hom}_X(f,i')$, is contractible.
\end{cor}

The image of a map $f:A\to X$ can now be defined using the propositional truncation:

\begin{defn}
For any map $f:A\to X$ we define the \define{image}\index{image} of $f$ to be the type
\begin{equation*}
\im(f) \defeq \sm{x:X}\brck{\fib{f}{x}}.
\end{equation*}
Furthermore, we define:
\begin{enumerate}
\item The \define{image inclusion}
  \begin{equation*}
    i_f:\im(f)\to X
  \end{equation*}
  to be the projection $\proj 1$.
\item The map
  \begin{equation*}
    q_f:A\to\im(f)
  \end{equation*}
  to be the map given by $q_f(x)\defeq(f(x),\eta(x,\refl{f(x)}))$.
\item The homotopy $I_f:f\htpy i_f\circ q_f$ witnessing that the triangle
  \begin{equation*}
    \begin{tikzcd}[column sep=tiny]
      A \arrow[rr,"q_f"] \arrow[dr,swap,"f"] & & \im(f) \arrow[dl,"i_f"] \\
      \phantom{\im(f)} & X
    \end{tikzcd}
  \end{equation*}
  commutes, to be given by $I_f(x)\defeq\refl{f(x)}$.
\end{enumerate}
\end{defn}

\begin{prp}
  The image inclusion $i_f:\im(f)\to X$ of any map $f:A\to X$ is an embedding.
\end{prp}

\begin{proof}
  The fiber of $i_f$ at $x:X$ is equivalent to the type $\brck{\fib{f}{x}}$. In particular we see that the fibers are propositions, so $i_f$ is an embedding.
\end{proof}

\begin{thm}
  The image inclusion $i_f:\im(f)\to X$ of any map $f:A\to X$ satisfies the universal property of the image inclusion of $f$.
\end{thm}

\begin{proof}
  Consider an embedding $m:B\to X$. Note that we have a commuting square
  \begin{equation*}
    \begin{tikzcd}[column sep=6em]
      \mathrm{hom}_X(i_f,m) \arrow[d] \arrow[r] & \mathrm{hom}_X(f,m) \arrow[d] \\
      \Big(\prd{x:X}\fib{i_f}{x}\to\fib{m}{x}\Big) \arrow[r,swap,"h\mapsto{\lam{x}h_x\circ\varphi_x}"] & \Big(\prd{x:X}\fib{f}{x}\to\fib{m}{x}\Big)
    \end{tikzcd}
  \end{equation*}
  The vertical maps are of the form
  \begin{equation*}
    (h,H) \mapsto \lam{x}{(y,p)}(h(y),\ct{H(y)^{-1}}{p}),
  \end{equation*}
  and they are both equivalences. The map
  \begin{equation*}
    \varphi_x:\fib{f}{x}\to\fib{i_f}{x}
  \end{equation*}
  given by $\varphi_x(a,p)\defeq((h(a),\eta(a,p)),p)$ is a propositional truncation for every $x:X$. Therefore it follows that the map
  \begin{equation*}
    (\fib{i_f}{x}\to\fib{m}{x})\to(\fib{f}{x}\to\fib{m}{x})
  \end{equation*}
  is an equivalence, for every $x:X$. Thus we conclude that the bottom map in the above square is an equivalence, which implies that the top map is an equivalence.
\end{proof}

\begin{eg}
  An important special case of the homotopy image of a map is the image of the terminal projection
\begin{equation*}
  \const_\ttt : A \to \unit,
\end{equation*}
which results in an embedding $I\hookrightarrow \unit$. Embeddings into the unit type are in fact just propositions. To see this, note that
\begin{align*}
\sm{A:\UU}{f:A\to\unit}\isemb(f)
& \eqvsym \sm{A:\UU}\isemb(\const_\ttt) \\
& \eqvsym \sm{A:\UU}\prd{x:\unit}\isprop(\fib{\const_\ttt}{x}) \\
& \eqvsym \sm{A:\UU}\isprop(\fib{\const_\ttt}{\ttt}) \\
& \eqvsym \sm{A:\UU}\isprop(A).
\end{align*}
Therefore, the universal property of the image of the map $A\to\unit$ is equivalently described as a proposition $P$ satisfying the universal property of the propositional truncation.
\end{eg}

\subsection{Surjective maps}

Another application of the propositional truncation is the notion of surjective map.

\begin{defn}
A map $f:A\to B$ is said to be \define{surjective} if there is a term of type
\begin{equation*}
\issurj(f)\defeq \prd{y:B}\brck{\fib{f}{b}}.
\end{equation*}
\end{defn}

\begin{eg}
Any equivalence is a surjective map, and so is any map that has a section (those are sometimes called \define{split epimorphisms}). Other examples include the base point inclusion $\unit\to\sphere{n}$ for any $n\geq 1$.
\end{eg}

\begin{prp}\label{prp:surjective}
  Consider a map $f:A\to B$. Then the following are equivalent:
  \begin{enumerate}
  \item The map $f:A\to B$ is surjective.
  \item For any family $P$ of propositions over $B$, the precomposition map
    \begin{equation*}
      \blank\circ f : \Big(\prd{y:B}P(y)\Big)\to\Big(\prd{x:A}P(f(x))\Big)
    \end{equation*}
    is an equivalence.
  \end{enumerate}
\end{prp}

\begin{proof}
  Suppose first that $f$ is surjective, and consider the commuting square
  \begin{equation*}
    \begin{tikzcd}[column sep=6em]
      \Big(\prd{y:B}P(y)\Big) \arrow[r,"\blank\circ f"] \arrow[d,swap,"h\mapsto\lam{y}\const_{h(y)}"] & \Big(\prd{x:A}P(f(x))\Big)  \\
      \Big(\prd{y:B}\brck{\fib{f}{y}}\to P(y)\Big) \arrow[r,swap,"h\mapsto\lam{y}h(y)\circ\eta"] & \Big(\prd{y:B}\fib{f}{y}\to P(y)\Big) \arrow[u,swap,"{h\mapsto\lam{x}h(f(x),(x,\refl{f(x)}))}"]
    \end{tikzcd}
  \end{equation*}
  In this square, the bottom map is an equivalence by the universal property of the propositional truncation of $\fib{f}{y}$. The map on the right is also easily seen to be an equivalence. Furthermore, the map on the left is an equivalence by the assumption that $f$ is surjective, from which it follows that the types $\brck{\fib{f}{y}}$ are contractible. Therefore it follows that the top map is an equivalence, which completes the proof that (i) implies (ii).

  For the converse, it follows immediately from the assumption (ii) that
  \begin{equation*}
    \blank\circ f : \Big(\prd{y:B}\brck{\fib{f}{y}}\Big)\to\Big(\prd{x:A}\brck{\fib{f}{f(x)}}\Big)
  \end{equation*}
  is an equivalence. Hence it suffices to construct a term of type $\brck{\fib{f}{f(x)}}$ for each $x:A$. This is easy, because we have
  \begin{equation*}
    \eta(x,\refl{f(x)}):\brck{\fib{f}{f(x)}}.\qedhere.
  \end{equation*}
\end{proof}

\begin{thm}\label{thm:surjective}
Consider a commuting triangle
\begin{equation*}
\begin{tikzcd}[column sep=tiny]
A \arrow[rr,"q"] \arrow[dr,swap,"f"] & & B \arrow[dl,"m"] \\
& X
\end{tikzcd}
\end{equation*}
in which $m$ is an embedding. Then the following are equivalent:
\begin{enumerate}
\item The embedding $m$ satisfies the universal property of the image inclusion of $f$.
\item The map $q$ is surjective.
\end{enumerate}
\end{thm}

\begin{proof}
  First assume that $m$ satisfies the universal property of the image inclusion of $f$, and consider the composite function
  \begin{equation*}
    \begin{tikzcd}
      \Big(\sm{y:B}\brck{\fib{q}{y}}\Big) \arrow[r,"\proj 1"] & B \arrow[r,"m"] & X.
    \end{tikzcd}
  \end{equation*}
  Note that $m\circ\proj 1$ is a composition of embeddings, so it is an embedding. By the universal property of $m$ there is a unique map $h$ for which the triangle
  \begin{equation*}
    \begin{tikzcd}[column sep=0]
      B \arrow[dr,swap,"m"] \arrow[rr,densely dotted,"h"] & & \sm{y:B}\brck{\fib{q}{y}} \arrow[dl,"m\circ\proj 1"] \\
      \phantom{\sm{y:B}\brck{\fib{q}{y}}} & X
    \end{tikzcd}
  \end{equation*}
  commutes. Now note that $\proj 1\circ h$ is a map such that $m\circ (\proj 1\circ h)\htpy m$. The identity function is another map for which we have $m\circ\idfunc\htpy m$, so it follows by uniqueness that $\proj 1\circ h\htpy \idfunc$. In other words, the map $h$ is a section of the projection map. Therefore we obtain by \cref{ex:pi_sec} a dependent function
  \begin{equation*}
    \prd{b:B}\brck{\fib{q}{b}},
  \end{equation*}
  showing that $q$ is surjective.

  For the converse, suppose that $q$ is surjective. To prove that $m$ satisfies the universal property of the image factorization of $f$, it suffices to construct an equivalence
  \begin{equation*}
    \mathrm{hom}_X(f,m')\to\mathrm{hom}_X(m,m'),
  \end{equation*}
  for any embedding $m':B'\to X$. To see that there is such an equivalence, we make the following calculation
  \begin{align*}
    \mathrm{hom}_X(m,m') & \simeq \prd{x:X}\fib{m}{x}\to\fib{m'}{x} \\
                         & \simeq \prd{b:B}\fib{m'}{m(b)} \\
                         & \simeq \prd{a:A}\fib{m'}{m(q(a))} \\
                         & \simeq \prd{a:A}\fib{m'}{f(a)} \\
                         & \simeq \prd{x:X}\fib{f}{x}\to\fib{m'}{x} \\
                         & \simeq \mathrm{hom}_X(f,m').
  \end{align*}
  In this calculation, the first and last equivalence hold by \cref{ex:triangle_fib}. The second and second to last equivalences hold by \cref{ex:pi-fib}. The third equivalence holds by \cref{prp:surjective}, since $q$ is assumed to be surjective, and the fourth equivalence holds since we have a homotopy $f\htpy m\circ f$.
\end{proof}

\begin{cor}
  Every map factors uniquely as a surjective map followed by an embedding.
\end{cor}

\begin{proof}
  Consider a map $f:A\to X$, and two factorizations
  \begin{equation*}
    \begin{tikzcd}[column sep=tiny]
      A \arrow[rr,"q"] \arrow[dr,swap,"f"] & & B \arrow[dl,"i"] &[3em] A \arrow[rr,"{q'}"] \arrow[dr,swap,"f"] & & B' \arrow[dl,"{i'}"] \\
      & X & & & X
    \end{tikzcd}
  \end{equation*}
  of $f$ where $m$ and $m'$ are embeddings, and $q$ and $q'$ are surjective. Then both $m$ and $m'$ satisfy the universal property of the image factorization of $f$ by \cref{thm:surjective}. Now it follows by \cref{cor:uniqueness-image} that the type of $(e,H):\mathrm{hom}_X(i,i')$ in which $e$ is an equivalence, equipped with an identification
  \begin{equation*}
    (e,H)\circ(q,I)=(q',I')
  \end{equation*}
  in $\mathrm{hom}_X(f,i')$, is contractible.
\end{proof}

\subsection{Type theoretic replacement}

\begin{comment}
We have constructed the set quotient $A/R$ as the image of the equivalence relation
\begin{equation*}
  R:A\to \UU^A.
\end{equation*}
However, the type $\UU^A$ is itself in the next universe $\UU^+$. Hence the quotient is also in the universe $\UU^+$. We prove in this section that $A/R$ is nevertheless equivalent to a type in $\UU$. In other words, we show that $A/R$ is \emph{essentially} small.
\end{comment}

\begin{defn}\label{defn:ess_small}
\begin{enumerate}
\item A type $A$ is said to be \define{essentially small}\index{essentially small!type} if there is a type $X:\UU$ and an equivalence $\eqv{A}{X}$. We write\index{ess_small(A)@{$\mathsf{ess\usc{}small}(A)$}}
\begin{equation*}
\mathsf{ess\usc{}small}(A)\defeq\sm{X:\UU}\eqv{A}{X}.
\end{equation*}
\item A map $f:A\to B$ is said to be \define{essentially small}\index{essentially small!map} if for each $b:B$ the fiber $\fib{f}{b}$ is essentially small.
We write\index{ess_small(f)@{$\mathsf{ess\usc{}small}(f)$}}
\begin{equation*}
\mathsf{ess\usc{}small}(f)\defeq\prd{b:B}\mathsf{ess\usc{}small}(\fib{f}{b}).
\end{equation*}
\item A type $A$ is said to be \define{locally small}\index{locally small!type} if for every $x,y:A$ the identity type $x=y$ is essentially small.
We write\index{loc_small(A)@{$\mathsf{loc\usc{}small}(A)$}}
\begin{equation*}
\mathsf{loc\usc{}small}(A)\defeq \prd{x,y:A}\mathsf{ess\usc{}small}(x=y).
\end{equation*}
\end{enumerate}
\end{defn}

\begin{eg}
  \begin{enumerate}
  \item Any essentially $\UU$-small type is also locally $\UU$-small.
  \item Any univalent universe $\UU$ is locally $\UU$-small, because by the univalence axiom we have equivalences
    \begin{equation*}
      (A=B)\simeq (A\simeq B)
    \end{equation*}
    for each $A,B:\UU$, and the type $A\simeq B$ is in $\UU$.
  \item Any proposition is locally small with respect to any universe $\UU$.
  \item For any family $P$ of locally $\UU$-small types over a essentially $\UU$-small type $A$, the dependent product $\prd{x:A}P(x)$ is locally $\UU$-small. In particular, any type $A\to B$ of functions from an essentially small type into a locally small type is again locally small.
  \end{enumerate}
\end{eg}

\begin{lem}\label{lem:isprop_ess_small}
The type $\mathsf{ess\usc{}small}(A)$ is a proposition for any type $A$.\index{essentially small!is a proposition}
\end{lem}

\begin{proof}
Let $A$ be a type, not necessarily in $\UU$. In order to show that $\mathsf{ess\usc{}small}(A)$ is a proposition, we will use \cref{lem:isprop_eq} and show that for any $X:\UU$ and any equivalence $e:A\simeq X$, the type
\begin{equation*}
\sm{Y:\UU}\eqv{A}{Y}
\end{equation*}
is contractible. Note that we have an equivalence
\begin{equation*}
\eqv{\Big(\sm{Y:\UU}\eqv{X}{Y}\Big)}{\Big(\sm{Y:\UU}\eqv{A}{Y}\Big)}
\end{equation*}
because precomposing with the equivalence $e:A \simeq X$ is an equivalence. However, the type $\sm{Y:\UU}\eqv{X}{Y}$ is contractible by \cref{thm:univalence}. This shows that $\mathsf{ess\usc{}small}(A)$ is equivalent to a contractible type, assuming that $A$ is essentially small.
\end{proof}

\begin{cor}
For each function $f:A\to B$, the type $\mathsf{ess\usc{}small}(f)$ is a proposition, and for each type $X$ the type $\mathsf{loc\usc{}small}(X)$ is a proposition.
\end{cor}

\begin{proof}
This follows from the fact that propositions are closed under dependent products, established in \cref{thm:trunc_pi}.
\end{proof}

Recall that in set theory, the replacement axiom asserts that for any family of sets $\{X_i\}_{i\in I}$ indexed by a set $I$, there is a set $X[I]$ consisting of precisely those sets $x$ for which there exists an $i\in I$ such that $x\in X_i$. In other words: the image of a set-indexed family of sets is again a set. Without the replacement axiom, $X[I]$ would be a class. In the following corollary we establish a type-theoretic analogue of the replacement axiom: the image of a family of small types indexed by a small type is again (essentially) small.

\begin{axiom}\label{axiom:replacement}
  For any map $f:A\to B$ from an essentially small type $A$ into a locally small type $B$, the image of $f$ is again essentially small.
\end{axiom}

\begin{eg}
  For any type $A:\UU$, the image of the constant map $\const_A:\unit\to \UU$ is essentially small. This image is called the \define{connected component} of the universe at $A$. To see why, let us calculate
  \begin{align*}
    \im(\const_A) & \jdeq \sm{X:\UU}\Brck{\fib{\const_A}{X}} \\
                  & \jdeq \sm{X:\UU}\Brck{\sm{t:\unit}A=X} \\
                  & \simeq \sm{X:\UU}\brck{A=X}.
  \end{align*}
  We see that the image of $\const_A:\unit\to\UU$ is the type of all types that are \emph{merely} equal to $A$. In other words, they are equal to $A$ in an unspecified way.
\end{eg}

\begin{eg}
  The type $\F$ of all finite types is defined to be the image of the map
  \begin{equation*}
    \Fin : \N\to\UU_0
  \end{equation*}
  By the replacement axiom, this type is essentially small.
\end{eg}

\begin{exercises}
  \exercise Consider a map $f:A\to P$ into a proposition $P$. Show that the following are equivalent:
  \begin{enumerate}
  \item The map $f$ is a propositional truncation of $A$.
  \item The map $f$ is surjective.
  \end{enumerate}
  \exercise Consider a map $f:A\to B$. Show that the following are equivalent:
  \begin{enumerate}
  \item $f$ is an equivalence.
  \item $f$ is both surjective and an embedding.
  \end{enumerate}
  \exercise Consider a commuting triangle
  \begin{equation*}
    \begin{tikzcd}[column sep=tiny]
      A \arrow[rr,"h"] \arrow[dr,swap,"f"] & & B \arrow[dl,"g"] \\
      & X
    \end{tikzcd}
  \end{equation*}
  with $H:f\htpy g\circ h$, and assume that $h$ is surjective. Show that the following are equivalent:
  \begin{enumerate}
  \item The map $f$ is surjective.
  \item The map $g$ is surjective.
  \end{enumerate}
  \exercise \label{ex:surjective-precomp}Consider a map $f:A\to B$. Show that the following are equivalent:
  \begin{enumerate}
  \item The map $f$ is surjective.
  \item For every set $C$, the precomposition function
    \begin{equation*}
      \blank\circ f:(B\to C)\to (A\to C)
    \end{equation*}
    is an embedding.
  \end{enumerate}
  Hint: To show that (ii) implies (i), use the assumption with the set $C\jdeq\prop_\UU$, where $\UU$ is a univalent universe containing both $A$ and $B$.
  \exercise Let us say that a type family $B$ over $A$ is \define{univalent} if the map
  \begin{equation*}
    (x=y)\to (B(x)\simeq B(y))
  \end{equation*}
  is an equivalence, for every $x,y:A$.
  \begin{subexenum}
  \item Show that a family $B:A\to\UU$ is univalent if and only if the map $B:A\to\UU$ is an embedding.
  \item For any family $B:A\to\UU$, show that the type family $\hat{B}:\hat{A}\to\UU$ defined by
    \begin{align*}
      \hat{A} & \defeq \im(B) \\
      \hat{B}(X,p) & \defeq X
    \end{align*}
    is univalent.
  \item For any two families $B:A\to\UU$ and $D:C\to\mathcal{V}$, define the type of \define{cartesian morphisms}
    \begin{equation*}
      \carthomFam((A,B),(C,D)) \defeq \sm{f:A\to C}\prd{x:A}B(x)\simeq D(f(x)).
    \end{equation*}
    Construct a cartesian morphism
    \begin{equation*}
      (\eta,\alpha) : \carthomFam((A,B),(\hat{A},\hat{B})).
    \end{equation*}
  \item Show that for any family $B:A\to\UU$ and any \emph{univalent} family $D:C\to\mathcal{V}$, the map
    \begin{equation*}
      \carthomFam((\hat{A},\hat{B}),(C,D))\to\carthomFam((A,B),(C,D))
    \end{equation*}
    given by
    \begin{equation*}
      (f,e)\mapsto (f\circ\eta,\lam{x}e_x\circ \alpha_x)
    \end{equation*}
    is an equivalence. This is the \define{universal property} of the univalent completion of $A$.
  \end{subexenum}
  %\exercise \label{also}(Mart\'in Escard\'o) For any two propositions $P$ and $Q$, define
  %\begin{equation*}
  %P\boxplus Q \defeq ((P\to Q)\to Q)\times ((Q\to P)\to P).
  %\end{equation*}
  %\begin{subexenum}
  %\item Show that $P\lor Q\to P\boxplus Q$ and $P\boxplus Q\to\neg(\neg P\land \neg Q)$.
  %\end{subexenum}
  %\item \label{ex:brck_comp} Formulate the computation rule corresponding to the path constructor $\mu$. That is, compute the type of $\apd{\rec{\brck{\blank}}(f,g)}{\mu(x,y)}$, and find a canonical element in it.
  %\exercise Let $f:A\to X$ be a map. Construct an equivalence
  %\begin{equation*}
  %\eqv{\Big(\sm{y:\mathsf{join\usc{}power}_X(n,A)}f(x)=f^{\ast n}(y)\Big)}{\Big(\sm{y:A}f(x)=f(y)\Big)^{\ast n}}
  %\end{equation*}
  %for any $x:A$.
\end{exercises}

\endinput

\begin{thm}
Consider a commuting triangle
\begin{equation*}
\begin{tikzcd}[column sep=small]
A \arrow[rr,"i"] \arrow[dr,swap,"f"] & & B \arrow[dl,"m"] \\
& X
\end{tikzcd}
\end{equation*}
with $I:f\htpy m\circ i$, where $m$ is an embedding. The following are equivalent:
\begin{enumerate}
\item $m$ satisfies the universal property of the image of $f$.
\item for each $x:X$, the proposition $\fib{m}{x}$ satisfies the universal property of the propositional truncation of $\fib{f}{x}$.
\end{enumerate}
\end{thm}


%
% CMPT 473: Software Quality Assurance - A Course Overview
% Section: Performance
%
% Author: Jeffrey Leung
%

\section{Performance}
	\label{sec:performance}
\begin{easylist}

& Performance areas include:
	&& Speed/runtime
	&& Resource management
	&& Throughput
	&& Responsiveness

& Analyzed differently depending on component granularity (e.g. system-level, instruction-level)

& Strategies of measuring performance:
	&& Identify area of interest

& Evaluating results:
	&& Be aware of:
		&&& Warm-up time
		&&& Caching
	&& Measure and compare across changes
	&& Run many executions and take the average

& Measurement of results:

	&& \textbf{Arithmetic mean:} Average of measurements which measure the same value
		&&& Equation:
\begin{equation}
\frac{ \displaystyle \sum_{i=1}^N r_i }{N}
\end{equation}

	&& \textbf{Harmonic mean:} Average of measurements which report rates (e.g. throughput for multiple tasks)
		&&& Represents the constant rate required for the same amount of time
		&&& Calculated by dividing the total number of rates by the rate per unit (inversion of the rate)
		&&& Equation:
\begin{equation}
\frac{N}{ \displaystyle \sum_{i=1}^N \frac{1}{r_i} }
\end{equation}
	&& \textbf{Geometric mean:} Average of measurements which represent different values
		&&& A change in any benchmark affects the final value proportionally
		&&& Represents a multiplied score of performance
		&&& Equation:
\begin{equation}
\sqrt[N]{ \prod_{i=1}^N r_i }
\end{equation}
	&& \textbf{Standard deviation:} Measure of confidence in the mean
		&&& Large values imply needing more samples or correction of methodological error

\end{easylist}
\clearpage


\documentclass[pre,aps,superscriptaddress,nofootinbib]{revtex4}

\usepackage{amsmath,amsfonts,amssymb,bm,graphicx,hyperref,listings,xcolor,float,mathrsfs}
\usepackage{fontawesome}
\setlength{\parindent}{0pt}

\begin{document}

\title{Polarisation-biased trajectories of independent Brownian rotors\\{\small Analytic considerations with Mathieu functions}}

\author{Yann-Edwin Keta}
\maketitle

\section{Scaled cumulant generating function and rate function}

We wish to bias the trajectories of the system with respect to the mean vectorial polarisation over a trajectory,
\begin{equation}
\bm{p}_{\tau} = \frac{1}{\tau} \int_0^{\tau} \text{d}t \, \bm{p}(t) = \frac{1}{N \tau} \int_0^{\tau} \text{d}t \, \sum_i \bm{u}(\theta_i(t)),
\end{equation}
and therefore need to compute the scaled cumulant generating function (SCGF)
\begin{equation}
N \psi_{\bm{p}}(\bm{s}) = \lim_{\tau \rightarrow \infty} \frac{1}{\tau} \log\left<\exp\left(- \bm{s} \cdot N \int_0^{\tau} \text{d}t \, \bm{p}(t)\right)\right>_0
\end{equation}
where we will assume that $\bm{s}$ is along the $x$-axis, and compute
\begin{equation}
\psi_ {p}(s) = \lim_{\tau \rightarrow \infty} \frac{1}{\tau} \log\left<\exp\left(- s N \int_0^{\tau} \text{d}t \, p_x(t)\right)\right>_0,
\end{equation}
from which we can recover
\begin{equation}
\psi_{\bm{p}}(\bm{s}) = \psi_{p}\left(\sqrt{s_x^2 + s_y^2}\right)
\end{equation}
from the fact that $\psi_{\bm{p}}(\bm{s})$ has to be cylindrically symmetric. We then have that $N \psi_s(p)$ is the largest eigenvalue of the eigenproblem
\begin{equation}
N \psi_{p}(s) P[\{\theta_i\}] = \mathscr{W}_{s, p} P[\{\theta_i\}],
\end{equation}
where we introduced the tilted generator \cite{touchette2018introduction}
\begin{equation}
\mathscr{W}_{s, p} = \mathcal{L} - s N p = D_r \sum_i \frac{\partial^2}{\partial \theta_i^2} - s \sum_i \cos\theta_i,
\end{equation}
using here $s \equiv s_x$ and $p \equiv p_x$, so that
\begin{equation}
\sum_i \left(\prod_{j \neq i} P(\theta_j)\right) \left[\psi_{p}(s) P(\theta_i) - \left(D_r \frac{\partial^2}{\partial \theta_i^2} P(\theta_i) - s \cos(\theta_i) P(\theta_i)\right)\right] = 0,
\end{equation}
is equivalent to the $1$-particle eigenproblem,
\begin{equation}
\psi_{p}(s) P(\theta) - \left(D_r \frac{\partial^2}{\partial \theta^2} P(\theta) - s \cos\theta P(\theta)\right) = 0 \Leftrightarrow \frac{\partial^2}{\partial \theta^2} \tilde{P}(\theta^{\prime}) + (a - 2 q \cos 2 \theta^{\prime}) \tilde{P}(\theta^{\prime}) = 0,
\label{1P_eigenproblem}
\end{equation}
with $2 \theta^{\prime} = \theta$, $\tilde{P}(\theta/2) = P(\theta)$, and
\begin{equation}
a = - \frac{4 \psi_{p}(s)}{D_r},~ q = \frac{2 s}{D_r},
\end{equation}
and which solutions are known as the Mathieu functions \cite{mathieu}.\\

Most notably, we have that for any $q \in \mathbb{R}$ there is a countable infinity of $a$. Inspired by \cite{grandpre2018current} we will choose $a_{\mathrm{Mathieu}, 0}(q)$ the characteristic value of the $0$-th Mathieu function which is $\pi$-periodic and even. We thus have the SCGF
\begin{equation}
N \psi_{\bm{p}}(\bm{s}) = N \psi_{p}\left(\sqrt{s_x^2 + s_y^2}\right) = - N \frac{D_r}{4} a_{\mathrm{Mathieu}, 0}\left(\frac{2}{D_r} \sqrt{s_x^2 + s_y^2}\right),
\end{equation}
and by Legendre transform
\begin{equation}
\begin{aligned}
N I(\bm{p}) &= \sup_{\bm{s} \in \mathbb{R}^2} \{- \bm{s} \cdot N \bm{p} - N \psi_{\bm{p}}(s)\}\\
&= \sup_{\bm{s} \in \mathbb{R}^2} \left\{- \bm{s} \cdot N \bm{p} + N \frac{D_r}{4} a_{\mathrm{Mathieu}, 0}\left(\frac{2}{D_r} \sqrt{s_x^2 + s_y^2}\right)\right\},
\end{aligned}
\end{equation}
we obtain the rate function. We note that both $\psi_{\bm{p}}(\bm{s})$ and $I(\bm{p})$ are cylindrically symmetric.\\

We have the following expansion of the SCGF for small $s$ \cite{abramowitz1948handbook},
\begin{equation}
\begin{aligned}
\psi_{\bm{p}}(\bm{s}) &= - \frac{D_r}{4} \left(-\frac{1}{2} \left(\frac{2}{D_r} \sqrt{s_x^2 + s_y^2}\right)^2 + \mathcal{O}(|\bm{s}|^4)\right),~ s \to 0\\
&= \frac{1}{2} (s_x^2 + s_y^2) \frac{1}{D_r} + \mathcal{O}(|\bm{s}|^4) ,~ s \to 0,
\end{aligned}
\end{equation}
and thus with
\begin{equation}
\begin{aligned}
\left(\frac{\partial^2}{\partial s_x^2} + \frac{\partial^2}{\partial s_y^2}\right) \psi_{\bm{p}}(\bm{s}) &= N \tau \left(\left<p_{x,\tau}^2 + p_{y,\tau}^2\right>_{\bm{s}} - \left<p_{x,\tau}\right>^2_{\bm{s}} - \left<p_{y,\tau}\right>^2_{\bm{s}}\right) = N \tau \left(\left<\bm{p}_{\tau}^2\right>_{\bm{s}} - \left<\bm{p}_{\tau}\right>^2_{\bm{s}}\right)\\
&= N \tau \mathrm{Var}(\bm{p}_{\tau})_{\bm{s}},
\end{aligned}
\end{equation}
we get
\begin{equation}
\mathrm{Var}(\bm{p}_{\tau})_0 = \frac{2}{N \tau D_r},
\end{equation}
and thus
\begin{equation}
\begin{aligned}
I(\bm{p}) &= \frac{1}{2} \bm{p} \cdot \begin{pmatrix} \left. \frac{\partial^2 I}{\partial p_x^2} \right|_0 & \left. \frac{\partial^2 I}{\partial p_x \partial p_y}  \right|_0 \\ \left. \frac{\partial^2 I}{\partial p_x \partial p_y}  \right|_0 & \left. \frac{\partial^2 I}{\partial p_y^2} \right|_0 \end{pmatrix} \bm{p} + \mathcal{O}(\bm{p}^4)\\
&= \frac{1}{2} D_r \bm{p}^2 + \mathcal{O}(\bm{p}^4),
\end{aligned}
\end{equation}
the rate function (see Fig. \ref{mathieu_rate_fig} \textbf{(left)}), considering $I(0) = 0$ and $I(\bm{p}) = I(-\bm{p})$.\\

We can compare the rate function to numerical results from the cloning simulations of Brownian rotors biased with respect to their squared polarisation, and the semi-analytical upper bound $-\inf_s \tilde{B}_{s,p^2}$ obtained in this case \cite{brownian_rotors_ldp} (see Fig. \ref{mathieu_rate_fig} \textbf{(centre)}).

\begin{figure}[H]
\centering
\includegraphics[width=0.32\textwidth]{mathieu_rate.eps}
\includegraphics[width=0.32\textwidth]{rate_cloning.eps}
\includegraphics[width=0.32\textwidth]{polarisation_x.eps}
\caption{\textbf{(left)} Rate function. -- Using \textsc{Mathieu} in \href{https://github.com/yketa/active_work/blob/master/rotors.py}{\faGithub~ yketa/active\_work/rotors.py}. \textbf{(centre)} Comparison to numerics -- $N_{\mathrm{clones}} = 10^3$, $t_{\mathrm{max}} = 10^2$. \textbf{(right)} Biased average of the polarisation along the x-axis.}
\label{mathieu_rate_fig}
\end{figure}

\section{Optimal control potential}

We have that the tilted generator is self-adjoint (hermitian), so that the left and right eigenfunctions are identical. We thus have the eigenfunction for the $\bm{s} \parallel \bm{e}_x$ problem,
\begin{equation}
\mathcal{F}_s(\theta) \propto y_{Mathieu, 0}\left(\frac{\theta}{2}, \frac{2s}{D_r}\right)
\end{equation}
with $y_{Mathieu, 0}$ the $0$-th Mathieu function, even and $\pi$-periodic, so that $\mathcal{F}_s(\theta)$ is $2\pi$-periodic, associated to the eigenvalue $\psi_{p}(s)$. We can thus compute the optimal control potential \cite{jack2019ergodicity}
\begin{equation}
\phi_s(\theta) = - 2 \log \mathcal{F}_s(\theta)
\label{control_potential}
\end{equation}
to achieve the fluctuations characteristic to the biased trajectories. We can compute the curvature of this potential at $\theta = 0$,
\begin{equation}
\left. \frac{\partial^2}{\partial \theta^2} \phi_s(\theta) \right|_{s=0} = 2 \left(\mathcal{F}(\theta = 0)^{-2} \left(\frac{\partial}{\partial \theta} \mathcal{F}(\theta = 0)\right)^2 - \mathcal{F}(\theta = 0)^{-1}\frac{\partial^2}{\partial\theta^2} \mathcal{F}(\theta = 0)\right),
\end{equation}
where
\begin{equation}
\frac{\partial}{\partial \theta} \mathcal{F}(\theta = 0) = 0,
\end{equation}
by parity of $y_{Mathieu, 0}$, and
\begin{equation}
\begin{aligned}
- \mathcal{F}(\theta = 0)^{-1}\frac{\partial^2}{\partial\theta^2} \mathcal{F}(\theta = 0) &= \left. - y_{Mathieu, 0}\left(\frac{\theta}{2}, \frac{2s}{D_r}\right)^{-1}\frac{\partial^2}{\partial\theta^2} y_{Mathieu, 0}\left(\frac{\theta}{2}, \frac{2s}{D_r}\right) \right|_{\theta=0}\\
&= \frac{1}{4} \left(a_{\mathrm{Mathieu}, 0}\left(\frac{2s}{D_r}\right) - 2 \frac{2s}{D_r}\right)
\end{aligned}
\end{equation}
from the differential equation defining $y_{Mathieu, 0}$ (Eq. (\ref{1P_eigenproblem})), therefore
\begin{equation}
\left. \frac{\partial^2}{\partial \theta^2} \phi_s(\theta) \right|_{s=0} = \frac{1}{2} \left(a_{\mathrm{Mathieu}, 0}\left(\frac{2s}{D_r}\right) - 2 \frac{2s}{D_r}\right),
\end{equation}
so we can compare this optimal potential to a potential $\phi^{(g)}_s \propto g (1 - \cos\theta)$,
\begin{equation}
\phi^{(g)}_s(\theta) = \left. \frac{\partial^2}{\partial \theta^2} \phi_s(\theta) \right|_{s=0} (1 - \cos\theta) = \frac{1}{2} \left(a_{\mathrm{Mathieu}, 0}\left(\frac{2s}{D_r}\right) - 2 \frac{2s}{D_r}\right) (1 - \cos\theta),
\end{equation}
of identical curvature at $\theta = 0$ (see Fig. \ref{mathieu_potential_fig} \textbf{(left)}).

\begin{figure}[H]
\centering
\includegraphics[width=0.32\textwidth]{mathieu_potential.eps}
\includegraphics[width=0.32\textwidth]{distribution_g.eps}
\caption{\textbf{(left)} Optimal control potential. \textbf{(Right)} Torque parameter, where the factor $1/2$ stems from Eq. (\ref{control_potential}). -- Using \textsc{Mathieu} in \href{https://github.com/yketa/active_work/blob/master/rotors.py}{\faGithub~ yketa/active\_work/rotors.py}.}
\label{mathieu_potential_fig}
\end{figure}

\bibliographystyle{unsrt}
{\renewcommand{\bibname}{References}\bibliography{ref}}

\end{document}