\section{Temporal-Difference Learning}

\subsection{Exercise 6.1}
\subsubsection*{Q}
If $V$ changes during the episode, then (6.6) only holds approximately; what would the difference be between the two sides? Let $V_t$ denote the array of state values used at time $t$ in the TD error (6.5) and in the TD update (6.2). Redo the derivation above to determine the additional amount that must be added to the sum of TD errors in order to equal the Monte Carlo error.
\subsubsection*{A}
Write
\[
    \delta_t \doteq R_{t+1} + \gamma V_t(S_{t+1}) - V_t(S_t),
\]
then
\begin{align}
    G_t - V_t(S_t) &= R_{t+1} + \gamma G_{t+1} - V_t(S_t) \\
                   &= \delta_t + \gamma [ G_{t+1} - V_{t+1}(S_{t+1})] + \gamma [V_{t+1}(S_{t+1}) - V_t(S_{t+1})]\\
                   &= \delta_t + \gamma [ G_{t+1} - V_{t+1}(S_{t+1})] + \alpha \gamma [R_{t+2} + \gamma V_t(S_{t+2}) - V_t(S_{t+1})]\\
                   &\vdotswithin{=} \notag\\
                   &= \sum_{k=t}^{T-1} \gamma^{k-t} \delta_k + \alpha \sum_{k=t}^{T-2} \gamma^{k-t+1} [R_{k+2} + \gamma V_k(S_{k+2}) - V_k(S_{k+1})].
\end{align}

\subsection{Exercise 6.2}
\subsubsection*{Q}
This is an exercise to help develop your intuition about why TD methods are often more efficient than Monte Carlo methods. Consider the driving home example and how it is addressed by TD and Monte Carlo methods. Can you imagine a scenario in which a TD update would be better on average than a Monte Carlo update? Give an example scenario—a description of past experience and a current state—in which you would expect the TD update to be better. Here’s a hint: Suppose you have lots of experience driving home from work. Then you move to a new building and a new parking lot (but you still enter the highway at the same place). Now you are starting to learn predictions for the new building. Can you see why TD updates are likely to be much better, at least initially, in this case? Might the same sort of thing happen in the original task?
\subsubsection*{A}
TD updates incorporate prior information. Suppose we had a good value estimate for a trajectory $\tau = S_1, S_2, \dots, S_T$, then if we try to estimate the trajectory $\tau = S_0, S_1, S_2, \dots, S_T$ using MC then we need to see multiple episodes of this to get a good estimate of $V(S_0)$, not leveraging the info we already have on $\tau$. TD would use info on $\tau$ to back up the value of $S_0$ and hence converge much quicker. The key differences here are bootstrapping and online learning.

\subsection{Exercise 6.3}
\subsubsection*{Q}
From the results shown in the left graph of the random walk example it appears that the first episode results in a change in only $V(A)$. What does this tell you about what happened on the first episode? Why was only the estimate for this one state changed? By exactly how much was it changed?

\subsubsection*{A}
All states apart from the terminal states were initialised to the same value (the terminal states must be initialised to 0) and the reward for non-terminal transitions is 0, so the TD(0) updates do nothing on the first pass to states that cannot lead directly to termination. \\

In the first run, the agent terminated on the left.
\begin{align}
    V_1(A) &= V_0(A) + \alpha[ 0 + \gamma \times 0 + V_0(A)]\\
           &= (1-\alpha) V_0(A) \\
           &= 0.9 \times 0.5 \\
           &= \frac{9}{20}.
\end{align}
The value of the estimate for $A$ reduced by $\alpha V_0(A) = 0.05$.

\subsection{Exercise 6.4}
\subsubsection*{Q}
The specific results shown in the right graph of the random walk example are dependent on the value of the step-size parameter, $\alpha$. Do you think the conclusions about which algorithm is better would be affected if a wider range of $\alpha$ values were used? Is there a different, fixed value of $\alpha$ at which either algorithm would have performed significantly better than shown? Why or why not?
\subsubsection*{A}

\begin{itemize}
    \item General arguments given earlier about the benefits of TD are independent of $\alpha$
    \item Increases in $\alpha$ make the curve more
    \item Decreases in $\alpha$ make the curve more smooth but make it converge slower.
    \item We see enough of a range here to decide between the two methods
\end{itemize}

\subsection{*Exercise 6.5}
\subsubsection*{Q}
In the right graph of the random walk example, the RMS error of the TD method seems to go down and then up again, particularly at high $\alpha$s. What could have caused this? Do you think this always occurs, or might it be a function of how the approximate value function was initialized?
\subsubsection*{A}
The state C happens to have een initialised to its true value. As training starts, updates occur on outer states (making them more accurate individually) which makes the error across all states reduce. This happens until the residual inaccuracies in the outer states propagate to C. The higher values of $\alpha$ make this effect more pronounced, because the value estimate for C changes more readily in these cases.

\subsection{Exercise 6.6}
\subsubsection*{Q}
In Example 6.2 we stated that the true values for the random walk example are $\frac16$, $\frac26$, $\frac36$, $\frac46$, and $\frac56$ for states A through E. Describe at least two different ways that these could have been computed. Which would you guess we actually used? Why?
\subsubsection*{A}
Could have used DP, but probably did the following calculation.\\

First note that
\[
    V(s) = \E{}[\mathds{1}\{\textsf{terminate on right from } S\}] = \P{}(\textsf{terminate on right from } S).
\]
Also recognise that symmetry now implies $V(C) = 0.5$. Now
\begin{align*}
    V(E) &= \frac12 \times 1 + \frac12 \times V(D) \\
         &= \frac12 + \frac14[V(C) + V(E)],
\end{align*}
so $V(E) = \frac56$. We then get $V(D) = \frac46$ and we can calculate the other states in the same way.

\subsection{*Exercise 6.7}
\subsubsection*{Q}
Design an off-policy version of the $TD(0)$ update that can be used with arbitrary target policy $\pi$ and covering behavior policy $b$, using at each step $t$ the importance sampling ratio $\rho_{t:t}$(5.3).
\subsubsection*{A}
Let $G_t$ be returns from an episode generated by $b$. Then
\begin{align*}
    v_\pi(s) &= \E{}[\rho_{t:T-1}G_t | S_t=s]\\
             &= \E{}[\rho_{t:T-1} R_{t+1} + \gamma \rho_{t:T-1}G_{t+1} | S_t=s] \\
             &= \rho_{t:t} \E{}[R_{t+1}|S_t=s] + \gamma \rho_{t:t}\E{}[\rho_{t+1:T-1}G_{t+1} |S_t=s] \\
             &= \rho_{t:t} \left( \E{}[R_{t+1} | S_t=s] + \E{}[\rho_{t+1:T-1}G_t| S_t = s] \right)\\
             &= \rho_{t:t} \left( r(s, A_t) + v_\pi(S_{t+1})\right).
\end{align*}
So the update for off-policy TD(0) (by sampling approximation) is
\begin{equation}
    V(S_t) \leftarrow V(S_t) + \alpha \left[ \rho_{t:t} R_{t+1} + \rho_{t:t} \gamma V(S_{t+1}) - V(S_t) \right].
\end{equation}

\subsection{Exercise 6.8}
\subsubsection*{Q}
Show that an action-value version of (6.6) holds for the action-value form of the TD error $\delta_t = R_{t+1} + \gamma Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t)$, again assuming that the values don’t change from step to step.
\subsubsection*{A}
Write $\delta_t \doteq R_{t+1} + \gamma Q(S_{t+1}, A_{t+1}) -  Q(S_t, A_t)$. Then the Monte-Carlo error is
\begin{align*}
    G_t - Q(S_t, A_t) &= R_{t+1} + \gamma G_{t+1} - Q(S_t, A_t) \\
                      &= \delta_t - \gamma [Q(S_{t+1}, A_{t+1}) + G_{t+1}]\\
                      &= \sum_{k=t}^{T-1}\gamma^{k-t}\delta_k
\end{align*}

\subsection{Exercise 6.9 (programming): Windy Grid World with King's Moves}
\subsubsection*{Q}
Re-solve the windy gridworld assuming eight possible actions, including the diagonal moves, rather than the usual four. How much better can you do with the extra actions? Can you do even better by including a ninth action that causes no movement at all other than that caused by the wind?

\subsubsection*{A}
\ProgrammingExercise\\

Optimal trajectory is now 7 steps, rather than 15. Including the do-nothing action is not helpful in this example because the wind blows vertically and the goal position is not vertically separated from the start position. It could be useful in other wind environments though.\\

Below are the optimal trajectories for the book example (no diagonal moves) and for the exercise (king's moves). The numbers represent the wind strength in that position.\\

\includegraphics[width=\textwidth]{\ExerciseOutput/ex_6_9/optimal_no_diagonal_moves.png}

\includegraphics[width=\textwidth]{\ExerciseOutput/ex_6_9/optimal_with_diagonal_moves.png}

\subsection{Exercise 6.10 (programming): Stochastic Wind}
\subsubsection*{Q}
Re-solve the windy gridworld task with King’s moves, assuming that the effect of the wind, if there is any, is stochastic, sometimes varying by 1 from the mean values given for each column. That is, a third of the time you move exactly according to these values, as in the previous exercise, but also a third of the time you move one cell above that, and another third of the time you move one cell below that. For example, if you are one cell to the right of the goal and you move left, then one-third of the time you move one cell above the goal, one-third of the time you move two cells above the goal, and one-third of the time you move to the goal.

\subsubsection*{A}
\ProgrammingExercise\\

Greedy trajectory and learning curve shown below. Note that although the gradient of the learning curve becomes constant (so the algorithm converges), the greedy episode shown suffers from stochasticity in the wind.

\includegraphics[width=\textwidth]{\ExerciseOutput/ex_6_10/optimal.png}

\includegraphics[width=\textwidth]{\ExerciseOutput/ex_6_10/learning_curve.eps}

\subsection{Exercise 6.11}
\subsubsection*{Q}
Why is Q-learning considered an \emph{off-policy} control method?

\subsubsection*{A}
The returns are sampled as if the agent followed the greedy policy with respect to $Q$.

\subsection{Exercise 6.12}
\subsubsection*{Q}
Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as Sarsa? Will they make exactly the same action selections and weight updates?
\subsubsection*{A}
Yes (?)

\subsection{Exercise 6.13}
\subsubsection*{Q}
What are the update equations for Double Expected Sarsa with an $\varepsilon$-greedy target policy?
\subsubsection*{A}
Expected SARSA has the update
\[
    Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \left[ R_{t+1} + \gamma \E{}[Q(S_{t+1}) | S_{t+1}] - Q(S_t, A_t)\right]
\]

The update for $S_t, A_t$ is
\[
    R_{t+1} + \gamma \sum_a \pi(a|S_{t+1}) Q(S_{t+1}, a) - Q(S_t, A_t).
\]
Double expected SARSA would be keeping two $Q$ arrays and updating one of them each timestep, chosen with equal probability.\\

For an $\varepsilon$-greedy policy we would increment $Q_1(S_t, A_t)$ by
\begin{equation}
    \alpha \left[ R_{t+1} + \gamma \left( \frac{\varepsilon}{|\mathcal{A}(a)|} \sum_a Q_2(S_{t+1}, a) + (1-\varepsilon)\max_a\{Q_2(S_{t+1}, a)\} \right) - Q_1(S_t, A_t)  \right]
\end{equation}
and the same with 1 and 2 reversed.

\subsection{Exercise 6.14}
\subsubsection*{Q}
Describe how the task of Jack’s Car Rental (Example 4.2) could be reformulated in terms of afterstates. Why, in terms of this specific task, would such a reformulation be likely to speed convergence?

\subsubsection*{A}
One might have coded this up initially with the states as the number of cars in each garage each evening. The agent then takes some action (moves some cars) and we transition stochastically to some state.\\

An alternative would be to introduce the number of cars in the morning (after the agent has moved cars) as an afterstate. This is because the agent is able to deterministically change the environment from evening to next morning (before rentals or returns).\\

In this case we would speed convergence by reducing the number of action-values to be calculated. For instance, we can now evaluate $(10, 0)$ moving one car and $(9, 1)$ moving no cars as the same afterstate $(9, 1)$.


% This is part of the TFTB Reference Manual.
% Copyright (C) 1996 CNRS (France) and Rice University (US).
% See the file refguide.tex for copying conditions.

\markright{tfrmh}
\section*{\hspace*{-1.6cm} tfrmh}

\vspace*{-.4cm}
\hspace*{-1.6cm}\rule[0in]{16.5cm}{.02cm}
\vspace*{.2cm}

{\bf \large \sf Purpose}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
Margenau-Hill time-frequency distribution.
\end{minipage}
\vspace*{.2cm}

{\bf \large \sf Synopsis}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
\begin{verbatim}
[tfr,t,f] = tfrmh(x)
[tfr,t,f] = tfrmh(x,t)
[tfr,t,f] = tfrmh(x,t,N)
[tfr,t,f] = tfrmh(x,t,N,trace)
\end{verbatim}
\end{minipage}
\vspace*{.3cm}

{\bf \large \sf Description}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
        {\ty tfrmh} computes the Margenau-Hill distribution of a
        discrete-time signal {\ty x}, or the cross Margenau-Hill
        representation between two signals. This distribution has the
        following expression :
\begin{eqnarray*}
MH_x(t,\nu)&=&\Re\left\{x(t)\ X^*(\nu)\ e^{-j2\pi \nu t}\right\}\\
&=&\int_{-\infty}^{+\infty} \frac{1}{2}\ (x(t+\tau)\ x^*(t)+x(t)\
x^*(t-\tau))\ e^{-j2\pi \nu \tau}\ d\tau.
\end{eqnarray*}
It corresponds to the real part of the Rihaczek distribution (see {\ty
tfrri}).\\

\hspace*{-.5cm}\begin{tabular*}{14cm}{p{1.5cm} p{10cm} c}
Name & Description & Default\\
\hline
        {\ty x}     & signal if auto-MH, or {\ty [x1,x2]} if cross-MH. {\ty
			(Nx=length(x))}\\
        {\ty t}     & time instant(s)          & {\ty (1:Nx)}\\
        {\ty N}     & number of frequency bins & {\ty Nx}\\
        {\ty trace} & if nonzero, the progression of the algorithm is shown
                                         & {\ty 0}\\
     \hline {\ty tfr}   & time-frequency representation\\
        {\ty f}     & vector of normalized frequencies\\

\hline
\end{tabular*}
\vspace*{.1cm}

When called without output arguments, {\ty tfrmh} runs {\ty tfrqview}.
\end{minipage}
\vspace*{.3cm}

{\bf \large \sf Example}
\begin{verbatim}
         sig=fmlin(128,0.1,0.4); tfrmh(sig,1:128,128,1);
\end{verbatim}
\vspace*{.3cm}

{\bf \large \sf See Also}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
all the {\ty tfr*} functions.
\end{minipage}
\vspace*{.3cm}

{\bf \large \sf Reference}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
[1] H. Margenhau, R. Hill ``Correlation between Measurements in Quantum
Theory'', Prog. Theor. Phys. Vol. 26, pp. 722-738, 1961.
\end{minipage}


\subsection{Partial Fractions}

If we have a function that is the quotient of two polynomials $f(x) = \frac{P(x)}{Q(x)}$, it's often easier to break this quotient into a sum of parts where the denominator is a linear or quadratic factor and the numerator is always a smaller polynomial degree than the denominator.

\input{./backgroundReview/algebraPreCalc/partialFractions_example.tex}

One natural way to find these small denominators comes from the linear factors of the denominator where we keep complex roots in quadratic form.
This way, when making a common denominator, we get back the original big denominator.
However, there are a few special cases we have to take care of.

\input{./backgroundReview/algebraPreCalc/linearFactors.tex}
\input{./backgroundReview/algebraPreCalc/repeatedLinearFactors.tex}
\input{./backgroundReview/algebraPreCalc/quadraticFactors.tex}
\input{./backgroundReview/algebraPreCalc/repeatedQuadraticFactors.tex}
\input{./backgroundReview/algebraPreCalc/improperFractions.tex}


% This is part of the TFTB Reference Manual.
% Copyright (C) 1996 CNRS (France) and Rice University (US).
% See the file refguide.tex for copying conditions.

\markright{tfrrgab}
\section*{\hspace*{-1.6cm} tfrrgab}

\vspace*{-.4cm}
\hspace*{-1.6cm}\rule[0in]{16.5cm}{.02cm}
\vspace*{.2cm}

{\bf \large \sf Purpose}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
Reassigned Gabor spectrogram time-frequency distribution.
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Synopsis}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
\begin{verbatim}
[tfr,rtfr,hat] = tfrrgab(x)
[tfr,rtfr,hat] = tfrrgab(x,t)
[tfr,rtfr,hat] = tfrrgab(x,t,N)
[tfr,rtfr,hat] = tfrrgab(x,t,N,Nh)
[tfr,rtfr,hat] = tfrrgab(x,t,N,Nh,trace)
[tfr,rtfr,hat] = tfrrgab(x,t,N,Nh,trace,k)
\end{verbatim}
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Description}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
        {\ty tfrrgab} computes the Gabor spectrogram and its reassigned
        version.  The analysis window $h$ used in this spectrogram is a
        gaussian window, which allows a 20 \% faster algorithm than with
        the {\ty tfrrsp} function (windows $\ens{T}_h$ and $\ens{D}_h$
        defined above are colinear in this case). The reassigned Gabor
        spectrogram is given by the following expressions :
\begin{eqnarray*}
 S_x^{(r)}(t',\nu';h)=\iint_{-\infty}^{+\infty} S_x(t,\nu;h)\
\delta(t'-\hat{t}(x;t,\nu))\ \delta(\nu'-\hat{\nu}(x;t,\nu))\ dt\ d\nu,
\end{eqnarray*}
where
\begin{eqnarray*}
\hat{t}(x;t,\nu)=t-\Re\left\{\dfrac{F_x(t,\nu;\ens{T}_h)\ F_x^*(t,\nu;h)}
{|F_x(t,\nu;h)|^2}\right\} \\
\hat{\nu}(x;t,\nu)=\nu+\Im\left\{\dfrac{F_x(t,\nu;\ens{D}_h)\ F_x^*(t,\nu;h)}
{2\pi\ |F_x(t,\nu;h)|^2}\right\}
\end{eqnarray*}
with $\ens{T}_h(t)=t\ h(t)$ and $\ens{D}_h(t)=\frac{dh}{dt}(t)$.\\

\hspace*{-.5cm}\begin{tabular*}{14cm}{p{1.5cm} p{8cm} c}
Name & Description & Default value\\
\hline
        {\ty x}     & analyzed signal ({\ty Nx=length(x)})\\
        {\ty t}     & the time instant(s)           & {\ty (1:Nx)}\\
        {\ty N}     & number of frequency bins      & {\ty Nx}\\
        {\ty Nh}    & length of the gaussian window & {\ty N/4}\\
        {\ty trace} & if nonzero, the progression of the algorithm is shown
                                              & {\ty 0}\\
        {\ty k}     & value at both extremities     & {\ty 0.001}\\

\hline \end{tabular*} \end{minipage}

%\newpage

\hspace*{1.5cm} \begin{minipage}[t]{13.5cm}
\hspace*{-.5cm}\begin{tabular*}{14cm}{p{1.5cm} p{8cm} c}
Name & Description & Default value\\
\hline
        {\ty tfr, rtfr} & time-frequency representation and its reassigned
              version\\
        {\ty hat}   & complex matrix of the reassignment vectors\\
\hline
\end{tabular*}
\vspace*{.2cm}

When called without output arguments, {\ty tfrrgab} runs {\ty tfrqview}.
\end{minipage}
\vspace*{1cm}

{\bf \large \sf Example}
\begin{verbatim}
         sig=fmlin(128,0.1,0.4);
         tfrrgab(sig,1:128,128,19,1);
\end{verbatim}
\vspace*{.5cm}

{\bf \large \sf See Also}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
all the {\ty tfr*} functions.
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Reference}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
[1] F. Auger, P. Flandrin ``Improving the Readability of Time-Frequency and
Time-Scale Representations by the Reassignment Method'' IEEE Transactions
on Signal Processing, Vol. 43, No. 5, pp. 1068-89, 1995.
\end{minipage}


\section{Photons}
Suppose you have a metal box at temperature $T$.
There is an electromagnetic field inside in equilibrium with the walls.
The field vanishes at the walls though.

Why should anybody study this?
Because all bodies which absorb all radiation shining on them (black bodies)
emit radiation the same way when heated and a metal cavity is the best or
easiest way to study.

The energy emitted by A equals the energy absorbed by a black body B equals the
energy emitted by B.
This is true frequency by frequency,
so $u(\nu)$ which is the energy density per unit is universal for every black
body.

We will find $u(\nu)$ by applying equilibrium statistical mechanics to the
electromagnetic field inside the box.

In order to have a Hamiltonian formalism,
we'll use the variables
$\vec{A}(\vec{r}, t)$ instead of $\vec{E}$ and $\vec{B}$.

Different $\vec{A}$'s differing by a gauge transformation,
give the same $\vec{E}$ and $\vec{B}$ and describe the same physics.
We'll impose the condition
\begin{align}
    \vec{\nabla}\cdot{A} &= 0\\
    \phi &= 0
\end{align}
to select one $\vec{A}$ for every physically distinct field $\vec{E},\vec{B}$.
Recalling that we can recover the electric and magnetic fields with
\begin{align}
    \vec{E} &= -\frac{\partial A}{c \partial t} - \vec{\nabla}\phi
    =
    -\frac{1}{c}\frac{\partial \vec{A}}{\partial t}\\
    \vec{B} &= \vec{\nabla}\times\vec{A}
\end{align}
The Lagrangian
\begin{align}
    L &=
    \frac{1}{8\pi} \int d^3 r\,
    \left[
    \left( \frac{\partial \vec{A}}{c \partial t} \right)^2
    - \left( \vec{\nabla}\times\vec{A} \right)^2
    \right]
\end{align}
leads to the equation of motion
\begin{align}
    \frac{1}{c^2} \frac{\partial A}{\partial t^2}
    - \nabla^2 \vec{A} &= 0
\end{align}
which is equivalent to Maxwell's equations.
The Hamiltonian is
\begin{align}
    H &=
    \int d^3r\left[
    \vec{\Pi}\cdot \frac{\partial \vec{A}}{\partial t} - H
    \right]\\
    &=
    \int d^3r\,\left[
    \Pi^2 4\pi c^2
    - \frac{(4\pi c)^2}{8\pi} \Pi^2
    + \frac{\vec{\nabla}\times\vec{A}}{8\pi}
    \right]
\end{align}
where
\begin{align}
    \vec{\Pi} &:=
    \frac{\partial H}{\partial \dot{\vec{A}}(\vec{r})}
    =
    \frac{1}{4\pi} \frac{1}{c^2} \frac{\partial \vec{A}}{\partial t}\\
\end{align}
so then
\begin{align}
    H
    &=
    \frac{1}{8\pi} \int d^3r\,\left[
    \underbrace{\left( 4\pi c \right)^2 \Pi^2}_{E^2}
    +
    \underbrace{\vec{\nabla}\times\vec{A}}_{B^2}
    \right]
\end{align}
Applying a change of variables to
\begin{align}
    \vec{A}(\vec{r}, t) &=
    \frac{1}{\sqrt{V}} \sum_k
    \vec{Q}(\vec{k}, t) e^{i\vec{k}\cdot \vec{r}}\\
    \vec{\Pi}(\vec{r}, t) &=
    \frac{1}{\sqrt{V}} \sum_k
    \frac{\vec{P}(\vec{k}, t)}{\sqrt{4\pi} c^2} e^{i\vec{k}\cdot\vec{r}}
\end{align}
where $\vec{k}=\frac{2\pi}{L}\vec{n}$ with $\vec{n}=0,\pm 1,\ldots$
and
\begin{align}
    \vec{Q}_{\vec{k}}^* &= \vec{Q}_{-\vec{k}}\\
    \vec{P}_{\vec{k}}^* &= \vec{P}_{-\vec{k}}
\end{align}
so then the Hamiltonian becomes decoupled as
\begin{align}
    H &=
    \frac{1}{2}\sum_{k}\left[
    \vec{P}_{\vec{k}}\cdot\vec{P}_{-\vec{k}}
    +
    \underbrace{c^2 k^2}_{=:\omega_k^2}
    \vec{Q}_{\vec{k}}\cdot\vec{Q}_{-\vec{k}}
    \right]
\end{align}
which is very much like the phonon Hamiltonian.
Classically,
the energy density is given by the equipartition theorem
\begin{align}
    E &=
    \sum_{k} \frac{1}{2} k_B
    \times \underbrace{2}_{\textrm{$P$ and $Q$ are quadratic}}
    \times \underbrace{2}_{\textrm{2 independent polarizations}}\\
    &\approx V \int d^3 k 2k_B T\\
    &\to \infty
\end{align}
The divergence comes because contrary to the phonon case,
there is not cutoff at high momentum $k$.
Still,
the contribution from every frequency range
\begin{align}
    \frac{E}{V} &=
    \frac{2k_B T}{2\pi^2} \int_{0}^{\infty} dk\, k^2\\
    &=
    \int_{0}^{\infty}
    \underbrace{\frac{k_B T}{\pi^2} \frac{(2\pi)^3}{c^3} \nu^2}_{u(\nu)}
    \, d\nu
\end{align}
and with $\nu = \frac{kc}{2\pi}$,
the energy density between $\nu$ and $\nu + d\nu $ is
\begin{align}
    u(\nu) \, d\nu &=
    \frac{8k_B T \pi}{c^3} \nu^2\, d\nu
\end{align}
this is known as the \emph{Rayleigh-Jeans law},
which agrees with experiments in the low frequency regime.
But not for high frequencies as it diverges!

The quantum theory is obtained by making
$\vec{A}$, $\vec{\Pi}$, $\vec{Q}$ and $\vec{P}$
into operators.

Assuming canonical commutation relations for $\vec{A}$ and $\vec{\Pi}$.
\begin{align}
    \left[ \hat{A}_i(\vec{r}), \hat{\Pi}_j(\vec{r}') \right]
    =
    i\hbar \delta_{ij} \delta\left( \vec{r} - \vec{r}' \right)
\end{align}
we find
\begin{align}
    \left[ \hat{A}_i(\vec{r}), \hat{\Pi}_j(\vec{r}') \right] &=
    \frac{1}{V} \sum_{k,k'}
    e^{i\left( \vec{k}\cdot\vec{r} + \vec{k}'\cdot\vec{r}' \right)}
    \underbrace{ \left[ Q_i(\vec{k}), P_j(\vec{k}') \right]}_{%
    i\hbar \delta_{ij} \delta_{\vec{k},-\vec{k}'}
    }\\
    &=
    \frac{i\hbar}{V} \sum_{\vec{k}}
    e^{i\vec{k}\cdot\left( \vec{r} - \vec{r}' \right)} \delta_{ij}\\
    &=
    i\hbar \delta_{ij} \delta\left( \vec{r} - \vec{r}' \right)
\end{align}
so then
\begin{align}
    \left[ Q_i\left( \vec{r} \right), P_j\left( \vec{r}' \right) \right]
    =
    i\hbar \delta_{ij} \delta_{\vec{k}, -\vec{k}'}
\end{align}
and all the other commutators between $Q_k$ and $P_k$ vanish.
We then define
\begin{align}
    \hat{a}_{\vec{k}}^{\alpha} &=
    \left(
    \sqrt{\frac{\omega_k}{2\hbar}} \vec{Q}(\vec{k})
    + \frac{i}{\sqrt{2\omega_k\hbar}} \vec{P}(\vec{k})
    \right)
    \cdot e_{\vec{k}}^{\alpha}\\
    {\hat{a}_{\vec{k}}^{\alpha}}^\dagger &=
    \left(
    \sqrt{\frac{\omega_k}{2\hbar}} \vec{Q}(\vec{k})
    - \frac{i}{\sqrt{2\omega_k\hbar}} \vec{P}(\vec{k})
    \right)
    \cdot e_{\vec{k}}^{\alpha}
\end{align}
where $e_{\vec{k}}^1$ and $e_{\vec{k}}^2$
are two unit vectors orthogonal to $\vec{k}$,
which are the polarization directions.
And the commutation relations
\begin{align}
    \left[
    \hat{a}_{\vec{k}}^{\alpha},
    {\hat{a}_{\vec{k}}^{\beta}}^\dagger,
    \right]
    &=
    -i \sqrt{\frac{\omega_k}{2\hbar}}
    \frac{1}{\sqrt{2\omega_{k'}\hbar}}
    \underbrace{\left[ Q_i(\vec{k}), P_j(\vec{k}'), \right]}_{%
    i\hbar \delta_{ij} \delta_{k,k'}
    }
    e_i^{\alpha}(k)
    e_j^{\beta}(k)\nonumber\\
    &\qquad
    - \sqrt{\frac{\omega_{k'}}{2\hbar}}
    \frac{1}{\sqrt{2\omega_k \hbar}}
    \underbrace{\left[ P_j(\vec{k}), Q_i(\vec{k}') \right]}_{%
    -i\hbar \delta_{ij} \delta_{k,k'}
    }
    e_j^{\alpha}(k)
    e_i^{\alpha}(k')\\
    &=
    \delta_{\alpha\beta} \delta_{\vec{k},\vec{k}'}
\end{align}
and also
\begin{align}
    \left[ \hat{a}_{\vec{k}}^{\alpha}, \hat{a}_{\vec{k}'}^{\beta} \right] =
    \left[
    {\hat{a}_{\vec{k}}^{\alpha}}^\dagger,
    {\hat{a}_{\vec{k}'}^{\beta}}^\dagger
    \right] = 0
\end{align}
Thus,
$\hat{a}^\alpha(k)$ and ${\hat{a}^{\alpha}}^\dagger(k)$
are creation and annihilation operators for photons with momentum $k$ and
polarization $e^{\alpha}(k)$.
In terms of them,
the Hamiltonian is
\begin{align}
    \hat{H} &=
    \frac{1}{2} \sum_{\alpha=1,2} V \int d^3k\,\left[
    -\frac{\hbar\omega_k}{2}
    \left(
    \hat{a}_{\vec{k}}^{\alpha}
    - {\hat{a}_{-\vec{k}}^{\alpha}}^\dagger
    \right)
    \left(
    \hat{a}_{-\vec{k}}^{\alpha}
    - {\hat{a}_{\vec{k}}^{\alpha}}^\dagger
    \right)
    +
    \omega_k^2
    \left(
    \hat{a}_{\vec{k}}^{\alpha}
    - {\hat{a}_{-\vec{k}}^{\alpha}}^\dagger
    \right)
    \left(
    \hat{a}_{-\vec{k}}^{\alpha}
    - {\hat{a}_{\vec{k}}^{\alpha}}^\dagger
    \right)
    \right]\\
    &\approx
    \frac{1}{2} \sum_{\alpha=1,2} \int d^3k\left[
    \frac{\hbar\omega_k}{2} 2
    \hat{a}_{k}^{\alpha}
    {\hat{a}_{k}^{\alpha}}^\dagger
    +
    {\hat{a}_{k}^{\alpha}}^\dagger
    \hat{a}_{k}^{\alpha}
    2
    \frac{\hbar\omega_k}{2}
    \right]\\
    &=
    \sum_{\alpha=1,2} \int d^3k\left[
    {\hat{a}_{k}^{\alpha}}^\dagger
    \hat{a}_{k}^{\alpha}
    +
    \frac{1}{2}
    \right]
    \hbar\omega_k
\end{align}
so that's one harmonic oscillator for every $\vec{k}$ and $\alpha$.

How we can calculate the partition function
\begin{align}
    Z &= \Tr e^{-\beta \hat{H}}\\
    &=
    \prod_{\vec{k},\alpha} \sum_{n_k=0}^{\infty}
    e^{-\beta\hbar\omega_k\left( n_k + \frac{1}{2} \right)}\\
    &= \prod_{\vec{k}, \alpha}
    \frac{e^{-\beta\hbar\omega_k/2}}{1 - e^{-\beta\omega_k\hbar}}\\
    &= \prod_{\vec{k},\alpha} \frac{2}{\sinh(\beta\hbar\omega_k/2)}
\end{align}
and the free energy
\begin{align}
    F &=
    -k_B T \ln Z\\
    &=
    k_B T \sum_{\vec{k}, \alpha} \ln\left(
    \frac{1 - e^{-\beta \omega_k \hbar}}{e^{-\beta\hbar\omega_k/2}}
    \right)\\
    &=
    k_B T \sum_{\vec{k},\alpha}\left[
    \ln\left(
    1 - e^{-\beta\hbar\omega_k}
    \right)
    +
    k_B T \beta \hbar\omega_k
    \right]\\
    &=
    k_B T \sum_{\vec{k},\alpha}\left[
    \ln\left(
    1 - e^{-\beta\hbar\omega_k}
    \right)
    +
    \underbrace{\hbar\omega_k}_{\textrm{$T$-independent zero-point energy}}
    \right]\\
    &\approx
    2 k_B T V \int d^3k \ln\left(
    1 - e^{-\beta \hbar\omega_k}
    \right)
    + \textrm{const}
\end{align}
which is the free energy of an ideal gas of bosons with dispersion relation
$\omega_k=kc$ and two polarizations plus a constant.

The energy is then
\begin{align}
    \frac{E}{V} &=
    2 \int d^3 k \frac{\hbar ck}{e^{\beta \hbar c k} - 1}\\
    &=
    \frac{2}{\left( \beta\hbar c \right)^3 }
    \frac{1}{2\pi^2}
    \frac{1}{\beta}
    \underbrace{\int_{0}^{\infty}
    dx\,
    \frac{x^3}{e^{x} - 1}}_{%
    \pi^4/15
    }\\
    &=
    \frac{\pi^2 \left( k_B T \right)^4}{15 \hbar^3 c^3}
\end{align}
This is the \emph{Steffan-Boltzmann law}.

Also, to get the spectrum,
\begin{align}
    \frac{E}{V} &=
    2 \frac{1}{2\pi^2} \int_{0}^{\infty} dk\,
    \frac{k^2 \hbar ck}{e^{\beta\hbar ck} - 1}\\
    &=
    \frac{\hbar c}{\pi^2} \left( \frac{2\pi}{c} \right)^4
    \int_{0}^{\infty} d\nu\,
    \frac{\nu^3}{e^{\beta h\nu} - 1}\\
    &=
    \int_{0}^{\infty} d\nu\,
    \underbrace{%
    \frac{16\pi^2\hbar}{c^3}
    \frac{\nu^3}{e^{\beta h \nu} - 1}}_{%
    u(\nu)
    }
\end{align}
where
\begin{align}
    u(\nu) =
    \frac{16\pi^2\hbar}{c^3}
    \frac{\nu^3}{e^{\beta h \nu} - 1}
\end{align}
is the energy per volume per frequency.
This is \emph{Planck's law}.


\subsection{The Space of Germs}
Let $D\subset\mathbb C$ a domain.
\begin{definition}
    Let $(f,U)$ and $(g,V)$ be function elements on $D$.
    For any $z\in U\cap V$, write $(f,U)\equiv_z(g,V)$ if $f,g$ agree on a neighbourhood of $z$.
\end{definition}
Easily $\equiv_z$ is an equivalence relation.
\begin{definition}
    Let $(f,U)$ be a function element and $z\in U$.
    The equivalence class of $(f,U)$ under $\equiv_z$ is called the germ of $f$ at $z$ and is denoted by $[f]_z$.
\end{definition}
So two germs $[f]_z$ and $[g]_w$ are equal iff $z=w$ and $f=g$ on a neighbourhood of $z=w$.
We want to study all possible germs on a domain $D$.
\begin{definition}
    The space of germ over $D$ is
    $$\mathcal G=\{[f]_z:z\in D,(f,D)\text{ a function element with $z\in U$}\}$$
\end{definition}
Now we defined it as a set, it is natural to endow a topology on it.
For any function element $(f,U)$ on $D$, let $[f]_U=\{[f]_z:z\in U\}$.
\begin{lemma}
    The collection $\{[f]_U\}$ for $U$ open in $D$ is a basis of a topology on $\mathcal G$.
\end{lemma}
\begin{proof}
    Let $(f,U)$ and $(g,V)$ be function elements on $D$.
    For any $[h]_z\in [f]_U\cap [g]_V$, then $h$ agrees with $f,g$ on a neighbourhood $W$ of $z$, therefore $[h]_W\subset [f]_U\cap[g]_V$.
\end{proof}
This is the topology we want.
\begin{lemma}
    $\mathcal G$ is Hausdorff.
\end{lemma}
\begin{proof}
    Consider elements of $[f]_z,[g]_z\in\mathcal G$ with $[f]_z\neq [g]_z$.\\
    If $z\neq w$ then we can choose function elements $(f,U)\in [f]_z,(g,V)\in[g]_w$ such that $U\cap V=\varnothing$, therefore $[f]_U$ and $[g]_V$ are disjoint.\\
    If $z=w$, then we can choose an open neighbourhood $U$ such that $(f,U)\in[f]_z$ and $(g,U)\in [g]_z$.
    Unless $[f]_U\cap [g]_U=\varnothing$, there is a germ $[h]_z\subset [f]_U\cap[g]_U$.
    By the identity principle $f|_U=h|_U=g|_U$, which means $[f]_z=[h]_z=[g]_z$ due to the connectedness of $U$, contradiction.
\end{proof}
\begin{definition}
    Let $\mathcal G$ be the space of germs over a domain $D$.
    The forgetful map $\pi:\mathcal G\to D$ is defined by $\pi([f]_z)=z$.
\end{definition}
\begin{lemma}
    For each component $G\subset\mathcal G$, the restriction $\pi:G\to D$ is a covering map.
\end{lemma}
\begin{proof}
    Take an open $U\subset D$.
    Then the pre-image of $U$ has to be
    $$\pi^{-1}(U)=\bigcup_{(f,V)\text{ function element on }U}[f]_V$$
    which is open.
    So $\pi$ is continuous.\\
    For each open set in the form $[f]_U$, we have
    $$(\pi|_{[f]_U})^{-1}(z)=[f]_z$$
    which is a continuous inverse of $\pi|_{[f]_U}$.
    This shows that $\pi$ is a local homeomorphism, hence a covering map.
\end{proof}
Hence, by Lemma \ref{covering_conformal}, $\pi$ induces a well-defined conformal structure on $\mathcal G$ (well, on each of its connected components) such that $\pi$ is analytic.
Explicitly, the atlas we have in mind consists of charts $(\pi|_{[f]_U},[f]_U)$ across all the function elements $(f,U)$ on $D$.
\begin{definition}
    Let $\mathcal G$ be the space of germs on a domain $D$.
    The evaluation map $\mathcal E:\mathcal G\to\mathbb C$ is defined by $\mathcal E([f]_z)=f(z)$.
\end{definition}
In the chart $(\pi|_{[f]_U},[f]_U)$, we have
$$\mathcal E\circ(\pi|_{[f]_U})^{-1}(z)=\mathcal E([f]_z)=f(z)$$
Therefore $\mathcal E$ is analytic.