\chapter{Heat}

Let's say you put a 1 kg aluminum pan that is $80^\circ$ C into
3 liters of water that is $20^\circ$ C. Energy, in the form of heat,
will be transferred from the pan to the water until they are at the same
temperature. (We call this ``thermal equilibrium.'')\index{thermal equilibrium}

What will the temperature of the water be?

\section{Specific Heat Capacity}

If you are heating something, the amount of energy you need to
transfer to it depends on three things: the mass of the thing you are
heating, the amount of temperature change you want, and the
\textit{specific heat capacity} of that substance.\index{specific heat capacity}

\begin{mdframed}[style=important, frametitle={Energy in Heat Transfer}]

  The energy moved in a heat transfer is given by

  $$E = m c \Delta_T$$

  where $m$ is the mass, $\Delta_T$ is the change in temperature, and
  $c$ is the specific heat capacity of the substance.
% ADD: q=mcat

  (Note that this
  assumes no phase change. For example, this formula works nicely on
  warming liquid water, but it gets more complicated if you warm the
  water past its boiling point.)

\end{mdframed}

Can we guess the specific heat capacity of a substance? It is very,
very difficult to guess the specific heat of a substance, so we determine
it by experimentation.

For example, someone determined that it took about 0.9 joules to raise
the temperature of solid aluminum one degree Celsius. So we say ``The
specific heat capacity of aluminum is 0.9 J/g $^\circ$C.''

The specific heat capacity of liquid water is about 4.2 J/g $^\circ$C.

To answer the question, then, the amount of energy given off by the
pan must equal the amount of energy absorbed by the water. And they
need to be the same temperature at the end.  Let $T$ be the final
temperature of both.

\includegraphics[width=0.8\textwidth]{Specific_Heat_Diagram.png}

% KA: https://www.khanacademy.org/science/ap-chemistry-beta/x2eef969c74e0d802:thermodynamics/x2eef969c74e0d802:heat-capacity-and-calorimetry/v/heat-capacity

Three liters of water weighs 3,000 grams, so the
change in energy in the water will be:

$$E_W = m c \Delta_T = (3000)(4.2)(T - 20) = 12600T - 252000 \text{ joules}$$

The pan weighs 1000 grams, so the change in energy in the pan will be::

$$E_P = m c \Delta_T = (1000)(0.9)(T - 80) = 900T - 72000 \text{ joules}$$

Total energy stays the same so $E_W + E_P = 0$.  So you need to solve

$$(12600T - 252000) + (900T - 72000) = 0$$

And find that the temperature at equilibrium will be

$$T = 24^\circ \text{C}$$

\begin{Exercise}[title={Thermal Equilibrium}, label=thermal_equilibrium]

Just as you put the aluminium pan in the water as described above,
someone also puts a 1.2 kg block of copper cooled to 10 $^\circ$ C.
The specific heat of solid copper is about 0.4 J/g $^\circ$C.

What is the new temperature at equilibrium?

\end{Exercise}
\begin{Answer}[ref=thermal_equilibrium]

  $$E_C = (1200)(0.4)(T - 10) = 480T - 4800$$

Total energy stays constant:

$$0 = (12600T - 252000) + (900T - 72000) + (480T - 4800)$$

Solving for $T$ gets you $T = 23.52^\circ$ C.

\end{Answer}

\section{Getting to Equilibrium}

When two objects with different temperatures are touching, the speed
at which they exchange heat is proportional to the differences in
their temperatures. Thus, as their temperatures get closer together,
the heat exchange slows down.
% ADD: explain which object, water or metal has a greater tempature change

In our example, the pan and the water will get close to equilibrium
quickly, but they may never actually reach equilibrium.

\begin{tikzpicture}
    \begin{axis}[
        xmin=0,xmax=4.25,
        ymin=15,ymax=85,
        axis x line=middle,
        axis y line=middle,
        axis line style=<->,
        xlabel={minutes},
        ylabel={degrees celsius},
        ]
        \addplot[no marks,sdkblue] expression[domain=0:4,samples=100]{24 + 56 * pow(2,-1.5 * x)} node[above, xshift=-1cm, yshift=0.1cm]{Pan};
        \addplot[no marks,sdkblue] expression[domain=0:4,samples=100]{24 - 4 * pow(2,-0.8 * x)} node[below, xshift=-2.5cm]{Water};
        \addplot[no marks,dashed,gray] coordinates {(0,24)(6,24)} node[above, xshift=-8cm]{equilibrium};
    \end{axis}
\end{tikzpicture}

\begin{Exercise}[title={Cooling Your Coffee}, label=cool_coffee]

  You have been given a ridiculously hot cup of coffee and a small pitcher of chilled milk.

  You need to start chugging your coffee in three minutes, and you want it as cool as possible at that time. When should you add the milk to the coffee?

\end{Exercise}
\begin{Answer}[ref=cool_coffee]

  During the 3 minutes, you want the coffee to give off as much of its
  heat as possible, so you want to maximize the difference between the
  temperature of the coffee and the temperature of the room around
  it.

  You wait until the last moment to put the milk in.

\end{Answer}

\section{Specific Heat Capacity Details}

For any given substance, the specific heat capacity often changes a
lot when the substance changes state. For example, ice is 2.1 J/g
$^\circ$C, whereas liquid water is 4.2 J/g$^\circ$C.
% KA: https://www.khanacademy.org/science/biology/water-acids-and-bases/water-as-a-solid-liquid-and-gas/v/specific-heat-of-water

Even within a given state, the specific heat capacity varies a bit
based on the temperature and pressure. If you are trying to do these
sorts of calculations with great accuracy, you will want to find the
specific heat capacity that matches your situation. For example, I
might look for the specific heat capacity for water at $22^\circ$C at
1 atmosphere of pressure( atm).


\section{Notation}

\input{tab-graphblas-notation}

\autoref{tab:graphblas-notation} contains the notation of \grb operations

Additionally, we use $\grbm{D} = \grbdiag{\grba{J}, \grbs{n}}$ to construct a diagonal matrix $\grbm{D} \grbassign \{ \grba{J}, \grba{J}, [1, 1, \ldots, 1] \}$. The elements of the matrix are $\grbm{D}(\grbs{j}, \grbs{j}) = 1$ for $\grbs{j} \in \grba{J}$.

Initializing scalars, vectors, and matrices (GraphBLAS methods):

\begin{itemize}
    \item $\grbnewscalar{\grbss}{\grbfloat}{64}$
    \item $\grbnewvector{\grbvu}{\grbfloat}{32}{n}$
    \item $\grbnewmatrix{\grbmA}{\grbuint}{16}{m}{n}$
    \item $\grbnewmatrix{\grbmA}{\grbint}{64}{k}{m}$
\end{itemize}


\section{Recap}
Suppose you have a box at temperature $T$.
Maxwell equations gives you $\vec{A}(\vec{r}, t)$,
which has a lot of degrees of freedom.
The Lagrangian is
\begin{align}
    L_0 &=
    \frac{1}{8\pi} \int d^3r \left[
    \left( \frac{\partial A}{c\partial t} \right)^2
    - \left( \vec{\nabla}\times \vec{A} \right)^2
    \right]
\end{align}
There is some exercise to show this is equivalent to Maxwell's equations,
but that is a classical physics problem.
You could keep going and find the classical Hamiltonian,
which you know how to find and is a function of the degrees of freedom $\vec{A}$
and their canonical momentum $\Pi$.
\begin{align}
    H &=
    \frac{1}{8\pi}
    \int d^3 r\, \left[
    \left( 4\pi c \right)^2
    \Pi^2 (r, t)
    + \left( \vec{\nabla}\times\vec{A}  \right)^2
    \right]
\end{align}
where the canonical momenta are
\begin{align}
    \Pi(\vec{R}, t) &=
    \frac{\partial L}{\partial A(\vec{r}, t)}
\end{align}
It's an infinite-dimensional phase space but that's okay.
It's a quadratic Hamiltonian,
so you can carry out all the calculations.
The way to do this is that you have to go to normal coordinates.
If I had only terms like $A^2(\vec{R})$,
it only couples to itself.
But since there are derivatives of $A$,
I have terms like
$\left( A(\vec{r} + d\vec{r}, t) - A(\vec{r}, t) \right)^2$,
which is very similar to the chain of masses connected by springs
with $-k(x_1 - x_2)^2$.
If you remember the normal coordinates of the mass and springs system,
you just take the Fourier transform to decouple the Fourier modes to get the
normal coordinates.

Again,
for the electromagnetic field,
I can write the transformation to normal coordinates like
\begin{align}
    A(\vec{r},t) &=
    \sqrt{4\pi c}
    \sum_{\vec{k}}
    e^{i\vec{k}\cdot\vec{r}} Q(\vec{k}, t)
\end{align}
where $Q$ are the cavity modes inside the box,
and the wave vectors are of the form
\begin{align}
    \vec{k} &= \frac{2\pi}{L} \vec{n}
\end{align}
where $\vec{n}$ are vectors with integer components.
Then the canonical momenta are
\begin{align}
    \Pi(\vec{r}, t) &=
    \frac{1}{\sqrt{V}\sqrt{4\pi c}} \sum_{\vec{k}}
    e^{i\vec{k}\cdot\vec{r}} P(\vec{k}, t)
\end{align}
and now the degrees of freedom are decoupled and I can calculate the partition
function in my head.

When we found the Lagrangian,
we picked a certain gauge,
that is divergence of the potential is zero and the curl of $A$ is zero,
so what extra constraints do we have?
\begin{align}
    \nabla\cdot A &= 0
    \implies
    \vec{k}\cdot\vec{Q}(\vec{k}, t) = 0
\end{align}
which means one less degree of freedom,
so electromagnetic waves are not longitudinal.
It's hard to do the counting,
but I start with a real vector $A$,
and I'm left with a complex $Q$,
so the degrees of freedom doubles,
but that can't be.
The trick here is that because $A$ is real,
that implies something about these $Q$'s.
\begin{align}
    \vec{Q}(k) = \vec{Q}^*(-k)
\end{align}
which is because if you take the complex conjugate of the definition,
you should get the same thing for $-k$.
And the same thing applies for the $\vec{P}$.
One way of thinking is that if I know the real value of $Q$,
I also know its imaginary value,
so I don't double the number of degrees of freedom,
it's the same.

This transformation is canonical because it satisfies certain conditions you
should know from Chacko's class.
I am allowed to do this and what's going to happen is that I'm going to find out
then what my Hamiltonian is going to look like in terms of $P$ and $Q$.
\begin{align}
    H &= \frac{1}{2} \sum_{\vec{k}}\left[
    \vec{P}(\vec{k}, t)\cdot\vec{P}^*(\vec{k}, t)
    + c^2 k^2 \vec{Q}(\vec{k}, t)\cdot \vec{Q}^*(\vec{k}, t)
    \right]
\end{align}
Before the $A$'s of neighbouring parts are coupled,
but now they are not.
A valid $k$ is not next to any other valid $k$,
just a sum of completely independent parts,
one for each $k$.
One harmonic oscillator for every value of $k$.

I don't know if you know that,
but EM an all that complication with curls and divergences,
is just a bunch of this.

Do you recognize this?
\begin{align}
    H &= \frac{1}{2}p^2 + \frac{1}{2}q^2
\end{align}
That's a harmonic oscillator.
This is for many oscillators.
\begin{align}
    H &= \sum_{i}\frac{1}{2}p_i^2 + \frac{1}{2}q_i^2
\end{align}
And the EM Hamiltonian is just an infinite number of oscillators.

Interesting thing to observe is that it tells us the frequency of each mode.
$c^2 k_^2 = \omega^2$.
If you calculate the velocity of the waves,
you get the velocity of the light $c$,
which is the same for any wavelength of light.
It's much like low-momentum phonons.

Note that I could have written
\begin{align}
    \vec{P}\cdot\vec{P} =
    \left( \Re P \right)^2 + \left( \Im P \right)^2
\end{align}

So I'm done right?

I can find classically what the average energy is going to be.
By equipartition,
I should have $\frac{1}{2}k_BT$ for every quadratic term in the Hamiltonian.
\begin{align}
    E &= \sum_{\vec{k}}\frac{1}{2}k_B T\left(
    \underbrace{2}_{\textrm{``kinetic'' + ``potential''}}
    + \underbrace{2}_{\textrm{polarization}}
    \right) = \infty
\end{align}
Only the transverse polarizations survives because
$vec{k}\cdot\vec{Q}=0$,
so you only have 2 polarizations.
Classically,
the energy does not depend on the frequency of the oscillators.
Differently from the phonons though,
this never ends,
because the maximum $k$ you could have is $\pi/a$,
but there is no lattice spacing $a$,
so this sum diverges!

It diverges because it can have modes of arbitrarily small wavelengths and each
one contributes exactly the same.

And so of course,
this is the famous ultraviolet catastrophe.
That's how quantum mechancis was invinented.
They did this calculation,
the result was not just slightly wrong,
but profoundly wrong.
And there you go.

There are some steps,
you may ask why you should believe,
but there is a reason to believe that this result is off because it's infinitely
off.

The reason is this.
Suppose now that you calculate the energy for a particular magnitude of
$k=|\vec{k}|$,
which corresponds to many different $k$ on a sphere of that radius.
Then you could try to calculate the energy density.

It's hard to do the sum because it lives on lattice,
but for large $k$,
you can do an approximation.

\begin{align}
    \frac{E}{V} &=
    =
    \frac{2k_B T}{2\pi^2} \int_{0}^{\infty} dk\, k^2\\
    &=
    \frac{8 k_B T \pi}{c^3} \int_{0}^{\infty} d\nu\, \nu^2
\end{align}
and if we define the Rayleigh-Jeans density
\begin{align}
    u_T(\nu) &= \frac{8 k_B T \pi}{c^3} \nu^2,
\end{align}
we find $u_T(\nu)$ is a quadratic.
At the time,
people did precise measurements of this thing.
I don't know how did this,
but I don't know how they measured the energy so precisely.
But they did it,
and then know that the correct curve was not quadratic.
They knew that it increased, peaked,
then decreased.

However,
they kind of agreed in the low-$\nu$ limit.
So they got something right.

And we know why.
It's because small frequencies corresponds to bulk quantum energies in the
regime
\begin{align}
    k_B T \gg \hbar \omega.
\end{align}
However, for large $\nu$,
this is violated,
and those high-frequency modes are actually frozen,
and we no longer satisfy the equipartition theorem.

That just tells you our story nowadays.

Let's take this Hamiltnoian here and don't treat it classically.
The same way that you have a particle moving on line,
you make the coordaintea nd mometa ot be operaotr,s
adn you know they have acertain commutation relation,
all that thing holds true,
except that you have an $a$ and $a^\dagger$
for every polarziation Foufier mode.
I'm not going to do this because we don't have the time,
but this is what we call \emph{quanutm field theory},
where you take a classical field,
apply quantum mechanicss to it.
the states of hte field are interpeted as particles,
like photons for example.
The particles ocrrespdongin to quantize leectromeatice field sare like phonons,
ecept hte lattice is the EM field.

To keep long story short,
quantum mechanically,
I'm going to have a Hamiltonian exactly like this,
except it doesn't change the fact I have a bunch of quantum harmonic
oscillators.

First of all,
let's find what the eigenstates of this Hamiltnoian are.
It could be that every harmonic oscilllator is in the ground sate,
the minimum amount of energy.
That's goingto have some energy.
Imagine if I take all of hte modes.

And I say only for that particular oscilators and iwht this poarlizatoin here,
I'm going to t tae the stateo f that oscillatora nd move it to the first excited
state.
Howmuch am I going to raise the energy?
It's going to be $\hbar\omega_k_1 = \hbar c k_1$.
I state like this,
we have created a photon of wavelength $2\pi/k_1$.
This is identical to what we have if we had a particle moving wit momentum
$\hbar k$.
This is the energy of the particle $cp_1$,
and rememver relativity,
this i the energy of a massless particle,
and htat's why we say it has zero mass.

What if I take that very same oscillator and I raise that energy level to the
second excited state?
Because the energy levels are equally spaced,
I would just get $2\habar\omega_{_k_1}$.

I could get two different oscillators,
and that would ocrrespond to the sum of theenrgies.
The fact hte nerg is just the sum of the enrgies of each one,
it means they are not interacting,
no potential enegy between them and that's it.

That's hte beginning of quantum electrodynamics,
let someone else teach you that.
But now I just want to compute the paritiotn function.

You've done this a million times.
Look at your notes.
After that we considered the case of phonos,
just a bunch of harmoinc oscilaotrs.
Here again,
we hve a bunchof harmoinc oscillators.
We're doing statistical mechanics of quantum field theory,
it's veyr fancy stuff.
Anyway,
what is the parittion function?

I have to take htis Hamilontina here,
put some hats on them and do the sum.
\begin{align}
    Z &=
    \Tr e^{-\beta H}\\
    &=
    \prod_{\vec{k},{\alpha}}
    \sum_{n_k=0}^{\infty}
    e^{-\beta \hbar \omega \left(n_{k} + \frac{1}{2}  \right)}\\
    &= \prod_{\vec{k},\alpha}
    \frac{e^{-\beta \omega_{\vec{k}}} \hbar/2}{%
    1 - e^{-\beta \omega_{\vec{k}}\hbar}}
\end{align}
where $\alpha=1,2$ are polarization indices.
To convince yourself it's right to take products,
\begin{align}
    \Tr e^{-\beta \left( H_1 + H_2 \right)} &=
    \Tr\left[
    e^{-\beta H_1} e^{-\beta H_2}
    \right]\\
    &=
    \sum_{n_1,n_2} \bra{n_1} e^{-\beta H_1} \ket{n_1}
    \bra{n_2} e^{-\beta H_2}\ket{n_2}
\end{align}
where $H_1$ and $H_2$ are independent.
Let me give you some advice.
Ther'es just so much more I could explain,
but I'm just feeling the time here.
Ifyo usee a large expression that doesn't have meaning to you hard to digest,
you can break it down,
by working out the trivial cases.
After you do 2,
everything becomes clear.
If you do this tomrrow,
and the day after,
and then it becomes eh,
not a big deal this infintie oscillaotrs,
it takes 30 years.
The alternative is ot manipulate formulas using rules no one told you,
and you'll feel all around.

And now we can calculate the Helmholtz free energy.
\begin{align}
    F(T, V) &=
    -k_B T \ln Z\\
    &=
    -k_B T \sum_{\vec{k},\alpha}\left[
    \ln\left( 1 - e^{-\beta\omega_k\hbar} \right)
    + \frac{\beta\hbar\omega_k}{2}
    \right]
\end{align}
By the way,
we invented QM becauseo f hte embarassing divergeence,
buti fy ou calcualte the ground stae of the infinite harmonic oscillaotrs,
you get $\hbar\omega/2$,
but how many ar ethere?
Infintiely many.
This theory predicts the enrgy density of hte vacuum is infinite.
That's bad right?
But that's hte predition.
We do't wrry about it,
is because if you add a photon,
you add energy,
so hte infinite amount of energy goes on both sides of the quation,
but we can ignore.
So delete that second term.
\begin{align}
    F(T, V) &=
    -k_B T \sum_{\vec{k},\alpha}\left[
    \ln\left( 1 - e^{-\beta\omega_k\hbar} \right)
    \right]
\end{align}

That is the energy by the way people on the internet want ot harness to move
spaceships or someting.
The rest is just like phonons.
The dispersion is not linear forever.
Here htere is no maximum value of $\vec{k}$.
Hopefuly thisp art that is temperature dependent.

There's one step I'm going to do that I'm going to defy detail.
I want ot transform this sum into an integral,
by an approxiamtion,
and when I do this,
I claim the answer is.
\begin{align}
    F(T, V) &=
    2k_B T V \int d^3k\, \ln\left( 1 - e^{-\beta \hbar ck} \right)
\end{align}
And this $2$ comes from the polarizations.
The free energy is an extensive quantity,
so you should expect it to be proportional to volume,
which makes sense.

Then we're going to move to the next topic of quantum gases next class.
Monday we have office hours.


\section{Online learning approach}
Most of the work in theoretical machine learning has been focused on an off-line setting, where one uses a batch of data to produce a predictive model and then apply it to new data. However, this paper focuses on an on-line setting, where predictions are made sequentially based on all previously processed data, which gives rise to the name itself as we process observations one by one. In this setting we start by observing the first example $x_1$ while predicting its label $y_1$. After that we observe the true label of $y_1$ and also the next data point $x_2$. This process goes on until we either reach the end of our data set or indefinitely. Foremost attention can be drawn to the expectation of improved prediction quality with increasing number of processed data.

Such supervised learning approach perfectly suits the case when data is available sequentially with time, for example, in predicting a stock price given its historic data and supplemented by availability of true price, as compared against the prediction of a learning system. Furthermore, as compared to off-line setting, on-line algorithm is expected to show a drastic decrease in memory usage and processing time required until the first prediction is made.

More formally, a mathematical description of the learning problem is defined by finding a minimum of an expected risk:
$$E(f)=\int l(f(x),y)dP(z)$$
where function $f$ in a family of functions $\mathcal{F}$ parameterized by weight vector $w$ minimizes the averaged loss $Q(z,w)=l(f_w(x),y)$ over all data points $z$. Ultimately, we'd prefer to average over the unknown distribution $dP(z)$, however, in practical setting we perform the calculation over a fixed number of examples $z_1, z_2,...z_n$, which is referred to as minimization of an empirical risk:
$$E_n(f)=\frac{1}{n} \sum\limits_{i=1}^{n}l(f(x_i),y_i)$$

Let us further consider how one could tackle such minimization task with some of the convergence results (Bottou, 2010 \cite{bottou-2010}).

\subsection{Gradient Descent Methods}
One of the most common approaches to address minimization of empirical risk function is by applying an iterative gradient descent method(GD) where each iteration $t$ updates optimization weights $w$ based on the gradient of $E_n(f_w)$ with learning rate $\gamma$:
$$w_{t+1}=w_t-\gamma \frac{1}{n} \sum\limits_{i=1}^{n} \nabla_w Q(z_i, w_t)$$
When the initial estimate of the optimum $w_0$ is sufficiently close and under certain regularity assumptions this method reaches linear convergence, described by a residual error $\rho$, that is $-\log\rho \sim t$.

A better optimization algorithm can be composed by replacing the learning rate with a positive-definite matrix $\Gamma_t$ which approaches inverse of the Hessian of the cost at optimum:
$$w_{t+1}=w_t-\Gamma_t \frac{1}{n} \sum\limits_{i=1}^{n} \nabla_w Q(z_i, w_t)$$
This is known as a 2nd order gradient descent method(2GD) and is a variant of well known Newton algorithm. Under analogous to GD assumptions of initial estimate and regularity this algorithm reaches quadratic convergence $-\log\log\rho \sim t$.

\subsection{Stochastic Gradient Descent Methods}
Stochastic setting of GD slightly simplifies the process, instead of computing the gradient of empirical loss function $E_n(f_w)$ exactly one uses a single randomly picked data point $z_t$ for its estimation:
$$w_{t+1}=w_t-\gamma \nabla_w Q(z_t, w_t)$$
In this simplification we assume that our optimization parameter behaves as a stochastic process ${w_t, t=1,...}$, which greatly depends on the data sampling and introduces certain amount of noise. Utilization of stochastic approximation framework (Bottou, 1998 \cite{bottou-98x}) fully addresses proofs of convergence including cases where loss function is not everywhere differentiable. Results of the aforementioned analysis show that speed of SGD convergence is limited by noisy approximation of the true gradient and is in fact optimal with learning rate $\gamma_t \sim t^{-1}$, at which point expectation of the residual error decreases similarly $E\rho \sim t^{-1}$.

Alike 2GD, one can multiply gradients by positive-definite matrix approaching inverse of the Hessian:
$$w_{t+1}=w_t-\gamma_t\Gamma_t \nabla_w Q(z_t, w_t)$$
which yields the 2nd order stochastic gradient descent method. However, such change doesn't decrease the bias introduced by stochastic noise and retains the convergence rate $E\rho \sim t^{-1}$.

\subsection{Sample SGD pseudocode}
A simplistic pseudocode of SGD with in-place optimization parameter update may look as follows:
\lstinputlisting[frame=single]{sgd.py}

\section{SGD in large-scale setting}
Following results for asymptotic analysis of SGD (Bottou, 2010 \cite{bottou-2010}) as summarized in \figurename{1}, where $\rho$ is the residual and excess error is defined as
$$\mathcal{E} = \mathcal{E}_{app} + \mathcal{E}_{est} + \mathcal{E}_{opt} \sim \mathcal{E}_{app} + (\frac{\log n}{n})^\alpha + \rho, \alpha \in [1/2, 1]$$
with $\mathcal{E}_{app}$ measuring how closely functions in our function space can approximate the optimal solution, $\mathcal{E}_{est}$ measuring the effect of minimizing empirical risk instead of the expected risk, $\mathcal{E}_{opp}$ measuring the impact of approximate optimization on expected risk,

\begin{figure}[h]
	\begin{center}
		\includegraphics[scale=0.3]{img/gd-convergence.png}
		\caption{Asymptotic results for various optimization methods}\label{1}
	\end{center}
\end{figure}

%\begin{table}[h]
%	\begin{center}
%		\caption{Asymptotic results for various optimization methods}\label{1}%
%		\begin{tabular}{lcccc}
%			& GD & 2GD & SGD & 2SGD\\
%			\hline
%			Time per iteration: 				& $n$ & $n$ & 1 & 1\\
%			Iterations to accuracy $\rho$: 		& $\log \frac{1}{\rho}$ & $\log \log \frac{1}{\rho}$ & $\frac{1}{\rho}$ & $ %\frac{1}{\rho}$\\
%			Time to accuracy $\rho$:			& $n\log \frac{1}{\rho}$ & $n\log \log \frac{1}{\rho}$ & $\frac{1}{\rho}$ & $\frac{1}{\rho}$\\
%			Time to excess error $\mathcal{E}$: & $\frac{1}{\mathcal{E}^{1/\alpha}} \log \frac{1}{\mathcal{E}}$ & $\frac{1}{\mathcal{E}^{1/\alpha}} \log \frac{1}{\mathcal{E}}\log\log \frac{1}{\mathcal{E}}$ & $\frac{1}{\mathcal{E}}$ & $\frac{1}{\mathcal{E}}$\\
%		\end{tabular}
%	\end{center}
%\end{table}

one can conclude that whilst SGD and 2SGD are worst optimization algorithms (as depicted in third row of \figurename{1}), they require less time to reach predefined expected risk (as in fourth row of \figurename{1}). In other words, in a large-scale setting where an algorithm is restricted by its computing time rather then the amount of data provided, stochastic gradient methods perform asymptotically better.

\section{Parallelization of SGD}
Unlike batch GD, SGD has inherently sequential nature, which essentially limits how one can parallelize the algorithm. However, pressing industry demands have led researchers to find feasible ways of achieving comparable convergence rates, as compared against its sequential counterpart, medium to high scalability on multiple compute units, which not seldom includes both GPUs and CPUs, while keeping computational complexity and data transfer low. We will further describe several parallelization ideas and their pitfalls (Zinkevich, Weimer, Li and Smola, 2010 \cite{NIPS2010_4006}), as compared to a hybrid method briefly depicted later in this paper.

\subsection{Distributed subgradient approach}
This approach essentially distributes parallelization of gradient computation among several machines, which hold part of the data, and subsequent aggregation of the result on a master machine. While showing great linear scalability relative to amount of data and log-linear relative to number of computers, this approach suffers from excessive communication overhead which results from multiple passes through data by MapReduce. In addition, since several MapReduce iterations are required to ensure fault tolerance, this approach generally doesn't fit into a setting where computation units are not closely coupled, as in multiple machines compared to multicore approach.

\subsection{Distributed convex solver approach}
This approach is comprised of a relatively simple idea of performing a minibatch optimization by means of breaking down a small subset of the data set into several mini-batches and then solving each and every problem on a separate machine for further aggregation of the solutions and averaging obtained values. Most notable part of this idea is that MapReduce needs to perform only one pass, which dramatically reduces overall communication between machines. Despite the fact that this approach significantly reduces variance relative to the sequential counterpart, the bias introduced by stochastic nature of the algorithm doesn't reduce at all, as compared to sequential version. Moreover, this approach requires a batch-solver to run on every single machine, thus making the whole algorithm rather computationally complex. More importantly, analysis of given approach showed that the convergence of the method greatly depends on degree of strong convexity of regularization.

\subsection{Distributed SGD approach}
A balance between MapReduce communication overhead and valid asymptotic analysis can be achieved by modifying the distributed convex solver approach to incorporate an SGD minimizer. More precisely, each processor would carry out an SGD on the set of loss functions $Q_i(w)$ with a fixed learning rate $\gamma$ for $T$ steps as described in Algorithm 1.

\begin{table}[h]
	\begin{flushleft}
		\begin{tabular}{l}
			Algorithm 1: $SGD({Q_1, \dots, Q_m}, T, \gamma, w_0)$\\
			\hline
			for $t = 1$ to $T$ do\\
			\enskip\enskip	Draw $j \in (1 \dots m)$ uniformly at random.\\
			\enskip\enskip	$w_{t+1}=w_t-\gamma*dQ_j(w_t)$\\
			end for\\
			return $w_T$.\\
		\end{tabular}
	\end{flushleft}
\end{table}

To aggregate computed parameters one would use a master routine. Full procedure pseudocode is presented in Algorithm 2.

\begin{table}[h]
	\begin{flushleft}
		\begin{tabular}{l}
			Algorithm 2: SimuParallelSGD(Examples: ${Q_1, \dots, Q_m}$, $\gamma$, Machines: $k$)\\
			\hline
			Define $T=[m/k]$\\
			Randomly partition the examples, giving $T$ examples to each machine\\
			for all $i \in {1 \dots, k}$ parallel do\\
			\enskip\enskip Randomly shuffle the data on machine $i$\\
			\enskip\enskip Initialize $w_{i,0}=0$\\
			\enskip\enskip for all $t \in {1, \dots, T}$: do\\
			\enskip\enskip\enskip\enskip Gather the $t$th example on the $i$th machine, $Q_{i,t}$\\
			\enskip\enskip\enskip\enskip $w_{i,t+1} = w_{i,t}-\gamma d_w Q_i(w_{i,t})$\\
			\enskip\enskip\enskip\enskip end for\\
			\enskip\enskip end for\\
			Agregate from all computers $v=\frac{1}{k} \sum\limits_{i=1}^{k} w_{i,t+1}$ and return $v$\\
		\end{tabular}
	\end{flushleft}
\end{table}

While this approach in a nutshell is rather simple, its mathematical analysis is nontrivial and can be found in the original work (Zinkevich, Weimer, Li and Smola, 2010 \cite{NIPS2010_4006}).

\subsection{Empirical results of distributed SGD approach}
To demonstrate scalability potential of the algorithm we can take a look at obtained experimental results (Zinkevich, Weimer, Li and Smola, 2010 \cite{NIPS2010_4006}) where number of training instances per machine was compared to a relative root-mean square error(RMSE) on the test set as executed on 1, 10 and 100 machines as in \figurename{2} and also to a normalized objective function as in \figurename{3}. All tests were performed on a $\sim$3M examples database of emails with $\sim$785M features in the whole data set, as resulted after hashing.

\begin{figure}[h]
	\begin{center}
		\includegraphics[scale=0.3]{img/parallel-sgd-experiment.png}
		\caption{Relative Test-RMSE with $\gamma= 1e^{-3}$. Depicts decreasing training time for various number of machines.}\label{2}
	\end{center}
\end{figure}

\begin{figure}[h]
	\begin{center}
		\includegraphics[scale=0.3]{img/parallel-sgd-experiment2.png}
		\caption{Relative train error using Huber loss $\gamma= 1e^{-3}$. Depicts how fast an algorithm reaches certain model quality for various number of machines.}\label{3}
	\end{center}
\end{figure}

It is worth mentioning that scaling of 1 to 10 machines gives much better improvement compared to the one from 10 to 100 machines, which signifies that the proposed algorithm scales poorly in a highly distributed scenario.

\section{Asynchronous SGD}
While provinding good results, the distributed SGD approach, described in the previous section, has several drawbacks. Namely, it suffers from parameter locking, which happens when parameters $w_i$ are being averaged. This prevents further processing of data by slave SGD machines, since they need to read current parameter for further processing. Moreover, even though the use of MapReduce in this setting does not result in high communication costs, in general it is recommended to avoid its usage in a large-scale numerically intensive applications, due to its inability to cope with iterative nature of solvers and also due to its bandwidth overhead, which results from fault tolerance redundancies. For example, one can experience a potential drop from 12GB/s throughput in a multicore shared memory to only tens of MB/s. Thus we advocate two approaches which do not use MapReduce and perform asynchronous parameter updates as follows.

\subsection{Hogwild!}
In order to eliminate the overhead caused by parameter variable locking, presented algorithm incorporates a simple idea of using high throughput memory shared across all compute units where optimization parameter $w$ is to be stored. Any processor can update the parameter at any given time. Even though such approach might seem excessively prone to parameter overwrites, it works extremely well in case where data access is sparse, meaning that each SGD modifies only a small part of the optimization parameter. Particularly, when our goal is to minimize a loss function
$$f(w)=\sum\limits_{e \in E} f_e(w_e)$$
where $e$ denotes a small subset of ${1, \dots, n}$ and $w$ denotes values of parameter vector $w$ indexed by $e$, the sparsity of loss function means that $|E|$ and $n$ are very large while each individual $f_e$ influences only few values of the whole parameter $w$. This concept is illustrated in more detail with precise mathematical examples of particular cost functions in the original work (Recht, Re, Wright and Niu, 2011 \cite{NIPS2011_4390}).

Hogwild! pseudocode running on each processor would look as described in Algorithm 3.

\begin{table}[h]
	\begin{flushleft}
		\begin{tabular}{l}
			Algorithm 3: Hogwild! update for individual processors\\
			\hline
			loop
			\enskip\enskip Sample $e$ uniformly at random from $E$\\
			\enskip\enskip Read current state $w_e$ and evaluate $G_e(x)$\\
			\enskip\enskip for $v \in e$ do $w_v = w_v - \gamma b^T_v G_e(w)$\\
			end loop\\
		\end{tabular}
	\end{flushleft}
\end{table}

where $G=(V,E)$ is a hypergraph induced by $f(w)$ whose nodes are the individual components of $w$ and where each subvector $w_e$ induces an edge in the graph $e \in E$ and $b_v$ is equal to 1 on the $v$th component and 0 otherwise. It is important to note that the processor modifies only values in $e$ leaving all others untouched.  Such approach does not require a locking mechanism on most modern hardware, such as GPUs or general purpose multicore CPUs.

Analytical results for convergence of the algorithm are rather non-trivial and can be found with exhaustive explanations in the original paper (Recht, Re, Wright and Niu, 2011 \cite{NIPS2011_4390}). Important to note, however, that the algorithm converges in nearly the same number of iterations as its sequential counterpart and provided experimental results outperformed theoretical analysis. In terms of scalability it reaches near linear speedup. Some of the experimental results as compared to almost identical rooundrobin implementation with exception of gradient updates can be found in \figurename{4}. Those results present a comparison of wall clock time while parallelized over 10 cores, both algorithms were implemented in C++ and were ran on the same hardware setup with a dual Xeon X650 CPUs (6 cores each x 2 hyperthreading) and 24GB of RAM.

\begin{figure}[h]
	\begin{center}
		\includegraphics[scale=0.25]{img/hogwild-res.png}
		\caption{Comparison of wall clock for various data sets and loss functions for Hogwild! and Roundrobin implementations}\label{4}
	\end{center}
\end{figure}

\subsection{Downpour SGD}
Unlike previous examples, this parallelization approach was used to train a deep neural network with image recognition research project \cite{NIPS2012_4687} and was developed as part of DistBelief framework at Google. However, the ideas used and results obtained are still worth mentioning as they give birth to potential further work.

In a nutshell the approach is as follows: the training data is divided into a number of subsets and a copy of the model is ran on each of those subsets. This approach leverages the idea of Hogwild! asynchronous parameter updates in the form of centralized parameter server which is connected to several model replicas (10 in the example) each holding subset of the parameter $w$ (1/10 in the example). This is schematically presented in \figurename{5}. Thus, after the model is ran on the assigned subset of training data, computed gradients are communicated to the central parameter server.

\begin{figure}[h]
	\begin{center}
		\includegraphics[scale=0.4]{img/downpour-sgd-param.png}
		\caption{Model replicas asynchronously fetch parameters $w$ and push gradients $\nabla w$ to the parameter server}\label{5}
	\end{center}
\end{figure}

Downpour SGD approach is more robust against machine failures than synchronous parallelized SGD, because in case of machine failure in synchronous SGD the entire training process is delayed. However, in asynchronous SGD when a model replica fails the other model replicas continue to operate and process their updates via parameter server.

In addition, Downpour SGD uses AdaGrad \cite{adagrad} adaptive learning rates which increase robustness of the whole implementation and tackle problems of stability within deep neural network context. Corresponding learning rate is as follows:
$$ \eta_{i,K}=\gamma / \sqrt{\sum\limits_{j=1}^{K}} \nabla w_{i,j}^2$$
One can notice that the learning rates $\eta_{i,K}$ are computed from the summed squared gradients of parameter part and thus can be easily implemented locally within each model replica.

Highly scalable nature of the algorithm can be seen as depicted in \figurename{6}.
\begin{figure}[h]
	\begin{center}
		\includegraphics[scale=0.4]{img/downpour-sgd.png}
		\caption{Training speed-up for four different deep networks as a function of machines allocated to a single model instance}\label{6}
	\end{center}
\end{figure}


\documentclass{article}
\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{graphicx}

\newtheorem{theorem}{Theorem}[section]
\renewcommand{\implies}{\Rightarrow}
\begin{document}

\title{Rational approximations of real numbers}

\section{Introduction}

Our goal is to prove that we can approximate any positive real number arbitrarily closely
with $\frac{2^a}{3^b}$ with positive integers $a,b$. This surprising result will bring us
into the world of infinite and infinitessimal analysis.

This will be a quick tour around some of the wonders of infinity in analysis, and how to
generate arbitrarily accurate rational approximations for any real number.

\section{Continued fractions}

One way of generating arbitrarily close approximations of irrational numbers is by taking
the partial terms (called "convergents") of the continued fraction expression. Generating
a continued fraction for $x$ involves taking the floor of $x$, subtracting it, taking the
floor of the inverse, and repeating the operation to infinity.

An example will help to illustrate. The continued fraction for $\sqrt{2} = 1.414\cdots$ has
a leading term of 1 (the whole number part of $\sqrt{2}$) and a remainder of $\sqrt{2}-1$.
Inverting the remainder, we find that it is $\frac{1}{\sqrt{2}-1} = \sqrt{2}+1$ - giving
our next term to be $\lfloor 1+\sqrt{2}\rfloor = 2$, and a remainder of $\sqrt{2}-1$.
We can repeat this process ad infinitem to generate the continued fraction
$\left[1;2,2,2,\cdots\right]$.

\textbf{Exercise:} Prove that the continued fraction of $\sqrt{n^2+1}$ in general is
$\left[n;2n,2n,2n,\cdots\right]$.

We can now generate a convergent set of rational approximations to $\sqrt{2}$ which are
guaranteed to be the best possible rational approximations with a given denominator (or
smaller) by taking the partial sums:
\[ 1+\frac{1}{2} = \frac{3}{2}\]
\[ 1+\frac{1}{2+\frac{1}{2}} = 1+\frac{2}{5} = \frac{7}{5} \]
\[ 1+\frac{1}{2+\frac{1}{2+\frac{1}{2}}} = 1+\frac{5}{12} = \frac{17}{12} \]
\[ 1+\frac{1}{2+\frac{1}{2+\frac{1}{2 + \frac{1}{2}}}} = 1+\frac{12}{29} = \frac{41}{29} \]

In general, if the $n$th approximation is $\frac{p}{q}$, the $(n+1)$th approximation is
$\frac{p+2q}{p+q}$.

\textbf{Exercise:} Prove that this sequence converges to $\sqrt{2}$. How does it converge?
Is it monotonic decreasing, or does it oscillate around the eventual limit? Does this
converge to $\sqrt{2}$ if we start with a different starting point for $\frac{p}{q}$?

Larger numbers in a continued fraction representation imply faster convergence, smaller
numbers mean slower convergence. The slowest converging continued fraction is thus:
\[\left[1;1,1,1,\cdots\right] = 1+\frac{1}{\left[1;1,1,1,\cdots\right]}\]

We can evaluate this exactly using the quadratic equation:
\[ x = 1+\frac{1}{x} \]
or $x = \varphi = \frac{1+\sqrt{5}}{2}$, the golden ratio.

The start continued fraction for $\pi$ is
$\left[3;7,15,1,292,1,1,1,2,1,3,1,14,2,1,1,2,2,2,2,\cdots\right]$ and its first few
convergents are the well known approximations of $\pi: 3, \frac{22}{7}, \frac{333}{106}, \frac{355}{113}$.
Since the next term is 292, we know that this last approximating is extremely close to the actual
value of $\pi$.

\section{Finding general rational approximations}

When given a number $x$, one naive strategy for finding a rational approximation is to multiply
it by a positive integer $n$, take the whole number closest to the result, and call $\frac{m}{n}$ our
estimate. This approach works, and will converge to $x$ over time, but it is far from efficient. For
example, $\frac{22}{7}$ is a much better estimate of $\pi$ than $\frac{31}{10}$ - by increasing the
denominator to 10, we have made our estimate a lot worse.

We can do better by using an approach described by Dirichlet. Dirichlet's approximation theorem
says that if $x$ is irrational, that we can always find $a,b \in \mathbb{Z}$ such that
$|ax-b|<\frac{1}{n}$ for any $n\in \mathbb{N}$. To prove this, consider $\{ax\} = ax -
\lfloor ax \rfloor$ for all $a < n$. $0<\{ax\}<1$ for all $a$, by definition. If we partition the
interval $[0,1)$ into $n-1$ buckets $[0,\frac{1}{n}) \cup [\frac{1}{n},\frac{2}{n}) \cup \cdots
\cup [\frac{n-1}{n}, 1)$, we are guaranteed that of the $n+1$ values $\{a_i\}_{i=0}^n$, at
least 2 will land in the same bucket, by the Pigeonhole principle. If these two values are $a_i, a_j$,
then $a_i = \{ix\} = ix - \lfloor ix \rfloor, a_j = \{jx\} = jx - \lfloor jx \rfloor$, and

\[ \frac{1}{n} > |\{ix\}-\{jx\}| = | (ix-\lfloor ix \rfloor) - (jx - \lfloor jx \rfloor) | \]
\[ \frac{1}{n} > | (i-j)x  + b | = | ax+b | \]

for an integer $b = |\lfloor ix \rfloor - \lfloor jx \rfloor|$. What's more, regardless of how close
the estimate is for $n$, we can find a better estimate by choosing $n > |\frac{1}{ax-b}|$, when
$\frac{1}{n}$ is smaller than the smallest fractional difference for $a<n$.

This is a bit abstract, so let's work an example through to see how this works in practice. Let's try
to approximate $\pi$ using this method with increasing values of $a$. With $a=10$, for example:

\begin{center}
	\begin{tabular}{|c|c|}
		\hline
		$i$ & $\{i\pi\}$ \\
		\hline
		$0$ & $0$ \\
		$1$ & $0.14159$ \\
		$2$ & $0.28318$ \\
		$3$ & $0.42277$ \\
		$4$ & $0.56637$ \\
		$5$ & $0.70796$ \\
		$6$ & $0.84955$ \\
		$7$ & $0.99114$ \\
		$8$ & $0.13274$ \\
		$9$ & $0.27433$ \\
		$10$ & $0.41592$ \\
		\hline
	\end{tabular}
\end{center}

Now, since we have 10 buckets, representing the $\frac{1}{10}$ths, we are guaranteed that at
least two of these will be in the same bucket (in fact, we see that many pairs are in the same
bucket). Looking down the list at the $\frac{1}{10}$ths digit, we can easily see that the
entries corresponding to $i=1, i=8$, $i=2, i=9$, and $i=3, i=10$ all land in the same buckets.
In other words:
\[ \frac{1}{10}> |\{8\pi\}-\{1\pi\}| = | 8\pi - 25 - (1\pi - 3)| = |7\pi -22| \]
which gives us the well-known estimate for $\pi$ of $\pi \approx \frac{22}{7}$.

Given a good estimate $\frac{a}{b}$ for $x$, we can guarantee to find a better estimate by
considering $|bx-a| = y$. Again, considering $\pi$: $|7\pi - 22| \approx 0.00885 >
\frac{1}{113}$, so we are guaranteed to be able to find a more accurate rational estimate by setting
$a \geq 113$.

This iterative nature of always being able to find a closer approximation by choosing a sufficiently
large value for $n$, is what enables arbitrary approximation of any rational number.

\section{A Pell equation approach}

\textbf{Question:} Source: The Riddler from FiveThirtyEight:
https://fivethirtyeight.com/features/can-you-find-an-extra-perfect-square/

For some perfect squares, when you remove the last digit, the result is also a perfect square. The
first few squares for which this happens are 16, 49, 169, 256, and 361. Can you find the next 3
numbers for which this is true? How many such numbers are there?

For extra credit, note that $(169,16,1)$ is a triple with this property. Are
there other similar triples?

If we write the larger number as $a^2$, and the smaller number as $b^2$, we get that:
\[ 0 \leq a^2 - 10b^2 < 10 \]

And by looking at $n^2 \pmod{10}$, we can see that $a^2 - 10b^2 \in \{0,1,4,5,6,9\}$.
In addition, if $a^2-10b^2 = 5$ that means that $a=10k+5, a^2 = 100(k^2+k) + 25$, so the last
digit of $b^2$ would have to be 2, which is not an option. And if $a^2-10b^2 = 0$, that means that:
$\frac{a^2}{b^2} = 10 \implies \frac{a}{b} = \sqrt{10}$ - but since $\sqrt{n}$ is irrational unless
$n$ is a perfect square, that is not a possibility either. Thus, we need to find the solutions to
$a^2 - 10b^2 \in \{1,4,6,9\}$.

Let's look first at $a^2-10b^2 = 1$.  We can factor this as a difference of two squares:
\[ a^2 - 10b^2 = (a-\sqrt{10}b)(a+\sqrt{10}b) = 1 \]

Additionally, we can show that if $a,b \in \mathbb{Z}$, then:
\[ (a-\sqrt{10}b)^n = A - \sqrt{10}B \]
\[ (a+\sqrt{10}b)^n = A + \sqrt{10}B \]
for some $A,B \in \mathbb{Z}$ using the binomial expansion. For $n=2$, we get $A=a^2+10b^2, B=2ab$.

From the question, we already have the smallest solution to this equation: $19^2 - 10(6^2) = 1$ -
and from this we can generate as many solutions as we want by using different values of $n$. Indeed:
\[A = 19^2 + 10\cdot 6^2 = 361 + 360 = 721,  B = 2 \cdot 19 \cdot 6 = 228 \]
\[ 721^2 = 51984, 228^2 = 51984, 721^2 - 10\cdot 228^2 = 1 \]

By defining an appropriate norm on numbers in this set (called formally $\mathbb{Z}[\sqrt{10}]$),
we can also find all solutions satisfying the question. Define the norm as:
\[ N(a+\sqrt{10}b) = (a+\sqrt{10}b)(a-\sqrt{10}b) = a^2-10b^2 \]
Then:
\begin{align*}
	N((a+\sqrt{10}b)(c+\sqrt{10}d)) &= N((ac+10bd) + \sqrt{10}(bc + ad)) \\
	&= (ac+10bd)^2-10(bc+ad)^2 \\
	&= a^2c^2 - 10b^2c^2 - 10a^2b^2 + 100b^2d^2 \\
	N(a+\sqrt{10}b)N(c+\sqrt{10}d) &= (a^2-10b^2)(c^2-10d^2) \\
	&= a^2c^2 -10b^2c^2 -10 a^2d^2 +100b^2d^2 \\
	&= N((a+\sqrt{10}b)(c+\sqrt{10}d))
\end{align*}

So for $\mathbb{Z}[\sqrt{10}]$, if we have a number with a norm $N(a+\sqrt{10}b) = 4$ then we can
generate infinitely many others with the same norm by multiplying by a number that has a norm of 1
(that is, by any number of the form $(19+6\sqrt{10})^n$). Since $N(2 + 0\sqrt{10}) = 4$, any number
of the form $2(19+6\sqrt{10})^n$ will have $a^2-10b^2 = 4$.

For $a^2 - 10b^2 = 6$, we can similarly generate solutions of the form $(4+\sqrt{10})(19+6\sqrt{10})^n$
and $(16+5\sqrt{10})(19+6\sqrt{10})^n$. And finally, for $a^2 - 10b^2 = 9$, there are several numbers with
$N(a+\sqrt{10}b) = 9$: $3, 7+2\sqrt{10}, 13 + 4\sqrt{10}$, and so we can generate solutions for all of
$3(19 + 6\sqrt{10})^n, (7 + 2\sqrt{10})(19 + 6\sqrt{10})^n, (13 + 4\sqrt{10})(19 + 6\sqrt{10})^n$.

It's worth noting that for all solutions of the form $a^2-10b^2 = 1$:
\[ \frac{a}{b} = \sqrt{10 + \frac{1}{b^2}} \approx \sqrt{10} \]

so the solutions to the question above are also increasingly accurate rational approximations of
$\sqrt{10}$. As we saw in the section on continued fractions, $\sqrt{10} = [3;6,6,6,\cdots]$ - so we
also can create solutions with convergents of the continued fraction above (taken after an
even number of terms, because it oscillates around $\sqrt{10}$). The first convergent,
$\frac{19}{6}$, corresponds to the smallest solution of $a^2-10b^2=1$. The next convergents are
\[ 3 + \frac{1}{6 + \frac{1}{6 + \frac{1}{6}}} = \frac{721}{228} \]
\[ 3 + \frac{1}{6 + \frac{1}{6 + \frac{1}{6 + \frac{1}{6 + \frac{1}{6}}}}} = \frac{27379}{8658} \]
which correspond to the next two solutions to $a^2-10b^2=1$.

For the extra credit: if $0<a^2-10b^2 <10, 0<10b^2 - (10c)^2 \leq 90$, then adding these,

\[0 < a^2 - (10c)^2 = (a-10c)(a+10c) < 100 \]

and $a > 10c$, giving us:

\[ 20c < a+10c < (a+10c)(a-10c) = a^2 - (10c)^2 < 100 \]
Which means $c < 5$, and $a^2$ must begin with $1,4,9,16$. The only option for $b$ with this
constraint is $169$, and there are no other solutions.

\section{Approximating any positive real number with $\frac{2^a}{3^b}, a,b \in \mathbb{Z}^+$}

Given $\delta>0, x \in \mathbb{R}^+$, we can find $a,b \in \mathbb{Z}^+$ such that:
\[ \left|x - \frac{2^a}{3^b} \right| < \delta \]

Consider:
\begin{align*}
	y &= \frac{2^a}{3^b} \\
	\ln(y) &= a\ln(2) - b\ln(3) \\
	\log_3(y) &= a\log_3(2) - b
\end{align*}

Since $\log_3(2)$ is irrational, we can use Dirichelet's approximation theorem to find
$a,b \in \mathbb{Z}$ such that:
\[ \left| a\log_3(2) - b \right| < \frac{1}{n} \]
for any $n \in \mathbb{N}$.

\end{document}


%
% Chapter 5.3
%

\section*{5.3 Volumes by Cylindrical Shells}

\subsubsection*{Formula for the Volume of a Cylindrical Shell}

\[ \text{V = [circumference][height][thickness]} \quad \Leftrightarrow \quad V = 2 \pi rh \Delta r \]


\section{Fibrations and cofibrations}
\subsection{Comparing fibers over different points}
Let $p:E\to B$ be a fibration.
Above, we saw that this implies that paths in $B$ ``lift'' to paths in $E$.
Let us consider a path $\omega:I\to B$ with $\omega(0) = a$ and $\omega(1) = b$.
Denote by $F_a$ the fiber over $a$.
If the world plays fairly, the path lifting property of fibrations should beget a (unique\footnote{At least up to homotopy.})
map $F_a\to F_b$.
The goal of this subsection is to construct such a map.

Consider the diagram:
\begin{equation*}
    \xymatrix{
	F_a\ar[d]_{\mathrm{in}_0}\ar[rr] & & E\ar[d]^p\\
	I\times F_a\ar@{-->}[urr]^{h}\ar[r]_{\mathrm{pr}_1} & I\ar[r]_{\omega} & B.
    }
\end{equation*}
This commutes since $\omega(0) = a$.
Utilizing the homotopy lifting property, there is a dotted arrow that makes the entire diagram commute.
If $x\in F_a$, the image $h(1,x)$ is in $F_b$, and $h(0,x) = x$.
This supplies us with a map $f:F_a\to F_b$, given by $f(x) = h(1,x)$.

We're now faced with a natural question: is $f$ unique up to homotopy?
Namely: if we have two homotopic paths $\omega_0,\omega_1$ with $\omega_0(0) = \omega_1(0) = a$,
and $\omega_0(1) = \omega_1(1) = b$, along with a given homotopy $g:I\times I\to B$ between $\omega_0$ and $\omega_1$,
such that $f_0,f_1:F_a \to F_b$ are the associated maps (defined by $h_0(1,x)$ and $h_1(1,x)$),
respectively, are $f_0$ and $f_1$ homotopic?

We have a diagram of the form:
\begin{equation*}
    \xymatrix{
	((\partial I\times I)\cup (I\times \{0\}))\times F_a\ar[d]_{\mathrm{in}_0}\ar[rr] & & E\ar[d]^p\\
	I\times I\times F_a\ar[r]_{\mathrm{pr}_1}\ar@{-->}[urr] & I\times I\ar[r]_g & B
    }
\end{equation*}
To get a homotopy between $f_0$ and $f_1$, we need the dotted arrow to exist.

% Add a picture, maybe?
%
%Think of the space $(\partial I\times I)\cup (I\times 0)$ as follows:
%\begin{equation*}
%\begin{tikzpicture}
%    \draw (2,2) -- (0,2) -- (0,0) -- (2,0);
%    \node [above] at (1,2) {$h_1$};
%    \node [below] at (1,0) {$h_0$};
%    \node [left] at (0,1) {$in_0$};
%\end{tikzpicture}
%\end{equation*}
%and $I\times I$ looks like:
%\begin{equation*}
%    \begin{tikzpicture}
%	\draw (0,0) -- (0,2) -- (2,2) -- (2,0) -- (0,0);
%	\draw[fill] (2,2) circle [radius=0.05];
%	\draw[fill] (2,0) circle [radius=0.05];
%	\node [left] at (0,1) {$a$};
%	\node [above] at (1,2) {$\omega_1$};
%	\node [below] at (1,0) {$\omega_0$};
%	\node [right] at (2,1) {$b$};
%	\node [above right] at (2,2) {$f_1$};
%	\node [below right] at (2,0) {$f_0$};
%    \end{tikzpicture}
%\end{equation*}
%
%Does the dotted map exist? Clearly this is crying out for us to use the homotopy lifting property. But what should our space $W$ be? Well the following pair:
%\begin{equation*}
%    \begin{tikzpicture}
%	\draw (2,2) -- (0,2) -- (0,0) -- (2,0);
%    \end{tikzpicture}\subseteq
%    \begin{tikzpicture}
%	\draw (0,0) -- (0,2) -- (2,2) -- (2,0) -- (0,0);
%    \end{tikzpicture}
%\end{equation*}
%is homotopy equivalent to $(0\subseteq I)\times I$. Hence in our diagram we now have:

It's an easy exercise to recognize that our diagram is equivalent to the following.
\begin{equation*}
    \xymatrix{
	I\times F_a\ar[rr]^{\simeq} & & ((\partial I\times I)\cup (I\times 0))\times F_a\ar[d]_{\mathrm{in}_0}\ar[rr] & & E\ar[d]^p\\
	I\times I\times F_a\ar[rr]_{\simeq} & & I\times I\times F_a\ar[r]_{\mathrm{pr}_1}\ar@{-->}[urr] & I\times I\ar[r]_g & B
    }
\end{equation*}
Letting $W=I\times F_a$ in the definition of a fibration (Definition \ref{fibration}) thus gives us the desired lift,
i.e., a homotopy $f_0\simeq f_1$.

We can express the uniqueness (up to homotopy) of lifts of homotopic paths in a functorial fashion.
To do so, we must introduce the fundamental groupoid of a space.
\begin{definition}
    Let $X$ be a topological space.
    The \emph{fundamental groupoid} $\Pi_1(X)$ of $X$ is a category (in fact, groupoid),
    whose objects are the points of $X$, and maps are homotopy classes of paths in $X$.
    The composition of compatible paths $\sigma$ and $\omega$ is defined by:
    \begin{equation*}
	(\sigma\cdot\omega)(t) = \begin{cases}
	    \omega(2t) & 0\leq t\leq 1/2\\
	    \sigma(2t - 1) & 1/2\leq t\leq 1.
	\end{cases}
    \end{equation*}
\end{definition}
The results of the previous sections can be succinctly summarized in the following neat statement.
\begin{prop}
    Any fibration $p:E\to B$ gives a functor $\Pi_1(B)\to \Top$.
\end{prop}
This is the beginning of a beautiful story involving fibrations.
(The interested reader should look up ``Grothendieck construction''.)

\subsection{Cofibrations}
Let $i:A\to X$ be a map of spaces.
If $Y$ is another topological space, when is the induced map $Y^X\to Y^A$ a fibration?
This is asking for the map $i$ to be ``dual'' to a fibration.

By the definition of a fibration, we want a lifting:
\begin{equation*}
    \xymatrix{
	W\ar[r]\ar[d]_{\mathrm{in}_0} & Y^X\ar[d]\\
	I\times W\ar@{-->}[ur]\ar[r] & Y^A.
    }
\end{equation*}
Adjointing over, we get:
\begin{equation*}
    \xymatrix{
	A\times W\ar[r]^{i\times 1}\ar[d]_{1\times \mathrm{in}_0} & X\times W\ar[d]\ar[ddr]& \\
	A\times W\times I \ar[r]\ar[drr] & X\times I\times W\ar@{-->}[dr] & \\
	& & Y.
    }
\end{equation*}
Again adjointing over, this diagram transforms to:
\begin{equation*}
    \xymatrix{
	A\ar[r]\ar[d] & X\ar[d]\ar[ddr] & \\
	A\times I\ar[r]\ar[drr] & X\times I\ar@{-->}[dr] & \\
	& & Y^W.
    }
\end{equation*}
This discussion motivates the following definition of a ``cofibration'':
as mentioned above, this is ``dual'' to the notion of fibration.

\begin{definition}\label{cofibration}
    A map $i:A\to X$ of spaces is said to be a \emph{cofibration} if it satisfies the
    \emph{homotopy extension property} (sometimes abbreviated as ``HEP''):
    for any space $Y$, there is a dotted map in the following diagram that makes it commute:
    \begin{equation*}
    \xymatrix{
	A\ar[r]\ar[d] & X\ar[d]\ar[ddr] & \\
	A\times I\ar[r]\ar[drr] & X\times I\ar@{-->}[dr] & \\
	& & Y.
    }
    \end{equation*}
\end{definition}

Again, using the definition of a pushout, the universal example of such a space $Y$ is the pushout $X\cup_A (A\times I)$.
Equivalently, we are therefore asking for the existence of a dotted arrow in the following diagram.
\begin{equation*}
    \xymatrix{
	X\cup_A (A\times I)\ar[r]\ar[dr] & X\times I\ar@{-->}[d]\\
	& Z,
    }
\end{equation*}
for any $Z$.
Using the universal property of a pushout, this is equivalent to the existence of a dotted arrow in the following diagram.
\begin{equation*}
    \xymatrix{
	X\cup_A (A\times I)\ar[r]\ar[dr] & X\times I\ar@{-->}[d]\\
	& X\cup_A(A\times I)\ar[d]\\
	& Z,
    }
\end{equation*}
which is, in turn, equivalent to asking $X\cup_A (A\times I)$ to be a retract of $X\times I$.

\begin{example}\label{intervalcofib}
    $S^{n-1}\hookrightarrow D^n$ is a cofibration.
    \todo{Properly draw out this figure!}
    \begin{figure}[H]
	\centering
	\includegraphics[scale=0.75]{assets/retract-cofibration}
	\caption{Drawing by John Ni.}
    \end{figure}
    In particular, setting $n=1$ in this example, $\{0,1\}\hookrightarrow I$ is a cofibration.
\end{example}

Here are some properties of the class of cofibrations of CGWH spaces.
\begin{itemize}
    \item It's closed under cobase change: if $A\to X$ is a cofibration, and $A\to B$ is any map,
	the pushout $B\to X\cup_A B$ is also cofibration. (Exercise!)
    \item It's closed under finite products. (This is surprising.)
    \item It's closed under composition. (Exercise!)
    \item Any cofibration is a closed inclusion\footnote{Note that the dual statement for
	fibrations would state:	any fibration $p:E\to B$ is a quotient map.
	This is definitely not true: fibrations do not have to be surjective!
	For instance, the trivial map $\emptyset\to B$ is a fibration.
	(Fibrations are surjective on path components though, because of path lifting.)}.
	\todo{This is not obvious; should we include a proof?}
\end{itemize}


\subsection{Division rings}

A divison ring is a ring where every non-zero element has a multiplicative inverse.

\subsubsection{Example}

The rational numbers are a divison ring.

\subsubsection{Relationship between divison rings and fields}

Fields (not yet introduced) are different from division rings only in that multiplication for a field must be commutative.


\lab{Gaussian Quadrature}{Gaussian Quadrature}
\label{Lab:GaussQuad}

% TODO: points and weights calculations aren't great when compared to NumPy via
% np.polynomial.chebyshev.chebgauss() or np.polynomial.legendre.leggauss().

\objective{
Learn the basics of Gaussian quadrature and its application to numerical integration.
Build a class to perform numerical integration using Legendre and Chebyshev polynomials.
Compare the accuracy and speed of both types of Gaussian quadrature with the built-in Scipy package.
Perform multivariate Gaussian quadrature.
}

\section*{Legendre and Chebyshev Gaussian Quadrature} % =======================

It can be shown that for any class of orthogonal polynomials $p\in \mathbb{R}[x;2n+1]$ with corresponding weight function $w(x)$, there exists a set of points $\{x_i\}_{i=0}^n$ and weights $\{w_i\}_{i=0}^n$ such that
\[
\int_a^b p(x) w(x) dx = \sum_{i=0}^n p(x_i)w_i.
\]
Since this relationship is exact, a good approximation for the integral
\[
\int_a^b f(x) w(x) dx
\]
can be expected as long as the function $f(x)$ can be reasonably interpolated by a polynomial at the points $x_i$ for $i=0,1,\dots,n$.
In fact, it can be shown that if $f(x)$ is $2n+1$ times differentiable, the error of the approximation will decrease as $n$ increases.

Gaussian quadrature can be performed using any basis of orthonormal polynomials, but the most commonly used are the Legendre polynomials and the Chebyshev polynomials.
Their weight functions are $w_l(x)=1$ and $w_c(x)=\frac{1}{\sqrt{1-x^2}}$, respectively, both defined on the open interval $(-1,1)$.

\begin{problem} % GaussianQuadrature.__init__().
\label{prob:gaussquad-init}
Define a class for performing Gaussian quadrature.
The constructor should accept an integer $n$ denoting the number of points and weights to use (this will be explained later) and a label indicating which class of polynomials to use.
If the label is not either \li{"legendre"} or \li{"chebyshev"}, raise a \li{ValueError}; otherwise, store it as an attribute.

The weight function $w(x)$ will show up later in the denominator of certain computations.
Define the reciprocal function $w(x)^{-1} = 1/w(x)$ as a \li{lambda} function and save it as an attribute.
\end{problem}

\subsection*{Calculating Points and Weights} % --------------------------------

% The first step of Gaussian quadrature is finding the weights and points that will be used.
% There are several important algorithms that will find the weights and points of quadrature.
% One based on a recurrence relationship inherent in all orthogonal polynomials will be considered in this lab.

All sets of orthogonal polynomials $\{u_k\}_{k=0}^{n}$ satisfy the three-term recurrence relation
\[
u_0=1,\qquad u_1 = x - \alpha_1,\qquad u_{k+1} = (x-\alpha_k)u_k - \beta_ku_{k-1}
\]
for some coefficients $\{\alpha_k\}_{k=1}^n$ and $\{\beta_k\}_{k=1}^n$.
% These constants have been calculated for several classes of orthogonal polynomials;
% there also exist algorithms for finding the coefficients for arbitrary classes of orthogonal polynomials.
%and may be determined for an arbitrary class using the procedure found in ``Calculation of Gauss Quadrature Rules'' by Golub and Welsch.
For the Legendre polynomials, they are given by
\[
\alpha_k=0, \qquad\qquad \beta_k=\frac{k^2}{4k^2-1},
\]
and for the Chebyshev polynomials, they are
\[
\alpha_k=0, \qquad\qquad \beta_k=\begin{cases} \frac{1}{2} &\mbox{ if } k = 1\\ \frac{1}{4} &\text{otherwise.} \\ \end{cases}
\]
Given these values, the corresponding \emph{Jacobi matrix} is defined as follows.
\[
J = \begin{bmatrix}
\alpha_1 & \sqrt{\beta_1} & 0 & \dots & 0 \\
\sqrt{\beta_1} & \alpha_2 & \sqrt{\beta_2} & \dots & 0 \\
0 & \sqrt{\beta_2} & \alpha_3 &  \ddots & 0 \\
\vdots & & \ddots & \ddots & \vdots \\
%\vdots & & \ddots & & \vdots \\
0 & \dots & & & \sqrt{\beta_{n-1}} \\
0 & \dots & & \sqrt{\beta_{n-1}} & \alpha_n
\end{bmatrix}
\]
According to the \emph{Golub-Welsch algorithm},\footnote{See \url{http://gubner.ece.wisc.edu/gaussquad.pdf} for a complete treatment of the Golub-Welsch algorithm, including the computation of the recurrence relation coefficients for arbitrary orthogonal polynomials.} the $n$ eigenvalues of $J$ are the points $x_i$ to use in Gaussian quadrature, and the corresponding weights are given by $w_i=\mu_w(\mathbb{R})v_{i,0}^2$ where $v_{i,0}$ is the first entry of the $i$th eigenvector and
$\mu_w(\mathbb{R}) = \int_{-\infty}^\infty w(x)dx$ is the \emph{measure} of the weight function.
Since the weight functions for Legendre and Chebyshev polynomials have compact support on the interval $(-1, 1)$, their measures are given as follows.
\[
\mu_{w_l}(\mathbb{R}) = \int_{-\infty}^\infty w_l(x)dx = \int_{-1}^{1} 1dx = 2
\qquad \qquad
\mu_{w_c}(\mathbb{R}) = \int_{-\infty}^\infty w_c(x)dx = \int_{-1}^{1} \frac{1}{\sqrt{1 - x^2}} dx = \pi
\]

%Finding eigenvalues and eigenvectors for a tridiagonal matrix is a well conditioned problem.
%Using a good eigenvalue solver gives the Golub-Welsch algorithm a complexity of $O(n^2)$.

\begin{problem} % Jacobi matrix + points and weights.
\label{prob:jacobi}
Write a method for your class from Problem \ref{prob:gaussquad-init} that accepts an integer $n$.
Construct the $n\times n$ Jacobi matrix $J$ for the polynomial family indicated in the constructor.
Use SciPy to compute the eigenvalues and eigenvectors of $J$, then compute the points $\{x_i\}_{i=1}^n$ and weights $\{w_i\}_{i=1}^n$ for the quadrature.
Return both the array of points and the array weights.

Test your method by checking your points and weights against the following values using the Legendre polynomials with $n=5$.
{\small
\[
\centering
\begin{array}{c|c|c|c|c|c}
    x_i
    & -\frac{1}{3}\sqrt{5 + 2\sqrt{\frac{10}{7}}}
    & -\frac{1}{3}\sqrt{5 - 2\sqrt{\frac{10}{7}}}
    & 0
    & \frac{1}{3}\sqrt{5 - 2\sqrt{\frac{10}{7}}}
    & \frac{1}{3}\sqrt{5 + 2\sqrt{\frac{10}{7}}}
    \\[1em] \hline
    w_i
    & \dfrac{322-13\sqrt{70}}{900}
    & \dfrac{322+13\sqrt{70}}{900}
    & \dfrac{128}{225}
    & \dfrac{322+13\sqrt{70}}{900}
    & \dfrac{322-13\sqrt{70}}{900}
\end{array}
\]
}
% Note that the order of the points and weights in the given table may differ.

Finally, modify the constructor of your class so that it calls your new function and stores the resulting points and weights as attributes.
\end{problem}

\subsection*{Integrating with Given Weights and Points} % ---------------------

Now that the points and weights have been obtained, they can be used to approximate the integrals of different functions.
For a given function $f(x)$ with points $x_i$ and weights $w_i$,
\[
\int_{-1}^{1} f(x) w(x) dx \approx \sum_{i=1}^n f(x_i)w_i.
\]
There are two problems with the preceding formula.
First, the weight function is part of the integral being approximated, and second, the points obtained are only found on the interval $(-1,1)$ (in the case of the Legendre and Chebyshev polynomials).
To solve the first problem, define a new function $g(x) = f(x) / w(x)$ so that
\begin{equation}
\int_{-1}^{1} f(x) dx
= \int_{-1}^{1} g(x) w(x) dx
\approx \sum_{i=1}^n g(x_i)w_i.
\label{eq:quadrature-no-shift}
\end{equation}

The integral of $f(x)$ on $[-1,1]$ can thus be approximated with the inner product $\w\trp g(\x)$, where $g(\x) = [g(x_1),\dots,g(x_n)]\trp$ and $\w = [w_1,\dots,w_n]\trp$.

\begin{problem} % Integrate with given points and weights.
Write a method for your class that accepts a callable function $f$.
Use \eqref{eq:quadrature-no-shift} and the stored points and weights to approximate of the integral of $f$ on the interval $[-1,1]$.
\\(Hint: Use $w(x)^{-1}$ from Problem \ref{prob:gaussquad-init} to compute $g(x)$ without division.)

% Remember that the weight function depends on the type of polynomial (the type should be stored as an attribute).
Test your method with examples that are easy to compute by hand and by comparing your results to \li{scipy.integrate.quad()}.
\begin{lstlisting}
>>> import numpy as np
>>> from scipy.integrate import quad

# Integrate f(x) = 1 / sqrt(1 - x**2) from -1 to 1.
>>> f = lambda x: 1 / np.sqrt(1 - x**2)
>>> quad(f, -1, 1)[0]
3.141592653589591
\end{lstlisting}
\label{prob:gaussquad-no-shift}
\end{problem}

\begin{info}
Since the points and weights for Gaussian quadrature do not depend on $f$, they only need to be computed once and can then be reused to approximate the integral of any function.
The class structure in Problems \ref{prob:gaussquad-init}--\ref{prob:gaussquad-shift} takes advantage of this fact, but \li{scipy.integrate.quad()} does not.
If a larger $n$ is needed for higher accuracy, however, the computations must be repeated to get a new set of points and weights.
\end{info}

\subsection*{Shifting the Interval of Integration} % --------------------------

Since the weight functions for the Legendre and Chebyshev polynomials have compact support on the interval $(-1,1)$, all of the quadrature points are found on that interval as well.
To integrate a function on an arbitrary interval $[a,b]$ requires a change of variables.
Let \[u = \frac{2x - b - a}{b - a}\] so that $u = -1$ when $x = a$ and $u = 1$ when $x=b$.
Then
\[
x = \frac{b - a}{2}u + \frac{a + b}{2}\qquad \text{and}\qquad dx = \frac{b - a}{2}du,
\]
so the transformed integral is given by
\[
\int_a^b f(x) dx = \frac{b-a}{2}\int_{-1}^1 f\left(\frac{b-a}{2}u + \frac{a+b}{2}\right)du.
\]
By defining a new function $h(x)$ as
\[
h(x) = f\left(\frac{(b-a)}{2}x + \frac{(a+b)}{2}\right),
\]
the integral of $f$ can be approximated by integrating $h$ over $[-1,1]$ with \eqref{eq:quadrature-no-shift}.
This results in the final quadrature formula
\begin{equation}
\int_{a}^{b} f(x) dx = \frac{b-a}{2}\int_{-1}^1 h(x)dx = \frac{b-a}{2}\int_{-1}^1 g(x)w(x)dx \approx \frac{b-a}{2}\sum_{i=1}^n g(x_i)w_i,
\label{eq:quadrature-complete}
\end{equation}
where now $g(x) = h(x) / w(x)$.

\begin{problem} % Shift the integral.
Write a method for your class that accepts a callable function $f$ and bounds of integration $a$ and $b$.
Use \eqref{eq:quadrature-complete} to approximate the integral of $f$ from $a$ to $b$.
\\(Hint: Define $h(x)$ and use your method from Problem \ref{prob:gaussquad-no-shift}.)
\label{prob:gaussquad-shift}
\end{problem}

\begin{comment}
\section*{Numerical Integration with SciPy} % =================================

There are many other techniques for finding the weights and points for a given weighting function.
SciPy's \li{integrate} module provides some general-purpose integration tools.
For example, \li{scipy.integrate.quad()} is a reasonably fast Gaussian quadrature implementation.
Also included in the \li{integrate} module are fixed-precision and fixed-order Gaussian quadrature methods.
\begin{lstlisting}
>>> from scipy.integrate import quad

>>> f = lambda x: np.cos(x) * np.sin(x)**2 # Function to integrate.
>>> g = lambda x: np.sin(x)**3 / 3         # Antiderivative.

# quad returns an array, the first entry is the computed value.
>>> calc = quad(f, -2, 3)[0]
>>> exact = g(3) - g(-2)                   # Exact value of the integral.
>>> np.<<abs>>(exact - calc)                   # Error of the approximation.
0.0
\end{lstlisting}
\end{comment}

% TODO: need to demonstrate this if it's to be included.
\begin{comment}
Another common hallmark of quadrature is that it can be used adaptively.
It is common in practice to refine the points of a quadrature estimate on an interval where a function is observed to be changing rapidly.
This allows for more accurate computation at a relatively low computational cost.
This is the approach used by the function \li{scipy.integrate.quad()}.
\end{comment}

\begin{problem} % Get error estimates by integrating the standard normal.
The \emph{standard normal distribution} has the following probability density function.
\[f(x) = \frac{1}{\sqrt{2 \pi}} e^{-x^2/2}\]
This function has no symbolic antiderivative, so it can only be integrated numerically.
The following code gives an ``exact'' value of the integral of $f(x)$ from $-\infty$ to a specified value.

\begin{lstlisting}
>>> from scipy.stats import norm

>>> norm.cdf(1)                     # Integrate f from -infty to 1.
0.84134474606854293
>>> norm.cdf(1) - norm.cdf(-1)      # Integrate f from -1 to 1.
0.68268949213708585
\end{lstlisting}
% The probability that a normally distributed random variable $X$ will take on a value less than (or equal to) a given value $x$ is \[ P(X \le x) = \int_{-\infty}^x p(t)dt = \int_{-\infty}^x \frac{1}{\sqrt{2 \pi}} e^{-t^2/2} dt. \]

Write a function that uses \li{scipy.stats} to calculate the ``exact'' value
\[
F = \int_{-3}^2 f(x)dx.
\]
Then repeat the following experiment for $n=5,10,15,\ldots,50$.
\begin{enumerate}
\item Use your class from Problems \ref{prob:gaussquad-init}--\ref{prob:gaussquad-shift} with the Legendre polynomials to approximate $F$ using $n$ points and weights.
Calculate and record the error of the approximation.
\item Use your class with the Chebyshev polynomials to approximate $F$ using $n$ points and weights.
Calculate and record the error of the approximation.
\end{enumerate}
Plot the errors against the number of points and weights $n$, using a log scale for the $y$-axis.
Finally, plot a horizontal line showing the error of \li{scipy.integrate.quad()} (which doesn't depend on $n$).
\end{problem}

\section*{Multivariate Quadrature} % ==========================================

The extension of Gaussian quadrature to higher dimensions is fairly straightforward.
The same set of points $\{z_i\}_{i=1}^n$ and weights $\{w_i\}_{i=1}^n$ can be used in each direction, so the only difference from 1-D quadrature is how the function is shifted and scaled.
To begin, let $h:\mathbb{R}^2\rightarrow\mathbb{R}$ and define $g:\mathbb{R}^2\rightarrow\mathbb{R}$ by $g(x,y) = h(x,y)/(w(x)w(y))$ so that
\begin{equation}
\int_{-1}^1\int_{-1}^1 h(x,y)dx\:dy.
= \int_{-1}^1\int_{-1}^1 g(x,y)w(x)w(y)dx\:dy
\approx \sum_{i=1}^n \sum_{j=1}^n w_i w_j g(z_i,z_j).
\label{eq:gaussquad-multivariable-scale}
\end{equation}
To integrate $f:\mathbb{R}^2\rightarrow\mathbb{R}$ over an arbitrary box $[a_1,b_1]\times[a_2,b_2]$, set
\[
h(x,y) = f\left(\frac{b_1 - a_1}{2}x + \frac{a_1 + b_1}{2},\
                \frac{b_2 - a_2}{2}y + \frac{a_2 + b_2}{2}\right)
\]
so that
\begin{equation}
\int_{a_2}^{b_2}\int_{a_1}^{b_1} f(x) dx\:dy
= \frac{(b_1 - a_1) (b_2 - a_2)}{4}\int_{-1}^1\int_{-1}^1 h(x,y)dx\:dy.
\label{eq:gaussquad-multivariable-shift}
\end{equation}
Combining \eqref{eq:gaussquad-multivariable-scale} and \eqref{eq:gaussquad-multivariable-shift} gives the final 2-D Gaussian quadrature formula.
Compare it to \eqref{eq:quadrature-complete}.
\begin{equation}
\int_{a_2}^{b_2}\int_{a_1}^{b_1} f(x) dx\:dy
\approx \frac{(b_1 - a_1) (b_2 - a_2)}{4}
\sum_{i=1}^n \sum_{j=1}^n w_i w_j g(z_i,z_j)
\label{eq:quadrature-multivariate-complete}
\end{equation}

\begin{problem}
Write a method for your class that accepts a function $f:\mathbb{R}^2\rightarrow\mathbb{R}$ (which actually accepts two separate arguments, not one array with two elements) and bounds of integration $a_1$, $a_2$, $b_1$, and $b_2$.
Use \eqref{eq:quadrature-multivariate-complete} to compute the double integral
\[\int_{a_2}^{b_2}\int_{a_1}^{b_1} f(x) dx\:dy.\]
% Note that we have assumed that we will be using the same number of points in both dimensions so the previous equation is valid.

Validate your method by comparing it \li{scipy.integrate.nquad()}.
Note carefully that this function has slightly different syntax for the bounds of integration.
\begin{lstlisting}
>>> from scipy.integrate import nquad

# Integrate f(x,y) = sin(x) + cos(y) over [-10,10] in x and [-1,1] in y.
>>> f = lambda x, y: np.sin(x) + np.cos(y)
>>> nquad(f, [[-10, 10], [-1, 1]])[0]
33.658839392315855
\end{lstlisting}
\end{problem}

\begin{info}
Although Gaussian quadrature can obtain reasonable approximations in lower dimensions, it quickly becomes intractable in higher dimensions due to the curse of dimensionality.
In other words, the number of points and weights required to obtain a good approximation becomes so large that Gaussian quadrature become computationally infeasible.
For this reason, high-dimensional integrals are often computed via \emph{Monte Carlo methods}, numerical integration techniques based on random sampling.
However, quadrature methods are generally significantly more accurate in lower dimensions than Monte Carlo methods.
\end{info}


\section{Definitions}

% BINOMIAL THEOREMS

\vspace{0.5cm}\subsection{Binomial Theorems}

\vspace{0.5cm}\begin{definition}\label{def-biomial-theorems}

    \begin{equation}
        (a+b)^2 = a^2 + 2ab + b^2 \label{eq-1}
    \end{equation}

    \begin{equation}
        (a-b)^2 = a^2 - 2ab + b^2 \label{eq-2}
    \end{equation}

    \begin{equation}
        (a+b)\,(a-b) = a^2 - b^2 \label{eq-3}
    \end{equation}

    \flushleft \normalfont For higher exponentiations:

    \begin{equation}
        (a+b)^3 = a^3 +3a^2b + 3ab^2 + b^3 \label{eq-4}
    \end{equation}

    \begin{equation}
        (a-b)^3 = a^3 - 3a^2b + 3ab^2 - b^3 \label{eq-5}
    \end{equation}

    \begin{equation}
        (-a-b)^3 = -a^3 - 3a^2b - 3ab^2 - b^3 \label{eq-6}
    \end{equation}

    Binomial Formula:

    \begin{equation}
        (x+y)^{n}=\sum _{k=0}^{n}{n \choose k}x^{n-k}y^{k}=\sum _{k=0}^{n}{n \choose k}x^{k}y^{n-k} \label{eq-7}
    \end{equation}

    where

    \begin{equation}
        {\displaystyle {\binom {n}{k}}={\frac {n!}{k!(n-k)!}},} \label{eq-8}
    \end{equation}

\end{definition}

% FRACTIONS

\vspace{0.5cm}\subsection{Fractions}

\vspace{0.5cm}\begin{definition}\label{def-fractions}

    \begin{equation}
        \frac{a}{b} + \frac{c}{b} = \frac{a + c}{b} \label{eq-9}
    \end{equation}

    \begin{equation}
        \frac{a}{b} - \frac{c}{b} = \frac{a - c}{b} \label{eq-10}
    \end{equation}

    \begin{equation}
        \frac{a}{b} \cdot  \frac{c}{d} = \frac{a \cdot c}{b \cdot d} \label{eq-11}
    \end{equation}

    \begin{equation}
        \frac{a}{b} + \frac{c}{d} = \frac{ad}{bd} + \frac{bc}{bd} = \frac{ad + bc}{bd} \label{eq-12}
    \end{equation}

    \begin{equation}
        \frac{a}{b} - \frac{c}{d} = \frac{ad}{bd} - \frac{bc}{bd} = \frac{ad - bc}{bd} \label{eq-13}
    \end{equation}

    \flushleft \normalfont Inverse:

    \begin{equation}
        \frac{a}{b} \div \frac{c}{d} = \frac{a}{b} \cdot \frac{d}{c} = \frac{ad}{bc} \label{eq-14}
    \end{equation}

    \begin{equation}
        \dfrac{\dfrac{a}{b}}{\dfrac{c}{d}} = \frac{a}{b} \cdot \frac{d}{c} = \frac{a \cdot b}{d \cdot c} \label{eq-15}
    \end{equation}

    \begin{equation}
        \dfrac{a \cdot \dfrac{b}{c}}{\dfrac{d}{e}} = \dfrac{\dfrac{a \cdot c + b}{c}}{\dfrac{d}{c}} = \dfrac{a \cdot c + b}{c} \cdot \dfrac{c}{d} = \dfrac{(a \cdot c + b) \cdot c}{c \cdot d} \label{eq-16}
    \end{equation}

    \begin{equation}
        \frac{ab}{c} = \frac{a}{c} \cdot b \label{eq-17}
    \end{equation}

    \begin{equation}
        \frac{a}{b} = \frac{1}{b} \cdot a \label{eq-18}
    \end{equation}

    \begin{equation}
        \frac{a \div b}{c} = \frac{a}{c} \div b \label{eq-19}
    \end{equation}

\end{definition}

% PARENTHESES

\vspace{0.5cm}\subsection{Parentheses Rules}

\vspace{0.5cm}\begin{definition}\label{def-parentheses}

    \begin{equation}
    +\,(a+b) = a+b \label{eq-20}
    \end{equation}

    \begin{equation}
        +\,(-a-b) = -a-b \label{eq-21}
    \end{equation}

    \begin{equation}
        -\,(a-b) = -a+b \label{eq-22}
    \end{equation}

    \begin{equation}
        -\,(-a+b) = +a-b \label{eq-23}
    \end{equation}

    \begin{equation}
        -\,(a+b) = -a-b \label{eq-24}
    \end{equation}

    \flushleft \normalfont Associative properties:

    \begin{equation}
        (a+b)+c = a+(b+c) \label{eq-25}
    \end{equation}

    \begin{equation}
        (a \cdot b) \cdot c = a \cdot (b \cdot c) \label{eq-26}
    \end{equation}

    Distributive properties:

    \begin{equation}
        a \cdot (b+c) = (a \cdot b) + (a \cdot c) \label{eq-27}
    \end{equation}

    \begin{equation}
        (a+b) \cdot c = (a \cdot c) + (b \cdot c) \label{eq-28}
    \end{equation}

    Commutative properties:

    \begin{equation}
        a+b = b+a \label{eq-29}
    \end{equation}

    \begin{equation}
        a \cdot b = b \cdot a \label{eq-30}
    \end{equation}

\end{definition}

% MULTIPLY WITH -1

\vspace{0.5cm}\subsection{Multiply with -1}
\vspace{0.5cm}\begin{definition}\label{def-multiply-with-minus-one}

    Mathematical operators may be swapped by multiplying with -1, because the result does not change.

    \begin{equation}
         a+b = c\; \;\boldsymbol{\leftrightarrow}\; \;-1 \cdot (-a-b) = c \label{eq-31}
    \end{equation}

    \flushleft \normalfont \bfseries Example:

    \begin{equation}
        (a-b)^2 = (b-a)^2 \label{eq-32}
    \end{equation}

\end{definition}

% SQUARE ROOTS

\vspace{0.5cm}\subsection{Square Roots}
\vspace{0.5cm}\begin{definition}\label{def-square-roots}

    \begin{equation}
        \sqrt[1]{a} = a \label{eq-33}
    \end{equation}

    \begin{equation}
        \sqrt[2]{a} = \sqrt{a} \label{eq-34}
    \end{equation}

    \begin{equation}
        \sqrt{a^2} = a \label{eq-35}
    \end{equation}

    \begin{equation}
        \left(\sqrt{a}\right)^2 = a \label{eq-36}
    \end{equation}

    \begin{equation}
        \frac{1}{\sqrt{n}} \cdot \frac{1}{\sqrt{n}} = \frac{1}{n} \label{eq-37}
    \end{equation}

    \begin{equation}
        \sqrt{n} \cdot \sqrt{n} = n \label{eq-38}
    \end{equation}

    \flushleft \normalfont Addition:

    \begin{equation}
        a\sqrt[n]{x} + b\sqrt[n]{x} = (a+b)\sqrt[n]{x} \label{eq-39}
    \end{equation}

    Subtraction:

    \begin{equation}
        a\sqrt[n]{x} - b\sqrt[n]{x} = (a-b)\sqrt[n]{x} \label{eq-40}
    \end{equation}

    Multiplication:

    \begin{equation}
        \sqrt[n]{a} \cdot \sqrt[n]{b} = \sqrt[n]{a \cdot b} \label{eq-41}
    \end{equation}

    Division:

    \begin{equation}
        \dfrac{\sqrt[n]{a}}{\sqrt[n]{b}} = \sqrt[n]{\dfrac{a}{b}} \label{eq-42}
    \end{equation}

    Root exponentiation:

    \begin{equation}
       \Big(\sqrt[n]{a}\Big)^m = \sqrt[n]{m} \label{eq-43}
    \end{equation}

    Root extraction:

    \begin{equation}
        \sqrt[m]{\sqrt[n]{a}} = \sqrt[m \cdot n]{a} \label{eq-44}
    \end{equation}

    Transforming roots into exponents:

    \begin{equation}
        \sqrt[n]{a} = a \cdot \frac{1}{n} \label{eq-45}
    \end{equation}

    \begin{equation}
        \sqrt{a} = a \cdot \frac{1}{2} \label{eq-46}
    \end{equation}

    \begin{equation}
        \sqrt[n]{a^m} = a^\frac{m}{n} \label{eq-47}
    \end{equation}

\end{definition}

% EXPONENTIATION

\vspace{0.5cm}\subsection{Exponentiation}
\vspace{0.5cm}\begin{definition}\label{def-exponentiation}

    \begin{equation}
        x^n \cdot x^b = x^{n+b} \label{eq-48}
    \end{equation}

    \begin{equation}
        x^n \div x^b = \frac{x^n}{x^b} = x^{n-b} \label{eq-49}
    \end{equation}

    \begin{equation}
       \Big(x^a\Big)^b = x^{a \cdot b} \label{eq-50}
    \end{equation}

    \begin{equation}
        a^n \cdot a^b = (a \cdot b)^n \label{eq-51}
    \end{equation}

    \begin{equation}
        a^n \div b^n = \frac{a^n}{b^n} = \left(\frac{a}{b}\right)^n \label{eq-52}
    \end{equation}

    \begin{equation}
        x^0 = 1 \label{eq-53}
    \end{equation}

    \begin{equation}
        x^1 = x \label{eq-54}
    \end{equation}

    \begin{equation}
        x^{-n} = \frac{1}{x^n} \label{eq-55}
    \end{equation}

    \begin{equation}
        \frac{1}{x} = x^{-1} \label{eq-56}
    \end{equation}

    \begin{equation}
        x^{\frac{1}{n}} = \sqrt[n]{x} \label{eq-57}
    \end{equation}

    \flushleft \normalfont {\bfseries Disclaimer (for 56):} If n is even, then x must be > 0!

    \begin{equation}
        x^{\frac{m}{n}} = \sqrt[n]{m} \label{eq-58}
    \end{equation}

    \begin{equation}
        x^{-\frac{m}{n}} = \frac{1}{\sqrt[n]{x^m}} \label{eq-59}
    \end{equation}

    Addition: %(cf. \ref{eq-power-to-sqrt})

    \begin{equation}
        ax^n + bx^n = (a+b)x^n \label{eq-60}
    \end{equation}

    Subtraction:

    \begin{equation}
        ax^n - bx^n = (a-b)x^n \label{eq-61}
    \end{equation}

    Transform a single root into a exponent:

    \begin{equation}
        \sqrt{a} = (a)^{\frac{1}{2}} \cdots \sqrt[3]{a} = (a)^{\frac{1}{3}} \cdots \label{eq-62}
    \end{equation}

\end{definition}


\clearpage
\section{Glossy Analysis}
\subsection{Linearly connected network}
In a linear connected network, assuming every nodes are connected linearly with two neighbors.
Namely, only two nodes in the RF range of single transmission except the terminal nodes.

\begin{figure}[h]
\centering
	\includegraphics[width=0.6\columnwidth]{glossy_linear_topology}
\end{figure}

With Glossy protocol, the number of transmission $N$ is a parameter which limits the maximum
number of transmission of each node. Assuming the packet length in time $T_p$, TXRX turnaround
time $T_t$ and the number of hops $K$.
In the steady state, every nodes in the network wake up in the same time. Assuming all nodes
wake up at time $t = 0$ and the {\bf Initiator (I)} starts transmission after time delay
$T_\gamma$. Figure~\ref{fig:glossy_linear_timing} is the example with I = {\bf n\subscript{1}}
and $N = 3$.

\begin{figure}[h]
\centering
	\includegraphics[width=0.9\columnwidth]{glossy_linear_timing}
	\caption{Timing diagram of a simple glossy flooding with N = 3}
	\label{fig:glossy_linear_timing}
\end{figure}

From figure~\ref{fig:glossy_linear_timing}, we can infer the radio on time for each node
in a linear connected network. Note that the following analysis is base on the assumption
that {\bf every transmission is success}, which is likely hold if no interference present.

The radio on time of initiator I is the shortest. It has to transmit $N$ times. Between $N$
transmissions, there are $N-1$ receiving slot. Therefore, sum up the total number of transmission
and receiving. We can find $2N-1$ packet length $T_p$ and $2N-2$ turnaround time $T_t$.
We can write the radio on time of Initiator $T_I$ as following:
\begin{equation}
	T_I = T_\gamma + (2N-1)T_P + (2N-2)T_t
\end{equation}
From the figure~\ref{fig:glossy_linear_timing}, we can infer that the radio on time of the node
which is 1 hop distance from the initiator $T_{D=1}$ is $T_p + T_t$ longer than $T_I$. Since
the node has the exactly the same pattern once it starts transmission. The time between the radio on
and starts transmitting is $T_p + T_t$.
\begin{equation}
	T_{D=1} = T_I + T_p + T_t
\end{equation}
Expand the radio on time for any nodes which is $i$ hops away from initiator $T_{D=i}$
\begin{equation}
	T_{D=i} = T_I + i\times(T_p + T_t)
\end{equation}

We define a {\bf Total Radio On Time (TROT)} as the {\bf sum of radio on time in the network}. Since
Glossy uses flooding to send packet through the entire network, the TROT of a linearly connected
network is not a constant. However, it's correlated with the {\bf position of initiator in the
network}. The intuition view is that the nodes far from initiator would have longer on time. Therefore,
the TROT is minimum if the initiator is located in the middle of the linearly connected network.
We define another term {\bf Total Distance (TD)} as the {\bf sum of distance between all nodes in the network
and initiator I}. We can correlate the TROT to TD. To calculate the TROT, we can simply
calculate the TD.
\begin{equation}
	TROT = TD\times(T_p + T_t) + K\times T_I
	\label{eq:TROT_TD}
\end{equation}
For example, if the initiator I is located in the edge of the network ($n_1$ or $n_K$), the TD
is following:
\begin{equation}
	TD_1 = \displaystyle\sum\limits_{i=0}^{K-1} i = \tfrac{K(K-1)}{2}
\end{equation}
Similarly, if the initiator I is located in 2\superscript{rd} position of the network ($n_2$ or $n_{K-1}$).
TD can be written as following:
\begin{equation}
	TD_2 = \displaystyle\sum\limits_{i=0}^{K-2} i + 1 = \tfrac{(K-1)(K-2)}{2} + 1
\end{equation}
Expand to j\superscript{th} position of the network. Where $1\le j\le K$.
\begin{align}
	TD_j = \displaystyle\sum\limits_{i=0}^{K-j} i + \displaystyle\sum\limits_{i=0}^{j-1} i &= \tfrac{(K-j+1)\times(K-j)}{2} + \tfrac{j(j-1)}{2}\\
	&=\tfrac{K(K+1)}{2} + j^2 - j\times(K+1)
\end{align}
Another assumption made here is that every nodes in the network has the same probability (uniform
distribution) to act as initiator I. Namely, $p_{j=1} = p_{j=2} \cdots = p_{j=K-1} = p_{j=K} = 1/K$

Therefore, the average TD can be expressed as follows:
\begin{align}
	TD_{avg} 	&= \tfrac{1}{K} \displaystyle\sum\limits_{j=1}^K TD_j\\
				&= \tfrac{1}{K} \displaystyle\sum\limits_{j=1}^K \tfrac{K(K+1)}{2} + j^2 - j\times(K+1)\\
				&= \tfrac{1}{K} [\tfrac{K(K+1)}{2}\times K + \tfrac{K(K+1)(2K+1)}{6} - \tfrac{K(K+1)}{2}\times(K+1)]\\
				&= \tfrac{K(K+1)}{2} + \tfrac{(K+1)(2K+1)}{6} - \tfrac{(K+1)^2}{2}\\
				&= \tfrac{K^2-1}{3}
\end{align}
From equation~\ref{eq:TROT_TD}, the average TROT can be written as following:
\begin{align}
	TROT_{avg} 	&= TD_{avg}\times(T_p + T_t) + K\times T_I\\
				&= \tfrac{K^2-1}{3}\times(T_p + T_t) + K\times[T_\gamma + (2N-1)T_P + (2N-2)T_t]
\end{align}
The average radio on time for each node $T_{on, avg} = TROT_{avg}/K$. If $K^2\gg1$
\begin{equation}
	T_{on, avg} \approx \{\tfrac{K}{3}\times(T_p + T_t)\} + \{T_\gamma + (2N-1)T_P + (2N-2)T_t\}
	\label{eq:ton_avg}
\end{equation}
From equation~\ref{eq:ton_avg}, we can find the average radio on time in a linear connected network
is composed of two parts. The first half is only correlated to {\bf the size of the network K} and the
second half is only correlated to {\bf the number of transmission N}. In a practical example,
$K$ = 10, $T_p$ = 0.8~ms (8 bytes data + 17 overhead~\footnote{ 4 bytes preamble, 1 byte SFD, 1 byte Length,
2 bytes FCF, 1 byte DSN, 6 bytes address (2 bytes for source/destination/PAN), 2 bytes FCS
}), $T_t$ = 192~$\mu$s + 23.5~$\mu$s (192~$\mu$s is the standard
turnaround time, 23.5~$\mu$s is SW delay implemented by Ferrai et al.~\cite{ferrari:efficient}), $N$ = 6 is
reasonable in a linear connected network since every node has only two neighbors. If $N$ is too small,
single transmission fails cause the packet unable to deliver to every node in the network.  $T_\gamma$ is
affected by three factors.
1) clock stability, 2) duty cycle period ($T$) and 3) the radio on time. Assume clock stability $= \pm$40~ppm,
$T$ = 1~s and crystal start-up time = 1~ms. We assume $T_\gamma$ = 1.5~ms in this case. Given these parameters,
the $T_{on, avg}$ in a linearly connected network is calculated as 15.84~ms.
\begin{table}[h]
\centering
	\begin{tabular}{|c|c|}
	\hline
	{\bf parameters} & {\bf value}\\ \hline
	$K$ 	& 10\\ \hline
	$T_p$ 	& 0.8~ms\\ \hline
	$T_t$	& 0.2155~ms\\ \hline
	$N$		& 6\\ \hline
	$T_\gamma$& 1.5~ms\\ \hline\hline
	$T_{on, avg}$& 15.84~ms\\ \hline
	\end{tabular}
\end{table}

\clearpage
\subsection{Grid network}
Considering a network structure that each node is {\bf on grid} and the minimum distance between
each node is 1. In addition, all the radio has its {\bf radio range = 1}. In other words, the nodes
in diagonal is out of 1 hop distance.
\begin{figure}[h]
\centering
	\includegraphics[width=0.4\columnwidth]{glossy_grid_topology}
	\caption{Topology of grid network. The small circles are nodes and big circle indicates the
	1 hop radio range}
\end{figure}

From previous session, we know the average of total distance for every initiator position in a linearly
connected network is equal to $\tfrac{K^2-1}{3}\equiv TD_0$.
The total distance of the column which next to the one hosts the initiator I. $TD_{D=1}$ can be written
as:
\begin{equation}
 TD_{D=1} = TD_0 + K
\end{equation}
We can calculate the total distance for the grid network with any initiator location.
For example, if the initiator is located in the left most column. $TD_{grid\ network,\ I=W_0}$ can be written as:
\begin{align}
	\nonumber
	TD_{grid\ network,\ I=W_0} 	&= TD_0 + TD_{D=1} + TD_{D=2} +\cdots TD_{D=W-2} + TD_{D=W-1}\\
						&= TD_0 + (TD_0+K) + (TD_0+2K) +\cdots (TD_0 + (W-2)K) + (TD_0 + (W-1)K)\\
						&= W\times TD_0 + K\times \displaystyle\sum\limits_{j=1}^{W-1} j
\end{align}
The average distance over entire nodes in a grid network in this case is:
\begin{equation}
	D_{avg,\ I=W_0} = \tfrac{TD_{grid\ network,\ I=W_0}}{W\times K}
\end{equation}
Similarly, the initiator could located in any column. The sum of total distance for every columns is following:
\begin{align}
	\nonumber
	TD_{total\ distance\ for\ every\ column}	&= TD_{grid\ network,\ I=W_0} + TD_{grid\ network,\ I=W_1}
	+ \cdots TD_{grid\ network,\ I=W_W}\\
	&= W^2\times TD_0 + K\displaystyle\sum\limits_{j=1}^W\{\displaystyle\sum\limits_{i=0}^{W-j} i + \displaystyle\sum\limits_{i=0}^{j-1} i\}\\
	&= W^2\times TD_0 + K\times\tfrac{W(W^2-1)}{3}
\end{align}
Therefore, the averaged total distance of the grid network is:
\begin{align}
	TD_{total\ distance,\ avg}	&= TD_{total\ distance\ for\ every\ column}/W\\
	&= W\times TD_0 + \tfrac{K(W^2-1)}{3}\\
	&= W\times\tfrac{K^2-1}{3} + K\times\tfrac{W^2-1}{3}
\end{align}
The average distance for each node is:
\begin{align}
	D_{avg} &= TD_{total\ distance,\ avg}/(W\times K)\\
	&= \tfrac{K^2-1}{3W} + \tfrac{W^2-1}{3K}
\end{align}
From equation~\ref{eq:ton_avg}, we can conclude that the average on time of a node in a grid
network $T_{on,\ avg,\ grid}$
\begin{equation}
	T_{on,\ avg,\ grid} = {[\tfrac{K^2-1}{3W} + \tfrac{W^2-1}{3K}]\times(T_p + T_t)} + {T_\gamma + (2N-1)T_P + (2N-2)T_t}
\end{equation}

Considering the grid network has the same depth (K) and width (W) equal to 5 hops and diagonal of the
network is 8 hops. Since any node in the network has at least 2 neighbors, the number of transmission
can reduce to 3 with reliable flooding. We calculate the average on time of the node in grid network
with following parameters:
\begin{table}[h]
\centering
	\begin{tabular}{|c|c|}
	\hline
	{\bf parameters} & {\bf value}\\ \hline
	$K$ 	& 5\\ \hline
	$W$		& 5\\ \hline
	$T_p$ 	& 0.8~ms\\ \hline
	$T_t$	& 0.2155~ms\\ \hline
	$N$		& 3\\ \hline
	$T_\gamma$& 1.5~ms\\ \hline\hline
	$T_{on, avg}$& 9.6116~ms\\ \hline
	\end{tabular}
\end{table}


\section{Bias and Variance for Learning in 1-D}
In this project, you will compare the difference between \textit{approximation} (finding the best possible candidate to model a function) and \textit{learning} (finding the best possible candidate given limited information).
You will also analyze the extra error that is incurred by using limited information, and you will explore one possible way of reducing this error.\\

\noindent\textit{Approximation:}\\
Let $f(x) = x^2(1-x)$ where $0 \leq x \leq 1$. We wish to find the `best' linear model (or approximation) for $f$. That is, for functions in the form $g(x; \alpha, \beta) =  \alpha +  \beta x$,  we wish to find the parameters $ \alpha$ and $ \beta$ that minimize the error we would expect to incur if we used the function $g$ to model (or approximate) the function $f$.
\begin{center}
\begin{tikzpicture}[xscale=5,yscale=10]
\draw[thick,domain=0:1,variable=\x,orange!90!black] plot ({\x},{.02 + .1*\x}) node[above right] {$y = g(x; \alpha, \beta)$};
\draw[thick,domain=0:1,variable=\x,blue] plot ({\x},{\x*\x*(1-\x)}) node[above right] {$y = f(x)$};
  \draw[->] (-.11, 0) -- (1.1, 0) node[right] {$x$};
  \draw[->] (0, -.02) -- (0, .2) node[above] {$y$};
\end{tikzpicture}
\end{center}
For this problem, we will define the error as
\begin{equation*}
\text{Error}(x;\alpha,\beta) = \Big(f(x) - g(x; \alpha, \beta)\Big)^2
\quad \text{and} \quad
\text{TotalError}(\alpha,\beta) = \int_0^1 \text{Error}(x; \alpha,\beta) dx.
% \text{Error}(f,g) = \int_0^1 \bigg(f(x) - g(x; \alpha, \beta)\bigg)^2 dx.
\end{equation*}
Use pencil-and-paper (or Mathematica, WolframAlpha, etc) to find a simplified expression for the error between $f$ and $g$ as a function of the parameters $ \alpha$ and $ \beta$. Find the values of $ \alpha$ and $ \beta$ that minimize the total error and find the corresponding total error.\\

\noindent\textit{Learning:}\\
In many practical applications, we only have imperfect knowledge of $f$, and therefore it is impossible to know the \textit{best possible} parameters $\alpha, \beta$.
Often, we must commit to a choice of sub-optimal parameters that are chosen after only a small glimpse of $f$.
In this project, you will estimate and explore the extra error incurred when using an imperfect choice of parameters.\\
\noindent\textit{Terminology:} For each value of $x$, the \textit{bias} of a model is defined as the the difference between $\bar{g}(x;\alpha,\beta)$, the expected (or average) value of $g(x;\alpha,\beta)$, and the true value of $f$.
\begin{equation}
\text{Bias}(x) = \bar{g}(x;\alpha,\beta) - f(x)
\end{equation}
\noindent\textit{Terminology:} For each value of $x$, the \textit{variance} of a model is defined as the expected (or average) value of the square of the difference between the value of $g(s;\alpha,\beta)$ and the expected value of $g(x;\alpha,\beta)$.
\begin{equation}
\text{Variance}(x) = \bigg\langle \big( g(x;\alpha,\beta) - \bar{g}(x;\alpha,\beta)\big)^2 \bigg\rangle
\end{equation}
Implement the following process to estimate how well a linear function that is parameterized by only two random data points can be used to model $f$.
\begin{enumerate}\setlength{\itemsep}{0pt}
  \item Create the variable \texttt{xx} by discretizing the interval $0 \leq x \leq 1$ into 101 points. That is, set \texttt{xx}$ = (0,\, 0.01,\, 0.02,\, \dots,\, 1)$.  Plot $f(x)$ in blue.
    \item Generate two random data points on $x_1, x_2 \in [0,1]$.
    \item Evaluate $y_1 =f(x_1)$ and $y_2 = f(x_2)$.  Plot $(x_1,y_1)$ and $(x_2,y_2)$ as two orange dots on the same figure as $f$.
    \item Using \textbf{only} $(x_1,y_1)$ and $(x_2,y_2)$, compute a `best guess' for parameters $\alpha$ and $\beta$.
    \item Use your best guess parameters $\alpha$ and $\beta$ to evaluate the linear model $g(x;\alpha, \beta)$ for each $x \in \texttt{xx}$ and record the results. Plot $g(x;\alpha,\beta)$ in orange on the same figure.
\end{enumerate}
Repeat steps 1-5 several times and save their plots.\\

Repeat steps 1-5 1000 times (without plotting) and record the results in a $101 \times 1000$ array.
For each $x$, compute $\bar{g}(x)$, the average value predicted by the linear model $g(x;\alpha,\beta)$ and plot it on the same figure as a plot of $f(x)$.
On the same figure, plot $g_{5}(x)$, $g_{25}(x)$, $g_{75}(x)$ and $g_{95}(x)$, the 5\textsuperscript{th}, 25\textsuperscript{th}, 75\textsuperscript{th} and 95\textsuperscript{th} quantiles of the values predicted by the linear model.  \\

\noindent\textit{Error Reduction for Learning:}\\
Someone proposes that the variance is so big that it makes the linear model unusable.
They suggest that we should approximate $f$ by a constant function  $h(x;\alpha)$.
Repeat the steps above to estimate the bias and the variance of the constant model. \\