\section{Preliminaries and exploratory data analysis}

Consider some data from Manly (you will need to get the file from the portal), OR try this with an example from Johnson and Wichern.   A link to their data is given in the portal.

\begin{Schunk}
\begin{Sinput}
> econ.df <- read.csv("econ.csv", row.names = 1)
> econ <- econ.df[, -10]
> econ.cor <- cor(econ)
> econ.ev <- eigen(econ.cor)
\end{Sinput}
\end{Schunk}

\section{PRINCIPAL COMPONENT EXTRACTION}

Extracts loadings, uniqueness and residuals for 4 factors based on principal component extraction.

\begin{Schunk}
\begin{Sinput}
> loadings4 <- matrix(0, 9, 4)
> for (i in 1:4) {
+     loadings4[, i] <- sqrt(econ.ev$values[i]) * econ.ev$vectors[,
+         i]
+ }
> LLt <- loadings4 %*% t(loadings4)
> unique <- diag(econ.cor - LLt)
> error <- econ.cor - (LLt + unique)
\end{Sinput}
\end{Schunk}

\begin{itemize}
  \item Now consider how to amend this code to give you a two or three factor solution.
\end{itemize}

\subsection{RESULTS FROM PRINCIPAL COMPONENT EXTRACTION}

The following labels the loadings for ease of use

\begin{Schunk}
\begin{Sinput}
> row.names(loadings4) <- row.names(econ.cor)
\end{Sinput}
\end{Schunk}
The following gives us the values

\begin{Schunk}
\begin{Sinput}
> loadings4
> unique
> error
\end{Sinput}
\end{Schunk}

The following extracts the eigenvalues, and then calculates the proportion and cumulative proportion of variance explained by each component (for 9 variables)

\begin{Schunk}
\begin{Sinput}
> econ.ev$values
> econ.ev$values/0.09
> cumsum(econ.ev$values/0.09)
\end{Sinput}
\end{Schunk}

The following calculates the communalities

\begin{Schunk}
\begin{Sinput}
> apply(loadings4^2, 1, sum)
\end{Sinput}
\end{Schunk}

The following carries out a rotation (you should also consider promax rotation)

\begin{Schunk}
\begin{Sinput}
> varimax(loadings4)
\end{Sinput}
\end{Schunk}

\section{PRINCIPAL FACTORING WITH ITERATION}

This is rather crude code, but it lets you see what's going on (i.e. try this once and then forget it)

Firstly, an initial estimate of the uniquenesses (can you explain what's going on here?)

\begin{Schunk}
\begin{Sinput}
> r2s <- vector("numeric", 9)
> for (i in 1:9) {
+     y <- econ[, i]
+     x <- econ[, -i]
+     mod <- lm(y ~ as.matrix(x))
+     r2s[i] <- summary(mod)$r.squared
+ }
> unique <- diag(1 - r2s)
> diag(unique)
\end{Sinput}
\end{Schunk}

The following step requires to be repeated until the values of the uniquenesses and the loadings converge (10 - 20 runs will do for now).   Type this code in a text editor and paste it all in as one (when you have it working).

\begin{Schunk}
\begin{Sinput}
> new <- econ.cor - unique
> new.ev <- eigen(new)
> loadings4pf <- matrix(0, 9, 4)
> for (i in 1:4) {
+     loadings4pf[, i] <- sqrt(new.ev$values[i]) * new.ev$vectors[,
+         i]
+ }
> LLt <- loadings4pf %*% t(loadings4pf)
> unique.f <- econ.cor - LLt
> diag(unique) <- diag(unique.f)
> diag(unique)
> loadings4pf
\end{Sinput}
\end{Schunk}

Repeat that until you get convergence in the loadings estimates.   If
you get convergence in the loadings estimates (one of the pitfalls of
this technique)

\section{MAXIMUM LIKELIHOOD SOLUTION}

For our purposes, we will trust R to do the optimisation.  In other
words, to carry out the analysis we need one line of code!!!!!!!!!!!!)

\begin{Schunk}
\begin{Sinput}
> econ.fact <- factanal(econ, factors = 4, rotation = "none")
> econ.fact
\end{Sinput}
\end{Schunk}

Extract the loadings, calculate the uniquenesses, calculate the residuals

\begin{Schunk}
\begin{Sinput}
> loadml <- loadings(econ.fact)
> loadml
> class(loadml) <- "matrix"
> uniqueml <- econ.fact$uniquenesses
> resid <- econ.cor - (loadml %*% t(loadml) + diag(uniqueml))
> resid
\end{Sinput}
\end{Schunk}

Calculate the communalities

\begin{Schunk}
\begin{Sinput}
> apply(loadml^2, 1, sum)
\end{Sinput}
\end{Schunk}

\section{PLOTTING THE ROTATIONS}

In case it helps, you could try plotting the rotations (the code here
illustrates the pcfa method), with numbers
for the unrotated loadings and letters for the rotated loadings.

\begin{Schunk}
\begin{Sinput}
> plot(loadings4[, c(1:2)], pch = as.character(c(1:9)), xlab = expression(paste(gamma,
+     "1")), ylab = expression(paste(gamma, "2")), main = "First and second loadings",
+     xlim = c(-1, 1), ylim = c(-1, 1))
> points(varimax(loadings4)$loadings[, c(1:2)], pch = letters[c(1:9)],
+     col = "red")
> abline(h = 0)
> abline(v = 0)
\end{Sinput}
\end{Schunk}
\includegraphics{week11fa-plotloadings}

\begin{itemize}
\item Do this for maximum likelihood method
\end{itemize}

\begin{itemize}
  \item How do you interpret the loadings??????
    \end{itemize}

\section{Summary}

\fbox{\parbox[c]{0.9\textwidth}{\color{blue}
We have left factor analysis until last - you should be very clear that despite some superficial similarities it is a very different technique to p.c.a.   By the end of this week we should:

\begin{itemize}
\item Understand the role of factor analysis as a means of explaining the relationships between variables in a given dataset
\item Be able to interpret the results (loadings) of a particular factor analysis, aided or not by rotations
\item To explain the difference between p.c.a. based and maximum likelihood based methods of factor analysis
  \item To interpret the ``signficance'' of solutions of different size, and to justify the use of a particular size solution in all cases
 \end{itemize}

}}


\documentclass[twoside]{MATH77}
\usepackage{multicol}
\usepackage[fleqn,reqno,centertags]{amsmath}
\begin{document}
\begmath 2.13 Inverse Error Function and Inverse \hbox{Complementary Error Function}

\silentfootnote{$^\copyright$1997 Calif. Inst. of Technology, \thisyear \ Math \`a la Carte, Inc.}

\subsection{Purpose}

Given $x$, this pair of subprograms computes the value $t$ such that%
\begin{equation*}
x=\text{erf}(t)=\frac 2{\sqrt{\pi }}\int_0^te^{-s^2}ds
\end{equation*}
for the Inverse Error Function, or%
\begin{equation*}
x=\text{erfc}(t)=\frac 2{\sqrt{\pi }}\int_t^\infty e^{-s^2}ds
\end{equation*}
for the Inverse Complementary Error Function.

Procedures in Chapter~15.2 may be used to calculate the inverse of
the Gaussian or normal probability integral.

Reference \cite{ams55:erf} provides further discussion of the properties
of the error function.

\subsection{Usage}

\subsubsection{Program Prototype, Single Precision}

\begin{description}
\item[REAL]  \ {\bf X, SERFI, SERFCI, T}
\end{description}

Assign a value to X and use one of the following function references.

To compute the Inverse Error Function:
$$
\fbox{{\bf T = SERFI(X)}}
$$
To compute the Inverse Complementary Error Function:
$$
\fbox{{\bf T = SERFCI(X)}}
$$

\subsubsection{Argument Definitions}

\begin{description}
\item[X]  \ [in] Argument of function. Require $-1<\text{X}<1$ for SERFI, $0<%
\text{X}<2$ for SERFCI.
\end{description}

\subsubsection{Modification for Double Precision}

For double precision usage change the REAL type statement to DOUBLE
PRECISION and change the function names to DERFI and DERFCI respectively.

\subsection{Examples and Remarks}

See DRDERFI and ODDERFI for an example of the usage of these subprograms.

$|\frac{\text{d}}{\text{dx}}\text{erf}^{-1}(x)|$ and
$|\frac{\text{d}}{\text{dx}}\text{erfc}^{-1}(x)|$ are $\frac{\sqrt{\pi}}2
e^{x^2}$.  Therefore, the relative accuracy to be expected in evaluating
these functions decreases rapidly as $x$ increases.  For example, when using
IEEE single-precision arithmetic, one should expect no more than one digit of
the result to be correct when $x > \approx 3.6$

\subsection{Functional Description}

The computer approximations for these functions were developed by A.
Strecok, \cite{Strecok:1968:OCI}, using Chebyshev polynomial expansions.

These subprograms were tested on the IBM PC/AT, which uses IEEE
arithmetic with precision, $\rho \approx 1.19 \times 10^{-7}$ in single precision and
$\rho \approx 2.22 \times 10^{-16}$ in double precision.  We checked how well the
Error Function subprograms (SERF and SERFC or DERF and DERFC) and the
present subprograms are inverses. Two sets of relative error tests were
performed:
\begin{gather*}
\varepsilon _1 = |\erf(\erfi(x))/x - 1|\ /\ \rho,\ \ \text{and}\\
\varepsilon _2 = |\erfi(\erf(x))/x - 1|\ /\ \rho
\end{gather*}

and similarly for the complementary functions. The latter test does not
measure relative error precisely for the complementary functions, and was
not carried out in the range 1.0E$-$35 $\leq x \leq $ 1.0E$-$5 (1.0E$-$300 $%
\leq x \leq $ 1.0E$-$5 in double precision). The results are summarized below.

\begin{tabular}{llll}
 & \bf Argument & \bf Max. & \bf Max.\\
\bf Function & \bf \ \,Interval & \bf \ \ \ $\varepsilon _1$ &
\bf \ \ \ $\varepsilon _2$\\
SERFI & [0.1E$-$2, 0.8] & 0.956 & 0.993\\
 & [0.8, 0.9975] & 0.625 & 1.66\\
 & [.9975, .99999] & 0.0 & 1.50\\
SERFCI & [1.0E$-$35, 1.0E$-$5] & 113\\
 & [1.E$-$5, 2.5E$-$3] & 16.3 & 1.91E+4\\
 & [2.5E$-$3, 0.2] & 14.2 & 86.7\\
 & [0.2, 0.999] & 2.14 & 1.92\\
DERFI & [0.1E$-$2, 0.8] & 0.896 & 0.973\\
 &[0.8, 0.9975] & 0.625 & 1.59\\
 &[.9975, .99999] & 0.0 & 1.50\\
DERFCI & [1.0E$-$300, 1.0E$-$5] & 964\\
 & [1.E$-$5, 2.5E$-$3] & 13.6 & 1.90E+4\\
 &[2.5E$-$3, 0.2] & 8.29 & 64.6\\
 &[0.2, 0.999] & 1.69 & 1.60
\end{tabular}

\bibliography{math77}
\bibliographystyle{math77}

\subsection{Error Procedures and Restrictions}

These subprograms issue an error message and terminate execution if

\hspace{.4in}(a) \ \ X $\leq -1$ or X $\geq 1$ for SERFI, or

\hspace{.4in}(b) \ \ X $\leq 0$ or X $\geq 2$ for SERFCI.

The error message is issued by way of the error message processor at level 0,
and the returned function value is 0.0.  If it is desired, in the event an error
occurs, to issue the error message and then terminate program execution

\hspace{.4in}CALL ERMSET (2)

before invoking SERFI or SERFCI.

See Chapter~19.2 for further description of the error message processor.

\subsection{Supporting Information}

The source language is ANSI Fortran~77.

Subprograms designed and developed by W. V. Snyder, JPL, 1986.

\begin{tabular}{@{\bf}l@{\hspace{5pt}}l}
\bf Entry & \hspace{.35in} {\bf Required Files}\vspace{2pt} \\
DERFCI & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, DERFI, DERM1, DERV1, ERFIN, ERMSG\rule[-5pt]{0pt}{8pt}}\\DERFI & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, DERFI, DERM1, DERV1, ERFIN, ERMSG\rule[-5pt]{0pt}{8pt}}\\SERFCI & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, ERFIN, ERMSG, SERFI, SERM1, SERV1\rule[-5pt]{0pt}{8pt}}\\SERFI & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, ERFIN, ERMSG, SERFI, SERM1, SERV1}\\\end{tabular}

\begcode

\bigskip

\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRDERFI}\vspace{10pt}
\lstinputlisting{\codeloc{derfi}}

\vspace{30pt}\centerline{\bf \large ODDERFI}\vspace{10pt}
\lstset{language={}}
\lstinputlisting{\outputloc{derfi}}

\end{document}


\section{The \minmaxelement algorithm}
\Label{sec:minmaxelement}

The \minmaxelement algorithm in the
\cxx Standard Library \cite[\S 28.7.8]{cxx-17-draft} searches
\emph{both} the minimum \emph{and} the maximum in a sequence.
The signature of our version of \minelement reads:

\begin{lstlisting}[style = acsl-block]

        size_type_pair minmax_element(const value_type* a, size_type n);
\end{lstlisting}

Note that \minmaxelement returns a \emph{pair} of indices (see \S\ref{sec:makepair}).
This pair contains the \emph{first} position where the minimum occurs in
the sequence \inl{a[0..n-1]} and the \emph{last} position where maximum occurs.

The properties of the index for the minimum value are the same as the properties
of \specref{minelement}.
However, the properties of the index that marks the maximum
element, are slightly different from the properties of \specref{maxelement}.
The \maxelement algorithm returns the position of the \emph{first} occurrence of
the maximum element if it occurs multiple times in the sequence. The
\minmaxelement algorithm returns the position of the last occurrence of the
maximum element.

%\clearpage

\subsection{Formal specification of \minmaxelement}

The following listing shows the acsl specification of \specref{minmaxelement}.
Note that we use the predicates \logicref{StrictLowerBound} and
\logicref{StrictUpperBound} in order to express that the algorithm returns
the positions of both the \emph{first minimum} and the \emph{last maximum}.
We also use the predicates \logicref{MinElement} and \logicref{MaxElement}.
Thus reflects of course the use of this predicates for the algorithms
\specref{minelement} and \specref{maxelement}.

\input{Listings/minmax_element.h.tex}

The specification is similar to the specifications of \minelement and
\maxelement. The only difference lies in the postcondition \inl{last}. Here the
postcondition states that after the position of the maximum element there is no
value greater or equal the maximum element. This differs from the specification
of \maxelement, where the first occurrence of the maximum value has to be
returned.

%\clearpage

\subsection{Implementation of \minmaxelement}

The implementation of \implref{minmaxelement} uses the auxiliary
function \specref{makepair} to construct a pair of indices.
We will focus on the loop invariant \inl{last}, because it is the
only loop invariant that differs from the implementations of \implref{minelement} and
\implref{maxelement}.

\input{Listings/minmax_element.c.tex}

As already mentioned we had to alter the range for the predicate
\logicref{StrictUpperBound} to fit into the property of returning
the last maximum position that occurred.


\documentclass{article}

\usepackage{mathrsfs,amssymb,amsmath}

\begin{document}

\section{Prove that for a finite topological spaces, discrete is equivalent to Hausdorf}

Assume a finite topological space X is discrete. Then given two points $x_1$ and $x_2$ there are the open sets $\{x_1\}$ and $\{x_2\}$ that are non-intersecting, thus X is Hausdorf. Note the assumption that the space was finite was not needed.

Assume a finite topological space X is Hausdorf. Pick an arbitrary point $x$. For every $x_i \in X$ where $1 \le i \le n=|X|$, there exists an open set containing $x$ that does not contain $x_i$, call it $U_i$. Take $U=\bigcap_{i=0}^n U_i$. $U = \{x\}$ since it is in every set $U_i$ but every other point in $X$ is not in at least one $U_i$. $U$ is open, since it is the finite intersection of open sets.

Since $x$ was chosen arbitrarily, the singleton set around every point is open. Thus since the union of open sets is open, any subset of $X$ is open, thus $X$ is discrete.

\end{document}


\documentclass[twoside]{MATH77}
\usepackage{multicol}
\usepackage[fleqn,reqno,centertags]{amsmath}
\begin{document}
\begmath 5.3 Eigenvalues of an Unsymmetric Matrix

\silentfootnote{$^\copyright$1997 Calif. Inst. of Technology, \thisyear \ Math \`a la Carte, Inc.}

\subsection{Purpose}

Compute all eigenvalues of a real N $\times $ N unsymmetric matrix $A$. Some or all
of the eigenvalues may be complex.

\subsection{Usage}

\subsubsection{Program Prototype, Single Precision}

\begin{description}
\item[REAL]  \ {\bf A}(LDA, $\geq $ N) \ [LDA$\geq $N]{\bf , VR}($\geq $ N)%
{\bf , VI}($\geq $N)

\item[INTEGER]  \ {\bf LDA, N, IFLAG}($\geq $ N)
\end{description}

Assign values to A(,), LDA, and N.
$$
\fbox{{\bf CALL SEVUN(A, LDA, N, VR, VI, IFLAG)}}
$$
Results are returned in VR(), VI(), IFLAG(1). The contents of A(,) will be
modified.

\subsubsection{Argument Definitions}

\begin{description}
\item[A(,), LDA, N]  \ A(,) is [inout], LDA and N are [in]. On entry A(,) must
contain the N $\times $ N matrix $A$ whose eigenvalues are to be computed. The
integer LDA is the dimension of the first subscript of the array A(,).
Require LDA $\geq $ N. On return the contents of A(,) will be modified.

\item[VR(), VI()]  \ [out] The subroutine will store the $J^{th}$ eigenvalue
in VR($J$) and VI($J$), $J =1$, ..., N. The real part is stored in
VR($J$), and
the imaginary part in VI($J$). If the $J^{th}$ eigenvalue is real VI($J$) will
be zero. The eigenvalues will be sorted so that VR(1) $\leq $ VR(2) $\leq $
... $\leq $ VR(N), and if VR($J$) = VR($J$+1) for some $J$ then $|$VI$(J)|
\leq |$VI$(J+1)|.$

Complex eigenvalues will occur in conjugate pairs. Such pairs will be stored
in adjacent locations with the eigenvalue having positive imaginary part
preceding its conjugate partner.

\item[IFLAG()]  \ [out, scratch] The N-array IFLAG() will be used as INTEGER
working space. In addition, the first location, IFLAG(1), will be used to
pass information back to the user as follows:

\begin{itemize}
\item[= 1]  If successful and all eigenvalues are real.

\item[= 2]  If successful and some eigenvalues are complex.
\end{itemize}

See Section E for use of IFLAG(1) in error conditions.
\end{description}

\subsubsection{Modifications for Double Precision}

Change SEVUN to DEVUN, and the REAL type statement to DOUBLE PRECISION.

\subsection{Examples and Remarks}

{\bf Example~1:} Define
\begin{equation*}
A=\left[
\begin{array}{rrr}
-954 & -464 & -2088 \\
792 & 387 & 1728 \\
264 & 128 & 579
\end{array}
\right] .
\end{equation*}
The eigenvalues of $A$ are $\lambda _1 = 3$, $\lambda _2 = 3$, $\lambda _3
= 6$.  This example was constructed by using the matrix called $A^{-1}$ in
Example~3.1, page~29, of \cite{Gregory:1969:ACM} as the matrix of
eigenvectors.  It illustrates the case of a double eigenvalue having a
full set of eigenvectors.

{\bf Example~2:} Define
\begin{equation*}
B=\left[
\begin{array}{ccc}
4 & 1 & 1 \\
2 & 4 & 1 \\
0 & 1 & 4
\end{array}
\right] .
\end{equation*}
This is Example~5.2 of page~82 of~\cite{Gregory:1969:ACM}.  The
eigenvalues are $\lambda _1 = 3 $, $\lambda _2 = 3$, and $\lambda _3 = 6$.
The matrix is defective in that there is only a one-dimensional space of
eigenvectors associated with the eigenvalue~3.

{\bf Example 3:} Define
\begin{equation*}
C=\left[
\begin{array}{rrr}
8 & -1 & -5 \\
-4 & 4 & -2 \\
18 & -5 & -7
\end{array}
\right] .
\end{equation*}
This is Example~5.4 of page~84 of \cite{Gregory:1969:ACM}.  The
eigenvalues are $\lambda _1 = 1 $, $\lambda _2 = 2 + 4i$, and $\lambda _3
= 2 - 4i.$

The demonstration program DRSEVUN below applies SEVUN to compute eigenvalues
for the above three matrices. Results are in the file ODSEVUN.

\subsection{Functional Description}

Given an N $\times $ N real unsymmetric matrix $A$ there exists an N $\times $ N
nonsingular matrix $C$ such that the matrix
\begin{equation*}
U=C^{-1}AC
\end{equation*}
is N $\times $ N upper triangular. The matrices $C$ and $U$ may be complex. The
diagonal elements of $U$ are called the eigenvalues of $A$. This set of N
numbers is uniquely determined by $A$ although $C$ and $U$ are not unique.
Note that $\lambda $ is an eigenvalue of $A$ if and only if $A - \lambda I$
is singular.

This subroutine SEVUN was developed using the subroutines BALANC, ELMHES,
and HQR from the EISPACK package of eigenvalue-eigenvector subroutines,
\cite{Smith:1974:MER}. The Fortran subroutines in EISPACK
are based directly on the earlier set of Algol procedures described in
\cite{Wilkinson:1971:HAC}.

Subroutine SEVUN first calls SEVBH which consists of the two EISPACK
subroutines, BALANC and ELMHES.

BALANC applies similarity permutations to isolate eigenvalues available by
inspection, if any. It then applies diagonal similarity scaling to balance
the size of the matrix elements%
\begin{equation*}
B=D^{-1}P^TAPD.
\end{equation*}
ELMHES reduces $B$ to upper Hessenberg form using stabilized elementary
transformations%
\begin{equation*}
H=G^{-1}BG.
\end{equation*}
The remainder of SEVUN consists of a minor modification of HQR
which applies the QR algorithm to $H$. This is an iterative process that
reduces $H$ to a real nearly-upper-triangular matrix $R$%
\begin{equation*}
R=Q^THQ.
\end{equation*}
The matrix $R$ has a mixture of single elements and $2 \times 2$ blocks on its
diagonal and is otherwise upper triangular.

The eigenvalues of $A$ are the single diagonal elements of $R$ along with the
eigenvalues of the $2 \times 2$ blocks on the diagonal of $R$. These latter
eigenvalues are computed by direct formulas.

SEVUN reorders the eigenvalues to achieve the ordering described
previously in Section~B.

\bibliography{math77}
\bibliographystyle{math77}

\subsection{Error Procedures and Restrictions}

If N $\leq 0$ or if there is convergence failure in the QR algorithm the
error processing subroutine ERMSG of Chapter 19.2 will be called with an
error level 0 to print an error message. Upon return, IFLAG(1) = 3~or~4 to
indicate N $\leq  0$ or convergence failure, respectively.

In these error conditions all computed eigenvalues should be regarded as
invalid.

\subsection{Supporting Information}

The source language is ANSI Fortran~77.

\begin{tabular}{@{\bf}l@{\hspace{5pt}}l}
\bf Entry & \hspace{.35in} {\bf Required Files}\vspace{2pt} \\
DEVUN & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, DEVBH, DEVUN, ERFIN, ERMSG\rule[-5pt]{0pt}{8pt}}\\
SEVUN & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, ERFIN, ERMSG, SEVBH, SEVUN}\\
\end{tabular}

The EISPACK package of Fortran subroutines was acquired at JPL from Argonne
National Laboratories where it was developed with financial support from the
AEC and the NSF. The subroutine SEVUN was written by F. T. Krogh, JPL,
October~1991.

\begcode

\medskip\
\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRSEVUN}\vspace{10pt}
\lstinputlisting{\codeloc{sevun}}

\vspace{30pt}\centerline{\bf \large ODSEVUN}\vspace{10pt}
\lstset{language={}}
\lstinputlisting{\outputloc{sevun}}
\end{document}


% This is a model template for the solutions in computational science. You can find a very useful documentation for LaTeX in Finnish at ftp://ftp.funet.fi/pub/TeX/CTAN/info/lshort/finnish/ or in English at ftp://ftp.funet.fi/pub/TeX/CTAN/info/lshort/english/. The section List of mathematical symbols in Chapter 3 is especially useful for the typesetting of mathematical formulas.

% Compile the document to PDF by command 'pdflatex model.tex' in the terminal. The command must be run twice for the references in the text to be correct.

\documentclass[a4paper,11pt]{article}
\usepackage[utf8]{inputenc}
% This includes letters such as � and �
\usepackage[T1]{fontenc}
% Use here 'Finnish' for Finnish hyphenation. You may have to compile the code twice after the change.
\usepackage[english]{babel}
\usepackage{graphicx}
% Some math stuff
\usepackage{amsmath,amsfonts,amssymb,amsbsy,commath,booktabs,hyperref}
% This is just to include the urls
\usepackage{hyperref,subcaption}
\usepackage[margin=2cm]{geometry}

\setlength{\parindent}{0mm}
\setlength{\parskip}{1.0\baselineskip}

\usepackage{listings}
\usepackage{color}
\usepackage{pdfpages}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}

\lstset{frame=tb,
	language=Python,
	aboveskip=3mm,
	belowskip=3mm,
	showstringspaces=false,
	columns=flexible,
	basicstyle={\tiny\ttfamily},
	numbers=none,
	numberstyle=\tiny\color{gray},
	keywordstyle=\color{blue},
	commentstyle=\color{dkgreen},
	stringstyle=\color{mauve},
	breaklines=true,
	breakatwhitespace=true,
	tabsize=4
}

\begin{document}

\title{Becs-114.1100 Computational Science -- exercise round 8} % Replace the exercise round number
\author{Kunal Ghosh, 546247} % Replace with your name and student number
\maketitle
\section{Solution to Question 3}
\subsection{Using Box-Muller algorithm}\label{prob3a}
In this exercise we use the Box-Muller algorithm to generate two gaussing distributed random variables. We generate $10^5$ random number pairs and plot them separately in two histograms shown below.\\
Here we have used the standard python random number generator which uses \textbf{Mersenel Primes} to generate a random number in the range of [0,1]
\begin{figure}[ht]
    \begin{subfigure}{.5 \textwidth}
	\centering
    \includegraphics[scale=0.40]{figure3_y1s.png}
    \caption{$Y_1$ as per $\sqrt(-2\log(x1))*\cos(2*\pi*x2)$}
	\label{fig:Y1}
\end{subfigure}
    \begin{subfigure}{.5 \textwidth}
	\centering
    \includegraphics[scale=0.40]{figure3_y2s.png}
    \caption{$Y_1$ as per $\sqrt(-2\log(x1))*\cos(2*\pi*x2)$}
	\label{fig:Y1}
\end{subfigure}
    \caption{This plot shows Y1 and Y2, two random variables generated from two uniformly distributed random variables(x1 and x2) and transformed as showing under the respective graphs.}
\end{figure}

These are two zero mean and unit variance gaussian distributed random numbers.

The corresponding python code can be found at \ref{code:problem3a}
\section{Solution 4a}\label{prob4a}

In this section we compare the 1st, 2nd and 3rd moments of two random number generators. LCG(128, 0, 509) and Mersene Twister. We compare these to the true moments defined by:
\begin{equation}
    \langle x^{k} \rangle = \frac{1}{(k+1)}
\end{equation}
In the above equation k is the moment. i.e. k = 1 gives us the 1st moment, k = 2 gives us the second moment and so on.
We calculate the moments for different random number generators by generating a sequence of $N$ random numbers from the generator and using the formula below.
\begin{equation}
    \langle x^k \rangle = \frac{1}{N}\sum_{i=1}^{N}x_{i}^{k}
\end{equation}

In this experiment we are trying to see whether the the two random number generators are truly uniform. If they are then their moments must be close to the true moments of the uniform distribution.

\begin{table}[ht]
\centering
\label{1moment}
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{N} & \textbf{LCG(128,0,509)}&\textbf{Mersene Twister}&\textbf{True Moment} \\ \hline
10 & 0.0264746457362 & 0.518080782167 & 0.5 \\
100 & 0.00240679565587 & 0.474926738729 & 0.5 \\
1000 & 0.000238511281212 & 0.494459184012 & 0.5 \\
10000 & 2.38296599591e-05 & 0.493928278685 & 0.5 \\
100000 & 2.38275152683e-06 & 0.500197518937 & 0.5 \\
1000000 & 2.38273008204e-07 & 0.499728476939 & 0.5 \\
10000000 & 2.38272793759e-08 & 0.500054377307 & 0.5 \\
\hline
\end{tabular}
\caption{Values for the 1st Moment}
\end{table}

\begin{table}[ht]
\centering
\label{1moment}
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{N} & \textbf{LCG(128,0,509)}&\textbf{Mersene Twister}&\textbf{True Moment} \\ \hline
10 & 0.00377304392308 & 0.310275753489 & 0.333333333333 \\
100 & 0.000343003993013 & 0.311212396032 & 0.333333333333 \\
1000 & 3.3991386695e-05 & 0.328312667816 & 0.333333333333 \\
10000 & 3.39607913874e-06 & 0.326612059681 & 0.333333333333 \\
100000 & 3.39577348856e-07 & 0.333317261858 & 0.333333333333 \\
1000000 & 3.39574292657e-08 & 0.333169017655 & 0.333333333333 \\
10000000 & 3.3957398704e-09 & 0.3333760948 & 0.333333333333 \\
\hline
\end{tabular}
\caption{Values for the 2nd Moment}
\end{table}

\begin{table}[ht]
\centering
\label{1moment}
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{N} & \textbf{LCG(128,0,509)}&\textbf{Mersene Twister}&\textbf{True Moment} \\ \hline
10 & 0.000640566707312 & 0.199266027703 & 0.25 \\
100 & 5.82333370283e-05 & 0.228229711742 & 0.25 \\
1000 & 5.77087123704e-06 & 0.246069547559 & 0.25 \\
10000 & 5.7656769335e-07 & 0.243561537242 & 0.25 \\
100000 & 5.76515801739e-08 & 0.249764456768 & 0.25 \\
1000000 & 5.76510613091e-09 & 0.249906635608 & 0.25 \\
10000000 & 5.76510094232e-10 & 0.250034371239 & 0.25 \\
\hline
\end{tabular}
\caption{Values for the 3rd Moment}
\end{table}
We can see that in all the 3 cases above, the LCG deviates from each of the moments quite a bit as we take more samples from the random number generator. However, the Mersent Twister stays quite close to the true moment even when we take a lot of samples from the generator which means that the random number it generates are close to uniformly distributed.
% \begin{figure}[ht]
% 	\center
%     \includegraphics[scale=0.75]{matlabS_dash.png}
%     \caption{Plot showing the first derivative of the natural cubic interpoland S'(X) as calculated using Matlab. Values of $S'(X_i)$ for $X_i$ in the range of (min(knot values), max(knot values)) are shown as red circles.}
% 	\label{fig:sdash_matlab}
% \end{figure}
% The corresponding plots created by matlab are much more smoother because matlab implements an additional constraint that the third derivatives of the piecewise polynomials are also equal at the second and the last knots. This are apparently called the "Not-A-Knot" end conditions. This is only done when the length of \textbf{t} and \textbf{y} are same.
%
% Natural splines are a good choice only when the functions have 0 second derivative at the end points. I read it up from here. \url{http://www.mathworks.com/matlabcentral/newsreader/view_thread/172988} would really appreciate some more information about why using the Not-A-Knot condition is better.
% \\
% The corresponding matlab code can be found at \ref{code:problem2b}

The corresponding python code can be found at \ref{code:problem4a}

\section{Solution 4b}\label{prob4b}
In this solution we are trying to validate the central limit theorem. \textit{An average of measured quantities, taken from the same distribution (which can be any distribution), will asymptotically follow the normal distribution.}
Here we draw random numbers from the uniform distribution and since the generator generates values close to the uniform distribution the mean of the values as we draw more samples tends to the normal distribution.
\begin{figure}[ht]
    \begin{subfigure}{.5 \textwidth}
	\centering
    \includegraphics[scale=0.45]{fig4b.png}
    \caption{N = 100, m = 10000}
	\label{fig:clt_10000}
    \end{subfigure}
    \begin{subfigure}{.5 \textwidth}
	\centering
    \includegraphics[scale=0.45]{fig4b_100000.png}
    \caption{N = 100, m = 100000}
	\label{fig:clt_100000}
    \end{subfigure}
    \caption{Figure validating the central limit theorem in the figure to the left we have plotted 10000 means of $\langle x^2 \rangle$100 random numbers and can see that the figure is almost like the normal distribution. This is confirmed as the number of means are increased to 100,000 of 100 random numbers and the graph looks very much like the normal distribution. Bins = 100 interval = [0.2,0.5] this interval is chosen because the mean of the values is around 0.33333}
\end{figure}

The corresponding python code can be found at \ref{code:problem4b}
\clearpage
\section{Appendix A}\label{code:problem3a}
Python source code for \ref{prob3a}.
{\footnotesize
\begin{lstlisting}
from __future__ import division
import random
import math
import pylab as pl

pi    = math.pi
log   = math.log
cos   = math.cos
sin   = math.sin
sqrt  = math.sqrt
floor = math.floor

def get_gauss_random():
    x1 = random.random()
    x2 = random.random()

    y1 = sqrt(-2*log(x1))*cos(2*pi*x2)
    y2 = sqrt(-2*log(x1))*sin(2*pi*x2)

    # We don't multiply by sigma
    # implying the value is unit variance

    # Since are random numbers are uniform
    # between 0 and 1, we have zero mean.

    # Hence our distributionis zero mean and
    # unit variance.
    return y1,y2

def get_bin(yi, ymin, ymax, Nbin):
    val = ((yi - ymin) / (ymax - ymin)) * Nbin
    return floor(val)

if __name__ == '__main__':
    seed = 7777
    random.seed(seed)
    Nbin = 100
    bins = [0] * Nbin
    ymin, ymax = -5, 5
    binVals = []
    ybins = []
    for _ in xrange(100000):
        y = get_gauss_random()
        ybins.append(y)
        #binVals.append(get_bin(yi, ymin, ymax, Nbin))
        # binNum = get_bin(yi, ymin, ymax, Nbin)
        # bins[binNum]+=1
    y1s,y2s = zip(*ybins)
    pl.figure()
    pl.grid()
    pl.hist(y1s,bins=Nbin,range=(ymin,ymax))
    #pl.hist(binVals, bins=Nbin)
    pl.xlabel("Bins")
    pl.title("Y1s")
    pl.savefig("figure3_y1s.png")
    pl.figure()
    pl.grid()
    pl.hist(y2s,bins=Nbin,range=(ymin,ymax))
    #pl.hist(binVals, bins=Nbin)
    pl.xlabel("Bins")
    pl.title("Y2s")
    pl.savefig("figure3_y2s.png")
    pl.show()
\end{lstlisting}
}
\clearpage
\section{Appendix B}\label{code:problem4a}
Python source code \ref{prob4a}.
{\footnotesize
\begin{lstlisting}

from __future__ import division
import random
import math
import pylab as pl

pi    = math.pi
log   = math.log
cos   = math.cos
sin   = math.sin
sqrt  = math.sqrt
floor = math.floor

def lcg(x):
    # Implementing lcg(a,b,m)
    # a = 128 b = 0 m = 509
    a = 128
    b = 0
    m = 509
    randVal = (a*x + b)%m
    return randVal / m # scale the rand num to be in 0,1 range

def get_moment(rand_nums, m):
    return sum([x**m for x in rand_nums])/len(rand_nums)

if __name__ == '__main__':
    seed = 7777
    random.seed(seed)
    randLcg = lcg(seed)
    lcgs = [randLcg]
    srgl = []
    N = 10**7
    moment = [[],[],[]]
    for _ in xrange(N):
        randLcg = lcg(randLcg)
        lcgs.append(randLcg)
        srgl.append(random.random())
    for m in [1,2,3]:
        trueMoment = 1/(m+1)
        print "M = {}".format(m)
        for N in [10, 100, 1000, 10000, 100000, 1000000, 10000000]:
            lcgMoment = get_moment(lcgs[1:N], m)
            srglMoment = get_moment(srgl[1:N], m)
            moment[m-1].append((lcgMoment, srglMoment, trueMoment))
            print("{} & {} & {} & {} \\\\".format(N,lcgMoment,srglMoment,trueMoment))

\end{lstlisting}
}
\section{Appendix C}\label{code:problem4b}
Python source code for \ref{prob4b}.
{\footnotesize
\begin{lstlisting}
from __future__ import division
import random
import math
import pylab as pl

pi    = math.pi
log   = math.log
cos   = math.cos
sin   = math.sin
sqrt  = math.sqrt
floor = math.floor

def get_moment(rand_nums, m):
    return sum([x**m for x in rand_nums])/len(rand_nums)

if __name__ == '__main__':
    seed = 7777
    random.seed(seed)
    N = 100
    m = 10000
    means = []
    for _ in xrange(m):
        temp = 0
        for _ in xrange(N):
            temp += random.random() ** 2
        means.append(temp/N)
    pl.hist(means, bins=100, range=(0.2,0.5))
    pl.savefig("fig4b.png")
    pl.show()
\end{lstlisting}
}
\end{document}


\documentclass[11pt]{article}

\usepackage[margin=1in]{geometry}

\usepackage{mathpazo}

\usepackage{amsmath}

\newcommand{\numpy}{{\sffamily NumPy}}

\usepackage[defaultsans]{cantarell} %% Use option ``defaultsans'' to use cantarell as sans serif only
\usepackage[T1]{fontenc}

% this needs to be done before the fncychap style, since that will
% redo the chapter stuff
\usepackage{sectsty}
\allsectionsfont{\sffamily}

\usepackage{graphicx}

% prefer PDF to PNG
\DeclareGraphicsExtensions{%
  .pdf}

\begin{document}

\begin{center}
{\bfseries \sffamily \LARGE Making Sense of Power Spectra} \\
Michael Zingale
\end{center}

We are interested in computing the numerical power spectrum of a
discretely-sampled three-dimensional dataset.  These notes serve
to verify the procedure by starting with a real-space
function with a known analytic power spectrum and computing the
power spectrum of this test function.

Consider a function that is a superposition of sines with different
wavelengths.  The power will be encoded in the amplitude of the sines.
Now imagine this function discretely sampled at $N$ points.  For a domain of
length $L$ the highest possible wavenumber is:
\begin{equation}
k_\mathrm{max} = \frac{1}{\Delta x}
\end{equation}
with $\Delta x = L/N$.  However, for real-valued data, Nyquist sampling
means that $k_\mathrm{max}/2$ is the meaningful maximum.
The lowest wavenumber is:
\begin{equation}
k_\mathrm{min} = \frac{1}{L}
\end{equation}
Figure~\ref{fig:wavenumbers} shows two different sine modes.

\suppressfloats

\begin{figure}[t]
\centering
\includegraphics[width=0.75\linewidth]{wavenumber}
\caption{\label{fig:wavenumbers} The $k = 1/L$ (top) and $k = 4/L$ (bottom)
modes.}
\end{figure}

The definition of the Fourier transform is:
\begin{equation}
\mathcal{F}(k) = \int_{-\infty}^{\infty} f(x) e^{-2\pi i x k} dx
\end{equation}
with the inverse transform as
\begin{equation}
f(x) = \int_{-\infty}^{\infty} \mathcal{F}(x) e^{2\pi i x k} dk
\end{equation}
However, we need to work with discretely-sampled data, so we will need
the {\em discrete Fourier transform}, which is efficiently implemented by
the FFT algorithm.

\section*{\numpy\ details}

The \numpy\ FFTs have the discrete form:
\begin{equation}
\mathcal{F}_k = \sum_{m=0}^{N-1} f_m \exp \left \{ -2\pi i \frac{mk}{N} \right \}
\end{equation}
and the inverse is
\begin{equation}
f_m = \frac{1}{N} \sum_{k=0}^{N-1} \mathcal{F}_k \exp \left \{ 2\pi i \frac{mk}{N} \right \}
\end{equation}
We see from the inverse that if we want to know the amplitude of a single
mode at a given spatial location (denoted by index $m$), that $\mathcal{F}/N$
is the normalized amplitude, i.e., we can think of this expansion as:
\begin{equation}
f_m = \sum_{k=0}^{N-1} \left (\frac{\mathcal{F}_k}{N} \right )
    \exp \left \{ 2\pi i \frac{mk}{N} \right \}
\end{equation}

The {\tt rfftfreq()} function returns the wavenumbers as
\begin{equation}
k = \left \{ 0, \frac{1}{N}, \frac{2}{N}, \ldots, \frac{N-2}{2N}, \frac{1}{2} \right \}
\end{equation}
(for even $n$, and in the mode where we do not specify the sample spacing).
We convert these into a physical quantity (with units of cm$^{-1}$) by
multiplying by $1/\Delta x$, giving the set:
\begin{equation}
k = \left \{ 0, \frac{1}{L}, \frac{2}{L}, \ldots, \frac{1}{2\Delta x} \right \}
\end{equation}

\numpy\ does not have a real-valued FFT function for 3-d data, so we
need to use the {\tt fftn} function.  Because $N$ real numbers results
in $2N$ complex numbers, only 1/2 of the transformed data (in each
dimension) will be unique.  In 3-d this means we consider only an
octant of data.  In practice, this means keeping the positive
frequencies and dropping the negative ones (this is what the {\tt
  rfftfreq} function does for us).

\section*{Power}

For a continuous function, $f(x)$, the ``energy'' is defined as
\begin{equation}
E = \int_{-\infty}^\infty |f(x)|^2 dx
\end{equation}
Using Parseval's theorem, we have
\begin{equation}
E = \int_{-\infty}^\infty |\mathcal{F}(k)|^2 dk
\end{equation}
%
So when we compute the power spectrum, we will be looking at
$|\mathcal{F}(k)|^2$. \\

\noindent [see ``spectral density'' Wikipedia] \\

The discrete version of Parseval's theorem in 1-d is
\begin{equation}
\sum_{n=0}^{N-1} | f(n) |^2 = \frac{1}{N} \sum_{k=0}^{N-1} | \mathcal{F}(k) |^2
\end{equation}

\section*{One-dimensional Example}

The idea of the power spectrum is very straightforward in 1-d.  To illustrate
this we will create a function in real space that we expect to have a
power spectrum scaling as:
\begin{equation}
P \sim k^{-\eta}
\end{equation}
This is accomplished by setting the amplitude of each mode to yield
this power.  Our function is:
\begin{equation}
f(x) = \sum_{m=1}^{N_\mathrm{modes}} a_m \sin \left ( \frac{2\pi m x}{L}\right )
\end{equation}
where $L$ is the width of the physical domain, and
\begin{equation}
a_m = \left (A_0 k^{-\eta} \right )^{1/2}
\end{equation}
This function is shown in Figure~\ref{fig:phi1d}.

\begin{figure}[t]
\centering
\includegraphics[width=0.5\linewidth]{phi_1d}
\begin{minipage}[b]{0.45\linewidth}
\caption{\label{fig:phi1d} The real-space, one-dimensional function $f(x)$,
for the case $N_\mathrm{modes} = 16$, with $A_0 = 1000$ and $\eta = -4$.\newline}
\end{minipage}
\end{figure}

We then transform this function, denoting the transform as $\mathcal{F}(k)$.  Plotting
$|\mathcal{F}(k)|^2$ vs.\ $k$ gives the power spectrum.  Figure~\ref{fig:ps1d} shows
the result, together with the expected power-law scaling, $P$, shown as the
dashed line.

\begin{figure}[t]
\centering
\includegraphics[width=0.5\linewidth]{ps1d}
\begin{minipage}[b]{0.45\linewidth}
\caption{\label{fig:ps1d} The power spectrum of $f(x)$, compared to the
expected result (dashed line).\newline}
\end{minipage}
\end{figure}

See {\tt ps1d.py} for the implementation of this test.

\section*{Three-dimensional Case}

In 3-d, things are more complex---traditionally we express the power
spectrum by integrating over spherical shells with a radius $k$.  The
power spectrum is defined as
\begin{equation}
E(k) = \frac{1}{\Omega} \int_{S(k)} \mathcal{F}(k) \mathcal{F}^\star(k) dS
\end{equation}
where $\Omega$ is the volume in physical space, and the integral is done
over the surface $S(k)$, defined by $|\vec{k}| = k$.  We can
write $dS = 4\pi k^2 dk$ and integrate along radial $|\vec{k}|$.

In discrete form, we bin the $k$ in our transformed space into a
number of uniformly spaced bins, $\kappa_l$, where the index
$l$ denote the bin boundaries, with $\kappa_0$ the leftmost bin boundary.
We also denote the bin centers as $\bar{\kappa}_l$ such that
\begin{equation}
\bar{\kappa}_l = \frac{1}{2} ( \kappa_l + \kappa_{l+1} )
\end{equation}
Our power spectrum becomes
\begin{equation}
E(\bar{\kappa}_l) = \sum_{\kappa_{l} \le |\vec{k}| < \kappa_{l+1}}
   \mathcal{F}(k) \mathcal{F}^\star(k)
\end{equation}
where the sum is over all the zones in the $k_x \times k_y \times k_z$
cube whose wavenumber magnitude, $k$, falls into our bin,
$[\kappa_l,\kappa_{l+1})$.  As we go to higher wavenumbers, the number
  of zones will increase as $k^2$---this captures the $dS$ scaling in
  the continuous-form of the power spectrum.

Again, to test the algorithm, we define a test function and set the power at
different wavenumbers {\em in physical space}, and then do the transform
and power spectrum to see if we get the expected result.  We will design
our function to have the same scaling as the 1-d case, $P \sim k^{-\eta}$.
Our test function
is:
\begin{equation}
f(x,y,z) = \sum_{m=1}^{N_\mathrm{modes}}
              \sum_{n=1}^{N_\mathrm{modes}}
              \sum_{p=1}^{N_\mathrm{modes}}
     a_{m,n,p} \sin \left ( \frac{2\pi m x}{L_x} +
                            \frac{2\pi n y}{L_y} +
                            \frac{2\pi p z}{L_z} \right )
\end{equation}
where we can identify the wavenumbers in each direction as
\begin{equation}
k_m = \frac{m}{L_x} \, ; \quad
k_n = \frac{n}{L_y} \, ; \quad
k_p = \frac{p}{L_z} \,
\end{equation}
This is equivalent to defining a vector wavenumber
\begin{equation}
\vec{k} = k_m \hat{x} + k_n \hat{y} + k_p \hat{z}
\end{equation}
and expressing a single component as
\begin{equation}
f_{m,n,p} = a_{m,n,p} \sin(2\pi \vec{k}\cdot \vec{x})
\end{equation}

We can set the amplitude of component based on our desired
power-law scaling.  Defining $k = |k|$, and $A_0$ as a
reference amplitude of the power, we take
\begin{equation}
a_{m,n,p}^2 = \frac{A_0 k^{-\eta}}{W(k)} \frac{1}{\Delta S}
\end{equation}
where $W(k)$ is the weight that accounts for the fact that multiple
combinations of $m, n, p$ can result in the same $k$.  Using Iverson bracket
notation, defined as
\begin{equation}
[P] = \begin{cases}
  1 & \text{if P is true} \\
  0 & \text{otherwise}
      \end{cases}
\end{equation}
we find $W(k)$ as
\begin{equation}
W(k) = \sum_{m=1}^{N_\mathrm{modes}}
       \sum_{n=1}^{N_\mathrm{modes}}
       \sum_{p=1}^{N_\mathrm{modes}} [ k_m^2 + k_n^2 + k_p^2 = k^2 ]
\end{equation}

The $\Delta S$ in the weighting reflects the fact that the power
in mode $k$ resides in a spherical shell---we need to divide this
area out so when we integrate over $dS$, we get the correct power
in the shell.  For $|k| \gg \Delta \kappa$, we can take $\Delta S \sim
4\pi k^2$.  But we can do better by recognizing that eventually we
will bin the data up into the bins given by $\{\kappa_l\}$.  If we
denote the bin $\xi$ such that $\kappa_\xi \le k < \kappa_{\xi+1}$,
then we can take
\begin{equation}
\Delta S = 4 \pi \bar{\kappa}_\xi^2
\end{equation}

This normalization should give an amplitude in the power spectrum at the lowest
$k$ that agrees with the corresponding amplitude in real-space for that same $k$.
Figure~\ref{fig:phireal} shows the function in three orthogonal slice planes
through the center of the domain.

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{phi}
\caption{\label{fig:phireal} $f(x,y,z)$ shown in three orthogonal slice
planes through the center of the domain.}
\end{figure}

Since the power spectrum integrated over shells, we don't include the $4\pi$
in the real space amplitude when comparing our power spectrum to the $P \sim k^{-\eta}$
expected result.  Figure~\ref{fig:ps} shows the power spectrum of our function.

\begin{figure}[h]
\centering
\includegraphics[width=0.5\linewidth]{ps}
\begin{minipage}[b]{0.45\linewidth}
\caption{\label{fig:ps} The power spectrum of our function $f(x,y,z)$ compared to
the expected result (dashed line).\newline}
\end{minipage}
\end{figure}

As a final check on the overall normalization, we can use Parseval's theorem
to see if we have the correct amount of energy in the spectrum.  In 3-d,
Parseval's theorem says:
\begin{equation}
E = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} |f(x,y,z)|^2 dx dy dz =
  \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty}  |\mathcal{F}(k_x,k_y,k_z)|^2 dk_x dk_y dk_z
\end{equation}
or in discrete form:
\begin{equation}
\sum_{j=0}^{N_x} \sum_{k=0}^{N_y} \sum_{l=0}^{N_z}
   | f(x_j, y_k, z_l) |^2 = \frac{1}{N_x N_y N_z}
 \sum_{k_x=0}^{N_x-1} \sum_{k_y=0}^{N_y-1} \sum_{k_z=0}^{N_z-1}
   | \mathcal{F}(k_x, k_y, k_z) |^2
\end{equation}
We can express the integral over $d^3k$ in terms of the spherical $|k|$ as
\begin{equation}
E = 4\pi \int_{-\infty}^{\infty} |\mathcal{F}(k)|^2 k^2 dk = \int_{-\infty}^{\infty} E(k) dk
\end{equation}
where we substitute in the power spectrum, $E(k)$.  This means that if we integrate up our
power spectrum, we should get the same energy as the integral
of $|f(x,y,z)|^2$ over physical space.  We need to keep track of the normalizations---in
creating the power spectrum, we used $\mathcal{F}/N^3$, so our discrete sum over the energy
spectrum will have a $N^{-6}$ normalization, while the discrete form of Parseval's theorem
has an $N^{-3}$ normalization.

\end{document}


\section{Orthogonal projections and Fourier series}

\begin{outcome}
  \begin{enumerate}
  \item Compute the orthogonal projection of a vector onto a subspace.
  \item Find the least squares approximation of a function by a
    polynomial of a given degree.
  \item Calculate the generalized Fourier series of a vector.
  \item Use Fourier series to approximate a function by polynomials or
    trigonometric functions.
  \end{enumerate}
\end{outcome}

We now reconsider a problem that we briefly encountered, only in the
context of $\R^3$, in Section~\ref{sec:planes}: how to find the
shortest distance between a point and a subspace. The method we used
in Section~\ref{sec:planes} (see
Example~\ref{exa:shortest-distance-plane}) relies on the existence of
normal vectors and does not generalize beyond $\R^3$. The following
proposition gives a much better method for solving this problem,
provided that we have an orthogonal basis of the subspace.

\begin{proposition}{Orthogonal projection onto a subspace}{projection-subspace}
  Let $V$ be an inner product space, and let $W$ be a subspace of\/
  $V$. Assume $\set{\vect{u}_1,\ldots,\vect{u}_k}$ is an orthogonal
  basis of\/ $W$, and $\vect{v}\in V$ is any vector. Then the
  following vector $\vect{v}'$ is the element of $W$ that is closest
  to $\vect{v}$, i.e., such that $\norm{\vect{v}-\vect{v}'}$ is as
  small as possible.
  \begin{equation*}
    \vect{v}' =
    \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}\,\vect{u}_1
    + \frac{\iprod{\vect{u}_2,\vect{v}}}{\iprod{\vect{u}_2,\vect{u}_2}}\,\vect{u}_2
    + \ldots
    + \frac{\iprod{\vect{u}_k,\vect{v}}}{\iprod{\vect{u}_k,\vect{u}_k}}\,\vect{u}_k.
  \end{equation*}
  Moreover, the vector $\vect{v}-\vect{v}'$ is orthogonal to $W$.
  \begin{center}
    \begin{tikzpicture}[x={(1cm,-0.1cm)},y={(0.4cm,0.5cm)},z={(0cm,1cm)}]
      \filldraw[draw=red!80,fill=red!10](-4,-4,0) -- (4,-4,0) -- (4,4,0) -- (-4,4,0) -- cycle;
      \path[red] (4.5,0,0) node[right] {$W$};
      \draw[thin,red!50] (-2,-4,0) -- (-2,4,0);
      \draw[thin,red!50] (0,-4,0) -- (0,4,0);
      \draw[thin,red!50] (2,-4,0) -- (2,4,0);
      \draw[thin,red!50] (-4,-2,0) -- (4,-2,0);
      \draw[thin,red!50] (-4,0,0) -- (4,0,0);
      \draw[thin,red!50] (-4,2,0) -- (4,2,0);
      \draw[->,thick,blue!80!black](0,0,0) -- node[below, pos=0.45]{$\vect{u}_1$} (2,0,0);
      \draw[->,thick,blue!80!black](0,0,0) -- node[left, pos=0.55]{$\vect{u}_2$} (0,2,0);
      \draw[->,thick,green!50!black](0,0,0) -- node[above, pos=0.7] {$\vect{v}$} (3,2.5,2);
      \draw[->,thick,green!50!black](0,0,0) -- node[above, pos=0.7] {$\vect{v}'$} (3,2.5,0);
      \draw[dashed](3,2.5,0) -- (3,2.5,2);
      \fill (0,0,0) circle [radius=2.2pt] node [left=3pt] {$\vect{0}$};
      \fill (3,2.5,2) circle [radius=2.2pt];
      \fill (3,2.5,0) circle [radius=2.2pt];
    \end{tikzpicture}
  \end{center}
  The vector $\vect{v}'$ is called the \textbf{orthogonal projection
    of $\vect{v}$ onto $W$}%
  \index{orthogonal projection!onto subspace}%
  \index{projection!onto subspace}. We also say that $\vect{v}'$ is
  the \textbf{best approximation}%
  \index{approximation} of $\vect{v}$ in $W$.
\end{proposition}

\begin{proof}
  In case $\vect{v}\in W$, then by
  Proposition~\ref{prop:fourier-coefficients}, we have
  \begin{equation*}
    \vect{v} =
    \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}\,\vect{u}_1
    + \frac{\iprod{\vect{u}_2,\vect{v}}}{\iprod{\vect{u}_2,\vect{u}_2}}\,\vect{u}_2
    + \ldots
    + \frac{\iprod{\vect{u}_k,\vect{v}}}{\iprod{\vect{u}_k,\vect{u}_k}}\,\vect{u}_k,
  \end{equation*}
  and therefore $\vect{v}'=\vect{v}$. In that case
  $\norm{\vect{v}-\vect{v}'}=0$, which is clearly as small as
  possible, and $\vect{v}-\vect{v}'=\vect{0}$ is orthogonal to $W$, so
  we are done.

  Now assume that $\vect{v}\not\in W$. Let
  $W'=\sspan\set{\vect{u}_1,\ldots,\vect{u}_k,\vect{v}}$. Apply the
  Gram-Schmidt procedure to the $k+1$ vectors
  $\vect{u}_1,\ldots,\vect{u}_k,\vect{v}$ to obtain an orthogonal
  basis of $W'$. Since the first $k$ vectors are already orthogonal,
  the Gram-Schmidt procedure does not change them, and we therefore
  obtain an orthogonal basis of $W'$ of the form
  $\vect{u}_1,\ldots,\vect{u}_k,\vect{u}_{k+1}$. Since $\vect{v}\in
  W'$, we can write $\vect{v}$ as a linear combination of these basis
  vectors, i.e., $\vect{v}=a_1\vect{u}_1 + \ldots + a_k\vect{u}_k +
  a_{k+1}\vect{u}_{k+1}$. Moreover, by
  Proposition~\ref{prop:fourier-coefficients}, we know that each $a_j$
  is the corresponding Fourier coefficient of $\vect{v}$, i.e., $a_j =
  \frac{\iprod{\vect{u}_j,\vect{v}}}{\iprod{\vect{u}_j,\vect{u}_j}}$.

  Now consider any $\vect{w}\in W$. Since
  $\set{\vect{u}_1,\ldots,\vect{u}_k}$ is a basis of $W$, we can write
  $\vect{w} = x_1\vect{u}_1 + \ldots + x_k\vect{u}_k$. Using
  Proposition~\ref{prop:norm-orthogonal}, we have
  \begin{eqnarray*}
    \norm{\vect{v}-\vect{w}}^2
    &=& \norm{(a_1-x_1)\vect{u}_1 + \ldots + (a_k-x_k)\vect{u}_k + a_{k+1}\vect{u}_{k+1}}^2 \\
    &=& (a_1-x_1)^2\norm{\vect{u}_1}^2 ~+~ \ldots ~+~ (a_k-x_k)^2\norm{\vect{u}_k}^2 ~+~ a_{k+1}^2\norm{\vect{u}_{k+1}}^2.
  \end{eqnarray*}
  Here, $a_1,\ldots,a_{k+1}$ are the fixed coordinates of $\vect{v}$,
  and $x_1,\ldots,x_k$ depend on $\vect{w}$.  Therefore,
  $\norm{\vect{v}-\vect{w}}$ takes its smallest value when $x_j=a_j$,
  for $j=1,\ldots,k$. So the minimum value of
  $\norm{\vect{v}-\vect{w}}$ occurs when
  \begin{equation*}
    \vect{w}
    ~~=~~ a_1\vect{u}_1 + \ldots + a_k\vect{u}_k
    ~~=~~
    \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}\,\vect{u}_1
    + \frac{\iprod{\vect{u}_2,\vect{v}}}{\iprod{\vect{u}_2,\vect{u}_2}}\,\vect{u}_2
    + \ldots
    + \frac{\iprod{\vect{u}_k,\vect{v}}}{\iprod{\vect{u}_k,\vect{u}_k}}\,\vect{u}_k
    ~~=~~ \vect{v}'.
  \end{equation*}
  This is what had to be shown. Finally, to show that
  $\vect{v}-\vect{v}'$ is orthogonal to $W$, note that
  $\vect{v}-\vect{v}' = a_{k+1}\vect{u}_{k+1}$, which is orthogonal to
  each $\vect{u}_1,\ldots,\vect{u}_k$, and therefore to $W$.
\end{proof}

\begin{example}{Orthogonal projection onto a subspace}{projection-subspace}
  Consider $\R^4$ with the usual dot product. Let
  $W=\sspan\set{\vect{u}_1,\vect{u}_2}$, where
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 1 \\ 1 \\ 2 \\ 0 \end{mymatrix}, \quad
    \vect{u}_2 = \begin{mymatrix}{r} -1 \\ -1 \\ 1 \\ 3 \end{mymatrix}.
  \end{equation*}
  Note that $\vect{u}_1$ and $\vect{u}_2$ are orthogonal. Let
  \begin{equation*}
    \vect{v} = \begin{mymatrix}{r} 1 \\ 2 \\ 3 \\ 4 \end{mymatrix}.
  \end{equation*}
  Find the best approximation of $\vect{v}$ in $W$.
\end{example}

\begin{solution}
  We calculate $\iprod{\vect{u}_1,\vect{v}}=9$,
  $\iprod{\vect{u}_1,\vect{u}_1}=6$, $\iprod{\vect{u}_2,\vect{v}}=12$,
  and $\iprod{\vect{u}_2,\vect{u}_2}=12$. Therefore, by
    Proposition~\ref{prop:projection-subspace}, the desired vector is
  \begin{equation*}
    \vect{v}'
    ~=~ \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}\,\vect{u}_1
    + \frac{\iprod{\vect{u}_2,\vect{v}}}{\iprod{\vect{u}_2,\vect{u}_2}}\,\vect{u}_2
    ~=~ \frac{9}{6} \begin{mymatrix}{r} 1 \\ 1 \\ 2 \\ 0 \end{mymatrix}
    + \frac{12}{12} \begin{mymatrix}{r} -1 \\ -1 \\ 1 \\ 3 \end{mymatrix}
    ~=~ \begin{mymatrix}{c} 1/2 \\ 1/2 \\ 4 \\ 3 \end{mymatrix}.
  \end{equation*}
\end{solution}

Note how straightforward the calculations in this example are. All we
had to do is calculate a few inner products. It is not even necessary
to solve a system of equations. Such is the power of orthogonal bases.

The next example shows that we can use exactly the same method to find
approximations%
\index{approximation!of functions} of functions.

\begin{example}{Approximating a function by a polynomial}{projection-subspace-polynomial}
  Let $V=C[-1,1]$ be the vector space of continuous functions, with the
  inner product given by
  \begin{equation*}
    \iprod{f,g} = \int_{-1}^{1} f(x)g(x)\,dx.
  \end{equation*}
  Consider the function $f\in V$ given by
  \begin{equation*}
    f(x)
    ~=~ 1-\abs{x}
    ~=~ \begin{cases}
      1+x & \text{if $x<0$,} \\
      1-x & \text{if $x\geq 0$.}
    \end{cases}
  \end{equation*}
  Find the closest approximation to $f$ by a polynomial of degree at
  most 2. Graph both $f$ and the approximating polynomial.
\end{example}

\begin{solution}
  Let $W=\sspan\set{1,x,x^2}$ be the subspace of $V$ consisting of
  polynomials of degree at most 2. What we are looking for is an
  element $g\in W$ such that $\norm{f-g}$ is as small as possible.  We
  can solve this problem using
  Proposition~\ref{prop:projection-subspace}.

  First we need an orthogonal basis for $W$. We found such an
  orthogonal basis in Example~\ref{exa:legendre-polynomials}, namely
  the Legendre polynomials $p_0(x) = 1$, $p_1(x) = x$, and
  $p_2(x) = x^2-\frac{1}{3}$.  By
  Proposition~\ref{prop:projection-subspace}, the desired
  approximation $g\in W$ is given by:
  \begin{equation*}
    g
    ~=~
    \frac{\iprod{p_0,f}}{\iprod{p_0,p_0}}\,p_0
    + \frac{\iprod{p_1,f}}{\iprod{p_1,p_1}}\,p_1
    + \frac{\iprod{p_2,f}}{\iprod{p_2,p_2}}\,p_2.
  \end{equation*}
  We already computed the inner products $\iprod{p_0,p_0}=2$,
  $\iprod{p_1,p_1}=\frac{2}{3}$, and $\iprod{p_2,p_2}=\frac{8}{45}$ in
  Example~\ref{exa:legendre-polynomials}. We calculate the remaining
  inner products:
  \begin{eqnarray*}
    \iprod{p_0,f}
    &=& \int_{-1}^{1} 1\cdot f(x)\,dx
        ~=~ \int_{-1}^{0} 1\cdot (1+x)\,dx
        +   \int_{0}^{1} 1\cdot (1-x)\,dx
        ~=~ \frac{1}{2} + \frac{1}{2}
        ~=~ 1, \\
    \iprod{p_1,f}
    &=& \int_{-1}^{1} x\cdot f(x)\,\,dx
        ~=~ \int_{-1}^{0} x\cdot (1+x)\,\,dx
        +   \int_{0}^{1} x\cdot (1-x)\,\,dx
        ~=~ -\frac{1}{6} + \frac{1}{6}
        ~=~ 0, \\
    \iprod{p_2,f}
    &=& \int_{-1}^{1} (x^2-\frac{1}{3})\cdot f(x)\,dx \\
    &=& \int_{-1}^{0} (x^2-\frac{1}{3})\cdot (1+x)\,dx
        +   \int_{0}^{1} (x^2-\frac{1}{3})\cdot (1-x)\,dx
        ~=~ - \frac{1}{12} - \frac{1}{12}
        ~=~ -\frac{1}{6}.
  \end{eqnarray*}
  Therefore,
  \begin{eqnarray*}
    g
    &=&
    \frac{\iprod{p_0,f}}{\iprod{p_0,p_0}}\,p_0
        + \frac{\iprod{p_1,f}}{\iprod{p_1,p_1}}\,p_1
        + \frac{\iprod{p_2,f}}{\iprod{p_2,p_2}}\,p_2
    \\
    &=& \frac{1}{2}\,p_0
        + \frac{0}{2/3}\,p_1
        - \frac{1/6}{8/45}\,p_2
    \\
    &=& \frac{1}{2}
        - \frac{15}{16}(x^2-\frac{1}{3})
    \\
    &=& \frac{13}{16} - \frac{15}{16}x^2.
  \end{eqnarray*}
  The following graph shows the function $f(x)$ as well as the
  polynomial $g(x)$:
  \begin{center}
    \begin{tikzpicture}[domain=-1:1, scale=3, samples=25]
      \def\fA#1{1}
      \def\fC#1{(abs((#1)^2)-1/3)}
      \draw[thick,color=fourier0] (-1,0) -- (0,1) -- (1,0);
      \draw[thick,color=fourier2] plot (\x,{0.5*\fA{\x} - 0.9375*\fC{\x}});
      \path[color=fourier2t] (0.6,0.6) node[right]{$g(x)= \frac{13}{16} - \frac{15}{16}x^2$};
      \path[color=fourier0t] (0.2,0.9) node[right]{$f(x) = 1-\abs{x}$};
      \draw[->] (-1.5,0) -- (1.5,0) node[right] {$x$};
      \draw[->] (0,-0.3) -- (0,1.1);
      \draw (1,0) -- (1,-0.05) node[below] {$1$};
      \draw (-1,0) -- (-1,-0.05) node[below] {$-1$};
    \end{tikzpicture}
  \end{center}
\end{solution}

When approximating a function $f$ by a polynomial, as in the last
example, there is no need to stop with polynomials of degree 2. We can
also ask what is the best approximation%
\index{approximation!of functions} of $f$ by a polynomial of
degree 3, of degree 4, of degree 5, and so on. By increasing the
degree of the polynomials, we get better and better approximations to
$f$. This leads us to the concept of a generalized Fourier series.

\begin{definition}{Generalized Fourier series}{generalized-fourier-series}
  Let $V$ be an inner product space and let
  $\set{\vect{u}_1,\vect{u}_2,\vect{u}_3,\ldots}$ be an infinite
  orthogonal set of vectors.  Let $\vect{v}\in V$. The
  \textbf{generalized Fourier series}%
  \index{generalized Fourier series}%
  \index{Fourier series!generalized}%
  \index{series!Fourier series} of $\vect{v}$ (with respect to
  $\vect{u}_1,\vect{u}_2,\vect{u}_3,\ldots$) consists of the following
  sequence of vectors:
  \begin{eqnarray*}
    \vect{v}_1
    &=& \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}\,\vect{u}_1, \\
    \vect{v}_2
    &=& \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}\,\vect{u}_1
        + \frac{\iprod{\vect{u}_2,\vect{v}}}{\iprod{\vect{u}_2,\vect{u}_2}}\,\vect{u}_2, \\
    \vect{v}_3
    &=& \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}\,\vect{u}_1
        + \frac{\iprod{\vect{u}_2,\vect{v}}}{\iprod{\vect{u}_2,\vect{u}_2}}\,\vect{u}_2
        + \frac{\iprod{\vect{u}_3,\vect{v}}}{\iprod{\vect{u}_3,\vect{u}_3}}\,\vect{u}_3, \\
    \vect{v}_4
    &=& \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}\,\vect{u}_1
        + \frac{\iprod{\vect{u}_2,\vect{v}}}{\iprod{\vect{u}_2,\vect{u}_2}}\,\vect{u}_2
        + \frac{\iprod{\vect{u}_3,\vect{v}}}{\iprod{\vect{u}_3,\vect{u}_3}}\,\vect{u}_3
        + \frac{\iprod{\vect{u}_4,\vect{v}}}{\iprod{\vect{u}_4,\vect{u}_4}}\,\vect{u}_4, \\
    &\vdots        &
  \end{eqnarray*}
  The vectors $\vect{v}_i$ are also called \textbf{generalized Fourier
    approximations}%
  \index{Fourier approximation}%
  \index{approximation!Fourier} of $\vect{v}$.
\end{definition}

By Proposition~\ref{prop:projection-subspace}, we know that each
$\vect{v}_i$ is the best approximation of $\vect{v}$ in the subspace
$\sspan\set{\vect{u}_1,\ldots,\vect{u}_i}$. In particular,
$\norm{\vect{v}-\vect{v}_{i+1}} \leq \norm{\vect{v}-\vect{v}_i}$, so
each $\vect{v}_i$ is potentially a better approximation of $\vect{v}$
than the previous one. In a course on analysis%
\footnote{``Algebra'' is the name for those subjects where all sums
  are finite. This includes, for example, linear algebra and abstract
  algebra. ``Analysis'' is the name for those subjects where sums are
  potentially infinite (and may or may not converge). This includes,
  for example, calculus, complex analysis, and functional analysis.},
you will learn that in many situations, the sequence
$\vect{v}_1,\vect{v}_2,\vect{v}_3,\ldots$ can be shown to converge to
$\vect{v}$.

\begin{example}{Generalized Fourier series}{generalized-fourier-series}
  Proceeding as in Example~\ref{exa:projection-subspace-polynomial},
  find the generalized Fourier approximations of $f(x)=1-\abs{x}$ up
  to degree 8.
\end{example}

\begin{solution}
  We use the Legendre polynomials $p_0,\ldots,p_8$ from
  Section~\ref{sec:gram-schmidt} and calculate the relevant inner
  products, each of which requires solving an integral. As these
  integrals get a bit complicated, it is best to use a computer
  algebra system to compute them.
  \begin{equation*}
    \def\arraystretch{1.4}
    \begin{array}{rcl}
      \iprod{p_0,f} &=& 1, \\
      \iprod{p_1,f} &=& 0, \\
      \iprod{p_2,f} &=& -\frac{1}{6}, \\
      \iprod{p_3,f} &=& 0, \\
      \iprod{p_4,f} &=& \frac{1}{105}, \\
      \iprod{p_5,f} &=& 0, \\
      \iprod{p_6,f} &=& -\frac{1}{924}, \\
      \iprod{p_7,f} &=& 0, \\
      \iprod{p_8,f} &=& \frac{1}{6435}, \\
    \end{array}
    \quad
    \begin{array}{rcl}
      \iprod{p_0,p_0} &=& 2, \\
      \iprod{p_1,p_1} &=& \frac{2}{3}, \\
      \iprod{p_2,p_2} &=& \frac{8}{45}, \\
      \iprod{p_3,p_3} &=& \frac{8}{175}, \\
      \iprod{p_4,p_4} &=& \frac{128}{11025}, \\
      \iprod{p_5,p_5} &=& \frac{128}{43659}, \\
      \iprod{p_6,p_6} &=& \frac{512}{693693}, \\
      \iprod{p_7,p_7} &=& \frac{512}{2760615}, \\
      \iprod{p_8,p_8} &=& \frac{32768}{703956825}. \\
    \end{array}
  \end{equation*}
  We therefore have the following approximations:
  \begin{eqnarray*}
    f_0 &=& f_1 ~~=~~
            \frac{1}{2}\,p_0,
    \\
    f_2 &=& f_3 ~~=~~
            \frac{1}{2}\,p_0
            - \frac{15}{16}\,p_2,
    \\
    f_4 &=& f_5 ~~=~~
            \frac{1}{2}\,p_0
            - \frac{15}{16}\,p_2
            + \frac{105}{128}\,p_4,
    \\
    f_6 &=& f_7 ~~=~~
            \frac{1}{2}\,p_0
            - \frac{15}{16}\,p_2
            + \frac{105}{128}\,p_4
            - \frac{3003}{2048}\,p_6,
    \\
    f_8 &=& f_9 ~~=~~
            \frac{1}{2}\,p_0
            - \frac{15}{16}\,p_2
            + \frac{105}{128}\,p_4
            - \frac{3003}{2048}\,p_6
            + \frac{109395}{32768}\,p_8.
  \end{eqnarray*}
  The following graph shows the function $f$ as well as its
  approximations $f_0$, $f_2$, $f_4$, $f_6$, and $f_8$. It can be seen
  that each successive approximation is closer to the function $f$
  than the previous one.
  \begin{center}
    \begin{tikzpicture}[domain=-1:1, scale=4, samples=25]
      \def\fA#1{1}
      \def\fB#1{(#1)}
      \def\fC#1{(abs((#1)^2)-1/3)}
      \def\fD#1{((#1)^3-3/5*(#1))}
      \def\fE#1{(abs((#1)^4)-6/7*abs((#1)^2)+3/35)}
      \def\fF#1{((#1)^5 - 10/9*(#1)^3+5/21*(#1))}
      \def\fG#1{(abs((#1)^6)-15/11*abs((#1)^4)+5/11*abs((#1)^2)-5/231)}
      \def\fH#1{((429*(#1)^7 - 693*(#1)^5 + 315*(#1)^3 - 35*(#1))/429)}
      \def\fI#1{((6435*abs((#1)^8)-12012*abs((#1)^6)+6930*abs((#1)^4)-1260*abs((#1)^2)+35)/6435)}
      \draw[thick,color=fourier0] (-1,0) -- (0,1) -- (1,0);
      \path[color=fourier0t] (0,1) node[right=2ex] {$f$};
      \draw[thick,color=fourier1] plot (\x,0.5*\fA{\x}) (1,0.5) node[right=1ex,color=fourier1t] {$f_0$};
      \draw[thick,color=fourier2] plot (\x,{0.5*\fA{\x} - 0.9375*\fC{\x}}) (1,-0.18) node[right=1ex,color=fourier2t] {$f_2$};
      \draw[thick,color=fourier4,samples=50] plot (\x,{0.5*\fA{\x} - 0.9375*\fC{\x} + 0.8203125*\fE{\x}}) (1,0.18) node[right=1ex,color=fourier4t] {$f_4$};
      \draw[thick,color=fourier5,samples=50] plot (\x,{0.5*\fA{\x} - 0.9375*\fC{\x} + 0.8203125*\fE{\x} -1.46630859375*\fG{\x}}) (1,-0.06) node[right=1ex,color=fourier5t] {$f_6$};
      \draw[thick,color=fourier6,samples=75] plot (\x,{0.5*\fA{\x} - 0.9375*\fC{\x} + 0.8203125*\fE{\x} -1.46630859375*\fG{\x}+3.338470458984375*\fI{\x}}) (1,0.06) node[right=1ex,color=fourier6t] {$f_8$};
      \draw[->] (-1.5,0) -- (1.5,0) node[right] {$x$};
      \draw[->] (0,-0.3) -- (0,1.1);
      \draw (1,0) -- (1,-0.05) node[below] {$1$};
      \draw (-1,0) -- (-1,-0.05) node[below] {$-1$};
    \end{tikzpicture}
  \end{center}
\end{solution}

\begin{example}{Generalized Fourier series}{generalized-fourier-series2}
  The following graph shows the generalized Fourier approximations of
  the function
  \begin{equation*}
    f(x) ~=~ \begin{cases}
      0 & \mbox{if $x<0$,} \\
      1 & \mbox{if $x\geq 0$}
    \end{cases}
  \end{equation*}
  by polynomials up to degree 7. (Note: the function $f$ is not
  continuous, so not technically an element of $C[-1,1]$, but we
  ignore this here. Instead of the vector space of continuous
  functions, we can work in the vector space of piecewise continuous
  functions).
\end{example}

\begin{center}
  \begin{tikzpicture}[domain=-1:1, scale=4, samples=25]
    \def\fA#1{1}
    \def\fB#1{(#1)}
    \def\fC#1{(abs((#1)^2)-1/3)}
    \def\fD#1{((#1)^3-3/5*(#1))}
    \def\fE#1{(abs((#1)^4)-6/7*abs((#1)^2)+3/35)}
    \def\fF#1{((#1)^5 - 10/9*(#1)^3+5/21*(#1))}
    \def\fG#1{(abs((#1)^6)-15/11*abs((#1)^4)+5/11*abs((#1)^2)-5/231)}
    \def\fH#1{((429*(#1)^7 - 693*(#1)^5 + 315*(#1)^3 - 35*(#1))/429)}
    \def\fI#1{((6435*abs((#1)^8)-12012*abs((#1)^6)+6930*abs((#1)^4)-1260*abs((#1)^2)+35)/6435)}
    \draw[thick,color=fourier0] (-1,0) -- (0,0) -- (0,1) -- (1,1);
    \path[color=fourier0t] (0.1,1) node[above] {$f$};
    \draw[thick,color=fourier1] plot (\x,0.5*\fA{\x}) (1,0.5) node[right=1ex,color=fourier1t] {$f_0$};
    \draw[thick,color=fourier2]     plot (\x,{0.5*\fA{\x} + 0.75*\fB{\x}}) (1,1.24) node[right=1ex,color=fourier2t] {$f_1$};
    \draw[thick,color=fourier4,samples=50]   plot (\x,{0.5*\fA{\x} + 0.75*\fB{\x} - 1.09375*\fD{\x}}) (1,0.82) node[right=1ex,color=fourier4t] {$f_3$};
    \draw[thick,color=fourier5,samples=50]       plot (\x,{0.5*\fA{\x} + 0.75*\fB{\x} - 1.09375*\fD{\x} + 2.70703125*\fF{\x}}) (1,1.10) node[right=1ex,color=fourier5t] {$f_5$};
    \draw[thick,color=fourier6,samples=75]      plot (\x,{0.5*\fA{\x} + 0.75*\fB{\x} - 1.09375*\fD{\x} + 2.70703125*\fF{\x} - 7.855224609375*\fH{\x}}) (1,0.96) node[right=1ex,color=fourier6t] {$f_7$};
    \draw[->] (-1.5,0) -- (1.5,0) node[right] {$x$};
    \draw[->] (0,-0.3) -- (0,1.1);
    \draw (1,0) -- (1,-0.05) node[below] {$1$};
    \draw (-1,0) -- (-1,-0.05) node[below] {$-1$};
  \end{tikzpicture}
\end{center}

Comparing the last two examples, we see that the function in
Example~\ref{exa:generalized-fourier-series2} is much harder to
approximate well by polynomials. This is due to the discontinuity in
the latter function. Nevertheless, the sequence of approximations
eventually converges to $f$.

\begin{example}{Generalized Fourier series}{generalized-fourier-series3}
  The following graph shows the generalized Fourier approximations of
  the function
  \begin{equation*}
    f(x) ~=~ \begin{cases}
      0 & \mbox{if $x<-\frac{1}{2}$,} \\
      x+\frac{1}{2} & \mbox{if $-\frac{1}{2}\leq x<\frac{1}{2}$,} \\
      1 & \mbox{if $x\geq \frac{1}{2}$} \\
    \end{cases}
  \end{equation*}
  by polynomials up to degree 7.
\end{example}

\begin{center}
  \begin{tikzpicture}[domain=-1:1, scale=4, samples=25]
    \def\fA#1{1}
    \def\fB#1{(#1)}
    \def\fC#1{(abs((#1)^2)-1/3)}
    \def\fD#1{((#1)^3-3/5*(#1))}
    \def\fE#1{(abs((#1)^4)-6/7*abs((#1)^2)+3/35)}
    \def\fF#1{((#1)^5 - 10/9*(#1)^3+5/21*(#1))}
    \def\fG#1{(abs((#1)^6)-15/11*abs((#1)^4)+5/11*abs((#1)^2)-5/231)}
    \def\fH#1{((429*(#1)^7 - 693*(#1)^5 + 315*(#1)^3 - 35*(#1))/429)}
    \def\fI#1{((6435*abs((#1)^8)-12012*abs((#1)^6)+6930*abs((#1)^4)-1260*abs((#1)^2)+35)/6435)}
    \draw[thick,color=fourier0] (-1,0) -- (-0.5,0) -- (0.5,1) -- (1,1);
    \path[color=fourier0t] (0.5,1) node[above] {$f$};
    \draw[thick,color=fourier1] plot (\x,0.5*\fA{\x}) (1,0.5) node[right=1ex,color=fourier1t] {$f_0$};
    \draw[thick,color=fourier3]     plot (\x,{0.5*\fA{\x} + 0.6875*\fB{\x}}) (1,1.2) node[right=1ex,color=fourier3t] {$f_1$};
    \draw[thick,color=fourier4,samples=50]   plot (\x,{0.5*\fA{\x} + 0.6875*\fB{\x} - 0.615234375*\fD{\x}}) (1,0.9) node[right=1ex,color=fourier4t] {$f_3$};
    \draw[thick,color=fourier5]       plot (\x,{0.5*\fA{\x} + 0.6875*\fB{\x} - 0.615234375*\fD{\x} + 0.38067626953125*\fF{\x}}) (1,1.0) node[right=1ex,color=fourier5t] {$f_5$};
    \draw[thick,color=fourier6,samples=50]      plot (\x,{0.5*\fA{\x} + 0.6875*\fB{\x} - 0.615234375*\fD{\x} + 0.38067626953125*\fF{\x} + 1.0494089126587303*\fH{\x}}) (1,1.1) node[right=1ex,color=fourier6t] {$f_7$};
    \draw[->] (-1.5,0) -- (1.5,0) node[right] {$x$};
    \draw[->] (0,-0.3) -- (0,1.1);
    \draw (1,0) -- (1,-0.05) node[below] {$1$};
    \draw (-1,0) -- (-1,-0.05) node[below] {$-1$};
  \end{tikzpicture}
\end{center}

So far, we have worked with Legendre polynomials, but there are of
course other examples of orthogonal sets of functions. An important
such set is given by sine and cosine waves. We will see that every
periodic function can be decomposed into sine and cosine waves of
varying frequencies. This was Fourier's original discovery, and the
corresponding series are just known as \textbf{Fourier series}%
\index{Fourier series}%
\index{series!Fourier series} (i.e., not ``generalized'').

\begin{example}{Fourier series}{fourier-series}
  Consider the inner product space $C[0,2\pi]$. Recall from
  Example~\ref{exa:orthogonal-set-sin-cos} that the following
  functions form an orthogonal set:
  \begin{eqnarray*}
    \vect{u}_0 &=& 1 \\
    \vect{u}_1 &=& \sin x \\
    \vect{u}_2 &=& \cos x \\
    \vect{u}_3 &=& \sin 2x \\
    \vect{u}_4 &=& \cos 2x \\
    \vect{u}_5 &=& \sin 3x \\
    \vect{u}_6 &=& \cos 3x \\
    &\vdots&
  \end{eqnarray*}
  Consider the function $f(x) = x - \pi$, where $x\in[0,2\pi]$. Find
  its Fourier series.
\end{example}

\begin{solution}
  Following Definition~\ref{def:generalized-fourier-series}, we must
  calculate a number of inner products. We have
  $\iprod{\vect{u}_0,\vect{u}_0}=2\pi$. Also, for all $i\geq 1$, we
  have $\iprod{\vect{u}_i,\vect{u}_i}=\pi$. We note the following
  antiderivatives, for $k\geq 1$:
  \begin{equation*}
    \int x\sin kx\,dx ~=~ -\frac{x}{k}\cos kx + \frac{1}{k^2}\sin kx, \qquad
    \int x\cos kx\,dx ~=~ \frac{x}{k}\sin kx + \frac{1}{k^2}\cos kx.
  \end{equation*}
  Using these formulas, we can compute the following inner products
  quite easily:
  \begin{eqnarray*}
    \iprod{\vect{u}_0, f}
    &=& \int_{0}^{2\pi} (x-\pi)
        ~~=~~ 0, \\
    \iprod{\vect{u}_1, f}
    &=& \int_{0}^{2\pi} (x-\pi)\sin x
        \,~~=~~ -2\pi, \\
    \iprod{\vect{u}_2, f}
    &=& \int_{0}^{2\pi} (x-\pi)\cos x
        ~~=~~ 0, \\
    \iprod{\vect{u}_3, f}
    &=& \int_{0}^{2\pi} (x-\pi)\sin 2x
        \,~~=~~ -\frac{2\pi}{2}, \\
    \iprod{\vect{u}_4, f}
    &=& \int_{0}^{2\pi} (x-\pi)\cos 2x
        ~~=~~ 0, \\
    \iprod{\vect{u}_5, f}
    &=& \int_{0}^{2\pi} (x-\pi)\sin 3x
        \,~~=~~ -\frac{2\pi}{3}, \\
    \iprod{\vect{u}_6, f}
    &=& \int_{0}^{2\pi} (x-\pi)\cos 3x
        ~~=~~ 0,
  \end{eqnarray*}
  and so on. We therefore find the following Fourier series:
  \begin{eqnarray*}
    f_1 &=& -2 \sin x, \\
    f_3 &=& -2 \sin x - \frac{2}{2}\sin 2x, \\
    f_5 &=& -2 \sin x - \frac{2}{2}\sin 2x - \frac{2}{3}\sin 3x, \\
    f_7 &=& -2 \sin x - \frac{2}{2}\sin 2x - \frac{2}{3}\sin 3x - \frac{2}{4}\sin 4x, \\
    f_9 &=& -2 \sin x - \frac{2}{2}\sin 2x - \frac{2}{3}\sin 3x - \frac{2}{4}\sin 4x - \frac{2}{5}\sin 5x,
  \end{eqnarray*}
  and so on.
\end{solution}

The following graph illustrates the successive approximations of this
Fourier series.
\begin{center}
  \begin{tikzpicture}[domain=0:6*pi, scale=0.8, samples=100]
    \def\fA#1{1}
    \def\fB#1{(sin((#1)/pi*180))}
    \def\fC#1{(cos((#1)/pi*180))}
    \def\fD#1{(sin(2*(#1)/pi*180))}
    \def\fE#1{(cos(2*(#1)/pi*180))}
    \def\fF#1{(sin(3*(#1)/pi*180))}
    \def\fG#1{(cos(3*(#1)/pi*180))}
    \def\fH#1{(sin(4*(#1)/pi*180))}
    \def\fI#1{(cos(4*(#1)/pi*180))}
    \draw[thick,color=fourier0] (0,0) -- (0,-pi) -- (2*pi,pi) -- (2*pi,-pi) -- (4*pi,pi) -- (4*pi,-pi) -- (6*pi, pi) -- (6*pi, 0) (2*pi, pi) node[right,color=fourier0t] {$f$};
    \draw[thick,color=fourier1]                 plot (\x,{((-2)*\fB{\x})}) (pi,0) +(0.6*pi,0.6*pi) node[left=4ex,color=fourier1t] {$f_1$};
    \draw[thick,color=fourier4,samples=200] plot (\x,{((-2)*\fB{\x})+((-1)*\fD{\x})}) (pi,0) +(0.75*pi,0.75*pi) node[left=4ex,color=fourier4t] {$f_3$};
    \draw[thick,color=fourier5,samples=200]       plot (\x,{((-2)*\fB{\x})+((-1)*\fD{\x})+((-2/3)*\fF{\x})}) (pi,0) +(0.9*pi,0.9*pi) node[left=4ex,color=fourier5t] {$f_5$};
    \draw[thick,color=fourier6,samples=300]      plot (\x,{((-2)*\fB{\x})+((-1)*\fD{\x})+((-2/3)*\fF{\x})+((-1/2)*\fH{\x})}) (pi,0) +(1.05*pi,1.05*pi) node[left=4ex,color=fourier6t] {$f_7$};
    \draw[->] (-0.8,0) -- (6*pi+0.8,0) node[right] {$x$};
    \draw[->] (0,-3.5) -- (0,3.5);
    \draw (pi,0) -- (pi,-0.05) node[below] {$\pi$};
    \draw (2*pi,0) -- (2*pi,-0.05) node[below] {$2\pi$};
    \draw (3*pi,0) -- (3*pi,-0.05) node[below] {$3\pi$};
    \draw (4*pi,0) -- (4*pi,-0.05) node[below] {$4\pi$};
    \draw (5*pi,0) -- (5*pi,-0.05) node[below] {$5\pi$};
    \draw (6*pi,0) -- (6*pi,-0.05) node[below] {$6\pi$};
  \end{tikzpicture}
\end{center}
Note that, although the function $f$ is defined on the interval
$[0,2\pi]$, we have extended it periodically for $[2\pi,4\pi]$,
$[4\pi,6\pi]$, and so on. This makes sense because all of the
orthogonal functions $1$, $\sin x$, $\cos x$, $\sin 2x$, and so on,
have period $2\pi$.

We can also illustrate the same information differently, by showing
the individual sine waves making up the wave form of the function
$f(x)$. In the context of audio signals, these sine waves are also
called the \textbf{harmonics}%
\index{harmonic} of the signal.

\begin{center}
  \begin{tikzpicture}[domain=0:6*pi, scale=0.3, samples=100]
    \def\fA#1{1}
    \def\fB#1{(sin((#1)/pi*180))}
    \def\fC#1{(cos((#1)/pi*180))}
    \def\fD#1{(sin(2*(#1)/pi*180))}
    \def\fE#1{(cos(2*(#1)/pi*180))}
    \def\fF#1{(sin(3*(#1)/pi*180))}
    \def\fG#1{(cos(3*(#1)/pi*180))}
    \def\fH#1{(sin(4*(#1)/pi*180))}
    \def\fI#1{(cos(4*(#1)/pi*180))}
    \begin{scope}[yshift=-12cm,xshift=-29cm]
      \draw[thick,color=fourier0] (0,0) -- (0,-pi) -- (2*pi,pi) -- (2*pi,-pi) --
      (4*pi,pi) -- (4*pi,-pi) -- (6*pi, pi) -- (6*pi, 0);
      \draw[->] (-1.5,0) -- (6*pi+1.5,0) node[right]{$~~=~~\left\{\rule{0mm}{3cm}\right.$};
    \end{scope}
    \begin{scope}[yshift=-4cm]
      \draw[thick,color=fourier1]   plot (\x,{((-2)*\fB{\x})}) node[right=2ex,color=fourier1t] {$-\frac{2}{1}\sin(x)$};
      \draw[->] (-1.5,0) -- (6*pi+1.5,0);
    \end{scope}
    \begin{scope}[yshift=-8cm]
      \draw[thick,color=fourier4] plot (\x,{((-1)*\fD{\x})}) node[right=2ex,color=fourier4t] {$-\frac{2}{2}\sin(2x)$};
      \draw[->] (-1.5,0) node[left]{$+$} -- (6*pi+1.5,0);
    \end{scope}
    \begin{scope}[yshift=-12cm]
      \draw[thick,color=fourier5,samples=200]     plot (\x,{((-2/3)*\fF{\x})}) node[right=2ex,color=fourier5t] {$-\frac{2}{3}\sin(3x)$};
      \draw[->] (-1.5,0) node[left]{$+$} -- (6*pi+1.5,0);
    \end{scope}
    \begin{scope}[yshift=-16cm]
      \draw[thick,color=fourier6,samples=200]    plot (\x,{((-1/2)*\fH{\x})}) node[right=2ex,color=fourier6t] {$-\frac{2}{4}\sin(4x)$};
      \draw[->] (-1.5,0) node[left]{$+$} -- (6*pi+1.5,0);
    \end{scope}
    \begin{scope}[yshift=-20cm]
      \path (-1.5,0) node[left]{$+$} node[right]{$\cdots$};
    \end{scope}
  \end{tikzpicture}
\end{center}

\begin{example}{Fourier series}{fourier-series2}
  The following graph shows the first few Fourier approximations of the function
  \begin{equation*}
    f(x) ~=~ \begin{cases}
      x - \frac{\pi}{2} & \mbox{if $0\leq x<\pi$,} \\
      \frac{3\pi}{2} - x & \mbox{if $\pi\leq x\leq 2\pi$.}
    \end{cases}
  \end{equation*}
\end{example}

\begin{center}
  \begin{tikzpicture}[domain=0:6*pi, scale=0.8, samples=100]
    \def\fC#1{(cos((#1)/pi*180))}
    \def\fG#1{(cos(3*(#1)/pi*180))}
    \def\fK#1{(cos(5*(#1)/pi*180))}
    \def\fO#1{(cos(7*(#1)/pi*180))}
    \draw[thick,color=fourier0] (0,-pi/2) -- (pi,pi/2) -- (2*pi,-pi/2) -- (3*pi,pi/2) -- (4*pi,-pi/2) -- (5*pi,pi/2) -- (6*pi, -pi/2) (pi,pi/2) node [above,color=fourier0t] {$f$};
    \draw[thick,color=fourier1]   plot (\x,{((-1.2732395447351623)*\fC{\x})}) (0.6*pi,-pi/2+0.6*pi) node[left=2ex,color=fourier1t] {$f_2$};
    \draw[thick,color=fourier4,samples=200] plot (\x,{((-1.2732395447351623)*\fC{\x})+((-0.14147106052612904)*\fG{\x})}) (0.75*pi,-pi/2+0.75*pi) node[left=2ex,color=fourier4t] {$f_4$};
    \draw[thick,color=fourier5,samples=200]     plot (\x,{((-1.2732395447351623)*\fC{\x})+((-0.14147106052612904)*\fG{\x})+((-0.05092958178940641)*\fK{\x})}) (0.9*pi,-pi/2+0.9*pi) node[left=2ex,color=fourier5t] {$f_6$};
    \draw[thick,color=fourier6,samples=200]    plot (\x,{((-1.2732395447351623)*\fC{\x})+((-0.14147106052612904)*\fG{\x})+((-0.05092958178940641)*\fK{\x})+((-0.025984480504798825)*\fO{\x})}) (1.05*pi,-pi/2+1.05*pi) node[left=2ex,color=fourier6t] {$f_8$};
    \draw[->] (-0.8,0) -- (6*pi+0.8,0) node[right] {$x$};
    \draw[->] (0,-2) -- (0,2);
    \draw (pi,0) -- (pi,-0.05) node[below] {$\pi$};
    \draw (2*pi,0) -- (2*pi,-0.05) node[below] {$2\pi$};
    \draw (3*pi,0) -- (3*pi,-0.05) node[below] {$3\pi$};
    \draw (4*pi,0) -- (4*pi,-0.05) node[below] {$4\pi$};
    \draw (5*pi,0) -- (5*pi,-0.05) node[below] {$5\pi$};
    \draw (6*pi,0) -- (6*pi,-0.05) node[below] {$6\pi$};
  \end{tikzpicture}
\end{center}

\noindent Note how rapidly this Fourier series converges to $f$. The
following image shows the function $f$ (extended periodically outside
the interval $[0,2\pi]$) as a sum of its harmonics. Compared to
Example~\ref{exa:fourier-series}, we see that the higher harmonics
have much smaller amplitudes, which explains the rapid convergence.

\begin{center}
  \begin{tikzpicture}[domain=0:6*pi, scale=0.3, samples=100]
    \def\fC#1{(cos((#1)/pi*180))}
    \def\fG#1{(cos(3*(#1)/pi*180))}
    \def\fK#1{(cos(5*(#1)/pi*180))}
    \def\fO#1{(cos(7*(#1)/pi*180))}
    \begin{scope}[yshift=-12cm,xshift=-29cm]
      \draw[thick,color=fourier0] (0,-pi/2) -- (pi,pi/2) -- (2*pi,-pi/2) -- (3*pi,pi/2) --
      (4*pi,-pi/2) -- (5*pi,pi/2) -- (6*pi, -pi/2);
      \draw[->] (-1.5,0) -- (6*pi+1.5,0) node[right]{$~~=~~\left\{\rule{0mm}{3cm}\right.$};
    \end{scope}
    \begin{scope}[yshift=-4cm]
      \draw[thick,color=fourier1]   plot (\x,{((-1.2732395447351623)*\fC{\x})}) (6*pi,0) node[right=2ex,color=fourier1t] {$-\frac{4}{\pi}cos(x)$};
      \draw[->] (-1.5,0) -- (6*pi+1.5,0);
    \end{scope}
    \begin{scope}[yshift=-8cm]
      \draw[thick,color=fourier4] plot (\x,{((-0.14147106052612904)*\fG{\x})}) (6*pi,0) node[right=2ex,color=fourier4t] {$-\frac{4}{9\pi}\cos(3x)$};
      \draw[->] (-1.5,0) node[left]{$+$} -- (6*pi+1.5,0);
    \end{scope}
    \begin{scope}[yshift=-12cm]
      \draw[->] (-1.5,0) node[left]{$+$} -- (6*pi+1.5,0);
      \draw[thick,color=fourier5]     plot (\x,{((-0.05092958178940641)*\fK{\x})}) (6*pi,0) node[right=2ex,color=fourier5t] {$-\frac{4}{25\pi}\cos(5x)$};
    \end{scope}
    \begin{scope}[yshift=-16cm]
      \draw[->] (-1.5,0) node[left]{$+$} -- (6*pi+1.5,0);
      \draw[thick,color=fourier6]    plot (\x,{((-0.025984480504798825)*\fO{\x})}) (6*pi,0) node[right=2ex,color=fourier6t] {$-\frac{4}{49\pi}\cos(7x)$};
    \end{scope}
    \begin{scope}[yshift=-20cm]
      \path (-1.5,0) node[left]{$+$} node[right]{$\cdots$};
    \end{scope}
  \end{tikzpicture}
\end{center}

\begin{example}{Fourier series}{fourier-series3}
  The following graph shows the first few Fourier approximations of the function
  \begin{equation*}
    f(x) ~=~ \begin{cases}
      1 & \mbox{if $0\leq x<\pi$,} \\
      -1 & \mbox{if $\pi\leq x\leq 2\pi$.}
    \end{cases}
  \end{equation*}
\end{example}

\begin{center}
  \begin{tikzpicture}[domain=0:6*pi, scale=0.8, samples=100]
    \def\fB#1{(sin((#1)/pi*180))}
    \def\fF#1{(sin(3*(#1)/pi*180))}
    \def\fJ#1{(sin(5*(#1)/pi*180))}
    \def\fN#1{(sin(7*(#1)/pi*180))}
    \draw[thick,color=fourier0] (0,0) -- (0,1) -- (pi,1) -- (pi,-1) --
    (2*pi,-1) -- (2*pi,1) --
    (3*pi,1) -- (3*pi,-1) --
    (4*pi,-1) -- (4*pi,1) --
    (5*pi,1) -- (5*pi,-1) --
    (6*pi,-1) -- (6*pi,0) (pi, 1) node[right,color=fourier0t] {$f$};
    \draw[thick,color=fourier1]   plot (\x,{((1.2732395447351623)*\fB{\x})}) (0.4*pi, 1.2) node[above,color=fourier1t] {$f_1$};
    \draw[thick,color=fourier4,samples=200] plot (\x,{((1.2732395447351623)*\fB{\x})+((0.4244131815783881)*\fF{\x})}) (0.6*pi, 1.2) node[above,color=fourier4t] {$f_3$};
    \draw[thick,color=fourier5,samples=200]     plot (\x,{((1.2732395447351623)*\fB{\x})+((0.4244131815783881)*\fF{\x})+((0.25464790894703326)*\fJ{\x})}) (0.8*pi, 1.2) node[above,color=fourier5t] {$f_5$};
    \draw[thick,color=fourier6,samples=300]    plot (\x,{((1.2732395447351623)*\fB{\x})+((0.4244131815783881)*\fF{\x})+((0.25464790894703326)*\fJ{\x})+((0.18189136353359486)*\fN{\x})}) (1.0*pi, 1.2) node[above,color=fourier6t] {$f_7$};
    \draw[->] (-0.8,0) -- (6*pi+0.8,0) node[right] {$x$};
    \draw[->] (0,-2) -- (0,2);
    \draw (pi,0) -- (pi,-0.05) node[below] {$\pi$};
    \draw (2*pi,0) -- (2*pi,-0.05) node[below] {$2\pi$};
    \draw (3*pi,0) -- (3*pi,-0.05) node[below] {$3\pi$};
    \draw (4*pi,0) -- (4*pi,-0.05) node[below] {$4\pi$};
    \draw (5*pi,0) -- (5*pi,-0.05) node[below] {$5\pi$};
    \draw (6*pi,0) -- (6*pi,-0.05) node[below] {$6\pi$};
  \end{tikzpicture}
\end{center}

\noindent Once again, here is an image showing the function $f$ as a
sum of its harmonics:

\begin{center}
  \begin{tikzpicture}[domain=0:6*pi, scale=0.3, samples=100]
    \def\fB#1{(sin((#1)/pi*180))}
    \def\fF#1{(sin(3*(#1)/pi*180))}
    \def\fJ#1{(sin(5*(#1)/pi*180))}
    \def\fN#1{(sin(7*(#1)/pi*180))}
    \begin{scope}[yshift=-12cm,xshift=-29cm]
      \draw[thick,color=fourier0] (0,0) -- (0,1) -- (pi,1) -- (pi,-1) --
      (2*pi,-1) -- (2*pi,1) --
      (3*pi,1) -- (3*pi,-1) --
      (4*pi,-1) -- (4*pi,1) --
      (5*pi,1) -- (5*pi,-1) --
      (6*pi,-1) -- (6*pi,0);
      \draw[->] (-1.5,0) -- (6*pi+1.5,0)
      node[right]{$~~=~~\left\{\rule{0mm}{3cm}\right.$};
    \end{scope}
    \begin{scope}[yshift=-4cm]
      \draw[thick,color=fourier1]   plot (\x,{((1.2732395447351623)*\fB{\x})}) node[right=2ex,color=fourier1t] {$\frac{4}{\pi}\sin(x)$};
      \draw[->] (-1.5,0) -- (6*pi+1.5,0);
    \end{scope}
    \begin{scope}[yshift=-8cm]
      \draw[thick,color=fourier4] plot (\x,{((0.4244131815783881)*\fF{\x})}) node[right=2ex,color=fourier4t] {$\frac{4}{3\pi}\sin(3x)$};
      \draw[->] (-1.5,0) node[left]{$+$} -- (6*pi+1.5,0);
    \end{scope}
    \begin{scope}[yshift=-12cm]
      \draw[thick,color=fourier5,samples=150]     plot (\x,{((0.25464790894703326)*\fJ{\x})}) node[right=2ex,color=fourier5t] {$\frac{4}{5\pi}\sin(5x)$};
      \draw[->] (-1.5,0) node[left]{$+$} -- (6*pi+1.5,0);
    \end{scope}
    \begin{scope}[yshift=-16cm]
      \draw[thick,color=fourier6,samples=200]    plot (\x,{((0.18189136353359486)*\fN{\x})}) node[right=2ex,color=fourier6t] {$\frac{4}{7\pi}\sin(7x)$};
      \draw[->] (-1.5,0) node[left]{$+$} -- (6*pi+1.5,0);
    \end{scope}
    \begin{scope}[yshift=-20cm]
      \path (-1.5,0) node[left]{$+$} node[right]{$\cdots$};
    \end{scope}
  \end{tikzpicture}
\end{center}

\noindent
Watch the video at \url{http://y2u.be/3IAMpH4xF9Q} for a demonstration
of what the functions from
Examples~\ref{exa:fourier-series}--\ref{exa:fourier-series3}, and
their harmonics, sound like as audio signals.


%---------------------------Condition-----------------------------
\section{Condition}

\[
q = \frac{1}{2} \max \left\{  \frac {\normvec{L_0}^2 + \normvec{L_3}^2 } { \alpha_0 },
                        \frac {\normvec{L_1}^2 + \normvec{L_0}^2 } { \alpha_1 },
                        \frac {\normvec{L_2}^2 + \normvec{L_1}^2 } { \alpha_2 },
                        \frac {\normvec{L_3}^2 + \normvec{L_2}^2 } { \alpha_3 }
                \right\}
\]

Note that if $\alpha_i< DBL\_MIN$, we set $q = DBL\_MAX$.

\quadmetrictable{condition}%
{$1$}%                                      Dimension
{$[1,4]$}%                                  Acceptable range
{$[1,DBL\_MAX]$}%                           Normal range
{$[1,DBL\_MAX]$}%                           Full range
{$1$}%                                      Unit square
{\cite{knu:00}}%                            Citation
{v\_quad\_condition}%                       Verdict function name