\documentclass[../main.tex]{subfile}

\begin{document}

\section{Classification} \label{sec:classification}

\subsection{Feature Extraction} \label{subsec:features-extraction}
There are several ways to represent an image and, for this work, the following ones were chosen: RGB image, grayscale image, and image representation through features.

The first method to represent the images is in the form of an RGB image. It can be considered the default method, since it does not require any additional computation after the pre-processing phase.

The second one transforms an image into a grayscale image, as the colors may not only be irrelevant but also misleading, and the difference in lighting between a healthy and a rotten fruit peel may be enough to classify the fruits correctly.

The third and last one represents an image through some textural properties from a Gray Level Co-occurrence Matrix (GLCM) \cite{IV-item1}.
A GLCM is a matrix defined over an image, and it represents the distribution of co-occurring gray values (intensity) at a given offset. In other words, it calculates the spatial distribution of intensity values in a neighborhood from a one-color channel image, like a grayscale image, which is why a GLCM is good to obtain textural information about an image.

For an image $I$ of size $N \times N$, a co-occurrence matrix $M$ of order $N_{g} \times N_{g}$ can be defined as:
\begin{equation}
    M(i,j) = \sum^{N}_{x=1}\sum^{N}_{y=1}
    \begin{cases}
        1 & \mbox{if } I(x,y) = i \mbox{ and }           \\
          & \quad I(x + \Delta_{x}, y + \Delta_{y}) = j, \\
        0 & \mbox{otherwise},
    \end{cases}
\end{equation}
In which $i$ and $j$ are the intensity values, $x$ and $y$ are the spatial coordinates. The offset ($\Delta_{x}, \Delta_{y}$) specifies the distance between the pixel-of-interest and its neighbors. This offset can also be parameterized in terms of a distance $d$ and an angle $\theta$. Note that when $d = 1$, it is called a normalized co-occurence matrix. Having that said, $m(i,j)$ will be referenced as the $(i,j)$-th entry in a normalized co-occurence matrix.

In order to calculate the textural properties yet to be mentioned, the following equations will be needed:
\begin{align*}
    m_{x}(i) = \sum^{N_{g}}_{j=1} m(i,j),                                 & \quad
    m_{y}(j) = \sum^{N_{g}}_{i=1} m(i,j)                                          \\
    \mu_{x} = \sum^{N_{g}}_{i=1} i \cdot m_{x}(i),                        & \quad
    \mu_{y} = \sum^{N_{g}}_{j=1} j \cdot m_{y}(j)                                 \\
    \sigma^{2}_{x} = \sum^{N_{g}}_{i=1} (1 - \mu_{x})^{2} \cdot m_{x}(i), & \quad
    \sigma^{2}_{y} = \sum^{N_{g}}_{j=1} (1 - \mu_{y})^{2} \cdot m_{y}(j)
\end{align*}

Concerning the GLCM method, to calculate the textural information necessary to classify the fruits, the following statistical descriptors will be used:

\begin{description}
    \item[Contrast] --- measures the intensity contrast between a pixel and its neighbours over the whole image:
          \begin{equation}
              Contrast = \sum^{N_{g}}_{i=1} \sum^{N_{g}}_{j=1} m(i,j) \cdot \lvert i - j \rvert^{2}
              \label{eq:contrast}
          \end{equation}

    \item[Dissimilarity] --- measures the distance between pairs of \mbox{pixels} in a region of interest:
          \begin{equation}
              Dissimilarity = \sum^{N_{g}}_{i=1} \sum^{N_{g}}_{j=1} m(i,j) \cdot \lvert i - j \rvert
              \label{eq:dissimilarity}
          \end{equation}

    \item[Homogeneity] --- measures the closeness of the distribution of elements in the GLCM to the GLCM diagonal:
          \begin{equation}
              Homogeneity = \sum^{N_{g}}_{i=1} \sum^{N_{g}}_{j=1} \frac{m(i,j)}{1 + \lvert i - j \rvert^{2}}
              \label{eq:homogeneity}
          \end{equation}

    \item[Angular Second Moment (ASM)] --- Provides the sum of the squared elements in the GLCM, it is used as a measure of orderliness:
          \begin{equation}
              ASM = \sum^{N_{g}}_{i=1} \sum^{N_{g}}_{j=1} m(i,j)^{2}
              \label{eq:asm}
          \end{equation}

    \item[Energy] --- Similar to the ASM, however it is considered a better texture measurement:
          \begin{equation}
              Energy = \sqrt{ASM}
              \label{eq:energy}
          \end{equation}

    \item[Correlation] --- measures how correlated a pixel  is to its neighbour over the over whole image:
          \begin{equation}
              Correlation = \sum^{N_{g}}_{i=1} \sum^{N_{g}}_{j=1} \frac{m(i,j) \left(1 - \mu_{x}\right) \left(1 - \mu_{y}\right)}{\sigma_{x} \cdot \sigma_{y}}
              \label{eq:correlation}
          \end{equation}
\end{description}

Considering that a GLCM uses gray levels values or intensity values, an RGB image cannot be used as an input, neither a grayscale image, as it loses relevant information about each color channel. So far, 3 GLCMs are needed, one for each channel in the RGB color scheme. In addition, as mentioned above, the distance used is one pixel, moreover, four angles were used: $0^{\circ}$, $45^{\circ} (\frac{\pi}{4})$, $90^{\circ} (\frac{2\pi}{4})$, and $135^{\circ} (\frac{3\pi}{4})$. A combination between an angle and a distance makes an offset. Now, 4 GLCMs are needed for each color channel. Therefore, in total, 12 GLCMs need to be built. Then, as 6 textural descriptors are calculated from one GLCM \eqref{eq:contrast} to \eqref{eq:correlation}. To represent a single image, 72 textural descriptors are calculated.

Moreover, for every method discussed above, a Gabor filter is applied. If it is a grayscale image, the Gabor filter is applied after the image is converted into grayscale. If a GLCM is used, the filter is applied to the image before the GLCM is calculated.

A Gabor filter can be defined as a sinusoidal signal of a particular frequency and orientation, modulated by a Gaussian wave. In other words, it allows certain frequencies and rejects others. When a Gabor Filter is applied to an image, it emphasizes the edges and points where texture changes accordingly to the filter pattern, i.e., its parameters. Therefore, these Gabor Filters are well suited for texture analysis. Mathematically, a Gabor Filter has a real and imaginary component representing orthogonal directions. For this work, only the real component was used.

\subsection{Neural Network} \label{subsec:neural-network}

There were 120 samples of each of the six fruits, i.e., 720 images in total, and all the fruit images were split equally into those three classes. In other words, 40 images for each class (A, B, or C). 80\% of fruit images were used to train the models and the remaining 20\% for the validation step. Since there were defined very distinct ways to represent the images, it demands different neural networks architectures to fit better the data fed into them.

\subfile{../figures/cnn-diagram}

The architecture seen in Figure \ref{fig:cnn} represents the structure of a Convolution Neural Network (CNN). A CNN is mainly composed of four different layers: Convolution, Pooling, Flattening, and Classification.

Convolutional layers allow one to systematically create and apply a series of filters over an array input (e.g., an image). These filters act like feature detectors, and then the layer outputs a "feature map" that summarizes the features detected (e.g., lines and edges) by the filters. Although the Convolutional layers have their limitations, they are translation-invariant, because they are capable of detecting patterns locally rather than globally. It means that if a Convolution layer leaned a certain pattern, and if the same pattern occurs again, but in a different location, they would still be able to detect that same pattern.

A Pooling Layer is used to perform a downsample on feature maps, which reduces their dimensions, but they still carry the main elements. It works by aggregating a group of pixels and applying a pooling operation. The two most commons are Average Pooling, which calculates the average of each group, and Max Pooling, which calculates the maximum value in each group. The purpose of using a Pooling Layer is to decrease the computational power needed to process the data. And as a side-effect, it helps to extract dominant features, making the training process more efficient.

Before the Pooling layer, an activation function (e.g., ReLU (Rectified Linear Unit)) is used upon the features maps. This function usually breaks the linearity and adds complexity, much like it is done for a Dense Layer.

The combination between a Convolutional Layer, a Pooling Layer, and an activation function makes a Convolution Group, seen in Figure \ref{fig:cnn}. When stacking these Convolutional Layers, instead of detecting simple features (e.g., lines and edges), deeper layers learn to detect more abstract features, like shapes or specific objects.

After that, a Flattening Layer is used because the output of the combination of multiple Convolutional Groups is multi-dimensional, composed of several features maps. So, it needs to be flattened.

Finally, the Classification part is composed of a sequence of Dense Groups. A Dense Group is made of Dense Layers, also known as Fully Connected Layers, followed by an activation function. Before the Output Layer, a Dropout Layer is used to help reducing overfitting while training (useful for a small dataset). It works by randomly deactivating a percentage of the neurons from the previous layer. Then for the Output Layer, which is also a Dense Layer, the activation function chosen is often different from the ones used in the hidden layers.

\subfile{../figures/dnn-diagram}

The other architecture, DNN, seen in Figure \ref{fig:dnn}, is a simple deep neural network composed of a Flattening Layer at the beginning, a sequence of Dense Groups, and a Dropout Layer right before the Output Layer. This architecture may look very similar to the one used in the Flattening and Classification part inside the CNN. However, in reality, a CNN uses a combination of a convolutional technique to extract useful information from images and passes it as input to a fully connected deep neural network.

Data Augmentation technique was used at the beginning of the CNN seen in Figure \ref{fig:cnn}. Since there was a total of 120 images for each fruit, and each model was trained for only one type of fruit, meaning the models were on a small dataset. It helps by reducing overfitting by generating new images from existing ones by doing simple modifications such as random horizontal and vertical flipping with a random rotation. For the same reason, a Spatial Dropout Layer was used right before the Flattening part. It works similarly to a regular Dropout Layer. However, it is applied to 2D feature maps instead of individual elements.

Regarding the Flattening Layer in the CNN, a Global Average Filter was used. It uses an average pooling to reduce the size of feature maps, and then it flattens and passes it as input to the next part. This Global Average Filter is important because its input came from either RGB or Grayscale image methods. Thus, if it were simply flattened, there would be too much information for the dense neural network, i.e., the Classification part, to process and classify efficiently. That was unnecessary for the DNN since its input came from the GLCM textural properties, and it consisted of only 72 elements, the GLCM descriptors when flattened.

Inside the Convolutional Groups, the activation functions used were the ReLU \eqref{eq:relu} or the swish \eqref{eq:swish} function, which behaves similarly to the ReLU. Concerning the Dense Groups, they use either the swish or the ELU (Exponential Linear Unit) \eqref{eq:elu} function, which is also similar to the ReLU.

The softmax \eqref{eq:softmax} function is used at the Output Layer because it converts its input into a probability distribution. Useful for classification tasks when there are more than two classes.
\begin{align}
    ReLU(x)        & = max(0, x) \label{eq:relu}                                                       \\
    swish(x)       & = \frac{x}{1 + \mathrm{e}^{-x}} \label{eq:swish}                                  \\
    ELU(x)         & =
    \begin{cases}
        x                  & \mbox{if } x > 0, \\
        \mathrm{e}^{x} - 1 & \mbox{otherwise},
    \end{cases} \label{eq:elu}                                                           \\
    softmax(x_{i}) & = \frac{\mathrm{e}^{x_{i}}}{\sum^{K}_{j=1} \mathrm{e}^{x_{j}}} \label{eq:softmax}
\end{align}

\end{document}


\subsection{Regression dilution}

Noise in \(y\) doesn't cause bias.

Noise in \(x\) does cause bias.

Need to correct.


\subsection{Dynamic time warping}

We may want to examine the similarity between two sequences.

We want to match a sample from one sequence to a sample from the other sequence.

Simply matching at the same time point is naive, as samples may move at different speeds, or have offsets.


\documentclass[12pt,letterpaper,oneside,notitlepage,onecolumn]{article}%
\usepackage[utf8]{inputenc}%
\usepackage{textcomp}%
\usepackage{lastpage}%
\usepackage[margin=0.69in]{geometry}%
%
\title{Vector{-}Rotation to Euler Angles}%
\author{Buck Baskin}%
\date{\today}%
\usepackage[english]{babel}%
\usepackage[margin=0.69in]{geometry}%
\usepackage{parskip}%
\usepackage{listings}%
\usepackage{color}%
\usepackage{verbatim}%
\usepackage{soul}%
\usepackage{amsmath}%
\usepackage{amssymb}%
\usepackage{amsthm}%
\usepackage{gensymb}%
\usepackage{graphicx}%
\definecolor{dkgreen}{rgb}{0, 0.6, 0}%
\definecolor{gray}{rgb}{0.5, 0.5, 0.5}%
\definecolor{mauve}{rgb}{0.58, 0, 0.82}%
\lstset{frame=tb,
  language=Matlab,
  aboveskip=3mm,
  belowskip=3mm,
  showstringspaces=false,
  columns=flexible,
  basicstyle={\small\ttfamily},
  numbers=none,
  numberstyle=\tiny\color{gray},
  keywordstyle=\color{blue},
  commentstyle=\color{dkgreen},
  stringstyle=\color{mauve},
  breaklines=true,
  breakatwhitespace=true,
  tabsize=3
}%
\DeclareMathOperator*{\argmax}{arg\,max}%
\DeclareMathOperator*{\argmin}{arg\,min}%
\newcommand{\subsubsubsection}{\paragraph}%
\newcommand{\bbs}[1]{\section{#1}}%
\newcommand{\bbbs}[1]{\subsection{#1}}%
\newcommand{\bbbbs}[1]{\subsubsection{#1}}%
\newcommand{\bbbbbs}[1]{\subsubsubsection{#1}}%
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}%
%
\begin{document}%
\normalsize%
\maketitle%
Some calculations were verified using Python.%
\section{Answer}%
$R_{BA} = R_{y}(\alpha) R_{x}(\beta) R_{y}(\gamma)$%
\subsection{Solving for $\alpha, \beta, \gamma$}%
\subsubsection{$\beta$}%
Using the atan method, one can solve for $\beta$ using the middle column of the matrix. The middle value, $r_{22} = cos(\beta)$. Using $r_{12}^{2} + r_{32}^{2} = sin^{2}(\beta)(sin^{2}(\gamma) + cos^{2}(\gamma)) = sin^{2}(\beta)$, $sin(\beta) = \sqrt{r_{12}^{2} + r_{32}^{2}}$. Therefore, $\beta = atan2(\sqrt{r_{12}^{2} + r_{32}^{2}}, r_{22})$.

%
\subsubsection{$\gamma$}%
Looking at two elements in the middle row of the matrix, one can solve for $\gamma$ using the atan method. $r_{21} = sin(\beta) sin(\gamma)$, so $sin(\gamma) = r_{21} / sin(\beta)$. $r_{23} = -sin(\beta) cos(\gamma)$, so $cos(\gamma) = -r_{23} / sin(\beta)$. Therefore, $\gamma = atan2(r_{21} / sin(\beta), -r_{23} / sin(\beta))$.

%
\subsubsection{$\alpha$}%
$r_{12} = sin(\alpha) sin(\beta)$, $sin(\alpha) = r_{12} / sin(\beta)$\\$r_{32} = cos(\alpha) sin(\beta)$, $cos(\alpha) = r_{32} / sin(\beta)$\\$\alpha = atan2(r_{12} / sin(\beta), r_{32} / sin(\beta))$

%
\subsection{Conclusion}%
Substituting the correct values from $R_{BA}$ calculated by Rodriguez Formula into the spaces allocated by $r_{ij}$ that were solved into an $atan2$ formula using $R_{BA}$ calculated by the rotation matrices gives the solution for the equivalent rotations in radians (rounded to 3 decimal places).%
\newline%
$\alpha = 1.833, \beta = 1.231, \gamma = -2.880$

%
\end{document}


\section{Invariants of nonassociative algebras}

Converting an algebra to a tensor enables Magma to compute standard invariants
of any algebra. We note that there are known errors for $\mathbb{R}$ and
$\mathbb{C}$ due to the numerical stability of the linear algebra involved in
the computations.

\index{Center}\index{Centre}
\begin{intrinsics}
Center(A) : Alg -> Alg
Centre(A) : Alg -> Alg
\end{intrinsics}

Returns the center of the algebra $A$.

\index{Centroid!algebra}
\begin{intrinsics}
Centroid(A) : Alg -> AlgMat
\end{intrinsics}

Returns the centroid of the $K$-algebra $A$ as a subalgebra of $\text{End}_K(A)$.

\begin{example}[CenterCentroids]

We will construct a representation of $\mathfrak{sl}_2(9)$ in $\mathbb{M}_4(\mathbb{F}_3)$.
First we construct $\mathfrak{gl}_2(9)$ from $\mathbb{M}_4(\mathbb{F}_3)$.
\begin{code}
> M := MatrixAlgebra(GF(3), 4);
> f := ConwayPolynomial(3, 2);
> C := CompanionMatrix(f);
> I := IdentityMatrix(GF(3), 2);
> A := sub< M | [InsertBlock(M!0, X, i, j) : \
>     X in [I, C], i in [1, 3], j in [1, 3]] >;
> T := CommutatorTensor(A);
> T;
Tensor of valence 3, U2 x U1 >-> U0
U2 : Full Vector space of degree 8 over GF(3)
U1 : Full Vector space of degree 8 over GF(3)
U0 : Full Vector space of degree 8 over GF(3)
> gl2 := HeisenbergAlgebra(T);
> gl2;
Algebra of dimension 8 with base ring GF(3)
\end{code}

Our Lie algebra is not simple as it has a nontrivial center, so we will obtain
$\mathfrak{sl}_2$ by factoring out the center. Note that our algebras are over
the prime field $\mathbb{F}_3$, so the center is 2-dimensional (over
$\mathbb{F}_3$). Notice that $\mathfrak{sl}_2(9)$ has a trivial center but has a
2-dimensional centroid.
\begin{code}
> sl2 := gl2/Center(gl2);
> sl2;
Algebra of dimension 6 with base ring GF(3)
> Center(sl2);
Algebra of dimension 0 with base ring GF(3)
> Centroid(sl2);
Matrix Algebra of degree 6 with 2 generators over GF(3)
\end{code}
\end{example}

\index{LeftNucleus!algebra}\index{RightNucleus!algebra}\index{MidNucleus!algebra}
\begin{intrinsics}
LeftNucleus(A) : Alg -> AlgMat
RightNucleus(A) : Alg -> AlgMat
MidNucleus(A) : Alg -> AlgMat
\end{intrinsics}

Returns the nucleus of the algebra $A$ as a subalgebra of the enveloping algebra
of right multiplication $\mathcal{R}(A)$.

\index{DerivationAlgebra!algebra}
\begin{intrinsics}
DerivationAlgebra(A) : Alg -> AlgMatLie
\end{intrinsics}

Returns the derivation algebra of the algebra $A$ as a Lie subalgebra of $\text{End}_K(A)$.

\begin{example}[DerivationAlg]

We will compute the derivation algebra of the (rational) octonions $\mathbb{O}$
and also the 27 dimension exceptional Jordan algebra
$\mathfrak{H}_3(\mathbb{O})$. Because the intrinsics use exact linear algebra,
we do not use the more familiar field $\mathbb{C}$ in this context. First we
consider $\mathbb{O}$. We verify that $\Der(\mathbb{O})\cong G_2$.
\begin{code}
> A := OctonionAlgebra(Rationals(), -1, -1, -1);
> A;
Algebra of dimension 8 with base ring Rational Field
> D := DerivationAlgebra(A);
> D;
Matrix Lie Algebra of degree 8 over Rational Field
> SemisimpleType(D);
G2
\end{code}

Now we will just briefly perform a sanity check and verify that \texttt{D} acts as it should.
\begin{code}
> a := Random(Basis(A));
> b := Random(Basis(A));
> del := Random(Basis(D));
> (a*b)*del eq (a*del)*b + a*(b*del);
true
\end{code}

Finally, we construct $\mathfrak{H}_3(\mathbb{O})$ the $3\times 3$ Hermitian
matrices, and we verify that $\Der(\mathfrak{H}_3(\mathbb{O})\cong F_4$.
\begin{code}
> J := ExceptionalJordanCSA(A);
> J;
Algebra of dimension 27 with base ring Rational Field
> D_J := DerivationAlgebra(J);
> Dimension(D_J);
52
> SemisimpleType(D_J);
F4
\end{code}
\end{example}


\subsection{LES}

A means of assessing the quality of grid resolution in LES \cite{Gant2009}:
\begin{itemize}
	\item Estimations based on prior RANS results
	\item Single-grid estimators
	\item Multi-grid estimators
\end{itemize}

\subsubsection{LES filtering}
In case instantaneous values of flow is desired and solving all scale turbulent structure by DNS is too expensive. LES filtering is employed to solve only large structure of flow, while universal small ones are still modelled. Using Farve filtering, any quantities $Q$ is decomposed to filtered quantities $\widetilde{Q}$ and unfiltered quantities $Q''$. Filtering instantaneous Navier-Stokes equations results below equations:
\begin{equation} \label{eq:LES-continuity}
\pdv{\overline{\rho}}{t}  +
\pdv{}{x_i} \left( \rho \widetilde{u_i} \right) = 0
\end{equation}

\begin{equation} \label{eq:LES-momentum}
\pdv{}{t} \left( \overline{\rho} \widetilde{u_i} \right) +
\pdv{}{x_j} (\overline{\rho} \widetilde{u_i} \widetilde{u_j}) =
-\pdv{\overline{p}}{x_i}
+ \pdv{\overline\tau_{ij}}{x_j}
+ \pdv{}{x_j} (-\overline\rho (\widetilde{u_i u_j} - \widetilde{u_i} \widetilde{u_j} ))
+ \overline b_i
\end{equation}

Farve filtered equation for  species mass fraction $Y_i$:
\begin{equation} \label{eq:LES_species}
\pdv{}{t} \left( \overline{\rho} \widetilde{Y_i} \right) +
\pdv{}{x_j} (\overline{\rho} \widetilde{u_j} \widetilde{Y_i}) =
\pdv{\overline j_{ij}}{x_j}
+ \pdv{}{x_j} \left[-\overline\rho (\widetilde{ u_j Y_i} - \widetilde{u_j} \widetilde{Y_i}) \right]
+ \overline\omega_i
\end{equation}

Unclosed quantities are \cite{Poinsot2005}: subgrid (unresolved) Reynold stresses $\tau_{ij}^s = -\rho(\widetilde{u_i u_j} - \widetilde{u_i} \widetilde{u_j})$; unresolved species fluxes: $j_{j} = -\rho (\widetilde{u_j Y_i} - \widetilde{u_j} \widetilde{Y_i}$), filtered laminar diffusion fluxes $\overline\tau_{ij}$, $\overline j_{ij}$  and filtered chemical reaction rate $\overline\omega_i$.

\subsubsection{Smagorinsky model}
With eddy viscosity assumption, subgrid scale Reynold stresses $\tau_{ij}^s$ can be modelled as:
\begin{equation} \label{eq:sgs_stress}
\tau_{ij}^{s} -\frac{1}{3}\tau_{kk}^{s} \delta_{ij} =
2 \mu_t \left( \widetilde{S_{ij}} - \frac{1}{3}\widetilde{S_{kk}} \delta_{ij} \right)
\end{equation}

The Farve-filtered rate-of-strain tensor $\widetilde{S_{ij}}$ is defined the same as Equation~\ref{eq:rate_of_strain_tensor} with velocity field is replaced by filtered velocity:
\begin{equation} \label{eq:sgs_rate_of_strain_tensor}
\widetilde{S_{ij}} = \frac{1}{2} \left( \pdv{\widetilde{u_i}}{x_j} + \pdv{\widetilde{u_j}}{x_i} \right)
\end{equation}
The second term in LHS is included to ensure that in case of isotropic stress tensor, its trace is equal to minus twice the kinetic energy \cite{Lilly1992}.

$\nu_t = \mu_t/\rho$ is \emph{subgrid scale viscosity}. It is modelled using dimensional argument of characteristic length scale and characteristic velocity scale:
\begin{equation} \label{eq:sgs_viscoscity}
\nu_t =2 C_\mu \Delta^2 (2 \widetilde{S_{ij}} \widetilde{S_{ij}})^{1/2}
\end{equation}

With gradient-diffusion assumption, turbulent scalar flux can be written as:
\begin{equation} \label{eq:sgs_turb_scalar}
-\bar{\rho} (\widetilde{u_j \phi_i} - \widetilde{u_j} \widetilde{\phi_i}) =
\bar{\rho} \alpha_t \pdv{\phi_i}{x_j}
\end{equation}
$\alpha_t$ is turbulent diffusivity and calculated from:
\begin{equation} \label{eq:sgs_turb_diffusivity}
\bar{\rho} \alpha_t  =
C_\alpha \bar{\rho} \Delta^2 |\tilde{S}|
\end{equation}

\subsubsection{Dynamic approach}
Dynamic modelling concept is used to derive dimensionless scaling coefficients in subgrid scale model instead of using constant coefficients as classical models. Assume we have a term $t(u)$ is a function of field variable $u$. A filtered value of $t(u)$ can be decomposed into resolved and modelled parts: $\overline{t(u)} = t(\overline{u}) + m(\overline{u})$. We use another filter, called \emph{test filter} to examine the variance of $t(\overline{u})$ and $m(\overline{u})$. Filtered term calculated at this test filter are denoted by hat symbol: $\widehat{ \overline{t(u)}} = t(\widehat{\overline{u}}) + m(\widehat{\overline{u}})$. If we have subgrid scale identity:
\begin{equation} \label{eq:sgs_identity}
\widehat{t({\overline{u}})} - t(\widehat{\overline{u}}) =
m(\widehat{\overline{u}}) - \widehat{m({\overline{u}})}
\end{equation}

Model for modelled part $m(\overline{u})$ Equation~\ref{eq:sgs_modelled_part}. $c$ is dimensionless coefficient, which can vary in both space and time. Substituting to Equation~\ref{eq:sgs_identity}, we have Equation~\ref{eq:sgs_identity_final}, where $c^*$ is an coefficient at test filter level.

\begin{equation} \label{eq:sgs_modelled_part}
m(\overline{u}) = c\; s(\overline{u},\Delta)
\end{equation}

\begin{equation} \label{eq:sgs_identity_final}
\widehat{t({\overline{u}})} - t(\widehat{\overline{u}}) =
c^*\; s(\widehat{\overline{u}},\widehat{\Delta}) - \widehat{c\; s(\overline{u},\Delta)}
\end{equation}

Assuming $c^* = c$ and allowing $c$ pass through the test filtering operator. Leonard term $\mathcal{L} = \widehat{t({\overline{u}})} - t(\widehat{\overline{u}})$ and model term $\mathcal{M} = s(\widehat{\overline{u}}, \widehat{\Delta}) - \widehat{s(\overline{u},\Delta)}$. We have $\mathcal{L} = c \mathcal{M}$. This relation can be solve by least-square to determine single value of $c$, as Equation~\ref{eq:sgs_coefficient}.

\begin{equation} \label{eq:sgs_coefficient}
c= \frac{\langle \mathcal{L} \cdot \mathcal{M}\rangle}{\langle \mathcal{M} \cdot \mathcal{M}\rangle}
\end{equation}

Apply dynamic approach to find coefficient $C_\mu$, $C_\alpha$, where density-weighted test filtering is denoted as $\check{\overline{u}} = \widehat{\overline{\rho}\widetilde{u}}/\widehat{\overline{\rho}}$.

\begin{equation} \label{eq:sgs_turb_stress}
C_\mu= \frac{\langle \mathcal{L}_{ij} \mathcal{M}_{ij} \rangle}{2 \langle \mathcal{M}_{ij} \mathcal{M}_{ij} \rangle}
\quad
\mathcal{L}_{ij} = -
\widehat{\overline{\rho}\widetilde{u_i}\widetilde{u_j}} +
\widehat{\overline{\rho}} \check{\overline{u_i}} \check{\overline{u_j}},
\quad
\mathcal{M}_{ij} =
\widehat{\overline{\rho}} \widehat{\Delta}^2 \check{| {\widetilde{S}} |} \check{\widetilde{S_{ij}}} -
\widehat{\overline{\rho} {\Delta}^2 | {\widetilde{S}} | \widetilde{S_{ij}}}
\end{equation}

\begin{equation} \label{eq:sgs_turb_diffusivity_dyn}
C_\alpha= \frac{\langle \mathcal{L}_{ij} \mathcal{M}_{ij} \rangle}{\langle \mathcal{M}_{ij} \mathcal{M}_{ij} \rangle}
\quad
\mathcal{L}_i = -
\widehat{\overline{\rho}\widetilde{u_i}\widetilde{\phi}} +
\widehat{\overline{\rho}} \check{\overline{u_i}} \check{\overline{\phi}},
\quad
\mathcal{M}_{i} =
\widehat{\overline{\rho}} \widehat{\Delta}^2 \check{| {\widetilde{S}} |} \check{\widetilde{\pdv{\phi}{x_i}}} -
\widehat{\overline{\rho} {\Delta}^2 | {\widetilde{S}} | \widetilde{\pdv{\phi}{x_i}}}
\end{equation}

\begin{equation} \label{eq:sgs_scalar_var_dyn}
C_\phi= \frac{\langle \mathcal{L} \mathcal{M} \rangle}{\langle \mathcal{M} \mathcal{M} \rangle}
\quad
\mathcal{L} =
\widehat{\overline{\rho}\widetilde{\phi}\widetilde{\phi}} -
\widehat{\overline{\rho}} \check{\widetilde{\phi}} \check{\widetilde{\phi}},
\quad
\mathcal{M} =
\widehat{\overline{\rho}} \widehat{\Delta}^2 | \grad{\check{\widetilde{\phi}}} |^2  -
\widehat{ \overline{\rho} {\Delta}^2 | \grad{\widetilde{\phi}} |^2 }
\end{equation}

\subsubsection{Implementation in OpenFOAM}
The anisotropic part of turbulence shear stress $\tau_{ij}$ is approximated by relating it to the resolved rate of strain tensor $D_{ij}$
\begin{equation}
\tau_{ij} - \frac{1}{3} \tau_{kk} \delta_{ij} \approx - 2 \nu_{sgs} \text{dev}(\overline{D}_{ij})
\end{equation}
$\text{dev}(\overline{D}_{ij})$ is deviatoric part of strain tensor $D_{ij}$.

The sub-grid scale viscosity is defined as:
\begin{equation}\label{eq:foam_nu_sgs}
\nu_{sgs} = C_{k} \Delta \sqrt{k_{sgs}}
\end{equation}
$C_{k}$ is a model constant whose default value is $0.094$ and $\Delta$ is the filter width that defines the subgrid length scale.

The SGS kinetic energy $k_{sgs}$ is defined as:
\begin{equation} \label{eq:foam_k_sgs}
k_{sgs} = \frac{1}{2} \tau_{kk} =
\frac{1}{2} \left( \overline{u_{k}u_{k}} -
\overline{u}_{k}\overline{u}_{k} \right)
\end{equation}
$k_{sgs}$ is computed with the assumption of balance between the subgrid scale energy production and dissipation Equation~\ref{eq:foam_k_sgs_equilibrium}:
\begin{equation}  \label{eq:foam_k_sgs_equilibrium}
\overline{D} : \tau_{ij} + C_{\epsilon} \frac{k_{sgs}^{1.5}}{\Delta} = 0
\end{equation}

\subsubsection{SGS models}
There are various options of SGS models implementation in OpenFOAM
\begin{itemize}
	\item \bera{Smagorinsky}: Smagorinsky SGS model
	\item \bera{kEqn}: One equation eddy-viscosity model
	\item \bera{dynamicLagrangian}: Dynamic SGS model with Lagrangian averaging
	\item \bera{dynamicKEqn}: Dynamic one equation eddy-viscosity model
	\item \bera{WALE}: Wall-adapting local eddy-viscosity (WALE) SGS model
	\item \bera{DeardorffDiffStress}: Differential SGS Stress Equation Model
\end{itemize}

For the dynamic SGS models, the spatial averaging operations of the coefficients are often performed to stabilize the calculation. The \bera{homogeneousDynSmagorinsky} model that had been implemented in older versions takes the average of the coefficient in the whole computational domain.

\subsubsection{Calculation of filter width $\Delta$}
The method for calculating the filter width $\Delta$ is specified in the \bera{turbulenceProperties} file. Available options in OpenFOAM are as follows:
\begin{itemize}
	\item cubeRootVol
	\item maxDeltaxyz
	\item maxDeltaxyzCubeRoot
	\item smooth
	\item vanDriest
	\item Prandtl
	\item IDDESDelta
\end{itemize}

The \bera{maxDeltaxyz} option calculates  filter width of the $i$ cell $\Delta_i$ by taking the maximum distance between the cell centre $P_i$ and each face centre $F_j$:
\begin{equation}  \label{eq:deltaxyz}
\Delta_i = \text{deltaCoeff} \times \max_{1 \le j \le n_i} \left\{ \overline{P_iF_j} \right\}
\end{equation}
\text{deltaCoeff} is the user specified constant of proportion.


%!TEX root =  ../main.tex

\subsection{Local Linearity}

\objective{Define derivatives and find numerical approximations}

When two variables grow proportionally to each other, they obey the algebraic relationship
$y=ax$.  This means when one goes up, the other goes up, and visa versa.  But many things
in nature are \emph{inversely} proportional to each.  \index{proportion!inverse}
That is, when one goes up, the other goes
\emph{down}.  This is the algebraic relationship $y=\frac{a}{x}$.

\personfeature[-1in]{\chapdir/pics/Henri-Poincare}{Jules Henri Poincar\'{e}
    }{1854-1912, French}{was a French mathematician and physicist
    who discovered many amazing facets of modern physics and non-
    traditional mathematics.  He was true polymath of his time, and
    never worked very long on a problem, relying (successfully)
    on his subconscious to keep working on the ideas.}

The fundamental postulate of calculus is that smooth, continuous functions are \textbf{locally
linear}, that is, if you zoom in far enough, they all appear as lines.  Algebraically, we should be able
to construct a tangent line to any function, using the difference quotient.  Chaos Theory, and
Fractal Geometry oppose this precept, in much the same way that Non-Euclidean Geometries
are build from a rejection of Euclid's Parallel Postulate (i.e., that parallel lines never meet).  Euclidean
Geometry and Calculus may be two viewpoints among many, but they are certainly useful ones to know!

\subsection{Derivative}
\index{Derivative!definition}
For all the aura surrounding calculus as the \textit{summum bonum} --- highest good --- it is not
some process to hard to just grasp.  We just did it graphically.  Algebraically, it is the difference
quotient, an tiny step away from the point producing a line which therefore touches the
curve at only one point.  This line gets more and more accurate, the smaller our step away becomes.

\subsubsection{Theorems of Derivatives}

\begin{itemize}
\item[\textbf{sum}] the derivative of a sum is the sum of the derivatives
\item[\textbf{scalar}] the derivative of a constant times a function is a constant times a derivative
\item[\textbf{constant}] the derivative of a constant is 0
\end{itemize}

Do no assume beyond this!  Note that the first two allow you to build a ``difference property'' too.


\section{Graph Isomorphism}

\frame{
{Part 1: Graph Isomorphism}

\tableofcontents[currentsection,hideallsubsections, firstsection=1, sections={1-4}]
}

\begin{frame}{Directed Graphs and Simple Graphs}
  \begin{columns}[T]
    \column{0.5\textwidth}
    \begin{center}
      Directed Graph:\\

      \begin{tikzpicture}[scale=.8,auto,swap]
        \tikzset{edge/.style = {->,>=latex'}}
        \node[vertex] (a) at (0,0) {a};
        \node[vertex] (b) at (2,3) {b};
        \node[vertex] (c) at (4,2) {c};
        \node[vertex] (d) at (4,0) {d};
        \draw[edge] (a) to (b);
        \draw[edge] (a) to (c);
        \draw[edge] (c) to (b);
        \draw[edge] (d) to (c);
        \draw[edge] (d) to[bend left] (a);
        \draw[edge] (a) to[bend left] (d);
        \draw[edge] (a) to[loop left] (a);
      \end{tikzpicture}
    \end{center}
    \column{0.5\textwidth}
    \begin{center}
      Simple Graph:\\

      \begin{tikzpicture}[scale=.8,auto,swap]
        %\tikzset{edge/.style = {->,>=latex'}}
        \node[vertex] (a) at (0,0) {a};
        \node[vertex] (b) at (2,3) {b};
        \node[vertex] (c) at (4,2) {c};
        \node[vertex] (d) at (4,0) {d};
        \draw[edge] (a) to (b);
        \draw[edge] (a) to (c);
        \draw[edge] (c) to (b);
        \draw[edge] (d) to (c);
        \draw[edge] (d) to (a);
      \end{tikzpicture}

      \vspace{2em}
    \end{center}
    \begin{itemize}
    \item No double edges allowed;
    \item No self-loop allowed;
    \end{itemize}
  \end{columns}
\end{frame}

\begin{frame}{Simple Graphs}{Some definitions}
  A Simple Graph \structure{$G$} consists of:
  \begin{itemize}
  \item A \emph{non-empty} set \structure{$V$} of vertices;
  \item A set \structure{$E$} of edges so that:
    \begin{itemize}
    \item Each edge has \structure{two endpoints} in $V$:
      \hfill (\alert{not an {\bf start} and an {\bf end}})

      \bigskip
    \item The order of the vertices in an edge does not matter:
      \hfill $e_1 = \{v_1,v_2\} = \{v_2, v_1\}$

      \bigskip
    \item Two vertices with an edge between them are
      \structure{adjacent}

      \bigskip
    \item An edge that connects two vertices is \structure{incident}
      to them.
      \hfill Ex: $e_1$ is \structure{incident} to $v_1$ and $v_2$
    \end{itemize}
  \end{itemize}
\end{frame}

\begin{frame}{Vertice Degrees}

    The \structure{degree} of a vertex is the \structure{number of
      incident edges}.

    \begin{center}
      \begin{tikzpicture}[scale=.8,auto,swap]
        %\tikzset{edge/.style = {->,>=latex'}}
        \node[vertex] (a) at (0,0) {a};
        \node[vertex] (b) at (2,3) {b};
        \node[vertex] (c) at (4,2) {c};
        \node[vertex] (d) at (4,0) {d};
        \draw[edge] (a) to (b);
        \draw[edge] (a) to (c);
        \draw[edge] (c) to (b);
        \draw[edge] (d) to (c);
        \draw[edge] (d) to (a);
      \end{tikzpicture}

      deg(a) = 3 \hspace{1cm} deg(d) = 2
    \end{center}

    \hfill

    \alert{Quiz:} Can you build a graph with following vertice degrees?
    \begin{itemize}
    \item 3, 2, 2, 1 (four vertices)
    \item 3, 2, 2, 2 (four vertices)
    \end{itemize}
\end{frame}

\begin{frame}{Verdice Degrees}{The Handshaking Lemma}

  {\bf Lemma:} The sum of vertice degrees in a graph is 2x the number of edges.
  \begin{equation}
    2|E| = \sum_{v\in V} \text{deg}(v)
  \end{equation}

  \begin{proof}
    \begin{itemize}
      \item Every edge in a graph connects two vertices;
      \item If we begin with a graph with 0 edges, for every edge $(v_i,v_j)$ that we add to the graph, we add 2 vertice degrees (one for $v_i$, one for $v_j$).
      \item So the total of vertices is 2 times the total of edges.
    \end{itemize}
  \end{proof}\bigskip

  Because of the lemma, it is impossible to make a graph with vertice degrees 3, 2, 2, 2.
\end{frame}

\subsection{Isomorphism}

\begin{frame}{Review: Isomorphism in graphs}{Remember: an isomorphism is an edge preserving bijection}
    \begin{columns}
      \column{0.5\textwidth}
      \begin{tikzpicture}[scale=1.5,auto,swap]
        %\tikzset{edge/.style = {->,>=latex'}}
        \node[vertex] (257) at (0,2) {257};
        \node[vertex] (122) at (1,2) {122};
        \node[vertex] (145) at (2,2) {145};
        \node[vertex] (306) at (0,0) {306};
        \node[vertex] (67) at (2,0) {67};
        \node[vertex] (99) at (1,-1) {99};
        \draw[edge] (257) to (122);
        \draw[edge] (257) to (99);
        \draw[edge] (122) to (99);
        \draw[edge] (306) to (99);
        \draw[edge] (67) to (99);
        \draw[edge] (306) to (67);
        \draw[edge] (306) to (145);
        \draw[edge] (145) to (99);
      \end{tikzpicture}

      \column{0.5\textwidth}
      \begin{tikzpicture}[scale=1.5,auto,swap]
        %\tikzset{edge/.style = {->,>=latex'}}
        \node[vertex] (257) at (0,2) {257};
        \node[vertex] (122) at (2,2) {122};
        \node[vertex] (145) at (2,-1) {145};
        \node[vertex] (306) at (1,-1) {306};
        \node[vertex] (67) at (0,-1) {67};
        \node[vertex] (99) at (1,1) {99};
        \draw[edge] (257) to (122);
        \draw[edge] (257) to (99);
        \draw[edge] (122) to (99);
        \draw[edge] (306) to (99);
        \draw[edge] (67) to (99);
        \draw[edge] (306) to (67);
        \draw[edge] (306) to (145);
        \draw[edge] (145) to (99);
      \end{tikzpicture}

    \end{columns}\bigskip

    The left and the right are \structure{the same graph}, but with different positions for the vertices.
\end{frame}

\begin{frame}{Review: Isomorphism in graphs}{Remember: Isomorphism is an edge preserving vertex bijection}
    \begin{columns}
      \column{0.5\textwidth}
      \begin{tikzpicture}[scale=1.5,auto,swap]
        %\tikzset{edge/.style = {->,>=latex'}}
        \node[vertex] (257) at (0,2) {257};
        \node[vertex] (122) at (1,2) {122};
        \node[vertex] (145) at (2,2) {145};
        \node[vertex] (306) at (0,0) {306};
        \node[vertex] (67) at (2,0) {67};
        \node[vertex] (99) at (1,-1) {99};
        \draw[edge] (257) to (122);
        \draw[edge] (257) to (99);
        \draw[edge] (122) to (99);
        \draw[edge] (306) to (99);
        \draw[edge] (67) to (99);
        \draw[edge] (306) to (67);
        \draw[edge] (306) to (145);
        \draw[edge] (145) to (99);
      \end{tikzpicture}

      \column{0.5\textwidth}
      \begin{tikzpicture}[scale=1.5,auto,swap]
        %\tikzset{edge/.style = {->,>=latex'}}
        \node[vertex] (257) at (0,2) {Aki};
        \node[vertex] (122) at (1,2) {Bob};
        \node[vertex] (145) at (2,2) {Gus};
        \node[vertex] (306) at (0,0) {Lyn};
        \node[vertex] (67) at (2,0) {Taro};
        \node[vertex] (99) at (1,-1) {Emi};
        \draw[edge] (257) to (122);
        \draw[edge] (257) to (99);
        \draw[edge] (122) to (99);
        \draw[edge] (306) to (99);
        \draw[edge] (67) to (99);
        \draw[edge] (306) to (67);
        \draw[edge] (306) to (145);
        \draw[edge] (145) to (99);
      \end{tikzpicture}

    \end{columns}\bigskip

    The left and the right are \structure{the same graph}, but with different {\bf labels} for the vertices.
\end{frame}

\begin{frame}{Isomorphism}

    \begin{itemize}
    \item Graph Isomorphism is determined solely by the edges between vertices;\bigskip

    \item Two graphs with the same edge connections are \structure{isomorphic};\bigskip

    \item Formally, wwo graphs are isomorphic if there is an \structure{Edge Preserving Matching Relation} between their vertices;\bigskip
    \end{itemize}
\end{frame}

\begin{frame}{Isomorphism}{Are these graphs Isomorphic?}

    \begin{columns}
      \column{0.5\textwidth}
      \begin{tikzpicture}[scale=1.5,auto,swap]
        %\tikzset{edge/.style = {->,>=latex'}}
        \node[vertex] (a) at (0,0) {Cow};
        \node[vertex] (b) at (2,0) {Cat};
        \node[vertex] (c) at (0,2) {Dog};
        \node[vertex] (d) at (2,2) {Pig};
        \draw[edge] (a) to (b);
        \draw[edge] (b) to (d);
        \draw[edge] (d) to (c);
        \draw[edge] (c) to (a);
        \draw[edge] (c) to (b);
      \end{tikzpicture}
      \column{0.5\textwidth}
      \begin{tikzpicture}[scale=1.5,auto,swap]
        %\tikzset{edge/.style = {->,>=latex'}}
        \node[vertex] (a) at (1,2) {Hay};
        \node[vertex] (b) at (2,0) {Tuna};
        \node[vertex] (c) at (0,0) {Beef};
        \node[vertex] (d) at (1,1) {Corn};
        \draw[edge] (a) to (b);
        \draw[edge] (b) to (d);
        \draw[edge] (d) to (c);
        \draw[edge] (c) to (a);
        \draw[edge] (c) to (b);
      \end{tikzpicture}
    \end{columns}\bigskip

    Edge Preserving Bijection:\\
    f(dog) = Beef; \hspace{2cm} f(cow) = Hay\\
    f(cat) = Tuna; \hspace{2cm} f(pig) = Corn
\end{frame}

\begin{frame}{Graph Isomorphism}{Edge Preserving Bijection}

    $G_1$ \structure{isomorphic} to $G_2$ means that $\exists$
    Edge Preserving Vertex Matching:
    \begin{equation*}
      \exists f:V_1 \rightarrow V_2,
      (u,v) \in E_1 \iff (f(u),f(v)) \in E_2
    \end{equation*}\bigskip

    It is easy to quickly identify {\bf \alert{non-isomorphic}} graphs:
    \begin{itemize}
    \item Not the same number of vertices;
    \item Not the same number of edges;
    \item Not the same degree distribution;
    \item Differences in Paths, Distances, etc...
    \end{itemize}
\end{frame}

\begin{frame}{How to find Graph Isomorphism?}
    \begin{itemize}
      \item Finding the bijection is very hard:
      \begin{itemize}
        \item Number of bijections: permutation on $|V|$
      \end{itemize}\bigskip

    \item If the graph is "small", can check the permutations by hand;\bigskip

    \item If the graph is "large", create random matchings $f: V_1 \rightarrow V_2$, and check:
      \begin{itemize}
        \item Quickly prune matchings that are {\bf not} isomorphic:
        \item Vertices in the bijection must have the same degree. (ex: a vertice with edge 4 must match to another vertice with edge 4)
        \item \emph{Adjacent vertices} must match degree as well. (ex: A vertice with degree 3, and neighbors with degree 4, 2, 1)
      \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{How to find Graph Isomorphism?}

  Finding an isomorphism for two graphs is a very expensive, and important, problem. In theory, there is no algorithm that is better than just checking every possible bijection.

  \begin{center}
    \includegraphics[width=0.8\textwidth]{../img/isomorphism}
    \pagenote{Isomorphism image from MIT OCW materials}
  \end{center}
\end{frame}


\section{Empirical Risk Minimisation (ERM)}

% ===
\emph{Cost:}\enspace
$R(c,X,Y) = \sum_{i\leq N} \norm{y_i - c^\top \bm x_i}^2$\quad (regr.) \\
\quad\textit{or}\quad
$R(c,X,Y) = \sum_{i\leq N} \max(0, -y_i c^\top \bm x)$\quad (class.) \\
\quad\textit{or}\quad
$R(c,\theta,X) = \sum_{i\leq N} \norm{\bm x_i - \theta_{c(i)}}^2$\quad (clust.)

\emph{Goal:}\enspace
$\arg\min\limits_c \E[\mathcal{X}]{ R(c,\mathcal{X} } \approx \arg\min\limits_c \frac1N R(c, X)$

% ===
\subsection{Bayesianism / Frequentism}

\emph{Bayesianism:}\enspace
Define prior $P(\theta)$, define likelihood $P(X\mid\theta)$, compute posterior $P(\theta\mid x_{1...n})$.
\\
\textbf{Bayes:}\enspace
$P(\theta\mid X) = \frac{P(X\mid\theta)P(\theta)}{P(X)}$,
{\footnotesize $P(X) {=} \sum_\theta P(X\vert\theta_i) P(\theta_i)$}

\emph{Frequentism:}\enspace
%Define a parametric model $\theta$ (e.g. $\Gauss{\theta,1}$), compute likelihood of data and compute MLE: $\hat\theta\ped{MLE} = \arg\max_\theta P(y_{1...n}\mid\theta)$.
Define param. model $P(Y\vert X,\theta)$, compute likelihood of data $P((X,Y)\mid\theta)$ and compute $\hat\theta\ped{MLE}$ via $\arg\max_\theta$ of likelihood.

% ===
\subsection{Linear Regression
\hfill{\normalfont\sffamily model:\enspace $\color{subsection-text-color} \hat{\mathbf y} = \bm X \beta$}}

\emph{Ridge:}\enspace
$\epsilon\ped{RSS}(\beta,{\color{OrangeRed}{\lambda}}) = \paren{\bm y - \bm X^\top \beta}^\top \paren{\bm y - \bm X^\top \beta} \color{OrangeRed} + \lambda \beta^\top \beta$ \\\quad
$\hat\beta = (\bm X^\top \bm X {\color{OrangeRed} + \lambda \mathbb I})^{-1} \bm X^\top \bm y$,
\quad prior: $\beta \sim \Gauss{0, \frac{\sigma^2}{\lambda} \mathbb I}$

\emph{Lasso:}\enspace
%$\epsilon\ped{RSS}(\beta,\lambda) = \sum_{i\leq n} (y_i - x_i^\top \beta)^2 + \lambda \norm{\beta}_2$ \\\quad
$\hat\beta = \arg\min_\beta \sum_{i\leq n} (y_i - x_i^\top \beta)^2 \color{OrangeRed} + \lambda \norm{\beta}_2$ \\\quad
\textit{(no closed form)},
\quad prior: $p(\beta_i) = \frac{\lambda}{4\sigma^2} \exp(-\abs{\beta_i} \frac{\lambda}{2\sigma^2})$

% ===
% === === === === === === ===
% ===
\iffalse
    \subsection{Estimation - MLE Properties}

    \emph{Consistency:}
    $\forall\epsilon>0, \; \mathbb P\{ \abs{\hat\theta_n - \theta^\ast} > \epsilon \} \overset{n\to\infty}{\longrightarrow} 0$

    \emph{Equivariance:}
    If $\hat\theta_n$ is MLE of $\theta$, then $g(\hat\theta_n)$ is MLE of $g(\theta)$.

    \emph{Asympt. normality:}\\
    $\sqrt{N} (\hat\theta_n - \theta^\ast) \to \Gauss{0,\, J^{-1}(\theta^\ast) I(\theta^\ast) J^{-1}(\theta^\ast)}$%, where $J = -\E*[x\mid\theta^\ast]{\frac{\partial^2 \log \P{x\mid\theta}}{\partial\theta\partial\theta^\top}}$ and $I(\theta_0) \triangleq \textrm{Fisher info}$.

    \emph{Asympt. efficiency:}
    $\hat\theta_n$ minimises $\E{(\hat\theta_n - \theta^\ast)^2}$ as $n\to\infty$, i.e. $\E{(\hat\theta_n - \theta^\ast)^2} = \frac{1}{I^{(n)}(\theta^\ast)}$ (Rao Cr.)\\
    Among all consistent estimators $\hat\theta_n$ has \textit{smallest variance}: $\lim_{n\to\infty} (\V{\hat\theta_n} I^{(n)}(\theta^\ast))^{-1} = 1$
\fi

% ===
\iffalse
    \subsection{Rao Cramer inequality \hfill {\normalfont\footnotesize all $\mathbb E$ w.r.t. $P(x\mid\theta^\ast)$}}
    %(all $\E{}$ w.r.t. $\P{x\mid\theta^\ast}$)

    Score func.: $\bm\Lambda = \pderiv{\log\P{x\mid\theta}}{\theta}$,\; $\E[x\mid\theta^\ast]{\bm\Lambda} = 0$\\
    Fisher info.: $I^{(n)}(\theta) = \V[x\mid\theta^\ast]{\bm\Lambda}$\\
    $J(\theta) = \E[x\mid\theta^\ast]{\bm\Lambda^2} = -\E*{\frac{\partial^2 \log \P{x\mid\theta}}{\partial\theta\partial\theta^\top}} = -\E*{\pderiv{\bm\Lambda}{\theta}}$

    \emph{General bound:} $\E{(\hat\theta - \theta^\ast)^2} \geq \frac{\paren*{ 1 + \partial \mathrm{b}_{\hat\theta} / \partial\theta }^2}{\E{\bm\Lambda^2}} + \mathrm{b}_{\hat\theta}$

    \emph{Unbiased case:} $\E{(\hat\theta - \theta^\ast)^2} = \V{\hat\theta_n} \geq \frac{1}{I^{(n)}(\theta^\ast)}$

    \emph{Tradeoff:}
    $\E{(\hat\theta_n - \theta^\ast)^2} = \V{\hat\theta_n} + \mathrm{bias}^2(\hat\theta_n)$

    \emph{Bias:}
    $\mathrm{bias}(\hat\theta_n) \equiv \mathrm{b}_{\hat\theta}(\theta^\ast) = \E{\hat\theta_n} - \theta^\ast \overset{\textrm{unbiased}}{=} 0$
\fi

% ===


\documentclass{article}
\usepackage{fullpage}
\usepackage{amsmath}

\newcommand{\F}{\mathbf{F}}
\newcommand{\N}{\mathbf{N}}
\renewcommand{\S}{\mathbf{S}}
\newcommand{\D}{\mathbf{D}}
\newcommand{\X}{\mathbf{X}}
\newcommand{\f}{\mathbf{f}}
\renewcommand{\P}{\mathbf{P}}
\newcommand{\Ds}{\D_s}
\newcommand{\Dm}{\D_m}
\renewcommand{\o}{\mathbf{1}}
\newcommand{\I}{\mathbf{I}}
\newcommand{\tr}{\mbox{tr}}
\newcommand{\m}[1]{\begin{pmatrix}#1\end{pmatrix}}
\newcommand{\pp}[2]{\frac{\partial #1}{\partial #2}}

\begin{document}

\section{Finite Elements}

\begin{tabular}{|c|c|cc|l|}
\hline
Symbol & Definition & Dimensions & Units & Meaning \\
\hline
$\o$ & $\o = \m{1 & \cdots & 1}^T$ & $d$ & $1$ & All ones vector \\
$\S$ & $\S = \m{\I & -\o}$ & $d \times (d+1)$ & $1$ & Scatter matrix \\
\hline
$\X_m$ & $\m{X_1 & X_2 & X_3 & X_4}_m$ & $d \times (d+1)$ & $m$ & Material space element node positions \\
$\X$ & $\m{X_1 & X_2 & X_3 & X_4}$ & $d \times (d+1)$ & $m$ & World space element node positions \\
$\Dm$ & $\Dm = \X_m \S^T$ & $d \times d$ & $m$ & Material space relative node positions \\
$\Ds$ & $\Ds = \X \S^T$ & $d \times d$ & $m$ & World space relative node positions \\
$\F$ & $\F = \Ds \Dm^{-1}$ & $d \times d$ & $1$ & Deformation gradient \\
\hline
$J$ & $J = \det(\F)$ & scalar & $1$ & Jacobian \\
$I_1$ & $I_1 = \tr(\F \F^T)$ & scalar & $1$ & First invariant \\
\hline
$V$ & $V = \frac{1}{d!} \det(\Dm)$ & scalar & $m^d$ & Material space element volume \\
$\N$ & $\N = V \Dm^{-T}$ & $d \times d$ & $m^{d-1}$ & Material space area weighted normals \\
\hline
$\psi$ & - & scalar & $kg \, m^{2-d} s^{-1}$ & Energy density \\
$\phi$ & $\phi = V \psi$ & scalar & $kg \, m^2 s^{-1}$ & Potential energy of element \\
$\f$ & $\f = -\pp{\phi}{\X}$ & $d \times (d+1)$ & $kg \, m s^{-1}$ & Force on element nodes \\
$\P$ & $\f = -\P \N \S^T$ & $d \times d$ & $kg \, m^{2-d} s^{-1}$ & First Piola-Kirchoff stress \\
\hline
\end{tabular}
\vspace{.2in}

Consider that a small position change $\delta \X$ is made.  Then
\begin{eqnarray*}
0 & = & \f : \delta \X + \delta \phi \\
& = & \tr(\f \delta \X^T) + \delta \phi \\
& = & \tr(-\P \N \S^T \delta \X^T) + V \delta \psi \\
& = & -V \tr(\P \Dm^{-T} \S^T \delta \X^T) + V \delta \psi \\
\delta \psi & = & \tr(\P \Dm^{-T} \S^T \delta \X^T) \\
& = & \tr(\P \delta (\Dm^{-T} \S^T \X^T)) \\
& = & \tr(\P \delta (\Dm^{-T} \Ds^T)) \\
& = & \tr(\P \delta \F^T) \\
& = & \P : \delta \F \\
\P & = & \pp{\psi}{\F}
\end{eqnarray*}

\section{Potential Energy for Neo Hookean}

Let $\psi = \frac{\mu}{2} (I_1 - d) - \mu \ln J + \frac{\lambda}{2} \ln^2 J$.

\begin{eqnarray*}
\pp{J}{\F} & = & \pp{}{\F} \det(\F) = \det(\F) \F^{-1} = J \F^{-1} \\
\pp{I_1}{\F} & = & \pp{}{\F} \tr(\F \F^T) = 2 \F \\
\pp{}{\F} & = & \pp{}{\F} \ln J = \F^{-1} \\
\P = \pp{\psi}{\X} & = & \pp{}{\X} \left( \frac{\mu}{2} (I_1 - d) - \mu \ln J + \frac{\lambda}{2} \ln^2 J \right) \\
& = & \frac{\mu}{2} \pp{I_1}{\X} - \mu \pp{}{\F} \ln J + \lambda \ln J \pp{}{\F} \ln J \\
& = & \mu (\F - \F^{-1}) + \lambda \ln J \F^{-1}
\end{eqnarray*}

\end{document}


\section{Eigenvectors and eigenvalues}

\begin{outcome}
  \begin{enumerate}
  \item Determine whether a vector is an eigenvector of a matrix.
  \item Given an eigenvector, find the corresponding eigenvalue.
  \item Given an eigenvalue, find the corresponding eigenvectors.
  \item Find a basis for the eigenspace of a given eigenvalue.
  \end{enumerate}
\end{outcome}

When we multiply a square matrix $A$ by a non-zero vector $\vect{v}$,
we obtain another vector $A\vect{v}$. Most of the time, the vectors
$A\vect{v}$ and $\vect{v}$ are unrelated; they could point in
completely different directions. However, sometimes it can happen that
$A\vect{v}$ is a scalar multiple of $\vect{v}$. In that case,
$\vect{v}$ is called an \textbf{eigenvector} of $A$. We will see later
in this chapter that we can learn a lot about the matrix $A$ by
considering its eigenvectors.

\begin{definition}{Eigenvalues and eigenvectors}{eigenvalues-and-eigenvectors}
  Let $A$ be an $n\times n$-matrix. Suppose that $\vect{v}\in\R^n$ is
  a non-zero vector such that $A\vect{v}$ is a scalar multiple of
  $\vect{v}$. In other words, suppose that there exists a scalar
  $\eigenvar$ such that
  \begin{equation*}
    A\vect{v}=\eigenvar\vect{v}.
  \end{equation*}
  Then $\vect{v}$ is called an \textbf{eigenvector}%
  \index{eigenvector}%
  \index{vector!eigenvector}%
  \index{matrix!eigenvector} of $A$, and $\eigenvar$ is called the
  corresponding \textbf{eigenvalue}%
  \index{eigenvalue}%
  \index{matrix!eigenvalue}.
\end{definition}

\begin{example}{Eigenvalues and eigenvectors}{eigenvalues-and-eigenvectors}
  Consider the matrix
  \begin{equation*}
    A = \begin{mymatrix}{rrr}
      3 & -2 &  2 \\
      1 &  2 &  1 \\
      0 &  2 &  1 \\
    \end{mymatrix}.
  \end{equation*}
  Which of the following vectors are eigenvectors of $A$? Find the
  corresponding eigenvalues.
  \begin{equation*}
    \vect{v}_1 = \begin{mymatrix}{r} 2 \\ -1 \\ -2 \end{mymatrix},\quad
    \vect{v}_2 = \begin{mymatrix}{r} 0 \\ 1 \\ 1 \end{mymatrix},\quad
    \vect{v}_3 = \begin{mymatrix}{r} 1 \\ 1 \\ 1 \end{mymatrix},\quad
    \vect{v}_4 = \begin{mymatrix}{r} 0 \\ 0 \\ 0 \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  We compute
  \begin{equation*}
    A\vect{v}_1 = \begin{mymatrix}{r} 4 \\ -2 \\ -4 \end{mymatrix},\quad
    A\vect{v}_2 = \begin{mymatrix}{r} 0 \\ 3 \\ 3 \end{mymatrix},\quad
    A\vect{v}_3 = \begin{mymatrix}{r} 3 \\ 4 \\ 3 \end{mymatrix},\quad
    A\vect{v}_4 = \begin{mymatrix}{r} 0 \\ 0 \\ 0 \end{mymatrix}.
  \end{equation*}
  \begin{itemize}
  \item We see that $A\vect{v}_1$ is a scalar multiple of
    $\vect{v}_1$, namely $A\vect{v}_1=2\vect{v}_1$. Therefore,
    $\vect{v}_1$ is an eigenvector of $A$ with corresponding
    eigenvalue $\eigenvar=2$.
  \item Similarly, $A\vect{v}_2=3\vect{v}_2$, so $\vect{v}_2$ is an
    eigenvector of $A$ with corresponding eigenvalue $\eigenvar=3$.
  \item On the other hand, $A\vect{v}_3$ is not a scalar multiple of
    $\vect{v}_3$. Hence, $\vect{v}_3$ is not an eigenvector of $A$.
  \item Finally, although $A\vect{v}_4$ is a scalar multiple of
    $\vect{v}_4$, the zero vector is not considered an eigenvector.
  \end{itemize}
\end{solution}

\begin{example}{Find eigenvectors for the given eigenvalue}{find-eigenvectors-given-eigenvalue}
  Let
  \begin{equation*}
    A = \begin{mymatrix}{rrr}
      2  &  0 & 0 \\
      -1 &  3 & 1 \\
      2  & -2 & 0 \\
    \end{mymatrix}.
  \end{equation*}
  Find the eigenvectors corresponding to the eigenvalue $\eigenvar=2$.
\end{example}

\begin{solution}
  We have to solve the equation $A\vect{v}=2\vect{v}$. We can use
  algebra to rewrite this as
  \begin{eqnarray*}
    A\vect{v} = 2\vect{v}
    &\iff& A\vect{v} - 2\vect{v} = \vect{0} \\
    &\iff& (A-2I)\vect{v} = \vect{0} \\
    &\iff& \begin{mymatrix}{rrr}
      0  &  0 & 0 \\
      -1 &  1 & 1 \\
      2  & -2 & -2 \\
    \end{mymatrix}\vect{v}
    = \begin{mymatrix}{r} 0 \\ 0 \\ 0 \end{mymatrix}.
  \end{eqnarray*}
  This is a homogeneous system of equations with general
  solution
  \begin{equation*}
    \vect{v}
    ~=~ s \begin{mymatrix}{r} 1 \\ 1 \\ 0 \end{mymatrix}
    + t \begin{mymatrix}{r} 1 \\ 0 \\ 1 \end{mymatrix},
  \end{equation*}
  where $s$ and $t$ are parameters. These (except the zero vector) are
  exactly the eigenvectors corresponding to the eigenvalue $\eigenvar = 2$.
\end{solution}

As the last example shows, the eigenvectors for a given eigenvalue
$\eigenvar$, plus the zero vector, form a subspace of $\R^n$. This is
called the \textbf{eigenspace} of $\eigenvar$.

\begin{definition}{Eigenspace}{eigenspace}
  Let $A$ be an $n\times n$-matrix, and let $\eigenvar$ be an eigenvalue
  of $A$. The \textbf{eigenspace}%
  \index{eigenspace}%
  \index{matrix!eigenspace}%
  \index{subspace!eigenspace} of $\eigenvar$ is the set
  \begin{equation*}
    E_{\eigenvar} = \set{\vect{v} \mid A\vect{v}=\eigenvar\vect{v}}.
  \end{equation*}
  It is a subspace of $\R^n$.
\end{definition}

Instead of finding {\em all} eigenvectors for a given eigenvalue, it
is often sufficient to find a basis for the eigenspace. We also
sometimes call the basis vectors of the eigenspace \textbf{basic
  eigenvectors}%
\index{basic eigenvector}%
\index{eigenvector!basic}%
\index{matrix!eigenvector!basic}.

\begin{example}{Basis of eigenspace}{basis-eigenspace}
  Let
  \begin{equation*}
    A = \begin{mymatrix}{rrr}
      2  &  0 & 0 \\
      -1 &  3 & 1 \\
      2  & -2 & 0 \\
    \end{mymatrix}.
  \end{equation*}
  The matrix $A$ has eigenvalues $\eigenvar=1$ and $\eigenvar=2$. Find a
  basis for each eigenspace.
\end{example}

\begin{solution}
  We already found a basis for the eigenspace $E_2$ in
  Example~\ref{exa:find-eigenvectors-given-eigenvalue}.
  \begin{equation*}
    \mbox{Basis of $E_2$:}\quad\set{
      \begin{mymatrix}{r} 1 \\ 1 \\ 0 \end{mymatrix},\quad
      \begin{mymatrix}{r} 1 \\ 0 \\ 1 \end{mymatrix}
    }.
  \end{equation*}
  To find a basis for the eigenspace $E_1$, we proceed analogously.
  We must solve the equation $A\vect{v}=1\vect{v}$. We have:
  \begin{eqnarray*}
    A\vect{v} = 1\vect{v}
    &\iff& A\vect{v} - \vect{v} = \vect{0} \\
    &\iff& (A-I)\vect{v} = \vect{0} \\
    &\iff& \begin{mymatrix}{rrr}
      1  &  0 & 0 \\
      -1 &  2 & 1 \\
      2  & -2 & -1 \\
    \end{mymatrix}\vect{v}
    = \begin{mymatrix}{r} 0 \\ 0 \\ 0 \end{mymatrix}.
  \end{eqnarray*}
  This is a homogeneous system of rank 2, with general solution
  \begin{equation*}
    \vect{v}
    ~=~ t \begin{mymatrix}{r} 0 \\ -1 \\ 2 \end{mymatrix}.
  \end{equation*}
  Thus, the following is a basis for the eigenspace $E_1$:
  \begin{equation*}
    \mbox{Basis of $E_1$:}\quad\set{
      \begin{mymatrix}{r} 0 \\ -1 \\ 2 \end{mymatrix}
    }.
  \end{equation*}
\end{solution}


\lab{CVXOPT}{CVXOPT}
\objective{CVXOPT is a package of Python functions and classes designed for the purpose of convex optimization.
In this lab we use these tools for linear and quadratic programming.
We will solve various optimization problems using CVXOPT and optimize eating healthily on a budget.}

%\begin{warn}
%CVXOPT is not part of the standard library, and it is only included in the Anaconda distribution for Python 3.6 for Linux and Mac.
%We recommend avoiding Windows machines for this lab.
%
%To install CVXOPT, use \li{conda install cvxopt} or \li{pip install cvxopt}.
%\end{warn}

\section*{Linear Programs} % ==================================================

%%Cvxopt has linear program solver and can implement integer programming through the Gnu Linear Programming Kit, glpk.
%CVXOPT is a package of Python functions and classes designed for the purpose of convex optimization.
%In this lab we will focus on linear and quadratic programming.
A \emph{linear program} is a linear constrained optimization problem. Such a problem can be stated in several
different forms, one of which is
\begin{align*}
\text{minimize}\qquad &\c\trp \x \\
\text{subject to}\qquad &G\x \preceq \mathbf{h}\\
&A\x = \b.
\end{align*}

The symbol $\preceq$ denotes that the components of $G\x$ are less than the components of $\mathbf{h}$. In other words, if $\x\preceq\y$, then $x_i < y_i$ for all $x_i\in\x$ and $y_i\in\y$.

Define vector $\mathbf{s} \succeq \0$ such that the constraint $G\x + \mathbf{s} = \mathbf{h}$.
This vector is known as a \emph{slack variable}.
Since $\mathbf{s} \succeq \0$, the constraint
$G\x + \mathbf{s} = \mathbf{h}$ is equivalent to $G\x \preceq \mathbf{h}$.

With a slack variable, a new form of the linear program is found:
\begin{align*}
\text{minimize}\qquad &\c\trp \x \\
\text{subject to}\qquad &G\x + \mathbf{s} = \mathbf{h}\\
&A\x = \b \\
&\mathbf{s} \succeq \0.
\end{align*}

This is the formulation used by CVXOPT.
It requires that the matrix $A$ has full row rank,
and that the block matrix $[G \quad A]\trp $ has full column rank.

% \preceq \succeq

% Students have not yet learned about the dual problem. May be included in a later lab.
\begin{comment}
The corresponding \emph{dual program} for the above linear program has the form
\begin{align*}
\text{maximize}\qquad &-h\trp z - b\trp y \\
\text{subject to}\qquad &G\trp z + A\trp y + c = 0\\
 &z \geq 0.
\end{align*}
CVXOPT provides functions to solve both the original (\emph{primal}) linear program and its dual program.
\end{comment}

Consider the following example:
\begin{align*}
\text{minimize}\qquad &-4x_1-5x_2 \\
\text{subject to}\qquad &x_1+2x_2 \leq 3 \\
	        &2x_1+x_2 = 3 \\
		&x_1, x_2 \geq 0
\end{align*}
Recall that all inequalities must be less than or equal to, so that $G\x\preceq \mathbf{h}$.
Because the final two constraints are $x_1, x_2 \geq 0$, they need to be adjusted to be $\leq$ constraints.
This is easily done by multiplying by $-1$, resulting in the constraints $-x_1, -x_2 \leq 0$.
If we define
\[
G = \begin{bmatrix}
  1 & 2\\
  -1 & 0\\
  0 & -1
\end{bmatrix} \text{, } \qquad
\mathbf{h} = \begin{bmatrix}
  3\\
  0\\
  0
\end{bmatrix} \text{, } \qquad
A = \begin{bmatrix}
2 & 1
\end{bmatrix} \text{, } \quad \text{and } \qquad
\mathbf{b} = \begin{bmatrix}
3
\end{bmatrix}
\]
then we can express the constraints compactly as
\[
\begin{matrix}
G\x \preceq \mathbf{h},\\
A\x = \mathbf{b},
\end{matrix}  \qquad \text{where} \qquad
\x = \begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix}.
\]
By adding a slack variable $\mathbf{s}$, we can write our constraints as
\[
G\x + \mathbf{s} = \mathbf{h},
\]
which matches the form discussed above.
% In the case of this particular example, we ignore the extra constraint
%\[
%A\x = \b,
%\]
%since we were given no equality constraints.

To solve the problem using CVXOPT, initialize the arrays $\c$, $G$, $\mathbf{h}$, $A$, and $\mathbf{b}$ and pass them to the appropriate function.
CVXOPT uses its own data type for an array or matrix.
While similar to the NumPy array, it does have a few differences, especially when it comes to initialization.
Below, we initialize CVXOPT matrices for $\mathbf{c}$, $G$, $\mathbf{h}$, $A$, and $\mathbf{b}$.
We then use the CVXOPT function for linear programming \li{solvers.lp()}, which accepts $\c$, $G$, $\mathbf{h}$, $A$, and $\b$ as arguments.

\begin{lstlisting}
>>> from cvxopt import matrix, solvers

>>> c = matrix([-4., -5.])
>>> G = matrix([[1., -1., 0.],[2., 0., -1.]])
>>> h = matrix([ 3., 0., 0.])
>>> A = matrix([[2.],[1.]])
>>> b = matrix([3.])

>>> sol = solvers.lp(c, G, h, A, b)
     pcost       dcost       gap    pres   dres   k/t
 0: -8.5714e+00 -1.4143e+01  4e+00  0e+00  3e-01  1e+00
 1: -8.9385e+00 -9.2036e+00  2e-01  3e-16  1e-02  3e-02
 2: -8.9994e+00 -9.0021e+00  2e-03  3e-16  1e-04  3e-04
 3: -9.0000e+00 -9.0000e+00  2e-05  1e-16  1e-06  3e-06
 4: -9.0000e+00 -9.0000e+00  2e-07  1e-16  1e-08  3e-08
Optimal solution found.
>>> print(sol['x'])
[ 1.00e+00]
[ 1.00e+00]
>>> print(sol['primal objective'])
-8.999999939019435
>>> print(type(sol['x']))
<<<class 'cvxopt.base.matrix'>>>
\end{lstlisting}

\begin{warn}
CVXOPT matrices only accept floats.
Other data types will raise a \li{TypeError}.

Additionally, CVXOPT matrices are initialized column-wise rather than row-wise (as in the case of NumPy).
Alternatively, we can initialize the arrays first in NumPy (a process with which you should be familiar),
and then simply convert them to the CVXOPT matrix data type.
\begin{lstlisting}
>>> import numpy as np

>>> c = np.array([-4., -5.])
>>> G = np.array([[1., 2.],[-1., 0.],[0., -1]])
>>> h = np.array([3., 0., 0.])
>>> A = np.array([[2., 1.]])
>>> b = np.array([3.])

# Convert the arrays to the CVXOPT matrix type.
>>> c = matrix(c)
>>> G = matrix(G)
>>> h = matrix(h)
>>> A = matrix(A)
>>> b = matrix(b)
\end{lstlisting}
In this lab we will initialize non-trivial matrices first as NumPy arrays for consistency.

%Finally, be sure the entries in the matrices are floats!
\end{warn}

%Having initialized the necessary objects, we are now ready to solve the problem.

\begin{info}
Although it is often helpful to see the progress of each iteration of the algorithm, you may suppress this output by first running,
\begin{lstlisting}
solvers.options['show_progress'] = False
\end{lstlisting}
\end{info}

The function \li{solvers.lp()} returns a dictionary containing useful information.
For now, we will only focus on the value of $\x$ and the primal objective value (i.e. the minimum value achieved by the objective function).

\begin{warn}
Note that the minimizer \li{x} returned by the \li{solvers.lp()} function is a \li{cvxopt.base.matrix} object.
\li{np.ravel()} is a NumPy function that takes an object and returns its values as a flattened NumPy array.
Use \li{np.ravel()} to return all minimizers in this lab as flattened NumPy arrays.
\end{warn}

\begin{problem}
Solve the following convex optimization problem:
\begin{align*}
\text{minimize}\qquad &2x_1+x_2+3x_3 \\
\text{subject to}\qquad &x_1+2x_2 \geq 3 \\
	        &2x_1+10x_2+3x_3 \geq 10 \\
		&x_1 \geq 0 \\
		&x_2 \geq 0 \\
		&x_3 \geq 0
\end{align*}
Return the minimizer $\x$ and the primal objective value.
\\(Hint: make the necessary adjustments so that all inequality constraints are $\leq$ rather than $\geq$).
\end{problem}

\subsection*{$l_1$ Norm}
The $l_1$ norm is defined
\[||\x||_1=\sum_{i=1}^n |x_i|.\]
A $l_1$ minimization problem is minimizing a vector's $l_1$ norm, while fitting certain constraints. It can be written in the following form:
\begin{align*}
\text{minimize}\qquad &\|\x\|_1\\
\text{subject to} \qquad &A\x = \b.
\end{align*}

This problem can be converted into a linear program by introducing an additional vector $\u$ of length $n$.
Define $\u$ such that $|x_i|\leq u_i$.
Thus, $-u_i-x_i\leq 0$ and $-u_i+x_i\leq 0$.
These two inequalities can be added to the linear system as constraints.
Additionally, this means that $||\x||_1\leq ||\u||_1$.
So minimizing $||\u||_1$ subject to the given constraints will in turn minimize $||\x||_1$.
This can be written as follows:
\begin{align*}
\text{minimize}\qquad
&\begin{bmatrix}
\mathbf{1}\trp & \0\trp
\end{bmatrix}
\begin{bmatrix}
\u \\
\x
\end{bmatrix}\\
\text{subject to}\qquad
&\begin{bmatrix}
-I & I\\
-I & -I\\
-I & \0
\end{bmatrix}
\begin{bmatrix}
\u \\
\x
\end{bmatrix}
\preceq
\begin{bmatrix}
0\\
0
\end{bmatrix},\\
&\begin{bmatrix}
\0 & A
\end{bmatrix}
\begin{bmatrix}
\u \\
\x
\end{bmatrix}
=
\b.
\end{align*}
Solving this gives values for the optimal $\u$ and the optimal $\x$, but we only care about the optimal $\x$.

\begin{problem}
Write a function called \li{l1Min()} that accepts a matrix $A$ and vector $\mathbf{b}$ as NumPy arrays and solves the $l_1$ minimization problem.
Return the minimizer $\x$ and the primal objective value.
Remember to first discard the unnecessary $u$ values from the minimizer.

To test your function consider the matrix $A$ and vector $\mathbf{b}$ below.
\[
A = \begin{bmatrix}
1 & 2 & 1 & 1\\
0 & 3 & -2 & -1
\end{bmatrix} \qquad
\mathbf{b} = \begin{bmatrix}
7 \\
4
\end{bmatrix}
\]
The linear system $A\x = \b$ has infinitely many solutions.
Use \li{l1Min()} to verify that the solution which minimizes $||\mathbf{x}||_1$ is approximately $\x = [0., 2.571, 1.857, 0.]^T$ and the minimum objective value is approximately $4.429$.
\label{prob:l1}
\end{problem}

\section*{The Transportation Problem}

Consider the following transportation problem:
A piano company needs to transport thirteen pianos from their three  supply centers (denoted by 1, 2, 3) to two demand centers (4, 5).
Transporting a piano from a supply center to a demand center incurs a cost, listed in Table \ref{tab:cost}.
The company wants to minimize shipping costs for the pianos while meeting the demand.
%How many pianos should each supply center send to each demand center?

\begin{table}[H]
\centering
\begin{tabular}{|c|c|}
Supply Center & Number of pianos available\\
\hline
1 & 7\\
2 & 2\\
3 & 4\\
\end{tabular}

\caption{Number of pianos available at each supply center}
\label{tab:supply}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{|c|c|}
Demand Center & Number of pianos needed\\
\hline
4 & 5\\
5 & 8\\
\end{tabular}

\caption{Number of pianos needed at each demand center}
\label{tab:demand}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{|c|c|c|c|}
Supply Center & Demand Center & Cost of transportation & Number of pianos\\
\hline
1 & 4 & 4 & $p_1$\\
1 & 5 & 7 & $p_2$\\
2 & 4 & 6 & $p_3$\\
2 & 5 & 8 & $p_4$\\
3 & 4 & 8 & $p_5$\\
3 & 5 & 9 & $p_6$\\
\end{tabular}
\caption{Cost of transporting one piano from a supply center to a demand center}
\label{tab:cost}
\end{table}

A system of constraints is defined for the variables $p_1,p_2,p_3,p_4,p_5,$ and $p_6$,
First, there cannot be a negative number of pianos so the variables must be nonnegative.
Next, the Tables \ref{tab:supply} and \ref{tab:demand} define the following three supply constraints and two demand constraints:
\begin{align*}
p_1 + p_2  &= 7\\
p_3 + p_4  &= 2\\
p_5 + p_6  &= 4\\
p_1 + p_3 + p_5 &= 5\\
p_2 + p_4 + p_6 &= 8
\end{align*}

The objective function is the number of pianos shipped from each location multiplied by the respective cost (found in Table \ref{tab:cost}):
\[
4p_1 + 7p_2 + 6p_3 + 8p_4 + 8p_5 + 9p_6.
\]

\begin{info}
Since our answers must be integers, in general this problem turns out to be an NP-hard problem.
There is a whole field devoted to dealing with integer constraints, called \emph{integer linear programming}, which is beyond the scope of this lab.
Fortunately, we can treat this particular problem as a standard linear program and still obtain integer solutions.
\end{info}

Recall the variables are nonnegative, so $p_1,p_2,p_3,p_4,p_5,p_6\geq 0$.
Thus, $G$ and $\mathbf{h}$ constrain the variables to be non-negative.
Because CVXOPT uses the format $G\x \preceq \mathbf{h}$, we see that this inequality must be multiplied by $-1$.
So, $G$ must be a $6 \times 6$ identity matrix multiplied by $-1$, and

$\mathbf{h}$ is a column vector of zeros.
Since the supply and demand constraints are equality constraints, they are $A$ and $\b$.
Initialize these arrays and solve the linear program by entering the code below.
\begin{lstlisting}
>>> c = matrix(np.array([4., 7., 6., 8., 8., 9.]))
>>> G = matrix(-1*np.eye(6))
>>> h = matrix(np.zeros(6))
>>> A = matrix(np.array([[1.,1.,0.,0.,0.,0.],
                         [0.,0.,1.,1.,0.,0.],
                         [0.,0.,0.,0.,1.,1.],
                         [1.,0.,1.,0.,1.,0.],
                         [0.,1.,0.,1.,0.,1.]]))
>>> b = matrix(np.array([7., 2., 4., 5., 8.]))
>>> sol = solvers.lp(c, G, h, A, b)
     pcost       dcost       gap    pres   dres   k/t
 0:  8.9500e+01  8.9500e+01  2e+01  2e-16  2e-01  1e+00
 1:  8.7023e+01  8.7044e+01  3e+00  1e-15  3e-02  2e-01
Terminated (singular KKT matrix).
>>> print(sol['x'])
[ 4.31e+00]
[ 2.69e+00]
[ 3.56e-01]
[ 1.64e+00]
[ 3.34e-01]
[ 3.67e+00]
>>> print(sol['primal objective'])
87.023
\end{lstlisting}
Notice that some problems occurred. First, CVXOPT alerted us to the fact that the algorithm terminated prematurely (due to a singular matrix).
Second, the minimizer and solution obtained do not consist of integer entries.

So what went wrong? Recall that the matrix $A$ is required to have full row rank, but we can easily see that the rows of $A$
are linearly dependent. We rectify this by converting the last row of the equality constraints into two \emph{inequality} constraints, so that
the remaining equality constraints define a new matrix $A$ with linearly independent rows.

This is done as follows:

 Suppose we have the equality constraint
\[
x_1 + 2x_2 - 3x_3 = 4.
\]
This is equivalent to the pair of inequality
constraints
\begin{align*}
x_1 + 2x_2 - 3x_3 &\leq 4, \\
x_1 + 2x_2 - 3x_3 &\geq 4.
\end{align*}
The linear program requires only $\leq$ constraints, so we obtain the pair
of constraints
\begin{align*}
x_1 + 2x_2 - 3x_3 &\leq 4, \\
-x_1 - 2x_2 + 3x_3 &\leq -4.
\end{align*}

Apply this process to the last equality constraint of the transportation problem.
Then define a new matrix $G$ with several additional rows (to account for the new inequality
constraints), a new vector $\mathbf{h}$ with more entries, a smaller matrix $A$, and a smaller vector $\b$.
\begin{problem}
Solve the transportation problem by converting the last equality constraint into an inequality constraint.
Return the minimizer $\x$ and the primal objective value.
\end{problem}

\begin{comment}
\section*{Example}

Why are all of the terms in $G$ and $\mathbf{h}$ non-positive?

\begin{lstlisting}
>>> from cvxopt import matrix, solvers
>>> G = matrix([ [-1., 0., 0., -1., 0.,  -1., 0., 0., 0., 0., 0.],
             [-1., 0., 0., 0., -1.,  0., -1., 0., 0., 0., 0.],
             [0., -1., 0., -1., 0.,  0., 0., -1., 0., 0., 0.],
             [0., -1., 0., 0., -1.,  0., 0., 0., -1., 0., 0.],
             [0., 0., -1., -1., 0.,  0., 0., 0., 0., -1., 0.],
             [0., 0., -1., 0., -1.,  0., 0., 0., 0., 0., -1.] ])

>>> h = matrix([-7., -2., -4., -5., -8.,  0., 0., 0., 0., 0., 0.,])
>>> c = matrix([4., 7., 6., 8., 8., 9])
>>> sol = solvers.lp(c,G,h)
>>> print sol['x']
>>> print sol['primal objective']
\end{lstlisting}

Another method is to use an integer linear program.
Cvxopt is configured to work with  Gnu, which does have an integer linear program.
It will work with either of the methods above.

\textbf{Example}

glpk.ilp returns a tuple.
The first entry describes the optimality of the result, while the second gives the $x$ values.

\begin{lstlisting}
>>> from cvxopt import matrix, solvers, glpk
>>> G = matrix([ [-1., 0., 0., -1., 0.,  -1., 0., 0., 0., 0., 0.],
             [-1., 0., 0., 0., -1.,  0., -1., 0., 0., 0., 0.],
             [0., -1., 0., -1., 0.,  0., 0., -1., 0., 0., 0.],
             [0., -1., 0., 0., -1.,  0., 0., 0., -1., 0., 0.],
             [0., 0., -1., -1., 0.,  0., 0., 0., 0., -1., 0.],
             [0., 0., -1., 0., -1.,  0., 0., 0., 0., 0., -1.] ])

>>> h = matrix([-7., -2., -4., -5., -8.,  0., 0., 0., 0., 0., 0.,])
>>> o = matrix([4., 7., 6., 8., 8., 9])
>>> sol = glpk.ilp(o,G,h)
>>> print sol[1]
\end{lstlisting}

or
\begin{lstlisting}
>>> from cvxopt import matrix, solvers, glpk
>>> G = matrix([ [-1., 0., 0., 0., 0., 0.],
             [0., -1., 0., 0., 0., 0.],
             [0., 0., -1., 0., 0., 0.],
             [0., 0., 0., -1., 0., 0.],
             [0., 0., 0., 0., -1., 0.],
             [0., 0., 0., 0., 0., -1.] ])

>>> h = matrix([ 0., 0., 0., 0., 0., 0.,])
>>> o = matrix([4., 7., 6., 8., 8., 9])
>>> A = matrix([ [1., 0., 0., 1., 0.],
             [1., 0., 0., 0., 1.],
             [0., 1., 0., 1., 0.],
             [0., 1., 0., 0., 1.],
             [0., 0., 1., 1., 0.],
             [0., 0., 1., 0., 1.] ])
>>> b = matrix([7., 2., 4., 5., 8])
>>> sol = glpk.ilp(o,G,h,A,b)
>>> print sol[1]
\end{lstlisting}

\textbf{Problem 2}
Choose one of these methods and compare the optimal values for the integer linear program to the result you received above.

\textbf{Problem 3}
Create the dual problem for the linear program and solve.
Compare your answer to the dual value cvxopt returned.
\end{comment}

\section*{Quadratic Programming}

Quadratic programming is similar to linear programming, but the objective function is quadratic rather than linear.
The constraints, if there are any, are still of the same form.
Thus, $G, \mathbf{h}, A$, and $\b$ are optional.
The formulation that we will use is
\begin{align*}
\text{minimize}\qquad &\frac{1}{2}\x\trp Q\x + \mathbf{r}\trp \x \\
\text{subject to}\qquad &G\x \preceq \mathbf{h}\\
 &A\x = \b,
\end{align*}
where $Q$ is a positive semidefinite symmetric matrix.
In this formulation, we require again that $A$ has full row rank and that the block matrix
$[Q \quad G \quad A]\trp $ has full column rank.

As an example, consider the quadratic function
\[
f(x_1,x_2) = 2x_1^2 +2x_1x_2 + x_2^2 +x_1 -x_2.
\]
There are no constraints, so we only need to initialize the matrix $Q$ and the vector $\mathbf{r}$.
To find these, we first rewrite our function to match the formulation given above.
If we let
\[
Q = \begin{bmatrix}
  a & b\\
  b & c
\end{bmatrix}, \qquad
\mathbf{r} = \begin{bmatrix}
  d\\
  e
\end{bmatrix},
\qquad \text{and} \qquad
\x = \begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix},
\]
then
\begin{align*}
\frac{1}{2}\x\trp Q\x + \mathbf{r}\trp \x &=
\frac{1}{2}
\begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix}\trp
\begin{bmatrix}
  a & b\\
  b & c
\end{bmatrix}
\begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix} +
\begin{bmatrix}
  d\\
  e
\end{bmatrix}\trp
\begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix} \\
&= \frac{1}{2}ax_1^2 + bx_1x_2 + \frac{1}{2}cx_2^2 + dx_1 + ex_2
\end{align*}
Thus, we see that the proper values to initialize our matrix $Q$ and vector $\mathbf{r}$ are:
\begin{align*}
a &= 4  &d = 1 \\
b &= 2  &e = -1 \\
c &= 2
\end{align*}
Now that we have the matrix $Q$ and vector $\mathbf{r}$, we are ready to use the CVXOPT function for quadratic programming \li{solvers.qp()}.
\begin{lstlisting}
>>> Q = matrix(np.array([[4., 2.], [2., 2.]]))
>>> r = matrix([1., -1.])
>>> sol=solvers.qp(Q, r)
>>> print(sol['x'])
[-1.00e+00]
[ 1.50e+00]
>>> print sol['primal objective']
-1.25
\end{lstlisting}

\begin{problem}
Find the minimizer and minimum of
\begin{equation*}
g(x_1,x_2,x_3) = \frac{3}{2}x_1^2 +2x_1x_2 + x_1x_3+ 2x_2^2 +2x_2x_3+\frac{3}{2}x_3^2+3x_1 + x_3
\end{equation*}
\\(Hint: Write the function $g$ to match the formulation given above before coding.)
\begin{comment}
\begin{equation}
f(x) = \frac{1}{2}x\trp Qx - x\trp p
\end{equation}
where

\begin{center}
$Q =
\begin{bmatrix}
3 & 2 & 1\\
2 & 4 & 2\\
1 & 2 & 3\\
\end{bmatrix}
$
and $p =
\begin{bmatrix}
3\\
0\\
1\\
\end{bmatrix}
$
\end{center}
\end{comment}

\end{problem}

\begin{problem}
The $l_2$ minimization problem is to
\begin{align*}
\text{minimize}\qquad &\|\x\|_2\\
\text{subject to} \qquad &A\x = \b.
\end{align*}

This problem is equivalent to a quadratic program, since $\|\x\|_2 = \x\trp \x$.
Write a function that accepts a matrix $A$ and vector $\b$ and solves the $l_2$ minimization problem.
Return the minimizer $\x$ and the primal objective value.

To test your function, use the matrix $A$ and vector $\b$ from Problem \ref{prob:l1}.
The minimizer is approximately $\x=[0.966, 2.169, 0.809, 0.888]^T$ and the minimum primal objective value is approximately $7.079$.
\end{problem}

\begin{comment}
\section*{Allocation Models}
Allocation models lead to simple linear programs. An allocation model seeks to allocate a valuable resource among competing needs. Consider the following example taken from ``Optimization in Operations Research" by Ronald L. Rardin. %%pg 132

The U.S. Forest service has used an allocation model to deal with the task of managing national forests.
The model begins by dividing the land into a set of analysis areas. Several land management policies (also
called prescriptions) are then proposed and evaluated for each area.
An \emph{allocation} is how much land (in acreage) in each unique analysis area will be assigned to each of the possible prescriptions.
We seek to find the best possible allocation, subject to forest-wide restrictions on land use.

The file \li{ForestData.npy} contains data for a fictional national forest (you can also find the data
in Table \ref{tab:forest}). There are 7 areas of analysis and 3 prescriptions for each of them.

\begin{align*}
&\text{Column 1: $i$, area of analysis} \\
&\text{Column 2: $s_i$, size of the analysis area (in thousands of acres)} \\
&\text{Column 3: $j$, prescription number} \\
&\text{Column 4: $p_{i,j}$, net present value (NPV) per acre in area $i$ under prescription $j$} \\
&\text{Column 5: $t_{i,j}$, protected timber yield per acre in area $i$ under prescription $j$} \\
&\text{Column 6: $g_{i,j}$, protected animal grazing capability per acre for area $i$ under prescription $j$} \\
&\text{Column 7: $w_{i,j}$, wilderness index rating (0 to 100) for area $i$ under prescription $j$}
\end{align*}

\begin{table}[H]
\centering
    \begin{tabular}{c c c c c c c}
&&&Forest Data&&& \\
\hline
Analysis & Acres &Prescrip-&NPV&Timber&Grazing&Wilderness \\
Area&(1000)'s &tion&(per acre) &(per acre)&(per acre)& Index\\
$i$ &$s_i$&$j$& $p_{i,j}$ & $t_{i,j}$&$g_{i,j}$&$w_{i,j}$ \\\hline
1&	75	&1	&503	&310	&0.01&	40\\
&&		2&	140&	50&	0.04	&80\\
&&		3&	203&	0&	0&	95\\ \hline
2&	90&	1	&675&	198&	0.03&	55\\
&&		2&	100&	46&	0.06&	60\\
&&		3&	45&	0&	0&	65\\ \hline
3&	140&	1	&630&	210	&0.04&	45\\
&&		2&	105&	57&	0.07&	55\\
&&		3&	40	&0&	0&	60\\ \hline
4	&60&	1&	330&	112&	0.01&	30\\
&&		2	&40&	30&	0.02&	35\\
&&		3&	295&	0&	0	&90\\ \hline
5	&212&	1	&105	&40	&0.05&	60\\
&&		2	&460&	32	&0.08&	60\\
&& 3	&120&0&	0	&70\\ \hline
6	&98	&1	&490	&105	&0.02	&35\\
&&		2&	55	&25	&0.03	&50\\
&&		3	&180	&0	&0	&75\\ \hline
7&	113&	1	&705	&213&	0.02	&40\\
&&		2&	60	&40	&0.04&	45\\
&&		3	&400	&0	&0	&95\\
\hline
    \end{tabular}
\caption{}
\label{tab:forest}
\end{table}
Let $x_{i,j}$ be the amount of land in area $i$ allocated to prescription $j$.
Under this notation, an allocation is a one-dimensional vector consisting of the $x_{i,j}$'s.
For this particular
example, there are 7 acres, with 3 prescriptions each.
So the allocation vector is a one-dimensional vector with 21 entries.
Our goal is to find the allocation vector that maximizes net present value, while producing at least 40 million
board-feet of timber, at least 5 thousand units of grazing capability, and keeping the average wilderness index at least 70.
The allocation vector is also constrained to be nonnegative, and all of the land must be allocated precisely.

Since acres are in thousands, divide the constraints of timber and animal grazing by 1000 in the problem setup, and compensate for this after obtaining a solution.

The problem can be written as follows:
\begin{align*}
\text{maximize } &\sum\limits_{i=1}^7 \sum\limits_{j=1}^3 p_{i,j}x_{i,j} \\
\text{subject to } &\sum\limits_{j=1}^3 x_{i,j} = s_i  \text{ for } i=1,..,7 \\
	        &\sum\limits_{i=1}^7 \sum\limits_{j=1}^3 t_{i,j}x_{i,j} \geq 40,000 \\
		&\sum\limits_{i=1}^7 \sum\limits_{j=1}^3 g_{i,j}x_{i,j} \geq 5 \\
		&\frac{1}{788} \sum\limits_{i=1}^7 \sum\limits_{j=1}^3 w_{i,j}x_{i,j} \geq 70 \\
		&x_{i,j} \geq 0 \text{ for } i=1,...,7  \text{ and } j=1,2,3
\end{align*}

\begin{problem}
Solve the allocation problem above.
Return the minimizing allocation vector of $x_{i,j}$'s and the maximum total net present value.
Remember to consider the following:
\begin{enumerate}
\item The allocation vector should be a (21,1) NumPy array.
\item Recall that the constraints of timber and animal grazing were divided by 1000.
To compensate, the maximum total net value will be equal to the primal objective of the appropriately minimized linear function multiplied by -1000.
\end{enumerate}
\end{problem}

\end{comment}

\section*{Eating on a Budget}

In 2009, the inmates of Morgan County jail convinced Judge Clemon of the Federal District Court in Birmingham to put Sheriff Barlett in jail for malnutrition.
Under Alabama law, in order to encourage less spending, "the chief lawman could go light on prisoners' meals and pocket the leftover change."\footnote[1]{Nossiter, Adam, 8 Jan 2009, "As His Inmates Grew Thinner, a Sheriff’s Wallet Grew Fatter", \emph{New York Times},\url{https://www.nytimes.com/2009/01/09/us/09sheriff.html}}.
Sheriffs had to ensure a minimum amount of nutrition for inmates, but minimizing costs meant more money for the sheriffs themselves.
Judge Clemon jailed Sheriff Barlett one night until a plan was made to use all allotted funds, $1.75$ per inmate, to feed prisoners more nutritious meals.
While this case made national news, the controversy of feeding prisoners in Alabama continues as of 2019\footnote[2]{Sheets, Connor, 31 January 2019, "Alabama sheriffs urge lawmakers to get them out of the jail food business", \url{https://www.al.com/news/2019/01/alabama-sheriffs-urge-lawmakers-to-get-them-out-of-the-jail-food-business.html}}.

The problem of minimizing cost while reaching healthy nutritional requirements can be approached as a convex optimization problem.
Rather than viewing this problem from the sheriff's perspective, we view it from the perspective of a college student trying to minimize food cost in order to pay for higher education, all while meeting standard nutritional guidelines.

The file \li{food.npy} contains a dataset with nutritional facts for 18 foods that have been eaten frequently by college students working on this text.
A subset of this dataset can be found in Table \ref{tab:food-data}, where the "Food" column contains the list of all 18 foods.

The columns of the full dataset are:
\begin{align*}
& \text{Column 1: $p$, price (dollars)} \\
& \text{Column 2: $s$, number of servings} \\
& \text{Column 3: $c$, calories per serving} \\
& \text{Column 4: $f$, fat per serving (grams)} \\
& \text{Column 5: $\hat{s}$, sugar per serving (grams)} \\
& \text{Column 6: $\hat{c}$, calcium per serving (milligrams)} \\
& \text{Column 7: $\hat{f}$, fiber per serving (grams)} \\
& \text{Column 8: $\hat{p}$, protein per serving (grams)}
\end{align*}

 \begin{table}[H]
% \begin{adjustwidth}{-.5in}{-.5in}
\begin{tabular}{|c||c|c|c|c|c|c|c|c|c|c|}
\hline
\textbf{Food} & \textbf{Price} & \textbf{Serving Size} & \textbf{Calories} & \textbf{Fat} & \textbf{Sugar} & \textbf{Calcium} & \textbf{Fiber} & \textbf{Protein} \\
& $p$ & $s$ & $c$ & $f$ & $\hat{s}$ & $\hat{c}$ & $\hat{f}$ & $\hat{p}$ \\
& dollars & & & g & g & mg & g & g \\ \hline\hline
Ramen & 6.88 & 48 & 190 & 7 & 0 & 0 & 0 & 5 \\ \hline
Potatoes & 0.48 & 1 & 290 & 0.4 & 3.2 & 53.8 & 6.9 & 7.9 \\ \hline
Milk & 1.79 & 16 & 130 & 5 & 12 & 250 & 0 & 8 \\ \hline
Eggs & 1.32 & 12 & 70 & 5 & 0 & 28 & 0 & 6 \\ \hline
Pasta & 3.88 & 8 & 200 & 1 & 2 & 0 & 2 & 7 \\ \hline
Frozen Pizza & 2.78 & 5 & 350 & 11 & 5 & 150 & 2 & 14 \\ \hline
Potato Chips & 2.12 & 14 & 160 & 11 & 1 & 0 & 1 & 1 \\ \hline
Frozen Broccoli & 0.98 & 4 & 25 & 0 & 1 & 25 & 2 & 1 \\ \hline
Carrots & 0.98 & 2 & 52.5 & 0.3 & 6.1 & 42.2 & 3.6 & 1.2 \\ \hline
Bananas & 0.24 & 1 & 105 & 0.4 & 14.4 & 5.9 & 3.1 & 1.3 \\ \hline
Tortillas & 3.48 & 18 & 140 & 4 & 0 & 0 & 0 & 3 \\ \hline
Cheese & 1.88 & 8 & 110 & 8 & 0 & 191 & 0 & 6 \\ \hline
Yogurt & 3.47 & 5 & 90 & 0 & 7 & 190 & 0 & 17 \\ \hline
Bread & 1.28 & 6 & 120 & 2 & 2 & 60 & 0.01 & 4 \\ \hline
Chicken & 9.76 & 20 & 110 & 3 & 0 & 0 & 0 & 20 \\ \hline
Rice & 8.43 & 40 & 205 & 0.4 & 0.1 & 15.8 & 0.6 & 4.2 \\ \hline
Pasta Sauce & 3.57 & 15 & 60 & 1.5 & 7 & 20 & 2 & 2 \\ \hline
Lettuce & 1.78 & 6 & 8 & 0.1 & 0.6 & 15.5 & 1 & 0.6 \\ \hline
\end{tabular}
\caption{Subset of table containing food data}
\label{tab:food-data}
% \end{adjustwidth}
\end{table}

 According to the FDA\footnote[1]{url{https://www.accessdata.fda.gov/scripts/InteractiveNutritionFactsLabel/pdv.html}} and US Department of Health, someone on a $2000$ calorie diet should have no more than 2000 calories, no more than 65 grams of fat, no more than 50 grams of sugar\footnote[2]{https://www.today.com/health/4-rules-added-sugars-how-calculate-your-daily-limit-t34731}, at least 1000 milligrams of calcium\footnote[1]{26 Sept 2018, \url{https://ods.od.nih.gov/factsheets/Calcium-HealthProfessional/}}, at least 25 grams of fiber, and at least 46 grams of protein\footnote[2]{\url{https://www.accessdata.fda.gov/scripts/InteractiveNutritionFactsLabel/protein.html}} per day.

 We can rewrite this as a convex optimization problem below.

 \begin{align*}
\text{minimize } & \sum_{i=1}^{18}p_ix_i, \\
\text{subject to }& \sum_{i=1}^{18} c_ix_i \leq 2000, \\
			& \sum_{i=1}^{18} f_ix_i \leq 65, \\
			& \sum_{i=1}^{18} \hat{s}_ix_i \leq 50, \\
			& \sum_{i=1}^{18} \hat{c}_ix_i \geq 1000, \\
			& \sum_{i=1}^{18} \hat{f}_ix_i \geq 25, \\
			& \sum_{i=1}^{18} \hat{p}_ix_i \geq 46, \\
			& x_i \geq 0.
\end{align*}

 \begin{problem}
Read in the file \li{food.npy}.
Use CVXOPT to identify how much of each food item a college student should each to minimize cost spent each day.
Return the minimizing vector and the total amount of money spent.

What is the food you should eat most each day?
What are the three foods you should eat most each week?

(Hint: Each nutritional value must be multiplied by the number of servings to get the nutrition value of the whole product).
\label{prob:diet}
\end{problem}

You can learn more about CVXOPT at
\url{http://cvxopt.org/index.html}.


\documentclass[12pt,letterpaper]{article}
\usepackage[margin=1.00in]{geometry}
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{braket}
\usepackage{csquotes}
\usepackage{listings}
\usepackage{siunitx}
\usepackage{tikz}
\usepackage{wrapfig}

\title{Bresenham's Line Algorithm}
\date{}

\begin{document}
\maketitle

\section{Discrete vs. Continuous Values}
Before discussing how the line algorithm works, we should have a basic knowledge of what discrete and continuous values are.
Discrete data can only take particular distinct values, with no \enquote{gray area} in between; whereas continuous data occupies any value over a continuous range.
Why does this concern us when all we want to do is draw a line?
We can picture a raster image as a grid of distinct, discrete pixels that grow larger and larger as we zoom into the image.
In contrast, a line is continuous: an infinite number of points in between the two endpoints.
In any line algorithm, we try our best to map a continuous object onto a discrete grid of predefined positions such that they look as if they are continuous when zoomed out far enough.

\section{Assumptions}
Our goal is to find the pixels that best approximate a target line that can be expressed in the form $y = mx + b$.
To help us, we divide the grid into eight octants, with position $(0, 0)$ as the center.
In the degree system, octant I consists of \ang{0} to \ang{45}; octant II, \ang{45} to \ang{90}; and so on.
By splitting the coordinate system into eight octants, octant I now only contains lines with a slope $0 < m < 1$; octant II, lines with a slope $1 < m < \infty$.
We also assume that $\set{x_0, y_0, x_1, y_1} \in \mathbb{Z}$ (that is, $x_0$, $y_0$, $x_1$, and $y_1$ are integers).

\section{Octant I}
To keep things simple, we will always plot lines in this octant from left to right.
This requires that $x_0 < x_1$, which can be done by swapping $(x_0, y_0)$ with $(x_1, y_1)$ as needed.
Remember that in this octant, $0 < m < 1$.
Suppose we want to represent the line shown in the left graph of Figure~\ref{fig:oct1} as a set of pixels on the grid.

\begin{figure}
  \centering
  \begin{minipage}{0.31\textwidth}
    \begin{tikzpicture}
      \draw[step=1.0cm,gray,very thin] (0,0) grid (5,3);
      \draw[magenta,ultra thick] (0,0) -- (5,3);
    \end{tikzpicture}
  \end{minipage}
  \hfill
  \begin{minipage}{0.31\textwidth}
    \begin{tikzpicture}
      \draw[step=1.0cm,gray,very thin] (0,0) grid (5,3);
      \fill[blue!40!white] (0,0) rectangle (1,1);
      \draw[magenta,ultra thick] (0,0) -- (5,3);
      \node at (1.5,1) {\textbullet};
    \end{tikzpicture}
  \end{minipage}
  \hfill
  \begin{minipage}{0.31\textwidth}
    \begin{tikzpicture}
      \draw[step=1.0cm,gray,very thin] (0,0) grid (5,3);
      \fill[blue!40!white] (0,0) rectangle (1,1);
      \fill[blue!40!white] (1,0) rectangle (2,1);
      \fill[blue!40!white] (2,1) rectangle (3,2);
      \fill[blue!40!white] (3,2) rectangle (4,3);
      \fill[blue!40!white] (4,2) rectangle (5,3);
      \draw[magenta,ultra thick] (0,0) -- (5,3);
      \node at (1.5,1) {\textbullet};
      \node at (2.5,1) {\textbullet};
      \node at (3.5,2) {\textbullet};
      \node at (4.5,3) {\textbullet};
    \end{tikzpicture}
  \end{minipage}
  \caption{\label{fig:oct1} Plotting a line in octant I.}
\end{figure}

We start with the leftmost column (i.e., the smallest $x$-value) and work our way up.
Earlier on, we said that $x_0$ and $y_0$ are integers, so we can begin by plotting point $(x_0, y_0)$ (see the middle graph of Figure~\ref{fig:oct1}).

Since we are strictly moving from left to right and down to up, we must move onto the next column.
From here, we have two (logical) choices.
Do we fill in $(x_0 + 1, y_0)$ or $(x_0 + 1, y_0 + 1)$?
It turns out that we can actually figure this out with minimal work.
All lines are functions that can be expressed in the form $y_i = f(x_i)$.
So for a given $x_i$-value, we can calculate the corresponding $y_i$-value.
Then, we compare that calculated $y_i$ value to the \enquote{midway} value $y_j = f\left(x_i + \frac{1}{2}\right)$ to see which pixel contains \enquote{more} of the line.
We can express this like so:

\begin{quote}
  If $\left(x + 1, y + \frac{1}{2}\right)$ is above the line, draw the lower pixel.
  If it is below the line, draw the upper pixel.
  If it is exactly equivalent to the midpoint, pick either pixel.
\end{quote}

We avoid plotting two pixels for a single column so that the line does not turn thicker at various locations.

\subsection{Testing $\left(x + 1, y + \frac{1}{2}\right)$}
We can manipulate the point-slope equation $y = mx + b$ like so:
\begin{align*}
  0       &= mx - y + b \\
          &= \frac{\Delta y}{\Delta x} x - y + b \\
          &= (\Delta y) x - (\Delta x) y + (\Delta x) b \\
  f(x, y) &= Ax + By + C
\end{align*}
In this case, $A = \Delta y$, $B = -\Delta x$, and $C = (\Delta x) b$.
This gives us the first draft of our algorithm, written in psuedo-code:
\begin{lstlisting}[mathescape]
  $x = x_0$; $y = y_0$
  $d = f\left(x + 1, y + \frac{1}{2}\right)$
  while $x \leq x_1$
    plot($x$, $y$)
    $x$++
    if $d > 0$ then $y$++
    $d = f\left(x + 1, y + \frac{1}{2}\right)$
\end{lstlisting}
We realize that if $x$ increases by 1, add $A$ to $d$, and if $y$ increases by 1, add $B$ to $d$.
We can revise the algorithm to get our second draft (diff-mode below):
\begin{lstlisting}[mathescape]
   $x = x_0$; $y = y_0$
   $d = f\left(x + 1, y + \frac{1}{2}\right)$
   while $x \leq x_1$
     plot($x$, $y$)
-    $x$++
+    $x$++; $d$ += $A$
-    if $d > 0$ then $y$++
+    if $d > 0$ then $y$++; $d$ += $B$
-    $d = f\left(x + 1, y + \frac{1}{2}\right)$
\end{lstlisting}
We can further simplify the value of $d$:
\begin{align*}
  d_0   &= f\left(x_0 + 1, y_0 + \frac{1}{2}\right) \\
        &= A(x_0 + 1) + B\left(y_0 + \frac{1}{2}\right) + C \\
        &= f(x_0, y_0) + A + \frac{1}{2} B \\
        &= A + \frac{1}{2} B \\
  2 d_0 &= 2A + B
\end{align*}
This gives us the third draft of the algorithm, which is more or less as optimized as it can be while avoiding division, which might return a floating-point number (diff-mode below):
\begin{lstlisting}[mathescape]
   $x = x_0$; $y = y_0$
-  $d = f\left(x + 1, y + \frac{1}{2}\right)$
+  $d = 2A + B$
   while $x \leq x_1$
     plot($x$, $y$)
-    $x$++; $d$ += $A$
+    $x$++; $d$ += $2A$
-    if $d > 0$ then $y$++; $d$ += $B$
+    if $d > 0$ then $y$++; $d$ += $2B$
\end{lstlisting}

\section{Octant II}
In octant II, we essentially just swap $x$- and $y$-values.
The midpoint becomes $\left(x + \frac{1}{2}, y + 1\right)$, and so $d_0 = f\left(x_0 + \frac{1}{2}, y + 1\right)$.
After simplification, we get $d = \frac{1}{2} A + B$, or $2d = A + 2B$.
The conditional $d > 0$ is flipped, and becomes $d < 0$.

\section{Octant VIII}
Note that we have skipped many octants because one can draw in octant V by switching the points and drawing in octant I, and in octant VI by doing the same in octant II\@.
In octant VIII, the midpoint is $\left(x + 1, y - \frac{1}{2}\right)$, and so $d_0 = f\left(x_0 + 1, y_0 - \frac{1}{2}\right)$.
After simplification, $2d = 2A - B$.

\end{document}