% !TEX root = ../../../proposal.tex

\section{Background}

\subsection{Groups, orders, and generators}
\label{sec:group-background}

The two types of groups used for Diffie-Hellman key exchange in practice are
multiplicative groups over finite fields (``mod $p$'') and elliptic curve
groups. We focus on the ``mod $p$'' case, so a group is typically specified by
a prime $p$ and a generator $g$, which generates a multiplicative subgroup
modulo $p$.  Optionally, the group order $q$ can be specified; this is the
smallest positive integer $q$ satisfying $g^q \equiv 1 \bmod p$.  Equivalently,
it is the number of distinct elements of the subgroup $\{g, g^2, g^3, \dots
\bmod p\}$.

By Lagrange's theorem, the order $q$ of the subgroup generated by $g$ modulo
$p$ must be a divisor of $p-1$. Since $p$ is prime, $p-1$ will be even, and
there will always be a subgroup of order 2 generated by the element $-1$. For
the other factors $q_i$ of $p-1$, there are subgroups of order $q_i \bmod p$.
One can find a generator $g_i$ of a subgroup of order $q_i$ using a randomized
algorithm: try random integers $h$ until $h^{(p-1)/q_i} \ne 1 \bmod p$; $g_i =
h^{(p-1)/q_i} \bmod p$ is a generator of the subgroup.  A random $h$ will
satisfy this property with probability $1 - 1/q_i$.

In theory, neither $p$ nor $q$ is required to be prime. Diffie-Hellman key
exchange is possible with a composite modulus and with a composite group order.
In such cases, the order of the full multiplicative group modulo $p$ is
$\phi(p)$ where $\phi$ is Euler's totient function, and the order of the
subgroup generated by $g$ must divide $\phi(p)$. Outside of implementation
mistakes, Diffie-Hellman in practice is done modulo prime $p$.
%\looseness=-1

\subsection{Diffie-Hellman Key Exchange}

Diffie-Hellman key exchange allows two parties to agree on a shared secret in
the presence of an eavesdropper~\cite{new-directions-in-crypto-1976}. Alice and Bob begin by
agreeing on shared parameters (prime $p$, generator $g$, and optionally group
order $q$) for an algebraic group.  Depending on the protocol, the group may be
requested by the initiator (as in IKE), unilaterally chosen by the responder
(as in TLS), or fixed by the protocol itself (SSH originally built in support
for a single group).

Having agreed on a group, Alice chooses a secret $x_a < q$ and sends Bob $y_a =
g^{x_a}\bmod p$.  Likewise, Bob chooses a secret $x_b < q$ and sends Alice $y_b =
g^{x_b}\bmod p$. Each participant then computes the shared secret key
$g^{x_a x_b}\bmod p$.%  \looseness=-1

Depending on the implementation, the public values $y_a$ and $y_b$ might be
\emph{ephemeral}---freshly generated for each connection---or \emph{static} and
reused for many connections.

\subsection{Discrete log algorithms}

The best known attack against Diffie-Hellman is for the eavesdropper to compute
the the private exponent $x$ by calculating the discrete log of one of Alice or
Bob's public value $y$. With knowledge of the exponent, the attacker can
trivially compute the shared secret.  It is not known in general whether the
hardness of computing the shared secret from the public values is equivalent to
the hardness of discrete log.

The \emph{computational Diffie-Hellman assumption} states that computing the
shared secret $g^{x_ax_b}$ from $g^{x_a}$ and $g^{x_b}$ is hard for some choice
of groups.  A stronger assumption, the \emph{decisional Diffie-Hellman
problem}, states that given $g^{x_a}$ and $g^{x_b}$, the shared secret
$g^{x_ax_b}$ is computationally indistinguishable from random for some groups.
This assumption is often not true for groups used in practice; even with safe
primes as defined below, many implementations use a generator that generates
the full group of order $p-1$, rather than the subgroup of order $(p-1)/2$.  This means
that a passive attacker can always learn the value of the secret exponent modulo 2.
To avoid leaking this bit of information about the exponent, both sides could agree to
compute the shared secret as $y^{2x} \bmod p$.  We have not seen implementations
with this behavior.
%\looseness=-1

There are several families of discrete log algorithms, each of which apply to
special types of groups and parameter choices. Implementations must take care
to avoid choices vulnerable to any particular algorithm. These include:

\paragraph{Small-order groups}
The Pollard rho~\cite{pollard1975monte} and Shanks' baby step-giant step
algorithms~\cite{shanks1971class} each can be used to compute discrete logs in
groups of order $q$ in time $O(\sqrt{q})$.  To avoid being vulnerable,
implementations must choose a group order with bit length at least twice the
desired bit security of the key exchange. In practice, this means that group
orders $q$ should be at least 160 bits for an 80-bit security level.

\paragraph{Composite-order groups}
If the group order $q$ is a composite with prime factorization $q = \prod_i
q_i^{e_i}$, then the attacker can use the Pohlig-Hellman
algorithm~\cite{pohlig1978improved} to compute a discrete log in time $O(\sum_i
e_i \sqrt{q_i})$.  The Pohlig-Hellman algorithm computes the discrete log in
each subgroup of order $q_i^{e_i}$ and then uses the Chinese remainder theorem
to reconstruct the log modulo $q$.  Adrian et al.~\cite{logjam-2015} found
several thousand TLS hosts using primes with composite-order groups, and were
able to compute discrete logs for several hundred Diffie-Hellman key exchanges
using this algorithm.  To avoid being vulnerable, implementations should choose
$g$ so that it generates a subgroup of large prime order modulo $p$.
%\looseness=-1

\paragraph{Short exponents}
If the secret exponent $x_a$ is relatively small or lies within a known range of
values of a relatively small size, $m$, then the Pollard lambda ``kangaroo''
algorithm~\cite{Pollard2000} can be used to find $x_a$ in time $O(\sqrt{m})$.  To
avoid this attack, implementations should choose secret exponents to have bit
length at least twice the desired security level.  For example, using a 256-bit
exponent for for a 128-bit security level.

\paragraph{Small prime moduli} When the subgroup order is not small or
composite, and the prime modulus $p$ is relatively large, the fastest known
algorithm is the number field sieve~\cite{discrete-log-nfs-1993}, which runs in
subexponential time in the bit length of $p$, $\exp\left((1.923+o(1))(\log
p)^{1/3} (\log \log p)^{2/3}\right)$. Adrian et al.~recently applied the number field sieve
to attack 512-bit primes in about 90,000 core-hours~\cite{logjam-2015}, and
they argue that attacking 1024-bit primes---which are widely used in
practice---is within the resources of large governments. To avoid this attack,
current recommendations call for $p$ to be at least 2048
bits~\cite{sp800}. When selecting parameters, implementers should
ensure all attacks take at least as long as the number field sieve for their parameter set.

\subsection{Diffie-Hellman group characteristics}

\paragraph{``Safe'' primes}
In order to maximize the size of the subgroup used for Diffie-Hellman, one can
choose a $p$ such that $p = 2q + 1$ for some prime $q$. Such a $p$ is called a
``safe'' prime, and $q$ is a Sophie Germain prime.  For sufficiently large safe
primes, the best attack will be solving the discrete log using the number field
sieve.
Many standards explicitly specify the use of safe primes for Diffie-Hellman in
practice.  The Oakley protocol~\cite{rfc2412} specified five ``well-known''
groups for Diffie-Hellman in 1998. These included three safe primes of size
768, 1024, and 1536 bits, and was later expanded to include six more groups in
2003~\cite{rfc3526}. The Oakley groups have been built into numerous other
standards, including IKE~\cite{rfc2409} and SSH~\cite{rfc4253}.

%Given a finite field, with a cyclic group modulo some prime, $\mathbb{Z}/p$, and group operations $+$ and $\cdot$.
%
%Both participants choose an element, $a$ and $b$ respectively, randomly from the integers up to the order of the subgroup. The public keys take the form of $g^a$ and $g^b$ where $g$ is a generator of the subgroup and exponentiation is repeated application of the multiplication operation within the group. Both parties then exponentiate their partners public key by their secret to yield values of $g^{ba}$ and $g^{ab}$ respectively. By the abelian nature of the group these are the same group element, thus yielding a shared secret.
%
%For the case where the group is sufficiently large, not of smooth order, and $a$ and $b$ are of sufficient length, the Diffie-Hellman assumption posits that that the most efficient way to find the value of $g^{ab}$ given only $g^{a}$ and $g^{b}$ is equivalent to finding a solution to an instance of the discrete log problem, which is assumed hard. However, the above relies on three assumptions that can all be targeted in real implementations of the protocol.
%
%The solution to the DLP can be found if the group has a small number of elements or the exponent is small enough such that there is no 'wrap-around' when the generator is exponentiated, reducing the problem to solving the logarithm in the integers.
%
%A group can be chosen such that it contains a subgroup in which all the prime factors of the generator are smaller than some value $B$, which for integers is called a '$B-smooth$' integer. Then an attacker can solve the discrete log problem in the orders of the factors, and using the Chinese remainder theorem can reconstruct the unique solution in the original group.
%

\paragraph{DSA groups}
The DSA signature algorithm~\cite{fips186} is also based on the hardness of
discrete log.  DSA parameters have a subgroup order $q$ of much smaller size
than $p$.  In this case $p-1 = q r$ where $q$ is prime and $r$ is a large
composite, and $g$ generates a group of order $q$.  FIPS 186-4~\cite{fips186}
specifies 160-bit $q$ for 1024-bit $p$ and 224- or 256-bit $q$ for 2048-bit
$p$.  The small size of the subgroup allows the signature to be much shorter
than the size of $p$.

\subsection{DSA Group Standardization}

DSA-style parameters have also been recommended for use for Diffie-Hellman key
exchange.  NIST Special Publication 800-56A, ``Recommendation for Pair-Wise Key
Establishment Schemes Using Discrete Logarithm
Cryptography''~\cite{sp800}, first published in 2007, specifies that
finite field Diffie-Hellman should be done over a prime-order subgroup $q$ of
size 160 bits for a 1024-bit prime $p$, and a 224- or 256-bit subgroup for a
2048-bit prime.  While the order of the multiplicative subgroups is in line
with the hardness of computing discrete logs in these subgroups, no explanation
is given for recommending a subgroup of precisely this size rather than setting
a minimum subgroup size or using a safe prime.  Using a shorter exponent will
make modular exponentiation more efficient, but the order of the subgroup $q$
does not increase efficiency---on the contrary, the additional modular exponentiation
required to validate that a received key exchange message is contained in the correct
subgroup will render key exchange with DSA primes less efficient than using a ``safe''
prime for the same exponent length.  Choosing a small subgroup order is not known to have
much impact on other cryptanalytic attacks, although the number field sieve is
somewhat (not asymptotically) easier as the linear algebra step is performed
modulo the subgroup order $q$.~\cite{logjam-2015}

RFC 5114, ``Additional Diffie-Hellman Groups for Use with IETF
Standards''~\cite{rfc5114}, specifies three DSA groups with the above orders
``for use in IKE, TLS, SSH, etc.''  These groups were taken from test data
published by NIST~\cite{nistffcsamples}. They have been widely implemented in
IPsec and TLS, as we will show below. We refer to these groups as Group 22
(1024-bit group with 160-bit subgroup), Group 23 (2048-bit group with 224-bit
subgroup), and Group 24 (2048-bit group with 256-bit subgroup) throughout the
remainder of the paper to be consistent with the group numbers assigned for
IKE.

RFC 6989, ``Additional Diffie-Hellman Tests for the Internet Key Exchange
Protocol Version 2 (IKEv2)''~\cite{rfc6989}, notes that ``mod $p$'' groups with
small subgroups can be vulnerable to small subgroup attacks, and mandates that
IKE implementations should validate that the received value is in the correct
subgroup or never repeat exponents.

\subsection{Small subgroup attacks}
\label{subsec:small-subgroup-attack}

Since the security of Diffie-Hellman relies crucially on the group parameters,
implementations can be vulnerable to an attacker who provides maliciously
generated parameters that change the properties of the group.
With the right parameters and implementation decisions, an attaker may be able
to efficiently determine the Diffie-Hellman shared secret. In some cases, a
passive attacker may be able to break a transcript offline.

\paragraph{Small subgroup confinement attacks}
In a small subgroup confinement attack, an attacker (either a man-in-the-middle
or a malicious client or server) provides a key-exchange value $y$ that lies in a
subgroup of small order.  This forces the other party's view of the shared
secret, $y^x$, to lie in the subgroup generated by the attacker.  This
type of attack was described by van Oorschot and Wiener~\cite{van1996diffie}
and ascribed to Vanstone and Anderson and Vaudenay~\cite{anderson-1996}.  Small
subgroup confinement attacks are possible even when the server does not repeat
exponents---the only requirement is that an implementation does not validate
that received Diffie-Hellman key exchange values are in the correct subgroup.

%Consider a simplified Diffie-Hellman key exchange protocol where Alice and Bob negotiate a shared secret $k = g^{ab} \bmod p$ using Diffie-Hellman key exchange, and then Alice symmetrically encrypts a message using the secret $k$ and transmits it to Bob.  Alice might specify a safe prime $p$ and a generator $g$ of order $q = (p-1)/2$, and use a static Diffie-Hellman public value $y_a = g^a \bmod p$.

When working $\bmod\,p$, there is always a subgroup of order 2, since $p-1$ is
even. A malicious client Mallory could initiate a Diffie-Hellman key exchange
value with Alice and send her the value $y_M = p-1 \equiv -1 \bmod p$, which is
is a generator of the group of order $2 \bmod p$.  When Alice attempts to
compute her view of the shared secret as $k_a = y_M^a \bmod p$, there are only
two possible values, $1$ and $-1 \bmod p$.

The same type of attack works if $p-1$ has other small factors $q_i$.  Mallory
can send a generator $g_i$ of a group of order $q_i$ as her Diffie-Hellman key
exchange value. Alice's view of the shared secret will be an element of the
subgroup of order $q_i$. Mallory then has a $1/q_i$ chance of blindly guessing Alice's
shared secret in this invalid group. Given a message from Alice encrypted using
Alice's view of the shared secret, Mallory can brute force Alice's shared secret in $q_i$ guesses.

More recently, Bhargavan and
Delignat-Lavaud~\cite{bhargavan-channel-bindings-2015} describe ``key synchronization''
attacks against IKEv2 where a man-in-the-middle connects to both the initiator and
responder in different connections, uses a small subgroup confinement attack
against both, and observes that there is a $1/q_i$ probability of the shared
secrets being the same in both connections.  Bhargavan and Leurent~\cite{sloth-2016}
describe several attacks that use subgroup confinement attacks to obtain a
transcript collision and break protocol authentication.

To protect against subgroup confinement attacks, implementations should use
prime-order subgroups with known subgroup order. Both parties must validate
that the key exchange values they receive are in the proper subgroup. That is,
for a known subgroup order $q$, a received Diffie-Hellman key exchange value
$y$ should satisfy $y^q \equiv 1 \bmod p$.  For a safe prime, it suffices to
check that $y$ is strictly between $1$ and $p-1$.

\paragraph{Small subgroup key recovery attacks}
Lim and Lee~\cite{lim-1997} discovered a further attack that arises when an
implementation fails to validate subgroup order and resues a static
secret exponent for multiple key exchanges. A malicious party may be able
to perform multiple subgroup confinement attacks for different prime factors
$q_i$ of $p-1$ and then use the Chinese remainder theorem to reconstruct the
static secret exponent.

The attack works as follows.  Let $p-1$ have many small factors $p-1 = q_1 q_2
\dots q_n$.  Mallory, a malicious client, uses the procedure described in
Section~\ref{sec:group-background} to find a generator of the subgroup $g_i$ of
order $q_i \bmod p$.  Then Mallory transmits $g_i$ as her Diffie-Hellman key
exchange value, and receives a message encrypted with Alice's view of the
shared secret $g_i^{x_a}$, which Mallory can brute force to learn the value of
$x_a \bmod q_i$.  Once Mallory has repeated this process several times, she can
use the Chinese remainder theorem to reconstruct $x_a \bmod \prod_i q_i$.  The
running time of this attack is $\sum_i q_i$, assuming that Mallory performs an
offline brute-force search for each subgroup.

A randomly chosen prime $p$ is likely to have subgroups of large enough order
that this attack is infeasible to carry out for all subgroups.  However, if in
addition Alice's secret exponent $x_a$ is small, then Mallory only needs to carry out this
attack for a subset of subgroups of orders $q_1, \dots, q_k$ satisfying
$\prod_{i=0}^k q_i > x_a$, since the Chinese remainder theorem ensures that $x_a$
will be uniquely defined.
Mallory can also improve on the running time of the attack by taking advantage
of the Pollard lambda algorithm.  That is, she could use a small subgroup
attack to learn the value of $x_a \bmod \prod_{i=1}^k q_i$ for a subset of
subgroups $\prod_{i=1}^k q_i < x_a$, and then use the Pollard lambda algorithm to
reconstruct the full value of $a$, as it has now been confined to a smaller
interval.

In summary, an implementation is vulnerable to small subgroup key recovery
attacks if it does not verify that received Diffie-Hellman key exchange values
are in the correct subgroup; uses a prime $p$ such that $p-1$ has small
factors; and reuses Diffie-Hellman secret exponent values.  The attack is made
even more practical if the implementation uses small exponents.

A related attack exists for elliptic curve groups: an invalid curve attack.
Similarly to the case we describe above, the attacker generates a series of
elliptic curve points of small order and sends these
points as key exchange messages to the victim.  If the victim does not validate that the received
point is on the intended curve, they return a response that reveals information
about the secret key modulo different group orders.  After enough queries, the
attacker can learn the victim's entire secret.  Jager, Schwenk, and
Somorovsky~\cite{jager-2015} examined eight elliptic curve implementations and
discovered two that failed to validate the received curve point. For elliptic
curve groups, this attack can be much more devastating because the attacker has
much more freedom in generating different curves, and can thus find many
different small prime order subgroups.  For the finite field Diffie-Hellman
attack, the attacker is limited only to those subgroups whose orders are factors
of $p-1$.


\documentclass{myart}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{colortbl}

\begin{document}
\renewcommand\figurename{Fig}
\thispagestyle{empty}
\newpage

\pagestyle{plain}
\pagenumbering{arabic}

\begin{center}
\huge SOLUTION
\end{center}

\begin{center}
\large Author:Name \qquad Student ID:21*****
\end{center}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem 1}
First, use the bilineartransformation maps:
\begin{equation}
z=\frac{1+s}{1-s}
\end{equation}
\qquad We can get
\begin{equation}
(\frac{15}{4}-K)s^2+\frac{1}{2}s+(K-\frac{1}{4})=0
\end{equation}
\qquad Then, use Routh-Hurwitz criterion
\begin{equation}
\begin{aligned}
s^2& \qquad &\frac{15}{4}-K \qquad &K-\frac{1}{4}\\
s&   \qquad &\frac{1}{2}    \qquad &\\
1&   \qquad &K-\frac{1}{4}  \qquad &
\end{aligned}
\end{equation}
\qquad If the system is stable, it needs to meet the following conditions at the same time
\begin{equation}
\left.
\begin{cases}
\frac{15}{4}>0\\
K-\frac{1}{4}>0
\end{cases}
\right.
\end{equation}
\qquad So, the range of K is
\begin{equation}
\frac{1}{4}<K<\frac{15}{4}
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem 2}
\subsection{(a)}
\begin{equation}
\begin{split}
G(z)&=(1-z^{-1})\cdot Z[\frac{1}{s}\cdot \frac{5}{s(s+1)}] \\
&=(1-z^{-1})\cdot Z[5(\frac{1}{s^2}-\frac{1}{s}+\frac{1}{s+1})] \\
&=\frac{5[(T-1+e^{-T})z+(1-e^{-T}-Te^{-T})]}{(z-1)(z-e^{-T})}
\end{split}
\end{equation}

\begin{equation}
\begin{split}
T(z)&=\frac{G(z)}{1+GH(z)}\\
&=\frac{G(z)}{1+G(z)}\\
&=\frac{5[(T-1+e^{-T})z+(1-e^{-T}-Te^{-T})]}{(z-1)(z-e^{-T})+5[(T-1+e^{-T})z+(1-e^{-T}-Te^{-T})]}
\end{split}
\end{equation}

\subsection{(b)}
Let $a=T-1+e^{-T}$, and $b=1-e^{-T}-Te^{-T}$, then
\begin{equation}
T(z)=\frac{5az+5b}{(z-1)(z-e^{-T})+5az+5b}
\end{equation}
\qquad The characteristic equation of this system is
\begin{equation}
(z-1)(z-e^{-T})+5az+5b=0
\end{equation}
\qquad When T=0.1s, $a=0.1-1+e^{-0.1}\approx 0.0048$, $b=1-e^{-0.1}-0.1e^{-0.1}\approx 0.0047$.\par
The characteristic equation is
\begin{equation}
z^2-1.88z+0.93=0
\end{equation}
\qquad Use Jury's stability criterion
\begin{equation}
\begin{tabular}{p{1cm} p{2cm} p{2cm} p{2cm}}
   &       1&  -1.88& 0.93\\
 -)&    0.93&  -1.88&    1\\
\hline
   &   \cellcolor[rgb]{.99,.3,.3}0.135& -0.132&    0\\
 -)&  -0.132&  0.135&    \\
 \hline
   & \cellcolor[rgb]{.99,.3,.3}0.00593&      0&     \\
\end{tabular}
\end{equation}
\qquad So the system is STABLE when T=0.1s.

\subsection{(c)}
Similarly, when T=1s, $a=1-1+e^{-1}\approx 0.368$, $b=1-e^{-1}-e^{-1}\approx 0.264$.\par
The characteristic equation is
\begin{equation}
z^2+0.47z+1.69=0
\end{equation}
\qquad Use Jury's stability criterion
\begin{equation}
\begin{tabular}{p{1cm} p{2cm} p{2cm} p{2cm}}
   &       1&   0.47& 1.69\\
 -)&    1.69&   0.47&    1\\
\hline
   &   \cellcolor[rgb]{.99,.3,.3}-1.86& -0.324&    0\\
\end{tabular}
\end{equation}
\qquad So the system is UNSTABLE when T=1s.

\subsection{(d)}
The stability of the discrete system is related to the sampling period. If the sampling period is too long, a stable system may not be stable anymore.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem 3}
\subsection{(a)}
Use Matlab to solve this problem.
\lstinputlisting[language=Matlab]{./code/p3.m}
\qquad Root locus:
\midpic{1.eps}{Root Locus}

\newpage
\subsection{(b)}
From Figure 2 and Figure 3, we can get that, the root locus and the real axis has three intersections when $K=47.2$ or $K=1.92$ or $K=9.57\times 10^{-5}$. However, when the two real poles break away from the real axis, $K=9.57\times 10^{-5}$, and split point coordinates is $(0.998,0)$.
\doublepic{2.eps}{$K=47.2$}{3.eps}{$K=1.92$ and $K=9.57\times 10^{-5}$}

\subsection{(c)}
The closed-loop system's characteristic equation is
\begin{equation}
z^4+(K-3.7123)z^3+(10.3614K+5.1644)z^2+(9.7585K-3.195)z+0.8353K+0.7408=0
\end{equation}
\qquad From the root locus we can know that the root locus and the unit circle have only two intersections, and these two intersections are actually a complex conjugate pole pair. So we suppose the intersections of the root locus and the unit circle are$ z_1=e^{j\omega},z_2=e^{-j\omega} $. Substitute it into the original equation (14), we can get the solution:
\begin{equation}
\left.
\begin{cases}
\omega &=-0.000000001295873\\
K &=0.956515005888313
\end{cases}
\right.
\end{equation}
\qquad Thus, the maximum $K$ for stability is $0.956515005888313$.

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem 4}
Use Matlab to solve this problem.
\lstinputlisting[language=Matlab]{./code/p4.m}
\qquad We can get a bode diagram as shown in Figure 4.
\maxpic{4.eps}{Bode Diagram}
\newpage
I find the following equivalent digital controllers:

Use forward difference:
\begin{equation}
C(z)=\frac{z+9}{z+59}
\end{equation}

Use backward difference:
\begin{equation}
C(z)=\frac{11z-1}{61z-1}
\end{equation}

Use FOH:
\begin{equation}
C(z)=\frac{0.1806z-0.01389}{z-8.757\times 10^{-27}}
\end{equation}

Use tustin's approximation:
\begin{equation}
C(z)=\frac{0.1935z+0.129}{z+0.9355}
\end{equation}

Use tustin's approximation with frequency prewarping. And I choose the critical frequency $W_c = 2.4 rad/s $:
\begin{equation}
C(z)=\frac{0.1794z+0.1488}{z+0.9694}
\end{equation}

Use matched pole-zero method:
\begin{equation}
C(z)=\frac{0.1667z-7.567\times 10^{-6}}{z-8.757\times 10^{-27}}
\end{equation}

\end{document}


\lab{CVXOPT}{CVXOPT}
\objective{CVXOPT is a package of Python functions and classes designed for the purpose of convex optimization.
In this lab we use these tools for linear and quadratic programming.
We will solve various optimization problems using CVXOPT and optimize allocating land using linear programming.}

%\begin{warn}
%CVXOPT is not part of the standard library, and it is only included in the Anaconda distribution for Python 3.6 for Linux and Mac.
%We recommend avoiding Windows machines for this lab.
%
%To install CVXOPT, use \li{conda install cvxopt} or \li{pip install cvxopt}.
%\end{warn}

\section*{Linear Programs} % ==================================================

%%Cvxopt has linear program solver and can implement integer programming through the Gnu Linear Programming Kit, glpk.
%CVXOPT is a package of Python functions and classes designed for the purpose of convex optimization.
%In this lab we will focus on linear and quadratic programming.
A \emph{linear program} is a linear constrained optimization problem. Such a problem can be stated in several
different forms, one of which is
\begin{align*}
\text{minimize}\qquad &\c\trp \x \\
\text{subject to}\qquad &G\x \preceq \mathbf{h}\\
&A\x = \b.
\end{align*}

The symbol $\preceq$ denotes that the components of $G\x$ are less than the components of $\mathbf{h}$. In other words, if $\x\preceq\y$, then $x_i < y_i$ for all $x_i\in\x$ and $y_i\in\y$.

Define vector $\mathbf{s} \succeq \0$ such that the constraint $G\x + \mathbf{s} = \mathbf{h}$.
This vector is known as a \emph{slack variable}.
Since $\mathbf{s} \succeq \0$, the constraint
$G\x + \mathbf{s} = \mathbf{h}$ is equivalent to $G\x \preceq \mathbf{h}$.

With a slack variable, a new form of the linear program is found:
\begin{align*}
\text{minimize}\qquad &\c\trp \x \\
\text{subject to}\qquad &G\x + \mathbf{s} = \mathbf{h}\\
&A\x = \b \\
&\mathbf{s} \succeq \0.
\end{align*}

This is the formulation used by CVXOPT.
In this formulation, we require that the matrix $A$ has full row rank,
and that the block matrix $[G \quad A]\trp $ has full column rank.

% \preceq \succeq

% Students have not yet learned about the dual problem. May be included in a later lab.
\begin{comment}
The corresponding \emph{dual program} for the above linear program has the form
\begin{align*}
\text{maximize}\qquad &-h\trp z - b\trp y \\
\text{subject to}\qquad &G\trp z + A\trp y + c = 0\\
 &z \geq 0.
\end{align*}
CVXOPT provides functions to solve both the original (\emph{primal}) linear program and its dual program.
\end{comment}

Consider the following example:
\begin{align*}
\text{minimize}\qquad &-4x_1-5x_2 \\
\text{subject to}\qquad &x_1+2x_2 \leq 3 \\
	        &2x_1+x_2 = 3 \\
		&x_1, x_2 \geq 0
\end{align*}
Recall that all inequalities must be less than or equal to, such that $G\x\preceq \mathbf{h}$.
Because the final two constraints are $x_1, x_2 \geq 0$, they need to be adjusted to be $\leq$ constraints.
This is easily done by multiplying by $-1$, resulting in the constraints $-x_1, -x_2 \leq 0$.
If we define
\[
G = \begin{bmatrix}
  1 & 2\\
  -1 & 0\\
  0 & -1
\end{bmatrix} \text{, } \qquad
\mathbf{h} = \begin{bmatrix}
  3\\
  0\\
  0
\end{bmatrix} \text{, } \qquad
A = \begin{bmatrix}
2 & 1
\end{bmatrix} \text{, } \quad \text{and } \qquad
\mathbf{b} = \begin{bmatrix}
3
\end{bmatrix}
\]
then we can express the constraints compactly as
\[
\begin{matrix}
G\x \preceq \mathbf{h},\\
A\x = \mathbf{b},
\end{matrix}  \qquad \text{where} \qquad
\x = \begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix}.
\]
By adding a slack variable $\mathbf{s}$, we can write our constraints as
\[
G\x + \mathbf{s} = \mathbf{h},
\]
which matches the form discussed above.
% In the case of this particular example, we ignore the extra constraint
%\[
%A\x = \b,
%\]
%since we were given no equality constraints.

To solve the problem using CVXOPT, initialize the arrays $\c$, $G$, $\mathbf{h}$, $A$, and $\mathbf{b}$ and pass them to the appropriate function.
CVXOPT uses its own data type for an array or matrix.
While similar to the NumPy array, it does have a few differences, especially when it comes to initialization.
Below, we initialize CVXOPT matrices for $\mathbf{c}$, $G$, $\mathbf{h}$, $A$, and $\mathbf{b}$.
We then use the CVXOPT function for linear programming \li{solvers.lp()}, which accepts $\c$, $G$, $\mathbf{h}$, $A$, and $\b$ as arguments.

\begin{lstlisting}
>>> from cvxopt import matrix, solvers

>>> c = matrix([-4., -5.])
>>> G = matrix([[1., -1., 0.],[2., 0., -1.]])
>>> h = matrix([ 3., 0., 0.])
>>> A = matrix([[2.],[1.]])
>>> b = matrix([3.])

>>> sol = solvers.lp(c, G, h, A, b)
     pcost       dcost       gap    pres   dres   k/t
 0: -8.5714e+00 -1.4143e+01  4e+00  0e+00  3e-01  1e+00
 1: -8.9385e+00 -9.2036e+00  2e-01  3e-16  1e-02  3e-02
 2: -8.9994e+00 -9.0021e+00  2e-03  3e-16  1e-04  3e-04
 3: -9.0000e+00 -9.0000e+00  2e-05  1e-16  1e-06  3e-06
 4: -9.0000e+00 -9.0000e+00  2e-07  1e-16  1e-08  3e-08
Optimal solution found.
>>> print(sol['x'])
[ 1.00e+00]
[ 1.00e+00]
>>> print(sol['primal objective'])
-8.999999939019435
>>> print(type(sol['x']))
<<<class 'cvxopt.base.matrix'>>>
\end{lstlisting}

\begin{warn}
CVXOPT matrices only accept floats.
Other data types will raise a \li{TypeError}.

Additionally, CVXOPT matrices are initialized column-wise rather than row-wise (as in the case of NumPy).
Alternatively, we can initialize the arrays first in NumPy (a process with which you should be familiar),
and then simply convert them to the CVXOPT matrix data type.
\begin{lstlisting}
>>> import numpy as np

>>> c = np.array([-4., -5.])
>>> G = np.array([[1., 2.],[-1., 0.],[0., -1]])
>>> h = np.array([3., 0., 0.])
>>> A = np.array([[2., 1.]])
>>> b = np.array([3.])

# Convert the arrays to the CVXOPT matrix type.
>>> c = matrix(c)
>>> G = matrix(G)
>>> h = matrix(h)
>>> A = matrix(A)
>>> b = matrix(b)
\end{lstlisting}
In this lab, we will initialize non-trivial matrices first as NumPy arrays for consistency.

%Finally, be sure the entries in the matrices are floats!
\end{warn}

%Having initialized the necessary objects, we are now ready to solve the problem.

\begin{info}
Although it is often helpful to see the progress of each iteration of the algorithm, you may suppress this output by first running,
\begin{lstlisting}
solvers.options['show_progress'] = False
\end{lstlisting}
\end{info}

The function \li{solvers.lp()} returns a dictionary containing useful information.
For now, we will only focus on the value of $\x$ and the primal objective value (i.e. the minimum value achieved by the objective function).

\begin{warn}
Note that the minimizer \li{x} returned by the \li{solvers.lp()} function is a \li{cvxopt.base.matrix} object.
\li{np.ravel()} is a NumPy function that takes an object and returns its values as a flattened NumPy array.
Use \li{np.ravel()} to return all minimizers in this lab as flattened NumPy arrays.
\end{warn}

\begin{problem}
Solve the following convex optimization problem:
\begin{align*}
\text{minimize}\qquad &2x_1+x_2+3x_3 \\
\text{subject to}\qquad &x_1+2x_2 \geq 3 \\
	        &2x_1+10x_2+3x_3 \geq 10 \\
		&x_1 \geq 0 \\
		&x_2 \geq 0 \\
		&x_3 \geq 0
\end{align*}
Return the minimizer $\x$ and the primal objective value.
\\(Hint: make the necessary adjustments so that all inequality constraints are $\leq$ rather than $\geq$).
\end{problem}

\subsection*{$l_1$ Norm}
The $l_1$ norm is defined
\[||\x||_1=\sum_{i=1}^n |x_i|.\]
A $l_1$ minimization problem is minimizing a vector's $l_1$ norm, while fitting certain constraints. It can be written in the following form:
\begin{align*}
\text{minimize}\qquad &\|\x\|_1\\
\text{subject to} \qquad &A\x = \b.
\end{align*}

This problem can be converted into a linear program by introducing an additional vector $\u$ of length $n$.
Define $\u$ such that $|x_i|\leq |u_i|$.
Thus, $-u_i-x_i\leq 0$ and $-u_i+x_i\leq 0$.
These two inequalities can be added to the linear system as constraints.
Additionally, this means that $||\x||_1\leq ||\u||_1$.
So minimizing $||\u||_1$ subject to the given constraints will in turn minimize $||\x||_1$.
This can be written as follows:
\begin{align*}
\text{minimize}\qquad
&\begin{bmatrix}
\mathbf{1}\trp & \0\trp
\end{bmatrix}
\begin{bmatrix}
\u \\
\x
\end{bmatrix}\\
\text{subject to}\qquad
&\begin{bmatrix}
-I & I\\
-I & -I
\end{bmatrix}
\begin{bmatrix}
\u \\
\x
\end{bmatrix}
\preceq
\begin{bmatrix}
0\\
0
\end{bmatrix},\\
&\begin{bmatrix}
\0 & A
\end{bmatrix}
\begin{bmatrix}
\u \\
\x
\end{bmatrix}
=
\b.
\end{align*}
Solving this gives values for the optimal $\u$ and the optimal $\x$, but we only care about the optimal $\x$.

\begin{problem}
Write a function called \li{l1Min()} that accepts a matrix $A$ and vector $\mathbf{b}$ as NumPy arrays and solves the $l_1$ minimization problem.
Return the minimizer $\x$ and the primal objective value.
Remember to first discard the unnecessary $u$ values from the minimizer.

To test your function consider the matrix $A$ and vector $\mathbf{b}$ below.
\[
A = \begin{bmatrix}
1 & 2 & 1 & 1\\
0 & 3 & -2 & -1
\end{bmatrix} \qquad
\mathbf{b} = \begin{bmatrix}
7 \\
4
\end{bmatrix}
\]
The linear system $A\x = \b$ has infinitely many solutions.
Use \li{l1Min()} to verify that the solution which minimizes $||\mathbf{x}||_1$ is approximately $\x = [1.41, 2.40, 2.40, 0.79]^T$ and the minimum objective value is approximately 7.
\label{prob:l1}
\end{problem}

\section*{The Transportation Problem}

Consider the following transportation problem:
A piano company needs to transport thirteen pianos from their three  supply centers (denoted by 1, 2, 3) to two demand centers (4, 5).
Transporting a piano from a supply center to a demand center incurs a cost, listed in Table \ref{tab:cost}.
The company wants to minimize shipping costs for the pianos while meeting the demand.
%How many pianos should each supply center send to each demand center?

\begin{table}[H]
\centering
\begin{tabular}{|c|c|}
Supply Center & Number of pianos available\\
\hline
1 & 7\\
2 & 2\\
3 & 4\\
\end{tabular}

\caption{Number of pianos available at each supply center}
\label{tab:supply}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{|c|c|}
Demand Center & Number of pianos needed\\
\hline
4 & 5\\
5 & 8\\
\end{tabular}

\caption{Number of pianos needed at each demand center}
\label{tab:demand}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{|c|c|c|c|}
Supply Center & Demand Center & Cost of transportation & Number of pianos\\
\hline
1 & 4 & 4 & $p_1$\\
1 & 5 & 7 & $p_2$\\
2 & 4 & 6 & $p_3$\\
2 & 5 & 8 & $p_4$\\
3 & 4 & 8 & $p_5$\\
3 & 5 & 9 & $p_6$\\
\end{tabular}
\caption{Cost of transporting one piano from a supply center to a demand center}
\label{tab:cost}
\end{table}

A system of constraints is defined for the variables $p_1,p_2,p_3,p_4,p_5,$ and $p_6$,
First, there cannot be a negative number of pianos so the variables must be nonnegative.
Next, the Tables \ref{tab:supply} and \ref{tab:demand} define the following three supply constraints and two demand constraints:
\begin{align*}
p_1 + p_2  &= 7\\
p_3 + p_4  &= 2\\
p_5 + p_6  &= 4\\
p_1 + p_3 + p_5 &= 5\\
p_2 + p_4 + p_6 &= 8
\end{align*}

The objective function is the number of pianos shipped from each location multiplied by the respective cost (found in Table \ref{tab:cost}):
\[
4p_1 + 7p_2 + 6p_3 + 8p_4 + 8p_5 + 9p_6.
\]

\begin{info}
Since our answers must be integers, in general this problem turns out to be an NP-hard problem.
There is a whole field devoted to dealing with integer constraints, called \emph{integer linear programming}, which is beyond the scope of this lab.
Fortunately, we can treat this particular problem as a standard linear program and still obtain integer solutions.
\end{info}

Recall the variables are nonnegative, so $p_1,p_2,p_3,p_4,p_5,p_6\geq 0$.
Thus, $G$ and $\mathbf{h}$ constrain the variables to be non-negative.
Because CVXOPT uses the format $G\x \preceq \mathbf{h}$, we see that this inequality must be multiplied by $-1$.
So, $G$ must be a $6 \times 6$ identity matrix multiplied by $-1$, and

$\mathbf{h}$ is a column vector of zeros.
Since the supply and demand constraints are equality constraints, they are $A$ and $\b$.
Initialize these arrays and solve the linear program by entering the code below.
\begin{lstlisting}
>>> c = matrix(np.array([4., 7., 6., 8., 8., 9.]))
>>> G = matrix(-1*np.eye(6))
>>> h = matrix(np.zeros(6))
>>> A = matrix(np.array([[1.,1.,0.,0.,0.,0.],
                         [0.,0.,1.,1.,0.,0.],
                         [0.,0.,0.,0.,1.,1.],
                         [1.,0.,1.,0.,1.,0.],
                         [0.,1.,0.,1.,0.,1.]]))
>>> b = matrix(np.array([7., 2., 4., 5., 8.]))
>>> sol = solvers.lp(c, G, h, A, b)
     pcost       dcost       gap    pres   dres   k/t
 0:  8.9500e+01  8.9500e+01  2e+01  2e-16  2e-01  1e+00
 1:  8.7023e+01  8.7044e+01  3e+00  1e-15  3e-02  2e-01
Terminated (singular KKT matrix).
>>> print(sol['x'])
[ 4.31e+00]
[ 2.69e+00]
[ 3.56e-01]
[ 1.64e+00]
[ 3.34e-01]
[ 3.67e+00]
>>> print(sol['primal objective'])
87.023
\end{lstlisting}
Notice that some problems occurred. First, CVXOPT alerted us to the fact that the algorithm terminated prematurely (due to a singular matrix).
Second, the minimizer and solution obtained do not consist of integer entries.

So what went wrong? Recall that the matrix $A$ is required to have full row rank, but we can easily see that the rows of $A$
are linearly dependent. We rectify this by converting the last row of the equality constraints into two \emph{inequality} constraints, so that
the remaining equality constraints define a new matrix $A$ with linearly independent rows.

This is done as follows:

 Suppose we have the equality constraint
\[
x_1 + 2x_2 - 3x_3 = 4.
\]
This is equivalent to the pair of inequality
constraints
\begin{align*}
x_1 + 2x_2 - 3x_3 &\leq 4, \\
x_1 + 2x_2 - 3x_3 &\geq 4.
\end{align*}
The linear program requires only $\leq$ constraints, so we obtain the pair
of constraints
\begin{align*}
x_1 + 2x_2 - 3x_3 &\leq 4, \\
-x_1 - 2x_2 + 3x_3 &\leq -4.
\end{align*}

Apply this process to the last equality constraint of the transportation problem.
Then define a new matrix $G$ with several additional rows (to account for the new inequality
constraints), a new vector $\mathbf{h}$ with more entries, a smaller matrix $A$, and a smaller vector $\b$.
\begin{problem}
Solve the transportation problem by converting the last equality constraint into an inequality constraint.
Return the minimizer $\x$ and the primal objective value.
\end{problem}

\begin{comment}
\section*{Example}

Why are all of the terms in $G$ and $\mathbf{h}$ non-positive?

\begin{lstlisting}
>>> from cvxopt import matrix, solvers
>>> G = matrix([ [-1., 0., 0., -1., 0.,  -1., 0., 0., 0., 0., 0.],
             [-1., 0., 0., 0., -1.,  0., -1., 0., 0., 0., 0.],
             [0., -1., 0., -1., 0.,  0., 0., -1., 0., 0., 0.],
             [0., -1., 0., 0., -1.,  0., 0., 0., -1., 0., 0.],
             [0., 0., -1., -1., 0.,  0., 0., 0., 0., -1., 0.],
             [0., 0., -1., 0., -1.,  0., 0., 0., 0., 0., -1.] ])

>>> h = matrix([-7., -2., -4., -5., -8.,  0., 0., 0., 0., 0., 0.,])
>>> c = matrix([4., 7., 6., 8., 8., 9])
>>> sol = solvers.lp(c,G,h)
>>> print sol['x']
>>> print sol['primal objective']
\end{lstlisting}

Another method is to use an integer linear program.
Cvxopt is configured to work with  Gnu, which does have an integer linear program.
It will work with either of the methods above.

\textbf{Example}

glpk.ilp returns a tuple.
The first entry describes the optimality of the result, while the second gives the $x$ values.

\begin{lstlisting}
>>> from cvxopt import matrix, solvers, glpk
>>> G = matrix([ [-1., 0., 0., -1., 0.,  -1., 0., 0., 0., 0., 0.],
             [-1., 0., 0., 0., -1.,  0., -1., 0., 0., 0., 0.],
             [0., -1., 0., -1., 0.,  0., 0., -1., 0., 0., 0.],
             [0., -1., 0., 0., -1.,  0., 0., 0., -1., 0., 0.],
             [0., 0., -1., -1., 0.,  0., 0., 0., 0., -1., 0.],
             [0., 0., -1., 0., -1.,  0., 0., 0., 0., 0., -1.] ])

>>> h = matrix([-7., -2., -4., -5., -8.,  0., 0., 0., 0., 0., 0.,])
>>> o = matrix([4., 7., 6., 8., 8., 9])
>>> sol = glpk.ilp(o,G,h)
>>> print sol[1]
\end{lstlisting}

or
\begin{lstlisting}
>>> from cvxopt import matrix, solvers, glpk
>>> G = matrix([ [-1., 0., 0., 0., 0., 0.],
             [0., -1., 0., 0., 0., 0.],
             [0., 0., -1., 0., 0., 0.],
             [0., 0., 0., -1., 0., 0.],
             [0., 0., 0., 0., -1., 0.],
             [0., 0., 0., 0., 0., -1.] ])

>>> h = matrix([ 0., 0., 0., 0., 0., 0.,])
>>> o = matrix([4., 7., 6., 8., 8., 9])
>>> A = matrix([ [1., 0., 0., 1., 0.],
             [1., 0., 0., 0., 1.],
             [0., 1., 0., 1., 0.],
             [0., 1., 0., 0., 1.],
             [0., 0., 1., 1., 0.],
             [0., 0., 1., 0., 1.] ])
>>> b = matrix([7., 2., 4., 5., 8])
>>> sol = glpk.ilp(o,G,h,A,b)
>>> print sol[1]
\end{lstlisting}

\textbf{Problem 2}
Choose one of these methods and compare the optimal values for the integer linear program to the result you received above.

\textbf{Problem 3}
Create the dual problem for the linear program and solve.
Compare your answer to the dual value cvxopt returned.
\end{comment}

\section*{Quadratic Programming}

Quadratic programming is similar to linear programming, but the objective function is quadratic rather than linear.
The constraints, if there are any, are still of the same form.
Thus, $G, \mathbf{h}, A$, and $\b$ are optional.
The formulation that we will use is
\begin{align*}
\text{minimize}\qquad &\frac{1}{2}\x\trp Q\x + \mathbf{r}\trp \x \\
\text{subject to}\qquad &G\x \preceq \mathbf{h}\\
 &A\x = \b,
\end{align*}
where $Q$ is a positive semidefinite symmetric matrix.
In this formulation, we require again that $A$ has full row rank, and that the block matrix
$[Q \quad G \quad A]\trp $ has full column rank.

As an example, consider the quadratic function
\[
f(x_1,x_2) = 2x_1^2 +2x_1x_2 + x_2^2 +x_1 -x_2.
\]
There are no constraints, so we only need to initialize the matrix $Q$ and the vector $\mathbf{r}$.
To find these, we first rewrite our function to match the formulation given above.
If we let
\[
Q = \begin{bmatrix}
  a & b\\
  b & c
\end{bmatrix}, \qquad
\mathbf{r} = \begin{bmatrix}
  d\\
  e
\end{bmatrix},
\qquad \text{and} \qquad
\x = \begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix},
\]
then
\begin{align*}
\frac{1}{2}\x\trp Q\x + \mathbf{r}\trp \x &=
\frac{1}{2}
\begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix}\trp
\begin{bmatrix}
  a & b\\
  b & c
\end{bmatrix}
\begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix} +
\begin{bmatrix}
  d\\
  e
\end{bmatrix}\trp
\begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix} \\
&= \frac{1}{2}ax_1^2 + bx_1x_2 + \frac{1}{2}cx_2^2 + dx_1 + ex_2
\end{align*}
Thus, we see that the proper values to initialize our matrix $Q$ and vector $\mathbf{r}$ are:
\begin{align*}
a &= 4  &d = 1 \\
b &= 2  &e = -1 \\
c &= 2
\end{align*}
Now that we have the matrix $Q$ and vector $\mathbf{r}$, we are ready to use the CVXOPT function for quadratic programming \li{solvers.qp()}.
\begin{lstlisting}
>>> Q = matrix(np.array([[4., 2.], [2., 2.]]))
>>> r = matrix([1., -1.])
>>> sol=solvers.qp(Q, r)
>>> print(sol['x'])
[-1.00e+00]
[ 1.50e+00]
>>> print sol['primal objective']
-1.25
\end{lstlisting}

\begin{problem}
Find the minimizer and minimum of
\begin{equation*}
g(x_1,x_2,x_3) = \frac{3}{2}x_1^2 +2x_1x_2 + x_1x_3+ 2x_2^2 +2x_2x_3+\frac{3}{2}x_3^2+3x_1 + x_3
\end{equation*}
\\(Hint: Write the function $g$ to match the formulation given above before coding.)
\begin{comment}
\begin{equation}
f(x) = \frac{1}{2}x\trp Qx - x\trp p
\end{equation}
where

\begin{center}
$Q =
\begin{bmatrix}
3 & 2 & 1\\
2 & 4 & 2\\
1 & 2 & 3\\
\end{bmatrix}
$
and $p =
\begin{bmatrix}
3\\
0\\
1\\
\end{bmatrix}
$
\end{center}
\end{comment}

\end{problem}

\begin{problem}
The $l_2$ minimization problem is to
\begin{align*}
\text{minimize}\qquad &\|\x\|_2\\
\text{subject to} \qquad &A\x = \b.
\end{align*}

This problem is equivalent to a quadratic program, since $\|\x\|_2 = \x\trp \x$.
Write a function that accepts a matrix $A$ and vector $\b$ and solves the $l_2$ minimization problem.
Return the minimizer $\x$ and the primal objective value.

To test your function, use the matrix $A$ and vector $\b$ from Problem \ref{prob:l1}.
The minimizer is approximately $\x=[1.55, 2.36, 2.36, 0.73]^T$ and the minimum primal objective value is approximately 14.09.
\end{problem}

\section*{Allocation Models}
Allocation models lead to simple linear programs. An allocation model seeks to allocate a valuable resource among competing needs. Consider the following example taken from ``Optimization in Operations Research" by Ronald L. Rardin. %%pg 132

The U.S. Forest service has used an allocation model to deal with the task of managing national forests.
The model begins by dividing the land into a set of analysis areas. Several land management policies (also
called prescriptions) are then proposed and evaluated for each area.
An \emph{allocation} is how much land (in acreage) in each unique analysis area will be assigned to each of the possible prescriptions.
We seek to find the best possible allocation, subject to forest-wide restrictions on land use.

The file \li{ForestData.npy} contains data for a fictional national forest (you can also find the data
in Table \ref{tab:forest}). There are 7 areas of analysis and 3 prescriptions for each of them.

\begin{align*}
&\text{Column 1: $i$, area of analysis} \\
&\text{Column 2: $s_i$, size of the analysis area (in thousands of acres)} \\
&\text{Column 3: $j$, prescription number} \\
&\text{Column 4: $p_{i,j}$, net present value (NPV) per acre in area $i$ under prescription $j$} \\
&\text{Column 5: $t_{i,j}$, protected timber yield per acre in area $i$ under prescription $j$} \\
&\text{Column 6: $g_{i,j}$, protected animal grazing capability per acre for area $i$ under prescription $j$} \\
&\text{Column 7: $w_{i,j}$, wilderness index rating (0 to 100) for area $i$ under prescription $j$}
\end{align*}

\begin{table}[H]
\centering
    \begin{tabular}{c c c c c c c}
&&&Forest Data&&& \\
\hline
Analysis & Acres &Prescrip-&NPV&Timber&Grazing&Wilderness \\
Area&(1000)'s &tion&(per acre) &(per acre)&(per acre)& Index\\
$i$ &$s_i$&$j$& $p_{i,j}$ & $t_{i,j}$&$g_{i,j}$&$w_{i,j}$ \\\hline
1&	75	&1	&503	&310	&0.01&	40\\
&&		2&	140&	50&	0.04	&80\\
&&		3&	203&	0&	0&	95\\ \hline
2&	90&	1	&675&	198&	0.03&	55\\
&&		2&	100&	46&	0.06&	60\\
&&		3&	45&	0&	0&	65\\ \hline
3&	140&	1	&630&	210	&0.04&	45\\
&&		2&	105&	57&	0.07&	55\\
&&		3&	40	&0&	0&	60\\ \hline
4	&60&	1&	330&	112&	0.01&	30\\
&&		2	&40&	30&	0.02&	35\\
&&		3&	295&	0&	0	&90\\ \hline
5	&212&	1	&105	&40	&0.05&	60\\
&&		2	&460&	32	&0.08&	60\\
&& 3	&120&0&	0	&70\\ \hline
6	&98	&1	&490	&105	&0.02	&35\\
&&		2&	55	&25	&0.03	&50\\
&&		3	&180	&0	&0	&75\\ \hline
7&	113&	1	&705	&213&	0.02	&40\\
&&		2&	60	&40	&0.04&	45\\
&&		3	&400	&0	&0	&95\\
\hline
    \end{tabular}
\caption{}
\label{tab:forest}
\end{table}
Let $x_{i,j}$ be the amount of land in area $i$ allocated to prescription $j$.
Under this notation, an allocation is a one-dimensional vector consisting of the $x_{i,j}$'s.
For this particular
example, there are 7 acres, with 3 prescriptions each.
So the allocation vector is a one-dimensional vector with 21 entries.
Our goal is to find the allocation vector that maximizes net present value, while producing at least 40 million
board-feet of timber, at least 5 thousand units of grazing capability, and keeping the average wilderness index at least 70.
The allocation vector is also constrained to be nonnegative, and all of the land must be allocated precisely.

Since acres are in thousands, divide the constraints of timber and animal grazing by 1000 in the problem setup, and compensate for this after obtaining a solution.

The problem can be written as follows:
\begin{align*}
\text{maximize } &\sum\limits_{i=1}^7 \sum\limits_{j=1}^3 p_{i,j}x_{i,j} \\
\text{subject to } &\sum\limits_{j=1}^3 x_{i,j} = s_i  \text{ for } i=1,..,7 \\
	        &\sum\limits_{i=1}^7 \sum\limits_{j=1}^3 t_{i,j}x_{i,j} \geq 40,000 \\
		&\sum\limits_{i=1}^7 \sum\limits_{j=1}^3 g_{i,j}x_{i,j} \geq 5 \\
		&\frac{1}{788} \sum\limits_{i=1}^7 \sum\limits_{j=1}^3 w_{i,j}x_{i,j} \geq 70 \\
		&x_{i,j} \geq 0 \text{ for } i=1,...,7  \text{ and } j=1,2,3
\end{align*}

\begin{problem}
Solve the allocation problem above.
Return the minimizing allocation vector of $x_{i,j}$'s and the maximum total net present value.
Remember to consider the following:
\begin{enumerate}
\item The allocation vector should be a (21,1) NumPy array.
\item Recall that the constraints of timber and animal grazing were divided by 1000.
To compensate, the maximum total net value will be equal to the primal objective of the appropriately minimized linear function multiplied by -1000.
\end{enumerate}
\end{problem}

You can learn more about CVXOPT at
\url{http://cvxopt.org/index.html}.


\subsection{Sorted lists}

There can be a total ordering on elements in a list.

We want to return a list such that only the ordering is changed.

\(\forall nm [list[n]>list[m] \leftrightarrow n>m]\)

\subsection{Checking a sortable list}
\subsection{Sorting algorithms}

Efficient in time and memory

Popular algorithms are:

\begin{itemize}
\item Merge sort
\item Quick sort
\item Heap sort
\end{itemize}

\subsection{Sorting linked lists}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amssymb}

\title{Proving the Converse of Pythagorean Theorem}
\author{Ethan Xu}
\date{October 2021}

\begin{document}

\maketitle

\section{Introduction}
The Pythagorean Theorem is defined as: if $a^2+b^2=c^2$, then $\theta = 90^{\circ}$. By definition, then the converse of the Pythagorean Theorem is if $\theta = 90^{\circ}$, then $a^2+b^2=c^2$.

\section{Proof}
Let $\bigtriangleup ABC$ be a triangle with sides $a$ ($AB$), $b$ ($BC$), $c$ ($AC$) such that:

\begin{equation}
    a^2+b^2=c^2
\end{equation}

Let $\bigtriangleup DEF$ be a triangle with sides $a$ ($DE$), $b$ ($EF$), $d$ ($DF$). Let $\angle DEF = 90^{\circ}$, such that side $d$ is the hypotenuse. By the Pythagorean Theorem, we have that:

\begin{equation}
    a^2+b^2=d^2
\end{equation}

Now we can substitute $a^2+b^2$ in (1) for $d^2$ from (2), so we have that $c^2=d^2$. Since $c, d > 0$, $c=d$.

By SSS Congruence, we have that $\bigtriangleup ABC \cong \bigtriangleup DEF$, so $\angle ABC = \angle DEF = 90^{\circ}$. Hence, $\bigtriangleup ABC$ contains a right angle. $\hfill\square$
\end{document}


\subsection{Bagging}

\subsubsection{Introduction}

Bagging, or Bootstrap AGGregation, is a way of generating weak learners.

\subsubsection{Bootstrapping}

Bootstraping refers to taking samples with repacement from the training set. This is how the datasets for each of the weak learners are formed.

\subsubsection{How to do bagging}

We take samples from the training set, with replacement, and train each of these separately. This gives us our weak learners.


\chapter{Countably Infinite Types} \label{infinite}
In the previous sections we saw different flavours of finiteness which were
really just different flavours of relations to \(\mathbf{Fin}\).
In this section we will see that we can construct a similar classification of
relations to \(\mathbb{N}\), in the form of the countably infinite types.
\section{Two Countable Types}
The two types for countability we will consider are analogous to split
enumerability and cardinal finiteness.
The change will be a simple one: we will swap out lists for streams.
\begin{definition}[Streams]
  \begin{equation}
    \mathbf{Stream}(A) \coloneqq (\mathbb{N} \rightarrow A)
    \simeq \llbracket \top , \text{const}(\mathbb{N}) \rrbracket
  \end{equation}
\end{definition}
\begin{definition}[Split Countability]
  \begin{equation}
    \aleph_0!(A) \coloneqq \Sigma {(\mathit{xs} : \mathbf{Stream}(A))} , \Pi {(x : A)} , x \in \mathit{xs}
  \end{equation}
\end{definition}
This type is definitionally equal to it surjection equivalent (\(\mathbb{N}
\twoheadrightarrow ! \; A\)).
We construct the unordered, propositional version of the predicate in much the
same way as we constructed cardinal finiteness.
\begin{definition}[Countability]
  \begin{equation}
    \aleph_0(A) \coloneqq \lVert \aleph_0!(A) \rVert
  \end{equation}
\end{definition}

From both of these types we can derive decidable equality.
\begin{lemma}
  Any countable type has decidable equality.
\end{lemma}
\section{Closure}
\input{figures/pairing-functions}
We know that countable infinity is not closed under the exponential (function
arrow), so the only closure we need to prove is \(\Sigma\) to cover all of
what's left.
\begin{theorem} \label{split-countability-sigma}
  Split countability is closed under \(\Sigma\).
\end{theorem}
We know that countable infinity is not closed under the exponential (function
arrow), so the only closure we need to prove is \(\Sigma\) to cover all of
what's left.
To do this we have to take a slightly different approach to the functions we
defined before.
Figure~\ref{pairings} illustrates the reason why: previously, we used the
depth-first product pairing for each support list.
This diverges if the first list is infinite, never exploring anything other than
the first element in the second list.
Instead, we use here the cantor pairing function, which performs a breadth-first
search of the pairings of both lists.

Finally, while we have lost certain closure proofs by allowing for infinite
types, we also \emph{gain} some: in particular the Kleene star.
\begin{theorem}
  Split countability is closed under Kleene star.
  \begin{equation}
    \aleph_0!(A) \rightarrow \aleph_0!(\mathbf{List}(A))
  \end{equation}
\end{theorem}
Again, this proof requires a particular pattern to ensure productivity.
The pattern here builds an intermediate stream \(\mathcal{KV}\) of non-empty
lists from the input support stream \(\mathit{xs}\), which is subsequently
flattened.
\begin{equation}
  \mathcal{KV}_i \coloneqq \left[ \left[ \mathit{xs}_{j - 1} \mid j \in \mathit{js} \right] \mid \mathit{js} \in \mathbf{List}(\mathbb{N}) ; \text{sum}(\mathit{js}) = i ; 0 \notin \mathit{js}  \right]
\end{equation}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../paper"
%%% End:


\section{Rational Functions and Partial Fractions}{}{}\label{sec:Rational Functions}
A \dfont{rational function} is a fraction with polynomials in the numerator and
denominator.  For example,
$$
  {x^3\over x^2+x-6},
  \qquad\qquad
  {1\over (x-3)^2},
  \qquad\qquad
  {x^2+1\over x^2-1},
$$
are all rational functions of $x$.  We should mention a special type of rational function that we already know how to integrate: If the denominator has the form $\ds (ax+b)^n$,
the substitution $u=ax+b$ will always work.  The denominator becomes
$\ds u^n$, and each $x$ in the numerator is replaced by $(u-b)/a$, and
$dx=du/a$. While it may be tedious to complete the integration if the
numerator has high degree, it is merely a matter of algebra.

\begin{example}{Substitution and Splitting Up a Fraction}{Substitution and Splitting Up a Fraction}\label{Substitution and Splitting Up a Fraction}
Find $\ds\int{x^3\over(3-2x)^5}\,dx.$
\end{example}

\begin{solution}
Using the substitution
$u=3-2x$ we get
\begin{eqnarray*}
  \int{x^3\over(3-2x)^5}\,dx
  &=&{1\over -2}\int {\left({u-3\over-2}\right)^3\over u^5}\,du
  ={1\over 16}\int {u^3-9u^2+27u-27\over u^5}\,du\cr
  &=&{1\over 16}\int u^{-2}-9u^{-3}+27u^{-4}-27u^{-5}\,du\cr
  &=&{1\over 16}\left({u^{-1}\over-1}-{9u^{-2}\over-2}+{27u^{-3}\over-3}
  -{27u^{-4}\over-4}\right)+C\cr
  &=&{1\over 16}\left({(3-2x)^{-1}\over-1}-{9(3-2x)^{-2}\over-2}+
  {27(3-2x)^{-3}\over-3}
  -{27(3-2x)^{-4}\over-4}\right)+C\cr
  &=&-{1\over
    16(3-2x)}+{9\over32(3-2x)^2}-{9\over16(3-2x)^3}+{27\over64(3-2x)^4}+C
\end{eqnarray*}
\vglue-10pt
\end{solution}

Of course there are other situation in which we can apply known techniques, perhaps after some clever manipulation. The following example demonstrates one such case.

\begin{example}{Denominator Does Not Factor}{Denominator Does Not Factor}\label{Denominator Does Not Factor}
Evaluate $\ds\int {x+1\over x^2+4x+8}\,dx$.
\end{example}

\begin{solution}
The quadratic denominator
does not factor. We could complete the square and use a trigonometric
substitution, but it is simpler to rearrange the integrand:
$$
  \int {x+1\over x^2+4x+8}\,dx = \int {x+2\over x^2+4x+8}\,dx -
  \int {1\over x^2+4x+8}\,dx.
$$
The first integral is an easy substitution problem, using $u=x^2+4x+8$:
$$
  \int {x+2\over x^2+4x+8}\,dx={1\over2}\int {du\over u}=
  {1\over2}\ln|x^2+4x+8|.
$$
For the second integral we complete the square:
$$
  x^2+4x+8=(x+2)^2+4=4\left(\left({x+2\over2}\right)^2+1\right),
$$
making the integral
$$
  {1\over4}\int {1\over\left({x+2\over2}\right)^2+1}\,dx.
$$
Using $\ds u={x+2\over2}$ we get
$$
  {1\over4}\int {1\over\left({x+2\over2}\right)^2+1}\,dx=
  {1\over4}\int {2\over u^2+1}\,dx=
  {1\over2}\arctan\left({x+2\over2}\right).
$$
The final answer is now
$$
  \int {x+1\over x^2+4x+8}\,dx={1\over2}\ln|x^2+4x+8|-
  {1\over2}\arctan\left({x+2\over2}\right)+C.
$$
\end{solution}

More generally, we can not rely on a rational function having such ``nice" forms as those above.   There is a general technique
called ``partial fractions'' that, in principle, allows us to
integrate any rational function.

% We can always factor out the coefficient of $\ds x^2$ and put
%it outside the integral, so we can assume that the denominator has the
%form $\ds x^2+bx+c$.  There are three possible cases, depending on how
%the quadratic factors: either $\ds x^2+bx+c=(x-r)(x-s)$,
%$\ds x^2+bx+c=(x-r)^2$, or it doesn't factor. We can use the quadratic
%formula to decide which of these we have, and to factor the quadratic
%if it is possible.
%
%\begin{example}{Factoring a Quadratic}{Factoring a Quadratic}\label{Factoring a Quadratic}
%Determine whether $\ds x^2+x+1$ factors, and factor it if possible.
%\end{example}
%
%\begin{solution}
%The quadratic formula tells us that $\ds x^2+x+1=0$ when
%$$x={-1\pm\sqrt{1-4}\over 2}.$$
%Since there is no square root of $-3$, this quadratic does not factor.
%\end{solution}
%
%\begin{example}{Factoring a Quadratic with Real Roots}{Factoring a Quadratic with Real Roots}\label{Factoring a Quadratic with Real Roots}
%Determine whether $\ds x^2-x-1$ factors, and factor it if possible.
%\end{example}
%
%\begin{solution}
%The quadratic formula tells us that $\ds x^2-x-1=0$ when
%$$x={1\pm\sqrt{1+4}\over 2}={1\pm\sqrt{5}\over2}.$$
%Therefore
%$$
%  x^2-x-1=\left(x-{1+\sqrt{5}\over2}\right)\left(x-{1-\sqrt{5}\over2}\right).
%$$
%\end{solution}
%
%If $\ds x^2+bx+c=(x-r)^2$ then we have the special case we have already
%seen, that can be handled with a substitution. The other two cases
%require different approaches.
%
%If  $\ds x^2+bx+c=(x-r)(x-s)$, we have an integral of the form
%$$\int{p(x)\over (x-r)(x-s)}\,dx$$
%where $p(x)$ is a polynomial. The first step is to make sure that
%$p(x)$ has degree less than 2.
%
%\begin{example}{}{}\label{}
%Rewrite $$\ds\int {x^3\over (x-2)(x+3)}\,dx$$ in terms of an integral
%with a numerator that has degree less than 2.
%\end{example}
%
%\begin{solution}
%To do this we use long division of polynomials to
%discover that
%$$
%  {x^3\over (x-2)(x+3)}={x^3\over x^2+x-6}=x-1+{7x-6\over x^2+x-6}=
%  x-1+{7x-6\over (x-2)(x+3)}.
%$$
%See \url{http://en.wikipedia.org/wiki/Polynomial_long_division} for a review on long division.
%Then
%$$
%  \int {x^3\over (x-2)(x+3)}\,dx=\int x-1\,dx +\int {7x-6\over
%  (x-2)(x+3)}\,dx.
%$$
%The first integral is easy, so only the second requires some work.
%\end{solution}
%
%Now consider the following simple algebra of fractions:
%$$
%  {A\over x-r}+{B\over x-s}={A(x-s)+B(x-r)\over (x-r)(x-s)}=
%  {(A+B)x-As-Br\over (x-r)(x-s)}.
%$$
%That is, adding two fractions with constant numerator and denominators
%$(x-r)$ and $(x-s)$ produces a fraction with denominator $(x-r)(x-s)$
%and a polynomial of degree less than 2 for the numerator. We want to
%reverse this process: Starting with a single fraction, we want to
%write it as a sum of two simpler fractions. An example should make it
%clear how to proceed.

%\begin{example}{Partial Fraction Decomposition}{Partial Fraction Decomposition}\label{Partial Fraction Decomposition}
%Evaluate $\ds\int {x^3\over (x-2)(x+3)}\,dx$.
%\end{example}
%
%\begin{solution}
%We start by
%writing $\ds{7x-6\over (x-2)(x+3)}$ as the sum of two fractions.  We
%want to end up with
%$${7x-6\over (x-2)(x+3)}={A\over x-2}+{B\over x+3}.$$
%If we go ahead and add the fractions on the right hand side, we seek a common denominator, and get:
%$${7x-6\over (x-2)(x+3)}={(A+B)x+3A-2B\over (x-2)(x+3)}.$$
%So all we need to do is find $A$ and $B$ so that $7x-6=(A+B)x+3A-2B$,
%which is to say, we need $7=A+B$ and $-6=3A-2B$. This is a problem
%you've seen before: Solve a system of two equations in two
%unknowns. There are many ways to proceed; here's one: If $7=A+B$ then
%$B=7-A$ and so $-6=3A-2B=3A-2(7-A)=3A-14+2A=5A-14$. This is easy to
%solve for $A$: $\ds A= 8/5$, and then $B=7-A=7-8/5=27/5$. Thus
%$$
%  \int {7x-6\over (x-2)(x+3)}\,dx=
%  \int {8\over5}{1\over x-2}+{27\over5}{1\over x+3}\,dx=
%  {8\over5}\ln |x-2|+{27\over5}\ln|x+3|+C.
%$$
%The answer to the original problem is now
%\begin{eqnarray*}
%  \int {x^3\over (x-2)(x+3)}\,dx
%  &=&\int x-1\,dx +\int {7x-6\over (x-2)(x+3)}\,dx\cr
%  &=&{x^2\over 2}-x+{8\over5}\ln |x-2|+{27\over5}\ln|x+3|+C.\cr
%\end{eqnarray*}
%\vskip-10pt
%\end{solution}
%
%Now suppose that $\ds x^2+bx+c$ doesn't factor. Again we can use long
%division to ensure that the numerator has degree less than 2, then we
%complete the square.

\subsection{Partial Fraction Decomposition}\label{sec:partial_fraction}

Consider the integral $\ds\int \frac{1}{x^2-1}\ dx$. We do not have a simple formula for this (if the denominator were $x^2+1$, we would recognize the antiderivative as being the arctangent function). It can be solved using Trigonometric Substitution, but note how the integral is easy to evaluate once we realize:

%This integral is not difficult to evaluate once one realizes the following fact:
$$\frac{1}{x^2-1} = \frac{1/2}{x-1} - \frac{1/2}{x+1}.$$
Thus
\begin{align*}
\int\frac{1}{x^2-1}\ dx &= \int\frac{1/2}{x-1}\ dx - \int\frac{1/2}{x+1}\ dx \\
			&= \frac12\ln|x-1| - \frac12\ln|x+1| + C.
\end{align*}

This section teaches how to \textit{decompose} $$\frac{1}{x^2-1}\quad  \text{into}\quad  \frac{1/2}{x-1}-\frac{1/2}{x+1}.$$

We start with a rational function $f(x)=\frac{p(x)}{q(x)}$, where $p$ and $q$ do not have any common factors and the degree of $p$ is less than the degree of $q$. It can be shown that any polynomial, and hence $q$, can be factored into a product of linear and irreducible quadratic terms. The following Key Idea states how to decompose a rational function into a sum of rational functions whose denominators are all of lower degree than $q$.

If we start with a rational function $f(x)=\frac{p(x)}{q(x)}$, we can assume that perhaps after long division the the fraction is reduced, and is "proper", in that $p$ and $q$ do not have any common factors and the degree of $p$ is less than the degree of $q$. Any polynomial, and hence $q$, can be factored into a product of linear and irreducible quadratic terms. The following outlines how to decompose a rational function into a sum of rational functions whose denominators are all of lower degree than $q$.
%\clearpage

\begin{formulabox}[Partial Fraction Decomposition] \label{idea:partial_fraction}
{Let $\ds \frac{p(x)}{q(x)}$ be a rational function, where the degree of $p$ is less than the degree of $q$.\index{integration!partial fraction decomp.}
\begin{enumerate}
	\item	\textbf{Linear Terms:} Let $(x-a)$ divide $q(x)$, where $(x-a)^n$ is the highest power of $(x-a)$ that divides $q(x)$. Then the decomposition of $\frac{p(x)}{q(x)}$ will contain the sum
	$$\frac{A_1}{(x-a)} + \frac{A_2}{(x-a)^2} + \cdots +\frac{A_n}{(x-a)^n}.$$
	\item		\textbf{Quadratic Terms:} Let $x^2+bx+c$ divide $q(x)$, where $(x^2+bx+c)^n$ is the highest power of $x^2+bx+c$ that divides $q(x)$. Then the decomposition of $\frac{p(x)}{q(x)}$ will contain the sum
	$$\frac{B_1x+C_1}{x^2+bx+c}+\frac{B_2x+C_2}{(x^2+bx+c)^2}+\cdots+\frac{B_nx+C_n}{(x^2+bx+c)^n}.$$
	\end{enumerate}
	To find the coefficients $A_i$, $B_i$ and $C_i$:
	\begin{enumerate}
	\item	Multiply all fractions by $q(x)$, clearing the denominators. Collect like terms.
	\item		Equate the resulting coefficients of the powers of $x$ and solve the resulting system of linear equations.
	\end{enumerate}
}
\end{formulabox}

To find the coefficients $A_i$, $B_i$ and $C_i$, it is helpful to have a process:

\begin{formulabox}[Partial Fractions: Solving for Coefficients] \label{idea:partial_fraction2}
Let $\ds \frac{p(x)}{q(x)}$ be a rational function, set up an equation with $\ds \frac{p(x)}{q(x)}$ set equal to it's partial fraction form.
\begin{enumerate}
	\item	Multiply by the denominator $ q(x) $ to clear all fractions and obtain the 	``\textbf{Basic Equation}".

	\item	Solve the Basic Equation for the unknowns using the following guidelines:

	\begin{enumerate}
	\item Expand the Basic Equation, collect terms according to powers of $ x $ and equate coefficients of like powers of $ x $. This will give a system of linear equations to be solved.
	\item	Alternatively, for distinct linear factors, you may substitute the roots of the distinct linear factors
	to determine the constants.
	\item Another alternative: For repeated linear factors, you may also first substitute the roots of the linear factors, then rewrite the Basic Equation and use other “convenient" choices for $ x $ to solve for the remaining
	coefficients (or use the method of equating coefficients).
	\end{enumerate}
\end{enumerate}
\end{formulabox}

The following examples will demonstrate how to put this into practice. Example \ref{exa:ex_pf1} stresses the decomposition aspect of the Key Idea.\\

\begin{example}{Decomposing into partial fractions}{ex_pf1}
{
Decompose $\ds f(x)=\frac{1}{(x+5)(x-2)^3(x^2+x+2)(x^2+x+7)^2}$ without solving for the resulting coefficients.}
\end{example}

\begin{solution}
{The denominator is already factored, as both $x^2+x+2$ and $x^2+x+7$ cannot be factored further. We need to decompose $f(x)$ properly. Since $(x+5)$ is a linear term that divides the denominator, there will be a $$\frac{A}{x+5}$$ term in the decomposition.

As $(x-2)^3$ divides the denominator, we will have the following terms in the decomposition:
$$\frac{B}{x-2},\quad \frac{C}{(x-2)^2}\quad \text{and}\quad \frac{D}{(x-2)^3}.$$

The $x^2+x+2$ term in the denominator results in a $\ds\frac{Ex+F}{x^2+x+2}$ term.

Finally, the $(x^2+x+7)^2$ term results in the terms $$\frac{Gx+H}{x^2+x+7}\quad \text{and}\quad \frac{Ix+J}{(x^2+x+7)^2}.$$
All together, we have
\begin{align*}
\frac{1}{(x+5)(x-2)^3(x^2+x+2)(x^2+x+7)^2} &= \frac{A}{x+5} + \frac{B}{x-2}+ \frac{C}{(x-2)^2}+\frac{D}{(x-2)^3}+ \\
		& \frac{Ex+F}{x^2+x+2}+\frac{Gx+H}{x^2+x+7}+\frac{Ix+J}{(x^2+x+7)^2}
\end{align*}
Solving for the coefficients $A$, $B \ldots J$ would be a bit tedious but not ``hard.''
}
\end{solution}

\begin{example}{Decomposing into partial fractions}{ex_pf2}
{
Perform the partial fraction decomposition of $\ds \frac{1}{x^2-1}$.
}
\end{example}

\begin{solution}
{The denominator factors into two linear terms: $x^2-1 = (x-1)(x+1)$. Thus
$$\frac{1}{x^2-1} = \frac{A}{x-1} + \frac{B}{x+1}.$$
To solve for $A$ and $B$, first multiply through by $x^2-1 = (x-1)(x+1)$ to obtain the Basic Equation:
\begin{align}
1 &= \frac{A(x-1)(x+1)}{x-1}+\frac{B(x-1)(x+1)}{x+1} \\
	&= A(x+1) + B(x-1)  \label{basic2} \\
	&= Ax+A + Bx-B \\
	\intertext{Now collect like terms.}
	&= (A+B)x + (A-B).
\end{align}
The next step is key. Note the equality we have:
$$1 = (A+B)x+(A-B).$$
For clarity's sake, rewrite the left hand side as
$$0x+1 = (A+B)x+(A-B).$$
On the left, the coefficient of the $x$ term is $ 0 $; on the right, it is $(A+B)$. Since both sides are equal, we must have that $0=A+B$.

Likewise, on the left, we have a constant term of $ 1 $; on the right, the constant term is $(A-B)$. Therefore we have $1=A-B$.

We have two linear equations with two unknowns. This one is easy to solve by hand, leading to
$$\begin{array}{c} A+B = 0 \\ A-B = 1 \end{array} \Rightarrow \begin{array}{c} A=1/2 \\ B = -1/2\end{array}.$$

Note, that alternatively, we could have substituted the two roots ($ usedx=\pm 1 $) into the basic equation \ref{basic2} to solve for the coefficients. $ x=1 $ gives $ 1= 2A$, and $ x=-1 $ gives $ 1=-2B $.

Thus we arrive at the partial fraction decomposition $$\frac{1}{x^2-1} = \frac{1/2}{x-1}-\frac{1/2}{x+1}.$$
}
\end{solution}

\begin{example}{Integrating using partial fractions}{ex_pf3}
Use partial fraction decomposition to integrate $\ds\int\frac{1}{(x-1)(x+2)^2}\ dx.$
\end{example}

\begin{solution}
We decompose the integrand as follows, as described in the process \ref{idea:partial_fraction}:
$$\frac{1}{(x-1)(x+2)^2} = \frac{A}{x-1} + \frac{B}{x+2} + \frac{C}{(x+2)^2}.$$
To solve for $A$, $B$ and $C$, we multiply both sides by $(x-1)(x+2)^2$ to obtain the Basic Equation and collect like terms:
\begin{align}
1 &= A(x+2)^2 + B(x-1)(x+2) + C(x-1)\label{eq:pf3}\\
	&= Ax^2+4Ax+4A + Bx^2 + Bx-2B + Cx-C \notag \\
	&= (A+B)x^2 + (4A+B+C)x + (4A-2B-C)\notag
\end{align}

We have $$0x^2+0x+ 1 = (A+B)x^2 + (4A+B+C)x + (4A-2B-C)$$
leading to the equations
$$A+B = 0, \quad 4A+B+C = 0 \quad \text{and} \quad 4A-2B-C = 1.$$
These three equations of three unknowns lead to a unique solution:
$$A = 1/9,\quad B = -1/9 \quad \text{and} \quad C = -1/3.$$

Thus
$$\int\frac{1}{(x-1)(x+2)^2}\ dx = \int \frac{1/9}{x-1}\ dx + \int \frac{-1/9}{x+2}\ dx + \int \frac{-1/3}{(x+2)^2}\ dx.$$

Each can be integrated with a simple substitution with $u=x-1$ or $u=x+2$  as the denominators are linear functions). The end result is
$$\int\frac{1}{(x-1)(x+2)^2}\ dx = \frac19\ln|x-1| -\frac19\ln|x+2| +\frac1{3(x+2)}+C.$$

{\textbf{Note:} The Basic Equation \ref{eq:pf3} offers a direct route to finding the values of $A$, $B$ and $C$. When $x=1$, the right hand side simplifies to $A(1+2)^2 = 9A$. Since the left hand side is still $ 1 $, we have $1 = 9A$. Hence $A = 1/9$.  Likewise, when $x=-2$we obtain $1=-3C$, so $C = -1/3$. Knowing $A$ and $C$, we can find the value of $B$ by choosing yet another value of $x$, such as $x=0$, and solving for $B$, or by equating coefficients.}

\end{solution}

\begin{example}{Integrating using partial fractions}{ex_pf4}
{
Use partial fraction decomposition to integrate $\ds \int \frac{x^3}{(x-5)(x+3)}\ dx$.
}
\end{example}

\begin{solution}
{Our method presumes that the degree of the numerator is less than the degree of the denominator. Since this is not the case here, we begin by using polynomial division to reduce the degree of the numerator. We omit the steps, but encourage the reader to verify that $$\frac{x^3}{(x-5)(x+3)} = x+2+\frac{19x+30}{(x-5)(x+3)}.$$
We can rewrite the new rational function in partial fraction form:
$$\frac{19x+30}{(x-5)(x+3)} = \frac{A}{x-5} + \frac{B}{x+3}$$ for appropriate values of $A$ and $B$. Clearing denominators, we have the basic equation:
\begin{equation*}
19x+30 = A(x+3) + B(x-5)
\end{equation*}
Setting $ x=-3 $ gives $ -27=-8B $, so $ B=27/8 $.\\
Setting $ x=5 $ gives $ 125=8A $, so $ A=125/8 $.

We can now integrate.
\begin{align*}
\int \frac{x^3}{(x-5)(x+3)}\ dx &= \int\left(x+2+\frac{125/8}{x-5}+\frac{27/8}{x+3}\right)\ dx \\
					&= \frac{x^2}2 + 2x + \frac{125}{8}\ln|x-5| + \frac{27}8\ln|x+3| + C.
\end{align*}

Note: Alternatively, we could have used the method of equating coefficients to solve for $ A $ and $ B $.
\begin{align*}
19x+30 &= A(x+3) + B(x-5)\\
			&= (A+B)x + (3A-5B).
\intertext{This implies that:}
19&= A+B \\
30&= 3A-5B.\\
\intertext{Solving this system of linear equations gives}
125/8 &=A\\
27/8 &=B.
\end{align*}
}
\end{solution}

%\clearpage

\begin{example}{Integrating using partial fractions}{ex_pf5}
{
Use partial fraction decomposition to evaluate $\ds \int\frac{7x^2+31x+54}{(x+1)(x^2+6x+11)}\ dx.$
}
\end{example}

\begin{solution}
{The degree of the numerator is less than the degree of the denominator so we begin by setting up the partial fraction form. We have:
\begin{align*}
\frac{7x^2+31x+54}{(x+1)(x^2+6x+11)} &= \frac{A}{x+1} + \frac{Bx+C}{x^2+6x+11}. \\
\intertext{Now clear the denominators to get the Basic Equation.}
7x^2+31x+54 &= A(x^2+6x+11) + (Bx+C)(x+1)\\
\intertext{Now collect terms to equate coefficients.}
					&= (A+B)x^2 + (6A+B+C)x + (11A+C).\\
\intertext{This implies that:}
				7&=A+B\\
				31 &= 6A+B+C\\
				54 &= 11A+C.
\end{align*}
Solving this system of linear equations gives the nice result of $A=5$, $B = 2$ and $C=-1$. Thus
$$\int\frac{7x^2+31x+54}{(x+1)(x^2+6x+11)}\ dx = \int\left(\frac{5}{x+1} + \frac{2x-1}{x^2+6x+11}\right)\ dx.$$

The first term of this new integrand is easy to evaluate; it leads to a $5\ln|x+1|$ term. The second term is not hard, but takes several steps and uses substitution techniques.

The integrand $\ds \frac{2x-1}{x^2+6x+11}$ has a quadratic in the denominator and a linear term in the numerator. This leads us to try substitution. Let $u = x^2+6x+11$, so $du = (2x+6)\ dx$. The numerator is $2x-1$, not $2x+6$, but we can get a $2x+6$ term in the numerator by adding $ 0 $ in the form of ``$7-7$.''
\begin{align*}
\frac{2x-1}{x^2+6x+11} &= \frac{2x-1+7-7}{x^2+6x+11} \\
					&= \frac{2x+6}{x^2+6x+11} - \frac{7}{x^2+6x+11}.
\end{align*}
We can now integrate the first term with substitution, leading to a $\ln|x^2+6x+11|$ term. The final term can be integrated using arctangent. First, complete the square in the denominator:
$$\frac{7}{x^2+6x+11} = \frac{7}{(x+3)^2+2}.$$
An antiderivative of the latter term can be found using Theorem \ref{thm:int_inverse_trig} and substitution:
$$\int \frac{7}{x^2+6x+11}\ dx = \frac{7}{\sqrt{2}}\tan^{-1}\left(\frac{x+3}{\sqrt{2}}\right)+C.$$

Let's start at the beginning and put all of the steps together.
\small\begin{align*}
\int\frac{7x^2+31x+54}{(x+1)(x^2+6x+11)}\ dx &= \int\left(\frac{5}{x+1} + \frac{2x-1}{x^2+6x+11}\right)\ dx \\
			&= \int\frac{5}{x+1}\ dx  + \int\frac{2x+6}{x^2+6x+11}\ dx -\int\frac{7}{x^2+6x+11}\ dx \\
			&= 5\ln|x+1|+ \ln|x^2+6x+11| -\frac{7}{\sqrt{2}}\tan^{-1}\left(\frac{x+3}{\sqrt{2}}\right)+C.
\end{align*}\normalsize
As with many other problems in calculus, it is important to remember that one is not expected to ``see'' the final answer immediately after seeing the problem. Rather, given the initial problem, we break it down into smaller problems that are easier to solve. The final answer is a combination of the answers of the smaller problems.
}\\
\end{solution}

Partial Fraction Decomposition is an important tool when dealing with rational functions. Note that at its heart, it is a technique of algebra, not calculus, as we are rewriting a fraction in a new form. Regardless, it is very useful in the realm of calculus as it lets us evaluate a certain set of ``complicated'' integrals.

%The next section introduces new functions, called the Hyperbolic Functions. They will allow us to make substitutions similar to those found when studying Trigonometric Substitution, allowing us to approach even more integration problems.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:Rational Functions}}

\begin{enumialphparenastyle}

%%%%%%%%%%
\begin{ex}
 $\ds\int {1\over 4-x^2}\,dx$
\begin{sol}
 $-\ln|x-2|/4+\ln|x+2|/4+C$
\end{sol}
\end{ex}
%%%%%%%%%%

%%%%%%%%%%
\begin{ex}
 $\ds\int {x^4\over 4-x^2}\,dx$
\begin{sol}
 $\ds -x^3/3-4x-4\ln|x-2|+$\hfill\break$4\ln|x+2| +C$
\end{sol}
\end{ex}
%%%%%%%%%%

%%%%%%%%%%
\begin{ex}
 $\ds\int {1\over x^2+10x+25}\,dx$
\begin{sol}
 $-1/(x+5) +C$
\end{sol}
\end{ex}
%%%%%%%%%%

%%%%%%%%%%
\begin{ex}
 $\ds\int {x^2\over 4-x^2}\,dx$
\begin{sol}
 $-x-\ln|x-2|+\ln|x+2| +C$
\end{sol}
\end{ex}
%%%%%%%%%%

%%%%%%%%%%
\begin{ex}
 $\ds\int {x^4\over 4+x^2}\,dx$
\begin{sol}
 $\ds -4x+x^3/3+8\arctan(x/2) +C$
\end{sol}
\end{ex}
%%%%%%%%%%

%%%%%%%%%%
\begin{ex}
 $\ds\int {1\over x^2+10x+29}\,dx$
\begin{sol}
 $(1/2)\arctan(x/2+5/2) +C$
\end{sol}
\end{ex}
%%%%%%%%%%

%%%%%%%%%%
\begin{ex}
 $\ds\int {x^3\over 4+x^2}\,dx$
\begin{sol}
 $\ds x^2/2-2\ln(4+x^2) +C$
\end{sol}
\end{ex}
%%%%%%%%%%

%%%%%%%%%%
\begin{ex}
 $\ds\int {1\over x^2+10x+21}\,dx$
\begin{sol}
 $(1/4)\ln|x+3|-(1/4)\ln|x+7| +C$
\end{sol}
\end{ex}
%%%%%%%%%%

%%%%%%%%%%
\begin{ex}
 $\ds\int {1\over 2x^2-x-3}\,dx$
\begin{sol}
 $(1/5)\ln|2x-3|-(1/5)\ln|1+x| +C$
\end{sol}
\end{ex}
%%%%%%%%%%

%%%%%%%%%%
\begin{ex}
 $\ds\int {1\over x^2+3x}\,dx$
\begin{sol}
 $(1/3)\ln|x|-(1/3)\ln|x+3| +C$
\end{sol}
\end{ex}
%%%%%%%%%%

\end{enumialphparenastyle}


\section{Linear Equations}

\subsection{Elementary Operations}

\begin{definition}
	Let $A$ be an $m\times n$ matrix. there are three \cindex{elementary row operation}:
	\begin{enumerate}
		\item interchange any two row of $A$.
		\item multiply any row of $A$ by nonzero scalar.
		\item add any scalar multiple of a row of $A$ to another row.
	\end{enumerate}
\end{definition}

\begin{definition}
	An $n\times n$ \cindex{elementary matrix} is a matrix obtained by performing \emph{one} elementary operation on $I_n$.
\end{definition}

\begin{definition}
	The \cindex{rank} of $A_{m \times n}$, denoted $\text{rank}(A)$, is the rank\footnote{The rank of a linear transformation is defined in Definition (\ref{rankdefinition}) on page \pageref{rankdefinition}.} of linear transformation $L_A: F^n \rightarrow F^m$.
\end{definition}

\begin{theorem}
	the rank of a matrix equals the maximum number of linearly independent columns.
\end{theorem}
\begin{proof}
	For any $A \in M_{m\times n}(F)$,
	\begin{equation*}
		\begin{aligned}
			\text{rank}(A) &= \rank{L_A} = \dimension{R(L_A)} = \vectorspan{L_A(\beta)} \\
			&= \vectorspan{\set{ L_A(e_1), L_A(e_2), \dots, L_A(e_n) }}
		\end{aligned}
	\end{equation*}
	we have $L_A(e_j) = A e_j = a_j$ where $a_j$ is the $j$th column of A. Hence
	\begin{equation*}
		R(L_A) = \vectorspan{\set{ a_1, a_2, \dots, a_n}}
	\end{equation*}
\end{proof}

\begin{theorem}
    Let $A_{m \times n}$ has rank $r$. Then there exist invertible matrix $B_{m \times m}$ and $C_{n \times n}$ that $D=BAC$, where:
    \begin{equation*}
        D = \begin{bmatrix}
			I_r & 0 \\
			0 & 0 \\
		\end{bmatrix}
    \end{equation*}
\end{theorem}

\begin{theorem}
    Every invertible matrix is a product of elementary matrices.
\end{theorem}

\begin{definition}
	For system $Ax=b$, the matrix $(A|b)$ is the \cindex{augmented matrix}.
\end{definition}

\begin{theorem}
    If A is an invertible matrix, it is possible to transform augmented matrix $(A|I_n)$ into matrix $(I_n|A^{-1})$ by means of a finite number of elementary row operations.
\end{theorem}

\subsection{System of Equations}

\begin{definition}
	A system $A_{m \times n} x=b$ of $m$ linear equation in $n$ unknowns is \cindex{homogeneous} if $b=0$. Otherwise the system is \cindex{nonhomogeneous}.
\end{definition}

\begin{definition}
	A system is \cindex{consistent} if its solution set is not empty. otherwise it is called \cindex{inconsistent}.
\end{definition}

\begin{theorem}
	Let $K$ be the set of all solutions for $Ax=0$. Then $K=\nullspace{L_A}$ has dimension of $n- \rank{L_A}=n-\rank{A}$.
\end{theorem}

\begin{theorem}
	if $m < n$, the system $Ax=0$ has nonzero solution.
\end{theorem}
\begin{proof}
    $\rank{A} \leq m < n$, so $\nullspace{A} = n - \rank{A} > 0$.
\end{proof}

\begin{theorem}\label{equationfromoneandnullspace}
	Let $K$ be the solution set of $Ax=b$, $K_H$ be the solution set of $Ax=0$. Then for all solution $s$ to $Ax=b$,
	\begin{equation}
		K = \set{s} + K_H = \set{s+k: k \in K_H }
	\end{equation}
\end{theorem}

\begin{theorem}
	Let $A_{n \times n}x=b$ be a system of equations. If $A$ is invertible, the solution is $A^{-1}b$. Conversely, if the system has exactly one solution, $A$ is invertible.
\end{theorem}

\begin{theorem}
	Let $Ax=b$ be a system of linear equations. the system is consistent $\Leftrightarrow$ $\text{rank}(A) = \text{rank}(A|b)$.
\end{theorem}

\begin{proof}
    $R(L_A) = \vectorspan{\set{a_1, a_2, \dots, a_n }}$. Since $b \in R(L_A)$, the extended span is the same.
\end{proof}

\begin{definition}
	A matrix is in \cindex{reduced row echelon form} if:
	\begin{enumerate}
		\item any row containing a nonzero entry precedes any row in which all the entries are zero.
		\item the first nonzero entry in each row is the only nonzero entry in its column.
		\item the first nonzero entry in each row is $1$ and it occurs in a column to the right of the first nonzero entry in the preceding row.
	\end{enumerate}
\end{definition}

\begin{theorem}\label{rankoftwomatrix}
    For $A_{m \times n}$ and $B_{n \times p}$, we have:
    \begin{equation}
        \rank{AB} = \rank{B} - \dimension{\nullspace{A} \cap \rangespace{B}}
    \end{equation}
\end{theorem}
\begin{proof}
    Let $\beta_i$ be the basis of $\nullspace{A} \cap \rangespace{B}$, expand to the basis $\beta \cup \alpha$ of $B$. Prove $\alpha$ is a basis of $\rangespace{AB}$.
\end{proof}

\begin{theorem}\label{rankofadjoint}
    For $A_{m \times n}$, we have
    \begin{enumerate}
        \item $\rank{A^\top A} = \rank{A} = \rank{A A^\top}$.
        \item $\rangespace{A^\top A} = \rangespace{A^\top}$.
        \item $\nullspace{A^\top A} = \nullspace{A}$.
    \end{enumerate}
    $A^\top$ could be replaced by $A^*$ in $C$.
\end{theorem}
\begin{proof}
    If $\exists x \neq 0 \left(x \in \nullspace{A^\top} \cap \rangespace{A} \right)$. Then $(A^\top x = 0) \wedge \left(\exists y(x = A y) \right)$. So $x^\top x = y^\top A^\top x = y^\top ( A^\top x) = 0 $ and then $x =0$. According to \thmref{rankoftwomatrix}, $\rank{A^\top A} = \rank{A^\top} - \dimension{\nullspace{A^\top} \cap \rangespace{A}} = \rank{A}$.
\end{proof}

\begin{theorem}
    For a system of linear equation $Ax = b$, the associated system of \cindex{normal equations} is defined as $n \times n$ system
    \begin{equation}
        A^\top A x = A^\top b
    \end{equation}

    $A^\top A x = A^\top b$ is always consistent and has unique solution when $\rank{A} = n$. If $Ax=b$ is consistent, two solutions are the same. \qed
\end{theorem}


\documentclass{article}
\usepackage{amsmath}
\usepackage{amsfonts}
\title{Statistical Machine Learning \\ Reading Assignment 1 Report}

\author{Alireza Sadeghi - Mohsen Shojaee}
\newtheorem{theorem}{Theorem}
\begin{document}

\maketitle

\section{A Taste of Real Analysis}
Real Analysis is the field of mathematics on which probability theory is founded, it is therefore convinient to first introduce some basic concepts from real analysis. We begin the this chapter with the definiation of metric space and use the notion of metric to define two core concepts: convergence and continuity. In  \ref{topology1} we revie the natural topoly defined on a metric space based on the metric function and then in \ref{topology2} we provide a more general view of topological spaces.
\subsection{Metric Space}
A metric space is a set $M$ together with a metric $d: M \times M \to \mathcal{R}$ satisfying following four properties
\begin{itemize}
\item d is non-negative
\item $d(x,y)=0$ iff $x=y$
\item Symmetry: $d(x,y) = d(y,x)$
\item Triangle Inequality: $d(x,z) \leq d(x,y) + d(y,z) $
\end{itemize}

Strictly speaking, the pair $(M,d)$ is the metric space as different metric functions can be defined on same $M$, consider for example $\mathcal{R}^n$; a well-known class of metrics defined on $\mathcal{R}^n$ is \textbf{Minkowski Norm}:
\[
d_p(x,y) = \big ( \sum_{i=1}^n (x_i - y_i)^p \big )^{1/p}
\]
which for all values of $p \geq 1$ is a valid metric function.

\subsubsection{Convergence \& Continuity}
There are different ways for defining convergence, we follow \cite{pough} and use sequence/subsequence approach. A sequence $(p_n)$ is a list of points $p_1, p_2, \ldots$ in $M$. Formaly, a sequence is a function $f: \mathbb{N} \to M $ in which $f(n) = p_n$. The sequence $(p_n)$ {\bf converges to the limit} $p$ in $M$ (and denote this by $(p_n) \to p$ if:
\begin{align*}
& \forall \epsilon > 0 \quad \exists N \in \mathbb{N} \quad \text{such that} \\
& n \geq N \Rightarrow d(p_n, p) < \epsilon
\end{align*}
Having defined convergence, contiuity can be described: For a function $f: M \to N$ between two metric spaces $(M, d_M)$ and $(N, d_N)$, we say that function is continous if it preserves sequential convergence, that is if $(p_n) \to p$ then $(f(p_n)) \to f(p)$.

The sequence definitation of continiutity stated above is equivalent with the more familiar definiation using $(\epsilon, \delta)$ condition:
\begin{theorem}
$f:M \to N$ is continouts if and only if for each $\epsilon >0$ and $ p \in M$ there exisits $\delta > 0$ such $\forall x \in M : \quad d_M(x,p) < \delta \Rightarrow d_N(f(x), f(p)) < \epsilon$
\end{theorem}
\subsubsection{Topology of Metric Space} \label{topology1}
Althogh topology can be defined on non-metric spaces (as we will do so in the next section) there is a \textit{natural} topology induced on metric spaces induced by the distance function. To this end we need to define notion of {\bf openness} and {\bf closeness} based metric and convergences. We say that point $p \in M$ is a limit of $S \subset M$ if there exists a  sequence in $S$ like $(p_n)$ that $(p_n) \to p$

{\bf Closeness:} $S$ is a closed set if it contains all it limits.
{\bf Openness:} $S$ is an open set if for each $p \in S$, $r>0$ exists such that
\[
d(p,q) <r \Rightarrow q \in S.
\]
that is for each point in $S$ an small ball around it is also in $S$.

One can simply prove that complement of an open set is closed and vice versa. However (like doors) sets can be neighter open nor closed and unlike doors they can be both at the same time.

\begin{theorem}\label{topdef}
	Now the collection $\mathcal{T}$ of all open sets of $M$ is the topolgy of $M$, i.e. is satisfies the following three properties:
	\begin{itemize}
	\item $M, \Phi \in \mathcal{T}$
	\item The intersection of finitely many open sets is an open set
	\item The union of arbitrarily many open sets is an open set.
\end{itemize}
\end{theorem}

\subsection{Topology: a More General Perspective} \label{topology2}
The three properties stated in \ref{topdef} are the definiation of topology, one can {\it handcraft} a collection $\mathcal{T}$  that satisfies these properties and call it the collection of open sets of $M$, even if they does not satisfy the definiation of openness based on metric or even $M$ is not a metric space at all.

\section{Theory of Probability}

\subsection{Introduction}
In many cases where statistics and statistical inference is an essential component of situation analysis, one encounters many discrete and continuous random variables and vectors and matrices. These are all special cases of a more general type of random quantity. The generalization of these notions to random quantities is through a notion of \textit{measure}.

\subsection{Measure Theory}
Measure, to be defined shortly is a way of assigning numerical values to the sizes of sets. Since it's used to give sizes to sets, it's domain is a collection of sets.

In order to define this "collection of sets" notion more thoroughly, we call a collection of sets that is closed under taking complements and finite unions, a \textbf{\textit{field}}.

A field, that is closed under taking countable unions is called a \textbf{\textit{$\sigma$-field}}.

A $\sigma$-field that is generated by the collection C of open subsets of a topological space is called a \textbf{\textit{Borel $\sigma$-field}}
\subsection{Measurable Functions}

Suppose \textit{S} is a set with a $\sigma$-field \textit{A} of subsets, and let \textit{T} be another set with a $\sigma$-field \textit{C} of subsets. Now consider a function $f : S \rightarrow T$. We say \textit{f} is \textit{measurable} if for every B $\in$ C, $f^{-1} \in A$.

If \textit{f} is measurable, one-to-one and onto, and $f^-1$ is also measurable, we say that \textit{f} is \textit{bimeasurable}, and also in another case, if the two sets \textit{S, T} are topological spaces with Borel $\sigma$-fields, a measurable function is \textit{Borel measurable}.

\subsection{Mathematical Probability}

If we'd want to define probability with a measure theoretic approach, we'd say that a measure space \textit{(S, A, $\mu$)} is a probability space and $\mu$ is a probability if $\mu(S) = 1$. We call each element of \textit{A} an \textit{event}, and a measurable function \textit{X} from \textit{S} to some other space \textit{X, B} is called a random quantity.

When \textit{X} is $\Re$ with the \textit{Borel $\sigma$-field}, this random quantity is called a \textit{Random Variable}. The probability measure $\mu_x$ induced on \textit{(X, B}} by X from $\my$ is called the \textit{distribution} of X. The expected value or mean

\section{Stochastic Processes}

\subsection{Definition}

A stochastic process {X_t}_$t \in T$ is a collection of random variables X_t, taking values in a common measure space (S, X), indexed by a set T.

This definition means that for each $t \in T$, $X_t(\omega)$ is an S \rightarrow X-measurable function from $\Omega$ to S, which yields a probability on S.

Some examples of stochastic processes are as follows:

\begin{itemize}
    \item Every random variable is a trivial stochastic process.
    \item Let $T = {1, 2, \cdots, k}$ and $S = \Re$. Then {X_t}_$t \in T$ is a random vector in $\Re^k$.
    \item One-sided random sequence, in which we define T to be {1, 2, \cdots}.
    \item Two-sided random sequences, in which we define T to be \mathbb{Z}.
    \item Spatially-discrete random fields, in which we let T to be $\mathcal{Z}$.
    \item
and some other types.
\end{itemize}

\subsection{Random Functions}

$X(t, \omega)$ has two arguments, t and $\omega$. If we fix the value of t, X_t($\omega$ is a random variable. For each fixed value of $\omega$ though, X(t) is a function from T to S, we call it a random function.

\end{document}