\subsection{Formal languages}\label{subsec:formal_languages}

Languages are used to define formulas for expressing the \hyperref[def:zfc]{axioms of set theory}. Here, sets are used to formally define languages. A simple way out of this vicious cycle is via the theory-metatheory relationship discussed in \fullref{rem:metalogic} and \fullref{rem:set_definition_recursion}. In short, we define languages within the metatheory using the already available concept of set, and we later define formulas, again in the metatheory, which allows us to subsequently formally define sets via axioms within the object logic.

\begin{definition}\label{def:formal_language}
  Fix a nonempty set \( \mscrA \).

  \begin{thmenum}
    \thmitem{def:formal_language/alphabet} We call \( \mscrA \) an \term{alphabet}.

    \thmitem{def:formal_language/symbol} We call each element of \( \mscrA \) a \term{symbol}.

    \thmitem{def:formal_language/word} A \term{string} over \( \mscrA \) is a \hyperref[def:sequence]{finite sequence} of symbols. If \( (a, b, c) \) is a word, for convenience we write it as the string \( abc \). This is the reason words are also referred to as \term{strings}. This notation only makes sense if each symbol of the language is actually represented by one typographic symbol.

    The term \enquote{string} is common in programming practice. In the context of formal languages, strings are often called \term{words}. We will avoid the later term since it does not correspond to the everyday use of the term \enquote{word}.

    \thmitem{def:formal_language/empty_word} We denote the empty string by \( \varepsilon \).

    \thmitem{def:formal_language/word_length} The \term{length} \( \len(w) \) of a word \( w \) is the number of elements of the tuple \( w \).

    \thmitem{def:formal_language/concatenation} The \term{concatenation} of the words \( v = (v_1, \ldots, v_n) \) and \( w = (w_1, \ldots, w_m) \) is the word
    \begin{equation*}
      vw \coloneqq (v_1, \ldots, v_n, w_1, \ldots, w_m).
    \end{equation*}

    We abbreviate \( \overbrace{w w \ldots w}^{k \T{times}} \) as \( w^k \). This is only a notation. We do not distinguish, formally, between the words \( aaabbaa \) and \( a^3 b^2 a^2 \), nor between \( a \varepsilon b \) and \( ab \).

    \thmitem{def:formal_language/reverse} The \term{reverse word} of \( w = (w_1, \ldots, w_n) \) is
    \begin{equation*}
      \op{rev}(w) \coloneqq (w_n, \ldots, w_1).
    \end{equation*}

    \thmitem{def:formal_language/prefix} The word \( p = (p_1, \ldots, p_m) \) is a \term{prefix} of \( w = (w_1, \ldots, w_n) \) if
    \begin{equation*}
      w = (\underbrace{p_1, \ldots, p_m}_p, w_{m+1}, \ldots, w_n).
    \end{equation*}

    \thmitem{def:formal_language/suffix} The word \( s \) is a \term{suffix} of \( w \) if \( \op{rev}(s) \) is a prefix of \( \op{rev}(w) \).

    \thmitem{def:formal_language/subword} The word \( v \) is a \term{subword} of \( w \) if there exists a prefix \( p \) and a suffix \( s \) of \( v \) such that
    \begin{equation*}
      w = pvs.
    \end{equation*}

    \thmitem{def:formal_language/kleene_star} The \term{Kleene star} \( \mscrA^* \) of \( \mscrA \) is the set of all words over \( \mscrA \). If we wish to exclude the empty word, like we often do, we instead write \( \mscrA^+ \) for the set of all non-empty words over \( \mscrA \).

    \thmitem{def:formal_language/language} A \term{language} over \( \mscrA \) is any subset of \( \mscrA^* \). Note that in some contexts like \hyperref[subsec:propositional_logic]{propositional logic} or \hyperref[subsec:first_order_logic]{first-order logic} the term \enquote{language} may refer to the alphabet itself (see \fullref{rem:propositional_language_is_alphabet}).
  \end{thmenum}
\end{definition}

\begin{proposition}\label{thm:kleene_star_is_monoid}
  For any alphabet \( \mscrA \), the Kleene star \( \mscrA^* \) is a \hyperref[def:unital_magma/monoid]{monoid} under concatenation.
\end{proposition}
\begin{proof}
  Concatenation is clearly associative and the empty word \( \varepsilon \) is a \hyperref[def:magma_identity]{two-sided identity} under concatenation.
\end{proof}

\begin{definition}\label{def:formal_grammar}\mcite[def. 2.2]{Sipser2013}
  Let \( V \) and \( \Sigma \) be disjoint nonempty subsets of some \hyperref[def:formal_language/alphabet]{alphabet}.

  \begin{thmenum}
    \thmitem{def:formal_grammar/terminals} We call elements of \( \Sigma \) \term{terminals}. We denote terminals in abstract grammars using lowercase Greek letters, and we denote words using lowercase Latin letters.

    \thmitem{def:formal_grammar/non_terminals} We call elements of \( V \) \term{non-terminals}. By convention, variables are denoted using capital letters.

    \thmitem{def:formal_grammar/start} We assume that a special \term{start symbol} \( S \in V \) is fixed.

    \thmitem{def:formal_grammar/production_rules} We define a binary \hyperref[def:relation]{relation} \( \to \) of \term{production rules} over \( (V \cup \Sigma)^* \).

    We impose the restriction that no rules of the form \( \varepsilon \to v \) exist for any word \( v \). We do allow, however, production rules of the form \( v \to \varepsilon \). Such rules are called \term{\( \varepsilon \)-rules}.

    Rules describe transformations that define how a language is \enquote{generated} starting from \( S \). See \fullref{def:grammar_derivation} and \fullref{ex:natural_number_arithmetic_grammar/derivation}.

    \thmitem{def:formal_grammar/grammar} The quadruple \( G \coloneqq (V, S, \Sigma, \to) \) is called a \term{formal grammar} or simply a \term{grammar}.
  \end{thmenum}
\end{definition}

\begin{definition}\label{def:chomsky_hierarchy}
  We will define the \term{Chomsky hierarchy} of \hyperref[def:formal_grammar]{formal grammars}. We can classify a grammar \( G = (V, S, \Sigma, \to) \) as follows, based on their rules:
  \begin{thmenum}
    \thmitem{def:chomsky_hierarchy/unrestricted} In general, every production rule has the form \( v \to w \), where both \( v \) and \( w \) are words consisting of terminal and non-terminals, and \( v \) is nonempty.

    When no additional restrictions are imposed on the rules of the grammar, we call it \term{unrestricted grammar}. The other levels of the hierarchy are subsets of the unrestricted grammars.

    \thmitem{def:chomsky_hierarchy/non_contracting} The grammar is \term{non-contracting} if \( \len(v) \leq \len(w) \) for every production rule \( v \to w \).

    \thmitem{def:chomsky_hierarchy/context_sensitive} The grammar is \term{context-sensitive} if every rule has the form \( aAb \to w \) for some non-terminal \( A \), arbitrary words \( a \) and \( b \), and a nonempty word \( w \).

    The requirement for \( w \) to be nonempty is set up so that context-sensitive grammars are non-contracting.

    \thmitem{def:chomsky_hierarchy/context_free} The grammar is \term{context-free} every rule has the form \( A \to w \) for some non-terminal \( A \) and a nonempty word \( w \).

    Unlike for context-sensitive languages, \( w \) is allowed to be empty. Thus, a context-free grammar is non-contracting, however it may not be context-sensitive if it has \( \varepsilon \)-rules.

    \thmitem{def:chomsky_hierarchy/regular} Finally, the grammar is \term{regular} if every rule has one of the forms
    \begin{align*}
      &A \to \varepsilon, \\
      &A \to B \tau, \\
      &A \to \tau B, \\
      &A \to \tau,
    \end{align*}
    where \( A \) and \( B \) are non-terminals and \( \tau \) is a terminal.

    Regular grammars are obviously context-free.
  \end{thmenum}
\end{definition}

\begin{example}\label{ex:natural_number_arithmetic_grammar/backus_naur_form}
  We define a grammar for primary school notation of multiplication and division of \hyperref[def:set_of_natural_numbers]{natural numbers}. Note that we consider the numbers in \( \BbbN \) only as symbols, without any regard to semantics.

  Let \( V \coloneqq \set{ N, O, M, E } \) and \( \Sigma \coloneqq \BbbN \cup \set{ \times, \div, (, ) } \). Define the grammar
  \begin{equation}\label{eq:ex:natural_number_arithmetic_grammar/backus_naur_form/simple}
    \begin{aligned}
      N &\to 0 \\
      N &\to 1 \\
      \phantom{N} &\vdots \\
      N &\to n \\
      \phantom{N} &\vdots \\
      O &\to \times \\
      O &\to \div \\
      E &\to N \\
      E &\to (E O E)
    \end{aligned}
  \end{equation}

  It is convenient to use the following shorthands:
  \begin{equation}\label{eq:ex:natural_number_arithmetic_grammar/backus_naur_form/shorthand}
    \begin{aligned}
      N &\to 0 \mid 1 \mid 2 \mid \ldots \\
      O &\to \times \mid \div \\
      E &\to N \mid (E O E)
    \end{aligned}
  \end{equation}

  We can choose different non-terminals as the starting symbol. The symbol \( N \) corresponds to numbers, \( O \) corresponds to operations, and \( E \) can be either a number or an expression. We say that \eqref{eq:ex:natural_number_arithmetic_grammar/backus_naur_form/shorthand} specifies a \term{grammar schema}. With any starting symbol, the grammar is clearly \hyperref[def:chomsky_hierarchy]{context-free}.
\end{example}

\begin{remark}\label{rem:backus_naur_form}
  The infinitude of possible rules in \fullref{ex:natural_number_arithmetic_grammar/backus_naur_form} may not bother us formally, but when dealing with software implementations, we must have a finite number of rules. An example of a nontrivial grammar in the wild is the Python grammar that can be found in \cite{Python39Grammar}. There are also other advantages of introducing a more convenient metasyntax (a syntax for describing language syntax).

  For \hyperref[def:chomsky_hierarchy/context_free]{context-free grammars}, is often convenient to use the \term{Backus-Naur form (BNF)}. For \fullref{ex:natural_number_arithmetic_grammar/backus_naur_form}, the BNF is
  \begin{bnf*}
    \bnfprod{nonzero digit} {\bnfts{1} \bnfor \bnfts{2} \bnfor \bnfts{3} \bnfor \bnfts{4} \bnfor \bnfts{5} \bnfor \bnfts{6} \bnfor \bnfts{7} \bnfor \bnfts{8} \bnfor \bnfts{9}} \\
    \bnfprod{digit}         {\bnfts{0} \bnfor \bnfpn{nonzero digit}} \\
    \bnfprod{number}        {\bnfpn{nonzero digit} \bnfor \bnfpn{number} \bnfsp \bnfpn{digit}} \\
    \bnfprod{operation}     {\bnfts{\( \times \)} \bnfor \bnfts{\( \div \)}} \\
    \bnfprod{expression}    {\bnfpn{number} \bnfor \bnfts{(} \bnfsp \bnfpn{number} \bnfsp \bnfpn{operation} \bnfsp \bnfpn{number} \bnfsp \bnfts{)}}.
  \end{bnf*}

  The obvious difference is that we explicitly define numbers via their decimal representation, which means that we get a finite amount of rules. Compared to \eqref{eq:ex:natural_number_arithmetic_grammar/backus_naur_form/simple} some other differences are:
  \begin{itemize}
    \item Variables are denoted by \( \langle \)words enclosed in angle brackets\( \rangle \), so that we can name variables more descriptively using more than one symbol.
    \item Terminals are, by convention, put in \enquote{quotes}. In human-readable rich text documents like this one, it is sometimes possible to use different fonts, and so instead of using \enquote{quotes} we specify terminals using an \texttt{upright typewriter font}.
    \item Free-text rules can be specified using a normal font. This is also only used in human-readable rich text documents, however this usage is justified because such rules are only beneficial for human understanding and not for machine parsing.
    \item By convention, the symbol \( \Coloneqq \) is used instead of \( \to \) for specifying transition rules.
    \item Different rules with the same, source are concatenated as in \eqref{eq:ex:natural_number_arithmetic_grammar/backus_naur_form/shorthand}.
    \item In order to fully describe a context-free grammar, we must only specify its Backus-Naur form and its starting variable.
  \end{itemize}
\end{remark}

\begin{definition}\label{def:grammar_derivation}
  Fix a \hyperref[def:formal_grammar]{formal grammar} \( G = (V, S, \Sigma, \to) \).

  \begin{thmenum}
    \thmitem{def:grammar_derivation/derivation} We define the binary relation \( \Rightarrow \) on the Kleene star \( (V \cup \Sigma)^* \) by declaring that, for every two \hyperref[def:formal_language/word]{words} \( p \) and \( s \) over \( V \cup \Sigma \) and every production rule \( v \to w \), we have \( pvw \Rightarrow pws \). We also define the relation \( \Rightarrow_L \) as a restriction of \( \Rightarrow \) to the cases where \( p \) contains only terminal symbols and \( \Rightarrow_R \) --- if \( s \) contains only terminal symbols.

    A \term{derivation} of the word \( w_n \) from \( w_1 \) is a \hyperref[def:quiver_path/directed]{directed path} in the quiver induced by the relation \( \Rightarrow \), i.e.
    \begin{equation}\label{eq:def:grammar_derivation/derivation}
      w_1
      \reloset {u_1 \to v_1} \implies
      w_2
      \reloset {u_2 \to v_2} \implies
      \cdots
      \reloset {u_{n-2} \to v_{n-2}} \implies
      w_{n-1}
      \reloset {u_{n-1} \to v_{n-1}} \implies
      w_n.
    \end{equation}

    A \term{leftmost derivation} is a derivation performed using \( \Rightarrow_L \) rather than \( \Rightarrow \). \term{Rightmost derivations} are defined analogously.

    We say that \( w_n \) is \term{derivable} from \( w_1 \) if there exists a derivation from \( w_1 \) to \( w_n \).

    We denote the \hyperref[def:relation_closures/transitive]{transitive closure} of \( \Rightarrow \) by \( \reloset + \Rightarrow \) and the \hyperref[def:relation_closures/reflexive]{reflexive} closure of \( \reloset + \Rightarrow \) by \( \reloset {*} \Rightarrow \). Clearly \( w_1 \) is derivable from \( w_n \) if and only if \( w_1 \reloset {*} \Rightarrow w_n \).

    The leftmost and rightmost derivations generate the same derivability relation --- the only potential difference is in the order of rule applications in the derivation itself.

    \thmitem{def:grammar_derivation/unambiguous}\mcite[def. 2.7]{Sipser2013} We say that the word \( w \) can be derived \term{unambiguously} if it has a unique leftmost derivation.

    Define the set
    \begin{equation*}
      D \coloneqq \set{ w \in (V \cup \Sigma)^* \colon S \reloset + \Rightarrow w }
    \end{equation*}
    of all words derivable from the starting symbol \( S \).

    If every word in \( D \) can be derived unambiguously, we say that the grammar itself is \term{unambiguous}.

    In an unambiguous grammar, \fullref{thm:structural_induction_on_unambiguous_grammars} can be used on the \hyperref[def:quiver/simple]{simple directed graph} \( (D, \Rightarrow_L) \). Indeed, every word in \( D \) is derivable from \( S \) and every leftmost derivation is unique.

    \thmitem{def:grammar_derivation/grammar_language} The \term{language} of the grammar is the set
    \begin{equation*}
      \mscrL(G) \coloneqq \set{ w \in \Sigma^* \colon S \reloset + \Rightarrow w }
    \end{equation*}
    of all terminal-only words derivable from the starting symbol \( S \).

    We also say that strings in \( \mscrL(G) \) are \term{generated} by the grammar \( G \).

    If a language can be generated by a \hyperref[def:chomsky_hierarchy/regular]{regular} grammar, we say that the language itself is regular, and similarly for \hyperref[def:chomsky_hierarchy/context_free]{context-free} and \hyperref[def:chomsky_hierarchy/context_sensitive]{context-sensitive} grammars.

    For other grammars, for example \hyperref[def:chomsky_hierarchy/unrestricted]{unrestricted} or \hyperref[def:chomsky_hierarchy/non_contracting]{non-contracting}, such a terminology is not established.
  \end{thmenum}
\end{definition}

\begin{example}\label{ex:natural_number_arithmetic_grammar/derivation}
  We continue \fullref{ex:natural_number_arithmetic_grammar/backus_naur_form}. Depending on our choice of starting symbol, we can derive different sets of words.

  For the sake of simplifying our exposition and proof, however, we will assume the simpler grammar described in \eqref{eq:ex:natural_number_arithmetic_grammar/backus_naur_form/simple}.

  Choose the starting symbol to be \( E \). We will show that this grammar is unambiguous.

  \Cref{fig:ex:natural_number_arithmetic_grammar/derivation/ambiguous} demonstrates that removing the parentheses makes even this simple grammar ambiguous.

  \begin{figure}
    \hfill
    \includegraphics[page=1]{output/ex__natural_number_arithmetic_grammar__derivation.pdf}
    \hfill\hfill
    \caption{The unique way to produce the parenthesized arithmetic expression \( ((6 \div 2) \times 3) \)}
    \label{fig:ex:natural_number_arithmetic_grammar/derivation/unambiguous}
  \end{figure}

  \begin{figure}
    \hfill
    \includegraphics[page=2]{output/ex__natural_number_arithmetic_grammar__derivation.pdf}
    \hfill\hfill
    \caption{Different leftmost derivations of the parenthesis-less arithmetic expression \( 6 \div 3 \times 2 \)}
    \label{fig:ex:natural_number_arithmetic_grammar/derivation/ambiguous}
  \end{figure}

  We will show that \( G \) is unambiguous. Let \( w \) be a word in \( \mscrL(G) \). We explicitly build the leftmost derivation of \( w \) using recursion on \( \len(w) \):
  \begin{itemize}
    \item If \( \len(w) = 1 \), then \( w = n \in \BbbN \), and the word has been derived as \( E \to N \to n \).

    \item Assume that \( w \) is unambiguously derived for \( \len(w) < m + 2 \) and let \( \len(w) = m + 2 \), then \( w \) is necessarily enclosed in parentheses. Let \( w = ( \sigma_1 \ldots \sigma_m ) \) be the symbols of \( w \). Because of the parentheses, the only possibility for \( \sigma_1 \ldots \sigma_m \) is that it consists of two words in \( \mscrL(G) \) with either a multiplication symbol \( \times \) or a division symbol \( \div \) between them. Let \( k \) be the index of the operator symbol, that is, the index such that \( \sigma_1 \ldots \sigma_{k-1} \) and \( \sigma_{k+1} \ldots \sigma_m \) both belong to \( \mscrL(G) \).

    By the inductive hypothesis, both \( \sigma_1 \ldots \sigma_{k-1} \) and \( \sigma_{k+1} \ldots \sigma_m \) are unambiguously derived from \( E \). Then \( w \) is generated by the rule \( E \to (E O E) \), where the operator symbol \( \sigma_k \) determines the terminal of \( O \). Therefore, the derivation of \( w \) is also unambiguous.
  \end{itemize}
\end{example}

\begin{definition}\label{def:ordered_arborescence}
  An \term{ordered arborescence} is an \hyperref[def:arborescence]{arborescence} \( T = (G, A) \) with a \hyperref[def:partially_ordered_set]{partial order} \( \leq \) such that every set of \hyperref[def:arborescence/ancestry]{siblings} is a \hyperref[def:partially_ordered_set_chain_and_antichain]{chain}.
\end{definition}

\begin{definition}\label{def:grammar_syntax_tree}\mimprovised
  Fix a \hyperref[def:chomsky_hierarchy/context_free]{context-free} \hyperref[def:formal_grammar]{formal grammar} \( G = (V, S, \Sigma, \to) \).

  For every word \( w \) in \( \mscrL(G) \), we will build an \hyperref[def:ordered_arborescence]{ordered} \hyperref[def:arborescence/undirected]{rooted tree} whose \hyperref[def:arborescence/ancestry]{leaves} are the symbols of \( w \) and whose root is \( S \). We will call this a \term{syntax tree} for \( w \).

  \begin{figure}
    \hfill
    \includegraphics[page=1]{output/alg__grammar_syntax_tree.pdf}
    \hfill\hfill
    \caption{A grammar tree corresponding to the expression \( (6 \div (3 \times 2)) \) from \cref{fig:ex:natural_number_arithmetic_grammar/derivation/unambiguous}}
    \label{fig:def:grammar_syntax_tree}
  \end{figure}

  Fix a derivation
  \begin{equation}\label{eq:def:grammar_syntax_tree/derivation}
    S \Rightarrow w_1 \Rightarrow \cdots \Rightarrow w_{n-1} \Rightarrow w_n.
  \end{equation}

  We use \hyperref[rem:natural_number_recursion]{natural number recursion} on \( n \) to build the tree. Note that, for the purposes of recursion, we allow \( w_n \) to contain non-terminals.

  \begin{itemize}
    \item In the trivial case where \( n = 0 \), and there is no actual derivation, we build a single-vertex tree with root \( S \).

    \item Suppose that we can build a tree for all derivations of length \( m - 1 \) and fix a derivation \eqref{eq:def:grammar_syntax_tree/derivation} of length \( n \).

    First, build a tree \( T \) from the derivation
    \begin{equation*}
      S \Rightarrow w_1 \Rightarrow \cdots \Rightarrow w_{n-2} \Rightarrow w_{n-1}.
    \end{equation*}

    There must exist words \( p \), \( s \) and \( v \) and a non-terminal \( A \) such that
    \begin{equation*}
      w_{n-1} = pAs \Rightarrow pvs = w_n.
    \end{equation*}

    There already exists a leaf for \( A \) in \( T \). For every symbol in \( v \), add a new node as a child of this node.
  \end{itemize}
\end{definition}

\begin{proposition}\label{thm:unambiguous_grammar_syntax_trees}
  A \hyperref[def:chomsky_hierarchy/context_free]{context-free} grammar is \hyperref[def:chomsky_hierarchy/context_free]{unambiguous} if and only if every word has a unique \hyperref[def:grammar_syntax_tree]{syntax tree}.
\end{proposition}

\begin{theorem}[Structural induction on unambiguous grammars]\label{thm:structural_induction_on_unambiguous_grammars}\mimprovised
  Unlike for the other induction principles in \fullref{rem:induction}, we will not formulate this one via logical formulas. This will complicate us unnecessarily. We will instead describe how the principle is used in practice.

  Let \( G = (V, S, \Sigma, \to) \) be an \hyperref[def:chomsky_hierarchy/context_free]{unambiguous} \hyperref[def:chomsky_hierarchy/context_free]{context-free} \hyperref[def:formal_grammar]{formal grammar}.

  Suppose that we want to prove a statement for every word in \( \mscrL(G) \). It is sufficient to perform the following for every rule \( A \to w \):
  \begin{displayquote}
    Let \( A_1, \ldots, A_n \) be all non-terminals of \( w \) and let \( u_0, \ldots, u_n \) be subwords of \( w \) such that
    \begin{equation*}
      w = u_0 A_1 u_1 A_2 \ldots A_n u_n.
    \end{equation*}

    Let \( v_1, \ldots, v_n \) be arbitrary words in \( \mscrL(G) \) derivable from \( A_1, \ldots, A_n \), respectively, so that we have the syntax tree
    \begin{equation*}
      \begin{aligned}
        \includegraphics[page=1]{output/thm__structural_induction_on_unambiguous_grammars.pdf}
      \end{aligned}
    \end{equation*}

    Then we must prove the statement for the word
    \begin{equation*}
      u_0 v_1 u_1 \ldots v_n u_n.
    \end{equation*}
  \end{displayquote}

  Compare this principle to the more general \fullref{thm:well_founded_induction}.
\end{theorem}
\begin{proof}
  Clearly every word in \( \mscrL(G) \) can be obtained in this way.
  The role of non-ambiguity is discussed in the proof of \fullref{thm:well_founded_induction}.
\end{proof}


\section{Discrete Armijo Gradient}
\lab{sec:DAG}
The Discrete Armijo Gradient algorithm can be used to solve
problem $\mathbf P_c$ defined in~\eqref{sub:Proc} where
$f(\cdot)$ is continuously differentiable.\\

The Discrete Armijo Gradient algorithm approximates gradients by finite differences.
It can be used for problems where the cost function is evaluated by
computer code that defines a continuously differentiable function but for
which obtaining analytical expressions for the gradients is impractical
or impossible.\\

Since the Discrete Armijo Gradient algorithm is sensitive to discontinuities in
the cost function, we recommend not to use this algorithm if the simulation program
contains adaptive solvers with loose precision settings, such as
EnergyPlus~\cite{Crawley2001:1}.
On such functions, the algorithm is likely to fail.
In Section~\ref{sec:AlgSel}, we recommend algorithms that
are better suited for such situations.\\

We will now present the Discrete Armijo Gradient algorithm and the
Armijo step-size subprocedure.

\noindent
\begin{minipage}[b]{\textwidth}
\begin{algorithm}
[Discrete Armijo Gradient Algorithm]
~\\
{\em
\begin{tabularx}{\headwidth}{m{2cm}l}
\multicolumn{2}{l}{\hspace{\textwidth}~} \\ \\[-8ex]\\
\hline \\[-2ex]
 \textbf{Data}:
     & Initial iterate $x_0 \in \mathbf X$.\\
     & $\alpha, \beta \in (0, 1)$, $\gamma \in (0, \infty)$, $k^*, k_0 \in \mathbb Z$,\\
     & $l_{max}, \kappa \in \Na$ (for reseting the step-size calculation).\\
     & Termination criteria $\epsilon_m, \epsilon_x \in \Re_+$,
     $i_{max} \in \Na$.\\
  \textbf{Step 0}:
     & Initialize $i=0$ and $m = 0$.\\
  \textbf{Step 1}:
     & \underline{Compute the {\it search direction} $h_i$.}\\
     & If $\beta^m < \epsilon_m$, stop.\\
     & Else, set $\epsilon = \beta^{k_0 + m }$ and compute, for $j \in \{1, \ldots, n\}$,\\
     & $h_i^j = - \left({f(x_i + \epsilon \, e_j ) - f(x_i)}\right)/{\epsilon}$.\\
\textbf{Step 2} :
     & \underline{Check descent.} \\
     & Compute $\Delta(x_i; h_i) =
     \left( f(x_i + \epsilon \, h_i ) - f(x_i) \right) / \epsilon$.\\
     & If $\Delta(x_i; h_i) < 0$, go to Step 3.\\
     & Else, replace $m$ by $m+1$ and go to Step 1.\\
\textbf{Step 3} :
     & \underline{Line search.}\\
     & Use Algorithm~\ref{al:ArmijoSubPro} (which requires $k^*, l_{max}$ and $\kappa$)
     to compute $k_i$.\\
     & Set
\end{tabularx}
\vspace{-1ex}
 \begin{equation}
   \hspace{3cm} \lambda_i = \argmin_{\lambda \in \{\beta^{k_i}, \beta^{k_i-1} \}  }
     f(x_i + \lambda \, h_i).
\end{equation}
\vspace{-1ex}
\begin{tabularx}{\headwidth}{m{2cm}l}
\textbf{Step 4} :
     & If $f(x_i + \lambda_i \, h_i) - f(x_i) > - \gamma \, \epsilon$,
     replace $m$ by $m+1$ and go to Step 1.\\
\textbf{Step 5} :
     & Set $x_{i+1} = x_i + \lambda_i \, h_i$.\\
     & If $\| \lambda_i \, h_i \| < \epsilon_x$, stop. Else, replace $i$ by $i+1$ and go to Step 1.\\
\hline
\end{tabularx}
}
~\\ \lab{al:DAG}
\end{algorithm}
\end{minipage}

\noindent
\begin{minipage}[b]{\textwidth}
\begin{subequations}
\begin{algorithm}
[Armijo Step-Size Subprocedure]
~\\
{\em
\begin{tabularx}{\headwidth}{m{2cm}l}
\multicolumn{2}{l}{\hspace{\textwidth}~} \\ \\[-8ex]\\
\hline \\[-2ex]
 \textbf{Data}:
     & Iteration number $i \in \Na$, iterate $x_i \in \Re^n$,
     search direction $h_i \in \Re^n$,\\
     & $k^*, k_{i-1} \in \mathbb Z$, $\alpha, \beta \in (0, \ 1)$, and
     $\Delta(x_i;h_i) \in \Re$ with $\Delta(x_i;h_i) < 0$,\\
     & parameter for restart $l_{max}, \kappa \in \Na$.\\
\textbf{Step 0}:
     & Initialize $l=0$.\\
     & If $i = 0$, set $k' = k^*$, else set $k' = k_{i-1}$.\\
\textbf{Step 1}:
     & Replace $l$ by $l+1$, and test the conditions
\end{tabularx}
\vspace{-1ex}
\begin{eqnarray}
\hspace{2cm}     f(x_i + \beta^{k'} \, h_i ) - f(x_i) & \le &
     \beta^{k'} \, \alpha \, \Delta(x_i; h_i), \label{eq:ArmSteSizPro1} \\
\hspace{2cm}     f(x_i + \beta^{k' - 1} \, h_i ) - f(x_i) & > &
     \beta^{k' - 1} \, \alpha \, \Delta(x_i; h_i).
\label{eq:ArmSteSizPro2}
\end{eqnarray}
\vspace{-1ex}
\begin{tabularx}{\headwidth}{m{2cm}l}
\textbf{Step 2}:
& If $k'$ satisfies \eqref{eq:ArmSteSizPro1} and
 \eqref{eq:ArmSteSizPro2}, return $k'$.\\
\textbf{Step 3}:
   & If $k'$ satisfies \eqref{eq:ArmSteSizPro2}
     but not \eqref{eq:ArmSteSizPro1},\\
   & \hspace{1cm} replace $k'$ by $k'+1$.\\
   & else,\\
   & \hspace{1cm} replace $k'$ by $k'-1$.\\
   & If $l < l_{max}$ or $k_{i-1} \le  k^* + \kappa$, go to Step 1. Else, go to Step 4.\\
\textbf{Step 4}:
     & Set $\mathbf K \triangleq \{ k \in \mathbb Z \ | \ k \ge k^* \}$,
     and compute\\
     & $k'
     \triangleq \min_{k \in \mathbf K} \{ k \ | \
     f(x_i + \beta^k \, h_i ) - f(x_i) \le \beta^k \, \alpha \, \Delta(x_i; h_i) \}$.\\
     & Return $k'$.\\
\hline
\end{tabularx}
}
\lab{al:ArmijoSubPro}
\end{algorithm}
\end{subequations}
\end{minipage}
~\\
Note that in Algorithm~\ref{al:ArmijoSubPro}, as $\beta \to 1$,
the number of tries to compute the Armijo step-size is likely to go to infinity.
Under appropriate assumptions one can show that $\alpha = 1/2$ yields
fastest convergence~\cite{Pol97:1}.\\

The step-size Algorithm~\ref{al:ArmijoSubPro} requires often only a small number
of function evaluations.
However, occasionally, once a very small step-size has occurred,
Algorithm~\ref{al:ArmijoSubPro} can trap the Discrete Armijo Gradient algorithm
into using a very small step-size for all subsequent iterations.
Hence, if $k_{i-1} >  k^* + \kappa$, we reset the step-size by computing Step 4.\\

Algorithm~\ref{al:DAG} together with the step-size Algorithm~\ref{al:ArmijoSubPro}
have the following convergence properties~\cite{Pol97:1}.

\begin{theorem}
Let $f \colon \Re^n \to \Re$ be continuously differentiable and bounded below.
\begin{enumerate}
\item
If Algorithm~\ref{al:DAG} jams at $x_i$, cycling indefinitely in the loop
defined by Steps 1-2 or in the loop defined by Steps 1-4,
then $\nabla f(x_i) = 0$.
\item
If $\{ x_i \}_{i=0}^\infty$ is an infinite sequence constructed by
Algorithm~\ref{al:DAG} and Algorithm~\ref{al:ArmijoSubPro}
in solving~\eqref{sub:Proc},
then every accumulation point $\widehat x$ of $\{ x_i \}_{i=0}^\infty$
satisfies $\nabla f(\widehat x) = 0$.
\end{enumerate}
\rbox
\end{theorem}

Note that $\epsilon \, h_i$ has the same units as the cost function,
and the algorithm evaluates $x_i + \lambda \, h_i$ for some $\lambda \in \Re_+$.
Thus, the algorithm is sensitive to the scaling of the problem variables,
a rather undesirable effect.
Therefore, in the implementation of
Algorithm~\ref{al:DAG} and Algorithm~\ref{al:ArmijoSubPro},
we normalize the cost function values by replacing, for all $x \in \Re^n$,
$f(x)$ by $f(x)/f(x_0)$, where $x_0$ is the initial iterate.
Furthermore, we set $x_0 = 0$ and evaluate the cost function for the values
$\chi^j + x^j \, s^j$, $j \in \{1, \ldots, n\}$, where
$x^j \in \Re$ is the $j$-th component of the design parameter computed in
Algorithm~\ref{al:DAG} or Algorithm~\ref{al:ArmijoSubPro} and
$\chi^j \in \Re$ and $s^j \in \Re$ are the setting of the parameters
\texttt{Ini} and \texttt{Step}, respectively, for
the $j$-th design parameter in the optimization command file
(see page~\pageref{par:comFil}).

In view of the sensitivity of the Discrete Armijo Gradient algorithm
to the scaling of the problem variables and the cost function values,
the implementation of penalty and barrier functions may cause
numerical problems if the penalty is large compared to the unpenalized
cost function value.

If box-constraints for the independent parameters are specified,
then the transformations~\eqref{sub:traBoxCon} are used.

%-------------------------
\subsection{Keywords}
For the Discrete Armijo Gradient algorithm, the command file (see page~\pageref{par:comFil}) must only contain continuous parameters.\\

To invoke the algorithm,
the \texttt{Algorithm} section of the GenOpt command file must have the following form:
\begin{lstlisting}
Algorithm{
   Main  = DiscreteArmijoGradient;
   Alpha = Double;     // 0 < Alpha < 1
   Beta  = Double;     // 0 < Beta  < 1
   Gamma = Double;     // 0 < Gamma
   K0    = Integer;
   KStar = Integer;
   LMax  = Integer;    // 0 <= LMax
   Kappa = Integer;    // 0 <= LMax
   EpsilonM = Double;  // 0 < EpsilonM
   EpsilonX = Double;  // 0 < EpsilonX
}
\end{lstlisting}
The entries are defined as follows:
\begin{codedescription}
\item [Main]
The name of the main algorithm.
\item [Alpha]
The variable $\alpha$ used in Step 1 and in Step 4 of Algorithm~\ref{al:ArmijoSubPro}.
A typical value is $\alpha = 1/2$.
\item [Beta]
The variable $\beta$ used in approximating
the gradient and doing the line search.
A typical value is $\beta = 0.8$.
\item [Gamma]
The variable $\gamma$ used in Step 4 of Algorithm~\ref{al:DAG}
to determine
whether the accuracy of the gradient approximation will be increased.
\item [K0]
The variable $k_0$ that determines the initial accuracy
of the gradient approximation.
\item [KStar]
The variable $k^*$ used to initialize the line search.
\item [LMax]
The variable $l_{max}$ used in Step 3 of Algorithm~\ref{al:ArmijoSubPro}
to determine whether the line search needs to be reinitialized.
\item [Kappa]
The variable $\kappa$ used in Step 3 of Algorithm~\ref{al:ArmijoSubPro}
to determine whether the line search needs to be reinitialized.
\item [EpsilonM]
The variable $\epsilon_m$ used in the determination criteria
$\beta^m < \epsilon_m$ in Step 1 of Algorithm~\ref{al:DAG}.
\item [EpsilonX]
The variable $\epsilon_x$ used in the determination criteria
$\| \lambda_i \, h_i \| < \epsilon_x$ in Step 5 of Algorithm~\ref{al:DAG}.
\end{codedescription}

% ===============================================


\documentclass[11pt]{article}

\input{preamble}

%\usepackage{tikz}
%\usepackage{tikz-qtree}
\usepackage{qtree}
\usepackage{xfrac}

\title{Week 6: Decision Trees and Variational Autoencoders}
\author{\url{http://mlvu.github.io}}

\begin{document}

\maketitle

\section{Decision trees}

Imagine you do a newspaper round to help you get through these lean times.  On your round, you encounter a number of dogs that either bark or (try to) bite. The dogs are described by the following binary features: \emph{Heavy}, \emph{Smelly}, \emph{Big} and \emph{Growling}. Consider the following set of examples:

\begin{center}
\begin{tabular}{c  c  c  c  | c }
Heavy & Smelly &  Big & Growling & Bites \\
\hline
\rc{No}  & \rc{No} & \rc{No}  & \rc{No}  & \oc{No} \\
\rc{No}  & \rc{No} & \gc{Yes} & \rc{No}  & \oc{No} \\
\gc{Yes} & \gc{Yes}& \rc{No}  & \gc{Yes} & \oc{No} \\
\gc{Yes} & \rc{No} & \rc{No}  & \gc{Yes} & \bc{Yes} \\
\rc{No}  &\gc{Yes} & \gc{Yes} & \rc{No}  & \bc{Yes} \\
\rc{No}  & \rc{No} & \gc{Yes} & \gc{Yes} & \bc{Yes} \\
\rc{No}  & \rc{No} & \rc{No}  & \gc{Yes} & \bc{Yes} \\
\gc{Yes} &\gc{Yes} & \rc{No}  & \rc{No}  & \bc{Yes} \\
\hline
\end{tabular}
\end{center}

\qu What is the entropy of the target value \emph{Bites} in the data?
\ans{
\[H(\text{Bites})= - \sfrac{5}{8}\log_2\sfrac{5}{8} - \sfrac{3}{8}\log_2\sfrac{3}{8} \approx 0.9544\]
}{}

\qu Which attribute would the ID3 algorithm choose to use for the root of the tree (without pruning)?
\ans{Growling}{}

\qu What is the information gain of the attribute you chose in the previous question?
\ans{approximately 0.0487}{}

\qu Draw the full decision tree that would be learned for this data using ID3 without pruning.
\ans{

\begin{figure}[H]
\centering
\includegraphics[width=0.7\linewidth]{Tree}
\caption{Decision tree}
\end{figure}
}{}

\qu Suppose three new dogs appear in your round as listed in the table below. Classify them using the tree from the previous question.

\begin{center}
\begin{tabular}{c c c c c | c }
Dog & Heavy & Smelly &  Big & Growling & Bites \\
\hline
Buster & \gc{Yes} & \gc{Yes} & \gc{Yes} & \gc{Yes} & \ans{\oc{No}}{?} \\
Pluto  & \rc{No}  & \gc{Yes} & \rc{No}  & \gc{Yes} & \ans{\oc{No}}{?} \\
Zeus   & \gc{Yes} & \gc{Yes} & \rc{No}  & \rc{No} & \ans{\bc{Yes}}{?} \\
\hline
\end{tabular}
\end{center}

\qu Someone proposes a new scheme to prevent overfitting: she suggests to set a pre-defined maximum depth for the decision trees. When the standard algorithm reaches this depth, it terminates. Could this help to prevent overfitting? Why (not)?
\ans{
It ensures smaller, simpler trees (a smaller model space) and therefore can reduce the risk of overfitting. Overfitting is essentially memorizing too much of your data. Smaller trees can memorize less.
}{}

\qu In the maximum depth scheme introduced above, how would you determine a good value for the maximum depth for a given data set?

\ans{
The maximum depth is a \emph{hyperparameter}. We can choose good values for our hyperparameters by splitting our training data into a validation set and trying different values (either by cross validation or just single runs).}{}

\qu Why can't we apply L1-regularization to this decision tree learning problem?

\ans{L1 regularization assumes that your model is described by a real valued vector of parameters (i.e. your model space is continuous.) Decision trees have a \emph{discrete} model space.}{}

\section{Variational autoencoders}

The maximum likelihood principle tells us to optimize the quantity $p(x\mid \theta)$ as a function of $\theta$ (the model parameters).For complex models, this does not usually lead to a closed form solution. Instead, we will rewrite the maximum likelihood objective using the following decomposition.

\[
\ln p(\x\mid \theta) = L(\rc{q}, \theta) + KL(\rc{q}, p) \\
\]
with
\begin{align*}
\rc{q}(\z\mid \x)\;&\text{any distribution on $\z$} \\
KL(\rc{q}, p)\;&\text{the Kullback-Leibler divergence\footnotemark}\\
&\text{between $\rc{q}(\z \mid \x)$ and $p(\z \mid \x, \theta)$}\\
L(\rc{q}, \theta) &= \E_\rc{q} \ln \frac{p(\x, \z \mid \theta) }{\rc{q}(\z\mid \x)}
\end{align*}

\footnotetext{The KL divergence can be defined using $\log_2$ or using $\ln$. The first leads to a value in \emph{bits}, the second to a value in the slightly more abstract unit of \emph{nats}. That is, the information distance between the two distributions is the same, but expressed in different units. In the context of the VAE, the natural logarithm $\ln$ is preferred, mainly because it cancels out neatly against the exponent in the pdf of the normal distribution.
}

We will first prove that this equality holds. We start with the right hand side, fill in the components, and derive the left-hand side.

\qu Fill in the blanks. We have written everything in terms of \emph{expectations} $\E$ to simplify the notation. The expectation is over the random variable $\z$, while $\x$ has some definite value. Note that $\E f(\z) + \E g(\z) = \E\left [f(\z) + g(\z)\right]$.

\begin{align*}
L(\rc{q}, \theta) + KL(\rc{q}, p) &= \mathbb{E}_\rc{q} \ln \frac{p(\x, \z \mid \theta) }{\rc{q}(\z\mid \x)} - \mathbb{E}_\rc{q} \ln \frac{p(\z \mid \x, \theta)}{\rc{q}(\z\mid \x)}\\
&= \ans{\E_\rc{q}  \ln p(\x,\z\mid \theta) - \E_\rc{q} \ln \rc{q}(\z\mid \x) - \E_\rc{q} \ln p(\z\mid \x, \theta) + \E_\rc{q} \ln \rc{q}(\z \mid \x)}{\ldots}\\
&= \mathbb{E}_\rc{q} \ln p(\x, \z \mid \theta)  - \mathbb{E}_\rc{q} \ln p(\z \mid \x, \theta) \\
&= \mathbb{E}_\rc{q} \ln \frac{p(\x, \z \mid \theta) }{ p(\z \mid \x, \theta) } = \ans{\mathbb{E}_\rc{q} \ln \frac{p(\z\mid \x,  \theta)p(\x\mid \theta) }{ p(\z \mid \x, \theta)}}{\ldots}\\
&= \mathbb{E}_\rc{q} \ln p(\x \mid \theta)  = \ln p(\x \mid \theta)
\end{align*}

In EM, we search by alternately (1) optimizing $L(\rc{q}\mid \theta)$ with respect to $\theta$ and (2) setting $\rc{q}$ equal to $\gc{p}$ (so that the KL term becomes zero).

\qu For the variational autoencoder, we cannot (easily) perform this last step. Why not?
\ans{In the variational autoencoder, our model is a neural network that transforms $\z$ into a distribution on $\x$. To set the KL divergence term equal to zero, we would have to compute $p(\z \mid \x, \theta)$: i.e. a probability distribution on $\z$ that indicates for which $\z$ our observed $\x$ is most likely.

While sampling techniques exist to approximate this kind of distribution, they are costly and can be very inaccurate.
}{}

~\linebreak
Instead, we \emph{approximate} $p(\z \mid\x, \theta)$ with a neural network $\rc{q_\bv}(\z \mid \x)$ that produces a distribution on $\z$ given some $\x$. We call the neural network computing $p(\x \mid \x, \theta)$ $\gc{p_\bw}(\x\mid \z)$, to make the notation a little more friendly. Here, $\gc{\bw}$, stands for all parameters of the \gc{p} network, and $\rc{\bv}$ stands for all parameters of the \rc{q} network.\footnotemark

\footnotetext{We've turned $\theta$ into $\gc{\bw}$ and added parameters $\rc{\bv}$ for our approximation $\rc{q}$ on the conditional distribution on $\z$. We've also taken the parameters out of the conditional, because we will always talk about the function ``given the parameter''; we will never talk about the probability on the parameters themselves.}

This gives us an auto-encoder-like structure. An input is mapped to a distribution $\rc{q_\bv}(\z\mid \x)$ by the \rc{encoder}. We sample a single $\z$ from this distribution and pass it through the \gc{decoder} $\gc{p_\bw}(\x \mid \z)$ to produce a distribution on $\x$ (see the \href{https://mlvu.github.io/lectures/51.Deep\%20Learning2.annotated.pdf}{slides} for diagrams).

To find a way to train such an architecture, we turn again to our decomposition of the likelihood. In our new notation:
\[
\ln \gc{p_\bw}(\x) = L(\rc{\bv}, \gc{\bw}) + KL(\rc{q}, \gc{p}) \p \\
\]

The KL divergence term is difficult to compute: it's an expectation, and it contains the function $\gc{p_\bw}(\z \mid \x)$ which requires us to invert the \gc{decoder} neural network (that is, to reason about the inputs given the outputs).

However, because the KL divergence is always positive, we know that

\[
\ln \gc{p_\bw}(\x) \geq L(\rc{\bv}, \gc{\bw}) \\
\]
for \emph{any} $\rc{q_\bv}$ we choose. This is why $L$ is called the variational \emph{lower bound}.\footnotemark~If we choose our parameters $\gc{\bw}, \rc{\bv}$ to maximize $L$, we are also, indirectly, maximizing $\ln \gc{p_\bw}(\x)$.\footnotemark

\footnotetext{The word \emph{variational} comes from the fact that one of its arguments, \rc{q}, is a function (the calculus of functions is called \emph{variational} calculus). For our purposes, this distinction doesn't matter much, since the function $\rc{q}$ is defined by a set of parameters $\rc{\bv}$, so ultimately we will take the derivative over those parameters, as we are used to.}

\footnotetext{How close the lower bound $L$ comes to the true value $\gc{p_\bw}(\x)$ depends on how well our encoder network $\rc{q_\bv}$ approximates the true conditional distribution on $\z$: $\gc{p_\bw}(\z \mid\x)$. I.e. how small the KL term in the original decomposition is.}

To do so, we rewrite $L(\rc{\bv}, \gc{\bw})$ into two separate terms: a KL divergence and an expectation:

\[
L(\rc{\bv}, \gc{\bw}) = - KL(\rc{q_\bv}(\z\mid\x), \gc{p_\bv}(\z)) + \E_\rc{q_\bv} \ln \gc{p_\bw}(\x\mid \z)
\]

\qu Show that this equation holds. That is, rewrite the left part into the right. We will assume that all expectations are over $\rc{q}$.
\begin{align*}
	L(\rc{\bv}, \gc{\bw}) &= \E_\rc{q} \ln \frac{\gc{p_\bw}(\x, \z)}{\rc{q_\bv}(\z\mid \x)} \\
	&= \ans{\E \ln \gc{p_\bw}(\x, \z) - \E \rc{q_\bv}(\z \mid \x)}{\ldots}\\
	&= \ans{\E \ln \left[\gc{p_\bw}(\x\mid \z)\gc{p_\bw}(\z)\right] - \E \ln \rc{q_\bv}(\z \mid \x)}{\ldots} \\
	&= \ans{\E \ln \gc{p_\bw}(\x \mid \z) + \E \ln \gc{p_\bw}(\z) - \E \ln \rc{q_\bv}(\z \mid \x)}{\ldots} \\
	&= \E \ln \gc{p_\bw}(\x \mid \z) - \left [ \E \ln \rc{q_\bv}(\z \mid \x) - \E \ln \gc{p_\bw}(\z) \right ]\\
	&= \E \ln \gc{p_\bw} (\x\mid \z) - KL(\rc{q_\bv}(\z\mid\x), \gc{p_\bw}(\z))
\end{align*}

Thus, to optimize our variational autoencoder, we should maximize $L$. In other words, $-L$ is our loss function. The only problem left to solve is that the second term is an expectation (which we cannot compute explicitly).

\qu How is this solved in practice?

\ans{We take a single sample from $\rc{q_\bv}(\z\mid \x)$ and use $\ln \gc{p_\bw}(\x\mid \z)$ as a (very crude) estimate of the expectation term.

To let the gradient propagate through the sampling, we add a sample from the standard MVN to the input and transform it to a sample from $\rc{q_\bv(\z\mid \x)}$ by multiplying by a matrix $A$ (with $\Sigma = AA^T$) and adding the mean.}{}

\end{document}