\subsection{Transfinite arithmetic}\label{subsec:transfinite_arithmetic}

Our purpose is to extend natural number arithmetic to ordinals and cardinals. It turns out that the two are rather different. We will first introduce some additional concepts, however.

\begin{definition}
  Let \( A \) and \( B \) be sets of \hyperref[def:ordinal]{ordinals}. We say
\end{definition}

\begin{definition}\label{def:ordinal_arithmetic}
  We recursively define arithmetic operations for arbitrary \hyperref[def:ordinal]{ordinals} as extensions of the corresponding operations of \hyperref[def:peano_arithmetic]{Peano arithmetic}.

  \begin{thmenum}
    \thmitem{def:ordinal_arithmetic/addition}\mcite[lemma 66.6]{OpenLogicFull} The \term{sum} of \( \alpha \) and \( \beta \) extends \eqref{eq:def:peano_arithmetic/PA4} and \eqref{eq:def:peano_arithmetic/PA5} with a case for limit ordinals:
    \begin{equation}\label{eq:def:ordinal_arithmetic/addition}
      \alpha + \beta \coloneqq \begin{cases}
        \alpha,                                            &\beta = 0 \\
        \op{succ}(\alpha + \gamma),                        &\beta = \op{succ}(\gamma) \\
        \sup\set{ \alpha + \gamma \given \gamma < \beta }, &\beta \T{is a limit ordinal} \\
      \end{cases}
    \end{equation}

    From \fullref{thm:union_of_set_of_ordinals} it follows that the in limit case \( \alpha + \beta \) is the smallest ordinal strictly larger than \( \alpha + \gamma \) for any \( \gamma < \beta \).

    \thmitem{def:ordinal_arithmetic/multiplication}\mcite[lemma 66.13]{OpenLogicFull} Analogously, the \term{product} of \( \alpha \) and \( \beta \) extends \eqref{eq:def:peano_arithmetic/PA6} and \eqref{eq:def:peano_arithmetic/PA7}:
    \begin{equation}\label{eq:def:ordinal_arithmetic/multiplication}
      \alpha \cdot \beta \coloneqq \begin{cases}
        0,                                                     &\beta = 0 \\
        \alpha \cdot \gamma + \alpha,                          &\beta = \op{succ}(\gamma) \\
        \sup\set{ \alpha \cdot \gamma \given \gamma < \beta }, &\beta \T{is a limit ordinal} \\
      \end{cases}
    \end{equation}

    \thmitem{def:ordinal_arithmetic/exponentiation}\mcite[lemma 66.16]{OpenLogicFull} Exponentiation extends \fullref{def:unital_magma/exponentiation}:
    \begin{equation}\label{eq:def:ordinal_arithmetic/exponentiation}
      \alpha^\beta \coloneqq \begin{cases}
        1,                                               &\beta = 0 \\
        \alpha^\gamma \cdot \alpha,                      &\beta = \op{succ}(\gamma) \\
        \sup\set{ \alpha^\gamma \given \gamma < \beta }, &\beta \T{is a limit ordinal} \\
      \end{cases}
    \end{equation}
  \end{thmenum}
\end{definition}

\begin{remark}\label{rem:ordinal_successor_via_addition}
  For any ordinal \( \alpha \) we have
  \begin{equation*}
    \op{succ}(\alpha)
    \reloset {\ref{eq:def:ordinal_arithmetic/addition}} =
    \op{succ}(\alpha + 0)
    \reloset {\ref{eq:def:ordinal_arithmetic/addition}} =
    \alpha + \op{succ}(0)
    =
    \alpha + 1.
  \end{equation*}

  We will occasionally use the later notation.

  Note that for infinite ordinals \( \op{succ}(\alpha) = 1 + \alpha \) as discussed in \fullref{ex:ordinal_addition}.

  This is an extension of \fullref{rem:natural_number_successor_via_addition}.
\end{remark}

\begin{proposition}\label{thm:ordinal_addition_is_monotone}
  \hyperref[def:ordinal_arithmetic/addition]{Ordinal addition} has the following monotonicity properties:
  \begin{thmenum}
    \thmitem{thm:ordinal_addition_is_monotone/left} Left addition is \hyperref[eq:def:partially_ordered_set/homomorphism/strict]{strictly monotone}:
    \begin{equation}\label{eq:thm:ordinal_addition_is_monotone/left}
      \alpha < \beta \T{implies} \gamma + \alpha < \gamma + \beta.
    \end{equation}

    \thmitem{thm:ordinal_addition_is_monotone/right} Right addition is \hyperref[eq:def:partially_ordered_set/homomorphism/nonstrict]{nonstrictly monotone}:
    \begin{equation}\label{eq:thm:ordinal_addition_is_monotone/right}
      \alpha < \beta \T{implies} \alpha + \gamma \leq \beta + \gamma.
    \end{equation}

    See \fullref{ex:ordinal_addition} for examples where the strict inequality fails.
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:ordinal_addition_is_monotone/left} We proceed by induction on \( \beta \).
  \begin{itemize}
    \item The condition \( \alpha < \beta \) is vacuously false for the base case \( \beta = 0 \), hence by \eqref{eq:def:intuitionistic_propositional_deductive_systems/rules/efq} the statement vacuously holds.

    \item Fix some nonzero \( \beta \) and some \( \alpha < \beta \). If \( \gamma + \alpha < \gamma + \beta \), then
    \begin{equation*}
      \gamma + \alpha < \gamma + \beta < \op{succ}(\gamma + \beta) = \gamma + \op{succ}(\beta).
    \end{equation*}

    Since \( \beta < \op{succ}(\beta) \), we have used the inductive hypothesis to conclude that
    \begin{equation*}
      \alpha < \op{succ}(\beta) \T{implies} \gamma + \alpha < \gamma + \op{succ}(\beta).
    \end{equation*}

    \item Let \( \lambda \) be a limit ordinal and suppose that \eqref{eq:thm:ordinal_addition_is_monotone/left} holds for all \( \beta < \lambda \). Let \( \alpha < \lambda \). Then \( \op{succ}(\alpha) < \lambda \) since \( \lambda \) is a limit ordinal and thus
    \begin{equation*}
      \gamma + \alpha
      <
      \gamma + \op{succ}(\alpha)
      \leq
      \sup\set{ \gamma + \beta \given \beta < \lambda }
      =
      \gamma + \lambda.
    \end{equation*}
  \end{itemize}

  \SubProofOf{thm:ordinal_addition_is_monotone/right} We proceed by induction on \( \gamma \).
  \begin{itemize}
    \item The base case \( \gamma = 0 \) is vacuous.
    \item If \( \alpha + \gamma < \beta + \gamma \), then
    \begin{equation*}
      \alpha + \op{succ}(\gamma)
      \reloset {\eqref{eq:def:ordinal_arithmetic/addition}} =
      \op{succ}(\alpha + \gamma)
      \reloset {\ref{thm:ordinal_successor_strictly_monotone_on_ordinals}} <
      \op{succ}(\beta + \gamma)
      \reloset {\eqref{eq:def:ordinal_arithmetic/addition}} =
      \beta + \op{succ}(\gamma).
    \end{equation*}

    \item Let \( \lambda \) be a limit ordinal and suppose that the lemma holds for every \( \gamma < \lambda \). That is, for every \( \gamma < \lambda \) we have
    \begin{equation*}
      \alpha + \gamma < \beta + \gamma.
    \end{equation*}

    Thus,
    \begin{equation*}
      \alpha + \lambda
      =
      \sup\set{ \alpha + \gamma \given \gamma < \lambda }
      \leq
      \sup\set{ \beta + \gamma \given \gamma < \lambda }
      =
      \beta + \lambda.
    \end{equation*}

    We cannot make a stronger conclusion here --- see \fullref{ex:ordinal_addition} for a counterexample.
  \end{itemize}
\end{proof}

\begin{proposition}\label{thm:ordinal_ordering_via_addition}
  For any two ordinals \( \alpha \) and \( \beta \) it holds that \( \alpha \leq \beta \) if and only if there exists an ordinal \( \gamma \) such that \( \alpha + \gamma = \beta \). This ordinal is unique and satisfies \( \gamma \leq \beta \).

  The strict inequality \( \alpha < \beta \) holds if and only if \( \gamma \neq 0 \).
\end{proposition}
\begin{proof}
  \SufficiencySubProof By definition \( \beta + 0 = \beta \), hence we are not interested in the case \( \alpha = \beta \). That is, we will only consider the case \( \alpha < \beta \).

  We will first show uniqueness of \( \gamma \). Suppose that \( \alpha + \gamma_1 = \beta = \alpha + \gamma_2 \). From \eqref{thm:ordinal_addition_is_monotone/left} it follows that if either \( \gamma_1 < \gamma_2 \) or \( \gamma_1 > \gamma_2 \), we would have a strict inequality. Hence, it only remains for \( \gamma_1 = \gamma_2 \) to hold.

  We now use induction on \( \beta \) on prove the existence of \( \gamma \).
  \begin{itemize}
    \item The condition \( \alpha < \beta \) is vacuously false for the base case \( \beta = 0 \), hence by \eqref{eq:def:intuitionistic_propositional_deductive_systems/rules/efq} the statement vacuously holds.

    \item Suppose that \( \alpha < \beta \) and that there exists a unique \( \gamma \leq \beta \) such that \( \alpha + \gamma = \beta \). Then
    \begin{equation*}
      \alpha + \op{succ}(\gamma)
      \reloset {\eqref{eq:def:ordinal_arithmetic/addition}} =
      \op{succ}(\alpha + \gamma)
      =
      \op{succ}(\beta).
    \end{equation*}

    Since \( \alpha < \beta \) and \( \beta < \op{succ}(\beta) \), we have used the inductive hypothesis to conclude that
    \begin{equation*}
      \alpha < \op{succ}(\beta) \T{implies} \qexists {\underbrace{\delta}_{\mathclap{\op{succ}(\gamma)}}} \alpha + \delta = \op{succ}(\beta).
    \end{equation*}

    Furthermore, since \( \gamma \leq \beta \), then also \( \op{succ}(\gamma) \leq \op{succ}(\beta) \).

    \item Suppose that \( \lambda \) is a limit ordinal, \( \alpha < \lambda \) and for each \( \beta < \lambda \) there exists some \( \gamma_\beta \leq \beta \) such that \( \alpha + \gamma_\beta = \beta \). Define
    \begin{equation*}
      \gamma \coloneqq \sup\set{ \gamma_\beta \given \beta < \lambda }.
    \end{equation*}

    By \fullref{thm:union_of_set_of_ordinals} we have that \( \gamma \) is an ordinal and that \( \gamma_\beta \leq \gamma \) for every \( \beta < \lambda \). Thus,
    \begin{align*}
      \lambda
      &\reloset {\ref{thm:ordinal_is_set_of_smaller_ordinals}} =
      \sup\set{ \beta \given \beta < \lambda }
      \reloset {\T{ind.}} = \\ &=
      \sup\set{ \alpha + \gamma_\beta \given \beta < \lambda }
      \reloset {\eqref{eq:thm:ordinal_addition_is_monotone/right}} \leq \\ &\leq
      \sup\set[\Big]{ \alpha + \delta \given \delta < \sup\set{ \gamma_\beta \given \beta < \lambda } }
      = \\ &=
      \sup\set{ \alpha + \delta \given \delta < \gamma }
      \reloset {\eqref{eq:def:ordinal_arithmetic/addition}} = \\ &=
      \alpha + \gamma.
    \end{align*}

    Aiming at a contradiction, suppose that the strict inequality holds. That is, suppose that \( \lambda < \alpha + \gamma \). Then there exists some \( \delta_0 < \gamma \) such that \( \alpha + \delta_0 > \alpha + \gamma_\beta \) for any \( \beta < \lambda \). It follows from \fullref{thm:monotone_map_converse} that \( \gamma_\beta < \delta_0 \) for any \( \beta < \lambda \) and thus
    \begin{equation*}
      \underbrace{\sup\set{ \gamma_\beta \given \beta < \lambda }}_{\gamma} \leq \delta_0 < \gamma.
    \end{equation*}

    The obtained contradiction shows that such an ordinal \( \delta_0 \) cannot exist and hence \( \lambda = \alpha + \gamma \).

    Furthermore, since \( \gamma_\beta \leq \beta \) for each \( \beta < \lambda \), we have
    \begin{equation*}
      \gamma
      =
      \sup\set{ \gamma_\beta \given \beta < \lambda }
      \leq
      \sup\set{ \beta \given \beta < \lambda }
      =
      \lambda.
    \end{equation*}
  \end{itemize}

  \NecessitySubProof Suppose that \( \alpha \), \( \beta \) and \( \gamma \leq \beta \) are ordinals and that \( \alpha + \gamma = \beta \). Obviously \( \gamma = 0 \) implies that \( \alpha = \beta \). If \( \gamma > 0 \), then from \eqref{eq:thm:ordinal_addition_is_monotone/left} it follows that
  \begin{equation*}
     \beta = \alpha + \gamma > \alpha + 0 = 0.
  \end{equation*}
\end{proof}

\begin{proposition}\label{thm:def:ordinal_addition_algebraic/properties}
  Ordinal number addition is \hyperref[def:magma/associative]{associative} and \hyperref[def:magma/cancellative]{left cancellative}.

  As in \fullref{thm:ordinals_are_well_ordered}, we adapt the corresponding axioms due to \fullref{thm:burali_forti_paradox}. The more concrete result is:
  \begin{thmenum}
    \thmitem{thm:def:ordinal_addition_algebraic/properties/associative} For any three ordinals \( \alpha \), \( \beta \) and \( \gamma \) we have
    \begin{equation*}
      (\alpha + \beta) + \gamma = \alpha + (\beta + \gamma).
    \end{equation*}

    \thmitem{thm:def:ordinal_addition_algebraic/properties/left_cancellative} For any three ordinals \( \alpha \), \( \beta \) and \( \gamma \) such that \( \gamma + \alpha = \gamma + \beta \), we have \( \alpha = \beta \).
  \end{thmenum}

   See \fullref{ex:ordinal_addition} for counterexamples to \hyperref[def:magma/commutative]{commutativity}.

   Compare this with \fullref{thm:def:natural_number_addition/properties} and \fullref{thm:def:cardinal_addition_algebraic/properties}.
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:ordinal_addition_algebraic/properties/associative} We will use induction on \( \gamma \). \Fullref{thm:def:natural_number_addition/properties} already proves the base and successor cases.

  Fix some ordinals \( \alpha \) and \( \beta \). Let \( \lambda \) be a limit ordinal and suppose that
  \begin{equation*}
    (\alpha + \beta) + \gamma = \alpha + (\beta + \gamma)
  \end{equation*}
  holds for all \( \gamma < \lambda \). Then
  \begin{align*}
    (\alpha + \beta) + \lambda
    &\reloset {\eqref{eq:def:ordinal_arithmetic/addition}} =
    \sup\set{ (\alpha + \beta) + \gamma \given \gamma < \lambda }
    \reloset {\T{ind.}} = \\ &=
    \sup\set{ \alpha + (\beta + \gamma) \given \gamma < \lambda }
    \reloset {\eqref{eq:def:partially_ordered_set/homomorphism/strict}} = \\ &=
    \sup\set{ \alpha + \delta \given \delta < \beta + \lambda }
    =
    \alpha + (\beta + \lambda).
  \end{align*}

  \SubProofOf{thm:def:ordinal_addition_algebraic/properties/left_cancellative} Follows from \fullref{thm:monotone_map_converse} and \eqref{eq:thm:ordinal_addition_is_monotone/left}.
\end{proof}

\begin{proposition}\label{thm:ordinal_addition_disjoin_union}
  For any two ordinals \( \alpha \) and \( \beta \), their \hyperref[def:ordinal_arithmetic/addition]{sum} satisfies
  \begin{equation*}
    \alpha + \beta = \ord(\alpha \amalg \beta, \prec),
  \end{equation*}
  where \( \prec \) is the \hyperref[def:lexicographic_order]{lexicographic order} on the \hyperref[def:disjoint_union]{disjoint union} \( \alpha \amalg \beta \).
\end{proposition}
\begin{proof}
  We will explicitly build an \hyperref[def:partially_ordered_set/homomorphism]{order isomorphism} between \( (\alpha + \beta, \in) \) and \( (\alpha \amalg \beta, \prec) \). Define
  \begin{equation*}
    \begin{aligned}
      &f: (\alpha + \beta) \to (\alpha \amalg \beta) \\
      &f(\gamma) \coloneqq \begin{cases}
        (\gamma, 0), &\gamma < \alpha \\
        (\delta, 1), &\qexists \delta (\gamma = \alpha + \delta).
      \end{cases}
    \end{aligned}
  \end{equation*}

  From \fullref{thm:ordinal_ordering_via_addition} it follows that the existence of \( \delta \) such that \( \gamma = \alpha + \delta \) is equivalent to the condition \( \gamma \geq \alpha \). Since \( \gamma < \alpha + \beta \), we have \( \alpha + \delta < \alpha + \beta \) and from \fullref{thm:def:ordinal_addition_algebraic/properties/left_cancellative} we have \( \delta < \beta \). Therefore, \( f \) is a total function. Furthermore, it is single-valued because of the uniqueness of \( \delta \).

  We will first show that \( f \) is a strict order homomorphism. Let \( \gamma_1 < \gamma_2 \). We have the following possibilities:
  \begin{itemize}
    \item If \( \gamma_2 < \alpha \), then \( f(\gamma_1) = (\gamma_1, 0) < (\gamma_2, 0) = f(\gamma_2) \).
    \item If \( \gamma_1 \geq \alpha \), then \( f(\gamma_1) = (\gamma_1, 1) < (\gamma_2, 1) = f(\gamma_2) \).
    \item If \( \gamma_1 < \alpha \leq \gamma_2 \), then \( f(\gamma_1) = (\gamma_1, 0) < (\gamma_2, 1) = f(\gamma_2) \).
  \end{itemize}

  Therefore, \( f \) is a strict order homomorphism and from \fullref{thm:total_order_embedding_iff_strict} it follows that \( f \) is an order embedding. Due to \fullref{thm:totally_ordered_strict_isomorphisms}, in order to show that \( f \) is an order isomorphism it only remains to show that it is a surjective function.

  Let \( (\gamma, k) \in \alpha \amalg \beta \).
  \begin{itemize}
    \item If \( k = 0 \), then \( f(\gamma) = (\gamma, k) \) since \( \gamma \in \alpha \).
    \item If \( k = 1 \), then \( \gamma \in \beta \) and by \eqref{eq:thm:ordinal_addition_is_monotone/left} we have \( \alpha + \gamma < \alpha + \beta \), so \( \alpha + \gamma \) is within the domain of \( f \). Furthermore, as shown in \fullref{thm:ordinal_ordering_via_addition}, if \( \alpha + \delta = \alpha + \gamma \), then \( \delta = \gamma \), Thus, \( f(\alpha + \gamma) = (\gamma, 1) \) .
  \end{itemize}

  Therefore, \( f \) is an order isomorphism between \( (\alpha + \beta, \in) \) and \( (\alpha \amalg \beta, \prec) \) and hence
  \begin{equation*}
    \alpha + \beta = \ord(\alpha \amalg \beta, \prec).
  \end{equation*}
\end{proof}

\begin{example}\label{ex:ordinal_addition}
  The distinction between \eqref{eq:thm:monotone_map_converse} and \eqref{eq:thm:monotone_map_converse} is important. A simple example is provided by any limit ordinal \( \lambda \), in particular by \( \omega \). The example are inconvenient to demonstrate with the recursive definition, however \fullref{thm:ordinal_addition_disjoin_union} eases us.

  In particular \fullref{thm:ordinal_addition_disjoin_union} highlights that adding one ordinal to another, in fact, \enquote{appending} a copy of the second to a copy the first.

  It is clear that
  \begin{equation*}
    0 + \lambda = \ord(0 \sqcap \lambda) = \ord(\lambda) = \lambda.
  \end{equation*}

  That is, we \enquote{append} \( \lambda \) to an empty well-ordered set only to obtain \( \lambda \) again.

  This operation seems different from \( 1 + \lambda \), which \enquote{appends} \( \lambda \) to a well-ordered singleton set. But this operation only \enquote{shifts} \( \lambda \) --- the function
  \begin{equation*}
    \begin{aligned}
      &f: \ord(1 \sqcap \lambda) \to \ord(0 \sqcap \lambda) \\
      &f(k, \gamma) \coloneqq \begin{cases}
        (0, 0),          &k = 0 \\
        (0, \gamma + 1), &k = 1.
      \end{cases}
    \end{aligned}
  \end{equation*}
  is an order isomorphism and thus
  \begin{equation*}
    1 + \lambda = \ord(1 \sqcap \lambda) = \ord(\lambda) = \lambda.
  \end{equation*}

  What the inequality \eqref{eq:thm:monotone_map_converse} gives us is that
  \begin{equation*}
    \lambda \leq 1 + \lambda = \lambda.
  \end{equation*}

  This inequality is, of course, strict when dealing with finite ordinals exclusively, but for limit ordinals its results may be counterintuitive.

  What is more interesting is that, as a consequence of \eqref{eq:thm:monotone_map_converse}, we have \( \lambda < \lambda + 1 \). This can be explained as follows. Instead of \enquote{appending} an infinite set to a finite one, we append a finite set to an infinite one. This way \( \lambda \) cannot \enquote{absorb} \( 1 \) like it does in \( 1 + \lambda \).

  As a consequence of this example, addition of ordinals is not commutative and also not right-cancellative.

  As discussed in the proof of \fullref{thm:def:cardinal_addition_algebraic/properties}, this is only a restriction of well-orders and not of the resulting sets themselves.
\end{example}

\begin{proposition}\label{thm:ordinal_multiplication_cartesian_product}
  For any two ordinals \( \alpha \) and \( \beta \), their \hyperref[def:ordinal_arithmetic/multiplication]{product} satisfies
  \begin{equation*}
    \alpha \cdot \beta = \ord(\alpha \times \beta, \prec),
  \end{equation*}
  where \( \prec \) is the \hyperref[def:lexicographic_order]{lexicographic order} on the \hyperref[def:cartesian_product]{Cartesian product} \( \alpha \times \beta \).
\end{proposition}
\begin{proof}
  We will build an order isomorphism between \( (\alpha \cdot \beta, \in) \) and \( (\alpha \times \beta, \prec) \) using recursion on \( \beta \).
  \begin{itemize}
    \item Both sets \( \alpha \cdot 0 \) and \( \alpha \cdot 0 \) are empty and the empty function is an order isomorphism.
    \item Suppose that \( f: \alpha \cdot \beta \to \alpha \times \beta \) is an order isomorphism. We construct the function
    \begin{equation*}
      \begin{aligned}
        &\widehat f: \alpha \cdot (\beta + 1) \to \alpha \times (\beta + 1) \\
        &\widehat f \coloneqq \begin{cases}
          f(\gamma),       &\gamma < \alpha \cdot \beta \\
          (\delta, \beta), &\qexists \delta (\gamma = \alpha \cdot \beta + \delta).
        \end{cases}
      \end{aligned}
    \end{equation*}

    In complete analogy with \fullref{thm:ordinal_ordering_via_addition} we can prove that \( \widehat f \) is an order isomorphism.

    \item Let \( \lambda \) be a limit ordinal and let \( f_\beta: \alpha \cdot \beta \to \alpha \times \beta \) be an order isomorphism for every \( \beta < \lambda \). Take their union
    \begin{equation*}
      f \coloneqq \bigcup\set{ f_\beta \given \beta < \lambda }.
    \end{equation*}

    The uniqueness of each \( f_\beta \) from \fullref{thm:well_ordered_order_type_existence} shows that \( f_{\beta_1} \subseteq f_{\beta_2} \) for each pair \( \beta_1 < \beta_2 \). Therefore, the union \( f \) is a single-valued partial function. It is also total because every ordinal \( \gamma < \alpha \cdot \lambda \) is contains in the image of the function \( f_{\gamma + 1} \).

    The function \( f \) is an order embedding by construction. It is also surjective because, if \( (\delta, \beta) \in \alpha \times \lambda \), then from the successor step we can conclude that \( f(\alpha \cdot \beta + \delta) = (\delta, \beta) \).

    Therefore, \( f \) is an order isomorphism.
  \end{itemize}
\end{proof}

\begin{proposition}\label{thm:def:ordinal_multiplication_algebraic/properties}
  Similarly to \fullref{thm:def:ordinal_addition_algebraic/properties} for ordinal number addition, multiplication is also \hyperref[def:magma/associative]{associative} and \hyperref[def:magma/cancellative]{left cancellative}:
  \begin{thmenum}
    \thmitem{thm:def:ordinal_multiplication_algebraic/properties/associative} For any three ordinals \( \alpha \), \( \beta \) and \( \gamma \) we have
    \begin{equation*}
      (\alpha \cdot \beta) \cdot \gamma = \alpha \cdot (\beta \cdot \gamma).
    \end{equation*}

    \thmitem{thm:def:ordinal_multiplication_algebraic/properties/left_cancellative} For any three ordinals \( \alpha \), \( \beta \) and \( \gamma \) such that \( \gamma \cdot \alpha = \gamma \cdot \beta \), we have \( \alpha = \beta \).
  \end{thmenum}

  Compare this with \fullref{thm:def:natural_number_multiplication/properties} and \fullref{thm:def:cardinal_multiplication_algebraic/properties}.
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:ordinal_multiplication_algebraic/properties/associative} Associativity follows easily from the obvious isomorphisms between \( (\alpha \times \beta) \times \gamma \) and \( \alpha \times (\beta \times \gamma) \).

  \SubProofOf{thm:def:ordinal_multiplication_algebraic/properties/left_cancellative} Now suppose that \( \gamma \cdot \alpha = \gamma \cdot \beta \). Let \( f \) the unique order isomorphism between \( \gamma \times \alpha \) and \( \gamma \times \beta \).

  Suppose that \( \alpha \neq \beta \). Without loss of generality, suppose that \( \beta \subsetneq \alpha \). Then \( f\restr_{\gamma \times \alpha} \) is the identity mapping and hence any set from \( \gamma \times (\beta \setminus \alpha) \) would make \( f \) not injective.

  The obtained contradiction shows that \( \alpha = \beta \).
\end{proof}

\begin{example}\label{ex:countable_limit_ordinals}
  We already know that \( \omega \) is a limit ordinal. It is clear from \fullref{ex:ordinal_addition} that \( \omega + n \) is a successor ordinal for every natural number \( n \).

  What about \( \omega + \omega \)? This corresponds to \enquote{placing} two copies of the natural numbers one after another.

  Suppose that \( \omega + \omega = \omega \cdot 2 \) is the successor of \( \alpha \). Then \( \alpha < \omega + \omega \) and we can show by induction on the natural numbers that \( \alpha + n < \omega + \omega \). But \( \alpha + 1 = \omega \) by assumption, which contradicts trichotomy of ordinals.

  Therefore, \( \omega + \omega \) is a limit ordinal. Furthermore, \( \omega + \omega \) is the second smallest limit ordinal since only ordinals of the form \( \omega + n \) for nonzero finite \( n \) satisfy \( \omega < \omega + n < \omega + \omega \).

  Both ordinals \( \omega \) and \( \omega + \omega \) are countable by \fullref{thm:omega_equinumerous_with_omega_squared}.

  Another limit ordinal is \( \omega \cdot \omega = \omega^2 \). It is also countable by \fullref{thm:countable_product_of_countable_sets}. Actually \( \omega^n \) for any natural number \( n \) is countable by the same theorem.

  Therefore, any \enquote{\hyperref[def:polynomial]{polynomial}} of the form
  \begin{equation*}
    \alpha_n \omega^n + \alpha_{n-1} \omega^{n-1} + \cdots + \alpha_1 \omega + \alpha_0
  \end{equation*}
  with countable coefficients is also countable.
\end{example}

\begin{definition}\label{def:cardinal_arithmetic}
  We will define arithmetic operations for them. Unlike in \fullref{def:ordinal_arithmetic}, we will directly define the operations as \hyperref[thm:cardinality_existence]{cardinal numbers} of some sets rather than via some form of recursion.

  Fix two ordinals \( \kappa \) and \( \mu \).
  \begin{thmenum}
    \thmitem{def:cardinal_arithmetic/addition} Based on \fullref{thm:ordinal_addition_disjoin_union}, we define their \term{sum} as
    \begin{equation*}
      \kappa + \mu \coloneqq \card(\kappa \amalg \mu),
    \end{equation*}
    where \( \kappa \amalg \mu \) is their \hyperref[def:disjoint_union]{disjoint union}.

    \thmitem{def:cardinal_arithmetic/multiplication} Based on \fullref{thm:ordinal_multiplication_cartesian_product}, we define their \term{product} as
    \begin{equation*}
      \kappa \cdot \mu \coloneqq \card(\kappa \times \mu).
    \end{equation*}

    \thmitem{def:cardinal_arithmetic/exponentiation} We define \term{exponentiation} as
    \begin{equation*}
      \kappa^\mu \coloneqq \card(\fun(\kappa, \mu)).
    \end{equation*}
  \end{thmenum}
\end{definition}

\begin{proposition}\label{thm:def:cardinal_addition_algebraic/properties}
  Cardinal number addition is \hyperref[def:magma/associative]{associative}, \hyperref[def:magma/associative]{commutative} and \hyperref[def:magma/cancellative]{cancellative}.

  Compare this with \fullref{thm:def:natural_number_addition/properties} and \fullref{thm:def:ordinal_addition_algebraic/properties}.
\end{proposition}
\begin{proof}
  Associativity and left cancellation is inherited from the ordinals. Commutativity and right cancellation hold because we are considering arbitrary bijective functions rather than the more restrictive order isomorphisms. Indeed, \( \kappa \amalg \mu \) and \( \mu \amalg \kappa \) may have different order types as demonstrated in \fullref{ex:ordinal_addition}, however there is an obvious bijective function between them.
\end{proof}

\begin{proposition}\label{thm:def:cardinal_multiplication_algebraic/properties}
  Cardinal number multiplication is \hyperref[def:magma/associative]{associative}, \hyperref[def:magma/associative]{commutative} and \hyperref[def:magma/cancellative]{cancellative}.

  Compare this with \fullref{thm:def:natural_number_multiplication/properties} and \fullref{thm:def:ordinal_addition_algebraic/properties}.
\end{proposition}
\begin{proof}
  The result follows from the same considerations as in \fullref{thm:def:cardinal_addition_algebraic/properties}.
\end{proof}

\begin{proposition}\label{thm:double_and_square_of_cardinal}
  For every cardinal \( \kappa \) we have
  \begin{thmenum}
    \thmitem{thm:double_and_square_of_cardinal/double} \( \kappa + \kappa = 2\kappa \).
    \thmitem{thm:double_and_square_of_cardinal/square} \( \kappa \cdot \kappa = \kappa^2 \).
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:double_and_square_of_cardinal/double} Obviously \( \kappa \amalg \kappa = 2 \times \kappa \).
  \SubProofOf{thm:double_and_square_of_cardinal/square} The function
  \begin{equation*}
    \begin{aligned}
      &T: \fun(2, \kappa) \to \kappa \times \kappa \\
      &T(f) \coloneqq (f(0), f(1))
    \end{aligned}
  \end{equation*}
  is clearly injective. It is also surjective because for any ordered pair \( (\gamma, \delta) \in \kappa \times \kappa \) we can define the function
  \begin{equation*}
    \begin{aligned}
      &f: 2 \to \kappa \\
      &f(k) \coloneqq \begin{cases}
        \gamma, k = 0 \\
        \delta, k = 1
      \end{cases}
    \end{aligned}
  \end{equation*}

  Then \( T(f) = (\gamma, \delta) \).
\end{proof}

\begin{lemma}\label{thm:square_of_infinite_cardinal}\mcite[thm. 3.1]{nLab:cardinal_arithmetic}
  If \( \kappa \) is an infinite cardinal, then \( \kappa = \kappa^2 \).
\end{lemma}
\begin{proof}
  Obviously \( \kappa \leq \kappa^2 \). Suppose that the lemma does not always hold and let \( \kappa \) be the smallest ordinal for which \( \kappa < \kappa^2 \).

  Consider the \hyperref[def:well_ordered_set]{well-ordered set}
  \begin{equation*}
    (\kappa \times \kappa \times \kappa, \prec),
  \end{equation*}
  where \( \prec \) denotes the corresponding \hyperref[def:lexicographic_order]{lexicographic order}. The set is well-ordered as a consequence of \fullref{thm:well_ordered_lexicographic_order_is_well_ordered}.

  Define the subset
  \begin{equation*}
    S \coloneqq \set{ (\alpha, \beta, \gamma) \in \kappa \times \kappa \times \kappa \given \alpha = \max\set{ \beta, \gamma } }.
  \end{equation*}

  The elements of \( S \) are determined exactly by any two of its three coordinates, hence \( S \) is equinumerous with \( \kappa \times \kappa \). Since \( \kappa < \kappa^2 \), there exists some \hyperref[def:partially_ordered_set_interval/ray]{initial segment}
  \begin{equation*}
    S_{<(\alpha_0, \beta_0, \gamma_0)} = \set{ (\alpha, \beta, \gamma) \in S \given \alpha < \alpha_0 \T{and} \beta < \beta_0 \T{and} \gamma < \gamma_0 }.
  \end{equation*}

  Note that \( \alpha_0^2 = \alpha_0 \) since \( \alpha_0 < \kappa \). Then
  \begin{equation*}
    \kappa = \card(S_{<(\alpha_0, \beta_0, \gamma_0)}) \leq \card(S_{<(\alpha_0, \alpha_0, \alpha_0)}) = \alpha_0^2 = \alpha_0 < \kappa,
  \end{equation*}
  which is a contradiction.

  Therefore, \( \kappa = \kappa^2 \) for every infinite cardinal \( \kappa \).
\end{proof}

\begin{proposition}\label{thm:simplified_cardinal_arithmetic}
  Unlike \hyperref[def:ordinal_arithmetic]{ordinal arithmetic} with its intricacies like \fullref{ex:ordinal_addition}, \hyperref[def:cardinal_arithmetic]{cardinal arithmetic} has a simpler behavior:
  \begin{thmenum}
    \thmitem{thm:simplified_cardinal_arithmetic/finite} If \( \kappa \) and \( \mu \) are finite cardinals, then \( \kappa + \mu \) and \( \kappa \cdot \mu \) are the familiar operations on \hyperref[def:set_of_natural_numbers]{natural numbers}.

    \thmitem{thm:simplified_cardinal_arithmetic/infinite} If either \( \kappa \) or \( \mu \) is infinite, then
    \begin{equation*}
      \kappa + \mu = \max\set{ \kappa, \mu }.
    \end{equation*}

    If, additionally, both are either zero or nonzero, then
    \begin{equation*}
      \kappa \cdot \mu = \max\set{ \kappa, \mu }.
    \end{equation*}
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:simplified_cardinal_arithmetic/finite} The addition of ordinals defined in \fullref{def:ordinal_arithmetic/addition} is an extension of addition of natural numbers, hence the two are equivalent for finite ordinals. The equivalence with cardinal addition defined in \fullref{def:cardinal_arithmetic/addition} comes from \fullref{thm:ordinal_addition_disjoin_union} and the fact that every finite ordinal is a cardinal as demonstrated in \fullref{thm:natural_numbers_are_cardinals}.

  Analogously, equivalence of cardinal and ordinal multiplication follows from \fullref{thm:ordinal_multiplication_cartesian_product}.

  \SubProofOf{thm:simplified_cardinal_arithmetic/infinite} Suppose that either \( \kappa \) or \( \mu \) is infinite and let \( \nu \coloneqq \max\set{ \kappa, \mu } \). The cases where either of them is zero are trivial, hence suppose that both are nonzero.

  We have
  \begin{equation*}
    \kappa \amalg \mu \subseteq \nu \amalg \nu = 2 \amalg \nu \subseteq \nu \times \nu,
  \end{equation*}
  hence \( \kappa + \mu \leq \nu \cdot \nu = \nu^2 \).

  Furthermore there exists an obvious injective function from \( \nu \) to \( \kappa \amalg \mu \) (which is different depending on whether \( \nu = \kappa \) or \( \nu = \mu \)). Therefore,
  \begin{equation*}
    \nu \leq \kappa + \mu \leq \nu^2.
  \end{equation*}

  For multiplication we have \( \kappa \times \mu \subseteq \nu \times \nu \), hence
  \begin{equation*}
    \nu \leq \kappa \cdot \mu \leq \nu^2.
  \end{equation*}

  The rest follows from \fullref{thm:square_of_infinite_cardinal}.
\end{proof}

\begin{corollary}\label{thm:aleph_zero_is_strong_limit}
  The first infinite cardinal \( \aleph_0 \) is a \hyperref[def:successor_and_limit_cardinal/strong_limit]{strong limit cardinal}.

  See also \fullref{thm:aleph_zero_is_regular}
\end{corollary}
\begin{proof}
  \Fullref{thm:simplified_cardinal_arithmetic/finite} states that cardinal exponentiation extends natural number exponentiation. Hence, we can conclude that \( 2^n < \aleph_0 \) for any \( n < \aleph_0 \) since the former are finite and the latter is not.
\end{proof}

\begin{lemma}\label{thm:power_set_via_subsets}
  Fix a set \( A \). The power set \( \pow(A) \) is equinumerous with the set of \hyperref[def:boolean_function]{Boolean-valued functions} \( \pow(A, \set{ T, F }) \).

  More precisely, then the operator
  \begin{equation*}
    \begin{aligned}
      &T: \fun(A, \set{ T, F }) \to \pow(A) \\
      &T(f) \coloneqq \set{ f(x) = T \given x \in A }.
    \end{aligned}
  \end{equation*}
  is bijective.
\end{lemma}
\begin{proof}
  Injectivity is clear. To see surjectivity, fix some subset \( B \subset A \) and define
  \begin{equation*}
    \begin{aligned}
      &f: A \to \set{ T, F } \\
      &f(x) \coloneqq \begin{cases}
        T, &x \in B \\
        F, \T{otherwise}.
      \end{cases}
    \end{aligned}
  \end{equation*}

  Clearly \( f \in \fun(A, \set{ T, F }) \) and \( T(f) = B \).
\end{proof}

\begin{proposition}\label{thm:cardinal_exponentiation_power_set}
  For every set \( A \) we have
  \begin{equation*}
    \card(\pow(A)) = 2^{\card(A)}.
  \end{equation*}
\end{proposition}
\begin{proof}
  The proof is similar to \fullref{thm:power_set_via_subsets}, but much more convoluted.

  Let \( \varphi: A \to \card(A) \) be \( \psi: \pow(A) \to \card(\pow(A)) \) bijective functions. Note that
  \begin{equation*}
    2^{\card(A)} = \card(\fun(\card(A), \set{ 0, 1 }))
  \end{equation*}
  by definition. Let \( \theta: \fun(\card(A), \set{ 0, 1 }) \to 2^{\card(A)} \) be a bijective function.

  Define the operator
  \begin{equation*}
    \begin{aligned}
      &T: 2^{\card(A)} \to \card(\pow(A)) \\
      &T(p) \coloneqq \psi\parens{ \set{ \varphi^{-1}(\gamma) \given \gamma \in \card(A) \T{and} \theta^{-1}(p)(\gamma) = 1 } }.
    \end{aligned}
  \end{equation*}

  This operator is bijective since for any \( \delta \in \card(\pow(A)) \) we can define
  \begin{equation*}
    \begin{aligned}
      &f: \card(A) \to \set{ 0, 1 } \\
      &f(\gamma) \coloneqq \begin{cases}
        1, &\varphi^{-1}(\gamma) \in \psi^{-1}(\delta) \\
        0, &\T{otherwise.}
      \end{cases}
    \end{aligned}
  \end{equation*}
  so that \( T(\theta(f)) = \delta \).

  Therefore, \( \card(A) \) and \( \card(\pow(A)) \) are equinumerous.
\end{proof}

\begin{proposition}\label{thm:strong_limit_cardinal_is_weak_limit}
  If \( \mu \) is the successor cardinal of \( \kappa \), then \( \mu \leq 2^\kappa \).

  In particular, every \hyperref[def:successor_and_limit_cardinal/strong_limit]{strong limit cardinal} is a \hyperref[def:successor_and_limit_cardinal/strong_limit]{weak limit cardinal}.

  Furthermore, if \( \kappa \) is infinite, then \fullref{hyp:generalized_continuum_hypothesis} implies that \( \mu = 2^\kappa \).
\end{proposition}
\begin{proof}
  It is clear from \fullref{thm:cantor_power_set_theorem} and \fullref{thm:cardinal_exponentiation_power_set} that \( \kappa < 2^\kappa \). By definition, \( \mu \) is the smallest cardinal such that \( \kappa < \mu \). Therefore, \( \mu \leq 2^\kappa \).
\end{proof}

\begin{conjecture}[Generalized continuum hypothesis]\label{hyp:generalized_continuum_hypothesis}
  For every ordinal \( \alpha \) we have
  \begin{equation*}
    \aleph_{\alpha + 1} = 2^{\aleph_\alpha},
  \end{equation*}

  For the definition and the properties of the \( \aleph \) hierarchy, see \fullref{def:aleph_hierarchy}. For the related \( \beth \) hierarchy, see \fullref{def:beth_hierarchy}.

  This is a vast generalization of \fullref{hyp:continuum_hypothesis} from the case \( \alpha = 0 \) to arbitrary ordinals.
\end{conjecture}

\begin{corollary}\label{thm:limit_cardinals_and_gch}
  \Fullref{hyp:generalized_continuum_hypothesis} implies that every \hyperref[def:successor_and_limit_cardinal/weak_limit]{weak limit cardinal} is a \hyperref[def:successor_and_limit_cardinal/strong_limit]{strong limit cardinal}.

  The converse holds in \logic{ZFC} as shown in \fullref{thm:strong_limit_cardinal_is_weak_limit}.
\end{corollary}
\begin{proof}
  Follows from \fullref{thm:infinite_cardinal_is_aleph} and \fullref{hyp:generalized_continuum_hypothesis}.
\end{proof}

\begin{definition}\label{def:beth_hierarchy}\mcite[def. 68.17]{OpenLogicFull}
  Similarly to \fullref{def:aleph_hierarchy}, we use transfinite recursion to define, for each ordinal \( \alpha \), the cardinal
  \begin{equation}\label{eq:def:beth_hierarchy}
    \beth_\alpha \coloneqq \begin{cases}
      \omega,                                       &\alpha = 0 \\
      2^\beta,                                      &\alpha = \op{succ}(\beta) \\
      \sup\set{ \beth_\beta \given \beta < \alpha } &\alpha \T{is a limit ordinal}.
    \end{cases}
  \end{equation}

  Unlike \fullref{def:aleph_hierarchy}, it is able to explicitly build the successor of any member of the hierarchy. \Fullref{hyp:generalized_continuum_hypothesis} states that \( \aleph_\alpha = \beta_\alpha \) for every ordinal \( \alpha \), however in general it is only provable that \( \aleph_\alpha \leq \beta_\alpha \) --- see \fullref{thm:strong_limit_cardinal_is_weak_limit}.
\end{definition}


\documentclass[../main.tex]{subfiles}
\begin{document}
\chapter{Recurrence Relations}
As we mentioned briefly about the power of recursion is in the whole algorithm design and analysis, we dedicate this chapter to recurrence relation. To summarize, recurrence relation can help with:
\begin{itemize}
    \item Recurrence relation naturally represent the relation of recursion. Examples will be shown in Chapter.~\ref{chapter_divide_conquer}.
    \item Any iteration can be translated into recurrence relation. Some examples can be found in Chapter.~\ref{chapter_complexity_analysis}.
    \item Recurrence relation together with mathematical induction is the most powerful tool to \textbf{design} and \textbf{prove} the correctness of algorithm(chapter.~\ref{chapter_divide_conquer} and Chapter.~\ref{chapter_complexity_analysis}).
    \item Recurrence relation can be applied to algorithm complexity analysis( Chapter.~\ref{chapter_complexity_analysis}).
\end{itemize}

In  the following chapters of this part, we endow application meanings to these formulas and discuss how to realize the mentioned uses.

\section{Introduction}
\paragraph{Definition and Concepts} A recurrence relation is  function expressed with the same function. More precisely, as defined in mathematics, recurrence relation is an equation that recursively defines a sequence or multidimensional array of values; once one or more initial terms are given, each further term of the sequence or array is defined as a function of the preceding terms. Fibonacci sequence is one of the most famous recurrence relation which is defined as $f(n)=f(n-1)+f(n-2), f(0)=0, f(1)=1$.
\begin{equation}
    a_n = \Psi(n, a_{n-1})  \text{ for $n \leq 0$,}
\end{equation}

We use $a_n$ to denote the value at index $n$, and the recurrence function is marked as $\Psi(n, P)$, $P$ is all preceding terms that needed to build up this recurrence relation. Like the case of factorial, each factorial number only relies on the result of the previous number and its current index, this recurrence relation can be written as the following equation:

A recurrence relation needs to start from \textit{initial value(s)}. For the above relation, $a_0$ needs to be defined and it will be the first element of a recurrence relation. The above relation is only related to the very first preceding terms, which is called recurrence relation of \textit{first order}. If $P$ includes multiple preceding terms, a recurrence relation of order $k$ can be easily extended as:
\begin{equation}
    a_n = \Psi(n, a_{n-1}, a_{n-2}, ..., a_{n-k})  \text{ for $n \leq k$,}
\end{equation}
In this case, $k$ initial values are needed for defining a sequence. Initial values can be given any values but then once initial values are decided,  the recurrence determines the sequence uniquely. Thus, initial values are also called the \textit{degree of freedom} for solutions to the recurrence.

Many natural functions are easily expressed as recurrence:
\begin{itemize}
    \item Polynomial: $a_n = a_{n-1}+1, a_1=1 \xrightarrow{} a_n = n$.
    \item Exponential: $a_n = 2 \times a_{n-1}, a_1=1 \xrightarrow{} a_n = 2^{n-1}$.
    \item Factorial: $a_n = n\times a_{n-1}, a_1=1 \xrightarrow{}a_n = n!$
\end{itemize}

\paragraph{Solving Recurrence Relation} In real problems, we might care about the value of recursion at $n$, that is compute $a_n$ for any given $n$, and there are two ways to do it:
\begin{itemize}
    \item Programming: we utilize the computational power of computer and code in either iteration or recursion to build up the value at any given $n$. For example, $f(2)=f(1)+f(0)=1$, $f(3)=f(2)+f(1)=2$, and so on. With this iteration, we would need $n-1$ steps to compute $f(n)$.
    \item Math: we solve the recurrence relation by obtaining an explicit or closed-form expression which is a non-recursive function of $n$. With the solution at hand, we can get  $a_n$ right away.
\end{itemize}
 Recurrence relations plays an important role in the analysis of algorithms. Usually, time recurrence relation $T(n)$ is defined to analyze the time complexity of solving a problem with input instance of size $n$. The field of complexity analysis studies the closed-form solution of $T(n)$; that is to say the functional relation between $T(n)$ with $n$ that it cares, not each exact value.

In this section, we focus on solving the recurrence relation using math to get a closed-form solution.  Categorizing the recurrence relation can help us pinpoint each type's solving methods.

\paragraph{Categorizes} Recurrence relation is essentially discreet function, which can be naturally categorized as \textbf{linear} (such as function $y=mx+b)$ and \textbf{non-linear}; quadratic, cubic and so on (such as $y=ax^2+bx+c, y=ax^3+bx^2+cx+d$). In the field of algorithmic problem solving, linear recurrence relation is commonly used and researched, thus we deliberately leave the non-linear recurrence relation and its method of solving out of the scope of this book.
% \begin{itemize}
%     \item Linear Recurrence Relation:
    \begin{itemize}
        \item \textbf{Homogeneous linear recurrence relation:} When the recurrent relation is linear homogeneous of degree $k$ with constant coefficients, it is in the form, and is also called order-k homogeneous linear recurrence with constant coefficients.
        \begin{equation}
            a_n=c_1a_{n-1} + c_2a_{n-2} + ... + c_k a_{n-k}.
            \label{eq_homogeneous_recurrence_relation}
        \end{equation}
        $a_0, a_1, ..., a_{k-1}$ will be initial values.

        \item \textbf{Non-homogeneous linear recurrence relation:} An order-k non-homogeneous linear recurrence with constant coefficients is defined in the form:
                \begin{equation}
            a_n=c_1a_{n-1} + c_2a_{n-2} + ... + c_k a_{n-k}+f(n).
             \label{eq_non_homogeneous_recurrence_relation}
        \end{equation}
        f(n) can be 1 or $n$ or $n^2$ and so on.
        \item \textbf{Divide-and-conquer recurrence relation}: When $n$ is not decreasing by a constant as does in Eq.~\ref{eq_homogeneous_recurrence_relation} and Eq.~\ref{eq_non_homogeneous_recurrence_relation}, instead by a constant factor, with the equality as shown below, it is called divide and conquer recurrence relation.
        \begin{equation}
    a_n=a_{n/b}+f(n)
    \label{divide_conquer_eq1}
\end{equation}
where $a\leq 1, b>1$, and $f(n)$ is a given function, which usually has $f(n)= cn^k$.
%The special method for solving divide and conquer which is named \textit{master method} will be introduced in Chapter.~\ref{chapter_divide_conquer} when we have enough understanding of the term--divide and conquer.

    \end{itemize}
%     \item Non-linear Recurrence Relation
% \end{itemize}
We will introduce general methods to solve a linear recurrence relation but leave out the part of divide and conquer recurrence relation in this chapter for reason that divide and conquer recurrence relation will most likely to be solved with just roughly, as shown in Chapter.~\ref{chapter_complexity_analysis} to just estimate the time complexity resulted from the divide and conquer method.

\section{General Methods to Solve Linear Recurrence Relation}
 No general method for solving recurrence function is known yet, however, linear recurrence relation with finite initial values and previous states, constant coefficients can always be solved. Due to the fact that the recursion is essentially mathematical induction, the most general way of solving any recurrence relation is to use \textit{mathematical induction} and \textit{iterative method}. This also makes the the mathematical induction, in some form, the foundation of all correctness proofs for computer programs.  We  examine these two methods by solving two recurrence relation: $a_n = 2\times a_{n-1} + 1, a_0 = 0$ and $a_n=a_{n/2} + 1$.

\subsection{Iterative Method}
The most straightforward method for solving recurrence relation no matter its linear or non-linear is the \textit{iterative method}. Iterative method is a technique or procedure in computational mathematics that it iteratively replace/substitute each $a_n$ with its recurrence relation $\Psi(n, a_{n-1}, a_{n-2}, ..., a_{n-k})$ till all items ``disappear'' other than the initial values. Iterative method is also called substitution method.

We demonstrate iteration with a simple non-overlapping recursion.
\begin{align}
\label{complexity_eq_binary_search}
    T(n)&=T(n/2)+O(1)\\
    &=T(n/2^2)+O(1)+O(1)\notag\\
    &=T(n/2^3)+3O(1)\notag\\
    &=...\notag\\
    &=T(1)+kO(1)
\end{align}
We have $\frac{n}{2^k}=1$, we solve this equation and will get $k=\log_2 n$. Most likely $T(1)=O(1)$ will be the initial condition, we replace this, and we get $T(n)=O(\log_2 n)$.

However, when we try to apply iteration on the third recursion:  $T(n)=3T(n/4)+O(n)$. It might be tempting to assume that $T(n)=O(n\log n)$ due to the fact that $T(n)=2T(n/2)+O(n)$ leads to this time complexity.
\begin{align}
\label{complexity_non_overlap_1}
    T(n)&=3T(n/4)+O(n)\\
    &=3(3T(n/4^2)+n/4)+n=3^2T(n/4^2)+n(1+3/4)\notag\\
    &=3^2(3T(n/4^3)+n/4^2)+n(1+3/4)=3^3T(n/4^3)+n(1+3/4+3/4^2)\\
    &=...\\
    &=3^kT(n/4^k)+n\sum_{i=0}^{k-1}(\frac{3}{4})^{i}
\end{align}
\subsection{Recursion Tree}
Since the term of T(n) grows, the iteration can look messy. We can use recursion tree to better visualize the process of iteration. In a recursive tree, each node represents the value of a single subproblem, and a leaf would be a subproblem. As a start, we expand $T(n)$ as a node with value $n$ as root, and it would have three children each represents a subproblem $T(n/4)$. We further do the same with each leaf node, until the subproblem is trivial and be a base case. In practice, we just need to draw a few layers to find the rule. The cost will be the sum of costs of all layers.  The process can be seen in  Fig.~\ref{fig:recursive_tree}.
\begin{figure}[!ht]
    \centering
    \includegraphics[width=0.98\columnwidth]{fig/recursion_tree_non_overlap.png}
    \caption{The process to construct a recursive tree for $T(n) = 3T(\floor*{n/4}) + O(n)$. There are totally k+1 levels. Use a better figure.  }
    \label{fig:recursive_tree}
\end{figure}
 In this case, it is the base case $T(1)$. Through the expansion with iteration and recursion tree, our time complexity function becomes:
\begin{align}
\label{complexity_non_overlap_2}
    T(n)&=\sum_{i=1}^{k}L_i + L_{k+1}\\
    &=n\sum_{i=1}^{k}(3/4)^{i-1}+3^kT(n/4^k)
\end{align}

In the process, we can see that Eq.~\ref{complexity_non_overlap_2} and Eq.~\ref{complexity_non_overlap_1} are the same.  Because $T(n/4^k)=T(1)=1$, we have $k=\log_4 n$.
\begin{align}
\label{complexity_non_overlap_2}
    T(n)&\leq n\sum_{i=1}^{\infty}(3/4)^{k-1}+3^kT(n/4^k)\\
    &\leq 1/(1-3/4)n+3^{\log_4 n} T(1)= 4n+n^{log_4 3}
    &\leq 5n \\
    &=O(n)
\end{align}

\subsection{Mathematical Induction}
Mathematical induction is a mathematical proof technique, and is essentially used to prove that a property $P(n)$ holds for every natural number $n$, i.e. for $n=0, 1, 2, 3$, and so on. Therefore, in order to use induction, we need to make a \textit{guess} of the closed-form solution for $a_n$. Induction requires two cases to be proved.
\begin{enumerate}
    \item
 \textit{Base case:} proves that the property holds for the number $0$.
\item \textit{Induction step:} proves that, if the property holds for one natural number $n$, then it holds for the next natural number $n+1$.
\end{enumerate}

For $T(n)=2\times T(n-1) +1, T_0 = 0$, we can have the following result by expanding $T(i), i \in [0, 7]$.
\begin{lstlisting}[numbers=none]
n    0 1 2 3 4 5 6 7
T_n  0 3 7 15 31 63 127
\end{lstlisting}
It is not hard that we find the rule and guess $T(n) = 2^n-1$. Now, we prove this equation by induction:
\begin{enumerate}
    \item Show that the basis is true: $T(0) = 2^0 -1 = 0$.
    \item Assume it holds true for $T(n-1)$. By induction, we get
    \begin{align}
        T(n)&=2T(n-1) + 1 \\
        &=2 (2^{n-1} - 1) + 1 \\
        &= 2^n -1
    \end{align}
    Now we show that the induction step holds true too.
\end{enumerate}

\begin{bclogo}[couleur = blue!30, arrondi=0.1,logo=\bccrayon,ombre=true]{Solve $T(n)=T(n/2)+O(1)$ and $T(2n)\leq2T(n)+2n-1, T(2)=1$.}
\end{bclogo}

\paragraph{Briefying on Other Methods}
When the form of the linear recurrence is more complex, say large degree of $k$, more complex of the $f(n)$, none of the iterative and induction methods is practical and managable. For iterative method, the expansion will be way too messy for us to handle. On the side of induction method, it is quite challenging or sometimes impossible for us just to ``guess'' or ``generalize'' the exact closed-form of recurrence relation solution purely based on observing a range of expansion.

The more general and approachable method  for solving homogeneous linear recurrence relation derives from making a rough guess rather than exact guess, and then solve it via \textit{characteristic equation}. This general method is pinpointed in Section.~\ref{subsec_homogeneous_linear_recurrence} with examples. For non-homogeneous linear recurrence relation (Section.~\ref{subsec_non_homogeneous}), there are generally two ways -- \textit{symbolic differentiation} and \textit{method of undetermined coefficients} to solve non-homogeneous linear recurrence relation and both of them relates to solving homogeneous linear relation. The study of the remaining content is most math saturated in the book, while we later on will find out its tremendous help in complexity analysis in Chapter.~\ref{chapter_complexity_analysis} and potentially in problem solving.

% \paragraph{Examples} Maybe

% \subsection{Solving Linear Recursion}
\section{Solve Homogeneous Linear Recurrence Relation}

\label{subsec_homogeneous_linear_recurrence}
In this section, we offer a more general and more managable method for solving recurrence relation that is homogeneous defined in Eq.~\ref{eq_homogeneous_recurrence_relation}. There are three broad methods: using characteristic equation which we will learn  in this section, and the other two-- {linear algebra, and Z-transofrm}~\footnote{Visit \url{https://en.wikipedia.org/wiki/Recurrence_relation} for details.} will not be included.
\paragraph{Make a General ``Guess''} From our previous examples, we can figure out the closed-form solution for simplied homogeneous linear recurrence such as the fibonacci recurrence relation:
\begin{equation}
a_n = a_{n-1}+a_{n-2}, a_0=0, a_1=1
\label{homogeneous_linear_recurrence_guess}
\end{equation}
A reasonable guess would be that $a_n$ is doubled every time; namely, it is approximately $2^n$. Let's guess $a_n=c2^n$ for some constant $c$. Now we substitute Eq.~\ref{homogeneous_linear_recurrence_guess}, we get
\begin{equation}
c2^n = c2^{n-1} + c2^{n-2} = c2^n
\label{homogeneous_linear_recurrence_guess}
\end{equation}
We can see that $c$ will be canceled and the left side is always greater than the right side. Thus we learned that $c2^n$ is a too large guess, and the multiplicative constant $c$ plays no role in the induction step.

Based on the above example, we introduce a parameter $\gamma$ as a base, $a_n = \gamma ^{n}$ for some $\gamma$. We then compute its value through solving \textit{Characteristic Equation} as introduced below.
\paragraph{Characteristic Equation}
Now, we substitute our guess into the Eq.\ref{eq_homogeneous_recurrence_relation}, then
\begin{align}
    \gamma^n & = a_n \\
    &= c_1 \gamma^{n-1} + c_2 \gamma^{n-2} + ... +  c_k \gamma^{n-k}.
    \label{eq_characteristic_equation_1}
\end{align}
We rewrite Eq.~\ref{eq_characteristic_equation_1} as:
\begin{align}
    \gamma^n  - c_1 \gamma^{n-1} - c_2 \gamma^{n-2} - ... -  c_k \gamma^{n-k} = 0.
    \label{eq_characteristic_equation_2}
\end{align}
By dividing $\gamma^{n-k}$ from left and right side of the equation, we get the simplified equation, which is called the \textit{characteristic equation} of the recurrence relation in the form of Eq.~\ref{eq_homogeneous_recurrence_relation}.
\begin{align}
    \gamma^k  - c_1 \gamma^{k-1} - c_2 \gamma^{k-2} - ... -  c_k = 0.
    \label{eq_characteristic_equation_3}
\end{align}
The concept of characteristic equation is related to generating function\footnote{}.  The solutons of characteristic equation are called \textit{characteristic roots}.

\paragraph{Characteristic Roots and Solution} Now, we have a linear homogeneous recurrence relation and its characteristic equation,
% \begin{align}
%   a_n&=c_1a_{n-1} + c_2a_{n-2} + ... + c_k a_{n-k}. \\
% 0&= \gamma^k  - c_1 \gamma^{k-1} - c_2 \gamma^{k-2} - ... -  c_k.
% \end{align}
and assume that the equation has $k$ distinct roots, $\gamma_1, \gamma_2, ..., \gamma_k$, then we can build upon these chracteristic roots, the general guess, and some other $k$ constants, $d_1, d_2, ,,, d_k$ of $\{a_n\}$ as:
\begin{align}
    a_n = d_1\gamma_1^n + d_2\gamma_2^n +...+d_k\gamma_k^n
\end{align}
The unknown constants, $d_1, d_2, ,,, d_k$ of $\{a_n\}$ can be found using the initial values $a_0, a_1, ..., a_{k-1}$ by solving the following equations:
\begin{align}
    a_0 &= d_1\gamma_1^0 + d_2\gamma_2^0 +...+d_k\gamma_k^0,\\
    a_1 &= d_1\gamma_1^1 + d_2\gamma_2^1 +...+d_k\gamma_k^1, \\
    &...,\\
    a_{k-1} &= d_1\gamma_1^{k-1} + d_2\gamma_2^{k-1} +...+d_k\gamma_k^{k-1}.
\end{align}
Within the context of computer science, the degree is mostly within 2. Here, we introduce the formula solving the character roots for  characteristic equation with the following form:
\begin{equation}
   0 = ax^2+bx+c
\end{equation}
% The root(s) of the function is the value(s) of $x$ which makes $f(x)=0$.
The root(s) can be computed from the following formula~\footnote{Visit {http://www.biology.arizona.edu/biomath/tutorials/Quadratic/Roots.html} for derivation} :
\begin{equation}
    x = \frac{-b \pm \sqrt{b^2-4ac}}{2a}
\end{equation}
\paragraph{Hands-on Example}
For $a_n = 2a_{n-1} + 3a_{n-2}, a_0=3, a_1=5$, we can write the characteristic equation as $\gamma^2-2\gamma-3=0$. Because $\gamma^2-2\gamma-3 = (\gamma-3)+(\gamma+1)$, which make the characteristic roots $\gamma_1=3, \gamma_2=-1$. Now our solution has the form:
\begin{align}
    a_n = d_13^n+d_2{(-1)}^{n}
\end{align}
Now, we find the constants via listing the initial values we know:
\begin{align}
    a_0 &= d_13^0+d_2{(-1)}^{0} = d_1+d_2=3, \\
 a_1 &= d_13^1+d_2{(-1)}^{1} = 3d_1-d_2=5.
\end{align}
We would get $d_1=2, d_2=1$. Finally, we have a solution $a_n = 2*3^n+(-1)^n$.
% \paragraph{Linear Algebra}
% \paragraph{Z-transform}
\begin{bclogo}[couleur = blue!30, arrondi=0.1,logo=\bccrayon,ombre=true]{Continue to solve $a_n=a_{n-1}+a_{n-2}$.}
\end{bclogo}

\section{Solve Non-homogeneous Linear Recurrence Relation}
\label{subsec_non_homogeneous}
 \textit{method of undetermined coefficients} where the solution is comprised of the solution of the homogeneous part and the particular $f(n)$ part by summing up; and the method of \textit{symbolic differentiation} which converts from the equation the same form of homogeneous linear recurrence relation.

The complexity analysis for most algorithms fall into the form of non-homogeneous linear recurrence relation. For examples: in fibonacci sequence, if it is be solved by using recursion shown in Chapter.~\ref{chapter_dynamic-programming} without caching mechanism, the time recurrence relation is $T(n)=T(n-1)+T(n-2)+1$; in the merge sort discussed in Chapter.~\ref{chapter_divide_conquer}, the recurrence relation is $T(n)=T(n/2)+n$. Examples of recurrence relation $T(n)=T(n-1)+n$ can be easily found, such as the maximum subarray.
% \subsection{Solving None-linear Recursion}

\paragraph{Method of Undetermined Coefficients} Suppose we have a recurrence relation in the form of Eq.~\ref{eq_non_homogeneous_recurrence_relation}.

Suppose we ignore the non-linear part and just look at the homogeneous part:
\begin{equation}
    h_n=c_1h_{n-1} + c_2h_{n-2} + ... + c_k h_{n-k}.
    \label{eq_non_homogeneous_recurrence_relation_2}
\end{equation}

\paragraph{Symbolic Differentiation}

% \section{Hands-on Examples}
% \label{sec_iter_recur_examples}
\section{Useful Math Formulas}
Knowing these facts can be very important in practice, we can treat each as an element in the problem solving. Sometimes, when its hard to get the closed form of a recurrence relation or finding the recurrence relation, we decompse it to multiple parts with these elements. Put some examples.
\paragraph{binomial theorem}
\begin{align}
    \sum_{k=0}^{n}C_{n}^{k}x^k = (1+x)^n
\end{align}
An example of using this the cost of generating a powerset, where $x=1$.
\section{Exercises}
\begin{enumerate}
    \item Compute factorial sequence using \texttt{while} loop.
    \item Greatest common divisor: The Euclidean algorithm, which computes the greatest common divisor of two integers, can be written recursively.
    \begin{equation}
            gcd(x, y)=
        \begin{cases}
  x & \text{if $y=0$,}\\
  gcd(y, x\% y) & \text{if $y>0$}
\end{cases}
    \end{equation}

Function definition:
\end{enumerate}

\section{Summary}
If a cursive algorithm can be further optimized, the optimization method can either be divide and conquer or decrease and conquer. We have put much effort into solving recurrence relation of both: the linear recurrence relation for decrease and conquer, the divide and conquer recurrence relation for divide and conquer.  Right now, do not struggle and eager to know what is divide or decrease and conquer, it will be explained in the next two chapters.

Further, Akra-Bazzi Method~\footnote{} applies to recurrence such that $T(n)=T(n/3)+T(2n/3)+O(n)$. Please look into more details if interested. Generating function is used to solve the linear recurrence.
\end{document}