\section{Introduction to train a Neural Network} %Jianhong / b-j8497 / Jul 10
\subsection{PyTorch Basics} % Author: Jonathan Siegel
In this section, we will introduce PyTorch \footnote{\url{https://pytorch.org}} which is an open source machine learning framework that accelerates the path from research prototyping to production deployment.

The basic class underlying pytorch is the tensor class. This class simply represents an array of numeric values (usually floats or doubles) and behaves very similar to the corresponding class in numpy.
\begin{python}
import torch
import numpy as np

# We can initialize a pytorch tensor from a python list.
x = torch.tensor([[1,2],[3,4]])
print('x is', x)

# Or from a numpy array.
y = torch.tensor(np.array([[1,2],[3,4]]))
print('y is', y)

# We can also initialize a zeroed tensor.
a = torch.zeros([2,2])
b = torch.zeros([3,3], dtype=torch.int32)

# Note that the default type of such a tensor is a 32-bit float.
print('Type of a:', a.dtype)
print('Type of b:', b.dtype)
\end{python}
The output will be
\begin{python}
x is tensor([[1, 2],
        [3, 4]])
y is tensor([[1, 2],
        [3, 4]])
Type of a: torch.float32
Type of b: torch.int32
\end{python}
Arithmetic operations on PyTorch tensors work as one would expect.
\begin{python}
z = 2*x
print(z)

z = y - z
print(z)

z = torch.matmul(x,y)
print(z)

# etc. see the documentation for more details.
\end{python}
It gives
\begin{python}
tensor([[2, 4],
        [6, 8]])
tensor([[-1, -2],
        [-3, -4]])
tensor([[ 7, 10],
        [15, 22]])
\end{python}

At this point, there isn't much difference between Pytorch and Numpy. What distinguishes Pytorch and makes it useful for Deep Learning is the automatic differentiation supported by Pytorch (essentially, Pytorch can automatically apply the chain rule to calculate derivatives). We illustrate this below with a simple example.
\begin{python}
# In this example we calculate the derivative of (x.y)^2 w.r.t x.

# In order to calculate the gradient with respect to a tensor,
# we must set the requires_grad flag to True.
x = torch.tensor([1.0, 0.0], requires_grad=True)
y = torch.tensor([1.0, 1.0])

# By default, requires_grad is set to False if possible.
print('y.requires_grad:', y.requires_grad)

z = torch.dot(x, y)

# Because z depends on x, which requires a gradient, z.requires_grad
# is automatically set to true. This is done because the gradient of z
# is needed to calculate the gradient of x.
print('z.requires_grad:', z.requires_grad)

out = z * z

# Calculate the derivative of out w.r.t. x. Automatically applies
# the chain rule as needed.
grad = torch.autograd.grad(outputs=out, inputs=x)
print(grad)
\end{python}
The output will be
\begin{python}
y.requires_grad: False
z.requires_grad: True
(tensor([2., 2.]),)
\end{python}

Using what we have so far, we can automatically differentiate polynomials (since they are just compositions of elementary arithmetic operations). But what about more complicated differentiable functions? We can deal with those by subclassing torch.autograd.Function as we show in the next example.
\begin{python}
# We illustrate this by implementing the pointwise ReLU function
# (which is of course already implemented in Pytorch, but this
# process can be used to define more complex custom functions as well).

# We subclass torch.autograd.Function to define new functions
class MyReLU(torch.autograd.Function):

  # The forward method applies the function to the input argument.
  # The ctx argument can be used to cache information for the subsequent
  # gradient computation. You can cache an object using using the
  # ctx.save_for_backward method.
  @staticmethod
  def forward(ctx, input):
    # The input is needed later to calculate the gradient.
    ctx.save_for_backward(input)
    return input.clamp(min=0)

  # The backward method calculates the gradient of the input, given that
  # the gradient of the output is the output_grad argument.
  def backward(ctx, output_grad):
    # Recover the original input from ctx.
    input, = ctx.saved_tensors
    grad_input = output_grad.clone()
    # Zero out the output gradient where the input is negative to obtain
    # the input gradient.
    grad_input[input < 0] = 0
    return grad_input

# Apply the new function to a tensor.
print(MyReLU.apply(x - 0.5))

# Calculating derivatives utilizes the backward method.
z = torch.mul(MyReLU.apply(x - 0.5), y)
# In this case z is not a scalar, so we must provide the derivatives
# of the outputs. What is actually computed is the gradient of
# z dot output_grad with respect to x.
output_grad = torch.ones([2])
print(torch.autograd.grad(outputs=z, inputs=x, grad_outputs = output_grad))
\end{python}
\begin{python}
tensor([0.5000, 0.0000], grad_fn=<MyReLUBackward>)
(tensor([1., 0.]),)
\end{python}
We have seen how to define a custom function. Luckily, we won't have to do this much work very often. As we will see, many common functions in machine learning have already been implemented for us.

Next we introduce the Variable class in torch.autograd. This class used to be necessary for autograd, but since regular tensors now allow for automatic differentiation, it is basically just a wrapper around the tensor class which introduces a few additional methods, most notably the .backward() method.
\begin{python}
from torch.autograd import Variable

# Variables can be copy constructed from tensors. When constructed, we can
# also specify whether the gradient is required or not.
x = Variable(x, requires_grad=True)
y = Variable(y, requires_grad=False)

# All functions which can be applied to tensors can also be applied to
# Variables. In the code below, z will be a Variable since x and y are.
# In general, if at least one input is a Variable, then the output will be as
# well.
z = torch.mul(MyReLU.apply(x - 0.5), y)

# Notably, we can now call .backward() from z_variable, which will compute
# the gradient with respect to z for all tensors it depends on (if their
# requires_grad field is True). Note that since z isn't a scalar, we still
# need to pass the gradients of each of its components.
z.backward(gradient=output_grad)

# The resulting gradients are stored in the .grad field of the corresponding
# tensors. The gradients are only calculated if requires_grad = True.
print('x gradient:', x.grad)
print('y gradient:', y.grad)
\end{python}
\begin{python}
x gradient: tensor([1., 0.])
y gradient: None
\end{python}
Since Variables contain all of the functionality as tensors, but contain additional methods (especially the very useful .backward() method) as well, we recommend using Variables for all non-constant quantities in your code. We illustrate some of the additional behaviors of the .backward() function below.
\begin{python}
# A very important note is that gradients are accumulated, i.e. added to
# whatever is already stored in the .grad field.
z = torch.mul(MyReLU.apply(x - 0.5), y)
z.backward(gradient=output_grad)
print('x.grad has now been doubled:', x.grad)

# In order to avoid this, we must first clear the gradient of x. Note also
# that in the above example we need to recompute z in terms of x and y
# in order to recompute the gradient. We can avoid this by setting
# retain_graph = True
x.grad.data.zero_()
z = torch.mul(MyReLU.apply(x - 0.5), y)
z.backward(gradient=output_grad, retain_graph=True)
print('x gradient:', x.grad)

# Since we set retain_graph=True during the last backward call, we can
# recompute the gradients without recalculating z. This feature is not
# commonly used and considered bad practice, though.
x.grad.data.zero_()
z.backward(gradient=output_grad)
print('x gradient:', x.grad)
\end{python}
\begin{python}
x.grad has now been doubled: tensor([2., 0.])
x gradient: tensor([1., 0.])
x gradient: tensor([1., 0.])
\end{python}

\subsection{Building Neural Networks} % Author: Jonathan Siegel
Building and training neural networks directly using Pytorch Variables and automatic differentiation would be very cumbersome. Luckily, Pytorch provides built-in functionality which makes it much easier to build and train neural networks. In this section, we will describe how to use this functionality to build (i.e. define) a network.

The most important library for building neural networks is the torch.nn library. This library allows us to build neural networks by concatenating different types of layers.
\begin{python}
import torch.nn as nn

# Here we define a sample neural network. The class defining our
# network should inherit from nn.Module.
class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    # Here we define the layers of the network.
    # The nn.Sequential method constructs a model by concatenating
    # the layers which are input to it.
    self.apply = nn.Sequential( # Sequentially apply a
        nn.Linear(10,20),       # Linear function R^10 -> R^20
        nn.ReLU(),              # Pointwise ReLU
        nn.Linear(20,20),       # Linear function R^20 -> R^20
        nn.ReLU(),              # Pointwise ReLU
        nn.Linear(20,10))       # Linear function R^20 -> R^10

  # The following method must be overloaded. It specifies how to
  # evaluate the model given an input x.
  def forward(self, x):
    return self.apply(x) # In our instance we simply pass x through
                         # the previously defined layers. This could
                         # potentially contains something more complex.
\end{python}
Implementing a Logistic Regression
We will now consider the problem of implementing a logistic regression (trained using stochastic gradient descent) using Pytorch. We will test this model on the MNIST handwritten data imageset.
\begin{python}
import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np

# Transforms images from [0,255] to [0,1] range.
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize(mean=[0], std=[1])])

# Load the set of training images.
trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)

# Load the set of test images.
testset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)

# Let's plot some of the images to see what we're dealing with.
def plot_images(imgs):
  for i in range(imgs.size()[0]):
    npimg = imgs.numpy()[i,0,:,:]
    plt.imshow(npimg, cmap='gray')
    plt.ion()
    plt.show()
    plt.pause(.05)

data = iter(testloader)
images, labels = data.next()
print(labels)
plot_images(images)
\end{python}
Now that we've loaded our dataset, let's build the logistic regression model. In the language of deep learning, the logistic regression is the same as a neural network with no hidden layers trained with a cross entropy loss (exercise: work this out yourself!).
\begin{python}
# This library contains a lot of useful classes for constructing
# neural networks and other machine learning models.
import torch.nn as nn

# This model simply takes an input of dimension 784 and multiplies it
# by a 784x10 matrix of parameters.
class LogisticRegression(nn.Module):
  def __init__(self):
    super(LogisticRegression, self).__init__()
    self.dense_linear = nn.Sequential(
      nn.Linear(28 * 28, 10))

  def forward(self, x):
    x = x.view(-1,28*28)
    x = self.dense_linear(x)
    return x

# Construct an instance of the model.
model = LogisticRegression()
\end{python}

Now we need to fit the parameters of this logistic regression model (the 784x10 matrix). We do this by defining a loss function (the cross entropy) and using the stochastic gradient descent optimization algorithm. Luckily, both of these have already been implemented for you in Pytorch!
\begin{python}
# This library contains implementations of a number of useful optimization algorithms.
import torch.optim as optim

# The cross entropy loss is already implemented in Pytorch.
criterion = nn.CrossEntropyLoss()

# The stochastic gradient descent algorithm with a step size of 0.1.
optimizer = optim.SGD(model.parameters(), lr=0.1)

# Write a loop to train the model using the given optimizer and loss functions.
for i in range(20):
  for data in trainloader:
    # extract the images and labels.
    inputs, labels = data

    # This must be called to zero out the accumulated gradients.
    optimizer.zero_grad()

    # Calculate the predictions made based on the model.
    outputs = model(inputs)

    # Evaluate the loss function based on the model predictions.
    loss = criterion(outputs, labels)

    # Calculate the gradient of the parameters with respect to the loss.
    loss.backward()

    # Take a optimization step.
    optimizer.step()

  print('Completed epoch %d' % i)
print('Completed training')
\end{python}
Now that we've trained our model, we will test its accuracy on the test dataset.
\begin{python}
# Calculate the total number of test samples and the number of correctly
# classified test samples
correct = 0
total = 0
for data in testloader:
  images, labels = data
  outputs = model(images)
  # Take the most likely label as the predicted label.
  _, predicted = torch.max(outputs.data, 1)
  total += labels.size(0)
  correct += (predicted == labels).sum()

print('Out of %d samples, the model correctly classified %d' % (total, correct))
\end{python}

\subsection{Datasets}
In this section, we will talk about how to train a neural network. According to the course in the morning, there are a lot of pictures in a dataset, so, at the beginning, we will introduce some famous and popular datasets.
\begin{itemize}
\item {\bf MNIST}\footnote{\url{http://yann.lecun.com/exdb/mnist/}} contains 60,000 pictures of the ten handwritten digits from 0 to 9. The size of each picture is $28\times 28$ pixels. There are 60,000 pictures for training and 10,000 for testing.
\begin{figure}
\centering
\includegraphics[scale=0.5]{./figures/497Proj_mnist}
\caption{MNIST}
\end{figure}
\item {\bf CIFAR10}\footnote{\url{https://www.cs.toronto.edu/~kriz/cifar.html}} is a datasets consists of $60,000$ color images in $10$ classes, with $6,000$ images per class. There are $50,000$ training data and $10,000$ testing data. Since there are colored pictures, there are three channels, red, green and blue, of the input data. The size is $32\times 32$ pixels for each. It's a subclass of CIFAR100.
\begin{figure}
\centering
\includegraphics[scale=0.5]{./figures/497Proj_cifar10}
\caption{CIFAR10}
\end{figure}
\item {\bf ImageNet}\footnote{\url{http://www.image-net.org}} is a much larger dataset containing 1.2 millions pictures. The size is $224\times 224$ pixels. The size is much bigger and the quantity is also much larger. So you may get a good performance with a simple neural network on MNIST or CIFAR10, but it's much hard to get good accuracy on ImageNet. %So that why we need   some optimization algorithms on such large datasets.
\begin{figure}[H]
\centering
\includegraphics[scale=0.4]{./figures/497Proj_imagenet}
\caption{ImageNet}
\end{figure}
\end{itemize}
These are three common used datasets that we will use to train our neural networks.

\subsection{Train an image classifier}
Now we move to a brief PyTorch \footnote{see a quick tutorial \url{https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html}} tutorial. To train your first image classifier, these are basically five steps in order:
\begin{enumerate}
\item Load and normalizing the CIFAR10 training and test datasets using torchvision (Here we use CIFAR10 as an example).
\item Define a Convolutional Neural Network. (You can also use a Deep Neural Networks or Recurrent Neural Networks whichever you like.)
\item Define a loss function. (As we just defined before as the function $f$ that is the sum of the total losses.)
\item Train the network on the training data. (Given the loss function and training data, we can do the steepest descent, stochastic gradient descent or some other optimization algorithms to decrease the loss function on the training set.)
\item Test the network on the test data. (After you have got a good model on your neural network from the last step, then you test your neural network on the test set to see how well it performs. So the final criterion that judges your neural network is the test accuracy because the test set doesn't involve the training process. So it is very important to get a good test accuracy.)
\end{enumerate}
These are all the steps you need to train a neural network. Now let start from the first step.

\subsubsection{Load a dataset}
To load the dataset, we need the libraries, like torch and some others. So we need to import the libraries first.
\begin{python}
import torch
import torchvision
import torchvision.transforms as transforms
\end{python}
This is the first step for all the other codes. And then here is some preprocess of the dataset. Because pictures can be various, you need to preprocess and normalize the pictures to make them have better properties, like the zero mean and small variance, for better interpretation.
\begin{python}
transform = transforms.Compose(
    [transforms.ToTensor(), # Transform the picture to the torch tensor
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]) # and then normalize it
\end{python}
Then we define a training set using the \emph{`CIFAR10'} function since CIFAR10 is a default dataset in PyTorch.  The \emph{`root'} is the folder to which we want to store the data. Since we are using the training set, set \emph{`train'} is true. And because we haven't download it before, set \emph{`download'} is true. \emph{`transform=transform'} means we want to transform these pictures following the \emph{`transform'} function that we defined in the preprocess part.
\begin{python}
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
\end{python}
Now we have already defined our training set. But there are still some problems. The dataset is too large to load them all to compute the gradient on the full batch. That is why we choose the stochastic gradient descent. So we choose a small batch (mini-batch) to calculate the corresponding gradient. Here we set the batch size to be 4. So we can compute the loss and the gradient very conveniently on each step.
\begin{python}
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)
\end{python}
It is the same for the test set except that the \emph{`train'} and \emph{`shuffle'} are false.
\begin{python}
testset = torchvision.datasets.CIFAR10(root='./data', train=False,download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,shuffle=False, num_workers=2)
\end{python}

Here are just some parts to show the images. If you are interested about what the images look like, you can use the \emph{`matplotlib'} library to plot them. If not, just skip this part.
\begin{python}
import matplotlib.pyplot as plt
import numpy as np

# functions to show an image

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()

# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))
\end{python}

That's basically how you load a dataset. Let's go to the second step.
\subsubsection{Define a Convolution Neural Network}
Since we are dealing with the image classification, we prefer to using the Convolution Neural Network. And we will introduce more details the part later. Now we just go through the codes for the simple example line by line.
The first part is to initialize the class. Then we define how we compute a forward process by given an input as \emph{`x'} in second part.  First, the input will be convoluted with the 2-d convolution operation, then use the ReLU function as the nonlinear activation function, after that we make a pooling. This is what a convolution layer composed, first a convolution operator, then a nonlinear activation, and then a pooling. We define two convolution layers this way.  After that, the function \emph{`x.view'} is used to flat the picture, because the output of the convolution layer is still a two dimensional picture. Here, we use the \emph{`view'} function to flat the two dimensional picture to be one dimensional of length 400. Then we make a fully connect operator and a ReLU function. After that, we have the final classification layer to output the prediction outcome. So that is basically what the CNN composed.
\begin{python}
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):  # Initialize the class
        super(Net, self).__init__()
        # the construction of the neural network
        self.conv1 = nn.Conv2d(3, 6, 5) # a 2-d convolution layer named as `conv1'
        self.pool = nn.MaxPool2d(2, 2) # a max pooling layer named `pool'
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120) # linear layer
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
\end{python}
Different choices of these hyper-parameters may lead to different results. We usually need to tune these hyper-parameters to get a better model.
Since the input of each layer in the neural network is the output of its previous layer, there are two parts of hyper-parameters of an operator, like \emph{`Conv2d'}, \emph{`Linear'}. One part depends on the input, which is the output of the previous layer, of this operator. The other can be arbitrarily choose to get a good model, and this part may decide a part of parameter of the next operator/layer. For example, `Linear(input dimension, output dimension)', the dimension of output of \emph{`fc1'} is 120, then the first parameter of \emph{`fc2'} should be same as it, 120.

Another important hyper-parameters of convolution layers is channel. For \emph{`Conv2d (number of input channels, number of output channels, kernel size)'}, we need keep the number of input channel same as the number of output channel of the previous layer.

To choose parameters (called hyper-parameters in neural network) of the torch functions, we recommend to read the corresponding documents in PyTorch website for more precise details.

Here we just give a simple working example. Later on, we will talk about more details and how to modify them.

\subsubsection{Define a loss function and optimization}
There are actually a lot of possible loss functions. For image classification, the common choice is the cross-entropy. You can also use the square loss or any reasonable functions based on the problems. Actually, the common loss functions have already been defined in the \emph{`torch.nn'} library. The way to use them is like the \emph{`criterion'} in the following code. And the \emph{`optimizer'} is the corresponding optimization algorithm used to decrease the loss function. The common algorithms can also be found in the \emph{`torch.optim'} library. In the example, we choose stochastic gradient descent (SGD). And there are a few optional parameters, like learning rate \emph{`lr'}, momentum. For large learning rate, you may behave very widely, but for small learning rate, you may zig-zaging around very small range. The learning rate is one of the most important hyper-parameter that we need to tune to get a good model.
\begin{python}
import torch.optim as optim

criterion = nn.CrossEntropyLoss() #use the cross-entropy loss defined in torch library
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
\end{python}

Now we have almost defined everything. Then we can start to train our networks.
\subsubsection{Train the network}
For the training, we first define what the epoch means. Because we have like 60,000 pictures in this training set, and we divided it into a few small batches (mini-batch). Whenever you go over all the pictures in the training set, then it means you finish an epoch.
And when we do one optimization step (calculate gradient and update the parameters) in one mini-batch, we say it an iteration.
\begin{python}
for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0): # i is the index and data are pictures in the corresponding mini-batch
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs) # forward propagation to get the output
        loss = criterion(outputs, labels) # compute the loss, see if it satisfy the terminal criterion
        loss.backward() # backward propagation to get the gradient
        optimizer.step() # update the parameters by optimization algorithm (like SGD) to decrease the loss

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training') # After 2 epochs, we finish the training.
\end{python}

Now, after training process, we want to see how well the model behave. So let's start to test the model we got on the test set, which is independent to the training.
\subsubsection{Test the network on the test data}
\begin{python}
correct = 0
total = 0
with torch.no_grad(): # We don't need compute the gradients in the test process since we don't need optimization
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1) # get the predicted class by neural network
        total += labels.size(0)
        correct += (predicted == labels).sum().item() # compare the predicted and the real label

print('Accuracy of the network on the 10000 test images: %d %%' % (100 * correct / total))
\end{python}
Now we finish training a simple neural network here.


\subsection{Ordinals}\label{subsec:ordinals}

\begin{remark}\label{rem:ordinal_definition}
  Ordinals are generalizations of \hyperref[def:set_of_natural_numbers]{natural numbers}. We will find characterizing properties of the natural numbers (defined as members of \hyperref[thm:smallest_inductive_set_existence]{\( \omega \)}), so that it is clear what we want to generalize.

  Every natural number is defined as a set of other natural numbers:
  \begin{align*}
    &0 = \varnothing \\
    &1 = \set{ \varnothing } = \set{ 0 } \\
    &2 = \set{ \varnothing, \set{ \varnothing } } = \set{ 0, 1 } \\
    &3 = \set{ \varnothing, \set{ \varnothing }, \set{ \varnothing, \set{ \varnothing } } } = \set{ 0, 1, 2 }
  \end{align*}

  It just, so happens that each natural number \( n \) is the set of natural numbers that are smaller with respect to the strict order relation \( < \) defined in \eqref{eq:def:natural_number_ordering/strict_predicate}.

  Therefore, \( \in \) and \( < \) are equivalent on the set \( \omega \). It follows from \fullref{thm:natural_numbers_are_well_ordered} that \( \in \) is a \hyperref[def:totally_ordered_set]{strict total order} on \( \omega \). It is even \hyperref[def:well_ordered_set]{well-ordered} by \( \in \), but the latter condition is redundant due to \fullref{thm:set_membership_is_well_founded}.

  For an arbitrary set \( A \), set membership is not even a \hyperref[def:partially_ordered_set/strict]{strict partial order} --- irreflexivity is implied by \fullref{thm:set_membership_is_well_founded}, but transitivity of \( \in \) as a binary relation on \( A \) fails to hold in general, not speaking about trichotomy.

  A very simple counterexample for transitivity of \( \in \) is the set \( \set{ \varnothing, \set{ \varnothing }, \set{ \set{ \varnothing } } } \). Clearly \( \varnothing \in \set{ \varnothing } \) and \( \set{ \varnothing } \in \set{ \set{ \varnothing } } \), but \( \varnothing \not\in \set{ \set{ \varnothing } } \).

  In order for a set \( A \) to be a member of \( \omega \), it is not sufficient for \( \in \) to be a strict total ordered on \( A \). Except for the members of \( \omega \), another set that is totally ordered by \( \in \) is \( A = \set{ 0, 2, 4 } \).

  If we require \( A = \set{ 0, 2, 4 } \) to be a \hyperref[def:transitive_set]{transitive set}, however, it will be a natural number. Indeed, since \( 4 \) is a member of \( A \) and \( 1 \) and \( 3 \) are members of \( 4 \), then by adding \( 1 \) and \( 3 \) to \( A \) we obtain the set \( \set{ 0, 1, 2, 3, 4 } \), which by our definition of natural numbers is \( 5 \).

  note that transitivity of the relation \( \in \) on \( A \) and transitivity of the set \( A \) itself are entirely different concepts, although we will use both. Every member of \( \omega \) is a transitive set by \fullref{thm:omega_is_transitive} and the relation \( \in \) is a strict total order by \fullref{thm:natural_numbers_are_well_ordered}.

  This is the reasoning behind our definition of an ordinal --- \fullref{def:ordinal}. From this definition it will follow that the ordinals are unique representatives of order-isomorphisms classes of well-ordered sets.

  As a final note, the above two conditions are not sufficient for \( A \) to be a member of \( \omega \) (they are too general), but if we additionally require \( A \) to be a \hyperref[def:set_finiteness]{finite set}, then \( A \) will be a member of \( \omega \). We have yet to define finiteness, however.
\end{remark}

\begin{definition}\label{def:ordinal}\mcite[def. 63.12]{OpenLogicFull}
  An \term{ordinal number} or simply \term{ordinal} is a \hyperref[def:transitive_set]{transitive set} \( A \) such that set membership (as a binary relation on \( A \)) \hyperref[def:well_ordered_set]{well-orders} \( A \). By tradition, ordinals are denoted by initial small Greek letters like \( \alpha \) and \( \beta \).

  Because of \fullref{thm:set_membership_is_well_founded}, it is sufficient for set membership to be a \hyperref[def:totally_ordered_set]{strict total order} on \( A \). Since well-foundedness also implies \hyperref[def:binary_relation/irreflexive]{irreflexivity}, it follows that set membership must only be \hyperref[def:binary_relation/transitive]{transitive} and \hyperref[def:binary_relation/trichotomic]{trichotomic} on \( A \).

  In the absence of the \hyperref[def:zfc/foundation]{axiom of foundation}, we additionally require set membership to be a \hyperref[def:well_founded_relation]{well-founded relation} on \( A \), so that \( A \) is well-ordered.

  See \fullref{rem:ordinal_definition} for a further discussion of the definition, especially the different notions of transitivity.

  We introduce the notation \( \alpha < \beta \) for \( \alpha \in \beta \) in analogy with natural numbers. This is not a binary relation since there is no set of all ordinals by \fullref{thm:burali_forti_paradox}, however it does satisfy the properties of a well-order due to \fullref{thm:ordinals_are_well_ordered/trichotomy}.

  Finally, we introduce the following \hyperref[rem:predicate_formula]{predicate formula}
  \begin{equation*}\taglabel[\op{IsOrdinal}]{eq:def:ordinal/predicate}
    \begin{aligned}
      \ref{eq:def:ordinal/predicate}[\tau] \coloneqq
        &\ref{eq:def:transitive_set/predicate}[\tau]
        \wedge \\ \wedge&
        \parens[\Big]
        {
          \qforall {\xi \in \tau}
          \qforall {\eta \in \tau}
          \parens[\Big]
            {
              \eta \in \xi \vee \eta \doteq \xi \vee \xi \in \eta
            }
        }
        \wedge \\ \wedge&
        \parens[\Big]
        {
          \qforall {\xi \in \tau}
          \qforall {\eta \in \tau}
          \qforall {\zeta \in \tau}
          \parens[\Big]
          {
            (\xi \in \eta \wedge \eta \in \zeta) \rightarrow \xi \in \zeta
          }
        }.
    \end{aligned}
  \end{equation*}
\end{definition}

\begin{proposition}\label{thm:omega_is_an_ordinal}
  The \hyperref[thm:smallest_inductive_set_existence]{smallest inductive set} \( \omega \) is an \hyperref[def:ordinal]{ordinal}.
\end{proposition}
\begin{proof}
  From \fullref{thm:omega_is_transitive} it follows that \( \omega \) is a transitive set.

  Also, as discussed in \fullref{rem:ordinal_definition}, from \fullref{thm:natural_numbers_are_well_ordered} it follows that set membership is a strict total order on \( \omega \).

  Therefore, \( \omega \) is an \hyperref[def:ordinal]{ordinal}.
\end{proof}

\begin{proposition}\label{thm:member_of_ordinal_is_ordinal}\mcite[lemma 63.13]{OpenLogicFull}
  Every member of an ordinal is an ordinal.
\end{proposition}
\begin{proof}
  Let \( \alpha \) be an ordinal and let \( \beta \in \alpha \). We will show that \( \beta \) is an ordinal.

  By transitivity of \( \alpha \), we have \( \beta \subseteq \alpha \) and by \fullref{def:partially_ordered_set/submodel}, \( (\beta, \in) \) is a (strictly) totally ordered set as a \hyperref[def:first_order_substructure]{substructure} of \( (\alpha, \in) \).

  It remains to show that \( \beta \) is itself transitive. Let \( x \in \beta \). We have that \( \beta \subseteq \alpha \) since \( \alpha \) is transitive, hence \( x \in \alpha \).

  Fix \( y \in x \). Again from the transitivity of \( \alpha \) it follows that \( y \in \alpha \). Also, \( \in \) is a total order on \( \alpha \) and hence from \( y \in x \) and \( x \in \beta \) it follows that \( y \in \beta \).

  Since \( y \in x \) was chosen arbitrarily, it follows that \( x \subseteq \beta \). Since \( x \) was chosen arbitrarily, it follows that \( \beta \) is transitive.
\end{proof}

\begin{proposition}\label{thm:initial_segment_of_ordinal}
  Let \( \alpha \) be an ordinal. For any \( \beta \in \alpha \), the \hyperref[def:partially_ordered_set_interval/ray]{open initial segment} \( \alpha_\beta \) equals \( \beta \).

  This is the bounded version of \fullref{thm:ordinal_is_set_of_smaller_ordinals}.
\end{proposition}
\begin{proof}
  Let \( \beta \in \alpha \). Consider the initial segment
  \begin{equation*}
    \alpha_{<\beta} = \set{ \gamma \in \alpha \given \gamma \in \beta }.
  \end{equation*}

  Clearly \( \alpha_{<\beta} = \alpha \cap \beta \). Given that \( \alpha \) is a transitive set, however, we have \( \beta \subseteq \alpha \) and thus \( \alpha \cap \beta = \beta \).

  Therefore, \( \alpha_{<\beta} = \beta \).
\end{proof}

\begin{corollary}\label{thm:natural_numbers_are_ordinals}
  The natural numbers (as members of \hyperref[thm:smallest_inductive_set_existence]{\( \omega \)}) are ordinals.
\end{corollary}
\begin{proof}
  Follows from \fullref{thm:omega_is_an_ordinal} and \fullref{thm:member_of_ordinal_is_ordinal}.
\end{proof}

\begin{definition}\label{def:transfinite_sequence}
  For any ordinal \( \alpha \) we call any function with \( \alpha \) as its domain a \( \alpha \)-indexed \term{transfinite sequence}.

  In particular, the case \( \alpha = \omega \) corresponds to the standard \hyperref[def:sequence]{natural number sequences}.
\end{definition}

\begin{theorem}[Bounded transfinite induction]\label{thm:bounded_transfinite_induction}
  For every formula \( \varphi \) in the language of set theory not containing \( \tau \), \( \eta \) nor \( \zeta \) as free variables, the following is a theorem of \logic{ZFC}:
  \small
  \begin{equation*}
    \qforall \tau
    \parens[\Bigg]
    {
      \ref{eq:def:ordinal/predicate}[\tau]
      \rightarrow
      \parens[\Big]
        {
          \qforall {\eta \in \tau}
          \overbrace
            {
              \underbrace{ \parens[\Big]{ \qforall {\zeta \in \eta} \varphi[\xi \mapsto \zeta] } }_{\mathclap{\substack{\T{inductive} \\ \T{hypothesis}}}}
              \rightarrow
              \underbrace{ \varphi[\xi \mapsto \eta] }_{\mathclap{\substack{\T{inductive step} \\ \T{conclusion}}}}
            }^{\T{inductive step}}
        }
      \rightarrow
      \underbrace{ \qforall {\eta \in \tau} \varphi[\xi \mapsto \eta] }_{\T{conclusion}}
    }.
  \end{equation*}
  \normalsize

  See the comments in \fullref{def:peano_arithmetic/PA3} regarding variables and quantification in axiom schemas and \fullref{rem:induction} for a general discussion of induction.

  See \fullref{rem:transfinite_induction} about a reformulation that is often useful in practice.
\end{theorem}
\begin{proof}
  This theorem is a special case of \fullref{thm:epsilon_induction} with the formula \( \xi \in \tau \rightarrow \varphi \) that is explicitly universally quantified by the parameter \( \tau \) which ranges over all ordinals.

  Note that it is unnecessary to verify that \( \eta \) and \( \zeta \) are ordinals because \fullref{thm:member_of_ordinal_is_ordinal} ensures that \( \eta \) is only quantified over ordinals.
\end{proof}

\begin{theorem}[Bounded transfinite recursion]\label{thm:bounded_transfinite_recursion}\mcite[177]{Enderton1977Sets}
  Fix an \hyperref[def:ordinal]{ordinal} \( \alpha \) and a nonempty set \( A \). Suppose that we are given some transformation \( T: \pow(\alpha \times A) \to A \) whose value for any relation between \( \alpha \). Then there exists a unique \( \alpha \)-indexed \hyperref[def:transfinite_sequence]{transfinite sequence} \( f: \alpha \to A \) such that for any \( \beta \in \alpha \) we have \( f(\beta) = T(f\restr_\beta) \).

  This is a vast generalization of \fullref{thm:omega_recursion} from \hyperref[def:sequence]{natural number sequences} to \hyperref[def:transfinite_sequence]{transfinite sequences}.

  See \fullref{rem:transfinite_induction} about a reformulation that is often useful in practice.
\end{theorem}
\begin{proof}
  The proof is analogous to that of \fullref{thm:omega_recursion}, but we will give it anyway to highlight the difference between using \fullref{thm:omega_induction} and \fullref{thm:bounded_transfinite_induction}.

  Let \( G \subseteq \pow(\alpha \times A) \) be the set of all \hyperref[def:partial_function]{partial single-valued functions} \( g: \alpha \to A \) such that
  \begin{itemize}
    \item There exists some \( \beta_g \in \alpha \) such that \( g \) is defined only in the \hyperref[def:partially_ordered_set_interval/ray]{open initial segment} \( \alpha_{< \beta_g} \). That is, \( g \) is defined for all \( \beta \) up to not including \( \beta_g \).

    \item \( g(\beta) = T(g\restr_\beta) \) for all \( \beta < \beta_g \).
  \end{itemize}

  Clearly \( G \) is nonempty because the function \( \set{ (\varnothing, T(\varnothing)) } \) belongs to \( G \).

  Define \( f \coloneqq \bigcup G \). At this point \( f \) is a \hyperref[def:multi_valued_function]{multi-valued function}. We must now show that \( f \) has all the properties that we want.

  \SubProofOf[def:multi_valued_function/total]{totality} First, we will use \fullref{thm:bounded_transfinite_induction} to show that \( f \) is total.

  Fix \( \beta \in \dom f \). Then there exists a function \( g \in G \) defined for all \( \gamma < \beta \).

  \begin{itemize}
    \item If \( g \) is also defined at \( \beta \) also, this directly proves that \( \beta \in \dom f \).
    \item If \( g \) is not defined at \( \beta \), consider
    \begin{equation*}
      \widehat g \coloneqq g \cup \set{ (\beta, T(g\restr_\beta) }.
    \end{equation*}

    The function \( \widehat g \) is again a single-valued partial function and thus it belongs to \( G \), hence \( \beta \in \dom f \).
  \end{itemize}

  Therefore, \fullref{thm:bounded_transfinite_induction} allows us to conclude that \( f: \alpha \multto A \) is a total multi-valued function.

  \SubProofOf[def:function]{single-valuedness} Now that we know that \( f \) is total, we will prove that it is single-valued and thus is a function in the usual sense of the term.

  Fix \( \beta \in \alpha \). Suppose that \( f \) is single-valued for all \( \gamma < \beta \). Since \( f \) is total, there exist at least one partial function \( g \) in \( G \) that is defined at \( \beta \). Let \( g \) and \( h \) both be such (single-valued partial) functions.

  Then
  \begin{equation*}
    g(\beta) = T(g\restr_\beta) = T(f\restr_\beta) = T(h\restr_\beta) = h(\beta),
  \end{equation*}
  hence \( g \) and \( h \) coincide at \( \beta \), which in turn implies that \( f \) is single-valued at \( \beta \).

  Therefore, \fullref{thm:omega_induction} allows us to conclude that \( f \) is a single-valued total function.

  \SubProofOf[def:function]{uniqueness} Now that it is clear that \( f \) satisfies the theorem, we must verify that it is unique.

  Suppose that \( f_1 \) and \( f_2 \) both satisfy the theorem. Fix some \( \beta \in \alpha \) and suppose that for every \( \gamma < \beta \), we have \( f_1(\gamma) = f_2(\gamma) \). Then
  \begin{equation*}
    f_2(\beta) = T(f_1\restr_\beta)) = T(f_2\restr_\beta) = f_2(\beta).
  \end{equation*}

  Therefore, \fullref{thm:omega_induction} allows us to conclude that \( f_1 = f_2 \). So there is at most one function that satisfies the theorem and we have already shown that \( f \) is such a function.
\end{proof}

\begin{theorem}[Transfinite induction]\label{thm:transfinite_induction}\mcite[thm. 63.15]{OpenLogicFull}
  It turns out that \fullref{thm:bounded_transfinite_recursion} is valid for all ordinals simultaneously.

  For every formula \( \varphi \) in the language of set theory not containing \( \eta \) nor \( \zeta \) as free variables, the following is a theorem of \logic{ZFC}:
  \small
  \begin{equation*}
    \qforall \eta
    \parens[\Bigg]
      {
        \ref{eq:def:ordinal/predicate}[\eta]
        \rightarrow
        \parens[\Big]
        {
          \overbrace
            {
              \underbrace{ \parens[\Big] { \qforall {\zeta \in \eta} \varphi[\xi \mapsto \zeta] } }_{\mathclap{\substack{\T{inductive} \\ \T{hypothesis}}}}
              \rightarrow
              \underbrace{ \varphi[\xi \mapsto \eta] }_{\mathclap{\substack{\T{inductive step} \\ \T{conclusion}}}}
            }^{\T{inductive step}}
        }
      }
    \rightarrow
    \qforall \eta \parens[\Bigg]
    {
      \ref{eq:def:ordinal/predicate}[\eta]
      \rightarrow
      \underbrace{ \varphi[\xi \mapsto \eta] }_{\T{conclusion}}
    }.
  \end{equation*}
  \normalsize

  This theorem could be a special case of \fullref{thm:bounded_transfinite_recursion}, but there exists no set of all ordinals due to \fullref{thm:burali_forti_paradox}.

  See the comments in \fullref{def:peano_arithmetic/PA3} regarding variables and quantification in axiom schemas and \fullref{rem:induction} for a general discussion of induction.

  See \fullref{rem:transfinite_induction} about a reformulation that is often useful in practice.
\end{theorem}
\begin{proof}
  The proof is similar to the proof of \fullref{thm:well_founded_induction}.
\end{proof}

\begin{proposition}\label{thm:ordinals_are_well_ordered}
  The ordinals are \hyperref[def:well_ordered_set]{well-ordered}. Since there exists no set of all ordinals due to \fullref{thm:burali_forti_paradox}, we cannot say that the ordinals form a well-ordered set. We will instead state a more concrete result.

  \begin{thmenum}
    \thmitem{thm:ordinals_are_well_ordered/transitivity} For any three ordinals \( \alpha \), \( \beta \) and \( \gamma \) such that \( \alpha < \beta < \gamma \) we have \( \alpha < \gamma \).

    \thmitem{thm:ordinals_are_well_ordered/trichotomy} For any two ordinals \( \alpha \) and \( \beta \), exactly one of \( \alpha = \beta \), \( \alpha < \beta \) or \( \alpha > \beta \) holds.
  \end{thmenum}

  As discussed in \fullref{def:ordinal}, irreflexivity and well-foundedness hold by \fullref{thm:set_membership_is_well_founded}.
\end{proposition}
\begin{proof}
  \SubProofOf{thm:ordinals_are_well_ordered/transitivity} Let \( \alpha \), \( \beta \) and \( \gamma \) be ordinals and let \( \alpha \in \beta \in \gamma \). Since \( \gamma \) is a transitive set, it follows that \( \beta \subseteq \gamma \) and thus \( \alpha \in \gamma \).

  Therefore, we have used the fact that \( \gamma \) is a transitive set to prove that set membership is a transitive relation, thus obtaining a connection between two distinct concepts both named \enquote{transitivity}.

  \SubProofOf{thm:ordinals_are_well_ordered/trichotomy} Let \( \alpha \) and \( \beta \) be ordinals.

  Due to \fullref{thm:simple_foundation_theorems/member_of_itself}, it is not possible for both \( \alpha \in \beta \) and \( \alpha = \beta \) to hold simultaneously. Due to \fullref{thm:simple_foundation_theorems/mutual_membership}, it is not possible for both \( \alpha \in \beta \) and \( \beta \in \alpha \) to hold simultaneously.

  Thus, at most one of \( \alpha = \beta \), \( \alpha \in \beta \) or \( \beta \in \alpha \) holds.

  We will use \fullref{thm:transfinite_induction} on \( \beta \) to show that at least one of them holds. Fix an ordinal \( \beta_0 \). Our inductive hypothesis is that for every ordinal \( \alpha \) and every \( \gamma \in \beta_0 \) at least one of \( \alpha = \gamma \), \( \alpha \in \gamma \) or \( \gamma \in \alpha \) holds.

  Fix some ordinal \( \alpha_0 \). We will show that at least one of \( \alpha_0 = \beta_0 \), \( \alpha_0 \in \beta_0 \) or \( \beta_0 \in \alpha_0 \) holds. Since the case \( \beta_0 = \alpha_0 \) is trivial, we exclude it from consideration.

  \begin{itemize}
    \item If there exists \( \gamma \in \beta_0 \) such that \( \gamma = \alpha_0 \), clearly \( \alpha_0 \in \beta_0 \).
    \item If there exists \( \gamma \in \beta_0 \) such that \( \alpha_0 \in \gamma \), then by transitivity \( \alpha_0 \in \beta_0 \).
    \item If for every \( \gamma \in \beta_0 \) we have \( \gamma \in \alpha_0 \), then \( \beta_0 \subsetneq \alpha_0 \). Let \( \gamma_0 \) be the smallest member of \( \alpha_0 \setminus \beta_0 \). We will show that \( \gamma_0 = \beta_0 \).

    Our first goal is to show that \( \gamma_0 \subseteq \beta_0 \). Aiming at a contradiction, suppose that there exists some \( \lambda \in \gamma_0 \setminus \beta \). Since \( \gamma_0 \) is a transitive set, we have \( \lambda \in \alpha \). Thus, \( \lambda \in \alpha \setminus \beta \) and \( \lambda \in \gamma_0 \), contradicting the minimality of \( \gamma_0 \). Therefore, \( \gamma_0 \subseteq \beta \).

    Now we will use the existing inductive hypothesis for \( \alpha = \gamma_0 \) to show that \( \beta_0 \subseteq \gamma_0 \).

    \begin{itemize}
      \item If there exists \( \lambda \in \beta_0 \) such that \( \lambda = \gamma_0 \), clearly \( \gamma_0 \in \beta_0 \). But that contradicts our choice of \( \gamma_0 \) as a member of \( \alpha_0 \setminus \beta_0 \).

      \item If there exists \( \lambda \in \beta_0 \) such that \( \gamma_0 \in \lambda \), then by transitivity \( \gamma_0 \in \beta_0 \), which again contradicts our choice of \( \gamma_0 \).

      \item Finally, if for every \( \gamma \in \beta_0 \) we have \( \gamma \in \gamma_0 \), then \( \beta_0 \subseteq \gamma_0 \).
    \end{itemize}

    Thus, both \( \gamma_0 \subseteq \beta_0 \) and \( \beta_0 \subseteq \gamma_0 \), giving us the equality \( \beta_0 = \gamma_0 \). This allows us to conclude that \( \beta_0 \in \alpha_0 \).
  \end{itemize}

  We have shown by transfinite induction that for a fixed ordinal \( \beta_0 \), for every other ordinal \( \alpha \) at least one of \( \beta_0 = \alpha \), \( \beta_0 \in \alpha \) or \( \alpha \in \beta_0 \) holds. We have already shown that at most one of the three holds, hence exactly one of the three holds.

  Since \( \beta_0 \) is itself arbitrary, we can conclude that trichotomy holds for any two ordinals \( \alpha \) and \( \beta \).
\end{proof}

\begin{proposition}\label{thm:ordinal_ordering_via_subsets}
  For any two ordinals \( \alpha \) and \( \beta \) we have \( \beta \in \alpha \) if and only if \( \beta \subsetneq \alpha \).
\end{proposition}
\begin{proof}
  \NecessitySubProof Since \( \alpha \) is a transitive set, from \( \beta \in \alpha \) it follows that \( \beta \subseteq \alpha \).

  We cannot have \( \beta = \alpha \) due to \fullref{thm:simple_foundation_theorems/member_of_itself}, hence \( \beta \subsetneq \alpha \).

  \SufficiencySubProof Suppose that \( \beta \subsetneq \alpha \).

  By \fullref{thm:ordinals_are_well_ordered/trichotomy}, the ordinals \( \beta \) and \( \alpha \) must be related by set membership.
  \begin{itemize}
    \item If \( \alpha = \beta \), this directly contradicts our assumption that \( \beta \subsetneq \alpha \).
    \item If \( \alpha \in \beta \), then \( \alpha \in \alpha \) which contradicts \fullref{thm:simple_foundation_theorems/member_of_itself}.
    \item It remains for \( \beta \in \alpha \) to hold.
  \end{itemize}
\end{proof}

\begin{proposition}\label{thm:ordinal_successor_strictly_monotone_on_ordinals}
  The ordinal successor operation is strictly monotone on ordinals. That is, if \( \alpha < \beta \), then \( \op{succ}(\alpha) < \op{succ}(\beta) \).
\end{proposition}
\begin{proof}
  Let \( \alpha \in \beta \) and let \( \gamma \in \op{succ}(\alpha) \).
  \begin{itemize}
    \item If \( \gamma \in \alpha \), clearly \( \gamma \in \beta \) because \( \beta \) is a transitive set.

    \item If \( \gamma = \alpha \), then \( \gamma = \alpha \in \beta \).
  \end{itemize}

  We have shown that \( \op{succ}(\alpha) \subseteq \beta \). Thus, we either have \( \op{succ}(\alpha) = \beta \in \op{succ}(\beta) \) or else by \fullref{thm:ordinal_ordering_via_subsets} we have \( \op{succ}(\alpha) \in \beta \in \op{succ}(\beta) \).
\end{proof}

\begin{proposition}\label{thm:transitive_set_of_transitive_sets}\mcite{MathSE:transitive_set_of_transitive_sets}
  A \hyperref[def:transitive_set]{transitive set} whose members are transitive sets is an ordinal.

  In particular, a transitive set of ordinals is an ordinal. If a set of ordinals is not transitive, we can instead take its \hyperref[def:transitive_closure_of_a_set]{transitive closure}.
\end{proposition}
\begin{proof}
  Let \( A \) be a set whose members are themselves transitive sets.

  We will first show that set membership is transitive on \( A \). If \( x \), \( y \) and \( z \) are members of \( A \) such that \( x \in y \in z \), then since \( z \) is transitive we have \( y \subseteq z \) and thus \( x \in z \).

  Therefore, we can conclude that set membership is a \hyperref[def:partially_ordered_set/strict]{strict partial order} on \( A \). Now define the set
  \begin{equation*}
    B \coloneqq \set{ x \in A \given \qexists {y \in A} x \not\in y \wedge y \not\in x }
  \end{equation*}
  of all members of \( A \) which are not related to at least one other member. If \( B \) is empty, then set membership is trichotomous on \( A \).

  If \( B \) is nonempty, let \( b \) be a minimal element of \( B \). A minimal element must exist by \fullref{thm:infinite_descent_partial_order} because set membership is a well-founded partial order on \( A \). We have chosen \( b \), so that every member of \( b \) is related to every other member of \( A \).

  Define the set
  \begin{equation*}
    C \coloneqq \set{ x \in A \given x \neq b \wedge x \not\in b \wedge b \not\in x }
  \end{equation*}
  of all members of \( A \) which are not related to \( b \) and let \( c \) be a minimal element of \( C \). We will now show that \( b = c \), which is a contradictions with our choice of \( c \).

  Let \( x \in b \). As we have already mentioned, \( x \) is related to every other member of \( A \), including \( c \).
  \begin{itemize}
    \item If \( c = x \), then \( c \in b \), which contradicts our choice of \( c \).

    \item Suppose that \( c \in x \). We have chosen \( x \) to be a member of \( b \) and we thus have \( c \in x \in b \). The set \( b \) is transitive as a member of \( A \), hence \( x \) is also a member of \( A \). Since set membership is a transitive relation on \( A \), it follows that \( c \in b \), which contradicts our choice of \( c \).

    \item It remains for \( x \in c \) to hold.
  \end{itemize}

  Therefore, \( b \subseteq c \). The converse inclusion \( c \subseteq b \) can be obtained analogously by noting that \( c \) is a minimal element of \( C \) and hence every \( x \in c \) is related to \( b \). Thus, we obtain \( b = c \), which contradicts our choice of \( c \) as a member of \( C \).

  The obtained contradiction shows that the set \( B \) is empty and thus every member of \( A \) is related to every other member, proving trichotomy of set membership on \( A \). This allows us to conclude that \( A \) is an ordinal because it is a transitive set and we have already shown in the beginning of the proof that set membership is a transitive relation on \( A \).
\end{proof}

\begin{definition}\label{def:transitive_closure_of_a_set}\mcite[def. 64.8]{OpenLogicFull}
  To show that every set has a rank we must introduce additional definitions. We use \hyperref[rem:natural_number_recursion]{natural number recursion} to define the \term{transitive closure} of a set \( A \) as
  \begin{equation*}
    \cl^T(A) \coloneqq \bigcup \set{ \cl_n^T(A) \given n \in \omega },
  \end{equation*}
  where
  \begin{equation*}
    \cl_n^T(A) \coloneqq \begin{cases}
      A,              &n = 0 \\
      \bigcup \cl_{n-1}^T(A), &n > 0
    \end{cases}
  \end{equation*}

  Note that this is different from the transitive closure of a relation defined in \fullref{def:relation_closures/transitive}.
\end{definition}

\begin{proposition}\label{thm:transitive_closure_of_a_set}\mcite[thm. 64.9]{OpenLogicFull}
  The \hyperref[def:transitive_closure_of_a_set]{transitive closure} \( \cl^T(A) \) of the set \( A \) is the smallest transitive set containing \( A \).
\end{proposition}
\begin{proof}
  It is clear that \( A = \cl_0^T(A) \) is a subset of \( \cl^T(A) \).

  If \( x \in \cl^T(A) \), then there exists some natural number \( n \) for which \( x \in \cl_n^T(A) \). Therefore, \( x \subseteq \bigcup \cl_n^T(A) = \cl_{n+1}^T(A) \) and thus \( x \subseteq \cl^T(A) \).

  Now suppose that \( B \) is a transitive subset of \( \cl^T(A) \) which contains \( A \). Let \( x_0 \in \cl^T(A) \).

  Suppose that \( x_0 \not\in B \). Then there must exist a smallest nonzero number \( n \) such that \( x_0 \in \cl_n^T(A) \). Then \( x_0 \) belongs to some member \( x_1 \) of \( \cl_{n-1}^T(A) \). If \( x_1 \in A \), then \( x_0 \) must belong to \( B \) since it is transitive. But this contradicts out choice of \( x_0 \). Then \( x_1 \not\in A \), in which case there exists some \( x_2 \in \cl_{n-2}^T(A) \) such that \( x_1 \in x_2 \). If \( x_2 \in A \), then again \( x_0 \in B \), which contradicts our choice of \( x_0 \). We can thus recursively construct a sequence \( \set{ x_k }_{k=0}^\infty \) such that for every \( k \geq 0 \) both \( x_k \in x_{k+1} \) and \( x_k \not\in A \) hold. The existence of such a sequence contradicts \fullref{thm:set_membership_is_well_founded}.

  Therefore, \( \cl^T(A) \subseteq B \), hence \( \cl^T(A) \) is the smallest transitive set containing \( A \).
\end{proof}

\begin{proposition}\label{thm:ordinal_is_set_of_smaller_ordinals}
  Every ordinal equals the set of all smaller ordinals.

  This is the unbounded version of \fullref{thm:initial_segment_of_ordinal}.
\end{proposition}
\begin{proof}
  Let \( \alpha \) be an ordinal and let \( A \) be the set of all ordinals smaller than \( \alpha \). We will show that \( A = \alpha \). We will first show that \( A \) is a transitive set. Let \( \beta \in A \) and \( \gamma \in \beta \). Since \( \alpha \) is a transitive set that contains \( \beta \), we have \( \gamma \in \alpha \). Thus, \( \gamma \) is smaller than \( \alpha \) and hence it belongs to \( A \). Therefore, \( A \) is a transitive set of ordinals and by \fullref{thm:transitive_set_of_transitive_sets}, it is itself an ordinal.

  \Fullref{thm:ordinals_are_well_ordered} implies that \( \alpha \) and \( A \) are either equal or related by set membership.
  \begin{itemize}
    \item If \( \alpha \in A \), then \( \alpha \) is smaller than itself, which contradicts \fullref{thm:simple_foundation_theorems/member_of_itself}.

    \item If \( A \in \alpha \), then \( A \) is smaller than itself, which again contradicts \fullref{thm:simple_foundation_theorems/member_of_itself}.

    \item It remains for \( A \) to be equal to \( \alpha \).
  \end{itemize}
\end{proof}

\begin{theorem}[Burali-Forti paradox]\label{thm:burali_forti_paradox}\mcite[thm. 63.19]{OpenLogicFull}
  Assuming \logic{ZFC}, there is no set of all ordinals.
\end{theorem}
\begin{proof}
  Aiming at a contradiction, suppose that \( A \) is a containing all ordinals. If \( \alpha \in A \) and \( \beta \in \alpha \), transitivity \( \beta \) implies \( \beta \in A \). Thus, \( A \) is a transitive set of ordinals, which \fullref{thm:transitive_set_of_transitive_sets} is itself an ordinal. Hence, \( A \in A \).

  But this contradicts \fullref{thm:simple_foundation_theorems/member_of_itself}. Hence, there is no set of all ordinals.
\end{proof}

\begin{proposition}\label{thm:union_of_set_of_ordinals}\mcite{MathSE:union_of_set_of_ordinals}
  Let \( A \) be a set of ordinals and denote \( \alpha \coloneqq \bigcup A \).

  \begin{thmenum}
    \thmitem{thm:union_of_set_of_ordinals/is_ordinal} The union \( \alpha \) is itself an ordinal.

    \thmitem{thm:union_of_set_of_ordinals/supremum} The union \( \alpha \), which is the \hyperref[def:partially_ordered_set_extremal_points/supremum_and_infimum]{supremum} of \( A \) with respect to set inclusion, is also the supremum of \( A \) with respect to ordinal ordering.

    That is, either \( \alpha = A \) or \( \alpha \) is the smallest ordinal that is larger than every member of \( A \).

    \thmitem{thm:union_of_set_of_ordinals/ordinal} If \( A \) is an ordinal, then \( \alpha \leq A \). Furthermore, in the case \( \alpha < A \), there is no ordinal between \( A \) and \( \alpha \). That is, if \( \alpha < A \), then \( A \) is the smallest ordinal strictly larger than \( \alpha \).

    See \fullref{def:successor_and_limit_ordinal/union} for a further distinction between \( \alpha = A \) and \( \alpha < A \).
  \end{thmenum}

  For recursive definitions like \fullref{def:ordinal_arithmetic/addition} this proposition justifies using \( \sup A \) instead of the more confusing \( \bigcup A \).

  Compare this result to \fullref{thm:union_of_set_of_cardinals}.
\end{proposition}
\begin{proof}
  \SubProofOf{thm:union_of_set_of_ordinals/is_ordinal} Let \( A \) be a set of ordinals. Denote its union by \( \alpha \coloneqq \bigcup A \).

  We will show that \( \alpha \) is a transitive set. Due to \fullref{thm:transitive_set_of_transitive_sets}, this is sufficient for \( \alpha \) to be an ordinal.

  Let \( \beta \in \alpha \). Then there exists an ordinal \( \gamma \) in \( A \) such that \( \beta \in \gamma \). Since \( \gamma \) is itself a transitive set, we have \( \beta \subseteq \gamma \). But \( \gamma \subseteq \alpha \), hence \( \beta \subseteq \alpha \).

  Therefore, \( \alpha = \bigcup A \) is a transitive set and thus an ordinal.

  \SubProofOf{thm:union_of_set_of_ordinals/supremum} For every \( \beta \in A \) we have \( \beta \subseteq \alpha \), which by \fullref{thm:ordinal_ordering_via_subsets} corresponds to \( \beta \leq \alpha \) with respect to ordinal ordering.

  We will show that \( \alpha \) is the smallest ordinal with this property. Let \( \lambda_0 \) be any other ordinal such that \( \gamma \leq \lambda_0 \) for any \( \gamma \in A \). Fix some \( \gamma_0 \in \alpha \). Then there exists an ordinal \( \beta_0 \in A \) such that \( \gamma_0 \in \beta_0 \). Since \( \beta_0 \in A \) and \( A \subseteq \lambda_0 \), since \( \lambda_0 \) itself is a transitive set we have \( \gamma_0 \in \lambda_0 \).

  Therefore, \( \alpha \) is the least upper bound of \( A \) with respect to ordinal ordering.

  \SubProofOf{thm:union_of_set_of_ordinals/ordinal} Assume that \( A \) is an ordinal.

  If \( \gamma \in \alpha \), by transitivity of \( \alpha \) we have \( \gamma \subseteq \alpha \). Thus, \( \gamma \in A \). Hence, \( \alpha \subseteq A \).

  Now suppose that there exists some \( \gamma \in A \) such that \( \alpha \in \gamma \). Then
  \begin{equation*}
    \gamma \subseteq \bigcup A = \alpha \in \gamma,
  \end{equation*}
  which contradicts \fullref{thm:simple_foundation_theorems/member_of_itself}.

  Hence, there is not ordinal between \( \alpha = \bigcup A \) and \( A \).
\end{proof}

\begin{proposition}\label{thm:successor_of_ordinal}
  The \hyperref[def:ordinal_successor]{successor} \( \alpha \coloneqq \op{succ}(\beta) \) of an ordinal \( \beta \) is the smallest ordinal larger than \( \alpha \).
\end{proposition}
\begin{proof}
  We must show that \( \alpha \) is a transitive set and thus by \fullref{thm:transitive_set_of_transitive_sets} an ordinal. Note that \( \alpha = \op{succ}(\beta) = \beta \cup \set{ \beta } \).

  Let \( \gamma \in \alpha \).
  \begin{itemize}
    \item If \( \gamma \in \beta \), then since \( \beta \) is a transitive set, we have \( \gamma \subseteq \beta \). Furthermore, since \( \beta \subseteq \alpha \), by transitivity of set inclusion \( \gamma \subseteq \alpha \).

    \item If \( \gamma = \beta \), then \( \gamma \subseteq \alpha \) by definition of successor.
  \end{itemize}

  Therefore, \( \alpha \) is a transitive set and thus an ordinal.

  Now suppose that there is another ordinal \( \gamma \) such that \( \beta \in \gamma \in \alpha = \beta \cup \set{ \beta } \).
  \begin{itemize}
    \item If \( \gamma \in \beta \), this would contradict \fullref{thm:simple_foundation_theorems/mutual_membership}.

    \item If \( \gamma = \beta \), this would contradict \fullref{thm:simple_foundation_theorems/member_of_itself}.
  \end{itemize}

  The obtained contradictions show that there is no ordinal between \( \alpha \) and \( \beta \).
\end{proof}

\begin{remark}\label{rem:def:ordinal_successor}
  It follows from \fullref{thm:ordinal_is_set_of_smaller_ordinals} and \fullref{thm:successor_of_ordinal} and that for any ordinal \( \beta \) we have
  \begin{align*}
    \beta            &= \set{ \gamma \given \gamma \T{is an ordinal and} \gamma < \beta }, \\
    \op{succ}(\beta) &= \set{ \gamma \given \gamma \T{is an ordinal and} \gamma \leq \beta }.
  \end{align*}

  This shows that the \hyperref[def:ordinal_successor]{ordinal successor} operation is very natural in the context of ordinals.
\end{remark}

\begin{definition}\label{def:successor_and_limit_ordinal}
  We say that the ordinal \( \alpha \) is a \term{successor ordinal} if any of the following equivalent conditions hold:

  \begin{thmenum}
    \thmitem{def:successor_and_limit_ordinal/successor} The ordinal \( \alpha \) is the successor of another ordinal. That is, there exists another ordinal \( \beta \) such that \( \alpha = \op{succ}(\beta) \).

    \thmitem{def:successor_and_limit_ordinal/smaller_successor} There exists some \( \beta \in \alpha \) such that \( \op{succ}(\beta) \) does not belong to \( \alpha \).

    \thmitem{def:successor_and_limit_ordinal/union} We have \( \bigcup \alpha \in \alpha \).
  \end{thmenum}

  If \( \alpha \) is neither zero nor a successor ordinal, we call it a \term{limit ordinal}. See \fullref{thm:limit_ordinal_order_topology} for a more involved equivalent condition.

  These notions should not be confused with \hyperref[def:successor_and_limit_cardinal]{successor and weak/strong limit cardinals}.
\end{definition}
\begin{proof}
  \ImplicationSubProof{def:successor_and_limit_ordinal/successor}{def:successor_and_limit_ordinal/smaller_successor} If \( \alpha = \op{succ}(\beta) \), then \( \beta \) satisfies \fullref{def:successor_and_limit_ordinal/smaller_successor}.

  \ImplicationSubProof{def:successor_and_limit_ordinal/smaller_successor}{def:successor_and_limit_ordinal/successor} Let \( \beta \in \alpha \) be such that \( \op{succ}(\beta) \not\in \alpha \). Then by trichotomy we have that either \( \op{succ}(\beta) = \alpha \) or \( \op{succ}(\beta) > \alpha \).

  If \( \op{succ}(\beta) > \alpha \), then either \( \alpha = \beta \), which would contradict \fullref{thm:simple_foundation_theorems/member_of_itself}, or \( \alpha \in \beta \), which would contradict \fullref{thm:simple_foundation_theorems/mutual_membership}.

  Thus, it remains for \( \op{succ}(\beta) \) to be equal to \( \alpha \).

  \ImplicationSubProof{def:successor_and_limit_ordinal/successor}{def:successor_and_limit_ordinal/union} Suppose that \( \alpha = \op{succ}(\beta) \).

  We have
  \begin{balign*}
    \bigcup \alpha
    &=
    \bigcup (\beta \cup \set{ \beta })
    = \\ &=
    \set{ \gamma \given \qexists \delta (\delta \in \beta \T{or} \delta = \beta) \T{and} \gamma \in \delta }
    \reloset {\eqref{eq:def:semilattice/distributive_lattice/finite/meet_over_join}} = \\ &=
    \set{ \gamma \given \qexists \delta (\delta \in \beta \T{and} \gamma \in \delta) \T{or} (\delta = \beta \T{and} \gamma \in \delta }
    = \\ &=
    \set*{ \gamma \given* \parens[\Big]{ \qexists {\delta \in \beta} \gamma \in \delta } \T{or} \gamma \in \beta }
    = \\ &=
    \parens*{ \bigcup \beta } \cup \beta
    \reloset{ \bigcup \beta \subseteq \beta } = \\ &=
    \beta.
  \end{balign*}

  Thus, \( \beta = \bigcup \alpha \in \alpha \).

  \ImplicationSubProof{def:successor_and_limit_ordinal/union}{def:successor_and_limit_ordinal/successor} Let \( \bigcup \alpha \in \alpha \). From \fullref{rem:def:ordinal_successor} we have that
  \begin{equation*}
    \op{succ}\parens*{ \bigcup \alpha }
    =
    \set*{ \gamma \given* \gamma \subseteq \bigcup \alpha }
    =
    \set{ \gamma \given \gamma \subsetneq \alpha }
    =
    \alpha.
  \end{equation*}
\end{proof}

\begin{remark}\label{rem:transfinite_induction}
  It is sometimes efficient to reformulate transfinite induction and recursion. Although analogous principles hold for \fullref{thm:bounded_transfinite_induction} and \fullref{thm:bounded_transfinite_recursion}, we will only demonstrate them for \fullref{thm:transfinite_induction}. The original statement is that in order to prove that some formula is satisfied for all sets, it is sufficient to only prove one inductive step.

  More precisely, let \( \varphi \) be a formula in the language of set theory, let \( \mscrV = (V, I) \) be a \hyperref[rem:standard_model_of_set_theory]{standard} \hyperref[rem:transitive_model_of_set_theory]{transitive} model of set theory and let \( v: \boldop{Var} \to V \) be some variable assignment. This allows us to fix any parameters that would otherwise be present in the induction schema. We will say that the set \( A \) satisfies \( \varphi \) if \( \Bracks{v_{\xi \mapsto A}} = T \).

  \Fullref{thm:transfinite_induction} states that in order to prove that \( \varphi \) holds for any ordinal, the following is sufficient:
  \begin{thmenum}[series=rem:transfinite_induction]
    \thmitem{rem:transfinite_induction/single_inductive_step} For every ordinal \( \alpha \), by assuming that every smaller ordinal satisfies \( \varphi \), we must prove that \( \alpha \) does.
  \end{thmenum}

  We have just defined in \fullref{def:successor_and_limit_ordinal} three mutually exclusive types of ordinals. We can now restate the principles of transfinite induction as follows:
  \begin{thmenum}[resume=rem:transfinite_induction]
    \thmitem{rem:transfinite_induction/base_case} In the \term{base case}, we must prove that \( 0 \) satisfies \( \varphi \).

    \thmitem{rem:transfinite_induction/successor_case} In the \term{successor case}, by assuming that \( \alpha \) satisfies \( \varphi \), we must prove that \( \op{succ}(\alpha) \) does. It is in line with \fullref{thm:transfinite_induction} assume that every ordinal smaller that or equal to \( \alpha \) satisfies \( \varphi \), however it is often enough to do, so only for \( \alpha \) itself.

    \thmitem{rem:transfinite_induction/limit_case} In the \term{limit case}, if \( \lambda \) is a limit ordinal, by assuming that every smaller ordinal satisfies \( \varphi \) we must prove that \( \lambda \) satisfies \( \varphi \).
  \end{thmenum}

  We must note that \fullref{rem:transfinite_induction/limit_case} is essentially the same as the single inductive step \fullref{rem:transfinite_induction/single_inductive_step} except that it is restricted to limit ordinals. The reasoning for this is that the proofs for zero and every successor ordinal can be different from those for limit ordinals.

  See \fullref{thm:ordinal_bijection_and_choice} and \fullref{def:cumulative_hierarchy} for concrete examples.
\end{remark}

\begin{theorem}[Structural recursion]\label{thm:structural_recursion}\mimprovised
  The most general recursion principle we will consider is \term{structural recursion}. It is sometimes also called \term{mutual recursion}. See \fullref{thm:well_founded_induction} for its corresponding induction principle.

  Let \( X \) be a set and let \( T: \pow(X) \to \pow(X) \) be a transformation. Then \( T \) has a \hyperref[def:partially_ordered_set_extremal_points/maximum_and_minimum]{smallest} \hyperref[def:fixed_point]{fixed point}. That is, there exists a unique subset \( A_0 \subseteq X \) such that \( T(A_0) = A_0 \) and no proper subset of \( A_0 \) has this property.

  See \fullref{rem:structural_recursion_and_induction} for how this theorem is used.
\end{theorem}
\begin{proof}
  The result follows by applying \fullref{thm:knaster_tarski_theorem} to the \hyperref[thm:boolean_algebra_of_subsets]{Boolean algebra of all subsets} of \( X \) with \( R(A) \coloneqq A \cup T(A) \) as the monotone operator.
\end{proof}

\begin{remark}\label{rem:structural_recursion_and_induction}\mimprovised
  We will now show the connection between \fullref{thm:well_founded_induction} and \fullref{thm:structural_recursion}.

  Let \( \mscrF \) be a set of functions, where each function \( f \) has a signature \( f: X^{\#f} \to X \) for some nonnegative integer \( \#f \). Define the operator
  \begin{equation*}
    \begin{aligned}
      &T: \pow(X) \to \pow(X) \\
      &T(A) \coloneqq \set*{ x \in X \given[\Big] \qexists{f \in \mscrF} \qexists{t_1, \ldots, t_{\#f} \in A_0} f(t_1, \ldots, t_{\#f}) = x }.
    \end{aligned}
  \end{equation*}

  Now we can use \fullref{thm:structural_recursion} to obtain the smallest fixed point \( A_0 \) of \( T \). The set \( A_0 \) is closed under all the functions in \( \mscrF \), i.e. \( f\bracks*{ A_0^{\#f} } \subseteq A_0 \) for any \( f \in \mscrF \).

  Define a binary relation \( \to \) on \( A_0 \) by declaring that, for every function \( f \) in \( \mscrF \), every sequence \( x_1, \ldots, x_{\#f} \) and every index \( k = 1, \ldots, \#f \), we have
  \begin{equation*}
    x_k \to f(x_1, \ldots, x_{\#f}).
  \end{equation*}

  This relation can be defined by taking unions of smaller relations rather than via recursion.

  If the \hyperref[def:quiver]{quiver} happens to be \hyperref[def:well_founded_relation]{well-founded}, we can use \fullref{thm:well_founded_induction} to prove universal statements about \( A_0 \).

  In \fullref{subsec:first_order_satisfiability}, for example, we use structural recursion to define first-order substitution in \fullref{def:first_order_substitution} and then use structural induction to prove certain semantic equivalences like \fullref{thm:renaming_assignment_compatibility} and \fullref{thm:first_order_substitution_equivalence}.
\end{remark}

\begin{remark}\label{rem:induction}
  \term{Mathematical induction} is a very valuable proof technique for universal statements. The proof of \fullref{thm:nonzero_natural_numbers_have_predecessors} contains remarks regarding its usage and its difference from deduction principles that are formalized via \hyperref[def:deductive_system]{deductive systems}. Although it is a logical tool, this remark belongs to this section because it contains several induction and recursion principles.

  More generally, given a \hyperref[def:first_order_syntax/formula]{first-order formula} \( \varphi \) over some \hyperref[def:first_order_syntax]{first-order language}, certain \hyperref[def:first_order_theory]{logical theories} allow us to prove indirectly \( \qforall \xi \varphi[\xi \mapsto \eta] \) by proving simpler statements (\fullref{def:peano_arithmetic/PA3} contains very important remarks regarding the free variables of \( \varphi \)). This can be done in cases where every \hyperref[def:first_order_semantics/satisfiability]{model} \( X = (X, I) \) of the theory allows us to exhaust its universe \( X \) in a small finite number of steps. We can sometimes use the same steps to instead build objects. The latter principles is called \term{recursion}.

  It should be noted that induction and recursion are used interchangeably in the literature, especially regarding \hyperref[rem:structural_recursion_and_induction]{structural induction}, however we will aim to distinguish between the two.

  Not much more can be said at this level of generality, so we list several induction principles and give examples of their usage:
  \begin{thmenum}
    \thmitem{rem:induction/peano_arithmetic} The most basic induction principles is the (weak) natural number induction. It is best described via the axiom schema \eqref{eq:def:peano_arithmetic/PA3}. \Fullref{thm:nonzero_natural_numbers_have_predecessors} contains detailed commentary regarding its usage and most of the proofs in \fullref{subsec:natural_numbers} are performed inductively.

    It its set-theoretic form \fullref{thm:omega_induction} it is important as a tool for introducing a model of Peano arithmetic. It is used directly for proving \fullref{thm:omega_is_transitive} and \fullref{thm:omega_recursion}. It is essentially the same as \fullref{rem:transfinite_induction} without \fullref{rem:transfinite_induction/limit_case}.

    \Fullref{thm:omega_recursion} is an important standalone tool that allows us to perform recursive definitions for natural numbers. The latter is used, often implicitly, in a great variety of places, from the ability to define natural number operations in \fullref{def:omega_operations} to the definition of magma exponentiation in \fullref{def:magma/exponentiation}. \Fullref{rem:natural_number_recursion} contains notes regarding its practical usage.

    \thmitem{rem:induction/well_founded} A vast generalization of natural number induction is \fullref{thm:well_founded_induction}. It is stated in a very general setting, but is not frequently used. It can be used to prove \fullref{thm:epsilon_induction}, which however is even less frequently used. We do not use neither in practice, however the special case where \( X = \BbbN \) is called strong induction on natural numbers.

    The usual (weak) natural number induction which is performed by proving the statement for \( 0 \) and then proceeding to prove it for \( n + 1 \) by assuming that it holds for \( n \). Strong induction instead has no base cases and is performed by proving a statement for \( n \) by assuming that it holds for all natural numbers strictly smaller than \( n \).

    Well-founded induction and epsilon-induction have no corresponding recursion principle.

    \thmitem{rem:induction/transfinite} Another vast generalization of natural number induction is \fullref{thm:bounded_transfinite_induction}, which is further generalized by \fullref{thm:transfinite_induction}.

    Both principles are used to prove fundamental properties of the ordinals. Outside of set theory, transfinite induction is immensely useful, but it is rarely used directly. Instead, it is usually combined with the \hyperref[def:zfc/choice]{axiom of choice} via \fullref{thm:zorns_lemma}. \Fullref{rem:transfinite_induction} contains notes on how it is used directly.

    Only bounded transfinite induction has a corresponding recursion principle --- \fullref{thm:bounded_transfinite_recursion}. Unbounded transfinite induction cannot define such a principle because that would easily lead to \fullref{thm:burali_forti_paradox}. \Fullref{rem:unbounded_transfinite_recursion} shows how to circumvent this, however.

    Transfinite recursion is used to construct the cumulative hierarchy in \fullref{def:cumulative_hierarchy}.

    \Fullref{rem:cardinal_transfinite_recursion_and_induction} provides alternative transfinite recursion and induction principles for cardinals rather than for ordinals.

    \thmitem{rem:induction/structural} A very general recursion principle is \fullref{thm:structural_recursion}. Inductive proofs can be performed via \fullref{thm:well_founded_induction} sometimes --- this is discussed in \fullref{rem:structural_recursion_and_induction}.

    Within this document, we use \fullref{thm:structural_induction_on_unambiguous_grammars} in certain special cases, for example \hyperref[def:propositional_syntax/formula]{propositional} and \hyperref[def:first_order_syntax/formula]{first-order formulas}.
  \end{thmenum}
\end{remark}

\begin{proposition}\label{thm:ordinal_isomorphism}
  Two ordinals are equal if and only if they are order-isomorphic.
\end{proposition}
\begin{proof}
  \SufficiencySubProof Trivial.

  \NecessitySubProof Let \( \alpha \) and \( \beta \) be two ordinals. The case \( \alpha = \beta \) is clear. Without loss of generality, suppose that \( \beta < \alpha \). \Fullref{thm:ordinal_ordering_via_subsets} implies that \( \beta \subsetneq \alpha \).

  Let \( f: \alpha \to \beta \) be an order isomorphism. Let \( \gamma_0 \) be the smallest value in \( \alpha \setminus \beta \).

  From \fullref{thm:well_ordered_embedding_inflationary} it follows that \( \gamma_0 \leq f(\gamma_0) \). But \( f(\gamma_0) \in \beta \), hence \( \gamma_0 < \beta \). But this contradicts our choice of \( \gamma_0 \) as a member of \( \alpha \setminus \beta \).

  Therefore, \( \gamma_0 \in \beta \). Since \( \gamma_0 \) was arbitrary, we conclude that \( \beta < \alpha \) leads to a contradiction.

  We can obtain an analogous contradiction for \( \alpha < \beta \), hence it only remains possible for \( \alpha \) and \( \beta \) to be equal.
\end{proof}

\begin{theorem}[Hartogs' lemma]\label{thm:hartogs_lemma}
  For every set \( A \) there exists a smallest ordinal \( \alpha \) such that no function from \( \alpha \) to \( A \) is injective.
\end{theorem}
\begin{proof}
  Define the set
  \begin{equation*}
    W \coloneqq \set{ (P, \leq) \given P \subseteq A \T*{and} \leq \T*{well-orders} P }.
  \end{equation*}

  Let \( \beta \) be an ordinal for which there exists an injective function \( f: \beta \to A \). At least one such pair of a ordinal and function exists because \( \varnothing: 0 \to A \) is an injective function.

  The image of \( f \) can be well-ordered by
  \begin{equation*}
    f(\delta) < f(\gamma) \T{if and only if} \delta \in \gamma,
  \end{equation*}
  where \( \delta \) and \( \gamma \) are members of \( \beta \). Thus, if we restrict the range of \( f \) to its image, it would be an explicit order isomorphism of \( (\beta, \in) \) and \( (\img f, <) \).

  We have shown that every ordinal \( \beta \) and every injective function \( f: \beta \to A \) induces a well-ordered set which belongs to \( W \). Furthermore, if \( f_1: \beta_1 \to A \) and \( f_2: \beta_2 \to A \) are two such injective functions and if the induced well-ordered sets \( (\img f_1, \leq_1) \) and \( (\img f_2, \leq_2) \) are order-isomorphic, then \( \beta_1 \) and \( \beta_2 \) are themselves order-isomorphic and thus \( \beta_1 = \beta_2 \) by \fullref{thm:ordinal_isomorphism}.

  Therefore, for any well-ordered set in \( W \) there exists at most one ordinal that induces it via some function. Let \( W' \) be the subset of all well-ordered sets in \( W \) induced by exactly one ordinal.

  We have that \( W' \) is a set and for each member of \( W' \) there corresponds exactly one ordinal. It follows from the \hyperref[def:zfc/replacement]{axiom schema of replacement} that these ordinals form a set. Denote this set by \( B \).

  There is no set of all ordinals by \fullref{thm:burali_forti_paradox}, hence there must exist an ordinal not in \( B \). But every ordinal that has an injective function into \( A \) belongs to \( B \). Hence, there exists some ordinal \( \alpha \) such that no function from \( \alpha \) to \( A \) is injective.

  If \( \alpha \) is not the smallest ordinal with this property, we can now easily take the smallest member of \( \alpha \) with this property.
\end{proof}

\begin{lemma}\label{thm:ordinal_bijection_and_choice}\mcite{MathSE:order_type_existence}
  Let \( A \) be a nonempty set and let \( c \) be a \hyperref[def:choice_function]{choice function} on \( A \). Then there exists an ordinal \( \alpha \) and a bijection between \( A \) and \( \alpha \).
\end{lemma}
\begin{proof}
  We will explicitly build the desired ordinal. We will use \fullref{thm:bounded_transfinite_recursion} in its structured form presented in \fullref{rem:transfinite_induction} to build a \hyperref[def:transfinite_sequence]{transfinite sequence} of injective maps from ordinals into \( A \).

  Let \( \delta \) be the smallest ordinal such that no function from \( \delta \) to \( A \) is injective. Such an ordinal exists by \fullref{thm:hartogs_lemma}. Note that \( \delta \) cannot be zero because the empty function is always injective.

  Any smaller than \( \delta \) ordinal has at least one injective function into \( A \). We will use recursion bounded by \( \delta \) to construct a \hyperref[def:transfinite_sequence]{transfinite sequence} \( \seq{ \iota_\beta }_{\beta < \delta} \) satisfying the invariant that for any \( \beta \in \delta \), the function \( \iota_\beta \) is injective and strictly contains \( \iota_\gamma \) for any \( \gamma < \beta \).

  \begin{itemize}
    \item The zero ordinal has only one possible injective function into \( A \), namely the empty function \( \varnothing: 0 \to P \).

    \item Now let \( \iota_\beta: \beta \to A \) be an injective function.

    If \( \iota_\beta \) is surjective, then it is the desired bijection and the rest of the construction is irrelevant.

    If \( \iota_\beta \) is not surjective, define
    \begin{equation*}
      \begin{aligned}
        &\iota_{\op{succ}(\beta)}: \op{succ}(\beta) \to P \\
        &\iota_{\op{succ}(\beta)}(\gamma) \coloneqq \begin{cases}
          \iota_\beta(\gamma),         &\gamma < \beta \\
          c(A \setminus \img(\iota_\beta)), &\gamma = \beta.
        \end{cases}
      \end{aligned}
    \end{equation*}

    This function is clearly injective.

    \item Let \( \lambda \) be a limit ordinal and let \( \iota_\beta \) be an injective function for any \( \beta < \lambda \). Note that the domain of \( \iota_\beta \) is not necessarily \( \beta \) --- it may be a smaller ordinal \( \gamma \) in case \( \iota_\gamma \) is surjective (and thus the desired bijection).

    In any case, we have a family \( \set{ \iota_\beta }_{\beta < \lambda} \) of functions such that \( \iota_\gamma \subseteq \iota_\beta \) if and only if \( \gamma < \beta \). We simply take their union
    \begin{equation*}
      \iota_\lambda \coloneqq \bigcup \set{ \iota_\beta \given \beta < \lambda }.
    \end{equation*}

    This function is injective because each of the constituent functions it injective.
  \end{itemize}

  We will now thoroughly build the order isomorphism itself.

  Let \( \alpha \leq \delta \) be the (nonstrict) initial segment of \( \delta \) such that \( \iota_\beta \) is fully defined on \( \beta \) for every \( \beta \in \alpha \). More precisely, let
  \begin{equation*}
    \alpha \coloneqq \set{ \beta \in \delta \given \dom(\iota_\beta) = \beta }.
  \end{equation*}

  We will show that \( \alpha \) is a successor ordinal. Note that \( \alpha \) cannot be zero because \( \dom(\iota_0) = \varnothing = 0 \). Aiming at a contradiction, assume that \( \alpha \) is a limit ordinal.

  Since for every \( \beta \in \alpha \) the value \( \iota_{\op{succ}(\beta)}(\beta) \) is defined, we can conclude that the union
  \begin{equation*}
    \bigcup\set{ \iota_\beta \given \beta \in \alpha }
  \end{equation*}
  is an injective function from \( \alpha \) to \( A \).

  If \( \alpha = \delta \), this contradicts our choice of \( \delta \). If \( \alpha \in \delta \), this means that the function \( \iota_\alpha \) is equal to \( \iota_\gamma \) for some \( \gamma < \alpha \). But then \( \iota_{\op{succ}(\gamma)} \) is not defined on the entirely of \( \gamma \). This implies that \( \gamma \geq \alpha \), which contradicts our assumption that \( \gamma < \alpha \).

  Therefore, \( \alpha \) cannot be a limit ordinal. It remains for \( \alpha \) to be a successor ordinal. Then there exists \( \beta \) such that \( \alpha = \op{succ}(\beta) \).

  Suppose that \( \iota_\beta \) is not surjective. We can then construct \( \iota_{\op{succ}(\beta)} \) as in the corresponding recursion step. This will be an injective function from \( \alpha = \op{succ}(\beta) \) to \( A \). But then we would have \( \alpha \in \alpha \), which contradicts \fullref{thm:simple_foundation_theorems/member_of_itself}.

  Therefore, \( \iota_\beta \) is a surjective function.
\end{proof}

\begin{theorem}\label{thm:well_ordered_order_type_existence}
  Any \hyperref[def:well_ordered_set]{well-ordered set} \( (P, \leq) \) is \hyperref[def:partially_ordered_set/homomorphism]{order-isomorphic} to a unique ordinal. This ordinal is called the \term{order type} of \( (P, \leq) \) and is denoted by \( \ord(P, \leq) \) or simply \( \ord(P) \).

  Furthermore, this isomorphism is unique.
\end{theorem}
\begin{proof}
  For \( P = \varnothing \), then the empty function \( \iota: 0 \to P \) is the desired isomorphism.

  We use \fullref{thm:ordinal_bijection_and_choice} on \( P \) with the choice function
  \begin{equation*}
    \begin{aligned}
      &c: \pow(P) \setminus \set{ \varnothing } \to P \\
      &c(B) \coloneqq \min B
    \end{aligned}
  \end{equation*}
  to obtain an ordinal \( \alpha \) and a bijection \( \iota: \alpha \to P \).

  The function \( \iota \) is directly verified to be an order embedding in each of the recursion cases. Therefore, \( \iota \) is a bijective order homomorphism and it follows from \fullref{thm:totally_ordered_strict_isomorphisms} that \( \iota \) is an order isomorphism.

  Uniqueness of \( \iota \) follows from \fullref{thm:well_ordered_isomorphism_is_unique}.
\end{proof}

\begin{theorem}[Zermelo's well-ordering theorem]\label{thm:well_ordering_theorem}\mcite[thm. 69.6]{OpenLogicFull}
  Any \hyperref[def:set]{set} can be \hyperref[def:well_ordered_set]{well-ordered}.

  Within \hyperref[def:zfc]{\logic{ZF}}, this theorem is equivalent to the \hyperref[def:zfc/choice]{axiom of choice} --- see \fullref{thm:axiom_of_choice_equivalences/well_ordering}.
\end{theorem}
\begin{proof}
  \ImplicationSubProof[def:zfc/choice]{the axiom of choice}[thm:well_ordering_theorem]{well-ordering theorem} The empty set is trivially well-ordered.

  Let \( A \) be any nonempty set. By the axiom of choice, there exists a \hyperref[def:choice_function]{choice function} \( c \) for \( A \). We use \fullref{thm:ordinal_bijection_and_choice} on \( A \) and \( c \) to obtain an ordinal \( \alpha \) and a bijection \( \iota: \alpha \to A \). Since \( \alpha \) itself is well-ordered, then the relation
  \begin{equation*}
    x <_A y \T{if and only if} \iota^{-1}(x) <_\alpha \iota^{-1}(y)
  \end{equation*}
  defines a well-order on \( A \).

  \ImplicationSubProof[thm:well_ordering_theorem]{well-ordering theorem}[def:zfc/choice]{axiom of choice} Let \( A \) be any set and suppose that \( < \) well-orders \( A \).

  Define the function
  \begin{equation*}
    \begin{aligned}
      &c: \pow(A) \setminus \set{ \varnothing } \to A \\
      &c(B) \coloneqq \min B.
    \end{aligned}
  \end{equation*}

  In the case where \( A \) is the empty set, \( c \) is the empty function.

  It is clear that \( c(B) \in B \) for every subset \( B \) of \( A \). Therefore, \( c \) is a choice function for \( A \).
\end{proof}


\section{Formulas}
\begin{frame}
\frametitle{Formulas}
\framesubtitle{Use Of Mathematical Formulas}

\begin{exampleblock}{New packages in this section}
\begin{multicols}{2}
\begin{itemize}
\item amsmath
\item amsthm
\item amssymb
\item mathtools
\end{itemize}
\end{multicols}
\end{exampleblock}

\begin{block}{New commands in this section}
\begin{multicols}{2}
\begin{itemize}
\item \color{nounibaredI}\textbackslash sqrt\color{black}\{\}
\item \color{nounibaredI}\textbackslash frac\color{black}\{\}\{\}
\item \color{nounibaredI}\textbackslash int\color{black}\_X
\item \color{nounibaredI}\textbackslash sum\color{black}\_\{\}
\item \color{nounibaredI}\textbackslash lim\color{black}\_\{\}
\item \color{nounibaredI}\textbackslash prod\color{black}
\item \color{nounibaredI}\textbackslash limits\color{black}\_\{\}
\item \color{nounibaredI}\textbackslash dots\color{black}
\item \color{nounibaredI}\textbackslash cdot\color{black}
\item \color{nounibaredI}\_\color{black}
\item \color{nounibaredI}\^~\color{black}
\end{itemize}
\end{multicols}
\end{block}

\end{frame}

%-------------------------------------------------------------------------------
\begin{frame}
\frametitle{Formulas}
\framesubtitle{\ldots ~A Marvel Of Beauty In \LaTeX !}

\begin{columns}
\begin{column}{.3\textwidth}
{\huge $2 \sqrt{\frac{\pi ^2}{3}\cdot c_{2}}$}
\end{column}

\begin{column}{.7\textwidth}
	$\underbrace{
	\color{unibayellowI}\text{\$}
	\color{black}2
	\color{nounibaredI}\backslash \text{sqrt}
	\color{black}\{
	\color{nounibaredI}\backslash \text{frac}
	\color{black}\{
	\color{nounibaredI}\backslash \text{pi}\color{nounibaredI}
	~\hat{}~\color{black}2\}\{3\color{black}\}
	\color{nounibaredI}\backslash
	\color{nounibaredI}\text{cdot}~
	\color{black} \text{c}
	\color{nounibaredI}\_
	\color{black}2\}
	\color{unibayellowI}\text{\$}
}$
\color{black}

The formula-environment begins and ends with \color{unibayellowI}\$ \color{black} .

\medskip
	$\underbrace{
	\color{nounibaredI}\backslash \text{sqrt}
	\color{black}\{
	\color{nounibaredI}\backslash \text{frac}
	\color{black}\{
	\color{nounibaredI}\backslash \text{pi}\color{nounibaredI}
	~\hat{}~\color{black}2\}\{3\color{black}\}
	\color{nounibaredI}\backslash
	\color{nounibaredI}\text{cdot}~
	\color{black} \text{c}
	\color{nounibaredI}\_
	\color{black}2\}
}$
\color{black}

This whole part is the radical.

\bigskip
$\underbrace{
	\color{nounibaredI}\backslash \text{frac}
	\color{black}\{
	\color{nounibaredI}\backslash \text{pi}\color{nounibaredI}
	~\hat{}~\color{black}2\}\{3\color{black}\}
}$

A fraction has always a numerator and a denominator.
\end{column}
\end{columns}
\end{frame}

%-------------------------------------------------------------------------------

\begin{frame}
\frametitle{Formulas}
\framesubtitle{\ldots ~A Marvel Of Beauty In \LaTeX !}
\begin{columns}
	\begin{column}{.4\textwidth}
		\flushright
		$\int_0^\infty$
	\end{column}
	\begin{column}{.6\textwidth}
		\flushleft
		{\ttfamily\color{unibayellowI}\$\color{nounibaredI}\textbackslash\color{nounibaredI}int\_\color{black}0\color{nounibaredI}\textasciicircum \textbackslash infty\color{unibayellowI}\$}
	\end{column}
\end{columns}
\begin{columns}
	\begin{column}{.4\textwidth}
		\flushright
		$\sum_{i=1}^n$
	\end{column}
	\begin{column}{.6\textwidth}
		\flushleft
		{\ttfamily \color{unibayellowI}\$\color{nounibaredI}\textbackslash
			\color{nounibaredI}sum\_\color{black}\{i=1\}\color{nounibaredI}\textasciicircum
			\color{black}n\color{unibayellowI}\$}
	\end{column}
\end{columns}

\begin{columns}
	\begin{column}{.4\textwidth}
		\flushright
		$\lim_{n \rightarrow \infty}$
	\end{column}
	\begin{column}{.6\textwidth}
		\flushleft
		{\ttfamily \color{unibayellowI}\$\color{nounibaredI}\textbackslash
			\color{nounibaredI}lim\_\color{black}\{n \color{nounibaredI}\textbackslash
			\color{nounibaredI}rightarrow \color{nounibaredI}\textbackslash infty\color{black}\}\color{unibayellowI}\$}
	\end{column}
\end{columns}

\begin{columns}
	\begin{column}{.4\textwidth}
		\flushright
		$\prod\limits_{i=1}^{n+1}i = 1 \cdot 2 \cdot \ldots \cdot n \cdot (n+1)$
	\end{column}
	\begin{column}{.6\textwidth}
		\flushleft
		{\ttfamily \color{unibayellowI}\$%
			\color{nounibaredI}\textbackslash\color{nounibaredI}prod\textbackslash  limits\_\color{black}\{i=1\}\color{nounibaredI}\^{}\color{black}\{n+1\} i = 1 \color{nounibaredI}\textbackslash \color{nounibaredI}cdot \color{black}2 \color{nounibaredI}\textbackslash \color{nounibaredI}cdot \color{nounibaredI}\textbackslash \color{nounibaredI}ldots \color{nounibaredI}\textbackslash\color{nounibaredI}cdot \color{black}n \color{nounibaredI}\textbackslash \color{nounibaredI}cdot \color{black}(n+1)\color{unibayellowI}\$}
	\end{column}
\end{columns}
\bigskip
The American Mathematical Society has a wonderful guide for the {\ttfamily amsmath}-package.\footnote{ftp://ftp.ams.org/pub/tex/doc/amsmath/amsldoc.pdf}
\end{frame}