\chapter{Mutable Objects}
\label{mutable}

\index{String class}
\index{type!String}

As you learned in the previous chapter, an object is a collection of data that provides a set of methods.
For example, a \java{String} is a collection of characters that provides methods like \java{charAt} and \java{substring}.

In this chapter, we'll explore two new types of objects: \java{Point} and \java{Rectangle}.
We'll see how to write methods that take objects as parameters and produce objects as return values.

We will also take a first look at the source code for the Java library.

\section{Point Objects}
\label{point}

\index{coordinate}

In math, 2D ``points'' are often written in parentheses with a comma separating the coordinates.
For example, $(0,0)$ indicates the origin, and $(x,y)$ indicates the point $x$ units to the right and $y$ units up from the origin.

\index{AWT}
\index{java.awt}
\index{Point}
\index{class!Point}

% TODO: ML suggest introducing AWT here with a short high-level description

The \java{java.awt} package provides a class named \java{Point} that represents a location in a Cartesian plane.
In order to use the \java{Point} class, you have to import it:

\begin{code}
import java.awt.Point;
\end{code}

\index{new}
\index{operator!new}

Then, to create a new point, you use the \java{new} operator:

\begin{code}
Point blank;
blank = new Point(3, 4);
\end{code}

\index{declaration}
\index{statement!declaration}
\index{reference}

The first line declares that \java{blank} has type \java{Point}.
The second line creates the new \java{Point} with the coordinates $x=3$ and $y=4$.
The result of the \java{new} operator is a {\em reference} to the object.
%So \java{blank} contains a reference to the new \java{Point} object.
Figure~\ref{fig.reference} shows the result.

\index{memory diagram}
\index{diagram!memory}

\begin{figure}[!ht]
\begin{center}
\includegraphics{figs/reference.pdf}
\caption{Memory diagram showing a variable that refers to a \java{Point} object.}
\label{fig.reference}
\end{center}
\end{figure}

As usual, the name of the variable \java{blank} appears outside the box, and its value appears inside the box.
In this case, the value is a reference, which is represented with an arrow.
The arrow points to the \java{Point} object, which contains two variables, \java{x} and \java{y}.

%\section{Attributes}

\index{attribute}
\index{dot notation}

Variables that belong to an object are called {\bf attributes}.
In some documentation, you also see them called ``fields''.
To access an attribute of an object, Java uses {\bf dot notation}.
For example:

\begin{code}
int x = blank.x;
\end{code}

The expression \java{blank.x} means ``go to the object \java{blank} refers to, and get the value of the attribute \java{x}.''
In this case, we assign that value to a local variable named \java{x}.

There is no conflict between the local variable \java{x} and the attribute \java{x}.
The purpose of dot notation is to identify {\em which} variable you are referring to unambiguously.

You can use dot notation as part of an expression.
%slr: 12-17-19
Here is an example:
% height = 130 + 11 * num_lines
\begin{trinket} [230] {Points.java}
import java.awt.Point;
public class Points {

    public static void main(String[] args) {
       Point blank;
       blank = new Point(3, 4);
       System.out.println(blank.x + ", " + blank.y);
       int sum = blank.x * blank.x + blank.y * blank.y;
       System.out.println(sum);
    }
}
\end{trinket}
%\begin{code}
%System.out.println(blank.x + ", " + blank.y);
%int sum = blank.x * blank.x + blank.y * blank.y;
%\end{code}
%
%The first line displays \java{3, 4}.
%The second line calculates the value \java{25}.

Run the trinket and see the result.

\textbf{Section Exercises:}
\begin{enumerate}
\item Modify the trinket \java{Points} to compute and print the distance from the origin to the \java{Point blank}.  (This is the square root of the sum of the squares of the x-coordinate and the y-coordiante.)
\item Modify the trinket \java{Points} to create a second \java{Point} and print out its x and y coordinates.  Experiment to determine which primitive type the x and y coordinates are.  Check your answer by looking up the Java API documentation for \java{java.awt.Point} at \url{https://docs.oracle.com/javase/8/docs/api/}
\end{enumerate}
%slr: end 12-17-19

\section{Objects as Parameters}

\index{parameter}
\index{object!as parameter}

You can pass objects as parameters in the usual way.
For example:

%slr: 12-17-19
% height = 130 + 11 * num_lines
\begin{trinket} [270] {PointParameter.java}
import java.awt.Point;
public class PointParameter {

    public static void printPoint(Point p) {
       System.out.println("(" + p.x + ", " + p.y + ")");
    }

    public static void main(String[] args) {
       Point blank;
       blank = new Point(3, 4);
       printPoint(blank);
    }
}
\end{trinket}
%\begin{code}
%public static void printPoint(Point p) {
%    System.out.println("(" + p.x + ", " + p.y + ")");
%}
%\end{code}

The \java{void} method \java{printPoint} takes a \java{Point} as an argument and displays its attributes in parentheses.
%If you invoke \java{printPoint(blank)}, it displays \java{(3, 4)}.

Run the trinket to see what is produced.
%slr: end 12-17-19

As another example, we can rewrite the \java{distance} method from Section~\ref{distance} so that it takes two \java{Point}s as parameters instead of four \java{double}s.

%slr: 12-17-19
% height = 130 + 11 * num_lines
\begin{trinket} [300] {Dist1.java}
import java.awt.Point;
public class Dist1 {
    //local method to compute distance between two Point objects
    public static double distance(Point p1, Point p2) {
       int dx = p2.x - p1.x;
       int dy = p2.y - p1.y;
       return Math.sqrt(dx * dx + dy * dy);
    }

    public static void main(String[] args) {
       Point p = new Point(3, 4);
       Point q = new Point(6, 8);
       System.out.println("The distance between p and q is " + distance(p,q));
    }
}
\end{trinket}
%\begin{code}
%public static double distance(Point p1, Point p2) {
%    int dx = p2.x - p1.x;
%    int dy = p2.y - p1.y;
%    return Math.sqrt(dx * dx + dy * dy);
%}
%\end{code}

Passing objects as parameters makes the source code more readable and less error-prone because related values are bundled together.

You actually don't need to write a \java{distance} method, because \java{Point} objects already have one. Again, refer to the API documentation at \url{https://docs.oracle.com/javase/8/docs/api/}. To compute the distance between two points, we invoke \java{distance} on one and pass the other as an argument.

%slr: 12-17-19
% height = 130 + 11 * num_lines
\begin{trinket} [240] {Dist.java}
import java.awt.Point;
public class Dist {

    public static void main(String[] args) {
       Point p = new Point(3, 4);
       Point q = new Point(6, 8);
       double dist = p.distance(q);  //Point method
       System.out.println("The distance between p and q is " + dist);
    }
}
\end{trinket}
%\begin{code}
%Point p1 = new Point(0, 0);
%Point p2 = new Point(3, 4);
%double dist = p1.distance(p2);  // dist is 5.0
%\end{code}
%slr: end 12-17-19

It turns out you don't need the \java{printPoint} method (from the \java{PointParameter} trinket above) either.
If you invoke \java{System.out.println(blank)}, it prints the type of the object and the values of the attributes:

\begin{stdout}
java.awt.Point[x=3,y=4]
\end{stdout}

\index{toString}

\java{Point} objects provide a method called \java{toString} that returns a string representation of a point.
When you call \java{println} with objects, it {\em automatically} calls \java{toString} and displays the result.

%slr: 12-17-19
\textbf{Section Exercises:}
\begin{enumerate}
\item Modify the trinket \java{Dist} to print out the \java{Point} objects \java{p} and \java{q} by calling the \java{toString} method explicity, (for example \java{p.toString()}).
\item Modify the trinket \java{Dist} to print out the \java{Point} objects \java{p} and \java{q} directly. (The \java{println} method invokes \java{goString} implicitly.) Interpret what is produced.
\end{enumerate}
%slr; end 12-17-19

\section{Objects as Return Values}
\label{sec:Rectangle}

\index{Rectangle}
\index{class!Rectangle}

The \java{java.awt} package also provides a class named \java{Rectangle}.
To use it, you have to import it:

\begin{code}
import java.awt.Rectangle;
\end{code}

\java{Rectangle} objects are similar to points, but they have four attributes: \java{x}, \java{y}, \java{width}, and \java{height}.
The following example creates a \java{Rectangle} object and makes the variable \java{box} refer to it:

\begin{code}
Rectangle box = new Rectangle(0, 0, 100, 200);
\end{code}

Figure~\ref{fig.rectangle} shows the effect of this assignment.

\begin{figure}[!ht]
\begin{center}
\includegraphics{figs/rectangle.pdf}
\caption{Memory diagram showing a \java{Rectangle} object.}
\label{fig.rectangle}
\end{center}
\end{figure}

If you run \java{System.out.println(box)}, you get:

\begin{stdout}
java.awt.Rectangle[x=0,y=0,width=100,height=200]
\end{stdout}

Again, \java{println} uses the \java{toString} method provided by \java{Rectangle}, which knows how to represent \java{Rectangle} objects as strings.

\index{return}
\index{statement!return}

You can also write methods that return new objects.
%slr:  12-17-19 mods + added trinket
For example, the \java{findCenter} method in the trinket below takes a \java{Rectangle} as an argument and returns a \java{Point} with coordinates the center of the rectangle:

%slr: 12-17-19
% height = 130 + 11 * num_lines
\begin{trinket} [280] {RectCenter.java}
import java.awt.Rectangle;
import java.awt.Point;
public class RectCenter {

    public static Point findCenter(Rectangle box) {
       int x = box.x + box.width / 2;
       int y = box.y + box.height / 2;
       return new Point(x, y);
    }

    public static void main(String[] args) {
       Rectangle rect = new Rectangle(20, 40, 100, 50);
       Point center = findCenter(rect);
       System.out.println("The center of rect is " + center);
    }
}
\end{trinket}
%\begin{code}
%public static Point findCenter(Rectangle box) {
%    int x = box.x + box.width / 2;
%    int y = box.y + box.height / 2;
%    return new Point(x, y);
%}
%\end{code}

%The return type of this method is \java{Point}.
%The last line creates a new \java{Point} object and returns a reference to it.

%The \java{Rectangle} we created using the arguments \java{(0, 0, 100, 200)} has its upper-left corner in the origin.
%The center of this rectangle is \java{(50, 100)}, which is 50 pixels to the right and 100 pixels down from the origin.

\textbf{Section Exercises:}
\begin{enumerate}
\item add find concentric method?  Or as a lab?
\end{enumerate}
%slr: end 12-17-19

\section{Rectangles are Mutable}

\index{mutable}
\index{object!mutable}

You can change the contents of an object by making an assignment to one of its attributes.
For example, to ``move'' a rectangle without changing its size, you can modify the \java{x} and \java{y} values:

\begin{code}
Rectangle box = new Rectangle(0, 0, 100, 200);
box.x = box.x + 50;
box.y = box.y + 100;
\end{code}

The result is shown in Figure~\ref{fig.rectangle2}.

\begin{figure}[!ht]
\begin{center}
\includegraphics{figs/rectangle2.pdf}
\caption{Memory diagram showing updated attributes.}
\label{fig.rectangle2}
\end{center}
\end{figure}

\index{encapsulation}
\index{generalization}

We can encapsulate this code in a method and generalize it to move the rectangle by any amount:

\begin{code}
public static void moveRect(Rectangle box, int dx, int dy) {
    box.x = box.x + dx;
    box.y = box.y + dy;
}
\end{code}

The variables \java{dx} and \java{dy} indicate how far to move the rectangle in each direction.
Invoking this method has the effect of modifying the \java{Rectangle} that is passed as an argument.

\begin{code}
Rectangle box = new Rectangle(0, 0, 100, 200);
moveRect(box, 50, 100);  // now at (50, 100, 100, 200)
\end{code}

%The code displays \java{java.awt.Rectangle[x=50,y=100,width=100,height=200]}.

Modifying objects by passing them as arguments to methods can be useful.
But it can also make debugging difficult, because it is not always clear which method invocations modify their arguments.

Java provides a number of methods that operate on \java{Point}s and \java{Rectangle}s.
For example, \java{translate} has the same effect as \java{moveRect}, but instead of passing the rectangle as an argument, you use dot notation:

\begin{code}
box.translate(50, 100);
\end{code}

This line invokes the \java{translate} method on the object that \java{box} refers to, which modifies the object.

\index{object-oriented}

This syntax---using dot notation to invoke a method on an object, rather than passing it as a parameter---is more consistent with the style of object-oriented programming.

\section{Aliasing Revisited}
\label{aliasing}

\index{reference}

Remember that when you assign an object to a variable, you are assigning a {\em reference} to an object.
It is possible to have multiple variables that refer to the same object.
For example, this code creates two variables that refer to the same \java{Rectangle}:

\begin{code}
Rectangle box1 = new Rectangle(0, 0, 100, 200);
Rectangle box2 = box1;
\end{code}

Figure~\ref{fig.aliasing} shows the result: \java{box1} and \java{box2} refer to the same object, so any changes that affect one variable also affect the other.

\begin{figure}[!ht]
\begin{center}
\includegraphics{figs/aliasing.pdf}
\caption{Memory diagram showing two variables that refer to the same \java{Rectangle} object.}
\label{fig.aliasing}
\end{center}
\end{figure}

For example, the following code uses \java{grow} to make \java{box1} bigger by 50 units in all directions.
It decreases \java{x} and \java{y} by 50, and it increases \java{height} and \java{width} by 100:

\begin{code}
box1.grow(50, 50);                // grow box1 (alias)
\end{code}

The result is shown in Figure~\ref{fig.aliasing2}.

\begin{figure}[!ht]
\begin{center}
\includegraphics{figs/aliasing2.pdf}
\caption{Memory diagram showing the effect of invoking \java{grow}.}
\label{fig.aliasing2}
\end{center}
\end{figure}

Now, if we print \java{box1}, we are not surprised to see that it has changed.

\begin{code}
java.awt.Rectangle[x=-50,y=-50,width=200,height=300]
\end{code}

And if we print \java{box2}, we should not be surprised to see that it has changed, too, because it refers to the same object:

\begin{code}
java.awt.Rectangle[x=-50,y=-50,width=200,height=300]
\end{code}

This scenario is called ``aliasing'' because a single object has multiple names, or aliases, that refer to it.

As you can tell from this simple example, code that involves aliasing can get confusing fast, and it can be difficult to debug.

\section{Java Library Source}
\label{src.zip}

\index{library}
\index{source code}

So far we have used several classes from the Java library, including \java{System}, \java{String}, \java{Scanner}, \java{Math}, and \java{Random}.
These classes are written in Java, so you can read the source code to see how they work.

\index{src.zip}

The Java library contains thousands of files, many of which are thousands of lines of code.
That's more than one person could read and understand fully, but don't be intimidated!

Because it's so large, the library source code is stored in a ZIP archive named \java{src.zip}.
If you have Java installed on your computer, you should already have this file somewhere.

\begin{itemize}
\item On Linux, it's likely under: \verb"/usr/lib/jvm/.../lib"
\\ If not, you might have to install the {\tt openjdk-...-source} package.

\item On MacOS, it's likely under: \\ \verb"/Library/Java/JavaVirtualMachines/.../Contents/Home/lib"

\item On Windows, it's likely under: \verb"C:\Program Files\Java\...\lib"
\end{itemize}

When you open (or unzip) the file, you will see folders that correspond to Java packages.
For example, open the {\tt java} folder, and then open the {\tt awt} folder.
(If you don't see a {\tt java} folder at first, open the {\tt java.desktop} folder.)
You should now see {\tt Point.java} and {\tt Rectangle.java}, along with the other classes in the \java{java.awt} package.

Open {\tt Point.java} in your editor and skim through the file.
It uses language features we haven't discussed yet, so you probably won't understand every line.
But you can get a sense of what professional Java source code looks like by browsing through the library.

\index{documentation}
\index{HTML}
\index{Javadoc}

Notice how much of {\tt Point.java} is documentation (see Appendix~\ref{javadoc}).
Each method includes comments and tags like \java{@param} and \java{@return}.
Javadoc reads these comments and generates documentation in HTML.
You can see the same documentation online by doing a web search for ``Java Point''.

Now take a look at the \java{grow} and \java{translate} methods in the \java{Rectangle} class.
There is more to them than you may have expected.

% ABD: I'm not sure these conclusions follow from the example

%But that doesn't limit your ability to use these methods in a program.

%Object-oriented programming makes it possible to hide messy details so that you can more easily use and understand code that other people wrote.

%By looking at the source code for \java{Point}, \java{Rectangle}, and other classes, we hope you will learn two things.
%1) Objects encapsulate data and provide methods to access and modify the data directly.

\section{Class Diagrams}
\label{UML}

To summarize what we've learned so far, \java{Point} and \java{Rectangle} objects have attributes and methods.
Attributes are an object's {\em data}; methods are an object's {\em code}.
An object's {\em class} definition specifies the attributes and methods that it has.

\index{UML}

{\bf Unified Modeling Language} (UML) defines a graphical way to summarize this information.
Figure~\ref{fig.umlPoint}, shows two examples, the {\bf UML class diagrams} for the \java{Point} and \java{Rectangle} classes.

\begin{figure}[!ht]
\begin{center}
\includegraphics{figs/point-rect.pdf}
\caption{UML class diagrams for \java{Point} and \java{Rectangle}.}
\label{fig.umlPoint}
\end{center}
\end{figure}

\index{class diagram}
\index{diagram!class}

Each class is represented by a box with the name of the class,
a list of attributes, and a list of methods.

\index{private}
\index{variable!private}

To identify the types of attributes and parameters, UML uses a language-independent syntax, like {\tt x:~int} rather than Java syntax, \java{int x}.

The plus sign (\java{+}) identifies \java{public} attributes and methods.
A minus sign (\java{-}) identifies \java{private} attributes and methods, which we discuss in the next chapter.

Both \java{Point} and \java{Rectangle} have additional methods; we only show the ones introduced in this chapter.

In contrast to memory diagrams, which visualize objects (and variables) at run-time, a class diagram visualizes the source code at compile-time.

\section{Scope Revisited}

\index{scope}

In Section~\ref{stack}, we introduced the idea that variables have scope.
The scope of a variable is the part of a program where a variable can be used.

Consider the first few lines of the \java{Rectangle.translate} method from the Java library source code:

% TODO: ML suggests adding the header line of the class and the attribute declatrations

\begin{code}
public void translate(int dx, int dy) {
    int oldv = this.x;
    int newv = oldv + dx;
    if (dx < 0) {
    ...
\end{code}

This example uses three kinds of variables:

\begin{enumerate}

\item Parameters (\java{dx} and \java{dy})

\item Local variables (\java{oldv} and \java{newv})

\index{this}
\item Attributes (\java{this.x})

\end{enumerate}

Parameters and local variables are created in a stack frame when a method is invoked (see Figure~\ref{fig.stack}).  The method's stack frame, and therefore its parameters and local variables, disappear when the method returns.
They can be used anywhere inside the method, but are not visible in other methods or in other classes.

Attributes are created and stored within an object when created, and they disappear when the object is destroyed.
They can be used \textit{anywhere} within the object including its methods using the keyword \java{this} even if declared private.
And if they are public, they can be used in other classes via references to the object, for example \java{box1.x}.

When the Java compiler encounters a variable name, it searches backwards for its declaration in ever-widening scope.
The compiler first looks for local variables, then parameters, then attributes.

% ABD: The previous three paragraphs pack in a lot.  Some of it is a little early, and some of it is not explained carefully here.  So I think this is a soft spot to come back to in future revisions.

%slr: 12-17-19.  I agree.  Slight modifications made to above three paragraphs.

%\section{Shadowing}
%
% CSM: We now discuss shadowing in Chapter 11.
% @Deprecated is a nice idea, but unnecessary.
%
%It's possible to declare a local variable (or parameter) with the same name as an attribute.
%For example, you can declare \java{int x} in a method of the \java{Rectangle} class.
%In that method, the expression \java{x + 5} will refer to the local variable, even though there is an attribute named \java{x}.
%
%\index{shadowing}
%
%This situation is called {\bf shadowing}, because the local variable ``hides'' the attribute.
%Java provides the keyword \java{this} to refer to attributes explicitly.
%For example, look at the \java{Rectangle.reshape} method:
%
%\begin{code}
%@Deprecated
%public void move(int x, int y) {
%    this.x = x;
%    this.y = y;
%}
%\end{code}
%
%\index{scope}
%
%The variables \java{x} and \java{y} are parameters of the \java{move} method, and \java{this.x} and \java{this.y} are attributes of the rectangle being moved.
%These variables have the same name, but they have different scope.
%
%\index{deprecated}
%
%Notice that the \java{move} method has been {\bf deprecated}.
%Sometimes when new versions of Java are released, the library classes are revised or enhanced.
%The documentation for \java{Rectangle.move} indicates that the \java{setLocation} method should be used instead.
%
%The word ``deprecate'' means to express strong disapproval.
%Java discourages using deprecated methods, either because they are dangerous, or because better alternatives exist.
%The compiler will display a warning if you attempt to use a deprecated method.

\section{Garbage Collection}

In the previous section, we said that attributes exist as long as the object exists.
But when does an object cease to exist?
Here is a simple example:

%In Section~\ref{aliasing}, we saw what happens when more than one variable refers to the same object.
%What happens when {\em no} variables refer to an object?

\begin{code}
Point blank = new Point(3, 4);
blank = null;
\end{code}

The first line creates a new \java{Point} object and makes \java{blank} refer to it.
The second line changes \java{blank} so that instead of referring to the object, it refers to nothing.
As shown in Figure~\ref{fig.reference3}, after the second assignment, there are no references to the \java{Point} object.

% TODO: ML suggests revising this diagram to show before and after

\begin{figure}[!ht]
\begin{center}
\includegraphics{figs/reference3.pdf}
\caption{Memory diagram showing the effect of setting a variable to \java{null}.}
\label{fig.reference3}
\end{center}
\end{figure}

If there are no references to an object, there is no way to access its attributes or invoke a method on it.
From the program's point of view, it ceases to exist.
However, it's still present in the computer's memory, taking up space.

\index{garbage collection}

As your program runs, the system automatically looks for stranded objects and deletes them; then the space can be reused for new objects.
This process is called {\bf garbage collection}.
%You can manually run the garbage collector by invoking \java{System.gc()} method.

You don't have to do anything to make garbage collection happen, and in general, you don't have to be aware of it.
But in high-performance applications, you may notice a slight delay every now and then while Java reclaims space from discarded objects.

\section{Mutable vs Immutable}

\index{mutable}
\index{immutable}

\java{Point}s and \java{Rectangle}s are {\bf mutable} objects, because their attributes can be modified.
You can modify their attributes directly, like \java{box.x = 15}, or you can invoke methods that modify their attributes, like \java{box.translate(15, 0)}.

In contrast, immutable objects like \java{String}s and \java{Integer}s cannot be modified.
They don't allow direct access to their attributes or provide methods that change them.

Immutable objects have a number of advantages that help improve the reliability and performance of programs.
You can pass strings (and other immutable objects) to methods without worrying about their contents changing as a ``side-effect'' of the method.
That makes programs easier to debug, and more reliable.

Also, two strings that contain the same characters can be stored in memory only once.
That can reduces the amount of memory the program uses, and can speed it up.

% ABD: Do we want trinket markup in this version?  It seems to require cmplete class definitions, which involve code that's extraneous to the example.

In the following example, \java{s1} and \java{s2} are created differently, but they refer to equivalent strings; that is, the two strings contains the same characters.

\index{Surprise.java}

\begin{trinket}[200]{Surprise.java}
public class Surprise {
    public static void main(String[] args) {
        String s1 = "Hi, Mom!";
        String s2 = "Hi, " + "Mom!";
        if (s1 == s2) {                // true!
            System.out.println("s1 and s2 are the same");
        }
    }
}
\end{trinket}

Because both strings are specified at compile time, the compiler can tell that they are equivalent.
And because strings are immutable, there is no need to make two copies; the compiler can create one \java{String} and make both variables refer to it.

As a result, the test \java{s1 == s2} turns out to be true, which means that \java{s1} and \java{s2} refer to the same object.
In other words, they are not just equivalent; they are identical.

%Since neither variable can change the string itself, both \java{s1} and \java{s2} will be \java{"Hi, Mom!"} until they are reassigned.

Although immutable objects have some advantages, mutable objects have other advantages.
Sometimes it is more efficient to modify an existing object, rather than creating a new one.
And some computations can be expressed more naturally using mutation.

Neither design is always better, which is why you will see both.

\section{StringBuilder Objects}
\label{stringbuilder}

Here's an example where mutable objects are efficient and arguably more natural: building a long string by concatenating lots of small pieces.

Strings are particularly inefficient for this operation.
For example, consider the following program, which reads ten lines from \java{System.in} and concatenates them into a single \java{String}.

\index{Append.java}

\begin{code}
String text = "";
for (int i = 0; i < 10; i++) {
    String line = in.nextLine();        // new string
    text = text + line + '\n';    // two more strings
}
\end{code}

Inside the \java{for} loop, \java{in.nextLine()} returns a new string each time it is invoked.
The next line of code concatenates \java{text} and \java{line}, which creates another string, and then appends the newline character, which creates yet another string.

As a result, this loop creates 30 \java{String} objects!
At the end, \java{text} refers to the most recent \java{String}.
Garbage collection deletes the rest, but that's a lot of garbage for a seemly simple program.

\index{StringBuilder}

The Java library provides the \java{StringBuilder} class for just this reason.
It's part of the \java{java.lang} package, so you don't need to import it.
Because \java{StringBuilder} objects are mutable, they can implement concatenation more efficiently.

Here's a version of the program that uses \java{StringBuilder}:

%TODO: ML suggests also showing how to print the results
%slr:  12-20-19
% height = 130 + 11 * num_lines
\begin{trinket}[300]{MutableString.java}
import java.util.Scanner;
public class MutableString {
    public static void main(String[] args) {
       Scanner in = new Scanner(System.in);
       StringBuilder text = new StringBuilder();
       System.out.println("Enter 10 lines of lines of text followed by <return>:");
       for (int i = 0; i < 3; i++) {
          String line = in.nextLine();
          text.append(line);
          text.append('\n');
       }
       System.out.println("Here is what you entered:\n" + text);
    }
}
\end{trinket}
%\begin{code}
%StringBuilder text = new StringBuilder();
%for (int i = 0; i < 10; i++) {
%    String line = in.nextLine();
%    text.append(line);
%    text.append('\n');
%}
%\end{code}
%slr: end 12-10-19

The \java{StringBuilder append} method takes a \java{String} as a parameter and appends it to the end of the \java{StringBuilder}.
Each time it is invoked, it modifies the \java{StringBuilder} object; it doesn't create any new objects.

The \java{StringBuilder} class also provides methods for inserting and deleting parts of strings efficiently.
Programs that manipulate large amounts of text run much faster if you use \java{StringBuilder} instead of \java{String}.

\section{Vocabulary}

\begin{description}

\term{attribute}
One of the named data items that make up an object.
%Each object has its own copy of the attributes for its class.

\term{dot notation}
Use of the dot operator (\java{.}) to access an object's attributes or methods.

\term{UML}
Unified Modeling Language, a standard way to draw diagrams for software engineering.

\term{class diagram}
An illustration of the attributes and methods for a class.

%\term{deprecated}
%A library method that has been replaced by another method, but left in the class for backwards compatibility.

\term{garbage collection}
The process of finding objects that have no references and reclaiming their storage space.

\term{mutable}
An object that can be modified at any time.
Points and rectangles are mutable by design.

\end{description}

\section{Exercises}

The code for this chapter is in the {\tt ch10} directory of {\tt ThinkJavaCode2}.
See page~\pageref{code} for instructions on how to download the repository.
Before you start the exercises, we recommend that you compile and run the examples.

At this point you know enough to read Appendix~\ref{graphics}, which is about simple 2D graphics and animations.
During the next few chapters, you should take a detour to read this appendix and work through the exercises.

\begin{exercise}  %%V6 Ex10.1

The point of this exercise is to make sure you understand the mechanism for passing objects as parameters.

\begin{enumerate}

\item For the following program, draw a stack diagram showing the local variables and parameters of \java{main} and \java{riddle} just before \java{riddle} returns.
Use arrows to show which objects each variable references.

\item What is the output of the program?

\item Is the \java{blank} object mutable or immutable?
How can you tell?

\end{enumerate}

\begin{code}
public static int riddle(int x, Point p) {
    x = x + 7;
    return x + p.x + p.y;
}
\end{code}

\begin{code}
public static void main(String[] args) {
    int x = 5;
    Point blank = new Point(1, 2);

    System.out.println(riddle(x, blank));
    System.out.println(x);
    System.out.println(blank.x);
    System.out.println(blank.y);
}
\end{code}

\end{exercise}

\begin{exercise}  %%V6 Ex10.2

The point of this exercise is to make sure you understand the mechanism for returning new objects from methods.
The following code uses \java{findCenter} and \java{distance} as defined in this chapter.

\begin{enumerate}

\item Draw a stack diagram showing the state of the program just before \java{findCenter} returns.
Include all variables and parameters, and show the objects those variables refer to.

\item Draw a stack diagram showing the state of the program just before \java{distance} returns.
Show all variables, parameters, and objects.

\item What is the output of this program?
(Can you tell without running it?)

\end{enumerate}

\begin{code}
public static void main(String[] args) {
    Point blank = new Point(5, 8);

    Rectangle rect = new Rectangle(0, 2, 4, 4);
    Point center = findCenter(rect);

    double dist = distance(center, blank);
    System.out.println(dist);
}
\end{code}

\end{exercise}

\begin{exercise}  %%V6 Ex10.3

This exercise is about aliasing.
Recall that aliases are two variables that refer to the same object.
The following code uses \java{findCenter} and \java{printPoint} as defined in this chapter.

\begin{enumerate}

\item Draw a diagram that shows the state of the program just before the end of \java{main}.
Include all local variables and the objects they refer to.

\item What is the output of the program?

\item At the end of \java{main}, are \java{p1} and \java{p2} aliased?
Why or why not?

\end{enumerate}

\begin{code}
public static void main(String[] args) {
    Rectangle box1 = new Rectangle(2, 4, 7, 9);
    Point p1 = findCenter(box1);
    printPoint(p1);

    box1.grow(1, 1);
    Point p2 = findCenter(box1);
    printPoint(p2);
}
\end{code}

\end{exercise}


\paragraph{Nodal value interpolant}
\Label{sc-seio}

For any continuous function $u$, we define its linear finite element
interpolation, $(I_h u)(x)\in V_{h,0}$,  as follows:
\begin{equation}
  \label{u-interp}
(I_h u)(x)= \sum_{i=1}^{n_h}u(x_i)\varphi_i(x).
\end{equation}
Usually, we also denote $(I_h u)(x)$ as $u_I(x)$. Using interpolation, we can obtain the following approximation property of
linear finite element space.
%For any $v\in\Shz$, we can obviously write
%$$
%        v(x)=\sum_{i=1}^{n_h}v(x_i)\phi_i(x).
%$$
%The nodal value interpolation\index{interpolation} operator $I_h: C(\bar\Om)\mapsto V_h$ is defined as follows \index{$I_h$}
%$$
%        (I_h u)(x_i)=u(x_i),\qall x_i\in {\cal N}_h,
%$$
%where ${\cal N}_h$ is the set of the vertexes for the partition $\mathcal T_h$.
\begin{figure}[hpt]
\begin{center}
\includegraphics*[height=2.5in, width=3in]{figures/fdsolutions.pdf}
\caption{Approximation of finite element space.}
\label{Interpolation}
\end{center}
\end{figure}

\begin{theorem}\label{interp00}
Assume that $\mathcal T_h$ is quasi-uniform and $V_h$ is the linear finite element space associated with $\mathcal T_h$, then
\begin{equation}
\label{error0}
\inf_{v_h\in V_h} \|v-v_h\|+h |v-v_h|_{1}\lc h^2 |v|_2
        \qall v\in H^2(\Om).
\end{equation}
 \end{theorem}
 \begin{proof}
Let us first prove Theorem \ref{interp00} for $d=1, 2, 3$.
This proof presented here follows from Xu~\cite{xu1982estimate} (see also Xu~\cite{xu2013estimate}).
Let $x=(x^1,\ldots, x^d)$ and $a_i=(a^1_{i}, \ldots, a^d_{i})$. Introducing
the auxiliary functions
$$
g_i(t)=v(a_i(t)),\mbox{  with  }  a_i(t)=a_i+t(x-a_i),
$$
we have
$$
g_i'(t)=(\nabla v)(a_i(t))\cdot (x-a_i)
=\sum_{l=1}^d(\partial_lv)(a_i(t))(x^l-a_i^l)
$$
and
\begin{equation}\label{gpp}
g_i''(t)=\sum_{k,l=1}^d\partial^2_{kl}v)(a_i(t))(x^k-a_i^k)(x^l-a_i^l).
\end{equation}
Note Taylor expansion
$$
        g_i(0)=g_i(1)-g_i'(1)+\int_0^1tg''_i(t)dt,
$$
namely
\begin{equation}\label{Taylor_vi}
v(a_i)=v(x)-(\nabla v)(x)\cdot (x-a_i)+\int_0^1tg''_i(t)dt,
\end{equation}
and note that
$$
(I_hv)(x)=\sum_{i=1}^{d+1}v(a_i)\lambda_i(x), \quad \sum_{i=1}^{d+1}\lambda_i(x)=1,
$$
and
$$
\sum_{i=1}^{d+1}(x-a_i)\lambda_i(x)=0.
$$
It follows that
\begin{equation}\label{Ihvv}
(I_hv-v)(x)=\sum_{i=1}^{d+1}\lambda_i(x)\int_0^1tg''_i(t)dt.
\end{equation}
Using \rf{gpp} and the trivial fact that $|x^l-a_i^l|\le h$,
we obtain
\begin{eqnarray*}
\|g''_i(t)\|_{L^2(\tau)}\le h^2
\sum_{k,l=1}^d\|(\partial^2_{kl}v)(a_i(t))\|_{L^2(\tau_i^t)}
\le h^2t^{-d/2}\sum_{k,l=1}^d\|\partial^2_{kl}v\|_{L^2(\tau)},
\end{eqnarray*}
where we have used the following change of variable
$$
y=a_i+t(x-a_i): \tau\mapsto \tau_i^t\subset\tau \mbox{ with } dy=t^ddx.
$$
Now taking the $L^2(\tau)$ norm on both hand of sides of
\rf{Ihvv}, we get
\begin{eqnarray*}
\|I_hv-v\|_{L^2(\tau)}
&\le& h^2\sum_{i=1}^{d+1}\max_{x\in\tau}|\lambda_i(x)|
\int_0^1t\|g''_i(t)\|_{L^2(\tau)}\;dt\\
&\le& (d+1)\int_0^1t^{-d/2}dt\;h^2\;
\sum_{k,l=1}^d\|\partial^2_{kl}v\|_{L^2(\tau)}\\
&\le&\frac{2(d+1)}{4-d}h^2
\sum_{k,l=1}^d\|\partial^2_{kl}v\|_{L^2(\tau)}\\
&\le&\frac{4d(d+1)}{4-d}h^2|v|_{H^2(\tau)}.
\end{eqnarray*}
Now we prove the $H^1$ error estimate. Notice that
$$
[\partial_{j}( I_{h} v - v)](x) = \sum_{i} (\partial_{j} \lambda_{i} )(x) \int_{0}^{1} t g''_{i}(t) dt + \sum_{i} \lambda_{i}(x) \partial_{j} \int_{0}^{1} t g''_{i}(t) dt.
$$
By \rf{Taylor_vi},
$$
\int_0^1tg''_i(t)dt = v(a_i) - v(x) + (\nabla v)(x)\cdot (x-a_i)
$$
therefore,
\begin{eqnarray*}
\lefteqn{\partial_{j} \int_0^1tg''_i(t)dt} \\
& = & - \partial_{j} v + (\nabla \partial_{j} v )(x) (x - a_{i}) + \nabla v \cdot e_{j}
\hcomment{$e_{j}$ is the $j$-th standard basis}  \\
& = & (\nabla \partial_{j} v )(x) (x - a_{i}).
\end{eqnarray*}
Noting that $\sum_{i} \lambda_{i}( \nabla \partial_{j} v )(x) (x - a_{i}) = 0$, we have
$$
[\partial_{j}( I_{h} v - v)](x) = \sum_{i} (\partial_{j} \lambda_{i} )(x) \int_{0}^{1} t g''_{i}(t) dt.
$$
Then the estimate for $|\nabla(I_hv-v)|_{L^2(\tau)}$
follows by a similar argument and the following obvious
estimate
$$
|(\nabla\lambda_i)(x)|\lc\frac{1}{h}.
$$

On the proof of Theorem \ref{interp0} for $d\ge 4$, the above proof does not
apply for $d \ge 4$. This is because when $d \ge 4$, the embedding
relation between $H^{2}(\Om) \hookrightarrow C(\bar{\Om})$ is no longer true.  Only
continuous functions can have interpolations. In this case, one approach is to use the
so-called Scott-Zhang interpolation \cite{scott1990finite}, the
details can be found in \cite{Xu.J2015a}.
\end{proof}
As a result of Theorem \ref{interp00}, we have
\begin{theorem}\label{interp0}
Let $V_N$ be linear finite element space on a quasi-uniform
triangulation consisting of $N$ element.  Then
\begin{equation}
\label{error0N}
\inf_{v_h\in V_N} \|v-v_h\|+N^{-{1\over d}} |v-v_h|_{1}\lc N^{-{2\over d}} |v|_2
        \qall v\in H^2(\Om).
\end{equation}
\end{theorem}

497-6DL/6 Finite Element Method/6.2-Nodal-Interpolation.tex


\include{config/config}

\begin{document}
% ========== Edit your name here
\title{MATH 2901 Basic Probability Lecture Notes 7}
\author{Instructor: Richard Kleeman}
\date{}
\maketitle

%\medskip

% ========== Contents begin here ==============
Most of our study of probability has dealt with independent trials processes, in which we have i.i.d. random variables $X_i$. This is useful for modeling and sampling situations. But when study random dynamical systems, they are less useful. In these cases, $\{X_i\}$ are no longer independent because they are all part of the system, and they have to interact with each other. So one way to modeling this is to use conditional probability.

\section{Markov chains: basic definitions}
Consider a countable sequence $X_1, X_2, \dots, X_n, \dots$ which
\begin{itemize}
    \item the subscript represents time, thus we have discrete time sequence;
    \item each $X_i$ has finite outcomes.
\end{itemize}
In other words, each $X_i$ is a discrete random variable that takes one of $N$ possible values, where
$N = \card(S)$ and $S$ is the outcome space; it may be the case that $N = \infty$.

\begin{definition}
The process $X$ is a \textbf{Markov chain} if it satisfies the \textbf{Markov condition}:
\begin{equation*}
    \Prob\left(X_{n}=s | X_{1}=x_{1}, X_{2}=x_{2}, \ldots, X_{n-1}=x_{n-1}\right)=\Prob\left(X_{n}=s | X_{n-1}=x_{n-1}\right)
\end{equation*}
for all $n\geq 1$ ans all $s, x_1, \dots, x_{n-1} \in S$.
\end{definition}
Since $S$ is assumed countable, it can be put in one-to-one correspondence with some subset $S'$ of the integers, and without loss of generality we can assume that $S$ is this set $S'$ of integers. Then the evolution of a chain is described by its \textbf{transition probabilities}
\begin{equation}
    \label{eq:7.1}
    \tag{7-1}
    \Prob(X_{n} = j \spacevert X_{n-1} = i) := p_{ij}(n-1).
\end{equation}

\begin{theorem}
The transition matrix $P$ is a \textbf{stochastic matrix}, which is to say that:
\begin{enumerate}[(a)]
    \item $P$ has non-negative entries, or $p_{ij} \geq 0$ for all $i, j$,
    \item $P$ has row sums equal to one, or $\sum_j p_{ij} = 1$ for all $i$.
\end{enumerate}
\end{theorem}

%Note that the Markov property is equivalent to the following stipulation, which is also called \textbf{$k$-step transition probabilities}.

\begin{definition}
The \textbf{$k$-step transition probabilities} is defined as
\begin{equation}
    \label{eq:7.2}
    \tag{7-2}
    \Prob(X_{n+k-1} = j \spacevert X_{n-1} = i) := p_{ij}^{(k)}(n-1) \quad \text{for any } n, k \geq 1.
\end{equation}
\end{definition}

%Based on this, we can define
\begin{definition}
The \textbf{transition matrix} $P(n-1) = (p_{ij}(n-1))$ is the $\abs{S} \times \abs{S}$ matrix of \textbf{transition probabilities} $p_{ij}(n-1)$. The \textbf{$k$-step transition matrix} $P^{(k)}(n-1) = (p_{ij}^{(k)}(n-1))$ is the matrix of \textbf{$k$-step transition probabilities} $p_{ij}^{(k)}(n-1)$.
\end{definition}

\begin{remark}
We can check that the Markov property is equivalent to each of the following two stipulations: for each $s \in S$ and for every sequence $\{ x_i \spacevert i \geq 0\}$ in $S$,
\begin{equation}
    \label{eq:7.3a}
    \tag{7-3a}
    \begin{split}
        \Prob\left(X_{n+1}=s | X_{n_{1}}=x_{n_{1}}, X_{n_{2}}=x_{n_{2}}, X_{n_{k}}=x_{n_{k}}\right)=\Prob\left(X_{n+1}=s | X_{n_{k}}=x_{n_{k}}\right) \\
        \text{for all } n_1 < n_2 < \cdots < n_k \leq n.
    \end{split}
\end{equation}
\begin{equation}
    \label{eq:7.3b}
    \tag{7-3b}
    \begin{split}
        \Prob\left(X_{m+n}=s | X_{0}=x_{0}, X_{1}=x_{1}, \ldots, X_{m}=x_{m}\right)=\Prob\left(X_{m+n}=s | X_{m}=x_{m}\right) \\
        \text{for any } m, n \geq 0.
    \end{split}
\end{equation}
\end{remark}

Next we introduce another assumption:
\begin{definition}
The chain $X$ is called \textbf{homogeneous} if
\begin{equation*}
    \Prob(X_n = j \spacevert X_{n-1} = i) = \Prob(X_2 = j \spacevert X_1 = i), \quad \text{for all } n, i, j.
\end{equation*}
\end{definition}

\begin{remark}
Note the following two remarks.
\begin{enumerate}[(a)]
    \item Markov property and homogeneity property are \textbf{independent}. There are also Markov chains which are not homogeneous.
    \item If a Markov chain $X$ is homogeneous, then $p_{ij}(n-1)$ doesn't depends on the time $(n-1)$, \ie, the probability transition matrix $P$ is \textbf{fixed} at each step.
\end{enumerate}
\end{remark}

\section{Properties of Markov chains}
By the assumption of homogeneity, we know $P(n-1) = P$. That $P^{(k)}(n-1)$ doesn't depend on $(n-1)$ is a consequence of the following important fact.

\begin{theorem}[Chapman-Kolmogorov equations]
\begin{equation*}
    p_{ij}^{(n+r)}(m) = \sum_{k} p_{ik}^{(n)}(m) p_{kj}^{(r)}(m+n).
\end{equation*}
Therefore,
\begin{equation*}
    P^{(n+r)}(m) = P^{(n)}(m) P^{(r)}(m+n).
\end{equation*}
Furthermore, if we have homogeneity, then
\begin{equation*}
    P^{n}(m) = P^n,
\end{equation*}
the $n$th power of $P$.
\end{theorem}

\begin{proof}
    \begin{equation*}
        \begin{split}
            p_{ij}^{n+r}(m) &= \Prob(X_{m+n+r} = j \spacevert X_m = i) \\
            &= \sum_{k} \Prob(X_{m+n+r} = j, X_{m+n} = k \spacevert X_m = i) \\
            &= \sum_{k} \Prob(X_{m+n+r} = j \spacevert X_{m+n} = k, X_m = i) \Prob(X_{m+n} = k \spacevert X_m = i) \quad \text{(marginalization)} \\
            &= \sum_{k} \Prob(X_{m+n+r} = j \spacevert X_{m+n} = k) \Prob(X_{m+n} = k \spacevert X_m = i) \quad \text{(Markov property)} \\
            &= \sum_{k} p_{kj}^{(r)}(m+n) p_{ik}^{(n)}(m) .
        \end{split}
    \end{equation*}
    The marginalization step uses the fact that
    \begin{equation*}
        \Prob(A \cap B \spacevert C) = \Prob(A \spacevert B \cap C) \Prob(B \spacevert C).
    \end{equation*}
    The Markov property step uses \eqref{eq:7.3a}. The $n$th power is obtained by iteration.
\end{proof}

One consequence of the preceding theorem is that $P^{(n)}(m) = P^{(n)}(0)$. Note that this consequence is obtained \textbf{with homogeneous assumption}. We write $P_n$ for $P^{(n)}(m)$ and $p_{ij}(n)$ for $p_{ij}^{(n)}(m)$. This theorem relates long-term development to short-term development, and tells us how $X_n$ depends on the initial variable $X_0$. Let $\mu_i^{(n)} = \Prob(X_n = i)$ be the mass function of $X_n$, and write $\mu^{(n)}$ for the \textbf{row vector} with entries $(\mu_i^{(n)} : i \in S)$. Then we have the following lemma.

\begin{lemma}
$\mu^{(m+n)} = \mu^{(m)}P_n$, and hence $\mu^{(n)} = \mu^{(0)}P^n$.
\end{lemma}

\begin{proof}
    We have that
    \begin{equation*}
        \begin{split}
            \mu_j^{(m+n)} &= \Prob(X_{m+n} = j) \\
            &= \sum_{i} \Prob(X_{m+n} = j \spacevert X_m = i) \Prob(X_m = i) \\
            &= \sum_{i} \mu_i^{(m)} p_{ij}(n) \\
            &= (\mu^{(m)} P_n)_j
        \end{split}
    \end{equation*}
    and the result follows from the preceding theorem.
\end{proof}

The random evolution of the chain is determined by the transition matrix $P$ and the initial mass function $\mu^{(0)}$.

\section{Absorbing Markov chains}
An absorbing Markov chain is a special type of Markov chains.

\begin{definition}
A state $s_i$ of a Markov chain is called \textbf{absorbing} if it is impossible to leave it (\ie, $p_{ii} = 1$). A Markov chain is \textbf{absorbing} if it has at least one absorbing state, and if from every state it is possible to go to an absorbing state (not necessarily in one step).
\end{definition}

\begin{definition}
In an absorbing Markov chain, a state which is not absorbing is called \textbf{transient}.
\end{definition}

\subsection{Canonical form of absorbing Markov chains}
Consider an arbitrary absorbing Markov chain. Renumber the states so that the transient states come first. If there are $r$ absorbing states and $t$ transient states, the transition matrix will have the following canonical form
\begin{equation*}
    P =
    \begin{blockarray}{rcc}
        & TR. & ABS. \\
    \begin{block}{r[cc]}
        TR. & Q &  R \\
        ABS. & 0 &  I \\
    \end{block}
    \end{blockarray}
\end{equation*}
Here $I$ is an $r\times r$ identity matrix, $0$ is an $r\times t$ zero matrix, $R$ is a nonzero $t\times r$ matrix, and $Q$ is an $t\times t$ matrix. The first $t$ states are transient and the last $r$ states are absorbing.

A standard matrix algebra argument shows that $P^n$ is of the form
\begin{equation*}
    P^n =
    \begin{blockarray}{rcc}
        & TR. & ABS. \\
    \begin{block}{r[cc]}
        TR. & Q^n &  * \\
        ABS. & 0 &  I \\
    \end{block}
    \end{blockarray}
\end{equation*}
where the asterisk $*$ stands for the $t\times r$ matrix in the upper right-hand corner of $P^n$.

The entries of $Q_n$ give the probabilities for being in each of the transient states after $n$ steps for each possible transient starting state. The following theorem will show that every entry of $Q_n$ will approach zero as $n$ approaches infinity (\ie, $Q_n \to 0$).

\begin{theorem}
In an absorbing Markov chain, the probability that the process will be absorbed is 1 (\ie, $Q_n \to 0$ as $n \to \infty$).
\end{theorem}

\begin{proof}
From each non-absorbing state $s_j$ it is possible to reach an absorbing state. Let $m_j$ be the minimum number of steps required to reach an absorbing state, starting from $s_j$. Let $p_j$ be the probability that, starting from $s_j$, the process will not reach an absorbing state in $m_j$ steps. Then $p_j < 1$. Let $m$ be the largest of the $m_j$ and let $p$ be the largest of $p_j$. The probability of not being absorbed in $m$ steps is less than or equal to $p$, in $2m$ steps less than or equal to $p^2$, etc. Since $p < 1$, these probabilities tend to 0. Since the probability of not being absorbed in $n$ steps is monotone decreasing, these probabilities also tend to 0, hence $\lim_{n\to\infty} Q_n = 0$.
\end{proof}

\subsection{The fundamental matrix}
\begin{definition}
For an absorbing Markov chain $P$, the matrix $N = (I-Q)^{-1}$ is called the \textbf{fundamental matrix} for $P$. The entry $n_{ij}$ of $N$ gives the expected number of times that the process is in the transient state $s_j$ if it is started in the transient state $s_i$.
\end{definition}

\begin{theorem}
For an absorbing Markov chain the matrix $I-Q$ has an inverse $N$ and $N = I + Q + Q^2 + \cdots$. The $ij$-entry $n_{ij}$ of the matrix $N$ is the expected number of times the chain is in state $s_j$, given that it starts in state $s_i$. The initial state is counted if $i = j$.
\end{theorem}

\begin{proof}
Let $(I-Q)x = 0$; that is $x = Qx$. Then, iterating this we see that $x = Q^n x$. Since $Q_n \to 0$, we have $Q_n x \to 0$, so $x = 0$. Thus $(I-Q)^{-1} = N$
exists. Note next that
\begin{equation*}
    (I-Q)(I+Q + Q^2 + \cdots + Q^n) = I - Q^{n+1}.
\end{equation*}
Thus multiplying both sides by $N$ gives
\begin{equation*}
    I + Q + Q^2 + \cdots + Q^n = N(I-Q^{n+1}) .
\end{equation*}
Letting $n$ tend to infinity we have
\begin{equation*}
    N = I + Q + Q^2 + \cdots .
\end{equation*}
Let $s_i$ and $s_j$ be two transient states, and assume throughout the remainder of the proof that $i$ and $j$ are fixed. Let $X^{(k)}$ be a random variable which equals 1 if the chain is in state $s_j$ after $k$ steps, and equals 0 otherwise. For each $k$, this random variable depends upon both $i$ and $j$; we choose not to explicitly show this dependence in the interest of clarity. We have
\begin{equation*}
    \Prob \left(X^{(k)}=1\right)=q_{i j}^{(k)},
\end{equation*}
and
\begin{equation*}
    \Prob \left(X^{(k)}=0\right)=1-q_{i j}^{(k)},
\end{equation*}
where $q_{i j}^{(k)}$ is the $ij$th entry of $Q^k$. These equations hold for $k = 0$ since $Q^0 = I$. Therefore, since $X^{(k)}$ is a 0-1 random variable, $\Exp(X^{(k)}) = q_{i j}^{(k)}$.

The expected number of times the chain is in state $s_j$ in the first $n$ steps, given that it starts in state $s_i$, is clearly
\begin{equation*}
    \Exp\left(X^{(0)}+X^{(1)}+\cdots+X^{(n)}\right)=q_{i j}^{(0)}+q_{i j}^{(1)}+\cdots+q_{i j}^{(n)}.
\end{equation*}
Letting $n$ tend to infinity we have
\begin{equation*}
    \Exp\left(X^{(0)}+X^{(1)}+\cdots\right)=q_{i j}^{(0)}+q_{i j}^{(1)}+\cdots=n_{i j}.
\end{equation*}
\end{proof}

\subsection{Time to absorption}
Given that the chain starts in state $s_i$, what is the expected number of steps before the chain is absorbed? The following theorem gives the answer.
\begin{theorem}
Let $t_i$ be the expected number of steps before the chain is absorbed, given that the chain starts in state $s_i$, and let $t$ be the \textbf{column vector} whose $i$th entry is $t_i$. Then
\begin{equation*}
    t = N \1
\end{equation*}
where $\1$ is a column vector all of whose entries are 1.
\end{theorem}

\begin{proof}
If we add all the entries in the $i$th row of $N$, we will have the expected number of times in any of the transient states for a given starting state $s_i$, that is, the expected time required before being absorbed. Thus, $t_i$ is the sum of the entries in the $i$th row of $N$. If we write this statement in matrix form, we obtain the theorem.
\end{proof}

\subsection{Absorption probabilities}
\begin{theorem}
Let $b_{ij}$ be the probability that an absorbing chain will be absorbed in the absorbing state $s_j$ if it starts in the transient state $s_i$. Let $B$ be the matrix with entries $b_{ij}$ . Then $B$ is an $t\times r$ matrix, and
\begin{equation*}
    B = NR,
\end{equation*}
where $N$ is the fundamental matrix and $R$ is as in the canonical form.
\end{theorem}

\begin{proof}
We have
\begin{equation*}
    \begin{aligned}
        B_{i j} &=\sum_{n} \sum_{k} q_{i k}^{(n)} r_{k j} \\ &=\sum_{k} \sum_{n} q_{i k}^{(n)} r_{k j} \\ &=\sum_{k} n_{i k} r_{k j} \\ &=(N R)_{i j}.
    \end{aligned}
\end{equation*}
This completes the proof.
\end{proof}

\end{document}


\section{Fundamentals of Estimation}
%This section is under Jazz-\work\\{\scriptsize [and will evolve from face-to-face interactions with students of mathematics at Uppsala University]}.

\subsection{Introduction}
Now that we have been introduced to two notions of convergence for RV sequences, we can begin to appreciate the basic limit theorems used in statistical inference.  The problem of estimation is of fundamental importance in statistical inference and learning.  We will formalise the general estimation problem here.  There are two basic types of estimation.  In point estimation we are interested in estimating a particular point of interest that is supposed to belong to a set of points.  In (confidence) set estimation, we are interested in estimating a set with a particular form that has a specified probability of ``trapping'' the particular point of interest from a set of points.  Here, a point should be interpreted as an element of a collection of elements from some space.

\subsection{Point Estimation}\label{S:PointEstimation}
{\bf Point estimation} is any statistical methodology that provides one with a ``{\bf single best guess}'' of some specific quantity of interest.  Traditionally, we denote this {\bf quantity of interest as $\theta^*$} and {\bf its point estimate as $\widehat{\theta}$ or $\widehat{\theta}_n$}.  The subscript $n$ in the point estimate $\widehat{\theta}_n$ emphasises that our estimate is based on $n$ observations or data points from a given statistical experiment to estimate $\theta^*$.  This quantity of interest, which is usually unknown, can be: %, namely $\theta$
%, may be an {\bf integral} $\Iz$ of a real valued function $h(x)$, i.e.~$\theta=\Iz := \int_a^b h(x)\,dx \in \Rz$, or simply a
\begin{itemize}
\item an {\bf integral} $\vartheta^* := \int_A h(x)\,dx \in \BB{\varTheta}$.  If $\vartheta^*$ is finite, then $\BB{\varTheta} =  \Rz$, or %  For e.g.~see \ref{S:BMC} on Monte Carlo integration.
\item a {\bf parameter} $\theta^*$ which is an element of the {\bf parameter space} $\BB{\Theta}$, denoted $\theta^* \in \BB{\Theta}$,
\item a {\bf distribution function (DF)} $F^* \in \Fz := \text{the set of all DFs}$
\item a {\bf density function (pdf)} $f \in \{ \text{``not too wiggly Sobolev functions''} \}$, or
\item a {\bf regression function} $g^* \in \Gz$, where $\Gz$ is a class of regression functions in a regression experiment with model: $Y=g^*(X)+\epsilon$, such that $\E(\epsilon)=0$, from pairs of observations $\{(X_i,Y_i)\}_{i=1}^n$, or
\item a {\bf classifier} $g^* \in \Gz$, i.e.~a regression experiment with discrete $Y = g^*(X)+\epsilon$, or
\item a {\bf prediction} in a regression experiment, i.e.~when you want to estimate $Y_i$ given $X_i$.
\end{itemize}
%\begin{table}[htpb]
%\begin{center}
%\begin{tabular}{|c c c|}
%\hline
%Quantity of Interest $\theta$ & Point Estimation & Sections \\ \hline
%Parameter $\theta \in \BB{\Theta} \subset \Rz^n$ & Parametric Estimation & Section~\ref*{S:ParametricEstimation} \\
%Integral $\Iz := \int_a^bh(x)\,dx$ & Monte Carlo Integration & Section~\ref*{S:BMC} \\ \hline
%\end{tabular}
%\end{center}
%\end{table}

Recall that a statistic is an RV $T(X)$ that maps every data point $x$ in the data space $\Xz$ with $T(x)=t$ in its range $\Tz$, i.e.~$T(x):\Xz \to \Tz$ (\hyperref[D:Statistic]{Definition~\ref*{D:Statistic}}).  Next, we look at a specific class of statistics whose range is the parameter space $\BB{\Theta}$.
\begin{definition}[Point Estimator]\label{D:Estimator}
A {\bf point estimator} $\widehat{\Theta}$ of some {\bf fixed and possibly unknown} $\theta^* \in \BB{\Theta}$ is a statistic that associates each data point $x \in \Xz$ with an estimate $\widehat{\Theta}(x)=\widehat{\theta} \in \BB{\Theta}$,
\[
\boxed{
 \widehat{\Theta} := \widehat{\Theta}(x)=\widehat{\theta}: \Xz \to \BB{\Theta}
 } \ .
\]
If our data point $x := (x_1,x_2,\ldots,x_n)$ is an $n$-vector or a point in the $n$-dimensional real space, i.e.~$x := (x_1,x_2,\ldots,x_n) \in \Xz_n \subset \Rz^n$, then we emphasise the dimension $n$ in our point estimator $\widehat{\Theta}_n$ of $\theta^* \in \BB{\Theta}$.
\[
\boxed{
\widehat{\Theta}_n :=  \widehat{\Theta}_n(x:=(x_1,x_2,\ldots,x_n))=\widehat{\theta}_n : \Xz_n \to \BB{\Theta}, \quad \Xz_n \subset \Rz^n
} \ .
\]
 The typical situation for us involves point estimation of $\theta^* \in \BB{\Theta}$ on the basis of one realisation $x \in\Xz_n \subset \Rz^n$ of an independent and identically distributed (IID) random vector $X=(X_1,X_2,\ldots,X_n)$, such that $X_1,X_2,\ldots,X_n \overset{\IID}{\sim} X_1$ and the DF of $X_1$ is $F(x_1; \theta^*)$, i.e.~the distribution of the IID RVs, $X_1, X_2,\ldots,X_n$, is parameterised by $\theta^* \in \BB{\Theta}$.
\end{definition}

\begin{example}[Coin Tossing Experiment ($X_1,\ldots,X_n \overset{IID}{\sim} \bernoulli(\theta^*)$)]\label{EX:CoinTossing}
I tossed a coin that has an unknown probability $\theta^*$ of landing Heads independently and identically $10$ times in a row.  Four of my outcomes were Heads and the remaining six were Tails, with the actual sequence of Bernoulli outcomes (Heads $\to 1$ and Tails $\to 0$) being $(1,0,0,0,1,1,0,0,1,0)$.  I would like to estimate the probability $\theta^* \in \BB{\Theta} = [0,1]$ of observing Heads using the natural estimator $\widehat{\Theta}_n((X_1,X_2,\ldots,X_n))$ of $\theta^*$:
\[
\widehat{\Theta}_n((X_1,X_2,\ldots,X_n)) := \widehat{\Theta}_n = \frac{1}{n} \sum_{i=1}^n X_i =: \overline{X}_n
\]
For the coin tossing experiment I just performed ($n=10$ times), the point estimate of the unknown $\theta^*$ is:
\begin{eqnarray}
\widehat{\theta}_{10} = \widehat{\Theta}_{10}((x_1,x_2,\ldots,x_{10}))
&=&\widehat{\Theta}_{10}((1,0,0,0,1,1,0,0,1,0)) \notag \\
&=& \frac{1+0+0+0+1+1+0+0+1+0}{10}=\frac{4}{10}=0.40 \notag \ .
\end{eqnarray}
\end{example}

\begin{labwork}[$\bernoulli(38/75)$ Computer Experiment]\label{LW:1000CoinTossingExp}
Simulate one thousand IID samples from a $\bernoulli(\theta^*=38/75)$ RV and store this data in an array called $\tt Samples$.  Use your student ID to initialise the fundamental sampler.  Now, pretend that you don't know the true $\theta^*$ and estimate $\theta^*$ using our estimator $\widehat{\Theta}_n=\overline{X}_n$ from the data array $\tt Samples$ for each sample size $n=1,2,\ldots,1000$.  Plot the one thousand estimates $\widehat{\theta}_1,\widehat{\theta}_2,\ldots,\widehat{\theta}_{1000}$ as a function of the corresponding sample size.  Report your observations regarding the behaviour of our estimator as the sample size increases.
\end{labwork}

\subsection{Some Properties of Point Estimators}\label{S:PropPointEstim}
Given that an estimator is merely a function from the data space to the parameter space, we need choose only the best estimators available.  Recall that a point estimator $\widehat{\Theta}_n$, being a statistic or an RV of the data has a probability distribution over its range $\BB{\Theta}$.  This distribution over $\BB{\Theta}$ is called the {\bf sampling distribution} of $\widehat{\Theta}_n$.  Note that the sampling distribution not only depends on the statistic $\widehat{\Theta}_n := \widehat{\Theta}_n(X_1,X_2,\ldots,X_n)$ but also on $\theta^*$ which in turn determines the distribution of the IID data vector $(X_1,X_2,\ldots,X_n)$.  The following definitions are useful for selecting better estimators from some lot of them.

\begin{definition}[Bias of a Point Estimator]\label{D:Bias}
The $\mathsf{bias}_n$ of an estimator $\widehat{\Theta}_n$ of  $\theta^* \in \BB{\Theta}$ is:
\begin{equation}\label{E:Bias}
\boxed{
\mathsf{bias}_n=\mathsf{bias}_n(\widehat{\Theta}_n) := \E_{\theta^*}(\widehat{\Theta}_n) - \theta^* = \int_{\Xz_n} \widehat{\Theta}_n(x) \, dF(x;\theta^*) - \theta^*
}
 \ .
\end{equation}
We say that the estimator $\widehat{\Theta}_n$ is {\bf unbiased} if $\mathsf{bias}_n(\widehat{\Theta}_n)=0$ or if $\E_{\theta^*}(\widehat{\Theta}_n)=\theta^*$ for every $n$.  If $\lim_{n \to \infty}\mathsf{bias}_n(\widehat{\Theta}_n)=0$, we say that the estimator is {\bf asymptotically unbiased}.
\end{definition}
Since the expectation of the sampling distribution of the point estimator $\widehat{\Theta}_n$ depends on the unknown $\theta^*$, we emphasise the $\theta^*$-dependence by $\E_{\theta^*}(\widehat{\Theta}_n)$.

\begin{example}[Bias of our Estimator of $\theta^*$]\label{EX:BiasEstimatePFromNIIDBernoulliTrials}
Consider the sample mean estimator $\widehat{\Theta}_n := \overline{X}_n$ of $\theta^*$, from $X_1,X_2,\ldots,X_n \overset{\IID}{\sim} \bernoulli(\theta^*)$.  That is, we take the sample mean of the $n$ IID $\bernoulli(\theta^*)$ trials to be our point estimator of $\theta^*\in[0,1]$.  Then, {\bf this estimator is unbiased} since:
\[
\E_{\theta^*}(\widehat{\Theta}_n) = \E_{\theta^*} \left( n^{-1} \sum_{i=1}^n X_i \right) = n^{-1} \E_{\theta^*} \left(  \sum_{i=1}^n X_i \right) = n^{-1} \sum_{i=1}^n \E_{\theta^*}(X_i) = n^{-1} n \theta^* = \theta^* \ .
\]
\end{example}

\begin{definition}[Standard Error of a Point Estimator]
The standard deviation of the point estimator $\widehat{\Theta}_n$ of  $\theta^* \in \BB{\Theta}$ is called the {\bf standard error}:
\begin{equation}\label{E:se}
\boxed{
\mathsf{se}_n = \mathsf{se}_n(\widehat{\Theta}_n)=\sqrt{\V_{\theta^*}(\widehat{\Theta}_n)} = \sqrt{ \int_{\Xz_n} \left( \widehat{\Theta}_n(x) - \E_{\theta^*}(\widehat{\Theta}_n) \right)^2 \, dF(x;\theta^*)}
}\ .
\end{equation}
\end{definition}
Since the variance of the sampling distribution of the point estimator $\widehat{\Theta}_n$ depends on the fixed and possibly unknown $\theta^*$, as emphasised by $\V_{\theta^*}$ in \eqref{E:se}, the $\mathsf{se}_n$ is also a possibly unknown quantity and may itself be estimated from the data.  The estimated standard error, denoted by $\widehat{\mathsf{se}}_n$, is calculated by replacing $\V_{\theta^*}(\widehat{\Theta}_n)$ in \eqref{E:se} with its appropriate estimate.

\begin{example}[Standard Error of our Estimator of $\theta^*$]\label{EX:StdErrEstimatePFromNIIDBernoulliTrials}
Consider the sample mean estimator $\widehat{\Theta}_n := \overline{X}_n$ of $\theta^*$, from $X_1,X_2,\ldots,X_n \overset{\IID}{\sim} \bernoulli(\theta^*)$.  Observe that the statistic:
$$T_n((X_1,X_2,\ldots,X_n)) := n \,  \widehat{\Theta}_n((X_1,X_2,\ldots,X_n)) = \sum_{i=1}^n X_i$$ is the $\binomial(n,\theta^*)$ RV.
The standard error  $\mathsf{se}_n$ of this estimator is:
\[
\mathsf{se}_n=\sqrt{\V_{\theta^*}(\widehat{\Theta}_n)}
= \sqrt{\V_{\theta^*}\left(\sum_{i=1}^n \frac{X_i}{n} \right)}
= \sqrt{\left(\sum_{i=1}^n \frac{1}{n^2}\V_{\theta^*}(X_i) \right)}
= \sqrt{\frac{n}{n^2}\V_{\theta^*}(X_i)}
=\sqrt{{\theta^*}(1-{\theta^*})/n} \ .
\]
\end{example}

Another reasonable property of an estimator is that it converge to the ``true'' parameter $\theta^*$ -- here ``true'' means the supposedly fixed and possibly unknown $\theta^*$, as we gather more and more IID data from a $\theta^*$-specified DF $F(x; \theta^*)$.  This property is stated precisely next.
\begin{definition}[Asymptotic Consistency of a Point Estimator]\label{D:Consistency}
A point estimator $\widehat{\Theta}_n$ of $\theta^* \in \BB{\Theta}$ is said to be {\bf asymptotically consistent} if:
\[
\boxed{
\widehat{\Theta}_n \overset{P}{\longrightarrow} \theta^*
} \qquad \text{i.e., for any real $\epsilon > 0$,} \quad
\boxed{
\lim_{n \to \infty} \P (| \widehat{\Theta}_n - \theta^* | > \epsilon) = 0
} \ .
\]
\end{definition}

\begin{definition}[Mean Squared Error (MSE) of a Point Estimator]\label{D:MSE}
Often, the quality of a point estimator $\widehat{\Theta}_n$ of $\theta^* \in \BB{\Theta}$ is assessed by the {\bf mean squared error} or $\mathsf{MSE}_n$ defined by:
\begin{equation}\label{E:MSE}
\boxed{
\mathsf{MSE}_n=\mathsf{MSE}_n(\widehat{\Theta}_n) := \E_{\theta^*} \left((\widehat{\Theta}_n-\theta^*)^2 \right)
= \int_{\Xz} (\widehat{\Theta}_n(x)-\theta^*)^2 \, dF(x;\theta^*)
} \ .
\end{equation}
\end{definition}

The following proposition shows a simple relationship between the mean square error, bias and variance of an estimator $\widehat{\Theta}_n$ of $\theta^*$.
\begin{prop}[The $\sqrt{\mathsf{MSE}_n}:\mathsf{se}_n:\mathsf{bias}_n$--Sided Right Triangle of an Estimator]
Let $\widehat{\Theta}_n$ be an estimator of $\theta^* \in \BB{\Theta}$.  Then:
\begin{equation}\label{E:RMseSeBiasTriangle}
\boxed{
\mathsf{MSE}_n(\widehat{\Theta}_n)
= (\mathsf{se}_n(\widehat{\Theta}_n))^2 + (\mathsf{bias}_n(\widehat{\Theta}_n))^2
} \ .
\end{equation}
{\scriptsize
\begin{proof}
\begin{eqnarray}
& & LHS \notag \\
&=& \mathsf{MSE}_n(\widehat{\Theta}_n) \notag \\
&:=& \E_{\theta^*} \left((\widehat{\Theta}_n-\theta^*)^2 \right), \qquad  \text{by definition of $\mathsf{MSE}_n$ \eqref{E:MSE}} \notag \\
&=& \E_{\theta^*} \left( \left( \ \underset{A}{\underbrace{\left(\widehat{\Theta}_n -\E_{\theta^*}(\widehat{\Theta}_n) \right)}} + \underset{B}{\underbrace{\left(\E_{\theta^*}(\widehat{\Theta}_n) -\theta^* \right)}} \ \right)^2 \right), \qquad \text{by subtracting and adding the constant $\E_{\theta^*}(\widehat{\Theta}_n)$} \notag \\
&=& \E_{\theta^*} \left( \underset{A^2}{\underbrace{{\left(\widehat{\Theta}_n -\E_{\theta^*}(\widehat{\Theta}_n) \right)}^2}} + \underset{2AB}{\underbrace{2  {\left(\widehat{\Theta}_n -\E_{\theta^*}(\widehat{\Theta}_n) \right)} {\left(\E_{\theta^*}(\widehat{\Theta}_n) -\theta^* \right)}}} + \underset{B^2}{\underbrace{{\left(\E_{\theta^*}(\widehat{\Theta}_n) -\theta^* \right)}^2}}  \right), \qquad \text{$\because \ (A+B)^2=A^2+2AB+B^2$} \notag \\
&=& \E_{\theta^*} \left( {\left(\widehat{\Theta}_n -\E_{\theta^*}(\widehat{\Theta}_n) \right)}^2 \right) +
\E_{\theta^*} \left( 2  {\left(\widehat{\Theta}_n -\E_{\theta^*}(\widehat{\Theta}_n) \right)} {\left(\E_{\theta^*}(\widehat{\Theta}_n) -\theta^* \right)} \right) +
\E_{\theta^*} \left( {\left(\E_{\theta^*}(\widehat{\Theta}_n) -\theta^* \right)}^2  \right), %\quad \text{taking $\E_{\theta^*}(\cdot)$ of $A^2$, $2AB$ and $B^2$}
\notag \\
&=& \E_{\theta^*} \left( {\left(\widehat{\Theta}_n -\E_{\theta^*}(\widehat{\Theta}_n) \right)}^2 \right) +
\underset{C}{\underbrace{2 {\left(\E_{\theta^*}(\widehat{\Theta}_n) -\theta^* \right)}}} \underset{D}{\underbrace{ \E_{\theta^*} \left(  {\left(\widehat{\Theta}_n -\E_{\theta^*}(\widehat{\Theta}_n) \right)} \right)}} +
\E_{\theta^*} \left( {\left(\E_{\theta^*}(\widehat{\Theta}_n) -\theta^* \right)}^2  \right), \quad \text{$\because \ C$
% := 2 {\left(\E_{\theta^*}(\widehat{\Theta}_n) -\theta^* \right)}$
is constant} \notag \\
&=& \E_{\theta^*} \left( {\left(\widehat{\Theta}_n -\E_{\theta^*}(\widehat{\Theta}_n) \right)}^2 \right) +
0 +
\E_{\theta^*} \left( {\left(\E_{\theta^*}(\widehat{\Theta}_n) -\theta^* \right)}^2  \right), \qquad \text{$\because \ D := \E_{\theta^*} \left(  {\left(\widehat{\Theta}_n -\E_{\theta^*}(\widehat{\Theta}_n) \right)} \right)=\E_{\theta^*}(\widehat{\Theta}_n ) - \E_{\theta^*}(\widehat{\Theta}_n) = 0$} \notag \\
&=& \V_{\theta^*}(\widehat{\Theta}_n)+
\E_{\theta^*} \left( {\left(\E_{\theta^*}(\widehat{\Theta}_n) -\theta^* \right)}^2  \right), \qquad \text{$\because \ \V_{\theta^*}(\widehat{\Theta}_n) := \E_{\theta^*} \left( {\left(\widehat{\Theta}_n -\E_{\theta^*}(\widehat{\Theta}_n) \right)}^2 \right)$, by definition of variance} \notag \\
&=& \left( \sqrt{\V_{\theta^*}(\widehat{\Theta}_n)} \right)^2+
\E_{\theta^*} \left( {\left( \mathsf{bias}_n(\widehat{\Theta}_n) \right)}^2  \right), \qquad \text{$\because \  \mathsf{bias}_n(\widehat{\Theta}_n) = \E_{\theta^*}(\widehat{\Theta}_n) -\theta^* $, by definition of $\mathsf{bias}_n$ of an estimator $\widehat{\Theta}_n$} \notag \\
&=&  \left(  \mathsf{se}_n(\widehat{\Theta}_n) \right)^2 +
\E_{\theta^*} \left( {\left( \mathsf{bias}_n(\widehat{\Theta}_n) \right)}^2  \right) , \qquad
\text{$\because \ \mathsf{se}_n(\widehat{\Theta}_n) := \sqrt{\V_{\theta^*}(\widehat{\Theta}_n)}$, by definition \eqref{E:se}} \notag \\
&=&  \left(  \mathsf{se}_n(\widehat{\Theta}_n) \right)^2 +
 {\left( \mathsf{bias}_n(\widehat{\Theta}_n) \right)}^2, \qquad
\text{$\because \  \mathsf{bias}_n(\widehat{\Theta}_n) = \E_{\theta^*}(\widehat{\Theta}_n) -\theta^*$ and $\left( \mathsf{bias}_n(\widehat{\Theta}_n) \right)^2$ are constants.} \notag \\
&=& RHS \notag
\end{eqnarray}
\end{proof}
}
\end{prop}

\begin{prop}[Asymptotic consistency of a point estimator]\label{P:AsympConsistencyUnbiasedSE0}
Let $\widehat{\Theta}_n$ be an estimator of $\theta^* \in \BB{\Theta}$.  Then, if $\mathsf{bias}_n(\widehat{\Theta}_n) \to 0$ and $\mathsf{se}_n(\widehat{\Theta}_n) \to 0$ as $n \to \infty$, the estimator $\widehat{\Theta}_n$ is asymptotically consistent:
\[
\widehat{\Theta}_n \overset{P}{\longrightarrow} \theta^* \ .
\]
{\scriptsize
\begin{proof}
If $\mathsf{bias}_n(\widehat{\Theta}_n) \to 0$ and $\mathsf{se}_n(\widehat{\Theta}_n) \to 0$, then by \eqref{E:RMseSeBiasTriangle}, $\mathsf{MSE}_n(\widehat{\Theta}_n) \to 0$, i.e.~that $\E_{\theta^*}\left( (\widehat{\Theta}_n-\theta^*)^2 \right) \to 0$.  This type of convergence of the RV $\widehat{\Theta}_n$ to the $Point~Mass(\theta^*)$ RV as $n \to \infty$ is called convergence in {\bf quadratic mean} or {\bf convergence in} $\B{L_2}$ and denoted by $\widehat{\Theta}_n \overset{qm}{\longrightarrow} \theta^*$.  Convergence in quadratic mean is a stronger notion of convergence than convergence in probability, in the sense that
\[
\E_{\theta^*}\left( (\widehat{\Theta}_n-\theta^*)^2 \right) \to 0 \quad \text{ or } \quad \widehat{\Theta}_n \overset{qm}{\longrightarrow} \theta^* \implies \widehat{\Theta}_n \overset{P}{\longrightarrow} \theta^* \ .
\]
Thus, if we prove the above implication we are done with the proof of our proposition.  To show that convergence in quadratic mean implies convergence in probability for general sequence of RVs $X_n$ converging to an RV $X$, we first assume that $X_n \overset{qm}{\longrightarrow} X$.
Now, fix any $\epsilon>0$.  Then by Markov's inequality \eqref{E:MarkovNeq},
\[
\P(|X_n-X|>\epsilon) = \P(|X_n-X|^2>\epsilon^2) \leq \frac{\E(|X_n-X|^2)}{\epsilon^2} \to 0 \ ,
\]
and we have shown that the definition of convergence in probability holds provided convergence in quadratic mean holds.
\end{proof}
}
\end{prop}
We want our estimator to be unbiased with small standard errors as the sample size $n$ gets large.  The {\bf point estimator} $\widehat{\Theta}_n$ will then produce a {\bf point estimate} $\widehat{\theta}_n$:
$$\widehat{\Theta}_n((x_1,x_2,\ldots,x_n)) = \widehat{\theta}_n \in \BB{\Theta} \enspace ,$$ on the basis of the {\bf observed data} $(x_1,x_2,\ldots,x_n)$, that is close to the {\bf true parameter} $\theta^* \in \BB{\Theta}$.

\begin{example}[Asymptotic consistency of our Estimator of $\theta^*$]
Consider the sample mean estimator $\widehat{\Theta}_n := \overline{X}_n$ of $\theta^*$, from $X_1,X_2,\ldots,X_n \overset{\IID}{\sim} \bernoulli(\theta^*)$.  Since $\mathsf{bias}_n(\widehat{\Theta}_n)=0$ for any $n$ and
$\mathsf{se}_n=\sqrt{\theta^*(1-\theta^*)/n} \to 0$, as $n \to \infty$, by
\hyperref[P:AsympConsistencyUnbiasedSE0]{Proposition \ref*{P:AsympConsistencyUnbiasedSE0}},
$\widehat{\Theta}_n \overset{P}{\longrightarrow} \theta^*$.  That is $\widehat{\Theta}_n$ is an {\bf asymptotically consistent estimator} of $\theta^*$.
\end{example}

\subsection{Confidence Set Estimation}\label{S:ConfidenceSets}
As we saw in Section~\ref*{S:PointEstimation}, the point estimate $\widehat{\theta}_n$ is a ``single best guess'' of  the fixed and possibly unknown parameter $\theta^* \in \BB{\Theta}$.  However, if we wanted to make a statement about our confidence in an estimation procedure, then one possibility is to produce subsets from the parameter space $\BB{\Theta}$ called {\bf confidence sets} that ``engulf'' $\theta^*$ with a probability of at least $1-\alpha$.

Formally, an $1-\alpha$ {\bf confidence interval} for the parameter $\theta^* \in \BB{\Theta} \subset \Rz$, based on $n$ observations or data points $X_1,X_2,\ldots,X_n$, is an interval $C_n$ that is a function of the data:
\[
C_n := [\underline{C}_{\, n}, \overline{C}_{\, n}]
= [\underline{C}_{\, n}(X_1,X_2,\ldots,X_n), \overline{C}_{\, n}(X_1,X_2,\ldots,X_n)] \ ,
\]
such that:
\[
\P_{\theta^*} \left(  \theta^* \in C_n :=  [\underline{C}_{\, n}, \overline{C}_{\, n}] \right) \geq 1-\alpha \ .
\]
Note that the confidence interval $C_n := [\underline{C}_{\, n}, \overline{C}_{\, n}]$ is a two-dimensional RV or a random vector in $\Rz^2$ that depends on the two statistics $\underline{C}_{\, n} (X_1,X_2,\ldots,X_n) $ and $\overline{C}_{\, n} (X_1,X_2,\ldots,X_n) $, as well as $\theta^*$, which in turn determines the distribution of the data $(X_1,X_2,\ldots,X_n)$.  In words, $C_n$ engulfs the true parameter $\theta^* \in \BB{\Theta}$ with a probability of at least $1-\alpha$.  We call $1-\alpha$ as the {\bf coverage} of the confidence interval $C_n$.

Formally, a $1-\alpha$ {\bf confidence set} $C_n$ for a vector-valued $\theta^* \in \BB{\Theta} \subset \Rz^k$ is any subset of $\BB{\Theta}$ such that $\P_{\theta^*}( \theta^* \in C_n) \geq 1-\alpha$.  The typical forms taken by $C_n$ are $k$-dimensional boxes or hyper-cuboids, hyper-ellipsoids and subsets defined by inequalities involving level sets of some estimator of $\theta^*$.

Typically, we take $\alpha=0.05$ because we are interested in the $1-\alpha=0.95$ or $95\%$ confidence interval/set $C_n \subset \BB{\Theta}$ of $\theta^* \in \BB{\Theta}$ from an estimator $\widehat{\Theta}_n$ of $\theta^*$.

Let us look at an example that makes use of the CLT next (Exercise in Prob.~Theor.I).
\begin{example}[Errors in computer code (Wasserman03, p.~78)]\label{EXCLTPoisson}
Suppose the collection of RVs $X_1,X_2, \ldots, X_n$ model the number of errors in $n$ computer programs named $1,2,\ldots,n$, respectively.  Suppose that the RV $X_i$ modelling the number of errors in the $i^{\text{th}}$ program is the $Poisson(\lambda^*=5)$ for any $i=1,2,\ldots,n$.  Also suppose that they are independently distributed.  In short, we suppose that:
\[
X_1,X_2,\ldots,X_n \overset{\IID}{\sim} \poisson(\lambda^*=5) \ .
\]
Suppose we have $n=125$ programs and want to make a probability statement about $\overline{X}_n$ which is the average number of errors per program out of these $125$ programs.  Since $\E(X_i) = \lambda^*=5$ and $\V(X_i)=\lambda^*=5$, we may want to know how often our sample mean $\overline{X}_{125}$ differs from the expectation of $5$ errors per program.  Using the CLT, we can approximate $\P(\overline{X}_n < 5.5)$, for instance, as follows:
\begin{eqnarray}
\P(\overline{X}_n < 5.5)
&=& \P \left( \frac{\sqrt{n}(\overline{X}_n - \E(X_1))}{\sqrt{\V(X_1)}} < \frac{\sqrt{n}(5.5-\E(X_1))}{\sqrt{\V(X_1)}} \right) \notag \\
&\approxeq& \P \left( Z < \frac{\sqrt{n}(5.5-\lambda^*)}{\sqrt{\lambda^*}} \right) \qquad \text{{\scriptsize [by CLT, and $\E(X_1)=\V(X_1)=\lambda^*$]}} \notag \\
&=& \P \left( Z < \frac{\sqrt{125}(5.5-5)}{\sqrt{5}} \right) \qquad \text{{\scriptsize [Since, $\lambda^*=5$ and $n=125$ in this Example]}} \notag \\
&=& \P(Z \leq 2.5) = \Phi(2.5) =  \int_{- \infty}^{2.5} \left( \frac{1}{\sqrt{2 \pi}} \ \exp \left( \frac{-x^2}{2} \right) \right) dx \approxeq 0.993790334674224 \ . \notag
\end{eqnarray}
To obtain the final number in this approximation, we need the following:
\begin{labwork}
The numerical approximation of $\Phi(2.5)$ was obtained via the call shown below to our $\erf$-based {\tt NormalCdf} function from \ref*{Mf: NormalCdfPdf}.  We could have also found it from a pre-computed Table for $\Phi(x)$.
\begin{VrbM}
>> format long
>> disp(NormalCdf(2.5,0,1))
   0.993790334674224
\end{VrbM}
\end{labwork}
\end{example}

The CLT says that if $X_1,X_2,\ldots \overset{\IID}{\sim} X_1$ then $Z_n := \sqrt{n}(\overline{X}_n-\E(X_1))/\sqrt{\V(X_1)}$ is approximately distributed as $\normal(0,1)$.  In \hyperref[EXCLTPoisson]{Example \ref*{EXCLTPoisson}}, we knew $\sqrt{\V(X_1)}$ since we assumed knowledge of $\lambda^*=5$.   However, in general, we may not know $\sqrt{\V(X_1)}$.  The next proposition says that we may estimate $\sqrt{\V(X_1)}$ using the sample standard deviation $S_n$ of $X_1,X_2,\ldots,X_n$, according to \eqref{E:SampleStdDevRV}, and still make probability statements about the sample mean $\overline{X}_n$ using a $\normal$ distribution, {\bf provided $\mathbf{n}$ is not too small}, for e.g.~$n \geq 30$.
\begin{prop}[CLT based on Sample Variance]
Let $X_1,X_2,\ldots \overset{\IID}{\sim} X_1$ and suppose $\E(X_1)$ and $\V(X_1)$ exists, then:
\begin{equation}\label{E:CLTApproxSn}
\frac{\sqrt{n} \left( \overline{X}_n - \E(X_1) \right)}{S_n} \rightsquigarrow \normal(0,1) \ .
\end{equation}
\end{prop}

The following property of an estimator makes it easy to obtain confidence intervals.
\begin{definition}[Asymptotic Normality of Estimators]
An estimator $\widehat{\Theta}_n$ of a fixed and possibly unknown parameter $\theta^* \in \BB{\Theta}$ is {\bf asymptotically normal} if:
\begin{equation}\label{E:AsymptoticNormalEstimator}
\frac{\widehat{\Theta}_n - \theta^*}{\mathsf{se}_n} \rightsquigarrow \normal(0,1) \ .
\end{equation}
That is, $\widehat{\Theta}_n \rightsquigarrow \normal(\theta^*,\mathsf{se}_n^2)$.  By a further estimation of $\mathsf{se}_n := \sqrt{\V_{\theta^*}(\widehat{\Theta}_n)}$ by $\widehat{\mathsf{se}}_n$, we can see that $\widehat{\Theta}_n \rightsquigarrow \normal(\theta^*,\widehat{\mathsf{se}}_n^2)$ on the basis of \eqref{E:CLTApproxSn}.
\end{definition}

\begin{prop}[Normal-based Asymptotic Confidence Interval]\label{P:NormalBasedAsympCI}
Suppose an estimator $\widehat{\Theta}_n$ of  parameter $\theta^* \in \BB{\Theta} \subset \Rz$ is asymptotically normal:
\[
\widehat{\Theta}_n \rightsquigarrow \normal(\theta^*,\widehat{\mathsf{se}}_n^2) \ .
\]
Let the RV $Z \sim \normal(0,1)$ have DF $\Phi$ and inverse DF $\Phi^{-1}$.  Let:
\[
z_{\alpha/2}=\Phi^{-1}(1-(\alpha/2)), \quad \text{ that is, } \quad \P(Z>z_{\alpha/2})=\alpha/2 \ \text{ and } \ \P(-z_{\alpha/2} < Z < z_{\alpha/2}) = 1-\alpha \ .
\]
Then:
\[
\P_{\theta^*}(\theta^* \in C_n)  = \P \left( \theta^* \in [\widehat{\Theta}_n - z_{\alpha/2} \widehat{\mathsf{se}}_n, \widehat{\Theta}_n + z_{\alpha/2} \widehat{\mathsf{se}}_n] \right) \to 1-\alpha \ .
\]
Therefore:
\[
C_n := [\underline{C}_{\, n}, \overline{C}_{\, n}]
= [\widehat{\Theta}_n - z_{\alpha/2} \widehat{\mathsf{se}}_n, \widehat{\Theta}_n + z_{\alpha/2} \widehat{\mathsf{se}}_n]
\]
is the $1-\alpha$ Normal-based asymptotic confidence interval that relies on the asymptotic normality of the estimator $\widehat{\Theta}_n$ of $\theta^* \in \BB{\Theta} \subset \Rz$.
{\scriptsize
\begin{proof}
Define the centralised and scaled estimator as $Z_n := (\widehat{\Theta}_n-\theta^*)/\widehat{\mathsf{se}}_n$.  By assumption, $Z_n \rightsquigarrow Z \sim \normal(0,1)$.  Therefore,
\begin{eqnarray}
\P_{\theta^*}(\theta^* \in C_n)
&=& \P_{\theta^*} \left( \theta^* \in [\widehat{\Theta}_n - z_{\alpha/2} \widehat{\mathsf{se}}_n, \widehat{\Theta}_n + z_{\alpha/2} \widehat{\mathsf{se}}_n] \right) \notag \\
&=& \P_{\theta^*} \left( \widehat{\Theta}_n - z_{\alpha/2} \widehat{\mathsf{se}}_n \leq \theta^* \leq  \widehat{\Theta}_n + z_{\alpha/2} \widehat{\mathsf{se}}_n \right) \notag \\
&=& \P_{\theta^*} \left(  - z_{\alpha/2} \widehat{\mathsf{se}}_n \leq \widehat{\Theta}_n - \theta^* \leq   z_{\alpha/2} \widehat{\mathsf{se}}_n \right) \notag \\
&=& \P_{\theta^*} \left(  - z_{\alpha/2}  \leq \frac{\widehat{\Theta}_n - \theta^*}{\widehat{\mathsf{se}}_n} \leq   z_{\alpha/2} \right) \notag \\
&\to& \P_{\theta^*} \left(  - z_{\alpha/2}  \leq Z \leq   z_{\alpha/2} \right) \notag \\
&=& 1-\alpha \notag
\end{eqnarray}
\end{proof}
}
\begin{figure}[htb]
\caption{Density and Confidence Interval of the Asymptotically Normal Point Estimator}
\vspace{4cm}
\end{figure}
For $95\%$ confidence intervals, $\alpha=0.05$ and $z_{\alpha/2}=z_{0.025}=1.96\approxeq 2$.  This leads to the {\bf approximate ${95\%}$ confidence interval} of $\widehat{\theta}_n \pm 2 \widehat{\mathsf{se}}_n$, where $\widehat{\theta}_n=\widehat{\Theta}_n(x_1,x_2,\ldots,x_n)$ and $x_1,x_2,\ldots,x_n$ are the data or observations of the RVs $X_1,X_2,\ldots,X_n$.
\end{prop}

\begin{example}[Confidence interval for $\theta^*$ from $n$ $\bernoulli(\theta^*)$ trials]\label{EX:EstimatePFromNIIDBernoulliTrials}
Let $X_1,X_2,\ldots,X_n \overset{\IID}{\sim} \bernoulli(\theta^*)$ for some fixed but unknown parameter $\theta^* \in \BB{\Theta}=[0,1]$.  Consider the following point estimator of $\theta^*$:
\[
\widehat{\Theta}_n((X_1,X_2,\ldots,X_n)) = n^{-1} \sum_{i=1}^n X_i \ .
\]
That is, we take the sample mean of the $n$ IID $\bernoulli(\theta^*)$ trials to be our point estimator of $\theta^*\in[0,1]$.  Then, we already saw that {\bf this estimator is unbiased}

We already saw that the standard error  $\mathsf{se}_n$ of this estimator is:
\[
\mathsf{se}_n=\sqrt{{\theta^*}(1-{\theta^*})/n} \ .
\]
Since ${\theta^*}$ is unknown, we obtain the estimated standard error $\widehat{\mathsf{se}}_n$ from the point estimate $\widehat{\theta}_n$ of ${\theta^*}$ on the basis of $n$ observed data points $x=(x_1,x_2,\ldots,x_n)$ of the experiment:
\[
\widehat{\mathsf{se}}_n = \sqrt{\widehat{\theta}_n(1-\widehat{\theta}_n)/n}, \quad \text{where, } \quad \widehat{\theta}_n=\widehat{\Theta}_n((x_1,x_2,\ldots,x_n))=n^{-1}\sum_{i=1}^n x_i \ .
\]
By the central limit theorem, $\widehat{\Theta}_n \rightsquigarrow \normal(\theta^*,\widehat{\mathsf{se}}_n)$, i.e.~$\widehat{\Theta}_n$ is asymptotically normal.  Therefore, an asymptotically (for large sample size $n$) approximate $1-\alpha$ normal-based confidence interval is:
\[
\widehat{\theta}_n \pm z_{\alpha/2} \widehat{\mathsf{se}}_n
= \widehat{\theta}_n \pm z_{\alpha/2} \sqrt{\frac{ \widehat{\theta}_n (1- \widehat{\theta}_n )}{n}}
:= \left[ \, \widehat{\theta}_n - z_{\alpha/2} \sqrt{\frac{ \widehat{\theta}_n (1- \widehat{\theta}_n )}{n}} \ , \ \widehat{\theta}_n + z_{\alpha/2} \sqrt{\frac{ \widehat{\theta}_n (1- \widehat{\theta}_n )}{n}} \, \right]
\]
We also saw that $\widehat{\Theta}_n$ is an {\bf asymptotically consistent estimator} of $\theta^*$  due to \hyperref[P:AsympConsistencyUnbiasedSE0]{Proposition \ref*{P:AsympConsistencyUnbiasedSE0}}.

The confidence Interval for the coin tossing experiment in \hyperref[EX:CoinTossing]{Example \ref*{EX:CoinTossing}} with the observed sequence of Bernoulli outcomes (Heads $\to 1$ and Tails $\to 0$) being $(1,0,0,0,1,1,0,0,1,0)$.  We estimated the probability $\theta^*$ of observing Heads with the {\bf unbiased, asymptotically consistent estimator} $\widehat{\Theta}_n((X_1,X_2,\ldots,X_n))=n^{-1}\sum_{i=1}^{n} X_i$ of $\theta^*$.  The point estimate of $\theta^*$ was:
\begin{eqnarray}
\widehat{\theta}_{10} = \widehat{\Theta}_{10}((x_1,x_2,\ldots,x_{10}))
&=&\widehat{\Theta}_{10}((1,0,0,0,1,1,0,0,1,0)) \notag \\
&=& \frac{1+0+0+0+1+1+0+0+1+0}{10}=\frac{4}{10}=0.40 \notag \ .
\end{eqnarray}
The normal-based confidence interval for $\theta^*$ may not be a valid approximation here with just $n=10$ samples.  Nevertheless, we will compute a $95\%$ normal-based confidence interval:
\[
C_{10}
= 0.40 \pm 1.96 \sqrt{\frac{0.40(1-0.40)}{10}}
=0.40 \pm 0.3036
=[0.0964, 0.7036]
\]
with a width of $0.6072$.  When I increased the sample size $n$ of the experiment from $10$ to $100$ by tossing the same coin another $90$ times, I discovered that a total of $57$ trials landed as Heads.  Thus my point estimate and confidence interval for $\theta^*$ are:
\[
\widehat{\theta}_{100} = \frac{57}{100} = 0.57 \qquad and \qquad
C_{100}
= 0.57 \pm 1.96 \sqrt{\frac{0.57(1-0.57)}{100}}
 = 0.57 \pm 0.0495
 =[0.5205, 0.6195]
\]
with a much smaller width of $0.0990$.  Thus our confidence interval shrank considerably from a width of $0.6072$ after an additional $90$ Bernoulli trials.  Thus, we can make the width of the confidence interval as small as we want by making the number of observations or sample size $n$ as large as we can.
\end{example}

\remove{
\subsection{Likelihood}\label{S:Likelihood}
We take a look at one of the most fundamental concepts in Statistics.

\begin{definition}[Likelihood Function]\label{D:LklFn}
Suppose $X_1,X_2,\ldots,X_n$ have joint density $f(x_1,x_2,\ldots,x_n; \theta)$ specified by parameter $\theta \in \BB{\Theta}$.  Let the observed data be $x_1,x_2,\ldots,x_n$.

The {\bf likeihood} function given by $L_n(\theta)$ is proportional to $f(x_1,x_2,\ldots,x_n; \theta)$, the joint probability of the data, with the exception that we see it as a function of the parameter:
\begin{equation}
L_n(\theta) := L_n(x_1,x_2,\ldots,x_n; \theta) = f(x_1,x_2,\ldots,x_n; \theta) \enspace .
\end{equation}
The likelihood function has a simple product structure when the observations are independently and identically distributed:
\begin{equation}
X_1,X_2,\ldots,X_n \overset{IID}{\sim} f(x;\theta) \implies
\boxed{
L_n(\theta) := L_n(x_1,x_2,\ldots,x_n;\theta) = f(x_1,x_2,\ldots,x_n; \theta) := \prod_{i=1}^n f(x_i ; \theta)
}
\enspace .
\end{equation}
The {\bf log-likelihood} function is defined by:
\begin{equation}
\boxed{
\ell_n(\theta) := \log(L_n(\theta))
} \enspace
\end{equation}
\end{definition}

\begin{example}[Likelihood of the IID $\bernoulli(\theta^*)$ experiment]
Consider our IID Bernoulli experiment:
$$
X_1,X_2,\ldots,X_n \overset{IID}{\sim} \bernoulli(\theta^*), \text{ with PDF } f(x;\theta)=\theta^x(1-\theta)^{1-x} \BB{1}_{\{0,1\}}(x) \enspace .
$$
Let us understand the likelihood function for one observation first.  There are two possibilities for the first observation.

If we only have one observation and it happens to be $x_1=1$, then our likelihood function is:
$$L_1(\theta)=L_1(x_1;\theta)
= f(x_1;\theta)
=\theta^{1}(1-\theta)^{1-1} \BB{1}_{\{0,1\}}(1)
=\theta (1-\theta)^0 1
=\theta \enspace
$$
If we only have one observation and it happens to be $x_1=0$, then our likelihood function is:
$$L_1(\theta)=L_1(x_1;\theta)
= f(x_1;\theta)
=\theta^{0}(1-\theta)^{1-0} \BB{1}_{\{0,1\}}(0)
=1 (1-\theta)^1 1
=1-\theta \enspace
$$
If we have $n$ observations $(x_1,x_2,\ldots,x_n)$, i.e.~a vertex point of the unit hyper-cube $\{0,1\}^n$, then our likelihood function is obtained by multiplying the densities:
\begin{eqnarray}
L_n(\theta) &:=& L_n(x_1,x_2,\ldots,x_n; \theta)  = f(x_1,x_2,\ldots,x_n ; \theta) \notag \\
&=& f(x_1 ; \theta) f(x_2 ; \theta) \cdots f(x_n ; \theta) := \prod_{i=1}^n f(x_i ; \theta) \notag \\
&=& \theta^{\sum_{i=1}^n x_i} (1-\theta)^ {n-\sum_{i=1}^n x_i} := \theta^{t_n} (1-\theta)^{n-t_n} \notag
\end{eqnarray}
In the last step, we have formally defined the following statistic of the data:
$$T_n(X_1,X_2,\ldots,X_n)=\sum_{i=1}^n X_i :  \Xz_n \rightarrow \Tz_n$$ with the corresponding realisation $t_n := T_n(x_1,x_2,\ldots,x_n)=\sum_{i=1}^n x_i \in \Tz_n$.
\end{example}

\begin{figure}[ht]
\caption{Data Spaces $\Xz_1=\{0,1\}$, $\Xz_2=\{0,1\}^2$ and $\Xz_3=\{0,1\}^3$ for one, two and three IID Bernoulli trials, respectively and the corresponding likelihood functions.\label{F:BernoulliSampleLkl}}
\centering   \makebox{\includegraphics[width=4.5in]{figures/BernoulliSampleLkl}}
\end{figure}

\begin{figure}[ht]
\caption{{\small $100$ realisations of $C_{10}, C_{100}, C_{1000}$ based on samples of size $n$ $=$ $10$, $100$ and $1000$ drawn from the $\bernoulli(\theta^*=0.5)$ RV as per \hyperref[Mf:BernoulliMLEConsistency]{Labwork~\ref*{Mf:BernoulliMLEConsistency}}.  The MLE $\widehat{\theta}_n$ (cyan dot) and the log-likelihood function (magenta curve) for each of the $100$ replications of the experiment for each sample size $n$ are depicted.  The approximate normal-based $95\%$ confidence intervals with blue boundaries are based on the exact $\mathsf{se}_n=\sqrt{\theta^*(1-\theta^*)/n}=\sqrt{1/4}$, while those with red boundaries are based on the estimated $\widehat{\mathsf{se}_n}=\sqrt{\widehat{\theta}_n(1-\widehat{\theta}_n)/n}$.  The fraction of times the true parameter $\theta^*=0.5$ was engulfed by the exact and approximate confidence interval (empirical coverage) over the $100$ replications of the experiment for each of the three sample sizes are given by the numbers after {\tt Cvrg.=} and {\tt $\sim$=}, above each sub-plot, respectively.}\label{F:BernoulliMLEConsistency}}
\begin{center}
\makebox{\includegraphics[width=4.50in]{figures/BernoulliMLEConsistency}}
\end{center}
\end{figure}

%Let us look at some examples of estimation.
}