\documentclass[twoside,12pt]{article}

\setlength{\textwidth}{6.5 in}
\setlength{\textheight}{8.5 in}
\setlength{\topmargin}{0 in}
\setlength{\headheight}{0.25 in}
\setlength{\headsep}{0.25 in}
\setlength{\evensidemargin}{0 in}
\setlength{\oddsidemargin}{0 in}
\setlength{\parskip}{1ex}

\usepackage{graphics}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{latexsym}

\newenvironment{comment}[1]{}{}
\newenvironment{proof}{\noindent {\em Proof.}}{$\Box$}

\newtheorem{theorem}{Theorem}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{claim}[theorem]{Claim}

\newcommand{\beqn}{\begin{eqnarray*}}
\newcommand{\eeqn}{\end{eqnarray*}}
\newcommand{\pr}[1]{\Pr\{#1\}}

\begin{document}

\begin{center}
{\Large Lecture 4 - Conditional Probability} \\
6.042 - February 13, 2003 \\
\bigskip
\end{center}

\footnotetext{Copyright \copyright 2003, Charles Leiserson, Srinivas
Devadas, Eric Lehman.  All rights reserved.}

\section{Conditional Probability}

Suppose that we pick a random person in the world.  Everyone has an
equal chance of being picked.  Let $A$ be the event that the person is
an MIT student, and let $B$ be the event that the person lives in
Cambridge.  The situation is shown in Figure~\ref{fig:lec4-cond}.  Clearly,
both events $A$ and $B$ have low probability.  But what is the
probability that a person is an MIT student, {\em given} that the
person lives in Cambridge?

\begin{figure}[hpb]
\centerline{\resizebox{!}{2.0in}{\includegraphics{lec4-cond.eps}}}
\caption{What is the probability that a random person in the world
is an MIT student, given that the person is a Cambridge resident?}
\label{fig:lec4-cond}
\end{figure}

This is a conditional probability question.  It can be concisely
expressed in a special notation.  In general, $\pr{A \mid B}$ denotes
the probability of event $A$, given that event $B$ occurs.  In this example,
$\pr{A \mid B}$ is the probability that the person is an MIT student,
given that he or she is a Cambridge resident.

How do we compute $\pr{A \mid B}$?  Since we are {\em given} that the
person lives in Cambridge, all outcomes outside of event $B$ are
irrelevant; these irrelevant outcomes are diagonally shaded in the
figure.  Intuitively, $\pr{A \mid B}$ should be the fraction of
Cambridge residents that are also MIT students.  That is, the answer
should be the probability that the person is in set $A \cap B$
(horizontally shaded) divided by the probability that the person is in
set $B$.  This leads us to the following definition.

\begin{definition}
Let $A$ and $B$ be events such that $\pr{B} > 0$.  Then the {\em
conditional probability of $A$ given $B$} is denoted $\pr{A \mid B}$
and defined by:

\beqn
\pr{A \mid B} & = & \frac{\pr{A \cap B}}{\pr{B}}
\eeqn
\end{definition}

As an example, what is $\pr{A \mid B}$ when event $B$ is a subset of
event $A$?  This is the case if $A$ is the event that a random person
is an MIT student and $B$ is the event that the person is a $6.042$
student.  From the definition of conditional probability, we have:

\beqn
\pr{A \mid B}
	& = &	\frac{\pr{A \cap B}}{\pr{B}} \\
	& = &	\frac{\pr{B}}{\pr{B}} \\
	& = &	1
\eeqn

\noindent In words, the probability that a person is an MIT student,
given that they are a 6.042 student is 1.

Rearranging terms in this definition gives an equation called the {\em
product rule}:

\beqn
\pr{A \cap B} & = & \pr{B} \cdot \pr{A \mid B}
\eeqn

\noindent This is useful for calculating the probabilty that two
events {\em both} occur.  We've actually been using this formula on
the sly all along.  Exactly where will be clear shortly.

Conditional probabilities also play a central role in an important
equation called the {\em Total Probability Law}:

\beqn
\pr{A} & = &	\pr{A \mid B} \cdot \pr{B} +
		\pr{A \mid \overline{B}} \cdot \pr{\overline{B}}
\eeqn

\noindent Sometimes the probability of an event $A$ depends on whether
or not some other event $B$ occurs.  In such situations, one can
divide the analysis into two cases, when $B$ occurs and when it does
not, and then assemble the results to find the probability of $A$
using the Total Probability Law.

\section{The Halting Problem}

The {\em Halting Problem} is the canonical undecidable problem in
computation theory that was first introduced by Alan Turing in his
seminal 1936 paper, ''On Computable Numbers, with an Application to
the Entscheidungsproblem''.  The problem is to determine whether a
Turing machine halts on a given blah, blah, blah.  {\em Much more
importantly}, it is the name of the Laboratory for Computer Science
D-league hockey team.

In a best-of-three tournament, the Halting Problem wins the first game
with probability $\frac{1}{2}$.  In subsequent games, their
probability of winning is determined by the outcome of the previous
game.  If the Halting Problem won the previous game, then they are
energized by victory and win the current game with probability
$\frac{2}{3}$.  If they lost the previous game, then they are
demoralized by deefeat and win the current game with probablity
$\frac{1}{3}$.  What is the probability that the Halting Problem wins
the tournament, given that they win the first game?

We can cast this question in conditional probability terms.  Let $T$
be the event that the Halting Problem wins the tournament, and let $F$
be the event that they win the first game.  Our goal is then to
determine a conditional probability:

\beqn
\pr{T \mid F} & = & \frac{\pr{T \cap F}}{\pr{F}}
\eeqn

\begin{figure}[htbp]
\centerline{\resizebox{!}{3in}{\includegraphics{lec4-tree1.eps}}}
\caption{Tree diagram for the Halting Problem tournament.} \label{fig:lec4-tree1}
\end{figure}

We can evaluate the expression on the right by finding the
probabilities of events $T \cap F$ and $F$.  For this purpose, we can
use a tree diagram and the usual four-step method.  Such a tree
diagram is shown in Figure~\ref{fig:lec4-tree1}.  The edges are
annotated with victory probabilities based on the the Halting
Problem's psychology.  As usual, outcomes probabilities are computed
by multiplying edge probabilities along root-to-leaf paths.  The event
$T \cap F$ consists of two outcomes:

\[
\{\ WW,\ WLW\ \}
\]

\noindent The event $F$ consists of three outcomes:

\[
\{\ WW,\ WLW\, WLL \}
\]

\noindent The probability of an event is the some of the probabilities
of the outcomes it contains.  Therefore, we conclude:

\beqn
\pr{T \mid F}	& = &	\frac{\pr{T \cap F}}{\pr{F}} \\
		& = &	\frac{\frac{1}{3} + \frac{1}{18}}
			     {\frac{1}{3} + \frac{1}{18} + \frac{1}{9}} \\
		& = &	\frac{7}{9}
\eeqn

\noindent Thus, the Halting Problem is very likely to win the
tournament, given that they win the first game.

\subsection*{Why Tree Diagrams Work}

Before putting away the Halting Problem example, let's reconsider one
step.  {\em Why} is the probability of an outcome equal to the product
of the edge probabilities along the path from the root to that
outcome?  This is a fact we've relied upon since the first lecture,
but what is the mathematical justification?  For example, we concluded
above that:

\beqn
\pr{WW} & = & \frac{1}{2} \cdot \frac{2}{3} \\
	& = & \frac{1}{3}
\eeqn

\noindent The basis for this conclusion is actually the product rule.
We know that the Halting Problem wins the first game with probability
$\frac{1}{2}$.  Furthermore, we know that they win the second game
with probability $\frac{2}{3}$, given that they won the first game.
In other words, we know:

\beqn
\pr{\text{win first game}} & = & \frac{1}{2} \\
\pr{\text{win second game} \mid \text{win first game}} & = & \frac{2}{3}
\eeqn

\noindent Based on these two facts, we use the product rule to compute
the probability of outcome $WW$, the event that they win both the
first and second games:

\beqn
\lefteqn{\pr{\text{win first game} \cap \text{win second game}}}
		\hspace{0.5in} \\
	& = & \pr{\text{win first game}} \cdot
              \pr{\text{win second game} \mid \text{win first game}} \\
	& = & \frac{1}{2} \cdot \frac{2}{3} \\
	& = & \frac{1}{3}
\eeqn

This explains why tree diagrams work!  On each edge of a tree diagram,
we record the conditional probability of traversing that edge, given
that we reach the parent node.  The product rule implies that the
probability of an outcome is the product of the edge probabilities on
the path from the root to that outcome.

\section{A Posteriori Probability}

Probability theory sometimes allows us to compute the probability that
something happened, given observed consquences.  This is called an
{\em a posteriori} probability.  For example, suppose that I have two
coins.  One is normal and fair, but the other has heads on both sides.
Suppose that I pick a coin uniformly at random, flip it, and the
outcome is heads.  What is he probability that I picked the fair coin?

The sample space is worked out in the tree diagram in
Figure~\ref{fig:lec4-tree2}.  Let $F$ be the event that I picked the
fair coin, and let $H$ be the event that I flipped heads.  Our goal is
to compute a conditional probability:

\beqn
\pr{F \mid H} & = & \frac{\pr{F \cap H}}{\pr{H}}
\eeqn

\begin{figure}[htbp]
\centerline{\resizebox{!}{3in}{\includegraphics{lec4-tree2.eps}}}
\caption{Tree diagram for the two coins experiment.} \label{fig:lec4-tree2}
\end{figure}

As usual, we determine outcome probabilities by multiplying
edge probabilities along root-to-leaf paths.  The probability of the
events $F \cap H$ and $H$ are equal to the sum of the probabilities of
the outcomes they contain.  We find:

\beqn
\pr{F \mid H}
	& = &	\frac{\pr{F \cap H}}{\pr{H}} \\
	& = &	\frac{\frac{1}{4}}{\frac{1}{4} + \frac{1}{2}} \\
	& = &	\frac{1}{3}
\eeqn

People are sometimes troubled by a posteriori probabilities on a
philosophical level.  If I flip heads, then I've {\em already chosen}
either the normal coin or the double-headed coin.  Which coin I
flipped is at that point not an matter of probability; rather, it is
an already determined fact.

\section{The 3-Card Game}

Suppose that I have three cards.  One is red on both sides ($RR$), one
is blue on both sides ($BB$), and one is red on one side and blue on
the other ($RB$).  I jumble the cards in a hat, pick one out, and slap
it on the table.  The side you can see is blue.  What is the
probability that the other, hidden side is also blue?

One argument says that the hidden side is blue with probability
$\frac{1}{2}$.  After all, there are three equally-likely outcomes:
$RR$, $BB$, and $RB$.  We want the probability that the hidden face is
blue, given that the exposed face is blue.  Since the exposed face is
red, I could not have picked the $RR$ card out of the hat.  This
leaves two equally-likely possibilites: either I picked $BB$ or $RB$.
In one case, the hidden face is blue and in the other it is red.
Therefore, the hidden side is blue with probability $\frac{1}{2}$.

This argument is bogus.  The error goes back to the definition of an
outcome: {\em An outcome of an experiment consists of the total
information about the experiment after it has been performed,
including the values of all random choices.}

\begin{figure}[htbp]
\centerline{\resizebox{!}{3in}{\includegraphics{lec4-tree3.eps}}}
\caption{Tree diagram for the 3-card experiment.} \label{fig:lec4-tree3}
\end{figure}

\noindent There are two random choices in this experiment, not one: I
not only select a card from the hat, but also an orientation of that
card.  Thus, as the tree diagram in Figure~\ref{fig:lec4-tree3} shows,
there are six outcomes, not three.  Let $E$ be the event that the
exposed face is blue, and let $H$ be the event that the hidden face is
blue.  After computing outcome probabilities, we can determine the
desired conditional probability:

\beqn
\pr{H \mid E}
	& = &	\frac{\pr{H \cap E}}{\pr{E}} \\
	& = &	\frac{\frac{1}{6} + \frac{1}{6}}
                     {\frac{1}{6} + \frac{1}{6} + \frac{1}{6}} \\
	& = &	\frac{2}{3}
\eeqn

Therefore, if the exposed face is blue, then the hidden face is blue
with probability $\frac{2}{3}$.

\end{document}
