Massachusetts Institute of Technology Lecture ?? 6.042J/18.062J: Mathematics for Computer Science November 15, 1996 Professors Nancy Lynch and Tom Leighton

Notes 20

1 Conditional Probability

In a lot of problems (including the one we looked at in the last lecture), we want to consider the probability that an event A happens, given that we know a certain other event B has happened. We write

P r(A | B)

to denote the probability of event A, given B.

There is a very simple formula for this, which can be viewed as follows. Once we know that event B has happened, we are in the subset of the sample space equal to B. We now want to know what fraction of this set consists of outcomes in A. Thus,

P r(A | B) = \frac{P r(A ∩ B)}{P r(B)}.

We will take this formula as the definition of the conditional probability of A given B.

Let’s do an example. Suppose a baseball team is playing a best-2-out-of-3 series. The probability it wins the first game is \frac{1}{2}. The probability of winning a game following a win in the previous game is \frac{2}{3}; the probability of winning following a loss is only \frac{1}{3}.

The question is: what is the probability the team wins the series if it wins the first game?

There are actually two types of conditioning going on here:

1. The probability of winning a game depends on the probability of having won the previous game.

2. We’re asked for the probability of winning the series, given that the team wins the first game.

We solve this problem in the following step-by-step way.

1. Draw the sample space. (See the figure below.)

2. Assign probabilities to the outcomes. These are all calculated using the same principle; let’s show we calculate the probability of the outcome (tree branch) W LL.

P r(W LL) = P r(Win 1st) · P r(Lose 2nd | Win 1st) · P r(Lose 3rd | Win 1st, Lose 2nd)
= \frac{1}{2} · \frac{1}{3} · \frac{2}{3}
= \frac{1}{9}

W L  W L  W L WL  W L 1/3 1/3  1/18 1/9 1/9 1/18

Figure 1: Why the Red Sox didn’t make it to the World Series

3. Now we identify the events. A is the event that the team wins the series, and B is the event that the team wins the first game.

4. Finally, the answer we want is P r(A | B). We compute this using the formula, and adding up the probabilities from the figure:

P r(A | B) = \frac{P r(A ∩ B)}{P r(B)} = \frac{\frac{1}{3} + \frac{1}{18}}{\frac{1}{3} + \frac{1}{9} + \frac{1}{18}} = \frac{\frac{7}{18}}{\frac{9}{18}} = \frac{7}{9}.

5. We can also answer the question “What is the probability that the series lasts three games?”. From the picture, we add the probabilities of all the series of length three. 1/18 + 1/9 + 1/9 + 1/18 = 1/3.

2 Another Example

Conditional probability can lead to some counter-intuitive answers; but things really do work out if you just follow the definition. Here’s a problem that always confuses people. It’s based on calculating the probability of an event A, given an event B that comes after it in time. In such cases, the conditional probability is often called the a posteriori probability — though it’s really the same thing mathematically.

Suppose I have two coins — one is a fair coin, which comes up heads or tails with probability \frac{1}{2} each, and the other is an unfair coin, which always comes up heads. Now I choose one of the two coins with probability \frac{1}{2} each, and I flip this chosen coin once. If it comes up heads, what is the probability that I picked the fair coin?

The trickiness here is that our intuitive notion of conditional probability does not always match the definition as P r(A ∩ B)/P r(B) — in this case, the notion of “causality” seems to go backwards. But in such cases, we always have to follow the definition.

As in the previous example, we solve this problem in the following steps.

1. Draw the sample space. (See the figure below.)

2. Assign probabilities to the events, again as in the previous example.

3. Define the relevant events. Here, A is the event that we choose the fair coin, and B is the event that the result is heads.

4.

P r(A | B) = \frac{P r(A ∩ B)}{P r(B)} = \frac{\frac{1}{4}}{\frac{1}{4} + \frac{1}{2}} = \frac{1}{3}.

fair unfair  H T H 1/4 1/4 1/2

Figure 2: Flipping Coins

The key point here is that we knew the probability of picking the fair coin was exactly \frac{1}{2}. Without this, we could not have computed P r(A | B).

For example, say that we didn’t know that we picked the fair coin with probability 1/2, we performed the experiment 100 times and it came up heads every time. What can we say about the probability of having picked the fair coin? The answer is “absolutely NOTHING!”. We can see this by drawing the sample space of the new experiment (see figure 3).

fair unfair  H T H p2^{−100} p(1 − 2^{−100}) 1 − p

Figure 3: Flipping Coins

P r(A | B) = \frac{p2^{−100}}{1−p+2^{−100}} = \frac{p}{(1−p)2^{−100}+1}

The probability of this event appears to be very small, but if p = 1 then P r(A | B) = 1.

3 Yet Another Example

Assume that 10% of the American polulation have disease X. You don’t know if you have the disease but there is a test. If you have the disease, there is a 10% chance that the test will come out negative. If you don’t have the disease there is a 30% chance that the test will come out positive. Now suppose that a random American is selected and tested. What is the probability that this person has the disease given that the test is positive?

Let A be the event that the person has the disease and B be the event that the test is positive. The probability that both events occur is 0.1 · 0.9 = 0.09 , so P r(A ∩ B) = 0.09. The probability that the test comes out positive is P r(B) = 0.1 · 0.9 + 0.9 · 0.3 = 0.36 (draw the sample space to convince yourself), so we have

P r(A | B) = \frac{.09}{.36} = \frac{1}{4}

Even if you test positive, you are not likely to have the disease, even though the test has low false rates. Indeed, P r( test is correct ) = .09 + .63 = .72. The test is accurate more than 70% of the time, but that does not mean that it is likely to be accurate when it is positive. This is because the vast majority of people don’t have the disease and thus there are more people that don’t have the disease but test positive (27%) than people who do have the disease and test positive (9%). It’s worth looking at the alternative test which always says no. This test is accurate 90% of the time, which is better than 72%!

4 Probability and Set Theory

Before discussing some more complicated probability problems, let’s develop some useful connections between probabiliy and set theory. Recall from the last lecture that

Definition 4.1 A sample space for a problem (or experiment) is a set consisting of all possible outcomes.

Definition 4.2 A probability space consists of a sample space S and a probability function P r : S → ℝ, such that

1. For all s ∈ S, 0 ≤ P r(S) ≤ 1.

2. ∑_{s∈S} P r(s) = 1.

Definition 4.3 An event is a subset of the sample space.

Definition 4.4 The probability of an event S′ is the sum of the probabilities of all outcomes in the event.

Using these definitions, we can establish some basic facts.

Theorem 4.5 Let A and B be events such that A ⊆ B. Then P r(A) ≤ P r(B).

Proof.

P r(A) = ∑_{s∈A} P r(s) ≤ ∑_{s∈B} P r(s) = P r(B).

For an event A, let Ā denote the complement of A.

Theorem 4.6 For any event A,

P r(Ā) = 1 − P r(A).

Proof.

P r(Ā) + P r(A) = ∑_{s∈A} P r(s) + ∑_{s∉A} P r(s)
= ∑_{s∈S} P r(s)
= 1

where the first equality follows from the definition of complementation and the second follows from the definition of a probability space.

Also this has a simple proof, often it’s much easier to calculate a probability by first calculating the probability of the complement and then subtracting from 1.

We can use the inclusion-exclusion principle to get probabilities of unions.

Theorem 4.7 For any events A and B, P r(A ∪ B) = P r(A) + P r(B) − P r(A ∩ B).

Proof.

P r(A ∪ B) = ∑_{s∈A∪B} P r(s)
= ∑_{s∈A} P r(s) + ∑_{s∈B} P r(s) − ∑_{s∈A∩B} P r(s)
= P r(A) + P r(B) − P r(A ∩ B).

Sometimes it’s hard to calculate the exact probabilities of unions and intersections. But the advantage of a result like Theorem ?? is that it allows us to give bounds on these probabilties.

Corollary 4.8 (Boole’s Inequality) For any events A and B,

P r(A ∪ B) ≤ P r(A) + P r(B).

Note that equality holds if and only if P r(A ∩ B) = 0. By induction, Boole’s Inequality can be easily extended to n events.

Theorem 4.9 For all n ≥ 1 and any events A_{1}, …, A_{n}, we have

P r(∪_{i=1}^{n}A_{i}) ≤ ∑_{i=1}^{n} P r(A_{i}).

Proof. We prove this by induction on n. For n = 1, we just have P r(A_{1}) ≤ P r(A_{1}). Now, assuming the theorem is true for n−1, we apply Boole’s Inequality followed by the induction hypothesis:

P r(∪_{i=1}^{n}A_{i}) ≤ ≤ P r(∪_{i=1}^{n−1}A_{i}) + P r(A_{n})
≤ ∑_{i=1}^{n−1} P r(A_{i}) + P r(A_{n})
= ∑_{i=1}^{n} P r(A_{i}).

As an example, let’s suppose you’re wiring up a circuit. Overall, you are supposed to make n connections, and for each connection, the probability that it is made incorrectly is p. What is the probability that any of the connections is made incorrectly?

Well, if we let A_{i} denote the event that connection i is made incorrectly, we are just asking for P r(∪_{i=1}^{n}A_{i}). Without knowing the relationship among these events, we can’t give an exact answer — it could be that messing up one connection will cause you to mess up others as well.

However, we can still gives bounds on this probability. Using Theorem ??, we know

P r(∪_{i=1}^{n}A_{i}) ≤ ∑_{i=1}^{n} P r(A_{i}) = np.

Also, since A_{1} ⊆ ∪_{i=1}^{n}A_{i}, and P r(A_{1}) = p, we can use Theorem ?? to conclude

P r(∪_{i=1}^{n}A_{i}) ≥ P r(A_{1}) = p.

So for example, if n = 10 and p = .01, we get the following bounds:

.01 ≤ P r(∪_{i=1}^{10}A_{i}) ≤ .10.

Now the event that the chip gets wired correctly is the just the complement of the event that a connection is made incorrectly; so by Theorem ??, the probability that the chip gets wired correctly in this example is between .9 and .99.