Massachusetts Institute of Technology Lecture 25 6.042J/18.062J: Mathematics for Computer Science 4 Dec 97 Professor Tom Leighton

Lecture Notes

This lecture is devoted to one rather general probability question. Let A_{1}, A_{2}, …, A_{N} be mutually independent events over the same sample space. Let the random variable T be the number of these events that occur. What is the probability that at least k events occur? That is, what is Pr(T ≥ k)?

This question comes up often. For example, suppose we want to know the probability that at least k heads come up in N tosses of a coin. Here A_{i} is the event that that the coin is heads on the i-th toss, T is the total number of heads, and Pr(T ≥ k) is the probability that at least k heads come up.

As a second example, suppose that we want the probability of a student answering at least k questions correctly on an exam with N questions. In this case, A_{i} is the event that the student answers the i-th question correctly, T is the total number of questions answered correctly, and Pr(T ≥ k) is the probability that the student answers at least k questions correctly.

There is an important difference between these two examples. The first example is a special case in that all events A_{i} have equal probability; that is, the coin is as likely to come up heads on one flip as on another. In particular, suppose that the coin comes up heads with probability p. Then all events A_{i} have probability p, and T has the now-familiar binomial distribution:

Pr(T = k) = \binom{n}{k}p^{k}(1 − p)^{n−k} = f_{n,p}(k)

We studied the binomial distribution extensively two weeks ago. Therefore, we already have an answer to the question posed above in the special case where all events A_{i} have equal probability.

In the second example, however, some exam questions might be more difficult than others. If question 1 is easier than question 2, then the probability of event A_{1} is greater than the probability of event A_{2}. This lecture focuses on questions of this more general type in which the events A_{i} may have different probabilities.

1 The Probability of at Least One Event

Before computing the probability that at least k events occur, we will consider another special case, namely the probability that at least one event occurs. That is, we will determine Pr(T ≥ 1).

1.1 Markov’s Inequality

We can obtain a weak bound on the probability that at least one of the events A_{1}, A_{2}, …, A_{N} occurs using Markov’s Theorem:

Pr(T ≥ 1) ≤ \frac{Ex(T)}{1}

Markov’s Theorem applies because the random variable T is non-negative.

All that remains is to compute Ex(T), the expected number of events that occur. We can do this by introducing an indicator variable T_{i} for each event A_{i}; that is, T_{i} = 1 if event A_{i} occurs, and T_{i} = 0 if event A_{i} does not occur. Then T, the total number of events that occur, is the sum of these indicator variables:

T = T_{1} + T_{2} + … + T_{N}

We can now compute the expected value of T as follows:

Ex(T) = Ex(T_{1} + T_{2} + … + T_{N})
= Ex(T_{1}) + Ex(T_{2}) + … + Ex(T_{N})
= Pr(A_{1}) + Pr(A_{2}) + … Pr(A_{N})

In the first step, we take the expected value of both sides of the preceding equation. The second step uses linearity of expectation. In the last step, we use the fact that the expected value of an indicator variable is the probability that the indicator is 1.

Substituting the expression for Ex(T) into the inequality from Markov’s Theorem gives the following bound on the probability that some event occurs.

Fact 1 Let A_{1}, A_{2}, …, A_{N} be events over the same sample space, and let the random variable T be the number of these events that occur. Then

Pr(T ≥ 1) ≤ ∑_{i=1}^{n} Pr(A_{i}).

This is a formula well worth remembering. In words, the probability that at least one event occurs is at most the sum of the event probabilities.

This result is weak for events with sizable probabilities. For example, if we flip a fair coin three times, then we can conclude that the probability of flipping heads at least once is at most \frac{1}{2} + \frac{1}{2} + \frac{1}{2} = \frac{3}{2}, which is not front-page news.

Nevertheless, Fact 1 is useful if the events A_{i} all have relatively low probability. A key advantage is that the events A_{i} need not be independent. (The only tools that we used in the derivation were Markov’s Theorem and linearity of expectation, and neither of these requires independence.) This makes the result easy to apply.

For example, suppose that we want to compute the probability of a meltdown at MIT’s nuclear plant on Albany Street. First, we enumerate all events A_{i} that could lead to a meltdown, e.g. operator error, earthquake, terrorist attack, meteor strike, etc. Next, we somehow determine the probability of each type of catastrophe. Then Fact 1 says that the probability of a meltdown is at most the sum of these event probabilities. Thankfully, we can totally ignore tricky independence issues such as, what if the terrorist team is wiped out by a meteor strike?

1.2 An Alternate Proof

We can obtain Fact 1 in another way. Recall Boole’s Inequality, which says that for any events A_{1}, A_{2}, …, A_{N}, we have:

Pr(A_{1} ∪ A_{2} ∪ … ∪ A_{N}) ≤ Pr(A_{1}) + Pr(A_{2}) + … + Pr(A_{N})

The left side of the inequality is the probability that at least one event A_{i} occurs, and the right side is the sum of the event probabilities. Therefore, Boole’s Inequality states that the probability that at least event occurs is at most the sum of the event probabilities. This is precisely the conclusion reached above!

1.3 A Lower Bound Using Independence

The previous method gives an upper bound on Pr(T ≥ 1). We can get a lower bound by computing the maximum of Pr(A_{i}) over all i.

Fact 2 Let A_{1}, A_{2}, …, A_{N} be events over the same sample space, and let the random variable T be the number of these events that occur. Then

Pr(T ≥ 1) ≥ max_{1≤i≤n} Pr(A_{i}).

The left side is Pr(A_{1}∪A_{2}∪…∪A_{n}). Since every event A_{i} is a subset of the union A_{1}∪A_{2}∪…∪A_{n}, we have Pr(A_{1} ∪A_{2} ∪…∪A_{n}) ≥ Pr(A_{i}) for all i. In particular, this holds for the i which maximizes Pr(A_{i}).

We can prove a stronger result if the events A_{i} are mutually independent. In this case, if we expect more than one event to occur, then almost certainly at least one event will occur.

Theorem 1.1 Let A_{1}, A_{2}, … A_{N} be independent events, and let T be the number of these events that occur. The probability that none of the events occur is at most e^{−Ex(T)}.

The proof of the theorem is surprisingly simple. The main trick is to use the fact that 1 − x ≤ e^{−x} for all x. This inequality follows from the Taylor expansion e^{−x} = 1 − x + \frac{x^{2}}{2!} − \frac{x^{3}}{3!} + . . ..

Proof.

Pr(T = 0) = Pr(Ā_{1} ∩ Ā_{2} ∩ … ∩ Ā_{N})
= ∏_{i=1}^{n} Pr(Ā_{i})

= ∏_{i=1}^{n} 1 − Pr(A_{i})

≤ ∏_{i=1}^{n} e^{− Pr(A_{i})}
= e^{− ∑_{i=1}^{n} Pr(A_{i})}
= e^{−Ex(T)}

In the first step, we reason that if zero of the events A_{i} occur, then all of the complementary events Ā_{i} must occur. Equality holds in the second step because the events are independent. The third step uses the fact that the probability of an event is one minus the probability of the complementary event. The fourth step relies on the inequality 1 − x ≤ e^{−x} mentioned above. The remainder is simplification.

Interestingly, this theorem does not depend on N, the number of events. The theorem gives the same bound whether there are 100 events each with probability 0.1 or 1000 events each with probability 0.01. In both cases, the expected number of events is 10, and so the probability of no event occurring is at most e^{−10} or about 1 in 22,000. (Though the bounds given by the theorem are equal, the actual probabilities are somewhat different in these two cases.)

This theorem can be interpreted as a sort of “Murphy’s Law”: if we expect some things to go wrong, then something probably will. For example, suppose that we are building a microprocessor, and the fabrication process is such that each transistor is faulty mutually independently with a probability of one in a million. This sounds good. However, microprocessors now contain about ten million transistors, so the expected number of faulty transistors is 10 per chip. Since we expect some things to go wrong, something probably will. In fact, Theorem 1.1 implies that the probability of a defect-free a chip is less than 1 in 22,000!

2 A Mind-Reading Trick

Theorem 1.1 is the basis for a neat card trick.

For this trick, every card has a “value”. The value of a number card less than 10 is just the number, and the value of a 10, face card, ace, or joker is 1. For example, 8♣ has value 8, Q♠ has value 1, and 10♢ has value 1.

A volunteer shuffles the deck and secretly chooses a number from 1 to 9. She deals this number of cards minus one from the top of the deck and looks at the new top card. For example, if she chooses the number 4, then she deals off 3 cards and looks at the 4th card.

She then takes the value of this top card, deals off that number of cards minus one, and again looks at the new top card. For example if she sees a 6♢, then she deals off 5 more cards and looks at the next one.

She repeats this procedure until she runs out of cards. “Her card” is the last card that she looked at to determine the number to deal off. For example, suppose that the deck is as follows:

Q♡ 4♠ \underline{2♠} J♣ \underline{K♠} \underline{5♡} 3♡ A♣ 7♢ 7♡ \underline{3♣} K♣

If the volunteer chooses 3 as her secret number, then she deals off 3 − 1 = 2 cards and looks at the 2♠. Since the value of this card is 2, she deals off 2 − 1 = 1 more card and looks at the K♠. This card has value 1, so she deals off 1 − 1 = 0 more cards and looks at the 5♡. Since this card has value 5, she deals of 5 − 1 = 4 more cards and looks at the 3♣. This card has value 3, so she tries to deal off 3 − 1 = 2 more cards. However, she then runs out of cards, so “her card” is 3♣.

After the volunteer completes the procedure, the magician is able to name her card!

The Secret

The secret is that the magician performs the same procedure as the volunteer, starting with the number 1. When he guesses “her card”, he is actually naming “his card”! Remarkably, the two are almost certain to be the same! The reason is that if the magician and volunteer ever land on the same card, then they will finish with the same card. In the example, the magician lands on the underlined sequence of cards:

\underline{Q♡} \underline{4♠} 2♠ J♣ K♠ \underline{5♡} 3♡ A♣ 7♢ 7♡ \underline{3♣} K♣

Even though the volunteer and the magician start with different numbers, they both coincidentally land on the 5♡. After this point, they both continue in the same way and both end up with 3♣.

All that remains is to prove that the magician and the volunteer are likely to land on the same card at some point. An exact analysis of this probability is difficult, but Theorem 1.1 gives some intuition.

Let A_{i} be the event that on his i-th jump the magician lands on a card also visited by the volunteer. If the volunteer’s next card is 1 away, then the magician has a \frac{1}{3} chance of landing on it (12 face cards, 3 10’s, and 2 jokers gives \frac{18}{54} = \frac{1}{3}). If the volunteer’s next card is more than 1 away, then the magician has a \frac{2}{27} chance of landing on it (4 correct number cards gives \frac{4}{54} = \frac{2}{27}).

Since the magician takes 10-15 jumps through the deck, the sum of these probabilities is rather high; that is, Ex(T) is around 3 or so. Consequently, Theorem 1.1 suggests that the magician is likely to eventually jump on a card visited by the volunteer. (As a ballpark estimate, the probability might be around 1 − e^{−3}.) In this case, the magician and volunteer end up with the same final card and the trick works.

Of course, Theorem 1.1 does not really apply, because the events A_{i} are not at all independent. For example, event A_{i} implies event A_{i+1}; once the magician and volunteer match, they always agree. This means that our probability estimates are bogus. However, computer tests suggest that for a full deck with jokers, the trick works more than 90% of the time.

3 The Chernoff Bound

We now address the general question posed at the beginning of lecture. Given mutually independent events A_{1}, A_{2}, …, A_{N}, what is the probability that at least k of the events occur? We will prove that the number of events that occur is almost always very close to the expectation. This result is called the Chernoff Bound.

For example, if we toss N coins, the expected number of heads is \frac{N}{2} heads. The Chernoff Bound implies that for sufficiently large N, the number of heads is almost always very close to \frac{N}{2} .

A nice feature of the Chernoff Bound is that we do not event need to know the probability of each event A_{i} or even the number of events N; rather, we need only the expected number of events that occur and the fact that the events are mutually independent.

The theorem is stated below in terms of Bernoulli variables instead of events. However, we can regard T_{i} as an indicator for the event A_{i}.

Theorem 3.1 (Chernoff Bound) Let T_{1}, T_{2}, …, T_{N} be mutually independent Bernoulli variables, and let T = T_{1} + T_{2} + … + T_{N}. Then for all c ≥ 1, we have

Pr(T ≥ cEx(T)) ≤ e^{−αcEx(T)}

where α = ln c + \frac{1}{c} − 1.

The formula for α is a little awkward. The situation is simpler when c = e = 2.718 …, as in the following corollary.

Corollary 3.2 Let T_{1}, T_{2}, …, T_{N} be mutually independent Bernoulli variables, and let T = T_{1} + T_{2} + … + T_{N}.

Pr(T ≥ eEx(T)) ≤ e^{−Ex(T)}

Proof. Setting c = e in the Chernoff Bound gives:

Pr(T ≥ eEx(T)) ≤ e^{−(ln e+ \frac{1}{e} −1)eEx(T)}
= e^{−Ex(T)}

We will prove the Chernoff Bound shortly. First, let’s see an example of how it is used.

3.1 Pick 4

There is a lottery game called Pick 4. In this game, each player picks 4 digits, defining a number in the range 0 to 9999. A winning number is drawn each week. The players who picked the winning number win some cash. A million people play the lottery, so the expected number of winners each week is \frac{1}{10,000} · 1, 000, 000 = 100. However, on some lucky day thousands of people might all pick the winning number, costing the lottery operators loads of money. How likely is this?

Assume that all players pick numbers uniformly and mutually independently. Let T_{i} be an indicator variable for the event that the i-th player picks the winning number. Let T = T_{1} + T_{2} + … + T_{N}. Then T is the total number of players that pick the winning number. As noted above, an average of 100 people win each week, so Ex(T) = 100. We can use Corollary 3.2 to bound the probability that number of winners is greater than 272 as follows:

Pr(T ≥ 272) ≤ Pr(T ≥ eEx(T))
≤ e^{−Ex(T)}
= e^{−100}

The probability of 272 or more people winning is absurdly small! It appears that the lottery operators should not worry that ten thousand people will pick correctly one day!

But there is a catch. The assumption that people choose Pick 4 numbers uniformly and mutually independently is empirically false; people choose certain “favorite” numbers far more frequently than others.

Chernoff used this fact in devising a scheme to actually make money on the lottery. In this case, a fraction of all money taken in by the lottery was divided up equally among the winners. A bad strategy would be to pick a popular number. Then, even if you pick the winning number, you must share the cash with many other players. A better strategy is to pick a lot of unpopular numbers. You are just as likely to win with an unpopular number, but will not have to share with anyone. Chernoff found that peoples’ picks were so highly correlated that he could actually turn a 7% profit by picking unpopular numbers!

3.2 Proof of the Chernoff Bound

The Chernoff Bound has the most difficult proof of the course. The intuition is in the first two steps; the rest is algebraic chicanery.

Pr(T ≥ cEx(T)) = Pr(c^{T} ≥ c^{cEx(T)})
≤ \frac{Ex(c^{T})}{c^{cEx(T)}}

The first step is a shocker; we raise both sides of the inequality in the probability to the power c. Since the new inequality describes the same event as the old, the probabilities are the same. The second step uses Markov’s Theorem.

Recall that Markov’s Theorem sometimes gives weak bounds and sometimes gives tight bounds. The motivation for the first step is to alter the distribution of the random variable T to hit the “sweet spot” of Markov’s Theorem. That is, Markov’s Theorem gives a tighter bound on the random variable c^{T} than on the original variable T.

All that remains is to evaluate Ex(c^{T}). This leads to a lot of algebra.

Ex(c^{T}) = Ex(c^{T_{1}+T_{2}+...+T_{N}})
= Ex(c^{T_{1}}) · Ex(c^{T_{2}}) ⋯ Ex(c^{T_{N}})
= ∏_{i=1}^{N} Ex(c^{T_{i}})

In the first step, we expand T according to its definition. The second step uses linearity of expectation, and in the third step we shift from … notation to product notation. To simplify the product, we can rewrite Ex(c^{T_{i}}) as follows.

Ex(c^{T_{i}}) = c^{1} Pr(T_{i} = 1) + c^{0} Pr(T_{i} = 0)
= c^{1} Pr(T_{i} = 0) + c^{0}(1 − Pr(T_{i} = 1))
= 1 + (c − 1) Pr(T_{i} = 0)
≤ e^{(c−1) Pr(T_{i}=1)}
= e^{(c−1)Ex(T_{i})}

The first step uses the definition of expected value. The second step uses the fact that the probability of an event is one minus the probability of the complement. The third step is simplification. The fourth line uses the inequality 1+x ≤ e^{x} which follows from the Taylor series e^{x} = 1+x+\frac{x^{2}}{2!} +\frac{x^{3}}{3!} +. . .. The last line relies on the fact that the expected value of a Bernoulli variable is equal to the probability that the variable is 1.

We can now substitute this bound on Ex(c^{T_{i}}) into the product for Ex(c^{T}). The motivation for the approximation 1 + x ≤ e^{x} used above is that terms of the form e^{x} are easy to work with inside a product.

Ex(c^{T}) ≤ ∏_{i=1}^{N} e^{(c−1)Ex(T_{i})}
= e^{∑_{i=1}^{N}(c−1)Ex(T_{i})}
= e^{(c−1) ∑_{i=1}^{N} Ex(T_{i})}
= e^{(c−1)Ex(∑_{i=1}^{N} T_{i})}
= e^{(c−1)Ex(T)}

In the first step, we substitute the bound on Ex(c^{T_{i}}) into the expression for Ex(c^{T}). The second equality follows from an exponent rule. In the third step, we pull the constant (c − 1) out of the summation. The fourth step uses linearity of expectation, and the last step uses the definition of the random variable T.

Now we can substitute this bound on Ex(c^{T}) into the expression derived from Markov’s Theorem. All of the remaining steps are simplifications.

Pr(T ≥ cEx(T)) = \frac{e^{(c−1)Ex(T)}}{c^{cEx(T)}}
= e^{(c−1)Ex(T)−c ln cEx(T)}
= e^{−(ln c+ \frac{1}{c} −1)cEx(T)}
= e^{−αcEx(T)}  where α = ln c + \frac{1}{c} − 1

4 A Phone Network Problem

Suppose that there is a phone network that handles a billion calls a day. Some of these call are routed through a certain switch. The exact number of calls passing through this switch is somewhat random and fluctuates over time, but on average the switch handles a million calls a day.

Our problem is to set the capacity of the switch; that is, we must determine the number of calls that a switch is able to handle in a day. If we make the capacity too small, then some phone calls will not go through. On the other hand, if we make the capacity too large, then we are wasting money. Of course, we can not rule out a freak situation in which a huge number of calls are all coincidentally routed through the same switch, thus overloading it. However, we would like to guarantee that a switch is rarely overloaded.

Assume that each call has some probability of passing through a particular switch. In particular, let T_{i} be an indicator variable for the event that the i-th call passes through the switch. That is, T_{i} = 1 if the call is routed through the switch, and T_{i} = 0 if the call does not pass through the switch. Then the total call load on the switch is T = T_{1} + T_{2} + … + T_{N}. We do not know the exact probability that the switch handles the i-th call, but we are given that the switch handles an average of a million calls a day; that is, Ex(T) = 1, 000, 000.

We will make the crucial assumption that the random variables T_{i} are mutually independent; that is, calls do or do not pass through the switch mutually independently.

4.1 How to Build One Switch

We can now compute the probability that the load on a switch fluctuates upwards by 1% due to the randomness of calling patterns. Substituting c = 1.01 and Ex(T) = 1, 000, 000 into the Chernoff Bound gives:

Pr(particular switch overloaded) = Pr(T ≥ 1.01 · 1, 000, 000)
≤ e^{−1.01(ln 1.01+ \frac{1}{1.01} −1)·1,000,000}
< e^{−1.01(0.00004934)·1,000,000}
< 2.3 · 10^{−22}

The probability that the load on the switch ever rises by even 1% is unbelievably small! (A terrorist attack on Albany Street during an earthquake is far more likely.) If we build the switch with capacity only 1% above the average load, then the switch will (almost) never be overloaded.

The strength of this result relies on the huge expected number of calls. For example, suppose that the average number of calls through the switch were 100 per day instead of 1,000,000. Then every million in the above calculation would be replaced by a hundred; no other numbers change. The final probability of overloading the switch would then be bounded above not by 2.3 · 10^{−22}, but by 0.995! If the switch handles only 100 calls on an average day, then the call load can very often fluctuate upward by 1% to 101 or more.

4.2 How to Build the Network

We now know that building 1% excess capacity into a switch ensures that it is effectively never overloaded. The next problem is to guarantee that no switch in the entire network is overloaded. Suppose that that are 1000 switches, and every switch handles an average of a million calls a day.

In Section 1 we saw that the probability that some event occurs is at most the sum of the event probabilities. In particular, the probability that some switch is overloaded is at most the sum of the probabilities that each of the 1000 switches is overloaded. Therefore, we have:

Pr(some switch overloaded) ≤ 1000 · Pr(particular switch overloaded)
< 2.3 · 10^{−19}

This means that building 1% excess capacity into every switch is sufficient to ensure that no switch is ever overloaded.

The above results are of limited practical value, however. One problem is that calls do not pass through a switch mutually independently. For example, after an earthquake on Albany Street, everyone would call through the eastern Cambridge switchboard to check on friends and family. Furthermore, there are many more phone calls on certain dates like Mother’s Day. On such occasions, 1% excess capacity is insufficient.