Massachusetts Institute of Technology Handout ?? 6.042J/18.062J: Mathematics for Computer Science December 5, 1996 Professors Nancy Lynch and Tom Leighton

Notes 25

For today’s lecture, we are going to apply what we have learned about probability to solve a particular kind of probability problem that arises in all sorts of applications. In the general form of the problem, we have a collection of independent events and we want to get some idea of how many of the events are likely to occur.

Given: mutually independent events A_{1}, A_{2}, ⋯, A_{n} ⊆ S.

For example, we can think of A_{i} as being the event that the ith component of a system fails. Or we can think of A_{i} as the event that the ith coin is heads when flipping n coins. r anything else–probability student answers the ith question on the final correctly.

What we want to know is, what is the probability that k of the events occur. Let T be the number of events that occur.

The question is easy to answer if each event is equally likely, that is, ∀iP r(A_{i}) = p. Then P r(T = k) = \binom{n}{k}p^{k}(1 − p)^{n−k}, the binomial distribution. We’ve already talked about important distribution. Very tightly centered around the mean = pn.

Today, we’re going to talk about the general case: P r(A_{i}) are different.

First thing is to look at the expected number of events to occur. To figure this out, we can use linearity of expectation and indicator variables.

Let T_{i} be an indicator for the ith event. (1 if A_{i} occurs, 0 otherwise)

T = T_{1} + T_{2} + ⋯ + T_{n}
E[T] = E[T_{1}] + E[T_{2}] + ⋯ + E[T_{n}]

= P r(A_{1}) + P r(A_{2}) + ⋯ + P r(A_{n})

= ∑_{i=1}^{n} P r(A_{i})

So to compute the expected number of events, we just add up the probabilities. We don’t even need the A_{i} independent for this, because we just used linearity of expectation.

Example: A_{i} is the probability the ith component fails. Then the expected number of failures is the sum of the individual component failures, even if there is dependence.

OK, nothing really new here. But say that we want to know the probability that there is one or more failures. That is, say that any fault is critical, so we don’t care how many there are, just if there is one. (Once there is one it is all over, so who cares how many more happen?) So we want P r(T ≥ 1).

First let’s get a simple upper bound. By Markov,

P r(T ≥ 1) ≤ \frac{E[T]}{1} = E[T] = ∑_{i=1}^{n} P r(A_{i})

So if the sum of the individual event probabilities is small, then the chance that one or more occurs is also small. Note we haven’t used independence yet.

We could show this another way, using Boole’s inequality. Let A be the event that 1 or more occurs. So A = A_{1} ∪ A_{2} ∪ ⋯ ∪ A_{n}.

P r(T ≥ 1) = P r(A) ≤ ∑_{i=1}^{n} P r(A_{i}) = E[T]

Note we still didn’t need independence.

So if the expected number of events to occur is << 1, then the probability at least one occurs is very small. For example, if you buy 1000 lottery tickets, each of which wins with probability 1 in a million, then the probability you have a winner is ≤ 1000/1000000 = 1/1000.

Now let’s look at the case where the expected number of events to occur is bigger than 1. For example if the expected number is 10, it seems likely that at least one will happen. This is the case if the events are independent. The following theorem captures this fact:

Theorem 0.1 Given independent events A_{1}, A_{2}, ⋯, A_{n}, the probability that none occur is ≤ e^{−m}, where m = ∑_{i=1}^{n} P r(A_{i}) = expected number to occur.

So probability none occur is exponentially small in the expected number of events to occur.

Proof. Probability that no event occurs

= P r(\overline{A_{1}} ∩ \overline{A_{2}} ∩ ⋯ ∩ \overline{A_{n}})

= ∏_{i=1}^{n} P r(\overline{A_{i}}) (by independence)

= ∏_{i=1}^{n}(1 − P r(A_{i}))

≤ ∏_{i=1}^{n} e^{−P r(A_{i})}

= e^{− ∑_{i=1}^{n} P r(A_{i})}

= e^{−m} Note independence is critical here. Also, this is another example where Taylor series bounds are important.

This is a really powerful result. For example, we immediately get the following corollary:

Corollary 0.2 Given independence, if we expect 10 or more events to occur, the probability that no event occurs is ≤ e^{−10} < 1/22000

Note there is no dependence on n here. This is a really strong result since it says that if we expect some things to happen then they almost certainly will. I.e, if you expect your system to have some faults, then it surely will–this is sort of a proof of Murphy’s Law. If there are a bunch of things that can go wrong, even if none of them is very likely, this result says that almost surely something will go wrong.

Example 1 (Murphy’s Law) n components, each fails with probability p = 10/n. expected number of failures = 10, probability there is no faults ≤ e^{−10}.

So even though the failure probability of any component is small, and only 10 are expected, probability of none is not 1/10 but ≤ e^{−10}.

This theorem is in fact so powerful that it will allow me to read your mind.

1 Mind Reading Game

You do the following: Pick a number i from 1 to 9. You then start dealing out a deck of cards one at a time. When you get to card i, you continue counting ahead, this time with the value of that card. 10, J, Q, K, A, and Joker all count as 1. Whatever card you are counting off of when the deck runs out is your card to be mind read.

Example (on a few cards): You pick 3. The sequence of cards in the deck is 4 5 2 3 10 Q 3 7 6 4 2. So the cards you use are 2, 10, Q, 3, 4, and the card to be mind read is the 4.

I get to read your mind because if I do the same thing as you it is likely we will have the same last card. The reason, is that is we ever land on the same card we will have the same last card. Each time one of use hits a card and starts counting again, there is some chance we will land on the others positions. If 1 away, 5/13 chance, if >1 away, 1/13 chance, and there are lots of times that we each jump. On average, we will each jump >10 times, so 20 about jumps total between the two sides, making it likely that we will collide. In particular, A_{i} is the event that on the ith jump we collide.

Now the intuition I just went thru was pretty fuzzy. In fact, the theorem doesn’t really apply in this example because the events are not independent. If we have not yet collided, it tells us something about the numbers we have had, and since we only have a 52 card deck, something about the numbers that are forthcoming, which conditions the probability that we collide. However, we could analyze it if the cards we selected at random from an infinite number of decks. Then the event of a hit would be independent for every jump.

We don’t have time in class to analyze this precisely. It turns out that the probability I read you mind is > 90% for a 52 card deck.

The point of all this can be summed up as follows:

If we expect several events to occur, then the probability that no events occur is exponentially small if the events are independent. If the events are dependent, then the probability of no events occurring could be anything.

2 Chernoff Bound

Now we are going to prove an even stronger result. If we have a collection of independent events and we expect m of them to occur, then with probability very close to 1, close to m will occur. It’s similar to the phenomenon with tossing n coins. We expect n/2 heads, and with probability close to 1 we will get close to n/2 heads.

For simplicity, we will only look at the case of getting much more than the expected value; a similar method gives similar results for the other side.

Theorem 2.1 (Chernoff Bound) Let T = T_{1} + T_{2} + ⋯ + T_{n}, where the T_{i} are independent Bernoulli random variables. Then ∀c > 1 P r(T ≥ cE[T]) ≤ e^{−αcE[T]}, where α = ln c + 1/c − 1 ≥ 0.

In general, this is much stronger than what Markov’s theorem gives (1/c); here we get exponential bounds on probability.

Corollary 2.2 With T as above, P r(T ≥ eE[T]) ≤ e^{−E[T]}.

Proof. c = e ⇒ α = ln e + 1/e − 1 = 1/e ⇒ αc = 1

Example 2 If E[T] = 100, P r(T ≥ 272) ≤ e^{−100}.

Example 3 1 million people play “Pick 4”. Each player picks a random 4 digit decimal number. So P r(winning) = 1/10000, E[#winners] = 1M/1000 = 100. We assume random picks for the sake of argument. If picks are independent, then P r(≥ 272winners) ≤ e^{−100}, which is really, really, tiny. Note answer is not dependent on the number of people playing or the individual probability of winning.

Of course, the result is not true if the picks are dependent. If everyone picks the same number, then the P r(≥ 272winners) = 1/10000 = chance that number comes up. (In effect, we’ve proved that it is incredibly unlikely that 272 pick the same number if they choose independently.)

Now in truth with lottery, people tend to pick in a structured fashion. (Digression) They pick the date, or something they saw on TV, or yesterdays number, etc. In fact, Chernoff himself (who was once on the faculty here) developed a strategy to make money on the state lottery. He did a statistical study of unpopular numbers and showed that by betting on them you would expect to earn money! This only works in lotteries where payoff is fraction of total pot. If zillions of people all bet on few numbers and you bet on all the rest, you are likely to win their zillions of dollars. Way it turned out made 7 percent return. Now state sets it up to give you random number.

Back to the proof of Chernoff’s theorem:

Proof. This uses an exponential transform of Markov’s theorem.

P r(T ≥ cE[T]) = P r(c^{T} ≥ c^{cE[T]})

This is because it is the same event, just stuffed in an exponent on one side. Then by Markov,

≤ \frac{E[c^{T}]}{c^{cE[T]}}

Need to evaluate E[c^{T}]:

T = T_{1} + T_{2} + ⋯ + T_{n}
⇒ c^{T} = c^{T_{1}}c^{T_{2}} ⋯ c^{T_{n}}
⇒ E[c^{T}] = E[c^{T_{1}}]E[c^{T_{2}}] ⋯ E[c^{T_{n}}] (by product rule (T_{i} ind. ⇒ c^{T_{i}} ind.))

= ∏_{i=1}^{n} E[c^{T_{i}}]

So just need to evaluate E[c^{T_{i}}]:

E[c^{T_{i}}] = c^{1}P r(T_{i} = 1) + c^{0}P r(T_{i} = 0)
= cP r(T_{i} = 1) + (1 − P r(T_{i} = 1))
= 1 + (c − 1)P r(T_{i} = 1)
≤ e^{(c−1)P r(T_{i}=1)} (since 1 + x ≤ e^{x})
= e^{(c−1)E[T_{i}]} (since indicator variables)

Now we can plug in:

E[c^{T}] ≤ ∏_{i=1}^{n} e^{(c−1)E[T_{i}]}
= e^{∑_{i=1}^{n}(c−1)E[T_{i}]}
= e^{(c−1) ∑_{i=1}^{n} E[T_{i}]}
= e^{(c−1)E[T]}

Now we can go back and plug into the bounds from Markov:

P r(T ≥ cE[T]) ≤ \frac{e^{(c−1)E[T]}}{c^{cE[T]}}
= e^{(c−1)E[T]−c ln cE[T]}
= e^{−c(ln c+1/c−1)E[T]}
= e^{−αcE[T]} where α = ln c + 1/c − 1

This is the deepest result in the class. Don’t feel like you’d have to come up with this on your own–it’s pretty tricky. But the basic method is not so bad:

P r(A > B) = P r(f(A) > f(B)) ≤ \frac{E[F (A)]}{f(B)}

and then just plug and chug.

You should try to remember the implications of the result. If you have a sum of 0-1 variables or a collection of independent events, then almost surely the number of events is close to the expected number. “Tails” are exponentially small.

Example 4 Say we have a phone network (or some kind of internet), and the network typically handles 1 billion calls (or messages) per day. In this problem, we are interested in the volume of traffic that is routed thru a particular switch. In particular, we are interested in the fluctuations in volume - want to buy big enough switch to handle random variations in traffic. Each particular call may or may not be routed thru the switch, depending on its destination and source. We will treat each call as a random variable T_{i}:

T_{i} = \begin{cases}1 if call i gets routed thru switch, \\ 0 otherwise. \end{cases}

Then the total number of calls routed through the switch is given by:

T = ∑_{i=1}^{10^{9}} T_{i}

Let P_{i} = Pr(T_{i} = 1). P_{i} is an unknown - we don’t know the probability that each of the billion calls goes thru switch. We do know the expected number of calls going thru the switch, however - it is about 1 million. So assume that E[T] = 10^{6}. Aslo, assume that the T_{i} are mutually independent.

We want to figure out what sort of variations T might have. In particular, we’d like to know if a 1% variation from the mean is likely to occur (on the high side). So let’s compute the probability of having more than 1.01 million calls going thru the switch. We use Chernoff’s bound with c = 1.01 and E[T] = 10^{6}, and get:

Pr(T ≥ 1.01 · 10^{6}) ≤ e^{αcE[T]}
≤ e^{−49.8}
≤ 2.3 · 10^{−22}

So the chance of being more than 1% above the mean is not really all that high!! This tells us that if we build in an extra 1% capacity, we will, for all practical purposes, never exceed the switch’s capacity. Notice that, if we only expected 100 calls, the probability of getting more than 101 calls would be pretty high. Things work in our example because of the volume being so high (this means that the exponent in Chernoff’s bound becomes very small).

So why do we sometimes see abnormally high loads on telephone switches in practice? The answer is that phone calls are not necessarily independent. Some events, like mother’s day, earth quakes, etc. etc. change calling patterns significantly.

One last point. Suppose that we have 1000 switches in our network. What is the probability that no switch has more traffic than 1% above the mean, supposing al the switches have an expected traffic of 10^{6} calls per day? Let O_{s} be the event that some switch s gets more traffic than 1.01 · 10^{6} on a given day. From Boole’s law we have that:

Pr(⋃_{s} O_{s}) ≤ 1000 · Pr(O_{∫})
≤ 2.3 · 10^{−19}