Massachusetts Institute of Technology Handout ?? 6.042J/18.062J: Mathematics for Computer Science December 3, 1996 Professors Nancy Lynch and Tom Leighton

Notes 24

Last week we spent a lot of time talking about the expected values of random variables. One thing we saw is that it is possible for a random variable to have expected value 1 even though it is never actually anywhere near 1 (e.g., if I flip two coins that are tied together, what is the expected number of heads?). This week, we are going to study deviations from expectation. We will study the likelihood that a random variable has a value distant from its expectation and give methods for proving that probability is small. Intuitively, the expected value of a random variable is the value that we expect it to obtain.

For example, if we flip 100 fair coins, the expected number of heads is 50. In fact, the probability that we get exactly 50 heads is low, but there is a pretty good chance that we get close to 50 heads. In particular, the probability that we deviate substantially from the expected value in this case is extremely small: The probability of getting more than 75 or less than 25 heads is less than one in a billion. This is because the distribution of the number of heads, which is a binomial distribution, is very tightly centered around its mean. This property holds for any binomial distribution, even biased ones. Now, for arbitrary random variables, it is not always the case that the distribution is tightly centered around the mean. For example, if we glue 100 coins together, the probability of getting close to the mean of 50 vanishes, as the random variable always takes on either the value 0 or 100. Even so, it turns out that we can usually say something about the probability of deviating from the mean. For example, in the latter example, the random variable never takes on values that are more than 2 times the mean.

1 Markov’s Theorem

Markov’s Theorem gives the most general bound on the probability of deviating from the mean. Markov’s Theorem says that if a random variable takes on only non-negative values, then it is unlikely to greatly exceed its mean;

Theorem 1.1 If R : S → R^{+} is a random variable, then for all x ≥ 0,

P r(R ≥ x) ≤ \frac{E[R]}{x}.

Proof. (of theorem)

E[R] = ∑_{z} z Pr(R = z)

≥ ∑_{z≥x} z Pr(R ≥ z)

≥ ∑_{z≥x} x Pr(R = z)

= x ∑_{z≥x} Pr(R = z)

= x Pr(R ≥ x)

In other words, if E[R] is small, then R is not likely to be large. Markov’s Theorem is often expressed in the follwoing alternative form, which can be stated as a corollary:

Corollary 1.2 If R : S → R^{+}, then for all positive real numbers c,

Pr(R ≥ cE[R]) ≤ \frac{1}{c}.

Proof. (of corollary) Just write x=cE[R] in the theorem. Then

Pr(R ≥ cE[R]) ≤ \frac{E[R]}{cE[R]} = 1/c

Example 1 Let R be the weight of a randomly selected person. Suppose we know that E[R] = 100, but we don’t know what the distribution of R looks like. Then the probability that R is greater than or equal to 200 is, by Markov’s Theorem, Pr(R ≥ 200) = Pr(R ≥ 2 · E[R]) ≤ 1/2. Similarly, Pr(R ≥ 300) = Pr(R ≥ 3 · E[R]) ≤ 1/3. Why can’t everyone have weight ≥ 200 in this example? The answer is that, if this were the case, then the expected weight of a randomly selected person would be greater than 200. Not everyone can “beat” the mean!

Example 2 If I give each person a random hat back at a hat check, what is the probability that at least k people get their own hat back? Let R be the number of people who get their own hat back. Markov’s Theorem doesn’t give us an exact answer, but, since E[R] = 1, it tells us that Pr(R ≥ k) ≤ 1/k.

Note that Markov’s theorem doesn’t hold if R can be negative. For example, consider a variable R that is ±1000 with probabilities 1/2 each. It’s expectation is 0! So Markov’s theorem is clearly violated.

So where in the proof of Markov’s Theorem did we use the non-negativity of R? If we go back and check, we see that non-negativity is required for E[R] = ∑_{x} x Pr(R = x) ≥ ∑_{x≥K} x Pr(R ≥ x) to hold.

There is a symmetric result that gives an upper bound on the probability that that a random variable takes on a small value.

Theorem 1.3 If R ≤ L for some L ∈ ℝ, then, for all x ≤ L:

Pr[R ≤ x] ≤ \frac{L − E[R]}{L − x}

Proof. Consider the event L − R. Since R ≤ L, we have that L − R ≥ 0, and hence the conditions of Markov’s Theorem are satisfied. Hence we have that Pr(R ≤ x) = Pr(L − R ≥ L − x) ≤ \frac{E[L−R]}{L−x} = \frac{L−E[R]}{L−x}

Example 3 Let R be the score of a randomly selected student on a test, and suppose the average score on a test is E[R] = 75, and the max score was 100. Then we can compute Pr(R ≤ 50).

Pr[R ≤ 50] ≤ \frac{100 − 75}{100 − 50} = 1/2

One problem with Markov’s Theorem and its corollaries is that it does not give an exact probability, just an upper bound. Sometimes the upper bound it gives is exactly the right answer, but sometimes it can also be pretty far off from the actual value.

Example 4 Suppose we flip 100 fair coins, and let R be the number of heads. Then E[R] = 50, and so from Markov’s Theorem, Pr(R ≥ 75) ≤ 2/3. However, Pr(R ≥ 75) is in fact less than one in a billion, so Markov’s Theorem is way off the mark!

In general, just knowing the range and expected value of a random variable is not enough to give a really good idea of what the distribution looks like. In an effort to get a better handle on the shape and nature of a distribution, mathematicians have defined several other properties of a distribution which help describe it. After the expected value, the next most important measure of a random variable is its variance.

2 Variance and Chebyshev’s Theorem

Definition 2.1 The variance of R is

Var[R] = E[(R − E[R])^{2}].

The variance gives us the average of the squares of the deviations from the mean. The idea behind variance is that it is large when R is likely to be far from its expectation, that is, when its distribution is “spread out”. If R is usually close to its expectation, the variance will be small.

Example 5 Consider the follwing two gambling games, A and B:

• Game A: With probability 2/3, you win 2$. Otherwise, you lose 1$

• Game B: With probability 2/3, you win 1002$. Otherwise, you lose 2001$

Which would you play? First, note that in either game, you win with probability 2/3. Let A, B be the returns from playing games A and B, respectively. Then for game A, E[A] = 2 · 2/3 − 1 · 1/3 = 1, and for game B, E[B] = 1002 · 2/3 − 2001 · 1/3 = 1. So the expected return is the same for both games, namely 1! Still, the games are very different. This difference is captures by the variances. Let’s see what the variance is for each game:

• Game A:

A − E[A] = \begin{cases}1 & with probability 2/3, \\ −2 & with prob. 1/3. \end{cases}

(A − E[A])^{2} = \begin{cases}1 & with probability 2/3 \\ 4 & with prob. 1/3 \end{cases}
Var[A] = 1 · 2/3 + 4 · 1/3

• Game B:

B − E[B] = \begin{cases}1001 & with probability 2/3, \\ −2002 & with prob. 1/3 \end{cases}

(B − E[B])^{2} = \begin{cases}1002001 & with probability 2/3 \\ 4008004 & with prob. 1/3 \end{cases}
Var[B] = 1002001 · 2/3) + 4008004 · 1/3

= 2004002

Hence we se that game B is much more risky than game A. Risk averse people should stay away from game B!!! In short, playing a game with high variance is a serious risk: we can win a lot, but we can loose a lot, too. Playing another game with the same expectation of winnings but less variance we can hope for the same average gains risking less. In investments, many people prefer bonds to stocks because, although the expected earnings in bonds are lower, the variance is much lower too. They are willing to pay for risk-reduction.

Why do we bother to square the deviation before taking expectations? Why not just compute the expected deviation from the mean? Let’s consider a random variable R with mean E[R]. What we are interested in, then, is the deviation of the random variable from its mean, namely R − E[R]. Unfortunately, E[R − E[R]] = E[R] − E[R] = 0, so the positive and negative deviations cancel out and give us no information. This problem occurs because the expected value is the center of mass for the distribution, so the positive and negative deviations must balance out, by definition of expectation. So, one advantage of squaring is that all deviations become positive, and hence there is no cancelling. Of course, we could also get around this cancellation problem by computing E[|R − E[R]|]. However, absolute values are painful in analysis (not differentiable, etc.). We could also take higher powers of R-E[R]. If we take the 4th power, for instance, we get what is called the kurtosis of R. It turns out however, for reasons what we will not go into, that squaring is pretty much the best way to go.

People often like to “renormalize” the variance:

Definition 2.2 The standard deviation of R is

σ_{R} = \sqrt{Var[R]}.

Thus the variance is the square of the standard deviation. It follows that the standard deviation has the same units (days, people, etc.) as the original random variable and its expectation. On an intuitive level, the standard deviation measures E[|R − E[R]|] (careful— they aren’t really equal!). The standard deviation is a very popular measure in statistics, particularly when fitting curves to experimental data.

Example 6 Returning to game B, we can compute its standard deviation σ_{B}:

σ_{B} = \sqrt{Var[R]} = \sqrt{2004002} ≈ 1416

So far, we have given a precise definition of variances of random variables, as well as their standard deviations. The intuition is that these measures tell you how likely a random variable is to be far from its mean. Unfortunately, this is not always true. Just like the expectation of a random variable does not completely specify the distribution, the expectation of the deviations does not completely specify the distribution of deviations. However, the variance and standard deviation of a distribution are still useful tools for analysis.

2.1 The Chebyshev Bound

A variable with small variance is unlikely to be far from its mean. With Markov’s Theorem, we can deduce, for example, that given a mean IQ of 100, only 1/2 the people can have IQ bigger than 200. But suppose we know that the standard deviation in IQ is 10. Then intuitively, it should quite rare to have an IQ bigger than 110. We can show that in fact only 1/100 people have IQ bigger than 200.

Chebyshev’s theorem formalizes this observation:

Theorem 2.3 For any random variable R,

Pr(|R − E[R]| ≥ x) ≤ \frac{Var[R]}{x^{2}}.

Proof.

Pr(|R − E[R]| ≥ x) = Pr((R − E[R])^{2} ≥ x^{2}

≤ \frac{E[(R − E[R]^{2}]}{x^{2}}

= \frac{Var[R]}{x^{2}}

Corollary 2.4 Pr(|R − E[R]| > c · σ_{R}) ≤ 1/c^{2}.

Proof. Set x = c · σ_{R}

Chebyshev’s theorem is very similar to Markov’s Theorem, but uses the added information supplied by the standard deviation to get a better upper bound. In fact, the proof uses Markov’s Theorem.

Example 7 Let R be the IQ of a random person. Then, assume R = geq0, σ_{R} = 10 and E[R] = 100. Therefore, using Chebyshev:

Pr(R ≥ 200( = Pr((R − 100) ≥ 100)
= Pr(|R − E[R]| ≥ 10σ_{R})
≤ 1/100

Had we used Markov’s Theorem here, we would have gotten a lower bound of 1/2 on this same probability.

In the previous example, we used Pr(|R − E[R]|) to give an upper bound on Pr(R − E[R]). Maybe the probability Pr(R − E[R]) is in fact half the probability Pr(|R − E[R]|)? In this case, the answer is no. If the distribution were symmetric about the mean, however, it would be true. We do not, however, have any reason to believe that the distribution is symmetric. In general, if we only want the probability mass of one of the tails, we can do a little better than what we just did, however.

Theorem 2.5 For any random variable R, the following holds:

Pr(R − E[R] ≥ x) ≤ \frac{Var[R]}{Var[R] + x^{2}}

Pr(R − E[R] ≤ −x) ≤ \frac{Var[R]}{Var[R] + x^{2}}

We will not prove this theorem. The following is a useful corollary:

Corollary 2.6 For any random variable R, the following holds:

Pr(R − E[R] ≥ cσ_{R}) ≤ \frac{σ_{R}^{2}}{σ_{R}^{2} + c^{2}σ_{R}^{2}} = \frac{1}{1 + c^{2}}

Pr(R − E[R] ≤ −cσ_{R}) ≤ \frac{σ_{R}^{2}}{σ_{R}^{2} + c^{2}σ_{R}^{2}} = \frac{1}{1 + c^{2}}

Example 8 Returning to our IQ example, we can now recompute the probability of having an IQ greater than 200;

Pr[R ≥ 200] ≤ 1/101

Example 9 At most 1/4 of the students score more than 2 standard deviations away from the mean, whatever the set of scores is. Indeed, if R is the score of a random student, then

|P r(R − E(R)| ≥ 2σ_{R}) ≤ \frac{1}{4},

which means that at most 1/4 of all the scores can be that far from the average. Moreover, at most 1/5 of the students score more than 2 standard deviations above the mean, whatever the set of scores is. Indeed, if R is the score of a random student, then

P r(R − E(R) ≥ 2σ_{R}) ≤ \frac{1}{1 + 2^{2}} = 1/5,

Note that the results we get from Markov’s Theorem and Chebyshev’s Theorem are only upper bounds on the probability of deviating from the mean. If there is more information available about the distribution, it might be possible to prove better bounds. For example, if we know we are dealing with an unbiased binomial distribution, then it is possible to prove a much better upper bound:

Theorem 2.7 For an unbiased binomially distributed random variable R,

Pr[R − E[R] ≥ cσ_{R}] < e^{−c^{2}}

Example 10 Suppose that the test scores are binomially distributed. Then the probability of a randomly selected student having a score greater than or equal to two times the standard deviation is bounded above by 1/e^{4}.

2.2 A Different Variance Formula

There is another popular way of defining the variance of a random variable, which we will state as a theorem.

Theorem 2.8 For all R, we have

V ar(R) = E[R^{2}] − E[R]^{2}.

Proof.

V ar(R) = E[(R − E[R])^{2}]
= E[R^{2} − 2RE[R] = E[R]^{2}]
= E[R^{2}] − 2E[R]E[R] + E[R]^{2}
= E[R^{2}] − E[R]^{2}.

This formula is often more useful to compute variances than the previous one.

Example 11 Let R be the outcome of rolling a 6-sided die. Then the variance of R is given by

Var[R] = \frac{1}{6}(1 + 4 + 9 + ⋯ + 36) − (3.5)^{2}

= 15\frac{1}{6} − 12\frac{1}{4} = 2\frac{11}{12}

This theorem is also useful from a computational point of view. Suppose, say, that you are given a stream of data (like test scores), and you are trying to compute the expectation and variance on the fly with only one pass through the data. Computing a running average is easy, we just compute a running sum and divide by the number of data points seen so far. Similarly, to compute the variance, we compute two running sums, one for the scores and one for the square of scores.

The theorem has a nice corollary:

Corollary 2.9 For all R, E(R^{2}) ≥ E(R)^{2}.

Proof. This follows from the previous theorem, since Var(R) is always nonnegative, being the expectation of a square.

2.3 Manipulating Variances

Theorem 2.10 If Var(R)=0, then R is a constant scalar.

10 Handout ??: Notes 24 Proof. Suppose that

Var[R] = E(R − E[R])^{2}] = 0.

Then (R − E[R])^{2} must be always 0, thus R must always be equal to E[R]. Theorem 2.11 For any constants a and b, Var[aX + b] = a^{2}Var[X]. Proof.

Var[aX + b] = E[(aX + b)^{2}] − (E[aX + b])^{2}
= E[a^{2}X^{2} + 2abX + b^{2}] − (aE[X] + b)^{2}
= a^{2}E[X^{2}] + 2abE[X] + b^{2} − a^{2}E[X]^{2} − 2abE[X] − b^{2}
= a^{2}(E[X^{2}] − E[X]^{2})
= a^{2}Var[X]

A Bernoulli Random variable with probability p has variance

p · 1^{2} + (1 − p) · 0^{2} − p^{2} = p(1 − p)

What about the binomial distribution? This is messy to do directly, but easy if we figure out how to sum variances: Theorem 2.12 If X and Y are independent, then

Var[X + Y] = Var[X] + Var[Y]

Proof. The assumption that X and Y are independent means that E[XY] = E[X]E[Y]. Therefore,

Var[X + Y] = E[(X + Y)^{2}] − E[X + Y]^{2}
= E[X^{2} + 2XY + Y^{2}] − (μ_{X} + μ_{Y})^{2}
= E[X^{2}] + 2E[XY] + E[Y^{2}] − μ_{X}^{2} − 2μ_{X}μ_{Y} − 2μ_{Y}^{2}
= E[X^{2}] − μ_{X}^{2} + E[Y^{2}] − μ_{Y}^{2} + 2E[XY] − 2μ_{X}μ_{Y}
= E[X^{2}] − μ_{X}^{2} + E[Y^{2}] − μ_{Y}^{2}

(since E[XY] = μ_{X}μ_{Y} by independence)

= Var[X] + Var[Y] Corollary 2.13 If X_{1}, …, X_{n} are pairwise independent random variables, then

Var[∑ X_{i}] = ∑ Var[X_{i}].

Note that mutual independence is not required, because the only fact we used that relied on independence was that E[X_{i}X_{j}] = E[X_{i}]E[X_{j}]. This follows from pairwise independence, since it means X_{i} and X_{j} are independent.

Therefore, the variance of the binomial distribution on n Bernoulli trials with probability p is np(1 − p). It follows that the standard deviation is σ = \sqrt{np(1 − p)}.