Massachusetts Institute of Technology Lecture 22 6.042J/18.062J: Mathematics for Computer Science November 21, 1996 Professors Nancy Lynch and Tom Leighton

Notes 22

1 Random Variables

Definition 1.1 A random variable R on sample space S is a function R : S → ℝ.

Note defined with respect to a sample space—associates a value with each point.

Example 1 Let S = outcomes of 3 coin tosses = {HHH, HHT, HTH, HTT, THH, THT, TTH, TTT}.

• R_{H} = number of H in outcome. So R_{H}(HHH) = 3, R_{H}(HHT) = 2.

• R_{M}(p) = 1 if all coins match, 0 otherwise. So R_{M}(HHH) = 1, R_{M}(HHT) = 0. R_{M} is an indicator or characteristic random variable: 1 if event happens, 0 otherwise.

Note functions: R_{H} : S → {0, 1, 2, 3}, R_{M} : S → {0, 1}

Definition 1.2 An indicator (or Bernoulli or characteristic ) random variable is a random variable with range 0,1.

A random variable partitions S into events (subsets of sample points).

“R = x” corresponds to the event {p ∈ S|R(p) = x}

Note that indicators partition the sample space into two events.

R_{H} = 3  R_{H} = 2  R_{H} = 1 R_{H} = 0 TTT HHH HHT THH HTH TTH THTHTT

Definition 1.3

P r(R = x) ≡ ∑_{p∈S:R(p)=x} P r(p)

For example, P r(R_{H} = 2) = P r({HHT, HT H, T HH}) = 3/8 P r(R_{M} = 1) = P r({HHH, T T T }) = 1/4

We can also define the probability that a random variable is within some range or in some subset of specific values.

Example 2 P r(R_{H} ≥ 2) = ∑_{i=2}^{∞} P r(R_{H} = i) = 1/4

or, for A ⊆ ℝ : P r(R_{H} ∈ A) = ∑_{a∈A} P r(R = a)

Our random variables will be discrete–range, finite, or countably infinite (ie, as here or R : S → ℕ). There are also continuous random variables with values in the reals; for these sums must be replaced by integrals.

2 Independence of Random Variables

Recall, events E_{1} and E_{2} are independent if P r(E_{1} ∩ E_{2}) = P r(E_{1})P r(E_{2}).

Definition 2.1 Random variables R_{1} and R_{2} are independent if ∀x, y ∈ ℝ,

P r(R_{1} = x ∧ R_{2} = y) = P r(R_{1} = x)P r(R_{2} = y)

The coins being fair and mutually independent (so each P ∈ S has probability 1/8) does not mean R_{H} and R_{M} are independent, because P r(R_{H} = 2 ∧ R_{M} = 1) = 0 ≠ P r(R_{H} = 2)P r(R_{M} = 1) = \frac{3}{8} · \frac{1}{4}.

Intuition: knowing the value of one variable biases the distribution of the other.

The picture, with area proportional to probability:

R_{1} = 1  R_{1} = 3 R_{1} = 2  R_{2} = 6 R_{2} = 7 R_{2} = 9 R_{2} = 8  R_{1} = 1  R_{1} = 2  R_{2} = 6 R_{2} = 7 R_{2} = 8 R^{1}, R^{2} independent  R^{1}, R^{2} not independent

Note that independence of random variables covers all pairs of “value events.”

Example 3 Suppose we roll two independent fair dice to get values D_{1} and D_{2}. Let T = D_{1} + D_{2} be the sum of values. Let S = 1 if T = 7 and 0 otherwise.

• T and D_{1} are not independent since, for example, Pr(D_{1} = 6 | T = 1) = 0 ≠ P r(T = 2) = 1/36.

• T and S are not independent since knowing T completely determines S

• S and D_{1} are independent. P r(S = 1 | D_{1} = 1) = 1/6 = P r(S = 1)

P r(S = 1 | D_{1} = 2) = 1/6 = P r(S = 1) P r(S = 1 | D_{1} = 6) = 1/6 = P r(S = 1) P r(S = 0 | D_{1} = 1) = 5/6 = P r(S = 0) P r(S = 0 | D_{1} = 2) = 5/6 = P r(S = 0) P r(S = 0 | D_{1} = 6) = 5/6 = P r(S = 0)

The definition of mutually independent random variables works in the same way.

Definition 2.2 Given a random variable R, the probability distribution function (pdf) for R is f(x) ≡ P r(R = x)

Definition 2.3 Given a random variable R, the cumulative distribution function for R is F (x) ≡ P r(R ≤ x) = ∑_{y≤x} f(y)

Example 4 For a Bernoulli random variable f(0) = p F (0) = p f(1) = 1 − p F (1) = 1

For a uniform random variable R : S → [1, N] f(k) = \frac{1}{N} F (k) = \frac{k}{N}

3 The Binomial Distribution

We now study what is perhaps the most common random variable distribution in computer science: the binomial distribution.

Definition 3.1 The binomial distribution is given by

f_{n,p}(k) = \binom{n}{k}p^{k}(1 − p)^{n−k}

Sanity check: is this a probability distribution?

∑_{k} f_{n,p}(k) = ∑ \binom{n}{k}p^{k}(1 − p)^{n−k}
= (p + (1 − p))^{n} (binomial theorem)
= 1

A special case is the unbiased binomial distribution which is the binomial distribution with p = 1/2.

f_{n}(k) = \binom{n}{k}2^{−n}

Example 5 Given a system with n components each of which fails with probability p find the probability distribution function for the number of failures.

Let R = number of failures. We wish to compute f(k) = P r(R = k)

solution 1. The sample space S consists of 2^{n} binary strings. (Say, a 0 corresponds to a failure)

2. The probability to get a sample point with k failures is p^{k}(1 − p)^{n−k}

3. Identify the event of interest. A = sample points with k failures.

4. Compute the probability of the event.

f(k) = P r(R = k) = P r(E) = ∑_{A} p^{k}(1 − p)^{n=k} = |A|p^{k}(1 − p)^{n−k} = \binom{n}{k}p^{k}(1 − p)^{n−k}

We get a binomial distribution. In general, we get a binomial distribution whenever we have n independent Bernoulli events with the same distribution.

3.1 Asymptotics

It’s hard to see the qualitative behavior of the binomial distribution for large k and n. Lets consider case p = 1/2: uniform distribution. Each outcome has probability

f_{n,1/2}(k) = \binom{n}{k} (\frac{1}{2})^{n}

Suppose I flip 100 coins

• what is most likely number of heads?

• what probability of exactly 50 heads?

• what probability of exactly 25 heads

• what probability less than 25 heads?

• what probability of exactly 25 heads, given at most 25?

vote: 1, .5, .1, .01, .000001

3.1.1 The central term

Where is f_{n,p}(k) maximized?

f_{n,p}(k)/f_{n.p}(k − 1) = \frac{n − k + 1)p}{k(1 − p)}

= 1 + \frac{(n + 1)p − k}{k(1 − p)}

Therefore, f_{n,p}(k) increases until k = np, decreases after.

What is the value at the central term? Let’s consider p = 1/2, so the central term is at k = n/2:

P r(\frac{n}{2}heads) = \binom{n}{n/2}2^{−n}

= \frac{n!}{\frac{n}{2}!\frac{n}{2}!2^{n}}

= \frac{\sqrt{2πn} (\frac{n}{e})^{n} e^{a_{n}}}{(\sqrt{πn} (\frac{n}{2e})^{n/2} e^{a_{n}/2})^{2} 2^{n}}

= \frac{\sqrt{2πn}}{πn} \frac{n^{n}(2e)^{n}}{e^{n}n^{n}2^{n}} \frac{e^{a_{n}}}{e^{2a_{n/2}}}

= \sqrt{\frac{2}{πn}}e^{a_{n}−2a_{n/2}}

Recall that \frac{1}{12n+1} ≤ a_{n} ≤ \frac{1}{12n} so \frac{1}{12n+1} − 26n ≤ a_{n} − 2a_{n/2} ≤ \frac{1}{12n} − \frac{2}{6n+1}. Therefore e^{a_{n}−2a_{n/2}} → 1. So P r(\frac{n}{2}heads) ∼ \sqrt{\frac{2}{πn}} = Θ(\frac{1}{\sqrt{n}})

For n = 100 we have .079589176 ≤ P r(\frac{n}{2}heads) ≤ .079589671 So Stirling’s formula gives very tight bounds.

3.1.2 The tails

In general, for k = αn, α ≤ 1/2,

P r(αn heads)

= \binom{n}{k}2^{−n}

= \frac{n!}{(αn)!(1 − αn)!2^{n}}

= \sqrt{\frac{1}{2πα(1 − α)n}}2^{n(H(α)−1)}

where H(α) = α log(1/α) + (1 − α) log(\frac{1}{1−α}) is the entropy function. It arises in the study of how much information is carried in a binary string with a fraction α of the bits set to one, and thus in the study of error correction over faulty communication links.

0^{0.2}^{0.4}^{0.6}^{0.8}^{1} ^{0}  ^{0.2}  ^{0.4}  ^{0.6}  ^{0.8}  ^{1} H(α)  α

Lemma 3.2 For α a constant ≠ 1/2, f_{n,1/2}(αn) is exponentially small in n.

In other words, as n increases, the likely values become concentrated more and more tightly around α = 1/2.

Example 6

\binom{n}{n/4} ∼ \sqrt{\frac{8}{3πn}}2^{0.811n}

f_{n,1/2}(n/4) ∼ \sqrt{\frac{8}{3πn}}2^{−0.189n} exponetially small

Returning to our beginning questions,

P r(25 heads) = f_{100,1/2}(25) ≈ 1.913 × 10^{−7}

3.1.3 Bounding Sums with Geometric Series

For a random variable R, the cumulative distribution function F_{R}(k) = Pr(R ≤ k).

Since f_{n,1/2}(k) is increasing with k < n/2, we can bound the probability F_{R}(25) ≤ 25 · f_{n,1/2}(25). But we can do better.

8 Lecture 22: Notes 22 Let k = αn where α ≤ 1/2. Then

P r(R ≤ αn) = ∑_{i=0}^{αn} P r(R = i)

= \frac{1}{2^{n}} [\binom{n}{αn} + \binom{n}{αn − 1} ⋯ \binom{n}{0}]

To upper bound the sum with a geometric series, we first compute the size of successive terms.

\begin{matrix}( & {n \atop r−1} & ) \\ ( & {n \atop r} & ) \end{matrix} = \frac{\frac{n!}{(r−1)!(n−r+1)!}}{\frac{n!}{r!(n−r)!}}

= \frac{r}{n − r + 1}

≤ \frac{αn}{n − αn + 1}

≤ \frac{α}{1 − α}

(Note that we used r ≤ αn.) Therefore,

\binom{n}{αn − r} ≤ (\frac{α}{1 − α})^{r} \binom{n}{αn}

Plugging in, we find

P r[R ≤ αn] ≤ 2^{−n} [\binom{n}{αn} + \frac{α}{1 − α}\binom{n}{αn} + (\frac{α}{1 − α})^{2} \binom{n}{αn} + ⋯ + (\frac{α}{1 − α})^{αn} \binom{n}{αn}]

≤ 2^{−n}\binom{n}{αn}\frac{1}{1 − \frac{α}{1−α}} (α < 1/2 ⇒ \frac{α}{1 − α} < 1)

= 2^{−n}\binom{n}{αn}\frac{1 − α}{1 − 2α}

Notice that \frac{1−α}{1−2α} is constant!.

P r(≤ 25 heads) (n = 100, α = 1/4) = (1.913 × 10^{−7})\frac{3/4}{1/2}

For β = o(\sqrt{n}), f(n/2 − β\sqrt{n}) ∼ \sqrt{\frac{2}{πn}}e^{−2β^{2}}.

Furthermore,

Pr(≤ 24 heads) ≤ \frac{α}{1 − α} Pr(≤ 25 heads)

so that

Pr(exactly 25 heads | at most 25 heads) > 1/2

3.1.4 Limiting Distribution

As n → ∞, f_{n,p} approaches “Bell curve” or “normal distribution” (continuous). (Main difference is in the tails, which are exponentially small anyway.)

Here is a plot of the “Bell curve”:

In fact, the binomial distribution is the discrete analog to the normal distribution. So far, we have been focussing on the unbiased binomial distribution (p = 1/2). It’s not too hard to extend the results to the general binomial distribution. f_{p}(αn) ≤ \frac{1}{\sqrt{2πα(1−α)n}}2^{αlog( \frac{p}{α})+(1−α)log( \frac{1−p}{1−α})n}

The maximum occurs at α = p so we have f_{p}(αn) ≤ \frac{1}{\sqrt{2πp(1−p)n}}

Similarilly, F_{p}(αn) ≤ \frac{1−α}{1−α/p}f_{p}(αn)

4 Applications

4.1 Polling

We want to estimate the fraction p of Democrats in the country by conducting a poll, that is, we will sample n people uniformly at random, with replication, from the entire population. Let D be the number of Democrats in the sample we obtain. Then the intuition is that D should be a reasonably close approximation to np, since each person is a Democrat with probability p , and we sample n times.

Theorem 4.1 Given ϵ > 0. Let δ be the sum of the probabilities of getting ϵn too many democrats and ϵn too few democrats in our sample, that is

δ = P r(D < (p + ϵ)n) + P r(D > (p − ϵ)n)

Then, with probability 1 − δ, p − ϵ < D < p + ϵ, that is, our poll will give a margin of error of ±ϵ, and be correct with probability 1 − δ.

Given ϵ, how do we compute δ ? We use the fact that D is binomially distributed. This allows us to apply the formulas that we derived in the previous section. In particular, we have that

δ = F_{p}((p − ϵ)n) + F_{1−p}((1 − p − ϵ)n)

The only problem with this is that we don’t know p yet. This is not a serious problem, however. We have the following fact:

Fact 1 For any ϵ > 0, δ is maximized by setting p = 1/2.

Given this fact, we may set p = 1/2 for the sake of analysis. Hence we have that

δ ≤ F_{1/2}((1/2 − ϵ)n) + F_{1/2}((1/2 − ϵ)n) ⇒ δ ≤ 2F_{1/2}((1/2 − ϵ)n)

Now we can just plug in values for ϵ. Say we want an error margin of ±.05. In this case, we have that

δ ≤ F_{1/2}(.45n) = 2\frac{1−.45}{1−.90} \frac{1}{\sqrt{2π·.45·.55n}}2^{H(.45)−1)n} ⇒ δ ≤ 11 · \frac{.802}{\sqrt{n}}2^{−.0072n}

Plugging in for some values of n, we get:

n = 1000 ⇒ δ ≈ .00186
n = 700 ⇒ δ ≈ .0100
n = 400 ⇒ δ ≈ .05949

We conclude that, by sampling 700 people, there is a 99% chance that the result of the poll will be within 5% of the correct value. That does not, of course, exclude the possibility that we are really unlucky in constructing our sample.

4.2 Counting Fish

There is a lake with 1 million fish in it. We conduct an experiment on the fish in two phases. In phase 1 we catch A fish, tag them and release them back into the lake. In phase 2 we repeatedly catch and release the fish - we do this B times. What is the minimum total number (A + B) of fish we should catch to make sure that we catch a tagged fish (during phase 2) with probability greater than 99%.

The probability we don’t catch even one tagged fish during phase 2 is (1 − \frac{A}{10^{6}} )^{B}. We want this probability to be less than 0.01. Now,

(1 − \frac{A}{10^{6}} )^{B} < e^{−( \frac{AB}{10^{6}} )}

. Hence if we assure that

e^{−( \frac{AB}{10^{6}} )} < 0.01

we are done. This implies that AB > 4.6 ∗ 10^{6}. A + B is minimized when A = B for a given AB. Hence the minimum total number of fish we should catch is 2 ∗ \sqrt{4.6 ∗ 10^{6}} ∼ 4292

4.3 Leader Election

There are n persons in a room. They wish to pick one of themselves as their leader. They wish to do this in a fair and democratic way, so that each and everyone has the same chance to be the leader. The scheme they employ is for everyone to toss a coin. If exactly one person tosses a head that person is elected the leader. If no persons or more than one person tosses heads then they repeat the entire process. If the coins they use have probability p of coming up heads then what should p be to maximize the probability of selecting a leader in a given round.

If n coins are tossed then the probability of having exactly one head is \binom{n}{1}p(1 − p)^{n−1}. Differentiating w.r.t. p and equating to 0 we find the maximum occurs when p = 1/n. Hence, they should use coins so that the probability of coming up heads is 1/n. When they use such coins then the probability of selecting a leader in a given round is \binom{n}{1}(1/n)(1 − (1/n))^{n−1} ∼ 1/e.

5 Summary

Unbiased (p = 1/2) Binomial Distribution:

f(\frac{n}{2}) ∼ \sqrt{\frac{2}{πn}} = Θ(\frac{1}{\sqrt{n}})

f(αn) ∼ \sqrt{\frac{2}{πn}}\sqrt{\frac{1}{4α(1 − α)}} \frac{1}{2^{n(1−H(α))}} (α < 1/2)

General (p ≤ 1/2) Binomial Distribution:

f(pn) ∼ \sqrt{\frac{1}{2πp(1 − p)n}} = Θ(\frac{1}{\sqrt{pn}})

f(αn) ∼ \sqrt{\frac{1}{2πα(1 − α)n}}2^{n(α log \frac{p}{α} +(1−α) log \frac{p}{1−α})}