6.858/18.428 Machine Learning Lecture 10: October 17, 1994

Lecturer: Ron Rivest Scribe: Mikael Goldmann

10.1 Outline

• A lower bound on sample complexity through VC-dimension.

• Cover’s coin problem.

10.2 A lower bound on the sample complexity through VC-dimension

In previous lectures it has been shown that the VC-dimension of a concept class gives an upper bound on the number of samples needed to PAC-learn concepts from the class. In particular it has been shown that for VC-dimension d it is possible to PAC-learn with parameters δ and ϵ given that the number of samples m is at least

m ≥ c_{0} (\frac{d}{ϵ} log \frac{1}{ϵ} + \frac{1}{ϵ} log \frac{1}{δ})  (10.1)

where c_{0} is a fixed constant. So, as long as VC-dim(C) is finite, it is possible to PAC-learn concepts from C even though |C| might be infinite.

One might ask if there are concept classes of infinite VC-dimension that can be learned from finite sample size, or if at least the upper bound of (10.1) can be improved in any substantial way to, say, O(d log(1/ϵ)). The answer to both these questions is “No.” It is pretty clear that one cannot PAC-learn a concept class C of infinite VCdimension. Say that we have an algorithm that makes m queries to the examples oracle (m depends only on δ and ϵ), and let Y be a subset of the domain of size ≥ 2m that is shattered by C. After m examples A knows nothing about at least half the points in Y, and therefore it cannot predict the labels on those points better than a random coin toss. In fact, for finite VC-dimension equation (10.1) is nearly tight, and we will show that for VC-dimension d, achieving error rate ϵ requires c_{1}d/ϵ samples for some fixed constant c_{1}.

10-2 Lecture 10: October 17, 1994

10-1 Let X = {x_{1}, …, x_{d}} and let C = 2^{X}. Obviously, VC-dim(C) = d. We will show that PAC-learning C requires Ω(d/ϵ) examples.

Theorem 1 Let X = {x_{1}, …, x_{d}} and let C = 2^{X}. Any algorithm A that PAC-learns C with parameters ϵ and δ ≤ 1/15 must use more than (d − 1)/(64ϵ) queries to the examples oracle.

This is in fact a general result since PAC-learning C is a sub-problem of PAC-learning any concept class of VC-dimension d.

Corollary 1 Let C be an arbitrary concept class and let d = VC-dim(C). To PAClearn C with parameters ϵ and δ ≤ 1/15 requires more than (d − 1)/(64ϵ) queries to the examples oracle.

Proof of Theorem 1: Let m = (d − 1)/(64ϵ), and A be an algorithm that makes at most m queries to the examples oracle and then produces a hypothesis h. We need to show that there are a distribution p on X and a concept c ∈ C such that er(h) > ϵ with probability at least 1/15.

Let us first fix p independently of A:

p(x_{1}) = 1 − 16ϵ

p(x_{2}) = p(x_{3}) = ⋯ = p(x_{d}) = \frac{16ϵ}{d − 1}

In what follows S is a random sample of X that we get by m independent draws from p. We want to establish that there is a c so that Pr_{S}[er(h) > ϵ] > \frac{1}{15}.

Let X′ = {x_{2}, …, x_{d}}. For any fixed c ∈ C and hypothesis h, let er′(h) = Pr[c(x) ≠ h(x)∧x ∈ X′]. For technical reasons, it is easier to prove that Pr_{S}[er′(h) > ϵ] > 1/15, which is enough since er′(h) ≤ er(h).

We pick a random c ∈ C and show that with positive probability c is hard to learn for A, thereby showing that there must be some fixed c that is hard to learn for A.

Let B be the event that A’s sample S contains less than (d − 1)/2 points in X′. We establish three things:

Pr_{S}[B] ≥ 1/2  (10.2)
E_{c,S}[ er′(h) | B] > 4ϵ  (10.3)
E_{S}[ er′(h) | er′(h) > ϵ] ≤ 16ϵ for any fixed c.  (10.4)

Lecture 10: October 17, 1994 10-3

Inequality (10.4) is obvious since for any h we have er′(h) ≤ Pr[x ∈ X′] = 16ϵ. However, if we consider er(h) rather than er′(h), then (10.4) does not necessarily hold, which is the reason for introducing er′(h). Both (10.2) and (10.3) make intuitive sense: if A only sees a few samples it is likely to miss a large fraction of X′ (this is (10.2)), and if A knows nothing about a large fraction of X′ then the error rate of the hypothesis is likely to be high (this is (10.3)). Before formally proving (10.2) and (10.3) we show that they imply that there is a c_{∗} ∈ C so that Pr_{S}[er′(h) > ϵ] > 1/15.

First we use (10.2) and (10.3) to get a lower bound on E_{c,S}[er′(h)].

E_{c,S}[er′(h)] ≥ Pr_{S}[B] · E_{c,S}[ er′(h) | B] > \frac{1}{2} · 4ϵ = 2ϵ.

In particular, there is some c_{∗} ∈ C such that E_{S}[er′(h)] > 2ϵ. We take c_{∗} as the target concept and show that A is likely to produce a hypothesis with high error rate. Using conditional expectation we get

2ϵ < E_{S}[er′(h)]
= Pr_{S}[er′(h) > ϵ] · E_{S}[ er′(h) | er′(h) > ϵ]
+(1 − Pr_{S}[er′(h) > ϵ]) · E_{S}[ er′(h) | er′(h) ≤ ϵ].

Next we apply (10.4) to get

2ϵ < E_{S}[er′(h)] ≤ Pr_{S}[er′(h) > ϵ] · 16ϵ + (1 − Pr_{S}[er′(h) > ϵ]) · ϵ
= 15ϵ Pr_{S}[er′(h) > ϵ] + ϵ,

which implies Pr_{S}[er′(h) > ϵ] > 1/15.

It remains to prove (10.2) and (10.3).

Proof of (10.2). Let T be the number of times A gets a sample point in X′. Clearly, E[T] = 16ϵm = (d − 1)/4. We have Pr_{S}[B] ≥ 1 − Pr[T ≥ (d − 1)/2], and by Markov’s inequality,^{1} Pr[T ≥ (d − 1)/2] ≤ 1/2.

Proof of (10.3). Let S be the set of sample points that A gets. Choosing a random c is equivalent to flipping a fair coin for each point in X to determine its label. Since h is independent of the labeling of X′ −S, the contribution to er′(h) is expected to be 16ϵ/(2(d−1)) for each point in X′ −S. When B occurs, we have |X′ −S| > (d−1)/2 and thus the expected value of er′(h) given B is strictly greater than 4ϵ.

The proof is now complete.

1Markov’s inequality: if X is a positive random variable, then Pr[X ≥ t] ≤ E[X]/t.

10.3 Cover’s coin problem

Cover’s coin problem is somewhat connected to the idea of “learning the bias of a coin.” The precise statement of the problem is as follows:

Cover’s coin problem INPUT: A coin. OUTPUT: “rational” if Pr[heads] is rational and “irrational” otherwise.

At first glance the question might look strange, especially since the maximum likelihood estimate p̄ of p = Pr[heads] after a finite number of coin flips is always rational (p̄ = #heads/#flips). On the face of it, the problem appears to be impossible. In order to get a better understanding, let us look at a simpler problem and see in what sense it can be solved.

Fair coin problem INPUT: A coin. OUTPUT: “fair” if Pr[heads] = 1/2 and “biased” otherwise.

It is possible to devise an algorithm F for the “Fair coin problem” that flips the coin a few times, guesses “fair” or “biased,” then flips again and guesses, and so on. With probability 1, the infinite sequence of guesses will contain only a finite number of mistakes.

F:

For i = 1, 2, 3, … δ_{i} = 2^{−i} N_{i} = 4^{i}i Flip N_{i} times and let H be the number of “heads” observed If |H − N_{i}/2| > δ_{i}N_{i} then output “fair” else output “biased”

End for

Consider first the case that the coin is fair, and let B_{i} be the event that the ith output is “biased.” Using the additive version of the Chernoff bounds we get

Pr[B_{i}] = Pr[|H − N_{i}/2| > δ_{i}N_{i}] ≤ 2e^{−2N_{i}δ_{i}^{2}} = 2e^{−2i}.

The following result is very useful when studying the behavior of F.

Theorem 2 (Borel–Cantelli lemmas) Let A_{1}, A_{2}, … be a countable sequence of events, and let A = ∩_{n≥1} ∪_{m≥n} A_{i} be the event that infinitely many of the A_{i}’s occur.

Then

Pr[A] = 0 if ∑_{i≥1} Pr[A_{i}] < ∞,
Pr[A] = 1 if ∑_{i≥1} Pr[A_{i}] < ∞ and A_{1}, A_{2}, … are independent events.

In our case ∑_{i≥1} Pr[B_{i}] < ∞, which implies that F ouputs “fair” all but finitely often.

The second case is when the probability of heads is p ≠ 1/2. In this case, with probability 1, it only happens finitely many times that H/N_{i} ∉ [p − δ_{j}, p + δ_{j}]. When j is sufficiently large, the intervals [p − δ_{j}, p + δ_{j}] and [\frac{1}{2} − δ_{j}, \frac{1}{2} + δ_{j}] are disjoint, so F will say “fair” only finitely many times with probability 1.

Let us now return to Cover’s coin problem. We will construct an algorithm with the following properties. If p = Pr[heads] is rational, then with probability 1 the algorithm will output “irrational” only a finite number of times. For all irrational p, except for a set of measure 0, the algorithm will output “rational” only finitely many times with probability 1.

What we would like to do is to put shrinking intervals around all rational numbers, but these intervals would of course cover [0, 1] and any irrational number would always be close to some rational number. The key is to consider a slowly growing, but always finite, set of rationals. If p is rational, then it will eventually become a member of the set, and with probability 1 the empiric average p̄ will be close to p all but finitely often. On the other hand, If p is irrational, the union of the intervals around the rationals in the finite set shrinks in size, and eventually p̄ should always fall outside this union. This is almost what happens. The problem is that while the union of intervals has smaller and smaller measure, we keep adding more rationals to our finite set, and therefore an irrational number that is not in an interval at some point may be too close to a rational number that is about to be added to the set.

While the ideas are the same as in the “Fair coin problem,” the technical details are a bit more involved. Let q_{1}, q_{2}, … be an arbitrary enumeration of ℚ ∩ [0, 1], and 𝕀_{k} be {q_{1}, q_{2}, …, q_{k}}. Define

𝕀_{k}(δ) = ∪_{i=1}^{k}[q_{i} − δ, q_{i} + δ]
N_{0} = 0
N_{k} = 2^{k}
δ_{k} = \sqrt{(ln k)/2^{k}}

G: h = 0 n = 0 guess = “rational” For k = 1 to ∞

For n = N_{k−1} + 1 to N_{k} h_{n} = flip h = h + h_{n} p̄_{n} = h/n output guess

End for

If p̄_{N_{k}} ∈ 𝕀_{k}(δ_{k}) then guess = “rational” else guess = “irrational” End for

The following lemma is useful when analyzing G.

Lemma 1 If p = Pr[heads] then p̄_{N_{k}} ∈ [p − δ_{k}, p + δ_{k}] for all but finitely many k with probability 1.

Proof: Let A_{k} be the event that p_{N_{k}} ∉ [p − δ_{N_{k}}, p + δ_{N_{k}}]. Using the additive version of the Chernoff bounds we

Pr[A_{k}] ≤ 2e^{−2N_{k}δ_{k}^{2}} = 2/k^{2}.

Therefore ∑_{k≥1} Pr[A_{k}] < ∞, and the lemma follows by the Borel–Cantelli lemmas.

We consider three cases.

Case 1: p is rational. There is an l such that p = q_{l} ∈ 𝕀_{k} for all k ≥ l. By Lemma 1 p̄_{k} ∈ [q_{l} − δ_{k}, q_{l} + δ_{k}] for all but finitely many k with probability 1, and [q_{l} − δ_{k}, q_{l} + δ_{k}] ⊂ 𝕀_{k}(δ_{k}) for k ≥ l. This implies that G will output “rational” all but finitely often with probability 1.

Case 2: p ∈ 𝕀_{k}(2δ_{k}) for finitely many k. In this case there is some l so that [p − δ_{k}, p + δ_{k}] ∩ 𝕀_{k}(δ_{k}) = ∅ for k ≥ l. Together with Lemma 1 this implies that the output will be “irrational” all but finitely often with probability 1.

Case 3: p is irrational, but p ∈ 𝕀_{k}(2δ_{k}) for infinitely many k. This case is bad since the output may be “rational” infinitely often. However, we will show that the Lebesgue measure of the set of such p is 0.

First note that

{ p | p ∈ 𝕀_{k}(2δ_{k}) for infinitely many k } ⊂ ⋃_{k≥m} 𝕀_{k}(2δ_{k}) for all m ≥ 0.

Since λ(𝕀_{k}(2δ_{k})) ≤ 4kδ_{k}, we have

λ ( ⋃_{k≥m} 𝕀_{k}(2δ_{k})) ≤ ∑_{k≥m} 4kδ_{k}.

The sum ∑_{k≥1} kδ_{k} converges, so ∑_{k≥m} 4kδ_{k} tends to 0 as m tends to ∞. This implies that λ{ p | p ∈ 𝕀_{k}(2δ_{k}) for infinitely many k } = 0.

References

[1] T. M. Cover, On determining the irrationality of the mean of a random variable. The Annals of Satistics, 1(5):862–871, 1973.

[2] A. Ehrenfeucht, D. Haussler, M. Kearns, and L. Valiant. A general lower bound on the number of examples needed for learning. Information and Computation, 82(3):247–251, 1989.

[3] M. Kearns and U. Vazirani. An Introduction to Computational Learning Theory, 52–64, 1994.