6.858/18.428 Machine Learning Lecture 12: October 24, 1994

Lecturer: Ron Rivest Scribe: Lillian Lee

12.1 Outline

• Definition of weak learnability

• Weak learnability implies PAC-learnability

–Boosting confidence

–Boosting accuracy

12.2 Weak Learning

Notice that the definition of PAC-learning imposes some rather stringent conditions, among them the requirement that a PAC-learning algorithm be able to output hypotheses that have arbitrarily low error rates, and be able to do so with arbitrarily high confidence. In his Ph.D. thesis, Schapire considered (among other topics) the weak learning model, which differs from the PAC-learning model in that the learning algorithm need only achieve a fixed (but small) error rate and confidence. We go through his proof of the rather surprising fact that if a concept class is weakly learnable, then it is PAC-learnable.

12.2.1 Definition of Weak Learnability

Let EX(c, D) be as before.

Definition 1 A concept class C is weakly learnable if there exists a learning algorithm L and constants ϵ_{0} < \frac{1}{2} and δ_{0} < 1 such that for every concept c_{∗} ∈ C and every distribution D on X, algorithm L, with access to EX(c, D), returns a hypothesis h such that with probability at least 1 − δ_{0}, err(h) is at most ϵ_{0}.

12-2 Lecture 12: October 24, 1994

12-1 Notice that we can define “weakly learnable using H” and “efficiently weakly learnable” in the same fashion as we did for PAC-learnability. Also note that this definition differs slightly from that given in the book (and in the next lecture).

12.2.2 Weak learnability implies PAC-learnability

Our main theorem is the following:

Theorem 1 If a concept class C is weakly learnable by H, then it is PAC-learnable by H′, where H′ is the closure of H under majority operations.

We prove this theorem by first showing that we can boost the confidence of certain learning algorithms, and then showing that we can boost the accuracy of certain learning algorithms.

12.2.3 Boosting confidence

Suppose we have a weak learning algorithm L that achieves arbitrarily low error rates, but only does so with confidence at most δ_{0}. Then, we can raise the confidence to any δ we want (thereby creating a PAC-learning algorithm) by running the algorithm many times and picking the output hypothesis with the lowest estimated error rate.

Suppose we run L k times with parameters ϵ/2 and δ_{0}. Let h_{1}, h_{2}, …, h_{k} be the independent hypotheses that L outputs. Then we have

(∀i)Pr\{err(h_{i}) ≤ \frac{ϵ}{2}\} ≥ 1 − δ_{0}
⇒ Pr\{(∀i)err(h_{i}) > \frac{ϵ}{2}\} ≤ δ_{0}^{k}
⇒ Pr\{(∃i)err(h_{i}) ≤ \frac{ϵ}{2}\} ≥ 1 − δ_{0}^{k}.

Choose k so that δ_{0}^{k} ≤ δ/2, that is, k ln δ_{0} ≤ ln(δ/2) ⇒ k ≥ log_{δ_{0}}(δ/2), since δ_{0} < 1. Then, with probability at least 1 − δ/2, we have some hypothesis h_{i} such that err(h_{i}) ≤ ϵ/2.

Lecture 12: October 24, 1994 12-3

Now it remains to figure out which of our h_{i}’s has the lowest error rate; we do this by drawing a sample of size m for each h_{i} to estimate each h_{i}’s error rate. It’s sufficient that our estimate be accurate within ϵ/4 because our aim is to output a hypothesis with error rate at most ϵ: if err(h_{i}) ≤ ϵ/2 then our error estimate will be at most 3ϵ/4, whereas if err(h_{i}) > ϵ, our error estimate will be above 3ϵ/4. The additive form of the Chernoff bound gives us that GE(p, m, (p + ϵ/4)) ≤ exp(−2m(ϵ/4)^{2}); setting the latter quantity less than or equal to δ/(2k) implies that m ≥ \frac{8}{ϵ^{2}} ln(2k/δ).

What is the chance that we output a hypothesis with error greater than ϵ with the above procedure? Well, either among our k tries we never get a hypothesis with error rate ≤ ϵ/2 or we make a bad mistake when estimating error rates. But the chance of either of these event occurring is ≤ δ/2 + δ/2 = δ, as desired.

We remark that there is also a method for taking an algorithm with fixed ϵ_{0} and δ_{0} and constructing an algorithm which, on inputs ϵ and δ, outputs with probability at least 1 − δ a hypothesis with error rate at worst only slightly larger than ϵ.

12.2.4 Boosting accuracy

Now suppose we have a weak learning algorithm L that outputs hypotheses with error rate at most β < 1/2 but does so with probability at least 1 − δ for any input δ. We then take advantage of the fact that weak learning algorithms can learn from any distribution to find hypotheses h_{1}, h_{2}, and h_{3}, formed on three different distributions. Our output is h = majority(h_{1}, h_{2}, h_{3}) (this is why we learn using H′); h will have a lower error rate with high probability.

Let c_{∗} be the concept we are trying to learn, and let D_{1} = D. The first step is to run L with EX(C, D) to get a hypothesis h_{1} such that err(h_{1}) = β_{1} ≤ β < 1/2. Then, we want to focus L’s attention where it does poorly by running L on a new distribution that gives greater weight to examples on which h_{1} errs.

A first attempt to do this would be to only give L examples where the target concept c_{∗} differs from h_{1}. However, the result of this might be that L outputs exactly the complement of h_{1}, which is not necessarily any better than h_{1}. So instead, let us consider a new oracle which flips a coin: on heads, it returns an example x such that h_{1}(x) ≠ c_{∗}(x), and on tails, it returns an example x such that h_{1}(x) = c_{∗}(x) (in the next lecture, we will discuss what to do if it is hard to get such a distribution). Let D_{2} be this new distribution. Then, run L using the oracle EX(c, D_{2}); let h_{2} be the output. Let β_{2} ≤ β be the error rate of h_{2} on D_{2}.

The next step is to find a hypothesis h_{3} to act as a tiebreaker. We will learn this hypothesis by using yet another distribution D_{3} that consists entirely of examples where h_{1} and h_{2} disagree (if drawing examples from D_{3} is too hard (which means that h_{1} and h_{2} are “too good”), we output h_{1} or h_{2}). Let β_{3} ≤ β be h_{3}’s error rate on D_{3}.

Finally, we output h = majority(h_{1}, h_{2}, h_{3}).

Claim 1 err_{D}(h) ≤ g(β), where g(ϵ) = 3ϵ^{2} − 2ϵ^{3}.

Proof:

D1  E1 B1  D2 B2  E2 A1 A2  C2=1/2 C1 D1 D2::  γ_{1}  γ_{2} 1 − β_{1}  β_{1}

Figure 12.1: Mapping from D_{1} to D_{2}

Consider the figure below. C1-E1 is the region where h_{1} is wrong, B2-D2 is the region where h_{2} is wrong, and C1-D1 is the region where both are wrong. B2-C2 has weight γ_{1} under D_{2}; C2-D2 has weight γ_{2} under D_{2}. Notice that regions to the right of C1 are mapped to regions with probability 1/(2β_{1}) times more under D_{2} than they had under D_{1}, whereas regions to the left of C1 are mapped to regions with probability 1/(2(1 − β_{1})) times less under D_{2} than they had under D_{1}. Applying this mapping backwards gives us the following information:

Region Weight under D_{1}
A1-B1 2(1 − β_{1})(1/2 − γ_{1})
B1-C1 2(1 − β_{1})γ_{1}
C1-D1 2β_{1}γ_{2}
D1-E1 2β_{1}(1/2 − γ_{2})

From now on, assume all weights are with respect to D. We have:

err(h) ≤ Pr{h_{1} ≠ c_{∗} and h_{2} ≠ c_{∗}} + Pr{h_{1} ≠ h_{2} and h_{3} wrong}
≤ weight(C1-D1) + (weight(B1-C1) + weight(D1-E1))(β_{3})
≤ 2β_{1}γ_{2} + (2(1 − β_{1})γ_{1} + 2β_{1}(1/2 − γ_{2}))β, since β_{3} ≤ β (∗)

= β_{1}(2γ_{2} − 2βγ_{1} + 2β(1/2 − γ_{2})) + 2βγ_{1}
= β_{1}(2γ_{2} − β(2(γ_{1} + γ_{2}) − 1)) + 2βγ_{1}
= β_{1}(2γ_{2} + β(1 − 2(γ_{1} + γ_{2})) + 2βγ_{1}.

But γ_{1} + γ_{2} = β_{2} ≤ β < 1/2 so 1 − 2(γ_{1} + γ_{2}) is positive ⇒ (2γ_{2} + β(1 − 2(γ_{1} + γ_{2})) is positive. So err(h) is an increasing function of β_{1}, which means we can replace β_{1} by β in (*). Thus,

err(h) ≤ 2βγ_{2} + β(2(1 − β)γ_{1} + 2β(1/2 − γ_{2}))
= 2βγ_{2} + 2βγ_{1} − 2β^{2}γ_{1} + β^{2} − 2β^{2}γ_{2}
= 2βγ_{2}(1 − β) + 2βγ_{1}(1 − β) + β^{2}
= β^{2} + 2β(1 − β)(γ_{1} + γ_{2})
≤ β^{2} + 2β(1 − β)β
= 3β^{2} − 2β^{3}
= g(β)

12.2.5 Next lecture

To finish the proof, we still need to show that we can simulate EX(c, D_{2}) and EX(c, D_{3}); after all, we would run into trouble if it were never the case that h_{1}(x) ≠ h_{2}(x). Also, it remains to be shown that we can iterate the above procedure to get the error rate down to arbitrary levels in a reasonable amount of time.

References

[1] R. E. Schapire. The Design and Analysis of Efficient Learning Algorithms. MIT Press, Cambridge, MA, 1992.