6.858/18.428 Machine Learning Lecture 19 : November 21, 1994

Lecturer: Javed Aslam Scribe: Gillian Lee

19.1 Outline

• Standard PAC Model vs. Noisy PAC Model

• Noise Models

• Minimizing disagreements to learn with classification noise

• Minimizing disagreements for conjunctions is NP-hard

The PAC model assumes access to a noise-free oracle for examples of the target concept. In reality, we need learning algorithms with at least some tolerance for mislabeled examples. In this lecture, we show that all finite concept classes can be learned with classification noise if we minimize disagreements. However, then we show that for the simple concept class of monotone conjunctions, minimizing disagreements is NP-hard. In the following lecture, we will show how to get efficient learning algorithms via the statistical query model.

19.2 Standard PAC Model vs. Noisy PAC Model

The standard PAC model assumes that the learning algorithms have access to a noise-free examples oracle for the target concept (see Figure 19.1). In the noisy PAC model, however, we may have noise which corrupts examples and/or the classification of examples (see Figure 19.2). We consider two types of noise models: the malicious noise model and the classification noise model.

19.2.1 Malicious Noise

EX_{MN}^{β}(c_{∗}, D) = \{ \begin{matrix}with probability 1 − η, returns ⟨x, c_{∗}(x)⟩ from EX(c_{∗}, D) \\ with probability η, returns ⟨x, l⟩ for maliciously chosen \\ x ∈ X and l ∈ \{0, 1\} \end{matrix}

19-1

19-2 Lecture 19 : November 21, 1994

Figure 19.1: Standard PAC Model

The malicious error noise model was first introduced by Valiant (1985). His results for CNF(n,k) hold only for a small rate of noise. Kearns and Li (1987) show that the malicious noise model can tolerate only a small rate of noise for any domain.

19.2.2 Random Classification Noise

EX_{CN}^{η}(c_{∗}, D) = \{ \begin{matrix}with probability 1 − η, returns ⟨x, c_{∗}(x)⟩ from EX(c_{∗}, D) \\ with probability η, returns ⟨x, ¬c_{∗}(x)⟩ from EX(c_{∗}, D) \end{matrix}

• label flipped with probability η

• example unchanged

This model was first introduced by Angluin and Laird (1988). We shall discuss their results below.

Lecture 19 : November 21, 1994 19-3

Figure 19.2: Noisy PAC Model

19.3 Learning With Classification Noise

We assume:

• The learner knows which type of noise exists.

• The learner has an upper bound η_{b} on noise rate (0 ≤ η ≤ η_{b} < 0.5)

As the noise rate approaches 0.5, the labels provided by the noisy oracle are providing less and less information about the target concept. The learning alorithm thus needs more oracle calls and more computation time as the noise rate approaches 0.5. When the noise rate is equal to 0.5, PAC learning becomes impossible, because every label seen by the algorithm is the outcome of an unbiased coin flip, and gives no information about the target concept.

Definition 1 C is PAC-learnable by H in the presence of noise if there exists a learning algorithm L with the property that (∀c ∈ C)(∀D on X)(∀ε, 0 < ε < 1) (∀δ, 0 < δ < 1) (∀η, 0 ≤ η < 0.5), if L is given inputs n,ε, δ, η_{b} (η ≤ η_{b} < 0.5) and access to EX^{η}(c_{∗}, D), then it will with probability ≥ 1−δ produce an ouptut hypothesis h ∈ H s.t. error(h) ≤ ε.

Definition 2 C is efficiently PAC-learnable if the running time of L is polynomial in n, \frac{1}{ε}, \frac{1}{δ}, \frac{1}{1−2η_{b}}.

Note:

• The error rate is measured with respect to the target concept and distribution.

error(h) = ∑_{x:c_{∗}(x)≠h(x)} Pr_{D}(x)

• The learner is allowed more time as η_{b} → 0.5.

19.3.1 Angluin and Laird Method

The typical noise-free PAC algorithm draws a large number of samples and outputs a consistent hypothesis. With classification noise, however, there may not be a consistent hypothesis. Angluin and Laird propose the following method.

• Draw a “large enough” sample.

• Output hypothesis c ∈ C which minimizes disagreements with the sample.

Suppose concept c_{i} has true error rate d_{i}. What is the probability p_{i} that c_{i} disagrees with a labelled example drawn from EX_{CN}^{η}(c_{∗}, D)? We have two cases.

1. EX_{CN}^{η}(c_{∗}, D) reports correctly, but c_{i} is incorrect: d_{i}(1 − η)

2. EX_{CN}^{η}(c_{∗}, D) reports incorrectly, but c_{i} is correct: (1 − d_{i})η

p_{i} = d_{i}(1 − η) + (1 − d_{i})η
p_{i} = η + d_{i}(1 − 2η)

An ε-good hypothesis has expected disagreement rate ≤ η + ε(1 − 2η). An ε-bad hypothesis has expected disagreement rate > η + ε(1 − 2η).

Figure 19.3: Disagreement Rates

We need m large enough such that an ε-bad hypothesis will not minimize disagreements with the hypothesis. Consider the point η + \frac{ε(1−2η_{b})}{2}. If an ε-bad hypothesis minimizes disagreements, then the target function must have at least as large of a disagreement rate. Thus, at least one of the following events must hold:

1. Some ε-bad hypothesis c_{i} has empirical disagreement rate ≤ η + \frac{ε(1−2η_{b})}{2}.

2. Target concept c_{∗} has empirical disagreement rate ≥ η + \frac{ε(1−2η_{b})}{2}.

We can make these probabilities small (i.e., ≤ \frac{δ}{2}) using Hoeffding Bounds:

GE(p, m, p + α) ≤ e^{−2mα^{2}}
LE(p, m, p − α) ≤ e^{−2mα^{2}}

We have:

Pr (event 2) = GE(η, m, η + \frac{ε(1 − 2η_{b})}{2})
≤ e^{−2m(\frac{ε(1−2η_{b})}{2})^{2}}

We want Pr (event 2) ≤ \frac{δ}{2}, so choose

m ≥ \frac{2}{ε^{2}(1 − 2η_{b})^{2}} ln (\frac{2}{δ}) .

We have:

Pr (event 1) ≤ LE(η + ε(1 − 2η), m, η + \frac{ε(1 − 2η_{b})}{2})|C|
≤ LE(η + ε(1 − 2η_{b}), m, η + \frac{ε(1 − 2η_{b})}{2})|C|
≤ |C|e^{−2m( \frac{ε(1−2η_{b})}{2} )^{2}}

We want Pr(event 1)≤ \frac{δ}{2}, so choose

m ≥ \frac{2}{ε^{2}(1 − 2η_{b})^{2}} ln (\frac{2|C|}{δ}) .

If we choose m larger than both values, then the probability that either of the events occurs is at most δ. That is, the probability that an ϵ-bad hypothesis minimizes disagreements is at most δ. Thus, if we draw a sample of size m as specified above, and then find a hypothesis which minimizes disagreements with the sample, then we have an algorithm which PAC learns in the presence of classification noise. This gives the following theorem.

Theorem 1 For all finite concept classes C, we can PAC learn in the presence of classification noise.

19.3.2 Minimizing disagreements can be NP-hard

Theorem 2 Finding monotone conjunctions which minimize disagreements with a given sample is NP-hard.

Sketch of Proof: Reduce the vertex cover problem to that of finding a monotone conjunction which minimizes disagreements with a given sample. Given a graph G = (V, E), and a constant c, the vertex cover problem is to determine whether there is a vertex cover of size at most c (i.e., whether there is a subset of the nodes of size at most c such that every edge in the graph is adjacent to some vertex in this subset). Our reduction is given by the following table.

Graph  Monotone Conjunction
V = {v_{1}, v_{2}, ...v_{n}} {x_{1}, x_{2}, ...x_{n}} define the instance space.
∀v_{i} ∈ V  example (⟨11 ⋯ 101 ⋯ 11⟩, +), with 0 in i-th position.
∀ edges (v_{i}, v_{j}) ∈ E c + 1 examples (⟨11 ⋯ 101 ⋯ 101 ⋯ 11⟩, −),

with 0’s in i-th and j-th positions.

If there are n vertices in the graph, then our instance space is {0, 1}^{n}. For each vertex in the graph, we introduce a positive example, and for each edge in the graph we introduce c + 1 negative examples. It is straightforward to show the following.

Claim 1 G has a vertex cover of size at most c iff there is a monotone conjunction with at most c disagreements.

References

[1] D. Angluin and P. Laird. Learning from noisy examples. Machine Learning, 2(4):343-370, 1988.

[2] M. Kearns and M. Li. Learning in the presence of malicious errors. (Technical Report TR-03-87). Cambridge, MA: Harvard University, Center for Research in Computing Technology.

[3] M. Kearns and U. Vazirani. An Introduction to Computational Learning Theory. MIT Press, 1994.

[4] L. G. Valiant. Learning disjunctions of conjunctions. Proceedings of the Ninth International Joint Conference of Artificial Intelligence (pp. 560-566). Los Angeles, CA: Morgan Kaufmann.