18.858 Machine Learning Lecture 5 : September 26, 1994

Lecturer: Ron Rivest Scribe: Hugh B. Morgenbesser

Outline

• PAC learning 3-term DNF by 3-term DNF is intractable.

• Occam’s algorithms (simple case).

• Learning 3-term DNF by 3-CNF.

5.1 Intractability of PAC learning 3-term DNF by 3-term DNF

After talking and thinking about PAC learning, it is interesting to try to determine what types of concept classes are PAC learnable. In the last lecture, we saw that the concept class consisting of conjunctions is PAC learnable. Here, we will show that the class of 3-term DNF is probably not PAC learnable [2]. The “probably not” comes from a reliance on the assumption that NP-complete problems cannot be solved efficiently by a randomized algorithm. In other words, we assume here that RP ≠ NP.

The general idea of this method is to choose a known NP-complete problem which we can reduce to the problem of PAC learning 3-term DNF formulae. This will prove that we cannot PAC learn 3-term DNF formulae by 3-term DNF formulae unless RP = NP.

In this learning problem, we are targeting 3-term DNF formulae. An example of a 3-term DNF formula is:

x_{1}\overline{x}_{2}x_{3}x_{4} ∨ x_{2}\overline{x}_{4}\overline{x}_{6} ∨ x_{7}x_{8}\overline{x}_{9}.

5-1

5-2 Lecture 5 : September 26, 1994

5.1.1 Graph 3-colorability

The problem of determining whether a given graph G with vertices V and edges E can be colored using 3 or less colors (where no edge connects two vertices of the same color) is a known NP-complete problem. For more information on this and other known NP-complete problems, see the Garey and Johnson text [1].

We desire some procedure to map these graphs G = (V, E) to 3-term DNF target formulae. In other words, given an instance G we would like to compute a set S of labelled examples such that G is 3-colorable if and only if S is consistent with some 3-term DNF formula.

Note that in our mapping from G to S, S will be the size of V + E. Note also that the method for determining S does not require that we know whether or not G is 3-colorable.

5.1.2 What does learner get?

We give the learner a source of labelled examples, and values for ϵ and δ.

• oracle for labelled examples: the examples are chosen uniformly at random from S (defines D).

• choose ϵ = \frac{1}{|S|+1} so that hypothesis must be consistent with all examples in S.

• and choose an arbitrary small δ, say δ = 0.00001.

We will now prove that there is a transformation that gives the desired property that: G is 3-colorable ⟺ S is consistent with some 3-term DNF formula. This will prove the equivalence of PAC learning 3-term DNF formula and the existence of an RP algorithm for determining graph 3-colorability.

Note that if G is not 3-colorable we are asking the learner to do the impossible. It can’t produce a 3-term DNF formula, because we have ensured that it doesn’t exist.

Furthermore, if we have an algorithm for learning 3-term DNF, we have a randomized algorithm for graph 3-colorability. The algorithm would be to convert G to S and to try to learn S. The algorithm would conclude that the graph was 3-colorable if and only if it was able to learn S.

If the graph is 3 colorable, the learner will come up with a 3-term DNF formula consistent with S. If not, the learner will do something else (either halt, return bad

Lecture 5 : September 26, 1994 5-3

[figure: .; 01011-; 11001-; 11010-; 10011-; 10110-; 01101-; 01111+; v1; 10111+; v2; 11101+; v4; 11110+; v5; v3; 11011+]

Figure 5.1: Example of the mapping from a graph to a set of labelled examples.

formula, etc.). So we can test the resultant formula against S. If we do not find a consistent hypothesis, then G is not 3-colorable (with probability ≥ 1 − δ).

Construction: The transformation from a graph G to a set of labelled examples S is done as follows (as shown in Figure 5.1). For each vertex v_{i}, a positive example x^{(i)} is created, where x^{(i)} = x_{1}x_{2}x_{3} … x_{n}, and x_{i} = 0 and x_{j} = 1 for all j ≠ i. For each edge, (v_{i}, v_{j}), a negative example is created e_{ij} = x_{1}x_{2}x_{3} … x_{n}, where x_{i} = x_{j} = 0 and x_{k} = 1 for all k ≠ i, j.

This is clearly constructible in polynomial time.

First we show the ⇒ direction. Assume G is 3-colorable. Then, by the definition of 3-colorability, there exist sets R, B, and Y of vertices that are disjoint and color the graph. For example, in the graph in the figure, sets of R = {v_{1}, v_{5}}, B = {v_{2}, v_{4}}, and Y = {v_{3}} represent a valid 3-coloring. That is, in our example, by coloring vertices v_{1} and v_{5} red, vertices v_{2} and v_{4} blue, and v_{3} yellow, no two adjacent vertices have the same color.

In this case, the following terms of a 3-term DNF will have the desired property.

T_{R} = ⋀_{x∈B∪Y} x = x_{2}x_{4}x_{3}

T_{B} = ⋀_{x∉B} x = x_{1}x_{3}x_{5}

T_{Y} = ⋀_{x∉Y} x = x_{1}x_{2}x_{4}x_{5}

Claim 1 F = T_{R} ∨ T_{B} ∨ T_{Y} is consistent with S.

1. F is true on all positive examples: If v_{i} ∈ R then x^{(i)} satisfies T_{R}. This implies that F is true.

2. F is false on all negative examples: Say (v_{i}, v_{j}) is an edge. Then v_{i} and v_{j} aren’t both red since this a valid 3-coloring. Assume that v_{i} isn’t colored red. Then, by definition, literal x_{i} is in T_{R} And thus the negative example corresponding to edge (v_{i}, v_{j}) doesn’t satisfy T_{R}. Similarly we can show it doesn’t satisfy T_{Y} or T_{B}.

So, since 1 and 2 are both true, we see that if G is 3-colorable, then S is consistent with some DNF formula.

For reverse direction ⇐, we assume that some 3-term DNF formula is consistent with S, i.e. T_{R} ∨ T_{B} ∨ T_{Y}. Call vertex v_{i} red if example x^{(i)} satisfies T_{R}, blue if it satisfies T_{B}, and yellow if it satisfies T_{Y}. (Ties may be broken arbitrarily).

Claim 2 No two adjacent vertices have the same color. Suppose that vertices v_{i} and v_{j} are both colored red, then the examples x^{(i)} and x^{(j)} satisfy T_{R} by definition. If v_{i} and v_{j} are adjacent, then e_{ij} = x^{(i)} ∧ x^{(j)} (bitwise). But x^{(j)} and e_{ij} differ only in i^{th} bit so then e_{ij} must also satisfy T_{R}. But e_{ij} is a negative example, so this is a contradiction. So no two adjacent vertices have the same color, and we have colored the graph using 3-colors.

From this we see that PAC learning of 3-term DNF is as “tough” as an NP-complete problem ^{1}.

This result is important, but we won’t give up yet. We still want efficient PAC learning algorithms. But how? One idea is explored below. The basic idea is to learn a concept class C using a larger hypothesis class, H.

But first, let us talk about Occam algorithms.

5.2 Occam Algorithms

An Occam algorithm is just a general structure for how to apply a PAC learning algorithm. Simply stated it is two steps:

• draw a sample of size m = poly(n, size(H))

• return any h ∈ H consistent with sample.

We define h to be “bad” if error(h) ≥ ϵ. Then the probability h survives m examples is ≤ (1 − ϵ)^{m}. The probability that some bad hypothesis survives m examples is then ≤ |H|(1 − ϵ)^{m}. We want that probability to be ≤ δ, so we require:

|H|(1 − ϵ)^{m} ≤ δ.

We can get this inequality into another form by taking the natural logarithm of both sides:

ln |H| + m ln(1 − ϵ) < ln(δ)

or equivalently:

m > \frac{ln(δ) − ln(|H|)}{ln(1 − ϵ)}

1Note to the curious: k-term DNF is hard for any k ≥ 2.

and since ln(1 − ϵ) < −ϵ, we see that:

m > \frac{1}{ϵ}(ln(|H|) + ln(\frac{1}{δ}))

.

5.3 PAC Learning 3-term DNF by 3-CNF

3-CNF is Conjunctive Normal Form with at most 3 literals per clause.

⋀_{i}(x_{i_{1}} ∨ x_{i_{2}} ∨ x_{i_{3}})

For example, (a ∨ b)(\overline{c} ∨ b ∨ d)(\overline{a} ∨ e), is a 3-CNF formula. Every 3-term DNF formula is equivalently representable by a 3-term CNF formula, i.e.

T_{1} ∨ T_{2} ∨ T_{3} = ⋀_{x∈T_{1},y∈T_{2},z∈T_{3}}(x ∨ y ∨ z)

.

The converse is not true. Many 3-CNF formulae cannot be expressed in 3-term DNF. 3-CNF is more expressive than 3-term DNF.

How big are these classes? Well, ln(|3-term DNF|) = θ(n), and ln(|3-CNF|) = θ (\binom{2n}{3}) = θ(n^{3}). So we see that 3-CNF is bigger than 3-term DNF.

To learn 3-term DNF by 3-CNF, we use an Occam style algorithm.

1. Pick enough samples (given by formula for m in Section 5.2).

2. To find a hypothesis consistent with labeled examples, reduce the problem to learning conjunctions. This is done by creating a new set of labelled examples on θ(n^{3}) variables, one for each possible 3-CNF clause. Then the problem is as easy as learning conjunctions.

References

[1] Michael R. Garey and David S. Johnson, Computers and Intractability: A Guide to the theory of NP-Completeness. Freeman, 1979.

[2] L. Pitt and L. Valiant. Computational limitations on learning from examples. Journal of the ACM, 35:965-984, 1988.