6.858/18.428 Machine Learning Lecture 22: November 30, 1994

Lecturer: Ron Rivest Scribe: Rebecca Hwa

22.1 Outline

• Finish up Learning Decision Trees using the Fourier Spectrum with Respect to Uniform Distribution

• Learning DNF with Membership Queries with Respect to Uniform Distribution

22.2 Learning Decision Trees using the Fourier Spectrum

In this lecture we continue the discussion on Kushilevitz and Mansour’s polynomial time algorithm for learning decision trees with respect to the uniform distribution using membership queries.

22.2.1 Approximation by sparse functions

We define the basis function χ_{z} for each z ∈ {0, 1}^{n}:

χ_{z}(x) = (−1)^{z·x mod 2}

Let f : {0, 1}^{n} → {0, 1} be the boolean function that represents the target decision tree. To express f as a Fourier series, we expand it as a linear combination of the χ_{z}’s:

f(x) = ∑_{z∈{0,1}^{n}} f̂(z)χ_{z}(x)

where the Fourier coefficients are:

f̂(z) = E[fχ_{z}].

22-1

22-2 Lecture 22: November 30, 1994

[figure: .; f̂(111); f; f_{α}; f̂(100)]

Figure 22.1: the coefficient tree.

We can visualize the Fourier coefficients of f as being arranged on a complete binary tree of depth n, where each path down the tree from the root to a leaf node represents some vector z ∈ {0, 1}^{n}, and the leaf node represents the coefficient f̂(z). We label each node in the tree by the path vector from the root to it. Let α ∈ {0, 1}^{k} be the label of an internal node at depth k. (See Figure 22.1). We define a function at node α, which is f_{α} : {0, 1}^{n−k} → ℜ:

f_{α}(x) = ∑_{β∈{0,1}^{n−k}} f̂(αβ)χ_{β}(x)

So the coefficients of f_{α}(x) are all the coefficients of f that start with α. The process of finding large coefficients for f, then, can be broken down into finding the large coefficients of the left and right subtrees recursively.

Lemma 1 Let f be a boolean function, and θ > 0. Then,

1. At most 1/θ^{2} of z satisfy |f̂(z)| ≥ θ.

2. At any level of the tree at most 1/θ^{2} of the nodes have E[f_{α}^{2}] ≥ θ^{2}.

Lecture 22: November 30, 1994 22-3

Proof: By Parseval’s Identity and remembering that f is a boolean function whose coefficients sum to 1, we have:

E[f^{2}] = ∑_{z∈{0,1}^{n}} f̂^{2}(z) = 1

It then follows that there are at most 1/θ^{2} large coefficients (i.e., |f̂(z)| ≥ θ). Similarly, for f_{α}:

E[f_{α}^{2}] = ∑_{β∈{0,1}^{n−k}} f̂^{2}(αβ)

But if |f̂(αβ)| ≥ θ for some β ∈ {0, 1}^{n−k}, then E[f_{α}^{2}] ≥ θ^{2}. By the above equalities we get

∑_{α∈{0,1}^{k}} E[f_{α}^{2}]

= ∑_{α∈{0,1}^{k}} ∑_{β∈{0,1}^{n−k}} f̂^{2}(αβ)
= ∑_{z∈{0,1}^{n}} f̂^{2}(z) = 1

So at most 1/θ^{2} nodes have E[f_{α}^{2}] ≥ θ^{2}.

We now present the algorithm Coef(α). The initial call is Coef(λ), where λ is the empty string.

Subroutine Coef(α)

if E[f_{α}^{2}] ≥ θ^{2} then

        if |α| = n then output α
            else Coef(α0);Coef(α1)

Corollary 1 Coef examines at most n/θ^{2} nodes of the tree (i.e., it’s invoked at most n/θ^{2} times).

Proof: From lemma 1 we know that the number of α’s for which E[f_{α}^{2}] ≥ θ^{2} is at most 1/θ^{2}, for each length of α. So the total number of recursive calls is in O(n/θ^{2}).

The algorithm calls for the exact value of E[f_{α}^{2}] which is computationally costly. The good news is, we can estimate E[f_{α}^{2}] accurately in polynomial time—with an error less than θ^{2}/4 with very high probability. To use the approximated values requires a minor change in the algorithm described above. Instead of testing on (E[f_{α}^{2}] ≥ θ^{2}), we will test on whether the approximate value of E[f_{α}^{2}] is greater than θ^{2}/2.

To approximate E[f_{α}^{2}] we approximate the value of f_{α}. Remembering from the last lecture that f_{α}(x) = E_{y∈{0,1}^{k}}[f(yx)χ_{α}(y)], we can then approximate f_{α} by picking m_{1} values of x, where |x| = n − k and picking m_{2} values of y, where |y| = k. To evaluate f(yx)χ_{α}(y) for each pair of (x, y), we make queries for each f(yx). The size of m_{1} and m_{2} can be determined by Chernoff bounds:

m_{1} = O (1/θ^{2} ln (\frac{n}{θ^{2}δ} ))
m_{2} = O (1/θ^{2} ln (\frac{m_{1}n}{θ^{2}δ}))

22.2.2 Functions with small L_{1} norm

We now turn our attention to functions with small L_{1} norms; i.e.,

L_{1}(f) = ∑_{z} |f̂(z)|,

where 1 ≤ L_{1}(f) ≤ 2^{n/2} for some boolean function f. The L_{1} norm measures how spread out things are. A function with a small norm has a small number of large coefficients. In the next lemma, we show that approximating the coefficients of f is good enough for an approximation of f.

Lemma 2 Given ϵ > 0, let S = {z : |f̂(z)| ≥ ϵ/L_{1}(f)}, and let g(x) = ∑_{z∈S} f̂(z)χ_{z}(x). Then

E[(f − g)^{2}] ≤ ϵ.

This implies that

P rob[f(x) ≠ sign(g(x))] ≤ ϵ.

Proof: By the definition of g, we have

(f − g)(x) = ∑_{z∉S} f̂(z)χ_{z}(x)

By Parseval’s identity, we have

E[(f − g)^{2}] = ∑_{z∉S} f̂^{2}(z).

The upper bound is

Max_{z∉S}|f̂(z)| · ( ∑_{z∈{0,1}^{n}} |f̂(z)|)

≤ \frac{ϵ}{L_{1}(f)} · L_{1}(f) = ϵ.

The above lemma implies that the threshold value θ is ϵ/L_{1}(f) and f can be approximated by the coefficients whose absolute values are greater than the threshold value. Thus we can approximate f by using procedure Coef to find all coefficients that are greater in absolute value than ϵ/L_{1}(f).

22.2.3 Decision trees

Now let us consider the decision tree that the boolean function f is representing. In the next lemma, we show that the L_{1} norm of f is bounded by m, the number of leaves in the decision tree.

Lemma 3 Let f be computed by a decision tree with m nodes, then L_{1}(f) ≤ m.

Proof: Let leaf(T_{f}) be the set of leaves in the decision tree T_{f}, and let d(v) be the number of nodes on the path from the root to node v, but not including v. For each node at depth d, exactly 1/2^{−d} fraction of the inputs reaches it. Let I(v) be the set of all the inputs that reach leaf v. Then for every z,

f̂(z) = E[fχ_{z}]
= ∑_{v∈leaf(T_{f})} 2^{−d(v)}val(v)E_{x∈I(v)}[χ_{z}(x)].

Kushilevitz and Mansour show that |E_{x∈I(v)}[χ_{z}(x)]| = 1 for exactly 2^{d(v)} values of z and 0 for the rest. Thus, each leaf v contributes to at most 2^{d(v)} coefficients f̂(z), and to each such coefficient, it contributes at most 2^{−d(v)}. Thus, leaf v contributes at most one to the sum of the absolute value of all the coefficients. So L_{1}(f) ≤ number of leaves ≤ total number of nodes. More formally:

L_{1}(f) = ∑_{z∈{0,1}^{n}} |E[fχ_{z}]|

= ∑_{z∈{0,1}^{n}} | ∑_{v∈leaf(T_{f})} 2^{−d(v)}val(v)E_{x∈I(v)}[χ_{z}]|
≤ ∑_{z∈{0,1}^{n}} ∑_{v∈leaf(T_{f})} 2^{−d(v)} |E_{x∈I(v)}[χ_{z}]|
= ∑_{v∈leaf(T_{f})} 2^{−d(v)} ∑_{z∈{0,1}^{n}} |E_{x∈I(v)}[χ_{z}]|
= ∑_{v∈leaf(T_{f})} 2^{−d(v)}2^{d(v)}
≤ |leaf(T_{f})| ≤ m

Now we have finished showing that there is a polynomial time algorithm that for any boolean function f that can be represented by an m node decision tree with linear operations, and for any ϵ, δ > 0, outputs a function g such that with probability at least 1 − δ, Pr[f ≠ sign(g)] ≤ ϵ, and the algorithm runs in time polynomial in n, m, 1/ϵ, and log 1/δ.

22.3 Learning DNF with Membership Queries

We now briefly discuss Jackson’s algorithm for learning DNF with respect to the uniform distribution. This algorithm is based on two ideas that we’ve seen in the course. One is Freund’s boosting technique to boost the weak learner into a strong learner. The other is Kushilevitz and Mansour’s decision tree learning algorithm that we saw in the previous sections.

The hypothesis boosting algorithm requires a vote from a set of functions g_{i}(x), where 1 ≤ i ≤ k. We let

g_{1}(x) = f̂(z)χ_{z}(x),

but we will need to have non-uniform distributions for g_{2}(x), …, g_{k}(x).

Fact 1 For every DNF f with s terms and for every distribution D on the instance space of f there exists a χ_{A} such that |E_{D}[fχ_{A}]| ≥ 1/(2s + 1).

We need an oracle that, when given x, computes D(x). We choose g(x) = 2^{n}f(x)D(x). We can then simulate an oracle for this g given oracles for f and D.

(∀A)E_{D}[fχ_{A}] = E[gχ_{A}] = ĝ(A).

Theorem 1 DNF is learnable with respect to the uniform distribution using membership queries.

Corollary 2 Every DNF f with s terms can be computed as the majority of O(ns^{2}) parity functions.

References

[1] Jeffrey Jackson. An efficient membership-query algorithm for learning DNF with respect to the uniform distribution. In Proceedings 35th Annual Symposium on Foundations of Computer Science, pages 42-53, Santa Fe, New Mexico, November 1994. IEEE.

[2] Eyal Kushilevitz and Yishay Mansour. Learning decision trees using the fourier spectrum. In Proceedings of the Twenty-Third Annual ACM Symposium on Theory of Computing, pages 455-464, New Orleans, Louisiana, May 1991. ACM.