6.858/18.428 Machine Learning Lecture 21 : November 28, 1994

Lecturer: Ron Rivest Scribe: Andreas Argiriou

Outline

• Learning decision trees with Fourier theory

In this lecture we see how we can apply Fourier techniques to learn decision trees, as described by Kushilevitz and Mansour in [1]. Such techniques have also been used in other problems, like DNF formulas.

21.1 Introduction

The type of learning game we need to consider now is special in the following ways:

1. The distribution is uniform and not arbitrary.

2. We want to learn decision trees.

3. There is no source of random examples.

4. The algorithm uses membership queries.

In the model of membership queries, the learner makes up the questions so that it may ask for the label of an example. With equivalence queries, on the other hand, the learner presents a hypothesis h and asks if h is the same as the target concept f. If h ≠ f, the learner receives a counterexample as an answer. In this problem we essentially need statistical queries, but we can use membership queries to get information of the type a statistical query would give.

We will consider boolean functions with a slight modification: a boolean function now maps {0, 1}^{n} to {±1} instead of {0, 1} in the following way

x ∈ {0, 1} ↦ (−1)^{x}.

21-1

21-2 Lecture 21 : November 28, 1994

x
z 00 01 10 11
00 1 1 1 1
01 1 -1 1 -1
10 1 1 -1 -1
11 1 -1 -1 1

Table 21.1: The χ functions for n = 2.

In other words, negative examples map to +1, and positive examples map to −1.

The concept class being learned is the class C of boolean functions computed by decision trees with linear operations. In this model each node computes a sum (modulo 2) of a subset of the n variables and if the result is 1 branches to the right subtree, else branches to the left subtree. The output is the value of the leaf where we end up.

Learning this class in a passive model of learning is an open problem. In our new model, when given ϵ, δ we need to produce a hypothesis h such that Pr[h(x) ≠ f(x)] ≤ ϵ with probability at least 1−δ (where x is drawn according to the uniform distribution and f is the target function).

21.2 The orthonormal basis

The basis functions for the Fourier transform we will use are given by the following definition.

Definition 1 For each z ∈ {0, 1}^{n}, define the function χ_{z} : {0, 1}^{n} → {±1} by

χ_{z}(x) = (−1)^{z·xmod2}

The value of the function depends on the exclusive or of those variables “selected” by z in the dot product. If ∑_{i=1}^{n} z_{i}x_{i} ≡ 0 mod 2 the output is +1, else −1. Table 21.1 shows the values of the χ functions if n = 2.

Our objective now is to define an inner product of functions such that

⟨χ_{z_{1}}, χ_{z_{2}}⟩ = \{ \begin{matrix}0 & if z_{1} ≠ z_{2} \\ 1 & if z_{1} = z_{2} \end{matrix}  (21.1)

Lecture 21 : November 28, 1994 21-3

and such that a function f on {0, 1}^{n} can be uniquely expressed as

f = ∑_{z∈{0,1}^{n}} f̂(z)χ_{z}.  (21.2)

where f̂(z) are real constants.

The Fourier transform of f is the expansion of f as a linear combination of the χ_{z}’s. Equations (21.1) and (21.2) together state that the χ functions form an orthonormal basis. Then the Fourier coefficients are given by

f̂(z) = ⟨f, χ_{z}⟩.

The method of using Fourier coefficients is helpful because now the objective will be constructing an algorithm that approximates a function f as a linear combination of the basis functions. The algorithm should find enough of the biggest Fourier coefficients to meet restrictions on the error.

To define the inner product, we use the concept of the expectation of f (over uniform distribution)

E[f] = \frac{1}{2^{n}} ∑_{x∈\{0,1\}^{n}} f(x).

Definition 2 For any functions f, g : {0, 1}^{n} → ℜ, define the inner product of f, g as

⟨f, g⟩ = E[fg] = \frac{1}{2^{n}} ∑_{x∈\{0,1\}^{n}} f(x)g(x)

and the L_{2} norm of f as

∥f∥_{2} = \sqrt{⟨f, f⟩}.

Note that for boolean functions ∥f∥_{2} = 1.

It is also clear from definition 1 that for all z_{1}, z_{2} ∈ {0, 1}^{n},

χ_{z_{1}}χ_{z_{2}} = χ_{z_{1}⊕z_{2}}

where ⊕ denotes exclusive or.

Using the above, one can see that definition 2 verifies equations (21.1), so that indeed the χ functions form an orthonormal basis.

Another useful relation is Parseval’s identity:

∥f∥_{2}^{2} = ∑_{z∈{0,1}^{n}} f̂^{2}(z)  (21.3)

or for boolean functions

∑_{z∈{0,1}^{n}} f̂^{2}(z) = 1.  (21.4)

Finally, we can define the L_{1} norm of f, as follows:

Definition 3 The L_{1} norm of f is

L_{1}(f) = ∑_{z∈{0,1}^{n}} |f̂(z)|.

The L_{1} norm is a measure of how spread out the coefficients are and ranges from 1 to 2^{\frac{n}{2}} .

21.3 Approximation by sparse functions

The following theorem will be proven later:

Theorem 1 If the function f is represented by a decision tree with m nodes, then L_{1}(f) ≤ m.

We will thus try to learn decision trees with a small number of nodes, since it is easier to approximate a function with a small L_{1} norm (intuitively, weight is not spread a lot and there are just a few big coefficients).

The approximation of the target function f will be another function g, real valued and not boolean. However

sign(g(x)) = \{ \begin{matrix}+1 & if g(x) ≥ 0 \\ −1 & otherwise \end{matrix}

is a boolean function. So we want sign(g(x)) to be a good approximation of f. We see now that this happens when g ϵ-approximates f in the L_{2} norm.

Definition 4 We say that a function g ϵ-approximates f in L_{2} norm, when

E[(f − g)^{2}] ≤ ϵ

.

Notice that when sign(g(x)) ≠ f(x), then (f(x) − g(x))^{2} ≥ 1 and hence Pr[f ≠ sign(g)] ≤ E[(f − g)^{2}]. Thus we have the following lemma:

Lemma 1 If g ϵ-approximates f in L_{2} norm, then Pr[f ≠ sign(g)] ≤ ϵ.

So, now the question is how many of the Fourier coefficients are needed to ϵapproximate the target function. For this reason it is helpful to define the concept of a t-sparse function.

Definition 5 The function g is t-sparse if it has at most t non-zero Fourier coefficients.

As we mentioned earlier, we prefer approximations with large coefficients and the next lemma shows that this can be achieved.

Lemma 2 If f can be ϵ-approximated (in L_{2} norm) by a t-sparse function g then f can be ϵ + O(ϵ^{2})-approximated by a t-sparse function h whose non-zero coefficients are at least ϵ/t.

Proof: Write g(x) = ∑_{i=1}^{t} ĝ(z_{i})χ_{z_{i}}(x). Parseval’s identity (21.3) gives E[(f − g)^{2}] = ∑_{z}(f̂(z) − ĝ(z))^{2} and this has a minimum when ĝ(z_{i}) = f̂(z_{i}), i = 1, …, t. Let g′(x) = ∑_{i=1}^{t} f̂(z_{i})χ_{z_{i}}(x). We can show that

h(x) = ∑_{f̂(z_{i})≥ \frac{ϵ}{t}} f̂(z_{i})χ_{z_{i}}(x)

is as desired. Indeed, E[(f − h)^{2}] − E[(f − g)^{2}] ≤ E[(f − h)^{2}] − E[(f − g′)^{2}] = ∑_{f̂(z_{i})< \frac{ϵ}{t}} f̂^{2}(z_{i}) < t(\frac{ϵ}{t})^{2} = O(ϵ^{2}). Since E[(f − g)^{2}] ≤ ϵ we are done. □

A consequence of lemma 2 is that our algorithm can look only for the larger Fourier coefficients. However, the space is huge and this is not an easy task. What we can do is use a recursive procedure to implement tree search over the values of z and find the large coefficients (see Figure 21.1). The problem is solved if at each node we can say whether there is a large coefficient in the subtree. With a perfect test like that we would need O(tn) work to find the t largest coefficients in a tree of depth n. We need only to figure out

[figure: .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; .; f(111); t coefficients; f(000); depth=n; 0; 0; 1; 1; 1; 0; 1; 0; 0; 1; 1; 0; 1; 0]

Figure 21.1: Search for large coefficients on the n-dimensional cube • how to do the test

• how many coefficients to find. The idea is to define a function that includes coefficients under a certain node. Definition 6 For each α ∈ {0, 1}^{k}, define the function f_{α} : {0, 1}^{n−k} → ℜ by

f_{α}(x) = ∑_{β∈{0,1}^{n−k}} f̂(αβ)χ_{β}(x).

By estimating E[f_{α}^{2}], which equals ∑_{β∈{0,1}^{n−k}} f̂^{2}(αβ) (Parseval), it will be possible to make a prediction about the size of the coefficients f̂(αβ). The following lemma will be necessary for the approximation of E[f_{α}^{2}]: Lemma 3 If 1 ≤ k < n and α ∈ {0, 1}^{k}, then

f_{α}(x) = E_{y∈{0,1}^{k}}[f(yx)χ_{α}(y)].

Proof: It is easy to see from the definitions that if z = z_{1}z_{2} and |z_{1}| = k, then χ_{z}(yx) = χ_{z_{1}}(y)χ_{z_{2}}(x). So, we have

E_{y}[f(yx)χ_{α}(y)] = E_{y}[∑_{z} f̂(z)χ_{z}(yx)χ_{α}(y)] = E_{y}[∑_{z_{1}} ∑_{z_{2}} f̂(z_{1}z_{2})χ_{z_{1}}(y)χ_{z_{2}}(x)χ_{α}(y)]

= ∑_{z_{1}} ∑_{z_{2}} f̂(z_{1}z_{2})χ_{z_{2}}(x)E_{y}[χ_{z_{1}}(y)χ_{α}(y)].

Now from equations (21.1) we see that

E_{y}[χ_{z_{1}}(y)χ_{α}(y)] = \{ \begin{matrix}0 & if z_{1} ≠ α \\ 1 & if z_{1} = α \end{matrix}

and therefore

E_{y}[f(yx)χ_{α}(y)] = ∑_{z_{2}} f̂(αz_{2})χ_{z_{2}}(x) = f_{α}(x)

□

The remaining issues are now the following:

• How to use an approximation technique to pick out large Fourier coefficients

• Showing the approximation is good enough

• Showing the algorithm applies to decision trees.

References

[1] Eyal Kushilevitz and Yishay Mansour. Learning Decision Trees using the Fourier Spectrum. In Proceedings of the 23rd Annual ACM Symposium on Theory of Computing, pages 455-464, 1991.