6.858 Machine Learning Lecture 2 : September 12, 1994

Lecturer: Ron Rivest Scribe: Deniz Yuret

2.1 Outline

• On-line learning with mistake bounds

• Optimal and Halving algorithms

• Winnow1 algorithm for {x_{i_{1}} ∨ x_{i_{2}} ∨ … ∨ x_{i_{k}}}

2.2 On-line learning with mistake bounds

2.2.1 Review of the learning model

x^{(1)}, x^{(2)}, … → c_{∗} The goal of concept learning is to discover an unknown target concept from labeled instances. Online learning makes a prediction for each instance. In the mistake bound model the goal is to minimize the number of prediction mistakes.

X = {0, 1}^{n} Instance space X consists of instances described by bit vectors of length n.

c ∈ C, c : X → {0, 1} Class of concepts C consists of concepts which classify each instance as negative or positive.

c_{∗} ∈ C : ; c_{∗}(x) = 1 ⇒ positive example ; c_{∗}(x) = 0 ⇒ negative example There is an unknown target concept the learner is

trying to discover.

for i = 1, 2, … learner given x^{(i)} ∈ X learner predicts c_{∗}(x^{(i)}) learner is told c_{∗}(x^{(i)})

The learning process keeps predicting and learning in a loop.

2-1 2-2 Lecture 2 : September 12, 1994

2.2.2 Minimum number of mistakes

With the mistake bound approach, we define the optimal algorithm as one that has the best worst-case bound:

Opt(C) = min_{learners L} max_{c_{∗}∈C} max_{examples x^{(i)}} (number of mistakes made by L when learning c_{∗} from ex- amples x^{(i)})

Some obvious bounds we can show on Opt(C) are:

Opt(C) ≤ 2^{n}  There are only 2^{n} possible instances.
Opt(C) ≤ |C| − 1  Learner predicts according to the first remaining c.
Opt(C) ≤ log(|C|)  We will achieve this bound by using the halving algorithm.

2.3 Optimal and Halving algorithms

2.3.1 Version space

Let V_{0} = C, V_{i} = {c ∈ C : c is consistent with the first i labeled examples} V_{i} is called the i’th “version space” in AI terminology due to Mitchell. V_{i} represents the remaining possible concepts.

Let {ξ_{0}(C, x) = \{c ∈ C : c(x) = 0\} \atop ξ_{1}(C, x) = \{c ∈ C : c(x) = 1\}}

Then for i > 1, V_{i} = \{ \begin{matrix}ξ_{0}(V_{i−1}, x^{(i)}) if c_{∗}(x^{(i)}) = 0 \\ ξ_{1}(V_{i−1}, x^{(i)}) if c_{∗}(x^{(i)}) = 1 \end{matrix}

2.3.2 Halving algorithm

The halving algorithm is an on-line learning algorithm which always predicts by the majority of concepts in the current version space.

Lecture 2 : September 12, 1994 2-3

On input x^{(i)}, predict 1 if |ξ_{1}(V_{i−1}, x^{(i)})| ≥ |ξ_{0}(V_{i−1}, x^{(i)})|, 0 otherwise.

Theorem: Opt(C) ≤ lg |C|

Proof: At each step the halving algorithm predicts according to the majority of concepts that are still in the version space. Thus when a mistake is made, more than half of the candidate concepts is eliminated.

Note that the halving algorithm may not be efficient, since determining and representing V_{i}’s and their sizes may be hard.

2.3.3 Optimal algorithm

On input x^{(i)} predict 1 if Opt(ξ_{1}(V_{i−1}, x^{(i)})) ≥ Opt(ξ_{0}(V_{i−1}, x^{(i)})), 0 otherwise.

The halving algorithm is not optimal. Instead of guessing in accordance with the majority of the valid concepts, we should guess according to the concept group that gives us the least number of expected mistakes. However this optimal algorithm is computationally even less efficient.

There are cases where the bound lg |C| is not tight. Take the example instance space of a large number of integers. The target concept is being one particular integer. By always guessing false, the optimal algorithm can only make one possible mistake.

2.3.4 Randomized halving algorithm

Predict according to c^{(i)} ∈ V_{i} selected at random.

On any step, the halving algorithm may not predict according to any concept. The prediction is based on a majority vote, and there may not be a single concept consistent with the majority votes. The randomized halving algorithm (due to Maass [2]), on the other hand, bases its predictions on a randomly selected c^{(i)} ∈ V_{i}.

Theorem: The expected number of mistakes randomized halving algorithm makes ≤ ln|C| + O(1).

2-4 Lecture 2 : September 12, 1994

Proof: Assume that the sequence of examples are chosen independent of the learner’s behavior. Thus x^{(1)}, x^{(2)}, … can be seen as a fixed sequence of examples. At step i, V_{i} is the set of remaining concepts. These concepts can be ordered by when they are going to be eliminated by the examples. Let c^{(1)}, c^{(2)}, …, c^{(r)} be this order, where r = |V_{i}|. Let M_{r} be the expected number of mistakes to be made. To derive a recurrence, note that the algorithm is equally likely to pick one of c^{(1)}, c^{(2)}, …, c^{(r)} as the concept which it will predict according to at this step. With probability 1/r, it will pick c^{(r)}, and make no more mistakes. With probability (r − 1)/r it will pick one of c^{(i)}, where i < r. It will then make at least one mistake, plus the expected number of mistakes for the remaining r − i concepts, i.e. M_{r−i}.

M_{1} = 0
M_{r} = \frac{r − 1}{r} [1 + \frac{M_{1} + M_{2} + ⋯ + M_{r−1}}{r − 1} ]
= \frac{r − 1}{r} + \frac{M_{1} + M_{2} + ⋯ + M_{r−1}}{r}
rM_{r} = (r − 1) + M_{1} + M_{2} + ⋯ + M_{r−1}
(r − 1)M_{r−1} = (r − 2) + M_{1} + M_{2} + ⋯ + M_{r−2}
r(M_{r} − M_{r−1}) + M_{r−1} = 1 + M_{r−1}
M_{r} = M_{r−1} + 1/r
M_{r} = ∑_{i=2}^{r} 1/i = ln(r) + O(1)

2.4 Winnow algorithm

So far, we have seen algorithms for general concept classes. In this section we will consider a restricted concept class: monotone disjunctions of at most k variables. We will analyze the Winnow algorithm (due to Littlestone [1]) which makes at most O(k lg n) mistakes. This bound is particularly useful when n is large and k is restricted (i.e., when there are many irrelevant attributes).

C = {x_{i_{1}} ∨ x_{i_{2}} ∨ … ∨ x_{i_{k}}}
|C| = ( \begin{matrix}n \\ k \end{matrix} ) + ( \begin{matrix}n \\ k − 1 \end{matrix} ) + ⋯ + ( \begin{matrix}n \\ 0 \end{matrix} )
lg |C| = Θ(k lg n)

We know that this is the bound for the optimal algorithm. However this does not tell us how to implement it efficiently.

We could try to use the on-line conjunction learning algorithm from the last lecture, based on the following duality principle:

c(x) = x_{i_{1}} ∨ x_{i_{2}} ∨ … ∨ x_{i_{k}} ⇒ \overline{c}(x) = \overline{x}_{i_{1}} \overline{x}_{i_{2}} … \overline{x}_{i_{k}}

However, this algorithm will give us the bound of n + 1 − k, which is linear in the total number of attributes. It would not be useful when there is a large number of irrelevant attributes compared to relevant ones.

Instead, we will learn disjunctions by an algorithm for learning linearly separable Boolean functions. A concept c is linearly separable if ∃w ∈ ℜ^{n}, Θ ∈ ℜ such that

∀x c(x) = 1 ⇔ w · x ≥ Θ

In other words, a function from {0, 1}^{n} to {0, 1} is linearly separable if there is a hyperplane in ℜ^{n} separating the points on which the function is 1 from the points on which it is 0. Monotone disjunctions are linearly separable. That is, for any concept x_{i_{1}} ∨ x_{i_{2}} ∨ … ∨ x_{i_{k}}, the separating hyperplane is given by x_{i_{1}} + x_{i_{2}} + ⋯ + x_{i_{k}} ≥ 1/2.

We now give the Winnow1 algorithm for learning monotone disjunctions. We present it as a linear-threshold algorithm, where the algorithm learns appropriate weights w.

Initialize Θ = n/2, w = (1, 1, …, 1) Predict c(x) = 1 iff w · x > Θ If prediction OK, do nothing If c(x) = 1 but w · x ≤ Θ, double all the weights w_{i} where x_{i} = 1 (promotion) If c(x) = 0 but w · x > Θ, zero all the weights w_{i} where x_{i} = 1 (elimination)

Theorem: Winnow1 makes O(k lg n) mistakes.

Proof: Let u be the number of false negatives (promotions), and v the number of false positives (eliminations). The total number of mistakes is u + v.

Lemma 1: v ≤ n/Θ + u In the case of a promotion, the increase in the weights is at most Θ. The algorithm doubles a subset of the weights which did not add up to Θ. Conversely, in the case of an elimination, the decrease in the weights is at least Θ. The algorithm zeroes a subset of the weights that added up to more than Θ. The sum of the weights can never go below zero, since they are either doubled or zeroed, and they start positive. Thus 0 ≤ ∑ w_{i} ≤ n + u.Θ − v.Θ ⇒ v ≤ n/Θ + u.

Lemma 2: w_{i} ≤ 2Θ A promotion occurs only if the sum of the relevant weights is less than Θ. Thus a single weight which is above Θ can never be doubled.

Lemma 3: After u promotions, ∃w_{i} such that lg w_{i} ≥ u/k. Each promotion must double one of w_{i_{1}}, w_{i_{2}}, …, w_{i_{k}}. Each doubling adds 1 to the logarithm.

Claim: u + v ≤ 2k(lg Θ + 1) + n/Θ

u/k ≤ lg w_{i} ≤ lg Θ + 1  by Lemmas 2 and 3
u ≤ k(lg Θ + 1)
v ≤ n/Θ + k(lg Θ + 1)  by Lemma 1
⇒ u + v ≤ n/Θ + 2k(lg Θ + 1)

With Θ = n/2, number of mistakes ≤ 2 + 2k lg n.

References

[1] Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning, 2:285-318, 1988.

[2] Wolfgang Maass. On-line learning with an oblivious environment and the power of randomization. Proceedings of the Fourth Annual ACM Workshop on Computational Learning Theory, pp. 167-175, August 1991.