6.858/18.428 Machine Learning Lecture 15: November 7, 1994

Lecturer: Ron Rivest Scribe: Jerry Pratt

15.1 Outline

• Neural Networks

• Back propagation

15.2 Neural Networks

Neural networks are used to learn arbitrary target functions which map a vector of real numbers to another vector of real numbers,

t : X ⊂ ℜ^{m} → Y ⊂ ℜ^{n}

The neural network is trained on a set, S, of m examples (x⃗, y⃗) such that y⃗ = t(x⃗). By training the net to do well on the given examples, we hope that it will do well later in predicting the value of new examples.

The most popular method of training neural networks is to use gradient descent to minimize the Mean Square Error of the training sample. Back propagation is a computationally clever means of doing this. These concepts are explained below.

15.2.1 Mean Squared Error (MSE)

Suppose we have a neural network which computes f_{w}(x⃗) but is trying to learn the target function t(x⃗). (Note that the function which the neural net computes depends on the architecture of the network, and on the weights of the network.) Given a fixed sample S, with m elements, we can define the Mean Squared Error (MSE) of the neural network on sample S to be:

E(w⃗) = \frac{1}{m} ∑_{x⃗∈S}(f_{w}(x⃗) − t(x⃗))^{2}

15-2 Lecture 15: November 7, 1994

15-1 Since the target function and sample are fixed, the MSE is only a function of the weights of the network.

The goal of our training routine is thus to minimize the MSE. The MSE is a useful cost function for several reasons. It can be used for arbitrary functions, is well defined, and is easily computed. Most importantly, however, is that it has a computable, continuous derivative (as long as f_{w}(x⃗) has a computable, continuous derivative). This will play an important role in the training of the neural network. The major disadvantage of using the MSE is that “outliers” have a large effect on it.

For boolean functions, minimizing the MSE is equivalent to minimizing errors. This is not true in general however.

15.2.2 Gradient Descent

To minimize the MSE cost function, we use gradient descent, also known as “hill climbing”. Gradient descent is done by taking small steps in the direction which reduces the error the most. In general, one starts with an initial weight vector w⃗_{o} ∈ ℜ^{k}. At each iteration, the weight vector is updated:

w⃗_{i+1} ← w⃗_{i} + ϵ △ w⃗_{i}

where ϵ is the learning rate parameter and △w⃗_{i} is the direction to move in. Essentially, we are taking a walk in weight space, always going down hill. The iteration is stopped after converging to a minima to within a given margin.

The direction which reduces the error the fastest is opposite the gradient ∇E, where

∇E = (\frac{∂E}{∂w_{1}} , \frac{∂E}{∂w_{2}} , ..., \frac{∂E}{∂w_{k}} )

so that the update is

w⃗_{i+1} ← w⃗_{i} − ϵ∇E

There are several important issues in using gradient descent which are discussed below.

Initial weight vector

Lecture 15: November 7, 1994 15-3

The choice of the initial weight vector, w⃗_{o}, is important in determining whether gradient descent converges to a local minima or to the true global minima. There are several methods for choosing the initial vector. These include random generation, having a fixed vector, or choosing the initial vector. Random generation is most popular in neural networks. In general, there is no known way to tell, given an initial weight vector, whether gradient descent will converge to a global or local minima. In practice, all that can be done is determine when a local minima is hit (when no more progress is being made) and start over with a new initial weight vector.

Learning parameter

The choice of a learning parameter ϵ determines whether or not gradient descent will converge to a minima, and how long it will take. If ϵ is too small, gradient descent will progress too slowly. If it is too large, gradient descent will overshoot the minima and hence possibly not converge. There is no systematic method for picking the “right” ϵ. However, there are adaptive methods and also methods for dynamically adjusting it during learning.

Batch vs. Incremental processing

Given a sample set, S, gradient descent can either be done on a point-by-point basis or by batch processing. If batch processing is done, the gradient is computed as follows:

∇E(S) = \frac{1}{m} (∇E(\{x⃗_{1}\}) + ∇E(\{x⃗_{2}\}) + ... + ∇E(\{x⃗_{m}\}))

Momentum

Gradient descent gets in trouble when it encounters narrow, slowly sloping valleys. In such a situation, it will tend to oscillate between the sides of the valley while slowly making progress down it. The short term oscillations can be cancelled out by maintaining some of the “momentum” of the previous updates. This is done by computing the direction to move in in the following way:

△w⃗_{i+1} = α △ w⃗_{i} − (1 − α)(∇E)

where 0 < α ≤ 1. This effectively is a low pass filter. If α = 0, no filtering occurs.

Convergence time

Convergence time may be decreased if the hessian is used instead of the gradient. The hessian is more accurate because it retains second order terms in addition to the first order terms of the gradient.

Using a dynamically adjusted variable step size may also decrease the convergence time.

When to stop training

Training is usually stopped when the MSE seems to have converged to a (local) minimum. However, in some cases it may be best to stop training early. Prediction of samples not contained in the training set may be more accurate if training is stopped short.

15.2.3 Application: Linear Functions

The following example shows how to use gradient descent to minimize the Mean Square Error when learning the linear function

f_{w}(x⃗) = w⃗ · x⃗

where w⃗, x⃗ ∈ ℜ^{k}. This function is similar to a perceptron except that there is no threshold. Using batch processing, the MSE and gradient can be calculated and the weight update made:

E(w⃗) = \frac{1}{m} ∑_{i}(w⃗ · x⃗_{i} − y_{i})^{2}

∇E = \frac{δE(w⃗)}{δw⃗} = \frac{2}{m} ∑_{i}(w⃗ · x⃗_{i} − y_{i})x⃗_{i}

w⃗ ← w⃗ − ϵ∇E

If single point processing is used, we have:

w⃗ ← w⃗ − ϵ(w⃗ · x⃗ − y)x⃗

This last update equation is very similar to the perceptron training rule except now the update is proportional to the error.

15.3 Back Propagation

By using the chain rule of differentiation, we can show that the derivative of the MSE with respect to the weight vector of a given node (\frac{δE}{δw⃗_{i}}) can be computed with access only to information found in the node and information in the nodes directly connected to the output of the given node. Therefore gradient information can be propagated backward through the network in order to update the weights. Hence the name back propagation. In order to propagate gradient information back through the threshold functions, it is important that they have a continuous derivative which is easily computed. The following sigmoid functions have such properties.

15.3.1 Sigmoid Functions

Sigmoid functions are similar to the threshold function except that they are smooth and have a computable, continuous derivative. Other properties of a sigmoid function is that its range is the real numbers, its domain is a bounded interval, and it is monotonic. Two popular sigmoid functions are the hyperbolic tangent and the logistic function.

Hyperbolic tangent

The hyperbolic tangent sigmoid function is

z = tanh(x) = \frac{e^{x} − e^{−x}}{e^{x} + e^{−x}}

Its derivative is

\frac{δz}{δx} =sech^{2}(x) = \frac{4}{(e^{x} + e^{−x})^{2}}

The domain of the hyperbolic tangent is ℜ and the range is the interval [−1, 1]. See Figure 15.1 for a graph of the hyperbolic tangent (solid) and its derivative (dotted).

15-6 Lecture 15: November 7, 1994 Figure 15.1: Hyperbolic tangent sigmoid function (solid line) and derivative (dotted line).

Logistic function

The logistic sigmoid function is

z = \frac{1}{1 + e^{−x}}

Its derivative is

\frac{δz}{δx} = z(1 − z)

The domain of the logistic function is ℜ and the range is the interval [0, 1]. See Figure 15.2 for a graph of the logistic function (solid) and its derivative (dotted). The logistic function is popular because its derivative can be written in terms of its output. Since its output is computed in order to compute the output of the network, not much more computation is necessary in computing the derivative.

15.3.2 Chain Rule

Let y be a function of x, y = f_{w}(x). The chain rule states that

\frac{∂E}{∂x} = \frac{∂E}{∂y} \frac{δy}{δx}

For the following, lets use the notation δ_{x} = \frac{∂E}{∂x}. Then the chain rule states δ_{x} = δ_{y} \frac{δy}{δx}. It is conceptually easier to see how back propagation works, via the chain rule, if a node of the network is redrawn in its expanded form, as in Figure 15.3.

Figure 15.4 contains all of the elementary blocks which are used in building a neural network. Using the chain rule we can calculate how the partial derivatives are propagated backward through each block:

a) y = x^{2}

δ_{x} = 2δ_{y}x

b) y = σ(x)

logistic:

δ_{x} = δ_{y}y(1 − y)

hyperbolic tangent:

δ_{x} = δ_{y}sech^{2}(x)

15-8 Lecture 15: November 7, 1994 Figure 15.2: Logistic sigmoid function (solid line) and derivative (dotted line).

Figure 15.3: Expanded node of network for visualization of back propagation. If the node is an output node then we can make the network compute the mean squared error, by subtracting the target value from the value output at the node, and squaring this value.

15-10 Lecture 15: November 7, 1994

Figure 15.4: Five different blocks found in neural networks.

c) y = x_{1} + x_{2}

δ_{x_{1}} = δ_{y}
δ_{x_{2}} = δ_{y}

d) y = wx

δ_{x} = δ_{y}w
δ_{w} = δ_{y}x

e) y_{1} = x, y_{2} = x

δ_{x} = δ_{y_{1}} + δ_{y_{2}}

By combining the blocks, as in Figure 15.3, we can now compute how the partial derivatives propagate backward through an entire node: If we are using a logistic sigmoid function we have:

δ_{x_{i}} = δ_{y}y(1 − y)w_{i}

δ_{w_{i}} = δ_{y}y(1 − y)x_{i}

With a hyperbolic tangent sigmoid function, we have:

δ_{x_{i}} = δ_{y}sech^{2}(x_{i})w_{i}

δ_{w_{i}} = δ_{y}sech^{2}(x_{i})x_{i}

For hidden nodes, we have:

δ_{y} = ∑ δ_{x_{i}}

where the sum is over the x_{i} of the nodes in the next higher layer that y branches out to.

For the output nodes, we have

δ_{y} = 2(y_{d} − y) = 2(t(x⃗) − f_{w}(x⃗))

15.3.3 Back propagation Algorithm

We can now write the pseudocode for a back propagation algorithm. The following uses the logistic sigmoid function, a constant learning parameter, ϵ, and batch processing.

Until progress is stopped {

For each input x⃗

{ Compute the output, y, of each node in a forward manner. For each node, in a backward manner, {

Compute δ_{y}(x⃗) using δ_{y}(x⃗) = 2(t(x⃗) − f_{w}(x⃗)) if node in output layer or

δ_{y}(x⃗) = ∑ δ_{x_{i}}(x⃗) if node in hidden layer.
For each input x_{i} to the node, compute δ_{x_{i}}(x⃗) = δ_{y}(x⃗)y(1 − y)w_{i}

For each weight w_{i} on edges to the node, compute δ_{w_{i}}(x⃗) = δ_{y}(x⃗)y(1 − y)x_{i} }

}

Add up the δ_{w}(x⃗) for each input x⃗ and update the weights using

w_{i} ← w_{i} − \frac{1}{m}ϵ ∑_{x⃗∈S} δ_{w_{i}}(x⃗)

}

References

[1] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning internal representations by error propagation. Parallel Distributed Processing - Explorations in the Microstructure of Cognition, pages 318-362, MIT Press, 1986.