%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% NOTE: delete ,psfig here if using on athena
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentstyle[12pt,twoside,alltt,psfig]{article}
\textwidth=6in
\oddsidemargin=0.25in
\evensidemargin=0.25in
\topmargin=-0.1in
\footskip=0.8in
\parindent=0.0cm
\parskip=0.3cm
\textheight=8.00in
\setcounter{tocdepth} {3}
\setcounter{secnumdepth} {2}
\sloppy

\newenvironment{proof}{\noindent{\bf Proof:}}{\qed\bigskip}
\newenvironment{proof_sketch}{\noindent{\bf Sketch of Proof:}\hspace*{1em}}{\qed\bigskip}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{claim}{Claim}
\newtheorem{fact}{Fact}
\newtheorem{definition}{Definition}
\newtheorem{assumption}{Assumption}
\newtheorem{observation}{Observation}
\newtheorem{example}{Example}
\newcommand{\qed}{\rule{7pt}{7pt}}



\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}



\newcommand{\lecture}[6]{
   \pagestyle{myheadings} \thispagestyle{plain} \newpage
\setcounter{lecnum}{#1} \setcounter{page}{1} \noindent \begin{center}
\framebox{ \vbox{ \hbox to 6.28in { {\bf 6.858/18.428 Machine Learning
\hfill Lecture #1: #2} } \vspace{4mm} 
\vspace{2mm} \hbox to 6.28in { {\it
Lecturer: #3 \hfill Scribe: #4} } } } \end{center}
\markboth{Lecture #1: #2}{Lecture #1: #2} \vspace*{4mm} }



\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Lecture command takes 4 arguments:
%		ordinal number of the lecture
%		date of the lecure
%		lecturer
%		scribe---that is you.
%
% Fill out the next line with this information !!!
\lecture{15}{November 7, 1994}{Ron Rivest}{Jerry Pratt} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes start here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% For theorems, lemmas, definitions, remarks, etc. use commands
% {\theorem{...}}, {\lemma{...}}, {\definition{...}}, etc.
% For proofs, use \begin{proof} ... \end{proof}
%
% For figures use the following block:
%
% \begin{figure}
% \begin{center}
% \mbox{\psfig{figure=lecture-nn-fig-mm.ps}}
% \caption{A very nice picture.}
% \label{fig:picture}
% \end{center}
% \end{figure}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Outline}
\begin{itemize}
\item Neural Networks
\item Back propagation
\end{itemize}

\section{Neural Networks}

Neural networks are used to learn arbitrary target functions which map
a vector of real numbers to another vector of real numbers,
\[ t: X \subset \Re^m \rightarrow Y \subset \Re^n \]
The neural network is trained on a set, $S$, of m examples
${(\vec{x},\vec{y})}$ such that $\vec{y} = t(\vec{x})$.  By training
the net to do well on the given examples, we hope that it will do well
later in predicting the value of new examples.

The most popular method of training neural networks is to use gradient
descent to minimize the Mean Square Error of the training sample.
Back propagation is a computationally clever means of doing this.
These concepts are explained below.

\subsection{Mean Squared Error (MSE)}

Suppose we have a neural network which computes $f_w(\vec{x})$ but is trying
to learn the target function $t(\vec{x})$.  (Note that the function
which the neural net computes depends on the architecture of the
network, and on the weights of the network.) Given a fixed sample $S$, with
$m$ elements, we can define the Mean Squared Error (MSE) of the neural 
network on sample $S$ to be:
\[ E(\vec{w}) = \frac{1}{m} \sum_{\vec{x} \in S} (f_w(\vec{x})-t(\vec{x}))^2 \]
Since the target function and sample are fixed, the MSE is only a
function of the weights of the network.

The goal of our training routine is thus to minimize the MSE.  The MSE
is a useful cost function for several reasons.  It can be used for
arbitrary functions, is well defined, and is easily computed.  Most
importantly, however, is that it has a computable, continuous
derivative (as long as $f_w(\vec{x})$ has a computable, continuous
derivative).  This will play an important role in the training of the
neural network.  The major disadvantage of using the MSE is that
``outliers'' have a large effect on it.

For boolean functions, minimizing the MSE is equivalent to minimizing
errors.  This is not true in general however.


\subsection{Gradient Descent}

To minimize the MSE cost function, we use gradient descent, also known as
``hill climbing''.  Gradient descent is done by taking small steps in the
direction which reduces the error the most.  In general, one starts
with an initial weight vector $\vec{w}_o \in \Re ^k$.  At each iteration,
the weight vector is updated:
\[ \vec{w}_{i+1} \leftarrow \vec{w}_i + \epsilon \bigtriangleup \vec{w}_i \]  
where $\epsilon$ is the learning rate parameter and $\bigtriangleup
\vec{w}_i$ is the direction to move in.  Essentially, we are taking a
walk in weight space, always going down hill.  The iteration is stopped
after converging to a minima to within a given margin.      

The direction which reduces the error the fastest
is opposite the gradient $\nabla E$, where
\[ \nabla E = \left( \frac{\partial E}{\partial w_1}, 
\frac{\partial E}{\partial w_2}, ..., \frac{\partial E}{\partial w_k} \right) \]
so that the update is 
\[ \vec{w}_{i+1} \leftarrow \vec{w}_i - \epsilon \nabla E \] 


There are several important issues in using gradient descent which are 
discussed below.

\subsubsection{Initial weight vector}

The choice of the initial weight vector, $\vec{w}_o$, is important in
determining whether gradient descent converges to a local minima or to
the true global minima.  There are several methods for choosing the
initial vector.  These include random generation, having a fixed
vector, or choosing the initial vector.  Random generation is most
popular in neural networks.  In general, there is no known way to
tell, given an initial weight vector, whether gradient descent will
converge to a global or local minima.  In practice, all that can be
done is determine when a local minima is hit (when no more progress is
being made) and start over with a new initial weight vector.

\subsubsection{Learning parameter}

The choice of a learning parameter $\epsilon$ determines whether or
not gradient descent will converge to a minima, and how long it will
take.  If $\epsilon$ is too small, gradient descent will progress too
slowly.  If it is too large, gradient descent will overshoot the minima
and hence possibly not converge.  There is no systematic method for picking
the ``right'' $\epsilon$.  However, there are adaptive methods and also
methods for dynamically adjusting it during learning.   

\subsubsection{Batch vs. Incremental processing}

Given a sample set, $S$, gradient descent can either be done on a
point-by-point basis or by batch processing.  If batch processing is
done, the gradient is computed as follows:
\[ \nabla E(S) = \frac{1}{m} \left( \nabla E(\{\vec{x}_1\})
+ \nabla E(\{\vec{x}_2\}) + ... + \nabla E(\{\vec{x}_m\})\right) \]

\subsubsection{Momentum}

Gradient descent gets in trouble when it encounters narrow, slowly sloping
valleys.  In such a situation, it will tend to oscillate between the sides
of the valley while slowly making progress down it.  The short term oscillations
can be cancelled out by maintaining some of the ``momentum'' of the previous
updates.  This is done by computing the direction to move in in the following
way:
\[ \bigtriangleup \vec{w}_{i+1} = \alpha \bigtriangleup \vec{w}_i - (1-\alpha)
(\nabla E) \] 
where $0 < \alpha \leq 1$.  This effectively
is a  low pass filter.  If $\alpha = 0$, no filtering occurs.

\subsubsection{Convergence time}
Convergence time may be decreased if the hessian is used instead of
the gradient.  The hessian is more accurate because it retains second
order terms in addition to the first order terms of the gradient.

Using a dynamically adjusted variable step size may also decrease the
convergence time.

\subsubsection{When to stop training}
Training is usually stopped when the MSE seems to have converged to a
(local) minimum.
However, in some cases it may be best to stop training early.
 Prediction of samples not contained in
the training set may be more accurate if training is stopped short.


\subsection{Application: Linear Functions}
The following example shows how to use gradient descent to minimize
the Mean Square Error when learning the linear function 
\[ f_w(\vec{x}) = \vec{w} \cdot \vec{x} \]
where $\vec{w},\vec{x} \in \Re^k$. This function is similar to a
perceptron except that there is no threshold.  Using batch
processing, the MSE and gradient can be calculated and the weight
update made:
\[ E(\vec{w}) = \frac{1}{m} \sum_i(\vec{w} \cdot \vec{x}_i - y_i)^2 \]
\[ \nabla E = \frac{\delta E(\vec{w})}{\delta \vec{w}} = \frac{2}{m}
\sum_i (\vec{w} \cdot \vec{x}_i - y_i)\vec{x}_i \]
\[ \vec{w} \leftarrow \vec{w} - \epsilon \nabla E \]
If single point processing is used, we have:
\[ \vec{w} \leftarrow \vec{w} - \epsilon (\vec{w} \cdot \vec{x} - y) \vec{x} \]
This last update equation is very similar to the perceptron training
rule except now the update is proportional to the error.


\section{Back Propagation}

By using the chain rule of differentiation, we can show that the
derivative of the MSE with respect to the weight vector of a given
node ($\frac{\delta E}{\delta \vec{w}_i}$) can be computed with access
only to information found in the node and information in the nodes
directly connected to the output of the given node.  Therefore
gradient information can be propagated backward through the network
in order to update the weights.  Hence the name back propagation.
In order to propagate gradient information back through the threshold
functions, it is important that they have a continuous derivative which
is easily computed.  The following sigmoid functions have such properties.

\subsection{Sigmoid Functions}

Sigmoid functions are similar to the threshold function except that
they are smooth and have a computable, continuous derivative.  Other
properties of a sigmoid function is that its range is the real
numbers, its domain is a bounded interval, and it is monotonic.  Two
popular sigmoid functions are the hyperbolic tangent and the logistic
function.

\subsubsection{Hyperbolic tangent}

The hyperbolic tangent sigmoid function is 
\[ z = \tanh(x) = \frac{e^x-e^{-x}}{e^x+e^{-x}} \]
Its derivative is\\ 
\begin{center}
$  \displaystyle \frac{\delta z}{\delta x} = $sech$^2(x) \displaystyle = \frac{4}{(e^x+e^{-x})^2} $
\end{center}
The domain of the hyperbolic tangent is $\Re$ and the range is the
interval $[-1,1]$.  See Figure \ref{fig:tanh} for a graph of the hyperbolic
tangent (solid) and its derivative (dotted).

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture15fig1.ps}}
\caption{Hyperbolic tangent sigmoid function (solid line) and derivative (dotted
line).}
\label{fig:tanh}
\end{center}
\end{figure}

\subsubsection{Logistic function}

The logistic sigmoid function is
\[ z = \frac{1}{1+e^{-x}} \]
Its derivative is
\[\frac{\delta z}{\delta x} = z(1-z) \]
The domain of the logistic function is $\Re$ and the range is the
interval $[0,1]$.  See Figure \ref{fig:logistic} for a graph of the logistic
function (solid) and its derivative (dotted).  The logistic function
is popular because its derivative can be written in terms of its
output.  Since its output is computed in order to compute the output
of the network, not much more computation is necessary in computing the
derivative.

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture15fig2.ps}}
\caption{Logistic sigmoid function (solid line) and derivative (dotted line).}
\label{fig:logistic}
\end{center}
\end{figure}



\subsection{Chain Rule}

Let $y$ be a function of $x$, $y=f_w(x)$.  The chain rule states that 
\[ \frac{\partial E}{\partial x} = \frac{\partial E}{\partial y} 
\frac{\delta y}{\delta x} \]
For the following, lets use the notation $\delta_x = \frac{\partial
E}{\partial x}$.  Then the chain rule states $\delta_x = \delta_y 
\frac{\delta y}{\delta x}$.  It is conceptually easier to see how 
back propagation works, via the chain rule, if a node of the network is
redrawn in its expanded form, as in Figure \ref{fig:expanded}.

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture15fig3.ps}}
\caption{Expanded node of network for visualization of back
propagation. If the node is an output node then we can make the
network compute the mean squared error, by subtracting the target
value from the value output at the node, and squaring this value.}
\label{fig:expanded}
\end{center}
\end{figure}

Figure \ref{fig:chain} contains all of the elementary blocks which are
used in building a neural network.  Using the chain rule we can calculate
how the partial derivatives are propagated backward through each block:

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture15fig4.ps}}
\caption{Five different blocks found in neural networks.}
\label{fig:chain}
\end{center}
\end{figure}


a) $y=x^2$
\[  \delta_x = 2 \delta_y x \]

b) $y=\sigma (x)$

$\mbox{      }$logistic:
\[ \delta_x = \delta_y y (1-y) \]
$\mbox{      }$hyperbolic tangent:
\begin{center}
$ \delta_x = \delta_y  $sech$^2(x) $
\end{center}

c) $y = x_1 + x_2$
\[ \delta_{x_1} = \delta_y \]
\[ \delta_{x_2} = \delta_y \]

d) $y = wx$
\[ \delta_x = \delta_y w \]
\[\delta_w = \delta_y x \]

e) $y_1 = x, y_2 = x$
\[ \delta_x = \delta_{y_1} + \delta_{y_2} \]



By combining the blocks, as in Figure \ref{fig:expanded}, we can now
compute how the partial derivatives propagate backward through an
entire node:

If we are using a logistic sigmoid function we have:
\[ \delta_{x_i} = \delta_y y (1-y) w_i \]
\[ \delta_{w_i} = \delta_y y (1-y) x_i \]

With a hyperbolic tangent sigmoid function, we have:
\begin{center}
$ \displaystyle \delta_{x_i} = \delta_y $sech$^2 \displaystyle (x_i) w_i $

$ \displaystyle \delta_{w_i} = \delta_y $sech$^2 \displaystyle (x_i) x_i $
\end{center}

For hidden nodes, we have:
\[ \delta_y = \sum \delta_{x_i} \]
where the sum is over the $x_i$ of the nodes in the next higher layer that
$y$ branches out to.  

For the output nodes, we have 
\[ \delta_y = 2(y_d-y) = 2(t(\vec{x})-f_w(\vec{x})) \]


\subsection{Back propagation Algorithm}

We can now write the pseudocode for a back propagation algorithm.  
The following uses the logistic sigmoid function, a constant learning 
parameter, $\epsilon$, and batch processing.
\vspace{-.1in}
\begin{tabbing}

Unt\=il progress is stopped \\ 
\{ \\
\>    For \= each input $\vec{x}$ \\
\>	\{ \\
\> \>	Compute the output, $y$, of each node in a forward manner.\\
\> \> For \= each node, in a backward manner, \\
\> \> \{ \\
\> \> \> Com\=pute $\delta_y(\vec{x}) $ using $ \delta_y(\vec{x}) = 2(t(\vec{x})  - f_w(\vec{x})) $ if node in output layer or \\
\> \> \> \>$ \delta_y(\vec{x}) = \sum \delta_{x_i}(\vec{x})$ if node in hidden layer. \\
\> \> \> For each input $x_i$ to the node, compute $\delta_{x_i}(\vec{x}) = \delta_y(\vec{x}) y (1-y) w_i$ \\
\> \> \> For each weight $w_i$ on edges to the node, compute 
$\delta_{w_i}(\vec{x}) = \delta_y (\vec{x}) y (1-y) x_i$ \\
\> \> \} \\
\> \}\\
\> Add up the $\delta_w (\vec{x})$ for each input $\vec{x}$ and update the weights using \\
\> $w_i \leftarrow w_i - \frac{1}{m} \epsilon \displaystyle \sum_{\vec{x} \in S} \delta_{w_i}(\vec{x})$ \\
\} \\
\end{tabbing}


\vspace{-.1in}

\nocite{Rumelhart86}
\begin{thebibliography}{1}

\bibitem{Rumelhart86}
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams.
\newblock Learning internal representations by error propagation. 
\newblock {\em Parallel Distributed Processing - Explorations in the 
Microstructure of Cognition}, pages 318-362, MIT Press, 1986.

\end{thebibliography}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes end here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}














