
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% NOTE: delete ,psfig here if using on athena
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentstyle[12pt,twoside,alltt,psfig]{article}
\textwidth=6in
\oddsidemargin=0.25in
\evensidemargin=0.25in
\topmargin=-0.1in
\footskip=0.8in
\parindent=0.0cm
\parskip=0.3cm
\textheight=8.00in
\setcounter{tocdepth} {3}
\setcounter{secnumdepth} {2}
\sloppy

\newenvironment{proof}{\noindent{\bf Proof:}}{\qed\bigskip}
\newenvironment{proof_sketch}{\noindent{\bf Sketch of Proof:}\hspace*{1em}}{\qed\bigskip}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{claim}{Claim}
\newtheorem{fact}{Fact}
\newtheorem{definition}{Definition}
\newtheorem{assumption}{Assumption}
\newtheorem{observation}{Observation}
\newtheorem{example}{Example}
\newcommand{\qed}{\rule{7pt}{7pt}}



\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}



\newcommand{\lecture}[6]{
   \pagestyle{myheadings} \thispagestyle{plain} \newpage
\setcounter{lecnum}{#1} \setcounter{page}{1} \noindent \begin{center}
\framebox{ \vbox{ \hbox to 6.28in { {\bf 6.858/18.428 Machine Learning
\hfill Lecture #1: #2} } \vspace{4mm} 
\vspace{2mm} \hbox to 6.28in { {\it
Lecturer: #3 \hfill Scribe: #4} } } } \end{center}
\markboth{Lecture #1: #2}{Lecture #1: #2} \vspace*{4mm} }



\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Lecture command takes 4 arguments:
%		ordinal number of the lecture
%		date of the lecure
%		lecturer
%		scribe---that is you.
%
% Fill out the next line with this information !!!
\lecture{14 }{October 31, 1994}{Ron Rivest}{Michael Best} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes start here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% For theorems, lemmas, definitions, remarks, etc. use commands
% {\theorem{...}}, {\lemma{...}}, {\definition{...}}, etc.
% For proofs, use \begin{proof} ... \end{proof}
%
% For figures use the following block:
%
% \begin{figure}
% \begin{center}
% \mbox{\psfig{figure=lecture-nn-fig-mm.ps}}
% \caption{A very nice picture.}
% \label{fig:picture}
% \end{center}
% \end{figure}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%
%%% BIBLIOGRAPHY %%%
%%%%%%%%%%%%%%%%%%%%

\def\thebibliography#1{\section{Bibliography}
  \list
  {[\arabic{enumi}]}{\settowidth\labelwidth{[#1]}\leftmargin\labelwidth
    \advance\leftmargin\labelsep
    \usecounter{enumi}}
    \def\newblock{\hskip .11em plus .33em minus -.07em}
    \sloppy
    \sfcode`\.=1000\relax}


\section{Outline}
\begin{itemize}
\item Finish weak $\Rightarrow$ strong learning and implications
\item Neural nets -- an introduction
\end{itemize}

\section{Weak learning}
\subsection{Weak learning -- initial bounds}

Recall our weak learning algorithm, $L$,  which gave us an error rate, 
$\beta \leq (1/2 - 1/p(n))$
with probability $1 - \delta$.  We built a ternary tree of majority
gates where the height of our tree was defined by 
$B = O(\log (p(n)) + \log \log (1/ \varepsilon))$.  The leaf nodes of our
execution tree are labeled with hypotheses from
$L$, using the oracle $EX(c, \cal D ^{'})$. 
Here $\cal D ^{'}$ are special distributions derived from the original
distribution $\cal D$.

We need to allocate a confidence measure to all points in the
execution tree which can fail.  We give a bound on the total number
of places where the algorithm can fail and allocate probability,
$\delta^{'} = \delta / 3^{B} $, to each point. We now wish to arrive
at some bounds on a strong learning algorithm $SL$, as a function of
similar bounds on our weak learning algorithm $L$.  Let the following
bounds, $t(\delta , n)$, $m(\delta , n)$,  and $u(\delta , n)$
be the running time, sample size required, and time to evaluate a
hypotheses respectively for algorithm $L$.

We can arrive at the following bounds:
$$U = {\rm time \: to \: evaluate \: hypothesis} = O(3^{B} \cdot u(\delta^{'},n)) 
    = {\rm poly} (n,1/\varepsilon, 1/\delta).$$

By ${\rm poly}(...)$ we mean a polynomial function in the 
given three variables.

In the last lecture we showed how to bound the total number of draws
required on the original distribution $\cal D$ to arrive at the desired
distribution of some leaf node, $\cal D^{'}$.  From these bounds we
arrive at,
\begin{eqnarray*}
M & = & {\rm sample \: size} = O(216^{B}/\varepsilon^{2}(m(\delta^{'}, n) 
+ p^{2}(n)\log (1/\delta^{'}))) \\
&  = & {\rm poly}(n,1/\varepsilon, 1/\delta)\\
T & = & {\rm total \: time} = O(648^{B}/\varepsilon^{2}(m(\delta^{'}, n) +
 p^{2}(n)\log (1/\delta^{'}) \cdot B \cdot u(\delta^{'}, n)) +
3^{B}t(\delta^{'},n)) \\
 &  = & {\rm poly}(n,1/\varepsilon, 1/\delta)
\end{eqnarray*}


\subsection{Weak learning -- improved bounds}

The above bounds are improveable by a few techniques.  In particular, it
has been shown that by ``damaging'' a hypothesis found to be too ``good'' we can
arrive at better asymptotic results.

$$M = {\rm sample \: size} = 1/\varepsilon \cdot {\rm poly}(n, \log (1/\varepsilon),
\log (1/\delta))$$
$$T = {\rm time} = 1/\varepsilon \cdot {\rm poly}(n, \log (1/\varepsilon),
\log (1/\delta))$$
$${\rm memory} =  {\rm poly}(n, \log (1/\varepsilon), \log (1/\delta))$$

Note that even though our sample size is growing with $1/\varepsilon$ the
memory requirements are growing with $\log (1/\varepsilon)$.  Therefore we
do not have to keep all of our examples in memory at all times.

$$size(h) = {\rm poly}(n, \log (1/\varepsilon))$$

We can see that this method produces a big compression factor from our
set  of samples
($O(1/\varepsilon)$) to our hypothesis ($O(\log (1/\varepsilon))$).

\subsection{On-line learning}

Now we'd like to consider how our bounds will change for an on-line
algorithm.

The probability of an error after $m$ trials is $1/m \cdot {\rm poly}(n,
\log (m))$.  So the expected number of mistakes in these first $m$
trials is ${\rm poly}(n, \log (m))$.  The computation time of the first $m$
trials is $m \cdot {\rm poly}(n, \log (m))$.  Finally the space used for the first
$m$ trials is ${\rm poly}(n, \log (m))$.

\subsection{An algorithm with a single majority gate}

Consider learning from a single majority gate.  In other words, where
our output is the result
of a single majority function applied to some set of $k$ hypothesis.  This
technique, due to Freund \cite{Freund92}, gives the best theoretic
results known to date.

Let $\gamma = 1/p(n)$, and our committee size, $k=1/(2\gamma^{2})
\log (1/\varepsilon)$.

Consider training the particular hypothesis, $h_{i+1}$.  Let $r$ be
the number of the first $i$ committee members which were correct on
the current example.  We wish to decide whether to give this
particular example to our current hypothesis, $h_{i+1}$.  We will
ignore the example if $r > \lfloor k/ 2 \rfloor$, or $r < i - \lceil
k/2 \rceil$.  This is because in the first case we would already have
a majority of hypothesis that are correct, and in the second case we
already have a majority that are incorrect.  Thus we consider training
our new hypothesis only on those samples in which it is possible that
an additional vote might influence the outcome.  Rather then always
excepting the sample in that case we accept it with the probability:

$$ \left( \begin{array}{c} k-i-1 \\ \lfloor k/2 \rfloor -r \end{array}
\right)  (1/2 + \gamma)^{\lfloor k/2-r \rfloor} 
(1/2-\gamma)^{\lceil k/2 \rceil -i-1+r} $$

If not enough examples are seen by our current hypothesis (because the
$i$ previous members are too good) then report a coin flip instead.
The bounds for this algorithm follow:

$$M = {\rm sample \: size} = 1/\varepsilon \cdot
\log ^2(1/\varepsilon)(\log \log (1/\varepsilon)+\log (1/\delta))
\cdot {\rm poly}(n) $$
$$T = {\rm time} = 1/\varepsilon \cdot \log ^2(1/\varepsilon)(\log \log (1/\varepsilon) +
\log (1/\delta)) \cdot {\rm poly}(n) $$
$${\rm memory} = \log (1/\varepsilon)(\log \log (1/\varepsilon) + \log (1/\delta)) \cdot
{\rm poly}(n)$$
$${\rm size \:  of \: hypothesis} = \log (1/\varepsilon) \cdot {\rm poly}(n) $$
$${\rm time \: to \: evaluate} = \log (1/\varepsilon) \cdot {\rm poly}(n) $$


\section{Neural networks -- an introduction}

We can think of a neural network as a layered directed acyclic graph.
We have $n$ layer 0 nodes of indegree 1 which we think of as our
inputs.  Usually there is a single node at the highest level of the
graph with outdegree 1 which we think of as our output.  Associated
with each directed edge is a weight, $w \epsilon \cal R$.  At any node
in the graph, the weighted sum of its inputs with their weights is
computed, $w \cdot x$, and the result is compared against some
thresholding function to determine whether the node's output is
positive or negative.


\begin{figure}[hbtp]
\begin{center}
\mbox{\psfig{figure=lecture14-fig-1.ps}}
\caption{A completely connected neural net with one hidden layer.}
\label{fig:g}
\end{center}
\end{figure}


We could use a sharp threshold function like we saw in the perceptron
model. However, more success has been found when using sigmoid
threshold functions.  One nice property of these functions is that
they, unlike a sharp threshold, are differentiable at all points.

Consider the following method generally referred to as
back-propagation:

1) Define an error measure for the weights, $Error(w)$. This could be
the mean-squared error for instance.

2)  Compute the gradient of the errors with respect to the weights,
$\bigtriangledown E$.

3)  Iteratively reduce the error by adjusting the weights, 
$w \leftarrow w - \varepsilon \cdot \bigtriangledown E$.

Continue this process until the error is sufficiently small.

We will continue this discussion of back-propagation in our next lecture.

\nocite{Schapire92}
\begin{thebibliography}{1}

\bibitem{Freund92}
Y. Freund.
\newblock An improved boosting algorithm and its implications on
learning complexity. 
\newblock {\em Proceedings of the 5th Workshop on Computational Learning
Theory}, pages 391-398, ACM Press, New York, NY, 1992.

\bibitem{Schapire92} R. E.  Schapire. {\em
The Design and Analysis of Efficient Learning Algorithms}.  MIT Press,
Cambridge, MA, 1992.

\end{thebibliography}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes end here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}














