%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% NOTE: delete ,psfig here if using on athena
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentstyle[12pt,twoside,alltt,eepic,epic]{article}
\textwidth=6in
\oddsidemargin=0.25in
\evensidemargin=0.25in
\topmargin=-0.1in
\footskip=0.8in
\parindent=0.0cm
\parskip=0.3cm
\textheight=8.00in
\setcounter{tocdepth} {3}
\setcounter{secnumdepth} {2}
\sloppy

\newenvironment{proof}{\noindent{\bf Proof:}}{\qed\bigskip}
\newenvironment{proof_sketch}{\noindent{\bf Sketch of Proof:}\hspace*{1em}}{\qed\bigskip}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{claim}{Claim}
\newtheorem{fact}{Fact}
\newtheorem{definition}{Definition}
\newtheorem{assumption}{Assumption}
\newtheorem{observation}{Observation}
\newtheorem{example}{Example}
\newcommand{\qed}{\rule{7pt}{7pt}}



\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}



\newcommand{\lecture}[6]{
   \pagestyle{myheadings} \thispagestyle{plain} \newpage
\setcounter{lecnum}{#1} \setcounter{page}{1} \noindent \begin{center}
\framebox{ \vbox{ \hbox to 6.28in { {\bf 6.858/18.428 Machine Learning
\hfill Lecture #1: #2} } \vspace{4mm} 
\vspace{2mm} \hbox to 6.28in { {\it
Lecturer: #3 \hfill Scribe: #4} } } } \end{center}
\markboth{Lecture #1: #2}{Lecture #1: #2} \vspace*{4mm} }



\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Lecture command takes 4 arguments:
%		ordinal number of the lecture
%		date of the lecure
%		lecturer
%		scribe---that is you.
%ls
% Fill out the next line with this information !!!
\lecture{12}{October 24, 1994}{Ron Rivest}{Lillian Lee} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes start here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% For theorems, lemmas, definitions, remarks, etc. use commands
% {\theorem{...}}, {\lemma{...}}, {\definition{...}}, etc.


% For proofs, use \begin{proof} ... \end{proof}
%
% For figures use the following block:
%
%  \begin{figure}[htb]                          
%  \begin{center}                               
%  \mbox{\psfig{figure=lecture-nn-fig-mm.ps}}   
%  \caption{A very nice picture.}               
%  \label{fig:picture}                          
%  \end{center}                                 
%  \end{figure}                                 
%                                               
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Outline}
\begin{itemize}
\item Definition of weak learnability
\item Weak learnability implies PAC-learnability
\subitem --Boosting confidence
\subitem --Boosting accuracy
\end{itemize}

\section{Weak Learning}

Notice that the definition of PAC-learning imposes some rather
stringent conditions, among them the requirement that a PAC-learning
algorithm be able to output hypotheses that have arbitrarily low error
rates, and be able to do so with arbitrarily high confidence.  In his
Ph.D. thesis, Schapire considered (among other topics) the {\it weak
learning} model, which differs from the PAC-learning model in that the
learning algorithm need only achieve a fixed (but small) error rate
and confidence.  We go through his proof of the rather surprising fact
that if a concept class is weakly learnable, then it is
PAC-learnable.\\

\subsection{Definition of Weak Learnability}

Let $EX(c,{\cal D})$ be as before. 

{\definition{A concept class ${\cal C}$ is {\bf weakly learnable} if
there exists a learning algorithm $L$ and constants $\epsilon_0 <
\frac{1}{2}$ and $\delta_0 < 1$  such that for every concept $c_* \in
{\cal C}$ and every distribution ${\cal D}$ on $X$, algorithm $L$,
with access to $EX(c,{\cal D})$, returns a hypothesis $h$ such that with
probability at least $1-\delta_0$, err(h) is at most $\epsilon_0$.}}

Notice that we can define ``weakly learnable using ${\cal H}$'' and
``efficiently weakly learnable'' in the same fashion as we did for
PAC-learnability.  Also note that this definition differs slightly
from that given in the book (and in the next lecture).  

\subsection{Weak learnability implies PAC-learnability}

Our main theorem is the following:

{\theorem{If a  concept class ${\cal C}$ is weakly learnable by ${\cal
H}$, then it is PAC-learnable by ${\cal H}'$, where ${\cal H}'$ is the
closure of ${\cal H}$ under majority operations.}}

We prove this theorem by first showing that we can boost the
confidence of certain learning algorithms, and then showing that we can
boost the accuracy of certain learning algorithms.

\subsection{Boosting confidence}

Suppose we have a weak learning algorithm $L$ that achieves
arbitrarily low error rates, but only does so with confidence at most
$\delta_0$.  Then, we can raise the confidence to any $\delta$ we want
(thereby creating a PAC-learning algorithm) by running the algorithm
many times and picking the output hypothesis with the lowest estimated
error rate.

Suppose we run $L$ $k$ times with parameters $\epsilon/2$ and
$\delta_0$.  Let $h_1,h_2,\ldots, h_k$ be the independent hypotheses
that $L$ outputs.  Then we have 
\begin{eqnarray*}
(\forall i){\rm Pr}\{err(h_i) \leq \frac{\epsilon}{2} \} & \geq & 1 -
\delta_0\\
\Rightarrow {\rm Pr}\{(\forall i) err(h_i) >\frac{\epsilon}{2} \}& \leq &
\delta_0^k \\
\Rightarrow {\rm Pr}\{(\exists i) err(h_i) \leq \frac{\epsilon}{2} \}
& \geq & 1 - \delta_0^k.\\
\end{eqnarray*}

Choose $k$ so that $\delta_0^k \leq \delta/2$, that is,  $k\ln\delta_0
\leq \ln(\delta/2) \Rightarrow k \geq \log_{\delta_0}(\delta/2)$,
since $\delta_0 < 1$.  Then, with probability at least $1 - \delta/2$,
we have some hypothesis $h_i$ such that $err(h_i) \leq \epsilon/2$.

Now it remains to figure out {\it which} of our $h_i$'s has the lowest
error rate; we do this by drawing a sample of size $m$ for each $h_i$
to estimate each $h_i$'s error rate.  It's sufficient that our
estimate be accurate within $\epsilon/4$ because our aim is to output
a hypothesis with error rate at most $\epsilon$: if $err(h_i) \leq
\epsilon/2$ then our error estimate will be at most $3\epsilon/4$,
whereas if $err(h_i) > \epsilon$, our error estimate will be above
$3\epsilon/4$.  The additive form of the Chernoff bound gives us
that $GE(p,m,(p+\epsilon/4)) \leq \exp(-2m(\epsilon/4)^2)$; setting
the latter quantity less than or equal to $\delta/(2k)$ implies that
$m \geq \frac{8}{\epsilon^2}\ln(2k/\delta)$.

What is the chance that we output a hypothesis with error greater than
$\epsilon$ with the above procedure?  Well, either among our $k$ tries
we never get a hypothesis with error rate $\leq \epsilon/2$ or we make
a bad mistake when estimating error rates.  But the chance of either
of these event occurring is $\leq \delta/2 + \delta/2 = \delta$, as
desired.

We remark that there is also a method for taking an algorithm with
fixed $\epsilon_0$ and $\delta_0$ and constructing an algorithm which,
on inputs $\epsilon$ and $\delta$, outputs with probability at least
$1 - \delta$ a hypothesis with error rate at worst only slightly
larger than $\epsilon$.

\subsection{Boosting accuracy}

Now suppose we have a weak learning algorithm $L$ that outputs
hypotheses with error rate at most $\beta < 1/2$ but does so with
probability at least $1 - \delta$ for any input $\delta$.  We then
take advantage of the fact that weak learning algorithms can learn
from {\it any} distribution to find hypotheses $h_1, h_2$, and $h_3$,
formed on three different distributions.  Our output is $h = {\rm
majority}(h_1,h_2,h_3)$ (this is why we learn using ${\cal H}'$); $h$
will have a lower error rate with high probability.

Let $c_*$ be the concept we are trying to learn, and let ${\cal D}_1 =
{\cal D}$.  The first step is to run $L$ with $EX(C,{\cal D})$ to get
a hypothesis $h_1$ such that $err(h_1) = \beta_1 \leq \beta < 1/2$.
Then, we want to focus $L$'s attention where it does poorly by running
$L$ on a new distribution that gives greater weight to examples on
which $h_1$ errs.

A first attempt to do this would be to only give $L$ examples where
the target concept $c_*$ differs from $h_1$.  However, the result of
this might be that $L$ outputs exactly the complement of $h_1$, which
is not necessarily any better than $h_1$.  So instead, let us consider
a new oracle which flips a coin: on heads, it returns an example $x$
such that $h_1(x) \neq c_*(x)$, and on tails, it returns an example
$x$ such that $h_1(x) = c_*(x)$ (in the next lecture, we will discuss
what to do if it is hard to get such a distribution).  Let ${\cal
D}_2$ be this new distribution.  Then, run $L$ using the oracle
$EX(c,{\cal D}_2)$; let $h_2$ be the output.  Let $\beta_2 \leq \beta$
be the error rate of $h_2$ on ${\cal D}_2$.

The next step is to find a hypothesis $h_3$ to act as a tiebreaker.
We will learn this hypothesis by using yet another distribution ${\cal
D}_3$ that consists entirely of examples where $h_1$ and $h_2$
disagree (if drawing examples from ${\cal D}_3$ is too hard (which
means that $h_1$ and $h_2$ are ``too good''), we output $h_1$ or
$h_2$).  Let $\beta_3 \leq \beta$ be $h_3$'s error rate on ${\cal
D}_3$.

Finally, we output $h={\rm majority}(h_1,h_2,h_3)$.

{\claim{$err_{{\cal D}}(h) \leq g(\beta)$, where $g(\epsilon) =
3\epsilon^2 - 2\epsilon^3$.}}

\begin{proof}

\begin{figure}[htb]                          
\begin{center}
\setlength{\unitlength}{0.0075in}
%
\begingroup\makeatletter\ifx\SetFigFont\undefined
% extract first six characters in \fmtname
\def\x#1#2#3#4#5#6#7\relax{\def\x{#1#2#3#4#5#6}}%
\expandafter\x\fmtname xxxxxx\relax \def\y{splain}%
\ifx\x\y   % LaTeX or SliTeX?
\gdef\SetFigFont#1#2#3{%
  \ifnum #1<17\tiny\else \ifnum #1<20\small\else
  \ifnum #1<24\normalsize\else \ifnum #1<29\large\else
  \ifnum #1<34\Large\else \ifnum #1<41\LARGE\else
     \huge\fi\fi\fi\fi\fi\fi
  \csname #3\endcsname}%
\else
\gdef\SetFigFont#1#2#3{\begingroup
  \count@#1\relax \ifnum 25<\count@\count@25\fi
  \def\x{\endgroup\@setsize\SetFigFont{#2pt}}%
  \expandafter\x
    \csname \romannumeral\the\count@ pt\expandafter\endcsname
    \csname @\romannumeral\the\count@ pt\endcsname
  \csname #3\endcsname}%
\fi
\fi\endgroup
\begin{picture}(500,263)(0,-10)
\path(100,210)(100,170)
\path(500,210)(500,170)
\path(500,50)(500,10)
\path(120,30)(500,30)
\path(100,50)(100,10)
\path(100,30)(120,30)
\path(300,40)(300,15)
\dashline{4.000}(200,190)(220,30)
\dashline{4.000}(400,190)(380,30)
\path(220,40)(220,15)
\path(380,40)(380,20)
\path(380,25)(380,35)(380,15)
\path(400,200)(400,175)
\path(220,25)(380,25)
\path(220,35)(380,35)
\path(100,190)(480,190)
\path(400,195)(500,195)
\path(400,185)(500,185)
\path(340,200)(340,180)
\path(340,195)(400,195)
\path(340,185)(400,185)
\dashline{4.000}(340,190)(300,30)
\dashline{4.000}(500,190)(500,30)
\dashline{4.000}(100,185)(100,40)
\path(223.000,62.000)(215.000,60.000)(223.000,58.000)
\path(215,60)(255,60)
\path(270,60)(305,60)
\path(297.000,58.000)(305.000,60.000)(297.000,62.000)
\path(318.000,62.000)(310.000,60.000)(318.000,58.000)
\path(310,60)(335,60)
\path(350,60)(380,60)
\path(372.000,58.000)(380.000,60.000)(372.000,62.000)
\path(108.000,237.000)(100.000,235.000)(108.000,233.000)
\path(100,235)(200,235)
\path(348.000,237.000)(340.000,235.000)(348.000,233.000)
\path(340,235)(415,235)
\path(430,235)(500,235)
\path(492.000,233.000)(500.000,235.000)(492.000,237.000)
\path(240,235)(340,235)
\path(332.000,233.000)(340.000,235.000)(332.000,237.000)
\path(480,190)	(487.531,190.000)
	(493.062,190.000)
	(500.000,190.000)

\path(500,190)	(500.000,190.000)

\put(375,170){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}D1}}}}}
\put(480,170){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}E1}}}}}
\put(175,170){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}B1}}}}}
\put(355,10){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}D2}}}}}
\put(195,10){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}B2}}}}}
\put(475,10){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}E2}}}}}
\put(75,170){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}A1}}}}}
\put(75,10){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}A2}}}}}
\put(280,0){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}C2=1/2}}}}}
\put(320,170){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}C1}}}}}
\put(0,190){\makebox(0,0)[lb]{\smash{{{\SetFigFont{10}{12.0}{rm}D}}}}}
\put(15,180){\makebox(0,0)[lb]{\smash{{{\SetFigFont{10}{12.0}{rm}1}}}}}
\put(0,30){\makebox(0,0)[lb]{\smash{{{\SetFigFont{10}{12.0}{rm}D}}}}}
\put(15,20){\makebox(0,0)[lb]{\smash{{{\SetFigFont{10}{12.0}{rm}2}}}}}
\put(25,190){\makebox(0,0)[lb]{\smash{{{\SetFigFont{10}{12.0}{rm}:}}}}}
\put(25,30){\makebox(0,0)[lb]{\smash{{{\SetFigFont{10}{12.0}{rm}:}}}}}
\put(255,55){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}$\gamma_1$}}}}}
\put(335,55){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}$\gamma_2$}}}}}
\put(205,230){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}$1-\beta_1$}}}}}
\put(415,230){\makebox(0,0)[lb]{\smash{{{\SetFigFont{8}{9.6}{rm}$\beta_1$}}}}}
\end{picture}
\end{center}
\caption{Mapping from ${\cal D}_1$ to ${\cal D}_2$}
\end{figure}

Consider the figure below.  C1-E1 is the region where $h_1$ is wrong,
B2-D2 is the region where $h_2$ is wrong, and C1-D1 is the region
where both are wrong. B2-C2 has weight $\gamma_1$ under ${\cal D}_2$;
C2-D2 has weight $\gamma_2$ under ${\cal D}_2$.  Notice that regions
to the right of C1 are mapped to regions with probability
$1/(2\beta_1)$ times more under ${\cal D}_2$ than they had under
${\cal D}_1$, whereas regions to the left of C1 are mapped to regions
with probability $1/(2(1-\beta_1))$ times less under ${\cal D}_2$ than
they had under ${\cal D}_1$.  Applying this mapping backwards gives us
the following information:

\begin{tabular}{ll}
Region & Weight under ${\cal D}_1$ \\ \hline
A1-B1 & $2(1 - \beta_1)(1/2 - \gamma_1)$ \\
B1-C1 & $2(1-\beta_1)\gamma_1$ \\
C1-D1 & $2\beta_1\gamma_2$ \\
D1-E1 & $2\beta_1(1/2-\gamma_2)$\\
\end{tabular}

From now on, assume all weights are with respect to ${\cal D}$. We have:
\begin{eqnarray*}
err(h) & \leq & {\rm Pr}\{h_1 \neq c_* \mbox{ and } h_2 \neq c_* \} + 
{\rm Pr}\{h_1 \neq h_2 \mbox{ and } h_3 \mbox{ wrong}\} \\
	& \leq & {\rm weight}(\mbox{C1-D1}) + ({\rm weight}(\mbox{B1-C1})
+ {\rm weight}(\mbox{D1-E1}))(\beta_3) \\
	& \leq & 2\beta_1\gamma_2 + (2(1-\beta_1)\gamma_1 +
2\beta_1(1/2-\gamma_2))\beta, \mbox{ since } \beta_3 \leq \beta \hskip
.1in (*)\\
	& = & \beta_1(2\gamma_2 - 2\beta\gamma_1 + 2\beta(1/2 -
\gamma_2)) + 2\beta\gamma_1 \\
	& = & \beta_1(2\gamma_2 - \beta(2(\gamma_1 + \gamma_2) -1)) +
2\beta\gamma_1 \\
	& = & \beta_1(2\gamma_2 + \beta(1 - 2(\gamma_1 + \gamma_2)) +
2\beta\gamma_1. \\
\end{eqnarray*}
But $\gamma_1 + \gamma_2 = \beta_2 \leq \beta < 1/2$ so $ 1-2(\gamma_1
+ \gamma_2)$ is positive $\Rightarrow (2\gamma_2 + \beta(1 -
2(\gamma_1 + \gamma_2))$ is positive.  So $err(h)$ is an increasing
function of $\beta_1$, which means we can replace $\beta_1$ by $\beta$
in (*).  Thus,
\begin{eqnarray*}
err(h) & \leq & 2\beta\gamma_2 + \beta(2(1-\beta)\gamma_1 + 2\beta(1/2
- \gamma_2)) \\
	& = &2\beta\gamma_2 + 2\beta\gamma_1 - 2\beta^2\gamma_1 +
\beta^2-2\beta^2\gamma_2\\
	& = & 2\beta\gamma_2(1 - \beta) + 2\beta\gamma_1(1 - \beta) +
\beta^2 \\ 
	& = & \beta^2 + 2\beta(1-\beta)(\gamma_1 + \gamma_2) \\
	& \leq & \beta^2 + 2\beta(1-\beta)\beta \\
	& = & 3\beta^2 -2\beta^3\\
	& = & g(\beta)\\
\end{eqnarray*}
\end{proof}  
\subsection{Next lecture}

To finish the proof, we still need to show that we can simulate
$EX(c,{\cal D}_2)$ and $EX(c,{\cal D}_3)$; after all, we would run
into trouble if it were never the case that $h_1(x) \neq h_2(x)$.
Also, it remains to be shown that we can iterate the above procedure
to get the error rate down to arbitrary levels in a reasonable amount
of time.


\begin{thebibliography}{99}

\bibitem{Schapire85} R. E.  Schapire. {\em
The Design and Analysis of Efficient Learning Algorithms}.  MIT Press,
Cambridge, MA, 1992.
\end{thebibliography}




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes end here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}














