%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% NOTE: delete ,psfig here if using on athena
%%%%%%%%%%%%%%\documentstyle[12pt,twoside,alltt,psfig]{article}
%%%% Athena style \documentstyle[12pt,twoside,alltt,psbox]{article}
\documentstyle[12pt,twoside,alltt,psfig]{article}
%%%\documentstyle[12pt,twoside,alltt]{article}
\textwidth=6in
\oddsidemargin=0.25in
\evensidemargin=0.25in
\topmargin=-0.1in
\footskip=0.8in
\parindent=0.0cm
\parskip=0.3cm
\textheight=8.00in
\setcounter{tocdepth} {3}
\setcounter{secnumdepth} {2}
\sloppy

\newenvironment{proof}{\noindent{\bf Proof:}}{\qed\bigskip}
\newenvironment{proof_sketch}{\noindent{\bf Sketch of Proof:}\hspace*{1em}}{\qed\bigskip}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{claim}{Claim}
\newtheorem{fact}{Fact}
\newtheorem{definition}{Definition}
\newtheorem{assumption}{Assumption}
\newtheorem{observation}{Observation}
\newtheorem{example}{Example}
\newcommand{\qed}{\rule{7pt}{7pt}}



\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}



\newcommand{\lecture}[6]{
\pagestyle{myheadings} \thispagestyle{plain} \newpage
\setcounter{lecnum}{#1} \setcounter{page}{1} \noindent \begin{center}
\framebox{ \vbox{ \hbox to 6.28in { {\bf 6.858 Machine Learning
\hfill Lecture #1: #2} } \vspace{4mm} 
\vspace{2mm} \hbox to 6.28in { {\it
Lecturer: #3 \hfill Scribe: #4} } } } \end{center}
\markboth{Lecture #1: #2}{Lecture #1: #2} \vspace*{4mm} }



\begin{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Lecture command takes 4 arguments:
%		ordinal number of the lecture
%		date of the lecure
%		lecturer
%		scribe---that is you.
%
% Fill out the next line with this information !!!
\lecture{9 }{October 12, 1994}{Ron Rivest}{Vladimir Roussakov} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes start here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% For theorems, lemmas, definitions, remarks, etc. use commands
% {\theorem{...}}, {\lemma{...}}, {\definition{...}}, etc.
% For proofs, use \begin{proof} ... \end{proof}
%
% For figures use the following block:
%
% \begin{figure}[h]
% \begin{center}
% \mbox{\psfig{figure=notes-nn-fig-mm.ps}}
% \caption{A very nice picture.}
% \label{fig:picture}
% \end{center}
% \end{figure}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Outline}
\begin{itemize}
\item Estimating Error Rates
\item Uniform Convergence and VC-dimension
\end{itemize}

The major result we have seen so far is the relationship between VC-dimension 
and PAC-learning. It turns out that {\em finite} VC-dimension enables PAC-
learnability. Finite VC-dimension is used in other areas, such as computational 
geometry.

\section{Estimating Error Rates}

\subsection{Introduction}

We are moving away from PAC learning to learning in a more general sense. The 
objective is to find a relationship between {\em observed} error rate for a sample and 
the {\em true} error rate. We hope that picking a concept $c$ with a small observed 
error rate gives us small true error rate.

Let \parbox[t]{5in} {$X = $ domain, \\
$C =$ concept class on $X$, \\
$c(x) =  \left\{ \begin{array}{ll}
			\mbox{0 $-$ negative example}, \\
			\mbox{1 $-$ positive example}
	         \end{array}\right.$ \\
$D =$ some unknown distribution on $X$ from which we draw labelled examples. }

As in the PAC learning framework we attempt to learn the target concept $c_{\ast}$. However, we do 
not guarantee that $c$ belongs to $C$ and we may not be able to come up with a 
perfect target concept. Our goal is to minimize the error rate.

{\bf Notation:} \\
Define $\tilde{c} = c \oplus c_{\ast} $ as the error region for $c$; \\
$\tilde{c} = 1$ if and only if $c$ makes a mistake on $x$:
$\tilde{c}(x) = \left\{ \begin{array}{ll}
0, c(x) = c_{\ast}(x) \\
1, c(x) \neq c_{\ast}(x)
\end{array}\right.$

Then $D\tilde{c}$ = true error rate of $c$ with respect to $D$, and we are looking for
a concept $c$ with a small error rate.

Let $S = \{ x_1, x_2, ..., x_m \}$ be a random sample of size $m$ drawn according to $D$.
Define $D_m$ to be a uniform distribution on $S$. That is, each point in $S$ has weight $\frac {1} {m}$ and 
the weight is zero outside $S$.

Then $D_m\tilde{c}$ = observed (empirical) error rate of concept $c$ on sample $S$. 
$D_m\tilde{c} = \frac { \#errors } {m}$. Note that  $D_m\tilde{c}$
 is a random variable which depends on sample $S$.

Learning algorithms are often in the situation of estimating true error rates 
from empirical error rates, in order to attempt to find a hypothesis $c$ with low 
true error rate. The questions that need to be answered are the following:

\begin{itemize}
\item Why should approximate minimization of observed error rate yield a concept 
$c$ that approximately minimizes true error rate?
\item How big a sample do we need?
\end{itemize}


\section{Simple Error Rate Estimation (for a single concept)}

For a fixed concept $c$, every example is classified as positive or negative. Let 
$p = D\tilde{c}$  be the true error rate. This case is equivalent to flipping a biased 
coin a certain number of times. We are trying to estimate the bias $p$, (i.e., probability of heads of the coin).

\subsection{Review of the Law of Large Numbers}

{\bf Theorem:} (Strong Law of Large Numbers)

	With probability 1, $lim_{m \rightarrow \infty } p_m = p$

In other words, the  observed frequency of heads converges to the true frequency of 
heads. Here $p_m = D_m\tilde{c}$ is the observed error rate. This theorem does not tell us what happens for a 
particular number of samples. It is useless for estimating the sample size. For 
practical purposes we need the {\em rate} of convergence.

Note that \parbox[t]{5in}{ expected error rate $E(p_m) = p$, \\
	  variance $Var(p_m) = \frac {pq} {m} \leq 1/4m, (q
= (1 - p))$, \\
	  standard deviation $\sigma_m = \sigma(p_m) =
\sqrt{pq/m} \leq 1/2\sqrt{m}$
}

This is a characteristic result for estimating $p$ with $p_m$ - the quadratic 
dependency between $\sigma_m$ and the sample size.

\subsection{Review of the Law of Iterated Logarithm}

{\bf Theorem:} (Law of Iterated Logarithm)

\parbox[t]{5in}{$limsup_{m \rightarrow \infty}\frac{p_m - p} {\sigma_m\sqrt{2 \ln \ln m}} = 1$,  this is the upper limit  \\
 \\
$limsup_{m \rightarrow \infty}\frac{p - p_m} {\sigma_m\sqrt{2 \ln \ln m}} = 1$,  this is the lower limit
}

In other words we expect $p_m$ to stay close to $p$. Asymptotically $p_m$ is 
never more than $\sqrt{2 \ln \ln m}$ standard deviations away, or at most $\sqrt{\frac {\ln \ln m} {2m}}$
away. This is still an asymptotic result, and we are more interested in finite bounds for 
$m$.


\subsection{Review of Chernoff Bounds}

{\bf Notation:} \\
$$GE(p, m, r) = Prob[p_m \geq r]$$
$$LE(p, m, r) = Prob[p_m \leq r]$$

That is, this is useful when we are trying to relate the empirical
bias to the true bias.

{\bf Theorem:} [Hoeffding] (Additive form of Chernoff Bounds)

$$GE(p, m, p + \epsilon) \leq e^{-2m\epsilon^2}$$
$$LE(p, m, p - \epsilon) \leq e^{-2m\epsilon^2}$$

This result has a practical use. We take a coin with true bias $p$,
flip it $m$ times,
and now we can bound the probability $p_m$ differs too much from $p$.

{\bf Corollary:}

We have error $|p_m - p| \leq \epsilon$ with probability $\geq 1 - \delta$ if
$$m \geq \frac {1} {2\epsilon^2}ln(\frac {2} {\delta})$$
{\bf Note:} The sample size grows quadratically with $1/\epsilon$. Recall that in 
PAC learning sample sizes grew only linearly with $1/\epsilon$. The discrepancy is due to the fact that

\parbox[t]{5in}{$\sigma_m = \Theta(1/\sqrt{m}),$ if $p = 1/2$, but
$\sigma_m = \Theta(1/{m})$, if $p \leq 1/m$  (or $p \rightarrow 0$)
}.

The region we are working with in PAC-learning is small, and we are dealing with small 
probabilities and small deviations. Here we are near the middle of the region and the
deviation is a lot larger. In practice, we should use the fact that $p$ is small and
stay within a linear dependency on $1/\epsilon$.

This problem can be fixed using relative bounds rather than absolute bounds.

{\bf Theorem:} [Angluin and Valiant] (Multiplicative form of Chernoff Bounds)

$$GE(p, m, (1 + \alpha)p) \leq e^{mp\alpha^2/3}$$
$$LE(p, m, (1 - \alpha)p) \leq e^{mp\alpha^2/2}$$

This bounds the probability that $p_m$ is off by a multiplicative
factor from $p$.
As before the dependency is exponential. In addition the bound also 
depends on $p$. 

{\bf Note:} If $p \rightarrow 0$, (i.e., the true probability is
small), then the required sample size to get a good estimate for $p_m$ grows.

{\bf Corollary:} \\
If we keep $p$ from being too small, i.e., $p \geq \epsilon$, then the number of 
points that suffice to achieve $(1 - \alpha)\leq \frac {p_m} {p} \leq (1 + \alpha)$ with probability $\geq (1 - \delta)$ is
$$m = \frac {2} {\epsilon \alpha^2} ln(\frac {2} {\delta} ) $$


\section{Uniform Convergence. Estimating Error in Case of Many Concepts}

Now the objective is to estimate many biased coins simultaneously. The question 
is whether we can estimate all $D_m\tilde{c}$ simultaneously.

{\bf Example:} \\
%%%%%%%%%%%%%%%% DID NOT WORK ON ATHENA
%%%%%%%%%%%%%%%% PLEASE REMOVE COMMENTS (EXCEPT PSBOX) BEFORE PROCESSING
\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture-09-fig1.new.ps}}
%%,bbllx=79pt,bblly=746pt,bburx=649pt,bbury=482pt}}
%%%% Athena style \PSbox{lecture-09-fig1.ps}{5in}{3in}
\caption{True and estimated error rates in case of multiple concepts}
\label{fig:picture}
\end{center}
\end{figure}




\parbox[t]{5in}{$X = [0, 1] $ is a unit interval \\
$D = $ uniform distribution \\
$C = $ set of all concepts on [0, 1]
}

When many concepts converge simultaneously the estimate for one concept does not 
give the estimate for all (see figure~\ref{fig:picture}). For any sample $S$ there exists concept $c$ whose observed
error rate $D_m\tilde{c} = 0$, and the true error rate $D\tilde{c} = 1$  for any $m$. In other words concept $c$ agrees with the target everywhere on $S$ and does not agree anywhere outside $S$ for any $m$. Such a concept class is too rich. The VC-dimension of this concept class is infinite.

We are interested in a bound on $sup_{c \in C} |D_m\tilde{c} - D\tilde{c}|$, so that even for the worst-case 
$c \in C$ the observed error rate is close to the true error rate. If 
$sup_{c \in C} |D_m\tilde{c} - D\tilde{c}| \rightarrow 0$ as $m \rightarrow \infty$ we have {\em uniform convergence} of observed error rates to true error
rates. 

The case of finite concept class $C$ is easy, the result is given by the 
additive form of Chernoff bounds:
$$2|C|e^{2m\epsilon^2} \leq \delta$$
So,
$$m \geq \frac {1} {2\epsilon^2}(ln|C| + ln(\frac {2} {\delta}))$$ 
and the observed error rate converges.

What if $C$ is infinite? We want to use finite VC-dimension (assuming $C$ has finite 
VC-dimension) to replace $|C|$ and prove uniform convergence.

{\bf Theorem:} [Vapnik and Chervonenkis; Improved by Devroye, J. {\em Multivariate 
               Analysis} 12, 1 (1982), 72-79]

If class $C$ has VC-dimension $d$, then empirical error rates converge uniformly to 
true error rates:
$$Pr\{sup_{c \in C} |D_m\tilde{c} - D\tilde{c}| \geq \epsilon\} \leq 
4e^{(4\epsilon + 4\epsilon^2)}\Pi_c(m^2)e^{-2m\epsilon^2}$$

Qualitatively, the proof is similar to the proof of the VC-dimension
theorem for PAC learning. Recall that $$|\Pi_c(m)| \leq \left(\frac {em}
{d}\right)^d$$ That is, it grows polynomially with $m$. Similar analysis yields
that $$m \geq \Omega(\frac {d} {\epsilon^2}log \frac {1} {\epsilon} +
\frac {1} {\epsilon^2} log \frac {1} {\delta})$$ This gives the lower
bound of the number of examples that suffice to ensure that, with
probability $\geq (1-\delta)$ all empirical error rates are within
$\epsilon$ of their true error rates.

{\bf Note:} Once again there is a quadratic dependency on $1/\epsilon$, if we ignore the logarithm factors.

%Variations:   <? missed>

The uniform convergence is a powerful notion, and it gives a good learning algorithm. 
Consider the following procedure.

{\bf Procedure:}
\begin{itemize}
\item Draw a sample of size $m \geq \Omega(\frac {d} {\epsilon^2}log \frac {1} {\epsilon} + \frac {1} {\epsilon^2} log \frac {1} {\delta})$
\item Return concept $c'$ with the smallest empirical $D_m\tilde{c}$. 
\end{itemize}

{\bf Theorem:} \\
Suppose we choose a sample big enough so that $$(\forall c) 
  |D_m\tilde{c} - D\tilde{c}| \leq \epsilon$$
then the distance between the error rate of the target $c_\ast$ and $c'$ is 
$$|D_m\tilde{c_\ast} - D\tilde{c'}| \leq 2\epsilon$$
Here we assume that $c_\ast$ belongs to the concept class $C$, so the
condition of  the theorem holds for $c_\ast$.

{\bf Proof:} \\ For the target concept $c_\ast$ the distance between
observed and true error rates is $$|D_m\tilde{c_\ast} -
D\tilde{c_\ast}| \leq \epsilon$$ Because $c'$ has the minimal observed
error rate $$D_m\tilde{c'} \leq D_m\tilde{c_\ast}$$ Consequently (see
figure~\ref{fig:picturetwo}), $$|D_m\tilde{c'} - D\tilde{c'}| \leq
\epsilon$$ \qed

%%%%%%%%%%%%%%%% DID NOT WORK ON ATHENA
%%%%%%%%%%%%%%%% PLEASE REMOVE COMMENTS (EXCEPT PSBOX) BEFORE PROCESSING
\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture-09-fig2.new.ps}}
%%,bbllx=71pt,bblly=760pt,bburx=578pt,bbury=315pt}}
%%  8 
%% just playing --- 
%%%% Athena style \PSbox{lecture-09-fig2.ps}{5in}{3in}
\caption{Error rates for $c_{\ast}$ and $c'$. $D$ is the true error rate, $D=m$ is the observed error rate.}
\label{fig:picturetwo}
\end{center}
\end{figure}






This result means that in practice we just need to minimize on $D_m\tilde{c}$.

{\bf Note:} We assume that we have infinite computation power and ignore how hard it 
is to compute $c'$. In some cases it is better not to find $c'$, but only to 
approximate it.

\begin{thebibliography}{99}
\bibitem{xxx}Devroye, J. {\em Multivariate Analysis} 12, 1:72-79 (1982), .
\bibitem{yyy}Vapnik, V.N. and Chervonenkis, A.Y. On the uniform covergence of relative frequences of events to their probabilities. {\em Theory of Probability and its Applications}, 16(2):264-280 (1971).
\end{thebibliography}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes end here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}

