
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% NOTE: delete ,psfig here if using on athena
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentstyle[12pt,twoside,alltt,psfig]{article}  %,psfig
\textwidth=6in
\oddsidemargin=0.25in
\evensidemargin=0.25in
\topmargin=-0.1in
\footskip=0.8in
\parindent=0.0cm
\parskip=0.3cm
\textheight=8.00in
\setcounter{tocdepth} {3}
\setcounter{secnumdepth} {2}
\sloppy

\newenvironment{proof}{\noindent{\bf Proof:}}{\qed\bigskip}
\newenvironment{proof_sketch}{\noindent{\bf Sketch of Proof:}\hspace*{1em}}{\qed\bigskip}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{claim}{Claim}
\newtheorem{fact}{Fact}
\newtheorem{definition}{Definition}
\newtheorem{assumption}{Assumption}
\newtheorem{observation}{Observation}
\newtheorem{example}{Example}
\newcommand{\qed}{\rule{7pt}{7pt}}



\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}



\newcommand{\lecture}[6]{
   \pagestyle{myheadings} \thispagestyle{plain} \newpage
\setcounter{lecnum}{#1} \setcounter{page}{1} \noindent \begin{center}
\framebox{ \vbox{ \hbox to 6.28in { {\bf 6.858/18.428 Machine Learning
\hfill Lecture #1: #2} } \vspace{4mm} 
\vspace{2mm} \hbox to 6.28in { {\it
Lecturer: #3 \hfill Scribe: #4} } } } \end{center}
\markboth{Lecture #1: #2}{Lecture #1: #2} \vspace*{4mm} }



\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Lecture command takes 4 arguments:
%		ordinal number of the lecture
%		date of the lecure
%		lecturer
%		scribe---that is you.
%
% Fill out the next line with this information !!!

\lecture{7}{10/3/94}{Ron Rivest}{Lewis Girod}

\section{}
\section{Outline}
\begin{itemize}
\item The VC-dimension and PAC Learning infinite classes
\item Definitions related to VC-dimension
\item VC-dimension examples
\item The Main property of $\Pi_c(m)$
\end{itemize}

\section{The Vapnik-Chervonenkis Dimension}
So far in our study of algorithms we have proven a number of finite
concept classes to be PAC learnable.  Recall from last lecture the
Occam�s razor result: for any {\em finite} concept class $\cal C$ and
any {\em finite} hypothesis class $\cal H$, if we draw
$\frac{1}{\epsilon}(\ln(|{\cal H}|)+\ln(\frac{1}{\delta}))$ examples,
and find a consistent hypothesis, then we PAC-learn.  This a very
strong result about finite classes -- but none of the PAC learning
results we have shown thus far apply to infinite concept classes.

Infinite-cardinality classes make up a large domain of important
learning problems.  As a simple example, the concept class formed by
the half spaces of the plane is clearly infinite.  Note that we have
shown this concept class to be learnable with a finite mistake bound
by the perceptron convergence algorithm, but we have not yet shown
this class to be PAC learnable.

In this lecture we will begin what will be a major result of this
course.  The result will prove an analogous Occam's razor result for
infinite concept classes.  Essentially, $VC_{DIM}({\cal H})$ replaces
$\ln(|{\cal H}|)$ in the number of samples we must draw.

\section{Definitions}

First of all, our notation is changing slightly; $\cal C$ represents a
concept class over an instance space $X$, both of which may now be
infinite.  We use $S$ to represent a finite sample chosen from $X$.
We will now define a function which characterizes the response of the
concepts in $\cal C$ to a given sample $S$.

{\definition{For any concept class $\cal C$ over instance space $X$,
for all finite samples $S \subseteq X$,
\[\Pi_{\cal C}(S) = \{c \cap S:c \in {\cal C}\}.\]}}

It is important at this point to give some kind of explanation of what
this means, especially since the set notation can be a little
confusing.  Essentially, $\Pi_{\cal C}(S)$ collects the set of all
subsets of the sample set $S$ which are made positive by some concept
$c$ in the concept class $\cal C$.  Thus $c \cap S$ represents the
elements of $S$ that are labelled postive by a concept $c$, and 
$\Pi_{\cal C}(S)$ is all such subsets for all concepts.

{\definition{If $|\Pi_{\cal C}(S)| = 2^{|S|}$ then $S$ is {\bf
shattered} by {\cal C}. 
In other words, fix a sample and a concept
class; if the following is true for every possible subset of the
sample, then the concept class shatters the sample: 
the subset is made positive by some concept in the concept class,
and that concept does not make any other element of $S$ (not in the
subset) positive.  That is, $S$ is shattered by {\cal C} if {\cal C}
realizes all possible dichotimies of $S$. }}

Finally, we define VC-dimension in these new terms.  When reading this
definition take careful note of the direction of the quantifiers.

{\definition{The {\bf Vapnik-Chervonenkis dimension} of ${\cal C}$,
denoted $VC_{DIM}({\cal C})$, is the largest cardinality $d$ such that
there exists a sample set of that cardinality $|S| = d$ that is
shattered by $\cal C$.  If no largest cardinality exists then
$VC_{DIM}({\cal C}) =
\infty$.}}


\section{Examples: Finding $VC_{DIM}$}

\begin{enumerate}
\begin{figure}[hbtp]
\begin{center}
\mbox{\psfig{figure=lecture07fig1.ps}}
\caption{A left-bounded interval on the real axis: 
No set of two samples is shattered.}
\label{fig:1}
\end{center}
\end{figure}
\item Consider the concept class of intervals $[0,a]$ on the real number line,
$0 < a < 1$.  Clearly samples of size 1 can be shattered by this
class, since if we pick some point $x$ as our sample, the point $a$
can be placed to the left or the right, thus excluding or including
$x$.  However, as \ref{fig:1} shows, no sample of size 2 can be
shattered since it is impossible to choose $a$ such that $x_2$ is
included and $x_1$ is excluded if $x_2 > x_1$.  Hence the $VC_{DIM} =
1$.

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture07fig2.ps}}
\caption{One interval on the real axis: No set of three points is shattered.}
\label{fig:2}
\end{center}
\end{figure}
\item Consider the class of subintervals $[a,b]$, $0 < a,b < 1$ 
(see \ref{fig:2}).  Here a sample of size 2 is shattered, but no
sample of size 3 is shattered, since no concept can satisfy a sample
whose middle point is negative and outer points are positive.  Hence,
$VC_{DIM} = 2$.

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture07fig3.ps}}
\caption{$k$ subintervals on the real axis.}
\label{fig:3}
\end{center}
\end{figure}
\item What about the class of $k$ non-intersecting subintervals (see 
\ref{fig:3})?  A sample of size $2k$ shatters (just treat each pair of 
points as a separate case
of example 2) but no sample of size $2k+1$ shatters, since if the
sample points are alternated positive/negative, starting with a
positive point, the positive points can't be covered by only $k$
intervals.  Hence $VC_{DIM} = 2k$.

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture07fig4.ps}}
\caption{Half-spaces of the plane.}
\label{fig:4}
\end{center}
\end{figure}
\item Consider the class of half-spaces of the plane.  Three samples can 
be shattered, but four cannot; hence $VC_{DIM} = 3$.  To see why four
points can never be shattered, consider two cases.  The trivial case is
when one point can be placed within a triangle formed by the other
three; then if the middle point is positive and the others are
negative, no half space can contain only the positive points.  If
however the points cannot be arranged in that pattern, then  label two
points diagonally across from each other as positive, and the other
two as negative
%%it is
%%possible to place three of the points on a convex hull which encloses
%%the fourth point.  If the outer two points on the hull are labelled
%%positively and the other two points are labelled negatively, no half
%%space can cover only the positive points 
(see \ref{fig:4}). Similar
results hold for higher dimensions: for half-spaces of ${\bf R^n}$,
$VC_{DIM} = n+1$.

\item The class of circles in the plane has $VC_{DIM} = 3$, using the same
argument as above.

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture07fig5.ps}}
\caption{Axis-aligned rectangles in the plane.}
\label{fig:5}
\end{center}
\end{figure}
\item The class of axis-aligned rectangles in the plane has $VC_{DIM} = 4$.  
The trick here is to note that for any collection of five points, at
least one of them must be interior to any rectangle bounded by the
other four; hence if the bounding points are positive, the interior
point cannot be made negative (see \ref{fig:5}).

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture07fig6.ps}}
\caption{Convex $d$-gons.}
\label{fig:6}
\end{center}
\end{figure}
\item The class of convex $d$-gons has $VC_{DIM} = 2d+1$.  To see that a 
set of $2d+1$ points can be shattered, place the vertices on a circle;
it is then possible to selectively include any of the vertices 
(see \ref{fig:6}).

\item The class of arbitrary convex polygons has infinite $VC_{DIM}$.
\end{enumerate}

\section{The Main Property of $\Pi_{\cal C}(m)$}

We begin our investigation of infinite concept classes by deriving a
polynomial bound on the magnitude of $\Pi_{\cal C}(S)$.  In order to
show this we need to introduce a new function which in the interests
of greater confusion is called $\Pi_{\cal C}(m)$:

{\definition{For any natural number $m$,
\[\Pi_{\cal C}(m) = \max\{|\Pi_{\cal C}(S)|:|S|=m\}.\]}}

In other words, $\Pi_{\cal C}(m)$ takes as its argument a sample size
$m$ and returns the maximum number of ways a sample of that size can
be labeled in the concept class $\cal C$.  This function behaves as a
measure of concept class complexity.  Suppose $d = VC_{DIM}({\cal
C})$; then $m \leq d$ implies $\Pi_{\cal C}(m) = 2^m$, and $m > d$
implies $\Pi_{\cal C}(m) < 2^m$.  We will further demonstrate that
$\Pi_{\cal C}(m)$ grows polynomially, within a constant factor of
$m^d$, whenever $m > d$.  In order to show this bound we need to
define a new function.

{\definition{For any natural numbers $m$ and $d$, 
\[\Phi_d(m) = \left\{ \begin{array}{l}
1 {\rm \,\, if \,\,} d=0 {\rm \,\, or \,\,} m=0 \\
\Phi_d(m-1) + \Phi_{d-1}(m-1) \end{array} \right. .\]}}

Our plan is to first show that $\Phi_d(m)$ grows polynomially, and
then show that $\Pi_{\cal C}(m) \leq \Phi_d(m)$, proving the main
property of $\Pi_{\cal C}(m)$: that it is polynomially bounded for $m
> d$.

{\lemma{$\Phi_d(m) = \sum_{i = 0}^d \left( \begin{array}{c} m \\ i
\end{array} \right)$}

{\rm
Base cases:
\begin{itemize}
\item if $d = 0$, $\left( \begin{array}{c} m \\ 0 \end{array}
\right) = 1 $
\item if $m = 0$, $\sum_{i=0}^d \left( \begin{array}{c} 0 \\ d \end{array} 
\right) = \left( \begin{array}{c} 0 \\ 0 \end{array} \right) = 1.$
\end{itemize}
Inductive step:
\begin{eqnarray*}
\Phi_d(m) & = & \Phi_d(m-1) + \Phi_{d-1}(m-1) \\
	  & = & \sum_{i = 0}^d \left( \begin{array}{c} m-1 \\ i \end{array} 
		\right) + \sum_{i = 0}^{d-1} \left( \begin{array}{c} m-1 \\ i 
		\end{array} \right) \\
	  & = & \sum_{i = 0}^d \left[ \left( \begin{array}{c} m-1 \\ i
		\end{array} \right) + \left( \begin{array}{c} m-1 \\ i-1
		\end{array} \right) \right] \\
	  & = & \sum_{i = 0}^d \left( \begin{array}{c} m \\ i \end{array} 
		\right)
\end{eqnarray*}}}

{\lemma{$\Phi_d(m)$ grows polynomially when $m > d$.}

{\rm
Since we are concerned with times when $m > d$, we have $0 \leq
\frac{d}{m} < 1$.  We can now write:
\[{\left(\frac{d}{m}\right)}^d \sum_{i=0}^d \left( \begin{array}{c} m \\ i
\end{array} \right) \leq \sum_{i=0}^d {\left( \frac{d}{m} \right)}^i \left( 
\begin{array}{c} m \\ i \end{array} \right) \leq \sum_{i=0}^m {\left( 
\frac{d}{m} \right)}^i \left( \begin{array}{c} m \\ i \end{array} \right) =
{\left( 1 + \frac{d}{m} \right)}^m \leq e^d.\]
Dividing both sides by ${\left( \frac{d}{m} \right)}^d$ , we get
\[ \Phi_d(m) \leq e^d {\left( \frac{m}{d} \right)}^d \leq { \left(
\frac{me}{d} \right)}^d.\]}}

We now wish to show the main property of $\Pi_{\cal C}(m)$, that it is
less than $\Phi_d(m)$ and is therefore polynomially bounded.

{\lemma{If $d = VC_{DIM}({\cal C})$, then for all $m$, $\Pi_{\cal C}(m) \leq \Phi_d(m)$}

{\rm The proof proceeds by double induction on $m$ and $d$.  We must
show the base cases whenever $m=0$ or $d=0$.  When $m = 0$, there can
only be one subset, hence $\Pi_{\cal C}(0) \leq 1 = \Phi_d(0)$.  When
$d=VC_{DIM}({\cal C}) = 0$, no set of points can be shattered, hence
all points can be labelled only one way.  From this we conclude that
$\Pi_{\cal C}(m) = 1 \leq \Phi_0(m)$.  So the lemma holds for the base
case.

We choose as the induction hypothesis that for all $m', d'$ such that
$m' \leq m$ and $d' \leq d$ and at least one of these inequalities is
strict, we assume $\Pi_{\cal C}(m') \leq \Phi_{d'}(m')$.

Now suppose we have a set $S$ of cardinality $m$.  In order to use the
induction hypothesis, we consider the number of ways of labelling $S$
after we remove some element $x$ from it:
\[|\Pi_{\cal C}(S-\{x\})| \leq \Pi_{\cal C}(m-1) \leq \Phi_d(m-1).\]
Our goal is to count the possible labellings of $S$; certainly there
are at least as many as there are ways of labelling $S-\{x\}$, but
this may not account for all of them.  So we need to count the number
of labellings in $S$ that correspond to a single labelling in
$S-\{x\}$ (i.e. this is the amount by which we're undercounting); if
we add in this correction term we will get our desired count.  In
order to do this, we consider the set of labellings of $S$ which may
be extended to include $x$:
\[{\cal C}' = \{c \in \Pi_{\cal C}(S) : x \not\in c, c \cup \{x\} \in 
\Pi_{\cal C}(S)\}. \]
By definition, $|{\cal C}'|$ is the number of labellings in $S$ that map to a
single labelling in $S-\{x\}$.  We also note that ${\cal C}' = 
\Pi_{\cal C'}(S-\{x\})$ since ${\cal C'}$ consists only of possible labellings
of $S-\{x\}$.  Furthermore, we know that the $VC_{DIM}({\cal C'}) \leq d-1$
because if $VC_{DIM}({\cal C'}) = d$ then $VC_{DIM}({\cal C}) = d+1$ since
by definition of $\cal C'$ there exists some element $x$ that can be 
arbitrarily included or excluded in a labelling of $S$, thus shattering $S$ 
if $S - \{x\}$ can be shattered.  From this we conclude:
\[ {\cal C'} = \Pi_{\cal C'}(S - \{x\}) \leq \Phi_{d-1}(m-1).\]
Putting this together with our previous result we have:
\[ |\Pi_{\cal C}(S)| \leq \Phi_d(m-1) + \Phi_{d-1}(m-1) =  \Phi_d(m).\]
This concludes the proof of the Lemma, demonstrating the main property of
$\Pi_{\cal C}(m)$.}}

\section{Next Lecture}

How do we PAC learn over an infinite concept class?  We need to show
that with high probability no bad concepts will survive.  It turns out
that a finite set of samples {\em can} kill off all the bad concepts!


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% For theorems, lemmas, definitions, remarks, etc. use commands
% {\theorem{...}}, {\lemma{...}}, {\definition{...}}, etc.
% For proofs, use \begin{proof} ... \end{proof}
%
% For figures use the following block:
%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%




\end{document}














