
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% NOTE: delete ,psfig here if using on athena
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentstyle[12pt,twoside,alltt,psfig]{article}
\textwidth=6in
\oddsidemargin=0.25in
\evensidemargin=0.25in
\topmargin=-0.1in
\footskip=0.8in
\parindent=0.0cm
\parskip=0.3cm
\textheight=8.00in
\setcounter{tocdepth} {3}
\setcounter{secnumdepth} {2}
\sloppy

\newenvironment{proof}{\noindent{\bf Proof:}}{\qed\bigskip}
\newenvironment{proof_sketch}{\noindent{\bf Sketch of Proof:}\hspace*{1em}}{\qed\bigskip}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{claim}{Claim}
\newtheorem{fact}{Fact}
\newtheorem{definition}{Definition}
\newtheorem{assumption}{Assumption}
\newtheorem{observation}{Observation}
\newtheorem{example}{Example}
\newcommand{\qed}{\rule{7pt}{7pt}}



\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}



\newcommand{\lecture}[6]{
   \pagestyle{myheadings} \thispagestyle{plain} \newpage
\setcounter{lecnum}{#1} \setcounter{page}{1} \noindent \begin{center}
\framebox{ \vbox{ \hbox to 6.28in { {\bf 6.858/18.428 Machine Learning
\hfill Lecture #1: #2} } \vspace{4mm} 
\vspace{2mm} \hbox to 6.28in { {\it
Lecturer: #3 \hfill Scribe: #4} } } } \end{center}
\markboth{Lecture #1: #2}{Lecture #1: #2} \vspace*{4mm} }



\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Lecture command takes 4 arguments:
%		ordinal number of the lecture
%		date of the lecure
%		lecturer
%		scribe---that is you.
%
% Fill out the next line with this information !!!
\lecture{17}{November 14, 1994}{Ron Rivest}{Milos Hauskrecht} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes start here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% For theorems, lemmas, definitions, remarks, etc. use commands
% {\theorem{...}}, {\lemma{...}}, {\definition{...}}, etc.
% For proofs, use \begin{proof} ... \end{proof}
%
% For figures use the following block:
%
% \begin{figure}
% \begin{center}
% \mbox{\psfig{figure=lecture-nn-fig-mm.ps}}
% \caption{A very nice picture.}
% \label{fig:picture}
% \end{center}
% \end{figure}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\section*{Outline}

\begin{enumerate}
\item Complexity of training neural nets

\begin{itemize}
\item Perceptrons/Linear programming
\item Judd's result
\item Blum/Rivest result
\end{itemize}

\item Approximation properties of neural nets
\begin{itemize}
\item Cybenko's result
\end{itemize}
\end{enumerate}

\section{Complexity of training neural nets}
So far in this course we have studied neural nets more from the practical side.
Now we will discuss properties of neural nets and address theoretical issues like:
\begin{itemize}
\item {\it computational complexity}:  
Given a training set of examples and a neural network, how hard is it to learn the weights of the net such that good performance on the training set is achieved?
\item {\it sample complexity}:  
How many samples do we need to see in the training phase in order to achieve good performance in the testing phase?
\end{itemize}


First we will focus on the issues related to computational complexity of neural nets and present some results achieved in recent years. 

\subsection{Complexity of training a one-node neural net}
Here we will assume the easiest neural net topology --- a neural net
with one n-input node (i.e., a perceptron). The question we would like
to answer is: Is it possible to train a one node neural net in a
reasonable time? That is, is there an algorithm that assigns weights
to the network such that it correctly classifies the given examples and
runs in time polynomial in the size of the input.

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture17-fig1.ps}}
\caption{One node neural net}
\label{fig:1}
\end{center}
\end{figure}

In the  case of a one node neural net (figure \ref{fig:1}), we are looking for a set of weights $\bar w$ such that\\
\hspace*{0.5 in} $ \bar{w} \bar{x} \geq  0 $ for positive examples and\\
\hspace*{0.5 in}
\( \bar{w} \bar{x} < 0 \) for negative examples. \\
In other words, we are looking for a hyperplane that separates the set of positive 
and negative examples.

We have already seen the perceptron algorithm. However, this algorithm gives no bound on the number of examples needed to do the training and thus cannot be used here.

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture17-fig2.ps}}
\caption{Problem of finding a feasible region of weights}
\label{fig:2}
\end{center}
\end{figure}

An alternate approach is based on the idea of viewing every example as a constraint 
in the hyperspace of weights (figure \ref{fig:2}). Then the problem of finding weights can be 
translated to the problem of finding a non-empty region in the weights space that 
satisfies all the constraints. But this problem is solvable in time polynomial 
in the size of the input using linear programming. Thus a one-node neural net 
can be trained in polynomial time.

The question that arises is whether there is an algorithm that can achieve 
the same result for larger neural networks. This problem was addressed
by the  works of Judd \cite{Judd90}, and Blum and Rivest \cite{Blum89}.

\subsection{Judd's result}

The main result of the Judd's work is showing that there are some intrinsic computational difficulties associated with training neural nets. This result is embodied in the following theorem.

{\theorem{(Judd) It is NP-complete to determine if a given neural net architecture can be trained to correctly classify a given set of examples}}

{\bf Idea of the proof:}\\
%%Use the standard approach to prove NP-completeness by:
%%\begin{itemize}
%\item showing that the problem is in NP
%\item showing a reduction of a known NP-complete problem to pairs of networks and examples 
%\end{itemize}
The reduction is from 3SAT. 
The following are conditions and properties of Judd's reduction:
\begin{enumerate}
\item The reduction produces multiple output networks (i.e., the neural network represents a mapping from inputs to a vector of outputs).
\item The architecture of the network varies with 3-SAT instance.
\item The network has ``don't cares'' in the outputs of the examples (partially specified examples).
\item The reduction uses only 3 training examples.
\end{enumerate} 


Judd also showed that the problem remains NP-complete even in the case
when only two thirds of the training examples are required to be correctly
classified. This implies that approximate training of neural nets is
also intrinsically hard.

Note that the theorem and the result for approximate training
 do not mean that for some specific network and/or some specific set of training 
examples the answer to the problem cannot be found in the polynomial time. It 
merely states that there are networks and examples for which the decision about 
the existence of weights that correctly classify the examples in the training
set is difficult.

Also note that the search problem (i.e.,  the problem of actually finding 
correct weights), is at least as hard as a decision problem. 

\subsection{Blum/Rivest result}

The theorem proven by Judd does not specify any particular network or
class of networks that is hard to train. This was done by Blum and
Rivest\cite{Blum89}, when they showed that training very simple
networks can in fact be NP-complete. The network they used is shown in
figure \ref{fig:3}. It is a completely connected 3-node neural net,
with  2
nodes in a hidden layer,  binary inputs and sharp threshold
functions.

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture17-fig3.ps}}
\caption{Three node neural net used by Blum and Rivest}
\label{fig:3}
\end{center}
\end{figure}

{\theorem {(Blum/Rivest) Training the 3-node neural network is
NP-complete.}}

\begin{proof}
The standard approach to prove NP-completeness will be
followed. First we will argue that the 3-node network is in NP. For
this we will use the following lemma (given without proof).

{\lemma {The number of bits needed to encode any weight (and threshold) is at most $O(n \lg n)$.}}

Since $O(n \lg n)$ bits are needed per weight and  the number
of weights is polynomial in $n$,  one can write down all bits
needed to encode the weights and thresholds in time polynomial in $n$.
Then since it is possible to verify in polynomial time that the network with
assigned weights correctly classifies all the examples, the 3-node
network training problem is in NP.

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture17-fig4.ps}}
\caption{Division of hyperspace to 4 quadrants by nodes $N_{1}$ and $N_{2}$.}
\label{fig:4}
\end{center}
\end{figure}

Now we need to show that some known NP-complete problem is reducible
to the problem of training the 3-node network. Before we show the
reduction we will give the geometric interpretation of the neural net
training problem.  Here the training example can be considered to be a
labelled point in $n$-dimensional Boolean space $\{0,1\}^{n}$,
and the linear functions thresholded by nodes $N_{1}$ and
$N_{2}$ can be thought of as $n-1$ dimensional hyperplanes in this
space. These hyperplanes define 4 quandrants of the space based on
four possible combinations of outputs for $N_{1}$ and $N_{2}$ (see
figure \ref{fig:4}).

Note that the structure of the network excludes some combinations of
the point labellings. This is because $N_{3}$ is a linear function of
outputs from $N_{1}$ and $N_{2}$, and therefore it cannot output the
same result for inputs $(+1 +1)$ and $(-1 -1)$ and the opposite result
for $(+1 -1)$ and $(-1 +1)$.

Using the above geometric interpretation, the 3-node network training problem can be equivalently described as:\\
Given a collection of points $\{0,1\}^{n}$, where each point is
labelled as positive  or negative, does there exist either:
\begin{enumerate}
\item a single plane that separates positive and negative points, or
\item two planes that partition the space such that any one quadrant
contains all the positive points and no negative points,  or all the
negative points and no positive points. 
\end{enumerate}

In the following we will base the reduction on the restricted version
of the above geometric problem: {\em Given points from $\{0,1\}^{n}$,
where each point is labelled positive or negative, do there exist 2 hyperplanes that
partition the points such that one quadrant contains all positive
points and no negative points?} Later, we will show how the general
case can be transformed to this restricted case.

First,  the Set-Splitting problem will be defined. This problem belongs to the family of NP-complete problems (we won't prove this here) and it will be used in the reduction.\\
{\definition {(Set-Splitting problem) \\
 Given a set of points $S = \{s_{1}, s_{2}, \cdots , s_{n} \} $ and family $C = \{c_{1}, c_{2}, \cdots, c_{k} \} $ of subsets of $S$, 
does there exist sets $S_1$ and $S_2$, satisfying $S=S_1\cup S_2$ and $S_1 \cap S_2 = \emptyset$
such that:
$$ \forall j \hspace{.2 in}   c_j \cap S_1 \neq \emptyset \mbox{ and }  c_j \cap S_2 \neq \emptyset $$}}

\pagebreak

Assume the following reduction from the Set-Splitting problem to the
restricted version of the geometric problem. 
Given an instance of the Set-Splitting problem 
$ S=\{s_{1}, s_{2}, \cdots , s_{k}\}$ and 
$C = \{c_{1}, c_{2}, \cdots , c_{l}\} \mbox{ where } c_{j} \subset S  \mbox{ for } 1\leq j \leq l$, 
create the following points in $\{0,1\}^{n}$:
\begin{enumerate}
\item  The origin $0^{n}$ is labelled positive.
\item For each $s_{i}$, point $p_{i} = 0^{i-1}10^{n-i}$ (i.e.,  $i-1$
$0's$ followed by 1, followed by $n-i$ $0's$) is labelled negative.
\item For each $c_{j} = \{s_{j_{1}}, s_{j_{2}}, \cdots, s_{j_{m}} \}$,
there is a point labeled positive with bits set to 1 at positions $j_{1},j_{2}, \cdots, j_{m}$ (i.e., $p_{j_{1}}+p_{j_{2}}+ \cdots + p_{j_{m}} $).
\end{enumerate} 

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture17-fig5.ps}}
\caption{An example of the transformation of the Set-Splitting problem to the geometric problem.}
\label{fig:5}
\end{center}
\end{figure}

{\bf Example} of the transformation (figure \ref{fig:5})\\
Let $S=\{s_{1},s_{2},s_{3} \}$, $C=\{c_{1},c_{2} \}$, $c_{1}= \{s_{1},s_{2} \}$, $c_{2} = \{s_{2}, s_{3} \}$.
Then the points created are:\\
\hspace*{.5 in} 
positive points : $(0 0 0)$, $(1 1 0)$, $(0 1 1)$;\\
\hspace*{.5 in}
negative points : $(0 0 1)$, $(0 1 0)$, $(1 0 0)$.


{\claim{An instance of the Set-Splitting problem has a solution iff
there exist a quadrant of $\{0,1\}^{n}$ with all the positive points and no negative points.}}

($ \Rightarrow$)\\ 
Assume that Set-Splitting has a solution $S_{1}$, $S_{2}$. Let planes $P_{1}$ and $P_{2}$
be two hyperplanes such that:\\
$P_{1}$: \(a_{1}x_{1} + a_{2}x_{2} + \cdots + a_{n}x_{n} = - \frac{1}{2} \)
where:\\
\hspace*{2.5 in} $a_{i} = -1$ if $s_{i} \in S_{1}$\\
\hspace*{2.5 in} $a_{i} = n$ if $s_{i} \notin S_{1}$;\\
$P_{2}$: \(b_{1}x_{1} + b_{2}x_{2} + \cdots + b_{n}x_{n} = - \frac{1}{2} \) where:\\
\hspace*{2.5 in} $b_{i} = -1$ if $s_{i} \in S_{2}$\\
\hspace*{2.5 in} $b_{i} = n$ if $s_{i} \notin S_{2}$.

Then plane $P_{1}$ separates all points $\{p_{i}\}$ corresponding to $s_{i} \in S_{1} $, and does not separate any positive point from the origin. Similarly $P_{2}$ separates all points corresponding to  $s_{i} \in S_{2}$, and does not separate any positive point from the origin. Therefore the quadrant for which $ \bar{a} \bar{x} > - \frac{1}{2}$ and $ \bar{b} \bar{x} > - \frac{1}{2} $ contains all the positive points and no negative points.


\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture17-fig6.ps}}
\caption{ }
\label{fig:6}
\end{center}
\end{figure}

($ \Leftarrow $)\\ Assume that there exist hyperplanes $P_{1}$ and
$P_{2}$ separating all positive points from negative points in
$\{0,1\}^{n}$. Let $S_{1}$ be the  set of all points separated from the
origin by $P_{1}$ and $S_2$ those separated by $P_{2}$ (break ties
arbitrarily). Note that the points of $S_{1}$ and $S_{2}$ cover all points
corresponding to $S$.

Now for the purpose of contradiction assume that all negative points
$p_{j_{1}}, p_{j_{2}}, \cdots , p_{j_{m}} $ corresponding to members
of $c_{j} = \{s_{j_{1}}, s_{j_{2}}, \cdots, s_{j_{m}}\}$ are separated
from the origin by $P_{1}$ (i.e., that $c_{j} \subset S_{1}$). Then
also $p_{j_{1}}+p_{j_{2}}+ \cdots + p_{j_{m}} $ must be separated from
the origin by $P_{1}$ (see figure \ref{fig:6}). As
$p_{j_{1}}+p_{j_{2}}+ \cdots + p_{j_{m}} $ is exactly a point
corresponding to $c_{j}$, it is labelled positive. But this contradicts
with the fact that $P_{1}$ and $P_{2}$ separate positive and negative
points. Therefore $c_{j} \subset S_{1}$ does not hold. Similarly,
$c_{j} \subset S_{2}$ does not hold.
 
This concludes the proof of the validity of the reduction for the case when positive points 
are confined to one quadrant. However the above reduction does not prove the general 
case.  

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture17-fig7.ps}}
\caption{The extension enforcing positive points to one quadrant}
\label{fig:7}
\end{center}
\end{figure}

In order to use the above idea for general case we need to enforce that all positive points 
be in one quadrant. This can be done by adding three new dimensions $x_{n+1}, x_{n+2}, x_{n+3}$ and the following set of additional points (see figure \ref{fig:7}):
\begin{itemize}
\item positive  points : \( ( \underbrace{00 \cdots0}_{n} \hspace{.1 in}101) \), \( (00 \cdots0 \hspace{.1 in}011) \);
\item negative points : $(00 \cdots 0 \hspace{.1 in}100)$, $(00 \cdots 0 \hspace{.1 in}010)$, $(00 \cdots 0 \hspace{.1 in}001)$, $(00 \cdots 0\hspace{0.1 in}111)$.
\end{itemize}

Using examples created in this manner, the only way positive and
negative points can be separated by 2 hyperplanes is by  confining all
positive points to one quadrant. But this allows us to use the
reduction described above and transform Set-Splitting problem to the
problem of finding weights of the 3-node neural network. Therefore
training the 3-node network is NP-complete.

\end{proof}


\section{Approximation properties of Neural Nets}

In this section we will focus on the problem  ``Can neural network approximate any bounded continuous function?''. This problem  was explored  by Cybenko \cite{Cybenko88} who showed that networks with 2-hidden layers are sufficient to achieve that. In the following a slightly modified version of Cybenko's theorem will be presented.      

\subsection{Cybenko's result}

{\theorem{ (Cybenko) Let $f$ be a bounded continuous function mapping $ [-1,1]^{n} \rightarrow R$ and $\sigma $ be any ``sigmoid'' function such that:
$$ \sigma (x) \rightarrow 1  \mbox{ as }  x \rightarrow + \infty$$
$$ \sigma (x) \rightarrow 0  \mbox{ as }  x \rightarrow - \infty.$$
Then $ \forall \epsilon > 0 $ there exists a neural network (NN) with nonlinearity $ \sigma$ and two hidden layers such that:\\
$$ \forall x \in [-1,1]^{n} \hspace{.5 in} \mid f(x)- NN(x) \mid < \epsilon$$}}

{\bf Proof sketch} :
A direct approach to approximate the function is to partition the input space to regions and approximate values of the function in every region by constant. In the proof the same ``partitioning'' idea will be used. 

Let $R$ be a set of regions that cover the input space. Then function $f$ can be approximated as : $$f(x) = \sum_{r \in R} I_{r}(x)f(x_{r})$$ where :\\
\hspace*{.5 in}$I_{r}(x)$ is function that is $1$ on any point in region $r$ and $0$ elsewhere;\\
\hspace*{0.5 in}$f(x_{r})$ is an approximation of $f$ on region $r$ (i.e., $x_{r}$ is a point in region $r$).

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture17-fig8.ps}}
\caption{Structure of the network approximating continuous function}
\label{fig:8}
\end{center}
\end{figure}

Assuming that we know how to compute $I_{r}(x)$ for all regions, then the function can be approximated by the network with the structure in figure \ref{fig:8}, where weights correspond to values of $f(x_{r})$ and the output node has no squashing function.

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture17-fig9.ps}}
\caption{Input space partitioned to square patches ($n=2$)}
\label{fig:9}
\end{center}
\end{figure}

Now we need to show how $I_{r}(x)$ can be computed. For simplicity assume that the number of input variables is 2 and that input regions are squares patches (see \ref{fig:9}).  
The idea of how to compute $I_{r}(x)$ is to:
\begin {itemize}
\item use two threshold functions per dimension such that both of them are 1 in the area of patch and less than 1 ouside;
\item add all such threshold functions and run the result through threshold of a $2n-0.5$, where n is the number of dimensions
\end{itemize}

\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture17-fig10.ps}}
\caption{An example of computation of  $I_{r}(x)$ using threshold functions }
\label{fig:10}
\end{center}
\end{figure}

The idea for $n=2$ is illustrated in figure \ref{fig:10}, where the threshold of $3.5$ filters all $x's$ outside the patch.

Note that the construction of $I_{r}(x)$ as described above can be realized by two layer network with sigmoid nonlinearities. As function to be approximated is continuous the required error bound can be always satisfied by altering the the size of patches. Therefore it is possible to construct a neural net with 2 hidden layers that approximates any bounded continuous function with error at most $\epsilon$, which concludes the proof. 

Note that no assumption on the number of nodes used was made by the
theorem. This is because the number of nodes in neural net grows with
decreasing $\epsilon$ (i.e., smaller $\epsilon$ values increase the
number of patches that cover the input space).

\begin{thebibliography}{99}

\bibitem{Blum89} A. Blum, R. Rivest. Training a 3-node nerural net is NP-complete. In D.S. Touretzky, editor, {\em Advances in Neural Information Processin Systems I}, Morgan Kaufmann, pp 494-501, 1989.

\bibitem{Cybenko88} G. Cybenko. Continuous-valued neural networks with two hidden layers are sufficient. (unpublished), 1988

\bibitem{Judd90} J.S. Judd. Neural network design and the complexity of learning. MIT Press, 1990

\end{thebibliography}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes end here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}














