%& latex

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% NOTE: delete ,psfig here if using on athena
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentstyle[12pt,twoside,alltt,psfig]{article}
%\documentstyle[12pt,twoside, epsf]{article}
\textwidth=6in
\oddsidemargin=0.25in
\evensidemargin=0.25in
\topmargin=-0.1in
\footskip=0.8in
\parindent=0.0cm
\parskip=0.3cm
\textheight=8.00in
\setcounter{tocdepth} {3}
\setcounter{secnumdepth} {2}
\sloppy

\newenvironment{proof}{\noindent{\bf Proof:}}{\qed\bigskip}
\newenvironment{proof_sketch}{\noindent{\bf Sketch of Proof:}\hspace*{1em}}{\qed\bigskip}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{claim}{Claim}
\newtheorem{fact}{Fact}
\newtheorem{definition}{Definition}
\newtheorem{assumption}{Assumption}
\newtheorem{observation}{Observation}
\newtheorem{example}{Example}
\newcommand{\qed}{\rule{7pt}{7pt}}



\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}



\newcommand{\lecture}[6]{
   \pagestyle{myheadings} \thispagestyle{plain} \newpage
\setcounter{lecnum}{#1} \setcounter{page}{1} \noindent \begin{center}
\framebox{ \vbox{ \hbox to 6.28in { {\bf 6.858/18.428 Machine Learning
\hfill Lecture #1: #2} } \vspace{4mm} 
\vspace{2mm} \hbox to 6.28in { {\it
Lecturer: #3 \hfill Scribe: #4} } } } \end{center}
\markboth{Lecture #1: #2}{Lecture #1: #2} \vspace*{4mm} }

\newcommand{\oracle}{EX(c, D)}
\newcommand{\aoracle}{EX(\overline{c}, D)}
\newcommand{\cnoracle}{EX_{CN}^\eta(c, D)}
\newcommand{\cnaoracle}{EX_{CN}^\eta(\overline{c}, D)}

\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Lecture command takes 4 arguments:
%		ordinal number of the lecture
%		date of the lecture
%		lecturer
%		scribe---that is you.
%
% Fill out the next line with this information !!!
\lecture{20}{November 23, 1994}{Javed Aslam}{Jon McAuliffe} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes start here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% For theorems, lemmas, definitions, remarks, etc. use commands
% {\theorem{...}}, {\lemma{...}}, {\definition{...}}, etc.
% For proofs, use \begin{proof} ... \end{proof}
%
% For figures use the following block:
%
% \begin{figure}
% \begin{center}
% \mbox{\psfig{figure=lecture-nn-fig-mm.ps}}
% \caption{A very nice picture.}
% \label{fig:picture}
% \end{center}
% \end{figure}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Outline}
\begin{itemize}
\item The statistical query model
\begin{itemize}
\item Statistical query learning algorithms
\item Simulators for noise-free statistics
\end{itemize}
\item{Choosing a best hypothesis in the statistical query model}
\end{itemize}

\section{The Statistical Query Model}
\subsection{Motivation}
In the previous lecture, we examined the result derived by Angluin and Laird in \cite{AL} that PAC learning is possible even in the presence of classification noise, as long as we are able to find a hypothesis $h \in {\cal H}$ which minimizes disagreement with a sample of bounded polynomial size. This finding was subject to the information-theoretic barrier of $1/2$ on the noise rate $\eta$. We also looked at some unfortunate complexity-theoretic results on finding a minimally disagreeing hypothesis, which told us that the problem is NP-hard even for relatively simple concept classes such as monotone conjunctions.

This new difficulty, as well as the obstacles faced by our original PAC algorithms when presented with noisy samples, stems in part from the fact that the PAC approach to learning depends quite sensitively on the classification of every example presented. The algorithms we have examined thus far usually make irretrievable decisions about the desired hypothesis based on individual labels for points $x^{(i)} \in X$.

In light of this realization, it makes sense to consider a formalism for learning which relies on gross statistical properties of a {\it population} of labelled examples, rather than on the information carried by each individual example. Consider, for example, Figure \ref{fig:subinterval}, where we wish to learn subintervals of the unit interval. Although no hypothesis exists which correctly classifies the given noisy sample, the indicated subinterval does have a certain statistical appeal: if the noise rate is $\frac{1}{4}$, then outside of $[a,b]$, roughly $75\%$ of the examples are negative, while within $[a,b]$ exactly the opposite holds.

\begin{figure}
\begin{center}
%\mbox{\epsffile{subinterval-example.ps}}
\mbox{\psfig{figure=lecture-20-subinterval-example.ps}}
\caption{Two samples from the target concept $[a,b]$: the left without and the right with noise.}
\label{fig:subinterval}
\end{center}
\end{figure}

So if noise affects the statistical characteristics of a sample in a uniform way, it should be possible to design a model of learning which first recovers statistical information about noiseless samples from the statistics of a noisy sample, then uses the recovered information to learn. We will spend the rest of the lecture developing a formalism which makes this statistical intuition precise and general.

\subsection{The Structure of a Statistical Query Learner}
As depicted in Figure \ref{fig:sqm}, a learning algorithm in the statistical query model relies on a {\it statistics oracle} for the information it uses to select an output hypothesis. In particular, the algorithm itself makes no use of information about particular labelled examples, and it expects the answers to its queries to be based on the noise-free characteristics of the target concept. The statistics oracle, in turn, relies on sample data from an examples oracle which may have no noise, classification noise, malicious adversarial noise, or another type of noise altogether. In cases where the data do indeed have noise, the statistics oracle acts as a {\it simulator}, producing noise-free statistics based on the noisy samples it draws. 

\begin{figure}
\begin{center}
%\mbox{\epsffile{stat-query-model.ps}}
\mbox{\psfig{figure=lecture-20-stat-query-model.ps}}
\caption{A schematic of the interface between algorithm and oracle in the statistical query model.}
\label{fig:sqm}
\end{center}
\end{figure}

The decision to divide the model into an algorithm based on noiseless statistics and an oracle which derives those noiseless statistics from potentially noisy data proves advantageous for two reasons.
\begin{description}
\item[Generality] Previous efforts to cope with noise involved heuristic techniques incorporated directly into the learning algorithm. Here, a single algorithm automatically succeeds in the presence of several different types of noise, by virtue of the noiseless statistics simulator on which it relies.
\item[Efficacy] We simply state that nearly all algorithms for PAC learning in the absence of noise can be recast in terms of queries to a noiseless statistics oracle, which immediately suggests the usefulness of the proposed formalism for robust learning.
\end{description}
We now proceed first to formalize the notion of learning by statistical query, setting aside issues of noise, then turn to the definition of simulators which produce noiseless statistics from noisy data.

\section{Statistical Query Learning Algorithms}
\subsection{Definitions for statistical query learning}
{\definition{A {\bf statistical query} is an ordered pair $(\chi, \tau)$, where $\chi : X \times \{0, 1\} \rightarrow \{0, 1\}$ is an indicator function on labelled examples from the instance space $X$, and $\tau$ is an error tolerance on the answer returned by the statistics oracle defined below.}}

{\definition{Let $P_\chi$ be the true probability with respect to noiseless examples from a target concept that $\chi = 1$. A {\bf statistics oracle} $STAT(c, D)$ for a target concept $c \in {\cal C}$ and distribution $D$ on $X$ takes a statistical query as defined above and returns an approximation $\widehat{P}_\chi$ such that $P_\chi - \tau \leq \widehat{P}_\chi \leq P_\chi + \tau$.}}

For example, $\chi$ may indicate the event $(x \in [a', b']) \wedge (\ell = 1)$. A statistical learning algorithm for the concept class depicted in Figure \ref{fig:subinterval} will certainly be able to make use of $P_\chi$ in this case. But since the statistics oracle may be using a noisy data source and must draw a bounded number of examples, it can only return the approximation $\widehat{P}_\chi$ which is within $\pm \tau$ of $P_\chi$.

We mention two relevant points about these definitions somewhat prematurely.
\begin{itemize}
\item This definition of $STAT(c, D)$ holds for arbitrary $\tau > 0$, though the number of examples will of course depend on how small $\tau$ is. We do not address exceptions, but they do exist ({\it e.g.} when learning from a malicious adversary).
\item The learning algorithm expects a correct $\widehat{P}_\chi$ with probability 1. The PAC-learning confidence parameter $\delta$ resurfaces in the discussion of simulator design.
\end{itemize}

{\definition{A concept class ${\cal C}$ is {\bf efficiently learnable by statistical queries} using hypothesis class ${\cal H}$ if there exists a learning algorithm $L$ and polynomials $p(\cdot,\cdot)$, $q(\cdot,\cdot)$, and $r(\cdot,\cdot)$ with the following property: for any probability distribution $D$ on $X$, $(\forall c \in {\cal C})(\forall \epsilon, 0 < \epsilon < 1)$ if $L$ is given access to $STAT(c, D)$ and input $\epsilon$, then:
\begin{enumerate}
\item for every query $(\chi, \tau)$ submitted,
	\begin{itemize}
	\item $\chi$ can be evaluated in time bounded above by $q(\frac{1}{\epsilon}, n)$
	\item $1/\tau$ is bounded above by $r(\frac{1}{\epsilon}, n)$
	\end{itemize}
\item $L$ will halt in time bounded above by $p(\frac{1}{\epsilon}, n)$
\item $L$ outputs a hypothesis $h \in {\cal H}$ such that error($h$) $\leq \epsilon$.
\end{enumerate}
(Here $n$ denotes the length of any example vector $x \in X$.)}}

If $L$ is deterministic, then an $\epsilon$-good $h$ will be returned with certainty. We will not consider the case where $L$ is randomized.

\subsection{An example: SQ-learning monotone conjunctions}
The concept class of monotone conjunctions is the set of all boolean formulae of the form $\bigwedge x_{i_k}$. Recall that the PAC-learning algorithm for monotone conjunctions begins with an initial conjunction of all variables $\bigwedge_{i = 1}^{n} x_i$. It then draws a sufficiently large sample from $EX(c, D)$ and removes from the initial conjunction all $x_i$'s which are false in any positive example.

The approach in the statistical query model is essentially the same. The statistical query algorithm also begins with a conjunction $h = \bigwedge_{i = 1}^{n} x_i$. Then, for each $i$ such that $ 1 \leq i \leq n$, the algorithm submits a query $(\chi_i, \tau_i)$, where $\chi_i$ is $[(x_i = 0) \wedge (\ell = 1)]$ and $\tau_i = \frac{\epsilon}{2n}$. We know that for any variable present in the target concept $c$, $P_{\chi_i}$ will be 0. Therefore, if $\widehat{P}_{\chi_i} > \frac{\epsilon}{2n}$, we have that $P_{\chi_i} > 0$ by choice of $\tau_i$. Hence we remove $x_i$ from $h$. Furthermore, if $\widehat{P}_{\chi_i} \leq \frac{\epsilon}{2n}$, we know $P_{\chi_i} \leq \frac{\epsilon}{n}$, again by choice of $\tau_i$.

Now, no negative examples can be misclassified by the output hypothesis $h$, because we never remove a variable $x_i$ from the initial conjunction unless it is guaranteed not to be in the target concept. Further, if a positive example is misclassified as negative by $h$, it is only because an errant $x_i$ is present in $h$ when it should have been removed. But the total probability of such an event is at most $\sum_{x_i \in h} P_{\chi_i} \leq \sum_{x_i \in h} \frac{\epsilon}{n} \leq n\frac{\epsilon}{n} = \epsilon$.

Hence we conclude that our algorithm outputs with certainty a hypothesis $h$ such that error$(h) \leq \epsilon$. The relevant polynomials $p(\cdot,\cdot)$, $q(\cdot,\cdot)$ and $r(\cdot,\cdot)$ are apparent. Thus monotone conjunctions are efficiently SQ-learnable.

We now claim that similar translations into the statistical query model exist for a wide range of efficiently PAC-learnable concept classes. One might speculate about the existence of a ``folk theorem'' to the effect that every PAC-learning algorithm has a statistical query counterpart. However, the PAC-learnable concept class of parity functions is known {\it not} to be efficiently SQ-learnable, so we can rule out the existence of such a theorem.

\subsection{Summary of advantages to SQ-learning algorithms}
We now present a brief synopsis of the reasons this modular approach to statistical learning proves especially useful to us.
\begin{enumerate}
\item ``Nearly'' every PAC-learning algorithm finds re-expression in the statistical query formalism.
\item A correct SQ-learning algorithm is also a correct PAC-learning algorithm in the absence of noise.
\item A correct SQ-learning algorithm is also a correct PAC-learning algorithm in the presence of classification noise, malicious adversarial noise, \ldots
\end{enumerate}

\section{Simulation of Noise-Free Statistics on Noisy Data}
We now turn our attention from learning algorithms which rely on an oracle for information about the statistical properties of noise-free samples to the design of the statistical oracle itself. First we consider how to produce statistical data given examples from a noise-free oracle of the type encountered in the PAC model, then we describe how to simulate the production of noise-free statistics given an oracle with classification noise. Other classes of noise are not considered here.

\subsection{Producing statistics from a noise-free source of examples}
Given that the statistical oracle need not concern itself with the possibility of incorrectly labelled examples, finding a $\widehat{P}_\chi$ within $\pm \tau$ of $P_\chi$ for a query $(\chi, \tau)$ is a straightforward exercise in elementary statistical sampling. The only complication lies in the requirement that the value returned by the statistical oracle is a correct $\widehat{P}_\chi$ with probability at least $1 - \delta$, where $\delta$ is the PAC-learning accuracy parameter. In other words, we must handle {\it $\delta$-budgeting} correctly.

If the number of queries is known beforehand to be fixed, such as in the case of SQ-learning monotone conjunctions, we simply distribute our $\delta$-budget uniformly over the samples drawn for each query. When the number of queries is unknown, two approaches suggest themselves. First, we can avail ourselves of the knowledge that $\sum_{i = 1}^{\infty} \frac{1}{i^2} = \frac{\pi^2}{6}$, so that if we let $\delta_i = \frac{6}{\pi^2} \cdot \frac{\delta}{i^2}$, then $\sum_{i} \delta_i \leq \delta$ as the series converges. Otherwise, we can choose to draw a single large sample after receiving all queries, basing the answer for each query on that sample alone. In this case, uniform convergence tells us that for a sample size only polynomially dependent on the VC-dimension of the query class, we can guarantee with probability at least $1 - \delta$ that all of the estimates $\widehat{P}_{\chi_i}$ will be within $\pm \tau$ of the true $P_{\chi_i}$, as we desire.

\subsection{Producing noise-free statistics from a sample oracle with classification noise}
First, note that the treatment of this topic provided in \cite{KV} has some significant errors. We now present an alternate treatment, which is covered in more detail in \cite{ASL}.

Suppose we have a query $(\chi, \tau)$ to the statistics oracle. We want to return the value $Pr_{\oracle}[\chi = 1]$, that is, a statistic taken with respect to the noise-free oracle $\oracle$, but we have access only to a source of labelled examples with classification noise $\cnoracle$. For the purposes of analysis only, consider these four different oracles:

\begin{tabular}{ll}
\parbox[t]{2.5in}{$\oracle$} &
\parbox[t]{3in}{The noise-free examples oracle} \\
\parbox[t]{2.5in}{$\aoracle$} &
\parbox[t]{3in}{The noise-free anti-examples oracle, that is, an oracle which outputs $\langle x, \overline{c(x)} \rangle$, where $c$ is the target concept} \\
\parbox[t]{2.5in}{$\cnoracle$} &
\parbox[t]{3in}{The examples oracle with classification noise rate $\eta < \frac{1}{2}$} \\
\parbox[t]{2.5in}{$\cnaoracle$} &
\parbox[t]{3in} {The anti-examples oracle (as above) with classification noise rate $\eta$}
\end{tabular}

{\claim{\[\cnoracle = \left\{ \begin{array}{ll}
								\oracle & {\it with\ probability}\ 1-\eta \\
                                \aoracle & {\it with\ probability}\ \eta
							\end{array}
						\right. \]
		\[\cnaoracle = \left\{ \begin{array}{ll}
                                \aoracle & {\it with\ probability}\ 1-\eta \\
                                \oracle & {\it with\ probability}\ \eta
							\end{array}
						\right. \]}}

\begin{proof} This follows immediately from the definition of the oracles and $\eta$.
\end{proof}

We can now find an expression for $Pr_{\cnoracle}[\chi = 1]$ in terms of $Pr_{\oracle}[\chi = 1]$,  $Pr_{\aoracle}[\chi = 1]$, and $\eta$, and do so similarly for $Pr_{\cnaoracle}[\chi = 1]$.

{\claim{
	\begin{equation}
		Pr_{\cnoracle}[\chi = 1] = (1-\eta) Pr_{\oracle}[\chi = 1] +
									\eta Pr_{\aoracle}[\chi = 1]
	\label{eq:cnor}
	\end{equation}
	\begin{equation}
		Pr_{\cnaoracle}[\chi = 1] = (1-\eta) Pr_{\aoracle}[\chi = 1] +
	   								\eta Pr_{\oracle}[\chi = 1]
	\label{eq:cnaor}
	\end{equation}}}

\begin{proof} 
These expressions can be derived using Claim 1.
\end{proof}

Remember that we wish to calculate $Pr_{\oracle}[\chi = 1]$. We can
easily compute $Pr_{\cnoracle}[\chi = 1]$ and $Pr_{\cnaoracle}[\chi =
1]$ via normal statistical sampling, and $\eta$ is an unknown we deal
with later. So, multiplying equation \ref{eq:cnor} by $(1-\eta)$,
equation \ref{eq:cnaor} by $\eta$, then subtracting \ref{eq:cnaor}
from \ref{eq:cnor}, we obtain
\begin{eqnarray*}
\lefteqn{(1-\eta) Pr_{\cnoracle}[\chi=1] - \eta Pr_{\cnaoracle}[\chi=1]} \\
& & = ((1-\eta)^2 Pr_{\oracle}[\chi=1] + \eta(1-\eta)Pr_{\aoracle}[\chi=1]) \\
& & \hspace{.2in} \mbox{} - (\eta(1-\eta)Pr_{\aoracle}[\chi=1] + \eta^2Pr_{\oracle}[\chi=1]) \\ 
& & = ((1-\eta)^2 - \eta^2) Pr_{\oracle}[\chi=1] \\
& & = Pr_{\oracle}[\chi=1] (1 - 2\eta + \eta^2 - \eta^2) \\
& & = Pr_{\oracle}[\chi=1] (1 - 2\eta)
\end{eqnarray*}
And therefore we have that

\begin{equation} Pr_{\oracle}[\chi=1] =
		\frac{(1-\eta) Pr_{\cnoracle}[\chi=1] - \eta Pr_{\cnaoracle}[\chi=1]}
		{(1 - 2\eta)}
\label{eq:bigresult}
\end{equation}

Note that we can remove reference to the classification noise anti-oracle in this equation by defining $\chi'(x,\ell) \stackrel{\triangle}{=} \chi(x,\overline{\ell})$, then replacing $Pr_{\cnaoracle}[\chi=1]$ with $Pr_{\cnoracle}[\chi'=1]$. Let us forego for a moment concerns about the unknown value $\eta$. We now have a method for estimating the value $P_\chi$ by calculating statistical estimates on noisy examples, but we need to insure that the returned value $\widehat{P}_\chi$ is within $\pm \tau$ of the true value. In order to accomplish this, we will need to determine what the accuracy bounds should be on the estimates from the noisy data. Finding these bounds is an exercise in {\it sensitivity analysis}.

\subsection{Sensitivity analysis}
{\claim{ Suppose we know that for values $0 \leq a, b, c, \tau \leq 1$,
\begin{eqnarray}
a & = & b - c \label{eq:gosh}\\
a & = & bc \label{eq:i-am}\\
a & = & b/c \label{eq:tired}
\end{eqnarray}
and we want to estimate $a$ within $\pm \tau$ in each case. Then for
\begin{description}
\item[\ref{eq:gosh}] we must estimate $b$ and $c$ to within $\pm \tau/2$
\item[\ref{eq:i-am}] we must estimate $b$ and $c$ to within $\pm \tau/3$
\item[\ref{eq:tired}] we must estimate $b$ and $c$ to within $\pm c\tau/3$.
\end{description}}}

\begin{proof}
\ref{eq:gosh} is obvious. For \ref{eq:i-am}, one boundary case is $(b+\frac{\tau}{3})(c+\frac{\tau}{3}) = bc + c\frac{\tau}{3} + b\frac{\tau}{3} + \frac{\tau^2}{9} \leq a + \frac{\tau}{3} + \frac{\tau}{3} + \frac{\tau^2}{9} \leq a + \tau$, which checks. \ref{eq:tired} is left as an exercise.
\end{proof}

Applying these results to our expression for $Pr_{\oracle}[\chi=1]$, it follows that we must estimate the numerator and denominator of the fraction to within $\pm (1-2\eta)\tau/3$. Repeated application of Claim 3 to the numerator tells us that the tightest estimation interval required for the entire fraction is $\pm (1-2\eta)\tau/18$.

Though we do not know $\eta$, we do know an upper bound $\eta_b$. Hence we can use an estimation interval of $\pm (1-2\eta_b)\frac{\tau}{18} \leq (1-2\eta)\frac{\tau}{18}$ and, by employing Chernoff bounds, draw a noisy sample(s) large enough to guarantee that $\widehat{P}_\chi$ can be within $\pm \tau$ of $P_\chi$. 

The only problem remaining is that equation \ref{eq:bigresult} depends on the unknown noise rate $\eta$ directly, and simply substituting $\eta_b$ will not guarantee a reasonable $\widehat{P}_\chi$. Further, there is no known method of sampling from $\cnoracle$ so as to compute an estimate of $\eta$.

\section{Choosing the Best $h \in {\cal H}$ in the Statistical Query Model}
In order to get around the problem that we have insufficient knowledge about $\eta$, we simply run the given statistical query algorithm (which relies on our simulated noise-free statistics oracle) many times, with many different guessed values of $\eta$, so that we are assured of having at least one run which produces an $h_i \in {\cal H}$ such that error$(h_i) \leq \epsilon$. This is depicted in figure \ref{fig:eta-guesses}.

\begin{figure}
\begin{center}
%\mbox{\epsffile{eta-guesses.ps}}
\mbox{\psfig{figure=lecture-20-eta-guesses.ps}}
\caption{Multiple runs of the SQ-learning algorithm, with different guesses at $\eta$.}
\label{fig:eta-guesses}
\end{center}
\end{figure}


We need to decide on a set of guesses at $\eta$ small enough to have size bounded by a polynomial, so that it will be feasible to search for the $h_i$ in the set of output hypotheses which minimizes disagreement with the sample drawn from $\cnoracle$. But the set of guesses must also be large enough to guarantee that some $\epsilon$-good hypothesis is produced. 

In the previous section, we found that $\eta$ needs to be known
within $\pm (1-2\eta) \tau / 18$.  However, the value of~$\eta$
is unknown, and the value of~$\tau$ may be different for each query.
Let~$\tau_{min}$ be a lower bound on the tolerance required for any
query submitted by the SQ learning algorithm.  (In practice,
$\tau_{min}$ is generally simple to calculate from the specification
of the SQ learning algorithm.)  Since $(1-2\eta_b) \tau_{min}/18 \leq
(1-2\eta) \tau / 18$, it is sufficient to find a value of $\eta$
within $\pm (1-2\eta_b) \tau_{min}/18$.
One simple way to achieve this is to guess values of~$\eta$
between~0 and~$\eta_b$ uniformly spaced $(1-2\eta_b) \tau_{min} / 9$ apart.
Note that this will require $\Theta (\frac{1}{\tau_{min} (1-2\eta_b)})$
guesses which is polynomial in the relevant learning parameters
as needed.

We can reduce the number of $\eta$-guesses by noting that finding a
value of $\eta$ within $\pm (1-2\eta) \tau_{min}/18$ is also
sufficient.  Thus, for ``large'' values of $\eta$, our guesses need to
be ``closely'' spaced, but for ``small'' values of $\eta$, our guesses
can be ``further'' apart.  By employing a guessing strategy similar to
that depicted in Figure~\ref{fig:large-to-small}, one can show that
only $\Theta (\frac{1}{\tau_{min}} \log \frac{1}{1-2\eta_b})$ guesses
are required.

%%%To derive our final result, we will need to assume that the smallest tolerance $\tau_{min}$ used by the SQ-learning algorithm is known to us beforehand. Empirically, this assumption is reasonable. Given $\tau_{min}$, we just need to make sure that only polynomially many guesses are required to arrive at a proposed $\eta$ which is within $\pm (1-2\eta_b)\frac{\tau_{min}}{18}$ of the true $\eta$, for this will provide an $\epsilon$-good $h$.

%%%Suppose guesses at $\eta$ are large initially and get smaller in the manner depicted in Figure \ref{fig:large-to-small}. That is, our first guess brings $(1-2\eta)$ close to zero, and in if this does not provide a sufficient estimate, then in subsequent guesses we repeatedly halve the distance between the proposed $eta$ and $eta_b$. The number of simulated runs of the SQ learning algorithm will then be $\Theta(\frac{1}{\tau_{min}} \log \frac{1}{1-2\eta_b})$.

\begin{figure}
\begin{center}
%\mbox{\epsffile{large-to-small.ps}}
\mbox{\psfig{figure=lecture-20-large-to-small.ps}}
\caption{A intuitive characterization of guessing at $\eta$.}
\label{fig:large-to-small}
\end{center}
\end{figure}

\begin{thebibliography}{99}
\bibitem{AL} Angluin, Dana, and Philip Laird. ``Learning from noisy examples.'' {\it Machine Learning}, 2(4): 343-370, 1988.
\bibitem{ASL} Aslam, Javed and Scott Decatur. ``Improved Noise-Tolerant Learning and Generalized Statistical Queries.'' {\it Harvard University Technical Report TR-17-94}, Center for Research in Computing Technology, Division of Applied Sciences.
\bibitem{KV} Kearns, Michael, and Umesh Vazirani. {\it An Introduction to Computational Learning Theory}. Cambridge, MA: MIT Press, 1994.
\end{thebibliography}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes end here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}














