
\documentstyle[12pt,twoside,alltt]{article}
\textwidth=6in
\oddsidemargin=0.25in
\evensidemargin=0.25in
\topmargin=-0.1in
\footskip=0.8in
\parindent=0.0cm
\parskip=0.3cm
\textheight=8.00in
\setcounter{tocdepth} {3}
\setcounter{secnumdepth} {2}
\sloppy

\newenvironment{proof}{\noindent{\bf Proof:}}{\qed\bigskip}
\newenvironment{proof_sketch}{\noindent{\bf Sketch of Proof:}\hspace*{1em}}{\qed\bigskip}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{claim}{Claim}
\newtheorem{fact}{Fact}
\newtheorem{definition}{Definition}
\newtheorem{assumption}{Assumption}
\newtheorem{observation}{Observation}
\newtheorem{example}{Example}
\newcommand{\qed}{\rule{7pt}{7pt}}

\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}

\newcommand{\lecture}[6]{
   \pagestyle{myheadings} \thispagestyle{plain} \newpage
\setcounter{lecnum}{#1} \setcounter{page}{1} \noindent \begin{center}
\framebox{ \vbox{ \hbox to 6.28in { {\bf 6.858/18.428 Machine Learning
\hfill Lecture #1: #2} } \vspace{4mm} 
\vspace{2mm} \hbox to 6.28in { {\it
Lecturer: #3 \hfill Scribe: #4} } } } \end{center}
\markboth{Lecture #1: #2}{Lecture #1: #2} \vspace*{4mm} }

\begin{document}

\lecture{24}{December 7, 1994}{Ron Rivest}{Ping Huang}{}

\section*{Outline}

\begin{itemize}
\item Learning DFAs: Angluin's L* algorithm
\item Learning DFAs without resets using homing sequences: Rivest \& Schapire
\end{itemize}

\section{Learning DFAs: Angluin's L* algorithm}

In review, Angluin's L* algorithm progresses by iteratively building a
table such as that in Figure~\ref{fig:example}.  Upon every iteration,
one of these actions is taken:

\begin{description}

\item[If table is not closed:] I.e., the bottom part of the table
contains a row which is different from every row in the top part of
the table.  The algorithm promotes that row to the top part, and
appropriate rows are added to the bottom part to preserve the $S \cdot
A-S$ property.

\item[If table is not consistent:] I.e., the top part of the table has
two states ($S_j$, $S_k$) whose rows are identical but those two
states' successors have rows which are {\em not} identical.  The
algorithm expands the set of experiments to differentiate between
$S_j$ and $S_k$ by adding the experiment $a \cdot e$, where $e$ is an
experiment which $S_{j \cdot a}$ and $S_{k \cdot a}$ differed on.

\item[If table is closed and consistent:] The algorithm constructs a
conjecture DFA and issues an equivalence query.  If it receives the
answer yes, then the algorithm successfully terminates.  If it
receives the answer no, it adds the received counterexample and all
its prefixes to $S$, and extends $S \cdot A-S$ accordingly.

\end{description}

Proof that the L* algorithm is correct is informally sketched:

\begin{enumerate}

\item For any regular language, there exists an unique minimal DFA
which recognizes that language.

\item We can generate a DFA consistent with a closed consistent table.

\item Any DFA which is consistent with the table has at least as many
states as unique rows in the top part of the table.

\item Once the algorithm builds the table to have $n$ different states
in $S$, then it will have the target DFA.

\end{enumerate}

Proof that the L* algorithm terminates is informally sketched:

\begin{enumerate}

\item Each closure failure increases the number of distinct rows in
$S$ by 1.

\item Each consistency failure increases the number of experiments in
$E$ by 1.  This must concomitantly increase the number of distinct row
patterns in $S$ by at least 1, because the added experiment is
precisely one which distinguishes between two formerly-identical rows.

\item Each counterexample increases the number of distinct rows in $S$
by at least 1, since the counterexample was not handled by the table
before.

\item The number of distinct rows in $S$ is therefore strictly
increasing, and reaches $n$, at which point the table must represent
the target DFA.

\end{enumerate}

\begin{figure}
\begin{center}
%% Generate file with:
%%   fig2dev -L latex lecture24-example.fig >! lecture24-example.tex
\include{lecture24-example}
\caption{Example of the table constructed by Angluin's L* algorithm
for learning DFAs when resets, membership queries, and equivalence
queries are available.\label{fig:example}}
\end{center}
\end{figure}

{\theorem{L* is a polynomial-time algorithm for inferring the exact
target DFA using membership queries and equivalence queries.}}

\begin{proof}
More specifically, the running time of the algorithm can be shown to
be polynomial in $n$ (number of states in the target machine) and $m$
(length of the longest counterexample).  Initially $|S|=|E|=1$.  At
all times, $|E| \leq n$ since at most $n$ consistency failures could
have occurred; similarly, the maximum length of the longest string in
$E$ is $n-1$.  At all times, $|S| \leq n+(n-1)m$ (the first term is a
bound on the number of states added to $S$ by closure or consistency
failures, the second term a bound on the number of states added to $S$
by counterexamples and their prefixes); the maximum string length in
$S$ is $n+m-1$.  The number of table entries is $|(S \cup S \cdot A)|
\cdot |E| \leq (k+1)(n+(n-1)m)n = O(mn^2)$, if the size $k$ of the
alphabet representation is assumed to be constant; this is also the
number of membership queries issued.  The maximum string length in $(S
\cup S \cdot A) \cdot E$ is $\leq m + 2n - 1$, so the size of the
table is polynomial.  Finally, at most $n-1$ equivalence queries are
issued.
\end{proof}

Note that while membership queries are quite reasonable, availability
of equivalence queries is a more difficult requirement to meet.  We
can, however, simulate equivalence queries by using randomized tests,
in a PAC-framework where $\epsilon$ and $\delta$ provide leeway for
statistical errors in the simulation process, allocating
$\frac{1}{2^i}$ of the $\delta$ budget on each successive equivalence
query if we do not know {\it a priori} how many will be issued.

\section{Learning DFAs without resets using homing sequences: Rivest/Schapire}

\subsection{Scenario}

\begin{figure}
\begin{center}
%% Generate file with:
%%   fig2dev -L latex lecture24-robot.fig >! lecture24-robot.tex
\include{lecture24-robot}
\caption{A robot can learn its environment by modelling it as a DFA,
but one which cannot be reset for learning with L*.  The robot's
location is marked by ``R'', its orientation by the arrow, and the
current DFA ``output'' is blue.  The black square represents an
inaccessible location.\label{fig:robot}}
\end{center}
\end{figure}

L* makes the assumption that it is possible to ``reset'' the DFA being
learned.  This may not be realistic; or, it may be too restrictive.
For example, suppose we have a robot trying to explore its
environment, 15 squares which the robot may travel to, plus 1 which it
cannot travel to, arranged in a 4x4 grid (Figure~\ref{fig:robot}).  In
addition to which of the 15 squares it is in, the robot's state
includes which of 4 directions it is facing.  The north and south
walls are red, the east and west walls are blue, and the barrier
square is black.  The robot may step forward ({\tt S}), turn left
({\tt L}), or turn right ({\tt R}); a step forward when the robot is
up against a wall or barrier is a no-op.

From the robot's point of view, the {\em environment} can be modeled
by a DFA: the robot feeds this DFA a string of symbols from the
alphabet \{{\tt S}, {\tt L}, {\tt R}\} (via its motor mechanisms), and
gets back the DFA output \{{\tt blue}, {\tt red}, {\tt black}\}, which
represents the color the robot sees in front of itself.

To do membership queries, strictly speaking, as required by the L*
algorithm, the robot would need a ``RESET'' button to get it back to
the state it was to start with; such teleportation is an unrealistic
requirement.  (Note that with prior knowledge about the semantics of
\{{\tt S}, {\tt L}, {\tt R}\} and under particular circumstances,
e.g., wrap-around world without barriers, after the robot issues some
membership query $q$, it could construct and issue $q\prime$ to undo
$q$.  However, in general there may not exist inverses, or even if one
existed it might not be constructable given just $q$.  Even in the
kind of ``strongly connected'' environment pictured in
Figure~\ref{fig:robot}, constructing $q\prime$ from $q$ would require
having {\em already} learned the environment.)

\subsection{What is a homing sequence?}

\begin{figure}
\begin{center}
%% Generate file with:
%%   fig2dev -L latex lecture24-home.fig >! lecture24-home.tex
\include{lecture24-home}
\caption{Homing sequences.  The observation sequence for the action
sequence {\tt X} determines exactly what state we end up in.\label{fig:home}}
\end{center}
\end{figure}

The key insight is that while there is no procedure for general finite
state automata to restore the initial state, a {\em homing sequence}
always exists, and can help us.  A homing sequence is a sequence of
actions whose output sequence fully determines what final state the
DFA is in, even if we did not know what the start state was.  For
example, in Figure~\ref{fig:home}, the action sequence {\tt X} is a
homing sequence; even if we do not know what was the start state, the
observation sequence can still tell us where we must have ended up.

{\theorem{Every minimal DFA has a homing sequence.}}

\begin{proof}
Given a minimal DFA, we construct a homing sequence as follows.  Check
if the null sequence suffices as a homing sequence (i.e., if every
state has an unique output).  If the null sequence is insufficient,
for every pair of states which currently look the same, by the
definition of a minimal DFA, there must exist some sequence to
distinguish them, of at most length $n$---we append that sequence to
our candidate homing sequence.  We iterate at most $n-1$ times to
distinguish every state; the resulting homing sequence has length
$O(n^2)$.
\end{proof}

(Note that the Rivest/Schapire procedure works much faster when used
with a homing sequence generator that adapts to the current state
rather than a statically generated and usually unnecessarily long one
as generated by the above construction.)

\subsection{Using homing sequences}

Assuming a robot is given some homing sequence {\tt HS}, what should
it do with it?  Let us plan to run $n$ copies of the L* algorithm,
each with one of the $n$ different states as its initial state.  (For
now, we assume $n$ is known; this assumption is actually unnecessary.)
We then can get the semantics of membership queries with resets by
time-slicing.  Suppose we are currently running copy $i$ of the L*
algorithm and it just issued a membership query; we then issue the
homing sequence, which takes us to state $S_k$.  We then put copy $i$
to sleep and wake up copy $k$ --- for which the DFA is in the correct
initial state.  Each time, {\em some} copy of the algorithm must make
progress, hence at worst we've introduced a factor of $n$ into the
running time.

\subsection{Generating homing sequences}

We've solved the problem that there is not a reset, but now we must be
able to generate a homing sequence for a DFA whose structure we do not
already know.  To do this, we add another layer of iterations to the
procedure.  We start with the trial homing sequence {\tt HS}$_1$ =
$\lambda$.  In the $i^{th}$ iteration, we try running $n$ copies of
the L* algorithm afresh, as above, using {\tt HS}$_i$ as the trial
homing sequence.  (Note: the ``afresh'' implies a tremendous loss of
efficiency, because all the work done by previous sets of $n$ copies
are thrown away, but there does not seem to be an easy way to
incorporate the learning already successfully done.)  If any copy of
the L* algorithm ever successfully completes in some iteration (i.e.,
its equivalence query returns the answer ``Yes''), then we're
definitely done.

How do we figure out when {\tt HS}$_i$ is not a homing sequence for
the DFA being learned (so that we know to stop running iteration $i$
and go on)?  During a given iteration, we know we're running $\leq n$
copies of L*, since any trial homing sequence can at generate at most
$n$ different observation sequences.  For any given copy of L*, the
generated table may be messed up, but we know that after a certain
number of wake-ups, it will have promoted more than $n$ unique row
patterns to the top part of its table (this assumes we know $n$).
When any of the copies of L* in an iteration has more than $n$ unique
row patterns in the top part of its table, we can stop that iteration
and go on to the next one, because clearly by then something has gone
wrong, and we will need to try a new homing sequence.

How do we construct {\tt HS}$_{i+1}$ from {\tt HS}$_i$?  We will have
need to extend our prospective homing sequence if and only if {\tt
HS}$_i$ could not distinguish between some two states $S_j$ and $S_k$.
Therefore, for the table of some copy of the L* algorithm, some table
entries were filled in using membership queries when the DFA was in
$S_j$, and some when the DFA was in $S_k$.  Choose an experiment
column ($e$) such that the table entries for the experiment in any two
randomly chosen rows ($S_1$, $S_2$) differ in value.  (These two rows
appear to be different, but may actually be the same.)  Pick one of
the table entries and test it (i.e., with $S_1 \cdot e$ or $S_2 \cdot
e$).  If we happened to have chosen the right two rows (probability
O($\frac{1}{n^2}$)) and the right experiment, then we will have found
how to extend the homing sequence: {\tt HS}$_{i+1}$ = {\tt HS}$_i
\cdot e$.

As for the assumption that $n$ is known, we can always add another
layer of iteration which keeps doubling our estimate for $n$.

\subsection{In practice\ldots}

Schapire implemented a program to try to learn the random graph walk
problem modeled as a DFA.  There were 400 nodes, each of degree 3,
which were randomly colored black or white.  Equivalence queries were
simulated with random walks.  One run to learn the graph used 258
copies of L* in parallel (in a particular iteration), issued 33,000
membership queries, took 1.7 million actions, and consumed 1 hour 20
minutes of CPU time.

\begin{thebibliography}{9}

\bibitem{A} Dana Angluin: ``Learning regular sets from queries and
counterexamples.'' {\it Information and  Computation}, 75:87-106, 1987.

\bibitem{RS} Ronald L. Rivest \& Robert E. Schapire: ``Inference of
Finite Automata Using Homing Sequences.''  {\it Information and
Computation}, 103:299-347, 1993.

\end{thebibliography}

\end{document}

