
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% NOTE: delete ,psfig here if using on athena
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentstyle[12pt,twoside,alltt,psfig]{article}
\textwidth=6in
\oddsidemargin=0.25in
\evensidemargin=0.25in
\topmargin=-0.1in
\footskip=0.8in
\parindent=0.0cm
\parskip=0.3cm
\textheight=8.00in
\setcounter{tocdepth} {3}
\setcounter{secnumdepth} {2}
\sloppy

\newenvironment{proof}{\noindent{\bf Proof:}}{\qed\bigskip}
\newenvironment{proof_sketch}{\noindent{\bf Sketch of Proof:}\hspace*{1em}}{\qed\bigskip}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{claim}{Claim}
\newtheorem{fact}{Fact}
\newtheorem{definition}{Definition}
\newtheorem{assumption}{Assumption}
\newtheorem{observation}{Observation}
\newtheorem{example}{Example}
\newcommand{\qed}{\rule{7pt}{7pt}}



\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}



\newcommand{\lecture}[6]{
   \pagestyle{myheadings} \thispagestyle{plain} \newpage
\setcounter{lecnum}{#1} \setcounter{page}{1} \noindent \begin{center}
\framebox{ \vbox{ \hbox to 6.28in { {\bf 6.858/18.428 Machine Learning
\hfill Lecture #1: #2} } \vspace{4mm} 
\vspace{2mm} \hbox to 6.28in { {\it
Lecturer: #3 \hfill Scribe: #4} } } } \end{center}
\markboth{Lecture #1: #2}{Lecture #1: #2} \vspace*{4mm} }



\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Lecture command takes 4 arguments:
%		ordinal number of the lecture
%		date of the lecure
%		lecturer
%		scribe---that is you.
%
% Fill out the next line with this information !!!
\lecture{11 }{October 19th, 1994}{Ron Rivest}{Matthew Andrews} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes start here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% For theorems, lemmas, definitions, remarks, etc. use commands
% {\theorem{...}}, {\lemma{...}}, {\definition{...}}, etc.
% For proofs, use \begin{proof} ... \end{proof}
%
% For figures use the following block:
%
% \begin{figure}
% \begin{center}
% \mbox{\psfig{figure=lecture-nn-fig-mm.ps}}
% \caption{A very nice picture.}
% \label{fig:picture}
% \end{center}
% \end{figure}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Outline}
\begin{itemize}
\item Bayesian learning
\item MDL (Minimum Description Length)
\end{itemize}

In this lecture we consider the technique of Bayesian Inference.  These notes
are largely taken from Chapter 5 of {\em Lecture Notes in Machine Learning}
by Ron Rivest.  





%%%%%%%%%%%%%%%%%%%%
%%% BIBLIOGRAPHY %%%
%%%%%%%%%%%%%%%%%%%%

\def\thebibliography#1{\section{Bibliography}
  \list
  {[\arabic{enumi}]}{\settowidth\labelwidth{[#1]}\leftmargin\labelwidth
    \advance\leftmargin\labelsep
    \usecounter{enumi}}
    \def\newblock{\hskip .11em plus .33em minus -.07em}
    \sloppy
    \sfcode`\.=1000\relax}


\section{Bayesian Inference -- Introduction}

In this chapter we review Bayesian inference.  In many ways the theory
of Bayesian inference is more advanced and sophisticated than the
formal theories of learning that have developed within the computer
science community; in other ways it is more primitive.  I believe that
it is instructive to include Bayesian inference in a study of machine
learning in order to highlight these differences.  Cheeseman has
several interesting articles (e.g., \cite{Cheeseman85,Cheeseman88})
that make a good starting point for reviewing the issues.  The journal
containing \cite{Cheeseman88} has many other articles discussing
Bayesian approaches versus others.

Bayesian inference handles the following issues reasonably well:
\begin{enumerate}
\item Prior knowledge of the learner; these are reflected in the learner's
      set of {\em prior} probabilities.
\item Evidence that does not eliminate hypotheses, 
      but which nonetheless gives preferential support
      to some hypotheses over others.
\item Degrees of belief other than certainty.
\item Consideration of how the learned information will be used.
\end{enumerate}
Issues of computational complexity are usually not addressed within this 
framework.

Bayesian inference is well-suited to situations where the amount of
data is sparse relative to the complexity of the hypothesis space, so
that the given data does not uniquely specify the correct answer.
When many possible answers remain, one needs a basis for deciding what
to do.  Bayesian inference is just such a theory.

In its simplest form, Bayesian inference is just the straightforward
application of Bayes' Rule.  Suppose we have a hypothesis space ${\cal
H} = \{H_1, H_2, \ldots\}$ which consists of a set of hypotheses which
are mutually exclusive (at most one of them is right) and exhaustive
(at least one of them is right).

The learner associates with each hypothesis $H_i$ a {\em prior
probability} $P(H_i)$ according to the learner's background knowledge
about the hypothesis space and the learning situation.  These
probabilities sum to one.  Here $P(H_i)$ is the learner's initial {\em
degree of belief} in the proposition that $H_i$ is correct.

The learning situation then presents the learner with some data $D$,
which provides some information about which hypothesis is correct.  In
our previous chapters, we only considered cases where $D$ was either
consistent with (or even determined by) each specific hypothesis, or
inconsistent with each specific hypothesis, so that hypotheses were
either eliminated or considered still viable.  Here we refine this
idea by introducing the {\em conditional probability} $P(D\mid H_i)$
that the data $D$ would be experienced in the learning situation, {\em
given that $H_i$ is the correct hypothesis}.  We assume that the
learner can in principle compute $P(D\mid H_i)$ from $D$ and $H_i$.
The extreme cases of $P(D\mid H_i)=1$ and $P(D\mid H_i)=0$ correspond
to our previous case where the data is either consistent with (forced
by) the hypothesis or inconsistent with the hypothesis.

We can then define $P(D)$ to be the {\em unconditional} probability that 
the learner expects to see the data $D$:
\begin{equation}
	P(D) = \sum_{i} P(D\mid H_i) P(H_i).
\label{Deqn}
\end{equation}

Bayes' Rule is just a straightforward fact about conditional probability:
\begin{equation}
	P(H_i \mid D) = \frac{P(D\mid H_i) P(H_i)}{P(D)}
\end{equation}
The interpretation of this rule is most interesting; it says that {\em
the learner's final degree of belief in $H_i$ is proportional to the
product of his initial degree of belief in $H_i$ and the conditional
probability of seeing $D$ given $H_i$.  } (The denominator $P(D)$ can
be considered as just a normalizing constant, so that the final
probabilities add up to one.)

What should the learner do if there is more than one hypothesis
consistent with the observed data $D$, and he is asked to specify some
single hypothesis as his ``inference''?  There are two standard
answers to this question:
\begin{itemize}
\item {\em (ML -- Maximum Likelihood)} 
Choose the hypothesis which has maximum likelihood
	$P(D\mid H_i)$ given the data $D$.
\item {\em (MAP -- Maximum A Posteriori)} 
Choose the hypothesis which has maximum {\em
        a posteriori} (i.e. final) probability $P(H_i\mid D)$.
\end{itemize}

The MAP method is the true Bayesian approach, since the ML technique
doesn't even make use of Bayes' Rule.  The second approach is better
(in my opinion), but the first is often used when one doesn't want to
work with prior probabilities for some reason.  The approaches
coincide when the prior distribution weights each hypothesis equally.

If the learner is asked to specify a particular hypothesis, and there
is a known cost for being wrong (i.e. the cost of outputting $H_i$
when $H_j$ is the truth), one can use the known set of final
probabilities to minimize the expected cost of making an error.

If the learner is not asked to specify a particular hypothesis as his
choice, but is instead asked to make a prediction based on his current
(final) set of beliefs, he can apply {\em transduction} (see Cheeseman
\cite{SelfCh87}).  Using $P(H_i)$ now to denote his current (as
updated) set of probabilities, this is merely the use of equation
(\ref{Deqn}) to predict the probability of any proposed new data $D$.
For example, if we are inferring Boolean functions, and there are many
more-or-less equally likely functions consistent with the data which
predict different values at a new point $x$, then we are not going to
get a strong prediction for the value of the unknown function at $x$.

\section{A Coin Example}

Suppose that we have a coin and we know that it is either fair
or else it has a 60\% bias in favour of heads.  That is, our two hypotheses
are:
\begin{itemize}
\item $H_1$ - fair coin
\item $H_2$ - 60\% heads, 40\% tails.
\end{itemize}
Usually when we are given a coin we assume that it is more likely
to be fair than to have a 60\% bias.  Hence we take for our 
{\em a priori} probabilities $P(H_1)=0.75$, $P(H_2)=0.25$.
Suppose that our first piece of evidence $D_1$ is a single coin toss which
comes up heads. 
\[
\begin{array}{cccccc}
P(D_1|H_1)&=&0.5&P(D_1|H_2)&=&0.60 \\
P(H_1)P(D_1|H_1)&=&0.375&P(H_2)P(D_1|H_2)&=&0.15 \\
\Rightarrow P(H_1|D_1)&=&0.714&P(H_2|D_1)&=&0.286.
\end{array}
\]
Hence after seeing one head we still believe that a fair coin
is the most likely hypothesis.  Our second piece of evidence
is a sequence of 100 coins tosses which contains 70 heads. Note that
we have seen a specific sequence of this form and so $P(D_2|H_1)=
2^{-100}$ (not ${100 \choose 70}2^{-100}$).  Similarly, $P(D_2|H_2)=
(0.6)^{70}(0.4)^{30}$. i.e.,
\[
\begin{array}{cccccc}
P(D_2|H_1)&=&7.9 \times 10^{-31}&P(D_2|H_2)&=&3.41 \times 10^{-28} \\
P(H_1|D_1)P(D_2|H_1)&=&5.63 \times 10^{-31}&P(H_2|D_1)P(D_2|H_2)
&=&9.75 \times 10^{-29} \\
\Rightarrow P(H_1|D_1,D_2)&=&0.0057&P(H_2|D_1,D_2)&=&0.9943.
\end{array}
\]
If our goal is to output a hypothesis then after the first flip
we would predict $H_1$. After the 101st flip we would predict $H_2$.
If however our goal is to make predictions then after the 101st flip
we would believe that the probability of heads is 
$(0.0057)(0.5)+(0.9943)(0.6) > 0.5$ and so we would predict heads.


\section{Probabilities as Degrees of Belief}

What is a {\em probability}?

There are a number of different kinds of probabilities, such as
``physical probability'', ``long run chance of success'', or
``subjective probability''.  As probability theory has matured, it has
become recognized that the most fundamental notion is that of {\em
subjective probability}, which represents a {\em degree of belief}.
The other notions can be viewed as derivative of this primary notion.
See Good\cite{Good59} for an excellent discussion of this issue.

The reason for mentioning this fact is that some people feel squeamish
about using probabilities to measure degrees of belief, since
elementary courses in probability theory often work with the
maximum-likelihood approach (and so avoid the use of prior or final
probabilities), or with situations where the prior probabilities are
given externally somehow.  It is not often emphasized that the notion
of degree of belief is really at the heart of probability theory.  See
Cheeseman \cite{Cheeseman85} for a fuller discussion of the role of
probability theory in AI.

\section{Prior Probabilities}

Prior probabilities are mildly controversial, in the sense that the
theory doesn't say much about how they are to be created, merely how
they are to be updated using Bayes' Rule.  This resultant ambiguity
motivates some people to avoid the issue altogether and stick to a
Maximum Likelihood approach.

Jaynes \cite{Jaynes68,Jaynes82} gives an nice overview of the problem,
together with an introduction to the principle of ``Maximum Entropy''
as a means for constructing prior probability distributions.  The
principle of maximum entropy suggests that one should use as a prior
the distribution with maximum entropy (i.e., choose
priors to maximise $\sum_i P(H_i)\log(\frac{1}{P(H_i)})$) 
subject to any known constraints.  This is a
formal way of picking the distribution which expresses maximum
ignorance or uncertainty, subject to the given constraints.

As an example of an application of the Maximum Entropy Principle,
suppose that you have a (six-sided) die, where you know that the die
is unfair since you know that the average roll is $4.5$.  That is, if
$p_i$ is the chance of rolling $i$, we know that 
$\sum_{i=1}^{6} ip_i =4.5$.
However, we know nothing else about the die.  {\em What is the chance
of rolling a 6?}

The problem is clearly underspecified, and there is no ``right''
answer.  We can construct a ``maximally ignorant'' prior probability
distribution by finding the distribution which maximizes
$\sum_{i=1}^{6} - p_i \log(p_i)$ subject to the given constraint:
\begin{equation}
	(p_1,\ldots,p_6) = (0.0543, 0.0788, 0.1142, 0.1654, 0.2398, 0.3475).
\end{equation}
The answer to our question is thus: {\em we expect to see a 6 with probability 0.3475}.

In a similar spirit, Rissanen \cite{Rissanen83} suggests using the following
prior probability distribution if the hypothesis space ${\cal H} =
\{H_1, H_2, \ldots, \}$ is countably infinite:
\begin{equation}
	P(H_i) = \frac{1}{c f^*(i)};
\end{equation}
here $f^*(n)$ is defined by the equation
\begin{equation}
	f^*(n) = n \cdot \log n \cdot \log \log n \cdots,
\end{equation}
where only the positive terms are included in the product, and
$c=2.865064\ldots$ is a normalizing constant.  This probability
distribution goes to $0$ as ``slowly as possible'' subject to the
condition that $\sum_i P(H_i) = 1$, and so is thus another way of
assuming ``maximum ignorance''.

\section{Prior Probabilities and Simplicity}

From another viewpoint, we can view the prior probability distribution
as a mechanism for expressing opinions about the {\em simplicity} of
hypotheses.  Suppose we have a situation where the each conditional
probability $P(D\mid H_i)$ is either $0$ or $1$ --- the data either
disqualifies a hypothesis or is determined by the hypothesis.  In this
case, in the absence of sufficient information to select a unique
hypothesis, one would like to select the ``simplest'' hypothesis which
is consistent with the data.  The MAP principle will select the one
with maximum prior probability.  Thus we can use the prior probability
distribution as a means to differentiate hypotheses according to their
simplicity, with simpler hypthoses getting larger prior probabilities.
For example, with Rissanen's prior, hypothesis $H_i$ is given a higher
initial probability that $H_j$ if $i<j$, so the ordering $\{H_1, H_2,
\ldots\}$ could be interpreted as a listing of the hypothesis space in
order of decreasing simplicity (increasing complexity).

\section{Minimum Description Length (MDL)}

Bayes Rule can be reformulated into an additive form using logarithms:
let us call
\begin{equation}
	L(H_i) = - \log(P(H_i))
\label{CPeqn}
\end{equation}
the initial (or prior) {\em complexity} of hypothesis $H_i$, in line
with our discussion above. (This term is nonstandard.)

Similarly, let us call
\begin{equation}
	L(D \mid H_i) = - \log(P(D\mid H_i))
\end{equation}
the {\em weight of evidence against $H_i$} provided by $D$.  Using
base-two logarithms, we can measure both complexities and weights of
evidence in {\em bits}.

If we define the {\em final complexity} of $H_i$, given $D$, as
\begin{equation}
	L(H_i\mid D) = -\log(P(H_i\mid D))
\end{equation}
then Bayes' Rule can be reinterpreted in additive form:
\begin{equation}
\label{CWeqn}
	L(H_i\mid D) = L(H_i) + L(D \mid H_i) + c 
\end{equation}
where $c$ is a suitable normalizing constant (which is a function of
$D$ and the priors only).

In some applications it is convenient to work with unnormalized
complexities $L'(H_i)$ only, since the relative probabilities of
various hypotheses depend only on the difference between their
relative complexities.

In any case, we see that Bayes' Rule provides a tradeoff between the
complexity of the hypothesis and the ``fit'' of the data to that
hypothesis.

\section{A Virus Example}

Suppose a person believes that he has a prior probability of carrying the
AIDS virus of $10^{-5}$, due to having had a blood transfusion before blood
screening was routine.  Suppose that a blood test is $99\%$ accurate
for carrying the virus, and that our poor friend tests positive. (This is the
data $D$.)  How likely is it that he actually has the virus? Using base 10
logarithms, we have
\begin{eqnarray*}
 L(\hbox{virus}) & = & -\log(10^{-5})= 5 \\
 L(\overline{\hbox{virus}}) & = & - \log(0.99999) = 4.3 \times 10^{-6} \\
 L(D\mid\hbox{virus}) & = & \log(0.99) = 4.3 \times 10^{-3} \\
 L(D\mid\overline{\hbox{virus}}) & = & \log(0.01) = 2 \\
 L'(\hbox{virus}\mid D) & = & 5 + 4.3 \times 10^{-3} = 5.043 \\
 L'(\overline{\hbox{virus}}\mid D) & = & 4.3 \times 10^{-6} + 2 = 2.000043 
\end{eqnarray*}

Thus, even after the positive test, our friend still is $1000$ times
more likely {\em not} to have the virus than to have it, since
$1000 \approx 10^{(5.043-2.000043)}$.

(This sort of analysis is obviously relevant when one begins to consider
the proposal to institute wide-spread testing for AIDS.)

\section{The coin example revisited}

Recall that $H_1$ was the hypothesis that we have a fair coin. $H_2$
was the hypothesis that the coin has a 60\% bias in favour of heads.
$D_1$ was the first coin toss which came up heads.  $D_2$ was a 
sequence of 100 coin tosses which contained 70 heads.  The initial
probabilities were $P(H_1)=0.75$, $P(H_2)=0.25$.  We now have,
\[
\begin{array}{cccccc}
L(H_1)&=&0.415&L(H_2)&=&2 \\
L(D_1|H_1)&=&1&L(D_1|H_2)&=&0.737 \\
\Rightarrow L(H_1)+L(D_1|H_1)&=&1.415&L(H_2)+L(D_1|H_2)&=&2.737 \\
\\
L(D_2|H_2)&=&100&L(D_2|H_2)&=&91.244 \\
\Rightarrow L(H_1|D_1)+L(D_2|H_2)&=&101.415&L(H_2|D_1)+L(D_2|H_2)&=&93.981
\end{array}
\]

\section{Relating MDL and MAP}

One can reverse the relationship between prior probabilities and
complexity.  That is, rather than determine the complexity of the
hypotheses from the prior probabilities, one can go in the other
direction and use a measure of the complexity of the hypotheses to
determine the prior probabilities, using equation (\ref{CPeqn}).

For example, Rissanen \cite{Rissanen78,Rissanen83} proposes that one
can use {\em syntactic} measures of the complexity of hypothesis --
how many bits does it take to encode the hypothesis?  This approach is
very natural for a computer scientist, since devising encodings for
hypotheses seems easier to think about than devising prior
probabilities.  This approach has been examined further by Hart
\cite{Hart87}.

Similarly, Rissanen proposes determining the weight of evidence
against $H_i$ provided by $D$ by actually encoding $D$ assuming $H_i$
as given.

Finally, the combined ``description length'' for $D$ using $H_i$ is
just the length of the encoding of $H_i$ plus the length of $D$ using
$H_i$ as given.  Except for the normalization factor (which is fixed
once $D$ is known), this is seen to be the same as the final
complexity as determined by equation (\ref{CWeqn}).

The hypothesis which gives $D$ the minimum description length (i.e.
which ``compresses'' $D$ the most, when the cost of encoding the
hypothesis is counted as well) is chosen as the ``best'' hypothesis
according to the Minimum Description Length Principle.

We see that we can view Rissanen's Minimum Description Length
Principle as a special case of Bayes' Rule.

Furthermore, we see that Bayesian inference and data compression
are related: a good theory about the data allows it to be compressed
better.

\section{A Cosmology Example}

Suppose that we would like to know whether or not galaxies are 
evenly spread about the universe.  We first of all divide the universe
into an $n \times n$ grid.  We have the following hypotheses.
\begin{itemize}  
\item $H_{p_0}$ is the hypothesis that there is no structure, every 
cell has a galaxy with probability $p_0$.  
\item $H_{p_0,\left((a_1,a_2),(b_1,b_2),p_1 \right)}$ is the hypothesis
that in the rectangle with lower left corner $(a_1,a_2)$ and
upper right corner $(b_1,b_2)$ there is a galaxy with probability $p_1$.  
Elsewhere there is a galaxy with probability $p_0$.
\item $H_{p_0,\left\{ \left((a_1^{(i)},a_2^{(i)}),
                (b_1^{(i)},b_2^{(i)}),p_i \right) \right\}}$ is the 
hypothesis that in the rectangle with lower left corner
$(a_1^{(i)},a_2^{(i)})$ and upper right corner 
$(b_1^{(i)},b_2^{(i)})$ there is a galaxy with probability $p_i$.
A cell which is outside all of the rectangles has a galaxy with
probability $p_0$.  Here we assume that $i \in \{1,2, \ldots , k\}$
and the rectangles are disjoint.
\end{itemize}
\begin{figure}
\begin{center}
\mbox{\psfig{figure=lecture11-fig-1.ps}}
\caption{A hypothesis with three probabilities.}
\label{fig:figure1}
\end{center}
\end{figure}
Let $H$ be the entropy function. That is, $H(p)=p\log (\frac{1}{p})
+(1-p)\log (\frac{1}{1-p})$. Let $\lambda$ be the number of bits
that we are using to encode probabilities and let
$D$ be the distribution of galaxies.  The following formulas
give the expected combined description lengths for $D$ for each of the 
hypotheses. 
\begin{itemize}
\item If the hypothesis is $H_{p_0}$ then the expected length is 
$\lambda + n^2H(p_0)$.  $\lambda$ bits are used to represent $p_0$ and 
$n^2H(p_0)$ is the expected number of bits required to represent $D$. 
\item If the hypothesis is $H_{p_0,\left\{ \left((a_1^{(i)},a_2^{(i)}),
    (b_1^{(i)},b_2^{(i)}),p_i \right) \right\}}$ then the expected 
length is, 
\[
(k+1)\lambda+4k\log n+\sum A_iH(p_i)+ (n^2-\sum A_i)H(p_0),
\] 
where $A_i$ is the area of the $i$th rectangle.
The first term in the above formula is the number of bits needed to 
represent the probabilities.
The second term corresponds to representing the rectangles.  The last two
terms are the expected number of bits needed to represent $D$.
\end{itemize}         

If we wanted to give prior probabilities to the hypotheses then
we would let,
\begin{eqnarray*}
P(H_{p_0})&=&c2^{-\lambda}, \\
P\left(H_{p_0,\left\{ \left((a_1^{(i)},a_2^{(i)}),
    (b_1^{(i)},b_2^{(i)}),p_i \right) \right\}}\right)&=&
  c2^{-(k+1)\lambda -4k\log n},
\end{eqnarray*}
where $c$ is a suitable normalizing constant.

\section{Summary}

Bayesian inference provides a way of handling the uncertainty that arises
from hypotheses of differing initial complexity and from uneven conditional
probabilities.  It also provides a way of formulating the tradeoff between
the complexity of a hypothesis and its fit to the data.


%% \bibliography{/th/u/rivest/papers/ai}

\begin{thebibliography}{10}

\bibitem{Cheeseman85}
Peter~C. Cheeseman.
\newblock In defense of probability.
\newblock In {\em Proceedings of the Ninth International Joint Conference on
  Artificial Intelligence}, pages 1002--1009, 1985.

\bibitem{Cheeseman88}
Peter~C. Cheeseman.
\newblock An inquiry into computer understanding.
\newblock {\em Computational Intelligence}, 4(1):58--66, February 1988.

\bibitem{Good59}
I.~J. Good.
\newblock Kinds of probability.
\newblock {\em Science}, 129(3347):443--447, February 1959.

\bibitem{Hart87}
George~W. Hart.
\newblock {\em Minimum Information Estimation of Structure}.
\newblock PhD thesis, MIT Dept.\ of Electrical Engineering and Computer
  Science, April 1987.
\newblock Appears as LIDS-TH-1664.

\bibitem{Jaynes68}
Edwin~T. Jaynes.
\newblock Prior probabilities.
\newblock {\em IEEE Transactions on Systems Science and Cybernetics},
  SSC-4(3):227--241, September 1968.

\bibitem{Jaynes82}
Edwin~T. Jaynes.
\newblock On the rationale of maximum-entropy methods.
\newblock {\em Proceedings of the IEEE}, 70(9):939--952, September 1982.

\bibitem{Levin80}
A.~Levin, Leonid.
\newblock A concept of independence with applications in various fields of
  mathematics.
\newblock Technical Report MIT/LCS/TR-235, MIT Laboratory for Computer Science,
  April 1980.

\bibitem{Rissanen78}
Jorma Rissanen.
\newblock Modeling by shortest data description.
\newblock {\em Automatica}, 14:465--471, 1978.

\bibitem{Rissanen83}
Jorma Rissanen.
\newblock A universal prior for integers and estimation by minimum description
  length.
\newblock {\em The Annals of Statistics}, 11(2):416--431, 1983.

\bibitem{SelfCh87}
Matthew Self and Peter~C. Cheeseman.
\newblock Bayesian prediction for artificial intelligence.
\newblock (unpublished manuscript).

\end{thebibliography}
\end{document}







%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your notes end here!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}














