\documentclass[11pt,twoside]{article}
\usepackage{macros-course-lectures}
%\renewcommand{\bar}[1]{\overline{#1}}

\begin{document}

\lecture{notes13}{Consensus; Generic Models for Distributed
Computing}{October 22, 2002}

\noindent
Readings: \\
*** Keidar, Rajsbaum.  On the cost of fault-tolerant consensus when
there are no faults---a tutorial. \\
*** Gafni, Round-by-round fault detectors:  Unifying synchrony and
asynchrony \\
\\
We didn't get to the following; will cover them at a later date: \\
*** Attiya, Welch \\
Ladin, Liskov, Shrira, Ghemawat \\
*** Fekete, Gupta, Luchangco, Lynch, Shvartsman \\
\\
Next time:  Performing work in distributed systems (Alex Shvartsman) \\
Dwork, Halpern, Waarts.  Performing work efficiently in the
presence of faults.  Sections 1, 2.1, 2.2.  Browse the rest. \\
Chlebus, De Prisco, Shvartsman.  Performing tasks...  Sections 1, 2,
3.1, 4.1.\\
*** Georgiou, Russell, Shvartsman.  The complexity of synchronous
iterative do-all with crashes.  Sections 1-4.\\
*** Malewicz, Russell, Shvartsman. Distributed cooperation in the absence
of communication.  Sections 1-4. \\
Georgiou, Shvartsman. Cooperative computing with fragmentable 
and mergeable groups. Sections 1-4. 

\section{Keidar, Rajsbaum}

\subsection{Overview}

\noindent
On the cost of fault-tolerant consensus when there are no faults---a
tutorial. \\
Overviews many results about time bounds on consensus,
and presents some new results and new proofs of old results. \\
I'll focus on Section 4. \\
\\
Their motivation starts with partially synchronous models (models with
timing assumptions, which may hold from some point on), and
asynchronous models with unreliable failure detectors like diamond-S. \\
But the actual technical results they get are for the round-based
synchronous model.  They then use those to get results about the other
models as corollaries. \\
\\
So, I'll focus on the results about the synchronous model. \\
Process stopping failures. \\
Assume a worst-case bound of $t$ on the number of failures, but they
also consider the number $f$ that occur in particular executions. \\
\\
The results I will describe deal with uniform consensus:
\begin{enumerate}
\item
A 2-round lower bound for the time for all nonfaulty processes to
decide, in executions with no failures. \\
This matches the bound for Mostefaoui, Raynal's algorithm,
when executed in the synchronous model (implement your own reliable
failure detector by observing when messages don't arrive).
%\item
%2.  Ordinary consensus:  A $f+1$-round lower bound for the time for
%every nonfaulty process to decide, in executions with at most $f$
%failures. \\
%\\
%This follows from the standard bound:  For any fixed $f$, we can
%restrict attention to the set of runs with at most $f$ failures.
%These all occur in a similar system in which $f$ is an actual upper
%bound on failures.  The standard lower bound yields $f+1$ for this
%case, so that carries over to the current, more general setting. \\
%\\
%However, they give a new proof. \\
%
\item
An $f+2$-round lower bound for the time for every nonfaulty process to
decide, in executions with at most $f$ failures. \\
Charron-Bost and Schiper have an earlier version of this, slightly less
general (assumes $f \geq 1$). \\
\item
Unsolvable in the ``mobile failures'' model, which simply allows
messages from at most one process to be lost at
each round.  (Can change from round to round.). \\
Re-proves a result previously proved by Santoro, Widmayer and
re-proved earlier by Moses, Rajsbaum. 
\end{enumerate}
Compare:  Dolev, Reischuk, Strong 90 algorithm:  min(f+2,t+1) rounds
(?). \\
That is, if $f \leq t-2$ then it decides within $f+2$ rounds (else
within the max, $t+1$). \\
\\
KR emphasize their technique.  It's pretty simple, like that of
Aguilera, Toueg's.  But it doesn't use bivalence.  In fact, they claim
that a bivalency argument can't be used to get the $f+2$-round lower
bound.  Not exactly clear how one would state/prove that precisely. 

\subsection{Layering, similarity, and connectivity}

\noindent
Everything in this section applies to general algorithms in the
round-based synchronous model---not just consensus. \\
\\
I don't see a careful definition of ``layering'' here.  I'll try to
fill in. \\
Restrict attention to executions in which at most one process fails
for the first time at each round. \\
Moreover, when it fails, it fails to send to some ``prefix'' of the
set of processes, $[k] = p_1,p_2,\ldots,p_k$ for some $k$. \\
\\
An execution is determined by an assignment $I$ of initial states to 
all the processes plus a sequence $E$ of ``environment actions'' of
the form $(i,k)$, where $p_i$ is a process that fails to send to all
of the processes in $[k]$.  \\
If $k = 0$, then $p_i$ sends to everyone.  In this case, no failure
occurs at this step. \\
\\
It's clear how such an initialization and sequence of actions entirely
determines an execution of the standard process-stopping-failure model.  \\
The process designated for each round indicates a new failure, except
when $k=0$. \\
The convention is that a failed process doesn't send any messages at later
rounds either. \\
\\
However, they ``overload'' their notation a bit by using it also for
the ``mobile failure'' model.  \\
Here, the convention is different:  process failures are for only one
round. \\
By this convention, their notation determines an execution for the
mobile failures model.  \\
\\
Let's restrict attention to the stopping failure model, to avoid
having to go back and forth thinking about the two different cases. \\
Then we can check that the results apply in the mobile model. \\
\\
Define an environment sequence $E$ to be {\em well-formed\/} if it
includes at most $t$ process failures and the same process fails at
most once in the sequence. \\
\\
I don't understand exactly what a ``layer'' is.  I think their ideas
are better understood in terms of executions and extensions of those
executions, as usual. \\
\\
Recast the $L^k$ definitions slightly: \\
Let $\cal E$ be a set of well-formed finite environment sequences of
the same length. \\
Then $L^k(\cal E)$ is the set of well-formed environment sequences that are
$k$-step extensions of sequences in $\cal E$. \\
\\
Define a {\em run\/} to be a pair $(I,E)$:  initialization and
well-formed environment sequence. \\
\\
Similarity:  Two runs $(I,E)$ and $(I',E')$ of the same length are
{\em similar\/} provided that there are two processes, $p_i$ and
$p_j$, such that: \\
(a) $p_i$ and $p_j$ are both non-failed in both of the executions
generated by these runs, and \\
(b) These two executions look identical to all processes except
$p_j$. \\
Thus, if two runs are similar, at least one nonfaulty process
$p_i$ can't distinguish them. \\
\\
Q:  Why should we care if $p_j$ is faulty or not? \\
Maybe we could just say that these runs look the same to all but one
process, and some process is non-failed in both executions. ??? \\
\\
A set $\Phi$ of runs is said to be {\em connected\/} if every
pair is in the reflexive transitive closure of the similarity
relation. \\
\\
Lemma 4.1:  The set of ``initial'' runs, in which the envt-seqs have
length $0$, is connected.  \\
Proof:  Change the initial state of one process at a time. 

\subsection{The 2-round lower bound for failure-free executions}

\noindent
Now assume that the algorithm is solving uniform consensus. \\
\\
Coloring:  For any run $\rho = (I,E)$, we define $val(\rho)$ to be the
unique decision value that results in the extension of $\rho$ in which
no failures occur. \\
\\
Key Lemma, 4.2:  Let $\Phi$ be a connected set of runs.
Suppose $\rho$ and $\rho'$ are two runs in the set such that $val(\rho) \neq
val(\rho')$. \\
(Also assume there are ``enough'' failures left to use.) \\
Then there is a run $\sigma \in \Phi$ such that, in the failure-free
extension of $\sigma$, some non-failed process has not decided within
one additional round. \\
\\
This immediately implies the first main theorem:  the $2$-round lower
bound for uniform consensus when there are no failures.  \\
Uses the fact, in Lemma 4.1, that the set of initial states is
connected. \\
\\
Proof of Lemma 4.2:  WLOG, we can assume that $\rho$ and $\rho'$ are
actually similar, not just connected (by the usual
argument---the two assumed runs are connected by some chain, so we
have a difference at some point in the chain). \\
Then we know that $\rho$ and $\rho'$ look identical to everyone except
for a particular $p_j$.  And there are enough failures left over for
us to play around with. \\
Choose $p_m$ to be some other process that is nonfaulty in both
runs---in fact, the one with the largest index that isn't faulty in
both runs. \\
(We seem to be using the fact here that the same processes are faulty
in both.  That seems to follow...) \\
\\
Assume for the sake of contradiction that everyone decides within one
round in the failure-free extensions of $\rho$ and $\rho'$. \\
Assume wlog that $val(\rho) = 1$; thus, all processes in the
failure-free extension of $\rho$ decide $1$, within one more step. \\
And $val(\rho') = 0$. \\
\\
Consider the execution constructed by extending $\rho$ with $(j,m-1)$,
that is, $j$ (who hasn't failed yet) now fails and misses sending to
everyone up to and not including $m$. \\
To process $m$, this looks the same as the failure-free extension of
$\rho$, so $m$ must decide $1$ after one round. \\
Then everyone else must also decide $1$ in {\em any extension\/} of
this extended run, though because they know this is not a failure-free
extension (they can see missing messages), they are allowed to take
more rounds to do it.  \\
\\
Likewise, extend $\rho'$ with $(j,m-1)$, conclude that $m$ must decide
$0$ after one round, so everyone else decides $0$ in any extension of
that extended run. \\
\\
But, among the extensions, there are two---one extending 
$\rho \cdot (j,m-1)$ and one extending $\rho' \cdot (j,m-1)$---
in which $m$ fails immediately after the decision round and no one
else ever fails.  \\
No remaining nonfaulty process can tell the difference between these
two runs.  \\
But they are forced to decide differently in these runs. \\
Contradiction.

\subsection{The $f+2$-round lower bound for $f$-failure executions}

\noindent
The previous result is like the special-case result used in the
Aguilera-Toueg proof.  This paper also has an analog of the
earlier-rounds argument (bivalence preserved for one round when a
failure occurs) of AT. \\
Stated in terms of preserving similarity. \\
This result doesn't say anything about consensus. \\
\\
Lemma 4.4:  Let $\Phi$ be a connected set of states.  \
(Assume there are enough failures left to use.) \\
Then $L(\Phi)$ (the set of one-round extensions of $\Phi$) is also
connected. \\
\\
Proof: This isn't hard---add lots of extra intermediate ``links'' in
the chain, removing one message at a time.  LTTR. \\
\\
Let $\Phi_0$ be the set of initializations for the processes. \\
\\
Lemma 4.5, part 1: \\
$L^k(\Phi_0)$ (the set of $k$-round runs) is connected
for all $k$, $0 \leq k \leq t$. \\
\\
This follows because $\Phi_0$ is connected and the previous
lemma says this is preserved (since we have ``enough'' failures to
play with). \\
\\
Now consider uniform consensus again.  \\
Lemma 4.5, part 2: \\
For every $k \geq 0$, there exist $\rho$, $\rho'$ in $L^k(\Phi_0)$
such that $val(\rho) \neq val(\rho')$. \\
\\
Just take non-failing runs that start with all $0$ or all $1$. \\
\\
Finally, we can prove the $f+2$-round lower bound on uniform consensus
with $f$ actual failures (out of a possible $t$): \\
\\
Theorem 4.9: ...\\
Proof:  Fix $f$, $0 \leq f \leq t-2$. \\
Use Lemma 4.5 to obtain runs $\rho, \rho'$ in $L^f(\Phi_0)$ such that
$val(\rho) \neq val(\rho')$. \\
Also by Lemma 4.5, $L^f(\Phi_0)$ is connected. \\
Then we use Lemma 4.2 (noting that there are ``enough'' failures
left), and conclude that there is a run $\sigma \in L^f(\Phi_0)$ such
that in the failure-free extension of $\sigma$, some non-failed
process hasn't decided within one additional round.  \\
That is, it takes at least $f+2$ rounds to decide. 

\subsection{Mobile failures}

\noindent
They use the same results to show impossibility of ever deciding in
the (uniform or nonuniform---the difference isn't meaningful here)
consensus problem, in the mobile failures model. \\
\\
This uses Lemma 4.5 again, but with no upper bound $t$---so after
any finite number of rounds, the set of runs is
connected. \\
Since each such set has runs with distinct vals, a simple argument
shows there's another run with no decision. \\
Use Konig's lemma to show that there's an infinite branch in which
some nonfaulty process doesn't decide.

\subsection{Discussion}

\noindent
Results have topological flavor. \\
\\
The definitions of similarity, layers, etc., might be polished a bit. \\
\\
Q: What's the minimum time for consensus to terminate after
stabilization time, in synchronous models where: \\
(a)  For any amount of time processes may exhibit any number of
omission failures, and may stop and restart. \\
(b)  After stabilization time, no more processes stop or restart, and
no messages are lost. \\
Since the failure model is worse-behaved than the one considered by
Keidar and Rajsbaum, it might be that a larger bound applies. 

\section{Gafni}

\noindent
Round-by-Round Fault Detectors:  Unifying Synchrony and Asynchrony \\
PODC 98 \\
\\
Gafni sets up a simple, general, round-based modeling framework that
can be specialized to describe variants of many of the models used
elsewhere, including the usual synchronous models,
the usual asynchronous models, models with failure detectors;
message-passing and shared-memory models;  models with different kinds
of failures. \\
\\
Casting the different models as special cases of a common general
model allows him to compare different models, e.g., describe
transformations from one to another. \\
This allows him to carry over impossibility results from one model to
another, e.g., to show that an impossibility result for a consensus
problem in an asynchronous model implies a number-of-rounds lower
bound in a synchronous model. 

\subsection{Introduction} 

\noindent
Assume a fixed set of processes $S = \{ 1,\ldots,n \}$. \\
The general framework is a round-based model, with rounds numbered
$1,\ldots$. \\
In each round $r$, every process $i$ broadcasts a single message to
all processes, including itself.  \\
Then it receives messages from some subset $S(i,r)$ of the processes
(possibly not including itself), and gets an indication that another
subset $D(i,r)$ may be faulty. \\
He assumes that: 
\begin{enumerate}
\item
$S(i,r) \cup D(i,r) = S$, the set of all processes,
that is, at each round, each process either hears from each other
process or has some explicit indication that the other process might
be faulty. 
\item
$D(i,r) \neq S$, that is, a process can't hear that everyone has
failed.
\end{enumerate}
He doesn't rule out $S(i,r) \cap D(i,r)$ being nonempty. \\
He does allow $i \in D(i,r)$. \\
Given these two sets, process $i$ makes a state transition and goes on
to the next round. \\
\\
This modeling framework gets specialized to a particular ``computation
model'' by the use of a particular ``predicate'' $P$ defining the
allowable sets of functions $D(i,r)$. \\
It gets further specialized to a particular ``system'' by fixing the
processes' sets of states, initial states, and transition relation. \\
The same system admits different inputs, represented by different
initial process states. \\
Once all this is fixed, the system can execute.  The execution can be
nondeterministic, based on which allowable $D(*,*)$ sets the
``adversary'' contributes. \\
\\
Define a task $T$ to be a set of (input-vector, output-vector
pairs). \\
E.g., the uniform consensus task is the set of pairs $(V,W)$ where
$W(i) = W(j)$ for all $j$, and $W(i) \in \{ V(j)| j \in S \}$. \\
\\
A particular system ``solves'' a task $T$ provided that, if the
processes in the system are started with inputs represented by $V$,
any outputs they produce are portions of a vector $W$ such that $(V,W)
\in T$. \\
That expresses safety.  \\
Moreover, he says that ``eventually, after enough rounds, processes
commit to outputs''.  \\
\\
But this isn't completely clear.  It seems that we would want to say
that ``nonfaulty'' processes eventually decide. \\
But he hasn't actually modeled process failures! ??? \\
\\
I'll read on, see later if this omission causes confusion...

\subsection{Examples of RRFD models} 

\noindent
He shows how to view many previously-defined computational models
within his framework, by particular ``predicates'' that capture
appropriate restrictions on the $D$ functions. \\
\\
In each case, he makes some argument that his representation is
faithful to the intention of the original model.  \\
These arguments say that the two models ``implement'' each other:
for any particular system $A$ in one, there is a system $B$ expressed
in the other, such that $B$ ``simulates'' $A$. 
Warning:  He doesn't define what ``simulates'' means.  Surely
that notion would have to say something about process failures, which
aren't modeled here...\\
\\
Anyway, read on...\\
\\
Example 1:  Round-based synchronous systems in which processes
broadcast just one message at each round, and in which at most $f$
processes fail by send-omission. \\
\\
Express this by the following restrictions on the $D$ functions:
\begin{enumerate}
\item
$i \notin D(i,r)$, that is, no one is ever told that he might be
faulty.
\item
There is a single set $D$ with $|D| \leq f$, such that every $D(i,r)
\subseteq D$.
\end{enumerate}
This is essentially the usual formulation, so no special
``implementation'' argument has to be made. \\
\\
Example 2:  Round-based synchronous systems...in which at most $f$
processes fail by stopping. \\
\\
Express by using the restrictions of Example 1 plus one other
restriction:
\begin{enumerate}
\item
$D(i,r) \subseteq D(j,r')$ for all $i, j, r, r'$ with $r < r'$. 
\end{enumerate}
Again, this is essentially the usual model. \\
\\
Example 3:  Asynchronous system with at most $f$ stopping failures. \\
\\
Express simply by:
\begin{enumerate}
\item
For every $i$ and $r$, $|D(i,r)| \leq f$.
\end{enumerate}
That is, independently at each round, and for each recipient, at most
$f$ nodes are designated as possibly faulty. \\
\\
He makes an informal argument that this representation implements the
usual asynchronous model, and vice versa. \\
\\
(a) Asynch implements the RRFD version: \\
Assume $A$ is an algorithm in this RRDF model; construct $B$ in the
asynch model: \\
Use asynchronous rounds.  \\
In each round $r$, each process $i$ of $B$ waits to hear from all
but $f$ processes.  (It can do that in the asynch model because it
eventually hears from all but $f$.) \\
Then process $i$ of $B$ simulates the transition of $i$ in $A$ by
letting $D(i,r)$ be the set of processes that $i$ doesn't wait for at
round $r$.  \\
\\
It should be reasonably clear that $B$ ``simulates'' $A$ pretty
directly, though this isn't formally defined (stating it carefully
would require modeling the actual failure events). \\
\\
This simulation corresponds to a very common strategy in constructing
asynchronous algorithms:  Asynchronous rounds, wait for all but $f$. \\
Ignore any of the remaining messages if they arrive ``too late'', when
the recipient has moved on to the next round. \\
\\
(b) The new RRFD model implements the usual asynch model: \\
Now assume $A$ is an algorithm in the standard asynch model; construct
$B$ in the RRFD version: \\
Now the problem is that the RRFD model seems to give the adversary
more options than the asynch model does.  In the RRFD model, different
collections of $f$ processes might appear to have failed to different
other processes, and at different rounds.  In the asynchronous model,
the adversary is more restricted:  there are only a particular set of
$f$ processes that ever fail. 
The implementation result has to show that the apparent freedom of
this model isn't really any greater than what's allowed in the
standard asynch model. \\
\\
Gafni shows this by letting the processes of $B$ turn missed
messages into late arrivals.  \\
The sketch is extremely rough; I'll try to fill in.  \\
Assume $A$ uses bcast messages, with eventual delivery (no ordering
assumptions).  \\
And assume that every process in $A$ is always enabled to bcast
(perhaps dummy messages). \\
\\
Each process $i$ of $B$ starts by sending, at round $1$, a bcast
message that it is enabled to send in the initial state of $i$ in
$A$. \\
In the RRFD model, it receives $n-f$ messages; it then does local
state transitions corresponding to the $n-f$ receive events in the
asynch model, in any order.  (The order doesn't matter---the asynch
model allows any order.) \\
Then it sends its second round message---a bcast message that it is
enabled to send in its simulated $A$-state. \\
\\
Well, not quite.  \\
If we kept doing rounds like this, we might not be correctly
simulating a fair execution of the asynch algorithm, because we're not
capturing the liveness assumption that the channels eventually deliver
all messages from the real nonfaulty processes.  We are throwing out 
some messages.  \\
\\
To get around this problem, we use a trick, by running the processes
in what he calls ``full information mode''.  This means that each
process keeps track of all messages it has ever received from everyone
(disambiguated somehow), and sends this history on every message.  A
little more precisely, what it sends in full info mode is
a ``super-message'' containing its actual message plus a history of
all the actual messages it has ever received from anyone in the past.  \\
When a process receives a super-message, it unpacks all the messages
contained within, figures out which ones it hasn't previously seen,
and simulates their receipt.  \\
Thus, in one round, the process in $B$ simulates the receipt of, not
just the new messages sent by everyone, but also any old messages that
those processes knew about.  \\
This is simulating delayed arrival of those old messages, but this is
fine because the asynch model allows those messages to arrive at any
time! \\
(I'm assuming that the channels between processes in the asynch model
don't need to be FIFO; if they do, we can further constrain the order
in which we simulate delivery using sequence numbers.  LTTR.) \\
\\
Example 4:  Asynchronous shared memory model, at most $f$ stopping
failures, reliable memory.  The shared objects are single-writer
multi-reader r/w registers. \\
\\
He models this in his framework by using the restriction of the asynch
network model, given above. \\
This isn't enough:  the asynch shared memory model is stronger than
the asynch network model in that the objects don't fail. \\
So he has to assume something is reliable.  What he assumes (not
self-evident) is the extra condition:
\begin{enumerate}
\item
In each round $r$, there is some process $j$ such that $j \notin
D(i,r)$ for all $i$. \\
That is, at each round separately, some process is not suspected by
anyone.
\end{enumerate}
He gives some very sketchy arguments---too sketchy for me to work out
quickly---for why this is actually equivalent to the standard
shared-memory model. \\
\\
Example 5:  Atomic snapshot shared memory (where a process can read
everyone's local variable all at once, atomically), with at most $f$
process stopping failures. \\
\\
He equates this with the RRFD that uses the restrictions of the asynch
network model, plus: 
\begin{enumerate}
\item
$i \notin D(i,r)$
\item
For each round $r$ and each pair of processes $i$ and $j$, either
$D(i,r) \subseteq D(j,r)$ or vice versa. \\
\end{enumerate}
No equivalence argument is given. \\
\\
Example 6:  Asynchronous message-passing system with failure detector
$S$ (some nonfaulty process is never suspected) \\
\\
He views this as the RRFD satisfying (the conditions of the asynch
network model) plus:
\begin{enumerate}
\item
There is some process $j$ that does not appear in any $D(i,r)$ set. 
\end{enumerate}

\subsection{RRFDs and $k$-agreement}

\noindent
What can we do with these models?  And what does this all have to do
with consensus, which is today's main topic?  \\
\\
Viewing all these models in a uniform framework makes it somewhat
easier to relate different models.  Gafni relates synchronous and
asynchronous models in this way (next section), and uses this to get
corollaries about impossibility and lower bounds for consensus.  \\
In this section, he characterizes solvability of $k$-consensus in
terms of RRFDs. \\
\\
Basically, he defines a particular RRFD and claims that it captures
the ability of a model to solve the $k$-consensus problem.  The one he
uses is:
\begin{enumerate}
\item
For each round $r$, there are at most $k-1$ processes that appear in
some $D(i,r)$ but not in all $D(i,r)$.
\end{enumerate}
That is, $k-1$ bounds the ``uncertainty'' of the fault detector---the
amount of disagreement in each round. \\
\\
Theorem 3.1:  With this fault detector, can solve $k$-consensus.  \\
Proof:  Trivial.  Use one round.  Everyone sends around their initial
values, choose the value from the process with the smallest id. \\
\\
Then he claims that this yields Chaudhuri's result asserting
solvability of $k$-consensus in asynch shared memory systems with at
most $k-1$ stopping failures. \\
But it takes a bit of work:  It first requires showing that
shared memory with $k-1$ failures can implement snapshot memory with
$k-1$ failures.  That's a known result, from the Borowsky paper.  The
proof could presumably be redone in the setting of this paper.  But
this isn't exactly self-contained. \\
Once we have the snapshot memory, it's easier because the predicate
defining the snapshot memory immediately implies the predicate given
here for $k$-consensus. \\
\\
Theorem 3.3:  They also give a result---the statement is a little
loose---that says that this predicate isn't too strong to model
$k$-consensus.  \\
In particular, having a solution to the standard $k$-consensus
problem in some form and having SMRW asynch shared memory, it's
possible to compute $D(i,r)$ sets satisfying the condition above.  \\
The precise statement and proof aren't so clear to me,
esp. when he's talking about the standard models and when about the
RRFD framework. 

\subsection{Relating synchronous and asynchronous systems}

\noindent
He uses his framework to show that that the asynchronous shared memory
model with $1$ stopping failure implements a model in which the
first $f$ rounds satisfy the restrictions of a synchronous system with
$f$ omission or stopping failure. \\
\\
This implies that a consensus algorithm that works in the synchronous
model within $f$ rounds also runs in the asynchronous model with $1$
stopping failure.  But that's impossible, by FLP. \\
\\
The result is given in a more general form:  for $k$ stopping failures
in the asynch model, and $\lfloor f/k \rfloor$ rounds of the
synchronous system.  \\
That means that the unsolvability result for $k$-consensus in the
asynch model with $k$ failures (proved by three different groups of
researchers in 1993) implies the lower bound result of Chaudhuri,
Herlihy, Lynch, Tuttle, also 1993, for $k$-consensus. \\
(Both of these proofs are pretty hard, and use similar topological
techniques.) \\
\\
I'll leave the stopping failure results to you.  Let's just look at
the key theorem for send-omission: \\
\\
Theorem 4.1:  The asynchronous atomic-snapshot shared-memory model
with $k$ stopping failures (example 5) implements the first $\lfloor
f/k \rfloor$ rounds of the synchronous message-passing model with $f$
send-omission failures. \\
\\
(He seems to want a special notion of ``implements the first $r$
rounds of''.  Remains to be cleared up.  Maybe we can talk
alternatively about a model that satisfies the needed predicate for
$r$ rounds but not necessarily afterwards. ???) \\
\\
He uses snapshot, but that's not supposed to be significantly
different from using ordinary shared-memory, because of snapshot
algorithm results. \\
\\
Proof:  The snapshot memory's predicates imply that, at each round,
the total size of its $D$ sets is at most $k$. \\
(Follows from the predicates in example 5). \\
Over $f/k$ rounds, that gives a total of at most $f$ failures. \\
That yields the predicate for the RRFD for a synchronous system with
at most $f$ omission failures.  (Well, just for the first $f/k$
rounds.) \\
\\
This is very rough.  But the idea seems right, and interesting. \\
The cited paper by Herlihy, Rajsbaum, and Tuttle, called
``Unifying synchronous and asynchronous message-passing models'',
sounds related and should be looked at. 

\end{document}

\section{Data replication and data consistency:  Introduction}

\noindent
The problem of maintaining data objects in a distributed system is one
of the most fundamental problems of distributed computing, both
theory and practice. \\
\\
The algorithms that manage such objects are supposed to make the
objects look as much as possible like objects in a
centralized memory. \\
\\
However, because of issues of availability (the objects should be
accessible even if some of the processes or channels fail), or
performance (want copies of the objects close to the users), the
objects are typically not kept in a single centralized location. \\
Replicated in various places in the system. \\
\\
Then the question is how to manage the copies, in order to keep them
``consistent'' in some sense.  \\
By far the most desirable kind of consistency is ``atomicity'', AKA
linearizability. \\
But that can be expensive to achieve. \\
\\
Most of you have already seen various strategies for implementing
atomic objects in distributed systems. \\
E.g., in my book, I describe various standard strategies, e.g.:
\begin{itemize}
\item
(Fault-tolerant)  Quorum-based approaches for reads/writes.  Attiya,
Bar-Noy, Dolev and extensions to allow multiple writers.  Reads and
writes can be carried out concurrently, using two-phase ($4d$)
protocols that access read-quorums in the first phase and
write-quorums in the second.  \\
Even with lots of concurrency, the intersection properties of quorums
yield atomicity. \\
Tolerates stopping failure of under half of the processes.
\item
(Not fault-tolerant)  Using Lamport timestamps.  Works for arbitrary
updates (which modify the state as a function of the previous state,
but do not return any value).  Attach timestamp to an update, send to
everyone, order updates by timestamp, apply them when you know that
you won't get any more with smaller timestamp.  
Since this is not fault-tolerant, we can tell when this is by waiting
to receive a larger timestamp from everyone.
\end{itemize}

\noindent
In a couple of weeks, I'll talk about our new Rambo algorithm (again),
which also gives atomic read/write data, with reasonable efficiency,
and in rather dynamic settings, tolerating failures. \\
I don't have an efficient algorithm to yield atomicity for more
general data types, e.g., read-modify-write, in fault-prone settings.
\\
Note that, if failures continue, then this is impossible---RMW allows
solution of consensus, with 1 failure. \\
But it should be possible to get good algorithms for situations in
which stabilization occurs. \\
\\
Today, I won't talk just about atomicity, but also about sequential
consistency and other conditions somewhat weaker than atomicity. \\
\\
Sequential consistency is the same as atomicity except that events at
different nodes can be reordered. \\
Formally, it is defined by the existence of a total ordering of all
the operations such that, if an operation completes at any location
before another operation starts at the same node, then the first
operation must precede the second in the total order. \\
in a certain order occur in the same order in the total order. \\
%
%[[[This isn't exactly the definition---the sentence at the beginning
%of the paragraph, which is expressed in terms of events, expresses
%the standard definition.  I think they are equivalent.  Check.]]]
%
Atomicity adds the requirement that, if an operation finishes
anywhere before another one starts anywhere, then the first operation
must precede the second in the total order.
\begin{enumerate}
\item
Revisit Lamport and other solutions based on replicated state
machines, to see what consistency conditions they yield for the data.
\item
Go over the paper by Attiya and Welch, which gives some basic
upper and lower bounds for latency of operations on objects
(read/write and some other kinds of objects), considering both
sequential consistency and atomicity.
\item
Consider the systems paper by Ladin, Liskov, et al. briefly,
summarizing an algorithm of theirs at a very high level, and
describing the rather novel kind of consistency condition it guarantees.
\item
Go over the more theoretical paper by Fekete et al., which
is based closely on the Ladin et al. paper, but does things more
abstractly.
\end{enumerate}
There is still a lot more to be done on this topic (and it's important).

\section{The replicated state machine approach}

\noindent
Lamport originated the ``replicated state machine approach''
to building distributed systems, in his famous ``Time, Clocks,...''
paper. \\
The idea is simply that everyone should keep a copy of the object
(total replication) and receive the same sequence of operations to
apply to that object. \\
He doesn't distinguish reads, writes, etc.---treats all operations
the same. \\
\\
Of course this yields sequential consistency.  \\
But in fact, he gets atomicity:  \\
He uses an ordering based on logical timestamps, as described above. \\
When an operation returns, it has already acquired a timestamp, and
everyone has a local timestamp that is higher.  \\
Therefore, no operation that gets invoked afterwards, anywhere, can
acquire a smaller timestamp. \\
\\
That paper does not handle failures. \\
In order to accommodate failures, something must be done to guarantee
a consistent sequence everywhere, even in the presence of some faulty
processes. \\
\\
The much later Paxos multidecree algorithm is again designed to yield a
consistent sequence of operations everywhere, this time in the face of
a wide variety of (non-Byzantine) failures. \\
Sequential consistency follows easily from the consistent ordering. \\
He may also get atomicity, though the paper doesn't exactly say. \\
However, if an operation returns, it has already been assigned an
index in the total ordering. \\
And if another operation is later invoked, the gap-filling policy
MIGHT succeed in preventing the later operation from getting assigned
to an earlier index. \\
\\
Q:  Does this work to give atomicity? \\
\\
Q:  What if we build a replicated data service on top of an ``atomic
broadcast service'' that is guaranteed to output all the operations
everywhere, in a consistent order?  Just apply the operations as they
arrive and respond to a local operation after you've applied it. \\
That's surely sequentially consistent---is it also atomic? \\
A:  Seems so---a node must know the position of a local op in the
order before it can return a response.  Any operation that gets
invoked, anywhere, later, must be assigned a later position in the
sequence. \\
\\
The next paper examines the two notions of atomicity and s.c. more
closely, and compares the costs of achieving them. 

\section{Attiya, Welch}

\noindent
Theoretical upper and lower bound results about the cost (worst-case
response time) of achieving sequential consistency vs. atomicity. \\
No faults considered. \\
Considers three kinds of objects:  read/write objects, FIFO queues,
and stacks. \\
I'll focus on r/w. \\
\\
Results: 
\begin{enumerate}
\item
If processes have no clocks, or have clocks that are approximately
synchronized, then there is a provable difference in the costs of the
two kinds of consistency conditions (for all three of the variable
types).  \\
They get lower bounds for achieving atomicity, and lower (in a sense)
upper bounds for sequential consistency. 
\item
With perfectly synchronized clocks, there's no difference in cost.
\end{enumerate}
They draw a lot of connections to multiprocessor memory consistency,
and cache coherence protocols.

\subsection{Model}

\noindent
{\em Application processes\/} are external user processes.  They
invoke read, write, whatever, on particular objects. \\
\\
The operations are implemented by a collection of MCS (memory
consistency system) processes, communicating using a reliable
network. \\
\\
Each MCS process has a real-time local clock, running at the exact
rate of real time. \\
For the first set of results, the local clocks are not synchronized
with other clocks. \\
For the second set, they're all perfectly synchronized. \\
In either case, the process can use them to schedule an event after
exactly a designated time has elapsed after another event. \\
\\
Message delay known to be in interval $[d-u,d]$. \\
(As in earlier Lundelius-Lynch clock synchronization paper.) 

\subsection{Imperfect clocks}

\noindent
Actually, these are perfect duration timers, but unsynchronized. 

\subsubsection{Lower bounds for atomicity}

\noindent
Theorem 1:  In any implementation of a single atomic read/write
object for at least 3 processes: \\
1. Some read must take time at least $u/4$. \\
2. Some write must take time at least $u/2$. \\
\\
Proof of part 1: \\
WLOG assume the initial value of $X$ is $0$. \\
Assume for contradiction that all reads take time $< u/4$ (writes may
take as long as we want).\\
Construct an execution $\alpha$ in which process $1$ reads $X$,
obtaining the value $0$. \\
Then process $3$ begins a write of $1$ to $X$. \\
While the write is going on, processes $1$ and $2$ alternate reading
$X$, each happening to start right after the other one finishes. \\
Since the write eventually completes, eventually one of these reads
must return $1$.  \\
Therefore, there exist a consecutive pair of reads, say $R_1$ by
process $1$ and $R_2$ by process $2$, where $R_1$ returns $0$ and
$R_2$ returns $1$. \\
\\
This execution is under-specified.  Suppose in addition that the
message delays everywhere in $\alpha$ are all exactly the midpoint of
the allowed range, $d - u/2$. \\
\\
Now we can construct a new, bad execution $\alpha'$ by shifting the
events at process $2$ earlier by $u/2$. \\
Recall this shifting from the Lundelius, Lynch clock synch lower bound
result.  \\
It means to move all events (at the indicated process) earlier in real
time, while adjusting the local clock of the shifted process so that
everything looks the same to it.  \\
The fact that the message delay is as indicated means that the shifting
produces allowable message delays. \\
\\
But in $\alpha'$, the read of $1$ by process $2$ completely precedes
(we can resolve the race conditions to say this is so)
the read of $0$ by process $1$.  \\
Violates atomicity. \\
\\
Proof of Part 2: \\
Similar.  This time construct $\alpha$ in which process $1$ writes
$1$ at time $0$, then after this finishes, process $2$ writes $2$ at
time $u/2$, then after this finishes, process $3$ reads, getting the
value $2$. \\
Also, suppose that message delays are $d$ from $1$ to $2$, $d-u$ from
$2$ to $1$, and $d - u/2$ between every other pair of processes. \\
\\
Now construct $\alpha'$ by shifting process $1$'s events later by time
$u/2$ and process $2$'s events earlier by $u/2$.  \\
This keeps the message delays in the allowed bounds. \\
But now, the write of $2$ precedes the write of $1$, which means that
process $3$ is supposed to get the value $2$, not $1$. 

\subsubsection{Upper bounds for sequential consistency}

\noindent
They describe two algorithms, one with fast reads and one with fast
writes.  In each case the fast operation is immediate and the slower
operation is bounded by an upper bound $h$ on the message delivery
latency in a separate atomic broadcast service. \\
\\
Atomic broadcast:  Their version of this has simple requirements,
since they do not deal with failures:  Has ABsend inputs, ABreceive
outputs.  Guarantees that every message sent is received at every
process, all messages are delivered in the same order at all
processes, and two messages sent by the same process are deliveredin
the same order they were sent. \\
\\
They describe (in the appendix), a fast implementation of atomic
bcast, in which the latency bound $h$ is only $2d$. \\
It's simple, looks like Lamport's timestamp-based totally-ordered
bcast. \\
Keep a vector of the latest local timestamps (counters) known for
everyone. \\
Tag messages with local timestamp (counter), send to all, accumulate
timestamped messages in a buffer.  \\
When you learn that everyone has reached a certain timestamp, you can
deliver all the smaller-timestamped messages, in order. \\
\\
Algorithm for fast reads:  \\
Everyone keeps a local copy of every object.  \\
A read returns the value of the local copy immediately. \\
When a write comes in to $i$, $i$ broadcasts it using ABsend, but
doesn't yet generate an ack to its client. \\
When a node receives a write operation using ABreceive, it writes to
its local copy. \\
In addition, if the write was originally generated by this process, it
then acks its client. \\
\\
The algorithm appears to work for read/update as well. \\
I believe it's the same as the algorithm used in the Orca system. \\
Seems reasonably practical. \\
\\
Yields sequential consistency:  the total order constructed is
consistent with the order of operations at any individual location.\\
The possibility of out-of-order reads happening at different locations
shows that it doesn't yield atomicity, however. \\
\\
Time $0$ for read, $2d$ for writes. \\
\\
Algorithm for fast writes: \\
A write results in an ABsend as before, but instead of waiting to
receive its own write back from the ABCast service, the process acks
to the client immediately. \\
It keeps track, however, of the number of writes it has acked but not
gotten back from the ABCast, using a counter $num$. \\
\\
A read results in waiting for the $num$ counter to be $0$, then
reading the local copy and returning the value to the client. \\
Thus, the process waits for all the writes that it originated locally
to pass through the ABCast service, before performing the read. \\
\\
A ``pipelined writes'' strategy, in the terminology of multiprocessor
memories. \\
Again, yields sequential consistency.  \\
Now time $0$ for writes, $2d$ for reads. \\
\\
The rest of this section of the paper deals with queues and stacks.
The results seem similar to the ones for reads/writes.

\subsection{Perfect clocks}

\noindent
Here, it turns out that there isn't a distinction.  So their results
take the form of upper and lower bounds results that apply to both
conditions.

\subsubsection{Lower bound}

\noindent
For sequential consistency, and hence for linearizability. \\
\\
Theorem 2:  (Lipton, Sandberg) In any implementation of a sequentially
consistent read/write memory, the sum of the worst-case latencies of
read and write operations is at least $d$. \\
\\
Proof:  By contradiction, as before.  Consider two objects, $X$ and $Y$.  \\
Construct execution $\alpha_1$:  process $1$ writes $1$ to $X$ then,
immediately after return, reads $Y = 0$.  \\
All messages take time exactly $d$. \\
Thus, by assumption that the sum of the latencies is $< d$, all this
is finished before any messages are delivered. \\
\\
Symmetrically, construct $\alpha_2$:  process $2$ writes $1$ to $Y$
then reads $X = 0$. \\
Again, all this is finished before any messages are delivered. \\
\\
Then we can ``paste'' these two executions into one.  \\
The resulting execution isn't sequentially consistent.  \\
To see this, suppose we have some sequence of the four operations
that's consistent with what is observed at each node. \\
In this sequence, one of the two writes must be ordered before the
other, say $W(X,1)_1$ comes before $W(Y,1)_2$. \\
But then $R(X,0)_2$ must come after the $W(Y,1)_2$, since the order is
consistent with the local order of operations at process $2$. \\
But this means that $R(X,0)_2$ must come after the $W(X,1)_1$, which
means that the response isn't correct. 

\subsubsection{Upper bounds}

\noindent
They give two algorithms that give atomicity.  Both have $d$ as the
sum of the worst-case latencies for reads and writes. \\
One puts the burden entirely on the reads, the other on the writes.
\\
Algorithm for fast reads: \\
Similar to the approximate-clocks case, but don't use ABCast. \\
Each process keeps a copy of all objects. \\
A read returns the value of the local copy immediately. \\
When a write request arrives at $i$, process $i$ sends it to everyone else,
waits time $d$, then performs the write and acks. \\
When anyone receives a write message, just perform the write (break
ties with process identifiers). \\
\\
Gives sequential consistency based on times of the invocations,
breaking ties with process ids. \\
Reads take time $0$, writes time $d$. \\
\\
Algorithm for fast writes: \\
Each process keeps a copy of all objects. \\
A write request triggers messages to all processes (including oneself)
and an immediate ack. \\
When a write message arrives, change the value in local memory.
When a read request occurs, wait for time $d$ and read the value then
in local memory. \\
\\
All these algorithms seem to work the same for read/update, a richer
object type than read/write. \\
%
%[[[Q:  Where is this in the object hierarchy?]]]
%
Later results by others (Mavronicolas, Roth) have shown that these
algorithms can be modified to yield any combination of upper bounds
$r$ and $w$ for reads and writes, such that $r+w = d$. \\
They also produce algorithms to show that the lower bounds (based on $u$)
given for the inexact clocks are tight (are these exact bounds, or
just $\Omega(u)$ bounds)? 

\subsection{Discussion}

\noindent
The lower bounds are cute, don't seem surprising. \\
The algorithms have some interesting ideas: \\
Using ABCast for writes only to get sequential consistency rather than
atomicity.  (Lamport produced total order for all operations.) \\
Pipelining writes to get fast response to writes at a cost of delaying
reads. \\
\\
The subtle distinctions between atomicity and sequential consistency
are good to know about---though the bounds here aren't high enough to
demonstate that this distinction matters in terms of practical
performance. (The lower bounds for atomicity aren't very high.) \\
\\
I'm not convinced that atomicity is impractical. \\
Or that, if it has to be weakened for practical considerations, that
sequential consistency is exactly the right notion.

\section{Ladin, Liskov, Shrira, Ghemawat}

\noindent
Now we switch to a different attempt to weaken atomicity.  \\
It defines a new, weaker consistency condition, and an algorithm that
achieves it.  \\
But (warning) this is not an algorithms paper, but a systems paper. \\
It does not try to present the specifications and algorithms
precisely, nor to generalize and abstract, and decompose as much as
possible.  \\
Rather, it describes a particular implementation, with many kinds of
theoretical and engineering issues mixed together. \\
We can try to pick them apart.  The paper by Fekete et al. is a decent
attempt to do this, though that doesn't seem so easy to read.  \\
\\
The new specification for memory consistency is given on p. 364, where
the API is given, and p. 365, Figure 1, where some of the correctness
conditions to be satisfied by the inputs and outputs are presented. \\
\\
The inputs are invocations and the outputs are responses, as usual. \\
The kinds of operations considered are ``update'' and ``query''. \\
An update, as before, can modify the state as a function of the
previous state, but does not return a response. \\
A query does not modify the state, but returns a function of the state
(essentially a read). \\
\\
Notice the extra stuff in the API:  prev, uid, newl. \\
Every update is supposed to get assigned a uid by the system, which is
returned to the user in the response to the update. \\
When a user then submits any operation (update or query), he is free
to include a ``label'', which represents a set of uids. \\
This ``prev set'' represents the set of updates that are required to
``precede'' the operation, that is, the new operation is supposed to
be ordered after all these updates in any ordering created by the
system. \\
\\
The query returns another label---the set of uids that the query
actually used in determining its response.  The query invocatio
specified a prev set of updates that the user said it had to depend
on.  This response set must include those, but may include more.  This
response is simply informing the user about what was used. \\
\\
The service guarantees, in Figure 1, say: \\
1.  The service can use only updates that were actually requested by
the users. \\
2.  The set of updates returned by a query (newl) includes the
set of updates requested by the user (prev). \\
3.  The set of updates returned by a query (newl) must be closed under
the ``dependency'' relation: update $u$ depends on update $v$ proviced
that $v$ is in $u$'s prev set. \\
4.  The value returned by a query is consistent with the newl set of
updates, applied in some order that is consistent with the dependency
order. \\
(It seems that this order need not be unique.  In fact, from what is
said here, the resulting value need not even be unique.) \\
\\
They make some assumptions about the client's use of the labels: \\
They assume that each client maintains a local label (set of
updates), which is what it uses for its prev set for each of its
operations. \\
Then when a response comes back, it takes the union of the response
set with its current label set, to get the new label. \\
\\
This specification seems to maintain a kind of causality. \\
\\
It's not clear to me exactly what the consistency guarantees
mean---how can they be used, how strong are they, what properties do
they have w.r.t. composition, etc. \\
The spec is a little messy-looking (esp. w.r.t. its division of
responsibility between system and user), which makes it harder to
tell.  Is it expressed in the simplest way? \\
\\
Anyway, they include two special additions to the basic service:  two
special kinds of updates, which they call ``forced'' and
``immediate''. \\
These have stronger consistency requirements:  \\
Forced updates are said to be ``totally ordered with respect to each
other''. \\
Immediate updates are ordered with respect to all other updates. \\
In terms of the spec, who is responsible for ensuring this? \\
Basically, they seem to be strengthening their dependency order among
updates to include the extra requirements, then using the stronger
dependency relation in their closure condition (3). \\
\\
Skip this part for now, just focus on the ``causal'' spec and its
algorithm. \\
\\
The main algorithm is described on p. 365-371. \\
But it is described in terms of vector timestamps---a particular way
that the sets of updates are represented. \\
That is, each update is assigned a vector timestamp as a uid.  \\
Each label that is attached to an invocation or response is again a
vector timestamp; the set of updates that it represents is (as we have
seen before) the set of updates whose vector timestamps are $\leq$ the
given vector. \\
The resulting algorithm description is quite complicated, and mixes
the timestamp handling with more fundamental issues involving update
dependencies. \\
\\
However, I think it's useful to describe the algorithm first in terms of
sets of updates, then think later about how the actual vector timestamps
are used to represent those sets (a lower level of refinement, an
optimization). \\
\\
For this, I switch to the other paper, by Fekete et al.

\section{Fekete et al.}

\subsection{Overview}

\noindent
This is an attempt to represent the essence of the algorithm in the
other paper in a more abstract, more general style. \\
Also, a student, Oleg Cheiner, implemented this version of the algorithm,
and implemented some small demo applications (a name service) on top
of it. \\
\\
This paper first points out a subtle issue with the LLSG paper: \\
They seem to rely on the clients to guarantee:  for any pair of
non-commutative updates, one must be specified as depending on the
other. \\
Without this, their algorithm may assign incomparable timestamps to
them, which may mean that, forever after the point where both of these
updates are known to everyone, no one will know what order to apply
them in.  \\
So this order will be forever undetermined, which means that forever,
clients may decide arbitrarily what order to use.  \\
This can lead to infinite inconsistencies. \\
\\
Discussion with the authors suggested that they didn't mean for this
to happen---they really did intend that eventually, a total ordering
of updates should be produced. \\
\\
That led us to produce a new version of the spec, that says that
eventually such a total ordering should be produced. \\
After presenting our abstract algorithm, we prove its correctness and
analyze its performance.  \\
It's all very carefully done, but I think the ideas are sometimes hard
to extract from the formalism. \\
\\
Most of our paper deals with general operations, not even divided into
reads and updates.  The optimization to handle reads specially is
described later. 

\subsection{Problem specification: ESDS}

\noindent
We consider operations that are ``ordinary'' and others that are
``strict''.  The ``strict'' ones have stronger consistency
requirements. \\
\\
Our clients (p. 8) are allowed to request any operation at any time,
as long as the prev set describes some previously-requested
operations. \\
We didn't include the assumption that the client must submit, in its
prev set, everything it has ever heard about. \\
That's not needed to prove that the algorithm satisfies the spec. \\
\\
Our main spec appears on p. 10. \\
\\
Conceptually, the service is building up a global partial order of
operations, $po$, which gravitates eventually to a total order. \\
\\
The spec is expressed in terms of some abstract internal
``milestones''. \\
An operation that is invoked first gets put into a buffer $wait$, and
its response comes from another buffer $rept$. \\
In between, the milestones are:  the operation gets ``entered'' into
$po$; at some point it may get ``stabilized'', when its prefix in the
eventual total order is determined; a response value gets
``calculated''. \\
For an ordinary operation, the response can be calculated anytime
after the operation is entered into the partial order. \\
For a strict operation, this can occur only after the operation has
been stabilized. 

\subsection{Our algorithm}

\noindent
p. 15-19. \\
It's based on LLSG, but it's simpler because it's more abstract.  \\
It's written in terms of a partial order, of sets of updates, etc. \\
\\
Our version, like LLSG, is written assuming separate ``front end''
automata and ``replica'' automata.  The front end automata simply
forward requests to replicas and collect return values, however.  The
interesting work is done by the replicas. \\
\\
The replica code is on p. 18. \\
Information is gossiped in the background, vis send and receive
actions. \\
\\
A replica keeps track ($rcvd$) of the set of operations it has learned
about. \\
At some point, it ``does'' each such operation, which means that it
assigns it a locally-generated ``label'' from a totally-ordered
set. \\
Furthermore, it keeps two vectors, $done$ and $stable$. \\
The $done$ vector records its knowledge of the set of operations that
it knows have been done at each node. \\
The $stable$ vector contains second-order information:  
An operation is said to be ``stable'' if it has been done everywhere. \\
Process $i$ puts an operation into its own stable set (in
$stable[i]_i$) when it is in the intersection of all its $done[j]_i$
sets. \\
Process $i$ puts an operation into $stable[j]_i$ when it learns, via
gossip, that the operation is in $stable[j]_j$. \\
\\
The local ordering on the operations is captured by the derived
variable $lc$.  \\
This is based completely on the labels for the operations. \\
An interesting complication:  the same operation can be assigned a
label at different locations, independently.  \\
The node keeps updating its label to be the minimum one that it has
heard for the update. \\
In this way, the ordering can change over time. \\
\\
When can the replica send a response? \\
If the operation is ordinary, this can be anytime after the operation
is {\em done locally}. \\
The value returned is a value that could be produced by ordering the
things in $done[i]_i$ according to the ordering of labels expressed by
$lc$.  (This looks unique to me, though it can change over time.) \\
\\
If the operation is strict, the process waits for the operation to
become {\em stable everywhere}---this implies that the ordering of the
prefix, and the value, will never change in the future.

\subsection{Discussion}

\noindent
Is this really doing the same thing as the LLSG algorithm? \\
\\
Is the specification, or some variant of it that expresses causal
consistency, a useful weakening of atomicity? \\
\\
Evidence that something like this is useful is the collection of
applications/implementations that have been built on top, but I don't
think this is well understood from a theoretical viewpoint.
\end{document}






