Massachusetts Institute of Technology Lecture 12 6.895: Advanced Distributed Algorithms October 17, 2002 Professor Nancy Lynch

More Consensus

Bring: Notes for scribe. Chalk

Today’s reading: Aguilera, Toueg. A simple bivalency proof that t-resilient consensus requires t+1 rounds. *** Keidar, Rajsbaum. On the cost of fault-tolerant consensus when there are no faults—a tutorial. *** Gafni, Round-by-round fault detectors: Unifying synchrony and asynchrony Datta, Guerraoui. The inherent price of indulgence (skim)

Next: Data replication and data consistency: Schneider *** Attiya, Welch *** Ladin, Liskov, Shrira, Ghemawat Fekete, Gupta, Luchangco, Lynch, Shvartsman Frolund, Guerraoui

1 Introduction

Today we have several results dealing with optimal consensus algorithms, with corresponding lower bounds and impossibility results. Some new results, and some new ways of obtaining old results. Especially interesting are some newly-observed connections between synchronous and asynchronous models.

2 Aguilera, Toueg

2.1 Overview

A really neat 4-page paper! It gives a very simple proof for the known lower bound of f + 1 rounds for f stopping failures in the (round-based) synchronous model, by using the technique of FLP for impossibility in the asynchronous case.

Historically, the lower bound on rounds was proved first. It was proved by constructing a “chain” of executions, where consecutive exections in the chain look the same to some nonfaulty process. The chain can be constructed to span from a failure-free execution where everyone starts with 0 to a failure-free execution where everyone starts with 1. Agreement and validity lead to a contradiction.

The first such proof was by Lynch, Fischer, for the Byzantine case. Around 1979 or 1980. A more complicated chain construction evolved, not attributed to anyone in particular. Dolev, Strong? Merritt? A version in my book. The construction is a little complicated.

The impossibility for consensus in the asynchronous model was proved a little later—the result was actually obtained in 1982. It introduced a new method based on “bivalence” and “univalence”. Until recently, no one seems to have tried out the new method on the former problem. It works much better than the old method.

The new proof seems to have been found not just by AT, but also by Moses and Rajsbaum. A similar argument was also used by Bar-Joseph (a student here, before he was here) and Ben-Or, but they applied it to the setting of randomized algorithms.

2.2 Model and problem

Synchronous round model. Known set of processes. In each round, each process sends messages to all other processes, receives messages, changes state.

Stopping failures. When a process stops, it may send any subset of its messages for the round in which it fails. Furthermore, at most one process fails per round. (Makes the impossibility result stronger.)

Consensus: Agreement, validity (decision value must be the initial value of some process), termination. The uniformity of agreement isn’t needed.

2.3 The proof

Assumes that n > f + 1.

By contradiction. Assume there is such an algorithm A, tolerating f failures and always terminating in exactly f rounds.

Restrict attention to executions with at most f failures. Define a execution that completes some finite number of rounds to be 0-valent (1-valent) if the only decision that can follow from extensions of the execution is 0 (1).

Univalent: 0-valent or 1-valent. Bivalent: Both decisions are possible.

Lemma 1: There is some initial state (0-round execution) that is bivalent.

Proof: As in FLP. Assume all are univalent, note that C^{0}, where everyone starts with 0, is 0valent and C^{1} is 1-valent. Then there must be two initial configurations with Hamming distance 1 (differ in state of one process) such that one is 0-valent and one is 1-valent. But this can’t be, because if that process fails, no one else can distinguish the two configs.

Lemma 2: There is a bivalent k-round execution for every k, 0 ≤ k ≤ f − 1.

Proof: By induction. The base, k = 0, follows from lemma 1. Inductive step: Assume true for k: we have a bivalent k-round bivalent execution r. Show for k + 1, where 0 ≤ k ≤ f − 2. Assume not—then every one-round extension of r is univalent.

Let r^{∗} be the one-round extension of r in which no new failures occur in round k + 1. This is univalent by assumption, wlog 1-valent. Since r is bivalent, there must be another 1-round extension of r, call it r^{0}, that is 0-valent. In r^{0}, some single process p fails in round k + 1. Let q_{1}, …, q_{m} be the processes to which p fails to send a message in round k + 1.

Now define a little chain of runs r^{1}, r^{2}, …, r^{m}. Each r^{i} is the same as r^{0} except that p sends a message to q_{1}, …, q_{i} in addition to the others that it sends to in r^{0}. That is, this chain represents adding in one message from p at a time. In all these executions, p still fails at round k + 1; however, in the case of r^{m}, it fails (think of a special “fail” event occurring) after sending all its messages.

Each r^{i}, 0 ≤ i ≤ m, is univalent, by assumption. Since r^{0} is assumed to be 0-valent, there are two possibilities:

1. At least one is 1-valent. Then there must some j such that r^{j−1} is 0-valent and r^{j} is 1-valent. But these two differ after round k + 1 only in the state of process q_{j}. This can’t be possible, because we can extend both executions by simply failing q_{j} at the beginning of round k + 2 (recall we have k + 2 ≤ f), and no one who is left could tell the difference.

2. All of them are 0-valent. Then compare r^{m}, in which p sends all its messages at round k + 1 and then fails, with r^{∗}, in which p sends all its messages at round k + 1 and does not fail. r^{m} is 0-valent and r^{∗} is 1-valent. But we can fail p at the beginning of round k + 1 in r^{∗}, and then no one else can tell the difference, again a contradiction.

So, we can keep the execution bivalent through f −1 rounds, while expending at most f −1 failures.

Now, what about the final round? The argument is essentially the same as the inductive argument. But instead of arguing that there is a bivalent f-round execution (which doesn’t exactly make sense, since we’ve required that everyone decide by the end of f rounds), we argue that there is an f-round execution in which two nonfaulty processes decide differently:

Lemma 3: There is an f-round execution in which two nonfaulty processes decide differently (which is a contradiction to the problem definition).

Proof: Use Lemma 2 to get a bivalent f − 1-round execution r (with at most f − 1 failures). We know that in every one-round extension of r, everyone who hasn’t failed decides (and agrees).

Let r^{∗} be the one-round extension of r in which no new failures occur in round f. Everyone decides in this extension. If two decide differently we are done, so assume they all decide the same. Wlog they decide 1. Since r is bivalent, there must be another 1-round extension of r, call it r^{0}, in which some (nonfaulty) process q decides 0.

In r^{0}, some single process p fails in round f. And p must fail to sends a message to q in round f (otherwise the execution would look to q just like r^{∗} and it would decide 1).

Now define a very small chain: an “intermediate” execution r^{0,1} that is identical to r^{0} but in which p sends its message to q. Then r^{0,1} looks to q like r^{∗}, so it decides 1. But r^{0,1} looks to every other nonfaulty process like r^{0}, so they decide 1. Thus, we have disagreement, as needed.

2.4 Discussion

The final round argument is really like the earlier round arguments. There should be an even shorter proof that treats this as a special case.

Note the slight trickiness that arises in these proofs: Every failure is apportioned to some round. There is an ambiguity when someone fails “between” rounds—can be assigned to either round. Has to be handled carefully in the proofs.

The key seems to be what happens in a round that happens right after a bivalent execution. If a single failure can occur at the next round, then the execution can be forced to remain bivalent. And if the decision must be made after that additional round, then disagreement is possible.

What can we say about the situation where the maximum number of failures is some value f, but the actual number that occur in some particular execution may be smaller?

Note that this proof assumed right at the beginning that everyone decides right at the end. We could generalize this to allow processes to decide earlier if possible. There are “early stopping” algorithms in the literature that terminate early in better cases, where fewer failures occur.

This lower bound doesn’t show that more than f rounds are needed in every execution—just that this is necessary in the worst case. The proof assumes we are free to expend the maximum number of failures f.

The next paper considers what is possible for early stopping algorithms.

3 Keidar, Rajsbaum

3.1 Overview

On the cost of fault-tolerant consensus when there are no faults—a tutorial. This 20-pager overviews many results about time bounds on consensus, but it also contains some new results, and new proofs of old results. I’ll just present the new results/proofs. They appear in Section 4.

Their motivation starts with partially synchronous models (models with timing assumptions, which may hold from some point on), and asynchronous models with unreliable failure detectors (diamond-S). But the actual technical results they get are for the round-based synchronous model. They then use those to get results about the other models as simple corollaries.

So, I’ll focus on the results about the synchronous model. Process stopping failures. Assume a worst-case bound of t on the number of failures, but they also consider the number f that occur in particular executions.

The results I will describe deal with uniform consensus:

1. A 2-round lower bound for the time for all nonfaulty processes to decide, in executions with no failures.

This matches what can be obtained from Mostefaoui, Raynal’s algorithm, when executed in the synchronous model (implement your own reliable failure detector by observing when messages don’t arrive).

2.

3. An f + 2-round lower bound for the time for every nonfaulty process to decide, in executions with at most f failures.

Charron-Bost and Schiper have an earlier version of this, slightly less general (assumes f ≥ 1).

4. Unsolvable in the “mobile failures” model, which simply allows messages from at most one process to be lost at each round. (Can change from round to round.). This is a re-proving of a result previously proved by Santoro, Widmayer and re-proved earlier by Moses, Rajsbaum.

Compare: Dolev, Reischuk, Strong 90 algorithm: min(f+2,t+1) rounds (?). That is, if f ≤ t − 2 then it decides within f + 2 rounds (else within the max, t + 1).

KR emphasize their technique. It’s pretty simple, like that of Aguilera, Toueg’s. But it doesn’t use bivalence. In fact, they claim that a bivalency argument can’t be used to get the f + 2-round lower bound. Not exactly clear how one would state/prove that precisely.

3.2 Layering, similarity, and connectivity

Everything in this section applies to general algorithms in the round-based synchronous model— not just consensus.

I don’t see a careful definition of “layering” here. I’ll try to fill in as needed. Restrict attention to executions in which at most one process fails for the first time at each round. Moreover, when it fails, it fails to send to some “prefix” of the set of processes, [k] = p_{1}, p_{2}, …, p_{k} for some k.

An execution is determined by an assignment I of initial states to all the processes plus a sequence E of “environment actions” of the form (i, k), where p_{i} is a process that fails to send to all of the processes in [k]. If k = 0, then p_{i} sends to everyone. In this case, no failure occurs at this step.

It’s clear how such an initialization and sequence of actions entirely determines an execution of the standard process-stopping-failure model. The process designated for each round indicates a new failure, except in the special case where k = 0. The convention is that a failed process doesn’t send any messages at later rounds either.

However, they “overload” their notation a bit by using it also for the “mobile failure” model. Here, the convention is different: process failures are for only one round. By this convention, their notation determines an execution for the mobile failures model.

Let’s restrict attention to the stopping failure model, to avoid having to go back and forth thinking about the two different cases. Then we can check that the results apply in the mobile model.

Define an environment sequence E to be well-formed if it includes at most t process failures and the same process fails at most once in the sequence.

I don’t understand exactly what a “layer” is. I think their ideas are better understood in terms of executions and extensions of those executions, as usual.

Recast the L^{k} definitions slightly: Let E be a set of well-formed finite environment sequences. Then L^{k}(E) is the set of well-formed environment sequences that are k-step extensions of sequences in E.

Define a run to be a pair (I, E): initialization and well-formed environment sequence.

Similarity: Two runs (I, E) and (I′, E′) of the same length are similar provided that there are two processes, p_{i} and p_{j}, such that:

(a) p_{i} and p_{j} are both non-failed in both of the executions generated by these runs, and

(b) These two executions look identical to all processes except p_{j}.

Q: Why would we care if p_{j} is faulty or not? Could we just say: These two executions look identical to all but one process, and Some process is non-failed in both executions. ???

Thus, if two runs are similar, at least one nonfaulty process p_{i} can’t distinguish them.

A set ρ of runs is said to be similarity-connected if every pair is in the reflexive transitive closure of the similarity relation.

Lemma 4.1: The set of “initial” runs, in which the envt-seqs have length 0 is similarity-connected.

Proof: Change the initial state of one process at a time.

3.3 The 2-round lower bound for failure-free executions

Now assume that the algorithm is solving uniform consensus.

Coloring: For any run ρ = (I, E), we define val(ρ) to be the unique decision value that results in the extension of ρ in which no failures occur.

Lemma 4.2: Let be a similarity-connected set of runs. Suppose ρ and ρ′ are two runs in the set such that val(ρ) ≠ val(ρ′). (Also assume there are “enough” failures left to use.) Then there is a run σ ∈ such that, in the (a?) failure-free extension of σ, some non-failed process has not decided within one additional round.

This implies the first main theorem: the 2-round lower bound for uniform consensus when there are no failures. Uses the fact, in Lemma 4.1, that the set of initial states is similarity-connected.

Proof of Lemma 4.2: WLOG, we can assume that ρ and ρ′ are actually similar, not just similarity connected (by the usual argument—the two assumed runs are connected by some chain, so we have a difference at some point in the chain). Then we know that ρ and ρ′ look identical to everyone except for a particular p_{j}. And there are enough failures left over for us to play around with. Choose p_{m} to be some other process that is nonfaulty in both runs—in fact, the one with the largest index that isn’t faulty in both runs. (We seem to be using the fact here that the same processes are faulty in both. That seems to follow...)

Assume for the sake of contradiction that everyone decides within one round in all failure-free extensions of ρ and ρ′. Assume wlog that val(ρ) = 1; thus, all processes in the failure-free extension of ρ decide 1, within one more step. And val(ρ′) = 0.

Consider the execution constructed by extending ρ with (j, m − 1), that is, j (who hasn’t failed yet) now fails and misses sending to everyone up to and not including m. To process m, this looks the same as the failure-free extension of ρ, so m must decide 1 after one round. Then everyone else must also decide 1 in any extension of this extended run, though because they know this is not a failure-free extension (they can see missing messages), they are allowed to take more rounds to do it.

Likewise, extend ρ′ with (j, m − 1), conclude that m must decide 0 after one round, so everyone else decides 0 in any extension of that extended run.

But, among the extensions, there are two—one extending ρ · (j, m − 1) and one extending ρ′ · (j, m − 1)— in which m fails immediately after the decision round and no one else ever fails. No remaning nonfaulty process can tell the difference between these two runs. But they are forced to decide differently in these runs. Contradiction.

3.4 The f + 2-round lower bound for f-failure executions

The previous result is like the special-case result used in the Aguilera-Toueg proof. This paper also has an analog of the earlier-rounds argument (bivalence preserved for one round when a failure occurs) of AT. Stated in terms of preserving similarity.

Lemma 4.4: Let be a similarity-connected set of states. (Assume there are enough failure left to use.) Then L() (the set of one-round extensions of ) is also similarity-connected.

Proof: This isn’t hard—add lots of extra intermediate “links” in the chain, removing one message at a time. LTTR.

Now consider uniform consensus again. Let _{0} be the set of initializations for the processes.

Lemma 4.5, part 1: L^{k}() (the set of k-round runs) is similarity connected for all k, 0 ≤ q ≤ t.

This follows because is similarity-connected and the previous lemma says this is preserved (since we have “enough” failures to play with).

Lemma 4.5, part 2: For every k ≥ 0, there exist ρ, ρ′ in L^{k}(Φ_{0}) such that val(ρ) ≠ val(ρ′).

Just take non-failing runs that start with all 0 or all 1.

Finally, we can prove the f +2-round lower bound on uniform consensus with f actual failures (out of a possible t):

Theorem 4.9: ... Proof: Fix f, 0 ≤ f ≤ t − 2. Use Lemma 4.5 to obtain runs ρ, ρ′ in L^{f}(_{0}) such that val(ρ) ≠ val(ρ′). Also by Lemma 4.5, L^{f}(_{0}) is similarity-connected. Then we use Lemma 4.2 (noting that there are “enough” failures left), and conclude that there is a run σ ∈ L^{f}(_{0}) such that in the failure-free extension of σ, some non-failed process hasn’t decided within one additional round. That is, it takes at least f + 2 rounds to decide.

3.5 Mobile failures

They use the same results to show impossibility of ever deciding in the (uniform or nonuniform— the difference isn’t meaningful here) consensus problem, in the mobile failures model.

This uses Lemma 4.5 again, but with no upper bound t—so after any finite number of rounds, the set of runs is similarity-connected. Since each such set has runs with distinct vals, a simple argument shows there’s another run with no decision. Use Konig’s lemma to show that there’s an infinite branch in which some nonfaulty process doesn’t decide.

3.6 Discussion

Results have nice topological flavor.

The definitions of layers, etc. are probably not the cleanest—could use some technical polishing.

Open questions? What’s the minimum time to terminate after stabilization time, in synchronous models where:

(a) For any amount of time processes may exhibit omission failures, any number, and may stop and restart. (b) After stabilization time, no more processes stop or restart, and no messages are lost.

What’s the exact (optimal) bound?

4 Gafni

5 Guerraoui