Massachusetts Institute of Technology Lecture 14 6.895: Advanced Distributed Algorithms October 24, 2002 Professor Nancy Lynch

Performing Work in Distributed Systems

Notes: Initial notes gathered and cobbled by Alex A. Shvartsman

Today’s reading:

• G. G. Malewicz, A. Russell, and A. A. Shvartsman. Distributed cooperation in the absence of communication. In Proceedings of the 14th International Conference on Distributed Computing, DISC, 2000. .ps

• C. Georgiou, A. Russell, and A.A. Shvartsman. The complexity of synchronous iterative do-all with crashes. Distributed algorithms (DISC 2001). .ps Next: TBD:

1 Introduction

Performing a set of tasks in a decentralized setting is a fundamental problem in distributed computing. This is often challenging because the set of processors available to the computation and their ability to communicate may dynamically change due to perturbations in the computation medium. An abstract statement of this problem, referred to as the Do-All problem — P faultprone processors perform N independent tasks — is one of the standard problems in the research on the complexity of fault-tolerant distributed computation [DHW-SIAM98,KS-DC92]. Variations of this problem have been studied in shared-memory models (Write-All^{1}) [KS97,KPRS-STOC91,MS-JA94], in message-passing models [DMY-PODC94,DHW-SIAM98,GMY-FOCS95], and in partitionable networks (Omni-Do) [DSS-SIR99,GS-JDA,MRS-DISC00]. Solutions for Do-All must perform all tasks efficiently in the presence of specific failure patterns. The efficiency is assessed in terms of work, time and communication complexity depending on the specific model of computation.

In the design of practical distributed/parallel programs one needs to ensure good performance and dependability under unpredictable load patterns caused by deviations from synchrony or by the failures of some processors to complete tasks on time. Here again, a common challenge is to perform N independent tasks on P processors [HG-DC01]. Such tasks could be copying a large array, searching a collection of data, or applying a function to all elements of a matrix [GHVM-DC01].

Computation and communication are two basic resource commodities in distributed computing. In a distributed system setting one normally computes to perform specific tasks. One also needs to communicate in order to coordinate multiple computation activities, for example, to increase computation efficiency by eliminating redundant work. As can be expected, there exists a trade-off between the efficiency of distributed computation and the amount of communication needed to coordinate the computation. Dwork, Halpern and Waarts [DHW] have considered a basic problem of distributed coordination, where a number of message-passing processors cooperate in executing a collection of independent and similarly-sized tasks (in the presence of processor crashes). They measure the efficiency of the computation in terms of its work complexity W (one task consumes one work unit), and the cost of communication in terms of the message complexity M. The authors also assess the overall performance in terms of effort that is defined as the sum of work W and message complexity M. Chlebus at al. [CGKS] solve a similar problem and they measure the effort of their solution as the sum of the number of all processing steps performed (this includes taskoriented work and any bookkeeping steps) and the number of messages sent. They observe that a processor can learn that a task is performed either by doing the task, or by receiving a message that tells it that the task was done. In such a setting, one can view computation and communication as comparable resources.

1The Do-All problem and the Write-All problem are subtly different. The main difference is that in Do-All the tasks may be supplied to the processors from some external sources, while in Write-All the tasks are stored in shared-mmeory accessible to all processors. In some contexts it is possible to abstract away from the sources and the nature of the tasks and to treat Do-All and Write-All as the same problem. The distinction between the problems is reinstated when one applies the results of such abstract analysis to specific message-passing and shared-memory algorthms.

2 The Write-All / Do-All Primitive

We define Do-All to be the following problem:

Definition 2.1 Given a set of N independent and idempotent tasks A[1..N] and P processors, perform each task at least once

Solving Do-All when P = N is one of the simplest computations that can be performed by a distributed system with P processors The following optimal synchronus program solves Do-All in constant time, thus achieving linear speed-up.

forall processors pid = 1..N parbegin

perform A[pid] parend

Next suppose that P ≠ N. It easy to modify the program above so that each processor performs ⌈N/P ⌉ tasks. (Padding is used to include “empty” tasks when P does not divide N.)

forall processors pid= 1..P parbegin

for i = 0 to ⌈N/P ⌉ − 1 do perform A[pid+i ∗ P]

rof parend

The above algorithm takes time T_{N,P} = Θ(N/P) and its cost (work) is C = P · T_{N,P} = P · Θ(N/P) = Θ(N). Thus this solution is also optimal, and in particular, it achieves linear speed-up: S = Θ(N)/T_{N,P} = Θ(N)/Θ(N/P) = Θ(P).

In shared-memory models, solving Do-All is equivalent (under some simple assumptions) to solving the following, more abstract problem, called Write-All.

Definition 2.2 Given a zero-valued array of N elements and P processors, the Write-All problem is to set each element of the array to 1.

The Write-All problem captures the computational progress that can be naturally accomplished in unit time by N synchronous processors in the absence of failures. Solving Do-All or Write-All efficiently in the presence of asynchrony and failures is very challenging.

Measuring efficiency. We are interested in studying the complexity of Do-All measured as work (cf. [KS-DC92,DHW-SIAM98,DMY-PODC94]). We assume that it takes a unit of time for a processor to perform a unit of work, and that a single task corresponds to a unit of work. Our definition of work complexity is based on the available processor steps measure [KS97]. Let F be the failure model. For a computation subject to a failure pattern F, F ∈ F, where F is a failure model, denote by P_{i}(F) the number of processors completing a unit of work in step i of the computation.

Definition 2.3 Given a problem of size N and a P -processor algorithm that solves the problem in the failure model F, if the algorithm solves the problem for a pattern F in F, with |F | ≤ f, by time step τ(F), then the work complexity W of the algorithm is:

W_{N,P,f} = max_{F ∈F, |F |≤f} { ∑_{i≤τ(F)} P_{i}(F)} .

The idling processors consume a unit of work per step even though they do not contribute to the computation. Definition 2.3 does not depend on the specifics of the target model of computation, e.g., whether it is message-passing or shared-memory. (Communication complexity is defined similarly in Section ??.)

Some research on Do-All in message-passing models uses a different definition of work, where processors are not charged for time spent idling or waiting for a message [DHW-SIAM98].

Do-All in message-passing models

Perform a set of N tasks using P fault-prone processors.

This problem was considered for the message-passing model by Dwork, Halpern and Waarts in their pioneering work [DHW92]. They developed several efficient algorithms for Do-All in the setting where the processors are subject to fail-stop (or crash) failures and where the tasks can be performed using the at-least-once execution semantics (i.e., the tasks can be made idempotent). Their model of computation assumes a synchronous system of processors, such that in one time unit a processor can perform a local computation and complete one task, and perform one round of communication consisting of sending and receiving messages.

Algorithmic techniques for distributed Do-All are motivated with the help of an observation that there is a qualitative duality between the shared-memory and the message-passing settings. In a shared-memory algorithm, a key datum, say reflecting an important component of the state of the computation, can be stored in a shared memory cell, and P processors can learn the value stored by performing P reads. In a message-passing algorithm, the only way to propagate information, stored anywhere, to P processors, is to do so by means of at least P −1 messages. Thus in message-passing algorithms, one sees a multicast or diffusion of a value to a subset of processors as a dual of these processors reading the value from a location in shared memory.

For the distributed message-passing models, one has to consider the costs of a computation in terms of the total work performed by the processors, and in terms of the number of messages sent by the processors. In the shared-memory models we have covered thus far, the algorithm efficiency was determined primarily by the available processor steps (or work) measure. Since in the worst case each instruction involves a constant number of memory accesses, work is also the measure of the number of memory accesses (i.e., the communication cost) performed by the computation. The one-to-one cost correspondence between work and communication does not apply to the messagepassing models, since the number of messages sent and received by one processor in one round of communication can be 0, or it can be comparable to P when broadcasts are enabled.

To measure the communication efficiency of distributed algorithms, the number of messages sent by the processors is counted. One can also consider the total number of bits sent when the messages can vary in size or can be large. Measuring the work of a computation depends on the interpretation of the relative costs associated with different processing steps. Specifically, one needs to consider the costs associated with local processing and the costs associated with doing the tasks in solving the Do-All problem.

The efficiency of message-passing algorithms is characterized in terms of their work and message complexity. We define message complexity similarly to Definition 2.3 of work: For a computation subject to a failure pattern F, F ∈ F, denote by M_{i}(F) the number of point-to-point messages sent during step i of the computation. For a given problem of size N, if the computation solves the problem by step τ(F) in the presence of the failure pattern F, where |F | ≤ f, then the message complexity M is:

M_{N,P,f} = max_{F ∈F, |F |≤f} { ∑_{i≤τ(F)} M_{i}(F)} .

3 Dwork Halpern Waarts

Optimizing Task-Oriented Work

In the setting of Dwork, Halpern and Waarts [DHW92], the local computation, whether performing low-level administrative tasks (such as maintaining the local clock) or idling, is considered to be negligible compared to the costs of performing the Do-All tasks, where processors may need to do constant size, but significant processing, or the costs of communication with external systems. This approach is consistent with the accepted accounting of costs in distributed algorithms. Additionally, Dwork, Halpern and Waarts consider processes, a notion that can be more general than processors. For example, when we interpret processes as threads of control, such that several threads may co-exist on the same processor, then the idle “steps” taken by threads need not be accounted for, since during such idle steps the processor hosting the thread can be executing other steps on behalf of other threads. Thus idle thread steps do not constitute a waste of the computational resources, and it is reasonable to account only for work spent on the Do-All tasks.

Dwork, Halpern and Waarts define the effort of an algorithm as the sum of its work and message complexity. This is done under the assumption that the cost of performing a unit of work and sending a message are comparable. They observe that at least P +N −1 units of work are necessary to solve the Do-All problem of size N using P processors. The goal then is the development of algorithms that perform O(N + P) work, thus achieving work-optimality, while keeping the total effort reasonable. In this synchronous model, the running time of an algorithm is also a meaningful measure of efficiency, and in the analysis of algorithms this measure is also considered. The first two algorithmsre sequential — at any point in time only one task is being performed, thus in these algorithms the redundancy of having multiple processors contributes to fault-tolerance, but not to speed-up. The third algorithm achieves fault-free time optimality, with graceful degradation of running time in the presence of failures.

One algorithm presented by the authors (protocol B) has effort O(N +P \sqrt{P}), with work contributing the cost O(N + P) towards the effort, and message complexity contributing the cost O(P \sqrt{P}). The running time of the algorithm is O(N + P). The use of synchrony in the algorithm is limited to failure detection by means of time-outs. Such algorithms can be converted to asynchronous algorithms when the underlying system is equipped with failure detectors. In this algorithm the N tasks are divided into chunks and each of these is divided into subchuncks. Processors checkpoint their progress by multicasting the completion information to subsets of processors after performing a subchunk, and broadcasting to all processors after completing chunks of work.

Another algorithm in [DHW92] (protocol C) has effort O(N + P log P). This includes optimal work of O(N + P), message complexity of (P log P), and time O(P^{2}(N + P)2^{N+P}). Thus the reduction in message complexity is traded-off for a significant increase in time.

The third work-optimal algorithm (protocol D) is designed for maximum speed-up, which is achieved with a more aggressive checkpointing strategy, thus trading-off time for messages. The message complexity is quadratic in P for the fault-free case, and in the presence of a failure pattern F, the message complexity degrades to Θ(|F | · P^{2}).

One interesting application of the Do-All algorithms is to solving Byzantine agreement with crash failures. Dwork, Halpern and Waarts show that a Do-All algorithm with P processors and N tasks can be used to solve agreement for N processors and less than P failures [DHW92]. In this application, the tasks performed by processors are sending messages to other processors informing them of the agreement value.

4 Chlebus De Prisco Shvartsman

Two new algorithms are presented and analyzed: AN and AR. All previous algorithms do not consider the possibility that a faulty processor is repaired and reintegrated into the system. [CDS] introduces a new algorithmic technique based on an aggressive coordination paradigm that permits multiple concurrent coordinators. This approach is suggested by the earlier observation that algorithms with only one coordinator cannot deal efficiently with restarts. The number of coordinators is managed adaptively. When failures of coordinators disrupt the progress of the computation, the number of coordinators is increased; when the failures subside, a single coordinator is appointed.

4.1 Algorithm AN

Algorithm AN presented by Chlebus et al. [CDS-DC01] uses a multiple-coordinator approach to solve Do-All(N, P, f) on crash-prone synchronous message-passing processors (P ≤ N). The model assumes that messages incur a known bounded delay and that reliable multicast [HT93] is available, however messages to/from faulty processors may be lost.

W: available processor steps (work)  M: message complexity
No  [DMY]  O(t + (f + 1)p)  O((f + 1)p)
restarts  [GMY]  O(t + (f + 1)p)  O(fp^{ε} + min{f + 1, log p}p)
(f < p)  AN  O((t + p log p/ log log p) log f)  O(t + p log p/ log log p + fp)

Restarts (f < p+r)

AR O((t + p log p + f) · min{log p, log f}) O(t + p log p + fp)

Figure 1: Efficiency of the solutions in [DMY,GMY] and algorithms AN and AR (the solutions in [DHW] consider a different notion of work complexity and focus on evaluation of effort).

4.1.1 Description of the Algorithm

We now give a brief description of the algorithm; additional details are given in Appendix A (but to avoid a complete restatement, we refer the reader to [CDS-DC01]). Algorithm AN proceeds in a loop which is iterated until all the tasks are executed. A single iteration of the loop is called a phase. A phase consists of three consecutive stages. Each stage consists of three steps. In each stage processors use the first step to receive messages sent in the previous stage, the second step to perform local computation, and the third step to send messages. A processor can be a coordinator or a worker. A phase may have multiple coordinators. The number of processors that assume the coordinator role is determined by the martingale principle: if none of the expected coordinators survive through the entire phase, then the number of coordinators for the next phase is doubled. If at least one coordinator survives in a given phase, then in the next phase there is only one coordinator. A phase that is completed with at least one coordinator alive is called attended, otherwise it is called unattended.

Processors become coordinators and balance their loads according to each processor’s local view. A local view contains the set of ids of the processors assumed to be alive. The local view is partitioned into layers. The first layer contains one processor id, the second two ids, the i^{th} contains 2^{i−1} ids.

Given a phase, in the first stage, the processors perform a task according to the load-balancing rule derived from their local views and report the completion of the task to the coordinators of that phase (determined by their local views). In the second stage, the coordinators gather the reports, they update the knowledge of the done tasks and they multicast this information to the processors that are assumed to be alive. In the last stage, the processors receive the information sent by the coordinators and update their knowledge of done tasks and their local views.

It is shown in [CDS-DC01] that the work of algorithm AN is W = O((N + N log N/ log log N) log f) and its message complexity is M = O(N + P log P/ log log P + fP), for P ≤ N.

In the rest of this section we present the new analysis of work and message complexity of algorithm AN. Throughout we assume that the algorithm correctness is shown as in [CDS-DC01].

4.1.2 Algorithmic details

The following information about algorithm AN is taken from [CDS-DC01]. More information can be obtained there. The algorithm proceeds in a loop which is repeated until all the tasks are executed. A single iteration of the loop is called a phase. A phase consists of three consecutive stages. Each stage consists of three steps (thus a phase consists of 9 steps). In each stage processors use the first step to receive messages sent in the previous stage, the second step to perform local computation, and the third step to send messages. We refer to these three step as the receive substage, the compute substage and the send substage.

Coordinators and workers. A processor can be a coordinator of a given phase. All processors (including coordinators) are workers in a given phase. Coordinators are responsible for recording progress, while workers perform tasks and report on that to the coordinators. In the first phase one processor acts as the coordinator. There may be multiple coordinators in subsequent phases. The number of processors that assume the coordinator role is determined by the martingale principle: if none of the expected coordinators survive through the entire phase, then the number of coordinators for the next phase is doubled. Whenever at least one coordinator survives a given phase, the number of coordinators for the next phase is reduced to one.

If at least one processor acts as a coordinator during a phase and it completes the phase without failing, we say that the phase is attended, the phase is unattended otherwise.

Local views. Processors assume the role of coordinator based on their local knowledge. During the computation each processor w maintains a list L_{w} = ⟨q_{1}, q_{2}, ..., q_{k}⟩ of supposed live processors. We call such list a local view. The processors in L_{w} are partitioned into layers consisting of consecutive sublists of L_{w}: L_{w} = ⟨Λ^{0}, Λ^{1}, ..., Λ^{j}⟩. The number of processors in layer Λ^{i+1}, for i = 0, 1, ..., j − 1, is the double of the number of processors in layer Λ^{i}. Layer Λ^{j} may contain less processors. When Λ^{0} = ⟨q_{1}⟩ the local view can be visualized as a binary tree rooted at processor q_{1}, where nodes are placed from left to right with respect to the linear order given by L_{w}. Thus, in a tree-like local view, layer Λ^{0} consists of processor q_{1}, layer Λ^{i} consists of 2^{i} consecutive processors starting at processor q_{2^{i}} and ending at processor q_{2^{i+1}−1}, with the exception of the very last layer that may contain a smaller number of processors. Processors in a local view do not necessarily appear in the order of processor identifiers.

The local view is used to implement the martingale principle of appointing coordinators as follows. Let L_{ℓ,w} = ⟨Λ^{0}, Λ^{1}, ..., Λ^{j}⟩ be the local view of worker w at the beginning of phase ℓ. Processor w expects processors in layer Λ^{0} to coordinate phase ℓ; if no processor in layer Λ^{0} completes phase ℓ, then processor w expects processors in layer Λ^{1} to coordinate phase ℓ + 1; in general processor w expects processors in layer Λ^{i} to coordinate phase ℓ + i if processors in all previous layers Λ^{k}, ℓ ≤ k < ℓ + i, did not complete phase ℓ + k. The local view is updated at the end of each phase.

Phase structure and task allocation. The structure of a phase of the algorithm is as follows. Each processor w keeps its local information about the set of tasks already performed, denoted D_{w}, and the set of live processors, denoted P_{w}, as known by processor w. Set D_{w} is always an underestimate of the set of tasks actually done and P_{w} is always an overestimate of the set of processors that are “available” from the start of the phase. We denote by U_{w} the set of unaccounted tasks, i.e., whose done status is unknown to w. Sets U_{w} and D_{w} are related by U_{w} = T \D_{w}, where T is the set of all the tasks. Given a phase ℓ we use P_{ℓ,w}, U_{ℓ,w} and D_{ℓ,w} to denote the values of the corresponding sets at the beginning of phase ℓ.

Computation starts with phase 0 and any processor q has all processors in L_{0,q} and has D_{0,q} empty. At the beginning of phase ℓ each worker (that is, each processor) w performs one task according to its local view L_{ℓ,w} and its knowledge of the set U_{ℓ,w} of unaccounted tasks, using the following load-balancing rule. Worker w executes the task whose rank is (i mod |U_{ℓ,w}|)^{th} in the set U_{ℓ,w} of unaccounted tasks, where i is the rank of processor w in the local view L_{ℓ,w}. Then the worker reports the execution of the task to all the processors that, according to the worker’s local view, are supposed to be coordinators of phase ℓ. For simplicity we assume that a processor sends a message to itself when it is both worker and coordinator. Any processor c that, according to its local view, is supposed to be coordinator, gathers reports from the workers, updates its information about P_{ℓ,c} and U_{ℓ,c} and broadcasts this new information causing the local views to be reorganized. In [CDS-DC01] it is shown that at the beginning of any phase ℓ all live processors have the same local view L_{ℓ} and the same set U_{ℓ} of unaccounted tasks and that accounted tasks have been actually executed. A new phase starts if U_{ℓ} is not empty.

The following is a detailed description of a phase.

Phase ℓ of algorithm AN:

stage 1. The receive substage is not used. In the compute substage, any processor w performs a specific task z according to the load-balancing rule. In the send substage processor w sends a report(z) to any coordinator, that is, to any processor in the first layer of the local view L_{ℓ,w}.

stage 2. In the receive substage the coordinators gather report messages. For any coordinator c, let z_{c}^{1}, …, z_{c}^{k_{c}} be the set of tids received. In the compute substage c sets D_{c} ← D_{c} ∪ ⋃_{i=1}^{k_{c}}{z_{c}^{i}}, and P_{c} to the set of processors from which c received report messages. In the send substage, coordinator c multicasts the message summary(D_{c}, P_{c}) to processors in P_{c}.

stage 3. During the receive substage summary messages are received by live processors. For any processor w, let (D_{w}^{1}, P_{w}^{1}), …, (D_{w}^{k_{w}}, P_{w}^{k_{w}}) be the sets received in summary messages In the compute substage w sets D_{w} ← D_{w}^{i} and P_{w} ← P_{w}^{i} for an arbitrary i ∈ {1, …, k_{w}} and updates its local view L_{w} as described below. The send substage is not used.

Local view update rule. In phase 0 the local view L_{0,w} of any processor w is a tree-like view containing all P processors ordered by their pids. Let L_{ℓ,w} = ⟨Λ^{0}, Λ^{1}, ..., Λ^{j}⟩ be the local view of processor w for phase ℓ. We distinguish two possible cases.

Case 1. Phase ℓ is unattended. Then the local view of processor w for phase ℓ + 1 is L_{ℓ+1,w} = ⟨Λ^{1}, ..., Λ^{j}⟩.

Case 2. Phase ℓ is attended. Then processor w receives summary messages from some coordinator in Λ^{0}. Processor w computes its set P_{w} as described in stage 3 (we will see that all processors compute the same set P_{w}). The local view L_{ℓ+1,w} of w for phase ℓ + 1 is a tree-like local view containing the processors in P_{w} ordered by their pids.

Finally, we demonstrate how a computation of algorithm AN proceeds. Given an execution of the algorithm, we enumerate the phases. We denote the attended phases of the execution by α_{1}, α_{2}, …, etc. We denote by π_{i} the sequence of unattended phases between the attended phases α_{i} and α_{i+1}. We refer to π_{i} as the i^{th} (unattended) period; an unattended period can be empty. Hence the computation proceeds as follows: unattended period π_{0}, attended phase α_{1}, unattended period π_{1}, attended phase α_{2}, and so on.

4.2 Algorithm AR

Algorithm AR is similar to algorithm AN; the difference is that there are added messages to handle the restart of processors. After the restart, processor q broadcasts restart(q) messages in each step until it receives a response. Processors receiving such messages, ignore them if these messages are not received in the receive substage of stage 2 of a phase. Thus we can imagine that a restarted processor q broadcasts a restart(q) in the send substage of stage 1 of a phase ℓ (however we will count all the restart messages in the message complexity). This message is then received by all the live and restarted processors of that phase, and, as we will see shortly, processor q is re-integrated in the view for phase ℓ + 1. Processor q needs to be informed about the status of the ongoing computation. Hence processors that have this information send the info(U_{ℓ}, L_{ℓ}) messages to processor q with the set U_{ℓ} of unaccounted tasks and the local view L_{ℓ}. Next we provide the detailed description for each phase. The parts that are new or that are different in algorithm AR as compared to algorithm AN are italicized.

Phase ℓ of algorithm AR:

stage 1. The receive substage is not used. In the compute substage any processor w performs a specific task z according to the load balancing rule. In the send substage w sends a report(z) to any coordinator, that is, to any processor in the first layer of L_{ℓ,w}. Any restarted processor q broadcasts the restart(q) message informing all live processors of its restart.

stage 2. In the receive substage the coordinators gather report messages and all processors gather restart messages. Let R be the set of processors that sent a restart message. For any coordinator c, let z_{c}^{1}, ..., z_{c}^{k_{c}} be the set of TIDs received in report messages. In the compute substage c sets D_{c} ← D_{c} ∪⋃_{i=1}^{k_{c}}{z_{c}^{i}} and P_{c} to the set of processors from which c received report messages. In the send substage, coordinator c multicasts the message summary(D_{c}, P_{c}) to the processors in P_{c} and R. Any processor in P_{c} sends the message info(U_{ℓ}, L_{ℓ}) to processors in R.

stage 3. In the receive substage processors in R receive info(U_{ℓ}, L_{ℓ}) messages and processors in P_{c} and R receive summary(D_{c}, P_{c}) messages. In the compute substage, a restarted processor q sets L_{ℓ,q} ← L_{ℓ} and U_{ℓ,q} ← U_{ℓ}. Let (D_{w}^{1}, P_{w}^{1}), ..., (D_{w}^{k_{w}}, P_{w}^{k_{w}}) be the sets received in summary messages by processor w. Processor w sets D_{w} ← D_{w}^{i} and P_{w} ← P_{w}^{i} for an arbitrary i ∈ 1, ..., k_{w} and updates its local view L_{ℓ,w} as described below. The send substage is not used.

[figure: Coordinator; Worker knows L,P,U,D; Restarted; Step 2; Step 1; Step 3; Perform one task; receive report; update D,P; summary; {update \atop R}; receive summary; update D,P,L; report; receive restart; info; restart; receive restart; {update \atop R}; receive info summary; update D,P,L]

Figure 2: A phase of algorithm AR (for algorithm AN ignore the bottom line, which represents restarted processors, and all the messages referring to it).

Loal view update rule. In phase 0 the local view L_{0,w} of any processor w contains all the processors in ¶ ordered by their PIDs, and the first layer is a singleton set. Let L_{ℓ,w} = ⟨Λ^{0}, Λ^{1}, ..., Λ^{j}⟩ be the local view of processor w for phase ℓ. We distinguish two possible cases.

Case 1. Phase ℓ is unattended. Let R^{ℓ} be the set of restarted processors which send restart messages. Let R′ be the set of processors of R^{ℓ} that are not already in the local view L_{ℓ,w}. Let ⟨R′⟩ be the processors in R′ ordered according to their PIDs. The local view for the next phase is L_{ℓ+1,w} = ⟨Λ^{1}, ..., Λ^{j}⟩ ⊕ ⟨R′⟩. The operator ⊕ places processors of R′, in the order ⟨R′⟩, into the last layer Λ^{j} till this layer contains exactly the double of the processors of layer Λ^{j−1} and possibly adds a new layer Λ^{j+1} to accommodate the remaining processors of ⟨R′⟩. That is, newly restarted processors which are not yet in the view, are appended at the end of the old view. Notice that restarted processors, which receive info messages, know the old view L_{ℓ}.

Case 2. Phase ℓ is attended. Let R^{ℓ} be the set of restarted processors. Since the phase is attended summary messages are received by all the live processors (including the restarted ones). Any processor w updates P_{w} as described in stage 3. Processor w knows the set R^{ℓ}. The local view L_{ℓ+1,w} for the next phase is structured according to the martingale principle and contains all the processors in P_{w} ∪ R^{ℓ} ordered according to their PIDs.

If there are no restarts, algorithm AR behaves as algorithm AN. Figure 2 provides a graphical description of both algorithms.

5 Georgiou Russell Shvartsman

We let Do-All(N, P, f) stand for the Do-All problem for N tasks, P processors and up to f failures. We let Do-All^{O}(N, P, f) denote the Do-All(N, P, f) problem that is solved with the use of an arbitrary, omniscient oracle O that may make available to the processors any deterministic function of the global history of the computation: in particular, the oracle may be used to assist the processors to load-balance and terminate (but unlike the oracle’s delphian colleague, it cannot predict the future). The oracle assumption is used as a tool for studying the work complexity patterns of any fault-tolerant algorithm: (i) of course, any lower bound developed for this strong model applies equally well to message-passing or shared memory models as the oracle may make available to each processor the message or memory-read history it would have observed, (ii) any algorithm in the oracle model may be simulated in a message-passing or shared memory model by suitably simulating the oracle. As most Do-All algorithms use communication to load-balance and detect termination, this framework allows for the complexity analysis of specific algorithms to be divided into two parts: (1) the analysis of the cost of tolerating failures while performing work assuming perfect load-balancing, and (2) the analysis of the cost of implementing perfect load-balancing. We use exactly this approach to derive new f-sensitive upper bounds for messagepassing and shared-memory models.

One may study the complexity of Do-All^{O}(·) in conjunction with a variety of different oracles. When we prove lower bounds, we consider arbitrary oracles that may make available, for example, the entire global history of the computation on every processor. When we prove upper bounds, we use the load-balancing oracle, O_{L}, described next. During each synchronous iteration of the computation, the oracle O_{L} makes available to each processor P_{i} two values: Oracle-complete, a Boolean which takes the value true if and only if all tasks were completed at the beginning of this iteration, and Oracle-task_{i}, an integer whose value is a task identifier with the property that the identifiers associated to the live processors are split evenly among the tasks. In particular, if processors i_{1}, …, i_{k} ∈ [P] are alive and tasks j_{1}, …, j_{ℓ} ∈ [N] are incomplete at the beginning of the iteration, then Oracle-task_{i_{s}} = j_{t}, where t = (s − 1 mod ℓ) + 1.

Prior to [GRS] no comprehensive failure-sensitive analysis of Do-All was known. Yet practical concerns would be well served by the knowledge of what happens as the number of failures grows from 1 to P − 1. In particular, it is important to understand the behavior of the best algorithms for the entire range of f.

The detailed contributions in [GRS] are as follows.

I. We provide upper bounds (Section ??) and matching lower bounds (Section 7) that are fsensitive, specifically we give a complete analysis of Do-All^{O}(N, P, f) for the entire range of f. The bounds on work W are:^{2}

(a) W = \{O|Ω\} (N + P + P \frac{log(min(N,P))}{log(P/f)})  when f ≤ \frac{P}{log(min(N,P))},
(b) W = \{O|Ω\} (N + P + P \frac{log(min(N,P))}{log log(min(N,P))}) when f > \frac{P}{log(min(N,P))}.

(1)

The lower bounds of course apply to algorithms in weaker models (e.g., a message passing or shared memory model).

In the rest of the paper we use a shorthand notation used to combine the statements such as (1)(a) and (1)(b) above. We define the quantity Λ_{N,P,f}^{r}, where r is the number of Do-All iterations (for r-iterative Do-All), and where N, P, and f appear in their usual roles, as follows.

Λ_{N,P,f}^{r} = \{ \begin{matrix}log(\frac{P r}{f}) & when f ≤ \frac{P r}{log(min(N,P))}, \\ log log(min(N, P)) & when f > \frac{P r}{log(min(N,P))}. \end{matrix}  (2)

For a single instance Do-All, i.e., when r = 1, we simplify this by defining Λ_{N,P,f} = Λ_{N,P,f}^{1}. Thus the result (1) is succinctly given by the following.

W = {O|Ω} (N + P + P log(min(N, P))/Λ_{N,P,f}) .  (3)

We use our bounds (3) (or equivalently (1)) to derive new bounds for algorithms where the extant analyses do not integrate f adequately. This is done by analyzing how load-balancing is implemented by the algorithms, e.g., by using coordinators or global data-structures. We show the following.

II. In Section ?? we provide new analysis of algorithm AN of Chlebus et al. [CDS-DC01] for Do-All in the message-passing model with crashes (P ≤ N). This algorithm has best known work for moderate number of failures. We show the complete analysis of work W and message complexity M:

W = O (log f (N + P log P/Λ_{N,P,f})) and M = O (N + P log P/Λ_{N,P,f} + P f) .

2We let “{O|Ω}” stand for “O” when describing an upper bound and for “Ω” when describing a lower bound.

All logarithms are to the base 2 unless explicitly specified otherwise. It is understood, that when f = 0, then X/ log(Y/f) = 0, for any X ≠ 0 and Y ≠ 0.

III. In Section ?? we give a complete analysis of the work complexity W of the algorithm of Kanellakis and Shvartsman [KS-DC92] that solves the Write-All problem in synchronous sharedmemory systems (shared memory Do-All) with processor crashes (P ≤ N):

W = O (N + P log N log P/Λ_{N,P,f}) .

Note that the two algorithms [CDS-DC01,KS-DC92] are designed for different models and use dissimilar data and control structures, however both algorithms make their load-balancing decisions by gathering global knowledge. By understanding what work is expended on load-balancing vs. the inherent work overhead due to the lower bounds (1), we are able to obtain the new results, and to demonstrate the utility and the generality of our approach. (We restrict our attention to the case P ≤ N for consistency with [CDS-DC01,KS-DC92].)

Do-All algorithms have been used in developing simulations of failure-free algorithms on failureprone processors, e.g., [KPS-STOC90,AS91]. This is done by iteratively using a Do-All algorithm to simulate the steps of the N failure-free “virtual” processors on P failure-prone “physical” processors (here the usual case is that the number of physical processors does not exceed the number of virtual processors, i.e., P ≤ N). In this paper we abstract this idea as the iterative Do-All problem as follows:

The r-iterative Do-All problem, denoted r-Do-All(N, P, f), is the problem of using P processors to solve r instances of N-task Do-All with the added restriction that every task of the ith instance must be completed before any task of the i + 1st instance is begun.

The oracle version r-Do-All^{O}(N, P, f) is defined analogously. An obvious solution for this problem is to run a Do-All algorithm r times. If the work complexity of Do-All in a given model is W_{N,P,f}, then the work of r-Do-All is clearly no more than r · W_{N,P,f}. This does not take into account the fact that f crashes occur throughout the r iterations. We present a substantially better analysis for crash-prone processors:

IV. In Section ?? we show matching upper and lower bounds on work W for r-Do-All^{O}(N, P, f), where f < P ≤ N, for specific ranges of failures.

W = {O|Ω} (r · (N + P log P/Λ_{N,P,f}^{r})) .  (4)

The improvement in the bounds (4) are twofold. First, the derivation of the bounds reflects our improved failure-sensitive analysis for a single Do-All. Additionally, we have Λ_{N,P,f}^{r} ≥ Λ_{N,P,f} for any r, moreover Λ_{N,P,f} is a constant with respect to r. The result is that our bounds (4) are asymptotically better than those obtained by computing the product of r and the (non-iterated) Do-All bounds (1) (for P ≤ N).

V. In Section ?? we show how to solve r-Do-All(N, P, f) (P ≤ N) on synchronous messagepassing processors with the following work (W) and message (M) complexity.

W = O (r · log(\frac{f}{r}) · (N + P log P/Λ_{N,P,f}^{r})) and M = O (r · (N + P log P/Λ_{N,P,f}^{r}) + fP ).

VI. In Section ?? we use r-Do-All(N, P, f) (P ≤ N) to show that P crash-prone processors can simulate any synchronous N-processor, r-time shared-memory algorithm (pram) with work:

W = O (r · (N + P log N log P/Λ_{N,P,f}^{r})) .

This last result is strictly better than the previously known deterministic bounds for deterministic parallel algorithm simulations using the Write-All algorithm [KS-DC92] (the best known to date) together with the simulation techniques such as [KPS-STOC90,AS91].

6 Malewicz Russell Shvartsman (00, 01, 02)

To study aspects of the trade-off between communication and computation in distributed cooperative applications, we consider the following abstract problem: n processors must perform t tasks and learn the results of all tasks. We assume that all tasks are known to all processors. A common impediment to effective coordination in distributed settings is asynchrony that manifests itself, for example, in disparate processor speeds and nondeterministic message latency. Fortunately, our problem can always be solved by a communication-oblivious algorithm that forces each processor to perform all tasks. Such a solution has work W = O(t · n), an requires no communication, i.e., M = 0. On the other hand, Ω(t) is the obvious lower bound on work and the best known lower bound is W = Ω(t + n log n), cf. [KS]. Therefore the trade-off expectation is that if we gradually increase the number of messages we should be able to decrease the amount of work performed.

Let us consider an asynchronous setting, where processors communicate by means of a rendezvous, i.e., two processors that are able to communicate can perform state exchange. The processors that are not able to communicate via rendezvous have no choice but to perform all t tasks. Consider the computation with a single rendezvous. There are n − 2 processors that are unable to communicate, and they collectively must perform exactly t·(n−2) work units to learn all results. Now what about the remaining pair of processors that are able to rendezvous? In the worst case they rendezvous after performing all tasks individually. In this case no savings in work are realized. Suppose they rendezvous having performed t/2 tasks each. In the best case, the two processors performed mutually-exclusive subsets of tasks and they learn the complete set of results as a consequence of the rendezvous. In particular if these two processors know that they will be able to rendezvous in the future, the could schedule their work as follows: one processors performs the tasks in the order 1, 2, …, t, the other in the order t, t−1, …, 1. No matter when they happen rendezvous, the number of tasks they both perform is minimized. Of course the processors do not know a priori what pair will be able to rendezvous. Thus it is interesting to produce task execution schedules for all processors, such that upon the first rendezvous of any two processors the number of tasks performed redundantly is minimized.

This setting we have just described is interesting for several reasons. If the communication links are subject to failures, then each processor must be ready to execute all of the t tasks, whether or not it is able to communicate. In realistic settings the processors may not initially be aware of the network configuration, which would require expenditure of computation resources to establish communication, for example in radio networks. In distributed environments involving autonomous agents, processors may choose not to communicate either because they need to conserve power or because they must maintain radio silence. Finally, during the initial configuration of a dynamic network or a middleware service (such as a group communication service [CACM96]) the individual processors may start working in isolation pending the completion of system configuration. Regardless of the reasons, it is important to direct any available computation resources to performing the required tasks as soon as possible. In all such scenarios, the t tasks have to be scheduled for execution by all processors. The goal of such scheduling must be to control redundant task executions in the absence of communication and during the period of time when the communication channels are being (re)established.

For a variation of the problem we deal with in this report, Dolev et al. [DSS] showed that for the case of dynamic changes in connectivity, the termination time of any on-line task assignment algorithm can be greater than the termination time of an off-line task assignment algorithm by a factor linear in n. This means that an on-line algorithm may not be able to do better than the trivial solution that incurs linear overhead by having each processor perform all the tasks. With this observation [DSS] develops an effective strategy for managing the task execution redundancy and proves that the strategy provides each of the n processors with a schedule of Θ(n^{1/3}) tasks such that at most one task is performed redundantly by any two processors.

6.1 Schedules, waste, and designs

We consider the abstract setting where n processors need to perform t independent and idempotent tasks. A task is idempotent if the execution of the task yields the same result when it is performed more than once. A task is independent if the result of its execution does not depend on the order in which other tasks are executed. The processors have unique identifiers from the set [n] = {1, …, n}, and the tasks have unique identifiers from the set [t] = {1, …, t}. Initially each processor knows the tasks that need to be performed and their identifiers.

We shall focus on the scheduling problem discussed above, abstracted as follows. An (n, t)-schedule is a tuple (σ_{1}, …, σ_{n}) of n permutations of the set [t]. When n = 1 it is elided and we simply write t-schedule.

An (n, t)-schedule immediately gives rise to a strategy for n isolated processors who must complete t tasks until communication between some pair (or group) is established: the processor i simply proceeds to complete the tasks in the order prescribed by σ_{i}. Suppose now that some k of these processors, say q_{1}, …, q_{k}, should rendezvous at a time when the ith processor in this group, q_{i}, has completed a_{i} tasks. Ideally, the processors would have completed disjoint sets of tasks, so that the total number of tasks completed is ∑_{i} a_{i}. As this too much to hope for in general, it is natural to attempt to bound the gap between ∑_{i} a_{i} and the actual number of distinct tasks completed. This gap we call waste:

Definition 6.1 If S is a (n, t)-schedule and (a_{1}, …, a_{k}) ∈ ℕ^{k}, the waste function for S is

W_{S}(a_{1}, …, a_{k}) = max_{(q_{1},... ,q_{k})} (∑_{i}^{k} a_{i} − |⋃_{i}^{k} σ_{q_{i}}([a_{i}])|) ,

this maximum taken over all k tuples (q_{1}, …, q_{k}) of distinct elements of [n].

Here (and throughout), if ϕ : X → Y is a function and S ⊂ X, we let ϕ(S) = {ϕ(x) | x ∈ S}. For a specific vector a = (a_{1}, …, a_{k}), W_{S}(a) captures the worst-case number of redundant tasks performed by any collection of k processors when the ith process has completed the first a_{i} tasks of its schedule.

One immediate observation is that bounds on pairwise waste W_{S}(·, ·) can be naturally extended to bounds on k-wise waste W_{S}(\underbrace{·, …, ·}_{k}): specifically, note that if S is an (n, t)-schedule then

W_{S}(a_{1}, …, a_{k}) ≤ ∑_{i<j} W_{S}(a_{i}, a_{j})

just by considering the first two terms of the standard inclusion-exclusion rule. Moreover, it appears that this relationship is fairly tight as it is nearly attained by randomized schedules—see Section 7.1.2. With this justification we shall content ourselves to focus the investigation on pairwise waste—the function W_{S}(a, b).

Set systems with prescribed intersection properties have been the object of intense study by both the design theory community and the extremal set theory community (see, e.g., [?] for a survey). Despite this, the study of waste appears to be new. We shall, however, make substantial use of some design-theoretic constructions, which we describe below.

Definition 6.2 A ℓ-(v, k, λ) design is a family of subsets S = (S_{1}, …, S_{n}) of the set [v] with the property that each |S_{i}| = k and any set of ℓ elements of [v] is a subset of precisely λ of the S_{i}. (N.B. The subsets S_{i} are typically referred to as blocks.)

Observe that if S is a ℓ-(v, k, λ) design, then it is also a (ℓ − 1)-(v, k, λ̂) design where

λ̂ = λ\frac{(v − ℓ + 1)}{(k − ℓ + 1)}.

To see this, note that if T is a subset of elements of size ℓ − 1, then there are exactly v − (ℓ − 1) sets of size ℓ which contain T; let U_{i}, i ∈ [v −(ℓ−1)], denote these sets. By assumption, each U_{i} appears in exactly λ of the S_{j}. Of course, if U_{i} is a subset of some S_{j}, then in fact exactly k − (ℓ − 1) if the U_{i} are subsets of S_{j}. Hence T appears in exactly λ(v − ℓ + 1)/(k − ℓ + 1) of the S_{j}, as desired.

To see the connection between such designs and our problem, let D be a 2-(n, k, λ) design consisting of t sets S_{1}, …, S_{t}. For each i ∈ [n], let T_{i} = {j | i ∈ S_{j}}. Note now that for any i ≠ j,

T_{i} ∩ T_{j} = {k | {i, j} ⊂ S_{k}}

and hence that |T_{i} ∩ T_{j}| = λ. Based on the observation above, we see also that ∀i, j, |T_{i}| = |T_{j}| and let a denote this common cardinality. Now, let Σ = (σ_{1}, …, σ_{n}) be any sequence of permutations of [t] for which σ_{i}([a]) = T_{i}. It is clear that these form an (n, t)-schedule for which

W_{Σ}(a, a) = λ.

Unfortunately, the above construction offers satisfactory control of 2-waste only for the specific pair (a, a). Furthermore, considering that the construction only determines the sets σ_{i}([a]) and σ_{i}([n] \ [a]), the ordering of these can be conspiratorially arranged to yield poor bounds on waste for other values. Our goal is construct schedules with satisfactory control on waste for all pairs (a, b).

While designs do not appear to immediately induce a solution to this problem, we will apply the following design-theoretic construction several times in the sequel. Let GF(q) denote the finite field with q elements, where q is a prime power. Treating GF(q)^{3} as a vector space over GF(q), the design will be given by the lattice of linear subspaces of GF(q)^{3}. It is easy to check that there are n = q^{2} + q + 1 distinct one dimensional subspaces of GF(q)^{3}, which we denote ℓ_{1}, …, ℓ_{n}. We say that two subspaces ℓ_{i} and ℓ_{j} are orthogonal if ∀u ∈ ℓ_{1}, ∀v ∈ ℓ_{2}, ⟨u, v⟩ = ∑ u_{j}v_{j} mod p = 0; in this case we write ℓ_{i} ⊥ ℓ_{j}. It is a fact that for any one dimensional subspace there are exactly q + 1 one dimensional subspaces to which it is orthogonal. The design consists of the n = q^{2} + q + 1 sets S_{u} = {ℓ_{i} | ℓ_{i} ⊥ ℓ_{u}}. It is easy to show that any pair of such sets intersect at a single ℓ_{i}, and that this forms a 2-(q^{2} + q + 1, q + 1, 1) design. See [?] for a proof and more discussion.

For concreteness, we fix a specific (arbitrary) ordering of each of these sets S_{u}: let L_{u} denote a canonical sequence ⟨t_{u}^{1}, …, t_{u}^{r}⟩ where S_{u} = {ℓ_{t_{u}^{i}} | 1 ≤ i ≤ q +1}; i.e., the one dimensional subspaces ℓ_{t_{u}^{i}}, i = 1, …, q + 1, are precisely those orthogonal to ℓ_{u}. For convenience, for two sequences A and B, we let A ∩ B and A ∪ B denote the corresponding union or intersection of the sets of objects in the sequences. We record the above discussion in the following theorem.

Theorem 6.3 Let n = q^{2} +q+1, where q is a prime power. Then the sequences L_{n} = ⟨L_{1}, …, L_{n}⟩ possess the following properties: each L_{u} has length q + 1, for each u ≠ v, |L_{u} ∩ L_{v}| = 1, and any element appears in exactly q+1 distinct sequences. We note also that if q is prime, the first element of each sequence can be calculated in O(log n) time; each subsequent element can be calculated in O(1) time.

(We assume throughout that addition or multiplication of two log (max{n, t})-bit numbers can be performed in O(1) time.)

7 Redundancy without communication: a lower bound

Controlling global computation redundancy in the absence of communication is a futile task. This is because no amount of algorithmic sophistication can compensate for the possibility of individual processors, or groups of processors, becoming disconnected during the computation. In general, an adversary that is able to partition the processors into g groups that cannot communicate with each other will cause any task-performing algorithm to have work Ω(t · g), even if each group of processors performs no more than the optimal number of Θ(t) tasks. In the extreme case where all processors are isolated from the beginning, the work of any algorithm is Ω(t · n), which is at least the work of an oblivious algorithm, where each processor performs all tasks.

Of course it is not surprising that substantial redundancy cannot be avoided in the absence of communication, furthermore, the lower bound on work of Ω(t · n) is not very interesting. However, as we pointed out earlier, it is possible to schedule the work of a pair of processors so that each can perform up to t/2 tasks without a single task performed redundantly. Thus it is very interesting to consider the intersection properties of pairs of processor schedules, i.e., 2-waste.

If we insist that among the n total processors, any two processors, having executed the same number of tasks t′, where t′ < t, perform no redundant work, then it must be the case that t′ ≤ ⌊t/n⌋. In particular, if n = t, then the pairwise waste jumps to one if any processor executes more than one task. The next natural question is: how many tasks can processors complete before the lower bound on pairwise redundant work is 2? In general, if any two processors perform t_{1} and t_{2} tasks respectively, what is the lower bound on pairwise redundant work? In this section we answer these questions. The answers contain both good and bad news: given a fixed t, the lower bound on pairwise redundant work starts growing slowly for small t_{1} and t_{2}, then grows quadratically in the schedule length as t_{1} and t_{2} approach t.

We proceed to the lower bound, which generalizes the second Johnson Bound [?] for the case when two processors execute different number of tasks prior to their rendezvous.

Theorem 7.1 ([MRS-DISC]) Let P = ⟨π_{1}, …, π_{n}⟩ be an (n, t)-schedule and let 0 ≤ a ≤ b ≤ t. Then

W_{P}(a, b) ≥ \frac{na^{2}}{(n − 1)(t − b + a)} − \frac{a}{n − 1}.

For example, when processors perform the same number of tasks a = b and n = t, then the worst case number of redundant tasks for any pair is at least \frac{a^{2}−a}{t−1}. This means that (for n = t) if a exceeds \sqrt{t} + 1, then the number of redundant task is at least 2.

Corollary 7.2 ([MRS-DISC]) For t = n, if a > \sqrt{n − 3/4} + \frac{1}{2} then any n-processor schedule of length a for t tasks has worst case pairwise waste at least 2.

7.1 Random schedules

As one would expect, schedules chosen at random perform quite well. In this section we explore the behavior of the (n, t)-schedules obtained when each permutation is selected uniformly (and independently) at random among all permutations of [t].

7.1.1 Randomized schedules

When the processors are endowed with a reasonable source of randomness, a natural candidate scheduling algorithm is one where processors select tasks by choosing them uniformly among all tasks they have not yet completed. This amounts to the selection, by each processor i, of a random permutation π_{i} ∈ S_{[t]} which determines the order in which this processor will complete the tasks. (S_{[t]} denotes the collection of all permutations of the set [t].) We let R be the resulting system of schedules.

Our objective now is to show that random schedules R have controlled waste with high probability. This amounts to bounding, for each pair i, j and each pair of numbers a, b, the overlap |π_{i}([a]) ∩ π_{j}([b])| . Observe that when these π_{i} are selected at random, the expected size of this intersection is ab/t. By showing that the actual waste is very likely to be close to this expected value, one can conclude the waste if bounded for all long enough prefixes.

Theorem 7.3 ([MRS-DISC]) Let R be a system of n random schedules for t tasks constructed as above. Then with probability at least 1 − \frac{1}{nt},

∀a, b such that 7\sqrt{t} ln (2nt) ≤ a, b ≤ t, W_{R}(a, b) ≤ \frac{ab}{t} + Δ(a, b) ,

where Δ(a, b) = 11\sqrt{\frac{ab}{t} ln(2nt)}

Observe that Theorem 7.1 shows that (n, t)-schedules must have waste W(a, a) = Ω(a^{2}/t) (as n → ∞); hence such randomized schedules offer nearly optimal waste for this case.

7.1.2 k-Waste for random schedules

For random schedules, one can apply martingale techniques to directly control k-wise waste. We mention one such result below.

Theorem 7.4 Consider the random schedule R from above. Then with probability at least 1−1/n,

W_{R}(a, …, a) ≤ ∑_{s=2}^{k}(−1)^{s}\binom{k}{s}\frac{a^{s}}{t^{s−1}} + Δ_{a,k},

where Δ_{a,k} = (2k + 1)\sqrt{a ln n}.

Note that again this bounds the distance of the k-waste from its expected value, which can be computed by inclusion-exclusion to be ∑_{s=2}^{k}(−1)^{s}\binom{k}{s} \frac{a^{s}}{t^{s−1}} .

7.1.3 Arbitrary rendezvous patterns (Georgiou Russell Shvartsman)

Thus far we have established bounds on wasted work for a single rendezvous. It is also interesting to understand the effectiveness of randomized schedules when encountering arbitrary patterns of (not necessarily pairwise) rendezvous. In general, an adversary that leaves the processors partitioned into g disconnected components will cause any task-performing algorithm to have work Ω(t·g) even if each group of processors performs no more than the optimal number of Θ(t) tasks. This lower bound forms a somewhat pessimistic landscape for our problem. Nevertheless, it is important to design and analyze algorithms that can be shown to be more efficient than the oblivious approach where each processor performs all tasks. In this case where the lower bound is quite sensitive to the interconnection pattern, we consider a competitive analysis [ST-CACM85] of randomized schedules.

Here we focus on the scenario where processors begin working (asynchronously) on the tasks in isolation and an adversary may choose to merge together various processors or groups of processors as the computation progresses. Once two processors find themselves in the same group, they cannot be separated–the adversary may never fragment a group of processors once it has been formed. An algorithm in this model is a rule which, given a collection of processors and a collection of tasks, determines the next task for this group to complete. A group of processors P who have collectively completed tasks T can use such a rule to determine the next task to work on. We shall consider the randomized algorithm, where a group of processors always selects a task at random from those it has not completed to work on next.

Adversaries in this model can be serialized in the following sense without loss of generality. An adversary A is described by a sequence of events; each event either (i.) allows a specific processor or group of processors to complete some number t of tasks, or (ii.) merges together some collection of processors and groups to form a new group. Note that when groups are formed by the adversary they inherit from their constituents full knowledge of all tasks they have previously completed.

We consider a competitive analysis that compares the work of the randomized algorithm with the work of an optimal algorithm that has complete information about the adversary (and hence the future pattern of rendezvous).

Let D be a deterministic algorithm for our problem and A an adversary, we let W_{(t,n)}(D, A) denote the total work (i.e., the number of tasks executed, counting multiplicities) of algorithm D with n processors and t tasks, where the pattern of rendezvous is determined by adversary A. Where t and n can be inferred from context, we simply write W (D, A).

We treat randomized algorithms as distributions over deterministic algorithms; for a set C and a family of deterministic algorithms {D_{r} | r ∈ C} we let R = Ra({D_{r} | r ∈ C}) denote the randomized algorithm where r is selected uniformly at random from C and scheduling is done according to D_{r}. We let OPT denote the optimal off-line algorithm, which may schedule tasks with full knowledge of the sequence of events caused by the adversary.

Definition 7.5 ([ST-CACM85,FKLMSY-JA91,BBKTW94])

A randomized algorithm R is α-competitive if for all adversaries A,

E_{r}[W (D_{r}, A)] ≤ α · W (OP T, A),

this expectation being taken over choice of r ∈ Ω.

The following lower bound holds for the competitive ratios.

Theorem 7.6 ([MERGE-DO]) Let Ra({D_{r} | r ∈ Ω}) be an α-competitive randomized algorithm for our problem. Then α ≥ 1 + \frac{1}{e}.

This lower bound suggests that it is indeed worthwhile to seek algorithms that are very competitive, despite the potentially high bounds on “absolute” work. Armed with this hope we analyze the competitive performance of randomized schedules.

As mentioned, we consider the natural randomized algorithm RS where a processor (or group) with knowledge that the tasks in a set K ⊂ [t] have been completed selects to next complete a task at random from the set [t]\K. More formally, let Π = (π_{1}, …, π_{n}) be a (n, t)-schedule (where each π_{i} is a permutation of [t]). We describe a deterministic algorithm D_{Π} so that RS = Ra({D_{Π} | Π ∈ (S_{t})^{n}}). Let G be a group of processors and γ ∈ G the processor in G with the lowest processor identifier. Then the deterministic algorithm D_{Π} specifies that the group G, should it know that the tasks in K ⊂ [t] have been completed, next completes the first task in the sequence π_{γ} which is not in K.

It turns out, not unexpectedly, that this algorithm is optimal with respect to the lower bound on competitive ratios.

Theorem 7.7 ([MERGE-DO]) Algorithm RS is (1 + \frac{1}{e})-competitive for any given pattern of rendezvous.

Currently we have obtained preliminary results extending this competitive analysis to arbitrary patterns of merges and fragmentations.

7.2 Derandomization via finite geometries

We now consider a method for derandomizing these schedules using the design discussed in Section 6.1.

Schedules for n = t. We construct a system of schedules of length n by arranging tasks from the sequences of L_{n} in a recursive fashion. (Recall that while the sequences of L_{n} have strong intersection properties, they are only roughly \sqrt{n} in length.) In preparation for the recursive construction, we record the following lemma about the pairwise intersections of the elements in the sequence of L_{n} indexed by a specific subspace L_{u}.

Lemma 7.8 Let L_{n} = ⟨L_{1}, …, L_{n}⟩ be the collection of sequences constructed in Theorem 6.3, and let L_{u} = ⟨t_{u}^{1}, …, t_{u}^{q+1}⟩, 1 ≤ u ≤ n. Then for any i ≠ j, we have L_{t_{u}^{i}} ∩ L_{t_{u}^{j}} = {u}.

As a result of this lemma, there is only a single repeated element in the sequences L_{t_{u}^{1}}, L_{t_{u}^{2}}, …, L_{t_{u}^{q+1}}; this element is u. This fact suggests the following construction of a system of schedules P_{n}. Let P_{u}, 1 ≤ u ≤ n, be the sequence whose first element is u, and whose remaining elements are given by concatenating the q + 1 sequences L_{t^{1}u}, …, L_{t_{u}^{q+1}} after removing u from each. Specifically,

P_{u} = ⟨u⟩ ∘ (◯_{i∈L_{u}}(L_{i} − u)),

where ∘ denotes concatenation and L_{i} − u denotes the sequence L_{i} with u deleted. Note now that since the total length of P_{u} is evidently (q + 1)q + 1 = n, each element of [n] must appear exactly once in each P_{u}; these P_{u} thus rive rise to a family of permutations π_{u}, where π_{u}(k) is the k element of P_{u}. Let P = (π_{1}, …, π_{n}).

We conceptually divide the sequences P_{u} (associated with the permutations π_{u}) into q +1 segments of elements. The first segment contains the first q + 1 elements (including the initial element u); the remaining q segments contain q consecutive elements each.

This recursive construction yields a straightforward bound on pairwise waste, recorded below.

Theorem 7.9 Let q be a prime power, n = q^{2} + q + 1. Let a = 1 + iq, b = 1 + jq, 0 ≤ i, j ≤ q + 1. Then

W_{P_{n}}(a, b) ≤ \begin{cases}0, & i + j = 0, \\ 1, & i = 0, j ≥ 1 or i ≥ 1, j = 0, \\ q + ij, & i · j ≥ 1. \end{cases}

We mention that the construction can be done on-line. For each schedule the first element can be calculated in O(1) time. For the remaining q(q + 1) elements, at the beginning of every sequence of q elements we need to invert at most two elements in GF(q). When q is prime this can be done in O(log n) using the extended Euclidean algorithm. Other elements of the schedule can be found in O(1) time.

Note that when t = κn for some κ ∈ ℕ, the above construction can be trivially applied by placing the t tasks into n chunks of size κ. In this case, of course, when a single overlap occurred in the original construction, this penalty is amplified by κ.

One disadvantage of P_{n} is that the first segment may repeat, so that (q + 1) waste may be incurred when a prefix of length â = (q+1) is executed. To postpone this increase one would like to rearrange the segments in each P_{u} so that the first segment is distinct across the resulting schedules. This can be accomplished by finding a permutation ρ : [n] → [n] such that the sequence L_{u} contains task ρ(u). (In other words ℓ_{u} must be orthogonal to ℓ_{ρ(u)}.) This permutation can then be used to select distinct segments as the first segments of schedules in P_{n}.