\documentclass[11pt,twoside]{article}
%\usepackage{/a/class/6.895/fall02/lectures/macros-course-lectures}
\usepackage{macros-course-lectures}
%\renewcommand{\bar}[1]{\overline{#1}}

\begin{document}

\newcommand{\DA}{{\it Do-All\/}}
\newcommand{\WA}{{\it Write-All\/}}
\newcommand{\Ana}[2]{#2}
\newcommand{\Inx}[1]{#1}
\newcommand{\Inv}[2]{#1 #2}
\newcommand{\Inxx}[2]{}
\newcommand{\DAP}[1]{\DA$(#1)$}
\newcommand{\lan}{\langle}
\newcommand{\ran}{\rangle}
\newcommand{\T}{{\mathcal T}}
\newcommand{\DAO}[1]{\DA\/$^{\cal O}(#1)$}
\newcommand{\F}{\vspace*{\smallskipamount}}
\newcommand{\FF}{\vspace*{\medskipamount}}
\newcommand{\FFF}{\vspace*{\bigskipamount}}
\newcommand{\B}{\vspace*{-\smallskipamount}}
\newcommand{\BB}{\vspace*{-\medskipamount}}
\newcommand{\BBB}{\vspace*{-\bigskipamount}}
\newcommand{\VX}{\vspace*{1mm}}
\newcommand{\Forall}{{\kf forall~}}
\newcommand{\Processors}{{\kf processors~}}
\newcommand{\PID}{{\sc pid}}
\newcommand{\Parbegin}{{\kf parbegin~}}
\newcommand{\HT}{\hspace*{2em}}
\newcommand{\Parend}{{\kf parend}}
\newcommand{\La}[2]{{\Lambda}_{#2}^{#1}}
\newcommand{\OM}{\left\{ O|\mathit\Omega \right\}}
\newcommand{\Wmega}{\mathit\Omega}
\newcommand{\Thita}{\mathit\Theta}
\newcommand{\rDA}[2]{$#1${\sl -Do-All}$(#2)$}
\newcommand{\Item}{\item}
\newcommand{\rDO}[2]{$#1${\sl -Do-All\/}$^{\cal O}(#2)$}
\newcommand{\CITE}[1]{{\small\sf [#1]}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\norm}[1]{\|#1\|}
\renewcommand{\vec}[1]{{\bf #1}}
\newcommand{\e}[1]{{\bf e}_{\boldsymbol{#1}}}\newcommand{\GF}{\textrm{GF}}
%\newcommand{\card}[1]{\left| #1 \right|}
\newcommand{\waste}{\mathcal{W}}
\newcommand{\LB}{\textrm{LB}}
\newcommand{\UB}{\textrm{UB}}
\newcommand{\tup}[1]{\langle #1 \rangle}
\newcommand{\setst}[2]{{\left\{#1\;\vrule\;#2\right\}}}
\newcommand{\Exp}[1]{\Expsymbol\left[#1\right]}
\newcommand{\Expsymbol}{\mathop\mathsf{Exp}}
\newcommand{\Varsymbol}{\mathop\mathsf{Var}}
%\newcommand{\Var}[1]{\Varsymbol\left[#1\right]}
\newcommand{\Cov}[1]{\Covsymbol\left[#1\right]}
\newcommand{\Covsymbol}{\mathop\mathsf{Cov}}
\font\kf=cmssbx10 at 9pt

\lecture{notes14}{Performing Work in Distributed Systems}{October 24, 2002}

\begin{tabbing}
XXX\=XXX\=XXX\=XXX\= \kill
Notes: Initial notes gathered and cobbled by Alex A. Shvartsman
\end{tabbing}

\noindent
Today's reading: \\
\begin{itemize}
\item  G. G. Malewicz, A. Russell, and A. A. Shvartsman. Distributed cooperation in the absence of communication. In Proceedings of the 14th International Conference on Distributed Computing, DISC,  2000. .ps

\item C. Georgiou, A. Russell, and A.A. Shvartsman. The complexity of synchronous iterative do-all with crashes. Distributed algorithms (DISC 2001). .ps 
Next:  TBD: 
\end{itemize}

\section{Introduction}

Performing a set of tasks in a decentralized setting is a fundamental
problem in distributed computing. This is often challenging
because the set of processors available to the computation and their
ability to communicate may dynamically change due to perturbations
in the computation medium.
An abstract statement of this problem, referred to as the
\DA{} problem
--- {\em $P$ fault-prone processors perform $N$ independent tasks\/}
--- is one of the standard problems in the research on the complexity
of fault-tolerant distributed computation~\CITE{DHW-SIAM98,KS-DC92}.
Variations of this problem have been studied in
% a variety of settings, e.g., in
shared-memory models (\WA{}\/\footnote{%
%
The \DA{} problem and the \WA{} problem are subtly different.
The main difference is that in \DA{} the tasks may be supplied
to the processors from some external sources, while in \WA{}
the tasks are stored in shared-mmeory accessible to all processors.
In some contexts it is possible to abstract away from the sources
and the nature of the tasks and to treat \DA{} and \WA{} as the same problem.
The distinction between the problems is reinstated when
one applies the results of such abstract analysis to specific
message-passing and shared-memory algorthms.
}%
%
)~\CITE{KS97,KPRS-STOC91,MS-JA94}, in
message-passing models~\CITE{DMY-PODC94,DHW-SIAM98,GMY-FOCS95}, and 
in partitionable
networks ({\sl Omni-Do\/})~\CITE{DSS-SIR99,GS-JDA,MRS-DISC00}.
Solutions for \DA{} must perform all tasks efficiently in the presence
of specific failure patterns. The efficiency is assessed in terms
of work, time and communication complexity depending on the
specific model of computation.


In the design of practical distributed/parallel programs one needs
to ensure good performance and dependability under unpredictable
load patterns caused by deviations from synchrony
or by the failures of some processors to complete tasks on time.
Here again, a common challenge is to perform $N$ independent
tasks on $P$ processors~\CITE{HG-DC01}. Such tasks could be copying
a large array, searching a collection of data, or
applying a function to all elements of a matrix~\CITE{GHVM-DC01}.

Computation and communication are two basic resource commodities in
distributed computing.  In a distributed system setting one normally
computes to perform specific tasks. One also needs to communicate in
order to coordinate multiple computation activities, for example, to
increase computation efficiency by eliminating redundant work.  As can
be expected, there exists a trade-off between the efficiency of
distributed computation and the amount of communication needed to
coordinate the computation. Dwork, Halpern and Waarts \CITE{DHW} have
considered a basic problem of distributed coordination, where a number
of message-passing processors cooperate in executing a collection of
independent and similarly-sized tasks (in the presence of processor
crashes).  They measure the efficiency of the computation in terms of
its {\em work\/} complexity $W$ (one task consumes one work unit), and
the cost of communication in terms of the {\em message\/}
complexity $M$.  The authors also assess the overall performance in
terms of {\em effort\/} that is defined as the sum of work $W$ and
message complexity $M$. 
%Whereas \cite{DHW} measures specifically the task-oriented work, 
Chlebus {\em at al.}~\CITE{CGKS} solve a similar
problem and they measure the effort of their solution as the sum of
the number of all processing steps performed (this includes
task-oriented work and any bookkeeping steps) and the number of
messages sent. They observe that a processor can learn that a task is
performed either by doing the task, or by receiving a message that
tells it that the task was done. In such a setting, one can view
computation and communication as comparable resources.

\section{The Write-All / Do-All Primitive
% and Robust Computation
}\label{SecWA}

We define Do-All to be the following problem:

\begin{definition}
Given a set of $N$ independent and idempotent tasks
$A[1..N]$ and $P$ processors, perform each task at least once
\end{definition}

Solving Do-All when $P=N$ is one of the simplest computations 
that can be performed by a distributed system with $P$ processors
The following optimal synchronus program solves Do-All
in constant time, thus achieving linear speed-up.

\begin{center}
{\small
\framebox[0.75\textwidth][c]{%
\parbox[c]{0.6\textwidth}{%
\VX
\Forall \Processors \PID${}=1..N$ \Parbegin \\
\HT   {\kf perform} $A[$\PID{}$]$ \\
\Parend \VX
} } }
\end{center}

Next suppose that $P\ne N$. It easy to modify the program above so that
each processor performs $\lceil N/P \rceil$ tasks. (Padding is used
to include ``empty" tasks when $P$ does not divide $N$.) 

\begin{center}
{\small
\framebox[0.75\textwidth][c]{%
\parbox[c]{0.6\textwidth}{%
\VX
\Forall \Processors \PID{}$=1..P$ \Parbegin \\
\HT  \For{} $i=0$ {\kf to} $\lceil N/P \rceil - 1$ {\kf do} \\
\HT\HT   {\kf perform} $A[$\PID{}$ + i*P]$ \\
\HT  {\kf rof}\\
\Parend \VX
} } }
\end{center}

The above algorithm takes time $T_{N,P} = \Theta(N/P)$
and its cost (work) is $C = P\cdot T_{N,P} = P\cdot\Theta(N/P) = \Theta(N)$.
Thus this solution is also optimal, and in particular, it achieves 
linear speed-up: $S = \Theta(N)/T_{N,P} = \Theta(N)/\Theta(N/P)
 = \Theta(P)$.

In shared-memory models, solving Do-All is equivalent (under some
simple assumptions) to solving the following, more abstract problem,
called Write-All.

\begin{definition}\label{WAP}
Given a zero-valued array of $N$ elements and $P$ 
processors,
% subject to failures defined by a model {\bf F}, 
the {\em Write-All\/} problem is to set each element of 
the array to 1.
% \hfill $\Box$
\end{definition}

The {\em Write-All} problem captures the computational progress 
that can be naturally accomplished in unit time by
$N$ synchronous processors in the absence of failures.
Solving Do-All or Write-All efficiently in the presence of asynchrony
and failures is very challenging.

\paragraph{\bf Measuring efficiency.}
We are interested in studying the complexity of \DA{}
measured as \emph{work} (cf.~\CITE{KS-DC92,DHW-SIAM98,DMY-PODC94}).
We assume that it takes a unit of time for a processor
to perform a unit of work,
%according to its local clock,
and that a single task
corresponds to a unit of work.
Our definition of {\em work complexity\/}
is based on the {\em available processor steps\/}
measure~\CITE{KS97}.
Let ${\mathcal F}$ be the failure model.
For a computation subject to a failure pattern $F$, $F \in{\mathcal F}$,
where $\mathcal F$ is a failure model,
denote by $P_i(F)$ the number of processors completing a unit of work
in step $i$ of the computation.


\begin{definition}\label{def:work}
Given a problem of size $N$ and a $P$-processor algorithm that
solves the problem in the failure model $\cal F$, if the algorithm
solves the problem for a pattern $F$ in $\cal F$, with $|F|\leq f$,
by time step $\tau(F)$, then the work complexity $W$ of
the algorithm is: $$W_{N,P,f} = \max_{F\in{\mathcal F},~|F|\leq f}
\left\{\sum_{i\leq \tau(F)} P_i(F)\right\}.$$
\end{definition}


The idling processors consume a unit of work per
step even though they do not contribute to the computation.
Definition~\ref{def:work} does not depend on the
specifics of the target model of computation, e.g.,
whether it is message-passing or shared-memory.
(Communication complexity is defined similarly
in Section~\ref{sec:ALG}.)

Some research on Do-All in message-passing models uses a different
definition of work, where processors are not charged for time spent idling
or waiting for a message \CITE{DHW-SIAM98}.

\paragraph{Do-All in message-passing models}

\begin{center}
%\begin{Def}\label{RefDA}
{\sl Perform a set of $N$ tasks 
%$\{ T_1, T_2, \ldots , T_N \}$, 
using $P$~fault-prone processors.
}
%\end{Def}
\end{center}

This problem was considered for the message-passing model
by \Ana{C.}{Dwork}, \Ana{J.}{Halpern} and \Ana{O.}{Waarts}
in their pioneering work~\CITE{DHW92}.  They developed several
efficient algorithms for \DA{} in the setting where the
processors are subject to fail-stop (or crash) failures
and where the tasks can be performed using the {\em at-least-once}
execution semantics (i.e., the tasks can be made idempotent).
Their model of computation assumes a synchronous
system of processors, such that in one time unit a processor
can perform a local computation and complete one task, and
perform one round of communication consisting of sending
and receiving messages.

Algorithmic techniques for distributed \DA{} are motivated
with the help of an observation that there is a qualitative duality between the
shared-memory and the message-passing settings.
In a shared-memory algorithm, a key datum, say reflecting an
important component of the state of the computation,
can be stored in a shared memory
cell, and $P$ processors can learn the value stored by performing
$P$ reads. In a message-passing algorithm, the only way to propagate 
information, stored anywhere, to $P$ processors, is to do so by means 
of at least $P-1$ messages. 
Thus in message-passing algorithms, one sees a
multicast or diffusion of a value to a subset of processors 
as a dual of these
processors reading the value from a location in shared memory.

For the distributed message-passing models, one has to consider
the costs of a computation in terms of the total work performed
by the processors, and in terms of the number of messages
sent by the processors.
In the shared-memory models we have covered thus far, the 
algorithm efficiency was determined primarily by the
{\em available processor steps\/} (or work) measure. Since in the worst
case each instruction involves a constant number of memory accesses, 
work is also the measure of the number of memory accesses 
(i.e., the communication cost) performed by the computation.
The one-to-one cost correspondence between work and communication
does not apply to the message-passing models, since the number of
messages sent and received by one processor in one round of 
communication can be 0, or it can be
comparable to $P$ when broadcasts are enabled.

To measure the communication efficiency of distributed algorithms,
the number of messages sent by the processors is counted. One can also
consider the total number of bits sent when the messages can vary
in size or can be large. Measuring the work of a computation
depends on the interpretation of the relative costs associated
with different processing steps. Specifically, one needs to consider
the costs associated with local processing and the costs
associated with doing the tasks in solving the \DA{} problem.

The efficiency of message-passing algorithms is characterized in terms
of their work and message complexity.
We define message complexity
similarly to Definition~\ref{def:work} of work:
%We now define the message complexity $M$. Let ${\mathcal F}$ be
%the set of allowed failure patterns, that is, the failure model
%considered.
For a computation subject to a failure pattern $F$, $F
\in{\mathcal F}$, denote by $M_i(F)$ the number of point-to-point
messages sent during step $i$ of the computation. For a given
problem of size $N$, if the computation solves the problem by step
$\tau(F)$ in the presence of the failure pattern $F$, where $|F|\leq f$,
 then the message complexity $M$ is:
$$M_{N,P,f} = \max_{F\in{\mathcal
F},~|F|\leq f} \left\{\sum_{i\leq \tau(F)} M_i(F)\right\}.$$


\section{Dwork Halpern Waarts}

\begin{quote}
{Optimizing Task-Oriented Work}
\end{quote}

In the setting of \Ana{C.}{Dwork}, \Ana{J.}{Halpern} and 
\Ana{O.}{Waarts}~\CITE{DHW92}, the local computation, whether
performing low-level administrative tasks (such as
maintaining the \Inv{local}{clock}) or idling, is considered to
be negligible compared to the costs of performing the
\DA{} tasks, where processors may need to do constant size,
but significant processing, or the costs of communication with external
systems. This approach is consistent with the accepted accounting
of costs in distributed algorithms. Additionally,
\Ana{C.}{Dwork}, \Ana{J.}{Halpern} and \Ana{O.}{Waarts}
consider {\em \Inx{process}es\/}, a notion that can be more general
than {\em processors\/}. For example, when we interpret processes
as {\em \Inx{threads} of control\/}, such that several threads may
co-exist on the same processor, then the idle ``steps'' taken
by threads need not be accounted for, since during such idle
steps the processor hosting the thread can be executing other steps 
on behalf of other threads. Thus idle thread steps do not constitute
a waste of the computational resources, and it is reasonable to
account only for work spent on the \DA{} tasks.

\Ana{C.}{Dwork}, \Ana{J.}{Halpern} and \Ana{O.}{Waarts}
define the {\em \Inx{effort}\/} of an algorithm as the sum of 
its work and \Inx{message complexity}. 
This is done under the assumption that the cost of
performing a unit of work and sending a message are comparable.
They observe that at least $P+N-1$ units of work are necessary
to solve the \DA{} problem of size $N$ using $P$ processors.
The goal then is the development of algorithms that perform
$O(N+P)$ work, thus achieving work-optimality, while keeping
the total effort reasonable. In this synchronous model, the
running time of an algorithm is also a meaningful measure of
efficiency, and in the analysis of algorithms this measure
is also considered. The first two algorithms\Inxx{algorithm,sequential}
are sequential ---
at any point in time only one task is being performed,
thus in these algorithms the redundancy of having multiple processors
contributes to fault-tolerance, but not to speed-up. The third
algorithm achieves fault-free time optimality, with graceful
degradation of running time in the presence of failures.

One algorithm presented by the authors (protocol $\cal B$)
has effort $O(N + P\sqrt{P})$, with work contributing
the cost $O(N+P)$ towards the effort, and message complexity
contributing the cost $O(P\sqrt{P})$.
The running time of the algorithm is $O(N+P)$.
The use of synchrony in the algorithm is limited to failure
detection by means of time-outs. Such algorithms can be converted
to asynchronous algorithms when the underlying system is equipped
with \Inx{failure detectors}.
%
In this algorithm the $N$ tasks are divided into {\em chunks} and
each of these is divided into {\em subchuncks\/}.  Processors
checkpoint their progress by multicasting the completion information
to subsets of processors after performing a subchunk, and
broadcasting to all processors after completing chunks of work.

Another algorithm in \CITE{DHW92} (protocol $\cal C$) has effort
$O(N+P\log P)$. This includes optimal work of $O(N+P)$, message
complexity of $(P\log P)$, and time $O(P^2(N+P)2^{N+P})$.
Thus the reduction in message complexity is traded-off for a
significant increase in time.

The third work-optimal algorithm (protocol $\cal D$) is designed 
for maximum
speed-up, which is achieved with a more aggressive checkpointing
strategy, thus trading-off time for messages. The message complexity
is quadratic in $P$ for the fault-free case, and in the presence of
a failure pattern $F$, the message complexity degrades to
$\Theta(|F|\cdot P^2)$.

One interesting application of the \DA{} algorithms is to
solving Byzantine agreement with crash failures. 
\Ana{C.}{Dwork}, \Ana{J.}{Halpern} and \Ana{O.}{Waarts} show that a \DA{}
algorithm with $P$ processors and $N$ tasks can be used
to solve agreement for $N$ processors and less than $P$
failures~\CITE{DHW92}. In this application, the tasks performed
by processors are sending messages to other processors
informing them of the agreement value.


\newcommand{\cO}{O}

\section{Chlebus De Prisco Shvartsman}

Two new algorithms are presented and analyzed: AN and AR.
All previous algorithms do not consider the possibility that a
faulty processor is repaired and reintegrated into the system.
\CITE{CDS}
introduces a new algorithmic technique based on an aggressive
coordination paradigm that permits multiple concurrent coordinators.
This approach is suggested by the earlier observation that algorithms
with only one coordinator cannot deal efficiently with restarts.  The
number of coordinators is managed adaptively. When failures of
coordinators disrupt the progress of the computation, the number of
coordinators is increased; when the failures subside, a single
coordinator is appointed.



\begin{figure}[ht]
\begin{center}
\begin{tabular}{||c|c|c|c||}
\cline{3-4}
\multicolumn{2}{r}{~}
& \multicolumn{1}{|c|}{$W${\sf : available processor steps (work)}}
& \multicolumn{1}{c||}{$M${\sf : message complexity}}\\
\cline{3-4}
\cline{1-4}
\sf No & \CITE{DMY} & $\cO(t+(f+1)p)$ & $\cO((f+1)p)$\\
\cline{2-4}
\sf restarts & \CITE{GMY} & $\cO(t + (f+1) p)$ & $\cO(f p^\varepsilon + \min\{f+1, \log p\}p)$\\
\cline{2-4}
$(f < p)$ & AN
& $\cO((t + p\log p/\log\log p)\log f)$ 
& $\cO(t + p\log p/\log\log p + fp)$\\
\hline
\hline
\parbox{4.5em}{\centering ~\\\sf Restarts $(f < p+r)$} & AR
& $\cO((t + p\log p + f)\cdot \min\{\log p,\log f\})$
& $\cO(t+p\log p+ f p)$\\
\hline
\end{tabular}
\end{center}
\caption{Efficiency of the solutions in~\CITE{DMY,GMY}
and algorithms AN and AR
(the solutions in~\CITE{DHW}
consider a different notion of work complexity and
focus on evaluation of effort).}
\label{Table}
\end{figure}

\subsection{Algorithm AN}

Algorithm AN presented by Chlebus \emph{et al.}~\CITE{CDS-DC01}
uses a multiple-coordinator approach to
solve \DAP{N,P,f} on crash-prone synchronous
message-passing processors ($P\leq N$).
The model assumes that messages incur a known bounded delay
and that reliable multicast~\CITE{HT93} is available, however
messages to/from faulty processors may be lost.


\subsubsection{Description of the Algorithm}
%\medskip
%\noindent
%{\bf Description of algorithm AN.}
%
We now give a brief
description of the algorithm; additional details
are given in Appendix A (but to avoid a complete restatement,
we refer the reader to~\CITE{CDS-DC01}).
Algorithm AN proceeds in a {\em loop\/} which is
iterated until all the tasks are executed. A single iteration of
the loop is called a {\em phase}. A phase consists of three
consecutive {\em stages}. Each stage consists of three steps. In
each stage processors use the first step to receive messages sent
in the previous stage, the second step to perform local
computation, and the third step to send messages. A processor can
be a {\em coordinator\/} or a {\em worker\/}. A phase may have
multiple coordinators. The number of processors
that assume the coordinator role is determined by the {\em
martingale principle\/}: if none of the expected coordinators
survive through the entire phase, then the number of coordinators
for the next phase is doubled. If at least one coordinator
survives in a given phase, then in the next phase there is only
one coordinator. A phase that is completed with at least one
coordinator alive is called {\em attended\/}, otherwise
it is called {\em unattended\/}.


Processors become coordinators and balance their loads
according to each processor's {\em local view\/}.
A local view contains the set of ids of the processors
assumed to be alive. The local view is partitioned into
{\em layers\/}. The first layer contains one processor id, 
the second two ids, the $i^{th}$ contains $2^{i-1}$ ids.


Given a phase, in the first stage, the processors perform a task
according to the load-balancing rule derived from their local
views and report the completion of the task to the coordinators of
that phase (determined by their local views). In the second stage,
the coordinators gather the reports, they update the knowledge of
the done tasks and they multicast this information to the
processors that are assumed to be alive. In the
last stage, the processors receive the information sent by the
coordinators and update their knowledge of done tasks and their
local views. 

It is shown in~\CITE{CDS-DC01} that the work of algorithm AN
is $W = O((N + N\log N /$ $\log\log N)\log f)$ and its message
complexity is $M = O(N+ P\log P/\log\log P + f P)$, for $P\leq N$.

In the rest of this section we present the new analysis of
work and message complexity of algorithm AN. Throughout we assume
that the algorithm correctness is shown as in~\CITE{CDS-DC01}.

\subsubsection{Algorithmic details}

The following information about algorithm AN is taken
from ~\CITE{CDS-DC01}. More information can be obtained there. The
algorithm proceeds in a {\em loop\/} which is repeated until all
the tasks are executed. A single iteration of the loop is called a
{\em phase}. A phase consists of three consecutive {\em stages}.
Each stage consists of three steps (thus a phase consists of 9
steps). In each stage processors use the first step to receive
messages sent in the previous stage, the second step to perform
local computation, and the third step to send messages. We refer
to these three step as the {\em receive\/} substage, the {\em
compute\/} substage and the {\em send\/} substage.


\medskip


\noindent {\bf Coordinators and workers.}  A processor can be a
{\em coordinator\/} of a given phase.  All processors (including
coordinators) are {\em workers\/} in a given phase. Coordinators
are responsible for recording progress, while workers perform
tasks and report on that to the coordinators.  In the first phase
one processor acts as the coordinator.  There may be multiple
coordinators in subsequent phases.  The number of processors that
assume the coordinator role is determined by the {\em martingale
principle\/}: if none of the expected coordinators survive through
the entire phase, then the number of coordinators for the next
phase is doubled. Whenever at least one coordinator survives a
given phase, the number of coordinators for the next phase is
reduced to one.


If at least one processor acts as a coordinator during a phase and
it completes the phase without failing, we say that the phase is
{\em attended\/}, the phase is {\em unattended\/} otherwise.


\medskip


\noindent {\bf Local views.} Processors assume the role of
coordinator based on their local knowledge.  During the
computation each processor $w$ maintains a list $L_w=\lan
q_1,q_2,...,q_k\ran$ of supposed live processors. We call such
list a {\em local view\/}. The processors in $L_w$ are partitioned
into {\em layers\/} consisting of consecutive sublists of $L_w$:
$L_w=\lan\Lambda^0,\Lambda^1,...,\Lambda^{j}\ran$.
%\footnote{%
%For sequences $L=\lan e_1, \ldots , e_n\ran$ and $K=\lan d_1, \ldots ,
%d_m\ran$ we define $\lan L, K\ran$ to be the sequence $\lan e_1,
%\ldots , e_n, d_1, \ldots , d_m\ran$.}.
The number of processors in layer $\Lambda^{i+1}$, for
$i=0,1,...,j-1$, is the double of the number of processors in
layer $\Lambda^{i}$. Layer $\Lambda^{j}$ may contain less
processors.
%
When $\Lambda^0 = \lan q_1\ran$ the local view can be visualized
as a
%complete :RDP taking ``complete'' out
binary tree rooted at processor $q_1$, where nodes are placed from
left to right with respect to the linear order given by $L_w$.
Thus, in a tree-like local view, layer $\Lambda^0$ consists of
processor $q_1$, layer $\Lambda^i$ consists of $2^i$ consecutive
processors starting at processor $q_{2^i}$ and ending at processor
$q_{2^{i+1}-1}$, with the exception of the very last layer that
may contain a smaller number of processors.  Processors in a local
view do not necessarily appear in the order of processor
identifiers.


The local view is used to implement the martingale principle of
appointing coordinators as follows. Let $L_{\ell,w}=\lan
\Lambda^0,\Lambda^1,...,\Lambda^{j}\ran$ be the local view of
worker $w$ at the beginning of phase $\ell$.  Processor $w$
expects processors in layer $\Lambda^0$ to coordinate phase
$\ell$; if no processor in layer $\Lambda^0$ completes phase
$\ell$, then processor $w$ expects processors in layer $\Lambda^1$
to coordinate phase $\ell+1$; in general processor $w$ expects
processors in layer $\Lambda^i$ to coordinate phase $\ell+i$ if
processors in all previous layers $\Lambda^k$, $\ell \leq k<
\ell+i$, did not complete phase $\ell+k$. The local view is
updated at the end of each phase.


\medskip


\noindent {\bf Phase structure and task allocation.}  The
structure of a phase of the algorithm is as follows.  Each
processor $w$ keeps its local information about the set of tasks
already performed, denoted $D_w$, and the set of live processors,
denoted $P_w$, as known by processor $w$.  Set $D_w$ is always an
underestimate of the set of tasks actually done and $P_w$ is
always an overestimate of the set of processors that are
``available" from the start of the phase. We denote by $U_w$ the
set of {\em unaccounted\/} tasks, i.e., whose done status is
unknown to $w$. Sets $U_w$ and $D_w$ are related by $U_w= \T
\setminus D_w$, where $\T$ is the set of all the tasks.  Given a
phase $\ell$ we use $P_{\ell,w}$, $U_{\ell,w}$ and $D_{\ell,w}$ to
denote the values of the corresponding sets at the beginning of
phase $\ell$.


Computation starts with phase $0$ and any processor $q$ has all
processors in $L_{0,q}$ and has $D_{0,q}$ empty. At the beginning
of phase $\ell$ each worker (that is, each processor) $w$ performs
one task according to its local view $L_{\ell,w}$ and its
knowledge of the set $U_{\ell,w}$ of unaccounted tasks, using the
following {\em load-balancing rule}.  Worker $w$ executes the task
whose rank is $(i\ {\tt mod}\ |U_{\ell,w}|)^{th}$ in the set
$U_{\ell,w}$ of unaccounted tasks, where $i$ is the rank of
processor $w$ in the local view $L_{\ell,w}$.  Then the worker
reports the execution of the task to all the processors that,
according to the worker's local view, are supposed to be
coordinators of phase $\ell$. For simplicity we assume that a
processor sends a message to itself when it is both worker and
coordinator. Any processor $c$ that, according to its local view,
is supposed to be coordinator, gathers reports from the workers,
updates its information about $P_{\ell,c}$ and $U_{\ell,c}$ and
broadcasts this new information causing the local views to be
reorganized. In~\CITE{CDS-DC01}  it is shown that at the beginning of
any phase $\ell$ all live processors have the same local view
$L_{\ell}$ and the same set $U_{\ell}$ of unaccounted tasks and
that accounted tasks have been actually executed.  A new phase
starts if $U_{\ell}$ is not empty.


The following is a detailed description of a phase.\\


{\bf Phase $\ell$ of algorithm AN:}
%
\begin{quote}
\begin{description}
\item[{\sc stage 1.}]
The receive substage is not used. In the compute substage, any
processor $w$ performs a specific task $z$ according to the load-balancing
rule. 
In the send substage processor $w$ sends a
{\tt report}$(z)$ to any coordinator, that is, to any processor
in the first layer of the local view~$L_{\ell,w}$.


\item[{\sc stage 2.}]
In the receive substage the coordinators gather {\tt report}
messages.  For any coordinator $c$, let ${z}_c^1, \ldots
,{z}_c^{k_c}$ be the set of \textsc{tid}s received.  In the compute
substage $c$ sets $D_c\gets D_c\cup \bigcup_{i=1}^{k_c}
\{{z}_c^i\}$, and $P_c$ to the set of processors from which $c$
received {\tt report} messages.  In the send substage,
coordinator $c$ multicasts the message {\tt summary}($D_c,P_c$)
to processors in~$P_c$.


\item[{\sc stage 3.}] During the receive substage {\tt summary}
messages are received by live processors.  For any processor $w$,
let $(D_w^1,P_w^1), \ldots, (D_w^{k_w},P_w^{k_w})$ be the sets
received in {\tt summary} messages In the compute substage $w$
sets $D_w\gets D_w^i$ and $P_w\gets P_w^i$ for an
arbitrary $i\in\{1, \ldots ,k_w\}$ and updates its local
view~$L_w$ as described below. The send substage is not used.
\end{description}
\end{quote}


\noindent {\bf Local view update rule.}  In phase 0 the local view
$L_{0,w}$ of any processor $w$ is a tree-like view containing all
$P$ processors ordered by their \textsc{pid}s. Let $L_{\ell,w}=\lan
\Lambda^0,\Lambda^1,...,\Lambda^{j}\ran$ be the local view of
processor $w$ for phase $\ell$.  We distinguish two possible
cases.


{\sc Case 1.} Phase $\ell$ is unattended. Then the local view of
processor $w$ for phase $\ell+1$ is $L_{\ell+1,w}=\lan
\Lambda^1,...,\Lambda^{j}\ran$.


{\sc Case 2.} Phase $\ell$ is attended. Then processor $w$
receives {\tt summary} messages from some coordinator in
$\Lambda^0$. Processor $w$ computes its set $P_w$ as described in
stage 3 (we will see that all processors compute the same set
$P_w$).  The local view $L_{\ell+1,w}$ of $w$ for phase $\ell+1$
is a tree-like local view containing the processors in $P_w$
ordered by their \textsc{pid}s.


\medskip


Finally, we demonstrate how a computation of algorithm AN
proceeds. Given an execution of the algorithm, we enumerate the
phases.  We denote the attended phases of the execution by
$\alpha_1,\alpha_2, \ldots,$ etc. We denote by $\pi_i$ the
sequence of unattended phases between the attended phases
$\alpha_i$ and $\alpha_{i+1}$. We refer to $\pi_i$ as the $i^{th}$
(unattended) period; an unattended period can be empty. Hence the
computation proceeds as follows: unattended period $\pi_0$,
attended phase $\alpha_1$, unattended period $\pi_1$, attended
phase $\alpha_2$, and so on.

\newcommand{\AN}{AN}
\newcommand{\AR}{AR}

\subsection{Algorithm \AR}

Algorithm \AR\ is similar to algorithm \AN; the difference is that
there are added messages to handle the restart of processors. After
the restart, processor $q$ broadcasts {\tt restart}($q$) messages in
each step until it receives a response. Processors receiving such
messages, ignore them if these messages are not received in the
receive substage of stage 2 of a phase. Thus we can imagine that a
restarted processor $q$ broadcasts a
{\tt restart}($q$) in the send substage of stage~1 of a phase $\ell$
(however we will count all the {\tt restart} messages in the message
complexity). This message is then received by all the live and
restarted processors of that phase, and, as we will see shortly,
processor $q$ is re-integrated in the view for phase
$\ell+1$. Processor $q$ needs to be informed about the status of the
ongoing computation. Hence processors that have this information send
the {\tt info}$(U_{\ell},L_{\ell})$ messages
to processor $q$ with the set
$U_{\ell}$ of unaccounted tasks and the local view~$L_{\ell}$.  Next
we provide the detailed description for each phase. 
The parts that are new or
that are different in algorithm \AR\ as compared to algorithm \AN\ 
are {\em italicized\/}.


\newcommand{\MK}[1]{{\em #1}}

\bigskip

{\bf Phase $\ell$ of algorithm \AR:}
%
\begin{quote}
\begin{description}
\item[{\sc stage 1.}]
The receive substage is not used. In the compute substage any processor
$w$ performs a specific task $z$ according to the load balancing
rule. In the send substage $w$ sends a
{\tt report}$(z)$ to any coordinator, that is, to any processor
in the first layer of $L_{\ell,w}$.  
\MK{Any restarted processor $q$
broadcasts the {\tt restart}($q$) message informing all live
processors of its restart.}

\item[{\sc stage 2.}]
In the receive substage the coordinators gather {\tt report}
messages 
\MK{and
all processors gather {\tt restart} messages. Let $R$ be the set
of processors that sent a {\tt restart} message.}
For any coordinator
$c$, let $z_c^1,...,z_c^{k_c}$ be the set of TIDs received in
{\tt report} messages.  In the
compute substage $c$ sets $D_c\leftarrow D_c\cup\bigcup_{i=1}^{k_c}
\{z_c^i\}$ and $P_c$ to the set of processors from which $c$ received
{\tt report} messages.  In the send substage, coordinator $c$
multicasts the message {\tt summary}($D_c,P_c$) to the processors in
$P_c$ \MK{and $R$.} 
\MK{Any processor in $P_c$ sends the message
{\tt info}($U_{\ell},L_{\ell}$) to processors in~$R$.}

\item[{\sc stage 3.}]~
\MK{In the receive substage processors in $R$ receive
{\tt info}($U_{\ell},L_{\ell})$ messages and processors in $P_c$ and $R$
receive {\tt summary}($D_c,P_c$) messages.  In the compute substage,
a restarted processor $q$ sets $L_{\ell,q}\leftarrow L_{\ell}$ and
$U_{\ell,q}\leftarrow U_{\ell}$.}
Let
$(D_w^1,P_w^1),...,(D_w^{k_w},P_w^{k_w})$ be the sets received in
{\tt summary} messages by processor $w$.
Processor $w$ sets $D_w\leftarrow D_w^i$ and $P_w\leftarrow P_w^i$ for
an arbitrary $i\in{1,...,k_w}$ and
updates its local view~$L_{\ell,w}$ \MK{as described below.} The send
substage is not used.
\end{description}
\end{quote}


\begin{figure}[thbp]
\begin{center}
%TexCad Options
%\grade{\on}
%\emlines{\off}
%\beziermacro{\off}
%\reduce{\on}
%\snapping{\on}
%\quality{2.00}
%\graddiff{0.01}
%\snapasp{1}
%\zoom{1.00}
\unitlength 0.82mm
\linethickness{0.4pt}
\begin{picture}(146.00,50.00)
\put(20.00,10.00){\dashbox{1.00}(0.00,40.00)[cc]{~}}
%\emline(20.00,34.00)(20.00,30.00)
\put(20.00,34.00){\line(0,-1){4.00}}
%\end
%\emline(34.00,34.00)(34.00,33.00)
\put(34.00,34.00){\line(0,-1){1.00}}
%\end
%\emline(34.00,31.00)(34.00,30.00)
\put(34.00,31.00){\line(0,-1){1.00}}
%\end
%\emline(48.00,34.00)(48.00,33.00)
\put(48.00,34.00){\line(0,-1){1.00}}
%\end
%\emline(48.00,31.00)(48.00,30.00)
\put(48.00,31.00){\line(0,-1){1.00}}
%\end
%\emline(62.00,34.00)(62.00,30.00)
\put(62.00,34.00){\line(0,-1){4.00}}
%\end
%\emline(104.00,34.00)(104.00,30.00)
\put(104.00,34.00){\line(0,-1){4.00}}
%\end
%\emline(76.00,34.00)(76.00,33.00)
\put(76.00,34.00){\line(0,-1){1.00}}
%\end
%\emline(118.00,34.00)(118.00,33.00)
\put(118.00,34.00){\line(0,-1){1.00}}
%\end
%\emline(76.00,31.00)(76.00,30.00)
\put(76.00,31.00){\line(0,-1){1.00}}
%\end
%\emline(118.00,31.00)(118.00,30.00)
\put(118.00,31.00){\line(0,-1){1.00}}
%\end
%\emline(90.00,34.00)(90.00,33.00)
\put(90.00,34.00){\line(0,-1){1.00}}
%\end
%\emline(132.00,34.00)(132.00,33.00)
\put(132.00,34.00){\line(0,-1){1.00}}
%\end
%\emline(90.00,31.00)(90.00,30.00)
\put(90.00,31.00){\line(0,-1){1.00}}
%\end
%\emline(132.00,31.00)(132.00,30.00)
\put(132.00,31.00){\line(0,-1){1.00}}
%\end
%\emline(146.00,34.00)(146.00,30.00)
\put(146.00,34.00){\line(0,-1){4.00}}
%\end
%\emline(20.00,15.00)(20.00,11.00)
\put(20.00,15.00){\line(0,-1){4.00}}
%\end
%\emline(34.00,15.00)(34.00,14.00)
\put(34.00,15.00){\line(0,-1){1.00}}
%\end
%\emline(34.00,12.00)(34.00,11.00)
\put(34.00,12.00){\line(0,-1){1.00}}
%\end
%\emline(48.00,15.00)(48.00,14.00)
\put(48.00,15.00){\line(0,-1){1.00}}
%\end
%\emline(48.00,12.00)(48.00,11.00)
\put(48.00,12.00){\line(0,-1){1.00}}
%\end
%\emline(62.00,15.00)(62.00,11.00)
\put(62.00,15.00){\line(0,-1){4.00}}
%\end
%\emline(104.00,15.00)(104.00,11.00)
\put(104.00,15.00){\line(0,-1){4.00}}
%\end
%\emline(76.00,15.00)(76.00,14.00)
\put(76.00,15.00){\line(0,-1){1.00}}
%\end
%\emline(118.00,15.00)(118.00,14.00)
\put(118.00,15.00){\line(0,-1){1.00}}
%\end
%\emline(76.00,12.00)(76.00,11.00)
\put(76.00,12.00){\line(0,-1){1.00}}
%\end
%\emline(118.00,12.00)(118.00,11.00)
\put(118.00,12.00){\line(0,-1){1.00}}
%\end
%\emline(90.00,15.00)(90.00,14.00)
\put(90.00,15.00){\line(0,-1){1.00}}
%\end
%\emline(132.00,15.00)(132.00,14.00)
\put(132.00,15.00){\line(0,-1){1.00}}
%\end
%\emline(90.00,12.00)(90.00,11.00)
\put(90.00,12.00){\line(0,-1){1.00}}
%\end
%\emline(132.00,12.00)(132.00,11.00)
\put(132.00,12.00){\line(0,-1){1.00}}
%\end
%\emline(146.00,15.00)(146.00,11.00)
\put(146.00,15.00){\line(0,-1){4.00}}
%\end
\put(146.00,10.00){\dashbox{1.00}(0.00,40.00)[cc]{~}}
\put(62.00,10.00){\dashbox{1.00}(0.00,40.00)[cc]{~}}
\put(104.00,10.00){\dashbox{1.00}(0.00,40.00)[cc]{~}}
\put(8.00,36.00){\makebox(0,0)[cc]{{\footnotesize\em Coordinator}}}
\put(8.00,20.00){\makebox(0,0)[cc]
{{\parbox{5em}{\centering\footnotesize\em  Worker\\\scriptsize\rm knows L,P,U,D}}}}
\put(8.00,8.00){\makebox(0,0)[cc]{{\footnotesize\em Restarted}}}
%\vector(51.00,21.00)(62.00,32.00)
\put(62.00,32.00){\vector(1,1){0.2}}
\multiput(51.00,21.00)(0.12,0.12){92}{\line(0,1){0.12}}
%\end
%\vector(51.00,2.00)(62.00,13.00)
\put(62.00,13.00){\vector(1,1){0.2}}
\multiput(51.00,2.00)(0.12,0.12){92}{\line(0,1){0.12}}
%\end
%\vector(93.00,24.00)(104.00,13.00)
\put(104.00,13.00){\vector(1,-1){0.2}}
\multiput(93.00,24.00)(0.12,-0.12){92}{\line(0,-1){0.12}}
%\end
%\vector(93.00,43.00)(104.00,32.00)
\put(104.00,32.00){\vector(1,-1){0.2}}
\multiput(93.00,43.00)(0.12,-0.12){92}{\line(0,-1){0.12}}
%\end
\put(82.00,48.00){\makebox(0,0)[cc]{{\small\sf Step 2}}}
\put(41.00,48.00){\makebox(0,0)[cc]{{\small\sf Step 1}}}
\put(125.00,48.00){\makebox(0,0)[cc]{{\small\sf Step 3}}}
\put(43.00,26.00){\makebox(0,0)[cc]{\parbox{4em}{\scriptsize Perform\\one task}}}
%\emline(20.00,32.00)(146.00,32.00)
\put(20.00,32.00){\line(1,0){126.00}}
%\end
%\emline(20.00,13.00)(146.00,13.00)
\put(20.00,13.00){\line(1,0){126.00}}
%\end
\put(69.00,37.00){\makebox(0,0)[cc]
{{\parbox{4em}{\centering\scriptsize receive\\\tt report}}}}
\put(83.00,37.00){\makebox(0,0)[cc]{{\parbox{4em}{\centering\scriptsize update\\D,P}}}}
\put(95.00,45.00){\makebox(0,0)[cc]{{\parbox{4em}{\centering\scriptsize\tt summary}}}}
\put(83.00,26.00){\makebox(0,0)[cc]{{\parbox{4em}{\centering\scriptsize update\\R}}}}
\put(111.00,26.00){\makebox(0,0)[cc]
{{\parbox{4em}{\centering\scriptsize receive\\\tt summary}}}}
\put(125.00,26.00){\makebox(0,0)[cc]
{{\parbox{4em}{\centering\scriptsize update\\ D,P,L}}}}
\put(56.00,20.00){\makebox(0,0)[cc]{{\parbox{4em}{\centering\scriptsize\tt report}}}}
\put(69.00,18.00){\makebox(0,0)[cc]
{{\parbox{4em}{\centering\scriptsize receive\\\tt restart}}}}
\put(93.00,18.00){\makebox(0,0)[cc]{{\parbox{4em}{\centering\scriptsize\tt info}}}}
\put(57.00,1.00){\makebox(0,0)[cc]{{\parbox{4em}{\centering\scriptsize\tt restart}}}}
\put(69.00,7.00){\makebox(0,0)[cc]
{{\parbox{4em}{\centering\scriptsize receive\\\tt restart}}}}
\put(83.00,7.00){\makebox(0,0)[cc]{{\parbox{4em}{\centering\scriptsize update\\ R}}}}
\put(111.00,6.00){\makebox(0,0)[cc]
{{\parbox{4em}{\centering\scriptsize receive\\\tt info summary}}}}
\put(125.00,7.00){\makebox(0,0)[cc]
{{\parbox{4em}{\centering\scriptsize update\\ D,P,L}}}}
%\vector(93.00,43.00)(103.00,16.00)
\put(103.00,16.00){\vector(1,-3){0.2}}
\multiput(93.00,43.00)(0.12,-0.32){84}{\line(0,-1){0.32}}
%\end
\end{picture}
\end{center}
\caption{\protect{\small A phase of algorithm \AR\ (for algorithm
        \AN\ ignore the bottom line, which represents restarted
        processors, and all the messages referring to it).}}
        \label{fig:phase}
\end{figure}

\noindent
{\bf Loal view update rule.} In phase 0 the local view $L_{0,w}$ of
any processor $w$ contains all the processors in $\P$ 
ordered by their PIDs, and the first layer 
is a singleton set.  Let
$L_{\ell,w}=\lan\Lambda^0,\Lambda^1,...,\Lambda^{j}\ran$ be the local
view of processor $w$ for phase $\ell$. We distinguish two possible
cases.

{\sc Case 1}. Phase $\ell$ is unattended. Let $R^{\ell}$ be the set of
restarted processors which send {\tt restart} messages.  Let $R'$ be
the set of processors of $R^{\ell}$ that are not already in the local
view $L_{\ell,w}$.  Let $\lan R' \ran$ be the processors in $R'$
ordered according to their PIDs.  The local view for the next phase is
$L_{\ell+1,w}=\lan \Lambda^1, ...,\Lambda^{j}
\ran
\oplus \lan R' \ran$. The operator $\oplus$ places processors of $R'$,
in the order $\lan R' \ran$, into the last layer $\Lambda^{j}$ till
this layer contains exactly the double of the processors of layer
$\Lambda^{j-1}$ and possibly adds a new layer $\Lambda^{j+1}$ to
accommodate the remaining processors of $\lan R'\ran$.  That is, newly
restarted processors which are not yet in the view, are appended at
the end of the old view. Notice that restarted processors, which
receive {\tt info} messages, know the old view $L_{\ell}$.

{\sc Case 2}. Phase $\ell$ is attended. Let $R^{\ell}$ be the set of
restarted processors.  Since the phase is attended
{\tt summary} messages are received by all the live processors 
(including the restarted ones). Any processor $w$ updates $P_w$ as
described in stage~3. Processor $w$ knows the set $R^{\ell}$. The
local view $L_{\ell+1,w}$ for the next phase is structured according to the
martingale principle and contains all the processors in $P_w\cup R^{\ell}$ 
ordered according to their PIDs.

\medskip

If there are no restarts, algorithm \AR\ behaves as algorithm \AN.
Figure~\ref{fig:phase} provides a graphical description of both
algorithms.

\section{Georgiou Russell Shvartsman}

We let \DAP{N,P,f} stand for the \DA\ problem for $N$ tasks, $P$
processors and up to $f$ failures. We let \DAO{N,P,f} denote the
\DAP{N,P,f} problem that is solved with the use of an arbitrary,
omniscient oracle $\mathcal{O}$ that may make available to the 
processors any
deterministic function of the global history of the computation: in
particular, the oracle may be used to assist the processors to
load-balance and terminate (but unlike the oracle's delphian
colleague, it cannot predict the future).  The oracle assumption is
used as a {\em tool\/} for studying the work complexity patterns of
{\em any\/} fault-tolerant algorithm: \textsl{(i)} of course, any
lower bound developed for this strong model applies equally well to
message-passing or shared memory models as the oracle may make
available to each processor the message or memory-read history it
would have observed, \textsl{(ii)} any algorithm in the oracle model
may be simulated in a message-passing or shared memory model by
suitably simulating the oracle. As most \DA\ algorithms use
communication to load-balance and detect termination, this framework
allows for the complexity analysis of specific algorithms to be
divided into two parts: \textsl{(1)}~the analysis of the cost of tolerating
failures while performing work assuming perfect load-balancing, and
\textsl{(2)}~the analysis of the cost of implementing perfect
load-balancing. We use exactly this approach to derive new
$f$-sensitive upper bounds for message-passing and shared-memory
models.


One may study the complexity of \DAO{\cdot}\ in conjunction with a
variety of different oracles. When we prove lower bounds, we consider
arbitrary oracles that may make available, for example, the entire
global history of the computation on every processor. When we prove upper
bounds, we use the \emph{load-balancing} oracle, $\mathcal{O}_L$,
described next.  During each synchronous iteration of the computation,
the oracle $\mathcal{O}_L$ makes available to each processor $P_i$ two
values: \textsl{Oracle-complete}, a Boolean which takes the value
\textsf{\small{true}} if and only if all tasks were completed at the
beginning of this iteration, and $\textsl{Oracle-task}_i$, an integer
whose value is a task identifier with the property that the
identifiers associated to the live processors are split evenly among
the tasks. In particular, if processors $i_1, \ldots, i_k \in [P]$ are
alive and tasks $j_1, \ldots, j_\ell \in [N]$ are incomplete at the beginning
of the iteration, then $\textsl{Oracle-task}_{i_s} = j_t$, where $t =
(s-1 \bmod \ell) + 1$.

Prior to \CITE{GRS} no comprehensive failure-sensitive analysis
of Do-All was known.
Yet practical concerns
would be well served by the knowledge of what happens
as the number of failures grows from 1 to $P-1$.
In particular, it is important to understand the behavior
of the best algorithms for the entire range of~$f$.

The detailed contributions in \CITE{GRS} are as follows.

\begin{itemize}
\Item[\bf I.]
We provide upper bounds (Section~\ref{sec:UB}) and matching lower
bounds (Section~\ref{sec:LB}) that are $f$-sensitive,
specifically we give a \emph{complete} analysis of \DAO{N,P,f}
for the \emph{entire} range of~$f$.
The bounds on work $W$ are:\footnote{
We let ``$\OM$" stand for ``$O$" when describing
an upper bound and for ``$\mathit\Wmega$" when describing
a lower bound. All logarithms are to the base 2
unless explicitly specified otherwise. It is understood, that
when $f=0$, then $X/\log (Y/f) = 0$, for any $X\neq 0$ and
$Y\neq 0$.
%The expression $\log X$ stands for
%$\max\{1,\; \log_2 X\}$ for the given $X$ in the
%description of complexity results.
}
\begin{equation}\label{eq:DA}
\begin{array}{cll}
(a) &
W=\OM\left(N + P + P
\frac{\log (\min(N,P))}{\log(P/f)}\right)
& {\rm ~when~} f\leq \frac{P}{\log (\min(N,P))} 
 ,\\
(b) &
W=\OM\left(N + P + P\frac{\log (\min(N,P))}{\log\log (\min(N,P))}\right)
& {\rm ~when~} f>\frac{P}{\log (\min(N,P))}
.
\end{array}
\end{equation}
The lower bounds of course apply to algorithms
in weaker models (e.g., a message passing or shared memory model). 
\end{itemize}

In the rest of the paper we use a shorthand notation
used to combine the statements such as (\ref{eq:DA})($a$)
and (\ref{eq:DA})($b$) above.
We define the quantity $\La{r}{N,P,f}$, where $r$ is the number
of \DA{} iterations (for \emph{r-iterative} \DA{}), and where $N$, $P$,
and $f$ appear in their usual roles,
as follows.

\begin{equation}\label{eq:L}
\La{r}{N,P,f} = \left\{
\begin{array}{cl}
\log (\frac{Pr}{f}) 
 & {\rm when~} f\leq\frac{Pr}{\log (\min(N,P))},
\vspace{2mm}\\
\log\log (\min(N,P))
 & {\rm when~} f> \frac{Pr}{\log (\min(N,P))}.
\end{array} \right.
\end{equation}

For a single instance \DA{}, i.e., when $r=1$,
we simplify this by defining
$\La{}{N,P,f} = \La{1}{N,P,f}$.
Thus the result (\ref{eq:DA}) is succinctly given by the following.

\begin{equation}\label{eq:DA-La}
\begin{array}{cll}
W=\OM\left(N + P + P\log (\min(N,P))/\La{}{N,P,f}\right).
\end{array}
\end{equation}

We use our bounds~(\ref{eq:DA-La}) (or equivalently~(\ref{eq:DA}))
to derive new bounds for algorithms where
the extant analyses do not integrate $f$ adequately.
This is done by analyzing
%how the information that may be provided by an oracle is modeled by the
how load-balancing is implemented by the
algorithms, e.g., by using coordinators or global data-structures.
We show the following.


\begin{itemize}
\Item[\bf II.]
In Section~\ref{sec:AN} we provide new
analysis of algorithm AN
of Chlebus \emph{et al.}~\CITE{CDS-DC01} for \DA\
in the message-passing model with crashes ($P\leq N)$.
This algorithm has best known work for moderate number of failures.
We show the complete analysis of work $W$ and
message complexity $M$:
%%%%The oracle assumption is replaced by the use of
%processors' {\em local views}.
%
$$
W = O\left(\log f\left(N + P\log P/\La{}{N,P,f}\right)\right)
{\rm ~~and~~}
M = O\left(N + P\log P/\La{}{N,P,f} + Pf\right).
$$\BBB\B
\Item[\bf III.]
In Section~\ref{sec:W} we give a \emph{complete} analysis of the work
complexity $W$ of the algorithm of Kanellakis and Shvartsman~\CITE{KS-DC92}
that solves the \WA{} problem in
synchronous shared-memory systems 
(shared memory \DA{}\/) 
with processor crashes ($P\leq N$):
$$
W=O\left(N + P\log N\log P/\La{}{N,P,f}\right).
$$
\end{itemize}
%
Note that the two algorithms~\CITE{CDS-DC01,KS-DC92}
are designed for different models
and use dissimilar data and control structures,
however both algorithms make their load-balancing decisions
by gathering global knowledge.
By understanding what work is expended on
load-balancing vs.\ the inherent work overhead due to the
lower bounds~(\ref{eq:DA}), we are able to obtain
the new results, and to demonstrate the utility and the
generality of our approach.
(We restrict our attention to the case $P\le N$
for consistency with~\CITE{CDS-DC01,KS-DC92}.)


\DA{} algorithms have been used in developing {\em simulations\/}
of failure-free algorithms on failure-prone processors,
e.g.,~\CITE{KPS-STOC90,AS91}.  This is done by iteratively using a
\DA{} algorithm to simulate the steps of the $N$ failure-free
``virtual" processors on $P$ failure-prone ``physical" processors
(here the usual case is that the number of physical processors
does not exceed the number of virtual processors, i.e., $P\le N$). 
In this paper we abstract this idea as the {\em
iterative\/} \DA{} problem as follows:
\begin{quote}
The $r${\sl -iterative} \DA\ problem, denoted \rDA{r}{N,P,f}, is the
problem of using $P$ processors to solve $r$~instances of $N$-task
\DA{}
%($P\leq N$)
with the added restriction that every task of the $i$th instance must
be completed before any task of the $i+1$st instance is begun.
%by doing one set of tasks at a time.
\end{quote}
%
The oracle version \rDO{r}{N,P,f} is defined analogously.  An obvious
solution for this problem is to run a \DA{} algorithm $r$ times.  If
the work complexity of \DA{} in a given model is $W_{N,P,f}$, then the
work of $r$-\DA{} is clearly no more than $r\cdot W_{N,P,f}$.
This does not take into account the fact that $f$ crashes
occur throughout the $r$ iterations.
We present a substantially better analysis
for crash-prone processors:

\begin{itemize}
%
\Item[\bf IV.]
In Section~\ref{sec:RDO} we show matching upper
and lower bounds
on work $W$ for \rDO{r}{N,P,f}, where $f<P\leq N$, for specific ranges
of failures. 

\begin{equation}\label{eq:RDA}
\begin{array}{cll}
W=\OM\left(r\cdot\left(N + P\log P/\La{r}{N,P,f}\right)\right).
%
\remove{%REMOVING (a) and (b)
(a) &
W=\OM\left(r\cdot\left(N + P\frac{\log P}{\log\log P}\right)\right)
& {\rm ~when~} f > \frac{Pr}{\log P},\\%%%%%%%, {\rm ~any~} c>0,\\
(b) &
W=\OM\left(r\cdot\left(N + P\frac{\log P}{\log \frac{Pr}{f}}\right)\right)
& {\rm ~when~} f\leq \frac{Pr}{\log P}.
}%% End remove 
\end{array}
\end{equation}
\end{itemize}

The improvement in the bounds~(\ref{eq:RDA}) are twofold.
First, the derivation of the bounds reflects our improved failure-sensitive
analysis for a single \DA{}.
Additionally, we have $\La{r}{N,P,f}\geq \La{}{N,P,f}$
for any $r$, moreover $\La{}{N,P,f}$ is a constant with respect to $r$.
The result is that our bounds~(\ref{eq:RDA}) are asymptotically better 
than those obtained by computing the product of $r$ and
the (non-iterated) \DA{} bounds~(\ref{eq:DA})~(for $P\leq N$).


\begin{itemize}
\Item[\bf V.]
In Section~\ref{sec:RDO:AN} we show how to solve
\rDA{r}{N,P,f} ($P\leq N$) on synchronous message-passing processors
with the following work ($W$) and message ($M$) complexity.
\begin{center}
$W = O\left(r\cdot\log (\frac{f}{r})\cdot
\left(N + P\log P/\La{r}{N,P,f}\right)\right)$
~and~
$M = O\left(r\cdot\left(N + P\log P/\La{r}{N,P,f}\right) + fP\right)$.
\end{center}


\Item[\bf VI.] In Section~\ref{sec:RDO:W} we use
\rDA{r}{N,P,f} ($P\leq N$) to show that $P$ crash-prone processors can
simulate any synchronous $N$-processor, $r$-time
shared-memory algorithm ({\sc pram}) with work:
$$
W = O\left(r\cdot\left(N + P\log N\log P/\La{r}{N,P,f}\right)\right).
$$
\end{itemize}
%
This last result is strictly better than the previously known
deterministic bounds for deterministic parallel algorithm simulations 
using the \WA{} algorithm~\CITE{KS-DC92} (the best known to date)
together with the simulation techniques such as~\CITE{KPS-STOC90,AS91}.
%%% CG %%%%%%%
%
%{\bf The improvement is due to the relationship between
%$\La{r}{N,P,f}$ and $\La{}{N,P,f}$ as pointed out above.}
%%%%%%%%%%%%%

\section{Malewicz Russell Shvartsman (00, 01, 02)}

To study aspects of the trade-off between communication and 
computation in distributed cooperative applications, we
consider the following abstract problem:
{\sl $n$ processors must perform $t$ tasks
and learn the results of all tasks\/}.
We assume that all tasks are known to all processors.
A common impediment to effective coordination in 
distributed settings is asynchrony that manifests itself, 
for example, in disparate processor speeds and 
nondeterministic message latency.
Fortunately, our problem can always be solved
by a communication-oblivious algorithm that
forces each processor to perform all tasks.
Such a solution has work $W = O(t\cdot n)$,
an requires no communication, i.e., $M = 0$.
On the other hand, $\Omega(t)$ is the obvious lower bound
on work and the best known
lower bound is $W = \Omega(t+n\log n)$, cf.~\CITE{KS}. 
Therefore the trade-off expectation is that if we gradually 
increase the number of messages we should be able to decrease
the amount of work performed.

Let us consider an asynchronous setting, where processors communicate
by means of a {\em rendezvous\/}, i.e., two processors that are able
to communicate can perform state exchange.  The processors that are
not able to communicate via rendezvous have no choice but to perform
all $t$ tasks.  Consider the computation with a single rendezvous.
There are $n-2$ processors that are unable to communicate, and they
collectively must perform exactly $t\cdot(n-2)$ work units to learn all
results.  Now what about the remaining pair of processors that are
able to rendezvous? In the worst case they rendezvous after performing
all tasks individually.  In this case no savings in work are realized.
Suppose they rendezvous having performed $t/2$ tasks each. In the best
case, the two processors performed mutually-exclusive subsets of tasks
and they learn the complete set of results as a consequence of the
rendezvous.  In particular if these two processors know that they will
be able to rendezvous in the future, the could schedule their work as
follows: one processors performs the tasks in the order $1, 2, \ldots, t$,
the other in the order $t, t-1, \ldots, 1$. No matter when they happen
rendezvous, the number of tasks they both perform is minimized.  Of
course the processors do not know {\em a priori\/} what pair will be
able to rendezvous. Thus it is interesting to produce task execution
schedules for all processors, such that upon the first rendezvous of
any two processors the number of tasks performed redundantly is
minimized.

This setting we have just described
is interesting for several reasons.
If the communication links are subject
to failures, then each processor must be ready to execute
all of the $t$ tasks, whether or not it is able
to communicate.
In realistic settings the processors may not initially
be aware of the network configuration,
which would require expenditure of computation resources
to establish communication, for example in radio networks.
In distributed environments involving autonomous agents,
processors may {\em choose} not to communicate either because
they need to conserve power or
because they must maintain radio silence.
Finally, during the initial configuration of a dynamic
network or a middleware service (such as a group communication
service~\CITE{CACM96}) the individual processors may start 
working in isolation pending the completion of system configuration.
Regardless of the reasons, it is important to direct
any available computation resources to performing
the required tasks as soon as possible.
%
In all such scenarios, the $t$ tasks have to be scheduled for
execution by all processors. The goal of such scheduling must be to
control redundant task executions in the absence of communication and
during the period of time when the communication channels are being
(re)established.

For a variation of the problem we deal with in
this report,
Dolev \emph{et al.}~\CITE{DSS} showed that for the case of
dynamic changes in connectivity, the termination time of any on-line
task assignment algorithm can be greater than the termination time of
an off-line task assignment algorithm by a factor linear in $n$. This
means that an on-line algorithm may not be able to do better than
the trivial solution that incurs linear overhead by having each
processor perform all the tasks.  With this observation~\CITE{DSS}
develops an effective strategy for managing the task
execution redundancy and proves that the strategy provides each of the
$n$ processors with a schedule of $\Theta (n^{1/3})$ tasks such that
at most one task is performed redundantly by any two processors.


\subsection{Schedules, waste, and designs}
\label{sec:Schedules}

We consider the abstract setting where $n$ processors need to perform
$t$ independent and idempotent tasks.  A task is \emph{idempotent} if
the execution of the task yields the same result when it is performed
more than once.  A task is independent if the result of its execution
does not depend on the order in which other tasks are executed.  The
processors have unique identifiers from the set $[n] = \{1, \ldots,
n\}$, and the tasks have unique identifiers from the set $[t] = \{1,
\ldots, t\}$.  Initially each processor knows the tasks that need to
be performed and their identifiers.


We shall focus on the scheduling problem discussed above, abstracted
as follows. An $(n,t)$-\emph{schedule} is a tuple 
$(\sigma_1, \ldots,\sigma_n)$ of
$n$ permutations of the set $[t]$. When $n=1$ it is elided and we
simply write $t$-\emph{schedule}.

%Consider initially isolated processors who must complete $t$ tasks. If
%the tasks are known to all processors

An $(n,t)$-schedule immediately gives rise to a strategy for $n$
isolated processors who must complete $t$ tasks until communication
between some pair (or group) is established: the processor $i$ simply
proceeds to complete the tasks in the order prescribed by $\sigma_i$.
Suppose now that some $k$ of these processors, say $q_1, \ldots, q_k$,
should rendezvous at a time when the $i$th processor in this group,
$q_i$, has completed $a_i$ tasks.  Ideally, the processors would have
completed disjoint sets of tasks, so that the total number of tasks
completed is $\sum_i a_i$. As this too much to hope for in general, it
is natural to attempt to bound the gap between $\sum_i a_i$ and the
actual number of distinct tasks completed. This gap we call
\emph{waste}:

\begin{definition}
If $S$ is a $(n,t)$-schedule and $(a_1, \ldots, a_k) \in \N^k$, the
\emph{waste} function for $S$ is
$$
\waste_S(a_1,\ldots, a_k) = \max_{( q_1, \ldots, q_k)} \left( \sum_i^k a_i -
  \card{\bigcup_i^k \sigma_{q_i}([a_i])}\right),
$$
this maximum taken over all $k$ tuples $(q_1, \ldots, q_k)$ of distinct
elements of $[n]$.
\end{definition}

Here (and throughout), if $\phi: X \to Y$ is a function and $S \subset X$, we
let $\phi(S) = \{ \phi(x) \mid x \in S\}$. 
For a specific vector $a = (a_1, \ldots,
a_k)$, $\waste_S(a)$ captures the worst-case number of redundant tasks
performed by any collection of $k$ processors when the $i$th process
has completed the first $a_i$ tasks of its schedule.

One immediate observation is that bounds on \emph{pairwise} waste
$\waste_S(\cdot,\cdot)$ can be naturally extended to bounds on
\emph{$k$-wise} waste $\waste_S(\underbrace{\cdot,\ldots,\cdot}_k)$:
specifically, note that if $S$ is an $(n,t)$-schedule then
$$
\waste_S(a_1,\ldots, a_k) \leq \sum_{i < j} \waste_S(a_i,a_j)
$$
just by considering the first two terms of the standard
inclusion-exclusion rule. Moreover, it appears that this relationship
is fairly tight as it is nearly attained by randomized schedules---see
Section~\ref{sec:randomized-martingale}. With this justification we
shall content ourselves to focus the investigation on pairwise
waste---the function $\waste_S(a,b)$.

Set systems with prescribed intersection properties have been the
object of intense study by both the design theory community and the
extremal set theory community (see, e.g., \cite{HP} for a survey).
Despite this, the study of \emph{waste} appears to be new. We shall,
however, make substantial use of some design-theoretic constructions,
which we describe below.

\begin{definition}
  A $\ell$-$(v,k,\lambda)$ \emph{design} is a family of subsets $\mathcal{S} =
  (S_1, \ldots, S_n)$ of the set $[v]$ with the property that each
  $\card{S_i} = k$ and any set of $\ell$ elements of $[v]$ is a subset
  of precisely $\lambda$ of the $S_i$.  (N.B. The subsets $S_i$ are
  typically referred to as \emph{blocks}.)
\end{definition}

Observe that if $\mathcal{S}$ is a $\ell$-$(v,k,\lambda)$ design, then it is
also a $(\ell-1)$-$(v,k,\hat{\lambda})$ design where
$$
\hat{\lambda} = \lambda \frac{(v-\ell+1)}{(k-\ell+1)}.
$$
To see this, note that if $T$ is a subset of elements of size
$\ell-1$, then there are exactly $v-(\ell-1)$ sets of size $\ell$ which
contain $T$; let $U_i, i \in [v-(\ell-1)],$ denote these sets. By
assumption, each $U_i$ appears in exactly $\lambda$ of the $S_j$. Of
course, if $U_i$ is a subset of some $S_j$, then in fact exactly $k -
(\ell-1)$ if the $U_i$ are subsets of $S_j$. Hence $T$ appears in
exactly $\lambda (v - \ell + 1) / (k - \ell + 1)$ of the $S_j$, as desired.

To see the connection between such designs and our problem, let
$\mathcal{D}$ be a $2$-$(n,k,\lambda)$ design consisting of $t$ sets $S_1,
\ldots, S_t$. For each $i \in [n]$, let $T_i = \{ j \mid i \in S_j \}$. 
Note now that for any $i \neq j$,
$$
T_i \cap T_j = \left\{ k \mid \{i,j\} \subset S_k\right\}
$$
and hence that $\card{T_i \cap T_j} = \lambda$. Based on the observation
above, we see also that $\forall i,j, \card{T_i} = \card{T_j}$ and let $a$
denote this common cardinality. 
Now, let $\Sigma = (\sigma_1, \ldots, \sigma_n)$ be
any sequence of permutations of $[t]$ for which $\sigma_i([a]) = T_i$. It
is clear that these form an $(n,t)$-schedule for which
$$
\waste_\Sigma (a,a) = \lambda.
$$

Unfortunately, the above construction offers satisfactory control of
2-waste only for the specific pair $(a,a)$. Furthermore, considering
that the construction only determines the \emph{sets} $\sigma_i([a])$ and
$\sigma_i([n] \setminus [a])$, the ordering of these can be conspiratorially
arranged to yield poor bounds on waste for other values. Our goal is
construct schedules with satisfactory control on waste for all pairs
$(a,b)$.

While designs do not appear to immediately induce a solution to this
problem, we will apply the following design-theoretic construction
several times in the sequel. Let $\GF(q)$ denote the finite field with
$q$ elements, where $q$ is a prime power. Treating $\GF(q)^3$ as a
vector space over $\GF(q)$, the design will be given by the lattice of
linear subspaces of $\GF(q)^3$. It is easy to check that there are $n
= q^2+q+1$ distinct one dimensional subspaces of $\GF(q)^3$, which we
denote $\ell_1,\ldots,\ell_{n}$. 
We say that two subspaces $\ell_i$ and $\ell_j$
are \emph{orthogonal} if 
$\forall u \in \ell_1, \forall v \in \ell_2$, $\langle u, v\rangle = \sum
u_jv_j \bmod p = 0$; in this case we write $\ell_i \perp \ell_j$. It is a
fact that for any one dimensional subspace there are exactly $q+1$ one
dimensional subspaces to which it is orthogonal. The design consists
of the $n = q^2 + q + 1$ sets $S_u = \{ \ell_i \mid \ell_i \perp \ell_u\}$. 
It is
easy to show that any pair of such sets intersect at a single $\ell_i$,
and that this forms a $2$-$(q^2+q+1,q+1,1)$ design.  See~\cite{HP} for
a proof and more discussion.

For concreteness, we fix a specific (arbitrary) ordering of each of
these sets $S_u$: let $L_u$ denote a canonical sequence $\langle
t^1_u,\ldots,t^r_u \rangle$ where 
$S_u = \{ \ell_{t_u^i} \mid 1 \leq i \leq q+1\}$;
i.e., the one dimensional subspaces $\ell_{t^i_u}$, $i=1,\ldots,q+1$, are
precisely those orthogonal to $\ell_u$. For convenience, for two
sequences $A$ and $B$, we let $A \cap B$ and $A \cup B$ denote the
corresponding union or intersection of the sets of objects in the
sequences. We record the above discussion in the following theorem.


\begin{theorem}\label{Ltime}
  Let $n=q^2+q+1$, where $q$ is a prime power. Then the sequences
  ${\mathcal L}_n=\langle L_1,\ldots,L_n\rangle$ possess the following
  properties: each $L_u$ has length $q+1$, for each $u \neq v$, $|L_u
  \cap L_v|=1$, and any element appears in exactly $q+1$ distinct
  sequences. We note also that if $q$ is prime, the first element of
  each sequence can be calculated in $O(\log{n})$ time; each
  subsequent element can be calculated in $O(1)$ time.
\end{theorem}
\noindent

(We assume throughout that addition or multiplication of two
$\log{(\max\{n,t\})}$-bit numbers can be performed in $O(1)$ time.)


\section{Redundancy without communication: a lower bound}
\label{sec:LB}

Controlling global computation redundancy in the absence 
of communication is a futile task. This is
because no amount of algorithmic sophistication can compensate
for the possibility of individual processors, or groups of
processors, becoming disconnected during the computation.
%
In general, an adversary that is able to partition
the processors into $g$ groups that cannot communicate
with each other will cause any task-performing algorithm 
to have work $\Omega(t\cdot g)$, 
even if each group of processors performs
no more than the optimal number of $\Theta(t)$ tasks.
In the extreme case where all processors are isolated from 
the beginning, the work of any algorithm is $\Omega(t\cdot n)$,
which is at least the work of an oblivious algorithm, where
each processor performs all tasks.

Of course it is not surprising that substantial redundancy cannot 
be avoided in the absence of communication, furthermore, 
the lower bound on work of $\Omega(t\cdot n)$ is not very interesting.
However, as we pointed out earlier,
it is possible to schedule the work of a pair of processors
so that each can perform up to $t/2$ tasks without a single task
performed redundantly.
Thus it is very interesting to consider the intersection
properties of pairs of processor schedules, i.e., 2-waste.

If we insist that among the $n$ total processors, any two processors,
having executed the same number of tasks $t'$, where $t'<t$, perform
{\it no\/} redundant work, then it must be the case that $t' \leq \lfloor t/n
\rfloor$. In particular, if $n=t$, then the pairwise waste jumps to one if
any processor executes more than one task.
%  In other words, if all
%processors complete $2$ tasks, the lower bound on pairwise redundancy is
%$1$.
The next natural question is: how many tasks can processors complete
before the lower bound on pairwise redundant work is $2$?  In general,
if any two processors perform $t_1$ and $t_2$ tasks respectively, what
is the lower bound on pairwise redundant work?  In this section we
answer these questions.  The answers contain both good and bad news:
given a fixed $t$, the lower bound on pairwise redundant work starts
growing slowly for small $t_1$ and $t_2$, then grows quadratically in
the schedule length as $t_1$ and $t_2$ approach $t$.

We proceed to the lower bound, which generalizes the second
Johnson Bound~\cite{JOH} for the case when two processors execute
\emph{different} number of tasks prior to their rendezvous.

\begin{theorem}[\CITE{MRS-DISC}]
\label{lowerBound}
Let ${\cal P} = \tup{\pi_1,\ldots,\pi_n}$ be an $(n,t)$-schedule and let $0
\leq a \leq b \leq t$.  Then
$$
\waste_{\mathcal{P}}(a,b) \geq \frac{na^2}{(n-1)(t-b+a)}-\frac{a}{n-1}.
$$
\end{theorem}

\renewcommand{\Pr}[1]{\Prsymbol\left[#1\right]}
\newcommand{\Prsymbol}{\mathop\mathsf{Pr}}

For example, when processors perform the same number of tasks $a=b$ 
and $n=t$, then the worst case number of redundant tasks
for any pair is at least $\frac{a^2-a}{t - 1}$.
This means that (for $n=t$) if $a$ exceeds $\sqrt{t}+1$,
then the number of redundant task is at least $2$.

\begin{corollary}[\CITE{MRS-DISC}]
\label{impPerm2}
For $t=n$, if $a > \sqrt{n-3/4}+\frac{1}{2}$ then any $n$-processor
schedule of length $a$ for $t$ tasks
has worst case pairwise waste at least $2$.
\end{corollary}


\subsection{Random schedules}
\label{sec:Random}

As one would expect, schedules chosen at random perform quite well. In
this section we explore the behavior of the $(n,t)$-schedules obtained
when each permutation is selected uniformly (and independently) at
random among all permutations of $[t]$.

\subsubsection{Randomized schedules}\label{s:RND}
\newcommand{\rand}{\textsc{Random}}

When the processors are endowed with a reasonable source of
randomness, a natural candidate scheduling algorithm is one where
processors select tasks by choosing them uniformly among all tasks
they have not yet completed. This amounts to the selection, by each
processor $i$, of a random permutation $\pi_i \in S_{[t]}$ which
determines the order in which this processor will complete the tasks.
($S_{[t]}$ denotes the collection of all permutations of the set
$[t]$.) We let ${\cal R}$ be the resulting system of schedules.

Our objective now is to show that random schedules ${\mathcal R}$ have
controlled waste with high probability.  This amounts to bounding, for
each pair $i, j$ and each pair of numbers $a, b$, the overlap
$\card{\pi_i([a]) \cap \pi_j([b])}.$ Observe that when these $\pi_i$ are
selected at random, the expected size of this intersection is $ab/t$.
By showing that the actual waste is very likely to be close to this
expected value, one can conclude the waste if bounded for \emph{all}
long enough prefixes.

\begin{theorem}[\CITE{MRS-DISC}]
  \label{thm:random-2}
  Let ${\mathcal R}$ be a system of $n$ random schedules 
  for $t$ tasks constructed as above. 
  Then with probability at least $1 - \frac{1}{nt}$, 
  $$
  \forall a,b \textrm{ such that } 7\sqrt{t}\ln{(2nt)} \leq a,b \leq t,~~
  \waste_{\mathcal R}(a,b) \leq \frac{ab}{t} + \Delta(a,b)~,
  $$
   where
  $\Delta(a,b) = 11 \sqrt{\frac{ab}{t}\ln(2nt)}$
\end{theorem}

\F

Observe that Theorem~\ref{lowerBound} shows that $(n,t)$-schedules
must have waste $\waste(a,a) = \Omega(a^2/t)$ (as $n \to \infty$); hence such
randomized schedules offer nearly optimal waste for this case.



\subsubsection{$k$-Waste for random schedules}
\label{sec:randomized-martingale}

For random schedules, one can apply martingale techniques to directly
control $k$-wise waste. We mention one such result below.


%Furthermore for randomized schedules, a martingale argument shows that
%$k$-waste is close to the expected $k$-waste with high probability:
\begin{theorem}
  \label{RNDkbound}
  Consider the random schedule $\mathcal R$ from above.  Then with
  probability at least $1 - 1/n$,
  $$
  \waste_{\mathcal R}(a,\ldots,a) \leq \sum_{s=2}^k (-1)^s\binom{k}{s}
  \frac{a^s}{t^{s-1}} + \Delta_{a,k},
  $$
  where $\Delta_{a,k} = (2k+1)\sqrt{a\ln n}$.
\end{theorem}
Note that again this bounds the distance of the $k$-waste from its
expected value, which can be computed by inclusion-exclusion to be
$\sum_{s=2}^k (-1)^s\binom{k}{s} \frac{a^s}{t^{s-1}}$.
% The proof, which
% we omit, proceeds by considering the martingale which exposes the
% $i$th element of all schedules at step $i$. The theorem then follows
% by noting that the expected value can change by at most $k$ during a
% single exposure and applying Azuma's inequality. 
% (See~\cite{AS} for a
% discussion of discrete exposure martingales and Azuma's inequality.)


\subsubsection{Arbitrary rendezvous patterns (Georgiou Russell Shvartsman)}

Thus far we have established bounds on wasted work for a single
rendezvous. It is also interesting to understand the effectiveness of
randomized schedules when encountering arbitrary patterns of (not
necessarily pairwise) rendezvous.  In general, an adversary that
leaves the processors partitioned into $g$ disconnected components
will cause any task-performing algorithm to have work $\Omega(t\cdot g)$ even
if each group of processors performs no more than the optimal number
of $\Theta(t)$ tasks. 
%In the extreme case when $g=n$ the work has to be
%$\Omega(t\cdot n)$.
This lower bound forms a somewhat pessimistic landscape for our
problem.  Nevertheless, it is important to design and analyze
algorithms that can be shown to be more efficient than the oblivious
approach where each processor performs all tasks. In this case where
the lower bound is quite sensitive to the interconnection pattern,
% In order to
%understand better the practical implications of performing work in
%partitionable settings, and especially in the settings where initially
%there may be several disconnected groups, it is interesting to
we consider a {\em competitive\/} analysis~\CITE{ST-CACM85} of randomized
schedules. 

Here we focus on the scenario where processors begin working
(asynchronously) on the tasks in isolation and an adversary may choose
to merge together various processors or groups of processors as the
computation progresses. Once two processors find themselves in the
same group, they cannot be separated--the adversary may never fragment
a group of processors once it has been formed. An algorithm in this
model is a rule which, given a collection of processors and a
collection of tasks, determines the \emph{next} task for this group to
complete. A group of processors $P$ who have collectively completed
tasks $T$ can use such a rule to determine the next task to work
on. We shall consider the randomized algorithm, where a group of
processors always selects a task at random from those it has not
completed to work on next.

Adversaries in this model can be serialized in the following sense
without loss of generality. An adversary $A$ is described by a
sequence of \emph{events}; each event either \textsl{(i.)} allows a
specific processor or group of processors to complete some number $t$ of tasks,
or \textsl{(ii.)} merges together some collection of processors and
groups to form a new group. Note that when groups are formed by the
adversary they inherit from their constituents full knowledge of all
tasks they have previously completed.

\newcommand{\MD}{our problem}
\newcommand{\OPT}{OPT}
\newcommand{\EXP}{RS}
\newcommand{\Ra}{{\mathcal R}a}

We consider a competitive analysis that compares the work of the
randomized algorithm with the work of an optimal algorithm that has
complete information about the adversary (and hence the future pattern of
rendezvous).

Let $D$ be a deterministic algorithm for \MD\ and $A$ an adversary, we
let $W_{(t,n)}(D,A)$ denote the total work 
(i.e., the number of tasks executed, counting multiplicities)
of algorithm $D$ with $n$ processors and $t$ tasks, 
where the pattern of rendezvous
is determined by adversary $A$. 
Where $t$ and $n$ can be inferred from context, we
simply write $W(D,A)$.
% (We will formally define the adversaries momentarily.)

We treat randomized algorithms as distributions over deterministic
algorithms; for a set $C$ and a family of deterministic algorithms $\{
D_r \mid r \in C \}$ we let $R = \Ra(\{ D_r \mid r \in C \})$ denote the
randomized algorithm where $r$ is selected uniformly at random from
$C$ and scheduling is done according to $D_r$.  We let \OPT\ denote
the optimal off-line algorithm, which may schedule tasks with full
knowledge of the sequence of events caused by the adversary.

\begin{definition}[\CITE{ST-CACM85,FKLMSY-JA91,BBKTW94}]~\\
  A randomized algorithm $R$ is {\bf $\alpha$-competitive} if for 
  all adversaries $A$,
 $$
 {\bf E}_r[W(D_r,A)] \leq \alpha\cdot W(\OPT,A),
 $$
  this expectation being taken over choice of $r\in \Omega$.  
\end{definition}

The following lower bound holds for the competitive ratios.

\begin{theorem}[\CITE{MERGE-DO}]
  Let $\Ra(\{D_r \mid r \in \Omega\})$ be an $\alpha$-competitive
  randomized algorithm for \MD. Then $\alpha \geq 1 + \frac{1}{e}$.
\end{theorem}

This lower bound suggests that it is indeed worthwhile to seek
algorithms that are very competitive, despite the potentially
high bounds on ``absolute" work. Armed with this hope we 
analyze the competitive performance of randomized schedules.

As mentioned, we consider the natural randomized algorithm \EXP\ where
a processor (or group) with knowledge that the tasks in a set $K \subset
[t]$ have been completed selects to next complete a task at random
from the set $[t]$$\setminus K$.  More formally, let $\Pi = (\pi_1, \ldots,
\pi_n)$ be a $(n,t)$-schedule (where each $\pi_i$ is a permutation of
$[t]$).  We describe a deterministic algorithm $D_\Pi$ so that $\EXP =
\Ra(\{ D_\Pi \mid \Pi \in (S_t)^n\})$. 
Let $G$ be a group of processors and $\gamma \in G$
the processor in $G$ with the lowest processor identifier. Then the
deterministic algorithm $D_\Pi$ specifies that the group $G$, should it
know that the tasks in $K \subset [t]$ have been completed, next completes
the first task in the sequence $\pi_\gamma$ which is not in $K$.

It turns out, not unexpectedly, that this algorithm is optimal with
respect to the lower bound on competitive ratios.

\begin{theorem}[\CITE{MERGE-DO}]
\label{thm:main}
Algorithm \EXP\ is $\left(1+\frac{1}{e}\right)$-competitive for any
given pattern of rendezvous.
%\MDAG.
\end{theorem}

Currently we have obtained preliminary results extending this
competitive analysis to arbitrary patterns of merges and
fragmentations.

\subsection{Derandomization via finite geometries}
\label{sec:Derandom}

We now consider a method for derandomizing these schedules using the
design discussed in Section~\ref{sec:Schedules}.

\paragraph{Schedules for $n=t$.}

We construct a system of schedules of length $n$ by arranging tasks
from the sequences of ${\mathcal L}_n$ in a recursive fashion. (Recall
that while the sequences of ${\mathcal L}_n$ have strong intersection
properties, they are only roughly $\sqrt{n}$ in length.) In
preparation for the recursive construction, we record the following
lemma about the pairwise intersections of the elements in the sequence
of ${\mathcal L}_n$ \emph{indexed} by a specific subspace $L_u$.

\begin{lemma}
\label{singleton}
Let ${\mathcal L}_n=\langle L_1,\ldots,L_n\rangle$ be the collection of sequences
constructed in Theorem~\ref{Ltime}, and let $L_u=\langle t^1_u,\ldots,t^{q+1}_u
\rangle$, $1\leq u \leq n$.  Then for any $i \neq j$, we have $L_{t_u^i} \cap
L_{t_u^j} = \{u\}$.
\end{lemma}

As a result of this lemma, there is \emph{only a single repeated
  element} in the sequences $L_{t^1_u}$, $L_{t^2_u}, \ldots,
L_{t^{q+1}_u}$; this element is $u$.  This fact suggests the following
construction of a system of schedules ${\mathcal P}_n$. Let $P_u$, $1
\leq u \leq n$, be the sequence whose first element is $u$, and whose
remaining elements are given by concatenating the $q+1$ sequences
$L_{t^1u}, \ldots, L_{t^{q+1}_u}$ after removing $u$ from each.
Specifically,
$$
P_u=\langle u \rangle \circ (\bigcirc_{i \in L_u} (L_i - u)),
$$
where $\circ$ denotes concatenation and $L_i - u$ denotes the sequence
$L_i$ with $u$ deleted.
% (the big circle stands for concatenation).  
Note now that since the total length of $P_u$ is evidently $(q + 1)q +
1 = n$, each element of $[n]$ must appear exactly once in each $P_u$;
these $P_u$ thus rive rise to a family of permutations $\pi_u$, where
$\pi_u(k)$ is the $k$ element of $P_u$. Let $\mathcal P = (\pi_1, \ldots,
\pi_n)$.

We conceptually divide the sequences $P_u$ (associated with the
permutations $\pi_u$) into $q+1$ \emph{segments} of elements.  The
first segment contains the first $q+1$ elements (including the initial
element $u$); the remaining $q$ segments contain $q$ consecutive
elements each.


This recursive construction yields a straightforward bound on pairwise
waste, recorded below.
%Now we show an upper bound on the 2-waste of the system of schedules
%${\mathcal P}_n$.

\begin{theorem}
\label{Pbound}
Let $q$ be a prime power, $n=q^2+q+1$. Let $a=1+iq$, $b=1+jq$,
$0\leq i,j\leq q+1$. 
Then
%The the waste for the system of schedules ${\mathcal P}_n$ is bounded by:
$$
\waste_{\mathcal{P}_n}(a,b) \leq
\begin{cases}
  0, & i+j=0,\\
  1, & i=0,j \geq 1\;\text{or}\;i\geq 1,j=0,\\
  q+ij, & i\cdot j \geq 1.
\end{cases}
$$
\end{theorem}


We mention that the construction can be done on-line.  For each
schedule the first element can be calculated in $O(1)$ time.  For the
remaining $q(q+1)$ elements, at the beginning of every sequence of $q$
elements we need to invert at most two elements in $\GF(q)$. When $q$
is prime this can be done in $O(\log{n})$ using the extended Euclidean
algorithm. Other elements of the schedule can be found in $O(1)$ time.

Note that when $t = \kappa n$ for some $\kappa \in \N$, the above construction
can be trivially applied by placing the $t$ tasks into $n$ chunks of
size $\kappa$. In this case, of course, when a single overlap occurred in
the original construction, this penalty is amplified by $\kappa$.

%\subsection{Controlling waste for short prefixes}

One disadvantage of ${\mathcal P}_n$ is that the first segment may
repeat, so that $(q+1)$ waste may be incurred when a prefix of length
$\hat{a}= (q+1)$ is executed.  To postpone this increase one would
like to rearrange the segments in each $P_u$ so that the first segment
is distinct across the resulting schedules. This can be accomplished
by finding a permutation $\rho: [n] \to [n]$ such that the sequence $L_u$
contains task $\rho(u)$. (In other words $\ell_u$ must be orthogonal to
$\ell_{\rho(u)}$.)  This permutation can then be used to select distinct
segments as the first segments of schedules in ${\mathcal P}_n$.

\end{document}
