% Lecture notes - Lecture 8 - 10/01/2003
% Scribes Kevin Matulef and Barbara Mack
% Edited by Sean Lie and Paul Youn

%\documentclass{article}

%\usepackage{6895}
%\input epsf

%\newcommand{\infinity}{\infty}

\begin{guestlecture}{8}{Cilk Scheduler}{Kevin Matulef, Barbara Mack, Paul Youn, and Sean Lie}{Michael Bender}

\begin{summary}
  
\sumitem{``Ideal'' Scheduling}{This section describes the desirable
properties of an ``ideal'' global scheduler.}
  
\sumitem{Cilk Computation}{This section gives a brief overview of the
Cilk computation model.}
  
\sumitem{Main Data Structure}{This section describes the main data
structure used in the Cilk scheduler, the ready deque.}

\sumitem{Cilk Scheduler}{This section describes the Cilk scheduling
algorithm}.

\sumitem{Work Stealing}{This section describes the Cilk scheduler's
work stealing algorithm.}

\sumitem{Ready Deque Structural Lemma}{This section proves a lemma
that shows some very important properties of the ready deque.}

\end{summary}

\section{``Ideal'' Scheduling}
\seclabel{ideal_scheduler}

The goal is to develop a distributed scheduler that approximates the
performance of an ``ideal'' global scheduler.  A global scheduler
knows the entire system state (the state of all processors), while a
distributed scheduler knows only the local state.

To design a good scheduler, we need to understand what properties we
would want in an "ideal" global scheduler. Three of these properties
are outlined here:

\begin{enumerate}
\item \textbf{The scheduler should be \emph{greedy}.}

  Recall that a greedy scheduler has a very good bound on total run
  time [Graham, Brent]:

  \begin{equation}
    T_p \leq \frac{T_1} {P} + T_\infty
  \end{equation}
  
  {\bf Moral:} In most practical instances, $T_\infty$ is much less
  than $\frac{T_1}{P}$.  Thus, if we have a choice, we would like for
  any overhead incurred by a distributed scheduler to be a multiple of
  \emph{$T_\infty$}, rather than $\frac{T_1}{P}$. So, ideally, we
  would like any additional overhead to take away from the critical
  path of the DAG.
  
\item \textbf{The scheduler should have the \emph{Busy Leaves} property.}
 
  As shown in a previous lecture, a busy leaves scheduler has a good
  bound on $S_p$, the space for $p$ processors:
  
  \begin{equation}
    S_p \leq P \cdot S_1 
  \end {equation}
  
  This property is desirable because we do not want the scheduler to
  take up too much memory in overhead. If our scheduler can have the
  busy leaves property, each processor on average is using at most the
  amount of memory to schedule tasks as is required by the serial
  execution.

\item \textbf{The scheduler should have a small number of
  steals/migrations.}

  When a processor does not have any work to do, it must get (steal)
  work from another processor. In general this is very efficient
  because every processor is being used as much as possible, but at
  the same time there is also a cost associated with the steal itself.
  It is important to keep the number of steals low since this overhead
  can be quite high. Later we will show that our scheduler can, with
  high probability, bound the number of steals:
  
  \begin{equation}
    \text{number of steals} = O(PT_\infty)
  \end{equation}
  
  If we can develop a greedy scheduler that satisfies that bound, then
  the cost of steals, averaged over the number of processors, is
  $O(T_\infty)$. Therefore, in accordance with the moral stated above,
  the total cost of steals is related to the critical path instead of
  $\frac{T_1}{P}$ term.
\end{enumerate}

\section{Cilk Computation}
\seclabel{cilk_computation}

We will develop a scheduler for the Cilk computation model. For
clarity, we define the terms \defn{thread}, \defn{procedure},
\defn{child thread}, and \defn{child procedure} in terms of a Cilk
DAG.

A \defn{thread} is node in the Cilk DAG. In \figref{DAG}, the circle
nodes ($t_{11}, t_{12}, \ldots$) are threads. Every node in the Cilk
DAG is a thread. We assume that each thread executes in the same
amount of time. If this is not the case, longer threads can be modeled
as several smaller threads executing in series.

A \defn{procedure} is a group of threads that must execute in
series. In \figref{DAG}, procedures are represented by the gray large
oval ($\alpha_1, \alpha_2, \ldots$) around some group of thread
nodes. For the purpose of the scheduler, we will often refer to the
execution of a Cilk program in terms of these procedures as opposed to
threads.

A \defn{child thread} of thread $i$ is one of thread $i$'s successor
nodes in the DAG. Each thread can have 0, 1, or 2 child threads. For
example, in \figref{DAG}, threads $t_{13}$ and $t_{21}$ are both child
threads of $t_{12}$.  A child thread is either a thread that follows
$i$ within the same procedure, or is a spawned parallel thread. Thus,
we can have serial or parallel child threads. Note that a spawned
parallel thread will be the first thread in a seperate procedure to be
executed in parallel. Lastly, the only thread with no child threads is
the final thread executed in the program.

A \defn{child procedure} of a procedure $\alpha$ is a procedure that
was spawned from a thread in $\alpha$. While every thread can have up
to 2 child threads, a procedure can have any number of parallel child
procedures.  Each procedure can consist of any number of threads, and
each of these threads can spawn off a parallel procedure.  For
example, in \figref{DAG}, procedures $\alpha_2$ and $\alpha_4$ are
both child procedures of $\alpha_1$.

\begin{figure}  
  \begin{center}
    \includegraphics[width=4in]{8/DAG.eps}
  \end{center}
  \caption{An example Cilk DAG. Nodes (white circles) are marked
    $t_{ij}$ where $i$ is the procedure number. Procedures (gray
    ovals) are marked $\alpha_i$.}  \figlabel{DAG}
\end{figure}

\section{Main Data Structure}
\seclabel{deque}

In this section, we describe the main data structure that we will use
in our scheduler. Each processor maintains a data structure called a
\defn{ready deque}. A ready deque is a double ended queue.

Every level of the ready deque contains a procedure that is ready to
execute. Specifically, every level contains the thread that is next
ready to execute in a given procedure, but it is easier to think of
each level as just containing the portion of a procedure that has not
yet executed. For the remainder of this lecture, levels of the deque
will be treated as containing procedures.

Procedures are inserted at the bottom of the deque, but they may be
deleted from either end.  As shown in \figref{deque}, the deque
resembles a queue at the top, and a stack at the bottom. Soon we will
see that deletion from the top of the deque corresponds to that
procedure being stolen, and adding a procedure to the bottom
represents to a procedure being spawned. Likewise, a procedure being
deleted from the bottom of the deque represents the processor
beginning/resuming execution on that procedure.

As shown in \figref{deque1}, when used in serial execution, the deque
operates similar to the stack of a normal C program. Note that a
processor's ready deque represents procedures that are ready to
execute but not already executing. In particular, this means that the
procedure currently being executed by a processor is \emph{not} in the
deque. A processor can be busy and still have an empty deque.

\begin{figure}
  \begin{center}
    \includegraphics{8/deque.eps}
  \end{center}
  \caption{The ready deque. Procedures can be pushed into the deque
  from the bottom. Procedures can be popped (deleted) from the deque
  from both the bottom and the top.} \figlabel{deque}
\end{figure}

\begin{figure}
  \begin{center}
    \includegraphics[width=\textwidth]{8/deque1.eps}
  \end{center}
  \caption{The ready deque as we execute the DAG from \figref{DAG} on
    one processor. When we spawn a procedure, we push the previously
    running procedure onto the deque and start executing the new
    procedure. When we return from a procedure, we pop the previous
    procedure from the deque and continue executing it. This resembles
    the stack of a normal C program.}  \figlabel{deque1}
\end{figure}

\section{Cilk Scheduler}
\seclabel{cilk_scheduler}

This section describes the Cilk scheduler. The Cilk scheduler uses a
ready deque as its main data structure. It also satisfies the
properties that we originally set. \\

In the Cilk scheduler, a processor works on procedure $\alpha$ until:

\begin{enumerate}
\item $\alpha$ \emph{spawns} procedure $\beta$
  
  \begin{itemize}
  \item In this case, the processor first pushes procedure $\alpha$
    onto the bottom of the ready deque. Putting $\alpha$ onto the
    deque means that $\alpha$ has not fully executed yet but that it
    is not being worked on currently. Then the processor starts to
    execute the procedure $\beta$.
  \end{itemize}

\item We reach a \texttt{sync} in $\alpha$
  
  \begin{itemize}
  \item If there exists outstanding children of $\alpha$ that have not
    completed, the computation cannot proceed. In this case, the
    processor steals work from another processor. Work stealing is
    described in detail in the next section.  Note that the deque is
    empty in this case. That means that $\alpha$ has not been
    completely executed and it is \emph{not} on any deque.
  \item If there are no outstanding children of $\alpha$ (there are no
    children or all children have completed), the processor just
    continues executing $\alpha$ past the \texttt{sync}.
  \end{itemize}

\item $\alpha$ \emph{returns}

  \begin{itemize}
  \item If the processor's deque is nonempty, the processor pops the 
    procedure at the bottom of its deque and begins working on it.
  \item If the deque is empty, the processor tries to execute
    $\alpha$'s parent (the procedure that spawned $\alpha$). If
    $\alpha$'s parent is currently running, the processor steals work
    from another processor.
  \end{itemize}
  
\end{enumerate}

Intuitively, the Cilk scheduler enables each processor to do as much
work as possible. When a processor runs out of work, it steals some
work from another processor. 

However, of all the steps in the algorithm described above, the last
step is the most odd. Why do we need to try to run $\alpha$'s parent
when we return? The answer becomes clear when we look at what the
scheduler does when we reach a \texttt{sync} statement in, say,
procedure $\beta$. If $\beta$ has outstanding children, we just drop
$\beta$ and steal work from another processor. At that point,
procedure $\beta$ is not finished but it is also not on any
deque. This may seem strange but it is fine since we rely on $\beta$'s
last outstanding child to execute $\beta$ when it returns. Note that
all of the $\beta$'s children will have empty deques when they
return. Therefore, when a procedure $\alpha$ returns and its deque is
empty, it must check if it is the last outstanding child. If so, it
needs to continue execution of its parent. The way we check if it is
the last outstanding child is by simply trying to execute the
parent. If we are not the last outstanding child, we will simply fail
and work steal. If we are the last outstanding child, we will continue
to execute the parent.

Another point that should be mentioned is that single processor
execution is the same as a normal C program. As discussed in
\secref{deque}, the deque acts just like the stack in a normal C
program when we are running on 1 processor. If we spawn, we simply
suspend the parent and work on the child. Once the child is fully
executed, execution on the parent resumes.  This corresponds to the
depth-first execution of the DAG as shown in \figref{deque1}.

\section{Work Stealing}
\seclabel{work_stealing}

Here we discuss the work stealing algorithm in detail. When a
processor $p$ begins work stealing, it operates as follows:

\begin{itemize}
\item Processor $p$ chooses a victim processor uniformly at random
  from the set of all processors. Let us call the victim processor
  $v$.
\item If processor $v$'s deque is empty, processor $p$ repeats the
  first step.
\item Otherwise if processor $v$'s deque is not empty, processor $p$
  steals the top procedure from $v$'s deque. This is the reason the
  deque data structure must support pops from the top. Processor $p$
  then begins to work on this stolen procedure.
\end{itemize}

The reason we steal work at random is because a distributed scheduler
has no global knowledge of which processors have available
work. Ideally, we would just steal from a processor that has a lot of
work. However, since this is not possible in a distributed scheduler,
we steal at random.

\section{Ready Deque Structural Lemma}
\seclabel{structural_lemma}

We proceed by proving a structural lemma about the ready deque. This
will help us understand the data structure. It will also begin to
explain why the scheduling, and ready deque data structure, as defined
above allow the Cilk scheduler to work steal in a manner that steals
the optimal procedure ``with high probability''.

The idea is that the procedure at the top of the deque is always the
oldest (and closest to the root of the DAG). Therefore, if we always
steal the top procedure, we will likely be taking away from the
critical path. Therefore, the steal overhead can be used towards the
critical path as we originally set out to do. We will prove this next
lecture.

In this section we will prove a structural lemma about the ready deque
that will set up the proof for next class. Consider any processor $p$
at any time $t$.  Let $\alpha_0$ be the procedure that processor $p$
is working on.  Let $\alpha_1, \ldots, \alpha_k$ be the procedures in
$p$'s deque from bottom to top, as shown in \figref{alpha deque}. 

\begin{figure}
  \begin{center}
    \epsfbox{8/alphadeque.eps}
  \end{center}
  \caption{The ready deque of a processor. Procedure $\alpha_0$ is
  being executed while procedures $\alpha_1 \ldots \alpha_k$ are in
  the deque.}  \figlabel{alpha deque}
\end{figure}.

\begin{lemma}
At time $t$:
\end{lemma}

\begin{enumerate}
\item 
  \begin{itemize}
  \item $\alpha_i$ is a child of $\alpha_{i+1}$, for $i=0, \ldots,
    k-1$
  \item $\alpha_i$ is the \emph{only} active child of $\alpha_{i+1}$,
    for $i = 0, \ldots, k-2$. An active procedure is a procedure that
    is on a deque or currently executing.
  \item $\alpha_{k-1}$ may be one of many children of $\alpha_k$
  \end{itemize}
  %\figref{spawntree1} illustrates these claims by showing the
  %spawn tree (by which we mean the spawns that have occured on 
  %$p$ since it stole $\alpha_k$) of procesor $p$ at time $t$.
\item The number of actives leaves is at most $P$, the number of
  processors. A leaf is defined as a procedure with no current
  outstanding children. 
\end{enumerate}

We notice that in order for the scheduler to maintain the Busy Leaves
property as discussed above, the second property in the lemma hold. \\

%\begin{figure}
%  
%  \begin{center}
%  \epsfbox{8/spawntree1.eps}
%  \end{center}
%  
%\caption{Spawn tree at time $t$}
%\figlabel{spawntree1}
%\end{figure}.
%

\begin{proof}
To prove this lemma, we do an induction on actions.  That is, we
examine what happens during each type of action (spawn, return, steal,
sync) and show that all properties in the lemma are preserved. Note
that the following proof is not complete, but should be sufficient to
convince the reader that the structural lemma is true.

\begin{itemize}
\item {\bf Spawn:} We consider what happens when $\alpha_0$ spawns a
  child procedure. 

  Let us label the child of $\alpha_0$ as $\alpha_{\text{child}}$.
  Then we push $\alpha_0$ onto the deque and start executing
  $\alpha_{\text{child}}$.

%\begin{figure}
%  \begin{center}
%    \epsfbox{8/spawntree2.eps}
%  \end{center}
%  \caption{What happens during a spawn} 
%  \figlabel{spawntreeduringspawn}
%\end{figure}
%

  Consider the last property in the lemma. While $\alpha_0$ was a leaf
  prior to the spawn, following the spawn, $\alpha_{\textrm{child}}$
  becomes a leaf and $\alpha_0$ ceases to be a leaf. Thus, the number
  of leaves remains the same and by the inductive hypothesis, the
  property is met.

  Also, $\alpha_{\textrm{child}}$ is the only child of
  $\alpha_0$. This implies the first two properties of the lemma by
  inductive hypothesis directly.

\item {\bf Sync:} We consider what happens when we hit a \texttt{sync}
  while executing $\alpha_0$.
  
  In the base case, $\alpha_0$ has no outstanding children, and
  execution on $\alpha_0$ continues past the sync. In this case, if
  the structural lemma held before the sync was reached, it holds
  afterwards since no changes have been made to any deque.

  In the second case, $\alpha_0$ has outstanding children so we give
  up on $\alpha_0$. Then we attempt to steal work from another
  processor. If there is no work to be stolen, we sit idle. In this
  case, the lemma holds as this deque is now empty and the other
  deques have not changed. If there is work to be stolen, then this is
  a trivial case of the steal argument described below.

%  At this time, however, it is important to consider a special case of
%  the return action.  In particular the case where a processor first
%  attempts to steal the parent of the procedure that returned. The
%  reason this case is necessary is precisley to deal with syncs while
%  maintaining the last property of the lemma.\\
%
%  Consider a procedure $s$ which is suspended because a sync has been
%  reached. Then, once $s$'s final child has returned to processor $p$,
%  $s$ becomes a leaf! If processor $p$ then did normal work stealing,
%  it is possible for $s$ to be a leaf that is not being executed on
%  any processor. Then, if all $P$ processors could be executing a
%  leaf, and $s$ is an additional leaf, we violate the final property
%  of the Lemma.\\

\item {\bf Steal:} We consider what happens in the case of a steal.

%\begin{figure}
%  \begin{center}
%    \epsfbox{8/spawntree3.eps}
%  \end{center}
%\caption{What happens during a steal}
%\figlabel{spawntreeduringsteal}
%\end{figure}

  When another processor steals from this deque, right after a steal
  \emph{all} the remaining processes $\alpha_{k-1}$ through $\alpha_0$
  left in the deque must be only-children. This is true since we
  assumed, in the inductive hypothesis, that $\alpha_k$ is the only
  process with multiple children.

  The stealing processor now has only one process $\alpha_k$ in its
  deque. This also satisfies the first three properties of the
  lemma. Moreover, the number of active leaves is clearly preserved
  because the stolen procedure is not a leaf. 
  
\item {\bf Return:} We consider what happens in the case of a return.

  If the deque is not empty when $\alpha_0$ returns, we simply begin
  working on $\alpha_1$. This preserves the properties of the lemma
  trivially since we only took the bottom procedure off the deque.

  If the deque is empty when $\alpha_0$ returns and we start working
  on $\alpha_0$'s parent, the parent must have become a leaf. The
  parent is a leaf since it has no more outstanding
  children. Therefore, the number of active leaves is
  maintained. Also, the other deques do not change in this case and
  the current deque is empty. Thus, the first three properties of the
  lemma have also been preserved.

  If the deque is empty when $\alpha_0$ returns and we cannot execute
  $\alpha_0$'s parent, we work steal. We already considered the case
  of steals and the lemma is preserved.
\end{itemize}

In the next lecture, we will see how this scheduler can ensure that
the steals take away from the critical path ``with high
probability''. We find that we would like to steal the procedure that
contains the shallowest thread in the DAG on that processor. We claim
that this procedure is ``always'' at the top of the deque. The
remainder of this idea is presented in the following lecture.

\end{proof}

\end{guestlecture}
\theend

%temporarily end here

\remove{

\section{The Ideal Thread to Steal}
\seclabel{The ideal thread}

Notice that now we switch gears a bit and are talking about threads
instead of procedures. The intuition being that we want to work steal
in a manner that allows us to shorten the critical path.  However,
since we can't see into the future to know which available thread is
on the critical path, we would like to steal the the shallowest thread
in the DAG that can be executed.\\

So, given that we steal from processor $p$, we would like to steal the
procedure that contains the shallowest thread in the DAG on that
processor. We claim that this procedure is ``always'' at the top of
$p$'s deque. On a side note, we examine the top procedure. Because
this procedure is at the top of the deque, it must have spawned a
child.  Thus, the next thread to be executed in this top procedure has
already traversed one of its successor edges (the spawn edge). Since
each thread has at most 2 successors/edges, once the procedure is
stolen the thread will finish executing completely because the only
remaining edge (if any) is a serial successor/edge.\\

The remainder of this idea is presented in the following lecture.\\

%Some designers might be tempted to steal the root.  However, we know
%that the root may be stalled at a sync and is not the most productive
%thread at that time.  Instead, processor $p$ should steal a thread
%near the beginning of the DAG, ensuring that it is not stalled at a
%sync and that it will assist with making progress on the critical
%path.  Roughly speaking, the threads at the top of each deque are
%those that are closest to the beginning of the DAG, so stealing one
%thread from the top of each deque should reduce the critical path by
%one.

  This is best illustrated by \figref{p Steals the Shallowest Thread at
Top of Each Deque}, below.

\begin{figure}
\begin{verbatim}

Insert Figure 9

P_1
P_2
...
P_p

\end{verbatim}
\caption{Rough View of the Computation}
\figlabel{p Steals the Shallowest Thread at Top of Each Deque}
\end{figure}.

In this model, are we sure that the shallowest thread is on a
processor at the top of each deque?  Unfortunately, we cannot be
certain.  In \figref{A Counter Example for the Shallowest Thread
Computation} below, we see that two threads may be at the same depth
at one time.

\begin{figure}
\begin{verbatim}

Insert Figure 10

\alpha[i] >> \alpha[i+1]
spawn 
\beta[1]

\alpha[i] >> \alpha[i+1]
                                                        \beta[1]

\end{verbatim}
\caption{$\alpha[i]$ Spawns $\beta[1]$}
\figlabel{A Counter Example for the Shallowest Thread Computation}
\end{figure}.

%\section{Structural Lemma Corollary}
%\seclabel{The Distance to the Root Thread}
%
%Consider any processor $p$ at time $t$.  Let $u_0$ be the
%current thread of processor $\alpha_0$, executing on processor $p$.
%
%Let $u_1, u_2,...,u_k$ be the threads of procedures
%$\alpha_1, \alpha_2,...,\alpha_k$ in $p$'s deque ordered from bottom
%to top.
%
%Let d($u_i$) be the distance in the DAG to the root thread.
%
%Then,
%\begin{equation*}
%d(u_0) \leq d(u_1) < d(u_2) <.....< d(u_k)
%\end{equation*}


%\seclabel{Proof of the Corollary} 
%
%This will be covered in class tonight.  The diagram associated with
%the Corollary should be shown as \figref{something about \upsilon}.
%
%\begin{figure}(h)
%\begin{verbatim}
%
%Insert Figure 11
%
%deque with \upsilon_k to \upsilon_1 in stack
%with procedures.
 
%\end{verbatim}
%\caption{Proof of Corollary}
%\figlabel{something about \upsilon}
%\end{figure}.                           

%\seclabel{Outline of Proof}

%Accounting Argument

%2 buckets

%work   steal

%On each step, a processor puts 1 dollar into one of the 2 buckets, depending on action in that step.
%
%\begin{enumerate}
%
%\item {execute instruction - 1 dollar into work bucket}
%\item{steal attempt - 1 dollar into steal bucket}

%\end{enumerate}
%Claim: T_p \leq \frac {number of dollars in work bucket + number of dollars in steal bucket} {P}

%Lemma: At the end of the computation, the number of dollars in the work bucket is T_1.
%Proof: Easy.
%Lemma: at the end of the computation, the number of dollars in the steal bucket is O(PT\infty) with high probability.
%Proof: Next time.

%Main idea: If a thread is on the top of a dequefor CP steal attempts, then the probability that it is not stolen is 
%\begin{equation}
%C*P - C.
%\end{equation}
%
%\end{lecture}

\end{guestlecture}
\theend
}
