% Lecture notes - Lecture 7 - 09/29/2003
% C. Scott Ananian and Barbara Mack
% Edited by Sean Lie
%\documentclass{article}
%\usepackage{6895}

\begin{lecture}{7}{Memory Contention}{C. Scott Ananian, Barbara Mack, and Sean Lie}
  
  \begin{summary}
    \sumitem{Introduction to Memory Contention}{This section
      introduces the memory contention model that we will use for the
      remainder of this lecture.}  

    \sumitem{The $(P,M,T)$ Recycling Game}{This section presents a
      game played by an adversary which models concurrent random
      accesses to memory.}
    
    \sumitem{Bounds on the Expected Total Delay}{This section proves
      that the expected total delay is bounded.}
  \end{summary}
  
  \section{Introduction to Memory Contention}
  \seclabel{intro}
  
  \begin{figure}[h]
    \providecommand{\color}[2][rgb]{}
    \begin{center}\input{7/lecture-7-net.tex}\end{center}
    \caption{A parallel computer with $P$ processors and $M$ memory
      banks connected by a network.} 
    \figlabel{model}
  \end{figure}

  Our model, illustrated in \figref{model}, consists of $P$ processors
  and $M$ memories distributed along an interconnect.  $P$ processors
  concurrently access $M$ memory banks.  All $P$ processors access the
  memory banks at once, but they do not necessarily all access the
  same memory bank.  In our model the network can carry an unlimited
  number of requests and replies at once.

  A memory bank can only serve one request per time step. If more than
  one request if made of a particular bank, we have contention. In
  that case, we must choose which request to serve. All other requests
  are queued and are served in some future time step.

  \subsection{Contention Delay}
  We are interesting in how much delay is introduced by
  contention. The length of the delay depends on the pattern of the
  accesses to the memory banks: if all $P$ processors access the same
  memory bank (say, $M_1$), the delay is larger than if the accesses
  are distributed (even randomly) across the banks.  As a trivial
  example, no access is ever delayed if $P_i$ always accesses $M_i$.

  Since the pattern matters, we will analyze a random pattern.  This
  approach is similar to the one we will use for Cilk work-stealing in
  the next lecture.  Our results with this assumption are useful
  enough that there have even been proposals to hash memory addresses
  to make the location access more uniformly random.

  \section{The ${\bf (P,M,T)}$ Recycling Game}
  \seclabel{game}
  
  %Robert Blumofe and Charles Leiserson came up with this game as a means
  %to model memory accesses; according to one participant, the game is 
  %more fun than hockey.

  To find an upper bound on contention delay, we assume that an
  adversary picks which request is served when there is
  contention. The adversary cannot choose to ignore every request.
  That is, if 1 or more requests are queued, it much choose one.  The
  adversary chooses the \emph{worst possible} request to service in
  banks where there is a choice.

  To model such a situation, we use a game. The $(P,M,T)$ recycling
  game [Blumofe 1995] is a ``balls and bins'' game with $P$ balls
  (representing processors), $M$ bins (representing memory banks), and
  $T$ tosses (representing memory accesses).

  The game is played by an adversary, with all the balls resting
  initially in a reservoir.  On each turn of the game:
  \begin{enumerate}
  \item[(a)] The adversary chooses a subset of the balls in the
    reservoir and tosses each ball into a random bin (uniform and
    independent).  Each ball tossed counts towards $T$.  A ``toss'' is
    \emph{not} a turn.
  \item[(b)] For each bin that contains 1 or more balls, the adversary
    chooses which ball to return to the reservoir.  The rest remain in
    the bin.
  \end{enumerate}
  The combination of (a) and (b) is a ``turn.''
  
  The adversary's goal is to make the delay as large as possible.  (We
  will define ``delay'' in a moment.)  Intuitively, the adversary's
  best strategy is always to toss \emph{all} the balls in the
  reservoir each turn. This will result in the largest delay since it
  has the highest probability of contention. The adversary does not
  know the outcome of future random events, and therefore cannot
  exploit any foreknowledge of the future.

  The adversary can leave a ball in a bin indefinitely, as long as
  there is at least one other ball in that bin.  Therefore, the
  adversary can make a given processor wait ``forever'' for its access
  to $M_i$ as long there is always at least one other request
  contending for $M_i$.  The chance of this scenario occurring depends
  on the sizes of $P$ and $M$.  For example, if $M=1$ and $P\geq 2$
  than the adversary can always starve a given processor.

  We now define \defn{delay}.  Let $b_j$ be the number of balls in
  bins immediately after turn $j$.  Then the \defn{total delay}, $D$,
  is given by
  \begin{equation}\equlabel{delay-sum-1}
    D = \sum_j b_j \ .
  \end{equation}
  
  \figref{game} illustrates the actions in a 3-turn sample game where
  $P=5$, $M=4$, and $T=8$.  The total delay $D$ for this game is 3,
  which is the sum of $b_1=1$, $b_2=2$, and $b_3=0$.  The adversary's
  strategy for this game is not optimal: balls are left in the
  reservoir turns 1 and 3.  Tossing these balls may have allowed the
  adversary to increase the total delay.

  \begin{figure}\figlabel{game}
    \providecommand{\color}[2][rgb]{}
    \begin{center}
      Initial state: $P$ balls in reservoir, $M$ empty bins\\
      \input{7/lecture-7-game-0.tex}
      
      \vspace{1em}
      \begin{tabular}{ccccc}
	Turn& Snapshot after part (a) &Tosses& Snapshot after part (b) & Returns \\\hline\\
	1&\raisebox{-1.5cm}{\input{7/lecture-7-game-1a.tex}}&$\begin{array}{l}P_1,\\P_2,\\P_4\end{array}$&\raisebox{-1.5cm}{\input{7/lecture-7-game-1b.tex}}&$\begin{array}{l}P_2,\\P_4\end{array}$\\
	    \\
	    2&\raisebox{-1.5cm}{\input{7/lecture-7-game-2a.tex}}&$\begin{array}{l}P_2,\\P_3,\\P_4,\\P_5\end{array}$&\raisebox{-1.5cm}{\input{7/lecture-7-game-2b.tex}}&$\begin{array}{l}P_2,\\P_3,\\P_5\end{array}$\\
		\\
		3&\raisebox{-1.5cm}{\input{7/lecture-7-game-3a.tex}}&$\begin{array}{l}P_5\end{array}$&\raisebox{-1.5cm}{\input{7/lecture-7-game-3b.tex}}&$\begin{array}{l}P_1,\\P_4,\\P_5\end{array}$\\
		    \\
		    &\hfill\textbf{Total Tosses:}&$T=8$&\textbf{Total Delay:} $D=3$&\\
      \end{tabular}
    \end{center}
    \caption{Three turns of the $(P,M,T)$ recycling game with $P=5$
      balls, $M=4$ bins, and $T=8$ tosses.  Bins are on the left and
      the reservoir is on the right.  Circles represent balls.  Each
      snapshot represents the state of the game after part (a) of a
      turn (on the left) or part (b) of a turn (on the right).  Time
      advances toward the bottom of the page.}
  \end{figure}
  
  \section{Bounds for the Expected Total Delay}
  \seclabel{bounds}
  
  We can bound the expected total delay in terms of the number of
  balls (processors) $P$, bins (memory banks) $M$, and tosses (total
  accesses) $T$.

  \begin{theorem}
    The expected total delay $\expect{D}$ is bounded by
    \begin{equation*}
      \expect{D} \leq \frac{PT}{M} \ .
    \end{equation*}
    In particular, if $P=M$ then $\expect{D} \leq T$. 
  \end{theorem}
  
%  If we perform the tosses over a period of time, we will find that a
%  constant fraction of bins has 0 balls and a constant fraction of
%  bins has 1 ball.
%
%  CLRS contains an expanded discussion of the balls and bins game in
%  section 5.4.2.  A summary of Chapter 5 and Appendix C from CLRS is
%  provided below in \appref{clrs-summary}.
%
  \begin{proof}
    Without loss of generality, the adversary's strategy in part (b)
    of the game is FIFO: the returned ball is always the one which has
    been in the bin the longest.  In terms of total delay, it doesn't
    matter which ball is chosen in part (b), because the number of
    balls in a bin is the same regardless of the identity of the ball
    which is removed.

    Let $\delta_r$ be the number of turns where ball $r$
    finishes in a bin (not a reservoir).  Then, the total delay is
    \begin{equation}\equlabel{D-expansion}
      D = \sum_{r=1}^P \delta_r \ ,
    \end{equation}
    which is just another way of looking at the total-delay summation
    from \equref{delay-sum-1}.
    
    Let the \defn{$i$th cycle} of a ball $r$ be those turns in which
    ball $r$ remains in a bin from its $i$th toss to its removal.
    Then we say that the \defn{$i$th delay} of a ball is the number of
    turns in the $i$th cycle.  This quantity may be 0 if the ball went
    into a bin and was immediately removed.

    All balls are symmetric, so let us focus on the delay of the
    \emph{first} ball $\delta_1$. Let $t_1$ be the number of times
    ball 1 is tossed. Note that the total number of tosses is
    $T=\sum_{r=1}^P t_r$.  Let $d_i\geq 0$ denote the $i$th delay of
    ball 1; i.e. how long this ball stays in a bin during the $i$th
    cycle.  Therefore, we have
    \begin{equation}\equlabel{delta-expansion}
      \delta_1 = \sum_{i=1}^{t_1} d_i \ ,
    \end{equation}
    which is to say that the sum of all the cycle delays of ball 1 is
    the total delay of ball 1.

    The expected $i$th delay of ball one, $\expect{d_i}$, is simply
    the expected number of balls in the bin that ball one lands in
    after the $i$th toss. However, this is the same as the expected
    number of balls in any bin at any time. Since there are $P$ balls
    and $M$ bins, $\expect{\text{number of balls in bin}} \leq
    P/M$. Therefore, $\expect{d_i}\leq P/M$.

    From \equref{delta-expansion}, we get
    \begin{equation*}
      \expect{\delta_1} \leq t_1 \frac{P}{M} \ ,
    \end{equation*}
    
    By symmetry, if $t_r$ is the number of tosses of ball $r$, then
    $\expect{\delta_r} \leq t_r P/M$.  Then, using
    \equref{D-expansion}, we have
    \begin{eqnarray}\equlabel{final-sum}
      \expect{D} &=& \sum_{r=1}^P \expect{\delta_r} \nonumber\\
      &\leq& \sum_{r=1}^P t_r \frac{P}{M} \\
      &=& \frac{PT}{M} \nonumber
    \end{eqnarray}
  \end{proof}

  From this result we see that, in general, the total delay is not
  that high. This is expected since memory contention is not the
  biggest problem that we face in parallel processing.  It is more
  important to worry about load-balancing, which we will cover later
  this semester.\\

  {\bf Alternative Proof} There is an alternative way of proving this
  theorem. The technique in the alternative proof is very useful in
  high probability analysis. Since we were only concerned with
  expected value, we did not use this technique. However, it is useful
  to see it.

  We say the $i$th cycle of ball 1 is \defn{delayed} by ball $r$ if
  ball 1 and ball $r$ are in the same bin when ball $r$ is removed.
  Recall that we're assuming FIFO order, so balls are delayed
  \emph{once} or \emph{not at all}.
    
  This binary choice means we can use \defn{indicator random
  variables} (IRV), which are random variables that only take on two
  values, typically 0 and 1.  Indicator random variables have many
  nice properties that make them easy to deal with [CLRS, 5.2].  We'll
  be seeing more IRVs throughout the semester.

  We define an indicator random variable $x_{ir}$ as follows:
  \begin{equation*}
    x_{ir} = \cases{\begin{array}{l} 1 \text{ if $i$th cycle of ball
	  1 is delayed by $r$,} \\ 0 \text{ otherwise.}
    \end{array}}
  \end{equation*}
  
  Then, we have
  \begin{equation*}
    d_i = \sum_{r=2}^P x_{ir} \ ,
  \end{equation*}
  
  and
  \begin{equation}\equlabel{delta-expansion1}
    \delta = \sum_{i=1}^{t_1} \sum_{r=2}^P x_{ir} \ .
  \end{equation}
  
  By the definition of expectation,
  \begin{eqnarray*}
    \expect{x_{ir}} &=& 1\cdot \Pr{x_{ir}=1} + 0\cdot \Pr{x_{ir}=0} \\
    &=& \Pr{x_{ir}=1} \\
    &=& \Pr{\text{$i$th cycle of ball 1 is delayed by ball $r$}} \ .
  \end{eqnarray*}
  
  The probability that the $i$th cycle of ball 1 is delayed by ball
  $r$ is either $1/M$ or $0$: if $r$ is in a bin when ball 1 is
  tossed, the probability that we get tossed in that same bin is
  $1/M$; and if $r$ is in the reservoir instead, the probability of
  landing in a bin with it is 0.  Although a tighter bound on
  $\expect{x_{ir}}$ is possible, the following bound is adequate for
  our purposes.
   
  \begin{equation*}
    \expect{x_{ir}} \leq \frac{1}{M}
  \end{equation*}
    
  Using this bound and \equref{delta-expansion1}, we obtain
  \begin{eqnarray*}
    \expect{\delta} &=& \sum_{i=1}^{t_1} \sum_{r=2}^P \expect{x_{ir}} \\
    &\leq& \sum_{i=1}^{t_1} \sum_{r=2}^P \frac{1}{M} \\
    &=& t_1\frac{(P-1)}{M} \\
    &\leq& t_1\frac{P}{M} 
  \end{eqnarray*}

  Then \equref{final-sum} follows directly.

%I took out the review of CLRS. I don't think that it's very useful. In
%a course like this the audience should know what indicator random
%variables are... or at least know where to find out if they don't
%know.
%
%\appendix
%  
%  \section{Summmary of CLRS Chapter 5: Probabilistic Analysis and Randomized Algorithms.}
%  \applabel{clrs-summary}
%
%  As a background for our consideration of the Memory Contention
%  Problem, Professor Leiserson recommended that we take a look at CLRS
%  Chapter 5 and Appendix C.  Chapter 5, Probablistic Analysis and
%  Randomized Algorithms, outlines the general process for analyzing
%  quantities, such as the cost of a particular endeavor, or the run
%  time of an algorithm.  We are shown how to use prior knowledge of,
%  or make assumptions about, the distribution of the inputs.  The
%  expectation is then taken over the distribution of the possible
%  inputs.  In the end, we average the quantity (e.g. in terms of cost
%  or run time) over all possible inputs.
%
%  \subsection{Example 1: The Hiring Problem}
%
%  CLRS describes a situation where your company needs to hire a new
%  office assistant.  Management has decided to use an employment
%  agency for the candidate search.  The agency will send one candidate
%  for an interview every day.  There is a small cost associated with
%  conducting the interview and there is a larger cost for firing your
%  current assistant and hiring a new assistant through the agency.
%  You are looking for the best possible candidate, so you are willing
%  to absorb these costs.  Still, you want to gain a better
%  understanding of what the costs will be.\
%
%  Clearly, the overall costs will be related to how many interviews
%  are conducted and how many replacements are made.  If $c_1$ is taken
%  as the cost of interviewing, $n$ the number of candidates
%  interviewed, and $m$ the number of people hired, then
%  \begin{equation*}
%    O(nc_1 + mc_h) \ ,
%  \end{equation*}
%  This equation represents the total cost associated with the
%  interviewing and hiring process.\
%  
%  In the worst case, we decide to hire every candidate that we
%  interview, for a total hiring cost of $O(nc_h)$.  This assumes that
%  each new candidate is better than the last and that we are never
%  satisfied with our current assistant.  It is more likely that
%  applicants will be interviewed in more or less random order
%  (e.g. that a reputable agency will not send them to us for the
%  interview in worst to best order).  So the queue of candidates will
%  form a uniform random permutation, spreading the probability of any
%  given permutation across the total range of possible permutations,
%  $n!$.
%
%  \subsection{Example 2: A More Complex Hiring Problem}
%  
%  Instead of office assistants, consider the case of a university
%  (let's say Harvard) that is planning to hire a new professor.  As
%  with the office assistant, there will be a small cost to interview
%  candidates and a larger cost to replace the existing professor.
%  However, we also know that the most qualified candidates will be
%  more aggressive about negotiating their compensation packages, so
%  now there is some variability in the total cost structure.  How
%  would you account for this variability (within certain reasonable
%  bounds) based on the initial equation of $O(nc_1 = mc_h)$ as above?
%  Assume that if you retain your existing professor, he will not
%  negotiate a raise successfully.
%
%  \subsection{Indicator Random Variables}
%  %\seclabel{IRVs assessed through Bernoulli Trial - A Coin Toss}
%  \applabel{coin-toss}
%
%  Indicator Random Variables (IRVs) provide a convenient method for
%  converting between probabilities and expectations.  Suppose that we
%  are given a sample space $S$ and an event $A$,
%
%  \begin{equation*}
%    I\{A\} = \cases{\begin{array}{l}
%	1 \text{ if A occurs,}\\
%	0 \text{ if A does not occur.}
%    \end{array}}
%  \end{equation*}
%  
%  Taking the results of a coin toss as an example, let's say that
%  $I\{A\} = 1$ if $Y = \text{Heads (H)}$ and $0$ if $Y = \text{Tails
%  (T)}$.  Then,
%
%  \begin{eqnarray*}
%    \expect{X_h} &=& \expect{I\{Y=H\}} \\
%    &=& 1 * \Pr{Y=H} + 0*\Pr{Y=T} \\
%    &=& 1* \frac{1}{2} + 0* \frac{1}{2}\\
%    &=& \frac{1}{2} \\
%  \end{eqnarray*}
%  
%  If we continue to toss the coin and are interested in the number of
%  heads obtained, we will find that multiple coin toss trials are
%  expressed by
%
%  \begin{equation*}
%    X = \sum_{i=1}^n X_{i} \,
%  \end{equation*}
%  
%  where $X$ is the random variable denoting the total number of heads
%  in the $n$ coin tosses.
%
%  If you wish to calculate the expected number of heads, take the
%  expectation of both sides of the equation:
%
%  \begin{equation*}
%    \expect{X} = \expect{\sum_{i=1}^n X_{i}},
%  \end{equation*}
%  
%  We will find that the expected number of heads is $n/2$.
%
%  Later in Chapter 5, we see how the probability of streaks
%  (consecutive heads in the coin toss, for example) may be calculated.
%  See CLRS Section 5.4.3.
%
%  In conclusion, linearity of expectations makes the use of IRVs a
%  powerful analytical technique; it applies even when there is
%  dependence among the random variables.
%
%  \section{Further Summary of Chapter 5 and Appendix C}
%  Table of additional topics covered in CLRS, Chapter 5 and Appendix
%  C.
%
%  \begin{enumerate}
%  \item[5.3] \emph{Randomized Algorithms.}  This section introduces a
%    method to analyze problems where we do not know the distribution
%    of the inputs and therefore cannot analyze the average-case
%    behavior of an algorithm.  In one method, we permute the given
%    input of an array to produce a randomly permuted array.
%  \item[5.4] \emph{Probabilistic analysis and further uses of
%    indicator random variables.}  This section introduces more
%    advanced concepts in probabilistic analysis, particularly analysis
%    where arrays are used to predict frequency of a given attribute in
%    a closed set or population.  The Birthday Paradox provides a clear
%    example of this operation and leads to a discussion of the Balls
%    and Bins game which we have outlined in the notes above.
%  \item[C] \emph{Counting and Probability.}
%    Provides a review of many basic concepts including:
%    \begin{enumerate}
%    \item[C.1] Counting
%      \begin{enumerate}
%      \item   Strings
%      \item   Permutations
%      \item   Combinations
%      \item   Binomial coefficients
%      \item   Binomial bounds
%      \end{enumerate}
%    \item[C.2] Probability
%      \begin{enumerate}
%      \item   Axioms of probability
%      \item   Discrete probability distributions
%      \item   Continuous uniform probability distribution
%      \item   Conditional probability and independence
%      \item   Baye's Theorem
%      \end{enumerate}
%    \item[C.3] Discrete Random Variables
%      \begin{enumerate}
%      \item   Probability density function
%      \item   Joint probability density function
%      \item   Expected value of a random variable
%      \item   Variance and standard deviation
%      \end{enumerate}
%    \item[C.4] The Geometric and Binomial Distributions
%      \begin{enumerate}
%      \item   The geometric distribution of a Bernoulli Trial
%      \item   The binomial distribution
%      \end{enumerate}
%    \item[C.5] The Tails of the Binomial Distribution
%      \begin{enumerate}
%      \item   More Bernoulli Trials
%      \end{enumerate}
%    \end{enumerate}
%  \end{enumerate}
%  
\end{lecture}
\theend
