Massachusetts Institute of Technology Lecture 8 6.042J/18.062J: Mathematics for Computer Science 30 Sep 97 Professor Tom Leighton

Lecture Notes

In this lecture we will cover methods for evaluating sums. Sums arise very frequently in the analysis of algorithms and systems. The main examples in this lecture, however, are drawn from finance and physics.

1 Annuities

Would you prefer a million dollars today or $50,000 a year for the rest of your life? This is a question about the value of an annuity. An annuity is a financial instrument that pays out a fixed amount of money at the beginning of every year for some specified number of years. In particular, an n-year, m-payment annuity pays m dollars at the start of each year for n years. In some cases, n is finite, but not always. Examples include lottery payouts, student loans, and home mortgages. There are even Wall Street people who specialize in trading annuities.

A key question is what an annuity is worth. For example, lotteries often pay out jackpots over many years. Intuitively, $50,000 a year for 20 years ought to be worth less than a million dollars right now. If you had all the cash right away, you could invest it and begin collecting interest. But what if the choice were between $50,000 a year for 20 years and a half million dollars today? Now it is not clear which option is better.

In order to answer such questions, we need to know what a dollar paid out in the future is worth today. We will to assume that money can be invested at a fixed annual interest rate p. These days a good estimate for p is around 8%; we’ll use this value for the rest of the lecture.

Here is why the interest rate p matters. Ten dollars invested today at interest rate p will become (1 + p) = $10.80 in a year, (1 + p)^{2} ≈ $11.66 in two years, and so forth. Looked at another way, ten dollars paid out a year from now are only really worth 1/(1 + p) ≈ $9.26 today. The reason is that if we had the $9.26 today, we could invest it and would have $10.00 in a year anyway. Therefore, p determines the value of money paid out in the future.

1.1 The Value of an Annuity

Our goal is to determine the value of an n-year, m-payment annuity. The first payment of m dollars is truly worth m dollars. But the second payment a year later is worth only m/(1 + p) dollars. Similarly, the third payment is worth m/(1+p)^{2}, and the n-th payment is worth only m/(1+p)^{n−1}. The total value V of the annuity is equal to the sum of the payment values. This gives:

V = ∑_{i=1}^{n} \frac{m}{(1 + p)^{i−1}}

To compute the real value of the annuity, we need to evaluate this sum. One way is to plug in m, n, and p, compute each term explicitly, and then add them up. However, this sum has a special closed form that makes the job easier. (The phrase “closed form” refers to a mathematical expression without any summation or product notation.) First, lets make the summation prettier with some substitutions.

V = ∑_{i=1}^{n} \frac{m}{(1 + p)^{i−1}}

= ∑_{j=0}^{n−1} \frac{m}{(1 + p)^{j}} (substitute j = i − 1)

= m ∑_{j=0}^{n−1} x^{j} (substitute x = \frac{1}{1 + p})

The goal of these substitutions is to put the summation into a special form so that we can bash it with a theorem given in the next section.

1.2 The Sum of a Geometric Series

Theorem 1.1 For all n ≥ 1 and all x ≠ 0,

∑_{i=0}^{n−1} x^{i} = \frac{1 − x^{n}}{1 − x} .

The terms of the summation in this theorem form a geometric series. The distinguishing feature of a geometric series is that each term is a constant times the one before; in this case, the constant is x. The theorem gives a closed form for the sum of a geometric series that starts with 1.

Proof. The proof is by induction on n. Let P (n) be the predicate that for all x ≠ 1, ∑_{i=0}^{n−1} x^{i} = \frac{1−x^{n}}{1−x}. In the base case, P (1) holds because ∑_{i=0}^{0} x^{i} = x^{0} = 1 and \frac{1−x^{1}}{1−x} = 1.

In the inductive step, for n ≥ 1 assume that for all x ≠ 1, ∑_{i=0}^{n−1} x^{i} = \frac{1−x^{n}}{1−x}. We will use this to prove that for all x ≠ 1, ∑_{i=0}^{n} x^{i} = \frac{1−x^{n+1}}{1−x}.

∑_{i=0}^{n} x^{i} = x^{n} + ∑_{i=0}^{n−1} x^{i}
= x^{n} + \frac{1 − x^{n}}{1 − x}
= \frac{x^{n}(1 − x) + 1 − x^{n}}{1 − x}
= \frac{1 − x^{n+1}}{1 − x}

The second line follows from the first by the induction hypothesis. The remaining steps are only simplifications.

As if often the case, the proof by induction gives no hint about how the formula was found in the first place. Here is a more insightful derivation. The trick is to let S be the value of the sum:

S = 1 + x + x^{2} + … + x^{n−1}
−xS = −x − x^{2} − x^{3} − … − x^{n}

Adding these two equations gives:

(1 − x)S = 1 − x^{n}
S = \frac{1 − x^{n}}{1 − x}

We’ll say more about finding summation formulas later in the next lecture.

1.3 Return of the Annuity Problem

Now we can solve the annuity pricing problem. The value of an annuity that pays m dollars at the start of each year for n years is computed as follows:

V = m ∑_{j=0}^{n−1} x^{j}

= m\frac{1 − x^{n}}{1 − x}
= m\frac{1 − (\frac{1}{1+p})^{n}}{1 − \frac{1}{1+p}}

= m\frac{1 + p − (\frac{1}{1+p})^{n−1}}{p}

The first line is a restatement of the summation we obtained earlier for the value of an annuity. The second line follows by applying the theorem for the summation of a geometric series. In the third line, we undo the earlier substitution x = 1/(1 + p). In the final step, both the numerator and denominator are multiplied by 1 + p to simplify the expression.

The resulting formula is much easier to use than a summation with dozens of terms. For example, what is the real value of a winning lottery ticket that pays $50, 000 per year for 20 years? Plugging in m = $50, 000, n = 20, and p = 0.8 gives V ≈ $530, 180. Because payments are deferred, the million dollar lottery is really only worth about a half million dollars!

The question at the beginning of this section was whether you would prefer a million dollars today or $50, 000 a year for the rest of your life. Of course, this depends on how long you live, so optimistically assume that the second option is to receive $50, 000 a year forever. This sounds like infinite money!

We can compute the value of an annuity with an infinite number of payments by taking the limit of our previous formula as n tends to infinity:

V = lim_{n→∞} m\frac{1 + p − (\frac{1}{1+p})^{n−1}}{p}
= m\frac{1 + p}{p}

Plugging in m = $50, 000 and p = 0.8 gives only $675, 000. Amazingly, a million dollars today is worth much more than $50, 000 paid every year forever! Then again, if we had a million dollars today in the bank earning 8% interest, we could take out and spend $80, 000 a year forever. So the answer makes some sense.

1.4 More About Geometric Series

Theorem 1.1 gives a formula for the sum of a geometric series with n terms. The following theorem gives the sum of a geometric series with an infinite number of terms. This one is worth remembering!

Theorem 1.2 If |x| < 1, then

∑_{i=0}^{∞} x^{i} = \frac{1}{1 − x}.

Proof.

∑_{i=0}^{∞} x^{i} = lim_{n→∞} ∑_{i=0}^{n−1} x^{i}
= lim_{n→∞} \frac{1 − x^{n}}{1 − x}
= \frac{1}{1 − x}

The first equality follows from the definition of an infinite summation. In the second line, we apply the formula for the sum of an n-term geometric series given in Theorem 1.1. The final line follows by evaluating the limit; the x^{n} term vanishes since we assumed that |x| < 1.

We now have formulas enabling us to sum both finite and infinite geometric series. Some examples are given below. In each case, the solution follows immediately from either Theorem 1.1 (for finite series) or Theorem 1.2 (for infinite series).

1 + 1/2 + 1/4 + 1/8 + … = ∑_{i=0}^{∞}(1/2)^{i}
= \frac{1}{1 − (1/2)}
= 2

1 − 1/2 + 1/4 − 1/8 + … = ∑_{i=0}^{∞}(−1/2)^{i}
= \frac{1}{1 − (−1/2)}
= 2/3

1 + 2 + 4 + 8 + … + 2^{n−1} = ∑_{i=0}^{n−1} 2^{i}
= \frac{1 − 2^{n}}{1 − 2}
= 2^{n} − 1

1 + 3 + 9 + 27 + … + 3^{n−1} = ∑_{i=0}^{n−1} 3^{i}
= \frac{1 − 3^{n}}{1 − 3}
= \frac{3^{n} − 1}{2}

If the terms in a geometric series grow smaller as in the first example, then the series is said to be geometrically decreasing. If the terms in a geometric series grow progressively larger as in the last two examples, then the series is said to be geometrically increasing.

Here is a good rule of thumb: the sum of a geometric series is approximately equal to the term with greatest absolute value. In the first two examples, the largest term is equal to 1 and the sums are 2 and 2/3, both relatively close to 1. In the third example, the sum is about twice the largest term. In the final example, the largest term is 3^{n−1} and the sum is (3^{n} − 1)/2, which is only about a factor of 1.5 greater.

1.5 Related Sums

We now know all about sums of geometric series. But in practice one often encounters sums that cannot be transformed by simple variable substitutions to the form ∑ x^{i}.

A non-obvious, but useful way to obtain new summation formulas from old is by differentiating or integrating with respect to x. As an example, consider the following series.

∑_{i=1}^{n} ix^{i} = x + 2x^{2} + 3x^{3} + … + nx^{n}

This is not a geometric series, since the ratio between successive terms is not constant. Our formula for the sum of a geometric series cannot be directly applied. But suppose that we differentiate that formula:

\frac{d}{dx} ∑_{i=0}^{n} x^{i} = \frac{d}{dx} \frac{1 − x^{n+1}}{1 − x}

∑_{i=0}^{n} ix^{i−1} = \frac{−(n + 1)x^{n}(1 − x) − (−1)(1 − x^{n+1})}{(1 − x)^{2}}

= \frac{−(n + 1)x^{n} + (n + 1)x^{n+1} + 1 − x^{n+1}}{(1 − x)^{2}}
= \frac{1 − (n + 1)x^{n} + nx^{n+1}}{(1 − x)^{2}}

Often differentiating or integrating messes up the exponent of x in every term. In this case, we now have a formula for a series of the form ∑ ix^{i−1}, but we want a formula for the series ∑ ix^{i}. The solution is simple: multiply by x. This gives:

∑_{i=0}^{n} ix^{i} = \frac{x − (n + 1)x^{n+1} + nx^{n+2}}{(1 − x)^{2}}

Since we could easily have made a mistake, it is a good idea to go back and validate a formula obtained this way with a proof by induction.

Notice that if |x| < 1, then this sum converges to a finite value even if there are infinitely many terms. Taking the limit as n tends infinity gives the following theorem:

Theorem 1.3 If |x| < 1, then

∑_{i=0}^{∞} ix^{i} = \frac{x}{(1 − x)^{2}} .

As a consequence, suppose there is an annuity that pays im dollars at the end of each year i forever. For example, if m = $50, 000, then the payouts are $50, 000 and then $100, 000 and then $150, 000 and so on. It is hard to believe that the value of this annuity is finite! But we can use the preceding theorem to compute the value:

V = ∑_{i=1}^{n} \frac{m}{(1 + p)^{i}}

= m\frac{\frac{1}{1+p}}{(1 − \frac{1}{1+p})^{2}}
= m\frac{1 + p}{p^{2}}

The second line follows by an application of Theorem 1.3. The third line is obtained by multiplying the numerator and denominator by (1 + p)^{2}.

For example, if m = $50, 000, and p = 0.08 as usual, then the value of the annuity is V = $8, 437, 500. Even though payments increase every year, they increase only linearly with time; by contrast, dollars paid out in the future decrease in value exponentially with time. Payments in the distant future are almost worthless, so the value of the annuity is finite.

The important thing to remember is the trick of taking the derivative (or integral) of a summation formula. Of course, this technique requires one to compute nasty derivatives correctly, but this is at least theoretically possible!

2 Book Stacking

Suppose n books are stacked on the edge of a table as shown in Figure 1. How far can the top book overhang the table edge without the whole stack falling over? One book length? Two? We will prove that if there are enough books, then the top book can overhang arbitrarily far!

Figure 1: If n books are stacked on a table, how far can the top book overhang the edge without the whole stack falling over?

2.1 Formalizing the Problem

First, we have to reduce the real-world problem to a mathematical question. Define the length of a book to be 1 and assume that all books have the same mass. For 1 ≤ i ≤ n, define x_{i} to be the distance that book i extends beyond the right edge of book i + 1 as shown in Figure 2. Treat the table as book n + 1; that is, x_{n} is the distance book n extends beyond the table. Note that while most x_{i} should be positive, some can be negative as shown by book 3 in the figure. We can now write an expression for how far book 1 extends past the table edge:

total extension = x_{1} + x_{2} + … + x_{n} = ∑_{i=1}^{n} x_{i}

Figure 2: For 1 ≤ i ≤ n, the variable x_{i} is the distance that book i extends beyond the right end of book i + 1. The table is regarded as book n + 1, so x_{n} is the distance that book n extends beyond the end of the table. Not all x_{i} need be positive, as shown by x_{3}.

The position of the books is constrained by the condition that the stack must not fall over. A stack of books is stable iff for 1 ≤ k ≤ n, the center of mass for the top k books lies above the (k + 1)-st book. We will not prove this, but will accept it as an princimple of physics (i.e., an axiom). This stability constraint is illustrated for n = 1 in Figure 3.

Figure 3: This figure illustrates the idea of a stable stack of books for the special case of a single book. In the left diagram, the stack is stable because the center of mass lies above the next “book” (in this case, the table). In the right diagram, the book is unstable because the center of mass overhangs.

We will introduce some new variables to formalize the condition that the stack of books must be stable. For 1 ≤ k ≤ n, define m_{k} to be the how far the center of mass of the top k books is beyond the right end of book k + 1. For example, if m_{k} is 0, then the center of mass of the top k books is directly above the right end of book k + 1. Similarly, if m_{k} is −1, then the center of mass of the top k books is centered over the extreme left end of book k + 1. The stability constraint can now be formalized as the condition that for all 1 ≤ k ≤ n, we have −1 ≤ m_{k} ≤ 0. Keep in mind that the values of the variables m_{k} are a little funny; it will be easy to become confused by thinking of them as positive.

In lecture, we DEFINED the m_{k} to all be positive. The equations below are exactly the same as given in lecture, except the sign of the m_{k}’s will all be opposite.

There are some relations between the m_{k} and x_{i} variables that will be helpful later on. First, note that the center of mass of a book is a half unit left of its right end. Now suppose that the right end of book 1 is distance x_{1} beyond the right end of book 2. This means that the center of mass of book 1 must be distance x_{1} − \frac{1}{2} beyond the right end of book 2. In other words, m_{1} = x_{1} − \frac{1}{2}. With similar reasoning, we can derive a sequence of relationships between the m_{k} and x_{i} variables:

m_{1} = x_{1} − \frac{1}{2}
m_{2} = \frac{(x_{2} − \frac{1}{2}) + (x_{2} + x_{1} − \frac{1}{2})}{2}
= \frac{2x_{2} + x_{1} − 1}{2}
m_{3} = \frac{(x_{3} − \frac{1}{2}) + (x_{3} + x_{2} − \frac{1}{2}) + (x_{3} + x_{2} + x_{1} − \frac{1}{2})}{3}
= \frac{3x_{3} + 2x_{2} + x_{1} − \frac{3}{2}}{3}
…

m_{k} = \frac{kx_{k} + (k − 1)x_{k−1} + … + 2x_{2} + x_{1} − \frac{k}{2}}{k}

The physics problem is now reduced to a purely mathematical question: what is the maximum value of ∑_{i=1}^{n} x_{i} given the stability constraints and the above relationships?

2.2 Reducing the Problem to a Summation

Since the m and x variables are so closely related, we can simplify matters by restating the entire problem in terms of one set of variables or the other. In fact, we can eliminate the x variables by expressing each one as a function of the m variables and then substituting. The trick is to obtain an expression for x_{k} by subtracting successive equations in the above list of relationships.

km_{k} = kx_{k} + (k − 1)x_{k−1} + … + 2x_{2} + x_{1} − \frac{k}{2}
(k − 1)m_{k−1} = (k − 1)x_{k−1} + … + 2x_{2} + x_{1} − \frac{k − 1}{2}

km_{k} − (k − 1)m_{k−1} = kx_{k} − \frac{1}{2}
kx_{k} = \frac{1}{2} + km_{k} − (k − 1)m_{k−1}
x_{k} = \frac{1}{2k} + m_{k} − (1 − \frac{1}{k})m_{k−1}

The first two equations are relations derived in the previous section in a slightly different form. The third equation is obtained by subtracting the second from the first. The next two equations are obtained by simple rewriting. For this computation to be correct in the special case k = 1, define m_{0} = 0.

We can finish restating the problem entirely in terms of the m variables by substituting this expression for x_{k} into the formula for the total extension of the book stack over the end of the table.

extension = x_{1} + x_{2} + … + x_{n}
= (\frac{1}{2} + m_{1}) + (\frac{1}{4} + m_{2} − \frac{1}{2}m_{1}) + (\frac{1}{6} + m_{3} − \frac{2}{3}m_{2}) + … + (\frac{1}{2n} + m_{n} − (1 − \frac{1}{n})m_{n−1})
= \frac{1}{2} + \frac{1}{4} + \frac{1}{6} + … \frac{1}{2n} + \frac{m_{1}}{2} + \frac{m_{2}}{3} + … + \frac{m_{n−1}}{n} + m_{n}
= \frac{1}{2} ∑_{i=1}^{n} \frac{1}{i} + ∑_{i=1}^{n−1} \frac{m_{i}}{i + 1} + m_{n}

Now the x variables have been completely eliminated from the problem! All that remains is to assign values to the m variables so as to maximize the total extension, subject to the stability constraint that −1 ≤ m_{i} ≤ 0.

Since the total extension is increasing in all variables m_{i}, the best option is to set all m_{i} to the largest possible value, which is 0. At last we can express the greatest possible extension of a stack of books beyond the table edge as a simple summation:

\frac{1}{2} ∑_{i=1}^{n} \frac{1}{i}

What is the arrangement of books that gives the greatest extension? Since all m_{i} are zero, x_{k} = \frac{1}{2k} + m_{k} − (1 − \frac{1}{k})m_{k−1} = \frac{1}{2k}. An example of this arrangement for n = 4 is shown in Figure 4.

Figure 4: This is a stacking of 4 books that overhangs the table by the greatest possible distance. In general, the k-th book from the top should extend distance \frac{1}{2k} beyond the book below.

2.3 Evaluating the Sum– The Integral Method

The best possible total extension is now expressed as a sum that, coincidentally, appears frequently in computer science. (Fancy that!) In fact, it is important enough to have a name.

Definition The n-th Harmonic number is equal to ∑_{i=1}^{n} \frac{1}{i} and is denoted H_{n}.

We can rewrite the formula for greatest possible extension simply as H_{n}/2.

The first few Harmonic numbers are easy to compute. For example, H_{1} = 1, H_{2} = 1 + \frac{1}{2} = \frac{3}{2}, H_{3} = 1 + \frac{1}{2} + \frac{1}{3} = \frac{11}{16}, H_{4} = 1 + \frac{1}{2} + \frac{1}{3} + \frac{1}{4} = \frac{25}{12}. The fact that H_{4} is greater than 2 has special significance; it implies that the total extension of a 4-book stack is greater than one full book! This is the situation shown in Figure 4.

It would be nice to answer questions like, “How many books are needed to build a stack extending 100 book lengths beyond the table?” One approach to this question would be to keep computing Harmonic numbers until we found one exceeding 200. However, as we will see, this is not such a keen idea. In fact, for all we know, building a stack extending so far may be impossible regardless of the number of books.

All such questions would be settled if we could express H_{n} in a closed form. Unfortunately, no closed form is known and probably none exists. As a second best, however, we can find closed forms for very good approximations to H_{n} using the Integral Method. The idea of the Integral Method is to bound terms of the sum above and below by simple functions as suggested in Figure 5. The integrals of these functions then bound the value of the sum above and below.

The Integral Method gives the following upper and lower bounds on the harmonic number H_{n}:

H_{n} ≤ 1 + ∫_{1}^{n} \frac{1}{x} dx = 1 + ln n
H_{n} ≥ ∫_{0}^{n} \frac{1}{x + 1} dx = ∫_{1}^{n+1} \frac{1}{x} dx = ln(n + 1)

Figure 5: This figure illustrates the Integral Method for bounding a sum. The area under the “stairstep” curve over the interval [0, n] is equal to H_{n} = ∑_{i=1}^{n} 1/i. The function 1/x is everywhere greater than or equal to the stairstep and so the integral of 1/x over this interval is an upper bound on the sum. Similarly, 1/(x+1) is everywhere less than or equal to the stairstep and so the integral of 1/(x + 1) is a lower bound on the sum.

These bounds imply that the harmonic number H_{n} is around ln n. Since ln n grows without bound, albeit slowly, we can make a stack of books that extends arbitrarily far.

For example, to build a stack extending 100 book lengths beyond the table, we need a number of books n so that H_{n} = 200. Exponentiating the above inequalities gives:

e^{H_{n}−1} ≤ n ≤ e^{H_{n}} − 1
e^{199} ≤ n ≤ e^{200} − 1

The number of books required has 86 digits!

2.4 More about Harmonic Numbers

In the preceding section, we showed that H_{n} is about ln n. A even better approximation is known:

H_{n} = ln n + γ + \frac{1}{2n} + \frac{1}{12n^{2}} + \frac{ϵ(n)}{120n^{4}}

Here γ is a value 0.577215664 … called Euler’s constant, and ϵ(n) is between 0 and 1 for all n. We will not prove this formula.

The shorthand H_{n} ∼ ln n is used to indicate that the leading term of H_{n} is ln n. More generally, the notation f(n) ∼ g(n) means that lim_{n→∞} \frac{f(n)}{g(n)} → 1.

We also might write H_{n} ∼ ln n + γ to indicate two leading terms. While this notation is widely used, it is not really right. Referring to the definition of ∼, we see that while H_{n} ∼ ln n + γ is a true statement, so is H_{n} ∼ ln n + c where c is any constant. The correct way to indicate that γ is the second-largest term is H_{n} − ln n ∼ γ.

The reason that the ∼ notation is useful is that often we do not care about lower order terms. For example, if n = 100, then we can compute H(n) to great precision using only the two leading terms:

|H_{n} − ln n − γ| ≤ |\frac{1}{200} − \frac{1}{120000} + \frac{1}{120 · 100^{4}} | < \frac{1}{200}