Massachusetts Institute of Technology Lecture ?? 6.042J/18.062J: Mathematics for Computer Science October 17, 1996 Professors Nancy Lynch and Tom Leighton

Notes 12

For the next eight lectures, we are going to be talking about methods for counting. Today we’ll begin by covering methods for evaluating sums and products. Then, next week we’ll show you how to solve recurrences. Later we’ll cover combinatorics and generating functions. This material is useful in several areas and will be particularly useful when you study the running time of programs in 6.046.

1 Annuities

We’ll start with an example many of you encountered when you got a student loan, the problem of annuity pricing. An annuity is a financial instrument which pays a fixed amount at the beginning of every year for some specified length of time. In particular,

Definition 1.1 An n-year, $m-payment annuity pays $m at the start of each year for n years.

In some cases n is finite, but it doesn’t have to be. Examples include student loans, home mortgages (pay every month for 30 years), “Megabucks” ($50,000 a year for 20 years), and retirement/life insurance benefits. Folks on Wall Street specialize in these things.

Key question is what an annuity is worth. It is clearly better to have $1 million today than $50K a year for 20 years, because there is inflation and interest and you get to use the money right away. But is it better to have $500K today?

In order to answer the question, we need to know what one dollar paid in a year is worth today. This is usually tied to some interest rate.

Assume that there is a fixed interest rate p. So $1 today is worth $(1 + p) after 1 year, and $(1 + p)^{2} after 2 years. (interest compounds).

Today p is about 8%.

Now we can see what the Megabucks payment is worth.

The first payment of m is worth m. The second payment of m is only worth \frac{m}{1+p}. (Its value goes up by (1 + p) over the year before it is paid, so it is worth m when paid.) Generalizing, the n-th payment payment is worth only \frac{m}{(1+p)^{n−1}} .

The total value is the sum of all the payments, which we write as

V = ∑_{i=1}^{n} \frac{m}{(1 + p)^{i−1}}

So now to answer the question we need to evaluate this sum. With values for m, n, p we could do this by actually adding the terms, but this sum has a special closed form which makes the job easier. That is, we can rewrite the expression for V as a simple expression without the ∑ notation.

First we rewrite the sum a little:

V = ∑_{i=0}^{n−1} \frac{m}{(1 + p)^{i}}

It is easy to see that we are still adding the same terms, so this sum is the same as the previous, just written differently. Now,

V = m ∑_{i=0}^{n−1} x^{i} where x = \frac{1}{1 + p}

Theorem 1.2

∀n ≥ 1, x ≠ 1, ∑_{i=0}^{n−1} x^{i} = \frac{1 − x^{n}}{1 − x}

Proof. By induction on n.

Base case: n = 1

∑_{i=0}^{0} x^{i} = x^{0} = 1 = \frac{1 − x^{1}}{1 − x}

Inductive step:

∑_{i=0}^{n−1} x^{i} = ∑_{i=0}^{n−2} x^{i} + x^{n−1}

= \frac{1 − x^{n−1}}{1 − x} + x^{n−1} (by inductive hypothesis)

= \frac{1 − x^{n−1} + x^{n−1}(1 − x)}{1 − x}

= \frac{1 − x^{n}}{1 − x}

This proof is fine, but it gives no intuition for how we got the answer. An alternate proof with better intuituion:

S = 1 + x + x^{2} + ⋯ + x^{n−1}
+ − xS = −x − x^{2} − ⋯ − x^{n−1} − x^{n}
(1 − x)S = 1 − x^{n}

So now we can solve the annuity pricing problem:

V = m\frac{1 − x^{n}}{1 − x}

= m\frac{1 − (\frac{1}{1+p})^{n}}{1 − (\frac{1}{1+p})}

= m\frac{1 + p − \frac{1}{(1+p)^{n−1}}}{p}

For Megabucks, m = $50, 000, n = 20, p = .08 ⇒ V = $530, 180. So million dollar prize is really only worth a half million!

Notice that if it were paid forever it wouldn’t make much difference.

lim_{n→∞} V = m\frac{1 + p}{p}

m = $50, 000, p = .08 ⇒ V = $675, 000

You should remember the formula for this sum (or how to derive it), because it comes up in all sorts of applications.

Theorem 1.3

|x| < 1 ⇒ ∑_{i=0}^{∞} x^{i} = \frac{1}{1 − x}

Some examples of these sums, finite and infinite:

1 + 1/2 + 1/4 + 1/8 + ⋯ = \frac{1}{1 − 1/2} = 2

1 + 2 + 4 + 8 + ⋯ + 2^{n−1} = \frac{1 − 2^{n}}{1 − 2} = 2^{n} − 1

1 + 3 + 9 + 27 + ⋯ + 3^{n−1} = \frac{1 − 3^{n}}{1 − 3} = \frac{3^{n} − 1}{2}

These are called geometric series. In general, the sum of a geometric series is about the same as the largest term. (Notice that whether this is the first or last term depends on whether |x| < 1)

2 Related sums

Sometimes we get sums that aren’t geometric, but are close. For example

∑_{i=1}^{t} ix^{i} = x + 2x^{2} + 3x^{3} + ⋯ + tx^{t}

It turns out that there is a nice trick for evaluating this sum that is fairly useful. The idea is to take the formula for a geometric series and differentiate with respect to x.

∑_{i=0}^{n−1} x^{i} = \frac{1 − x^{n}}{1 − x}

∑_{i=0}^{n−1} ix^{i−1} = \frac{−nx^{n−1}(1 − x) − (−1)(1 − x^{n})}{(1 − x)^{2}}

= \frac{−nx^{n−1} + nx^{n} + 1 − x^{n}}{(1 − x)^{2}}

= \frac{1 − nx^{n−1} + (n − 1)x^{n}}{(1 − x)^{2}}

But we wanted x^{i}, not x^{i−1}, so multiply by x.

∑_{i=0}^{n−1} ix^{i} = \frac{x − nx^{n} + (n − 1)x^{n+1}}{(1 − x)^{2}}

And we wanted the sum to t, not n − 1, so set t = n − 1 and substitute.

∑_{i=0}^{t} ix^{i} = \frac{x − (t + 1)x^{t+1} + tx^{t+2}}{(1 − x)^{2}}

And we wanted the sum from i = 1, not 0, but the i = 0 term = 0, so just delete it. Now finally,

∑_{i=1}^{t} ix^{i} = \frac{x − (t + 1)x^{t+1} + tx^{t+2}}{(1 − x)^{2}}

Now to make sure that we didn’t make a mistake we really ought to go verify this by induction. For homework you’ll do ∑ i^{2}x^{i} by taking another derivative.

Notice that if |x| < 1, then this sum converges even if t is infinite.

Theorem 2.1

|x| < 1 ⇒ ∑_{i=1}^{∞} ix^{i} = \frac{x}{(1 − x)^{2}}

For example, an annuity that pays $im at the end of year i is worth

\frac{m\frac{1}{1+p}}{(1 − \frac{1}{1+p})^{2}} = \frac{m(1 + p)}{p^{2}}

m = $50K, p = .08 ⇒ V = $8, 437, 500

So even though you get paid more and more each year, total current value is finite. Eventually, value of payments far into the future aren’t worth enough to matter, even though they are increasing.

Another example:

∑_{i=1}^{∞} i2^{−i} = \frac{1/2}{(1/2)^{2}} = 2

3 Book Stacking

We’re now going to look at a completely different type of sum that raises from a physics problem. The problem is to make a stack of n books on the edge of a table (that won’t fall over) where the top book hangs out completely off the table.

… n books  top book hanging out off edge! table

Is this even possible? Yes! The demo worked in class.

The trick is to put the top book so that its center of mass is at the edge of the second book, and then place the second book so that the center of mass of the top two is at the edge of the third book, and so on. So in general, we place the kth book such that the center of mass of the top k books is at the edge of the (k + 1)st book.

Assuming the book size is 2, how far do we place the right edge of the kth book out from the right edge of the (k + 1)st book?

Clearly we place the top book out 1.

Let x be the distance from the edge of the third book to the edge of the second. Then the center of mass m of the top two is given by 2m = (x + 1 − 1) + (x − 1) = 2x − 1. To be stable we need this over the third book, m ≤ 0, so 2x − 1 ≤ 0, which means x ≤ 1/2. We’ll put it right at the edge.

If we do this again for the top 3, we get that the center of mass m is given by 3m = (x + 1/2 + 1 − 1) + (x + 1/2 − 1) + (x − 1) = 3x − 1, so we place the third book 1/3 from the fourth.

Now it looks like a pattern. so let’s guess that we can place the kth book 1/k from the (k + 1)st book.

Center of mass for k books with respect to (k + 1)st book is then:

(1 + 1/2 + ⋯ + 1/k − 1) + (1/2 + 1/3 + ⋯ + 1/k − 1) + ⋯ + (1/k − 1)

= ∑_{i=1}^{k} m_{i} where m_{i} = (∑_{j=i}^{k} 1/j) − 1

= ∑_{i=1}^{k} [(∑_{j=i}^{k} 1/j) − 1]

= (∑_{i=1}^{k} ∑_{j=i}^{k} 1/j) − k

= (∑_{j=1}^{k} ∑_{i=1}^{j} 1/j) − k

= (∑_{j=1}^{k} 1) − k

= k − k So the center of mass is at the edge of the (k+)1st book, which means that it balances, as it is supposed to.

A bit more should be said about how we reversed i and j in the double sum. The idea is to write the double sum out in a table, and then it is clear that summing the columns first instead of the rows is helpful, and it is also clear what i and j should range over so that the sum is correct. So with j going across and i going down we write:

1+1/2+1/3 + ⋯ +1/k
1/2+1/3 + ⋯ +1/k
1/3 + ⋯ +1/k
⋯
1/k

It is easy to see that the columns each add up to 1, so the right thing to sum is i from 1 to j, and then sum that over all j (which is what we did).

So we know now that we have a stable scheme, but how far out is the top book? It is shifted out

H_{n} = ∑_{i=1}^{n} \frac{1}{i}

How big is this?

H_{1} = 1
H_{2} = 1 + 1/2 = 3/2
H_{3} = 11/6
h_{4} = 25/12 > 2

So 4 books is enough to get one out over edge!

The H_{n} are actually well studied sums called the Harmonic Numbers. They grow to infinity. Let’s try to get a handle on how fast. That is, how many books do we need to reach a given distance as a function of the distance?

We use what is known as the integral method for bounding a sum. We bound it above and below by an integral:

[figure: .; .; .; 2; 3; 4; 5; 6; 7; n − 1; n; 1; \frac{1}{x}; \frac{1}{x+1}]
Area in boxes = H_{n} = ∑_{i=1}^{n} \frac{1}{i}
H_{n} ≤ 1 + ∫_{1}^{n} \frac{1}{x}dx = 1 + ln n
H_{n} ≥ ∫_{0}^{n} \frac{1}{x + 1}dx = ∫_{1}^{n+1} \frac{1}{x}dx = ln(n + 1)
ln(n + 1) ≤ H_{n} ≤ ln(n) + 1

So to get top book out 100 beyond desk we need about e^{99} to e^{100} books. (That would take a while to build...) Actually, there is a better approximation, but it is beyond what we will do in this course, so we’ll just state it as a fact:

H_{n} = ln(n) + γ + \frac{1}{2n} − \frac{1}{12n^{2}} + \frac{ϵ(n)}{120n^{4}}

where 0 < ϵ(n) < 1 and γ = Euler’s Constant ≈ 0.577215664

Not all that much is known about γ, not even whether is is irrational. Note that it is usually good enough to get close to answer. We write

H_{n} ∼ ln(n)

to indicate that the leading term in series is ln(n). In general,

f(n) ∼ g(n) ⇒ lim_{n→∞} \frac{f(n)}{g(n)} → 1

Some people write H_{n} ∼ ln(n) + γ, meaning that the next term is γ, but it really should be H_{n} − ln(n) ∼ γ. There is some more related notation that will be covered in recitation.

4 Products

Now we’ll look at what happens if we have a product instead of a sum. In particular, about how big is n! = ∏_{i=1}^{n} i?

We don’t know much about products, so the best thing to do is to convert them to sums by taking logarithms of both sides. So,

ln(n!) = ln(1 · 2 · 3 · · · · · n)

= ln(1) + ln(2) + ln(3) + ⋯ + ln(n)

= ∑_{i=1}^{n} ln(i)

So we can get an expression for n! by evaluating this sum. Approximate it with the integral method as before, and we get

∫_{1}^{n} ln xdx ≤ ln n! ≤ ∫_{0}^{n} ln(x + 1)dx

∫_{1}^{n} ln xdx = x ln x − x|_{1}^{n}

= n ln n − n + 1

= n ln (\frac{n}{e}) + 1

and,

∫_{0}^{n} ln(x + 1)dx = ∫_{1}^{n+1} ln xdx

= x ln x − x|_{1}^{n+1}

= (n + 1) ln(n + 1) − (n + 1) + 1

= (n + 1) ln (\frac{n + 1}{e}) + 1

so,

n ln (\frac{n}{e}) + 1 ≤ ln(n!) ≤ (n + 1) ln (\frac{n + 1}{e}) + 1

(\frac{n}{e})^{n} e ≤ n! ≤ (\frac{n + 1}{e})^{n+1} e

So now we know that n! grows like (\frac{n}{e})^{n}.

We can go back an compute error terms, getting a better bound for n!:

(\frac{n}{e})^{n} \sqrt{2πn}e^{\frac{1}{12n+1}} ≤ n! ≤ (\frac{n}{e})^{n} \sqrt{2πn}e^{\frac{1}{12n}}

⇒ n! ∼ \sqrt{2πn} (\frac{n}{e})^{n} (Stirling’s Formula)

There is no closed form expression for n!, but from Stirling’s formula we can get nearly tight bounds on n! that are in closed form.

For example, if n = 100, then

(\frac{100}{e})^{100} \sqrt{200π}e^{\frac{1}{1201}} ≤ n! ≤ (\frac{100}{e})^{100} \sqrt{200π}e^{\frac{1}{1200}}

e^{\frac{1}{1201}} = 1.00083299 and e^{\frac{1}{1200}} = 1.00083368, so the upper bound is ≤ (1 + 10^{−6})lower bound, which means the approximation is correct to six digits! This is good enough for many practical purposes. If needed, the bounds can be made tighter with further error calculations.

Note that it is pretty neat that we were able to get such tight bounds on n!. It is clear that n! > 2^{n} and n! < n^{n}, but who would guess that n! ∼ \sqrt{2πn} (\frac{n}{e})^{n}?

Remember Stirling’s Formula because it is important and will come up again.

5 Some Additional Summations

5.1 Sums of Squares

You remember from an earlier problem set that we asked you to prove by induction that

∑_{i=0}^{n} i^{2} = \frac{n(n + 1)(2n + 1)}{6} = \frac{1}{3}n^{3} + \frac{1}{2}n^{2} + \frac{1}{6}n.

Now we’re in a part of the course where the idea is to derive formulas for summations. So it’s natural to ask — how might you be able to come up with a formula like this?

First, note that using the integral method, we can roughly approximate ∑_{i=0}^{n} i^{2} by

∫_{0}^{n} x^{2}dx = \frac{1}{3}n^{3}.

So it’s reasonable to guess that the answer is approximately \frac{1}{3}n^{3}. Based on this, let’s guess that the sum has the form an^{3} + bn^{2} + cn + d, for some values of a, b, c, d that we haven’t yet figured out.

Since ∑_{i=0}^{0} i^{2} = 0 = d, we know d = 0. Now, plugging in the sums for n = 1, 2, 3, we get the following three equations in the three unknowns a, b, c.

a + b + c = 1
8a + 4b + 2c = 5
27a + 9b + 3c = 14

Solving this system, we get (surprise, surprise),

a = \frac{1}{3}

b = \frac{1}{2}

c = \frac{1}{6}

Now — and this is very important — we have not proved that this is the correct formula for ∑_{i=0}^{n} i^{2}. We’ve only established that it is consistent with the first four values of n. But now we can go and verify that it works for all n by induction.

Another possible way to approach this problem is to notice that

∑_{i=1}^{n} i^{2} = ∑_{i=1}^{n} i^{2}x^{i−1} : for x = 1

We saw before that

∀x ≠ 1, ∑_{i=1}^{n} ix^{i} = \frac{x − (n + 1)x^{n+1} + nx^{n+2}}{(1 − x)^{2}}

So

∑_{i=1}^{n} i^{2}x^{i−1} = \frac{1 + x − (n + 1)^{2}x^{n} + (2n^{2} + 2n − 1)x^{n+1} − n^{2}x^{n+2}}{(1 − x)^{3}}

We want to evaluate at x = 0, but of course there is a problem because this is 0/0. So take the limit as x → 1 and use L’Hôpital’s rule. We’re not going to write it all out here. Take three derivatives on top and bottom and simplify and the right answer comes out. This is such a mess that checking by induction would be a good idea.

A third possible way is to use a trick to squeeze the exact answer out of the integral method. We know that

∑_{i=1}^{n} i^{2} ∼ \frac{n^{3}}{3}

12 Lecture ??: Notes 12 So we write an expression for how much this is wrong:

E_{n} = ∑_{i=1}^{n} i^{2} − \frac{n^{3}}{3}

Then

E_{n−1} = ∑_{i=1}^{n−1} i^{2} − \frac{(n − 1)^{3}}{3}

So

E_{n} − E_{n−1} = n^{2} − \frac{n^{3}}{3} + \frac{(n − 1)^{3}}{3}

= n^{2} − \frac{n^{3}}{3} + \frac{n^{3}}{3} − \frac{3n^{2}}{3} + \frac{3n}{3} − \frac{1}{3}

= n − \frac{1}{3}

⇒ E_{n} = (n − \frac{1}{3}) + E_{n−1}

= (n − \frac{1}{3}) + \overbrace{(n − 1 − \frac{1}{3}) + E_{n−2}}

= (n − \frac{1}{3}) + (n − 1 − \frac{1}{3}) \overbrace{(n − 2 − \frac{1}{3}) + E_{n−3}}

⋯

= (n − \frac{1}{3}) + (n − 1 − \frac{1}{3}) + ⋯ + (2 − \frac{1}{3}) + (1 − \frac{1}{3}) + E_{0}

= ∑_{i=1}^{n} (i − \frac{1}{3})

= ∑_{i=1}^{n} i − ∑_{i=1}^{n} \frac{1}{3}

= \frac{n(n + 1)}{2} − \frac{n}{3}

⇒ E_{n} = \frac{n^{2}}{2} + \frac{n}{6}
⇒ ∑_{i=1}^{n} i^{2} = \frac{n^{3}}{3} + \frac{n^{2}}{2} + \frac{n}{6}

5.2 Some Infinite Sums

Let’s see how the integral method proves that the sum

∑_{i=1}^{∞} \frac{1}{i^{2}}

converges. (Many of you may have seen this in calculus.) Using the technique from earlier in lecture, we know that this sum, minus its first term, is less than the area under the curve y = 1/x^{2} from 1 to ∞. That it,

∑_{i=1}^{∞} \frac{1}{i^{2}} = 1 + ∑_{i=2}^{∞} \frac{1}{i^{2}}
≤ 1 + ∫_{1}^{∞} x^{−2}dx
= 2.

Thus, our infinite sum is at most 2, and in particular it converges. In fact, one can prove (using complex analysis) that the exact value of the sum is π^{2}/6; we won’t prove this here.

Next let’s look at the following doubly infinite sum

∑_{i=0}^{∞} ∑_{j=1}^{∞} x^{i}y^{j},

where (critically)

0 < x, y < 1.

We can therefore re-write this as

∑_{i=0}^{∞} ∑_{j=1}^{∞} x^{i}y^{j} = ∑_{i=0}^{∞} x^{i} (∑_{j=1}^{∞} y^{j})

= ∑_{i=0}^{∞} x^{i} (\frac{1}{1 − y} )
= (\frac{1}{1 − y} ) ∑_{i=0}^{∞} x^{i}
= \frac{1}{(1 − x)(1 − y)}.

Note that these manipulations were justified because we had a series of positive real numbers whose sum converged.

Just to reinforce this, here’s a case in which re-arrangements of the sum are clearly not allowed. Let

S = ∑_{i=0}^{∞}(−1)^{i} = 1 + (−1) + 1 + (−1) + · · · .

Consider the following:

S = [1 + (−1)] + [1 + (−1)] + ⋯ = 0 + 0 + ⋯ = 0.

S = 1 + [(−1) + 1] + [(−1) + 1] + ⋯ = 1 + 0 + 0 + ⋯ = 1.

What’s the problem here? The point is that S is not well-defined, since it does not converge.

6 Asymptotic Notation

Above we wrote the fuzzy statement that the sum ∑_{i=0}^{n} i^{2} “is approximately \frac{1}{3}n^{3}.” Here, let’s talk about some ways to make statements like this more precise.

6.1 Big-Oh Notation

Definition 6.1 Given functions f and g, we say that f(x) is O(g(x)) if there exist constants c and x_{0} (independent of x) so that

∀x ≥ x_{0} : 0 ≤ f(x) ≤ c · g(x).

So big-Oh notation is a way of saying that f(x) is bounded from above by g(x), ignoring (i) constant factors, and (ii) small values of x. Just to make sure we have the definition straight, here are some examples: x^{2} is O(x^{3}). 100x is O(x). x + \sqrt{x} is O(x). 2^{x} is O(4^{x}). 4^{x} is not O(2^{x}). log_{2}(x) is O(log_{10}(x)). Just to show how to prove that a function f is not O(g) for some function g, let’s prove the fifth statement on the above list.

Theorem 6.2 4^{x} is not 2^{x}.

Proof. By contradiction. Suppose that 4^{x} is O(2^{x}); then there are constants c and x_{0} so that for all x ≥ x_{0}, we have 4^{x} ≤ c · 2^{x}. But then, dividing both sides by 2^{x}, we get that 2^{x} ≤ c for all x ≥ x_{0}. But when x > log_{2}(c), this is not true — a contradiction.

The final statement on our list follows because

log_{2}(x) = log_{2}(10) · log_{10}(x)

for all x.

6.2 Big-Omega Notation

Big-Omega notation works exactly the same way as Big-Oh notation, except that it provides lower bounds on the growth of a function.

Definition 6.3 Given functions f and g, we say that f(x) is Ω(g(x)) if there exist constants c and x_{0} (independent of x) so that

∀x ≥ x_{0} : 0 ≤ f(x) ≥ c · g(x).

Let’s go through the same five examples from above.

x^{2} is not Ω(x^{3}).

100x is Ω(x).

x + \sqrt{x} is Ω(x).

2^{x} is not Ω(4^{x}).

4^{x} is Ω(2^{x}).

log_{2}(x) is Ω(log_{10}(x)).

6.3 Big-Theta Notation

Finally, you’ll notice that in some of our examples above we had functions f and g so that f was O(g) and also f was Ω(g) — this corresponds to the notion that f(x) and g(x) are always within a constant factor of one another (for large enough x).

This is captured by Big-Theta notation.

Definition 6.4 Given functions f and g, we say that f(x) is Θ(g(x)) if f(x) is O(g(x)) and f(x) is Ω(g(x)). That is, if and only if there exist constants c_{1}, c_{2} and x_{0} (independent of x) so that

∀x ≥ x_{0} : 0 ≤ c_{1}g(x) ≤ f(x) ≤ c_{2} · g(x).

In this case, we can say that g is a tight asymptotic bound for f. Going through our examples again, we have x^{2} is not Θ(x^{3}). 100x is Θ(x). x + \sqrt{x} is Θ(x). 2^{x} is not Θ(4^{x}). 4^{x} is not Θ(2^{x}). log_{2}(x) is Θ(log_{10}(x)). Note that the third example shows that f can be Θ(g) even when f is not a fixed constant multiple of g.

6.4 Little-oh notation

Finally, we want to capture the notion that a function f grows strictly more slowly than a function g. That is, we want to model the case in which

lim_{x→∞} \frac{f(x)}{g(x)} = 0,

but we want our definition to make sense independent of the notion of taking a limit. So we write

Definition 6.5 Given functions f and g, we say that f(x) is o(g(x)) if for all constants c > 0, there exists an x_{0} such that for all x ≥ x_{0} we have 0 ≤ f(x) ≤ c · g(x).

Going through our six examples again, x^{2} is o(x^{3}). 100x is not o(x). x + \sqrt{x} is not o(x). 2^{x} is o(4^{x}). 4^{x} is not o(2^{x}). log_{2}(x) is not o(log_{10}(x)).