Massachusetts Institute of Technology Lecture 11 6.042J/18.062J: Mathematics for Computer Science 9 Oct 97 Professor Tom Leighton

Lecture Notes

Last lecture we saw how to solve Divide and Conquer recurrences. These come up all the time in the analysis of recursive algorithms such as Merge Sort.

In this lecture, we’ll consider another family of recurrences, called linear recurrences, that also frequently arise in computer science and other disciplines. We’ll first see how to solve a specific linear recurrence and then generalize our method to work for all linear recurrences.

1 Graduate Student Job Prospects

In a certain new academic field (say computer science), there are only so many faculty positions available in all the universities of the world. Initially, there were not enough qualified candidates, and many positions were unfilled. But, over time, new graduates are filling the positions, making job prospects for later computer science students ever more bleak. Worse, the increasing number of professors are able to train an increasing number of graduate students, causing positions to fill ever more rapidly. Eventually, the universities will be saturated; new computer science graduates will have no chance at an academic career. Our problem is to determine when the universities will stop hiring new computer science faculty and, in particular, to answer the question, “Are the 6.042 TAs doomed?” Here are the details of the problem.

• There are a total of N faculty positions available worldwide. This number never changes due to budgetary constraints.

• Congress has passed a law forbidding forced retirement, and no one will retire voluntarily. (This is true and a problem!) In our analysis, therefore, once a faculty position is filled, it never opens up again.

• Each year, every professor trains exactly 1 student who will go on to become a professor the following year. The only exception is that first year professors do not train students; they are too busy publishing, teaching, getting grants, and serving on committees.

• In year 0, there are no computer science professors in the world. In year 1, the first professor is hired.

1.1 Finding a Recurrence

Ideally, we could find a formula for the number of professors in the world in a given year. Then we could determine the year in which all N faculty positions are filled. Let f(n) be the number of professors during year n. To develop some intuition about the problem, we can compute values of this function for small n by hand.

f(0) = 0  No CS professors; the dark ages.
f(1) = 1  1 new professor; too busy to train a student.
f(2) = 1  1 old professor; now training a student.
f(3) = 2  1 new prof, 1 old prof; new prof too busy, old prof training.
f(4) = 3  1 new prof, 2 old profs; new prof too busy, both old profs training
f(5) = 5  2 new profs, 3 old profs
f(6) = 8  3 new profs, 5 old profs

In general, the number of professors in year n is equal to the number of professors last year plus the number of new hires. The number of professors last year is f(n − 1). The number of new hires is equal to the number of professors two years ago, f(n − 2), since each of these professors trained a student last year. These observations give the following recurrence equation for the number of professors:

f(n) = f(n − 1) + f(n − 2)  (n ≥ 2)

This is the familiar Fibonacci recurrence. Looking back, the values of f(n) that we computed by hand are indeed the first few Fibonacci numbers. Fibonacci numbers arise in all sorts of applications. For example, we have already seen that the running time of Euler’s GCD algorithm for inputs of size at most f(n) is bounded by n. So how big is f(n) anyway? Of course, we could compute as many Fibonacci numbers as we like using the recurrence, but it would be much nicer to find a closed form.

1.2 Solving the Recurrence

Solving the Fibonacci recurrence is easy because the recurrence is linear. (Well, “easy” in the sense that you can learn the technique in one lecture; discovering it actually took six centuries.)

Definition A recurrence is linear if it is of the form

f(n) = a_{1}f(n − 1) + a_{2}f(n − 2) + … + a_{d}f(n − d)
= ∑_{i=1}^{d} a_{i}f(n − i)

where a_{1}, a_{2}, … a_{d} are constants. The order of the recurrence is d.

For example, the Fibonacci recurrence is order 2 and has a_{1} = a_{2} = 1.

For now, let’s try to solve just the Fibonacci recurrence; we’ll see how to solve general linear recurrences later in the lecture. Our rule of thumb is that Guess and Verify is the first method to apply to a recurrence. It turns out that for a linear recurrence, an exponential solution is a good guess. However, since we know nothing beyond this, our initial Guess and Verify attempt will really only be a “dry run”; that is, we will not make an exact guess and will not verify it with a complete proof. Rather, the goal of this first attempt is only to clarify the form of the solution.

Guess f(n) = cα^{n}

Here c and α are parameters introduced to improve our odds of having a correct guess; in the verification step, we can pick values that make the proof work. To further improve our odds, for now let’s neglect the boundary conditions, f(0) = 0 and f(1) = 1.

Verification. (dry run) Plugging our guess into the recurrence f(n) = f(n − 1) + f(n − 2) gives:

cα^{n} = cα^{n−1} + cα^{n−2}
α^{2} = α + 1
α^{2} − α − 1 = 0
α = \frac{1 ± \sqrt{5}}{2}

In the first step, both sides of the equation are divided by cα^{n−2}. The second step only involves rearranging terms. In the final step, we can compute α by solving the quadratic equation.

This calculation suggests that c can be anything, but that α must be (1 ± \sqrt{5})/2. Evidently, there are two solutions to the recurrence:

f(n) = c (\frac{1 + \sqrt{5}}{2})^{n}  or  f(n) = c (\frac{1 − \sqrt{5}}{2})^{n}

There are actually more solutions than suggested by our “dry run” of Guess and Verify; in fact, any linear combination of the above two solutions is also a solution. The following theorem states that this is true in general for linear recurrences.

Theorem 1.1 If f(n) = α_{1}^{n} and f(n) = α_{2}^{n} are two solutions to a linear recurrence (without boundary conditions), then f(n) = c_{1}a_{1}^{n} + c_{2}a_{2}^{n} is also a solution for all constants c_{1} and c_{2}.

Proof. Every linear recurrence has the following form:

f(n) = ∑_{i=1}^{d} a_{i}f(n − i)

We must prove that f(n) = c_{1}α_{1}^{n} + c_{2}α_{2}^{n} is a solution; that is, we must prove:

c_{1}α_{1}^{n} + c_{2}α_{2}^{n} = ∑_{i=1}^{d} a_{i}(c_{1}α_{1}^{n−i} + c_{2}α_{2}^{n−i})

Substituting the two solutions f(n) = α_{1}^{n} and f(n) = α_{2}^{n} into the recurrence gives the following two equations:

α_{1}^{n} = ∑_{i=1}^{d} a_{i}α_{1}^{n−i}

α_{2}^{n} = ∑_{i=1}^{d} a_{i}α_{2}^{n−i}

Multiplying the first equation by c_{1}, multiplying the second equation by c_{2}, and adding the two gives the desired result.

The theorem implies that for all c_{1} and c_{2}, the following is a solution to the Fibonacci recurrence without boundary conditions:

f(n) = c_{1} (\frac{1 + \sqrt{5}}{2})^{n} + c_{2} (\frac{1 − \sqrt{5}}{2})^{n}

All that remains is to find a solution that satisfies the boundary conditions f(0) = 0 and f(1) = 1. The first boundary condition requires that:

f(0) = c_{1} (\frac{1 + \sqrt{5}}{2})^{0} + c_{2} (\frac{1 − \sqrt{5}}{2})^{0} = 0

⇒ c_{1} + c_{2} = 0

Similarly, the second boundary condition requires that:

f(1) = c_{1} (\frac{1 + \sqrt{5}}{2})^{1} + c_{2} (\frac{1 − \sqrt{5}}{2})^{1} = 1

⇒ c_{1} (\frac{1 + \sqrt{5}}{2}) + c_{2} (\frac{1 − \sqrt{5}}{2}) = 1

We now have two linear equations in two unknowns! The system of equations is not degenerate, so there is a unique solution: c_{1} = 1/\sqrt{5} and c_{2} = −1/\sqrt{5}. We now have a complete solution to the Fibonacci recurrence with boundary conditions:

f(n) = \frac{1}{\sqrt{5}} (\frac{1 + \sqrt{5}}{2})^{n} − \frac{1}{\sqrt{5}} (\frac{1 − \sqrt{5}}{2})^{n}

This looks completely wrong! All Fibonacci numbers are integers, but this expression is full of square roots of five! Amazingly, however, the square roots always cancel out. This expression really does give the Fibonacci numbers if we plug in n = 0, 1, 2, . . .. If one were required to guess the solution to the Fibonacci recurrence, then it is easy to imagine taking at least six centuries to come up with this!

1.3 Job Prospects

Let’s return to the original question: how long until all N faculty positions are taken?

To answer this question, we must find the smallest n such that f(n) ≥ N. This will imply that in year n, there are as many professors as university positions. Therefore, graduates after year n will have to find other employment, e.g. shuffling golden disks in an obscure monastic community for the next 10^{19} years.

Because f(n) has such a complicated form, it is hard to compute the right value of n exactly. However, we can make a good approximation. Notice that f(n) is the sum of two terms. The key observation is that the second term tends to zero as n grows large, since |\frac{1−\sqrt{5}}{2}| = 0.618 … < 1. This means that we can rewrite f(n) with asymptotic notation as follows:

f(n) = \frac{1}{\sqrt{5}} (\frac{1 + \sqrt{5}}{2})^{n} + o(1)

Using this approximation for f(n), we can determine the approximate year in which all faculty positions will be taken as follows:

f(n) ≥ N
⇔ \frac{1}{\sqrt{5}} (\frac{1 + \sqrt{5}}{2})^{n} + o(1) ≥ N

⇔ \frac{log(\sqrt{5} N + o(1))}{log(\frac{1+\sqrt{5}}{2})}

This implies that all the faculty jobs will be filled in Θ(log N) years. This makes sense, since the number of professors is increasing exponentially. For example, if there are N = 10000 jobs, then all will be taken in n = 20.8 years. Now, as to the fate of the TAs, there are fewer than ten thousand faculty jobs in computer science and hiring started about 20 years ago...

The solution to the Fibonacci recurrence has an interesting corollary: for large n, \frac{1}{\sqrt{5}} (\frac{1+\sqrt{5}}{2})^{n} is very close to an integer— namely, the n-th Fibonacci number. This expression is the first term in the closed form for the n-th Fibonacci number. Since the other term of the closed form goes to zero exponentially fast and Fibonacci numbers are always integers, the first term must be very close to an integer! This is one of many strange facts about the “Golden Ratio”, \frac{1+\sqrt{5}}{2} = 1.61803 . . ..

2 General Linear Recurrences

The method we used to solve the Fibonacci recurrence can be used to solve any linear recurrence! Recall that every linear recurrence has the following form:

f(n) = ∑_{i=1}^{d} a_{i}f(n − i)

Substituting the guess f(n) = α^{n} into the recurrence gives:

α^{n} = a_{1}α^{n−1} + a_{2}α^{n−2} + … + a_{d}α^{n−d}
α^{d} = a_{1}α^{d−1} + a_{2}α^{d−2} + … + a_{d−1}α + a_{d}
α^{d} − a_{1}α^{d−1} − a_{2}α^{d−2} − … − a_{d−1}α − a_{d} = 0

The equation is divided by α^{n−d} in the first step, and terms are rearranged in the second step. The last equation is called the characteristic equation of the recurrence. The characteristic equation of a recurrence can be read of very quickly; the coefficients of the equation are the same as the coefficients of the recurrence. In general, the characteristic equation will have d roots, where d is the order of the recurrence. The solution to the recurrence depends on whether all roots of the characteristic equation are distinct.

2.1 Simple Case: Distinct Roots

If the characteristic equation has d distinct roots r_{1}, r_{2}, …, r_{d}, then the solution to the recurrence has the form:

f(n) = c_{1}r_{1}^{n} + c_{2}r_{2}^{n} + … + c_{d}r_{d}^{n}

Note that the roots of the characteristic equation need not even be real! In the end, imaginary parts will cancel just as the \sqrt{5}’s canceled in the solution to the Fibonacci recurrence.

The constants c_{1}, c_{2}, …, c_{d} depend on the boundary conditions. Each boundary condition generates a linear equation in these constants. For example, the boundary conditions f(0) = b_{0}, f(1) = b_{1}, …, f(d − 1) = b_{d−1} generate the linear equations:

b_{0} = c_{1} + c_{2} + … + c_{d}
b_{1} = c_{1}r_{1} + c_{2}r_{2} + … + c_{d}r_{d}
. . . . . . . . .
b_{d−1} = c_{1}r_{1}^{d−1} + c_{2}r_{2}^{d−1} + … + c_{d}r_{d}^{d−1}

If there are d boundary conditions, then we obtain d linear equations in d unknowns. A neat linear algebra fact is that a system of equations of this particular type is never degenerate. This means that for every set of constants b_{0}, b_{1}, … b_{d−1} in the boundary conditions, we can find a unique set of constants c_{0}, c_{1}, … c_{d−1} that give a solution to the recurrence and satisfy the boundary conditions.

2.2 Tricky Case: Repeated Roots

If the characteristic equation of a recurrence has a repeated root, then the solution is slightly more complicated. The main tool is the following theorem, which we will not prove.

Theorem 2.1 If r is a root of the characteristic equation of a recurrence with multiplicity m, then all of r^{n}, nr^{n}, n^{2}r^{n}, … n^{m−1}r^{n} are solutions to the recurrence.

For example, if the characteristic equation of a recurrence has root r_{1} with multiplicity 2, root r_{2} with multiplicity 1, and root r_{3} with multiplicity 3, then the solution to the recurrence has the form:

f(n) = c_{1}r_{1}^{n} + c_{2}nr_{1}^{n} + c_{3}r_{2}^{n} + c_{4}r_{3}^{n} + c_{5}nr_{3}^{n} + c_{6}n^{2}r_{3}^{n}

As before, the constants depend on the boundary conditions. Each boundary condition generates a linear equation, and the constants are computed by solving the resulting system of linear equations. In this respect, solving linear recurrences is very similar to solving linear differential equations.

2.3 An Example of the Tricky Case

To understand recurrences of this tricky type, let’s work an example.

Suppose that there is a type of plant that lives forever, but only reproduces during the first year of life. How fast will the plant population grow? (Notice that this is just the reverse of the graduate student job problem where faculty “reproduce” in every year except the first.)

Let f(n) be the number of plants in year n. As boundary conditions, define f(0) = 0 and f(1) = 1. The plant population in year n is equal to the population from the year before plus the number of new plants. The population from the year before is f(n−1). The number of new plants this year is equal to the number of new plants last year or f(n − 1) − f(n − 2). We can now form a recurrence equation for the plant population:

f(n) = f(n − 1) + (f(n − 1) − f(n − 2))
= 2f(n − 1) − f(n − 2)
f(n) − 2f(n − 1) + f(n − 2) = 0

The characteristic equation is α^{2} − 2α + 1 = 0, which has the single root α = 1 with multiplicity 2. Therefore, the solution to the recurrence has the form:

f(n) = c_{1}(1)^{n} + c_{2}n(1)^{n}
= c_{1} + c_{2}n

The boundary conditions, f(0) = 0 and f(1) = 1, generate the following linear equations:

0 = c_{1} + c_{2}(0)
1 = c_{1} + c_{2}(1)

The solution to the linear system is c_{1} = 0 and c_{2} = 1. Therefore, the solution to the recurrence is:

f(n) = c_{1} + c_{2}n
= 0 + (1)n
= n

The answer turns out to be very simple! In year n, there are exactly n plants. Of course, we probably could have solved this problem more easily with Guess and Verify, but for many other examples, guessing is not so easy.

3 Inhomogeneous Recurrences

We can now solve all recurrences of the form:

f(n) − a_{1}f(n − 1) − … − a_{d}f(n − d) = 0

Strictly speaking, this is the family of linear homogeneous recurrences. These recurrences are called homogeneous because the right side of the equation is 0. If the right side were 1 or n^{2} or some arbitrary function g(n), then the recurrence would be called inhomogeneous.

Definition An inhomogeneous linear recurrence has the form:

f(n) − a_{1}f(n − 1) − a_{2}f(n − 2) − … − a_{d}f(n − d) = g(n)

where a_{1}, a_{2}, … a_{d} are constants.

There are three steps to solving an inhomogeneous linear recurrence.

1. Replace g(n) by 0 and solve the resulting homogeneous recurrence. (Ignore boundary conditions for now; that is, do not solve for constants c_{1}, c_{2}, …, c_{d}.) The solution to the homogeneous recurrence is called the homogeneous solution.

2. Restore g(n) and find a single solution to the recurrence, again ignoring boundary conditions. This is called the particular solution. (This is the hard part!)

3. Add the homogeneous and particular solutions together to obtain the general solution. Now use the boundary conditions to determine the constant factors by the usual method of generating and solving a system of linear equations.

Notice that the method for solving inhomogeneous linear recurrences is closely analogous to the method for solving inhomogeneous linear differential equations!

3.1 An Example of an Inhomogeneous Recurrence

Let f(n) = 4f(n − 1) + 3^{n} and, as a boundary condition, let f(1) = 1. This is an inhomogeneous linear recurrence.

Step 1: Solve the Homogeneous Recurrence

The homogeneous recurrence is f(n) − 4f(n − 1) = 0. The characteristic equation is α − 4 = 0. The only root is α = 4. Therefore, the homogeneous solution is c_{1}4^{n}.

Step 2: Find a Particular Solution

We must find a single solution to the recurrence f(n) = 4f(n − 1) + 3^{n}. This is usually the hard part of solving an inhomogeneous linear recurrence. Let’s guess that there is a solution of the form c3^{n}. Substituting this into the recurrence gives:

f(n) = 4f(n − 1) + 3^{n}
c3^{n} = 4c3^{n−1} + 3^{n}
3c = 4c + 3
c = −3

The equation was divided by 3^{n−1} in the first step. The second and third steps were only simplification. We now have a particular solution, f(n) = −3 · 3^{n} = −3^{n+1}.

Step 3: Add Solutions and Find Constants

We now add the homogeneous solution and the particular solution to obtain the general solution:

f(n) = c_{1}4^{n} − 3^{n+1}

The value of the constant is obtained from the boundary condition, f(1) = 1:

1 = c_{1}4^{1} − 3^{1+1}
= 4c_{1} − 9
c_{1} = 5/2

The solution to the recurrence is f(n) = \frac{5}{2}4^{n} − 3^{n+1}!

Since we could easily have made a mistake, let’s check to make sure that our solution at least works for n = 2. From the recurrence, f(2) = 4f(1) + 3^{2} = 13. From our closed form, f(2) = \frac{5}{2}4^{2} − 3^{3} = 40 − 27 = 13. It looks right!

3.2 A Final Example

Let f(n) = 4f(n − 1) − 4f(n − 2) + n and, as a boundary condition, let f(0) = 4 and f(1) = 7.

Step 1: Solve the Homogeneous Recurrence

The homogeneous recurrence is f(n) − 4f(n − 1) + 4f(n − 2) = 0. The characteristic equation is α^{2} − 4α + 4. The only root is α = 2 with multiplicity 2. Therefore, the homogeneous solution is c_{1}2^{n} + c_{2}n2^{n}.

Step 2: Find a Particular Solution

Now we need a single solution to the recurrence f(n) = 4f(n − 1) − 4f(n − 2) + n. Let’s guess a solution of the form an + b and substitute into the recurrence:

f(n) = 4f(n − 1) − 4f(n − 2) + n
an + b = 4(a(n − 1) + b) − 4(a(n − 2) + b) + n
= 4an − 4a + 4b − 4an + 8a − 4b + n
= 4a + n
⇒ a = 1, b = 4a = 4

This gives the particular solution f(n) = n + 4.

Step 3: Add Solutions and Find Constants

Adding the homogeneous solution and the particular solution gives the general solution:

f(n) = c_{1}2^{n} + c_{2}n2^{n} + n + 4

The values of the constants are obtained from the boundary conditions, f(0) = 4 and f(1) = 7.

4 = c_{1}2^{0} + 0c_{2}2^{0} + 0 + 4
= c_{1} + 4
c_{1} = 0
7 = c_{1}2^{1} + 1c_{2}2^{1} + 1 + 4
= 2c_{1} + 2c_{2} + 5
c_{2} = 1

The solution to the recurrence is f(n) = n2^{n} + n + 4.

3.3 Finding a Particular Solution

The hardest part in solving inhomogeneous recurrences is finding a particular solution. There are some good rules of thumb, however. Suppose the recurrence has the form:

f(n) − a_{1}f(n − 1) − a_{2}f(n − 2) − … − a_{d}f(n − d) = g(n)

If g(n) is a polynomial of degree r, then guess that f(n) is a polynomial of degree r. For example, if g(n) = n^{2}, then guess f(n) = an^{2} + bn + c and determine a, b, and c by substituting and verifying. Similarly, if g(n) is an exponential, then guess that f(n) is an exponential. For example, if g(n) = 2^{n} + 3^{n}, then guess f(n) = a2^{n} + b3^{n}. In general, guess a particular solution with the same form as g(n).