\documentstyle[11pt,twoside,amsfonts,amstex,psfig]{article}
\input{macros-course-lectures}

\newtheorem{falsethm}[theorem]{False Theorem}
\newtheorem{rul}[theorem]{Rule}
\newtheorem{guess}{Guess}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{method}{Method}
\newtheorem{axiom}{Axiom}

\begin{document}

\lecture{notes17}{Lecture Notes}{30 Oct 97}

This is the last lecture on counting the number of items in a finite
set.  The topic today is {\em generating functions}.  Generating
functions are one of the most surprising, useful, and clever
inventions in discrete math.  There is a huge chunk of mathematics
concerning generating functions, so we will only get a taste of the
subject today.  Generating functions will come up again later, though,
when we cover probability.

\section{Generating Functions}

Here is the idea behind generating functions.  We can represent a
sequence of numbers as a power series.  A power series defines a
function.  This lets us manipulate sequences of numbers with all the
tools that we have for working with functions.

\begin{defin*}
The {\em ordinary generating function} (OGF) of the infinite sequence
$<g_0, g_1, g_2, g_3 \ldots>$ is the formal power series:

\[
g_0 + g_1 x + g_2 x^2 + g_3 x^3 + \ldots
\]
\end{defin*}

This definition requires a few comments.  All of the generating
functions in this lecture will be ordinary, so this is the only
definition that we will need.  A generating function is a ``formal''
power series in that we typically regard $x$ as a placeholder, not as
a number.  In particular, the convergence of the power series is
almost never an issue, since we will almost never evaluate the power
series for a particular $x$.  Throughout the lecture, we will indicate
the correspondence between a sequence and its generating function with
a double-sided arrow as follows:

\[
<g_0, g_1, g_2, g_3, \ldots> \leftrightarrow
	g_0 + g_1 x + g_2 x^2 + g_3 x^3 + \ldots
\]

Here are some basic examples of sequences and the corresponding
generating functions:

\begin{eqnarray*}
<0, 0, 0, 0, \ldots> & \leftrightarrow & 0 + 0 x + 0 x^2 + 0 x^3 \ldots
= 0 \\
<1, 0, 0, 0, \ldots> & \leftrightarrow & 1 + 0 x + 0 x^2 + 0 x^3 \ldots
= 1 \\
<1, 1, 0, 0, \ldots> & \leftrightarrow & 1 + 1 x + 0 x^2 + 0 x^3 \ldots
= 1 + x \\
<1, 1, 1, 1, \ldots> & \leftrightarrow & 1 + 1 x + 1 x^2 + 1 x^3 \ldots
= \frac{1}{1-x} \\
\end{eqnarray*}

These examples follow directly from the definition; the $i$-th term in
the sequence is the coefficient of $x^i$ in the generating function.
The last example has an interesting feature.  Recall that $1 + x + x^2
+ x^3 + \ldots$ is an infinite geometric sequence with sum
$\frac{1}{1-x}$.  As a result, we can represent the infinite sequence
of 1's with a simple, closed-form expression!  This generating
function arises frequently and is good to remember.  Below are some
more tricky examples of sequences and their generating functions.

\begin{eqnarray*}
<1, a, a^2, a^3, \ldots> & \leftrightarrow &
    = 1 + a x + a^2 x^2 + a^3 x^3 \ldots \\
& & = 1 + a x + (a x)^2 + (a x)^3 \ldots \\
& & = \frac{1}{1 - ax} \\
<1, -1, 1, -1, \ldots> & \leftrightarrow &
    = \frac{1}{1 + x} \\
<1, 0, 1, 0, \ldots> & \leftrightarrow &
    = 1 + x^2 + x^4 + \ldots \\
& & = \frac{1}{1 - x^2}
\end{eqnarray*}

In the first and third examples, the closed forms for the power series
follow from the formula for the sum of an infinite geometric sequence.
The second example is a special case of the first with $a = -1$.

\section{Operations on Sequences}

The magic of generating functions is that they allow us to manipulate
sequences of numbers with all the tools usually applied to functions.
The sections below show how operations on sequences correspond to
operations on their generating functions.

\subsection{Adding Sequences}

Adding sequences term-by-term corresponds to adding the generating
functions.  For example:

\[
\begin{array}{clcl}
  & <1,  1,  1,  1, \ldots> & \leftrightarrow & \frac{1}{1-x} \\
+ & <1, -1,  1, -1, \ldots> & \leftrightarrow & \frac{1}{1+x} \\
\hline
  & <2,  0,  2,  0, \ldots> & \leftrightarrow & \frac{1}{1-x} + \frac{1}{1+x} =
	\frac{2}{1-x^2}\\
\\
\Rightarrow & <1,  0,  1,  0, \ldots> & \leftrightarrow & \frac{1}{1-x^2}\\
\end{array}
\]

The generating functions of the first two sequences were computed in
the preceding section.  On the third line, we sum the sequences and
sum their generating functions.  On the fourth line, we have divided
every term by two, which corresponds to dividing the generating
function by two.  We have ``derived'' the generating function of the
last sequence given in the preceding section!  The derivation is
justified by the following rule:

\begin{rul}
\label{rul:addition}
If $<f_0, f_1, f_2, \ldots> \leftrightarrow F(x)$ and
$<g_0, g_1, g_2, \ldots> \leftrightarrow G(x)$, then

\[
<\alpha f_0 + \beta g_0, \alpha f_1 + \beta g_1, \alpha f_2 + \beta
g_2, \ldots> \leftrightarrow \alpha F(x) + \beta G(x)
\]
\end{rul}

\begin{proof}
\begin{eqnarray*}
<\alpha f_0 + \beta g_0, \alpha f_1 + \beta g_1, \alpha f_2 + \beta
g_2, \ldots>
& \leftrightarrow & \sum_{n=0}^\infty (\alpha f_n + \beta g_n) x^n \\
& = & \alpha \sum_{n=0}^\infty f_n x^n +
      \beta \sum_{n=0}^\infty g_n x^n \\
& = & \alpha F(x) + \beta G(x)
\end{eqnarray*}
\end{proof}

\subsection{Shifting a Sequence}

Suppose that we add $k$ leading zeroes to a sequence; how does the
generating function change?  For example, in the first section we
showed that:

\[
<1, 1, 1, \ldots> \leftrightarrow \frac{1}{1-x}
\]

If we add $k$ leading zeroes to the sequence, we can compute the new
generating function as follows:

\begin{eqnarray*}
<\underbrace{0, 0, \ldots, 0}_{\mbox{$k$ zeroes}}, 1, 1, 1, \ldots>
	& \leftrightarrow & x^k + x^{k+1} + x^{k+2} + \ldots \\
	& = & x^k (1  + x + x^2 + \ldots) \\
	& = & \frac{x^k}{1-x} \\
\end{eqnarray*}

In this example, adding $k$ leading zeroes to the sequence corresponds
to multiplying the generating function by $x^k$.  In fact, this holds
in general.

\begin{rul}
\label{rul:shift}
If $<f_0, f_1, f_2, \ldots> \leftrightarrow F(x)$, then

\[
<\underbrace{0, 0, \ldots, 0}_{\mbox{$k$ zeroes}}, f_0, f_1, f_2, \ldots>
	\leftrightarrow x^k F(x)
\]
\end{rul}

\begin{proof}
\begin{eqnarray*}
<\underbrace{0, 0, \ldots, 0}_{\mbox{$k$ zeroes}}, f_0, f_1, f_2, \ldots>
& \leftrightarrow & f_0 x^k + f_1 x^{k+1} + f_2 x^{k+2} + \ldots \\
& = & x^k( f_0 + f_1 x + f_2 x^2 + \ldots) \\
& = & x^k F(x)
\end{eqnarray*}
\end{proof}

\subsection{Differentiation}

What happens if we take the derivative of a generating function?  As
an example, let's differentiate the now-familiar generating function
for an infinite sequence of 1's.

\begin{eqnarray*}
\frac{d}{dx} \ (1 + x + x^2 + x^3 + x^4 + \ldots)
	& = & \frac{d}{dx} \left( \frac{1}{1-x} \right) \\
1 + 2x + 3x^2 + 4x^3 + \ldots & = & \frac{1}{(1-x)^2} \\
<1, 2, 3, 4, \ldots> & \leftrightarrow & \frac{1}{(1-x)^2}
\end{eqnarray*}

We found a generating function for the sequence $<1, 2, 3, 4,
\ldots>$!  In general, differentiating a generating function has two
effects on the corresponding sequence:

\begin{enumerate}
\item Each term is multiplied by its index.
\item The sequence is shifted left one place.
\end{enumerate}

These effects are apparent in the following rule:

\begin{rul}
\label{rul:derivative}
If $<f_0, f_1, f_2, f_3, \ldots> \leftrightarrow F(x)$, then

\[
<f_1, 2f_2, 3f_3, \ldots> \leftrightarrow F'(x)
\]
\end{rul}

\begin{proof}
\begin{eqnarray*}
<f_1, 2f_2, 3f_3, \ldots> & = & f_1 + 2 f_2 x + 3 f_3 x^2 + \ldots \\
	& = & \frac{d}{dx} \ (f_0 + f_1 x + f_2 x^2 + f_3 x^3 + \ldots) \\
	& = & \frac{d}{dx} \ F(x)
\end{eqnarray*}
\end{proof}

The derivative rule is very useful!  In fact, there is frequent,
independent need for each of differentiation's two effects,
multiplying terms by their index and left-shifting one place.
Typically, we want just one effect and must somehow cancel out the
other; we'll see how to do this below and also in
Section~\ref{sec:rep}.

Suppose that we want the generating function for the sequence of
squares, $<0, 1, 4, 9, 16, \ldots>$.  We could start with the sequence
$<1, 1, 1, \ldots>$ and multiply each term by its index two times.
This would give the desired result:

\begin{eqnarray*}
<0 \cdot 0, 1 \cdot 1, 2 \cdot 2, 3 \cdot 3, \ldots>
& = &
<0, 1, 4, 9, \ldots>
\end{eqnarray*}

The problem is that differentiation not only multiplies each term by
its index, but also shifts a sequence left one place.  However,
Rule~\ref{rul:shift} tells how to cancel out this unwanted left-shift;
namely, multiply the generating function by $x$ to right-shift the
sequence back one place.

Our procedure, therefore, is to begin with the generating function for
$<1, 1, 1, \ldots>$, differentiate, multiply by $x$, and then again
differentiate and multiply by $x$.

\begin{eqnarray*}
<1, 1, 1, 1, \ldots>  & \leftrightarrow & \frac{1}{1-x} \\
<1, 2, 3, 4, \ldots>  & \leftrightarrow & \frac{d}{dx} \ \frac{1}{1-x}
                                          = \frac{1}{(1-x)^2} \\
<0, 1, 2, 3, \ldots>  & \leftrightarrow & x \cdot \frac{1}{(1-x)^2}
                                          = \frac{x}{(1-x)^2} \\
<1, 4, 9, 16, \ldots> & \leftrightarrow & \frac{d}{dx} \ \frac{x}{(1-x)^2}
		       			  = \frac{1+x}{(1-x)^3} \\
<0, 1, 4, 9, \ldots>  & \leftrightarrow & x \cdot \frac{1+x}{(1-x)^3}
		       			  = \frac{x(1+x)}{(1-x)^3}
\end{eqnarray*}

The generating function for squares is $\frac{x(1+x)}{(1-x)^3}$.

\section{The Fibonacci Sequence}

A sequence defined by a recurrence often has a simple generating
function.  An example is the Fibonacci sequence:

\[
<0, 1, 1, 2, 3, 5, 8, 13, 21, \ldots> \leftrightarrow \frac{x}{1-x-x^2}
\]

It is amazing that such a tricky sequence could have so simple a
generating function!  In the next two sections, we will derive this
generating function and then use it to find a closed form for the
$n$-th Fibonacci number.  The techniques used here are applicable to a
large class of recurrence equations.

\subsection{Finding a Generating Function}

Recall the definition of the Fibonacci sequence:

\[
f_0 = 0, \quad f_1 = 1, \quad f_n = f_{n-1} + f_{n-2} \mbox{ for } n \geq 2
\]

We can write the generating function for the Fibonacci numbers as follows.

\begin{eqnarray*}
F(x) & = & \sum_{n=0}^\infty f_n x^n \\
     & = & \underbrace{0 + x}_{
\begin{array}{c}
\mbox{break out} \\
\mbox{base cases}
\end{array}
} + \sum_{n=2}^\infty \underbrace{(f_{n-1} + f_{n-2}) x^n}_{
\begin{array}{c}
\mbox{expand other terms with} \\
\mbox{the recurrence relation}
\end{array}
}
\end{eqnarray*}

Our objective is now to write $F(x)$ in terms of itself.  We can then
solve for $F(x)$ to find a closed form for the generating function.
The following sequence of equations is explained in the text that
follows.

\begin{eqnarray*}
F(x) & = & x + \sum_{n=2}^\infty f_{n-1} x^n + \sum_{n=2}^\infty f_{n-2} x^n \\
     & = & x + x \sum_{n=2}^\infty f_{n-1} x^{n-1} +
               x^2 \sum_{n=2}^\infty f_{n-2} x^{n-2} \\
     & = & x + x \sum_{n=1}^\infty f_n x^n +
               x^2 \sum_{n=0}^\infty f_n x^n \\
     & = & x + x \sum_{n=0}^\infty f_n x^n +
               x^2 \sum_{n=0}^\infty f_n x^n \\
     & = & x + x F(x) + x^2 F(x)
\end{eqnarray*}

In the first equation, we split the previous sum into two parts.  The
idea behind all subsequent steps is to make each part look like
$F(x)$.  In the second equation, we make subscripts of coefficients
match exponents by pulling $x$'s out of the summations.  In the third
equation, we adjust the indices of summation so that subscripts and
exponents are $n$ instead of $n-1$ and $n-2$.  In the fourth equation,
the range of the first summation is extended to include $n=0$; this is
valid because the summand is 0 in this case anyway.  Finally, in two
places we substitute $F(x)$ for its definition, $\sum_{n=0}^\infty f_n
x^n$.

We now have an equation in $F(x)$.  Solving for $F(x)$ gives the
generating function for Fibonacci numbers.

\begin{eqnarray*}
            F(x) & = & x + x F(x) + x^2 F(x) \\
\Rightarrow F(x) & = & \frac{x}{1 - x - x^2}
\end{eqnarray*}

Amazingly, the power series of $F(x)$ really does encode all the
Fibonacci numbers:

\[
\frac{x}{1 - x - x^2} = 0 + x + x^2 + 2 x^3 + 3 x^4 + 5 x^5 + 8x^6 + \ldots
\]

\subsection{Finding a Closed Form}

Why should one care about the generating function for a sequence?
There are several answers, but here is one.  If we can find a
generating function for a sequence, then we can often find a closed
form for the $n$-th term in the sequence.

For example, we found a generating function for the Fibonacci
sequence.  From this we can find a closed form for the $n$-th
Fibonacci number.  In particular, the $n$-th Fibonacci number is the
coefficient of $x^n$ in the power series of $\frac{x}{1 - x - x^2}$.
Our problem, therefore, is to extract the $n$-th coefficient from the
power series of a generating function.  We will see two ways to do
this, one here and one in Section~\ref{sec:find_coeff}

The first way to extract a coefficient from a generating function
works if the generating function is a ratio of polynomials.  This is
often true in recurrence problems.  The idea is to rewrite the
generating function using partial fractions; in this form, one can
immediately read out the coefficients of the power series.

There are two steps in rewriting the generating function $\frac{x}{1 -
x - x^2}$ with partial fractions.  First, we factor the denominator:

\begin{eqnarray*}
1 - x - x^2 & = & (1 - \alpha_1 x) (1 - \alpha_2 x) \\
\mbox{where } \alpha_1 = \frac{1 + \sqrt{5}}{2} & , & \alpha_2 = \frac{1 - \sqrt{5}}{2}
\end{eqnarray*}

Second, we find $A_1$ and $A_2$ which satisfy the following equation:

\[
\frac{x}{1 - x - x^2} =
	\frac{A_1}{1 - \alpha_1 x} + \frac{A_2}{1 - \alpha_2 x}
\]

To do this, we plug in various values of $x$ to generate linear
equations in $A_1$ and $A_2$.  We can then find $A_1$ and $A_2$ by
solving a linear system.  In this case, we get:

\begin{eqnarray*}
A_1 = \frac{1}{\alpha_1 - \alpha_2} = \frac{1}{\sqrt{5}} \\
A_2 = \frac{-1}{\alpha_1 - \alpha_2} = -\frac{1}{\sqrt{5}}
\end{eqnarray*}

Substituting into the equation above gives the partial fractions
expansion of $F(x)$:

\[
\frac{x}{1 - x - x^2} =
	\frac{1}{\sqrt{5}}
	\left(
	\frac{1}{1 - \alpha_1 x} - \frac{1}{1 - \alpha_2 x}
	\right)
\]

After the generating function is written with partial fractions,
reading out coefficients is easy, because each fraction has a simple
power series.

\begin{eqnarray*}
\frac{1}{1 - \alpha_1 x} & = & 1 + \alpha_1 x + \alpha_1^2 x^2 + \ldots \\
\frac{1}{1 - \alpha_2 x} & = & 1 + \alpha_2 x + \alpha_2^2 x^2 + \ldots
\end{eqnarray*}

We can now find the coefficient of $x^n$ in the generating function:

\begin{eqnarray*}
F(x) & = &
	\frac{1}{\sqrt{5}}
	\left(
	\frac{1}{1 - \alpha_1 x} -
	\frac{1}{1 - \alpha_2 x}
	\right) \\
& = &
	\frac{1}{\sqrt{5}}
	\left(
	(1 + \alpha_1 x + \alpha_1^2 x^2 + \ldots) -
	(1 + \alpha_2 x + \alpha_2^2 x^2 + \ldots)
	\right) \\
\Rightarrow f_n & = & \frac{1}{\sqrt{5}} (\alpha_1^n - \alpha_2^n) \\
                & = & \frac{1}{\sqrt{5}}
\left(
	\left(\frac{1 + \sqrt{5}}{2}\right)^n -
	\left(\frac{1 - \sqrt{5}}{2}\right)^n
\right)
\end{eqnarray*}

This is the same scary formula for the $n$-th Fibonacci number that we
found using the method for solving linear recurrences!  The closed
form for Fibonacci numbers was actually first found by DeMoivre in
1718 using this technique.  The same approach can be used to solve any
linear recurrence and even some non-linear recurrences.

\subsection{Review of the Method}

Since we will see only this one example of using generating functions
to solve a recurrence, let's review the process.  There are two main
steps and some tricks in each.  First, we find a generating function
for the sequence defined by the recurrence as follows.

\begin{enumerate}
\item Write the generating function as $F(x) = \sum_{n=0}^\infty f_n x^n$.
\item Pull out base cases, apply the recurrence equation to remaining terms.
\item Massage this formula to get an equation in $F(x)$.
\item Solve the equation to obtain the generating function $F(x)$.
\end{enumerate}

Second, we have to extract coefficients from the generating function:

\begin{enumerate}
\item Expand the generating function into partial fractions.
\item Expand each fraction into a simple power series.
\item Read out the coefficients of the generating function.
\end{enumerate}

\section{Counting with Generating Functions}

Generating functions can be used to solve counting problems of the
type we encountered in the last two weeks.  Problems involving
choosing items from a set often lead to nice generating functions.
When generating functions are used in this way, the coefficient of
$x^n$ is the number of ways to choose $n$ items from a set.

\subsection{Choosing Items from a Set Without Repetition}

The generating function for binomial coefficients follows directly
from the Binomial Theorem:

\begin{eqnarray*}
< \binom{k}{0}, \binom{k}{1}, \binom{k}{2}, \ldots, \binom{k}{k},
	0, 0, 0, \ldots > & \leftrightarrow & \binom{k}{0} +
	\binom{k}{1} x + \binom{k}{2} x^2 + \ldots + \binom{k}{k} x^k \\
& = & (1 + x)^k
\end{eqnarray*}

This is the generating function for choosing items from a $k$-element
set without repetition.  More precisely, the coefficient of $x^n$ is
the number of ways to chose $n$ items from from a $k$-element set with
without repetition.  For example, the coefficient of $x^2$ is the
number of ways to choose 2 items from a $k$-element set, which is
$\binom{k}{2}$.  Similarly, the coefficient of $x^{k+1}$ is the number
of ways to choose $k+1$ items from a $k$-element set, which is zero.

Often we can find the generating function for such a counting problem
with very little work.  In fact, we could find the generating function
above without worrying about the Binomial Theorem or even binomial
coefficients!

Here is how.  First, consider a single-element set $\{ a_1 \}$.  The
generating function for the number of ways to choose $n$ elements from
this set is simply $1 + x$; we have one way to choose zero elements,
one way to choose one element, and zero ways to choose more than one
element.  Similarly, the number of ways to choose $n$ elements from
the set $\{ a_2 \}$ is also given by the generating function $1 + x$;
the fact that the element differs is irrelevant.

Here is the the main trick.  The generating function for choosing
elements from the union of disjoint sets is the product of the
generating functions for choosing from each set.  For example, the
generating for the number of ways to choose $n$ elements from the set
$\{ a_1, a_2 \}$ is:

\[
\underbrace{(1 + x)}_{
\begin{array}{cc}
\mbox{OGF for} \\
\mbox{$\{ a_1 \}$}
\end{array}}
\underbrace{(1 + x)}_{
\begin{array}{cc}
\mbox{OGF for} \\
\mbox{$\{ a_2 \}$}
\end{array}}
=
\underbrace{(1 + x)^2}_{
\begin{array}{cc}
\mbox{OGF for} \\
\mbox{$\{ a_1, a_2 \}$}
\end{array}}
= 1 + 2x + x^2
\]

Sure enough, for the set $\{ a_1, a_2 \}$ we have one way to choose
zero elements, two ways to choose one element, one way to choose two
elements, and zero ways to choose more than two elements.

\begin{rul}
Let $A(x)$ and $B(x)$ be ordinary generating functions for the number
of ways to choose items from disjoint sets ${\cal A}$ and ${\cal B}$,
respectively.  The generating function for the number of ways to
choose objects from the union ${\cal A} \cup {\cal B}$ is the product
$A(x) \cdot B(x)$.
\end{rul}

For example, by repeated application of this rule, we find that the
generating function for choosing $n$ items from a $k$-element set $\{
a_1, a_2, \ldots, a_k \}$ is:

\[
\underbrace{(1 + x)}_{
\begin{array}{cc}
\mbox{OGF for} \\
\mbox{$\{ a_1 \}$}
\end{array}}
\underbrace{(1 + x)}_{
\begin{array}{cc}
\mbox{OGF for} \\
\mbox{$\{ a_2 \}$}
\end{array}}
\ldots
\underbrace{(1 + x)}_{
\begin{array}{cc}
\mbox{OGF for} \\
\mbox{$\{ a_k \}$}
\end{array}}
=
\underbrace{(1 + x)^k}_{
\begin{array}{cc}
\mbox{OGF for} \\
\mbox{$\{ a_1, a_2, \ldots, a_k\}$}
\end{array}}
\]

This is the same generating function that we obtained by using the
Binomial Theorem.

\begin{proof}
Let $A(x) = \sum_{n=0}^\infty a_n x^n$, let $B(x) = \sum_{n=0}^\infty
b_n x^n$, and let $C(x) = A(x) \cdot B(x) = \sum_{n=0}^\infty c_n
x^n$.

We must show that $c_n$ is equal to the number of ways to choose $n$
items from ${\cal A} \cup {\cal B}$.  We can compute $c_n$ by
evaluating the product $A(x) \cdot B(x)$.  We can compute the number
of ways to choose $n$ items from ${\cal A} \cup {\cal B}$ by a
combinatorial argument.  The proof is completed by showing that these
two results are the same.

First, we compute $c_n$ by evaluating the product $A(x) \cdot B(x)$.
We can tabulate all of the terms in this product in a table like the
one shown below.

\[
\begin{array}{c|ccccc}
	& b_0 x^0 & b_1 x^1 & b_2 x^2 & b_3 x^3 & \ldots \\
\hline
a_0 x^0 & a_0 b_0 x^0 & a_0 b_1 x^1 & a_0 b_2 x^2 & a_0 b_3 x^3 & \ldots \\
a_1 x^1 & a_1 b_0 x^1 & a_1 b_1 x^2 & a_1 b_2 x^3 & \ldots \\
a_2 x^2 & a_2 b_0 x^2 & a_2 b_1 x^3 & \ldots \\
a_3 x^3 & a_3 b_0 x^3 & \ldots \\
\ldots & \ldots \\
\end{array}
\]

Note that all terms containing a particular power of $x$ lie on a
/-sloped diagonal.  Collecting terms, we find that the coefficient of
$x^n$ in the product is:

\[
c_n = a_0 b_n + a_1 b_{n-1} + a_2 b_{n-2} + \ldots a_n b_0
\]

Next, we compute the number of ways to choose $n$ items from ${\cal A}
\cup {\cal B}$.  We can break up the answer based on the number of
items chosen from set ${\cal A}$.  In particular, suppose we choose
$j$ items from ${\cal A}$ and $n-j$ items from ${\cal B}$.  The number
of ways to choose $j$ items from ${\cal A}$ is $a_j$, and the number
of ways to choose $n-j$ items from ${\cal B}$ is $b_{n-j}$.  By the
Product Rule, we can choose all $n$ items in $a_j b_{n-j}$ ways.
Since we could choose $j = 0, 1, \ldots, n$ items from ${\cal A}$, the
total number of ways to choose $n$ items from ${\cal A} \cup {\cal B}$
is the sum:

\[
a_0 b_n + a_1 b_{n-1} + a_2 b_{n-2} + \ldots a_n b_0
\]

This is equal to the value of $c_n$ computed above.
\end{proof}

In the above argument, we implicitly used the fact that ${\cal A}$ and
${\cal B}$ are disjoint sets when referring to the number of items
chosen from ${\cal A}$; were the sets intersecting, the number of
items drawn from ${\cal A}$ would not be well-defined, since the same
item might be in both sets.

The expression $c_n = a_0 b_n + a_1 b_{n-1} + a_2 b_{n-2} + \ldots a_n
b_0$ may be familiar from a signal processing course; the sequence
$<c_0, c_1, c_2, \ldots>$ is the {\em convolution} of sequences $<a_0,
a_1, a_2, \ldots>$ and $<b_0, b_1, b_2, \ldots>$.

\subsection{Choosing Items with Repetition}
\label{sec:rep}

In the last lecture, we saw that the number of ways to choose $n$
items {\em with repetition} from a $k$-element set is $\binom{n + k -
1}{k}$.  We can derive the same result using generating functions.
This will introduce some new techniques that will be helpful when we
do not know the answer in advance!

\subsubsection{Finding a Generating Function}

Suppose we choose $n$ items with repetition from a set containing a
single item.  Then there is one way to choose zero items, one way to
choose one item, one way to choose a two items, etc.  The generating
function for choosing $n$ elements with repetition from a 1-element
set is therefore:

\begin{eqnarray*}
<1, 1, 1, 1, \ldots> & \leftrightarrow & 1 + x + x^2 + x^3 + \ldots
                      = \frac{1}{1-x}
\end{eqnarray*}

As before, the generating function for choosing items from a union of
disjoint sets is the product of the generating functions for choosing
from each set.

\[
\underbrace{\frac{1}{1-x}}_{
\begin{array}{cc}
\mbox{OGF for} \\
\mbox{$\{ a_1 \}$}
\end{array}}
\underbrace{\frac{1}{1-x}}_{
\begin{array}{cc}
\mbox{OGF for} \\
\mbox{$\{ a_2 \}$}
\end{array}}
\ldots
\underbrace{\frac{1}{1-x}}_{
\begin{array}{cc}
\mbox{OGF for} \\
\mbox{$\{ a_k \}$}
\end{array}}
=
\underbrace{\frac{1}{(1-x)^k}}_{
\begin{array}{cc}
\mbox{OGF for} \\
\mbox{$\{ a_1, a_2, \ldots, a_k\}$}
\end{array}}
\]

The generating function for choosing $n$ items from a $k$-item set
with repetition allowed is $\frac{1}{(1-x)^k}$.

\subsubsection{Finding Coefficients}
\label{sec:find_coeff}

Now we need to find the coefficients of the generating function.  In
particular, the coefficient of $x^n$ in the generating function
$\frac{1}{(1-x)^k}$ gives the number of ways to choose $n$ items
from a $k$-item set with replacement.

Since the generating function is a ratio of polynomials, we could try
using partial fractions as before.  However, the partial fractions
expansion of $\frac{1}{(1-x)^k}$ actually {\em is} just
$\frac{1}{(1-x)^k}$; no progress there!

Another way to find coefficients of a generating function is by
repeatedly differentiating and then evaluating the result at 0.  Here
is how this technique can be used to find the first few coefficients
of a generating function $G(x)$.

\begin{eqnarray*}
G(x) = g_0 + g_1 x + g_2 x^2 + g_3 x^3 + g_4 x^4 + \ldots
	& G(0) = g_0 \\
G'(x) = g_1 + 2 g_2 x + 3 g_3 x^2 + 4 g_4 x^3 + \ldots
	& G'(0) = g_1 \\
G''(x) = 2 g_2 x + 3 \cdot 2 g_3 x + 4 \cdot 3 g_4 x^2 + \ldots
	& G''(0) = 2 g_2 \\
G'''(x) = 3 \cdot 2 g_3 + 4 \cdot 3 \cdot 2 g_4 x + \ldots
	& G'''(0) = 3 \cdot 2 g_3 \\
\end{eqnarray*}

In general, after $n$ derivatives we have $G^{(n)}(0) = n! g_n$.
Therefore, the coefficient of $x^n$ is $G^{(n)}(0) / n!$.

Recall that differentiating a generating function has two effects on
the corresponding sequence: terms are multiplied by their index and
then all terms are shifted left one place.  In this case, we intend to
left-shift $n$ times, but must cancel out the unwanted multiplication
by dividing by $n!$.

All that remains is to compute the $n$-th derivative of $G(x) =
\frac{1}{(1-x)^k}$.  This is not too hard:

\begin{eqnarray*}
G(x) & = & (1-x)^{-k} \\
G'(x) & = & k (1-x)^{-k-1} \\
G''(x) & = & k (k+1) (1-x)^{-k-2} \\
G^{(n)}(x) & = & k (k+1) \ldots (k + n - 1)(1-x)^{-k-n}
\end{eqnarray*}

The coefficient of $x^n$ in the generating function is computed below.

\begin{eqnarray*}
G^{(n)}(0) / n! & = & \frac{k (k+1) \ldots (k + n - 1)}{n!} \\
                & = & \frac{(k + n - 1)!}{n! \ (k-1)!} \\
                & = & \binom{k + n - 1}{n}
\end{eqnarray*}

Therefore, the number of ways to choose $n$ items from a $k$-element
set with repetition allowed is $\binom{k + n - 1}{n}$.  This is the
same result we derived in the last lecture!

\section{An ``Impossible'' Counting Problem}

Here is an absurd counting problem.  In how many ways can we fill a
bag with $n$ fruits subject to the following constraints?

\begin{itemize}
\item The number of apples must be even.
\item The number of bananas must be a multiple of 5.
\item There can be at most four oranges.
\item There can be at most one pear.
\end{itemize}

For example, there are 7 ways to form a bag with 6 fruits:

\[
\begin{array}{c|ccccccc}
\mbox{Apples}  & 6 & 4 & 4 & 2 & 2 & 0 & 0 \\
\mbox{Bananas} & 0 & 0 & 0 & 0 & 0 & 5 & 5 \\
\mbox{Oranges} & 0 & 2 & 1 & 4 & 3 & 1 & 0 \\
\mbox{Pears}   & 0 & 0 & 1 & 0 & 1 & 0 & 1
\end{array}
\]

This problem appears utterly hopeless for an arbitrary number of
fruits, but we can actually use generating functions to find an
answer.

First, we need a generating function for the number of fruit bags.
Initially, we can consider each fruit separately.  That is, we can
count the number of fruit bags with just apples, then just bananas,
petc.  For example, the number of fruit bags with 0 apples is one (the
empty bag), the number of bags with 1 apple is zero (the number of
apples must be even), the number of bags with 2 apples is one (the bag
with two apples), etc.  The generating functions for the four types of
fruit are worked out below.

\begin{eqnarray*}
A(x) = & 1 + x^2 + x^4 + x^6 + \ldots & =  \frac{1}{1-x^2} \\
B(x) = & 1 + x^5 + x^{10} + x^{15} + \ldots & = \frac{1}{1-x^5} \\
O(x) = & 1 + x + x^2 + x^3 + x^4 & = \frac{1-x^5}{1-x} \\
P(x) = & 1 + x
\end{eqnarray*}

As in previous counting problems, the generating function for picking
$n$ items from a union of disjoint sets is the product of the
generating functions for picking from each set.  That is, all fruit
bags are counted by the generating function $A(x) B(x) O(x) P(x)$.
This function is computed below.

\begin{eqnarray*}
\mbox{OGF for fruit bags} & = & A(x) B(x) O(x) P(x) \\
& = & \frac{1}{1-x^2} \frac{1}{1-x^5} \frac{1-x^5}{1-x} (1 + x) \\
& = & \frac{1}{(1-x)^2} \\
& = & 1 + 2x + 3x^2 + 4 x^3 + \ldots
\end{eqnarray*}

The coefficient of $x^n$ is simply $n+1$; that is, the number of ways
to form a bag of $n$ fruits is just $n+1$.  This is consistent with
the example we worked out, since there were 7 fruit bags with 6
fruits.  Amazing!

\end{document}
