Massachusetts Institute of Technology Lecture 15 6.042J/18.062J: Mathematics for Computer Science October 29, 1996 Professors Nancy Lynch and Tom Leighton

Notes 15

Readings: 4-1, 4-3, 5-4, 5-5 Next: 4-3, 5-4 After: 4-6, 4-7

1 Introduction

The next three lectures will present methods for counting the number of elements in various sets. Counting is one of the most important concepts of discrete math. Subject called combinatorics.

Counting arises in many different ways:

Examples: Counting addresses, keys How many phone numbers are available for use in the U.S.? Internet addresses? 200-digit primes for RSA cryptography?

Example: Measurement for programs Counting amount of time, memory, disk space, etc., used by programs.

Example: Programs that enumerate and test Some programs solve problems by enumerating all candidate solutions and trying them all. To see which is best (optimization problem), or just to try to find some solution.

Operations research optimization problems: travelling salesman problem, etc. Dynamic programming – a clever method for cutting down the search space. Circuit testing, by trying all inputs. Finite-state program verification, by testing all reachable states.

Nice to know how many candidates there are, so we know how long to expect the program to run.

Also, counting methods often yield ways of explicitly enumerating the elements. Use the enumeration in the program.

Example: Probability Want to know the probability that some event happens when some system or program operates. The probability of an event is closely related to the number of ways that event might happen. Calculating probabilities is based on counting number of ways.

You all understand counting informally. Actually, it’s a very formal notion, based on bijections: A set A has size (cardinality) n if and only if there is a bijection from A to the set {1, …, n}. To prove this, we don’t always exhibit a bijection directly to {1, …, n}; sometimes, show a bijection to another set B, which we already know is of size n. Uses the fact that two sets have the same cardinality if and only if there is a bijection between them.

Often counting arguments will look quite informal, but can always formalize them in terms of bijections.

Combinatorics contains a very large number of formulas to help in counting. It’s impossible to learn them all. The key is to learn the basics, then look up or derive the other formulas when you need them.

The complicated formulas can all be proved formally, using algebra. Many of them can also be understood informally, using what we will call a thought experiment. This involves giving an intuitive meaning to the expressions on the LHS and RHS of an equation, then verifying that these two meanings are the same.

2 Basic Counting Principles

Start with the basic counting principles: Sum rule, inclusion-exclusion principle, product rule, tree diagrams, division rule

2.1 Sum

Theorem 2.1 If A and B are disjoint, then |A ∪ B| = |A| + |B|.

This expresses the idea as an equation, which we will prove formally in terms of bijections.

Can also understand it by a “thought experiment”: LHS denotes the number of ways of choosing an element of A ∪ B, RHS denotes the number of ways of choosing an element of A plus the number of ways of choosing an element of B. Since the sets are disjoint, the LHS is the sum...(informal)

Proof. Let n_{A} = |A|, n_{B} = |B|. This means that there are bijections f_{A} and f_{B}, f_{A} from A to {1, …, n_{A}}, f_{B} from B to {1, …, n_{B}}. We want a bijection from A ∪ B to {1, …, n_{A} + n_{B}}.

So define a function g on A ∪ B by:

g(a) = \{ \begin{matrix}f_{A}(a) & if a ∈ A \\ n_{A} + f_{B}(a) & if a ∈ B \end{matrix}

The function g is well-defined since A and B are disjoint. We must show that g is a bijection. That is, one-to-one and onto.

One-to-one: Suppose that g(a) = g(b). If a, b ∈ A then g(a) = f_{A}(a) = f_{A}(b) = g(b); since f_{A} is one-to-one, we have a = b. If a, b ∈ B then g(a) = n_{A} + f_{B}(a) = n_{A} + f_{B}(a) = g(b); so f_{B}(a) = f_{B}(b); since f_{B} is one-to-one, we again have a = b. If a ∈ A, b ∈ B (wlog), then g(a) ≤ n_{A} and g(b) > n_{A}; contradiction.

Onto: Let n ∈ {1, …, n_{A} + n_{B}}. If n ≤ n_{A} then n = f_{A}(a) for some a ∈ A, since f_{A} is onto; so n = g(a). If n > n_{A}, then n = n_{A} + m for some m ∈ {1, …, n_{B}}; then m = f_{B}(b) for some b ∈ B, since f_{B} is onto; so g(b) = n_{A} + f_{B}(b) = n_{A} + m = n. Therefore, g is a bijection, as needed.

Recall that any bijection has an inverse function. Inverse here is a function from {1, …, n_{A} + n_{B}} to A ∪ B:

g^{−1}(n) = \{ \begin{matrix}f_{A}^{−1}(n) & if n ≤ n_{A} \\ f_{B}^{−1}(n − n_{A}) & if n > n_{A} \end{matrix}

In fact, note that a function has an inverse if and only if it is a bijection. Therefore, showing that a function has an inverse is another way of showing that it is a bijection. Here, define the inverse function, must show it’s defined everywhere, and well-defined.

Example 1 99 math majors, 179 cs majors, How many majors in math or cs (assume no double majors)? 99 + 179 = 278

Extend sum rule to any number of sets:

Theorem 2.2 If A_{1}, …, A_{n} are pairwise disjoint, then |A_{1}∪…∪A_{n}| = |A_{1}|+|A_{2}|+…+|A_{n}|.

Proof by induction on n.

Example 2 99 math, 179 cs, 24 philosophy Total in math or cs or philosophy = 302

2.2 Inclusion-Exclusion

What if the sets aren’t disjoint? E.g., math and cs majors, where some people are majoring in both? Consider two sets first.

Theorem 2.3 If A and B are any sets (not necessarily disjoint), then |A ∪ B| = |A| + |B| − |A ∩ B|.

Proof. (First proof) Construct a bijection from A∪B to {1, …, n_{A} +n_{B} −n_{AB}} explicitly, mapping all the elements of A to {1, …, n_{A}}, and all the elements of B not in the intersection to {n_{A} + 1, …, n_{A} + n_{B} − n_{AB}}. Here, n_{A}, n_{B}, n_{AB} denote the number of elements in A, B and the intersection respectively. There is a bijection from B to {1, …, n_{B}}, so B−(A∩B) has a bijection to {1, …, n_{B}−n_{AB}}. This assumes the elements in the intersection maps to the largest elements. We can then conver this to a bijection from B to {n_{A} + 1, …, n_{A} + n_{B} − n_{AB}}.

Figure 1: Venn diagram.

Proof. (Second proof) Use Theorem 2.1, for disjoint sets. But apply it to the disjoint sets A and B − A. Since A ∪ B = A ∪ (B − A), it’s enough to find the cardinality of A ∪ (B − A). By Theorem 2.1, this is |A| + |B − A|. We want it to be |A| + |B| − |B − A|.

So we need to show that |B − A| = |B| − |A ∩ B|. That is, show that |B| = |B − A| + |A ∩ B|. But this follows again from Theorem 2.1, this time applied to |B − A| and |A ∩ B| (which are disjoint and have B as their union).

The second proof above was constructed “by need”. A better way to write it, after you figure it out is in ordered steps, beginning to end:

• B = (B − A) ∪ (A ∩ B), by set identity; disjoint.

• |B| = |B − A| + |A ∩ B|, by Theorem 2.1.

• A ∪ B = A ∪ (B − A), by set identity; disjoint.

• |A ∪ B| = |A| + |B − A|, by Theorem 2.1.

• Now use algebra: |A ∪ B| = |A| + |B − A| = |A| + |B| − |A ∩ B|.

The order in which you think of the steps the first time is often not the same as the order in which the steps appear in the final version of the proof (that builds logically, step-by-step).

Example 3 99 math majors, 179 cs majors, 7 majoring in both. How many majors in math or cs? 99 + 179 - 7 = 271

Example 4 How many numbers in {1, …, 100} are divisible by 2 or 5?

• 50 divisible by 2

• 20 divisible by 5

• 10 divisible by both 2 and 5

• total: 50 + 20 − 10 = 60.

What about more than two sets? What is |A ∪ B ∪ C|, in general? Draw Venn diagram:

Well, |A ∪ B ∪ C| = |A| + |B| + |C| minus something for the overlapping part. What to remove for the overlapping part? Counted each 2-overlapped part twice, and the 3-overlapped part 3 times. Suppose we subtract |A ∩ B| + |B ∩ C| + |A ∩ C|. Now we’re counting the 2-overlapped parts only once, as we want, but the 3-overlapped part isn’t counted at all! So need to correct again, this time by adding in |A ∩ B ∩ C|, The full expression:

Theorem 2.4 |A ∪ B ∪ C| = (|A| + |B| + |C|) − (|A ∩ B| + |B ∩ C| + |A ∩ C|) + |A ∩ B ∩ C|.

Example 5 99 math, 179 cs, 24 philosophy math and cs: 7 (includes all 3 also) math and philo: 12 cs and philo: 2 all 3: 1 Total in math or cs or philosophy = (99 + 179 + 24) − (7 + 12 + 2) + 1 = 282.

In general, for m sets, alternate plus and minus terms:

Theorem 2.5 | ∪_{i} A_{i}| = Σ_{i}|A_{i}| − Σ_{i,j distinct} |A_{i} ∪ A_{j}| + Σ_{i,j,k distinct} |A_{i} ∪ A_{j} ∪ A_{k}| − … + (−1)^{m+1}| ∩ A_{i}|

Can be proved by induction on the number of sets (complicated details – LTTR).

But come back to this Thursday, with a “thought experiment” proof.

Example 6 How many primes less than 100? 99 minus numbers divisible by 2, 3, 5 or 7 (since non-prime means divisivle by some prime < \sqrt{100}) plus these 4 numbers. Let

P_{i} = numbers divisible by i note |P_{i}| = ⌊100/i⌋.

Then quantity is:

99 − ⌊\frac{100}{2}⌋ − ⌊\frac{100}{3}⌋ − ⌊\frac{100}{5}⌋ − ⌊\frac{100}{7}⌋
+ ⌊\frac{100}{2 · 3}⌋ + ⌊\frac{100}{2 · 5}⌋ + ⌊\frac{100}{2 · 7}⌋ + ⌊\frac{100}{3 · 5}⌋ + ⌊\frac{100}{3 · 7}⌋ + ⌊\frac{100}{5 · 7}⌋
− ⌊\frac{100}{2 · 3 · 5}⌋ − ⌊\frac{100}{2 · 3 · 7}⌋ − ⌊\frac{100}{2 · 5 · 7}⌋ − ⌊\frac{100}{3 · 5 · 7}⌋
+ ⌊\frac{100}{2 · 3 · 5 · 7}⌋

which is

99 − 50 − 33 − 20 − 14 + 16 + 10 + 7 + 6 + 4 + 2 − 3 − 2 − 1 − 0 + 0 = 21

so there are 25 primes.

2.3 Product

Theorem 2.6 If A and B are any sets, then |A × B| = |A| · |B|.

Thought experiment: LHS represents the number of ways of choosing a pair consisting of an element of A and an element of B. RHS represents number of ways to choose an element of A, multiplied by number of ways to choose an element of B. If there are n_{A} ways to choose an element of A, and n_{B} ways to choose an element of B, then there are n_{A} · n_{B} ways to choose a pair.

Proof. Let n_{A} = |A|, n_{B} = |B|. Let f_{A} be a bijection from A to {0, …, n_{A} − 1}, f_{B} from B to {0, …, n_{B} − 1}. These exist since these sets are the same size as the corresponding sets starting at 1. We want a bijection from A × B to {0, …, n_{A} · n_{B} − 1}. Define by:

g((a, b)) = n_{B}(f_{A}(a)) + f_{B}(b)

We can show that g is a bijection with inverse:

g^{−1}(n) = (f_{A}^{−1}(⌊\frac{n}{n_{B}} ⌋), f_{B}^{−1}(n mod n_{B})).

In other words, for any n ∈ {0, …, n_{A} · n_{B} − 1}, write n = qn_{B} + r, where 0 ≤ r < n_{B}, by division theorem. Then

g^{−1}(n) = (f_{A}^{−1}(q), f_{B}^{−1}(r)).

Example 7 How many ways can you choose 2 courses in different subjects from among 5 Math, 7 CS, and 10 EE courses?

• 1 Math and 1 CS: 5 × 7 = 35 ways (product rule)

• 1 Math and 1 EE: 5 × 10 = 50 ways

• 1 CS and 1 EE: 7 × 10 = 70 ways.

By sum rule, total of 155 ways.

Extend to m sets:

Theorem 2.7 |A_{1} × A_{2} × … × A_{m}| = |A_{1}| · |A_{2}| · . . . · |A_{m}|

This says that m successive choices, with exactly n_{k} choices possible at the k^{th} step, can produce a total of exactly n_{1}n_{2} ⋯ n_{m} different results.

Example 8 How many binary numbers of length n? By product rule, 2 × … × 2 (n terms). So 2^{n}.

Example 9 Number of subsets of n-element set? We already know this is 2^{n}. Each subset corresponds to a bit string of length n, with a 1 in position k meaning that the kth element is in the subset. That is, set of subsets has bijection to set of bit strings of length n, which has 2^{n} elements.

Example 10 License plates – 3 letters followed by 3 digits: 26 × 26 × 26 × 10 × 10 × 10.

Example 11 Number of passwords – length 8, letters and digits, at least one letter. First calculate number of length 8 strings of letters and digits, counting those witn no letters: 36^{8}. Then subtract those with no letters: 26^{8}. The difference is what’s needed: 36^{8} − 26^{8}.

Example 12 How many functions from set A with m elements to set B with n elements? Function corresponds to m choices from B, one for each element of A. That is, set of functions has bijection to set B × … × B (m terms). This set has size n^{m}, by the product rule.

Example 13 How many length-n palindromes over {0, 1}? Turns out exactly 2^{⌈n/2⌉}.

Proof. We consider the cases n even and odd separately.

• If n is even we define a bijection from the set {0, 1}^{n/2} to the set of palindromes of length n. This will imply that there are |{0, 1}^{n/2}| = 2^{n/2} palindromes. For a sequence a = (a_{1}, ⋯, a_{n/2}) define f(a) = (a_{1}, ⋯, a_{n/2}, a_{n/2}, ⋯, a_{1}). Clearly, f(a) is a palindrome and f is bijective.

• if n is odd, we define a bijection from the set {0, 1}^{⌈n/2⌉} to the set of palindromes of length n. This will imply that there are |{0, 1}^{⌈n/2⌉}| = 2^{⌈n/2⌉} palindromes. For a sequence a = (a_{1}, ⋯, a_{⌈n/2⌉}) define g(a) = (a_{1}, ⋯, a_{⌈n/2⌉}, ⋯, a_{1}). Clearly, g(a) is a palindrome and g is bijective.

Hence, in either case there are 2^{⌈n/2⌉} palindromes.

2.4 Tree Representation

Interpreting product rule in terms of number of successive choices. Can represent successive choices by tree, then total number of choices is number of leaves.

Example 14 Bit strings of length 3: Draw tree

Example 15 License places

This tree representation can also be used to count numbers of successive choices, in the more general case where later choices are from different sets, depending on the results of the earlier choices. Then the answer isn’t a product, but can still be calculated from the number of leaves of the tree.

Example 16 3 blue shirts, 2 red shirts, 1 green shirt, 2 grey pants and 3 brown pants. How many outfits if grey matches red/blue and brown matches red/green?

• 6 blue-grey, 4 red-grey, 6 red-brown and 3 green-brown makes 19.

Figure 2: Tree of outcomes.

• Alternatively, the tree of outcomes may be represented as below (See Fig. 1):

Example 17 Number of strings of letters of length 5, with no repeats: 26 × 25 × 24 × 23 × 22. Show as tree, with later sets depending on earlier choices.

2.5 Division Rule

One last basic rule: division rule:

Theorem 2.8 If there is a k to 1 mapping from A to B, and |A| = n, then |B| = n/k.

Example 18 How many subsets of size 2 are there, of a 5-element set A? Let S be this set of subsets.

First, consider set P of ordered pairs without repeats. Have |P | = 5 × 4 (since can’t repeat – consider tree).

There’s a 2 to 1 mapping from P to S – just ignore the order. So S has 20/2 = 10 elements.

3 Permutations and Combinations

Some of the examples we have seen (like the last one) deal with the number of ways of choosing some particular number of elements from a set. As you can see in this last example, there are two different ways to select them: in a particular order, or just in an (unordered) set. Selections are called permutations when we care about the order, and combinations when we don’t.

3.1 Permutations

Definition 3.1 An r-permutation of a set of n elements is an ordered arrangement of r of the n elements (all distinct).

Example 19 S = {A, B, C}. The 2-permutations are (A, B), (B, A), (A, C), (C, A), (B, C), (C, B). 6 2-permutations

How many r-permutations of a set of n elements?

• n ways to choose first element

• given first choice, n − 1 ways to choose second element

• n − 2 to choose third

• overall:

n(n − 1) ⋯ (n − r + 1) = \frac{n!}{(n − r)!}.

Denote this by P (n, r).

Definition 3.2 A permutation of n elements is an n-permutation of them.

P (n, n) = n!

Example 20 How many ways can n people stand in a circle?

• Map each permutation to a ring.

• n permutations mapped to each ring.

• By division rule, number of rings is n!/n = (n − 1)!.

3.2 Combinations

Definition 3.3 An r-combination of a set of n elements is an (unordered) subset of the n elements.

Example 21 S = {A, B, C}. The 2-combinations are {A, B} , {A, C} , {B, C}. 3 of them.

How many r-combinations of a set of n elements?

• Map each r-permutation to an r-combination by ignoring the order.

• How many r-permutations mapped to each r-combination? r! - because there are P (r, r) = r! permutations of a given r-combination.

• By division rule, P (n, r)/r!

Denote this by C(n, r), or more commonly as \binom{n}{r}.

Theorem 3.4 The number of r-combinations of an n-set is:

\binom{n}{r} = \frac{n!}{r!(n − r)!}

\binom{n}{r} is called a binomial coefficient for reasons that will become clear later.

Example 22 How many ways can 3 distinct numbers be chosen from 1, …, 10? This is \binom{10}{3} = \frac{10!}{3!7!} = 120.

Example 23 How many ways can 3 distinct numbers be chosen from 1, …, 10 so that their sum is divisible by 3?

• A = {3, 6, 9} (= 0 mod 3)

• B = {1, 4, 7, 10} (= 1 mod 3)

• C = {2, 5, 8} (= 2 mod 3)

Need sum to be zero mod 3. If any two are equal mod3, then the third must also be equal. So they are all same or all different. Thus the possibilities are:

• 3 from A: \binom{3}{3} = 1.

• 3 from B: \binom{4}{3} = 4.

• 3 from C: \binom{3}{3} = 1.

• One from each: 3 · 4 · 3 = 36 (product rule).

Total: 42.