\documentclass[11pt,twoside]{article}

\usepackage{latex-macros/course}

\begin{document} 
\inclassproblems{15, Mon.}

\begin{problem} \textbf{Explaining Sampling to a Jury.}

We used the Binomial Sampling Theorem (stated below) to show that merely
sampling 662 voters will yield a fraction that, 95\% of the time, is within
0.04 of the of the actual fraction of voters who prefer Kerry.  The actual
size of the voting population (10's of millions) was never considered
because \emph{it did not matter}.

Suppose you were going to serve as an expert witness in a trial.  How
would you explain why the number of people necessary to poll \emph{does
not depend on the population size}?

\begin{theorem*} [Binomial Sampling]
Let $K_1, K_2, \dots$, be a sequence of mutually independent 0-1-valued
random variables with the same expectation, $p$, and let
\[
S_n \eqdef \sum_{i=1}^n K_i.
\]
Then, for any $\epsilon > 0$,
\[
\pr{\abs{\frac{S_n}{n} - p} \geq \epsilon}
\leq 
\frac{1 - 2\epsilon}{2\epsilon} \cdot
		\frac{2^{-n (1- H((1/2) - \epsilon))}}{\sqrt{2 \pi (1/4 - \epsilon^2) n}}
\]
where
\[
H(\alpha) \eqdef - \alpha\log_2 \alpha - (1-\alpha)\log_2 (1-\alpha).
\]
\end{theorem*}


\solution{This was intended to be a thought-provoking, conceptual
question.  In past terms, although most of the class could follow the
derivations and crank through the formulas to calculate poll size and
confidence levels, many students couldn't articulate, and indeed didn't
really believe that the derived sample sizes were actually adequate to
produce reliable estimates.

Here's a way to explain why we model polling people about Kerry as
independent coin tosses that a jury might be able to follow:

\begin{quote}
Of the approximately 100,000,000 voters in the US, there are some
\emph{unknown} number, say 51,000,000, who favor Kerry.  So in this case,
the \emph{fraction} of voters favoring Kerry would be
51,000,000/100,000,000 = 0.51.

To estimate this unknown fraction, we randomly select one person from the
100,000,000 in such a way that \emph{everyone has an equal chance of being
picked}.  For example, we might get computer files from the Census Bureau
listing all 100,000,000 voters in the US.  Then we would generate a number
between 1 and 100,000,000 by some physical or computational process that
generated each number with equal probability, and then we would interview
the person whose number came up.  Picking a person this way amounts to
flipping a coin that had a chance of coming up ``Kerry'' that was equal to
the unknown fraction.  In our example, there would be a 51\% chance of the
``coin'' coming up ``Kerry'' and a 49\% chance of coming up ``Bush.''  

After we have picked a person and learned their voting preference, we
perform the procedure again, making sure that everyone is equally likely
to be picked the second time, and so on, for picking a third, fourth, \etc
person.  Each pick is like flipping a coin whose probability of coming up
``Kerry'' is the same unknown fraction.

Now we all understand that if we keep flipping a coin with a 51\% chance
of coming up Heads, then the more we flip, the closer the fraction of
Heads flipped will be to 51\%.  Mathematical theory lets us calculate us
how many times to flip coins to make the fraction of Heads very likely
close to 51\%, but we need't go into the details of the calculation.

Now suppose we had two coins, say a penny and a nickel, which had the same
51\% probability of coming up Heads.  Then it's not going to make any
difference which coin we use in our coin flips: the number of flips we
need to get the fraction of Heads flipped being very likely close to 51\%
will be the same whether we flip pennies or nickels.

Different size populations correspond to different coins: the nickel might
correspond to selecting a voter from a population 100,000,000 people, and
the dime might correspond to selecting one from a population of 100.  The
same number of flips of pennies or nickels will allow us to estimate the
probability of ``Heads,'' and hence to estimate the fraction of voters
favoring Kerry.  All that mattered is that the ``penny'' population had the
same probability of Heads, namely, 51,000,000 out of a population of
100,000,000, as the ``nickel'' population, namely, 51 out of 100.

So the number of ``flips'' needed does not depend on whether we're
flipping a ``51\% penny'' or a ``51\% nickel.''  That is, if two
populations have the same fraction of voters favoring Kerry, then \emph{the
number of people we need to poll is the same}, even if the populations are
of very different sizes.
\end{quote}
}

\end{problem}

\instatements{\newpage}
\begin{problem}
A new political journal announces as part of its editorial policy that
only poll results identifying a candidate preferred by more than half of
the voter at the 95\% confidence level will be acceptable for publication.
They will, however, publish any such poll that identifies the leading
candidate at this confidence level.  The editors reason that under this
policy, their readership can be confident that at most 5\% of their
listings of leading candidates will be incorrect.

The Republication National Committee plans to finesse the journal's policy
and make several Republican candidates who are trailing in tight races
appear to be in the lead.  Namely, the Committee identifies 200 districts
in which the Republican candidate is actually preferred by slightly fewer
than half of the voters in their districts.  It instructs each of the
managers in these district campaigns to conduct an \emph{honest},
\emph{random} poll of voter preference in their district.  The poll should
involve the minimum number of voters ensuring, at the 95\% confidence
level, that the fraction of voters in the poll who prefer the Republican
candidate will be less than 1\% larger than the actual fraction of voters
in the district who prefer the Republican candidate.

The Committee plans to submit to the journal the results of just those
polls where 51\% or more of the voters polled prefer the Republican
candidate.  The Committee expects that this will happen in around 5\% of
the polls.  That is, around 10 polls will show, at the 95\% confidence
level, that the Republican candidate has more than half the votes in his
district, when in fact he does not.

So the journal will publish its 95th\%-confidence-level list of leading
candidates, and its list will be \emph{wrong about every one} of the 10 or
so candidates whose polls are submitted by the Committee.

Will this strategy by the National Committee work?  If not explain why.
If so, discuss the fallacy in the editors' reasoning. 

\solution{Yes, the strategy will work.

Even though all the several hundred Republican candidates are slightly
behind in their campaigns, around 5\% of the poll results can be expected
to be more than 1\% larger than the actual fraction who prefer the
Republican.  So nearly 5\% of the polls will estimate that the Republican
is preferred by at least 51\% of the voters, when in fact the Republican
is preferred by slightly less than 50\%.

The editors have confused the nature of statistical confidence levels with
probability: confidence at the 95\% level does \textbf{not} mean that
there is only 1 chance in 20 that a poll is right.  It doesn't even make
sense to talk about the probability of a real-world fact such as which
candidate is leading.  We can only say that our prediction based on a
random sample has probability 0.95 of being right.  This means that if we
took many samples and made predictions based on each sample, there is a
high probability that 95\% of our predictions would be right.  But when
the National Committee selects just those sample results it likes, the
confidence level we assign to those results is no longer relevant.

There really is no clear alternative policy the journal editors could
adopt that would overcome this kind of problem, even if there wasn't a
National Committee trying to mislead readers.  For example, in medical
trials, there are many more than 200 ``alternative'' therapies being
evaluated by proponents who may be gullible, but are genuinely honest.
Nevertheless, at the 95\% confidence level, we can expect 10 of the 200
evaluations of useless or even damaging alternative therapies to yield
positive results.  The remaining 190 medical trials would not be submitted
for publication by honest proponents because the trials did not show
positive results at the 95\% level.  But the 10 that did would be
submitted by honest medical testers with no intention to mislead.

This is why, unless there is a explanation of \emph{why} a therapy works,
scientists and doctors usually doubt results claiming to confirm the
efficacy of some mysterious therapy at a high confidence level.}

\end{problem}


\end{document}
