 Introduction
 Sampling
 Total Survey Error
 Understanding Tables
 Glossary of Terminology
This tutorial offers a glimpse into the fundamentals of public opinion polling. Designed for the novice, Polling Fundamentals provides definitions, examples, and explanations that serve as an introduction to the field of public opinion research.
Total Survey Error
What is meant by the margin of error?
Most surveys report margin of error in a manner such as: “the results of this survey are accurate at the 95% confidence level plus or minus 3 percentage points.” That is the error that can result from the process of selecting the sample. It suggests what the upper and lower bounds of the results are. Sampling error is the only error that can be quantified, but there are many other errors to which surveys are susceptible. Emphasis on the sampling error does little to address the wide range of other opportunities for something to go wrong.
Total Survey Error includes Sampling Error and three other types of errors that you should be aware of when interpreting poll results: Coverage Error, Measurement Error, and NonResponse Error.
Sampling Error is the calculated statistical imprecision due to interviewing a random sample instead of the entire population. The margin of error provides an estimate of how much the results of the sample may differ due to chance when compared to what would have been found if the entire population was interviewed.
An annotated example:
There are close to 200 million adult U.S. residents. For comparison, let’s say you have a giant jar of 200 million jelly beans. The president has commissioned you to find out how many jelly beans are red, how many are purple, and how many are some other color. Since you have limited funds and time, you opt against counting and sorting all 200 million jelly beans. Instead you randomly select 500 jelly beans of which 30% are red, 10% are purple and 60% are some other color.
Looking at the matrix below, you find that with a sample of 500 jelly beans you can report that 30 percent of the jelly beans in the jar are red, +/ 4%. To further elaborate, you can say, with 95% confidence red jelly beans make up 30%, {+/ 4% or the range of 2634%} of the beans in the jar. Likewise you can report that purple jelly beans make up 10% {+/ 3% or the range of 713%} of the beans in the jar.
Recommended allowance for sampling error of a percentage *
In Percentage Points (at 95 in 100 confidence level)**
Sample Size
1,000 
750 
500 
250 
100 

Percentage near 10

2% 
2% 
3% 
4% 
6% 
Percentage near 20

3 
3 
4 
5 

Percentage near 30

3 
4 
4 
6 
10 
Percentage near 40

3 
4 
5 
7 
10 
Percentage near 50

3 
4 
5 
7 
11 
Percentage near 60

3 
4 
5 
7 
10 
Percentage near 70

3 
4 
4 
6 
10 
Percentage near 80

3 
3 
4 
5 
9 
Percentage near 90

2 
2 
3 
4 
6 
An Important Observation
As the sample size increases, there are diminishing returns in percentage error. At percentages near 50%, the statistical error drops from 7 to 5% as the sample size is increased from 250 to 500. But, if the sample size is increased from 750 to 1,000, the statistical error drops from 4 to 3%. As the sample size rises above 1,000, the decrease in marginal returns is even more noticeable.
Notes:
* Table extracted from ‘The Gallup Poll Monthly’.
** 95 in 100 confidence level: This means when a sample is drawn there are 95 chances in 100 that the sample will reflect the sampling frame at large within the sampling error (shown in chart).
Langer Research Associates offers a marginoferror calculator — MoE Machine — as a convenient tool for data producers and everyday data users. Access the MoE Machine at http://langerresearch.com/moe.php.
What is coverage error?
In a typical survey of US adults, some groups of people will not have the opportunity to be included, such a military personnel stationed overseas. This is an example of Coverage Error. That’s the error associated with the inability to contact portions of the population. Telephone surveys usually exclude the homeless and institutionalized populations. This error also includes people who are not home at the time of attempted contact because they are on vacation, living abroad, or otherwise unreachable for the period of time the interviewing (with call backs) takes place.
What is measurement error?
Measurement Error is error or bias that occurs when surveys do not survey what they intended to measure. This type of error results from flaws in the instrument, question wording, question order, interviewer error, timing, question response options, etc. This is perhaps the most common and most problematic collection of errors faced by the polling industry.
What happens when people can’t be reached? What about screening calls?
Nonresponse Error results from not being able to interview people who would be eligible to take the survey. Many households now use voice mail and caller ID to screen calls; other people simply do not want to respond to calls sometimes because the endless stream of telemarketing appeals make them wary of answering. Nonresponse bias is the difference in responses of those people who complete the survey vs. those who refuse to for any reason. While the error itself cannot be calculated, response rates can be calculated and there are countless ways to do so. The American Association for Public Opinion Research (AAPOR web site) provides recommended procedures for calculating response rates along with helpful tools and related definitions to assist interested researchers.
What happens when the final sample doesn’t look like the general public? For example, what if threequarters of your respondents are over fifty?
Survey firms apply a technique called weighting to adjust the poll results to account for possible sample biases caused by specific groups of individuals not responding. The weighting uses known estimates of the total population provided by the Census to adjust the final results.
It’s not uncommon to weight data by age, gender, education, race, etc. in order to achieve the correct demographic proportions.
What about people who only use cell phones? Don’t polls miss them?
There was a time when polls only sampled the population who had landlines. However, as increasing numbers of people have moved to using only cell phones, the industry has had to make changes in methodology. Now, most polls are conducted with both landline and cellphone samples.
For further information please contact The Roper Center at 860.486.4440 or support@ropercenter.org.