Roper Center for Public Opinion Research

Site   Datasets    Advanced Search
University of Connecticut

Fundamentals of Polling-Total Survey Error


Fundamentals of Polling - Total Survey Error


 

 

What is meant by the margin of error?

Most surveys report margin of error in a manner such as: "the results of this survey are accurate at the 95% confidence level plus or minus 3 percentage points." That is the error that can result from the process of selecting the sample. It suggests what the upper and lower bounds of the results are. Sampling error is the only error that can be quantified, but there are many other errors to which surveys are susceptible. Emphasis on the sampling error does little to address the wide range of other opportunities for something to go wrong.

Total Survey Error includes Sampling Error and three other types of errors that you should be aware of when interpreting poll results: Coverage Error, Measurement Error, and Non-Response Error.

Top


What is sampling error?

Sampling Error is the calculated statistical imprecision due to interviewing a random sample instead of the entire population. The margin of error provides an estimate of how much the results of the sample may differ due to chance when compared to what would have been found if the entire population was interviewed.

An annotated example:

There are close to 200 million adult U.S. residents. For comparison, lets say you have a giant jar of 200 million jelly beans. The president has commissioned you to find out how many jelly beans are red, how many are purple and how many are some other color. Since you have limited funds and time you opt against counting and sorting all 200 million jelly beans. Instead you randomly select 500 jelly beans of which 30% are red, 10% are purple and 60% are some other color.

Looking at the matrix below, you find that with a sample of 500 jelly beans you can report that 30 percent of the jelly beans in the jar are red, +/- 4%. To further elaborate, you can say, with 95% confidence red jelly beans make up 30%, {+/- 4% or the range of 26-34%} of the beans in the jar. Likewise you can report that purple jelly beans make up 10% {+/- 3% or the range of 7-13%} of the beans in the jar.

Recommended allowance for sampling error of a percentage *
In Percentage Points (at 95 in 100 confidence level)**

Sample Size

 

1,000

750

500

250

100

Percentage near 10

2%

2%

3%

4%

6%

Percentage near 20

3

3

4

5

9

Percentage near 30

3

4

4

6

10

Percentage near 40

3

4

5

7

10

Percentage near 50

3

4

5

7

11

Percentage near 60

3

4

5

7

10

Percentage near 70

3

4

4

6

10

Percentage near 80

3

3

4

5

9

Percentage near 90

2

2

3

4

6

 

An Important Observation

As the sample size increases, there are diminishing returns in percentage error. At percentages near 50%, the statistical error drops from 7 to 5% as the sample size is increased from 250 to 500. But, if the sample size is increased from 750 to 1,000, the statistical error drops from 4 to 3%. As the sample size rises above 1,000, the decrease in marginal returns is even more noticeable.
 

Notes:
* Table extracted from 'The Gallup Poll Monthly'.
** 95 in 100 confidence level: This means when a sample is drawn there are 95 chances in 100 that the sample will reflect the sampling frame at large within the sampling error (shown in chart).

Langer Research Associates offers a margin-of-error calculator -- MoE Machine -- as a convenient tool for data producers and everyday data users. Access the MoE Machine at http://langerresearch.com/moe.php.

Top


What about my brother who's in the army stationed in Europe--can he be interviewed?

No, not in a typical survey of US adults. This is an example of Coverage Error. That's the error associated with the inability to contact portions of the population. Telephone surveys usually exclude people who do not have landline telephones in their household, the homeless, and institutionalized populations. This error includes people who are not home at the time of attempted contact because they are on vacation, in the military overseas, along with a variety of other reasons that they are unreachable-for the period the interviewing (with call backs) takes place.

Coverage Error affects those who only use a cell phone, since Random Digit Dialing (RDD) samples do not include cell phone exchanges. Recently this problem has grown particularly in trying to reach young people.

Top

What is measurement error?

Measurement Error is error or bias that occurs when surveys do not survey what they intended to measure. This type of error results from flaws in the instrument, question wording, question order, interviewer error, timing, question response options, etc. This is perhaps the most common and most problematic collection of errors faced by the polling industry.

Top


What happens when people can't be reached? What about screening calls?

Non-response Error results from not being able to interview people who would be eligible to take the survey. Many households now have answering machines and caller ID that prevent easy contact; other people simply do not want to respond to calls sometimes because the endless stream of telemarketing appeals make them wary of answering. Non-response bias is the difference in responses of those people who complete the survey vs. those who refuse to for any reason. While the error itself cannot be calculated, response rates can be calculated and there are countless ways to do so. The American Association for Public Opinion Research (AAPOR web site) provides recommended procedures for calculating response rates along with helpful tools and related definitions to assist interested researchers.

Top


What happens when the final sample doesn't look like the general public--For example, what if three quarters of your respondents are over fifty?

Survey firms apply a technique called weighting to adjust the poll results to account for possible sample biases caused by specific groups of individuals not responding. The weighting uses known estimates of the total population provided by the Census to adjust the final results.

It's not uncommon to weight data by age, gender, education, race, etc. in order to achieve the correct demographic proportions.


For further information please contact The Roper Center at 860.486.4440 or rcweb@ropercenter.uconn.edu.

Top