TOPICS COVERED:

Introduction to Public Opinion Polling
Sampling
Total Survey Error
Reading Tables
Logical Links
A Poll About Polls

1. What is a public opinion poll?
2. What's a random sample?
3. When I receive some survey calls, they're trying to sell me something. Is this a public opinion poll?
4. Why haven't I been asked to participate in a national opinion poll?
5. Why should I participate in an opinion poll?

Top

SAMPLING

1. What is a scientific sample?
2. How are the surveys conducted?
3. How is the sample selected for a telephone survey?
4. How are face-to-face samples selected?
5. What's a self-administered survey?

Top

TOTAL SURVEY ERROR

1. What is meant by the margin or error?
2. What is sampling error?
3. What about my brother who's in the army stationed in Europe--can he be interviewed?
4. What is measurement error?
5. What happens when people can't be reached? What about screening calls?
6. What happens when the final sample doesn't look like the general public--For example, what if three quarters of your respondents are over fifty?

Top

READING TABLES

1. How do I read the numbers in tables reported?

Top

INTRODUCTION

1. What is a public opinion poll?

A poll is a type of survey or inquiry into public opinion conducted by interviewing a random sample of people.

Back

2. What's a random sample?

A random sample is the result of a process whereby a selection of participants is made from a larger population and each subject is chosen entirely by chance.

Back

3. When I receive some survey calls, they're trying to sell me something. Is this a public opinion poll?

No. Telemarketing calls are different from public opinion polls. A telemarketer’s objective is to sell you something, rather than learn of your opinions—although sometimes he or she will disguise the motive with a few questions first. The goal of the public opinion pollster is to measure the views of the targeted sample in the population.

Back

4. Why haven't I been asked to participate in a national opinion poll?

The US Census tells us there are more than 200 million American adults and most polls generally include about 1,000 respondents. So, let's say 2,500 national polls are completed each year, that's only 2,500,000 people. Assuming no one is interviewed more than once, the odds of being called in any given year are just over 1 in 100.

Back

5. Why should I participate in an opinion poll?

Public policy decisions are being made all the time. There are all sorts of interest groups who are making their positions known to those decision-makers. The public opinion poll provides an opportunity for the voices of the common man and woman to be heard. So, why wouldn’t you want your views heard—it’s your privilege in a democratic society!

Back

SAMPLING

1. What is a scientific sample?

A scientific sample is a process in which the respondents are chosen randomly by one of several methods. The key component in the scientific sample is that everyone within the designated group (sample frame) has a chance of being selected.

Back

2. How are the surveys conducted?

Two of the most common ways that public opinion polls are completed is by telephone and face-to-face interviews. Other methods include mail, on-line and self-administered surveys.

Back

3. How is the sample selected for a telephone survey?

Typically, survey organizations conducting telephone surveys purchase a Random Digit Dial (RDD) sample of randomly generated phone numbers from a firm that specializes in designing samples that have been purged to eliminate business numbers, dead lines, etc. Much could be said on this topic, but to keep it simple, a 10-digit phone number in the United States consists of 4 parts.

The Dissection of a Telephone Number

All 4 components are assigned by the telephone company. The first 3 components are based on location and the final component is randomly generated.

The interviewer will then randomly select a person in the household to be interviewed. One common method is to ask for the adult in the household who had the most recent birthday. This is done because certain parts of the population, such as young males, are more difficult to get on the phone than others, such as the elderly.

Back

4. How are face-to-face samples selected?

Face-to-face surveys, also known as 'in-person' interviews, are conducted with the interviewer and the interviewee next to each other. The interviewer reads material from the questionnaire and records the responses. At times the interviewer may hand a card to the respondent for him/her to select response(s).

Scientific face-to-face surveys are normally conducted using geographic area probability sampling. Some refer to this as 'block sampling'. Selecting a sample can be tedious work in order to represent the population you are targeting. The general way it works is referred to as multi-stage sampling. The population frame is first identified by blocks. For instance, you are starting a new business in Cincinnati, and you want to find out how many households have various items. You need to identify the sampling frame. In this case, to make the job cost effective, you divide the city into 1,000 'blocks' based on size so each block has roughly the same number of adults. Within each block there are 250 housing units. In order to get 500 completed interviews, with an estimated 80% completion rate, you first randomly choose 25 blocks then randomly choose 25 housing units within each block. Once at the housing unit level the final step is to randomly choose a respondent within the household.

Back

5. What's a self-administered survey?

In a self-administered survey, the respondent is directly handed the questionnaire to fill out. Exit polls are examples of self-administered surveys. Voters leaving polling booths are randomly selected to fill out a questionnaire in this type of survey.

Back

TOTAL SURVEY ERROR

Most surveys report margin of error in a manner such as: "the results of this survey are accurate at the 95% confidence level plus or minus 3 percentage points." That is the error that can result from the process of selecting the sample. It suggests what the upper and lower bounds of the results are. Sampling error is the only error than can be quantified, but there are many other errors to which surveys are susceptible. Emphasis on the sampling error does little to address the wide range of other opportunities for something to go wrong.

Total Survey Error includes Sampling Error and three other types of errors that you should be aware of when interpreting poll results: Coverage Error, Measurement Error, and Non-Response Error.

Back

2. What is sampling error?

Sampling Error is the calculated statistical imprecision due to interviewing a random sample instead of the entire population. The margin of error provides an estimate of how much the results of the sample may differ due to chance when compared to what would have been found if the entire population was interviewed.

An annotated example:

There are close to 200 million adult U.S. residents. For comparison, lets say you have a giant jar of 200 million jelly beans. The president has commissioned you to find out how many jelly beans are red, how many are purple and how many are some other color. Since you have limited funds and time you opt against counting and sorting all 200 million jelly beans. Instead you randomly select 500 jelly beans of which 30% are red, 10% are purple and 60% are some other color.

Looking at the matrix below, you find that with a sample of 500 jelly beans you can report that 30 percent of the jelly beans in the jar are red, +/- 4%. To further elaborate you can say that you can say with 95% confidence that red jelly beans make up 30%, {+/- 4% or the range of 26-34%} of the beans in the jar. Likewise you can report that purple jelly beans make up 10% {+/- 3% or the range of 7-13%} of the beans in the jar.

Recommended allowance for sampling error of a percentage *
In Percentage Points (at 95 in 100 confidence level)**

Sample Size

Back

3. What about my brother who's in the army stationed in Europe--can he be interviewed?

No, not in a typical survey of US adults. This is an example of Coverage Error. That's the error associated with the inability to contact portions of the population. Telephone surveys usually exclude people who do not have landline telephones in their household, the homeless, and institutionalized populations. This error includes people who are not home at the time of attempted contact because they are on vacation, in the military overseas, along with a variety of other reasons that they are unreachable-for the period the interviewing (with call backs) takes place.

Coverage Error affects those who only use a cell phone, since Random Digit Dialing (RDD) samples do not include cell phone exchanges. Recently this problem has grown particularly in trying to reach young people.

Back

4. What is measurement error?

Measurement Error is error or bias that occurs when surveys do not survey what they intended to measure. This type of error results from flaws in the instrument, question wording, question order, interviewer error, timing, question response options, etc. This is perhaps the most common and most problematic collection of errors faced by the polling industry.

Non-response Error results from not being able to interview people who would be eligible to take the survey. Many households now have answering machines and caller ID that prevent easy contact; other people simply do not want to respond to calls sometimes because the endless stream of telemarketing appeals make them wary of answering. Non-response bias is the difference in responses of those people who complete the survey vs. those who refuse to for any reason. While the error itself cannot be calculated, response rates can be calculated and there are countless ways to do so. The American Association for Public Opinion Research (AAPOR web site) provides recommended procedures for calculating response rates along with helpful tools and related definitions to assist interested researchers.

Back

6. What happens when the final sample doesn't look like the general public--For example, what if three quarters of your respondents are over fifty?

Survey firms apply a technique called weighting to adjust the poll results to account for possible sample biases caused by specific groups of individuals not responding. The weighting uses known estimates of the total population provided by the Census to adjust the final results.

It's not uncommon to weight data by age, gender, education, race, etc. in order to achieve the correct demographic proportions.

Back

READING TABLES

1. How do I read the numbers in tables reported?

There are some tables that are straightforward. The Roper Center's iPOLL database offers the top-line results to survey questions--toplines are how the full aggregated sample answered the questions.

iPOLL example:
You might say that the public is evenly split on judging the integrity of pollsters, according to this November 2002 telephone conducted by Harris Interactive and obtained from the Roper Center at the University of Connecticut.

Crosstabulation tables can be more complicated. Crosstabs offer a look at how different groups within the sample answered the question. In other words, the table below can be summarized in this manner:

A New York Times poll in June 2000 found that among whites, 81% thought race relations in their community were "good", while 72% of black respondents found this to be the case. Conversely, 14% of whites and 22% of blacks identified their community race relations as "bad". Among those who identified with the "other" race category, 79% responded good and 18% bad to the question of race relations in their community. There were too few Asians in the sample to be able to statistically rely upon the percentages. These data were provided by the Roper Center at the University of Connecticut.

Source: New York Times Poll, Race Relations in America, June 2000
Data provided by the Roper Center at the University of Connecticut.

Back

LOGICAL LINKS

To learn more about the field of polling and the methodological issues we invite you to visit these web sites:
American Association for Public Opinion Research (www.aapor.org)
National Council on Public Polls (www.ncpp.org)

Back

A POLL ABOUT POLLS

Roper Center Data Analysis Tool

Early in 2001 the Henry J. Kaiser Family Foundation in collaboration with Public Perspective magazine conducted a telephone survey on the Role of Polls in Policymaking. To review the results of this survey and use the Roper Center's Data Analysis Tool click on this link:
Role of Polls in Policymaking Survey [January, 2001].

Back