Survey Research Stories

For lack of anything better to talk about, I am going to tell a few stories.  My original starting point is a post at Far Outliers: Is Latin America Turning Protestant?

Latin America! The Catholic hemisphere, the last best wine the Church had counted on to see herself through the twenty-first century--Latin America is turning in its jar to Protestantism. At the beginning of this century, there were fewer than two hundred thousand Protestants in all of Latin America. Today there are more than fifty million Protestants. The rate of conversion leads some demographers to predict Latin America will be Protestant before the end of the next century. Not only Protestant but evangelical.

...

Evangelicals are fundamentalists. They read scripture literally. 

Yes, but what has that got to with survey research?  Read on ...

In 1997, I was looking for a survey research company to conduct field work in Central America (including Costa Rica, El Salvador, Guatemala, Honduras, Nicaragua and Panama).  Traditionally, I have found it very difficult to find qualified and quality survey research companies in Latin America.  For example, in Paraguay, there were only three survey research companies in the country.  We used one for year one and we fired them; we used another one for year two and we fired them; we used the third for year three and we fired them; and then we went back with the first one in year four; and so on.

Sometimes, it is not the fault of the survey research companies themselves, for it is their workers (often underpaid) who are taking short cuts.  These workers are often sent to remote locations without supervision, and their work cannot be verified.  However, in 1997, I thought that I found the best possible solution.

When I asked the company owner about what, if any, quality control procedures were in place to guarantee that the workers were not cheating, the answer was this: "We hire evangelicals to become interviewers.  They always speak the truth.  When they come back from the field and we ask them what they did, they speak the truth.  They do not lie."

I was sold.

Now for the next story which comes from Taiwan.  First, I will have to describe very briefly how survey respondents are selected.  The details of the sample design for one of my Latin American studies can be found here: Los Medios y Mercados de Latinoamérica: Description of the Survey Sample.

If I am drawing a sample for an urban area such as San Pedro Sula, Managua, San Salvador, or Taipei City, I would have information on the various political/administrative areas (which may be known under various names such as counties, towns, municipalities, school districts, electoral districts, census tracts, census block groups, census blocks, dioceses, etc) together with the population sizes.  This would allow me to draw a random sample of locations (or field clusters).

A field cluster may be a street block, and the interviewer is given a street map of that field cluster.  A random starting point (e.g. third house west starting from the southeast corner) is assigned to each block to indicate where the interviewing should start contacting households.  In addition, the interviewer is given a specific path (as indicated by directional arrows drawn on the map) to follow in order to complete the assigned number of interviews.  There are further instructions on screening households and persons.

The above works well for urban areas for which such maps and population figures exist.  What if you have to go into a rural village in the middle of nowhere without any street maps?  What is the interviewer to do?

In the case of Latin America, if this remote village was selected into the sample by chance, we told the interviewer to take the local bus to go out there.  When they get off the bus at the village, they are supposed to look for the church (or the mayor's office if necessary but there is usually a church), and then proceed about 100 yards to the left  (or right) of the church and start from there.  This is about the best that we can do, and we do not want the interviewer to decide whom they would rather interview.  Since there is no supervisor present, it was important that the interview did what they were supposed to instead of whatever they felt was easier for them (like go to a bar and interview someone there).

The story from Taiwan is about how such a procedure was 'gamed.'  Here is the translation of a section from a dated Boxun article:

Recently, Liberty Times has been advertising in the newspaper that "one newspaper beats five newspapers" and "one newspaper beats three newspapers."

So the advertisements says that the number of readers of Liberty Times is 1.77 times that of United Daily, and 1.15 times that of the five newspapers in the United Daily News group; also, the number of readers of Liberty Times is 1.55 times that of China Times, and 1.28 times that of the three newspapers in the China Times group.  The source of the information is A.C. Nielsen's survey between October and December 2001 of persons between 12 to 60 years old in Taiwan.

These advertisements were questioned by many knowledgeable people.  According to what one informed sourced told the reporter, Liberty Times worked to ensure that this would happen.  The research company is an American-capital company working in Taiwan with well-known operating procedures.  In the urban areas, they usually interview only people living on the first floor.  In the rural areas, they ordinarily proceed to the village government office and then they interview the households approximately 200 meters to the left of the office.  This procedure was noted by Liberty Times and they began to provide free copies of Liberty Times to people living on the first floor in urban areas, and to households about two hundred meters to the left of the village government officers in rural areas.  Thus, they manipulated the survey result.

The answer is simple: the survey research company should be varying its procedures (like going right instead of left).  By the way, if this happened in the United States, A.C. Nielsen would be suing Liberty Times in court for damaging the integrity of their commercial product and the outcome of such a lawsuit would never be in doubt.