The Global Competitiveness Report
There was an uproar in Hong Kong during the week:
(The Standard) Business ranking falls on political concerns. Vanson Soo. September 29, 2005.
Hong Kong has become less competitive, with political concerns helping to drag it down seven places to rank 28 in an annual worldwide table, according to the latest Global Competitiveness Report drawn up by the World Economic Forum.
Acting Financial Secretary John Tsang said the report's authors might not have a clear understanding of Hong Kong. "They may have miscalculated the points. We will be talking with these people in the near future to bring about a better understanding of where we are," he said. "They may not have a broad and macro understanding of what Hong Kong is all about. You can see the investments in Hong Kong have continued to improve. There is no indication of any abatement." He said the government will clarify with the forum the indicators used and the calculation method.
The findings gathered from the 26th report of its kind are drawn from a combination of hard data and a survey of 11,000 business leaders in the countries covered. The survey is designed to capture a broad range of factors relating to the prevailing business environment in each economy.
I am going to address the results from a technical viewpoint. My analysis is limited to the information available on the World Economic Forum website, where a number of pages from the more than 600 pages in the full report are posted. My qualifications are that I conduct C-suite surveys for commercial purposes. The C-suite refers to Fortune 500 employees with the word "Chief" in their job titles, such as Chief Executive Officer (CEO), Chief Financial Officer (CFO), (Chief Information Officer), Chief Technology Officer (CTO), but we usually also allow exceptions such as President and Executive Vice-President (but Vice-Presidents don't make the cut).
First, how is the Growth Competitive Index calculated? It is described in this two-page pdf file. Apparently, the index contains both survey data as well as some hard data (such as government surplus/deficit, national savings rate, cellular mobile subscribers per 100 inhabitants, etc). I will deal with the survey questions only. Here are some examples:
3.02. Are companies in your country unable/aggressive in absorbing new technology?
3.06. What is the extent of business collaboration in R&D with local universities?
3.12. Is there sufficient competition among ISPs in your country to ensure high quality, infrequent interruptions and low prices?
6.01. Is the judiciary in your country independent from political influences of members of government, citizens or firms?
6.08. Is your government neutral among bidders when deciding among public contracts?
6.16. Does organized crime impose significant costs on business?
6.20. How commonly are bribes paid when getting connected with public utilities?
6.21. How commonly are bribes paid in connection with annual tax payments?
I could get into a philosophical debate about the meaning of these questions. For example, for 6.21, an American might answer: "People don't commonly pay bribes to Internal Revenue Service agents. However, the super-rich people pay huge fees to their financial advisors and accountants in order to find loopholes to avoid paying any tax." So perhaps this is a payoff in spirit. As another example, for 6.16, a New York City resident might wonder: "The City of New York imposes an 8.25% sales tax on all sales within the city boundaries. This is perfectly legal, but I consider it criminal extortion." Anyway, I won't waste your time on those types of arguments.
Here is the technical description of the survey in the WEF press release.
This year nearly 11,000 business leaders were polled in a record 117 economies worldwide.
So one of the first things that critics said was that this was a large sample (and not just asking the opinion of 100 people), and that is why government officials must accept the results and then look within themselves for what has gone wrong. Here is Li Yi in Apple Daily): "Yet WEF interviewed almost 11,000 business leaders before coming to the conclusion. Could all these 11,000 business leaders be mistaken? If so, how do these 11,000 people lead their enterprises?" (然 而 WEF 是 訪 問 了 全 球 近 一 萬 一 千 名 商 界 領 袖 而 作 出 的 結 論 。 莫 非 全 球 的 這 些 商 界領 袖 都 看 走 眼 了 嗎 ？ 若 如 此 ， 這 一 萬 一 千 多 人 又 如 何 領 導 他 們 的 企 業 ？)
Hmm ... in truth, that is a completely wrong assertion, because the sample size is actually less than 100. How can that be? Wasn't the cited magic number 11,000? I ask you to go back and look at the survey questions again. Each and every one of them refers to "your country." After all, why would you ask Bill Gates about whether bribes are commonly paid to get connected with public utilities in Azerbaijan?
There were about 11,000 respondents in 117 countries. On the average, they interviewed 11,000 / 117 = 94 persons per country. Each respondent was asked about his/her country only. The results for any country is therefore based upon 94 people on the average. Therefore, the first point is that this is a small sample study. If I ran a public opinion poll in Hong Kong with a sample of 94, I'd be laughed out of the place.
Next, we come to the question of the sampling frame and the response rate. In the United States, I draw my C-suite samples by going to Dun and Bradstreet and I ask for the names and contact information for all those who qualify by job title from the Fortune 500 database. Most of these companies are publicly traded and must therefore provide public records on their officers. Generally, I know who these people are and what they do, so I will send only a very simple one-page questionnaire (usually about the television channels that they watch frequently: CNBC, CNN, Fox News, MSNBC, etc and the newspapers that they read) via courier service. My incentive may be something like US$100 sent to a charity of their choice. The typical response rate is between 5% to 10%. And I know that people like Bill Gates do not respond. Therefore, the second point is that the response rate is poor. If I ran a public opinion poll in Hong Kong with a response rate of 5%, I'd be laughed out of the place. The respondents may or may not have the same opinions as the non-responders, which are by definition unknowable.
But that would be for the United States where there is a Dun & Bradstreet database of corporate officers in publicly traded companies. Now, I read down the list of countries. Number 23 is Chile. That is a country in which I once tried to launch a business readership survey. There is no sampling frame to work with, so my project was dead on arrival. Ditto for Argentina and Peru. Numbers 55 and 57 are Mexico and Colombia. A directory of names and contact information for top business leaders? Just what the guerrillas and professional kidnappers need to know ... And how do you find 94 business leaders in Azerbaijan, Kazakhstan, Kyrgyz Republic, Tajikistan and other emerging countries? Therefore, the third point is that the sampling universe is highly uncertain in some countries.
The three points came to me just from the limited information that I have. Do not be impressed by the 11,000 'sample size.' There is a lot less than it seems.
[Digression: You may ask just why I was running C-suite studies when they are fraught with problems of small sample sizes and poor response rates. I do not run C-suite studies for decision-making purposes -- no one could take the data seriously. Instead, that information is used for advertising, as in XXXX is the preferred television channel among the C-suite, according to a survey conducted by marketing research company XYZ. In that sense, it is no more or less credible that a Coca Cola-Pepsi Cola taste test results taken at a shopping mall somewhere. The reason why this is done at all is that XXXX knows that it will win hands down (whereas it is a big-time loser in a general population study) and that it needs the reputation of research company XYZ to 'certify' the obvious.]