Hong Kong Legislative Council Election Post-Mortem Analysis

For the Hong Kong Legislative Council election of 2008, there had been a series of public opinion polls conducted by the Hong Kong University Public Opinion Programme, the Institute of Asia-Pacific Studies at the Chinese University of Hong Kong, the Lingnan University Public Governance Programme and the Hong Kong Research Association in the weeks leading up the vote itself.  On election data, a number of organizations conducted exit polls.  The question here is whether the public opinion polls and exit polls predictive of the actual vote.

In Hong Kong, this is not just a simple research question or some esoteric academic debate over survey methodology and executive.  Instead, this is a politically controversial subject.  Are the public opinion polls (or, at least, the selective release of certain parts of the data) serving some special political agenda (such as persuading a certain candidate to withdraw from an apparently hopeless situation)?  Are the exit polls used to guide strategic voting blocs on election day?

Here is the table of the actual voting data along with the four pre-election public opinion poll average and the Hong Kong Research Association exit poll results. 

Public Opinion Polls

For the 2004 election, it was noted that the HKU POP numbers were systematically understating the DAB votes everywhere.  I have extracted these numbers (see Table):

• Hong Kong Island: Ma Lik (DAB): HKU 15.5%, actual 21.1%.  Relative shortfall of 27%.
• Kowloon East: Chan Kam-lam (DAB): HKU 13.3%, actual 18.8%.  Relative shortfall of 29%.
• Kowloon West: Tsang Yok-sing (DAB): HKU 19.5%, actual 27.1%.  Relative shortfall of 28%.
• New Territories East: Lau Kong-wah/Li Kwok-ying (DAB) : HKU 17.3%, actual 22.1%.  Relative shortfall of 22%.
• New Territories West: Tam Yiu-chung/Cheung Hok-ming (DAB): HKU 18.7%, actual 24.9%.  Relative shortfall of 25.0%.

For the 2008 election, the corresponding numbers are:

• Hong Kong Island: Tsang Yok-sing (DAB): HKU 17.3%, actual 19.3%.  Relative shortfall of 10%.
• Kowloon East: Chan Kam-lam (DAB): HKU 18.7%, actual 22.6%.  Relative shortfall of 17%.
• Kowloon West: Starry Lee Wai-king (DAB): HKU 15.8%, actual 18.9%.  Relative shortfall of 16%.
• New Territories East:: Lau Kong Wah (DAB): HKU 23.7%, actual 28.4%.  Relative shortfall of 17%.
• New Territories West: Tam Yiu Chung (DAB): HKU 23.5%, actual 23.1%.  Relative shortfall of 2%.

For this election, the pre-election four-poll averages compared against the actual election outcomes resulted in:

• Hong Kong Island: 100% correct
• Kowloon East: 100% correct
• Kowloon West: Correct for all except Leung Mei-fun (IND) because two other persons (Claudia Mo (CP) and Lau Chin-shek (IND)) polled higher than her.
• New Territories East:: Correct for all except James Tien (Liberal Party) when Wong Shing-chi (DP) did a lot better than the polls suggested.
• New Territories West: Correct for all except Selina Chow (Liberal Party) when Chan Wai-yip (LSD) did a lot better than the polls suggested.

Out of thirty contested seats in the geographical constituency, the pre-election polls got 27 out of 30 winners (90% success rate).

Exit Polls

Exit polls had been controversial already (Banning Exit Polls in Hong Kong) and it became even more controversial earlier in the week.

(The Standard)  It's all a matter of opinion.  Mary Ma.  September 4, 2008.

A boat with a hole bored in its hull is in imminent danger of sinking.

Pollster Robert Chung Ting-yiu attempted yesterday to repair the hole he has drilled - unintentionally - in the University of Hong Kong's public opinion program with his contentious plan to release exit poll data to a number of electronic media outlets while voting is still ongoing during Sunday's Legislative Council election.  Only on Tuesday, Chung vigorously defended the decision to start releasing updated information to the media 41/2 hours after the first vote is cast. But he backtracked yesterday, saying updated exit poll data will only be given to sponsors from 8pm.

Although whether Chung's center will provide refunds to the sponsors is more an issue between him and the stations, the storm he has whipped up could prove to be a curse for the opinion program he so painstakingly set up in 1991. Prior to the pan-democrats' unanimous appeal for voters not to answer any exit poll questions, online discussion forums are already filled with calls urging electors to give the wrong answer to confuse the findings. Both scenarios are surely the last thing a pollster wants, since they would inevitably undermine the accuracy any trustworthy survey strives to achieve.

Should accuracy be in doubt, all polls will be stripped of any reference value and become worthless. This is the danger now facing Chung and his colleagues at the university's research center. Chung is fully aware of this, which helps explain why he published a statement after the pan-democrats' press conference. Warning that a boycott or giving the wrong answer would endanger the long- term development of a civic society and the professional development of polling and research, he called on the politicians to reconsider their opposition.

However, with the public outcry having reached such a fever pitch, it is doubtful whether the pollster's concession to delay the release of data to the evening will be good enough to repair the damage, since it will be very difficult for one to change his or her mind once a view is formed. Have the people formed their views after so many days of hues and cries? If anyone is to be blamed, it will be Chung himself who has helped to push his own boat to the edge of the waterfall.

If there is anything else to be learned from this saga, it is that it exposes the leniency of the election law in dealing with anyone breaching the guidelines that prohibit pollsters and concerned organizations to publicize the results of exit polls before voting is completed, as this could unduly influence those who have not yet cast their ballots. According to the guidelines, the maximum penalty for violating the rules is either a reprimand or a censure in the form of a public statement. Not much of a deterrent, I'd say.

The exit poll data were not released to the public while the voting was still going on.  At 11:00pm, the Hong Kong Research Association published its data on its website (HKRA Exit Poll ).  The numbers are included in the table above on this page.

The glaring discrepancy is in the exit poll data for the DAB.  In each case, the HKRA exit poll numbers for the DAB party are higher than the public opinion poll average, and the actual voting is closer to the public opinion polls than the exit polls.

• Hong Kong Island: Opinion poll 18.2%, Exit poll 26.3%, Actual vote 19.3%.  Exit poll overstated by 36%.
• Kowloon East: Opinion poll 19.2%, Exit poll 29.0%, Actual vote 22.6%.  Exit poll overstated by 28%.
• Kowloon West: Opinion poll 16.7%, Exit poll 25.4%, Actual vote 18.9%.  Exit poll overstated by 34%.
• New Territories East: Opinion poll 24.5%, Exit poll 36.1%, Actual vote 28.4%.  Exit poll overstated by 27%.
• New Territories West: Opinion poll 24.3%, Exit poll 27.3%, Actual vote 23.1%.  Exit poll overstated by 18%.

This will be known hereafter as the "Just say DAB effect."  Right before the election, there was a controversy whether the exit polls were being used exclusively by the DAB and/or other pro-establishment forces to allocate campaign resources.  Since the government refused to take action, some citizens started a "Just say DAB" campaign (see @Facebook Tell the pollster: "I voted for DAB!") -- when asked by a pollster after voting, the voter is supposed to say: "I voted for the DAB!"  This  tactic is supposed to distort the poll results and ruin its purported utility.  Indeed the poll results have been distorted systematically.  But there is no evidence that the exit poll data figured in any final-two-hour surge of 'iron' votes from the leftists towards designated candidates.

In comparing its predicted winners against the actual results, the HKRA exit poll performed as follows:

• Hong Kong Island: Correct for all except Audrey Eu (CP) over Choy So-yuk (DAB)
• Kowloon East: 100% correct
• Kowloon West: 100% correct
• New Territories East: Correct for all except Emily Lau (The Frontier) over James Tien (Liberal Party)
• New Territories West: 100% correct

Out of the thirty contested seats in the geographical constituency, the HKRA exit poll got 28 out of 30 winners (93% success rate). 

In spite of the "Just say DAB effect," the only major mishap to the DAB might be with its number two candidate Choy So-yuk.  Did the DAB think that they were already secure with two seats from their overwhelming lead and therefore mistakenly allocated resources to Regina Ip's list?  Here are some known facts.  At 2pm, Choy So-yuk issued a personal "urgent rescue" appeal.  At 8pm, the DAB list leader Tsang Yok-sing proclaimed that the sixth and last seat was a contest between Choy So-yuk and Yeung Sum (Democratic Party).  This was regarded as a tactic to draw anti-DAB voters towards the apparently hopeless Yeung instead of the better placed Audrey Eu and Cyd Ho.  These actions do not seem to indicate that the DAB was actively transferring votes to Regina Ip.

Why did the HKRA exit poll still work well in spite of the clear "Just say DAB effect"?  Because the DAB effect resulted only in transferring votes among the top vote-getters only and not outside the circle almost all of the time.  For example, in Kowloon East, there were six candidates contesting for four seats.  Four candidates (including the DAB one) had a big lead over the other two candidates.  The "Just say DAB effect" made the DAB number larger, but it does not change the fact that those four would win.  Similarly in Kowloon West, New Territories East and New Territories West, the DAB lists were the top vote-getters already and the "Just say DAB effect" was not big enough to suggest that they could get an extra seat.  The only exception was in Hong Kong Island, where the top vote-getter was the Civic Party list, and the "Just say DAB effect" was enough to push the DAB past them in the exit poll data.

This has the same characteristic as the HKRA exit poll data in terms of overstating the likelihood for the DAB (such as the third DAB candidates being competitive in New Territories East and West).  In Kowloon East, Fred Li ended up with 20.4% of the votes compared to 12.4% for Andrew To, and this should not have been close.  In all remaining districts, the candidates with 'extremely high chances' won and the remaining seats come from candidates with 'equal chances.'

The Hong Kong Research Association also provide 'qualitative' data (such as "extremely high chances," "relatively high chances," "relatively lower chances", "extremely little chances").

Hong Kong Island:
Extremely high chances: Cyd Ho (CA), Kam Nai-wai (DP), Jasper Tsang (DAB), Choy So-yuk (DAB), Tanya Chan (CP), Regina Ip (IND)
Relatively high chances: Lo Wing-lok (IND)
Relatively low chances: Tsang Kin-sing (LSD), Louis Shih (IND)
Relatively little chance: Audrey Eu (CP)

Kowloon East:
Extremely high chances: Chan Kam Lam (DAB), Wong Kwok Kin (FTU), Alan Leong (CP), Li Wah-ming (DP)
Relatively low chances: Wu Chi-ming (DP), Andrew To (LSD)

Kowloon West:
Extremely high chances: Raymond Wong (LSD), James To (DP), Frederick Fung (ADPL), Starry Lee (DAB), Leung Mei-fun (IND)
All others have 'relatively small or little chances.'

New Territories East:
Extremely high chances: Wong Shing Chi (DP), Leung Kwok Hung (LSD), Andrew Cheng Kar-foo (DP), Lau Kwong-wah (DAB), Chan Hak-kan (DAB)
Relatively high chances: James Tien (LP), Emily Lau (The Frontier), Ronny Tong (CP
Relatively low chances: Scarlett Pong (IND)

New Territories West:
Extremely high chances: Lee Cheuk-yan (CTU), Tam Yiu-chung (DAB), Cheung Hok-ming (DAB), Albert Ho (DP), Wong Kwok-hing (FTU), Lee Wing-tat (DP), Leung Yiu-chung (Neighbourhood and Worker's Service Centre)
Relatively high chances: Cheung Chiu-hung (CP), Chan Wai-yip (LSD), Selina Chow (LP)

These numbers are not that much different from the HKU POP data (except for the obvious case of Audrey Eu (CP) on Hong Kong Island).

Conclusions

Question 1: Were the pre-election public opinion polls useful?

It depends on how you want to use them.  Without the public opinion polls, you would have no idea who was leading and who was lagging.  You would have no idea about how things changed as a result of a turning point.  An example was the case of Raymond Wong (League of Social Democrats).  He began the campaign indifferently in the middle of the pack.  Then came the televised debates from which his segments went viral on the Internet.

Raymond Wong's numbers in the IAPS-CUHK tracking polls over time were:

    August 6-12: 6.6% (6th place)
    August 14-20: 13.2% (4th place)
    August 21-27: 12.9% (3rd place)
    August 28-September 3: 16.1% (2nd place)

And he finished second place in the actual election.

Were the public opinion polls useful in predicting the election outcomes? 

The above data showed that the pre-election four-poll average predicted 27 out of 30 winners (90% success rate).  There are some obvious explanations about the failures.  The known facts are (1) 90% of survey respondents said that they will definitely vote or are likely to vote and less than 10% said that they definitely won't vote; (2) the actual voter turnout was only 45.2%; (3) the no-show's are likely less committed to specific candidates.  The public opinion polls did not catch on that the Liberal Party candidates had soft support.  Their supporters may be willing to state their preferences over the telephone but they did not actually show up to vote.  Thus James Tien and Selina Chow flamed out.  The third case of the 'independent' Leung Mei-fun is less obvious.  We might guess that her voters were disproportionately "undecided/don't know/refuse to say" in the public opinion polls, but they showed up to choose a high-profile non-party-affiliated candidate.

Q2: Was the HKRA exit poll useful?

Here, I don't want to engage in any speculation about its possible role in the strategic decisions of insiders who had access to the data during the day.  I don't know.

Speaking as a citizen, there was only one time window (11pm when the data was first published and before the election results were announced at some time after 4am).  As it turns out, the HKRA exit poll predicted 28 out of 30 winners (93% success rate).  That is a respectable record.  Compared to the pre-election public opinion polls, the HKRA exit poll imparted the new information that Leung Mei-fun (IND) was a winner and Selina Chow was ousted.

The Apple Daily story on the "Just say DAB effect" used the headline: "Voters devastated exit polls."  If you need a precise estimate of the number of votes that a candidate is getting, you should wait for the actual election results.  If you want an overview on who the likely winners are, you will find the exit polls useful and interesting (with new information not present in the pre-election public opinions polls).

But you say: The exit polls are statistically biased due to respondent non-cooperation!  Well, you run a telephone survey and 40% of the contacted persons refuse to participate.  There is a potential survey bias because the non-responders may have different behaviors and attitudes than the responders.  But that does not mean that the survey is useless.  With experience in data usage over time (for example, comparing enough public opinion polls versus actual election results), you will know how to interpret the data correctly.