Saturday, June 20, 2009

Do public prediction markets really fail?

A few days ago I came across this article explaining why public prediction markets fail. The article gives an example where three different PMs failed to pick a winner in American Idol (Betfair) or Britain's got talent (Hubdub and Intrade). While it was definitely an interesting read, I feel that the author didn't take some things into account.
First, the success of PMs depends on the notion of participants playing risk neutral strategies (see, for example, the Manski 2005 paper) -- when people play with fake money, this may well not hold (they will tend to risk more than usual because they have nothing to lose).
Also, PMs are said to be more accurate than other ways of aggregating opinions such as polls, or exert opinions. It would be nice if the author had compared these predictions with some predictions made by opinion polls or experts and shown how the predictions differ.
Then there's the thing with data from Hubdub. As noted in the article, Hubdub's market concerned with Britain's got talent failed to predict the true outcome, giving Susan Boyle 78% chance of winning. What was not taken into account is that there were in fact more markets on Hubdub concerned with Britain's got talent. The one used in the article can be found here. However, here's another market that accurately predicted that "Susan Boyle OR Diversity" will win (Diversity won). Note the following: the market that got Susan Boyle wrong had 25k$ of activity, whereas the one that got it right had twice as much activity. So, if you were to make any decisions based on PM predictions, you would probably go for the one with more activity and you would be right. I don't know if Intrade of Betfair had more markets for the same event.
Anyway, I agree in the bottom line that accuracy of PMs depends on how much information its participants have, and that, ultimately, they will fail some of the time. However, we should be careful about making statements like "public prediction markets fail", especially since there are so many examples when they don't (this might be a topic for another post). And even when they do fail, it's important to understand why.

No comments: