New York Times on the Probability of Rare Events

The New York Times has published a short article about the commonness of rare events.  The article discusses the fraudulent scheme of a psychic sending out emails predicting the outcome of a ball game.  After 8 successful ballgame predictions, the psychic asks for $10 before providing you the next prediction for the ninth game.  The article states that the psychic is guessing, so how did he do it? 

This is a version  of a well known scheme that works like this.  The psychic sends out batches of 256 emails, 50% of them predicting the first team will win, 50% predicting the second will win for each ball game.  Half of the recipients receive a correct answer and half receive the wrong answer.  The next week, the psychic sends an email only to those that received the correct answer in the first week.  Again half of these are told the first team will win, half are told the second team will win.  This continues for eight weeks.  At the end of the eight weeks, one recipient will have received 8 correct predictions and a request for $10 for the next prediction.If the psychic starts with 25,600, emails, after 8 ballgames, there will be 100 recipients who have received perfect predictions.

The psychic’s success rate can be dramatically improved if he weights his predictions to the favorites for each game.  For example, suppose the average game has one team with betting odds of winning of 1.5:1, and that these odds are a reasonable predictor of the outcome.  This means that the psychic only has to send out 59.5 emails (instead of 256) on average to produce one recipient with 8 good predictions.  This is calculated as 59.5*(1.5/(1.5+1))^8 = 1.  If the psychic is choosing only one game from a weekly roster of games, he could choose even more lob-sided games and further improve the chance of producing a winning series of predictions.

A version of this occurs naturally in the financial markets, although with less of an intent to deceive.  Assume that there are 1,024 traders.  If each makes a key investment decision every 6 months, over a four year period, there will be roughly four that have a perfect record, assuming that each correct decision occurs with a probability of 50%.  These four traders are likely to be considered Wall Street geniuses.

Now real life is always more complicated than one might expect.  With traders, there is a strong likelihood that at least some of them are smarter than the rest and are more likely to make good decisions. Let’s assume that “Good” traders are 60% correct in their decisions, “Average” traders are 50% correct, and “Poor” traders are 40% correct.  Let’s further assume that the split of Good, Average and Poor traders is 10%, 80%, 10%.  The question that is most interesting is: if you come across a trader with 8 correct decisions, what is the probability that that trader comes from the “Good” category.  This is the key decision an investor must make when  looking to place his or her funds with an investment manager or trader (or a manager must make before paying out fat trader bonuses).

This can be solved in the following way:

Trader Strength	Probability of correct decision	Proportion of traders	Probability of 8 correct decisions	Percentage of decisions which are correct	Probability Good Decision came from the selected Class
Column	a	b	c	d	e
Good	60%	10%	1.7%	0.2%	34.5%
Average	50%	80%	0.4%	0.3%	64.2%
Poor	40%	10%	0.1%	0.0%	1.3%
Total		100%		0.5%	100.0%

Column c = a^8

Column d =b * c

Column e = d / (Total column d)

The model suggests that if you come across a trader with an astounding run of 8 correct decisions, you only have a 34.5% chance that that trader is “Good” and not “Average” or “Poor”.